Unhandled Exceptions in light turn on and off service

[mathiasolofsson]

The problem

After updating to 2024.10.1 I started getting a lot of Failed to deliver message: <sl_Status.ZIGBEE_DELIVERY_FAILED: 3074> errors and my lights would not respond to commands. I noticed the Ikea E14 opal 400lm lightbulb closest to my skyconnect coordinator had dropped out, so I tried repairing it but it did not help. I downgraded to 2024.10.0 again, but after downgrading that E14 lightbulb will not repair, it gets to the "configuring" state when resetting it and entering pairing mode, but it will never complete the configuring state when searching for new zigbee device in HA. I still have the same 3074 errors on 2024.10.0 now and my Zigbee network is not working anymore after having been stable for many months. I tried removing the E14 bulb entity completely in hopes the zigbee mesh would disregard it and repair itself but it has not helped.

I was able to find the following in the logs:

`Logger: homeassistant.components.websocket_api.http.connection
Source: components/websocket_api/commands.py:245
integration: Home Assistant WebSocket API (documentation, issues)
First occurred: 07:11:17 (14 occurrences)
Last logged: 09:41:52

[140462754609856] Unexpected exception
[140461825007552] Unexpected exception
[140461449062016] Unexpected exception
[140461515628336] Unexpected exception
[140461931824608] Unexpected exception
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 245, in handle_call_service
response = await hass.services.async_call(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/core.py", line 2761, in async_call
response_data = await coro
^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/core.py", line 2804, in _execute_service
return await target(service_call)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/config/custom_components/adaptive_lighting/hass_utils.py", line 62, in service_func_proxy
await existing_service.job.target(call)
File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 996, in entity_service_call
single_response = await _handle_entity_call(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1068, in _handle_entity_call
result = await task
^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/light/init.py", line 626, in async_handle_light_on_service
await async_handle_light_off_service(light, call)
File "/usr/src/homeassistant/homeassistant/components/light/init.py", line 639, in async_handle_light_off_service
await light.async_turn_off(**filter_turn_off_params(light, params))
File "/usr/src/homeassistant/homeassistant/components/group/light.py", line 194, in async_turn_off
await self.hass.services.async_call(
File "/usr/src/homeassistant/homeassistant/core.py", line 2761, in async_call
response_data = await coro
^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/core.py", line 2804, in _execute_service
return await target(service_call)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1021, in entity_service_call
raise result from None
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1610, in async_request_call
return await coro
^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1068, in _handle_entity_call
result = await task
^^^^^^^^^^
File "/usr/src/homeassistant/homeassistant/components/light/init.py", line 639, in async_handle_light_off_service
await light.async_turn_off(**filter_turn_off_params(light, params))
File "/usr/src/homeassistant/homeassistant/components/zha/helpers.py", line 1288, in handler
raise HomeAssistantError(err) from err
homeassistant.exceptions.HomeAssistantError: Failed to send request: Failed to deliver message: <sl_Status.ZIGBEE_DELIVERY_FAILED: 3074>`

What version of Home Assistant Core has the issue?

2024.10.1

What was the last working version of Home Assistant Core?

2024.10.0

What type of installation are you running?

Home Assistant OS

Integration causing the issue

Home Assistant WebSocket API

Link to integration documentation on our website

No response

Diagnostics information

No response

Example YAML snippet

No response

Anything in the logs that might be useful for us?

Logger: homeassistant.components.websocket_api.http.connection
Source: components/websocket_api/commands.py:245
integration: Home Assistant WebSocket API ([documentation](https://www.home-assistant.io/integrations/websocket_api), [issues](https://github.com/home-assistant/core/issues?q=is%3Aissue+is%3Aopen+label%3A%22integration%3A+websocket_api%22))
First occurred: 07:11:17 (14 occurrences)
Last logged: 09:41:52

[140462754609856] Unexpected exception
[140461825007552] Unexpected exception
[140461449062016] Unexpected exception
[140461515628336] Unexpected exception
[140461931824608] Unexpected exception
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 245, in handle_call_service
    response = await hass.services.async_call(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2761, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2804, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/adaptive_lighting/hass_utils.py", line 62, in service_func_proxy
    await existing_service.job.target(call)
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 996, in entity_service_call
    single_response = await _handle_entity_call(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1068, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 626, in async_handle_light_on_service
    await async_handle_light_off_service(light, call)
  File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 639, in async_handle_light_off_service
    await light.async_turn_off(**filter_turn_off_params(light, params))
  File "/usr/src/homeassistant/homeassistant/components/group/light.py", line 194, in async_turn_off
    await self.hass.services.async_call(
  File "/usr/src/homeassistant/homeassistant/core.py", line 2761, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2804, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1021, in entity_service_call
    raise result from None
  File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 1610, in async_request_call
    return await coro
           ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 1068, in _handle_entity_call
    result = await task
             ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/components/light/__init__.py", line 639, in async_handle_light_off_service
    await light.async_turn_off(**filter_turn_off_params(light, params))
  File "/usr/src/homeassistant/homeassistant/components/zha/helpers.py", line 1288, in handler
    raise HomeAssistantError(err) from err
homeassistant.exceptions.HomeAssistantError: Failed to send request: Failed to deliver message: <sl_Status.ZIGBEE_DELIVERY_FAILED: 3074>

Additional information

No response

[home-assistant]

Hey there @home-assistant/core, mind taking a look at this issue as it has been labeled with an integration (websocket_api) you are listed as a code owner for? Thanks!

Code owner commands

Code owners of websocket_api can trigger bot actions by commenting:

• @home-assistant close Closes the issue.
• @home-assistant rename Awesome new title Renames the issue.
• @home-assistant reopen Reopen the issue.
• @home-assistant unassign websocket_api Removes the current integration label and assignees on the issue, add the integration domain after the command.
• @home-assistant add-label needs-more-information Add a label (needs-more-information, problem in dependency, problem in custom component) to the issue.
• @home-assistant remove-label needs-more-information Remove a label (needs-more-information, problem in dependency, problem in custom component) on the issue.

_{^{(message by CodeOwnersMention)}}

---

websocket_api documentation
websocket_api source
_{^{(message by IssueLinks)}}

[mathiasolofsson]

Also found this in the logs, not sure if it's related:

`Logger: homeassistant
Source: /usr/src/homeassistant/homeassistant/runner.py:147
First occurred: 08:52:16 (1 occurrences)
Last logged: 08:52:16

Error doing job: Task exception was never retrieved (None)
Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/zha/zigbee/cluster_handlers/init.py", line 67, in wrap_zigpy_exceptions
yield
File "/usr/local/lib/python3.12/site-packages/zha/zigbee/cluster_handlers/init.py", line 85, in wrapper
return await RETRYABLE_REQUEST_DECORATOR(func)(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/zigpy/util.py", line 136, in retry
return await func()
^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/zigpy/zcl/init.py", line 378, in request
return await self._endpoint.request(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/zigpy/endpoint.py", line 265, in request
return await self.device.request(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/site-packages/zigpy/device.py", line 339, in request
await send_request()
File "/usr/local/lib/python3.12/site-packages/zigpy/application.py", line 834, in request
await self.send_packet(
File "/usr/local/lib/python3.12/site-packages/bellows/zigbee/application.py", line 827, in send_packet
raise zigpy.exceptions.DeliveryError(
zigpy.exceptions.DeliveryError: Failed to deliver message: <sl_Status.ZIGBEE_DELIVERY_FAILED: 3074>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.12/site-packages/zha/application/gateway.py", line 607, in async_device_initialized
await self._async_device_rejoined(zha_device)
File "/usr/local/lib/python3.12/site-packages/zha/application/gateway.py", line 648, in _async_device_rejoined
await zha_device.async_configure()
File "/usr/local/lib/python3.12/site-packages/zha/zigbee/device.py", line 745, in async_configure
await self.identify_ch.trigger_effect(
File "/usr/local/lib/python3.12/site-packages/zha/zigbee/cluster_handlers/init.py", line 84, in wrapper
with wrap_zigpy_exceptions():
File "/usr/local/lib/python3.12/contextlib.py", line 158, in exit
self.gen.throw(value)
File "/usr/local/lib/python3.12/site-packages/zha/zigbee/cluster_handlers/init.py", line 76, in wrap_zigpy_exceptions
raise ZHAException(message) from exc
zha.exceptions.ZHAException: Failed to send request: Failed to deliver message: <sl_Status.ZIGBEE_DELIVERY_FAILED: 3074>`

[mathiasolofsson]

Shut off the main circuit breaker to the whole house for a minute, and then on again to make all devices unpowered for a while. After about 1hr after switching on the circuit breaker the zigbee network now works again and seems stable so mustve been a conflict somewhere in the zigbee network or a device that had locked up.