Weird error - EZSP_BACKUP

[bartkummel]

Hi,

I tried to create a backup of my EZSP stick, but I got an error. Here's a snippet from my home-assistant.log:

2022-01-19 17:17:43 DEBUG (MainThread) [custom_components.zha_toolkit.utils] Parameters '{'command': 'ezsp_backup'}'
2022-01-19 17:17:43 DEBUG (MainThread) [custom_components.zha_toolkit] module is <module 'custom_components.zha_toolkit' from '/config/custom_components/zha_toolkit/__init__.py'>
2022-01-19 17:17:43 DEBUG (MainThread) [custom_components.zha_toolkit] event_data {'ieee_org': None, 'ieee': 'None', 'command': 'ezsp_backup', 'start_time': '2022-01-19T16:17:43.731062+00:00', 'errors': ['AttributeError("\'ControllerApplication\' object has no attribute \'_znp\'")'], 'params': {'cmd_id': None, 'endpoint_id': None, 'cluster_id': None, 'attr_id': None, 'attr_type': None, 'attr_val': None, 'min_interval': None, 'max_interval': None, 'reportable_change': None, 'dir': None, 'manf': None, 'tries': 1, 'expect_reply': True, 'args': [], 'state_id': None, 'state_attr': None, 'allow_create': False, 'event_success': None, 'event_fail': None, 'event_done': None, 'read_before_write': True, 'read_after_write': True, 'write_if_equal': False}, 'success': False}
2022-01-19 17:17:43 ERROR (MainThread) [homeassistant.helpers.script.websocket_api_script] websocket_api script: Error executing script. Unexpected error for call_service at pos 1: 'ControllerApplication' object has no attribute '_znp'

Apart from the fact that the error is not very descriptive for an end user, I also suspect that somethings going wrong. I'm calling the EZSP backup procedure, so, I don't expect something with znp in it. Can this be a bug? If not: what can I try to fix it?

[mdeweerd]

First issue, first pun of my name ;-) . Don't worry, nobody notices.

I have no ezsp so I made an attempt at the implementation, so yes I expect it needs adjustments.

I probably located that reason, you can update and test again.
There's no need to restart HA.

Edit : The title says "znp_backup: Backup ZNP network data" where ZNP suggests the dongle type. But this can always be repeated and clarified what that means.

[bartkummel]

Okay, that looks like an improvement. The log now says:

2022-01-19 20:45:15 INFO (MainThread) [custom_components.zha_toolkit] Running ZHA Toolkit service: <ServiceCall zha_toolkit.execute (c:fc8892a8a5a560d84bbc388d6b37e864): command=ezsp_backup>
2022-01-19 20:45:15 DEBUG (MainThread) [custom_components.zha_toolkit.utils] Parameters '{'command': 'ezsp_backup'}'
2022-01-19 20:45:15 DEBUG (MainThread) [custom_components.zha_toolkit] module is <module 'custom_components.zha_toolkit' from '/config/custom_components/zha_toolkit/__init__.py'>
2022-01-19 20:45:15 DEBUG (MainThread) [custom_components.zha_toolkit.ezsp] Network params: EmberNetworkParameters(extendedPanId=a0:7e:d0:9d:89:0c:c8:19, panId=0xdee3, radioTxPower=255, radioChannel=11, joinMethod=<EmberJoinMethod.USE_MAC_ASSOCIATION: 0>, nwkManagerId=0x0000, nwkUpdateId=0, channels=<Channels.ALL_CHANNELS: 134215680>)
2022-01-19 20:45:15 DEBUG (MainThread) [custom_components.zha_toolkit.ezsp] tc_link_key key: EmberKeyStruct(bitmask=<EmberKeyStructBitmask.KEY_IS_AUTHORIZED|KEY_HAS_PARTNER_EUI64|KEY_HAS_OUTGOING_FRAME_COUNTER: 26>, type=<EmberKeyType.TRUST_CENTER_LINK_KEY: 1>, key=[90, 105, 103, 66, 101, 101, 65, 108, 108, 105, 97, 110, 99, 101, 48, 57], outgoingFrameCounter=57344, incomingFrameCounter=0, sequenceNumber=0, partnerEUI64=00:0d:6f:00:0c:86:91:ba)
2022-01-19 20:45:15 DEBUG (MainThread) [custom_components.zha_toolkit.ezsp] network_key key: EmberKeyStruct(bitmask=<EmberKeyStructBitmask.KEY_HAS_OUTGOING_FRAME_COUNTER|KEY_HAS_SEQUENCE_NUMBER: 3>, type=<EmberKeyType.CURRENT_NETWORK_KEY: 3>, key=[20, 198, 161, 199, 203, 224, 195, 24, 142, 124, 54, 48, 220, 173, 165, 150], outgoingFrameCounter=3827760, incomingFrameCounter=0, sequenceNumber=0, partnerEUI64=00:00:00:00:00:00:00:00)
2022-01-19 20:45:15 DEBUG (MainThread) [custom_components.zha_toolkit] event_data {'ieee_org': None, 'ieee': 'None', 'command': 'ezsp_backup', 'start_time': '2022-01-19T19:45:15.176475+00:00', 'errors': [], 'params': {'cmd_id': None, 'endpoint_id': None, 'cluster_id': None, 'attr_id': None, 'attr_type': None, 'attr_val': None, 'min_interval': None, 'max_interval': None, 'reportable_change': None, 'dir': None, 'manf': None, 'tries': 1, 'expect_reply': True, 'args': [], 'state_id': None, 'state_attr': None, 'allow_create': False, 'event_success': None, 'event_fail': None, 'event_done': None, 'read_before_write': True, 'read_after_write': True, 'write_if_equal': False}, 'success': True}

And I have a backup file. However, I suspect that it doesn't contain everything. Here's a screenshot of the zigbee network visualisation:

And here's my backup file:

{
    "node_type": 1,
    "node_id": 0,
    "node_ieee": "00:0d:6f:00:0c:86:91:ba",
    "panId": 57059,
    "extendedPanId": "a0:7e:d0:9d:89:0c:c8:19",
    "radioChannel": 11,
    "radioTxPower": 255,
    "nwkUpdateId": 0,
    "channels": 134215680,
    "tc_link_key": {
        "bitmask": 26,
        "type": 1,
        "key": [
            90,
            105,
            103,
            66,
            101,
            101,
            65,
            108,
            108,
            105,
            97,
            110,
            99,
            101,
            48,
            57
        ],
        "outgoingFrameCounter": 57344,
        "incomingFrameCounter": 0,
        "sequenceNumber": 0,
        "partnerEUI64": "00:0d:6f:00:0c:86:91:ba"
    },
    "network_key": {
        "bitmask": 3,
        "type": 3,
        "key": [
            20,
            198,
            161,
            199,
            203,
            224,
            195,
            24,
            142,
            124,
            54,
            48,
            220,
            173,
            165,
            150
        ],
        "outgoingFrameCounter": 3827760,
        "incomingFrameCounter": 0,
        "sequenceNumber": 0,
        "partnerEUI64": "00:00:00:00:00:00:00:00"
    },
    "key_table": []
}

I think the backup should have more data?

[mdeweerd]

It's encouraging but the output does not look like the "open-coordinator-backup" format.

It's encouraging because you were able to make the backup live, without having to specify the port or doing anything special.

I dupped the code from https://github.com/zigpy/bellows/blob/dev/bellows/cli/backup.py#L85 and I now looked at the video mentioned here where we can notice the following output:

So I wonder: is ZHA in zigpy not containing this code or should I use something else as a reference?

The backup file should at least contain some devices, but only those that are known/managed by the coordinator - there is nothing in this file. And it should (eventually) record this in the open-coordinator-backup format.

So maybe somebody could find out where the good reference code is.

[mdeweerd]

I backed up a bit in the video and apparently it requires another version of bellows to export it this way:

So I can't do that in the custom_component, but at least there is a start.

[bartkummel]

Strange, it actually needs an older version of bellows? Otherwise I'd say we should wait until the bellows version of ZHA is updated, but now I'm not sure what to do. I guess I have to go for the manual way, as demonstrated in the video...

[Hedda]

I tried to create a backup of my EZSP stick,

Exactly which Zigbee Coordinator adapter brand + model (including exact chip it uses) and which exact firmware does it have?

Silicon Labs (a.k.a. Silabs) have a catalogue of Zigbee chips going back 20-years and some still ship very old firmware today.

For reference, could you try to stop the ZHA integration and test performing a backup to file using bellows CLI directly? See:

https://github.com/zigpy/zigpy/wiki/Coordinator-Backup-and-Migration

https://github.com/zigpy/bellows/blob/dev/README.md#nvram-backup-and-restore

https://community.home-assistant.io/t/backup-your-zha-husbzb-1-stick-and-even-seamlessly-migrate-to-a-new-stick-without-re-pairing/229044/

https://www.youtube.com/watch?v=ql58ZfXso1M&ab_channel=EverythingSmartHome

https://gist.github.com/EverythingSmartHome/990f2436608fd347cd20cf0009b5cef4

As mentioned, the goal is for zha-toolkit to produce a backup files compatible with Open ZigBee Coordinator Backup Format:

https://github.com/zigpy/open-coordinator-backup

Please see this sample file as an example how a such standard backup file should look like:

https://github.com/zigpy/open-coordinator-backup/blob/main/samples/z2m-sample-1.json

[mdeweerd]

I think it's a development version. I did dig enough to find the code that corresponds to it - possibly it's all in the CLI part of the code and worst case that could be copied over to zha-toolkit until it is release in HA.

[mdeweerd]

@Hedda It's not a matter of stopping ZHA, but having the code generating the open format - that's apparently the case for the ZNP tools currently in HA, but not for the bellows tools.

I haven't use all my means to find the bellows code in which 'zigpy/open-coordinator-backup' appears as we have it in the next lines for zigpy-znp:
https://github.com/zigpy/zigpy-znp/blob/06af078ab82aa8a3eec04128c2ee8b44bfea5fd6/zigpy_znp/tools/common.py#L24
https://github.com/zigpy/zigpy-znp/blob/48438ca84917c4f6f99e1f8eed2be200d7d34683/zigpy_znp/tools/network_backup.py#L58

In zigpy-znp the CLI code seems to suffice on its own.

[mdeweerd]

git grep  open-coordinator $(git rev-list --all)

did not find anything in the bellows repository, but found several occurrences in the zigpy-znp repository.

[mdeweerd]

The code is in puddly's fork: https://github.com/puddly/bellows/blob/puddly/open-coordinator-backup/bellows/cli/backup.py .

I update zha-toolkit with that code - it's likely compatible with HA's version. Needs to be tested.

[bartkummel]

Let me know when it's ready, and I will happily run a test backup with it.

[mdeweerd]

You can try it - I've updated the code.

[mdeweerd]

Apparently a merge messed up services.yaml - hopefully that did not affect you.

[Hedda]

In zigpy-znp the CLI code seems to suffice on its own.

@puddly Do you think a better solution here would be to move to zha-toolkit using zigpy-cli with this PR? -> zigpy/zigpy-cli#2

That PR, of course, is dependent on all radio libraries implementing that new radio api for zigpy:

This change introduces a network backup/restore utility for any adapter implementing the new radio API:

• zigpy: New radio API zigpy#848
• zigpy-znp: Implement new zigpy radio API zigpy-znp#97
• zigpy-deconz: Implement new radio API zigpy-deconz#172
• zigpy-xbee: Implement new zigpy radio API  zigpy-xbee#123
• zigpy-zigate
• bellows: Implement new zigpy network state bellows#445

Sample:

# Make a new venv, these packages are heavily modified
virtualenv -p 3 venv  # Python 3.7 or above is required
# Or python3 -m venv venv (if `virtualenv` isn't installed)
source venv/bin/activate

# Install modified versions of zigpy, zigpy-znp, zigpy-deconz, bellows, and the new zigpy-cli
pip install \
    git+https://github.com/puddly/zigpy.git@puddly/new-radio-settings-api \
    git+https://github.com/puddly/zigpy-cli.git@puddly/zigpy-radio-api \
    git+https://github.com/puddly/zigpy-znp.git@puddly/new-radio-settings-api \
    git+https://github.com/puddly/zigpy-deconz.git@puddly/new-radio-api \
    git+https://github.com/puddly/bellows.git@puddly/zigpy-radio-api

# Back up the old radio.      Note: this backup format will change in the future.
zigpy -vv radio deconz /dev/serial/by-id/... backup deconz.json
# zigpy -vv radio znp /dev/serial/by-id/... backup znp.json
# zigpy -vv radio ezsp /dev/serial/by-id/... backup ezsp.json

# Restore a backup to the new one
zigpy -vv radio znp /dev/serial/by-id/... restore deconz.json
# zigpy -vv radio ezsp /dev/serial/by-id/... restore deconz.json
# zigpy -vv radio deconz /dev/serial/by-id/... restore deconz.json


# Leave the virtualenv
deactivate

[bartkummel]

I'm now back at the original error:

2022-01-20 13:20:06 INFO (MainThread) [custom_components.zha_toolkit] Running ZHA Toolkit service: <ServiceCall zha_toolkit.execute (c:10225b2600db8573e86a55c595058765): command=ezsp_backup>
2022-01-20 13:20:06 DEBUG (MainThread) [custom_components.zha_toolkit.utils] Parameters '{'command': 'ezsp_backup'}'
2022-01-20 13:20:06 DEBUG (MainThread) [custom_components.zha_toolkit] module is <module 'custom_components.zha_toolkit' from '/config/custom_components/zha_toolkit/__init__.py'>
2022-01-20 13:20:06 DEBUG (MainThread) [custom_components.zha_toolkit] event_data {'ieee_org': None, 'ieee': 'None', 'command': 'ezsp_backup', 'start_time': '2022-01-20T12:20:06.550433+00:00', 'errors': ['AttributeError("\'ControllerApplication\' object has no attribute \'_znp\'")'], 'params': {'cmd_id': None, 'endpoint_id': None, 'cluster_id': None, 'attr_id': None, 'attr_type': None, 'attr_val': None, 'min_interval': None, 'max_interval': None, 'reportable_change': None, 'dir': None, 'manf': None, 'tries': 1, 'expect_reply': True, 'args': [], 'state_id': None, 'state_attr': None, 'allow_create': False, 'event_success': None, 'event_fail': None, 'event_done': None, 'read_before_write': True, 'read_after_write': True, 'write_if_equal': False}, 'success': False}
2022-01-20 13:20:06 ERROR (MainThread) [homeassistant.helpers.script.websocket_api_script] websocket_api script: Error executing script. Unexpected error for call_service at pos 1: 'ControllerApplication' object has no attribute '_znp'