Implement new radio API by puddly · Pull Request #2 · zigpy/zigpy-cli

puddly · 2021-11-14T17:12:50Z

This change introduces a network backup/restore utility for any adapter implementing the new radio API:

zigpy: New radio API zigpy#848
zigpy-znp: Implement new zigpy radio API zigpy-znp#97
zigpy-deconz: Implement new radio API zigpy-deconz#172
bellows: Implement new zigpy network state bellows#445

The following adapters also implement the new radio API but their firmware does not allow much network information to be read, making this form of migration impossible:

zigpy-zigate: Implement the new zigpy radio API zigpy-zigate#117
zigpy-xbee: Implement new zigpy radio API zigpy-xbee#123

Instructions

You'll need access to a Python environment. zigpy-znp has documentation that describes how to do this for every common platform.

# Make a new venv, these packages are heavily modified
virtualenv -p 3 venv # Python 3.7 or above is required# Or python3 -m venv venv (if `virtualenv` isn't installed)source venv/bin/activate
# Install HA OS packages (only if using HA OS)
apk add gcc musl-dev libffi-dev
# Install zigpy-cli
pip install git+https://github.com/zigpy/zigpy-cli.git
# Back up the old radio. Note: this backup format will change in the future.# Conbee/Raspbee
zigpy -vv radio deconz /dev/serial/by-id/... backup deconz.json
# CC2531, CC2652, etc.# zigpy -vv radio znp /dev/serial/by-id/... backup znp.json# HUSBZB-1, EFR32, etc.# zigpy -vv radio ezsp /dev/serial/by-id/... backup ezsp.json# Restore a backup to the new one
zigpy -vv radio znp /dev/serial/by-id/... restore deconz.json
# zigpy -vv radio ezsp /dev/serial/by-id/... restore deconz.json# zigpy -vv radio deconz /dev/serial/by-id/... restore deconz.json# Leave the virtualenv
deactivate

Once that's done, edit /config/.storage/core.config_entries (take a backup of the file first!) and change the coordinator's path to the new radio's /dev/serial/by-id/... path. If you're switching to a CC2652, make sure to delete the flow_control key and set the baudrate to 115200.

connoleg · 2022-01-12T17:56:29Z

Hi,
I managed to backup and restore from/to a ConBee II using zigpy. But when I try to restore to a Sonoff 3 plus I get the following exception:

zigpy -vv radio znp /dev/ttyUSB0 restore deconz.json
2022-01-12 17:51:04.390 parallels-Parallels-Virtual-Platform asyncio DEBUG Using selector: EpollSelector
Traceback (most recent call last):
File "/home/parallels/.local/bin/zigpy", line 8, in <module>
sys.exit(cli())
File "/usr/lib/python3/dist-packages/click/core.py", line 764, in __call__
return self.main(*args, **kwargs)
File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main
rv = self.invoke(ctx)
File "/usr/lib/python3/dist-packages/click/core.py", line 1137, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/lib/python3/dist-packages/click/core.py", line 1134, in invoke
Command.invoke(self, ctx)
File "/usr/lib/python3/dist-packages/click/core.py", line 956, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/lib/python3/dist-packages/click/core.py", line 555, in invoke
return callback(*args, **kwargs)
File "/usr/lib/python3/dist-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/home/parallels/.local/lib/python3.8/site-packages/zigpy_cli/cli.py", line 19, in inner
return loop.run_until_complete(cmd(*args, **kwargs))
File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/home/parallels/.local/lib/python3.8/site-packages/zigpy_cli/radio.py", line 51, in radio
app = app_cls(config)
TypeError: Can't instantiate abstract class ControllerApplication with abstract methods connect, disconnect, start_network

Any idea whats going wrong?
Thanks

puddly · 2022-01-12T17:57:52Z

Make sure you're using a virtualenv. I believe pip install won't overwrite existing packages when you install from Git.

Or pip uninstall zigpy-znp, re-install it from git, then uninstall it again and install the normal version from PyPI.

connoleg · 2022-01-12T18:07:25Z

Thanks puddly, bang on the money! virtualenv did the trick. I hadnt figured I needed to run that way as I was not on my HomeAssistant box but a standalone ubuntu

Cheers

toot · 2022-01-12T19:13:31Z

I would also like to switch from a ConBee II to a Sonoff 3 plus with your tool, but the setup fails ...

I get the following errors after (I tried to) installed wheel

Failed building wheel for zigpy
Failed building wheel for zigpy-cli
Failed building wheel for zigpy-deconz
Failed building wheel for bellows

But at the end:
Successfully installed bellows-0.30.0.dev0 zigpy-0.41.0.dev0 zigpy-cli-0.0.1 zigpy-deconz-0.14.0.dev0 zigpy-znp-0.6.4

Did the installation work anyway?

Also I'm still struggeling with the next step:

Using "lsusb" I get this info about my conbee II:
Bus 001 Device 003: ID 1cf1:0030 Dresden Elektronik

Where do I get the remaining information for the backup command?
zigpy -vv radio deconz /dev/serial/by-id/... backup deconz.json

Sorry, I'm pretty new to this and I'm still learning...
Thanks in advance :)

puddly · 2022-01-12T19:25:22Z

You can ignore the wheel errors, those are just saying that Python didn't cache .whl files so it'll have to rebuild the modules from scratch if you reinstall them.

Run ls /dev/serial/by-id/ to list the serial ports. You'll see which one corresponds to which device (you don't have to have them both plugged in at once).

toot · 2022-01-12T20:29:38Z

Thanks! :)

Now the backup command results in different errors, stating that some distributions are missing:

bellows 0.30.0.dev0 requires click-log>=0.2.1, which is not installed.
bellows 0.30.0.dev0 requires pure_pcapy3==1.0.1, which is not installed.
bellows 0.30.0.dev0 requires pyserial, which is not installed.
bellows 0.30.0.dev0 requires pyserial-asyncio, which is not installed.
bellows 0.30.0.dev0 requires voluptuous, which is not installed.

I already manually installed: scapy, click and coloredlogs, the current error is:
The 'voluptuous' distribution was not found and is required by zigpy

But I tried installing python-voluptuous with no effect.

I'm afraid I missed something at the beginning.

puddly · 2022-01-12T20:43:47Z

This may be my mistake, I just updated the pip install command in the example. Instead of running:

$ pip install --upgrade --no-deps --force-reinstall ...

Try just running:

$ pip install ...

The dependencies should then be correctly installed.

toot · 2022-01-13T18:06:41Z

Thanks Putty, now everything works as intended :)

I had a friend of mine help me undo the changes I made yesterday and after that your guide worked very well.

Some feedback from him (to make it more fool proof):

Adjusting the variable of the device from "..." and adding some additional input:

# Use TAB-autocomplete to get your device
YOUR_DEVICE=/dev/by-id/zigbeedevice
zigpy -vv radio deconz /dev/serial/by-id/$YOUR_DEVICE backup deconz.json

Using another name (less generic) for the venv, to prevent hiccups if there is already another "venv"

But thanks alot again for this really useful tool :)

puddly · 2022-01-28T21:25:56Z

@toot @connoleg would either of you mind testing out something with your Conbee? They need to be relatively running recent firmware (mid 2020 or newer) for this to work:

CONBEE=/dev/serial/by-id/...
zigpy -vv radio deconz $CONBEE backup conbee1.json
# Edit `conbee1.json` and change `network_key.frame_counter` to something bigger by adding a `0` to the end
zigpy -vv radio deconz $CONBEE restore conbee1.json
# Unplug the Conbee and plug it back in (very important!)
zigpy -vv radio deconz $CONBEE backup conbee2.json
# Compare the two files
diff conbee1.json conbee2.json

The contents of conbee1.json and conbee2.json should be nearly identical and the frame counter value should have changed.

stiang · 2022-01-29T11:12:14Z

I’m trying to migrate from a Raspberry Pi with RaspBee II to a proper server with the ConBee II USB dongle instead. I believe I have followed all the proper steps from the top of this issue, but getting this error when trying to backup the RaspBee:

(venv) bash-5.1# zigpy -vv radio deconz /dev/serial/by-id/usb-0658_0200-if00 backup deconz.json
2022-01-29 12:04:58.068 homeassistant asyncio DEBUG Using selector: EpollSelector
2022-01-29 12:04:58.242 homeassistant zigpy.appdb DEBUG SQLite version for <module 'sqlite3' from '/usr/local/lib/python3.9/sqlite3/__init__.py'>: 3.34.1
2022-01-29 12:04:58.273 homeassistant zigpy_deconz.uart DEBUG Connection made
2022-01-29 12:04:58.274 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.protocol_version: 34>, b'')
2022-01-29 12:04:58.275 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a02000800010022
2022-01-29 12:05:00.079 homeassistant zigpy_deconz.api WARNING No response to 'Command.read_parameter' command with seq id '0x02'
Traceback (most recent call last):
File "/usr/local/lib/python3.9/asyncio/tasks.py", line 492, in wait_for
fut.result()
asyncio.exceptions.CancelledError
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/config/venv/bin/zigpy", line 8, in <module>
sys.exit(cli())
File "/config/venv/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
return self.main(*args, **kwargs)
File "/config/venv/lib/python3.9/site-packages/click/core.py", line 1053, in main
rv = self.invoke(ctx)
File "/config/venv/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/config/venv/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/config/venv/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/config/venv/lib/python3.9/site-packages/click/core.py", line 754, in invoke
return __callback(*args, **kwargs)
File "/config/venv/lib/python3.9/site-packages/click/decorators.py", line 38, in new_func
return f(get_current_context().obj, *args, **kwargs)
File "/config/venv/lib/python3.9/site-packages/zigpy_cli/cli.py", line 19, in inner
return loop.run_until_complete(cmd(*args, **kwargs))
File "/usr/local/lib/python3.9/asyncio/base_events.py", line 642, in run_until_complete
return future.result()
File "/config/venv/lib/python3.9/site-packages/zigpy_cli/radio.py", line 90, in backup
await app.connect()
File "/config/venv/lib/python3.9/site-packages/zigpy_deconz/zigbee/application.py", line 71, in connect
self.version = await api.version()
File "/config/venv/lib/python3.9/site-packages/zigpy_deconz/api.py", line 470, in version
(self._proto_ver,) = await self[NetworkParameter.protocol_version]
File "/config/venv/lib/python3.9/site-packages/zigpy_deconz/api.py", line 435, in read_parameter
r = await self._command(Command.read_parameter, 1 + len(data), param, data)
File "/config/venv/lib/python3.9/site-packages/zigpy_deconz/api.py", line 320, in _command
return await asyncio.wait_for(fut, timeout=COMMAND_TIMEOUT)
File "/usr/local/lib/python3.9/asyncio/tasks.py", line 494, in wait_for
raise exceptions.TimeoutError() from exc
asyncio.exceptions.TimeoutError

Am I doing something wrong? I’m not 100% sure that the by-id ID is the correct one, but it’s the only entry in that folder and I get the same result when I use /dev/ttyACM0, which I know is the RaspBee.

puddly · 2022-01-29T11:36:29Z

0658:0200 looks like it's for an Aeotec Z-Wave stick. Try using /dev/ttyAMA0.

stiang · 2022-01-29T15:40:10Z

Oof, so much for "which I know is the RaspBee"... Thanks, you are right, that looks like the correct device. And I get what appears to be promising output with no errors:

(venv) bash-5.1# zigpy -vv radio deconz /dev/ttyAMA0 backup deconz.json
2022-01-29 16:31:57.184 homeassistant asyncio DEBUG Using selector: EpollSelector
2022-01-29 16:31:57.360 homeassistant zigpy.appdb DEBUG SQLite version for <module 'sqlite3' from '/usr/local/lib/python3.9/sqlite3/__init__.py'>: 3.34.1
2022-01-29 16:31:57.387 homeassistant zigpy_deconz.uart DEBUG Connection made
2022-01-29 16:31:57.388 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.protocol_version: 34>, b'')
2022-01-29 16:31:57.389 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a02000800010022
2022-01-29 16:31:57.399 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a02000a000300220e01
2022-01-29 16:31:57.400 homeassistant zigpy_deconz.api DEBUG Read parameter protocol_version response: [270]
2022-01-29 16:31:57.400 homeassistant zigpy_deconz.api DEBUG Command Command.version (0,)
2022-01-29 16:31:57.401 homeassistant zigpy_deconz.uart DEBUG Send: 0x0d0300090000000000
2022-01-29 16:31:57.411 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0d0300090000076926
2022-01-29 16:31:57.411 homeassistant zigpy_deconz.api DEBUG Version response: 26690700
2022-01-29 16:31:57.414 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.mac_address: 1>, b'')
2022-01-29 16:31:57.414 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a04000800010001
2022-01-29 16:31:57.426 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a04001000090001fefd08ffff2e2100
2022-01-29 16:31:57.427 homeassistant zigpy_deconz.api DEBUG Read parameter mac_address response: [00:21:2e:ff:ff:08:fd:fe]
2022-01-29 16:31:57.428 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.aps_designed_coordinator: 9>, b'')
2022-01-29 16:31:57.429 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a05000800010009
2022-01-29 16:31:57.438 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a0500090002000901
2022-01-29 16:31:57.439 homeassistant zigpy_deconz.api DEBUG Read parameter aps_designed_coordinator response: [1]
2022-01-29 16:31:57.440 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.nwk_address: 7>, b'')
2022-01-29 16:31:57.441 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a06000800010007
2022-01-29 16:31:57.451 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a06000a000300070000
2022-01-29 16:31:57.452 homeassistant zigpy_deconz.api DEBUG Read parameter nwk_address response: [0x0000]
2022-01-29 16:31:57.453 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.nwk_panid: 5>, b'')
2022-01-29 16:31:57.453 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a07000800010005
2022-01-29 16:31:57.463 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a07000a000300054b32
2022-01-29 16:31:57.464 homeassistant zigpy_deconz.api DEBUG Read parameter nwk_panid response: [0x324b]
2022-01-29 16:31:57.465 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.aps_extended_panid: 11>, b'')
2022-01-29 16:31:57.466 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a0800080001000b
2022-01-29 16:31:57.477 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a0800100009000b0000000000000000
2022-01-29 16:31:57.478 homeassistant zigpy_deconz.api DEBUG Read parameter aps_extended_panid response: [00:00:00:00:00:00:00:00]
2022-01-29 16:31:57.479 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.nwk_extended_panid: 8>, b'')
2022-01-29 16:31:57.480 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a09000800010008
2022-01-29 16:31:57.491 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a09001000090008fefd08ffff2e2100
2022-01-29 16:31:57.493 homeassistant zigpy_deconz.api DEBUG Read parameter nwk_extended_panid response: [00:21:2e:ff:ff:08:fd:fe]
2022-01-29 16:31:57.493 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.current_channel: 28>, b'')
2022-01-29 16:31:57.494 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a0a00080001001c
2022-01-29 16:31:57.504 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a0a00090002001c0f
2022-01-29 16:31:57.505 homeassistant zigpy_deconz.api DEBUG Read parameter current_channel response: [15]
2022-01-29 16:31:57.505 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.channel_mask: 10>, b'')
2022-01-29 16:31:57.506 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a0b00080001000a
2022-01-29 16:31:57.515 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a0b000c0005000a00800000
2022-01-29 16:31:57.516 homeassistant zigpy_deconz.api DEBUG Read parameter channel_mask response: [<Channels.CHANNEL_15: 32768>]
2022-01-29 16:31:57.517 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.nwk_update_id: 36>, b'')
2022-01-29 16:31:57.518 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a0c000800010024
2022-01-29 16:31:57.527 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a0c00090002002400
2022-01-29 16:31:57.528 homeassistant zigpy_deconz.api DEBUG Read parameter nwk_update_id response: [0]
2022-01-29 16:31:57.529 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (2, <NetworkParameter.network_key: 24>, b'\x00')
2022-01-29 16:31:57.530 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a0d00090002001800
2022-01-29 16:31:57.544 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a0d00190012001800af912adb53c0fe5b478542134b7796c9
2022-01-29 16:31:57.546 homeassistant zigpy_deconz.api DEBUG Read parameter network_key response: [0, [175, 145, 42, 219, 83, 192, 254, 91, 71, 133, 66, 19, 75, 119, 150, 201]]
2022-01-29 16:31:57.546 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.nwk_frame_counter: 39>, b'')
2022-01-29 16:31:57.547 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a0e000800010027
2022-01-29 16:31:57.557 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a0e000c00050027d6a60200
2022-01-29 16:31:57.558 homeassistant zigpy_deconz.api DEBUG Read parameter nwk_frame_counter response: [173782]
2022-01-29 16:31:57.559 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.trust_center_address: 14>, b'')
2022-01-29 16:31:57.560 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a0f00080001000e
2022-01-29 16:31:57.571 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a0f00100009000efefd08ffff2e2100
2022-01-29 16:31:57.572 homeassistant zigpy_deconz.api DEBUG Read parameter trust_center_address response: [00:21:2e:ff:ff:08:fd:fe]
2022-01-29 16:31:57.573 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (9, <NetworkParameter.link_key: 25>, b'\xfe\xfd\x08\xff\xff.!\x00')
2022-01-29 16:31:57.574 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a10001000090019fefd08ffff2e2100
2022-01-29 16:31:57.591 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a10002000190019fefd08ffff2e21005a6967426565416c6c69616e63653039
2022-01-29 16:31:57.593 homeassistant zigpy_deconz.api DEBUG Read parameter link_key response: [00:21:2e:ff:ff:08:fd:fe, [90, 105, 103, 66, 101, 101, 65, 108, 108, 105, 97, 110, 99, 101, 48, 57]]
2022-01-29 16:31:57.594 homeassistant zigpy_deconz.api DEBUG Command Command.read_parameter (1, <NetworkParameter.security_mode: 16>, b'')
2022-01-29 16:31:57.595 homeassistant zigpy_deconz.uart DEBUG Send: 0x0a11000800010010
2022-01-29 16:31:57.604 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0a1100090002001003
2022-01-29 16:31:57.605 homeassistant zigpy_deconz.api DEBUG Read parameter security_mode response: [3]
2022-01-29 16:31:57.615 homeassistant zigpy_deconz.api DEBUG Command Command.write_parameter (5, <NetworkParameter.watchdog_ttl: 38>, b'\x00\x00\x00\x00')
2022-01-29 16:31:57.616 homeassistant zigpy_deconz.uart DEBUG Send: 0x0b12000c0005002600000000
2022-01-29 16:31:57.627 homeassistant zigpy_deconz.uart DEBUG Frame received: 0x0b12000800010026
2022-01-29 16:31:57.627 homeassistant zigpy_deconz.api DEBUG Write parameter watchdog_ttl: SUCCESS
2022-01-29 16:31:57.630 homeassistant zigpy_deconz.uart DEBUG Closed serial connection

However, the backup file is very short and shows no devices ("devices": []):

{
"metadata": {
"version": 1,
"format": "zigpy/open-coordinator-backup",
"source": "zigpy-cli@0.0.1",
"internal": {
"node": {
"ieee": "00212effff08fdfe",
"nwk": "0000",
"type": "coordinator"
},
"network": {
"tc_link_key": {
"key": "5a69674265________69616e63653039",
"frame_counter": 0
},
"tc_address": "00212effff08fdfe",
"nwk_manager": "0000"
},
"link_key_seqs": {}
}
},
"stack_specific": {},
"coordinator_ieee": "00212effff08fdfe",
"pan_id": "324b",
"extended_pan_id": "00212effff08fdfe",
"nwk_update_id": 0,
"security_level": 5,
"channel": 15,
"channel_mask": [
15
],
"network_key": {
"key": "af912adb53c0fe________134b7796c9",
"sequence_number": 0,
"frame_counter": 173782
},
"devices": []
}

In HA I have 14 devices and 30 entities, which I successfully control via the RaspBee, so something’s still off:

Any ideas what might cause the device list to be empty? I have disabled zha in HA while doing this. If I try to enable it I get different errors when trying to create the backup (about possible simultaneous access etc., which makes sense).

puddly · 2022-01-29T17:07:19Z

The coordinator normally doesn't keep track of any devices that aren't its direct children, but even that information isn't critical. The Conbee however expects the program controlling it to load in the child table at runtime since it'll be stored in the program's database. This is normal.

After you restore to the Conbee, can you unplug it, plug it back in, and take a second backup of it and post it here?

stiang · 2022-01-29T17:24:25Z

@puddly Sorry if I’m being dense, but are you saying that the devices don’t need to be listed in the backup file, and are instead "mapped to" the RaspBee/ConBee through other means that are saved in the HA config/storage files? In other words, the zigbee backup file I have now should actually get me the devices I need if paired with a full HA backup?

I’m also unclear on what’s the best procedure to do the restore. Should I plug the ConBee into the Pi and do the restore there, then move the already restored ConBee to my Ubuntu server? Or plug the ConBee into the Ubuntu server, copy the backup file and then do the restore from there? Or doesn’t it matter?

I’ll post the info you want once I have successfully performed the restore. Thanks for helping out, really appreciate it!

puddly · 2022-01-29T17:27:51Z

Correct, there will be no devices if you're using a Raspbee or a Conbee.

It doesn't matter where you do the restore. Since you already have this virtual environment up and running, it's easiest to do it wherever you did the Raspbee backup.

stiang · 2022-01-29T18:27:38Z

@puddly Sweet, looks like I was able to restore to the ConBee now. Will try to get it up and running on the new server shortly. Thanks again!

I followed the instructions you gave a few comments back about increasing the frame_counter value before restoring, then creating another backup. As far as I can see the value was reset, but that was all:

(venv) bash-5.1# diff conbee1.json conbee2.json
--- conbee1.json
+++ conbee2.json
@@ -33,7 +33,7 @@
"network_key": {
"key": "af912adb53c0fe5b478542134b7796c9",
"sequence_number": 0,
- "frame_counter": 1
+ "frame_counter": 0
},
"devices": []
}
\ No newline at end of file
(venv) bash-5.1#

Let me know if your need anything else!

puddly · 2022-01-29T18:36:50Z

Looks like your Conbee is running an old firmware and doesn't support setting the frame counter. The one in your Raspbee backup was 173782 but the value read from the Conbee was 0.

Can you upgrade it to the latest firmware and re-run the migration from the Raspbee again? Some devices are strict about the frame counter rolling back and may not respond to commands sent from the Conbee.

toot · 2022-01-31T08:24:11Z

@toot @connoleg would either of you mind testing out something with your Conbee? They need to be relatively running recent firmware (mid 2020 or newer) for this to work:
The contents of conbee1.json and conbee2.json should be nearly identical and the frame counter value should have changed.

Unfortunately I've got my Conbee not around at the moment, I'm trying to convince a friend from HA and borrowed him mine :D

robtpaton · 2022-04-20T10:47:49Z

Done and fixed thank you! Can confirm I was able to move from a Conbee II to Sonoff Zigbee Dongle 👍

mattague · 2022-04-22T09:18:28Z

I'm not entirely sure what might have gone wrong, but I can't seem to get this owrking. I am running in a virtual environment, but I can not get the commands to work. After running pip install... I get the following error at the end:

Successfully built zigpy zigpy-cli zigpy-znp zigpy-deconz bellows pure_pcapy3 scapy pyrsistent
Failed to build aiohttp pycryptodome frozenlist multidict yarl
ERROR: Could not build wheels for aiohttp, frozenlist, multidict, yarl, which is required to install pyproject.toml-based projects

And then trying to run zigpy -vv radio znp /dev/ttyACM0 backup znp.json gets me bash: zigpy: command not found.

Edit:

I did not get this working on my raspberry pi running Home assistant OS but I tried from another linux machine and it worked flawlessly, Thank you very much for the instructions.

p71-yes · 2022-05-03T12:48:57Z

great work @puddly !!! huge thanks..

Once installed, it was a smooth move from deconz to a sonoff zigbee stick...

hcrohland · 2022-05-03T14:38:19Z

Just migrated from a raspbee to a conbee without problems

francodutch · 2022-05-16T08:51:29Z

great work @puddly !!! huge thanks..
Once installed, it was a smooth move from deconz to a sonoff zigbee stick...

I need to do the same/similar - raspbi on ZHA to sonoff dongle on ZHA. I've read through the chain but if you're interested we could do a clean procedure and test it for others to use?

Hedda · 2022-06-20T11:23:43Z

Originally posted by @pipiche38 in zigpy/zigpy-zigate#117 (comment)

@puddly , I found that https://github.com/fairecasoimeme/ZiGate/blob/5a64452e3cc4f32afe217a332d0c9b1d97e17d84/ModuleRadio/Firmware/src/ZiGate/Source/ZigbeeNodeControlBridge/app_start.c#L1191 which can be interested for you as regards to a Backup/Restore solution.
If we take in consideration this procedure is used when the PDM size is changed during a firmware upgrade, we might think that can be used for Backup and then Restore.

@doudz Do you have any input in regards to achieving ZiGate Zigbee network backup via this network backup/restore utility?

thefjordkeeper · 2022-07-27T01:08:07Z

@puddly - I'm fairly certain this worked okay for me on Ubuntu 22.04 but I am unable to locate the ezsp.json file. Where is that saved?
I'm not moving from one Coordinator to another, I just flashed a spare Sonoff EFR32MG21 dongle with the latest firmware in an attempt to iron out some Zigbee network behavior. I want to verify that the backup actually exists, it's a sickness that comes from working in IS.

Thank you!

Hedda · 2022-08-04T14:57:26Z

@puddly - I'm fairly certain this worked okay for me on Ubuntu 22.04 but I am unable to locate the ezsp.json file. Where is that saved?
I'm not moving from one Coordinator to another, I just flashed a spare Sonoff EFR32MG21 dongle with the latest firmware in an attempt to iron out some Zigbee network behavior. I want to verify that the backup actually exists, it's a sickness that comes from working in IS.

Suggest that now that this is merged (i.e. closed as merged) maybe instead post any additional questions as new issues https://github.com/zigpy/zigpy-cli/issues or discussions https://github.com/zigpy/zigpy-cli/discussions or ?

balucanb · 2022-12-31T15:57:38Z

@puddly If this is obvious sorry, I do not really understand yet how to use this tool but my question is this- I want to move my Zigbee2Mqtt network currently on Rpi4 w/ Sonoff 3.0 dongle to a a Home assistant Yellow and keep it on Z2M is that possible with this tool ( or any other way) Thanks

asriva13 · 2023-05-15T04:47:05Z

Hi, I am trying to move from conbee2 to sonoff. While executing the backup command I am getting a timeout error (attached in the png file). Any help or directions would be appreciated.

Hedda · 2023-05-15T13:25:56Z

@asriva13 Have you first upgraded to the latest deconz firmware on your ConBee/RaspBee adapter?

For procedure step reference see example(s):

https://skyconnect.home-assistant.io/migrate-conbee/

1. In the deCONZ app, under Settings > Gateway, check the firmware version. It should be version 26720700 or newer.

If the version is recent enough, skip the next two steps and continue with Step 4 Stop the deCONZ add-on.

2. If the version is not up-to-date, select Update to 26720700.

Also see:

Initial implementation of cross-adapter network backup/restore
bc5f9ec

puddly mentioned this pull request Nov 16, 2021
Conbee II as ZHA coordinator: can't make a backup zigpy/zigpy-znp#98
Closed

puddly added 2 commits November 28, 2021 18:32

Use new zigpy JSON state serialization format
688a793

Increment frame counter during restore
00bb7fc

puddly mentioned this pull request Nov 29, 2021
Worth taking a look at pyspinel CLI as a reference? #3
Open

puddly added the enhancement New feature or request label Dec 12, 2021

Use old frame counter attribute
b7478f9

Hedda mentioned this pull request Dec 16, 2021
[SUGGESTION] Use Nordic nRF52840 chip ... zigpy/zigpy#394
Open

puddly mentioned this pull request Dec 20, 2021
Add parameter to set channel #6
Closed

puddly mentioned this pull request Jan 3, 2022
[REQUEST] Open ZigBee Coordinator Backup Format support for Zigbee network backups and restoring from and to ConBee/RaspBee (deCONZ based adapters) zigpy/zigpy-deconz#177
Closed

Hedda mentioned this pull request Jan 3, 2022
[REQUEST] Open ZigBee Coordinator Backup Format support for Zigbee network backups and restoring from and to ZiGate zigpy/zigpy-zigate#112
Closed

Hedda mentioned this pull request Jan 20, 2022
Weird error - EZSP_BACKUP mdeweerd/zha-toolkit#1
Closed

puddly added 4 commits May 7, 2022 12:22

Explicitly handle missing modules
fbfc8bc

Do not unnecessarily call connect
a96fda4

Merge branch 'dev' into puddly/zigpy-radio-api
02868a7

Remove source, since radio libraries will now provide it
a5c2d4f

This was referenced May 16, 2022
[RFC] Unified radio command line interface and standard network backup format zigpy/zigpy#557
Closed
Is there an zigpy-znp API zigpy/zigpy-znp#77
Open

puddly mentioned this pull request May 17, 2022
how to send association request with zigpy-znp zigpy/zigpy-znp#152
Closed

puddly added 2 commits May 21, 2022 22:38

Remove click from setup.py runtime dependencies
3e6503e

Explicitly import importlib.util
759e918

Hedda mentioned this pull request May 30, 2022
[REQUEST] Telink radio library for zigpy + a Zigbee serial API proxy/bridge mode? Xinyuan-LilyGO/T-ZigBee#3
Open

umverteiler mentioned this pull request Jun 1, 2022
Migrate from local coordinate to remote / POE-based coordinator tube0013/tube_gateways#67
Closed

puddly marked this pull request as ready for review June 22, 2022 18:35

puddly added 3 commits June 27, 2022 20:37

Always install the common radio libraries
f962747

Provide a way to increment the frame counter during restore
89c18fe

Indicate the current channel in the energy scan
645bba0

puddly merged commit f27b67b into zigpy:devJun 28, 2022

Hedda mentioned this pull request Aug 3, 2022
[REQUEST] ZBOSS radio library for zigpy + a serial to TCP proxy/bridge/server for the Zigbee serial interface? (IDFGH-7711) espressif/esp-idf#9253
Open

Hedda mentioned this pull request May 16, 2023
ZBOSS Zigbee network backup and network restore + radio migration via zigpy radio API? kardia-as/zigpy-zboss#11
Open

Hedda mentioned this pull request Oct 24, 2023
zigpy-znp application examples zigpy/zigpy-znp#192
Open

Uh oh!

Conversation

puddly commented Nov 14, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Instructions

Uh oh!

connoleg commented Jan 12, 2022

Uh oh!

puddly commented Jan 12, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

connoleg commented Jan 12, 2022

Uh oh!

toot commented Jan 12, 2022

Uh oh!

puddly commented Jan 12, 2022

Uh oh!

toot commented Jan 12, 2022

Uh oh!

puddly commented Jan 12, 2022

Uh oh!

toot commented Jan 13, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

puddly commented Jan 28, 2022

Uh oh!

stiang commented Jan 29, 2022

Uh oh!

puddly commented Jan 29, 2022

Uh oh!

stiang commented Jan 29, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

puddly commented Jan 29, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stiang commented Jan 29, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

puddly commented Jan 29, 2022

Uh oh!

stiang commented Jan 29, 2022

Uh oh!

puddly commented Jan 29, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

toot commented Jan 31, 2022

Uh oh!

robtpaton commented Apr 20, 2022

Uh oh!

mattague commented Apr 22, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

p71-yes commented May 3, 2022

Uh oh!

hcrohland commented May 3, 2022

Uh oh!

francodutch commented May 16, 2022

Uh oh!

Hedda commented Jun 20, 2022

Uh oh!

thefjordkeeper commented Jul 27, 2022

Uh oh!

Hedda commented Aug 4, 2022

Uh oh!

balucanb commented Dec 31, 2022

Uh oh!

asriva13 commented May 15, 2023

Uh oh!

Hedda commented May 15, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

puddly commented Nov 14, 2021•
edited
Loading

puddly commented Jan 12, 2022•
edited
Loading

toot commented Jan 13, 2022•
edited
Loading

stiang commented Jan 29, 2022•
edited
Loading

puddly commented Jan 29, 2022•
edited
Loading

stiang commented Jan 29, 2022•
edited
Loading

puddly commented Jan 29, 2022•
edited
Loading

mattague commented Apr 22, 2022•
edited
Loading