BIG rework of integration Xiaomi Gateway 3

**IMPORTANT**
- The version is under development and will be called v4.0.0
- This post has been updated over time
- It is highly recommended to make a backup before upgrading!

**Why?** With the yet another Home Assistant update, the mechanism of creating entities was broken. They only worked when the Home Assistant was started. And became unavailable with any restart of the integration.

Creating entities is a very complicated part of integration. Some may be created on the first connection to the gateway. Some may appear when new devices are connected. If there are no converters for the device - entities can be created when the first data from the device appears.

But the most difficult part is that multiple integrations (gateways) may try to create the same entity. This works for BLE, Mesh, and sometimes even Zigbee devices. This behavior is not supported by the Home Assistant core. And needs to be coded very carefully.

So I decided to rework the entire component once again.

## Breaking changes
A lot of things have changed in the integration and something may not work and something has to be manually updated.

## Converters
All converters moved to `/xiaomi_gateway3/core/devices.py`.
All entities descriptions moved to `/xiaomi_gateway3/hass/entity_description.py`.

#### Basic converter class simplified
```python
# before
@dataclass
class Converter:
 attr: str # hass attribute
 domain: Optional[str] = None # hass domain

 mi: Optional[str] = None
 parent: Optional[str] = None

 enabled: Optional[bool] = True # support: True, False, None (lazy setup)
 poll: bool = False # hass should_poll

 # don't init with dataclass because no type:
 childs = None # set or dict? of children attributes
 zigbee = None # str or set? with zigbee cluster

# after
@dataclass
class BaseConv:
 attr: str
 domain: str = None
 mi: str | int = None
 entity: dict = None
```

#### Deprecated **enabled=False**
- Moved to `entity.enabled=False` param
```python
# before
BoolConv("led", "switch", mi="6.p.6", enabled=False),
# after
BoolConv("led", "switch", mi="6.p.6", entity={"enabled": False}),
```

#### Deprecated **enabled=None**
- Moved to `entity.lazy=True` param
```python
# before
Converter("battery", "sensor", enabled=None), # no in new firmwares
# after
Converter("battery", "sensor", entity={"lazy": True}),
```

#### Deprecated parent="..."
- Now the entities know what attributes they need to subscribe to
```python
# before
{
 "lumi.light.aqcn02": ["Aqara", "Bulb CN", "ZNLDP12LM"],
 "spec": [
 BoolConv("light", "light", mi="4.1.85"),
 ZXiaomiBrightnessConv("brightness", mi="14.1.85", parent="light"),
 ZXiaomiColorTempConv("color_temp", mi="14.2.85", parent="light"),
 MapConv("power_on_state", "select", mi="8.0.2030", map=BULB_MEMORY, enabled=False),
 ],
}
# after
{
 "lumi.light.aqcn02": ["Aqara", "Bulb CN", "ZNLDP12LM"],
 "spec": [
 BoolConv("light", "light", mi="4.1.85"),
 ZLumiBrightness("brightness", mi="14.1.85"),
 ZLumiColorTemp("color_temp", mi="14.2.85"),
 ZTransitionConv("transition"),
 MapConv("power_on_state", "select", mi="8.0.2030", map={0: "on", 1: "previous"}), # config
 ],
}
```

#### Setup entity description inside converter
```python
MathConv("occupancy_duration", "sensor", mi="2.p.3", entity={"category": "diagnostic", "enabled": False, "units": UNIT_MINUTES}),
```

#### Multiple models
- Now is OK to add multiple models to device
- For all BLE and Mesh devices it **is recommended** to add MiHome model (`hhcc.plantmonitor.v1`)
```python
# before
"lumi.gateway.mgl03": ["Xiaomi", "Multimode Gateway", "ZNDMWG03LM"]
"lumi.gateway.mgl001": ["Xiaomi", "Multimode Gateway 2 EU", "ZNDMWG04LM"],
152: ["Xiaomi", "Flower Care", "HHCCJCY01"],
1371: ["Xiaomi", "TH Sensor 2", "LYWSD03MMC"],

# after
"lumi.gateway.mgl03": ["Xiaomi", "Multimode Gateway", "ZNDMWG03LM", "ZNDMWG02LM", "YTC4044GL"],
"lumi.gateway.mgl001": ["Xiaomi", "Multimode Gateway 2 EU", "ZNDMWG04LM", "BHR6765GL"],
152: ["Xiaomi", "Flower Care", "HHCCJCY01", "hhcc.plantmonitor.v1"],
1371: ["Xiaomi", "TH Sensor 2", "LYWSD03MMC", "miaomiaoce.sensor_ht.t2"],
```

#### BLE old converters
- MiBeacon converter deprecated (`MiBeacon`)
- Prebuild converters deprecated (`BLETemperature`...)
- `mi` param now support `eid` from MiBeacon messages
- `BLEByteConv`, `BLEMathConv`, `BLEFloatConv` supports unpack MiBeacon data
```python
# before
{
 152: ["Xiaomi", "Flower Care", "HHCCJCY01"],
 "spec": [
 MiBeacon, BLETemperature, BLEMoisture, BLEConductivity, BLEIlluminance,
 Converter("battery", "sensor", enabled=None), # no in new firmwares
 ],
}
# after
{
 152: ["Xiaomi", "Flower Care", "HHCCJCY01", "hhcc.plantmonitor.v1"], # 4100,4103,4104,4105
 "spec": [
 BLEMathConv("temperature", "sensor", mi=4100, multiply=0.1, round=1, signed=True), # int16
 BLEMathConv("illuminance", "sensor", mi=4103), # uint24
 BLEByteConv("moisture", "sensor", mi=4104), # uint8
 BLEMathConv("conductivity", "sensor", mi=4105), # uint16
 BLEByteConv("battery", "sensor", mi=4106, entity=ENTITY_LAZY), # uint8
 ],
}
```

#### BLE new converters
```python
# before
{
 4611: ["Xiaomi", "TH Sensor", "XMWSDJ04MMC"],
 "spec": [
 MiBeacon, BLETemperature, BLEHumidity,
 # https://github.com/AlexxIT/XiaomiGateway3/issues/929
 MathConv("temperature", mi="3.p.1001", round=1),
 MathConv("humidity", mi="3.p.1008", round=1),
 Converter("battery", mi="2.p.1003"),
 Converter("battery", "sensor", enabled=None), # no in new firmwares
 ],
}
# after
{
 4611: ["Xiaomi", "TH Sensor", "XMWSDJ04MMC", "miaomiaoce.sensor_ht.t6"],
 "spec": [
 # mibeacon2 spec
 BLEFloatConv("temperature", "sensor", mi=19457, round=1), # float
 BLEFloatConv("humidity", "sensor", mi=19464, round=1), # float
 BLEByteConv("battery", "sensor", mi=18435), # uint8
 # miot https://github.com/AlexxIT/XiaomiGateway3/issues/929
 MathConv("temperature", mi="3.p.1001", round=1),
 MathConv("humidity", mi="3.p.1008", round=1),
 BaseConv("battery", mi="2.p.1003"),
 ],
}
```

#### MiOT spec buttons
- `ButtonMIConv("button")` deprecated, `ConstConv("action")` should be used
```python
# before
{
 11332: ["PTX", "Mesh Double Wall Switch", "090615.switch.aksk2"],
 "spec": [
 Converter("action", "sensor", enabled=False),
 ButtonMIConv("button_1", mi="8.e.1", value=1),
 ButtonMIConv("button_1", mi="8.e.2", value=2),
 ],
}
# after
{
 11332: ["PTX", "Mesh Double Wall Switch", "090615.switch.aksk2"],
 "spec": [
 BaseConv("action", "sensor", entity=ENTITY_DISABLED),
 ConstConv("action", mi="8.e.1", value=BUTTON_SINGLE),
 ConstConv("action", mi="8.e.2", value=BUTTON_DOUBLE),
 ],
}
```

#### MiOT events
- `mi="3.e.1012"` changed to `"3.e.1012.p.1"` if event has params
```python
# before
{
 # https://github.com/AlexxIT/XiaomiGateway3/issues/826
 7184: ["Linptech", "Wireless Button", "K11"],
 "spec": [
 MiBeacon, BLEAction, Button, BLEBattery,
 Converter("battery", mi="2.p.1003"),
 BLEEvent("action", mi="3.e.1012", map={1: SINGLE, 8: HOLD, 15: DOUBLE}),
 ],
}
# after
{
 # https://github.com/AlexxIT/XiaomiGateway3/issues/826
 7184: ["Linptech", "Wireless Button", "K11", "linp.remote.k9b01"],
 "spec": [
 # mibeacon2 spec
 BLEMapConv("action", "sensor", mi=19980, map={"01": BUTTON_SINGLE, "08": BUTTON_HOLD, "0F": BUTTON_DOUBLE}),
 BLEByteConv("battery", "sensor", mi=18435), # uint8
 # miot spec
 MapConv("action", mi="3.e.1012.p.1", map={1: BUTTON_SINGLE, 8: BUTTON_HOLD, 15: BUTTON_DOUBLE}),
 BaseConv("battery", mi="2.p.1003"),
 ],
}
```

#### Zigbee converters
- Now linked to ZHA contants to increase readability
```python
# before
class ZOnOffConv(ZBoolConv):
 zigbee = "on_off"
 zattr = "on_off"

 def encode(self, device: "XDevice", payload: dict, value: bool):
 cmd = zcl_on_off(device.nwk, self.ep, value)
 payload.setdefault("commands", []).extend(cmd)

# after
from zigpy.zcl.clusters.general import OnOff

class ZOnOffConv(ZBoolConv):
 cluster_id = OnOff.cluster_id
 attr_id = OnOff.AttributeDefs.on_off.id

 def encode(self, device: "XDevice", payload: dict, value: bool):
 cmd = zcl_on_off(device.nwk, self.ep, value)
 payload.setdefault("commands", []).extend(cmd)
```

#### Custom entities for models
```python
# before
def new_entity(gateway: XGateway, device: XDevice, conv: Converter) -> XEntity:
 if conv.mi == "4.21.85":
 return AqaraE1(gateway, device, conv)
 if device.model == 14050:
 return ScdvbHAVC(gateway, device, conv)
 else:
 return XiaomiClimate(gateway, device, conv)

# after
XEntity.NEW["climate.model.lumi.airrtc.tcpecn02"] = XAqaraS2 
XEntity.NEW["climate.model.lumi.airrtc.agl001"] = XAqaraE1 
XEntity.NEW["climate.model.14050"] = XScdvbHAVC
```

## Customize deprecated
- Moved to `xiaomi_gateway3.devices`
```yaml
# before
homeassistant:
 customize:
 binary_sensor.0x00158d00ccddeeff_motion:
 occupancy_timeout: 180

# after
xiaomi_gateway3:
 devices:
 "0x00158d00ccddeeff":
 occupancy_timeout: 180
```

## Stats sensors
Now can be either `sensor` or `binary_sensor`. The sensor data has also changed:
- Many useful info about `uid`, `did`, `mac`, `brand`, `model`, market and cloud names
- List of connected gateways
- Time from last message and info about last gateway
- Total payload from all messages

```yaml
extra:
 cloud_fw: 2.1.1_0037
 cloud_name: Home Lamp 1
 did: '1234567890'
 mac: 50:ec:50:aa:bb:cc
 market_brand: Xiaomi
 market_model: MJDP09YL, yeelink.light.mbulb3
 market_name: Mesh Bulb
 rssi_54ef44ccddff: -77
 rssi_6490c1ccddff: -52
 type: mesh
gateways: 54ef44ccddff, 6490c1ccddff
last_decode: 6m8s
last_decode_gw:
 fw_ver: 1.5.4_0090
 host: 192.168.1.123
 mac: 64:90:c1:cc:dd:ff
 model: lumi.gateway.mgl03
last_encode: 6m9s
last_report:
 light: false
model: 1771
payload:
 brightness: 255.0
 color_temp: 190
 flex_switch: true
 light: false
ttl: 20m
uid: 50ec50aabbcc
```

## Stats table

```yaml
type: custom:flex-table-card
clickable: true
columns:
 - data: name
 name: Name
 - data: device.uid
 name: UID
 - data: msg_received
 name: Recv
 modify: x+''
 - data: msg_missed
 name: Miss
 modify: x+''
 - data: device.extra.seq
 name: SEQ
 modify: x+''
 - data: device.extra.rssi
 name: RSSI
 modify: x+''
 - data: device.last_decode_gw.host
 name: Gateway
 modify: x+''
 - data: state
 name: Available
 - data: last_updated
 name: Last changed
entities:
 include:
 - binary_sensor.*_ble
 - binary_sensor.*_mesh
 - binary_sensor.*_zigbee
```

## Attributes template
```yaml
# before
xiaomi_gateway3:
 attributes_template: | 
 {% if attr in ('zigbee', 'ble', 'mesh') %} 
 {{{ 
 "name": device.info.name, 
 "device_fw_ver": device.fw_ver, 
 "device_model": device.model, 
 "device_market_model": device.info.model, 
 "device_manufacturer": device.info.manufacturer, 
 "integration": "gw3", 
 "gate": gateway.info.name, 
 "gateway_model": gateway.info.model, 
 "gateway_fw_ver": gateway.fw_ver 
 }}} 
 {% elif attr == 'gateway' %} 
 {{{ 
 "integration": "gw3", 
 "gate": gateway.info.name, 
 "gateway_model": gateway.info.model, 
 "gateway_fw_ver": gateway.fw_ver 
 }}} 
 {% elif attr == 'battery' %} 
 {{{ 
 "integration": "gw3", 
 "name": device.info.name, 
 "gate": gateway.info.name, 
 "battery": "true" 
 }}} 
 {% endif %}

# after
xiaomi_gateway3:
 attributes_template: |
 {% if attr in ('zigbee', 'ble', 'mesh') %}
 {{{
 "integration": "gw3",
 "name": device.human_name,
 "device_fw_ver": device.firmware,
 "device_model": device.model,
 "device_market_model": device.human_model,
 "device_manufacturer": device.extra.market_brand,
 "gate": gateway.human_name,
 "gateway_model": gateway.model,
 "gateway_fw_ver": gateway.firmware
 }}}
 {% elif attr == 'gateway' %}
 {{{
 "integration": "gw3",
 "gate": gateway.human_name,
 "gateway_model": gateway.human_model,
 "gateway_fw_ver": gateway.firmware
 }}}
 {% elif attr == 'battery' %}
 {{{
 "integration": "gw3",
 "name": device.human_name,
 "gate": gateway.human_name,
 "battery": "true"
 }}}
 {% endif %}
```
## Logging
Log format also have been reworked. Now you can control basic and mqtt logs from:
- Integration config (each type of data for each gateway)
- Global integration debug logs from Home Assistant Web UI
- `configuration.yaml`
```yaml
logger:
 default: warning
 logs:
 custom_components.xiaomi_gateway3: warning
 custom_components.xiaomi_gateway3.gate: info
 custom_components.xiaomi_gateway3.mqtt: info
 custom_components.xiaomi_gateway3.gate.192.168.1.123: debug
 custom_components.xiaomi_gateway3.mqtt.192.168.1.234: debug
```

## Command select
Now EVERY device has command select. Options list depends to device type:
- **Device info** - shows full device info in the Home Assistant notification. Same info you get in the stats sensors and in device diagnostics
- **Device update** - request state update from device. BLE devices and many battery Zigbee devices can't be requested
- **Device delete** - only Zigbee device option. Sends leave signal to Zigbee device

## Cloud Integration
- Now supports multiple accounts
- Now supports reloading the integration to update data
- Shows more information about devices
- Fixed name support for new Zigbee devices

## Zigbee force pairing
In the default "zigbee pairing" mode you can pair supported MiHome zigbee devices and 3rd party zigbee devices. But you can't pair unsupported MiHome zigbee devices.
In this mode you can pair any zigbee device, but it won't be displayed in the MiHome even if supported.

## RSSI
Now supported for BLE and Mesh. Data is logged separately for each gateway.

## Mesh groups
Now Mesh groups is another device type. Groups of different models (`light`, `cover`) are also supported.

## Matter
Support Matter **child devices** for Xiaomi Miltimode Gateway 2 (EU) on fw 1.0.7_0019.

## Device triggers
Was changed from multiple **types** and multiple **actions**. To one **type** - action and multiple **states**.
If you used device triggers - them should be updated manually.

## Delete device
Now deleting Home Assistant device won't delete it from Gateway. For deleting Zigbee device you can use "command select". For deleting other devices you should use MiHome.

## Gateway alarm
Now has trigger for enabling and disabling.

## Gateway disabling
Now gateways has disable and enable options via "command select". Them just for test, so you can check if your BLE/Mesh devices still can be controlled when some of your gateways down.

Navigation Menu

BIG rework of integration Xiaomi Gateway 3 #20

Breaking changes

Converters

Basic converter class simplified

Deprecated enabled=False

Deprecated enabled=None

Deprecated parent="..."

Setup entity description inside converter

Multiple models

BLE old converters

BLE new converters

MiOT spec buttons

MiOT events

Zigbee converters

Custom entities for models

Customize deprecated

Stats sensors

Stats table

Attributes template

Logging

Command select

Cloud Integration

Zigbee force pairing

RSSI

Mesh groups

Matter

Device triggers

Delete device

Gateway alarm

Gateway disabling

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

BIG rework of integration Xiaomi Gateway 3 #20

Description

Breaking changes

Converters

Basic converter class simplified

Deprecated enabled=False

Deprecated enabled=None

Deprecated parent="..."

Setup entity description inside converter

Multiple models

BLE old converters

BLE new converters

MiOT spec buttons

MiOT events

Zigbee converters

Custom entities for models

Customize deprecated

Stats sensors

Stats table

Attributes template

Logging

Command select

Cloud Integration

Zigbee force pairing

RSSI

Mesh groups

Matter

Device triggers

Delete device

Gateway alarm

Gateway disabling

Activity

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions