Latency reduction for bind


I have recently migrated from a Legrand gateway which I found limitating to the zzh adapter. The network comprehends 2 dimmers, 1 switch, 1 plug socket and 2 battery powered buttons, all branded “Legrand with Netatmo”.

The pairing and the binding configuration went well but the latency is really high (it can take up to 10 seconds between the push on a battery powered button and the light to be on).
When I switch the light on from HomeAssistant (linked by Mosquitto broker to Zigbee2MQTT), there is almost no sensible latency. When the devices were connected to the Legrand gateway, the was no sensible latency either.

Has someone already experienced a similar problem ?

Map of the network