Service interruption on Webhooks, Signalling logs and Messages endpoint

Incident Report for Onomondo

Resolved

all data propagated
Posted Nov 25, 2023 - 07:01 CET

Monitoring

Wystems have been upgrades. Webhooks are going out, backlog of messages i being worked through.
Posted Nov 25, 2023 - 05:48 CET

Investigating

All signalling logs have been backfilled. However we have issues service on webhooks and are investigating.
Posted Nov 25, 2023 - 04:44 CET

Update

signalling logs are being backfilled
Posted Nov 25, 2023 - 00:55 CET

Update

Signalling data is being backfilled and there is a small delay for messages on webhooks. All other services back to normal.
Posted Nov 24, 2023 - 19:03 CET

Monitoring

An internal configuration combined with an external network not respecting specific signaling caused a bottleneck which propagated in our system. We’ve rectified the configuration and are currently monitoring while the system recovers.
Posted Nov 24, 2023 - 15:51 CET

Investigating

Having issues with delays for messages, this is affecting Webhooks, IoT Connectors, and Signalling logs. Some location data got dropped, however other data is currently being backfilled. regular user data is unaffected.
Posted Nov 24, 2023 - 11:38 CET
This incident affected: Connectors (AWS IoT Core, Microsoft Azure DPS (beta)) and Webhooks.