2021-03-15
jose
Complete
We experienced an issue with our MQTT broker between 00:26 UTC to 00:49 UTC. During this time window, we experienced issues with MQTT clients attempting to connect to Ubidots by creating a new connection socket. All new sockets requests were rejected by our broker.
MQTT clients that attempted to send data and already had an open socket, did not experience any issue.
Major, new socket connections to our broker were rejected.
We had scheduled to turn off our educational servers during the first week of March, and because of this, all the traffic coming from things.ubidots.com was redirected to industrial.api.ubidots.com. Unfortunately, the workload was bigger than what we had expected, and thus, our broker did not escalate as expected.
A high amount of new socket connection and number of clients connected to our industrial broker.
To rollback the traffic redirection. Actually, data coming from socket connections to things.ubidots.com is not being redirected directly to our industrial cluster, but to an isolated environment that is in charge of the data ingestion to our main DB
Detected by the automated health checks robots
| Action Item | Type | Owner | Bug |
|---|---|---|---|
| Rollback the MQTT redirection change | mitigate | gustavo woakas@ubidots.com | DONE |
| Create an updated internal benchmark with our actual traffic to MQTT to know our actual broker limits | information | juan juan.agudelo@ubidots.com | IN PROGRESS |
| Based on the internal benchmark, to update our actual hardware deployment to MQTT to support the incoming traffic from the old educational endpoint | mitigate | juan juan.agudelo@ubidots.com | IN PROGRESS |
The automated checks alerted once the system began to gett degraded.
Support: support@ubidots.com