We experienced an issue with our MQTT broker between 00:26 UTC to 00:49 UTC. During this time window, we experienced issues with MQTT clients attempting to connect to Ubidots by creating a new connection socket. All new sockets requests were rejected by our broker.
MQTT clients that attempted to send data and already had an open socket, did not experience any issue.
Major, new socket connections to our broker were rejected.
We had scheduled to turn off our educational servers during the first week of March, and because of this, all the traffic coming from
things.ubidots.com was redirected to
industrial.api.ubidots.com. Unfortunately, the workload was bigger than what we had expected, and thus, our broker did not escalate as expected.
A high amount of new socket connection and number of clients connected to our industrial broker.
To rollback the traffic redirection. Actually, data coming from socket connections to
things.ubidots.com is not being redirected directly to our industrial cluster, but to an isolated environment that is in charge of the data ingestion to our main DB
Detected by the automated health checks robots
|Rollback the MQTT redirection change||mitigate||gustavo email@example.com||DONE|
|Create an updated internal benchmark with our actual traffic to MQTT to know our actual broker limits||information||juan firstname.lastname@example.org||IN PROGRESS|
|Based on the internal benchmark, to update our actual hardware deployment to MQTT to support the incoming traffic from the old educational endpoint||mitigate||juan email@example.com||IN PROGRESS|
The automated checks alerted once the system began to gett degraded.