We experienced issues with our private ingestion engine at Australia during different time windows:
## REST API response: From 18:40:23 UTC - 18:54:56 UTC
During this time window, our REST API responded with 50x error codes through HTTP, and may have lost data over other protocols as follows:
### From 17:54:35 UTC - 18:21:22 UTC
We responded to about 80% of the data received through TCP or UDP with a 50x response code. Through MQTT the response was without an ACK message. It is not possible to recover these Dots. Users may see data gaps in their variables.
### From 18:21:22 UTC - 18:54:56 UTC
We responded to about 100% of the data received through TCP or UDP with a 50x response code. Through MQTT the response was without an ACK message. It is not possible to recover these Dots. All our users will see data gaps in their variables during this time window.
The main root of the issue was related to our Kubernetes cluster, which suffered a non-expected task service demand increase, generating a general cluster degradation.
Please, remember that if you got a 50x response code or did not get an ACK message from our server, it means that something went wrong with the request and your request or socket message must be sent again.
Posted Nov 27, 2020 - 22:02 UTC
Monitoring
We have experienced issues receiving data through MQTT, TCP and HTTP at our private Australia deployment. We have released a hot patch and monitoring the services.
Posted Nov 27, 2020 - 19:18 UTC
This incident affected: Ubidots Australia (Private deployment) (MQTT Publish Sydney, TCP Sydney, HTTP).