The process of rolling back the changes applied yesterday has been completed. The latency is no longer being experienced.
Posted Jul 19, 2023 - 13:25 UTC
Monitoring
After reverting the deployment at 18/07/2023 18:00:00 UTC-5, we've seen the platform came back to normal operation. We will continue to monitor it.
Posted Jul 19, 2023 - 01:06 UTC
Identified
Our DevOps team found that the reason behind this latency was due to a process ran in the servers where it the team was deploying an optimization to the access the database storing time-series data.
The process began at 18/07/2023 15:00:00 UTC-5 and the performance impact was identified at around 18/07/2023 17:00:00 UTC-5. The process has since been reverted.
Posted Jul 18, 2023 - 23:37 UTC
Investigating
We're currently experiencing latency accessing the DB. This reflects in slowness while loading the platform, or querying the API to read values.
Data ingestion service is operational and no data is being lost.
Posted Jul 18, 2023 - 23:08 UTC
This incident affected: America (Login Apps Toronto).