Resolved -
This incident has been resolved.
Mar 11, 16:44 UTC
Update -
We are continuing to monitor for any further issues.
Feb 26, 19:41 UTC
Update -
We are still monitoring this latency. There's a substantial number of dots that are currently queued for ingestion.
To mitigate the TTS Plugin data ingestion issue, we have deployed a new version that no longer utilizes the affected endpoint.
Feb 26, 14:39 UTC
Monitoring -
We are actively working on implementing changes to mitigate the issue.
Feb 24, 11:00 UTC
Identified -
We are currently investigating a significant performance degradation in our bulk data processing system. This performance bottleneck is impacting the speed of the bulk data endpoint for all users and this incident has the potential to cause significant delays in triggering conditional events and alarms. Our team is currently investigating the root cause and working towards a resolution to restore normal operations as quickly as possible.
We have identified that the TTS Plugin is also affected. The reason is due to this plugin relies on the Bulk endpoint to ingest data.
We apologize for any inconvenience this may cause and appreciate your patience.
Further updates will be provided as we progress in resolving this issue.
Feb 24, 10:45 UTC
Completed -
The scheduled maintenance has been completed.
Mar 6, 13:00 UTC
In progress -
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Mar 6, 11:00 UTC
Scheduled -
As seen in the last 2 weeks, we have had issues with our bulk values ingestions services. These had been mitigated by adding more servers to consume the great amount of data sent that way and keep the system stable, however, this isn't sustainable.
To address this complex scalability situation, our back-end and DevOps team had rewritten the logic of the workers in charge of process the bulk queues.
After careful consideration and taking all preventing measures, namely:
– Unit testing – Testing in mirror deployments – Redirecting part of the traffic to a side server
We will deploy the new refactored logic to production.
We're aiming at confirming the new logic is faster and more capable of processing the bulk values queues.
Mar 5, 22:44 UTC