Access to applications: All end-user's applications have been applied their respective SSL certificate. Access to application is fully operational now
Schedule Reports: All queued reports have already been sent
Posted Dec 20, 2022 - 14:15 UTC
Monitoring
Access to applications: We have generated almost all SSL certificates to ensure access to end-user's applications are labeled as a secured connection. This will be ready today.
CDN static files: Some users will experience loss of some white-label assets (read previous update). Unfortunately, we cannot recover them, so the fastest way to get them back into the platform is simply upload them again.
Schedule Reports: There's still some user reports still repressed in the queue. This process is DB intensive because of the amount of data that needs to be extracted. We will continue to update as the queue clears.
Posted Dec 19, 2022 - 17:00 UTC
Update
Access to industrial.ubidots.com has been completely restored.
Some end-user application are still missing its SSL certificate to ensure a secure connection, so our DevOps team is head on making sure this is applied to all applications as soon as possible.
The hard drive failure corrupted static files hosted in the CDN, including assets such as those found in the applications' white-label (Loader logo, Favicon, Header logo, Login screen logo, Background image) configuration, as well as for example, the Profile's avatar image. Some user might have to upload this files again.
We have also identified that the Scheduled Reports engine was also compromised because of this hard drive failure. This means that as of this moment, there's a queue of reports to be generated and sent. Some of them might come up with errors.
Posted Dec 19, 2022 - 16:21 UTC
Update
Access to the main admin accounts at industrial.ubidots.com has been almost completely recovered as we have restored the CDN from which static files are loaded. Some users might experience degraded performance still.
End-user's applications static files backup has been mounted and we are generating the SSL certificates for the applications. As they're generated, they are being applied, which means that some applications will start working while others will take more time. Our DevOps team is estimating one more hour for all certificates to generate.
Posted Dec 19, 2022 - 02:53 UTC
Update
We are in direct contact with our infrastructure provider to find a solution to the disk failure.
At this moment, access to the main accounts at industrial.ubidots.com is intermittent, while the end-user's applications are still not accesible. We're mounting a backup of the application's static files to speed up the process of getting the UI back into operation while we have a more definitive solution to the disk failure from our provider.
Posted Dec 19, 2022 - 01:25 UTC
Identified
We have identified the error to be coming from a hard drive failure in the load balance and CDN server. We have moved all data traffic to a new load balancer to stop it from being lost, however, the UI is still not loading, and we're working on mounting a backup of the CDN.
As of this moment we know there will be a gap of data from 23:00 to 23:30 UTC.
Posted Dec 18, 2022 - 23:30 UTC
Update
We are continuing to investigate this issue.
Posted Dec 18, 2022 - 23:20 UTC
Investigating
We have experienced an error in one of our servers, specifically the one containing our load balancer for data traffic and the CDN from which static files are served. This is preventing data from being ingested through any of the supported protocols (HTTP, MQTT, TCP/UDP). Also, it is affecting the UI from loading, both at industrial.ubidots.com and the end-user's applications
We're investigating the root cause.
Posted Dec 18, 2022 - 23:00 UTC
This incident affected: America (Login Apps Toronto, HTTP Post Toronto, TCP Toronto, MQTT Publish Toronto, MQTT Subscribe Toronto, UDP Toronto).