Update - We have successfully finished the repair action of our NoSQL database and have included a new baremetal node in our cluster. We need to run a new Join + repair action with this new node to finish our migration. We expect minor latencies during this process that should not impact your daily Ubidots usage. We expect to finish this process next week.
May 4, 13:02 UTC

Update - We have identified a pattern where our time-series database cluster experiences a 15-minute degradation at 11:00, 12:00, 1:00, and 2:00 GMT-5 every day. This degradation is approximately 1/10th of the capacity (the SAN throughput goes from 200MB/s to 20MB/s), causing some dashboard widgets to display an error message during such times. The degradation slowly fades after a few minutes of such window, once the pending requests are processed.

About the SAN issues, our service provider is suggesting a server reboot will likely solve this, as it would discard OS-related issues. Unfortunately, this is not possible at this time because, given the initial degradation, we spun up a new server to increase overall cluster performance, and such a "cluster joining process" takes a few days, given the size of our database.

We are expecting that the new cluster node will successfully join within a week, after which we will be able to gradually reboot each cluster node to a new, fresh start.
Apr 29, 12:31 UTC

Update - We are still working on joining a new node to our cluster in order to alleviate the workload of our database access, unfortunately, this action is taking more than expected. We have detected additional SAN latency at our bare-metal infrastructure provider, and we are working with them in order to improve the write/read rate in our HDDs, this should improve the Database speed access. In the mean time, some users may experience latency when trying to access their data from the Dashboard or Variable view at certain times of the day. Synthetic variables calculation is expected to take longer than usually.
There are no issues with data ingestion.
Apr 27, 11:55 UTC

Update - We are experiencing latency for data retrieval for some requests, it is not a general issue but may affect some users that attempt to load dashboards or variables. The problem is not general and is raised just in certain moments during the daily Ubidots usage. We are working in the deployment of new cluster instances to solve it.
Apr 22, 14:36 UTC

Update - We have deployed a repair action to our cluster, this may derivate latency to our database for data retrieval and synthetic variables calculation. We do not expect any outage in our core systems, and this expected latency should not affect your daily Ubidots usage. This action may take the whole week to be finished
Apr 20, 12:46 UTC

Monitoring - We are experiencing latency during query execution at our database cluster, due to this, synthetic variables that usually take 1 minute to be calculated may take from 1:30 to 2:30, this behavior is extended to greater compute time variables. Dashboards and variables and device view may also experience a delay to be loaded. This delay should not be critical for our customer's applications.

We are working to solve this issues as soon as possible
Apr 12, 22:32 UTC