Token Authentication issues

Date

2021-03-15

Authors

jose

Status

Complete

Summary

We experienced issues with our data ingestion services between 00:26 UTC to 00:49 UTC. During this time window, we experienced issues with our token authentication rules, generating that temporal tokens created with an account API Key using the HTTP header X-Ubidots-ApiKey, which could not send data to Ubidots using the endpoint things.ubidots.com

Devices that attempted to send data to industrial.api.ubidots.com, did not experience any issue.

Impact

Major, all the requests coming from a token generated by an account API Key to things.ubidots.com were rejected

Root Causes

During a deployment routine, due to a human error, we did not add the rule to redirect the X-Ubidots-ApiKeyHTTP header from things.ubidots.com to industrial.api.ubidots.com in our Ngnix load balancer.

Trigger

The token authentication routine rejected new created tokens using an account APi key.

Resolution

To make available again the rule to redirect the API Key header from things.ubidots.com to industrial.api.ubidots.com

Detection

Detected by the DevOps team during a post-deployment verification

Action Items

Action Item	Type	Owner	Bug
Create again the redirect rule at Nginx	mitigate	gustavo woakas@ubidots.com	DONE
Create a functional check, to monitor data ingestion through all the available IoT protocols at Ubidots using things.ubidots.com	prevent	jose jose.garcia@ubidots.com	IN PROGRESS
Create functional checks to monitor data ingestion using created token with an account API Key	prevent	jose jose.garcia@ubidots.com	IN PROGRESS

Lessons Learned

What went well

The DevOps team could notice the issue early, and thus of this, the mitigation action took place fast.

What went wrong

• Our ingestion service experienced an issue due to a human error that we did not expect.
• There were not specific checks to verify if this way to ingest data was up and running.
• We did not monitor the endpoint things.ubidots.com

Supporting Information

Support: support@ubidots.com