Resolved -
Incident Summary:
Between 11:00 PM and 4:41 AM UTC-5, some users experienced 404 errors when attempting to access applications deployed in our US region.
Root Cause:
The issue was traced to an unexpected behavior in our caching mechanism. As part of recent DDoS attack mitigation improvements, changes were made to the cache that inadvertently resulted in these 404 errors. Our apps' cache is set to expire every 7 days, and when the cache expired, the affected applications returned 404 errors. This behavior required manual intervention or periodic tasks to refresh the cache.
Resolution:
A fix has been deployed to ensure the cache is automatically updated moving forward. We have also implemented additional monitoring checks every 5 minutes to proactively identify and address similar issues.
Next Steps:
We are exploring additional improvements to further enhance reliability and reduce the risk of similar incidents in the future.
We apologize for any inconvenience caused and appreciate your understanding as we continue to improve our systems.
Mar 18, 04:00 UTC