Serraview Detailed Root Cause Analysis | 4.08.2024
S1 – Login errors
We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident.
Description:
On April 8th, 2024, internal teams and customer support received reports of some Serraview instances experiencing inability to login without warning. This issue impacted a small number of Serraview customers.
This cause of the login issue was due to a routine migration that unfortunately had a performance error, causing it to get stuck and impacting some customers’ ability to login.
Upon receiving notice of the login issues, our dedicated team promptly took action to restart the migration process, ensuring its successful completion and resolving the issue.
Type of Event:
Service login disruption.
Remediation:
Upon immediate notice of the login issues, our dedicated Serraview team was able to restart the migration process to successful completion, resolving the issue.
Timeline:
April 8
Total Duration of Event:
(0 day/1 hours/17minutes)
Root Cause Analysis:
Our Engineering team determined that the root cause of the issue was that a routine migration to upgrade the database and some pods for Serraview had an error in processing, causing the process to get stuck. This resulted in some customers experiencing difficulty logging in and unable to receive a notice about the downtime.
Preventative Action:
The Serraview team was able to restart the migration process and monitor it through successful completion and review with internal testing to ensure the issue was resolved for customers. This incident has been closed, but our team is dedicated to closely monitoring future updates as they are released to ensure best customer experience.