S1 - Serraview 404 Not Found error
Incident Report for Serraview
Postmortem

Serraview Detailed Root Cause Analysis | 4.02.2024 

S1 – Login errors 

 

We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident.  

 

Description: 

On April 2, 2024,  internal teams and customer support received reports of some Serraview instances experiencing inability to login without warning. This issue impacted a small number of Serraview customers.   

This cause of the login issue was due to a routine migration that unfortunately had a performance error, causing it to get stuck and impacting some customers’ ability to login. 

Upon receiving notice of the login issues, our dedicated team promptly took action to restart the migration process, ensuring its successful completion and resolving the issue. 

 

Type of Event: 

Service login disruption. 

 

Remediation: 

Upon immediate notice of the login issues, our dedicated Serraview team was able to restart the migration process to successful completion, resolving the issue. 

 

Timeline: 

 April 2 

  • (5:40 AM) – Internal teams were notified that some customers were experiencing unexpected login issues due to the migration process to upgrade the product not completing properly.  
  • (1:12 PM) – The migration process to update Serraview was restarted by our team and monitored through successful completion.  

  • (1:18 PM) – Migration successfully completed. Monitoring to ensure resolution was successful began. All-clear was declared and incident was closed.  

 

Total Duration of Event: 

(0 day/7 hours/36minutes)  

 

Root Cause Analysis: 

 

Our Engineering team determined that the root cause of the issue was that a routine migration to upgrade the database and some pods for Serraview had an error in processing, causing the process to get stuck. This resulted in some customers experiencing difficulty logging in and unable to receive a notice about the downtime.  

 

Preventative Action:  

The Serraview team was able to restart the migration process and monitor it through successful completion and review with internal testing to ensure the issue was resolved for customers. This incident has been closed, but our team is dedicated to closely monitoring future updates as they are released to ensure best customer experience.

Posted Apr 12, 2024 - 16:09 UTC

Resolved
This incident has been resolved.
Posted Apr 02, 2024 - 13:52 UTC
Investigating
Serraview and Engage not working for a few clients. We are currently investigating the issue.
Posted Apr 02, 2024 - 13:04 UTC