S2 - APAC SFTP Imports Delay
Incident Report for Serraview
Postmortem

We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident. 

 

Description:  

On October 20, 2023, our customer experience team identified an issue affecting SFTP Imports for APAC clients. The root cause was identified and resolved on October 24. 

 

Type of Event:  

Unplanned outage.  

  

Services/Modules Impacted:  

Automated Imports via SFTP/File Upload. 

  

Remediation:  

  • A problematic badge import was identified as the cause and was subsequently addressed through updates to its preprocessor, ensuring that it no longer affects SFTP Automated Imports for other clients 
  • Following the resolution of the import, the SFTP file server was restarted and automated imports were restored 

  

Timeline (AEST):  

20th October 

  • 13:49 – Severity 2 issue identified and incident triggered 
  • 18:30 – Root cause identified 

24th October 

  • 11:00 – Incident Resolved, files began importing again 

  

Total Duration of Event:  

~ 3 days and 21 hours.  

  

Root Cause Analysis:  

Faulty badge import preprocessor. 

 

Preventative Action:   

After resolving the root cause, we’ve also identified forward-dated logs that are now being rectified to enhance our ability to detect similar issues in the future.

Posted Nov 03, 2023 - 04:12 UTC

Resolved
APAC Automated Imports experienced a delay in processing, which was resolved by restarting the scheduled tasks and all imports started coming through again.

19 Oct 2023 (AEST)
9:14 PM - Last successful import

20 Oct
4:30 PM - Issue resolved
Posted Oct 19, 2023 - 10:30 UTC