S2 - Automated service for processing of SFTP folders is offline. Clients will experience a delay on file updates
Incident Report for Serraview
Postmortem

We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident. 

 

Description:  

On October 19, 2023, our customer experience team identified an issue with the automated SFTP imports in the My Imports module for US clients.  

Upon investigation, our engineering team identified that the root cause of the problem was that the US SFTP server was not configured correctly. 

 

Type of Event:  

Unplanned outage for US clients automatically importing files.  

  

Services/Modules Impacted:  

Automated SFTP imports. 

  

Remediation:  

The US SFTP server had its allocated memory increased to 32GB and the app server was restarted. 

  

Timeline (AEST):  

19th October 

  • 04:48 – Issue raised 
  • 06:28 – App server restarted 
  • 07:34 – Issue resolved; imports began processing correctly again 

 

Total Duration of Event:  

~ 2 hours and 46 minutes.  

  

Root Cause Analysis:  

The US SFTP was identified to not have enough resources, and the memory was upgraded from 8GB to 32GB. 

 

Preventative Action:   

We will be migrating web servers used by SFTP imports to ensure the load is spread evenly with other web requests.

Posted Nov 16, 2023 - 05:15 UTC

Resolved
This incident has been resolved.
Posted Oct 19, 2023 - 12:55 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Oct 18, 2023 - 20:33 UTC
Identified
The issue has been identified and a fix is being implemented.
Posted Oct 18, 2023 - 19:42 UTC
Investigating
Automated service for processing of SFTP folders is offline. Clients will experience a delay on file updates. Next update 4pm CST
Posted Oct 18, 2023 - 17:48 UTC
This incident affected: Core Services (NA- Core Services).