We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident. 

Description:  

On September 26, 2023, our customer experience team identified an issue with the automated SFTP imports for US clients.  

Upon investigation, our engineering team identified that the root cause of the problem was that the US SFTP server was not configured correctly. 

Type of Event:  

Unplanned outage for US clients automatically importing files.  

Services/Modules Impacted:  

Automated SFTP imports (manual imports were unaffected). 

Remediation:  

The import task was configured to proceed with imports sequentially rather than in parallel. Additional memory has been added to the Import Server to better handle automated imports. 

Timeline (AEST):  

26th September 

• 04:48 – Issue raised 

27th September 

• 00:35 - App server restarted, and imports started coming through again; issue set to Solved 
• 05:04 - Issue reoccurred; incident reopened 
• 12:15 – App server restarted again, Imports were noted to not process 
• 20:43 – Import service task manually kicked off and changed to do each import sequentially rather than multiple in parallel. 

28th September 

• 11:44 – Imports set back to parallel from sequential, now that the backlog of imports succeeded. 

Total Duration of Event:  

~ 2 days and 7 hours.  

Root Cause Analysis:  

The US SFTP was identified as not having enough resources to import files simultaneously efficiently. 

Preventative Action:   

Monitoring was added to identify high usages of SFTP servers, and the SFTP server itself has had its memory increased from 8GB to 32GB to better handle simultaneous imports.