We are truly grateful for your continued support and loyalty. We value your feedback and appreciate your patience as we worked to resolve this incident. 

Description:  

On October 27, 2023, our customer experience team identified an issue affecting SVLive 2 for APAC clients due to the deployment of a recent update to the ETL Hub, stopping SVLive data from being processed temporarily. 

Our engineering team reverted the change, restarted the broker instance, and APAC SVLive 2 came back online and began processing the data again. 

Type of Event:  

Unplanned outage.  

Services/Modules Impacted:  

Serraview Live 2. 

Remediation:  

The change to the ETL Hub processes was reverted and the broker instance was restarted. 

Timeline (AEST):  

27th October 

• 10:35 - Incident Raised 
• 17:30 – Change reverted, broker instance restarted 
• 17:45 - Data processed, incident resolved  

Total Duration of Event:  

~ 7 hours and 10 minutes.  

Root Cause Analysis:  

The changes to the ETL Hub encountered issues during deployment but was not detected by automated monitoring. 

Preventative action:  

• Automated monitoring has been implemented to immediately alert the relevant teams 
• The servers of the broker instance will be upgraded to prevent the issue from reoccurring