Disrupted operation of cloud platform sites and services in Azure (EU) region
Date
Friday, February 21 2020 17:14 (GMT+2) - Friday, February 21 2020 17:19 (GMT+2)
Status
Complete
Summary
Access to cloud platform sites hosted in the Azure (EU) region is severely impacted. Some Storefront sites fail to load or display errors during loading. Some internal interworks.cloud platform services area also impacted and have stopped functioning.
Impact
Cloud Platform sites (BSS and Storefront web applications)
Users of the service may have experienced service disruption or web page load failures regarding the interworks.cloud platform web sites. This applies to both BSS and Storefront (v3 and v4) applications.
Cloud Platform services
The platform services that are responsible for operating the platform notification engine and internal platform support tickets have stopped working.
Root Causes
A significant use of platform's reporting module capabilities occurred within a very brief period of time requesting report generation on massive amounts of data from the platform databases. These operations started consuming all available resources allocated to temporary database objects that handle such requests and eventually resource congestion occurred causing further platform requests from BSS and Storefront applications to fail. Users of the platform services experienced failure to successfully load the contents of the platform sites. In addition certain platform services stopped working.
Trigger
The situation was triggered during heavy data operations involving multiple report generation using the relevant report module of the platform that occurred approximately at the same time frame.
Resolution
interworks.cloud senior engineering team was able to troubleshoot the situation by restarting the underlying platform database services. Further monitoring of the situation verified final resolution of the incident.
Detection
The issue was spotted by interworks.cloud platform monitoring systems that alerted the responsible response team.
Action Items
Re-configuration of the resources that are assigned to database services regarding temporary data storage and management in order to allow operations that require significant amounts of resources to function without causing I/O bottlenecks and to also allow enough time for the responsible response team to react in case similar platform behavior (regarding heavy report usage) is recorded again in the future.
Timeline
Friday, February 21 2020 17:08 (GMT+2) - 17:14 (GMT+2)
Multiple report generation operations where initiated through BSS web application and started consuming all available resources allocated to temporary database objects within the platform database servers. At approximately 17:14 (GMT+2) resource congestion occurred on the temporary database objects and this in turn caused failures to serve requests from the BSS and Storefront sites, as well as from platform services.
Friday, February 21 2020 17:14 (GM+2) - 17:18 (GM+2)
System alerts were received via the monitoring system, notifying of service and site unavailability.
Friday, February 21 2020 17:16 (GM+2) - 17:19 (GMT+2)
The engineering team was actively engaged in incident troubleshooting. Root cause was revealed and immediate actions were undertaken in order to remedy the situation.
Friday, February 21 2020 17:19 (GMT+2)
Cloud platform sites and services operation was restored. Engineering team maintained close monitoring of the previously affected resources.
Friday, February 21 2020 17:49 (GMT+2)
All services verified as up and running within established parameters. Incident was marked as resolved. Additional action items were scheduled in order to prevent similar issues from occurring again in the future.