Service Disruption in cloud infrastructure (INC-123)
- George Sartzetakis
Date
Incident start date & time: Sunday, August 28, 2022, 01:09 (GMT+3)
Incident end date & time: Thursday, September 28, 2022, 22:40 (GMT+3)
Status
Complete
Summary
A security incident occurred that resulted to broad scale loss of access to cloud services hosted on interworks.cloud infrastructure in the Thessaloniki data center region
Impact
Hosted Exchange
Users of the service could not connect to the service endpoints using any available means supported by the service. E-mail communications to/from the hosted exchange endpoints were brought offline.
Infrastructure as a Service
Users were unable to access the management portal, and all of their cloud resources (virtual machines, storage, networks, cloud firewalls) were placed in an offline state.
Platform as a Service
Cloud server communications to interworks.cloud public services (such as Cloud Databases) were impacted, causing applications to lose connections to the cloud services.
DNS Services
The public DNS servers were taken offline.
Monitoring Services
Customer hosted monitoring dashboards and services were not operational.
Acronis Backup Cloud on interworks.cloud
Users experienced errors when trying to restore data from existing backups.
Root Causes
Based on the findings of the forensics analysis it appears that the attacker was able to infiltrate a server within the Hosted Exchange 2016 environment by exploiting one or more known CVEs as described in the relevant blog post in Microsoft Security Response Center. Although the relevant security updates were installed almost immediately after Microsoft's initial release (as published in status.interworks.cloud status page), it is very likely that certain files were not correctly updated as per MS statement regarding known issues for those specific updates which were published long after the installation date. According to recorded indicators of compromise within the Hosted Exchange infrastructure, the attacker managed to gain initial access to one of the Exchange servers, prior to the very next set of security updates for the Hosted Exchange 2016 environment which could have mitigated the vulnerability (see relevant maintenance entry in status.interworks.cloud status page). Having gained access to the server and via privilege escalation, the attacker was able to gain access to an administrative workstation and from thereon, to other servers utilizing remote desktop connections. The attacker was then able to distribute and execute a malicious executable on those servers. It has been confirmed that the PLAY ransomware protocol was used in the attack.
Trigger
The situation was triggered after a threat actor was able to deploy and execute malicious software (ransomware) across several interworks.cloud (non-customer) endpoints within the interworks.cloud infrastructure.
Resolution
The security response teams proceeded swiftly in taking offline affected services and resources in order to contain the attack that was in progress. Although this action had a severe impact on customer services availability, it was necessary to protect the integrity of the customer data. in addition, a full-scale forensics investigation began in order to discover which endpoint had been breached and how the attacker managed to gain access and conduct the attack. The interworks.cloud team worked alongside with leading third-party cyber-security and technology experts to pinpoint the root cause and seal tightly any potential security holes or vulnerable endpoints. After the attack was contained, the engineering team initiated an extensive validation of all services backup data, in order to start recovery of the services and restoration of latest data available. At the same time the team conducted thorough checks and deep scans of all affected infrastructure endpoints, gradually bringing them online one by one. All system accounts were initially disabled, and users were forced to change their password. The physical cloud infrastructure was completely rebuilt and redeployed to a new separate, and completely isolated from all other networks environment in order to ensure isolation at the lowest physical network level. Furthermore, MFA was enforced for all administrative accounts and extended to all infrastructure servers, not only the ones tagged as critical. Restoration of services and data was conducted in two parallel phases, one targeting the Athens region environment and the other targeting the new Thessaloniki region environment, in an effort to bring customer systems online in a secure manner as quickly as possible.
Detection
The issue was spotted by interworks.cloud monitoring systems and by alerts that were sent by local intrusion detection systems.
Action Items
Extend enforced use of MFA to administrative accounts for remote/console access to all server systems
Further strengthen MFA resilience by use of hardware FIDO2 security keys
Enforce zero-trust policy on the endpoint detection and response (EDR) software running on all corporate server and workstation systems
Engage a 24x7 SOC to monitor and analyze traffic and activity patterns within the environment
Rebuild and isolate physical cloud infrastructure in Thessaloniki region
Decommissioning of vulnerable Hosted Exchange services
Review of current access policies on physical infrastructure
Review existing vulnerability management process
Schedule review of risk management process
Export Hosted Exchange services mailbox data and make them available to the end-users
Rebuild and redeploy Azure Pack tenant portal to manage and administer IaaS workloads
Timeline
Sunday, August 28, 2022, 01:09 (GMT+3)
System alerts were received via the monitoring system regarding failing services in various endpoints and suspicious activity in server systems.
Sunday, August 28, 2022, 01:10 - 01:32 (GMT+3)
The engineering and security response teams began initial investigation in order to troubleshoot the incoming alerts.
Sunday, August 28, 2022, 01:32 - 02:25 (GMT+3)
Initial investigation revealed an ongoing attack by an external threat actor targeting various systems within the interworks.cloud infrastructure. The engaged teams began taking all affected and all dependent systems and services offline to thwart further spreading of the attack and protect service data.
Sunday, August 28, 2022, 02:25 - 09:08 (GMT+3)
The attack was contained, and all affected services were taken offline. A full-scale investigation began in order to reveal the entry points the attacker used to gain access and initiate the attack. At the same time a series of immediate hardening security measures were implemented to increase system security and further restrict access within the internal environment.
Sunday, August 28, 2022, 09:08 (GMT+3) - Monday August 29, 2022
Data backup validation began in order to prepare for restoration of affected services. An external security advisor firm was engaged in order to assist and oversee the forensics investigation. The Athens data center region was activated and gradual restoration of affected services to that region rolled out.
Tuesday, August 30, 2022
The engineering team commenced redeployment of the affected infrastructure to a separate isolated environment in order to begin restoring customer VM workloads in the Thessaloniki data center.
Wednesday, August 31, 2022
Public DNS, Cloud Databases and Acronis Cloud Backup on interworks.cloud services were successfully restored and brought back to operational status.
Thursday, September 1, 2022 - Tuesday, September 27, 2022
All affected services have been gradually restored - either in their original location and configuration or their backups were made available to end-users for direct use in other environments.
Thursday, October 6, 2022
The forensics analysis report was completed, confirming our initial findings regarding the attacker's entry point. No evidence of data exfiltration was discovered by the two independent firms that conducted the forensics investigation and analysis. At the same time, no customer has reported unauthorized access to, copy of, transfer, alteration or deletion of their data.