Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 2 Current »

Date

Incident start date & time: Sunday, August 28, 2022, 01:09 (GMT+3)

Incident end date & time: Thursday, September 28, 2022, 22:40 (GMT+3)


Status

Complete, supplementary actions pending

Summary

A security incident occurred that resulted to broad scale loss of access to cloud services hosted on interworks.cloud infrastructure in the Thessaloniki data center region

Impact

Hosted Exchange

Users of the service could not connect to the service endpoints using any available means supported by the service. E-mail communications to/from the hosted exchange endpoints were brought offline.

Infrastructure as a Service

Users were unable to access the management portal, and all of their cloud resources (virtual machines, storage, networks, cloud firewalls) were placed in an offline state.

Platform as a Service

Cloud server communications to interworks.cloud public services (such as Cloud Databases) were impacted, causing applications to lose connections to the cloud services.

DNS Services

The public DNS servers were taken offline.

Monitoring Services

Customer hosted monitoring dashboards and services were not operational.

Acronis Backup Cloud on interworks.cloud

Users experienced errors when trying to restore data from existing backups.

Root Causes

Based on the findings of the forensics analysis it appears that the attacker was able to infiltrate a server within the Hosted Exchange 2016 environment by exploiting one or more known CVEs as described in the relevant blog post in Microsoft Security Response Center. Although the relevant security updates were installed almost immediately after Microsoft's initial release (as published in status.interworks.cloud status page), it is very likely that certain files were not correctly updated as per MS statement regarding known issues for those specific updates which were published long after the installation date. According to recorded indicators of compromise within the Hosted Exchange infrastructure, the attacker managed to gain initial access to one of the Exchange servers, prior to the very next set of security updates for the Hosted Exchange 2016 environment which could have mitigated the vulnerability (see relevant maintenance entry in status.interworks.cloud status page). Having gained access to the server and via privilege escalation, the attacker was able to gain access to an administrative workstation and from thereon, to other servers utilizing remote desktop connections. The attacker was then able to distribute and execute a malicious executable on those servers. It has been confirmed that the PLAY ransomware protocol was used in the attack.

Trigger 

The situation was triggered after a threat actor was able to deploy and execute malicious software (ransomware) across several interworks.cloud (non-customer) endpoints within the interworks.cloud infrastructure.

Resolution

The security response teams proceeded swiftly in taking offline affected services and resources in order to contain the attack that was in progress. Although this action had a severe impact on customer services availability, it was necessary to protect the integrity of the customer data. in addition, a full-scale forensics investigation began in order to discover which endpoint had been breached and how the attacker managed to gain access and conduct the attack. The interworks.cloud team worked alongside with leading third-party cyber-security and technology experts to pinpoint the root cause and seal tightly any potential security holes or vulnerable endpoints. After the attack was contained, the engineering team initiated an extensive validation of all services backup data, in order to start recovery of the services and restoration of latest data available. At the same time the team conducted thorough checks and deep scans of all affected infrastructure endpoints, gradually bringing them online one by one. All system accounts were initially disabled, and users were forced to change their password. The physical cloud infrastructure was completely rebuilt and redeployed to a new separate, and completely isolated from all other networks environment in order to ensure isolation at the lowest physical network level. Furthermore, MFA was enforced for all administrative accounts and extended to all infrastructure servers, not only the ones tagged as critical. Restoration of services and data was conducted in two parallel phases, one targeting the Athens region environment and the other targeting the new Thessaloniki region environment, in an effort to bring customer systems online in a secure manner as quickly as possible.

Detection

The issue was spotted by interworks.cloud monitoring systems and by alerts that were sent by local intrusion detection systems.

Action Items

  • Extend enforced use of MFA to administrative accounts for remote/console access to all server systems (tick)

  • Further strengthen MFA resilience by use of hardware FIDO2 security keys (tick)

  • Enforce zero-trust policy on the endpoint detection and response (EDR) software running on all corporate server and workstation systems (tick)

  • Engage a 24x7 SOC to monitor and analyze traffic and activity patterns within the environment (tick)

  • Rebuild and isolate physical cloud infrastructure in Thessaloniki region (tick)

  • Decommissioning of vulnerable Hosted Exchange services (tick)

  • Review of current access policies on physical infrastructure (tick)

  • Review existing vulnerability management process (tick)

  • Schedule review of risk management process (tick)

  • Export Hosted Exchange services mailbox data and make them available to the end-users (tick)

  • Rebuild and redeploy Azure Pack tenant portal to manage and administer IaaS workloads (tick)

Timeline

Sunday, August 28, 2022, 01:09 (GMT+3)

System alerts were received via the monitoring system regarding failing services in various endpoints and suspicious activity in server systems.

Sunday, August 28, 2022, 01:10 - 01:32 (GMT+3)

The engineering and security response teams began initial investigation in order to troubleshoot the incoming alerts. 

Sunday, August 28, 2022, 01:32 - 02:25 (GMT+3) 

Initial investigation revealed an ongoing attack by an external threat actor targeting various systems within the interworks.cloud infrastructure. The engaged teams began taking all affected and all dependent systems and services offline to thwart further spreading of the attack and protect service data.

Sunday, August 28, 2022, 02:25 - 09:08 (GMT+3)

The attack was contained, and all affected services were taken offline. A full-scale investigation began in order to reveal the entry points the attacker used to gain access and initiate the attack. At the same time a series of immediate hardening security measures were implemented to increase system security and further restrict access within the internal environment. 


Sunday, August 28, 2022, 09:08 (GMT+3) - Monday August 29, 2022

Data backup validation began in order to prepare for restoration of affected services. An external security advisor firm was engaged in order to assist and oversee the forensics investigation. The Athens data center region was activated and gradual restoration of affected services to that region rolled out.


Tuesday, August 30, 2022

The engineering team commenced redeployment of the affected infrastructure to a separate isolated environment in order to begin restoring customer VM workloads in the Thessaloniki data center.

Wednesday, August 31, 2022 

Public DNS, Cloud Databases and Acronis Cloud Backup on interworks.cloud services were successfully restored and brought back to operational status.


Thursday, September 1, 2022 - Tuesday, September 27, 2022

All affected services have been gradually restored - either in their original location and configuration or their backups were made available to end-users for direct use in other environments.


Thursday, October 6, 2022

The forensics analysis report was completed, confirming our initial findings regarding the attacker's entry point. No evidence of data exfiltration was discovered by the two independent firms that conducted the forensics investigation and analysis. At the same time, no customer has reported unauthorized access to, copy of, transfer, alteration or deletion of their data.


  • No labels