Credit: This article's content is the hard work of Nick Carr, Christopher Glyer, Doug Bienstock, Alex, Scott Runnels, Eric Scales and a few others. I'm merely presenting it in a simple abridged version for public consumption.
Solarwinds incident seems still fresh in my mind. It was almost Christmas, everyone was already in holiday mood and a sombre announcement came out. Kevin Mandia, whose name made the word Mandiant, came out to say that they were breached.
My initial response was utter shock. I spoke about Solarwinds in multiple conferences, highlighting the "advanced" aspect of threat actors which forms the A in the APT.
This article is a follow up from the original incident, some new or discoveries that were recently released has come to attention and renewed interest.
As the story goes...
Mandiant found themselves facing a very unique adversary which they were dumbfounded. They reached out to former Mandiant colleageus Nick Carr and Christopher Glyer, whose been investigating sophisticated campaigns. Through their investigation and data provided by Mandiant, they identified each victim is assigned a unique name mimicking a real system in the target organization. This is to minimise suspicion that the traffic is going out. Through this, Carr and Glyer knew they can identify other victims, and this lead them to confirm that Mandiant had been breached!
The attackers were not only attacking the client but also the watchers!
Mandiant installed the compromised version of Orion software on 27 July 2020 and discovered the breach 10 November 2020. The attackers had a good 107 days of dwell time, equivalent to 3 months and 15 days.
The CEO asked a question - how did they get in? To which, there were no answers.
The Threat Actor
This threat actor was seriously professional, and had good tradecraft. They showed restraint, activated contingency plans when discovered and actively monitored for detection and response team activities. Clear mark of an apex predator.
Picture from Nick Carr
If you notice that the VM was removed before the public exploitation started happening. This indicates the threat actors attempt to cover their tracks.
One of the important lessons or misses that was identified in the breach is that insertion of the malicious code into the build environment. An important artefact that was missed was that the timestamp was not modified (timestomp), which would have gave away the breach.
Threat actors didnt spare any punches either. One of their evasion methods is by using sleep function on their SUNBURST beacons, which countered defenders operational timelines.
Courtesy Microsoft
Note that Cobalt Strike was used. Perhaps beacon traffic identification and long tail analysis would have lead to means of detection.
Who detected Solarwinds first?
Interestingly, the first party to detect Solarwinds activities was Department of Justice. DOJ had called Mandiant to investigate the matter, but due to customer confidentiality and secrecy, there were different teams working on similar breaches who hadn't communicated with each other.
Attacker's Modus Operandi
The attackers, through Solarwinds had compromised thousands of networks through the supply chain attacks, but only focused on about 100 of these targets.
In January 2019, the attackers gained access through an employee;s VPN account. They used the account to exfiltrate 129 source code repositories containing various Solarwinds software and also grabbed customer information.
The attackers studied the source doe and identified that Orion was the best target. Orion not only was on 45% of customer install base but also had access to other systems within the network.
The attackers went dark for 6 months. They returned to the build system.The malicious code built to the doppelganger DLL file was written in the same style as an employee would, making it indistinguishable. Some suspicion arose whether an internal staff was in cahoots, though no evidence were ever found.
The attackers even did tests on the build environment. They deployed a benign code first to test the effectiveness. On Feb 2020, they dropped Sunspot in.
The attackers not only targetted the source codes, but also ensured that they had good foothold of the organization. They had compromised 71 email accounts belonging to staffs, in order to conduct surveillance to the internal activities. Solarwinds had consulted with Crowdstrike who had been brought in to investigate and give the green light on what can be used.
The attackers had been doing counter forensics all along. Clearing logs to make sure their activites were wiped clean. To their advantage, some logs didn't exist.
In order for the Solarwinds attack to be successful, the Orion server must be directly connected to the Internet. Most prudent organizations tend to have these servers behind firewall/NAT. Solarwinds had advised customers to enforce access-control list to only allow communication between Orion server and Solarwinds, however many didn't do so. This made external communications much easier for the attackers.
The Sunburst backdoor was compiled into one of the DLL. Once Orion server is installed, the backdoor remained dormant netween 12-14 days to prevent detection. Once woken up, the server began sending information to the attacker. If the target is of no interest, the attackers can simply turn off the backdoor.
If the target was intersting, then the attacker installed Teardrop, a second backdoor. All activities were conducted through Teardrop. Keeping Sunburst covert was top of their priority.
Courtesy Mandiant
Teardrop enabled the attackers to steal credentials to get access to more sensitive information/emails/systems. They even got access to Microsoft's source code, though Microsoft denies that the codes were altered.
Clues to the attacker emerged when an old virtual machine (VM) that had ben active a year ago was spotted. This virtual machine was used to build Orion back in 2020. Solarwinds had used TeamCity to orchestrate the build process. TeamCity will spin up VM's to compile and then spin down. Once done, they are deleted. However if the VM failes, a snapshot of the VM is kept. The dangling VM's will be deleted post snapshot, but somehow this one got missed. The VM had Sunspot, the malicious code that was integrated into Orion.
Sunspot was planted into the build server on February. Remained on the server until March, only then Solarwinds development team started building Orion uding TeamCity. Once in the build process, it renames the original file and gives its doppleganger the original name. Its an exact replica, but only has additonal malicious backdoor code. The system grabs ths malicious file and builds it into the production software. This was a split second operation. Once compilation done, the attackers restore the original file, making it look seamless. However on June 4, Sunspot was removed and attackers had erased their tracks.
The last time the attacker logged in via VPN was on November 26 until December 12, when Mandiant was investigating the matter.
Questions remained - why attackers stopped the attack on June 4th?
FBI explained that during that spring, they had identified rogue traffic coming from a server running Orion and contacted Solarwinds to discuss about it. It seems the attacker whose watching the wire, realising their operations may be discovered decided to wind up on that day. Fact is DOJ discovered the attack a month before FBI. Lack of inter agency communication had been the cause for the incident not being alerted much earlier.
Incident Response Management
Incident Response is a step by step methodology that can be applied in handling cyber incidents. However, different incidents require different type of handling. Attackers monitor internal chatter and communications to ensure that they are not discovered.
In one instance, (I can't remember which case), the attacker knew that the target was bringing in a cyber security company to do some cyber security work. Learning this, they went dark for 6 months and came back to check if everything was back to how it was. They confirmed it seeing that the DO was received and payment made to the security company, confirming that the engagement is over.
In such instances, electronic communication makes it easier for attackers to know whats happening in the ground. In such cases, can cyber security be implemented securely?
Without learning the full scale of an attack, it is difficult to ascertain whether DFIR (Digital Forensics/Incident Response) steps taken are effective and not being actively evaded.
Final thoughts
This incident has me sleepless. Some questions in my head remains unanswered. In fact, I ask the same when someone tries to sell security solutions to me.
- What do I have in my defense that would have allowed me to detect such incident from a supposedly legitimate software? Now don't just think about Solarwinds, but what about other software? Perhaps from Microsoft, or even from a security vendor themselves?
- If a company like Mandiant can have a dwell time of 107 days, I could only assume at best that my networks are already compromised and someone is actively in there. How do I draw them out without causing suspicion?
- I know that beacon detection would be an important piece of detection capability I need to have. What other capabilities or analysis methodology besides long tail analysis do I need to have better detection?
- Question to security sales people - on hindsight once everything is known then you can always position your product to say it can detect it. ON HINDSIGHT. I want proactive detection for such threats. How? (Be warned, I have dispelled many who had attempted to even begin to try). Often i'd say, if Microsoft and Mandiant themselves were hacked and they had the most brilliant minds in cybersecurity, what can you possibly offer ?
- The attackers were smart enough to deploy multiple tools, show restraint and very good amount of patience. At the same time, they monitored communications and acted proactively. In such instances, what are my go to method? (I call it the Sapphire Protocols).
- Do I pray and hope that the Threat Actors make mistake so that I can use that against them? Or is there better ways to do this?
Reference
[1] Nast, C. (2023). The Untold Story of the Boldest Supply-Chain Hack Ever. Retrieved 5 May 2023, from https://www.wired.com/story/the-untold-story-of-solarwinds-the-boldest-supply-chain-hack-ever/