Lessons from Log4Shell, MOVEit, xz utils and Snowflake

The best-known hacks of recent years rarely had an exotic cause. Log4Shell sat in a logging library used in a lot of Java software. With MOVEit, organisations that never ran the software themselves still lost data, through a supplier. With Snowflake, attackers logged in to accounts that had no two-factor authentication. So the lessons are practical: know what you run, patch quickly, put MFA on every account and agree clear terms with your suppliers.

Log4Shell: knowing what is inside your software

In December 2021, CVE-2021-44228 became public, better known as Log4Shell. The flaw was in Log4j 2, a widely used Java logging library. Anyone who could get text into a log message could make the server load and run code from elsewhere. Version 2.15.0 turned the feature off by default, which turned out not to be enough; 2.16.0 removed it entirely.

Applying the patch was not the hard part. Finding out where Log4j was running took longer: in your own applications, as a dependency of a dependency, or inside a supplier's product.

My advice is to keep a searchable software bill of materials (SBOM) for each application. When the next big vulnerability lands, you want to know within an hour which systems are affected. Ask your suppliers the same question: how quickly can they tell you whether their product is vulnerable?

MOVEit: flaws at your suppliers

From 27 May 2023, the Cl0p ransomware gang exploited an unknown flaw in MOVEit Transfer, software for exchanging files with other organisations. It was a SQL injection (CVE-2023-34362). According to the US cyber agency CISA, the group installed a web shell and used it to steal data from the databases behind MOVEit.

Some of the victims did not use MOVEit at all. The BBC, British Airways and Boots lost staff data through Zellis, their payroll provider. Zellis said data from eight of its clients had been stolen.

So ask your suppliers which systems holding your data are exposed to the internet and how fast they patch critical flaws. Record in your data processing agreement how quickly they must report an incident and how long they keep files, because nobody can steal a file that was deleted after the transfer.

xz utils: looking into odd signals

At the end of March 2024, developer Andres Freund reported a backdoor in xz utils (CVE-2024-3094). It is a compression library that ends up in the SSH server on Debian, Ubuntu, Fedora and similar systems through systemd. Freund noticed that SSH logins were using a lot of CPU: a test login took about 0.8 seconds instead of 0.3. When he dug into the cause, he found the backdoor.

The attacker had worked towards this for more than two years. Under the name Jia Tan, they made useful contributions, gained commit access and eventually published releases themselves. By the time it was found, the compromised version was already in Fedora Rawhide, the Fedora 40 beta and Debian unstable.

A small company can do little to stop an attack like this. You can make sure production does not run the newest versions without a reason, and that you know which version runs where. Also give your team room to investigate a strange slowdown, because that is how this backdoor was found. Milos explains how attacks through npm and PyPI work in supply chain attacks on npm and PyPI.

Snowflake: logging in without MFA

In 2024, a group Mandiant tracks as UNC5537 stole data from Snowflake customer environments. Mandiant and Snowflake notified around 165 organisations. According to Mandiant, Snowflake itself was not breached: the attackers logged in with usernames and passwords that infostealer malware had captured earlier.

Mandiant names three causes. The accounts had no MFA, so a username and password were enough. Stolen passwords sometimes still worked years later, because nobody had changed them. There was also no allow list limiting access to known networks.

At the time, turning on MFA at Snowflake was up to the customer. So check every SaaS service you use to see whether MFA is enforced for all users, including accounts that are hardly ever used. Change passwords and keys on a fixed schedule, and limit access to data platforms to your own network or VPN where you can.

What I would want in place as a client

If I were the CTO or founder of a scale-up, after these four incidents I would want the following in place:

• an up-to-date list of applications, dependencies and suppliers, with an owner for each;
• an agreed patch window, in days, for critical flaws in internet-facing systems;
• MFA on every account, including external developers and suppliers;
• in contracts: how fast a supplier reports an incident, which data they keep and for how long;
• a restore you have actually rehearsed, in case ransomware gets through anyway.

Aleksa covers that last point in backups and disaster recovery.

Our developers work on managed, encrypted laptops with centrally managed accounts, SSO with two-factor authentication and secured access (see how we work). That takes care of MFA for our developers. Which systems you run, how fast they get patched and what your contracts say is still your job as the client. If your organisation falls under NIS2, many of these points are also legal obligations. Erik wrote about that in NIS2 and your development team.

Sources

• CVE-2021-44228 (Log4Shell)
• CISA: CL0P Ransomware Gang Exploits CVE-2023-34362 MOVEit Vulnerability
• BBC: MOVEit hack, BBC, BA and Boots among cyber attack victims
• Andres Freund on oss-security: backdoor in upstream xz/liblzma
• Russ Cox: Timeline of the xz open source attack
• Mandiant: UNC5537 targets Snowflake customer instances