How developers scrambled to secure the Log4j vulnerability

Last weekend, the internet caught fire, and it is still unclear just how many developers with fire extinguishers will be needed to bring it under control. There was a set of first responders on the scene, however: largely unpaid maintainers or developers working in their spare time to patch vulnerabilities, issue guidance, and provide some much-needed clarity among the chaos.

On December 9, the Apache Foundation released an emergency update for a critical zero-day vulnerability called Log4Shell which had been identified in Log4j, an open source logging framework used in all kinds of Java applications. The bug, identified as CVE-2021-44228, allows an attacker to execute arbitrary code on any system that uses the Log4j library to write out log messages. It was immediately rated with the maximum severity of 10 on the CVSS scale.

[ Also on InfoWorld: How to detect the Log4j vulnerability in your applications ]

As Cloudflare CTO John Graham-Cumming wrote, “This is likely one of the most serious vulnerabilities on the internet since both Heartbleed and ShellShock.” Even Minecraft wasn’t safe.

The first responders

As developers and maintainers immediately scrambled over the weekend to patch as many of their Java applications as possible. The first line of defense was Log4j itself, which is maintained by the Logging Services team at the nonprofit Apache Software Foundation.

Apache’s Logging Services team is made up of 16 unpaid volunteers, distributed across almost every time zone around the world. “We do this because we love writing software and solving puzzles in our free time,” Gary Gregory, a software engineer and member of the Apache Logging Services Project Management Committee (PMC), told InfoWorld.

The PMC’s primary communication channel is email—and on Wednesday, November 24, at 7:51am GMT the group received an explosive one. Chen Zhaojun, a member of the cloud security team at Alibaba, was alerting them that a zero-day security bug had been discovered in their software.

“It was clear right away this would be a big problem,” Gregory said, operating on about four hours sleep over the weekend.

The team quickly got to work patching the issue in private, but their timeline accelerated rapidly when the exploit became public knowledge on Thursday, December 9. “Please hurry up,” Alibaba’s Chen urged.

Gregory and his fellow maintainers dropped everything and started working to fix the issue, putting together a version 2.15 update which they quickly decided “was not good enough” before issuing the update at 10:00am GMT on Friday, December 10. They followed up with a 2.16 release at 10:28pm GMT on December 13.

“I know these people—they all have families and things they have to do. But they put everything aside and just sat down for the whole weekend and worked on that,” former Log4j developer Christian Grobmeier told Bloomberg.

By this point in the weekend, the active members of the PMC had switched to communicating via a private Slack channel, where they continued to firefight the issue and work together to produce updates for users operating older versions of Java. They quickly produced the 2.12.2 release to fix the issue for Java 7 users. A fix for Java 6 is proving trickier, but is next on their backlog.

“Overall, I think despite the horrible consequences of this kind of vulnerability, things went as well as an experienced developer could expect,” Gregory said. “We were notified, provided a patch quickly and iterated on that release. That is something I have seen in professional environments time and time again.”

Hotpatches and urgent guidance

Another group that moved quickly over the weekend was the Amazon Corretto team within Amazon Web Services. Corretto is a distribution of the Open Java Development Kit (OpenJDK), putting this team on the front line of the Log4Shell issue.

Led by principal software engineer Volker Simonis, the Corretto team quickly built and open sourced a hotpatch for any organization where updating is not immediately possible.

As is described on its GitHub page:

This is a tool which injects a Java agent into a running JVM process. The agent will attempt to patch the lookup() method of all loaded org.apache.logging.log4j.core.lookup.JndiLookup instances to unconditionally return the string “Patched JndiLookup::lookup()”.

The hotpatch is designed to address the CVE-2021-44228 remote code execution vulnerability in Log4j without restarting the Java process. The dynamic and static agents are known to run on JDK 8 and JDK 11 on Linux, whereas on JDK 17 only the static agent is working.

“A huge thanks to the Amazon Corretto team for spending days, nights, and the weekend to write, harden, and ship this code,” AWS CISO Steve Schmidt wrote in a blog post. AWS has also posted an exhaustive list of service-specific security updates for impacted products.

Elsewhere, members of the Java team at Microsoft, led by principal engineering group manager for Java, Martijn Verburg, helped evaluate that patch and also issued more general advice for customers to protect themselves, including several recommended workarounds until a complete security update can be applied.

Google Cloud responded with an update to its Cloud Armor security product, which issued an urgent Web Application Firewall (WAF) rule on December 11 to help detect and block attempted exploits of CVE-2021-44228.

“In an attempt to help our customers address the Log4j vulnerability, we have introduced a new preconfigured WAF rule called “cve-canary” which can help detect and block exploit attempts of CVE-2021-44228,” Emil Kiner, a product manager for Cloud Armor and Dave Reisfeld, a network specialist manager at Google wrote in a blog post on Saturday.

What you can do

While these in-house developers hurried to secure their software for customers, many end users and enterprise developers are scrambling to assess their vulnerability and secure their own Java applications.

The first thing to do is detect whether Log4j is present in your applications. It’s also important to note that not all applications will be vulnerable to this exploit. Anyone using a Java version higher than 6u212, 7u202, 8u192, or 11.0.2 should be safe, thanks to the added protection for JNDI (Java Naming and Directory Interface) remote class loading in those versions.

Similarly, users of Log4j versions higher than 2.10 should mitigate the issue by setting the system property formatMsgNoLookups to true, setting the JVM parameter -Dlog4j2.formatMsgNoLookups=true, or by removing the JndiLookup class from the classpath.

It’s not over yet

Because the Log4j vulnerability not only impacts Java applications, but also any services that use the library, the Log4Shell attack surface is likely very large.

As Lucian Constantin wrote for CSO, “The community is still working to assess the attack surface, but it’s likely to be huge due to the complex ecosystem of dependencies. Some of the impacted components are extremely popular and are used by millions of enterprise applications and services.”

For its part, the Apache Logging Services team will “continue to evaluate features of Log4j that could have potential security risks and will make the changes necessary to remove them. While we will make every effort to maintain backward compatibility this may mean we have to disable features they may be using,” Ralph Goers, a member of the Apache Logging Services team, told InfoWorld.

Even as countless developers worked tirelessly over the weekend to patch the Log4j vulnerability, there will be plenty who are slower to respond. Thus the impact of Log4Shell will likely be long-term and wide-ranging.

As security analyst Tony Robinson tweeted: “While the good ones out there are fixing the problem quickly by patching it, there are going to be a lot of places that won’t patch, or can’t patch for a period of time. Then you start getting into software that’s end of life, or may not be getting patched.”