February 10, 2022

When prospective customers ask me how we handle security issues, I explain our Security Philosophy and describe our Seven-Point Security Solution. For many, that’s enough assurance, but others can be more skeptical, wanting to understand exactly why we are confident in our capabilities. To answer that, I’d like to offer the story of how we addressed a recent, notorious vulnerability, and how we were able to provide a response to customers much faster than Oracle could. 

The Vulnerability that Made the News 

On November 24, 2021, researchers discovered a new vulnerability in a key piece of open-source software. On Thursday, December 9, the National Vulnerability Database published the information. When the cybersecurity world woke up the next day, it learned of the zero-day vulnerability (one that is announced but has not yet been patched) and reviewed its attack vectors. It was evident that the scale of the exploitation was going to be astonishingly difficult to comprehend. 

Due to the popularity of the affected software tool and its usage in most enterprise software, this vulnerability sent shockwaves through the software industry. Publishers were in a race against time to patch their products to mitigate the vulnerability. And the saga was far from over: new variants of the exploit started to emerge as well. 

This vulnerability is called Log4Shell (base CVE number of CVE-2021-44228), and you’re likely to have heard of it by now. It’s also widely referred to as “Log4j,” the name of the Apache logging library behind it. Everyone was talking about it, and we published recommendations for JD Edwards users in December.  

What is the Log4j issue? 

Let’s dig a little deeper to understand why this vulnerability is so problematic. Log4j is an open-source java logging library developed by the Apache Foundation. This library is used to log events and messages for development, operational, and security purposes. It’s free, and so it’s widely used in enterprise applications from various vendors including Oracle, SAP, IBM to name a few. So, it’s really out there. 

The vulnerability within Log4j was down to its “message lookup substitution” feature being enabled by default. This feature allows users to specify custom code for formatting a log message. Using this feature based on JNDI, an attacker could execute arbitrary code on a vulnerable system, thus gaining control to the same with a view to perform more intrusive and disruptive actions. Hence this is classified as a Remote Code Execution (RCE) vulnerability.  

To put it simply: Log4j has an extremely low attack complexity coupled with high-impact metrics, which is why this CVE (Common Vulnerability & Exposure) gained the highest possible CVSS score of 10.0 (CRITICAL). The fact that enterprise applications with Log4j are widely deployed in the global software ecosystem means that most companies likely have it in their systems. Any organization that has deployed an enterprise software from one of the major vendors in IT should be concerned about this vulnerability. 

Here are a few good sources if you want to find more on this vulnerability. 

How did we handle it? 

Software publishers are concerned only about their software, so they practice security by throwing “one-size-fits-all” patches (one after the other) at customers. They have no regard to the stability of their complex and potentially customized systems or the rest of the technical stack.  

Spinnaker Support takes a holistic “customer first” view to security. Our mitigations are not only based on industry standards but are developed for the long-term and take existing customizations into consideration. Our global security team always stay ahead of the curve by keeping abreast of emerging vulnerabilities in the cybersecurity landscape.  

As soon as the Log4j vulnerability was published, we immediately started analyzing it and were able to distribute our first draft response within two days. We kept our customers continuously updated with the emerging developments on this vulnerability to provide additional assurance on our service. 

In the case of Log4jshell, there were five separate vulnerabilities (to date) associated with it. The attack vectors for the Log4Shell CVEs have fallen into two categories, either exploitable Java code or Log4 configuration.  

The process for the CVEs targeting the Java Class included steps to find all Log4j-core*.jar files and remove the offending class from each of them. The process for the 3 CVEs for configuration settings was to provide steps to check the configuration settings and remove the non-default configurations.   

In comparison, how did Oracle handle it? 

As mentioned above, Log4j is an open-source library developed by Apache Software Foundation. It is not code that Oracle specifically wrote. Hence, any enterprise software publisher that uses this library should wait for Apache Software Foundation to inform them of the mitigation and/or resolution before they start creating patches and putting them through their generalization and testing. Only then are they released to customers.  

In contrast, Spinnaker Support can move ahead far more quickly. We understand our customer’s security posture for their Oracle software system landscape, and we use those insights to accelerate the development of mitigation measures. We then distribute those measures to customers in thorough and easy-to-follow instructions.  

For this vulnerability, we not only advised our customers on how to locate the affected library and to patch it with corrected versions (2.17 and above), we also extended our advice on how to identify and put preventive measures in place for this vulnerability and its variants. 

Log4jblog

How was our response received? 

Our nimble response to this CVE exemplifies our ability prioritize customer needs. We received praise for the rapid response and informative documentation. One customer commented: “The experience has been very good – Spinnaker are the poster child for documentation, especially on escalation.” 

You should recognize, however, that Log4j is exceptional. Most CVEs are not ranked a CVSS score of 10.0. They receive neither a speedy patch response nor widespread press coverage. That is why we normally deliver a layered, Defense in Depth approach to security.  

Instead of targeting CVEs, we focus on the weakness category, or Common Weakness Enumeration (CWE), providing layered system protection rather than individual product protection. Using a CWE approach means that we don’t typically have to drop what we are doing to address individual CVEs. 

An emphasis on CWEs can also yield faster results. Vendor patches focus on addressing each published CVE by individual product, which means that vulnerabilities in products often go unannounced or patches are not released for extended periods of time. We see this as counterproductive.  

Let’s Talk Security 

Many organizations have now mitigated their Log4j issue, but who knows when another critical vulnerability will appear? At Spinnaker Support, we help our customers to prepare as best as they can through our Customer Risk Review, Attack Surface Reduction, Security Resource Library, and Proactive Security Tooling. 

These security solutions and more come standard with our support, and our mitigations are for all software versions, even those no longer under the Premier Support or Extended Support phases. If you’re considering a move to third-party software support and are concerned about security, reach out for a conversation. We’re glad to discuss the case above, share other examples, or introduce you to our team of global security experts.