Overview
There has always been a healthy tension between the cyber security team and business application owners when it comes to patching.
On one hand, cyber security is charged with increasing the security posture of an organization, and that means patching early and often. This patching often requires maintenance windows and downtime. On the other hand, business application owners need to maximize the uptime for their applications to keep the business running.
Why not have both security and availability? SUSE Kernel Live Patching (KLP) offers a solution to this age-old push and pull.
Introduction
The recent XZ Utils Backdoor (CVE-2024-3094) discovery reminds us that Linux Engineers and IT teams must stay vigilant with identifying and addressing vulnerabilities, but not all organizations are.
According to a recent study on the State of Enterprise Linux Security Management by the Ponemon® Institute, patching is not always done in a timely manner once a critical or high-priority vulnerability is detected. In fact, only 29% of organizations reported they can patch vulnerabilities on average within two weeks, with only 44% of all organizations patching vulnerabilities within a month. The majority of organizations, 56%, said it can take an average of five weeks to more than a year!
There are a few reasons why patching velocity might be slower than organizations would like. This could include lack of priority, tools, or process within an IT organization. Another common reason: Organizations can’t afford the downtime. This is where live patching can help.
Benefits of Live Patching
SUSE Linux Enterprise Live Patching provides a stream of packages to update a running kernel without interruption. With this approach, you can perform OS patching without rebooting your system, saving the cost of downtime and increasing service availability for enterprise applications.
Here at apiphani, we have incorporated live patching into our enterprise patching process automation. Our clients benefit from the increased security that comes from a regular, automated patching approach, plus the added benefit of reduced downtime for enterprise applications we manage, like SAP.
In one example, apiphani was able to reduce downtime due to patching of our client’s SAP and other enterprise applications by close to 70%. This was done by leveraging live patching for two of the three monthly patch windows in a quarter. This is tangible value that IT has been able to provide to the business without sacrificing security.
Live patching also allows apiphani and our clients to be more responsive to critical vulnerabilities. With live patching, we can apply critical updates shortly after they are made available. Because there is no required downtime, making changes through live patching does not need a high level of coordination with the business.
A Linux Engineer’s Perspective
Every passing day, information technology evolves to bring out solutions that we never thought possible. One such solution for Linux Engineers like me is Kernel Live Patching.
Throughout my career, we always used a traditional approach to patching our clients’ Linux environments. This was a time-consuming process and always involved a mandatory final step of rebooting the servers. This means downtime and loss of business activity during maintenance windows.
Thanks to SUSE Kernel Live Patching, we’re now able to patch our client’s servers without shutting down applications and rebooting.
The Details of Kernel Live Patching?
Kernel live patches are delivered as packages with modified code that are separate from the main kernel package. The live patches are cumulative, so the latest patch contains all fixes from the previous ones for the kernel package. Each kernel live package is tied to the exact kernel revision for which it is issued. The live patch package version number increases with every additional fix.
Apiphani uses SUSEManager to manage and patch our customer’s SUSE Linux landscape. With SUSEManager, we can enable Kernel Live Patching module on each of the clients’ systems and let SUSEManager take care of the details. That means SUSEManager will automatically download and install the latest live-patches on a system as soon as they become available, or, alternatively, it can be set to push the live patches according to user-defined schedule.
Kernel Live Patches can also be easily applied via command line, verifying the install with the command “klp -v patches,” as depicted in snippet below:
One point to note, an extra license is needed to implement Kernel Live Patch on a system, and applying Kernel Live Patching does not mean that a server never needs to be rebooted. In fact, it is highly recommended to reboot a server once every few patching cycles in order to boot the system into fully new kernel version.
Here at apiphani, we reboot our client’s systems every quarter with a fully new kernel version in order to minimize the risk of critical faults due to software issues. It just makes sense to not leave a systems completely dependent on Live Patching. Plainly put, Kernel Live Patching is not an alternative way to patch your systems; the purpose of Kernel Live Patching is to minimize downtime for business-critical applications.
Conclusion
Kernel Live Patching allows you to patch your Linux landscape without rebooting, which means you do not need to bring down your SAP or other mission-critical applications, avoid downtime.
This approach:
- Reduces security risk by applying Linux OS patches as they become available rather than when a maintenance window can be established. This enables organizations to respond to critical vulnerabilities more quickly.
- Increases availability of enterprise applications by reducing the downtime due to application shutdown and system reboot, by over two thirds with one of apiphani’s clients.
Increasing security or availability is no longer choice organizations must make. By taking advantage of automated and live patching techniques, organizations can better balance their critical cyber security needs while also keeping the business running by minimizing downtime of enterprise applications.
