The importance of live patching of Linux kernel came to light with the recent Dirty COW vulnerability. The Dirty COW privilege escalation vulnerability shook the Linux world – all the kernels released in the past nine years were affected, and what’s worse, the vulnerability was trivial to exploit.
With exploits appearing in the wild just mere hours after the vulnerability was discovered, system administrators were scrambling to fix the issue. As we suggested in this post last week, you need to get security patches applied to your system as soon as possible and make sure that the operating system is regularly updated with the latest patches.
But this is not a trivial task, especially in companies with thousands of servers running critical business applications and databases, as the kernel upgrade requires a reboot. In enterprise IT, it means waiting for a maintenance window, going through the risk assessment processes, and negotiating with different departments on the acceptable time frame for the maintenance window based on the criticality of the vulnerability. Even just rebooting thousands of servers afterward is not an easy task, as some of the servers tend not to come back up, getting stuck in fsck, and require human intervention.
The necessary maintenance window that causes downtime often leads to delaying the upgrade by months, bringing the company out of compliance – vulnerabilities in old kernels can create huge security holes for an enterprise, and act as an invitation to being hacked.
Recent progress with live kernel patching solves most of those issues, as it removes the need for the reboot, alleviating downtime and human intervention. The technique allows for fixing security issues in the kernel on the fly, saving time and money for the IT departments, while also keeping the software compliant with the best practices of running non-vulnerable, most up-to-date versions.
Here is how live patching works: vendors compile just the change needed (patch) into binary form and safely inject it into the running kernel. The process is somewhat complicated and requires a very skilled kernel developers and advanced testing setup to develop and release patches in a reliable manner.
Several Linux vendors today provide kernel live patching service. Ubuntu provides live patching services for their latest Ubuntu LTS 16.04. SuSE and Oracle Linux provide live patching for their Enterprise Linux offering as well. In each case, it is an add-on service that needs to be purchased.
The offering I find the most versatile and enterprise-ready comes from KernelCare.com, as it supports a broad range of kernels for Linux distributions including RHEL, CentOS, Ubuntu, Debian, and others. The solution has been on the market for two years and supports not only the newest kernels but also the older kernels – something missing from Ubuntu or SuSE offerings. KernelCare is used by thousands of customers, and some of our Enterprise customers use KernelCare quite successfully. Our engineers have also tested KernelCare technology and found it useful in fixing all known vulnerabilities in Linux kernels.
RHEL has live patching service too, but it is not available to general public yet. In addition, their service only supports latest RHEL 7 kernels and doesn’t cover CentOS kernels.