In early 2019, Pulse Secure released a patch for a VPN server vulnerability.
The company contacted customers by phone, email, in-product alerts, and online notifications to remind them to install the patch, but this past January the Department of Homeland Security's Cybersecurity and Infrastructure Security Agency released an alert that said it was seeing "wide exploitation" of the vulnerability.
CISA issued another alert in April about how far attackers were spreading through government and commercial networks after exploiting this vulnerability. Despite these efforts to inform organizations about the threat, some companies failed to patch. In fact, earlier this month a hacker leaked usernames and passwords for more than 900 Pulse Secure VPN servers.
What's going on here? Why are companies having so much trouble with such a basic part of cybersecurity hygiene, patching?
According to research released late last year by the Ponemon Institute, 60% of breaches that occurred over the previous two years could have been prevented if a patch had been applied in time. But only 32% of the respondents said that they were able to reduce the time it takes to patch critical vulnerabilities compared to the previous year -- white 36% said it actually took longer than it did before.
The problem of patch management is so big that organizations can run into "patch fatigue," said Angelos Keromytis, a professor at Georgia Institute of Technology focusing on systems and network security, and co-founder of cybersecurity firm Allure Security Technology.
"When you run a large data center, there's a limit to how much you can keep an eye on," he told Data Center Knowledge.
In addition, sometimes people think that the patches don't apply to them.
"Because of how they've configured networks and applications, many entities believe they're isolated, so they think they're not as vulnerable," he said. "There are a bunch of different rationalizations for not patching."
Data Centers Particularly Vulnerable
Patching is a problem for most large organizations, but data centers have particular challenges. These include potential business disruptions due to downtime, disruptions due to unforeseen consequences, and a lack of awareness of vulnerable systems.
According to Greg Touhill, president of AppGate Federal Group -- the cybersecurity firm spun off by data center provider Cyxtera last year -- and the country's first Chief Information Security Officer during the Obama administration, data center managers are not always aware of where the vulnerabilities are.
First, there's the thorny problem of shared responsibility. If the data center is in a colocation facility, or in the cloud, the provider will handle some but not all of the security tasks. From the providers' side, there's only so much they can do with customers' infrastructure.
"The actual maintenance and configuration of the equipment in the rack is often still retained by the customer," Touhill told DCK.
Even when the areas of responsibility are clear, there's the challenge of learning about the vulnerabilities that need patching.
"When I was in the Air Force, we would have our own red teams go out and use commercial off-the-shelf tools that potential hackers and adversaries would use, scanning tools, and we would scan from the inside and scan from the outside," Touhill said. "And we would use third-party pen testing that would do logical and physical pen tests."
Hackers often discover vulnerable, unpatched systems by scanning the Web -- or by gaining entrance to an environment and then scanning internal networks to find opportunities to spread further.
The Trouble With Downtime
The biggest problem of all for data centers is that patching can cause downtime.
Customers expect their data centers to be up 100% of the time, Josh Axelrod, a cybersecurity leader at Ernst & Young, told DCK.
As a result, patching might have to take place during very limited timeframes, when it will be least disruptive. That means it can take a long time to get around to patching some systems. One solution to this problem could come from the disaster recovery space. If some of a data center's infrastructure goes down, a good disaster recovery system would automatically shift the affected workloads to a backup environment. That same system could, in theory, be used for patching.
"Customers can see their workloads instantly rolled over to a clean environment where the patch has been applied," he said.
Although this kind of infrastructure is expensive to set up and isn't universally available, Axelrod noted, "That's where we need to get to."
Another major problem with patching data center systems is that the patches might break critical business operations.
"You don't know how that update is going to affect the applications being run," Keith Mularski, managing director for the cybersecurity practice at Ernst & Young, told us.
Running an auto-update on an endpoint computer can take down one employee's productivity if things go wrong. An auto-update on a server operating system could cause a whole company to shut down.
"You have to test and make sure the patch isn't impacting your business," he said.
One approach is to use development environments that an organization might already have in place for application testing to check for potential problems after a patch is applied, but that would require some coordination between data center managers and application development teams.
Automate, outsource, or move to the cloud
Vikas Shah, VP of IT at AArete, a management consulting firm, told us that if a data center doesn't have the resources to get a handle on the patching problem, there are some options. One would be to invest in testing environments to reduce patching complexity and downtime, and in automation technology. Microsoft, for example, offers Windows Server Update Services and the Systems Center Configuration Manager.
"Both of these products offer great solutions but are built for large organizations with dedicated teams," he said.
Other tools include the Solarwinds Patch Manager, LANDesk Patch Manager, ManageEngine Patch Manager Plus, and Ivanti Windows Patch.
"Most of these tools can work independently as well as can integrate with [Microsoft's] WSUS or SCCMs," he said, adding that there are also server providers that can handle the job, including Rackspace, Connection, Cognizant.
Finally, Shah suggested that enterprises consider moving their data centers from on-premises to the cloud. Moving to the cloud doesn't absolve the company from all responsibility for its own security, but cloud vendors do handle the patching requirements related to the underlying infrastructure.