The Definitive Guide to Patch and Release Management
Ryan Green and Matthew Flanagan round up their three-part series with a comprehensive overview of patch and release management, outlining deployment considerations and recommending patch schedules.
Patch management and release management are essential activities in IT environments that span the entire infrastructure (firmware) and software solution landscape. Within ITIL best practice, patch management falls under the label of Release Management and is necessary for a number of important reasons, including:
- Bug resolution – Poorly crafted code can at times impact the stability or performance of a product.
- Security vulnerability management – Buggy code sometimes leaves a system vulnerable to being exploited for the purposes of damaging the system or the data it maintains, obtaining remote administrative access, or leaking data.
- Feature enhancement – As vendors improve their products over time, customer who’ve already purchased are generally given access to improvements with firmware or software updates.
- Maintaining vendor support – Vendors often only support a number of the most recent versions or release levels. This is because they generally only manage test environments of a small number of releases and cannot cost effectively release updates to older code.
Start with the right inputs before you assess
Key inputs to a patch management assessment include:
- Reliable asset and configuration data – maintained in a Configuration Management Database (CMDB).
- Identification of systems by risk and criticality – business critical workloads, systems that are exposed to the internet or untrusted networks (i.e. DMZ servers), systems that are logged into by users other than trusted administrators.
- System dependency mappings – systems that are dependent need to be considered in parallel as changes to one system may impact another.
- Centralized log management – gathers logging data from the various systems and infrastructure to ensure it is retained for minimum periods and to make it easier to cross-reference logging data between various systems.
- Status, capacity and performance monitoring data – assists with the review and analysis of both expected or unexpected impacts of patch deployment on the environment.
These inputs require a baseline set of tools for patch management and vulnerability resolution. Without these, the environment must be manually catalogued and the impacts of a vulnerability investigated slowly and reactively. Additional configuration and patch management tooling can be deployed to automate operational tasks in the datacenter, systems and on edge devices, including the rollout of patches. Most importantly, IT operational maintenance policies and procedures are living documents that need to be maintained, robust and unambiguous.
Following the assessment, an implementation project plan can be developed and executed. The implementation plan identifies all of the tasks, order of operations, baseline and post-update testing or benchmarking, necessary tooling, roll back and decision points, outages and business impact assessments, change management, roles and responsibilities, risk management, communications plans, progress reporting, scope and cost management.
Deploy a patch or retain an earlier version?
The best practice is to regularly review posture and keep abreast of updates to minimize risk. Deployed infrastructure and software should be maintained within vendor support to ensure that security fixes are made available when identified and you aren’t left out in the cold. The decision of whether to deploy a patch, retain an earlier version, or do nothing needs to consider two key things:
- Assess the risk – including:
- What is the severity of the vulnerability/bug?
- Are there public exploits available for the vulnerability?
- Do I have systems that are exposed to untrusted networks that are vulnerable?
- What known issues are resolved in or are known to exist in the patches available?
- Interoperability – Inevitably there are components, be they hardware or software, that to operate properly in an integrated fashion, require supported patch-levels on other integrated components. To avoid inadvertently breaking interoperability and creating disruption in the environment, it is important to consider the whole integrated chain. This is the key reason to complete this review prior to making decisions on patching.
If yes, deploy the current patch, N-1 or N-2?
N-1 and N-2 approaches are common, where earlier than current versions are prioritized. These approaches limit the risk of hitting unidentified issues/bugs in new patches, as the earlier patch has likely had more field testing. However, taking this approach does not mean that the N-1 or N-2 patch is always deployed, for example:
- Fixes in the latest patch must be considered, as they may resolve critical issues or risks in the previous patch that cannot be ignored; and
- Often there is no option but to deploy an older or newer patch to satisfy interoperability requirements.
What is the recommended patch schedule?
The following classes of device are described in order of decreasing potential attack surface area, and aligns with the minimum suggested frequency of review and patching:
- Desktop operating system, malware and virus protection, VPN clients and security tooling, client applications – monthly
Client devices are often targeted first as an entry point into a more trusted network. Compromising an end point can be as simple as tricking an end user into executing malicious code with a carefully formed email or phishing for credentials over the phone. Once an endpoint is compromised it can be much easier to gain access to critical servers and systems via lateral movement inside an organisation’s network. Client devices are the most exposed area of the network, particularly in environments where users take devices between the internal network and untrusted networks, for example, public WiFi or home networks.
- Server operating system and applications – monthly
As the operating system and applications running on a server are key points of attack, especially those exposed to untrusted networks such as servers in the DMZ, these should be considered much more regularly. Monthly patching cycles for servers is recommended and in extremely sensitive environments more regular patching should be considered. To lower the overheads of managing this process, automated updating should be considered either internally or via a trusted managed service provider.
- Physical and virtual appliances, hypervisors, and management tooling – quarterly
With the higher complexity, more points of integration, and broader attack surface, it is recommended to review these items more regularly than firmware. It is recommended to prioritize appliances exposed to untrusted environments, such as the internet.
- Infrastructure firmware, drivers and management software – 6 monthly
Updating firmware is often also necessary to maintain vendor support levels. Some vendors do not retain older firmware releases in their labs to expedited support. This means that when you have a problem with an older revision, vendor support’s first request will be that you update to a current revision, potentially slowing resolution times and creating additional unplanned disruption
How can CSA help?
We know how hard it is to stay current with patch and release management – we proactively Monitor + Manage this for hundreds of customers across Australia – and we can help you alleviate the time and effort that is burdened on your internal resources. Our team of experts have the skills and experience to either support your team or to take it off their hands completely via a Managed Service. We offer flexible solutions to suit your requirements, contact us on the below form to talk to one of our experts.
- Article 1 in this series: Dealing with the fallout from Meltdown and Spectre
- Article 3 in this series: The definitive guide to Patch and Release Management
- Meltdown and Spectre: Everything you need to know
- Official Meltdown and Spectre website
About the Authors
Ryan Green is Portfolio Manager – Datacentre & Cloud at CSA and has over 20 years industry experience in a wide range of technologies. With a background supporting, operating (DevOps) and consulting in closed and open source environments, both locally and abroad, the last 10 years have seen him focus on Datacentre and Cloud solutions, Data Management and Disaster Recovery. With habitual interest in keeping abreast of technology, Ryan specialises in identifying complementary technologies, architecting solution offerings.Ryan Green
Matthew Flanagan is Platform Lead – Network and Security at CSA. Matt has over 20 years industry experience in a wide range of technologies focused around Internet and enterprise infrastructures. He specialises in Internet infrastructures, UNIX, security, network architecture and design. Matthew is a Member of the GIAC Advisory Board and holds the GIAC GXPN Certification for Exploit Research and Advanced Penetration Testing.Matthew Flanagan