Transcription
PATCH MANAGEMENT FOR ICSPATCH MANAGEMENT FOR ICSLIFECYCLE AND COMPLIANCEAuthorRoger Rademacher - Cyber Security Solution Architectfoxguardsolutions.comFoxGuard Solutions877.446.4732
PATCH MANAGEMENT FOR ICSINTRODUCTIONBACKGROUNDAfter defining a few criticalitems, this paper will present anumber of common problemsmost organizations experiencewhen establishing a patchmanagement solution. Solutionsto these problems will bepresented later in the paper inan effort to bridge any potentialgaps left by typical patchmanagement programs.In this whitepaper FoxGuard will examine, at a high level, common patch managementpractices and requirements as identified by the National Institute of Standards andTechnology (NIST), Department of Homeland Security (DHS), Industrial Control SystemsCyber Emergency Response Team (ICS-CERT), and North American Electric ReliabilityCorporation (NERC). It is the intent of this whitepaper to provide insight into manychallenges that organizations are faced with and suggest a path forward; taking intoaccount the specific challenges for Industrial Control System (ICS) environments.Patch management is one of several activities which comprise what is collectively referredto as vulnerability management, alongside vulnerability analysis, asset management, andconfiguration management, to name a few. In mature organizations, patch managementmay fall within a comprehensive Vulnerability Management Program, or it may stand on itsown as a discrete actionable process performed by IT or other relevant Operations teams.DEFINING PATCH MANAGEMENTPATCH MANAGEMENTLIFECYCLEThe image below shows patchmanagement as a cyclicalprocess defined by individualphases.Before discussing the challenges of patch management we need to define two criticalpieces of the puzzle: patch management and patch source. The idea of a patch sourceis of particular importance as it establishes critical compliance timelines and impact tovalidation processes.Patch management may assume many definitions when viewed from within a specificstandard or regulatory source. FoxGuard has developed its own definition of patchmanagement based on the NIST and NERC definitions provided below.NIST identifies patch management as “the process for identifying, acquiring,installing, and verifying patches for products and systems.”NERC CIP-007-5 R2.1 states Responsible Entities for High and Medium impact BESCyber Systems are required to implement a “process for tracking, evaluating, andinstalling cyber security patches for applicable Cyber Assets.”FoxGuard considers patch management to be “the act or practice of detecting, assessing,acquiring, validating, deploying and tracking updates to software in order to mitigate asecurity vulnerability.”A comprehensive definition takes into account the cyclical nature of patch managementand organizes the process into easily definable phases for individual processdevelopment. (See figure in column.)One additional process has been added: the need to have an aggregate softwarebaseline. Patch management must begin with an accurate account of what needs to bemonitored.foxguardsolutions.comFoxGuard Solutions877.446.4732
PATCH MANAGEMENT FOR ICSPATCH SOURCE DEFINITIONWho is the patch source? Is it the OEM, the Systems Integrator, the Vendor, or a service partner?NERC CIP-007 spells out some basic requirements.NERC CIP-007-5 R2.1NERC CIP-007-5 R2.1 states Responsible Entities for Highand Medium impact BES Cyber Systems are required toidentify the “source or sources that the Responsible Entitytracks for the release of cyber security patches for applicableCyber Assets.” Sources may be monitored at either theindividual BES Cyber Asset or BES Cyber System basis.This distinction is important when considering validation requirements andservice agreements for BES Cyber Assets. Either assets or systems mayfall under limiting service agreements which disallow updates that are notpreviously validated as part of a service agreement. As a result, patch sourcesfor updates may need to be shifted from the software manufacturer to aservice provider.NERC CIP-007-5 R2.2According to NERC CIP-007-5 R2.2 an organization must,“At least once every 35 calendar days, evaluate securitypatches for applicability that have been released since thelast evaluation from the source or sources.”Each source represents an independent window with whichto evaluate the applicability of security patches. Thus, thefewer sources to monitor the simpler it will be to maintain compliance whiletracking the evaluation and applicability of security patches.foxguardsolutions.comFoxGuard Solutions877.446.4732
PATCH MANAGEMENT FOR ICSPROBLEMSEach phase/process within patch management has its own challenges and some impact more than one area. The following challengesrepresent some of the major problems that ICS organizations are facing when implementing patch management programs.SOFTWARE BASELINESIn the spirit of starting with and ending with accuratebaselines we focus on asset management. The variousversions of software, whether applications or firmware, andconfigurations on each device establish its individual softwareand configuration baseline and roll up to an aggregatesoftware baseline when combined with all other devices.An aggregate software baseline represents the total uniquelist of software versions which need to be monitored forchanges on a regular cycle (every 35 days per NERC CIP).The difficulty here is establishing what software versions areinstalled on each asset and how to translate this informationinto an aggregate representation that minimizes managementoverhead.An organization should expect a modest amount of manualeffort when maintaining an accurate asset list whichincludes ICS equipment and other equipment that does notinteroperate with automated solutions. Translating softwareversions into aggregate software baselines is difficult andrequires some level of familiarity with software suites andvendor equipment. Version control and source consolidationrequire aggregation beyond common platform enumerationstandards.foxguardsolutions.comUPDATE IDENTIFICATIONOne sad truth we must accept is that proactive vendorsupport for security updates is not a common servicetrait. Each vendor has their own established processes fornotifying customers when an update is available; some donot notify at all and, instead, put the responsibility onto thecustomer to check public or private websites for updates. Itis considerably more difficult and time consuming to trackupdates when vendors have no online presence and rely onemail, phone, or post office communications to respond toupdate inquiries.Of particular difficulty are integrators who limit updateavailability through service agreements containing validationrequirements. The same Microsoft update may be availablefor multiple systems but those systems which fall under aservice agreement may be disallowed from applying theupdate. The logistics of tracking vendor requests andresponses and applicability of updates to individual assets canbecome overwhelming in large heterogeneous environmentsas aggregate software baselines reach into the hundreds ofitems and across multiple vendors and service providers.Regardless of how organizations become aware of softwareupdates, each item has an associated “mining window” whichindicates the time necessary to determine if an update isavailable. The more proactive and willing a vendor is torelease update information publically, the shorter this miningperiod tends to be. Short mining windows better enable anorganization to regularly discover updates within establishedcompliance periods.FoxGuard Solutions877.446.4732
PATCH MANAGEMENT FOR ICSMANUAL VS AUTOMATED TOOLSThe decision to automate any aspect of patch managementis fraught with complications. Asset management is difficultto automate when an environment includes non-prevalentequipment or devices that do not support standard querymethods.PATCH VS MITIGATE (PRIORITIZATION)NERC CIP-007-5 R2.3 states:“For applicable patches identified in Part 2.2, within 35calendar days of the evaluation completion, take one of thefollowing actions: Apply the applicable patches; orAutomated tools that look for and deploy updates may notrecognize industrial equipment, resulting in little to no benefitin highly industrialized environments; or it might provide an80% solution to discover, track, and deploy updates. Thechallenge is in solving the Patch Gap between automatic andmanual update discovery, acquisition and deployment. Create a dated mitigation plan; or Revise an existing mitigation plan.”As noted in the patch management lifecycle, updates shouldbe validated prior to being deployed; but not all patchesmay need to be deployed or may be allowed to be deployed.Patching may fully mitigate the security vulnerability but othermitigation techniques may lower the risk to an acceptablelevel by considering temporal or environmental factors. Still,some updates may invalidate system operation by reducingsystem performance or breaking critical services.Instances identified above where an update is known butdisallowed cause havoc when determining priority as well.The application of critical security updates may need to bedelayed through the use of other mitigation methods untilboth the service provider’s and organization’s validationtesting has been completed. Updating software may notbe the most reasonable approach given time constraintsor allowances, and mitigation plans allow organizations toreduce risk in the interim.foxguardsolutions.comFoxGuard Solutions877.446.4732
PATCH MANAGEMENT FOR ICSSOLUTIONComprehensive patch management solutions consider all aspects of the patch management lifecycle, including several other aspectsof vulnerability management and compliance requirements.BASELINE AGGREGATIONIt is near impossible to achieve compliance with standardssuch as NERC CIP without a comprehensive list of assetsand a corresponding Master Asset List (MAL) that representsan aggregate software baseline. The key to aggregatingsoftware baselines is proper common platform enumerationin the form of Vendor-Product-Version-Source (VPVS). Thisdiffers from Vendor-Product-Version (VPV) based namingstandards by providing a flexible approval source whichaccounts for service provider agreement requirements.The ability to account for service provider agreementsallows an organization to concentrate efforts on validatedupdates. This is important when availability, integrity, andreliability hold much higher value than confidentiality andother security principals. It may be more critical to safelymaintain services than preventing disclosure of information.The option to ignore manufacturer updates and concentrateon service provider approved updates allows an organizationto focus MALs according to service agreements.The monitoring of manufacturer updates along with serviceprovider approved updates presents a unique opportunityto evaluate the effectiveness of service agreements thatinclude validation requirements. Imagine an integratorthat has put restrictions on firmware updates across anoperational environment. This may prevent an organizationfrom updating unless the update has been validated. Havingknowledge of when an update is released by a manufacturerand subsequently released by a service provider enables anorganization to gauge the efficiency and value of serviceagreements that require service provider validation.SOFTWARE UPDATE MONITORINGComprehensive MALs may be used to provide the inventoryfor software update monitoring. Each item has its ownmining and mitigation windows and must be monitored ona consistent schedule in order to efficiently organize updatediscovery, evaluation and mitigation. Standards such as NERCCIP establish tolerances for patch management programsincluding maximum mining and mitigation windows.Some vendors or service providers may be proactive whenreporting the availability of updates; fewer are proactive inreporting a lack of updates. The burden to determine thepresence of lack of updates for every MAL item within anymining window lies with the customer. This may result inhundreds or thousands of individual mining windows that maybe audited.Checking for updates is tedious and requires a combinationof methods including: calling, email, reading through releasenotes, RSS feeds, online repositories, databases, and otherupdate information sources. Some of these sources providemore detail than others.SECURITY OR NON-SECURITY UPDATENot all patch sources are forthright about the security-relatedaspects of an update. A full spectrum of information isavailable when reviewing update information sources. Somesources spell out every change and others provide no detailsat all. It may be safe to say that when a patch source labelsan update as security-related we may assume the same; butwhat about updates that are not provided with a classification?It is prudent to read through release notes, query the vendor,and dig through documentation to determine if an updateimpacts security in any way. It may be the case that anupdate provides increased security in the form of strongerauthentication or increased availability. When a patch sourcedoes not classify Security vs Non-Security, someone has tomake the call.foxguardsolutions.comFoxGuard Solutions877.446.4732
PATCH MANAGEMENT FOR ICSUPDATE REPOSITORIESTrusting online file stores from vendors or third parties maybe difficult. The act of hashing files to assist with integrityverification is not a common practice, and weak hashingalgorithms present little challenge for dedicated attackers.Every file downloaded from the Internet, public or private,needs to be verified as authentic and harmless prior to enteringvalidation and deployment processes. An organization shouldscan, hash, and stage update files to an approved updaterepository.DEPLOYMENTAutomated and manual deployment methodologies shouldbe refined into a deployment package during validation. Thedeployment package may consist of the update file(s) andinstallation directions in the case of manual deployments orit may be pulled into an automated solution that tracks anddistributes the application of the deployment package acrossall applicable assets. Many automated platforms allow forcustom content and are flexible enough to deploy and verifyunattended installations to supported assets.An organization can be assured that when pulling files from anapproved update repository into validation environments, thosefiles do not contain known malicious code. Extensive testingserves to mitigate other factors such as bugs and unexpectedbehavior which pose risks to operational environments.Automated deployment methods do not alleviate the need totrack the application of updates to applicable assets. Basictracking should provide, at a minimum, a list of which assetsneed which updates. Historical inference of delta lists willindicate update progress but most automated solutions willprovide reports and logs with more specific information onsuccesses and failures to deploy update packages. Thepresence of non-standards-based assets will preclude theuse of most automated tools, and manual tracking will berequired for these assets.VALIDATIONValidation typically uses non-primary assets and sets thestage for deployment. The goal of validation is to ensure thatupdates are safe and do not impact operational environmentsadversely. In rare instances, validation may need to beperformed against primary assets; extreme care should betaken when limited test assets are available.Regardless of the validation environment used, all validationshould be done against representative assets. As an example,if there are 10 HMIs with duplicate software baselines asindicated by the asset inventory and MAL, only one need betested. Specific testing methodologies and wait periods foranomalous behavior should be defined and vary according toorganizational policies. Validation serves to verify configurationbaselines, update disaster recovery media, and prepare fordeployment into production environments.Software updates may include new features, new services, ornew ports and protocols to pay attention to. Configurationbaselines may need to be updated to include new options.Access lists and firewall rules may need to be altered to allow ordisallow new protocols or communication requirements. Builddocumentation may need to be updated along with disasterrecovery and continuity of operations plans. It is advisable toinstall as few updates as possible during validation to ensurechanges are attributed to the appropriate software. Validationserves to ensure that updates are safe, documented, approved,and ready for deployment.foxguardsolutions.comTRACKINGUpdates must be identified, evaluated for applicability,acquired, validated and deployed or mitigated. This requirescomprehensive tracking from end-to-end. Without propertracking it is near impossible to determine when compliancewindows start and end and, thus, compliance is difficult todetermine.Adopting aggregate software baselines will minimize thenumber of timelines which need to be monitored andminimizing patch sources will reduce the variance in timelinesto make reporting and tracking easier. It is much simplerto prioritize patch management efforts when timelines arealigned and there is a clear understanding of the impacts ofeach update across a production operational environment.FoxGuard Solutions877.446.4732
PATCH MANAGEMENT FOR ICSCONCLUSIONProper tracking of software updates from availability through applicability, acquisition, validation and deployment allows an organizationto maintain full visibility of patch management and enables other management processes such as asset management, configurationmanagement, and vulnerability management to succeed. Missteps during the patch management lifecycle can snowball into noncompliance and lead to vulnerable systems and unexpected downtime.Consider the following questions: What level of commitment are you receiving from your vendors to support compliance requirements? Are your vendors providing you with timely information on a consistent schedule? Are you scrambling to ensure that you have detected and assessed each update appropriately? Would you rather have a third-party track and monitor updates as a single patch source or invest in developing in-houseexpertise?It is the goal of FoxGuard to establish sound patch management solutions and enable customers to manage vulnerabilities to a degreethey are comfortable with.FOXGUARDWHY YOU SHOULD PARTNER WITH FOXGUARDFoxGuard has proven excellence in not only meeting compliance requirements but also solving functional issues and securityvulnerabilities. FoxGuard has deployed patching solutions in over 150 sites, in 15 countries over the past 10 years. Moreover, for afraction of the price and time it takes to monitor patches, FoxGuard can deliver an automated patch management solution, customizedfor your operation.LEARN MORE ABOUT OUR PATCH MANAGEMENT SOLUTIONSFoxGuard provides a wide range of patch management solutions that help entities identify and mitigate gaps in the security oftheir systems and prepare for NERC CIP audits. We host weekly webinar to discuss ways to develop and implement a robust patchmanagement program.To reserve a spot, please visit us ebinarIf you would like our security experts to contact you, thenkindly share your contact details and a brief summary ofyour challenges ES NIST 800-40r3, Guide to Enterprise Patch Management Technologies ICS CERT, Recommended Practice for Patch Management of Control Systems NERC CIP-007-5, Cyber Security – System Security Managementfoxguardsolutions.comFoxGuard Solutions877.446.4732
Patch management is one of several activities which comprise what is collectively referred to as vulnerability management, alongside vulnerability analysis, asset management, and . MANUAL VS AUTOMATED TOOLS The decision to automate any aspect of patch management is fraught with complications. Asset management is difficult
