Transcription
State of EnterpriseVulnerabilityDetection and PatchManagement
Making Open Source Linux Enterprise GradeFor maximum security and compliance, enterprises need to rapidly patchvulnerabilities, keep production Linux systems updated with the latest fixes, andhave a trusted technology partner for Linux support & maintenance always withinreach. TuxCare ticks all the boxes by helping organizations to take care ofsupport, maintenance, and security for Enterprise Linux systems.Our ServicesSupport, Maintain and Secure All CriticalComponents of Enterprise Linux SystemsLIVE PATCHING SERVICESPut an end to serviceinterruptions & non-compliancecaused by system rebootsEND-OF-LIFE LINUX SUPPORTSERVICESEliminate security vulnerabilitieswhile running End-of-Life LinuxLINUX SUPPORT SERVICESKeep all components of the productionLinux systems always up-to-date withvendor-level support servicesALL TUXCARE SERVICES INCLUDE INTEGRATIONS WITHPATCH MANAGEMENT TOOLS, VULNERABILITY SCANNERS,EPORTAL SECURE PATCH SERVER AND 24/7 SUPPORT.
ContentsExecutive summary3Introduction4Linux ecosystem diversity6Ideal vulnerability scanner and patchmanagement tool7Logging as a desired feature7Resource use and patch rollout8Vulnerability Management9Responding to a vulnerability10Automation11Maintenance and Patch deployment12Preparing a patch in-house13Downtime to deploy patches14Staff time consumed by patching tasks15When patching is delayed16Improvements17Conclusion18State of Enterprise Vulnerability Detection and Patch Management2
Executive summaryThe cybersecurity threat is broad, pervasive, and complex. Mitigatingsuch a comprehensive threat requires the use of a multi-prongedapproach. Technology teams must maximize the utility of every tool inthe security arsenal if they stand a chance to mount an effectivedefense. There are, however, different approaches to the use ofspecific tools.This survey looks at two critical cybersecurity tools. First, we examine the vulnerability detection toolsused to locate and map software vulnerabilities in an organization. We also look at patchmanagement, the process of closing vulnerabilities through software patches.Our intent is to analyze how different organizations treat these two key tenets of cybersecurity. Wewill outline the key variations we found from industry to industry – as well as aggregate statistics thatpoint to gaps in preparedness such resource limits and the reliance on manual methods.State of Enterprise Vulnerability Detection and Patch Management3
IntroductionVulnerability detection and patchmanagement are two importantoperational steps that underpin securetechnology infrastructure in everyindustry. With this study, we present anindustry-by-industry analysis to discoverhow different industries structurevulnerability detection and patchmanagement operations.We focus on three key factors:maintenance windows, patch deploymentpractices, and the overall securityawareness of the technology professionalsthat implement these processes. Resultswere collected in the first quarter of 2021.Respondents completed an online surveythat was publicly advertised to ITprofessionals around the globe.Interestingly, the geographic location of arespondent had no bearing on the surveyresults as varying locations reported similarpatterns across our questions, including ourquestions around the time spent and thefrequency of patch deployment andmaintenance operations. However, theindustry in which an organization operates didhave a clear impact on the question results.At the time of writing, we receivedresponses from 106 subjects, however, thesurvey is still running, and we will continueto update the results to reflect any changesin the findings. As it stands, we areconfident that the results provide helpfulinsights to the teams faced with thesechallenges.IT professionals that work in a systemsadministration role represent the largestnumber of respondents, while in aggregate,88.7% of our respondents were directlyinvolved in vulnerability managementoperations.State of Enterprise Vulnerability Detection and Patch Management4
Our analysis revealed the following key points:The majority of companies (76%) aredeploying automated patchingprocedures.75% of respondents said that theyrelied on manual online research as oneof their tools to find out more aboutdangerous vulnerabilities, making thisthe most commonly used tool.6.25%Sysadmin7.81%DevOps Professional7.81%Network Administrator42.19%IT Security ManagerChief InformationOfficerNot an IT staff17.19%18.72%Most respondents said that CentOSitself, or another CentOS fork, is theirpredominant server OS.73% of respondents said thattheir server fleets use just oneOS, with just 27% suggestingthat they use a mix ofoperating systems in theirserver fleets.Across industries, documenting thepatching process is not consuming asignificant amount of time whencompared to other patching-relatedtasks – in fact, documenting thepatching process consumes the leastamount of time.In some industries, obtaining approvalfor a maintenance window can be themost time- consuming element of thepatching process – in some casesconsuming more time than applying,documenting, or monitoring patching.RESPONDENT ROLE23%18%13%13%9%10%8%5%20,000 or 9100-499Less than 50COMPANY SIZERespondents worked at companies of a variety of sizes. Small,medium, and large companies are all represented in the survey results.State of Enterprise Vulnerability Detection and Patch Management5
Linux ecosystem diversityMost of our respondents reported that they used a single Linux distribution for their serverfleets, though a significant minority of respondents used multiple Linux distributions in theirserver deployments.There are pros and cons to each approach. Using a single Linux distribution for all server rolescarries benefits for server management by reducing the efforts required in managing servers andby simplifying the application of automation tools. On the other hand, choosing to use a rolespecific Linux distribution for different server roles such as a web server, file server orauthentication provider allows organizations to fully exploit the strengths of each distribution.Debian8CloudLinux OS26Other18Ubuntu38CentOS fork52Oracle Enterprise Linux15Red Hat Enterprise Linux28LINUX DISTRIBUTIONSThe “other” option resulted invarious entries including SUSE,Proxmox, Raspbian or Arch Linux.27%We found it revealing that singledistribution fleets dominate thelandscape. That suggests that thestandardization of procedures is animportant advantage for organizations.In our results, we found that thepreference for single distribution fleets isconsistent across industries, andcompany sizes.State of Enterprise Vulnerability Detection and Patch ManagementSingle DistributionMultipleDistributions73%COMPANIES WITH MULTIPLE ORSINGLE DISTRIBUTIONS FLEETS6
Ideal vulnerability scanner and patch managementtoolVulnerability management toolsare relatively complex andtypically carry a steep learningcurve while features andcapabilities vary from product toproduct. We asked respondentswhat features they would like tosee in their ideal vulnerabilityscanner and patch managementtool.Responses varied, with therespondents selecting nearlyequally from the available options,while a significant number ofrespondents opted to suggest afeature under “other”. Thesepreferences included “logging”,“minimal impact on systemresources”, “phased rollouts” and“detection of backported fixes”.IDEAL VULNERABILITYMANAGEMENT TOOL FEATURES87.5%75%69.79%69.79%6.25%Fast response to Live patching ofnew CVEsall componentswithout gging as a desired featureIt is not difficult to see why logging is mentioned. There is a need for transparency with respectto the inner workings of a tool. After all, security tools are directly touching organizationalsystems. What happens under the hood matters.Logging is also important because the user interface of a vulnerability management tool can hidethe underlying complexity of the tool. When a vulnerability management tool runs it will typicallyrun several complex, involved scripts that log into an OS or into applications to detect existingpackages, to check versions, and to test against known exploits.This automated script generates data that gives the user a bird’s eye view of key information viareports or a security operations center (SOC) dashboard. Yes, automation is helpful for day-today security operations activities, but automation can obstruct debugging steps.Current logging implementations sometimes provide so much information that it becomesoverwhelming to process and therefore obstructive. For other tools, the logging data providestoo little information to be of value.State of Enterprise Vulnerability Detection and Patch Management7
Resource use and patch rolloutRespondents that pointed to “minimal impact on system resources” reflected the fact thattoday’s tools can have a noticeable impact on system responsiveness or throughput. That isbecause of the characteristics of today’s tools: most work either through an agent deployed onevery server or via remote calls into existing daemons. The frequency of tests and the volumeof information gathered can easily cause a drain on resources.Another feature requested by respondents is “phased rollouts”. In other words, respondentswanted more granular control over patching so that servers could be grouped, and so thatpatches could be deployed to selected servers in order to test patches for wider distribution.It adds an essential extra stage in patch management – stepping patch deployment fromdevelopment to quality assurance, and then on to production. It is possible to set up selectivedeployment with current tools, but it is a time-consuming and manual exercise.Finally, the respondent that specified “detection of backported fixes” referred to the ability of asecurity scanner to detect vulnerability fixes that have already been applied, and to do so in away that does not rely purely on checking the version string.Detection of backported fixes matters because some patching mechanism will apply patchesto the affected code, without updating the version number. That can lead to false positives if apatching tool only relies on version numbers.State of Enterprise Vulnerability Detection and Patch Management8
Vulnerability ManagementAwareness is critical when dealing with vulnerabilities and it doesn’t matter how an organization gainsawareness: through online information, through vulnerability scanning, or thanks to internal teams.Understanding which systems are vulnerable is always the first step towards protecting those systemsbecause it is only possible to protect against danger when you’re aware of that danger.75.00%64.13%60.87%39.13%19.57%3%Information online aboutthe software versionsAutomated vulnerability scansManual vulnerability scansDedicated Security team (Blueand Red Operations team)The hacker leaves anote in your systemOtherVULNERABILITY AWARENESSFor our question on vulnerability awareness respondents were allowed to select multipleoptions. We found it particularly interesting that nearly 20% of respondents found out abouta vulnerability due to a note from a hacker on their system – which is a clear and obviousindicator of a successful breach.It is notable that online research is the most common source for vulnerability information asreported by our respondents. Given the growth in CVEs, the manual nature of onlineresearch may not remain an effective way to manage vulnerabilities in the long run.Vulnerability scans performed on an organization’s systems still remain an important way togain awareness about security flaws – whether these are performed manually or automatically.Using a dedicated security team to assess vulnerabilities is less widespread and possiblypoints to a lack of resources rather than a genuine lack of interest in appointing a dedicatedteam.State of Enterprise Vulnerability Detection and Patch Management9
Responding to a vulnerabilityIf a vulnerability is detected itmust be dealt with and, ifpossible, resolved. This is howrespondents said that theyhandled vulnerabilities.HOW DO YOU COPE WITHDETECTED VULNERABILITIES?73.91%Emergency maintenance windows isarguably the most disruptivemechanism, but it was nonethelessthe preferred choice when dealingwith a known vulnerability. It cantherefore be suggested that, for over70% of respondents, the risksassociated with a potential securitybreach outweighs operational andavailability considerations.The only non-disruptive mitigationmethod used, live patching, ischosen by nearly half ofrespondents. Interestingly, manyrespondents replied that they copewith vulnerabilities simply by waitingfor the next periodic maintenancewindow. This, in turn, implies thattheir systems will remain vulnerableduring the waiting period.State of Enterprise Vulnerability Detection and Patch Management48.91%46.74%44.57%1.09%Emergency maintenancewindowsWait for the next periodicmaintenance windowUse automatic patching toolOtherUse live patching tool10
AutomationThe response to the discovery of avulnerability can be actioned eithermanually or automatically. Thereare several steps involved in aresponse and it is interesting tolook at the steps that are mostcommonly automated – and thesteps that are least likely to beautomated.AUTOMATED STEPS76.19%61.90%30.95%21.43%Automating the patching processis clearly widespread whilereporting is also commonlyautomated. However, assigningtasks and prioritizing patches aresteps that are much less likely tobe sationNoneSurprisingly, despite the high numbers of CVEs that are filed, and the overall growth incybersecurity threats, a number of respondents reported that their organizations do notautomate any of the steps involved in the vulnerability management process.It is common for organizations to combine automated tasks – for example, combiningpatching and reporting. We asked respondents to report whether they automated multipleprocesses:NUMBER OF COMPANIES PERAUTOMATED TASKSPatching2412Patching, Reporting7ReportingPrioritisation, Assignment,Reporting, Patching6Assignment, Reporting,Prioritisation3Patching, Assignment, Reporting3Patching, Prioritisation3Reporting, Prioritisation2Assignment, Reporting2Prioritisation, Patching,Assignment1State of Enterprise Vulnerability Detection and Patch ManagementMost companies are onlyautomating the patching processand reporting is the mostcommonly automated function –with many companies automatingboth patching and reporting.Compliance law is increasinglydemanding evidence of patching,so it stands to reason that we willsee more and more reportingautomation.In our survey results we also found thatsmaller IT teams – less than twentyteam members – reported a higherreliance on reporting automation. Infact, small team’s reliance on reportingautomation outweighed the reliance onautomation of larger teams by a factorof 6 to 1.Reporting is the only automated taskwhere there was a significant differencein the behavior of small teams vs. largeteams.11
Maintenance and PatchdeploymentDetecting vulnerabilities in systems requires teamsto perform monitoring and testing tasks thatprobe their systems. The amount of time spentdoing so varies dramatically from one industry toanother.HOURS PERFORMING MONITORING TASKSAND LOOKING FOR VULNERABILITIES PERWEEKTechnology25Banking & Financial Services8Data Infrastructure, Telecom7Retail / Ecommerce3Professional Services (Law,Consulting, etc.)3Healthcare2Industrials (Manufacturing,Construction, etc.)2Education2Transport, Logistics2Agriculture, Forestry, Mining2Other2Media, Creative Industries2Public or Social Services1Compared to other industries, organizations inthe technology sector clearly spent much moretime actively searching for vulnerabilities –perhaps because tech firms are more familiarwith the risks.State of Enterprise Vulnerability Detection and Patch Management12
Preparing a patch in-houseWhen deploying patches for a vulnerability onekey part of the process is obtaining the rightpatch for the affected system or application. Itcan happen that an official or vendor patch hasnot been released, and that the only availableinformation covers the exploit itself with noadvice on how to mitigate.HAVE YOU EVER PREPARED APATCH IN-HOUSE?25.55%28.9%About half of our respondents have nevercreated or tried to create a patch for avulnerability in house, which indicates thatthese respondents are fully reliant on theavailability of vendor patches – or some formof public disclosure that points to remediationmeasure or remedial code. Another significantproportion, over 25%, suggested that theyattempted to create a patch – but didn’tachieve the desired outcome.YesNoI haveattempted it45.55%There are several obvious barriers todeveloping a patch. Doing so requires anextensive understanding of the OS orapplication that needs to be patched as wellas knowledge of how the patch may affectother subsystems. Extensive testing is alsocritical to validate that the patch is effective.State of Enterprise Vulnerability Detection and Patch Management13
Downtime to deploy patchesAfter a patch is obtained the patch must be deployed to the affected systems and that oftenrequires a reboot which results in downtime or service disruption. We asked our respondents tostate how many hours of downtime their workloads typically experience every week in response topatching.Transport, Logistics15Media, Creative Industries14Retail / Ecommerce8Technology4.5Industrials (Manufacturing, Construction,etc.)3.5Public or Social Services2Banking & Financial Services1.56Agriculture, Forestry, Mining1.5EducationTransport, LogisticsData Infrastructure, Telecom1.5HealthcareProfessional Services (Law, Consulting,etc.)1.1510.83AVERAGE HOURS OF DOWNTIME FORPATCHING PER WEEK, PER INDUSTRYMost industries reported less than two hours per week lost to patchingprocedures. However, two industries reported outsize numbers – transport andlogistics and media and creative industries both reported considerably higherhours lost due to patching.State of Enterprise Vulnerability Detection and Patch Management14
Staff time consumed by patching tasksPatching can be divided into subtasks. Teams must coordinate downtime with stakeholders,processes need to be documented, and some patches need to be installed outside themaintenance window.The overall time spent on patching, as broken down by industry, does not tell us that muchbecause different industries will have different software stacks and varying regulatoryrequirements. The expectations of end users around availability also vary, so some organizationsmay not be able to arrange for downtime as easily as others.Agriculture, Forestry, Mining1.54.52.04.52.52.5Banking & Financial Services1.612.34.32.52.5Data Infrastructure, are1.0Industrials (Manufacturing,Construction, etc.)3.53.07.03.57.01.013.02.048.0Media, Creative Industries14.0Professional Services (Law,Consulting, etc.)0.88.626.06.010.03.76.011.038.3Public or Social Services2.08.02.01.01.0Retail / 27.5Transport, Documentation60%6.03.52.080%100%Coordination With StakeholdersHowever, it is interesting to note that, for each industry, the proportion of timedevoted to each individual subtask varies dramatically.State of Enterprise Vulnerability Detection and Patch Management15
When patching is delayedIn an ideal world, patching will occur immediately after a vulnerability is disclosed, but there areseveral factors that affect this process. In the graph below, we allowed respondents to choosemultiple options – the numbers on the graph represent the number of times an option was selected.WHY DO YOU THINK PATCH INSTALLATION ISDELAYED?Not able to take critical applications & systems offline so they canbe patched quickly. It is difficult to prioritize what needs to bepatched4631No common view of applications and assets across security and ITteams30Human error29Not enough resources to keep up with the volume of patches27Technologies such as automation reduce the risk of not patchingquickly enoughWe can’t easily track whether vulnerabilities are being patched in atimely manner25Emails & spreadsheets are used to manage the process, so thingsslip between the cracks2519My organization has no tolerance for the downtime required forpatching15We don’t have the ability to hold IT or other departmentsaccountable for patching9Silo and turf issues4We do not think an attacker will exploit our vulnerabilitiesEOL systems can’t be patched1The results show that in many organizations there is no mechanism in place that ensures thatpatches can be deployed to business-critical systems in a timely manner.It may be that high-availability architectures are not resilient enough to cope with patching, andthat organizations are not using live patching mechanisms to deploy patches withoutdisruption. Either way, the result is that unpatched systems become high-value targets forattackers.State of Enterprise Vulnerability Detection and Patch Management16
ImprovementsFinally, we wanted to gauge which steps would help to improve the outcomes for technologyteams that are responsible for vulnerability and patch management.WHAT STEPS WOULD YOU TAKE TO IMPROVE YOURORGANIZATION’S PATCH MANAGEMENT59Use a live patching toolIncrease automation54Increase IT security staff38Define your process betterOtherNo new steps would be taken3332We received three responses under “other”, including “Fix lifecycle issues by not running EOLsystems”, “Enforce culpability for patch delayed for internal company politics” and “Have seniormanagement (outside of IT) hold business units responsible for patching”. The first answer is selfexplanatory, but the second and third answers are more interesting and may point to problemsaround silos and whose “turf” it is when it comes to patching.One pain point that is commonly mentioned is a lack of resources to deal with the mountingworkload generated by the growing number of vulnerabilities. We asked our respondentswhether they consider that their staff count was sufficient to meet the workload and if theyplanned to increase staff numbers within the next year.More than half,54.3%, ofrespondentsindicated that theirstaff is insufficient tomeet their workload– half of whichindicated that theyplan on hiringadditional staffmembers that arededicated topatching tasks.IS YOUR STAFF SUFFICIENT AND WILL YOU HIRE MORESTAFF DEDICATED TO PATCHING IN THE NEXT 12MONTHS?27.2%45.7%The organization hassufficient staff to patch ina timely mannerThe organization will hirestaff dedicated topatching in the next 12months27.2%State of Enterprise Vulnerability Detection and Patch ManagementComplete reporting17
ConclusionOne of the challenging aspects of mounting an effective cybersecurity response is the inherenttradeoffs. Consider, for example, the tradeoff between the availability of resources and the cost ofthose resources. Likewise, there is the still commonplace tradeoff between availability of services,and fast and timely patching.At times it can appear as if these tradeoffs are irreconcilable, but there are tools that can bridgethe gap – including automation. Indeed, automation is a must-have tool given the automatednature of the cyberthreat. Nonetheless, in our survey, we found that automation – amongst othertools – is unevenly embraced.It is true, of course, that every industry faces unique challenges and unique tradeoffs. However,irrespective of what these tradeoffs are, every organization must work to adapt to a growingthreat. Live patching and other tools will help, and many companies have already cottoned on tothe benefits of standardization.The cybersecurity threat is not going to recede. Vulnerability and patch management effortsmust be run intelligently, and must be adequately resourced to meet the growing securitychallenge.State of Enterprise Vulnerability Detection and Patch Management18
Thanks for reading the report!While we’ve received a meaningful number of responses, the survey is still running,and we are eager to increase the amount of answers to build a more complete picture ofvulnerability and patch management in the enterprise environment.Have your say!Participate in the survey and get a chance to win one of tenCertified Kubernetes Administrator Certification from The Linux Foundation*To avoid spam submissions, only users with corporate email addresses can participate in the raffle.State of Enterprise Vulnerability Detection and Patch Management19
PATCH MANAGEMENT TOOLS, VULNERABILITY SCANNERS, EPORTAL SECURE PATCH SERVER AND 24/7 SUPPORT. For maximum security and compliance, enterprises need to rapidly patch vulnerabilities, keep production Linux systems updated with the latest fixes, and have a trusted technology partner for Linux support & maintenance always within reach.
