Transcription
Document:Authors:Company:Date:File:EditionWebroot SecureAnywhere Business Endpoint Protection vs. Seven Competitors (April 2017)M. Baquiran, D. WrenPassMark Software26 April 2017Webroot SecureAnywhere Business vs competitors 2017 Edition 1.docx1
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwareTABLE OF CONTENTS. 2REVISION HISTORY. 3REFERENCES. 3EXECUTIVE SUMMARY . 4OVERALL SCORE . 5PRODUCTS AND VERSIONS . 6PERFORMANCE METRICS SUMMARY . 7TEST RESULTS . 10BENCHMARK 1 – INSTALLATION TIME . 10BENCHMARK 2 – INSTALLATION SIZE . 10BENCHMARK 3 – BOOT TIME . 11BENCHMARK 4 – CPU USAGE DURING IDLE . 11BENCHMARK 5 – CPU USAGE DURING SCAN . 12BENCHMARK 6 – MEMORY USAGE DURING SYSTEM IDLE . 12BENCHMARK 7 – MEMORY USAGE DURING INITIAL SCAN . 13BENCHMARK 8 – SCHEDULED SCAN TIME . 13BENCHMARK 9 – BROWSE TIME . 14BENCHMARK 10 – FILE COPY, MOVE, AND DELETE . 14BENCHMARK 11 – FILE COMPRESSION AND DECOMPRESSION . 15BENCHMARK 12 – FILE WRITE, OPEN, AND CLOSE . 15BENCHMARK 13 – NETWORK THROUGHPUT . 16BENCHMARK 14 – MEMORY USAGE DURING SCHEDULED SCAN . 16DISCLAIMER AND DISCLOSURE . 17CONTACT DETAILS . 17APPENDIX 1 – TEST ENVIRONMENT . 18APPENDIX 2 – METHODOLOGY DESCRIPTION . 19Performance BenchmarkWindows 10Page 2 of 2526 April 2017
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsRevEdition 1Revision History1DateInitial version of this report.Ref #DocumentWhat Really Slows Windows Down (URL)Performance BenchmarkWindows 10PassMark Software26 April 2017AuthorO. Warner,The PC SpyDate2001-2017Page 3 of 2526 April 2017
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwarePassMark Software conducted objective performance testing on eight (8) security software products, onWindows 10 (64-bit) during March and April 2017. This report presents our results and findings as a result ofperformance benchmark testing conducted for these endpoint security products.The aim of this benchmark was to compare the performance impact of Webroot’s SecureAnywhere BusinessEndpoint Protection product with seven (7) competitor products.Testing was performed on all products using fourteen (14) performance metrics. These performance metrics areas follows. Installation Time; Installation Size; Boot Time; CPU Usage during Idle; CPU Usage during Scan; Memory Usage during System Idle; Memory Usage during Initial Scan; Scheduled Scan Time; Browse Time; File Copy, Move, and Delete; File Compression and Decompression; File Write, Open, and Close; Network Throughput; and Memory Usage during Initial Scan.Performance BenchmarkWindows 10Page 4 of 2526 April 2017
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwarePassMark Software assigned every product a score depending on its ranking in each metric compared to otherproducts in the same category. In the following table the highest possible score attainable has been normalized to100. This would be the score given if a product attained first place in all fourteen (14) metrics. Endpoint productshave been ranked by their overall scores:Product NamePerformance BenchmarkWindows 10Overall ScoreWebroot SecureAnywhere Endpoint Protection95Trend Micro Worry-Free Business Security Advanced61Symantec Endpoint Protection Cloud53Sophos Endpoint Protection53ESET Endpoint Security52Bitdefender GravityZone Business Security51Malwarebytes Endpoint Security45Kaspersky Small Office Security41Page 5 of 2526 April 2017
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwareFor each security product, we have tested the most current and publicly available version. The names and versionsof products are given below:ManufacturerWebroot Inc.Trend Micro Inc.Kaspersky LabSophosBitdefenderSymantec CorpESET, spol. s r.o.MalwarebytesPerformance BenchmarkWindows 10Product VersionDateTestedWebroot SecureAnywhere Business EndpointProtection9.0.15.40March2017Trend Micro Worry Free Business SecurityAdvanced19.0.2166April 201717.0.0.611 (d)March2017EndpointSecurity andControl 10.7April 2017Bitdefender GravityZone Business Security6.2.18.884April 2017Symantec Endpoint Protection Cloud22.9.1.12April 2017ESET Endpoint Security6.5.2094.0April 2017Product NameKaspersky Small Office SecuritySophos Endpoint ProtectionMalwarebytes Endpoint omware0.9.17.689April 2017Page 6 of 2526 April 2017
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwareWe have selected a set of objective metrics which provide a comprehensive and realistic indication of the areas inwhich endpoint protection products may impact system performance for end users. Our metrics test the impactof the software on common tasks that end-users would perform on a daily basis.All of PassMark Software’s test methods can be replicated by third parties using the same environment to obtainsimilar benchmark results. Detailed descriptions of the methodologies used in our tests are available as “Appendix2 – Methodology Description” of this report.The speed and ease of the installation process will strongly influence the user’s first impression of the securitysoftware. This test measures the installation time required by the security software to be fully functional and readyfor use by the end user. Lower installation times represent security products which are quicker for a user to install.In offering new features and functionality to users, security software products tend to increase in size with eachnew release. Although new technologies push the size limits of hard drives each year, the growing disk spacerequirements of common applications and the increasing popularity of large media files (such as movies, photosand music) ensure that a product's installation size will remain of interest to home users.This metric aims to measure a product’s total installation size. This metric is defined as the total disk spaceconsumed by all new files added during a product's installation.This metric measures the amount of time taken for the machine to boot into the operating system. Securitysoftware is generally launched at Windows startup, adding an additional amount of time and delaying the startupof the operating system. Shorter boot times indicate that the application has had less impact on the normaloperation of the machine.The amount of memory used while the machine is idle provides a good indication of the amount of systemresources being consumed by the security software on a permanent basis. This metric measures the amount ofmemory (RAM) used by the product while the machine and security software are in an idle state. The total memoryusage was calculated by identifying all the security software’s processes and the amount of memory used by eachprocess.The amount of load on the CPU while security software conducts a malware scan may prevent the reasonable useof the endpoint machine until the scan has completed. This metric measured the percentage of CPU used bysecurity software when performing a scan.Performance BenchmarkWindows 10Page 7 of 2526 April 2017
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwareThis metric measures the amount of memory (RAM) used by the product while the machine and security softwareare in an idle state. The total memory usage was calculated by identifying all security software processes and theamount of memory used by each process.The amount of memory used while the machine is idle provides a good indication of the amount of systemresources being consumed by the security software on a permanent basis. Better performing products occupy lessmemory while the machine is idle.This metric measures the amount of memory (RAM) used by the product during an initial security scan on theendpoint’s main drive. The total memory usage was calculated by identifying all security software processes andthe amount of memory used by each process during the scan.Most security solutions are scheduled by default to scan the system regularly for viruses and malware. This metricmeasured the amount of time required to run a scheduled scan on the system. The scan is set to run at a specifiedtime via the client user interface.It is common behavior for security products to scan data for malware as it is downloaded from the internet orintranet. This behavior may negatively impact browsing speed as products scan web content for malware. Thismetric measures the time taken to browse a set of popular internet sites to consecutively load from a local serverin a user’s browser window.This metric measures the amount of time taken to copy, move and delete a sample set of files. The sample file setcontains several types of file formats that a Windows user would encounter in daily use. These formats includedocuments (e.g. Microsoft Office documents, Adobe PDF, Zip files, etc), media formats (e.g. images, movies andmusic) and system files (e.g. executables, libraries, etc).This metric measures the amount of time taken to compress and decompress different types of files. Files formatsused in this test included documents, movies and images.This benchmark was derived from Oli Warner’s File I/O test at http://www.thepcspy.com (please see Reference#1: What Really Slows Windows Down). This metric measures the amount of time taken to write a file, then openand close that file.The metric measures the amount of time taken to download a variety of files from a local server using theHyperText Transfer Protocol (HTTP), which is the main protocol used on the web for browsing, linking and dataPerformance BenchmarkWindows 10Page 8 of 2526 April 2017
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark Softwaretransfer. Files used in this test include file formats that users would typically download from the web, such asimages, archives, music files and movie files.This metric measures the amount of memory (RAM) used by the product during a scheduled security scan on thetarget endpoint. The total memory usage was calculated by identifying all security software processes and theamount of memory used by each process during the scan.Performance BenchmarkWindows 10Page 9 of 2526 April 2017
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwareIn the following charts, we have highlighted the results we obtained for Webroot SecureAnywhere BusinessEndpoint Protection in green. The competitor average has also been highlighted in blue for ease of comparison.The following chart compares the minimum installation time it takes for endpoint security products to be fullyfunctional and ready for use by the end user. Products with lower installation times are considered betterperforming products in this category.Webroot SecureAnywhere Endpoint Protection33Symantec EP Cloud42Kaspersky Small Office Security59Malwarebytes Endpoint Security63ESET Endpoint Security81Average98Sophos Endpoint Protection157Trend Micro WFBS Advanced170Bitdefender GravityZone Business Security1780s20 s40 s60 s80 s100 s120 s140 s160 s180 s200 sThe following chart compares the total size of files added during the installation of endpoint security products.Products with lower installation sizes are considered better performing products in this category.Webroot SecureAnywhere Endpoint Protection32.9Malwarebytes Endpoint Security211.5Kaspersky Small Office Security468.9Trend Micro WFBS Advanced548.9ESET Endpoint Security594.5Average640.4Sophos Endpoint Protection856.9Symantec EP Cloud1043.0Bitdefender GravityZone Business Security0 MBPerformance BenchmarkWindows 101366.5200 MB400 MB600 MB800 MB 1,000 MB 1,200 MB 1,400 MB 1,600 MBPage 10 of 2526 April 2017
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwareThe following chart compares the average time taken for the system to boot (from a sample of five boots) for eachendpoint security product tested. Products with lower boot times are considered better performing products inthis category.*Webroot SecureAnywhere Endpoint Protection7.3Trend Micro WFBS Advanced7.7Kaspersky Small Office Security7.7Bitdefender GravityZone Business Security8.5ESET Endpoint Security8.7Sophos Endpoint Protection9.2Average9.2Malwarebytes Endpoint Security15.50s2s4s6s8s10 s12 s14 s16 s18 s*Symantec was omitted from this test as the boot time test could not reach an idle state with the product installed.The following chart compares the average CPU usage during system idle. Products with lower CPU usage areconsidered better performing products in this category.Sophos Endpoint Protection0.39%Bitdefender GravityZone Business Security0.67%Webroot SecureAnywhere Endpoint Protection0.67%Kaspersky Small Office Security0.76%Malwarebytes Endpoint Security0.80%ESET Endpoint Security0.83%Trend Micro WFBS Advanced1.11%Average6.72%Symantec EP Cloud0.0%Performance BenchmarkWindows 1048.53%10.0%20.0%30.0%40.0%50.0%60.0%Page 11 of 2526 April 2017
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwareThe following chart compares the average CPU usage during a scan of a set of media files, system files andMicrosoft Office documents that totaled 5.42 GB. Products with lower CPU usage are considered betterperforming products in this category.Trend Micro WFBS Advanced21.1%Sophos Endpoint Protection23.3%Malwarebytes Endpoint Security24.4%Bitdefender GravityZone Business Security28.0%Webroot SecureAnywhere Endpoint Protection28.1%ESET Endpoint Security29.9%Average33.1%Symantec EP Cloud51.4%Kaspersky Small Office 0%The following chart compares the average amount of RAM in use by an endpoint security product during a periodof system idle. This average is taken from a sample of ten memory snapshots taken at roughly 60 seconds apartafter reboot. Products with lower idle RAM usage are considered better performing products in this category.Webroot SecureAnywhere Endpoint Protection5.8Trend Micro WFBS Advanced32.9Bitdefender GravityZone Business Security108.3Sophos Endpoint Protection126.7Average135.3Symantec EP Cloud175.4ESET Endpoint Security187.1Malwarebytes Endpoint Security218.6Kaspersky Small Office Security0 MBPerformance BenchmarkWindows 10227.950 MB100 MB150 MB200 MB250 MBPage 12 of 2526 April 2017
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwareThe following chart compares the average amount of RAM in use by an endpoint security product during an initialscan on the main drive. This average is taken from a sample of ten memory snapshots taken at twelve secondintervals during a scan of sample files which have not been previously scanned by the software. Products that useless memory during a scan are considered better performing products in this category.Webroot SecureAnywhere Endpoint Protection23.8Trend Micro WFBS Advanced106.3Sophos Endpoint Protection146.0Symantec EP Cloud173.9ESET Endpoint Security211.9Average217.6Bitdefender GravityZone Business Security331.5Malwarebytes Endpoint Security364.4Kaspersky Small Office Security382.50 MB50 MB 100 MB 150 MB 200 MB 250 MB 300 MB 350 MB 400 MB 450 MBThe following chart compares the average time taken to run a scheduled scan on the system for each securityproduct tested.Webroot SecureAnywhere Endpoint Protection18Symantec EP Cloud64Malwarebytes Endpoint Security152Bitdefender GravityZone Business Security710ESET Endpoint Security1219Average1567Sophos Endpoint Protection2436Trend Micro WFBS Advanced3255Kaspersky Small Office Security46810sPerformance BenchmarkWindows 10500 s1,000 s 1,500 s 2,000 s 2,500 s 3,000 s 3,500 s 4,000 s 4,500 s 5,000 sPage 13 of 2526 April 2017
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwareThe following chart compares the average time taken for Internet Explorer to successively load a set of popularwebsites through the local area network from a local server machine. Products with lower browse times areconsidered better performing products in this category.*Webroot SecureAnywhere Endpoint Protection16.8Malwarebytes Endpoint Security17.6Trend Micro WFBS Advanced19.7ESET Endpoint Security19.9Sophos Endpoint Protection22.4Average25.3Symantec EP Cloud34.6Kaspersky Small Office Security46.20s5s10 s15 s20 s25 s30 s35 s40 s45 s50 s* Bitdefender was excluded from this test as the test script was blocked by the application’s phishing filter.The following chart compares the average time taken to copy, move and delete several sets of sample files foreach endpoint security product tested. Products with lower times are considered better performing products inthis category.Webroot SecureAnywhere Endpoint Protection3.2Bitdefender GravityZone Business Security4.8Symantec EP Cloud4.9ESET Endpoint Security6.0Average8.2Trend Micro WFBS Advanced9.1Kaspersky Small Office Security9.4Malwarebytes Endpoint Security10.1Sophos Endpoint Protection18.20sPerformance BenchmarkWindows 102s4s6s8s10 s12 s14 s16 s18 sPage 14 of 2526 April 201720 s
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwareThe following chart compares the average time it takes for sample files to be compressed and decompressed foreach endpoint security product tested. Products with lower times are considered better performing products inthis category.Webroot SecureAnywhere Endpoint Protection39.1Bitdefender GravityZone Business Security40.4Symantec EP Cloud40.8ESET Endpoint Security41.4Average42.8Sophos Endpoint Protection43.4Kaspersky Small Office Security44.4Trend Micro WFBS Advanced44.8Malwarebytes Endpoint Security47.90s10 s20 s30 s40 s50 s60 sThe following chart compares the average time it takes for a file to be written to the hard drive then opened andclosed 180,000 times, for each endpoint security product tested. Products with lower times are considered betterperforming products in this category.Webroot SecureAnywhere Endpoint Protection10.2Kaspersky Small Office Security13.8Symantec EP Cloud20.0Sophos Endpoint Protection52.2ESET Endpoint Security79.9Bitdefender GravityZone Business Security185.1Trend Micro WFBS Advanced242.9Average741.4Malwarebytes Endpoint Security5327.50sPerformance BenchmarkWindows 101,000 s2,000 s3,000 s4,000 s5,000 s6,000 sPage 15 of 2526 April 2017
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwareThe following chart compares the average time to download a sample set of common file types for each endpointsecurity product tested. Products with lower times are considered better performing products in this category.Webroot SecureAnywhere Endpoint Protection6.0Trend Micro WFBS Advanced6.4ESET Endpoint Security6.5Bitdefender GravityZone Business Security6.7Malwarebytes Endpoint Security6.8Average7.3Symantec EP Cloud8.2Sophos Endpoint Protection8.5Kaspersky Small Office Security9.20s1s2s3s4s5s6s7s8s9s10 sThe following chart compares the average amount of RAM in use by an endpoint security product during ascheduled scan on the endpoint. This average is taken from a sample of five memory snapshots taken at fivesecond intervals. Products that use less memory during a scan are considered better performing products in thiscategory.Webroot SecureAnywhere Endpoint ProtectionTrend Micro WFBS Advanced7.624.0Symantec EP Cloud131.5Sophos Endpoint Protection158.9Average191.0ESET Endpoint Security221.3Kaspersky Small Office Security298.5Bitdefender GravityZone Business Security325.9Malwarebytes Endpoint Security0 MBPerformance BenchmarkWindows 10360.250 MB100 MB150 MB200 MB250 MB300 MB350 MB400 MBPage 16 of 2526 April 2017
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwareThis report only covers versions of products that were available at the time of testing. The tested versions are asnoted in the “Products and Versions” section of this report. The products we have tested are not an exhaustivelist of all products available in these very competitive product categories.While every effort has been made to ensure that the information presented in this report is accurate, PassMarkSoftware Pty Ltd assumes no responsibility for errors, omissions, or out-of-date information and shall not be liablein any manner whatsoever for direct, indirect, incidental, consequential, or punitive damages resulting from theavailability of, use of, access of, or inability to use this information.Webroot Inc. funded the production of this report. The list of products tested and the metrics included in thereport were selected by Webroot.All trademarks are the property of their respective owners.PassMark Software Pty LtdLevel 563 Foveaux St.Surry Hills, 2010Sydney, AustraliaPhone 61 (2) 9690 0444Fax 61 (2) 9690 0445Webwww.passmark.comPerformance BenchmarkWindows 10Page 17 of 2526 April 2017
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsModel:Lenovo H50W-50 i5CPU:Intel Core i5-4460 CPU @ 3.20GHz 3.20 GHzVideo Card:NVIDIA GeForce GT 705RAM:8GB DDR3 RAMSSD (Main Boot Drive):Samsung SSD 850 PRO 512 GBnd2 Drive:Samsung 1000GB 7200RPM HD103UJNetwork:Gigabit (1Gb/s) switchO/S:Windows 10 Home 10.0 (Build 10240)PassMark SoftwareThe Web and File server was not benchmarked directly, but served the web pages and files to the endpointmachine during performance testing.CPU:Video Card:Motherboard:RAM:SSD:Network:Performance BenchmarkWindows 10Intel Xeon E3-1220v2 CPUKingston 8GB (2 x 4GB ECC RAM)Intel S1200BTL ServerKingston 8GB (2 x 4GB) ECC RAM, 1333MhzOCZ 128GB 2.5” Solid State DiskGigabit (1GB/s)Page 18 of 2526 April 2017
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwareAs with testing on Windows Vista, Norton Ghost was used to create a “clean” baseline image prior to testing. Ouraim is to create a baseline image with the smallest possible footprint and reduce the possibility of variation causedby external operating system factors.The baseline image was restored prior to testing of each different product. This process ensures that we installand test all products on the same, “clean” machine.The steps taken to create the base Windows 10 image are as follows:1.Installation and activation of Windows 10.2.Disabled Automatic Updates.3.Changed User Account Control settings to “Never Notify”.4.Disable Windows Defender automatic scans to avoid unexpected background activity.5.Disable the Windows firewall to avoid interference with security software.6.Disabled Superfetch to ensure consistent results.7.Installed HTTP Watch for Browse Time testing.8.Installed Windows 10 Assessment and Deployment Kit (ADK) for Boot Time testing.9.Installed Active Perl for interpretation of some test scripts.10.Install OSForensics for testing (Installation Size and Registry Key Count tests) purposes.11.Disabled Windows updates.12.Install important Windows updates.13.Created a baseline image of the boot drive using OSForensics.This test measures the minimum Installation Time a product requires to be fully functional and ready for use bythe end user. Installation time can usually be divided in three major phases: The Extraction and Setup phase consists of file extraction, the EULA prompt, product activation and userconfigurable options for installation. The File Copy phase occurs when the product is being installed; usually this phase is indicated by a progressbar. The Post-Installation phase is any part of the installation that occurs after the File Copy phase. This phasevaries widely between products; the time recorded in this phase may include a required reboot to finalize theinstallation or include the time the program takes to become idle in the system tray.To reduce the impact of disk drive variables, each product was copied to the Desktop before initializing installation.Each step of the installation process was manually timed with a stopwatch and recorded in as much detail aspossible. Where input was required by the end user, the stopwatch was paused and the input noted in the rawresults in parenthesis after the phase description.Performance BenchmarkWindows 10Page 19 of 2526 April 2017
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwareWhere possible, all requests by products to pre-scan or post-install scan were declined or skipped. Where it wasnot possible to skip a scan, the time to scan was included as part of the installation time. Where an optionalcomponent of the installation formed a reasonable part of the functionality of the software, it was also installed(e.g. website link checking software as part of a Security Product).Installation time includes the time taken by the product installer to download components required in theinstallation. This may include mandatory updates or the delivery of the application itself from a downloadmanager. We have noted in our results where a product has downloaded components for product installation.We have excluded product activation times due to network variability in contacting vendor servers or time takenin account creation. For all products tested, the installation was performed directly on the endpoint, either usinga standalone installation package or via the management server web console.A product's Installation Size was previously defined as the difference between the initial snapshot of the Disk Space(C: drive) before installation and the subsequent snapshot taken after the product is installed on the system.Although this is a widely used methodology, we noticed that the results it yielded were not always reproduciblein Vista due to random OS operations that may take place between the two snapshots. We improved theInstallation Size methodology by removing as many Operating System and disk space variables as possible.Using PassMark’s OSForensics 2.2 we created initial and post-installation disk signatures for each product. Thesedisk signatures recorded the amount of files and directories, and complete details of all files on that drive (includingfile name, file size, checksum, etc) at the time the signature was taken.The initial disk signature was taken immediately prior to installation of the product. A subsequent disk signaturewas taken immediately following a system reboot after product installation. Using OSForensics, we compared thetwo signatures and calculated the total disk space consumed by files that were new, modified, and deleted duringproduct installation. Our result for this metric reflects the total size of all newly added files during installation.The scope of this metric includes only an ‘out of the box’ installation size for each product. Our result does notcover the size of file
Endpoint Security and Control 10.7 April 2017 Bitdefender Bitdefender GravityZone Business Security 6.2.18.884 April 2017 Symantec Corp Symantec Endpoint Protection Cloud 22.9.1.12 April 2017 ESET, spol. s r.o. ESET Endpoint Security 6.5.2094.0 April 2017 Malwarebytes Malwarebytes Endpoint Security Anti-Malware (Corporate) 1.80.2.1012
