Transcription
Document:Authors:Company:Date:File:EditionWebroot SecureAnywhere Business Endpoint Protection vs. Seven Competitors (February 2014)M. Baquiran, D. WrenPassMark Software19 February 2014Webroot SecureAnywhere endpoint protection vs competitors 19 February 2014.docx1
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwareTABLE OF CONTENTS. 2REVISION HISTORY. 3REFERENCES. 3EXECUTIVE SUMMARY . 4OVERALL SCORE . 5PRODUCTS AND VERSIONS . 6PERFORMANCE METRICS SUMMARY . 7TEST RESULTS . 10BENCHMARK 1 – INSTALLATION TIME . 10BENCHMARK 2 – INSTALLATION SIZE . 10BENCHMARK 3 – BOOT TIME . 11BENCHMARK 4 – CPU USAGE DURING IDLE . 11BENCHMARK 5 – CPU USAGE DURING SCAN . 12BENCHMARK 6 – MEMORY USAGE DURING SYSTEM IDLE . 12BENCHMARK 7 – MEMORY USAGE DURING INITIAL SCAN . 13BENCHMARK 8 – SCHEDULED SCAN TIME . 13BENCHMARK 9 – BROWSE TIME . 14BENCHMARK 10 – FILE COPY, MOVE, AND DELETE . 14BENCHMARK 11 – FILE COMPRESSION AND DECOMPRESSION . 15BENCHMARK 12 – FILE WRITE, OPEN, AND CLOSE . 15BENCHMARK 13 – NETWORK THROUGHPUT . 16DISCLAIMER AND DISCLOSURE . 17CONTACT DETAILS . 17APPENDIX 1 – TEST ENVIRONMENT . 18APPENDIX 2 – METHODOLOGY DESCRIPTION . 19Performance BenchmarkPage 2 of 2512 March 2014
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsRevEdition 1Revision History1DateInitial version of this report.Ref #DocumentWhat Really Slows Windows Down (URL)Performance BenchmarkPassMark Software5 February 2014AuthorO. Warner,The PC SpyDate2001-2014Page 3 of 2512 March 2014
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwarePassMark Software conducted objective performance testing on eight (8) security software products, onWindows 7 Ultimate Edition (64-bit) during January 2014. This report presents our results and findings as a resultof performance benchmark testing conducted for these endpoint security products.The aim of this benchmark was to compare the performance impact of Webroot’s SecureAnywhere BusinessEndpoint Protection product with seven (7) competitor products.Testing was performed on all products using thirteen (13) performance metrics. These performance metrics areas follows. Installation Time; Installation Size; Boot Time; CPU Usage during Idle; CPU Usage during Scan; Memory Usage during System Idle; Memory Usage during Initial Scan; Scheduled Scan Time; Browse Time; File Copy, Move, and Delete; File Compression and Decompression; File Write, Open, and Close; and Network Throughput.Performance BenchmarkPage 4 of 2512 March 2014
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwarePassMark Software assigned every product a score depending on its ranking in each metric compared to otherproducts in the same category. In the following table the highest possible score attainable is 104; in a hypotheticalsituation where a product has attained first place in all thirteen (13) metrics. Endpoint products have been rankedby their overall scores:Product NameWebroot SecureAnywhere Endpoint ProtectionESET NOD32 Antivirus BusinessPerformance BenchmarkOverall Score9770Microsoft Security Center Endpoint Protection69Symantec Endpoint Protection Small Business Edition67Kaspersky Endpoint Security55Sophos EndUser Protection – Business51McAfee Complete Endpoint Protection – Business48Trend Micro Worry Free Business Security Standard47Page 5 of 2512 March 2014
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwareFor each security product, we have tested the most current and available version.ManufacturerWebroot Software, Inc.Trend Micro Inc.Kaspersky LabSophosMcAfee, Inc.Symantec CorpESET, spol. s r.o.Microsoft CorporationPerformance BenchmarkProduct VersionDateTestedWebroot SecureAnywhere Endpoint Protection8.0.4.46Feb 2014Trend Micro Worry Free Business SecurityStandard7.0.1638Jan 2014Kaspersky Endpoint Security10.2.1.23Jan 2014SophosEndpointSecurity andControl 10.3Jan 2014VirusScan,AntiSpywareEnterprise 8.8Jan 2014Product NameSophos EndUser Protection – BusienssMcAfee Complete Endpoint Protection - BusinessSymantec Endpoint Protection Small BusinessEdition 2013 (Symantec .cloud)Cloud Agent x642.03.23.2539EndpointProtection NIS20.4.0.40Jan 2014ESET NOD32 Antivirus Business4.2.76.0Jan 2014Microsoft System Center Endpoint Protection4.3.220.0Jan 2014Page 6 of 2512 March 2014
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwareWe have selected a set of objective metrics which provide a comprehensive and realistic indication of the areas inwhich endpoint protection products may impact system performance for end users. Our metrics test the impactof the software on common tasks that end-users would perform on a daily basis.All of PassMark Software’s test methods can be replicated by third parties using the same environment to obtainsimilar benchmark results. Detailed descriptions of the methodologies used in our tests are available as “Appendix2 – Methodology Description” of this report.The speed and ease of the installation process will strongly influence the user’s first impression of the securitysoftware. This test measures the installation time required by the security software to be fully functional and readyfor use by the end user. Lower installation times represent security products which are quicker for a user to install.In offering new features and functionality to users, security software products tend to increase in size with eachnew release. Although new technologies push the size limits of hard drives each year, the growing disk spacerequirements of common applications and the increasing popularity of large media files (such as movies, photosand music) ensure that a product's installation size will remain of interest to home users.This metric aims to measure a product’s total installation size. This metric is defined as the total disk spaceconsumed by all new files added during a product's installation.This metric measures the amount of time taken for the machine to boot into the operating system. Securitysoftware is generally launched at Windows startup, adding an additional amount of time and delaying the startupof the operating system. Shorter boot times indicate that the application has had less impact on the normaloperation of the machine.The amount of memory used while the machine is idle provides a good indication of the amount of systemresources being consumed by the security software on a permanent basis. This metric measures the amount ofmemory (RAM) used by the product while the machine and security software are in an idle state. The total memoryusage was calculated by identifying all the security software’s processes and the amount of memory used by eachprocess.The amount of load on the CPU while security software conducts a malware scan may prevent the reasonable useof the endpoint machine until the scan has completed. This metric measured the percentage of CPU used bysecurity software when performing a scan.Performance BenchmarkPage 7 of 2512 March 2014
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwareThis metric measures the amount of memory (RAM) used by the product while the machine and security softwareare in an idle state. The total memory usage was calculated by identifying all security software processes and theamount of memory used by each process.The amount of memory used while the machine is idle provides a good indication of the amount of systemresources being consumed by the security software on a permanent basis. Better performing products occupy lessmemory while the machine is idle.This metric measures the amount of memory (RAM) used by the product during an initial security scan. The totalmemory usage was calculated by identifying all security software processes and the amount of memory used byeach process during the scan.Most antivirus solutions are scheduled by default to scan the system regularly for viruses and malware. This metricmeasured the amount of time required to run a scheduled scan on the system. The scan is set to run at a specifiedtime via the client user interface.It is common behavior for security products to scan data for malware as it is downloaded from the internet orintranet. This behavior may negatively impact browsing speed as products scan web content for malware. Thismetric measures the time taken to browse a set of popular internet sites to consecutively load from a local serverin a user’s browser window.This metric measures the amount of time taken to copy, move and delete a sample set of files. The sample file setcontains several types of file formats that a Windows user would encounter in daily use. These formats includedocuments (e.g. Microsoft Office documents, Adobe PDF, Zip files, etc), media formats (e.g. images, movies andmusic) and system files (e.g. executables, libraries, etc).This metric measures the amount of time taken to compress and decompress different types of files. Files formatsused in this test included documents, movies and images.This benchmark was derived from Oli Warner’s File I/O test at http://www.thepcspy.com (please see Reference#1: What Really Slows Windows Down). This metric measures the amount of time taken to write a file, then openand close that file.The metric measures the amount of time taken to download a variety of files from a local server using theHyperText Transfer Protocol (HTTP), which is the main protocol used on the web for browsing, linking and dataPerformance BenchmarkPage 8 of 2512 March 2014
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark Softwaretransfer. Files used in this test include file formats that users would typically download from the web, such asimages, archives, music files and movie files.Performance BenchmarkPage 9 of 2512 March 2014
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwareIn the following charts, we have highlighted the results we obtained for Webroot SecureAnywhere EndpointProtection in green. The competitor average has also been highlighted in blue for ease of comparison.The following chart compares the minimum installation time it takes for endpoint security products to be fullyfunctional and ready for use by the end user. Products with lower installation times are considered betterperforming products in this category.Webroot SecureAnywhere Business EP4Microsoft System Center EP39ESET NOD32 AV Business41Sophos EUP - Business134Kaspersky ES 10185Trend Micro WFBS Standard199Symantec EP SBE 2013200Average231McAfee CEP - Business10430s200 s400 s600 s800 s1,000 s1,200 sThe following chart compares the total size of files added during the installation of endpoint security products.Products with lower installation sizes are considered better performing products in this category.Webroot SecureAnywhere Business EP18.2ESET NOD32 AV Business237.9Microsoft System Center EP244.1Sophos EUP - Business476.3Average582.8Trend Micro WFBS Standard619.9McAfee CEP - Business769.9Symantec EP SBE 2013842.8Kaspersky ES 100 MBPerformance Benchmark1453.2200 MB400 MB600 MB800 MB 1,000 MB 1,200 MB 1,400 MB 1,600 MBPage 10 of 2512 March 2014
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwareThe following chart compares the average time taken for the system to boot (from a sample of five boots) for eachendpoint security product tested. Products with lower boot times are considered better performing products inthis category.Webroot SecureAnywhere Business EP14.3Microsoft System Center EP14.3ESET NOD32 AV Business16.1Sophos EUP - Business17.8Trend Micro WFBS Standard17.9Average18.4McAfee CEP - Business18.9Symantec EP SBE 201320.5Kaspersky ES 1027.10s5s10 s15 s20 s25 s30 sThe following chart compares the average CPU usage during system idle. Products with lower CPU usage areconsidered better performing products in this category.Kaspersky ES 100.01%Microsoft System Center EP0.02%ESET NOD32 AV Business0.02%Trend Micro WFBS Standard0.06%McAfee CEP - Business0.07%Webroot SecureAnywhere Business EPAverage0.10%0.13%Symantec EP SBE 20130.25%Sophos EUP - Business0.0%Performance Benchmark0.52%0.1%0.2%0.3%0.4%0.5%0.6%Page 11 of 2512 March 2014
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwareThe following chart compares the average CPU usage during a scan of a set of media files, system files andMicrosoft Office documents that totaled 5.42 GB. Products with lower CPU usage are considered betterperforming products in this category.Trend Micro WFBS Standard9.8%Webroot SecureAnywhere Business EP10.8%ESET NOD32 AV Business20.1%Sophos EUP - Business20.3%Symantec EP SBE 201320.5%Average23.7%McAfee CEP - Business26.9%Microsoft System Center EP39.8%Kaspersky ES 1041.2%0%5%10%15%20%25%30%35%40%45%The following chart compares the average amount of RAM in use by an endpoint security product during a periodof system idle. This average is taken from a sample of ten memory snapshots taken at roughly 60 seconds apartafter reboot. Products with lower idle RAM usage are considered better performing products in this category.Webroot SecureAnywhere Business EP5.6Microsoft System Center EP63.7ESET NOD32 AV Business94.3Symantec EP SBE 201399.8Average107.9Kaspersky ES 10109.3Trend Micro WFBS Standard122.3McAfee CEP - Business151.2Sophos EUP - Business0 MBPerformance Benchmark216.750 MB100 MB150 MB200 MB250 MBPage 12 of 2512 March 2014
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwareThe following chart compares the average amount of RAM in use by an endpoint security product during an initialscan on the main drive. This average is taken from a sample of ten memory snapshots taken at five second intervalsduring a scan of sample files which have not been previously scanned by the software. Products that use lessmemory during a scan are considered better performing products in this category.Webroot SecureAnywhere Business EP12.1ESET NOD32 AV Business102.9Microsoft System Center EP130.2Sophos EUP - Business175.8Kaspersky ES 10185.2Average195.9Trend Micro WFBS Standard246.3Symantec EP SBE 2013261.4McAfee CEP - Business453.50 MB50 MB 100 MB 150 MB 200 MB 250 MB 300 MB 350 MB 400 MB 450 MB 500 MBThe following chart compares the average time taken to run a scheduled scan on the system for each securityproduct tested.*Webroot SecureAnywhere Business EP26Symantec EP SBE 201334Microsoft System Center EP104Sophos EUP - Business357Average579Kaspersky ES 10986McAfee CEP - Business1218ESET NOD32 AV Business13290s200 s400 s600 s800 s1,000 s1,200 s1,400 s*Trend Micro’s product was omitted from the chart and given the lowest score. The scheduled scan time could not be run to completion due to what appears to be a bug.Performance BenchmarkPage 13 of 2512 March 2014
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwareThe following chart compares the average time taken for Internet Explorer to successively load a set of popularwebsites through the local area network from a local server machine. Products with lower browse times areconsidered better performing products in this category.*Webroot SecureAnywhere Business EP16.8Symantec EP SBE 201317.6Kaspersky ES 1028.9Trend Micro WFBS Standard35.2ESET NOD32 AV Business39.2Average43.9Microsoft System Center EP49.3Sophos EUP - Business71.0McAfee CEP - Business93.40s10 s20 s30 s40 s50 s60 s70 s80 s90 s100 s*To enable the browse time test to run without interruption, Webroot’s default settings were changed. An Agent Command was issued to “unprotect” Internet Explorer.The following chart compares the average time taken to copy, move and delete several sets of sample files foreach endpoint security product tested. Products with lower times are considered better performing products inthis category.Symantec EP SBE 201310.9Webroot SecureAnywhere Business EP11.7ESET NOD32 AV Business13.7Trend Micro WFBS Standard13.8McAfee CEP - Business14.2Average15.4Sophos EUP - Business18.3Microsoft System Center EP19.0Kaspersky ES 1021.60sPerformance Benchmark5s10 s15 s20 s25 sPage 14 of 2512 March 2014
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwareThe following chart compares the average time it takes for sample files to be compressed and decompressed foreach endpoint security product tested. Products with lower times are considered better performing products inthis category.Symantec EP SBE 201344.6Webroot SecureAnywhere Business EP45.8McAfee CEP - Business48.0Kaspersky ES 1048.3Average49.0Sophos EUP - Business49.0ESET NOD32 AV Business49.1Microsoft System Center EP50.6Trend Micro WFBS Standard56.60s10 s20 s30 s40 s50 s60 sThe following chart compares the average time it takes for a file to be written to the hard drive then opened andclosed 180,000 times, for each endpoint security product tested. Products with lower times are considered betterperforming products in this category.Webroot SecureAnywhere Business EP13.4Kaspersky ES 1018.9Symantec EP SBE 201319.3McAfee CEP - Business27.8Sophos EUP - Business72.6ESET NOD32 AV Business114.2Average165.6Microsoft System Center EP339.1Trend Micro WFBS Standard719.50sPerformance Benchmark100 s200 s300 s400 s500 s600 s700 s800 sPage 15 of 2512 March 2014
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwareThe following chart compares the average time to download a sample set of common file types for each endpointsecurity product tested. Products with lower times are considered better performing products in this category.Webroot SecureAnywhere Business EP6.3Microsoft System Center EP6.7McAfee CEP - Business7.3Symantec EP SBE 20137.6ESET NOD32 AV Business7.6Average7.6Sophos EUP - Business7.9Trend Micro WFBS Standard8.2Kaspersky ES 109.30sPerformance Benchmark1s2s3s4s5s6s7s8s9s10 sPage 16 of 2512 March 2014
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwareThis report only covers versions of products that were available at the time of testing. The tested versions are asnoted in the “Products and Versions” section of this report. The products we have tested are not an exhaustivelist of all products available in these very competitive product categories.While every effort has been made to ensure that the information presented in this report is accurate, PassMarkSoftware Pty Ltd assumes no responsibility for errors, omissions, or out-of-date information and shall not be liablein any manner whatsoever for direct, indirect, incidental, consequential, or punitive damages resulting from theavailability of, use of, access of, or inability to use this information.Webroot Software Inc. funded the production of this report. The list of products tested and the metrics includedin the report were selected by Webroot.All trademarks are the property of their respective owners.PassMark Software Pty LtdSuite 202, Level 235 Buckingham St.Surry Hills, 2010Sydney, AustraliaPhone 61 (2) 9690 0444Fax 61 (2) 9690 0445Webwww.passmark.comPerformance BenchmarkPage 17 of 2512 March 2014
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwareFor our testing, PassMark Software used a test environment running Windows 7 Ultimate (64-bit) with thefollowing hardware specifications:Model:CPU:Video Card:Motherboard:RAM:HDD:Network:HP Pavilion P6-2300AIntel Core i5 3330 @ 2.66GHz1GB nVIDIA GeForce GT 620MFoxconn 2ABF 3.106GB DDR3 RAMHitachi HDS721010CLA630 931.51GBGigabit (1GB/s)The Web and File server was not benchmarked directly, but served the web pages and files to the endpointmachine during performance testing.CPU:Video Card:Motherboard:RAM:SSD:Network:Performance BenchmarkIntel Xeon E3-1220v2 CPUKingston 8GB (2 x 4GB ECC RAM)Intel S1200BTL ServerKingston 8GB (2 x 4GB) ECC RAM, 1333MhzOCZ 128GB 2.5” Solid State DiskGigabit (1GB/s)Page 18 of 2512 March 2014
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwareAs with testing on Windows Vista, Norton Ghost was used to create a “clean” baseline image prior to testing. Ouraim is to create a baseline image with the smallest possible footprint and reduce the possibility of variation causedby external operating system factors.The baseline image was restored prior to testing of each different product. This process ensures that we installand test all products on the same, “clean” machine.The steps taken to create the base Windows 7 image are as follows:1.Installation and activation of Windows 7 Ultimate Edition.2.Disabled Automatic Updates.3.Changed User Account Control settings to “Never Notify”.4.Disable Windows Defender automatic scans to avoid unexpected background activity.5.Disable the Windows firewall to avoid interference with security software.6.Installed Norton Ghost for imaging purposes.7.Disabled Superfetch to ensure consistent results.8.Installed HTTP Watch for Browse Time testing.9.Installed Windows Performance Toolkit x64 for Boot Time testing.10.Installed Active Perl for interpretation of some test scripts.11.Install OSForensics for testing (Installation Size test) purposes.12.Disabled updates, accelerators and compatibility view updates in Internet Explorer 8.13.Update to Windows Service Pack 114.Created a baseline image using Norton Ghost.This test measures the minimum Installation Time a product requires to be fully functional and ready for use bythe end user. Installation time can usually be divided in three major phases: The Extraction and Setup phase consists of file extraction, the EULA prompt, product activation and userconfigurable options for installation. The File Copy phase occurs when the product is being installed; usually this phase is indicated by a progressbar. The Post-Installation phase is any part of the installation that occurs after the File Copy phase. This phasevaries widely between products; the time recorded in this phase may include a required reboot to finalize theinstallation or include the time the program takes to become idle in the system tray.To reduce the impact of disk drive variables, each product was copied to the Desktop before initializing installation.Each step of the installation process was manually timed with a stopwatch and recorded in as much detail aspossible. Where input was required by the end user, the stopwatch was paused and the input noted in the rawresults in parenthesis after the phase description.Performance BenchmarkPage 19 of 2512 March 2014
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwareWhere possible, all requests by products to pre-scan or post-install scan were declined or skipped. Where it wasnot possible to skip a scan, the time to scan was included as part of the installation time. Where an optionalcomponent of the installation formed a reasonable part of the functionality of the software, it was also installed(e.g. website link checking software as part of a Security Product).Installation time includes the time taken by the product installer to download components required in theinstallation. This may include mandatory updates or the delivery of the application itself from a downloadmanager. We have noted in our results where a product has downloaded components for product installation.We have excluded product activation times due to network variability in contacting vendor servers or time takenin account creation. For all products tested, the installation was performed directly on the endpoint, either usinga standalone installation package or via the management server web console.A product's Installation Size was previously defined as the difference between the initial snapshot of the Disk Space(C: drive) before installation and the subsequent snapshot taken after the product is installed on the system.Although this is a widely used methodology, we noticed that the results it yielded were not always reproduciblein Vista due to random OS operations that may take place between the two snapshots. We improved theInstallation Size methodology by removing as many Operating System and disk space variables as possible.Using PassMark’s OSForensics 2.2 we created initial and post-installation disk signatures for each product. Thesedisk signatures recorded the amount of files and directories, and complete details of all files on that drive (includingfile name, file size, checksum, etc) at the time the signature was taken.The initial disk signature was taken immediately prior to installation of the product. A subsequent disk signaturewas taken immediately following a system reboot after product installation. Using OSForensics, we compared thetwo signatures and calculated the total disk space consumed by files that were new, modified, and deleted duringproduct installation. Our result for this metric reflects the total size of all newly added files during installation.The scope of this metric includes only an ‘out of the box’ installation size for each product. Our result does notcover the size of files downloaded by the product after its installation (such as engine or signature updates), orany files created by system restore points, pre-fetch files and other temporary files.PassMark Software uses tools available from the Windows Performance Toolkit version 4.6 (as part of theMicrosoft Windows 7 SDK obtainable from the Microsoft Website) with a view to obtaining more precise andconsistent boot time results on the Windows 7 platform.The boot process is first optimized with xbootmgr.exe using the command “xbootmgr.exe -trace boot –prepSystem” which prepares the system for the test over six optimization boots. The boot traces obtained fromthe optimization process are discarded.After boot optimization, the benchmark is conducted using the command "xbootmgr.exe -trace boot -numruns 5”.This command boots the system five times in succession, taking detailed boot traces for each boot cycle.Performance BenchmarkPage 20 of 2512 March 2014
Webroot SecureAnywhere Endpoint Protection vs. Seven Endpoint Security ProductsPassMark SoftwareFinally, a post-processing tool was used to parse the boot traces and obtain the BootTimeViaPostBoot value. Thisvalue reflects the amount of time it takes the system to complete all (and only) boot time processes. Our finalresult is an average of five boot traces.CPUAvg is a command-line tool which samples the amount of CPU load two times per second. From this, CPUAvgcalculates and displays the average CPU load for the interval of time for which it has been active.For this metric, CPUAvg was used to measure the CPU load on average (as a percentage) during a period of systemidle for 500 samples. This tes
resources being consumed by the security software on a permanent basis. This metric measures the amount of memory (RAM) used by the product while the machine and security software are in an idle state. The total memory usage was calculated by identifying all the security software's processes and the amount of memory used by each process.
