Transcription
Consumer AntivirusPerformance Benchmarks (2011)Internet Security ProductsWindows 7September umer Antivirus Performance BenchmarksK. Lai, D. Wren, T. RowlingPassMark Software30 September 20102antivirus 11-performance-testing-ed2.docx
Internet SecurityConsumer Antivirus SoftwarePassMark SoftwareTable of ContentsTABLE OF CONTENTS. 2REVISION HISTORY. 4REFERENCES. 4EXECUTIVE SUMMARY . 5PRODUCTS AND VERSIONS . 6RANKING . 8PERFORMANCE METRICS SUMMARY . 9INTERNET SECURITY SOFTWARE – TEST RESULTS . 12BENCHMARK 1 – BOOT TIME . 12BENCHMARK 2 – SCAN TIME . 13BENCHMARK 3 – USER INTERFACE LAUNCH TIME . 13BENCHMARK 4 – MEMORY USAGE DURING SYSTEM IDLE . 14BENCHMARK 5 – BROWSE TIME . 14BENCHMARK 6 – INTERNET EXPLORER LAUNCH TIME . 15BENCHMARK 7 – INSTALLATION TIME . 15BENCHMARK 8 – INSTALLATION SIZE . 16BENCHMARK 9 – REGISTRY KEYS ADDED . 16BENCHMARK 10 – FILE COPY, MOVE AND DELETE . 17BENCHMARK 11 – INSTALLATION OF THIRD PARTY APPLICATIONS . 17BENCHMARK 12 – NETWORK THROUGHPUT . 18BENCHMARK 13 – FILE FORMAT CONVERSION . 18BENCHMARK 14 – FILE COMPRESSION AND DECOMPRESSION . 19BENCHMARK 15 – FILE WRITE, OPEN AND CLOSE . 19ANTIVIRUS SOFTWARE – TEST RESULTS . 20BENCHMARK 1 – BOOT TIME . 20BENCHMARK 2 – SCAN TIME . 20BENCHMARK 3 – USER INTERFACE LAUNCH TIME . 21BENCHMARK 4 – MEMORY USAGE DURING SYSTEM IDLE . 21BENCHMARK 5 – BROWSE TIME . 22BENCHMARK 6 – INTERNET EXPLORER LAUNCH TIME . 22BENCHMARK 7 – INSTALLATION TIME . 23BENCHMARK 8 – INSTALLATION SIZE . 23BENCHMARK 9 – REGISTRY KEYS ADDED . 24BENCHMARK 10 – FILE COPY, MOVE AND DELETE . 24BENCHMARK 11 – INSTALLATION OF THIRD PARTY APPLICATIONS . 25BENCHMARK 12 – NETWORK THROUGHPUT . 25BENCHMARK 13 – FILE FORMAT CONVERSION . 26BENCHMARK 14 – FILE COMPRESSION AND DECOMPRESSION . 26BENCHMARK 15 – FILE WRITE, OPEN AND CLOSE . 27BENCHMARK 16 – SCAN TIME OF A SOLID STATE DRIVE (SSD). 27Performance BenchmarkReport 2Page 2 of 3730 September 2010
Internet SecurityConsumer Antivirus SoftwarePassMark SoftwareDISCLAIMER AND DISCLOSURE . 28CONTACT DETAILS . 28APPENDIX 1 – TEST ENVIRONMENT . 29APPENDIX 2 – METHODOLOGY DESCRIPTION . 30Performance BenchmarkReport 2Page 3 of 3730 September 2010
Internet SecurityConsumer Antivirus SoftwarePassMark SoftwareRevision HistoryRevRevision HistoryDateReport 1Initial version of this report, includes new results for Norton Internet Security 2011, GData Internet Security 2011, Kaspersky Internet Security 2011, AVG Internet Security9.0, McAfee Internet Security 2010.Report 2Added Antivirus products, added new 2011 Internet Security products that havebecome available since the last report, added new SSD test metrics.24 August 201030 September 2010ReferencesRef #1DocumentWhat Really Slows Windows Down (URL)Performance BenchmarkReport 2AuthorO. Warner,The PC SpyDate2001-2009Page 4 of 3730 September 2010
Internet SecurityConsumer Antivirus SoftwarePassMark SoftwareExecutive SummaryPassMark Software conducted objective performance testing on eleven, publically available Internet Securitysoftware products and seven antivirus products on Windows 7 Ultimate Edition (64-bit) between July 2010 andSept 2010. This report includes newly obtained results from these current software products as well aspreviously obtained benchmark results from last generation software products released in 2010. Whereapplicable, we have tested new versions of software released for the year 2011.Subsequent editions of this report will include new products released for 2011 as they are made available. Formore details on which versions were tested, please see the section “Products and Versions”.Testing was performed on all products using sixteen performance metrics. These performance metrics are asfollows: Boot Time; Scan Time; User Interface Launch Time; Memory Usage during System Idle; Browse Time; Internet Explorer Launch Time; Installation Size; Installation Time; Registry Keys Added; File Copy, Move and Delete; Installation of Third Party Applications; Network Throughput (previously named “Binary Download Test”) File Format Conversion; File Compression and Decompression; and File Write, Open and Close Scan Time of a Solid State Drive (SSD) (Antivirus Products only)Performance BenchmarkReport 2Page 5 of 3730 September 2010
Internet SecurityConsumer Antivirus SoftwarePassMark SoftwareProducts and VersionsWhere possible, we have tested the full, retail release of the newest generation (2011 versions) of InternetSecurity and Anti-Virus software products. Where newer releases were not yet available, PassMark Softwarehas either re-tested the latest version of the software, or used previously obtained results for the lastgeneration (2010 versions) of Internet Security products.PassMark Software intends to test new, major releases of Internet Security products as they become publicallyavailable. In future editions on this report, results for newer versions of software will supersede previouslyobtained results for older versions of software.In this report, we have tested or included the following versions of Internet Security software:ManufacturerProduct NameRelease YearProduct VersionCurrentDate TestedTrend Micro Internet Security(Titanium) 20112010WebrootWebroot Internet Security 20112010Trend MicroTrend Micro VB 2011 (Japanese)2010BitDefenderBitDefender Internet Security 20112010SourceNext Virus Security ZERO2010Panda Internet Security 20112010Symantec CorporationNorton Internet Security 20112010Current18.0.0.128July 2010G Data Software AGG Data Internet Security 20112010Current21.0.2.1July 2010Kaspersky Internet Security 20112010Current11.0.0.239aJuly 2010AVG Internet Security 9.02010Current9.0.839July 2010McAfee Internet Security 20102010Current10.5.194July 2010ESET Smart Security 42010Legacy4.0.467.0August 2009F-Secure Internet Security 20102010Legacy10.00 build 246October 2009Trend MicroSourceNext CorporationPandaKaspersky LabsAVG TechnologiesMcAfee, IncESET,LLCF-Secure CorporationPerformance BenchmarkReport nt14.0.23.312Current10.0.0071Current16.00.00Sept 2010Sept 2010Sept 2010Sept 2010Sept 2010Sept 2010Page 6 of 3730 September 2010
Internet SecurityConsumer Antivirus SoftwarePassMark SoftwareIn this report, we have also tested or included the following versions of Antivirus software:ManufacturerProduct NameRelease YearNorton AV 20112010KasperskyKaspersky AV 20112010MicrosoftMicrosoft Security Essentials2010AvastAvast! Free AV 52010AviraAvira AntiVir Personal 102010G DataG Data AV 20112010PandaPanda AV 20112010Symantec CorporationPerformance BenchmarkReport 2Product 0.2.1Current16.00.00Date TestedSept 2010Sept 2010Sept 2010Sept 2010Sept 2010Sept 2010Sept 2010Page 7 of 3730 September 2010
Internet SecurityConsumer Antivirus SoftwarePassMark SoftwareRankingThis Overall Score provides a high level indication of that product’s performance compared to other productswe have tested. In this table,Internet Security ProductsIn the following table the highest possible score attainable is 195; in a hypothetical situation where a producthas attained first place in every metric. Internet Security products have been ranked by their overall score:Internet security Product NameOverall ScoreNorton IS 2011162ESET Smart Security 4134AVG IS 2010125Kaspersky IS 2011115G Data IS 2011114SourceNext Virus Security ZERO109McAfee IS 201098Panda IS 201195BitDefender IS 201188Trend Micro IS 201188Trend Micro VB 201188Webroot IS 201184F-Secure IS 201064Antivirus ProductsIn the following table the highest possible score attainable is 112; in a hypothetical situation where a producthas attained first place in every metric. Antivirus products have been ranked by their overall score:Product NamePerformance BenchmarkReport 2Overall ScoreNorton Antivirus 201181Avast Free Antivirus 578Avira AnitVir Personal 1076G Data Antivirus 201156Panda Antivirus 201155Kaspersky Antivirus 201154Microsoft Security Essentials48Page 8 of 3730 September 2010
Internet SecurityConsumer Antivirus SoftwarePassMark SoftwarePerformance Metrics SummaryWe have selected a set of objective metrics which provide a comprehensive and realistic indication of theareas in which an antivirus may impact system performance for end users. Our metrics test the impact of theantivirus software on common tasks that end-users would perform on a daily basis.All of PassMark Software’s test methods can be replicated by third parties using the same environment toobtain similar benchmark results. Detailed descriptions of the methodologies used in our tests are available as“Appendix 2 – Methodology Description” of this report.Benchmark 1 – Boot TimeThis metric measures the amount of time taken for the machine to boot into the operating system. Securitysoftware is generally launched at Windows startup, adding an additional amount of time and delaying thestartup of the operating system. Shorter boot times indicate that the application has had less impact on thenormal operation of the machine.Benchmark 2 – Scan TimeAll antivirus solutions have functionality designed to detect viruses and various other forms of malware byscanning files on the system. This metric measured the amount of time required to scan a set of clean files.Our sample file set comprised a total file size of 1.2 GB and was made up of files that would typically be foundon end-user machines, such as media files, system files and Microsoft Office documents.Benchmark 3 – User Interface Launch TimeThis metric provides an objective indication as to how responsive a security product appears to the user, bymeasuring the amount of time it takes for the user interface of the antivirus software to launch fromWindows. To allow for caching effects by the operating system, both the initial launch time and thesubsequent launch times were measured. Our final result is an average of these two measurements.Benchmark 4 – Browse TimeIt is common behavior for security products to scan data for malware as it is downloaded from the internet orintranet. This behavior may negatively impact browsing speed as products scan web content for malware. Thismetric measures the time taken to browse a set of popular internet sites to consecutively load from a localserver in a user’s browser window.Benchmark 5 – Memory Usage during System IdleThis metric measures the amount of memory (RAM) used by the product while the machine and antivirussoftware are in an idle state. The total memory usage was calculated by identifying all antivirus softwareprocesses and the amount of memory used by each process.The amount of memory used while the machine is idle provides a good indication of the amount of systemresources being consumed by the antivirus software on a permanent basis. Better performing products occupyless memory while the machine is idle.Performance BenchmarkReport 2Page 9 of 3730 September 2010
Internet SecurityConsumer Antivirus SoftwarePassMark SoftwareBenchmark 6 – Internet Explorer Launch TimeThis metric is one of many methods to objectively measure how much a security product impacts on theresponsiveness of the system. This metric measures the amount of time it takes to launch the user interface ofInternet Explorer 8. To allow for caching effects by the operating system, both the initial launch time and thesubsequent launch times were measured. Our final result is an average of these two measurements.Benchmark 7 – Installation TimeThe speed and ease of the installation process will strongly influence the user’s first impression of the antivirussoftware. This test measures the minimum installation time required by the antivirus software to be fullyfunctional and ready for use by the end user. Lower installation times represent antivirus products which arequicker for a user to install.Benchmark 8 – Installation SizeIn offering new features and functionality to users, antivirus software products tend to increase in size witheach new release. Although new technologies push the size limits of hard drives each year, the growing diskspace requirements of common applications and the increasing popularity of large media files (such as movies,photos and music) ensure that a product's installation size will remain of interest to home users.This metric aims to measure a product’s total installation size. This metric is defined as the total disk spaceconsumed by all new files added during a product's installation.Benchmark 9 – Registry Keys AddedA large registry increases a machine’s use of resources. This may negatively impact system performance,especially on much older machines. This test measures the amount of keys and values added to registry, afterrebooting the test machines, following a successful product installation. Lower numbers mean that a producthas added fewer keys during installation and had less impact on the registry.Benchmark 10 – File Copy, Move and DeleteThis metric measures the amount of time taken to move, copy and delete a sample set of files. The sample fileset contains several types of file formats that a Windows user would encounter in daily use. These formatsinclude documents (e.g. Microsoft Office documents, Adobe PDF, Zip files, etc), media formats (e.g. images,movies and music) and system files (e.g. executables, libraries, etc).Benchmark 11 – Installing Third Party ApplicationsThis metric measures the amount of time taken to install and uninstall third party programs. The installationspeed of third party applications may be impacted by antivirus behavior such as heuristics or real timemalware scanning.Benchmark 12 – Network ThroughputThe metric measures the amount of time taken to download a variety of files from a local server using theHyperText Transfer Protocol (HTTP), which is the main protocol used on the web for browsing, linking and datatransfer. Files used in this test include file formats that users would typically download from the web, such asimages, archives, music files and movie files.Performance BenchmarkReport 2Page 10 of 3730 September 2010
Internet SecurityConsumer Antivirus SoftwarePassMark SoftwareBenchmark 13 – File Format ConversionThis test measures the amount of time taken to convert an MP3 file to a WAV and subsequently, convert thesame MP3 file to a WMA format.Benchmark 14 – File Compression and DecompressionThis metric measures the amount of time taken to compress and decompress different types of files. Filesformats used in this test included documents, movies and images.Benchmark 15 – File Write, Open and CloseThis benchmark was derived from Oli Warner’s File I/O test at http://www.thepcspy.com (please seeReference #1: What Really Slows Windows Down). This metric measures the amount of time taken to write afile, then open and close that file.Benchmark 16 – Scan Time of a Solid State Drive (SSD)This metric measured the amount of time required to scan a set of clean files that are stored on a Solid StateDrive (SSD). Our sample file set comprised of 18,000 files and was made up of files that would typically befound on end-user machines, such as media files, system files and Microsoft Office documents.Performance BenchmarkReport 2Page 11 of 3730 September 2010
Internet SecurityConsumer Antivirus SoftwarePassMark SoftwareInternet Security Software – Test ResultsIn the following charts, we have highlighted the results we obtained for Norton Internet Security 2011 inyellow. The average has also been in highlighted in blue for ease of comparison.Benchmark 1 – Boot TimeThe following chart compares the average time taken for the system to boot (from a sample of five boots) foreach Internet Security product tested. Products with lower boot times are considered better performingproducts in this category.ESET Smart Security 425.90Trend Micro Virus Buster 201126.75Norton Internet Security 201126.92G Data Internet Security 201129.67SourceNext Virus Security ZERO29.92Trend Micro Internet Security 201130.21BitDefender Internet Security 201131.01Kaspersky Internet Security 201131.97McAfee Internet Security 201031.98Average33.69F-Secure Internet Security 201036.13AVG Internet Security 201037.38Panda Internet Security 201139.67Webroot Internet Security 201160.40010203040506070Boot Time (sec)Performance BenchmarkReport 2Page 12 of 3730 September 2010
Internet SecurityConsumer Antivirus SoftwarePassMark SoftwareBenchmark 2 – Scan TimeThe following chart compares the average time taken to scan a set of 6159 files (totaling 982 MB) for eachInternet Security product tested. This time is calculated by averaging the initial (Run 1) and subsequent (Runs2-5) scan times. Products with lower scan times are considered better performing products in this category.Norton Internet Security 201125.75G Data Internet Security 201129.00BitDefender Internet Security 201135.50F-Secure Internet Security 201040.00Webroot Internet Security 201151.88Kaspersky Internet Security 201152.50McAfee Internet Security 201053.23Average60.32Trend Micro Internet Security 201164.88ESET Smart Security 467.50SourceNext Virus Security ZERO75.61Panda Internet Security 201177.43Trend Micro Virus Buster 201185.63AVG Internet Security 2010125.25020406080100120140Scan Time (sec)Benchmark 3 – User Interface Launch TimeThe following chart compares the average time taken to launch a product’s user interface. Products with lowerlaunch times are considered better performing products in this category.McAfee Internet Security 201034.18ESET Smart Security 448.91BitDefender Internet Security 2011236.94G Data Internet Security 2011266.61SourceNext Virus Security ZERO395.27Norton Internet Security 2011430.41Kaspersky Internet Security 2011473.65AVG Internet Security 2010506.84Average731.36Webroot Internet Security 2011756.15Trend Micro Virus Buster 2011782.75Trend Micro Internet Security 2011784.25F-Secure Internet Security 20102,104.00Panda Internet Security 20112,687.6805001,0001,5002,0002,5003,000User Interface Launch Time (ms)Performance BenchmarkReport 2Page 13 of 3730 September 2010
Internet SecurityConsumer Antivirus SoftwarePassMark SoftwareBenchmark 4 – Memory Usage during System IdleThe following chart compares the average amount of RAM in use by an Internet Security product during aperiod of system idle. This average is taken from a sample of ten memory snapshots taken at roughly 60seconds apart after reboot. Products with lower idle RAM usage are considered better performing products inthis category.Panda Internet Security 2011Trend Micro Virus Buster 2011Norton Internet Security 2011SourceNext Virus Security ZEROKaspersky Internet Security 2011F-Secure Internet Security 2010BitDefender Internet Security 2011AverageAVG Internet Security 2010Trend Micro Internet Security 2011ESET Smart Security 4Webroot Internet Security 2011G Data Internet Security 2011McAfee Internet Security 9.5861.4266.5868.0570.0401020304050607080Memory Usage during System Idle (MB)Benchmark 5 – Browse TimeThe following chart compares the average time taken for Internet Explorer to successively load a set of popularwebsites through the local area network from a local server machine. Products with lower browse times areconsidered better performing products in this category.Norton Internet Security 201150.24Webroot Internet Security 201150.64Trend Micro Virus Buster 201155.00Panda Internet Security 201157.85Trend Micro Internet Security 201159.89Kaspersky Internet Security 201165.25AVG Internet Security 201065.47McAfee Internet Security 201071.57SourceNext Virus Security ZERO72.25Average75.83G Data Internet Security 201179.20ESET Smart Security 4106.27BitDefender Internet Security 2011106.43F-Secure Internet Security 2010145.79020406080100120140160Browse Time (sec)Performance BenchmarkReport 2Page 14 of 3730 September 2010
Internet SecurityConsumer Antivirus SoftwarePassMark SoftwareBenchmark 6 – Internet Explorer Launch TimeThe following chart compares the average launch times of Internet Explorer after rebooting the machine foreach Internet Security product we tested. Products with lower launch times are considered better performingproducts in this category.Kaspersky Internet Security 2011527.86Norton Internet Security 2011615.26McAfee Internet Security 2010663.21ESET Smart Security 4668.99Webroot Internet Security 2011698.50G Data Internet Security 2011791.54BitDefender Internet Security 2011815.73Trend Micro Internet Security 2011870.23SourceNext Virus Security ZERO1,015.39Average1,103.57Trend Micro Virus Buster 20111,124.55Panda Internet Security 20111,250.91AVG Internet Security 20101,328.92F-Secure Internet Security 04,500Internet Explorer Launch Time (ms)Benchmark 7 – Installation TimeThe following chart compares the minimum installation time it takes for Internet Security products to be fullyfunctional and ready for use by the end user. Products with lower installation times are considered betterperforming products in this category.Norton Internet Security 201141.31ESET Smart Security 442.85Kaspersky Internet Security 2011129.66SourceNext Virus Security ZERO169.19Panda Internet Security 2011180.82G Data Internet Security 2011191.29Trend Micro Internet Security 2011196.00Average266.13AVG Internet Security 2010312.65Webroot Internet Security 2011368.00BitDefender Internet Security 2011408.06Trend Micro Virus Buster 2011444.77F-Secure Internet Security 2010448.94McAfee Internet Security 2010526.120100200300400500600Installation Time (sec)* Our result for McAfee Internet Security 2010 includes time taken for the installer to download components as part of the installationprocess.Performance BenchmarkReport 2Page 15 of 3730 September 2010
Internet SecurityConsumer Antivirus SoftwarePassMark SoftwareBenchmark 8 – Installation SizeThe following chart compares the total size of files added during the installation of Internet Security products.Products with lower installation sizes are considered better performing products in this category.AVG Internet Security 2010149.57ESET Smart Security 4155.14SourceNext Virus Security ZERO176.45Norton Internet Security 2011217.07Panda Internet Security 2011250.21Trend Micro Internet Security 2011280.35McAfee Internet Security 2010307.82Average373.82Trend Micro Virus Buster 2011384.69Webroot Internet Security 2011408.16G Data Internet Security 2011435.24Kaspersky Internet Security 2011508.79F-Secure Internet Security 2010517.75BitDefender Internet Security 20111,068.4602004006008001,0001,200Installation Size (MB)Benchmark 9 – Registry Keys AddedThe following chart compares the amount of Registry Keys created during product installation for eachInternet Security product tested. Products with lower key counts are considered better performing products inthis category.ESET Smart Security 41,041.00SourceNext Virus Security ZERO2,005AVG Internet Security 20102,034F-Secure Internet Security 20102,115.00BitDefender Internet Security 20112,679Norton Internet Security 20113,047Panda Internet Security 20114,804G Data Internet Security 20116,563Kaspersky Internet Security 20117,431Webroot Internet Security 20118,574Trend Micro Virus Buster 20118,574Average9,700McAfee Internet Security 201015,919Trend Micro Internet Security 000Added Registry KeysPerformance BenchmarkReport 2Page 16 of 3730 September 2010
Internet SecurityConsumer Antivirus SoftwarePassMark SoftwareBenchmark 10 – File Copy, Move and DeleteThe following chart compares the average time taken to copy, move and delete several sets of sample files foreach Internet Security product tested. Products with lower times are considered better performing products inthis category.Norton Internet Security 201118.76AVG Internet Security 201020.16Trend Micro Internet Security 201120.22ESET Smart Security 420.28G Data Internet Security 201120.49Kaspersky Internet Security 201120.56McAfee Internet Security 201021.98Average22.61Panda Internet Security 201123.41BitDefender Internet Security 201123.55SourceNext Virus Security ZERO23.86Trend Micro Virus Buster 201125.60F-Secure Internet Security 201025.78Webroot Internet Security 201129.2305101520253035File Copy, Move and Delete (sec)Benchmark 11 – Installation of Third Party ApplicationsThe following chart compares the average time taken to install a third party application for each InternetSecurity product tested. Products with lower times are considered better performing products in this category.AVG Internet Security 201030.70Trend Micro Virus Buster 201133.65Norton Internet Security 201133.97Kaspersky Internet Security 201134.24ESET Smart Security 434.63Panda Internet Security 201134.88G Data Internet Security 201134.97BitDefender Internet Security 201135.62Average37.17McAfee Internet Security 201037.52SourceNext Virus Security ZERO37.85Trend Micro Internet Security 201143.90F-Secure Int
Data Internet Security 2011, Kaspersky Internet Security 2011, AVG Internet Security 9.0, McAfee Internet Security 2010. 24 August 2010 Report 2 Added Antivirus products, added new 2011 Internet Security products that have become available since the last report, added new SSD test metrics. 30 September 2010 References Ref # Document Author Date
