WIXLIB

TABLE OF CONTENTSOVERVIEW . 2Methodology . 2Objectives . 2Terminology . 2Recommended Test Protocol . 3CLOUD SERVICES VS. DEDICATED HOSTING. 4SYSTEM RESOURCE MONITORING. 4MAGENTO CONFIGURATION SETTINGS . 5Use Flat Catalog . 5Merge JavaScript & CSS . 5Enable Full-Page Cache . 6Magento Indexers. 7SERVER CONFIGURATION SETTINGS . 8.htaccess. 8Temporary File Storage . 8Cron Job . 9LAMP STACK . 9Linux . 9Apache. 11MySQL . 11PHP . 12LNMP STACK. 13Linux . 13Nginx . 13MySQL . 14PHP . 14Copyright 2015 Magento, Inc. All rights reserved.i

UNSUPPORTED CONFIGURATIONS AND SETTINGS . 15Linux . 15Apache. 15MySQL . 15PHP . 16SAMPLE CONFIGURATIONS . 17MySQL Configuration . 17Nginx Configuration . 19EXAMPLE SYSTEM CONFIGURATIONS . 20System Hardware Specifications . 20Physical Servers . 20Cloud Servers . 20System Software Versions . 21Developer Test Environments . 21SERVER TEST ENVIRONMENTS (1 SERVER) . 22SERVER TEST ENVIRONMENTS (2 SERVERS) . 23MAGENTO SCALABILITY . 24Magento Scalability Tiers . 24Performance vs. Scalability . 24Magento Performance Testing Guidelinesii

EXECUTIVE SUMMARYPerformance testing is a critical step in thedevelopment process for merchants lookingto optimize their shopping experiences.This technical document provides afoundation for performance testing bydefining a common set of software andenvironment configuration settings toproduce reliable, reproducible, and consistentperformance test results.These Guidelines are meant to be used inconjunction with the Magento PerformanceToolkit, and together they provide a consistentruler to compare performance across a varietyof Magento implementations.Using the Guidelines and Toolkit to gatherperformance measurements across differentimplementations provides merchants or theirdevelopers with information on theperformance impacts of custom hardware,software, or network configurations,empowering them to optimize their web storeexperience for growth.Copyright 2015 Magento, Inc. All rights reserved.1

OVERVIEWMethodology1. Load a Magento instance on a server, or set of servers.2. Use the Magento Performance Toolkit to prepare the database for testing.3. Use Apache jMeter to run benchmarking scripts from a load generator to send traffic to the server(s) and collectperformance data.Objectives Enable merchants to test the performance of Magento installations Enable developers to test the performance impact of customizations Enable performance testing across the Magento ecosystem Enable the comparison of performance tests (benchmarking) Provide commonly used methods, scenarios, and terminology to describe testing activitiesTerminologyThe following terms have specific meaning within the context of Magento performance testing:PerformanceResponsiveness and stability under a particular workload.BenchmarkingComparing system performance metrics, typically to recommended best practices orindustry standards.Scalability1. The ability to process growing amounts of requests in a capable manner.2. The ability to expand the system resources to increase capacity.Reliability1. The ability to yield the same or compatible results for different tests.2. The ability for the system to operate as designed.Reproducibility1. The ability to conduct a performance test and observe similar results on multiple testoccurrences.2. The ability for a third party to independently duplicate a test and observe comparableresults.Copyright 2015 Magento, Inc. All rights reserved.2

Recommended Test ProtocolDocument the environment and test procedures so the results are comparable and reproducible.Record all control variables.Performance test control variables are divided into the following groups: Load Generator Merchant Traffic Profile Reference Store Magento Platform Environment Hardware Software Reference System Architecture Test ProtocolMinimize test variables.Minimize test variables. When testing to compare results, minimize the number of input variables that are subjectto change.Drive tests from different servers.To isolate system resource usage, place the load generator on a different system than the application server thatruns Magento and supporting services. Large test configurations might need multiple load generators to createenough traffic to stress the system.Establish a baseline.Establish a baseline set of results with known values for the full list of control variables. The recommended valuesfor several control variables are described later in this document.Don’t use the first set of results.For cached configurations, the first test cycle populates only the caching layers. For each cycle, record the results ofthe second and subsequent test runs.Repeat to verify results.Never consider the first set of results as final. Repeat and verify each test to establish that the results are reliableand reproducible by comparing the variations in results for each test cycle. A minimum of two or three test cycles isrecommended.Include the minimum number of external variables in your test environment.The network metric distance from the load generator to the application server should be the minimum distancethat is required to establish the scope of the test. Your intent should be to minimize the impact of externalvariables on the test results, and know which variables are impacting your results. A test from a load generator to a server on the same switch can be used to establish the scope of applicationperformance and hardware configuration. A test from an Internet source to a private data center can be used to establish the scope of Internet latency,CDN and caching servers, infrastructure equipment, application performance, and hardware configuration.Magento Performance Testing Guidelines3

CLOUD SERVICES VS. DEDICATED HOSTINGMagento was originally designed to be installed and run on a dedicated, physical server, and deployed as an onpremise application. The emerging trend is to install Magento in a data center on either a dedicated physical server,or using a cloud service provided by the host.Servers that are managed by a hosting provider and provisioned for multiple customers are referred to as cloudservices. A physical server that is used by a single customer is referred to as a dedicated host.SYSTEM RESOURCE MONITORINGIt is recommended that you collect system-level metrics during the execution of the benchmark, to make sure theresults are not based on a system resource that is over utilized or limited causing a resource bottleneck. At aminimum, monitor the usage for each of the following primary system resources: CPU utilization Memory allocation Disk IO activity Network usage Database transactionsTo increase your understanding of how these resources are used you want to collect detailed usage data duringyour test activities. You can use various tools to collect these metrics including jMeter test plan listeners, individualsystem level commands, and system monitoring tools such as nmon. Resource usage results over several test runscan be captured and visualized in tools like Graphite. Also, you may consider using SaaS monitoring tools like NewRelic to capture and visualize data.Magento Performance Testing Guidelines4

MAGENTO CONFIGURATION SETTINGSThe following configuration settings are required for each Magento store to be tested: Use Flat Catalog Merge JavaScript & CSS Enable Full-Page Cache Indexers Update and Clean on ScheduleUse Flat CatalogTo configure Magento to enable Flat Catalog for categories and products, do the following:1. On the Admin menu, select System Configuration. In the panel on the left, under Catalog, select Catalog.2. Click to expand the Frontend section. Then, make the following settings:Use Flat Catalog Category“Yes”Use Flat Catalog Product“Yes”Merge JavaScript & CSSTo enable file merging for JavaScript and CSS files, do the following:1. In the panel on the left, under Advanced, select Developer.2. Click to expand the JavaScript Settings section. Then, set Merge JavaScript Files to “Yes.”3. Click to expand the CSS Settings section. Then, set Merge CSS Files to “Yes.”4. When complete, click the Save Config button.Magento Performance Testing Guidelines5

Enable Full-Page CacheFor Magento Enterprise Edition:The Full-Page Cache is enabled by default for Magento Enterprise Edition. To verify that the page cache is enabled,do the following:Step 1: Verify that Page Cache is Enabled1. On the Admin menu, select System Cache Management.2. If the Status column shows the Page Cache to be “Disabled,” do the following:a. On the Admin menu, select System Configuration. In the panel on the left, under Advanced, selectSystem.b. Click to expand the Full Page Cache Settings section. Then, set Enable Full Page Cache Auto Generationto “Yes.”3. When complete, click the Save Config button.Step 2: Verify that Page Cache is ValidOn the server, examine the contents of the cache to verify that it is working. It is not possible to determine thelocation of the cache from the Magento Admin. Use one of the following methods to verify the cache:File SystemIf using the file system, verify that cached pages are being saved in the following folder: magento root /var/full page cacheRedisIf using Redis, make an API call to query data used by Magento.Magento Performance Testing Guidelines6

Magento IndexersFor Magento Enterprise Edition:All indexers should “Update when scheduled,” and the system should periodically purge the indexer changelog.1. On the Admin menu, select System Configuration. In the panel on the left, under Advanced, select IndexManagement.2. Click to expand the Index Options section. Then, verify that all indexers are set to “Update when scheduled.”3. Click to expand the Index Clean Schedule section. Then, verify the following settings:Enable Scheduled Cleanup“Yes”Start Time(As needed)Frequency(As needed)4. When complete, click the Save Config button.Magento Performance Testing Guidelines7

SERVER CONFIGURATION SETTINGS.htaccessRewrite Engine ONIn vi .htaccess, make sure that this line is not comment: #RewriteEngine OnThis is the default setting for the htaccess file included with Magento.RewriteEngine OnExpire Headers ONThe correct Expire Headers setting can be found in the default Magento htaccess file.The default setting is access time 1 year.Temporary File StorageConfigure the location of the following temporary files as needed for your system architecture: magento root /var/session magento root /var/cache magento root /var/full page cacheSYSTEM TYPESTORAGE LOCATIONFile SystemStore all caches and session in memory backed by file system, or store on a RAMdisk.RedisStore all caches and session in Redis on the application server.1 1Store all caches and session in database.1 1 with RedisOn Application Server:Store in Redis on the application server.On Dedicated Server:Store in Redis on the Redis server instance.2 1Store all caches and session in database.2 1 with RedisStore all caches and session in Redis on the Redis server instance. In thisexample, the load balancer can be either hardware or a server instance.Magento Performance Testing Guidelines8