Transcription
DATA CENTER INTRUSION PREVENTION SYSTEMTEST REPORTFortinet FortiGate 3200D v5.4.10 GA Build 7811OCTOBER 30, 2018Authors – Keith Bormann, Ryan Turner, Matt Chips, Matt WheelerThis report is Confidential and is expressly limited to NSS Labs’ licensed users.
NSS LabsData Center Intrusion Prevention System (DCIPS) Test Report – Fortinet FortiGate 3200D v5.4.10 GABuild 7811 103018OverviewNSS Labs performed an independent test of the Fortinet FortiGate 3200D v5.4.10 GA Build 7811. The product wassubjected to thorough testing at the NSS facility in Austin, Texas, based on the Data Center Network Security(DCNS) Test Methodology v2.0,1 available at www.nsslabs.com. This test was conducted free of charge and NSS didnot receive any compensation in return for Fortinet’s participation.While the companion Comparative Reports on security, performance, and total cost of ownership (TCO) willprovide information about all tested products, this Test Report provides detailed information not availableelsewhere.NSS research indicates that DCIPS devices are typically deployed to protect data center assets, and mostenterprises will tune intrusion prevention system (IPS) modules within their DCIPS. Therefore, during NSS testing,DCIPS products are configured with a tuned policy setting in order to provide readers with relevant securityeffectiveness and performance dimensions based on their expected usage.ProductFortinet FortiGate 3200Dv5.4.10 GA Build 7811Exploit BlockRate2EvasionsBlockedStability &Reliability3-Year TCO(US )99.24%99/993PASS 116,100ResiliencyTransactionalUse CaseMultimediaUse CaseCorporate UseCase77.14%12,222 Mbps23,959 Mbps17,374 MbpsFigure 1 – Overall Test ResultsUsing the tuned policy, the Fortinet FortiGate 3200D v5.4.10 GA Build 7811 blocked 99.24% of exploits. The deviceproved effective against 99 out of 99 evasions it was tested against. The device passed all stability and reliabilitytests.To represent different types of traffic seen in a data center, NSS has created three different use cases:transactional, multimedia, and corporate. For each of these weighted use cases, NSS-Tested Throughput iscalculated by taking an average of the device’s IPv4 and IPv6 results. NSS Labs rates the FortiGate 3200Dthroughput as follows: Transactional use case: 12,222 Mbps Multimedia use case: 23,959 Mbps Corporate use case: 17,374 Mbps1This methodology covers a range of devices that provide network security for the data center, one of which is the data center intrusionprevention system (DCIPS). For more information, visit www.nsslabs.com.2Exploit block rate is defined as a percentage of the total number of exploits that are blocked under test.3In accordance with the industry standard for vulnerability disclosures and to provide vendors with sufficient time to add protection wherenecessary, NSS Labs will not publicly release information about which previously unpublished techniques were applied during testing until 90days after the publication of this document.This report is Confidential and is expressly limited to NSS Labs’ licensed users.2
NSS LabsData Center Intrusion Prevention System (DCIPS) Test Report – Fortinet FortiGate 3200D v5.4.10 GABuild 7811 103018Table of ContentsOverview. 2Security Effectiveness . 5False Positive Testing .5NSS Exploit Library .5Resiliency .5Coverage by Impact Type.5Coverage by Date.6Coverage by Target Vendor .6Resistance to Evasion Techniques .7Performance . 8Maximum Capacity .8HTTP Capacity .9Application Average Response Time – HTTP .10HTTP Capacity with HTTP Persistent Connections .10Single Application Flows .11Raw Packet Processing Performance (UDP Throughput) .11Raw Packet Processing Performance (UDP Latency) .12NSS-Tested Throughput: Use Cases . 13Stability and Reliability . 14Total Cost of Ownership (TCO) . 15Installation Hours .15Total Cost of Ownership .16Appendix A: Product Scorecard . 17Test Methodology . 23Contact Information . 23This report is Confidential and is expressly limited to NSS Labs’ licensed users.3
NSS LabsData Center Intrusion Prevention System (DCIPS) Test Report – Fortinet FortiGate 3200D v5.4.10 GABuild 7811 103018Table of FiguresFigure 1 – Overall Test Results.2Figure 2 – Number of Threats Blocked (%) .5Figure 3 – Resiliency Score .5Figure 4 – Product Coverage by Date .6Figure 5 –Product Coverage by Target Vendor .6Figure 6 – Resistance to Evasion Results .7Figure 7 – Concurrency and Connection Rates (IPv4 and IPv6) .9Figure 8 – HTTP Capacity with No Transaction Delays .9Figure 9 – Average Application Response Time (Milliseconds) .10Figure 10 – HTTP Capacity with HTTP Persistent Connections .10Figure 11 –– Single Application Flows .11Figure 12 – Raw Packet Processing Performance – UDP Traffic (IPv4) .12Figure 13 – UDP Latency in Microseconds.12Figure 14 – NSS-Tested Throughput Use Cases .13Figure 15 – Stability and Reliability Results .14Figure 16 – Device Installation Time (Hours) .15Figure 17 –3-Year TCO (US ) .16Figure 18 – Detailed Scorecard .22This report is Confidential and is expressly limited to NSS Labs’ licensed users.4
NSS LabsData Center Intrusion Prevention System (DCIPS) Test Report – Fortinet FortiGate 3200D v5.4.10 GABuild 7811 103018Security EffectivenessThis section verifies that the device can enforce the security policy effectively. Security effectiveness was testedover IPv4 only. For systems that may be exposed to threats over IPv6, NSS recommends that enterprises validatesecurity effectiveness using IPv6.False Positive TestingAny signature that blocks non-malicious traffic during false-positive testing is disabled for security testing.NSS Exploit LibraryNSS’ security effectiveness testing leverages the deep expertise of our engineers who utilize multiple commercial,open-source, and proprietary tools as appropriate. With more than 2,000 exploits, this is the industry’s mostcomprehensive test to date.ProductFortinet FortiGate 3200D v5.4.10 GA Build 7811Total Number ofThreats RunTotal Number ofThreats BlockedBlock Percentage2,2232,20699.24%Figure 2 – Number of Threats Blocked (%)ResiliencyNSS also measured the resiliency of a device by introducing previously unseen variations of a known exploit andmeasuring the device’s effectiveness against them. Figure 3 depicts the resiliency score.ProductBlock PercentageFortinet FortiGate 3200D v5.4.10 GA Build 781177.14%Figure 3 – Resiliency ScoreCoverage by Impact TypeThe most serious exploits are those that result in a remote system compromise, providing the attacker with theability to execute arbitrary system-level commands. Most exploits in this class are “weaponized” and offer theattacker a fully interactive remote shell on the target server. Slightly less serious are attacks that result in anindividual service compromise, but not arbitrary system-level command execution. Finally, there are attacks thatresult in a system- or service-level fault that crashes the targeted service or application and requires administrativeaction to restart the service or reboot the system. Clients can contact NSS for more information about these tests.This report is Confidential and is expressly limited to NSS Labs’ licensed users.5
Data Center Intrusion Prevention System (DCIPS) Test Report – Fortinet FortiGate 3200D v5.4.10 GABuild 7811 103018NSS LabsCoverage by DateFigure 4 provides insight into whether or not a vendor is aging out protection signatures aggressively enough topreserve performance levels. It also reveals whether a product lags behind in protection for the most currentvulnerabilities. NSS reports exploits by individual years for the past 10 years. Exploits older than 10 years aregrouped 0.0% 20082009201020112012201320142015201620172018Caught %Missed %Figure 4 – Product Coverage by DateCoverage by Target VendorExploits within the NSS Exploit Library target a wide range of protocols and applications. Figure 5 depicts thecoverage offered for five of the top vendors targeted in this test. Clients can contact NSS for more %0%Figure 5 –Product Coverage by Target VendorThis report is Confidential and is expressly limited to NSS Labs’ licensed users.6
NSS LabsData Center Intrusion Prevention System (DCIPS) Test Report – Fortinet FortiGate 3200D v5.4.10 GABuild 7811 103018Resistance to Evasion TechniquesEvasion techniques are a means of disguising and modifying attacks at the point of delivery to avoid detection andblocking by security products. Failure of a security device to correctly identify a specific type of evasion potentiallyallows an attacker to use an entire class of exploits for which the device is assumed to have protection. Thisrenders the device virtually useless. Many of the techniques used in this test have been widely known for yearsand should be considered minimum requirements for the DCIPS product category.Providing exploit protection results without fully factoring in evasion can be misleading. The more classes ofevasion that are missed (such as IP packet fragmentation, stream segmentation, RPC fragmentation, URLobfuscation, and FTP evasion), the less effective the device. For example, it is better to miss all techniques in oneevasion category, such as FTP evasion, than it is to miss one technique in each category, which would result in abroader attack surface.Furthermore, evasions operating at the lower layers of the network stack (IP packet fragmentation or streamsegmentation) have a greater impact on security effectiveness than those operating at the upper layers (URL orFTP obfuscation). Lower-level evasions will potentially impact a wider number of exploits; missing TCPsegmentation, for example, is a much more serious issue than missing FTP obfuscation.The resiliency of a system can be defined as its ability to absorb an attack and reorganize around a threat. When anattacker is presented with a vulnerability, the attacker can select one or more paths to trigger the vulnerability.NSS will measure a device’s resiliency by introducing a vulnerability along with its triggers and then asking thedevice to protect against the vulnerability. NSS will introduce various, previously unseen variations of exploits toexploit the vulnerability and measure the device’s effectiveness against them.A resilient device will be able to detect and prevent against different variations of the exploit. For more, see theEvasions Test Methodology v1.1 at www.nsslabs.com.Figure 6 provides the results of the evasion tests for the FortiGate 3200D. The FortiGate 3200D blocked all 99 ofthe evasions it was tested against. For further detail, please reference Appendix A.Test ProcedureResultRPC FragmentationPASSURL ObfuscationPASSFTP/Telnet EvasionPASSIP Packet Fragmentation TCP SegmentationPASSResiliency4PASSAttacks on nonstandard ports5Figure 6 – Resistance to Evasion Results4The results of resiliency testing are included in the Exploit Block Rate calculations.5Enterprises should be aware of the importance of performing deep packet inspection on all packets and ports and over all protocols in orderto secure applications effectively.This report is Confidential and is expressly limited to NSS Labs’ licensed users.7
NSS LabsData Center Intrusion Prevention System (DCIPS) Test Report – Fortinet FortiGate 3200D v5.4.10 GABuild 7811 103018PerformanceThere is frequently a trade-off between security effectiveness and performance. Because of this trade-off, it isimportant to judge a product’s security effectiveness within the context of its performance and vice versa. Thisensures that new security protections do not adversely impact performance and that security shortcuts are nottaken to maintain or improve performance. Performance was tested over IPv4 and IPv6 protocols for all testsexcept the UDP tests, where performance was tested only over the IPv4 protocol.In addition, when considering a security device (e.g., an IPS) for the data center rather than for the networkperimeter, there are several key metrics that must be adjusted. Performance metrics, while important in anysecurity device, become critical in a device that is intended for data center deployment. In a data center, thevolume of traffic is significantly higher than it would be for a device that is intended to protect end user desktopsbehind the corporate network perimeter. A data center security device also needs to support much higher datarates as it handles traffic for potentially hundreds of thousands of users who are accessing large applications in aserver farm inside the network perimeter. Connection rate and concurrent connection capacity are additionalmetrics that become even more important in a data center security device.The mix of traffic will differ significantly between a corporate network perimeter and a data center, and this canput additional load on the IPS inspection process. Stateless UDP traffic (such as that seen in a network file system[NFS]) and long-lived transmission control protocol (TCP) connections {as would be seen in an iSCSI storage areanetwork [SAN] or backup application) are common in many data center networks. These types of applicationspresent a continuous and heavy load for the network.Within the data center, application traffic puts a very different load on the network than does file system traffic.Communications between users and servers have different profiles than do communications between applications,databases, and directory servers. Application traffic is connection-intensive, with connections constantly being setup and torn down. A DCIPS that includes any application awareness capabilities will find significant challenges indata center deployments. Another critical concern is latency, since applications will be adversely affected if theDCIPS introduces delays.Maximum CapacityThe use of traffic generation appliances allows NSS engineers to create “real-world” traffic at multi-Gigabit speedsas a background load for the tests. The aim of these tests is to stress the inspection engine and determine how itcopes with high volumes of TCP connections per second, application layer transactions per second, and concurrentopen connections. All packets contain valid payload and address data, and these tests provide an excellentrepresentation of a live network at various connection/transaction rates.Note that in all tests the following critical “breaking points”—where the final measurements are taken—are used: Excessive concurrent TCP connections – Latency within the device is causing an unacceptable increase in openconnections. Excessive concurrent HTTP connections – Latency within the device is causing excessive delays and increasedresponse time. Unsuccessful HTTP transactions – Normally, there should be zero unsuccessful transactions. Once theseappear, it is an indication that excessive latency within the device is causing connections to time out.This report is Confidential and is expressly limited to NSS Labs’ licensed users.8
Data Center Intrusion Prevention System (DCIPS) Test Report – Fortinet FortiGate 3200D v5.4.10 GABuild 7811 103018NSS LabsFigure 7 depicts the results of the IPv4 and IPv6 tests for maximum 0,0004,000,000100,000Concurrent ConnectionsConnections / Transactions per Second600,0002,000,0000without dataMax. TCP CPS (IPv4)216,100Max. TCP CPS (IPv6)220,100Max. HTTP CPS (IPv4)192,000Max. HTTP CPS (IPv6)184,000Max. HTTP TPS (IPv4)499,900Max. HTTP TPS (IPv6)0with data476,400Max. Concurrent TCP Connections (IPv4)15,182,84518,903,883Max. Concurrent TCP Connections (IPv6)14,780,25718,718,874Figure 7 – Concurrency and Connection Rates (IPv4 and IPv6)HTTP CapacityThe aim of this test is to stress the HTTP detection engine and determine how the device copes with network loadsof varying average packet size and varying connections per second. By creating genuine session-based traffic withvarying session lengths, the device is forced to track valid TCP sessions, thus ensuring a higher workload than forsimple packet-based background traffic. This provides a test environment that is as close to real-world conditionsas possible, while ensuring absolute accuracy and repeatability.Each transaction consists of a single HTTP GET request. All packets contain valid payload (a mix of binary and ASCIIobjects) and address data. This test provides an excellent representation of a live network (albeit one biasedtoward HTTP traffic) at various network loads.Megabits per ,76410,00011,5605,00007,0404,1203,9987,00044 KBResponse21 KBResponse10 KBResponse4.5 KBResponse1.7 KBResponseIPv4 CPS48,40079,400118,400140,800164,800IPv6 CPS48,19078,820115,600140,000159,900IPv4 (Mbps)19,36015,88011,8407,0404,120IPv6 tions per SecondFigure 8 depicts the results of the IPv4 and IPv6 tests for HTTP capacity with no transaction delays.Figure 8 – HTTP Capacity with No Transaction DelaysThis report is Confidential and is expressly limited to NSS Labs’ licensed users.9
Data Center Intrusion Prevention System (DCIPS) Test Report – Fortinet FortiGate 3200D v5.4.10 GABuild 7811 103018NSS LabsApplication Average Response Time – HTTPApplication Average Response Time – HTTP (at 95% Maximum Load)IPv4 ResultsIPv6 Results2,500 Connections per Second – 44 KB Response2.903.125,000 Connections per Second – 21 KB Response3.593.8710,000 Connections per Second – 10 KB Response6.357.0920,000 Connections per Second – 4.5 KB Response5.605.7340,000 Connections per Second – 1.7 KB Response5.995.66Figure 9 – Average Application Response Time (Milliseconds)HTTP Capacity with HTTP Persistent ConnectionsThe aim of this test is to determine how the DCIPS copes with network loads of varying average packet size andvarying connections per second while inspecting traffic. By creating genuine session-based traffic with varyingsession lengths, the DCIPS is forced to track valid TCP sessions, thus ensuring a higher workload than for simplepacket-based background traffic. This provides a test environment that is as close to real-world conditions as it ispossible to achieve in a lab environment, while ensuring absolute accuracy and repeatability.This test will use HTTP persistent connections, with each TCP connection containing 10 HTTP GETs and associatedresponses. All packets contain valid payload (a mix of binary and ASCII objects) and address data, and this testprovides an excellent representation of a live network at various network loads. The stated response size is thetotal of all HTTP responses within a single TCP session.Megabits per ,00023,29018,04218,03215,00010,0005,0000HTTP 250 CPSHTTP 500 CPSHTTP 1000 CPSIPv4 CPS8,34510,68016,560IPv6 CPS8,04110,68016,550IPv4 (Mbps)35,72423,29018,042IPv6 ,0008,0006,0004,0002,0000Connections per SecondFigure 10 depicts the results of the IPv4 and IPv6 tests for HTTP capacity with HTTP persistent connections.Figure 10 – HTTP Capacity with HTTP Persistent ConnectionsThis report is Confidential and is expressly limited to NSS Labs’ licensed users.10
Data Center Intrusion Prevention System (DCIPS) Test Report – Fortinet FortiGate 3200D v5.4.10 GABuild 7811 103018NSS LabsSingle Application FlowsThis test measures the performance of the device with single application flows. For details about single applicationflow testing, see the NSS Labs Data Center Network Security (DCNS) Test Methodology v2.0, available atwww.nsslabs.com.35,00031,830 32,74030,00026,853 27,043Mbps25,00020,00015,00013,870 14,9279,900 10,18010,0005,211 5,8965,0000DBFinancialFile 614,9275,89632,74027,04310,180Figure 11 –– Single Application FlowsRaw Packet Processing Performance (UDP Throughput)This test uses UDP packets of varying sizes generated by test equipment. A constant stream of the appropriatepacket size, with variable source and destination IP addresses transmitting from a fixed source port to a fixeddestination port, is transmitted bidirectionally through each port pair of the device.Each packet contains dummy data and is targeted at a valid port on a valid IP address on the target subnet. Thepercentage load and frames per second (fps) figures across each inline port pair are verified by network monitoringtools before each test begins. Multiple tests are run and averages are taken where necessary.This traffic does not attempt to simulate any form of a “real-world” network condition. No TCP sessions arecreated during this test, and there is very little for the detection engine to do. However, each vendor is required towrite a signature to detect the test packets in order to ensure that they are being passed through the detectionengine and are not being “fast-pathed.”The aim of this test is to determine the raw packet processing capability of each inline port pair of the device, andto determine the device’s effectiveness at forwarding packets quickly in order to provide the highest level ofnetwork performance with the lowest amount of latency.This report is Confidential and is expressly limited to NSS Labs’ licensed users.11
Data Center Intrusion Prevention System (DCIPS) Test Report – Fortinet FortiGate 3200D v5.4.10 GABuild 7811 103018NSS LabsFigure 12 depicts the results of the IPv4 tests for raw packet processing its per 035,50035,000MbpsLatency (μs)6.05.05.536,1103.18.0Latency (μs)38,0001.064 BytePackets128 BytePackets256 BytePackets512 BytePackets1024 BytePackets1514 3.23.64.25.56.9-Figure 12 – Raw Packet Processing Performance – UDP Traffic (IPv4)Raw Packet Processing Performance (UDP Latency)DCIPS that introduce high levels of latency lead to unacceptable response times for users, especially wheremultiple security devices are placed in the data path. Figure 13 depicts UDP latency (in microseconds) as recordedduring the UDP throughput tests at 95% of the maximum load for IPv4.Latency – UDPIPv4 Results64-Byte Packets3128-Byte Packets3256-Byte Packets4512-Byte Packets41024-Byte Packets61514-Byte Packets7Figure 13 – UDP Latency in MicrosecondsThis report is Confidential and is expressly limited to NSS Labs’ licensed users.12
NSS LabsData Center Intrusion Prevention System (DCIPS) Test Report – Fortinet FortiGate 3200D v5.4.10 GABuild 7811 103018NSS-Tested Throughput: Use CasesBecause data center network traffic can vary greatly between industries and enterprises, NSS has created threeseparate use cases. Each use case weights test results in order to align them with the different use cases seen in adata center, i.e., transactional, multimedia or corporate.The transactional use case is intended to represent a data center with traffic that is more transactional in nature.An example of this may include B2B (business-to-business) or B2C (business-to-consumer) e-commerce. The ratedthroughput emphasizes smaller packet sizes and connections per second.The multimedia use case is intended to represent a data center whose purpose is to serve media content. Therated throughput emphasizes larger packet sizes, maximum concurrent sessions, and streaming protocols.The corporate use case may be best described as the data center footprint of a typical enterprise, where missioncritical applications such as email and ERP (enterprise resource planning software) are kept. The rated throughputemphasizes various packet sizes and protocols that are more likely to be found in those situations, such as email,database, and file sharing.Use CaseIPv4IPv6ResultsTransactional (small packets, database, email)12,27812,16512,222Multimedia (video, large packets, database, email)24,00623,91123,959Corporate (email, file share, database, mix of packet sizes)17,54217,20717,374Figure 14 – NSS-Tested Throughput Use CasesThis report is Confidential and is expressly limited to NSS Labs’ licensed users.13
NSS LabsData Center Intrusion Prevention System (DCIPS) Test Report – Fortinet FortiGate 3200D v5.4.10 GABuild 7811 103018Stability and ReliabilityLong-term stability is particularly important for an inline device, where failure can produce network outages. Thesetests verify the stability of the device along with its ability to maintain security effectiveness while under normalload and while passing malicious traffic. Products that cannot sustain legitimate traffic (or that crash) while underhostile attack will no
NSS Labs Data Center Intrusion Prevention System (DCIPS) Test Report - Fortinet FortiGate 3200D v5.4.10 GA Build 7811_103018 This report is Confidential and is expressly limited to NSS Labs' licensed users. 6 Coverage by Date Figure 4 provides insight into whether or not a vendor is aging out protection signatures aggressively enough to
