WIXLIB

Toward High-Performance and Scalable Network Functions VirtualizationServers run the Ubuntu 12.04long-term support (LTS) operatingsystem and are physically interconnected on one NIC port through afull-duplex 10-Gbps small form-factor pluggable (SFP ) cable. The virtualization layer uses Quick Emulator(QEMU 1.0) and Libvirt 0.9.8. EachVM has one virtual CPU (VCPU), 512Mbyte memory, 3-Gbyte disk space,and runs Ubuntu 12.04 LTS OS.We created virtual functions (VFs)from the 10-GbE port using SR-IOV.Each VM is assigned one VF, whichappears as an Ethernet interface. Themaximum number of VFs on a portis 63, hence the maximum number ofVMs sharing one port is 63.NFV ApplicationsTrafficgeneratorDPDK APPsDPDK APPsVF10G port10G portVFVirtual machine (VM)1 to 63 VMsDPDK APPsKVM hypervisorFigure 1. Testbed. One server acts as the traffic generator while the other isthe NFV server. (VF stands for virtual functions.)DPDK is a set of libraries and driversfor building fast packet-processingapplications. We build two NFV applicationsusing DPDK.Layer 2 forwarding. This reads the packet’sheader to get the destination media access control (MAC) address, which then is replaced withthe packet generator MAC address and forwardsthe packet back. There’s one receive (RX) queueand one transmission (TX) queue. This application has basic packet modification and forwarding operations with minimal queue/coreconfiguration. Hence, it serves as the baselinescenario for studying software packet-processing performance in baremetal and virtualization environments.Layer 3 forwarding using Cuckoo hashing. Compared to layer 2 forwarding, layer 3 forwardinghas sophisticated packet processing and providesthe router’s core functionality, which basicallyreads a packet’s IP header and searches the routing table to find the transmitting port. Examplelayer 3 applications in DPDK lack practicality,as they only support small routing tables. Tomake it more realistic and efficient, we leverage the CuckooSwitch technologies proposed byDong Zhou and colleagues3 and build a layer 3forwarding application supporting 1 billion forwarding information base (FIB) entries and linerate throughput (10 Gbps) with 256-byte or largerpackets.november/december 2016 NFV serverLayer 2 and layer 3 forwarding representbasic data plane operations shared across allVNFs: receiving, processing, and forwardingpackets. Hence, our experiments reveal the VNFdata plane performance on layer 3 and lower.The performance on higher layers can be studied using application-level VNFs such as videoserver middleboxes,4 which is out of the scopeof this article.Traffic GeneratorPktgen-DPDK (see https://github.com/Pktgen/Pktgen-DPDK) is a DPDK-based traffic generator running on a commodity x86 server. It generates 10-Gbps traffic with 64-byte networkframes. However, it can’t measure latency andjitter so we built an enhanced open source version available on github (see y-Jitter).The traffic generator sends packets to theNFV server. The DPDK applications in eitherVMs or the baremetal server forward the packets back to the traffic generator, which thencaptures the performance metrics. To test layer2 forwarding performance in baremetal, thetraffic generator sends packets with a line-rateof 10 Gbps. Fixing the total traffic throughput,we vary the packet sizes from 64 to 1,024 bytes.To study VNFs, the traffic generator generatespackets with the N MAC addresses in a roundrobin, assuming there are N VNFs. Each VNF13

(a)1248No. of VMs16120104Jitter (microseconds)14121086420Latency (microseconds log 10)Throughput (Gbps)Network Function Virtualization103102101100(b)Worst case1248No. of VMsRandomNUMA10080604020016(c)0.610.60.6248No. of VMs0.60.616Bare metalFigure 2. Layer 2 forwarding performance in the underprovisioning scenario. (a) Throughput. (b) Latency. (c) Jitter.receives the packet with its MAC address andforwards it back to the traffic generator, whichthen captures the aggregated performancemeasurements. In the layer 3 forwarding test,besides configuring MAC addresses in a roundrobin, the traffic generator randomly picks oneof the M IP addresses in a VNF’s lookup tableand writes it to the packet header. In this way,lookup keys are evenly distributed among hashtable entries in each VNF.Performance StudyHere, we consider the performance of our network in a variety of scenarios.Underprovisioning ScenarioThe x86 server has a NUMA architecture, inwhich lcores, NIC ports, and memory units connected to the same CPU socket are consideredin the same NUMA domain. Communicationbetween different NUMA domains is throughIntel QuickPath Interconnect (QPI). Accessinglocal memory is significantly faster than accessing memory in another NUMA domain. Our testbed has 32 lcores; 16 of them are in the sameNUMA domain as the NIC port, which is sharedby all the VMs. When the number of lcores islarger than the number of VNFs, there are threeVNF placement strategies: the NUMA-awarestrategy, which statically assigns a local lcore toeach VM; the random strategy, which relies onthe Linux kernel scheduler to dynamically allocate CPU resources at runtime; and the worstcase strategy, in which each VM was staticallyassigned an lcore in the remote NUMA domain.In the layer 2 forwarding application, thepacket generator evenly distributes 10 Gbps oftraffic to all the VNFs. As Figure 2 shows, the14www.computer.org/internet/ VNFs have overall lower performance thanbaremetal, even though in the baremetal casethe host uses only one lcore, while the VMs useone lcore each to forward the same amount oftraffic. For instance, in Figure 2b, the latency ofbaremetal is lower than the latency of the VMswith any strategy. This result points to the importance of looking at all the metrics other than justthroughput: if we solely looked at throughput, theperformance difference between NUMA-awareand baremetal would have been negligible.Based on the results, we learn that evenwith sufficient resources, NFV has lower performance than baremetal. With worst-caseplacement, VNFs need to access remote lcoresthrough QPI. As VNFs compete for using QPI,the packet-processing time is prolonged, leadingto queuing delay and packet loss. With randomplacement, the remote access penalties and QPIcontention are reduced because each VNF has aprobability to have a local lcore. When a VNF isallocated in a remote NUMA domain, however,the packet processing will be delayed. NUMAaware placement dedicates one local lcore foreach VNF, so it has the performance closest tobaremetal. However, NFV has an extra virtualization layer, the overhead in which largelycontributes to the end-to-end latency. We provide latency analysis later in the article.From a scalability perspective, the NFVperformance generally decreases as the number of VMs increases. Among the three placement strategies, scale has the smallest impacton NUMA-aware placement. In contrast, asthe number of VNFs increases, the worst-casestrategy’s performance further declines asthe resource contention on the QPI increasesaccordingly. The increased processing timeIEEE INTERNET COMPUTING

(a)108642064128256512 1,024Packet size (bytes)105100Jitter (microseconds)12Latency (microseconds log 10)Aggregated throughput (Gbps)Toward High-Performance and Scalable Network Functions Virtualization104103102101100(b)Worst case64128 256 512 1,024Packet size (bytes)RandomNUMA806040200(c)1.5640.81,7128 256 512 1,024Packet size (bytes)Bare metal – 16MFigure 3. Layer 3 forwarding performance in the underprovisioning scenario. (a) Throughput. (b) Latency. (c) Jitter.leads to significantly prolonged queuing delayin both hypervisor and VNF, causing about70 percent of the packets to be dropped. Fromthis, we learned that NUMA-aware scales well,whereas using random or worst-case strategy,packet drop, and performance degradationincrease significantly as NFV scales.In the layer 3 forwarding experiments, eachVNF has 1 million entries in the lookup tableand the counterpart baremetal lookup table hasa table size equal to the sum of all the entriesin all VNFs for each given scenario. Figures 3illustrates the performance (as a function ofpacket size) of 16 VNFs and the baremetal performance with 16 million entries. The throughput increases as the packet size increases,because the number of packets decreases whenthe total incoming traffic is fixed at 10 Gbps.The comparison among the three allocationstrategies provides the same insight we obtainedon NFV performance versus baremetal. We alsofind that for small packets (64 and 128 bytes),16 VNFs have better throughput performancethan baremetal. The reason is 16 VNFs have16 lcores to look up the 16 routing tables in parallel, while baremetal uses one lcore to search a16-times-larger routing table.We ran layer 3 experiments with differentVNF quantities and found similar results thatvalidate our lessons on NFV scalability. For brevity, we don’t present the detailed results here.Overprovisioning ScenarioIn the overprovisioning scenario, where thereare more VNFs than lcores, it isn’t possible topin a local lcore to every VNF as the NUMAaware placement, with neither worst-case strategy allocating to each VM a different dedicatednovember/december 2016 remote lcore. Therefore, we use the Linux scheduler here for VNF placement.Figure 4 shows the layer 2 forwarding performance of 32 and 63 VNFs (in our test, 63 is themaximum number of virtual functions supportedby SR-IOV on one NIC port). It’s apparent thatbaremetal substantially outperforms NFV. Fromthe layer 3 forwarding results shown in Figures 5and 6, 32 and 63 VNFs have packet loss at allpacket sizes, whereas baremetal reaches the linerate when packet size is 512 bytes or larger. Overprovisioning also has higher latency and jitterin most cases. The 63 VNFs have slightly lowerlatency than baremetal with small packets sizes(64 and 128 bytes), because they use all 32 lcoresbut baremetal only uses one lcore. The increase inCPU resources offsets the virtualization’s impacton latency.We also study the scalability transitionfrom underprovisioning to overprovisioning inFigure 4. We find that no more than 16 VNFswith the NUMA-aware strategy are able to sustain 10 Gpbs without packet loss. Scaling thenumber of VNFs over 16 can’t avoid substantialpacket loss, ranging from approximately 40 to60 percent.Hosting VNFs more than total lcores willresult in NUMA violation, which delays thepacket processing. The resource contention andcontext switch between VNFs increases the virtualization overheads. This lets us conclude thatNFV performance declines substantially in theoverprovisioning scenario as VNFs compete forresources and violate NUMA locality.Latency AnatomyTo study bottlenecks causing performance degradation in overprovisioning, we first use 2.5-Gbps15

64(a)128 256 512 1,024Packet size (bytes)Jitter (microseconds)1614121086420Latency (microseconds log 10)Throughput (Gbps)Network Function Virtualization10410310210110064(b)16 VMs NUMA128 256 512 1,024Packet size (bytes)16 VMs random32 VMs63 VMs1401201008060402000.564(c)0.70.60.11.4128 256 512 1,024Packet size (bytes)Bare metal108642064(a)128 256 512 1,024Packet size (bytes)105Jitter (microseconds log 10)Aggregated throughput (Gbps)12Latency (microseconds log 10)Figure 4. Layer 2 forwarding performance in oversubscription scenario. (a) Throughput. (b) Latency. (c) Jitter.10410310210110064(b)Random128 256 512 1,024Packet size (bytes)1031021010.810064(c)128 256 512 1,024Packet size (bytes)Bare metal 32M(a)108642064128 256 512 1,024Packet size (bytes)105Jitter (microseconds log 10)12Latency (microseconds log 10)Aggregated throughput (Gbps)Figure 5. Layer 3 forwarding performance of 32 virtual network functions (VNFs). (a) Throughput. (b) Latency. (c) Jitter.10410310210110064(b)Random128 256 512 1,024Packet size (bytes)1031021010.910064(c)128 256 512 1,024Packet size (bytes)Bare metal 63MFigure 6. Layer 3 forwarding performance of 63 VNFs. (a) Throughput. (b) Latency. (c) Jitter.input traffic (25 percent of the link capacity) torule out traffic congestion effects. Figures 7a and7b show the same insights learned from full loadscenarios. NUMA-aware placement with dedicated cores (the underprovisioning case) outper16www.computer.org/internet/ forms other strategies as well as ones withoutdedicated cores (the overprovisioning case).Overall, the latency is significantly lower thanthe latency with full-load input traffic. Especially, the 16 VNFs with NUMA-aware strategyIEEE INTERNET COMPUTING

Toward High-Performance and Scalable Network Functions Virtualization252015105024(a)816No. of VMs3263454035302520151050103Latency (microseconds)3530Latency (microseconds)Latency (microseconds)402(b)Worst case4816No. of VMsRandomNUMA321021011006316(c)32No. of VMs63Bare metalLessons LearnedThis empirical performance study teaches usthe following lessons: It’s difficult to scale the number of VNFs tomore than the number of lcores without substantial performance loss or low link use. A static, NUMA-aware placement strategyis crucial for achieving reliability and highperformance.november/december 2016 16 VM32 VM63 VM10080604020016 VM32 VM63 VMhave near-baremetal latency, which suggeststhat NFV can perform well when traffic is low.The latency of 32 VNFs is higher than that of 63VNFs, because their load per VNF nearly doubles. Results in Figure 7c validate the reasoning,because with the same load per VNF the latencyincreases as the number of VNFs increases.The same reasoning applies to Figure 2b,where the latency of 16 VNFs is lower than8 VNFs.The end-to-end latency includes three parts:the packet generator latency spent in the traffic generator; the VM latency spent in theVM to process the packets; and the hypvervisor latency spent in the virtualization layerfor activities such as queuing and scheduling.Results in Figure 8 show that the hypervisorconsumes up to 80 percent of the end-to-endlatency in both the partial- and full-load scenarios. Furthermore, the percentage of hypervisor latency increases when the traffic increasesfrom 2.5 to 10 Gbps, showing exaggerated overheads in the virtualization layer when the NFVserver has more traffic.PercentageFigure 7. Latencies with various load use. Non-uniform memory access (NUMA)-aware and worst-case placements areunavailable when the number of VNFs 32. (a) Layer 2 forwarding. (b) Layer 3 forwarding. (c) Layer 3 forwarding 156Mbps/virtual machine.2.5 Gbps10 GbpsHypervisorPacket generatorVMFigure 8. Latency breakdown. The hypervisorconsumes up to 80 percent of the end-toend latency in both the partial- and full-loadscenarios.Based on these lessons, we can derive ageneric model to extend our empirical study tomultiple NIC ports and analyze the maximumnumber of VNFs on any commodity NUMAserver with the following: a ports and a maximum of b VFs per port; and c lcores on d sockets with e NIC ports per socket. We assume thateach VNF only has one lcore, the server hassufficient memory and disk resources, and allpacket processing is in VNF.There are d NUMA domains and the numberof lcores per NUMA domain is c/d. For each port17

Network Function Virtualizationcontrol plane function (such as routing protocols and the firewall decision engine) in VNFs. Our layer 3VMVMVMVMforwarding application representssuch an architecture (see Figure 9a).Each VNF processes volumes of packFIBets continuously, leading to highHypervisorHypervisorfrequency hypervisor scheduling inan overprovisioning scenario withControlData(a) Control Data(b)significant overheads (as we saw inplane planeplaneplanethe performance study).To address this issue, we proFigure 9. NFV architectures. (a) Default architecture. (b) Proposed architecture. pose a new NFV design, aggregatingVNF forwarding functions into thehypervisor while maintaining theon a socket, the max number of lcores this port control function within the VNFs (see Figure 9b).can use will be c d e . Therefore, the maximum By doing so, the high-frequency packet pronumber of VNFs sharing an individual port is cessing can be handled in one place withoutmin {(c d e ) , b} , because each VNF requires scheduling VNFs.one VF and one lcore to guarantee performance.This architecture is functionally similar toOverall, the maximum number would be what Open vSwitch (see http://openvswitch.a min {(c d e ) , b} . Because d e a, the max- org) could support using OpenFlow5 to sepaimum number is finally min{a b, c}, where rate control and data planes. However, using thea b is the maximum number of VFs and c is long latency OpenFlow protocol will increasethe total number of lcores.FIB update overheads by orders of magnitude,From this model, the number of VNFs on considering that the CuckooSwitch can supporta server is limited by the smaller of the lcore 64,000 updates per second.3 As we show in thequantity and the max number of virtual I/O next section, our prototype yields better perforfunctions. For example, in our NFV server, mance than Open vSwitch as well.the maximum number of lcores is 32 and themaximum number of VFs on all four ports is Prototype and Performance Evaluation252 (63 4), so we can run at most 32 VNFs We implement a prototype using the new archion the server without substantial performance tecture. In the hypervisor, a layer 3 Cuckooloss. Because a large server that could support hashing-based forwarder processes data packets252 cores is substantially more expensive on a rather than relaying it to VNF. It uses the sumper-core basis and has an even more complex of all the VNF FIBs as a routing table. WhenNUMA design than a mainstream two-socket control packets arrive, the forwarder sends itsystem, the maximum number of VNFs in through a bridge to the VNF, which then writesany practical NFV system today is limited by update messages to the bridge. Upon receivingCPU resources rather than SR-IOV limitation. the message, the hypervisor updates the FIB.The issue will worsen because a 100-Gbps NICWe test the prototype’s performance onentering the mainstream as the CPU scalability 63 VNFs. The control plane traffic is set to 1can’t keep up with the increased rate of network Kbps (approximately two updates per second).speed.As Figure 10 shows, the new NFV architectureshows promising performance. The throughRethinking NFV Architectureput and latency are close to the counterpartTaking the lessons learned from our study,

Network functions virtualization (NFV) promises to bring significant flexibility and cost savings to networking. These improvements are predicated on being able to run many virtualized network elements on a server, which leads to a fundamental question on how scalable an NFV platform can be. With this in mind, the authors build an experimental platform with commonly used NFV technologies. They .