WIXLIB

Evaluating CoDel, FQ CoDel and PIE:how good are they really?Naeem Khademi naeemk@ifi.uio.no David Ros David.Ros@telecom-bretagne.eu Michael Welzl michawe@ifi.uio.no ICCRG – IETF 88Vancouver, BC, CanadaN. Khademi, D. Ros, M. Welzl ()Evaluating CoDel, FQ CoDel and PIENovember 5, 20131 / 23

OutlineOutlineNew AQM KidsExperimental SetupA Basic TestParameter SensitivityAQM on 802.11 WLANsFQ CoDel: Blending SFQ and AQMECNConclusions and Future WorkQ&AN. Khademi, D. Ros, M. Welzl ()Evaluating CoDel, FQ CoDel and PIENovember 5, 20132 / 23

New AQM KidsThe New AQM Kids on the Block.ITwo very recent proposals:IIISome older AQMs dating back to early 90’s/00’s (*RED, REM,BLUE, CHOKe,.)II(FQ )CoDel (IETF 84)PIE (IETF 85)Designed to be better than RED, just like CoDel and PIELittle academic literature available on CoDel and PIELiterature (bold peer-reviewed)Wired, simWired, real-lifeWireless (any)CoDel[NJ12][GRT 13][WP12][Whi13][GRT 13] XXPIE[Whi13]FQ CoDel[Whi13]XXX-NOTE: [WP12] and [Whi13] are on DOCSIS 3.0 while [GRT 13] has tests with LP CC.N. Khademi, D. Ros, M. Welzl ()Evaluating CoDel, FQ CoDel and PIENovember 5, 20133 / 23

New AQM KidsThe New AQM Kids on the Block (cont.)AQM Deployment StatusI(W)RED is available on plenty of HW but mostly "turned off"Mentioned Reasons for Lack of DeploymentIBad implementation (?)IHard to tune RED paramsISally Floyd’s ARED (2001 draft, available in Linux) adaptivelytunes RED params aiming for a certain target queuing withfixed BW maps to a "target delay"ITarget delay can be set in ARED, CoDel and PIEN. Khademi, D. Ros, M. Welzl ()Evaluating CoDel, FQ CoDel and PIENovember 5, 20134 / 23

Experimental SetupExperimental SetupIIIITraffic: 60/180/300 sec (wired/wireless/RTT 500 ms) of iperf, repeated for 10 runsAQM iface: GSO TSO off, BQL 1514, txqueuelen 1000TCP: Linux default with renoTopology: Dumbbell with 4 sender-receiver pairsModelCPURAMEthernetEthernet driver802.11 b/g802.11 driverOS kernelDell OptiPlex GX620Intel(R) Pentium(R) 4 CPU 3.00 GHz1 GB PC2-4200 (533 MHz)Broadcom NetXtreme BCM5751RTL-8139 (AQM interface)RTL8111/8168B (Dummynet router)tg38139too (AQM interface)r8168 (Dummynet router)D-Link DWL-G520 AR5001X ath5kLinux 3.8.2 (FC14)Linux 3.10.4 (AQM router) (FC16)N. Khademi, D. Ros, M. Welzl ()Evaluating CoDel, FQ CoDel and PIE100 Mbps100 MbpsAQM100 Mbps100 MbpsDummynet router100 Mbps100 Mbps100 Mbps100 Mbps100 Mbps10 MbpsAQM100 MbpsDummynet router100 Mbps100 MbpsBottleneck router100 MbpsNovember 5, 20135 / 23

Experimental SetupExperimental Setup (cont.)IAQM parameters used unless otherwise noted.CoDelinterval 100 mstarget 5 msPIEAREDparameters in [pie].PIE Parameter Default valuetupdate30 msTtarget20 msα0.125β1.25N. Khademi, D. Ros, M. Welzl ()parameters in [FGS01].ARED ParameterDefault valueinterval500 msαmin(0.01, pmax /4)β0.9Evaluating CoDel, FQ CoDel and PIENovember 5, 20136 / 23

Experimental SetupExperimental Setup (cont.)IRTT is measured on per-packet basis using Synthetic PacketPairs (SPP) tool [spp]IIGives a very precise distribution of perceived RTT on the pathGoodput is measured per 5-sec intervalsIlong-term throughput/goodput does not reflect AQM performanceover time (e.g. bursts of packet drops are not desired)N. Khademi, D. Ros, M. Welzl ()Evaluating CoDel, FQ CoDel and PIENovember 5, 20137 / 23

A Basic TestA Basic oodput (Mbps)RTT (ms)Single TCP Flow (RTTbase 100 ms)111CoDel555 10 10 10 20 20 20 30 30 30Target Delay ) Per-packet RTT555 10 10 10 20 20 20 30 30 30Target Delay (ms)PIEARED(b) GoodputPer-packet RTT and goodput. Bottom and top of whisker-box plots show 10th and 90thpercentiles respectively.N. Khademi, D. Ros, M. Welzl ()Evaluating CoDel, FQ CoDel and PIENovember 5, 20138 / 23

A Basic TestA Basic Test (cont.)CoDel vs. RED from K. Nichols, “Controlling QueueDelay” [NJ12]IA similar trend canbe observed betweenCoDel and RED in adifferent test in[NJ12]FTP traffic mix w/ and w/o web-browsing and CBRapplications and RTTs from 10 500 ms.N. Khademi, D. Ros, M. Welzl ()Evaluating CoDel, FQ CoDel and PIENovember 5, 20139 / 23

Parameter SensitivityParameter Sensitivity (cont.)1115CoDel55 10 10 10 20 20 20 30 30 30Target Delay (ms)PIEARED(c) 100RTT 00RTT (ms)RTT (ms)Target Delay (Per-packet RTT)111CoDel555 10 10 10 20 20 20 30 30 30Target Delay (ms)PIEARED(d) 1101001115CoDel55 10 10 10 20 20 20 30 30 30Target Delay (ms)PIEARED(e) HeavyPer-packet RTT. Light, moderate and heavy congestion scenarios (4 senders andRTTbase 100 ms).Light, moderate and heavy congestion correspond to 4, 16 and 64 concurrent TCP flowsrespectively.N. Khademi, D. Ros, M. Welzl ()Evaluating CoDel, FQ CoDel and PIENovember 5, 201310 / 23

Parameter SensitivityParameter Sensitivity (cont.)Goodput (Mbps)Target Delay (Goodput)109.598.518.518155CoDel5 10 10 10 20 20 20 30 30 30Target Delay (ms)PIEARED7.5(g) Moderate76.565.5111CoDel555 10 10 10 20 20 20 30 30 30Target Delay (ms)PIE(f) LightGoodput (Mbps)Goodput (Mbps)9109.5109.598.5111AREDCoDel555 10 10 10 20 20 20 30 30 30Target Delay (ms)PIEARED(h) HeavyGoodput. Light, Moderate and Heavy congestion scenarios (4 senders and RTTbase 100 ms).N. Khademi, D. Ros, M. Welzl ()Evaluating CoDel, FQ CoDel and PIENovember 5, 201311 / 23

AQM on 802.11 WLANsAQM on 802.11 WLANsI802.11 is a challenging environment for AQM deploymentIVarying MCS (BW) by RA and MAC retriesIIShared channel with various active STAsIIWe use TCP max achievable BW for testing (e.g. 27 Mbps in .11g)CoDel and PIE use delayIIIDirection-based unfairness (uplink vs. downlink)Hard to predict the BW as input to AREDIISome drivers still use SampleRate instead of Minstrel (e.g. in FBSD)CoDel: queuing delay by timestampingPIE: estimated queuing delay (queue length / departure rate)Public Wi-Fi e.g. at airports, hotels, corporations with bottleneckon wlanX interfaceN. Khademi, D. Ros, M. Welzl ()Evaluating CoDel, FQ CoDel and PIENovember 5, 201312 / 23

AQM on 802.11 WLANsAQM on 802.11 WLANs 014013012011010021.52120.5Goodput (Mbps)RTT (ms)802.11 Downlink Traffic Scenario – Target Delay 5 ms2019.51918.51817.51716.5111444888161616111Flows per sender #CoDelPIEAREDCoDel(i) Per-packet RTT44488Flows per sender #PIE8161616ARED(j) Goodput4 senders, RTTbase 100 ms.N. Khademi, D. Ros, M. Welzl ()Evaluating CoDel, FQ CoDel and PIENovember 5, 201313 / 23

AQM on 802.11 WLANsAQM on 802.11 WLANs (cont.)IIAQM on AP’s wlanX interface 65%/ 70% ACK loss at AQM (ARED) router for 16/32 flows.802.11 Mixed Traffic – Target Delay 5 ms (Uplink’s Stats)240022002000Goodput (Mbps)RTT 600CoDel40011144488816161644488Flows per sender #PIE8161616ARED(l) GoodputFlows per sender #CoDelPIEARED(k) Per-packet RTT4 senders, RTTbase 100 ms.N. Khademi, D. Ros, M. Welzl ()Evaluating CoDel, FQ CoDel and PIENovember 5, 201314 / 23

FQ CoDel: Blending SFQ and AQMFQ CoDel: Blending SFQ and AQMSFQ is highly likely to improvethe performance whencombined with any AQMIIIFlow isolation/protectionwith non-responsive trafficClose to 100% flow-levelfairness on the edgeSignificantly Lower 10thpercentile and median RTTsbut with longer (upper)distribution tailN. Khademi, D. Ros, M. Welzl ()Comparison between CoDel and FQ CoDel– target delay 5 ms210200190180170RTT (ms)I160150140130120110100114CoDel48Flows per sender #81616FQ-CoDelPer-packet RTT. 4 senders andRTTbase 100 ms.Evaluating CoDel, FQ CoDel and PIENovember 5, 201315 / 23

FQ CoDel: Blending SFQ and AQMFQ CoDel: Blending SFQ and AQMFQ CoDel: Lower median latency at the expense of higher jitterthan CoDel (with the increase of congestion level)IPer-packet RTT Samples of a Single Flow – target delay 5 0180160RTT 20120110110100120110100202530Time (Sec)35(m) Light Congestion40fq-codelcodel220210RTT (ms)RTT (ms)230fq-codelcodel220100202530Time (Sec)35(n) Moderate Congestion40202530Time (Sec)3540(o) Heavy Congestion4 senders, RTTbase 100 ms.N. Khademi, D. Ros, M. Welzl ()Evaluating CoDel, FQ CoDel and PIENovember 5, 201316 / 23

ECNECNIProposed two decades agoIStill turned off in clientsICan give a lot of benefits (out of this talk’s scope)RFC 3168For a router, the CE codepoint of an ECN- Capable packet SHOULDonly be set if the router would otherwise have dropped the packet asan indication of congestion to the end nodes.IRFC 3168 is the only guideline for general useIMarking affects the marking/dropping probability since itchanges the metrics AQMs use (e.g. queue length and delay)N. Khademi, D. Ros, M. Welzl ()Evaluating CoDel, FQ CoDel and PIENovember 5, 201317 / 23

ECNECN 10100RTT (ms)RTT (ms)Per-packet RTT. 4 senders, target delay 5 ms, RTTbase 100 0130120110100161114Flows per sender #CoDelPIE4488Flows per sender #CoDelARED(p) ECN disabledPIE8161616161616ARED(q) ECN enabled109.598.587.576.56Goodput (Mbps)Goodput lows per sender #CoDelPIE44888Flows per sender #ARED(r) ECN disabledN. Khademi, D. Ros, M. Welzl ()4CoDelPIEARED(s) ECN enabledEvaluating CoDel, FQ CoDel and PIENovember 5, 201318 / 23

ECNECN (cont.)IIConstant CE-marking under heavycongestionwith ECN, ARED’s goodput dropssignificantly as congestion leveldecreasesIIAggressive reaction to theincrease in average queue lengthCE-marking (w/ ECN) 3.5 6.6times more than dropping (w/oECN)N. Khademi, D. Ros, M. Welzl ()pmarking ecn /pdrop noecnFlows4163264Evaluating CoDel, FQ CoDel and 9ARED6.6213.4654.3033.873November 5, 201319 / 23

ECNECN (cont.)Our RecommendationIAQMs should modify their dropping/marking decision process toincorporate the impact of CE-marked packets on theirmeasurements.Possible SolutionsIExclude CE-marked packets from queue size (ARED), or excludedelay caused by CE-marked packets (PIE)ISet lower ECN thresholds for AQMs(fairness against non-ECN flows? dynamic threshold?)N. Khademi, D. Ros, M. Welzl ()Evaluating CoDel, FQ CoDel and PIENovember 5, 201320 / 23

Conclusions and Future WorkConclusive RemarksIARED only performed worse than CoDel or PIE in few scenarios1. With very small number of flows2. Public Wi-Fi with mixed (up- and down-link) traffic3. With the common ECN implementation that CE-marks only when itwould otherwise dropIARED outperforms CoDel and PIE in all other studied scenarios,most notably regarding delayFuture WorkIMore realistic traffic types (here, only bulk TCP traffic)IImplementing and testing SFQ AREDIDelay-based AREDN. Khademi, D. Ros, M. Welzl ()Evaluating CoDel, FQ CoDel and PIENovember 5, 201321 / 23

Q&ABibliography[FGS01]Sally Floyd, Ramakrishna Gummadi, and Scott Shenker.Adaptive RED: An Algorithm for Increasing the Robustness of RED’s Active Queue Management.Technical report, 2001.[GRT 13] Y. Gong, D. Rossi, C. Testa, S. Valenti, and D. Taht.Fighting the Bufferbloat: On the Coexistence of AQM and Low Priority Congestion Control.2013.[NJ12]Kathleen Nichols and Van Jacobson.Controlling Queue Delay.Queue, 10(5):20:20–20:34, May 2012.[pie]PIE Linux code (Cisco).ftp://ftpeng.cisco.com/pie/linux code/.[spp]Synthetic Packet eg White.A Simulation Study of CoDel, SFQ-CoDel and PIE in DOCSIS 3.0 Networks.Technical Report, CableLabs, April 2013.[WP12]Greg White and Joey Padden.Preliminary Study of CoDel AQM in a DOCSIS Network.Technical Report, CableLabs, November 2012.N. Khademi, D. Ros, M. Welzl ()Evaluating CoDel, FQ CoDel and PIENovember 5, 201322 / 23

Q&AQ&AMore on experimental results: N. Khademi, D. Ros, and M. Welzl,“The New AQM Kids on the Block: Much Ado About Nothing?”,Technical Report 434, Department of Informatics, University of Oslo,23 October 2013, available at http://urn.nb.no/URN:NBN:no-38868N. Khademi, D. Ros, M. Welzl ()Evaluating CoDel, FQ CoDel and PIENovember 5, 201323 / 23

New AQM Kids The New AQM Kids on the Block. I Two very recent proposals: I (FQ_)CoDel(IETF84) I PIE(IETF85) I Some older AQMs dating back to early 90's/00's (*RED, REM, BLUE, CHOKe,.) I Designed to be better than RED, just like CoDel and PIE I Little academic literature available on CoDel and PIE Literature (bold peer-reviewed) CoDel PIE FQ_CoDel Wired, sim