WIXLIB

Design of ExperimentsThe Science of TestThe Tooth Fairy in Experimental Design:Lying to Our Softwarepresented to:StatEase DOE SummitSept 28 2021Greg Hutto, Chief Ops Analyst - 96 Ops Group, Eglin AFB, Florida gregory.hutto@us.af.milPrincipal – Beta Associates LLC beta.hutto@gmail.com

Short BioMr. Gregory Hutto is Wing Operations Analyst for the 96thTest Wing, responsible for embedding designed experimentsas the principal test method for several hundred tests eachyear. He teaches an extensive series of short courses in testmethods to all testers in the Wing from the Wing Commanderto our 520 scientists and engineers. As a LtCol in the USAFReserves, he served as senior military advisor to AFOperational Test & Evaluation Center Test Support Directorand as special advisor for test design to the AF Flight TestCenter commander at Edwards AFB, California. Mr. Hutto isa distinguished graduate of the US Naval Academy inOperations Research and holds a Master’s in the same fieldfrom Stanford University. Over the past 41 years, he hasserved in nearly every branch of test and evaluation fromlaboratory basic science to joint operational field testing. Hewould like to publicly repent of his 11 years of testingwithout the benefits of the principles of well-designedexperiments.He is proud of his two children who make their livings in thesky – one serves martinis; the other mayhem. Ariel is a DeltaFlight Attendant and Daniel is a F-15E Strike Eagle WSO bomb/nav. He is inordinately proud of his highly educatedwife, Dr. Deb, with 6 degrees.

AbstractMany of us told our children about the mythical Tooth Fairy when they wereyoung, to add a little enchantment and mystique to childhood. Similarly, wedon’t always tell our software the whole truth. This talk will feature threeexamples making use of Design Expert’s comprehensive design-buildingfacilities to build the desired design while not revealing everything toDX. Case 1 -- Aircraft Single Engine Climb replaces numeric variables withcategoric in the build to achieve an even spread of values in the design. Oncebuilt, we switch the variables back to numeric. Case 2 -- Bomb Envelopebuilds a comprehensive design piecemeal – two overlapping constrainedoptimal designs are built, then combined into a single candidate set fromwhich the final design is built with one more twist. Case 3 explores a newdesign class Sensor Cross-blocks– a replicated geometric full-factorial crossedwith easy to change modal variables to achieve a split-plot-like structure withgood modeling properties.

But ThereReally is a ToothFairy! there is a veil covering the unseen world which not the strongestman could tear apart. Only faith can push aside that curtain andview the supernal beauty and glory beyond. Is it all real? Ah,VIRGINIA, in all this world there is nothing else real and abiding.No Santa Claus! Thank God! he lives, and he lives forever. A thousandyears from now, Virginia, nay, ten times ten thousand years from now,he will continue to make glad the heart of childhood.* I make a tradition of reading this beautifulpiece of prose every Christmas join me?* ia-there-is-a-santa-claus/

Eglin AFB Florida – Weapons, Spec Ops,Fighters & Joint OpsWe’re recruiting greatexperimentalists or at leastenthusiastic ones!

It All Started with Beer! Gosset – Dublin Guinness Brewery circa 1899-1908Brewing stouter porter, stout, or GuinnessGuinness decided to make brewing scientificFormed Experimental Brewery and Gosset as Brewerin-ChargeNeeded better hops and barley, small samples andmuch variationGosset, a chemist -- self-taught and workedfor Karl Pearson in statisticsPaper on Probable Error of A Mean forms the basisfor our statistical t testsTemperature, robust hops, and Archer breed of barley - plus experimentation key in Guiness findings!W. S. Gosset, 1899

Case 1 Aircraft Single Engine ClimbSource: ing-axis-forces-italy

Description: Twin Engine SingleEngine Climb* B-25 was notour test aircraftbut it is twinengine When taking off - safety calls for ability to minimallyclimb with single engine 150 feet per minute (fpm) Return to land from pattern altitude A number of factors – gross weight and air density -impact this mission planning check

Input Process OutputSingle Engine Climbweather, training, pilotINPUTS(Factors)Pressure AltitudeOUTPUTS(Responses)PROCESS:Aircraft WeightGear SettingClimb versus Chart)FlapsTemperatureNoiseB-25 – an Eglin favorite – the steed Doolittle’s Raiders rode to bomb Tokyo 14Apr ’42. Col Dick Cole died 9 Apr 21 @ 103 – Last Raider

The White Lie I Gross weight is primarilyf(fuel) Cannot ever be 0 or 100% Mostly in between extremes Factorials love edges andcorners Options – optimal polynomial,discrete Hit on building numeric grossweight as categoric forbalanced number of interiorlevels

The White Lie II Inconvenient physics –temperature lapse rate 3 deg /1000 feet Correlates density altitude Multiple (10 ) attempts togenerate balanced, constrainedoptimal design Pay attention to VIF for Temp XAltitude interaction Augment 4-8 points DesignTools Augment Optimal withTemp, Alt, TxA interaction

White Lie III Flaps can be at any degrees from 0 to 30;practically – only one or the other Gear is either up or down Treated as numeric, discrete, categoric

Final Design – Rube Goldberg Must have built 20 optimal designs Built & defaced Face Centered CCD1. Manually adjusted temperature &altitude2. Manually moved center pointvalues of gear & flaps to edges3. Accept multiple gross weight &temperature settings as set pointerrors4. Adjust unsafe combos

As-DesignedandAs-Executed Comparison not bad – knew AB was a problem

As executed – God has a sense ofhumor – all afternoons CNXDesign-Expert SoftwareDesign-Expert ointsbyvalueColor points by essAltB:PressAlt0.00193.003000.0041.501500.00As TempA:Temp90.0090.00100.00100.00

Final Design As ExecutedUnsafe test pointsDownUp0 differenceLessSafeSafe

Case 2: Building an uber-design fromconstrained sub-designs – AFranken-design!Team: Mike Huffman, Greg Hutto, Teresa Dailey, Blake Zessin,Francisco Ortiz, Dave Harmon, Nathan Selling

Case 2: Bomb Flight Test MatrixTest Objective: Four groups designing bomb drops Contractor has 6-7 shots; Agency 911, Air Force has approx 19 shots Task 1 – convince partners to conductIntegrated Test Task 2 – how to plan a robust programin face of likely budget cutsDOE Approach: Partner with Designers to selectconditions Exploit likely behavior based on similartechnologies Create menu of 8 different designs, eachwith pros and cons Objectively grade each; present ProgramOffice with recommendationsResult: Initial Design: N 19; 27% Mgt Reserve

Basic Report Card - Designed ExperimentsWheelPlanDesignExecuteAnalyzeDesign Alternative12Design NameL18 HunterFCDNumber of Factors55Levels ea Factor3x2 43Num Responses (MOPS)11Real-values?yyObjective?yyTest Events (N)1829Shot - Savings (-Increase)31%-12%Aliasing/Res/Ortho/ConfoundRIII ishRVa (0.05 for comparisons)5%5%2 s Power: AE 2 Fact Int %0.620.652 s Power: 3 3-lvl ME %0.750.952 s Power: 2 lvl ME ME %0.9399%Randomized?--Blocked or calibrated?--Replicates? True?03Pred Model SupportedMEQuadraticFDS Pred Err @50%1.20.46FDS Pred Err @95%3.10.7Leverage Avg0.530.3Leverage Max0.660.59VIF Avg1.11.05VIF Max1.21.5345678SCDIV Opt 2FI D Opt 2FI 2 5-1 FF 2 6-2 FF IV Opt Quad55556533x2 43x2 ----233333Quadratic ME 2FIME 2FI ME 2FI ME 2FI 491.11.011.02111.041.271.031.1111.07

Design Approach for fast-runningPhysics model Used simulation to evaluate 58 system variables Spanned spec range for “noise variables” (cannotcontrol in flight) Designed 6-8 run sets over ranges of control/noisevariablesD-optimal21Space-filling

Input Process OutputBomb Dropwind, aircraft, pilotINPUTS(Factors)OUTPUTS(Responses)Pressure AltitudeVelocityPROCESS:Climb-Dive AngleRangeMiss distance - aimpointXYZ V Errors from AircraftAircraft TypeCross-RangeDown-RangeNoiseNOT the actual bomb under test here

Traditional Designs Don’t Fit OurConstrained EnvelopesLEVEL RELEASE 1 0.5Factor B-2 2 1-0.5 0.5-0.5-1-1 1-1Factor AWe need a level (outer) and loft (inner) magenta LAR design andOBTW – don’t really want many lofts in 26 point design23

White Lie I – Just build this one set Clever Trick 1 use a well-schooled candidate set Design Expert Supports linear constraints &disallowed combos via “Candidate Sets” Ensure variables are common (different ranges) Build large Level design (optimal quartic) in DX 250 Build large Loft design (optimal quartic) in DX 250 Combine two design sets - points obey allconstraints Construct I-optimal designs from this merged candidateset24

White Lie II – I’ll want 4 blocks, please Clever Trick 2 – only get 26ish shots Want only a “few” lofts but more than a 1 or 2demo solution “block” design in 4 blocks for N 36 Block is design - discard loft runs inblocks 3 & 4 Loft model mostly orthogonal to blocks sometrics don’t suffer much

White Lie III – I’ll want all 26 points,please Clever Trick 3 -- Want mulligans (do-overs) at end Sort by leverage - last runs 22 ? can be mulligans Construct full set by augmentation Build runs 1-8 for Main Effects model Augment runs 9-19 for likely 2FI (we peeked) quads So - by run 21, likely model is well-supported Therefore 22, 23, 24, 25, 26 can either support modelor replace earlier failures

Displaying only 3 of 7 dimensions – ourbuild in blues

Case 3 Sensor Cross-Blocks Design

Critical features of target coordinatesproblem1.2.3.4.Estimating target lat/long error is a frequent, DOD-wide problemThe target battlespace is large, with many variablesThe response variable is skew with increasing varianceWe have restrictions on randomizing execution order (no transporter)

Complications – response distributionchanges with geometry Coordinate error increases as slant range (radius) squaredAt low graze angle long slant range errors more skew

Input Process OutputSensor Target Location ErrorINPUTS(Factors)weather, sensor number, pilotAltitudeOUTPUTS(Responses)PROCESS:Ground RangeAngle off the noseActual vs. Measured LocationSensor (visual or infrared)Target type (flat, vehicle, bldg.)Side of AircraftDirection (In-Out)NoiseThere are at least 34 * 23 648 unique points so a fractional design is essential

Which Coordinate System Shall We Use? Cartesian or Polar? In any case we must traverse space (alt range)

Which Coordinate System Shall We Use?CartesianPolar Cartesian or Polar? Collinearity in polar coordinates argues no

Optimal Designs Jump Quickly to MindOutAltitudeInDirectionRange Any issues? Fractional “holes” make us look dumb “Ummmm pilot take next two then skip 1 then skip 3 then ”

White Lies I & II - replicated geo factorial& the blocked modal factorial are separate Build sequential geometric design & replicates (96) Build convenient block size of modal variables Rotate or cross the blocks across the geo design

Cross-BlockDesign Left side 3 replicates ofa full geometric design(N 32 each) Right columns are 24modal variables in fourconvenient blocks Use Latin Square patternto cross blocks withgeometric replicates 1234 2341 3412 4123 Change block generatorsfor next replicate

Cross-Block Design Typical Evaluation Mild VIF inflation Mild 2FI and RSMcovariances Flown thesesince circa 2018 –a dozen times Models confirmand crossvalidate nicely

Tooth Fairies – Not All BadThe physical world of flight test offersendless challenges to our creativity DXa great partner toolSource: son-dont-take-my-tooth/