WIXLIB

The Undoing ProjectSusan Harkins Bertsch2018 Cost Symposium

Introduction This presentation title is shamelessly ripped off from MichaelLewis’ 2016 book of the same name While I riff loosely the principles of uncertainty andjudgement themes pioneered by Daniel Kahneman andAmos Tversky, as outlined in Lewis’ book, this paper is reallydesigned to share my own recent experiences in testing andundoing traditional approaches to cost estimation in dealingwith nontraditional developments2

I Blame Charley Hunt I wasn’t going to read this book Any mom who has raised teenagers knows quite well that the premise ofWhat You See Is All There Is always merits further investigation; surelythose Haloes are hiding some Horns. Furthermore, I dropped all myAnchors during Hurricane Harvey, while my dog killed all the Black Swansfloating through the floodwaters. And I long ago Regressed into Mean Did I really need to read another book on behavioral economics? But then Charley told me about the first chapter* Which is entirely devoted to the much-vaunted analytics models of DarylMorey and the Houston Rockets* The title of which is totally work inappropriate andone that you will have to research on your own3

I Blame The Houston Rockets Because I can Because every summer is spent indulging my inner Charles Barkley andhowling over an analytical model that tells me nothing and which hasdelivered nothing Does anyone really need analytics to know that James Harden is a greatplayer? 23 years and counting since back-to-back NBA championships that weredelivered without Morey’s stupid model, thankyouverymuch So I eagerly dove into the book looking for some Validation of my beliefsand biases, only to see that the Rockets’ front office struggles with manyof the same challenges faced by NASA’s cost estimators. Moreover, theycould see that the limits of their modeling invited human judgement andexpert opinion back into the evaluation4

How Is This All Relevant? This reading provided the inspiration to search for creative ways forassessing cost estimates, methodologies outside of the probabilisticevaluations that we have so carefully built over the past several years As with the Houston Rockets, this insight is meant to be additive to ourtools and especially when evaluating efforts in which we have little dataand experience Providing that human gut check against the probabilistic analyses, makesure that we keep asking the right questions I suppose that I should thank Charley for steering me to this book so thatI could glean these insights, but I blame him for forcing me to have somesympathies for Rockets analytics And I will remain grumpy about it until we bring Larry O’Brien trophy backto Houston5

SRB Support For Cubesat Program Tapped as programmatic lead for the Time-Resolved Observations ofPrecipitation structure and storm Intensity with a Constellation ofSmallsats (TROPICS) mission SRR review TROPICS is a constellation of 3U cubesats to provide high-value scienceinvestigations of tropical cyclones6

Review of ISE and ICE Independent schedule, cost estimates provided by the EarthSystem Science Pathfinder (ESSP) Program Relied heavily on smallsat models and relationships, which wereapplied to relevant cubesat data Analyses did note that many wraps and ratios might not apply tocubesats Reviewed methodology for the ISE, ICE Sound, well-explained, assumptions well documented In good NASA Best Practices fashion, the ISE and ICE were providedat a 70% confidence level, along with a probabilistic S-curve–Review preceded SMD Class D Mission Memo dropping required confidencelevel to 60% And, as is seemingly often the case, the results were a lot higherthan what was planned for the project7

Now The Hard Part The SRB chair asked if I believed the 70% cost confidence number The quick answer is that I had no doubts that the 70% confidencenumber had been generated correctly The not-so-quick takeaway was whether that number was really relevantto this particular project, requiring further investigation into the cubesatdata Thanks to the COMPACT data team for all the help and especially toTommy Paine for taking my calls and running down answers to my manyquestions8

Analysis Of Cubesat Cost Data No cubesat CERs or models Efforts to “scale” existing models very experimental Sparse data, both from industry and developer Used COMPACT data evaluate analogy data 18 data points, 17 of which were examined MarCO out of family at 6 U, planetary mission Remainder “in-family” at 1U to 3U, earth-orbiting Mission data compared against Basis of Estimate (BOE) provided forTROPICS mission No subsystem detail provided from either COMPACT data or from TROPICSBOE, some instrument data9

University versus Commercial Data needs to be delineated between university-ledversus commercial development Student “pizza money and course credit” Non-student “paycheck and 401k” (most analogous to TROPICS)Source: COMPACT, data normalized to FY 1610

Analyzing Commercial Cubesat Data Historical commercial developments break into two verybroad categories Lower end of scale mostly made up of tech demo missions, manywith university participation Higher end of scale is more mission driven developments BOE for TROPICS fits within this mission driven familySource: COMPACT, data normalized to FY 1611

What To Make Of The Data One obvious problem with the 5 data point mission cubesat “family” is avery large outlier on the upper end that skews a distribution generatedfrom this data The mission was one of the earliest cubesat developments, but a temporalanalysis did not indicate that earlier developments were necessarily moreexpensive than later ones Further research indicated that there were aspects to the lower end of thecost range that were hard to normalize out and could instead be theoutliers University participation and designs COTS bus Heritage from previous missions

Using The Data For Assessment Clearly, the data is pretty noisy Top-level data collection is not normalized for instrumentation, bus costs Only commonality among the data is size Moreover, there are rapidly emerging trends in cubesat manufacturing inwhich newer developments, using more COTS components Cost impact unclear And thanks to the Undoing Project, I now had the added burden of Iworrying that any biases in my data analysis would lead me craft a modelor analogy set for cubesat cost estimation that would yield the costestimation equivalent of Daryl Morey’s draft model that led him to pickJoey Dorsey over LaMarcus Aldridge** This happened, sadly13

Answering The Question Daniel Kahneman has observed that “when faced with a difficultquestion, we often answer an easier one instead” He has clearly never met our estimators While charging hard after trying to answer whether I believed the 70%confidence ICE, I realized that I had completely ignored the second halfof the question “Are you satisfied the project is on track?” By refocusing on the second half of the question, this led to a differentset of questions What makes the project go off track? What conditions break the budget? Rather than trying to make sense of the existing data, I decided the fullrange of unit costs against the budget

Multiple Cubesat Builds After fixing on how to handle the unit cost input, the next step was todetermine an appropriate learning curve to apply to the productionanalysis TROPICS is a constellation of cubesats, multiple builds Almost all the information in the COMPACT dataset represents single unitbuilds Little guidance in the way of choosing an appropriate learning curve forcubesat production

Learning Curve So I set out to exhaustively research learning curve theory Considered the production practices from the few multiple lot cubesat builds Processed in batches of 2 to 3, often in parallel Toured production facility Found all sorts of cool data The usual 85% guidance for aerospace A 75% recommendation for “high touch”, largely manual production Found a curve fit of 66.2% from data presented in “The Incremental Cost of Oneor More Copies” from APL Obsessed over what case best fit what I was trying to model As I said earlier, we never go simple with our stuff Decided to consider a range of cases for this variable as well

Analysis Matrix

Observations Matrix framed a set of conditions and input combinations that broke thebudget Not intuitively obvious without doing this where the BOE fit relative to thebudget Useful guidance tool going into PDR, where more data on unit costs will beavailable Helped hone risk posture for PDR, identifying potential risks, and how tobetter assess how candidate risks might impact these key variables Not a probabilistic analysis, but this matrix helped put the ICE in contextand is additive to that analysis18

Create Your Own Undoing Helpful when trying to evaluate how well our estimating tools and datawork with new paradigms Cubesats Cutting edge missions New business models (commercial space, international partnerships, publicprivate partnerships) Dealing with culture of optimism Cited recently in Inspector General report, congressional hearings Drives the technical and subject matter expert inputs for our work Our outputs are only as good as those inputs (including risk) Don’t undo our analytical efforts Instead add to the analysis by “undoing” inputs to test where models andplans break, see if risk posture is in alignment19

Because It HappensBlack Swans20

Contact Info susan.bertsch-1@nasa.gov 281-483-0208 (office) 713-376-1138 (cell)21

Questions?

Bibliography Lewis, M. (2017) The Undoing Project: A Friendship That Changed OurMinds. New York: W.W. Norton & Company, 2016 Kahneman, Daniel. Thinking, Fast And Slow. New York : Farrar, StrausAnd Giroux, 2011 S-2-Cubesat J. Mrozinski, “CubeSat Or Microsat Probabilistic Analogies Cost Tool(COMPACT)”, April 2016 S. Whitley “The Incremental Cost of One or More Copies: QuantifyingEfficiencies from Building Spacecraft and Instrument Constellations”, 27thAnnual AIAA/USU Conference on Small Satellites, August 13, 2013 G. Chen and D. D. McLennan, "A study of learning curve impact on threeidentical small spacecraft," 2004 IEEE Aerospace Conference Proceedings(IEEE Cat. No.04TH8720), Big Sky, MT, 2004

Cubesat Definition Extremely small, nanosat scale spacecraft ofstandard dimensions termed “U’s”, typicallylaunched into space along with traditionalspacecraft and payloadsSource: Joe Mrozinski, “CubeSat Or Microsat Probabilistic Analogies Cost Tool (COMPACT)”2016 NASA Cost Symposium

25

The Undoing Project Susan Harkins Bertsch 2018 Cost Symposium. 2 Introduction This presentation title is shamelessly ripped off from Michael Lewis’ 2016 book of the same name While I riff loosely the prin