Transcription
Roadsand Bridges:The Unseen Labor BehindOur Digital InfrastructureWRITTEN BYNadia Eghbal
Open up your phone.Your social media,your news, yourmedical records, yourbank: they are all usingfree and public code.2
C ontents3Table of Contents4Preface58Challenges Facing Digital Infrastructure5Foreword598Executive SummaryOpen source’s complicatedrelationship with money66Why digital infrastructure supportproblems are accelerating77The hidden costs of ignoring infrastructure89Sustaining Digital Infrastructure90Business models for digital infrastructure97Finding a sponsor or donor for11Introduction18History and Background ofDigital Infrastructure19How software gets built23How not charging for softwaretransformed society29A brief history of free and public softwareand the people who made it37How The Current System Works38What is digital infrastructure, and howdoes it get built?46How are digital infrastructure projectsmanaged and supported?53Why do people keep contributing tothese projects, when they’re notgetting paid for it?an infrastructure project106Why is it so hard to fund these projects?109Institutional efforts to support digital infrastructure124Opportunities Ahead125Developing effective support strategies127Priming the landscape136The crossroads we face139Appendix140Glossary142AcknowledgementsR O A D S A N D B R I D G E S : T H E U N S E E N L A B O R B E H I N D O U R D I G I TA L I N F R A S T R U C T U R E
PrefaceOur modern society—everythingfrom hospitals to stock markets tonewspapers to social media—runs onsoftware. But take a closer look, andyou’ll find that the tools we use to buildsoftware are buckling under demand.
fore w ordForewordI stumbled upon the problem described in this report on a hunch.Having previously worked in startups, and then venture capital, Isaw the enormous amounts of money being poured into softwarecompanies. But as an amateur software developer, I knew that I hadnever done any of it alone. I used free and publicly available code(also known as “open source” code), which I cobbled together andoffered up for personal or commercial purposes. Really, the peoplebehind those projects, whoever they were, had done mostof the work.I mulled over this observation for several years, as I watched theexplosion of coding “bootcamps” graduating new software developers left and right, and as I watched startups raise tens of millions ofdollars selling products which I knew, under the hood, wereprobably more public than proprietary code. Having previouslyworked in the nonprofit sector, I immediately thought of publicgoods and their associated challenges, yet this vocabulary wasstrangely absent among my peers in software.After I left my job in venture capital last year, I set off to explore theparadox I couldn’t stop thinking about: that there were valuablesoftware tools that couldn’t be supported by commercial models,and that they lacked any form of institutional support.Funnily enough, open source code wasn’t on my original list. I hadmistakenly assumed, as had my peers, that these tools were anexample of a particularly well-supported public good in software.When I brought up open source to friends and mentors, they gentlydissuaded me from pursuing the topic, encouraging me instead toR O A D S A N D B R I D G E S : T H E U N S E E N L A B O R B E H I N D O U R D I G I TA L I N F R A S T R U C T U R E5
fore w ordfind other examples that actually needed the help.A few open source projects crossed my radar, however, and shattered those assumptions. It turned out that sustainability challenges were well-known among those who contributed to open source.The more I dug, the more I found blog posts, articles, and frequentpublic conversations about the stress and exhaustion felt by thosewho maintain open source projects. Everybody knew someone elseI should talk to, and before I knew it, I had collected countlessstories on this topic.I realized I had walked into a problem with which the producers(open source contributors) were extremely familiar, but that theconsumers (software companies and other users of open sourcecode) were seemingly unaware of. That discrepancy made me wantto look more closely.In addition, it seemed that open source itself was changing, perhapseven bifurcating. I found myself having completely different conversations with different generations of open source contributors. Theyseemed to have divergent philosophies and values; they may as wellnot have been using the same terminology. I learned that opensource had seen an explosion of production as well as demand in thepast three to five years, thanks to improvements in developer toolsand workflows. Today’s open source contributor looked very differentfrom an open source contributor ten years ago, much less thirty yearsago. And yet these different generations weren’t talking to eachother, making productive conversations about sustainability difficult.R O A D S A N D B R I D G E S : T H E U N S E E N L A B O R B E H I N D O U R D I G I TA L I N F R A S T R U C T U R E6
fore w ord“A chance conversation with Ethan Zuckerman of the MITCenter for Civic Media gave me an opportunity to share thesefindings more widely.I described to Ethan the problem I was seeing, though I didn’t knowexactly what it all meant or the vocabulary I should be using, and hekindly put me in touch with Jenny Toomey of the Ford Foundation.Jenny suggested I aggregate my findings into a report. In the process,a narrative around our modern digital society, and the hidden infrastructure that powers it, has emerged.This report would not have happened without Ethan and Jennytaking a chance on a half-baked idea that now, through the processof writing, has been shaped into something more. I am extremelygrateful to both of them for their intuition. I am additionally gratefulto Michael Brennan and Lori McGlinchey for their guidance, perspective and enthusiasm in the editing process. Finally, and perhaps mostimportantly, I am indebted to every person working in open sourcewho made their stories public for people like me to read, and especially those who took a moment out of their busy schedules tohumor me with a conversation or an email. This report is a collection of their wisdom, not mine. I am particularly grateful for earlyconversations with Russell Keith-Magee, Eric Holscher, Jan Lehnardt,Andrey Petrov, and Mikeal Rogers, all of whom continue to inspireme with their patience and dedication to open source work. Thankyou for your kindness.R O A D S A N D B R I D G E S : T H E U N S E E N L A B O R B E H I N D O U R D I G I TA L I N F R A S T R U C T U R E7
e x ecuti v e su m m ar y8Executive SummaryOur modern society—everything from hospitals to stock marketsto newspapers to social media—runs on software. But take a closerlook, and you’ll find that the tools we use to build software arebuckling under demand.Nearly all software today relies on free, public code (called “opensource” code), written and maintained by communities of developersand other talent. Much like roads or bridges, which anyone can walkor drive on, open source code can be used by anyone—from companies to individuals—to build software. This type of code makes upthe digital infrastructure of our society today.Just like physical infrastructure, digital infrastructure needsregular upkeep and maintenance. In the United States, over halfof government spending on transportation and water infrastructure goes just to maintenance.1 But financial support for digitalinfrastructure is much harder to come by. Currently, any financialsupport usually comes through sponsorships, direct or indirect, fromsoftware companies.Maintaining open source code used to be more manageable.Following the personal computer revolution of the early 1980s, mostcommercial software was proprietary, not shared. Software toolswere built and used internally by companies, and their productswere licensed to customers. Many companies felt that open sourcecode was too nascent and unreliable for commercial use. In theirview, software was meant to be charged for, not given away for free.Today, everybody uses open source code, including Fortune 500R O A D S A N D B R I D G E S : T H E U N S E E N L A B O R B E H I N D O U R D I G I TA L I N F R A S T R U C T U R E1https://www.cbo.gov/publication/49910
e x ecuti v e su m m ar ycompanies, government, major software companies and startups.Sharing, rather than building proprietary code, turned out to becheaper, easier, and more efficient. This increased demand puts additional strain on those who maintain this infrastructure, yet becausethese communities are not highly visible, the rest of the world hasbeen slow to notice. Most of us take opening a software applicationfor granted, the way we take turning on the lights for granted. Wedon’t think about the human capital necessary to make that happen.In the face of unprecedented demand, the costs of not supportingour digital infrastructure are numerous. On the risk side, there aresecurity breaches and interruptions in service, due to infrastructuremaintainers not being able to provide adequate support. On theopportunity side, we need to maintain and improve these softwaretools in order to support today’s startup renaissance, which reliesheavily on this infrastructure. Additionally, open source work buildsdevelopers’ portfolios and helps them get hired, but the talent poolis remarkably less diverse than in tech overall. Expanding the poolof contributors can positively affect who participates in the techindustry at large.No individual company or organization is incentivized to address theproblem alone, because open source code is a public good. In orderto support our digital infrastructure, we must find ways to worktogether. Current examples of efforts to support digital infrastructure include the Linux Foundation’s Core Infrastructure Initiative andMozilla’s Open Source Support (MOSS) program, as well as numeroussoftware companies in various capacities.Sustaining our digital infrastructure is a new topic for many, andthe challenges are not well understood. In addition, infrastructureR O A D S A N D B R I D G E S : T H E U N S E E N L A B O R B E H I N D O U R D I G I TA L I N F R A S T R U C T U R E9
e x ecuti v e su m m ar yprojects are distributed across many people and organizations,defying common governance models. Many infrastructure projectshave no legal entity at all. Any support strategy needs to accept andwork with the decentralized, community-centric qualities of opensource code. Increasing awareness of the problem, making it easierfor institutions to contribute time and money, expanding the pool ofopen source contributors, and developing best practices and policiesacross infrastructure projects will all go a long way in building ahealthy and sustainable ecosystem.R O A D S A N D B R I D G E S : T H E U N S E E N L A B O R B E H I N D O U R D I G I TA L I N F R A S T R U C T U R E10
I ntroduction11IntroductionIn 1998, a group of security experts in the UK got together to build afree set of encryption tools for the Internet.Soon everybody was talking about their project, called OpenSSL.(The developers had used an existing Australian project, calledSSLeay, as their blueprint.) Not only was it comprehensive anddecently reliable, but it was free. Writing cryptography wasn’t easy,and OpenSSL had solved a major pain point for developers worldwide.By 2014, two-thirds of all Web servers were using OpenSSL, enablingwebsites to securely pass credit card and other sensitive informationover the Internet.2Meanwhile, the project continued to be informally managed by asmall handful of volunteers. A security consultant to the U.S.Department of Defense, Steve Marquess, noticed that one contributor, Stephen Henson, was working full time on OpenSSL. Curious,Marquess asked him what he did for income, and was shocked tolearn that Henson made one-fifth of Marquess’s salary.Marquess had always considered himself to be a strong programmer,but his skills paled in comparison to Henson’s. Like others, Marquesshad mistakenly assumed that someone as talented as Henson wouldhave a comfortable salary to match.Henson had been working on OpenSSL since 1998. Marquess wasnewer to the project, joining in the early 2000s, and had workedwith Henson for several years before learning of his income situation.R O A D S A N D B R I D G E S : T H E U N S E E N L A B O R B E H I N D O U R D I G I TA L I N F R A S T R U C T U R eartbleed-bug.html
I ntroduction12Having worked with the Department of Defense, Marquess saw howcritical OpenSSL was, not just to their software, but to other industries around the world, from enterprise to aeronautics to health care.Until that moment, he had “always assumed, (as had the rest of theworld) that the OpenSSL team was large, active, and well resourced.3” In3reality, OpenSSL wasn’t even able to support one person’s work.Email interview withSteve MarquessMarquess decided he wanted to help. Although he contributed codeoccasionally, he realized he could fill a more critical role on thebusiness side. Marquess started out by arranging small consultingcontracts through an existing nonprofit to help keep OpenSSL alivein its leanest years.As the volume of contracts grew, Marquess created the OpenSSLSoftware Foundation (OSF) to provide an official vehicle for revenue.Despite the number of individuals and companies relying on theirsoftware, OSF never received more than 2,000 in donations peryear. Gross revenues (which came from consulting and contractwork) never broke 1M, and much of that went toward security-related testing (which could cost hundreds of thousands of dollars)and server costs.There was enough to pay the salary of one developer, StephenHenson. That meant that two-thirds of the Web relied on encryptionsoftware maintained by just one full-time employee.The OpenSSL team continued to work in relative obscurity until April2014, when a Google engineer named Neel Mehta stumbled upon amajor flaw in OpenSSL’s software. Two days later, another engineerat the Finnish company Codenomicon discovered the same problem.Both of them immediately contacted the OpenSSL team.R O A D S A N D B R I D G E S : T H E U N S E E N L A B O R B E H I N D O U R D I G I TA L I N F R A S T R U C T U R E
I ntroduction13That bug, nicknamed Heartbleed, had been included in a 2011update. It had gone unnoticed for years. Heartbleed could allow anysophisticated hacker to capture secure information being passed tovulnerable web servers, including passwords, credit card information,and other sensitive data.Joseph Steinberg, a cybersecurity columnist for Forbes, wrote that“some might argue that [Heartbleed] is the worst vulnerabilityfound.since commercial traffic began to flow on the Internet.” 4Thanks to wide media reporting, much of the nontechnical worldbecame familiar with the security bug, at least by name. you-are-atrisk-what-you-need-to-do/services like Instagram, Gmail and Netflix were affected byHeartbleed.5 Reporters also drew attention to OpenSSL itself, andhow its team had struggled for years to support their work. OpenSSLwas a known concern among security experts, but the team did nothave adequate resources or attention to address the issues.Of Heartbleed, Marquess wrote, “The mystery is not that a few overworked volunteers missed this bug; the mystery is why it hasn’thappened more often.”People expressed their support by sending donations to the foundation. Although Marquess was grateful for their enthusiasm, the firstround of donations came out to roughly 9,000: not nearly enoughto sustain a team.R O A D S A N D B R I D G E S : T H E U N S E E N L A B O R B E H I N D O U R D I G I TA L I N F R A S T R U C T U R bsites-affected/#01gtseEchaqa
I ntroduction14Marquess took to the Internet to make an impassioned public pleafor funding:“These guys don’t work on OpenSSL for money. They don’t do itfor fame (who outside of geek circles ever heard of them orOpenSSL until ‘heartbleed’ [sic] hit the news?). They do it out ofpride in craftsmanship and the responsibility for somethingthey believe in.It takes nerves of steel to work for many years on hundreds ofthousands of lines of very complex code, with every line of codeyou touch visible to the world, knowing that code is used bybanks, firewalls, weapons systems, web sites, smart phones,industry, government, everywhere. Knowing that you’ll beignored and unappreciated until something goes wrong.There should be at least a half dozen full time OpenSSL teammembers, not just one, able to concentrate on the care andfeeding of OpenSSL without having to hustle commercialwork. If you’re a corporate or government decision maker in aposition to do something about it, give it some thought. Please.I’m getting old and weary and I’d like to retire someday. 6After Heartbleed, OpenSSL finally got more of the funding itneeded—at least for now. They currently have enough money to payfour full-time employees for three years. But a year and a half intothat funding, Marquess isn’t sure what will come next.Marquess said that Heartbleed was a boon for them, admitting itwas a “little ironic” that publicity had helped elevate their cause. Butafter funding runs out and the world moves on, Marquess thinksR O A D S A N D B R I D G E S : T H E U N S E E N L A B O R B E H I N D O U R D I G I TA L I N F R A S T R U C T U R sibility-and-pride/
I ntroduction15they could be back in the same situation as pre-Heartbleed, andperhaps even worse: the client work that took Marquess years tobuild has dried up, since the team works full-time on OpenSSL rightnow and no longer has time for contracts.77Phone interview with SteveMarquessMarquess himself is approaching retirement. He is the only personwilling to handle the business and operational tasks associated withOpenSSL, including taxes, sourcing client work, and managingdonors. The rest of his team prefers to focus on writing and maintaining code. He can’t hire someone else into his position when heretires, either, because he currently doesn’t take an income.Marquess remarked, “I don’t know that we can hold this together formore than a couple of years”.8OpenSSL’s story is not unique, and in many ways, Marquess thinksthey are the lucky ones. Countless other projects continue to gounheard of and unsupported. These projects make up the criticaldigital infrastructure that powers today’s software, and in turn, everyaspect of our daily lives.Checking email, reading the news, checking stock prices, shoppingonline, going to the doctor, calling customer service—whether werealize it or not, everything we do is made possible by projects likeOpenSSL. Without them, the technology that modern society reliesupon simply could not function.Many of these projects are built and maintained by volunteers andoffered to the public for free. Anyone, from Facebook to an amateurprogrammer, can use that code to build their own apps. And they do.If it sounds unbelievable that, as Marquess puts it, a “ragtag group ofR O A D S A N D B R I D G E S : T H E U N S E E N L A B O R B E H I N D O U R D I G I TA L I N F R A S T R U C T U R E8Email interview with SteveMarquess
I ntroduction16amateurs could outcompete huge corporations with their money andresources,” 9 consider how this work reflects the rise of peer-to-peercollaboration around the world.Unlikely startups like Uber or AirBnB exploded into major corporatepowerhouses in just a few years, challenging longstanding industrieslike transportation and hospitality. Musicians make a name forthemselves through YouTube or Soundcloud instead of big recordlabels
source had seen an explosion of production as well as demand in the past three to five years, thanks to improvements in developer tools and workflows. Today’s open source contributor looked very different from an open source contributor ten years ago, much less thirty years ago. An
