WIXLIB

THE ORIGIN OF CHAOS MONKEYWhy NetflixNeeded toCreate FailureIn this chapter we’ll take a deep dive into the origins and history ofChaos Monkey, how Netflix streaming services emerged, and whyNetflix needed to create failure within their systems to improve theirservice and customer experiences. We’ll also provide a brief overviewof The Simian Army and its relation to the original Chaos Monkeytechnology. Finally, we’ll jump into the present and future of ChaosMonkey, dig into the creation and implementation of Failure InjectionTesting at Netflix, and discuss the potential issues and limitationspresented by Chaos Monkey’s reliance on Spinnaker.The History of NetflixStreamingNetflix launched their streaming service in early 2007, as a free addonfor their existing DVD-by-mail subscribers. While their initial streaminglibrary contained only around 1,000 titles at launch, the popularity anddemand continued to rise, and Netflix kept adding to their streaminglibrary, reaching over 12,000 titles by June 2009.Netflix’s streaming service was initially built by Netflix engineerson top of Microsoft software and housed within vertically scaledserver racks. However, this single point of failure came back to bitethem in August 2008, when a major database corruption resultedin a three-day downtime during which DVDs couldn’t be shipped tocustomers. Following this event, Netflix engineers began migratingthe entire Netflix stack away from a monolithic architecture, and intoa distributed cloud architecture, deployed on Amazon Web Services.This major shift toward a distributed architecture of hundreds ofThe Origin of Chaos Monkey8

microservices presented a great deal of additional complexity. Thislevel of intricacy and interconnectedness in a distributed systemcreated something that was intractable and required a new approachto prevent seemingly random outages. But by using proper ChaosEngineering techniques, starting first with Chaos Monkey and evolvinginto more sophisticated tools like FIT, Netflix was able to engineer aresilient architecture.Netflix’s move toward a horizontally scaled software stack requiredsystems that were much more reliable and fault tolerant. One ofthe most critical lessons was that “the best way to avoid failure isto fail constantly.”. The engineering team needed a tool that couldproactively inject failure into the system. This would show the teamhow the system behaved under abnormal conditions, and would teachthem how to alter the system so other services could easily toleratefuture, unplanned failures. Thus, the Netflix team began their journeyinto Chaos.The Simian ArmyThe Simian Army is a suite of failure injection tools created by Netflixthat shore up some of the limitations of Chaos Monkey’s scope.Check out the Simian Army - Overview and Resources chapter for allthe details on what the Simian Army is, why it was created, the toolsthat make up the Army, the strategies used to perform various ChaosExperiments, and a tutorial to help you install and begin using theSimian Army tools.Chaos Monkey TodayChaos Monkey 2.0 was announced and publicly released on GitHubin late 2016. The new version includes a handful of major featurechanges and additions.SPINNAKER REQUIREMENTSpinnaker is an open-source, multi-cloud continuous delivery platformdeveloped by Netflix, which allows for automated deployments acrossmultiple cloud providers like AWS, Azure, Kubernetes, and a few more.One major drawback of using Chaos Monkey is that it forces you and yourorganization to build atop Spinnaker’s CD architecture. If you need someguidance on that, check out our Spinnaker deployment tutorials.IMPROVED SCHEDULINGInstance termination schedules are no longer determined by probabilisticalgorithms, but are instead based on the mean time betweenThe Origin of Chaos Monkey9

terminations. Check out How to Schedule Chaos Monkey Terminations fortechnical instructions.TRACKERSTrackers are Go language objects that report instance terminations toexternal services.LOSS OF ADDITIONAL CAPABILITIESPrior to 2.0, Chaos Monkey was capable of performing additional actionsbeyond just terminating instances. With version 2.0, those capabilitieshave been removed and moved to other Simian Army tools.Failure Injection TestingIn October 2014, dissatisfied with the lack of control introducedwhen unleashing some of The Simian Army tools, Netflix introduceda solution they called Failure Injection Testing (FIT). Built by a smallteam of Netflix engineers – including Gremlin Co-Founder and CEOKolton Andrus – FIT added dimensions to the failure injection process,allowing Netflix to more precisely determine what was failing andwhich components that failure impacted.FIT works by first pushing failure simulation metadata to Zuul, whichis an edge service developed by Netflix. Zuul handles all requestsfrom devices and applications that utilize the back end of Netflix’sstreaming service. As of version 2.0, Zuul can handle dynamicrouting, monitoring, security, resiliency, load balancing, connectionpooling, and more. The core functionality of Zuul’s business logiccomes from Filters, which behave like simple pass/fail tests appliedto each request and determine if a given action should be performedfor that request. A filter can handle actions such as adding debuglogging, determining if a response should be GZipped, or attachinginjected failure, as in the case of FIT.The introduction of FIT into Netflix’s failure injection strategy was agood move toward better, modern-day Chaos Engineering practices.Since FIT is a service unto itself, it allowed failure to be injectedby a variety of teams, who could then perform proactive ChaosExperiments with greater precision. This allowed Netflix to trulyemphasize a core discipline of Chaos Engineering, knowing they weretesting for failure in every nook and cranny, proving confidence thattheir systems were resilient to truly unexpected failures.The Origin of Chaos Monkey10

Unlike Chaos Monkey, tools like FIT and Gremlin are able to test fora wide range of failure states beyond simple instance destruction.In addition to killing instances, Gremlin can fill available disk space,hog CPU and memory, overload IO, perform advanced network trafficmanipulation, terminate processes, and much more.Chaos Monkey andSpinnakerAs discussed above and later in our Spinnaker Quick Start guide, ChaosMonkey can only be used to terminate instances within an applicationmanaged by Spinnaker.This requirement is not a problem for Netflix or those other companies (suchas Waze) that using Spinnaker to great success. However, limiting your ChaosEngineering tools and practices to just Chaos Monkey also means limitingyourself to only Spinnaker as your continuous delivery and deploymentsolution. This is a great solution if you’re looking to tightly integrate with all thetools Spinnaker brings with it. On the other hand, if you’re looking to expandout into other tools this may present a number of potential issues:SETUP AND PROPAGATIONSpinnaker requires quite a bit of investment in server setup andpropagation. As you may notice in even the streamlined, provider-specifictutorials found later in this guide, getting Spinnaker up and running on aproduction environment takes a lot of time (and a hefty number of CPUcycles).LIMITED DOCUMENTATIONSpinnaker’s official documentation is rather limited and somewhatoutdated in certain areas.PROVIDER SUPPORTSpinnaker currently supports most of the big name cloud providers, but ifyour use case requires a provider outside of this list you’re out of luck (orwill need to develop your own CloudDriver).The Origin of Chaos Monkey11

CHAOS MONKEY TUTORIALA Step-byStep Guideto CreatingFailure on AWSHow to Quickly DeploySpinnaker for ChaosMonkeyThis chapter will provide a step-by-step guide for setting up and usingChaos Monkey with AWS. We also examine a handful of scenariosin which Chaos Monkey is not always the most relevant solution forChaos Engineering implementation, due to its Spinnaker requirementsand limited scope of only handling instance terminations.Modern Chaos Monkey requires the use of Spinnaker, which is anopen-source, multi-cloud continuous delivery platform developed byNetflix. Spinnaker allows for automated deployments across multiplecloud platforms (such as AWS, Azure, Google Cloud Platform, andmore). Spinnaker can also be used to deploy across multiple accountsand regions, often using pipelines that define a series of eventsthat should occur every time a new version is released. Spinnaker isa powerful tool, but since both Spinnaker and Chaos Monkey weredeveloped by and for Netflix’s own architecture, you’ll need to do theextra legwork to configure Spinnaker to work within your applicationand infrastructure.That said, in this first section we’ll explore the fastest and simplestway to get Spinnaker up and running, which will then allow you tomove onto installing and then using.We’ll be deploying Spinnaker on AWS, and the easiest method forChaos Monkey Tutorial12

Looking to DeploySpinnaker In AnotherEnvironment?doing so is to use the CloudFormation Quick Start template.The AWS Spinnaker Quick Start will create a simple architecture forIf you’re looking for the utmost control overyou containing two subnets (one public and one private) in a Virtualyour Spinnaker deployment you shouldPrivate Cloud (VPC). The public subnet contains a Bastion hostcheck out our [How to Deploy a Spinnakerinstance designed to be strictly accessible, with just port 22 openStack for Chaos Monkey][#spinnaker-for SSH access. The Bastion host will then allow a pass throughmanual] guide, which provides a step-by-connection to the private subnet that is running Spinnaker.step tutorial for setting up Halyard andSpinnaker on a local or virtual machine ofyour choice.AWS Spinnaker Quick Start Architecture - Courtesy of AWS*This quick start process will take about 10 - 15 minutes and is mostlyautomated.Creating the SpinnakerStack1(Optional) If necessary, visit https://aws.amazon.com/ to sign upfor or login to your AWS account.2(Optional) You’ll need at least one AWS EC2 Key Pair for securelyconnecting via SSH.Chaos Monkey Tutorial13

1.If you don’t have a KeyPair already start by opening the AWSConsole and navigate to EC2 NETWORK & SECURITY KeyPairs.2.Click Create Key Pair and enter an identifying name in the Keypair name field.3.Click Create to download the private .pem key file to your localsystem.4.Save this key to an appropriate location (typically your localuser /.ssh directory).3After you’ve signed into the AWS console visit this page, whichshould load the quickstart-spinnakercf.template.4Click Next.5(Optional) If you haven’t already done so, you’ll need to create atleast one AWS Access Key.6Select the KeyName of the key pair you previously created.7Input a secure password in the Password field.8(Optional) Modify the IP address range in the SSHLocation fieldto indicate what IP range is allowed to SSH into the Bastionhost. For example, if your public IP address is 1.2.3.4 you mightenter 1.2.3.4/32 into this field. If you aren’t sure, you can enter0.0.0.0/0 to allow any IP address to connect, though this isobviously less secure.9Click Next.10(Optional) Select an IAM Role with proper CloudFormationpermissions necessary to deploy a stack. If you aren’t sure, leavethis blank and deployment will use your account’s permissions.11Modify any other fields on this screen you wish, then click Nextto continue.Chaos Monkey Tutorial14

12Check the I acknowledge that AWS CloudFormation mightIf your AWS account already contains thecreate IAM resources with custom names. checkbox and clickBaseIAMRole AWS::IAM::Role you mayCreate to generate the stack.have to delete it before this template willsucceed.13Once the Spinnaker stack has a CREATE COMPLETE Status,select the Outputs tab, which has some auto-generated stringsyou’ll need to paste in your terminal in the next section.Connecting to theBastion Host1Copy the Value of the SSHString1 field from the stack Outputstab above.2Execute the SSHString1 value in your terminal and enter yeswhen prompted to continue connecting to this host.Permission denied(publickey).ssh -A -L 9000:localhost:9000 -L 8084:localhost:8084 -L8087:localhost:8087 ec2-user@54.244.189.78If you received a permission deniedSSH error you may have forgotten toplace your .pem private key file that you3You should now be connected as the ec2-user to the Bastiondownloaded from the AWS EC2 Key Pairinstance. Before you can connect to the Spinnaker instancecreation page. Make sure it is located inyou’ll probably need to copy your .pem file to the Spinnakeryour /.ssh user directory. Otherwise youinstance’s /.ssh directory.can specify the key by adding an optionalºº Once the key is copied, make sure you set proper permissions-i identify file path flag, indicating theotherwise SSH will complain.path to the .pem file.chmod 400 /.ssh/my key.pemConnecting to theSpinnaker Host1To connect to the Spinnaker instance copy and paste theSSHString2 Value into the terminal.ssh –L 9000:localhost:9000 -L 8084:localhost:8084 -L8087:localhost:8087 ubuntu@10.100.10.167 -i /.ssh/mykey.pemChaos Monkey Tutorial15

2You should now be connected to the SpinnakerWebServer !Permission denied(publickey).Upon connecting to the Spinnakerinstance you may see a messageindicating the system needs to berestarted. You can do this through theAWS EC2 console, or just enter the sudoreboot command in the terminal, thenreconnect after a few moments.Configuring SpinnakerThe Spinnaker architecture is composed of a collection ofmicroservices that each handle various aspects of the entireservice. For example, Deck is the web interface you’ll spend mosttime interacting with, Gate is the API gateway that handles mostcommunication between microservices, and CloudDriver is theservice that communicates and configures all cloud providersSpinnaker is working with.Since so much of Spinnaker is blown out into smaller microservices,configuring Spinnaker can require messing with a few different files.If there’s an issue you’ll likely have to look through individual logs foreach different service, depending on the problem.Spinnaker is configured through /opt/spinnaker/config/spinnaker.ymlfile. However, this file will be overwritten by Halyard or other changes,so for user-generated configuration you should actually modify the /opt/spinnaker/config/spinnaker-local.yml file. Here’s a basic exampleof what that file looks like.# innaker:timezone: ‘America/Los Angeles’providers:aws:# For more information on configuring Amazon Web Services (aws), see# t-setup#section-amazon-web-services-setupChaos Monkey Tutorial16

enabled: {SPINNAKER AWS ENABLED:false}defaultRegion: {SPINNAKER AWS DEFAULT REGION:us-west-2}defaultIAMRole: ls:name: default# Store actual credentials in HOME/.aws/credentials. See spinnaker.yml# for more information, including alternatives.#will be interpolated with the aws account name (e.g. te: “-keypair”# .Standalone Spinnaker installations (such as the one created viathe AWS Spinnaker Quick Start) are configured directly through thespinnaker.yml and spinnaker-local.yml override configuration files.Creating an ApplicationIn this section we’ll manually create a Spinnaker applicationcontaining a pipeline that first bakes a virtual machine image and thendeploys that image to a cluster.1Open the Spinnaker web UI (Deck) and click Actions CreateApplication.2Input bookstore in the Name field.3Input your own email address in the Owner Email field.4(Optional) If you’ve enabled Chaos Monkey in Spinnaker you canopt to enable Chaos Monkey by checking the Chaos Monkey Enabled box.5Input My bookstore application in the Description field.6Under Instance Health, tick the Consider only cloud providerhealth when executing tasks checkbox.7Chaos Monkey TutorialClick Create to add your new application.17

Adding a Firewall1Navigate to the bookstore application, INFRASTRUCTURE FIREWALLS, and click Create Firewall.2Input dev in the Detail field.3Input Bookstore dev environment in the Description field.4Within the VPC dropdown select SpinnakerVPC.5Under the Ingress header click Add new Firewall Rule. Set thefollowing Firewall Rule settings.6Chaos Monkey TutorialººFirewall: defaultººProtocol: TCPººStart Port: 80ººEnd Port: 80Click the Create button to finalize the firewall settings.18

Adding a Load Balancer1Navigate to the bookstore application, INFRASTRUCTURE LOAD BALANCERS, and click Create Load Balancer.Chaos Monkey Tutorial2Select Classic (Legacy) and click Configure Load Balancer.3Input test in the Stack field.4In the VPC Subnet dropdown select internal (vpc-.).5In the Firewalls dropdown select bookstore--dev (.).6Click Create to generate the load balancer.19

Add Load Balancer Spinnaker DialogCreating a Pipeline inSpinnakerThe final step is to add a pipeline, which is where we tell Spinnakerwhat it should actually “do”! In this case we’ll tell it to bake a virtualmachine image containing Redis, then to deploy that image to ourwaiting EC2 instance.1Navigate to the bookstore application, PIPELINES and clickCreate Pipeline.Adding a Bake Stage2Select Pipeline in the Type dropdown.3Input Bookstore Dev to Test in the Pipeline Name field.4Click Create.1Click the Add stage button.2Under Type select Bake.20

3Input redis-server in the Package field.4Select trusty (v14.04) in the Base OS dropdown.5Click Save Changes to finalize the stage.Add Bake Deployment Stage Spinnaker DialogIgnoring Jenkins/TravisIn production environments you’ll likely also want to incorporate Travis, Jenkins, or another CIsolution as a preceding stage to the bake stage. Otherwise, Spinnaker will default to bakingand deploying the most recently built package. For our purposes here we don’t care, sincewe’re using an unchanging image.Adding a Deploy Stage123Click the Add stage button.Under Type select Deploy.Click the Add server group button to begin creating a newserver group.21

Adding a Server Group1Select internal (vpc-.) in the VPC Subnet dropdown.2Input dev in the Stack field.3Under Load Balancers Classic Load Balancers select thebookstore-dev load balancer we created.4Under Firewalls Firewalls select the bookstore--dev firewallwe also created.5Under Instance Type select the Custom Type of instance youthink you’ll need. For this example we’ll go with something smalland cheap, such as t2.large.6Input 3 in the Capacity Number of Instances field.7Under Advanced Settings Key Name select the key pair nameyou used when deploying your Spinnaker CloudFormation stack.8In the Advanced Settings IAM Instance Profile field input theInstance Profile ARN value of the BaseIAMRole found in theAWS IAM Roles BaseIAMRole dialog (e.g. MRol

Testing at Netflix, and discuss the potential issues and limitations presented by Chaos Monkey's reliance on Spinnaker. Netflix launched their streaming service in early 2007, as a free addon for their existing DVD-by-mail subscribers. While their initial streaming library contained only around 1,000 titles at launch, the popularity and