Transcription
Modern DevOpsPracticesImplement and secure DevOps in the public cloudwith cutting-edge tools, tips, tricks, and techniquesGaurav AgarwalBIRMINGHAM—MUMBAI
Modern DevOps PracticesCopyright 2021 Packt PublishingAll rights reserved. No part of this book may be reproduced, stored in a retrieval system, ortransmitted in any form or by any means, without the prior written permission of the publisher,except in the case of brief quotations embedded in critical articles or reviews.Every effort has been made in the preparation of this book to ensure the accuracy of theinformation presented. However, the information contained in this book is sold withoutwarranty, either express or implied. Neither the author(s), nor Packt Publishing or its dealers anddistributors, will be held liable for any damages caused or alleged to have been caused directly orindirectly by this book.Packt Publishing has endeavored to provide trademark information about all of the companiesand products mentioned in this book by the appropriate use of capitals. However, Packt Publishingcannot guarantee the accuracy of this information.Group Product Manager: Wilson D'SouzaPublishing Product Manager: Vijin BorichaSenior Editor: Arun NadarContent Development Editor: Mrudgandha KulkarniTechnical Editor: Nithik CheruvakodanCopy Editor: Safis EditingProject Coordinator: Ajesh DevavaramProofreader: Safis EditingIndexer: Rekha NairProduction Designer: Joshua MisquittaFirst published: July 2021Production reference: 2300821Published by Packt Publishing Ltd.Livery Place35 Livery StreetBirminghamB3 2PB, UK.ISBN 978-1-80056-238-7www.packt.com
I want to thank my wonderful wife, Deepti, for giving me the space andsupport I've needed to write this book. I'd also like to thank Vijin forgranting me the opportunity to complete this journey and Ajesh for keepingme on track. Special thanks to Aniket for reviewing the book. The wholePackt editing team has helped this first-time author immensely, but I'd liketo give special thanks to Mrudgandha and Arun, who edited most of mywork.
ContributorsAbout the authorGaurav Agarwal has a decade of experience as a site reliability engineer (SRE), architect,tech influencer, trainer, mentor, and developer. Currently, Gaurav works as a cloudSRE at ThoughtSpot Inc. Prior to that, Gaurav worked as a cloud solutions architect atCapgemini and as a software developer at TCS. Gaurav has a B.Tech. in electronics andcommunication engineering, and he is a Certified Kubernetes Administrator, CertifiedTerraform Associate, and a Google Cloud Certified Professional Cloud Architect. Whennot working, Gaurav enjoys time with his wonderful wife, Deepti, and loves to read abouthistory, human civilization, and the arts.To the doctors, nurses, public health officials, and first responders who areprotecting us from COVID-19.
About the reviewerAniket Mhala has more than 25 years of experience in solution architecture andimplementing legacy and cloud-native systems. He heads a technology practice thatmainly focuses on solution design, agile transformation, enterprise DevOps adoption,application modernization, and integration.He is particularly experienced in microservices, modern DevOps, Kafka, cloud platforms,Docker, Kubernetes, and other open source frameworks. He has published the Anixframework for organization transformation and innovation.
Table of ContentsPrefaceSection 1: Container Fundamentals andBest Practices1The Move to ContainersThe need for containers 4The matrix of hell 6Virtual machines 7Containers 8It works on my machine 9Container architecture Container networking Modern DevOps versustraditional DevOps Containers and modernDevOps practices Migrating from virtualmachines to containers 911141517Discovery 17Application requirement assessment 18Container infrastructure design 18Containerizing the application 19Testing 19Deployment and rollout 20What applications should go incontainers? 21Breaking the applications into smallerpieces 23Are we there yet? 24Summary 24Questions 25Answers 262Containerization with DockerTechnical requirements Installing tools 2828Installing Git Installing vim 2829
vi Table of ContentsInstalling Docker Introducing Docker storagedrivers and volumes 29Docker monitoring withPrometheus 4832Docker data storage options Mounting volumes Docker storage drivers Configuring a storage driver 32333435Running your first container 36Challenges with container monitoring 49Installing Prometheus 49Configuring cAdvisor and the nodeexporter to expose metrics 50Configuring Prometheus to scrapemetrics 50Launching a sample containerapplication 51Metrics to monitor 54Running containers from versionedimages 37Running Docker containers in thebackground 38Troubleshooting containers 38Putting it all together 40Restarting and removing containers 42Docker logging and loggingdrivers 43Container log management Logging drivers Configuring logging drivers Typical challenges and best practiceswith Docker logging 43434447Declarative containermanagement with DockerCompose 55Installing Docker Compose Deploying a sample application withDocker Compose Creating the docker-compose file Docker Compose best practices 56565861Summary 63Questions 63Answers 643Creating and Managing Container ImagesTechnical requirements 66Docker architecture 66Understanding Dockerimages 67The layered filesystem Image history 6869Understanding Dockerfiles,components, and directives 71Can we use ENTRYPOINT instead ofCMD? 72Are RUN and CMD the same? 72Building our first container 73Building and managing Dockerimages 81Single-stage builds Multi-stage builds Managing Docker images Flattening Docker images 82848691Optimizing containers withdistroless images 93Performance 93Security 94
Table of Contents viiCost 94Understanding Docker registries 96Hosting your private Docker registry Other public registries 97100Summary 101Questions 101Answers 1024Container Orchestration with Kubernetes – Part ITechnical requirements 106What is Kubernetes and why doI need it? 106Kubernetes architecture 109Installing Kubernetes (Minikubeand KinD) 111Understanding Kubernetespods 116Installing Minikube Installing KinD Summary 147Questions 147Answers 149111114Using port forwarding Troubleshooting pods Ensuring pod reliability Pod multi-container design patterns 1201211241275Container Orchestration with Kubernetes – Part IITechnical requirements 152Spinning up Google KubernetesEngine 152Kubernetes Deployments ReplicaSet resource Deployment resource Kubernetes Deployment strategies 153153156160Kubernetes Services andIngresses 169ClusterIP Services NodePort services LoadBalancer services Ingress resources 170174176178Horizontal Pod autoscaling 184Managing stateful applications 187StatefulSet resource Managing persistent volumes 188189Kubernetes command-line bestpractices 199Using alias Using kubectl bash autocompletion 199202Summary 202Questions 203Answers 204
viii Table of ContentsSection 2: Delivering Containers6Infrastructure as Code (IaC) with TerraformTechnical requirements Introduction to IaC Installing Terraform Terraform providers Authentication and authorizationwith Azure Using the Azure Terraform provider 208208211211212214Terraform variables 215Providing variable values 217Terraform workflow 218terraform init Creating the first resource – Azureresource group terraform fmt terraform validate terraform plan terraform apply terraform destroy 219219220220221222223terraform state Using the Azure Storage backend Terraform workspaces Inspecting resources Inspecting state files Cleaning up Terraform output, state,console, and graphs 225226230234236237238terraform output 238Managing Terraform state 239terraform console 241Terraform dependencies andgraph 242Cleaning up resources 243Summary 243Questions 244Answers 2457Configuration Management with AnsibleTechnical requirements 248Introduction to configmanagement 248Setting up Ansible 250Setting up inventory Installing Ansible in the control node Connecting the Ansible control nodewith inventory servers 250251252Setting up an inventory file Setting up the Ansibleconfiguration file 253255Ansible tasks and modules 257Introduction to Ansibleplaybooks 258Checking playbook syntax Applying the first playbook 259260
Table of Contents ixAnsible playbooks in action 261Updating packages andrepositories 261Installing application packages andservices 262Configuring applications 263Combining the playbooks 266Executing the playbooks 267Designing for reusability Ansible variables Sourcing variable values Jinja2 templates Ansible roles 267268270271272Summary 278Questions 278Answers 2808IaC and Config Management in ActionTechnical requirements Immutable infrastructurewith Hashicorp's Packer 282Ansible provisioners 282Prerequisites 288Defining the Packerconfiguration 290When to use immutableinfrastructure 284Installing Packer 286Creating the Apache andMySQL playbooks 287Building the Apache and MySQLimages using Packer and288Creating the requiredinfrastructure withTerraform 297Summary 305Questions 306Answers 3079Containers as a Service (CaaS) and Serverless Computing forContainersTechnical requirements 310The need for serverlessofferings 311Amazon ECS with EC2 andFargate 312ECS architecture Installing the AWS and ECS CLIs Spinning up an ECS cluster Creating task definitions 312315315317Scheduling EC2 tasks on ECS 318Scaling tasks 319Querying container logs fromCloudWatch 319Stopping tasks 320Scheduling Fargate tasks on ECS 320Scheduling services on ECS 324Browsing container logs using theECS CLI 326Deleting an ECS service 326
x Table of ContentsLoad balancing containers runningon ECS 327Other CaaS products 329Open source CaaS withKnative 330Knative architecture 332Spinning up Google KubernetesEngine 334Installing Knative 334Deploying a Python Flask app onKnative 336Load testing your app on Knative 340Summary 341Questions 341Answers 34210Continuous IntegrationTechnical requirements 344The importance of automation 344Building a CI pipeline withGitHub Actions 346Code Build Creating a GitHub repository Creating a GitHub Actions workflow Build performance bestpractices 380348349Scalable Jenkins on Kuberneteswith Kaniko 356Spinning up Google KubernetesEngine 359Installing Jenkins 359Connecting Jenkins with the cluster 364Running our first Jenkins job 371Automating a build withtriggers 375CI with AWS Code Commit and376Creating an AWS Code Commitrepository 377Creating an AWS Code Build job 378Aim for faster builds Always use post-commit triggers Configure build reporting Customize the build server size Ensure that your builds onlycontain what you need 380380380380381Summary 381Questions 381Answers 38211Continuous Deployment/Delivery with SpinnakerTechnical requirements Importance of ContinuousDeployment and automation 384Continuous deployment modelsand tools 386385Simple deployment model Complex deployment models 387388
Table of Contents xiIntroduction to Spinnaker Setting up Spinnaker 390392Deploying a sample applicationusing a Spinnaker pipeline 400Spinning up Google Kubernetes Engine 393Setting up service accounts andpermissions 394Creating a halyard host VM 395Installing halyard 396Setting up the required credentials 397Setting up the Spinnaker configuration 397Deploying Spinnaker 398Summary 414Questions 415Answers 416Creating a deployment manifest Creating a Spinnaker application Creating a Spinnaker pipeline Testing the pipeline 40040240441012Securing the Deployment PipelineTechnical requirements Securing CI/CD pipelines Managing secrets Sample application Creating a Secret manifest Creating a Cloud KMS secret Accessing the secret and deployingthe application 418419420422422423424Container vulnerabilityscanning 426Installing Anchore Grype Scanning images 426427Binary authorization 430Setting up binary authorization 431Creating a default binaryauthorization policy Attesting images 432434Security of modern DevOpspipelines 435Adopt a DevSecOps culture Establish access control Implement shift left Manage security risks consistently Implement vulnerability scanning Automate security 435435435436436436Summary 436Questions 437Answers 437Section 3: Modern DevOps with GitOps13Understanding DevOps with GitOpsTechnical requirements What is GitOps? 442442The principles of GitOps Why GitOps? 443444
xii Table of ContentsThe branching strategy andGitOps workflow 445The push model 445The pull model Structuring the Git repository 446447Declarative infrastructure andconfig management 451Summary 457Questions 457Answers 45814CI/CD Pipelines with GitOpsTechnical requirements Continuous integration withGitHub Actions 460460Creating an application repository onGitHub 463Creating a GitHub Actions workflow 463Release gating with pullrequests 467Continuous deployment withFlux CD 469Other Books You May EnjoyIndexIntroduction to Flux CD Installing Flux CD Managing sensitiveconfiguration and Secrets Installing the Sealed Secrets operator Installing kubeseal Creating Sealed Secrets 472473485486487487Summary 490Questions 490Answers 491
PrefaceThis book goes beyond just the fundamentals of DevOps tools and their deployments.It covers practical examples to get you up to speed with containers, infrastructureautomation, serverless container services, continuous integration and delivery, automateddeployments, deployment pipeline security, GitOps, and more.Who this book is forIf you are a software engineer, system administrator, or operations engineer looking tostep into the world of DevOps within public cloud platforms, this book is for you. CurrentDevOps engineers will also find this book useful as it covers best practices, tips, andtricks to implement DevOps with a cloud-native mindset. Although no containerizationexperience is necessary, a basic understanding of the software development life cycle anddelivery will help you get the most out of the book.What this book coversChapter 1, The Move to Containers, introduces containers. Containers are in vogue lately,and though the concept is well understood, it is worth introducing to you the book's scopeand how containers are changing the current IT landscape. As containers are a relativelynew concept, it is imperative that we understand the best practices and techniquessurrounding the building, deploying, and securing of container-based applications.Chapter 2, Containerization with Docker, will introduce Docker and cover installingDocker, configuring Docker storage drivers, running our first Docker container, andmonitoring Docker with journald and Splunk.Chapter 3, Creating and Managing Container Images, covers Docker images. Dockerimages are one of the key components when working with Docker. In this chapter, wewill learn about Docker images, the layered model, Dockerfile directives, how to flattenimages, building images, and the best practices surrounding image building. We will alsolook at distroless images and how they are good from a DevSecOps perspective.
xivPrefaceChapter 4, Container Orchestration with Kubernetes – Part I, introduces Kubernetes.We will install Kubernetes using Minikube and KinD, talk a bit about Kubernetes'architecture, and then move on to the fundamental building blocks of Kubernetes, whichinclude Pods, containers, ConfigMaps, secrets, and multi-container Pods.Chapter 5, Container Orchestration with Kubernetes – Part II, moves on to the advancedconcepts of Kubernetes, including networking, DNS, Services, Deployments, HorizontalPod Autoscaler, and StatefulSets.Chapter 6, Infrastructure as Code (IaC) with Terraform, introduces IaC with Terraformand explains the core concepts of IaC. We will then move on to a hands-on example wherewe will be building a resource group and a virtual machine from scratch on Azure usingTerraform while understanding the core Terraform concepts.Chapter 7, Configuration Management with Ansible, introduces configuration managementwith Ansible and explains its core concepts. We will then learn about the core Ansibleconcepts when configuring a MySQL and Apache application on Azure Virtual Machines.Chapter 8, IaC and Config Management in Action, talks about immutable infrastructureusing Packer and uses this, along with the concepts of Chapter 5, Container Orchestrationwith Kubernetes – Part II, and Chapter 6, Infrastructure as Code (IaC) with Terraform, toboot up an IaaS-based Linux, Apache, MySQL, and PHP (LAMP) stack on Azure.Chapter 9, Containers as a Service (CaaS) and Serverless Computing for Containers, looksat how Kubernetes forms a hybrid between IaaS and PaaS approaches. But when we don'twant to manage infrastructure and want something lightweight to host our container, wecan look at serverless container services such as AWS ECS. We will also briefly discussalternatives such as Google Cloud Run and Azure Container Instances. We will thendiscuss Knative, which is an open source, cloud-native, serverless technology.Chapter 10, Continuous Integration, looks at continuous integration from a containerperspective and talks about various tools and techniques for continuously building acontainer-based application. We will look at tools such as GitHub Actions, Jenkins, andAWS Cloud Build and discuss how and when to use each of them.Chapter 11, Continuous Deployment/Delivery with Spinnaker, looks into continuousdeployment/delivery using Spinnaker. Spinnaker is a modern continuous delivery toolthat helps you deploy and manage your container application seamlessly.Chapter 12, Securing the Deployment Pipeline, explores multiple ways of securing acontainer deployment pipeline, including managing secrets, storing secrets, containerimage analysis, vulnerability scanning, and binary authorization.Chapter 13, Understanding DevOps with GitOps, looks at the GitOps approach for doingDevOps and how it is expanding in popularity.
PrefacexvChapter 14, CI/CD Pipeline with GitOps, gets hands-on and sees you create a complete CI/CD pipeline using the GitOps approach. We will look at tools such as GitHub Actions andFlux CD.To get the most out of this bookFor this book, you will need the following: An Azure subscription to perform some of the exercises. Currently, Azure offers afree trial for 30 days with 200 worth of free credits; sign up at https://azure.microsoft.com/en-in/free. An AWS subscription. Currently, AWS offers a free tier for some products. Youcan sign up at https://aws.amazon.com/free. The book uses some paidservices, but we will try to minimize how many we use as much as possible duringthe exercises. A Google Cloud Platform subscription. Currently, Google Cloud Platform providesa free 300 trial for 90 days, which you can go ahead and sign up for at https://console.cloud.google.com/.For some chapters, you will need to clone the following GitHub repository to proceedwith the n-DevOps-Practices
xviPrefaceDownload the color imagesWe also provide a PDF file that has color images of the screenshots/diagrams used in thisbook. You can download it here: ads/9781800562387 ColorImages.pdf.Conventions usedThere are a number of text conventions used throughout this book.Code in text: Indicates code words in text, database table names, folder names,filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles.Here is an example: "We're doing two things in this command – first, we are runninggcloud secrets versions access latest --secret flask-appsecret to access the contents of the secret, and then we are piping it directly tokubectl apply -f -."A block of code is set as follows:.spec:containers:- image: ' your docker user /flask-app-secret:1'name: flask-appports:- containerPort: 5000env:- name: SECRETvalueFrom:secretKeyRef:name: flask-app-secretkey: SECRET
PrefacexviiBold: Indicates a new term, an important word, or words that you see onscreen. Forexample, words in menus or dialog boxes appear in the text like this. Here is an example:"Click Flash from Etcher to write the image."Any command-line input or output is written as follows: git clone Practices.git modern-devops cd modern-devops/ch7Tips or important notesAppear like this.Get in touchFeedback from our readers is always welcome.General feedback: If you have questions about any aspect of this book, mention the booktitle in the subject of your message and email us at customercare@packtpub.com.Errata: Although we have taken every care to ensure the accuracy of our content, mistakesdo happen. If you have found a mistake in this book, we would be grateful if you wouldreport this to us. Please visit www.packtpub.com/support/errata, selecting yourbook, clicking on the Errata Submission Form link, and entering the details.Piracy: If you come across any illegal copies of our works in any form on the Internet,we would be grateful if you would provide us with the location address or website name.Please contact us at copyright@packt.com with a link to the material.If you are interested in becoming an author: If there is a topic that you have expertise inand you are interested in either writing or contributing to a book, please visit authors.packtpub.com.
xviiiPrefaceShare your thoughtsOnce you've read Modern DevOps Practices, we'd love to hear your thoughts! Pleasehttps://packt.link/r/1-800-56238-1 for this book and share your feedback.Your review is important to us and the tech community and will help us make sure we'redelivering excellent quality content.
Section 1:ContainerFundamentals andBest PracticesThis section will introduce you to the world of containers and build a strong foundation ofknowledge regarding containers and container orchestration technologies. In this section,you will learn how containers help organizations build distributed, scalable, and reliablesystems in the cloud.This section comprises the following chapters: Chapter 1, The Move to Containers Chapter 2, Containerization with Docker Chapter 3, Creating and Managing Container Images Chapter 4, Container Orchestration with Kubernetes – Part I Chapter 5, Container Orchestration with Kubernetes – Part II
1The Move toContainersThis first chapter will provide you with background knowledge of containers and howthey change the entire IT landscape. While we understand that most DevOps practitionerswill already be familiar with this, it is worth providing a refresher to build the rest of thisbook's base. While this book does not entirely focus on containers and their orchestration,modern DevOps practices heavily emphasize it.In this chapter, we're going to cover the following main topics: The need for containers Container architecture Containers and modern DevOps practices Migrating to containers from virtual machinesBy the end of this chapter, you should be able to do the following: Understand and appreciate why we need containers in the first place and whatproblems they solve. Understand the container architecture and how it works.
4The Move to Containers Understand how containers contribute to modern DevOps practices. Understand the high-level steps of moving from a Virtual Machine-basedarchitecture to containers.The need for containersContainers are in vogue lately and for excellent reason. They solve the computerarchitecture's most critical problem – running reliable, distributed software with nearinfinite scalability in any computing environment.They have enabled an entirely new discipline in software engineering – microservices.They have also introduced the package once deploy anywhere concept in technology.Combined with the cloud and distributed applications, containers with containerorchestration technology has lead to a new buzzword in the industry – cloud-native –changing the IT ecosystem like never before.Before we delve into more technical details, let's understand containers in plain andsimple words.Containers derive their name from shipping containers. I will explain containersusing a shipping container analogy for better understanding. Historically, because oftransportation improvements, there was a lot of stuff moving across multiple geographies.With various goods being transported in different modes, loading and unloadinggoods was a massive issue at every transportation point. With rising labor costs, it wasimpractical for shipping companies to operate at scale while keeping the prices low.Also, it resulted in frequent damage to items, and goods used to get misplaced or mixedup with other consignments because there was no isolation. There was a need for astandard way of transporting goods that provided the necessary isolation betweenconsignments and allowed for easy loading and unloading of goods. The shippingindustry came up with shipping containers as an elegant solution to this problem.Now, shipping containers have simplified a lot of things in the shipping industry. Witha standard container, we can ship goods from one place to another by only moving thecontainer. The same container can be used on roads, loaded on trains, and transportedvia ships. The operators of these vehicles don't need to worry about what is inside thecontainer most of the time.
The need for containers5Figure 1.1 – Shipping container workflowSimilarly, there have been issues with software portability and compute resourcemanagement in the software industry. In a standard software development life cycle,a piece of software moves through multiple environments, and sometimes, numerousapplications share the same operating system. There may be differences in theconfiguration between environments, so software that may have worked on a developmentenvironment may not work on a test environment. Something that worked on test mayalso not work on production.Also, when you have multiple applications running within a single machine, there isno isolation between them. One application can drain compute resources from anotherapplication, and that may lead to runtime issues.Repackaging and reconfiguring applications are required in every step of deployment, soit takes a lot of time and effort and is sometimes error-prone.Containers in the software industry solve these problems by providing isolation betweenapplication and compute resource management, which provides an optimal solution tothese issues.The software industry's biggest challenge is to provide application isolation and manageexternal dependencies elegantly so that they can run on any platform, irrespectiveof the operating system (OS) or the infrastructure. Software is written in numerousprogramming languages and uses various dependencies and frameworks. This leads to ascenario called the matrix of hell.
6The Move to ContainersThe matrix of hellLet's say you're preparing a server that will run multiple applications for multiple teams.Now, assume that you don't have a virtualized infrastructure and that you need to runeverything on one physical machine, as shown in the following diagram:Figure 1.2 – Applications on a physical serverOne application uses one particular version of a dependency while another applicationuses a different one, and you end up managing two versions of the same software in onesystem. When you scale your system to fit multiple applications, you will be managinghundreds of dependencies and various versions catering to different applications. It willslowly turn out to be unmanageable within one physical system. This scenario is known asthe matrix of hell in popular computing nomenclature.There are multiple solutions that come out of the matrix of hell, but there are two notabletechnology contributions – virtual machines and containers.
The need for containers7Virtual machinesA virtual machine emulates an operating system using a technology called a Hypervisor.A Hypervisor can run as software on a physical host OS or run as firmware on a baremetal machine. Virtual machines run as a virtual guest OS on the Hypervisor. With thistechnology, you can subdivide a sizeable physical machine into multiple smaller virtualmachines, each catering to a particular application. This revolutionized computinginfrastructure for almost two decades and is still in use today. Some of the most popularHypervisors on the market are VMWare and Oracle VirtualBox.The following diagram shows the same stack on virtual machines. You can see that eachapplication now contains a dedicated guest OS, each of which has its own libraries anddependencies:Figure 1.3 – Applications on Virtual Machines
8The Move to ContainersThough the approach is acceptable, it is like using an entire ship for your goods ratherthan a simple container from the shipping container analogy. Virtual machines areheavy on resources as you need a heavy guest OS layer to isolate applications rather thansomething more lightweight. We need to allocate dedicated CPU and memory to a VirtualMachine; resource sharing is suboptimal since people tend to overprovision VirtualMachines to cater for peak load. They are also slower to start, and Virtual Machine scalingis traditionally more cumbersome as there are multiple moving parts and technologiesinvolved. Therefore, automating horizontal scaling using virtual machines is not verystraightforward. Also, sysadmins now have to deal with multiple servers rather thannumerous libraries and dependencies in one. It is better than before, but it is not optimalfrom a compute resource point of view.ContainersThat is where containers come into the picture. Containers solve the matrix of hell withoutinvolving a heavy guest OS layer in-between them. Instead, they isolate the applicationruntime and dependencies by encapsulating them to create an abstraction calledcontainers. Now, you have multiple containers that run on a single operating system.Numerous applications running on containers can share the same infrastructure. As aresult, they do not waste your computing resources. You also d
Other Books You May Enjoy Index. Preface This book goes beyond just the fundamentals of DevOps tools and their deployments. It covers practical examples to get you up to speed with containers, infrastructure automation, serverless container services, continuous integration and delivery, automated
