Scott Alan Miller Administration Best Practices

Transcription

Linux Administration Best Practices helps you to explore best practices for efficiently administeringLinux systems and servers. This Linux book covers a wide variety of topics, from installation anddeployment through to managing permissions, with each topic beginning with an overview of thekey concepts followed by practical examples of best practices and solutions. You'll find out howto approach system administration, Linux, and IT in general, put technology into proper businesscontext, and rethink your approach to technical decision making. Finally, the book concludes byhelping you to understand best practices for troubleshooting Linux systems and servers that'llenable you to grow in your career as well as in any aspect of IT and business.Linux AdministrationBest PracticesLinux is a well-known, open source Unix-family operating system that is the most widely usedOS today. Linux looks set for a bright future for decades to come, but system administration israrely studied beyond learning rote tasks or following vendor guidelines. To truly excel at Linuxadministration, you need to understand how these systems work and learn to make strategicdecisions regarding them.Scott Alan MillerLinux AdministrationBest PracticesLinuxAdministrationBest PracticesPractical solutions to approaching the design andmanagement of Linux systemsBy the end of this Linux administration book, you'll have gained the knowledge needed to takeyour Linux administration skills to the next level.Things you will learn: Find out how to conceptualize thesystem administrator roleUnderstand the key values of riskassessment in administrationApply technical skills to the ITbusiness contextDiscover best practices for working withLinux-specific system technologies Understand the reasoning behindsystem administration best practicesDevelop out-of-the-box thinking foreverything from reboots to backupsto triagePrioritize, triage, and plan for disastersand recoveriesDiscover the psychology behindadministration dutiesScott Alan Miller

LinuxAdministrationBest PracticesPractical solutions to approaching the design andmanagement of Linux systemsScott Alan MillerBIRMINGHAM—MUMBAI

Linux Administration Best PracticesCopyright 2022 Packt PublishingAll rights reserved. No part of this book may be reproduced, stored in a retrieval system, ortransmitted in any form or by any means, without the prior written permission of the publisher,except in the case of brief quotations embedded in critical articles or reviews.Every effort has been made in the preparation of this book to ensure the accuracy of theinformation presented. However, the information contained in this book is sold without warranty,either express or implied. Neither the author, nor Packt Publishing or its dealers and distributors,will be held liable for any damages caused or alleged to have been caused directly or indirectly bythis book.Packt Publishing has endeavored to provide trademark information about all of the companiesand products mentioned in this book by the appropriate use of capitals. However, Packt Publishingcannot guarantee the accuracy of this information.Group Product Manager: Rahul NairPublishing Product Manager: Rahul NairSenior Editor: Shazeen IqbalContent Development Editor: Rafiaa KhanTechnical Editor: Arjun VarmaCopy Editor: Safis EditingProject Coordinator: Shagun SainiProofreader: Safis EditingIndexer: Manju ArasanProduction Designer: Aparna BhagatMarketing Coordinator: Hemangi LotlikarFirst published: February 2022Production reference: 1120122Published by Packt Publishing Ltd.Livery Place35 Livery StreetBirminghamB3 2PB, UK.ISBN 978-1-80056-879-2www.packt.com

To my father, who had the wherewithal and foresight to introduce meto programming and computers at a very young age and taught me tosee technology as a business tool. And to my wife, Dominica, and mydaughters, Liesl and Luciana, for suffering through the writing of a book ontop of all of the normal craziness that life is always throwing at us. My teammakes this all possible.– Scott Alan Miller

ContributorsAbout the authorScott Alan Miller is an information technology and software engineering industryveteran of 30 years, with more than a quarter of a century on UNIX and Linux. Hisexperience has included companies of every size, in every region of the world, in nearlyevery industry. Scott has been a technician, lead, manager, educator, consultant, writer,author, speaker, and mentor. Today, and for more than the last 20 years, Scott has led theIT consulting team at NTG. He now lives in Nicaragua.

About the reviewerRené Jensen has 21 years of professional experience with UNIX/Linux administration,both as an employed administrator and, for the last 9 years, as a consultant. His experienceranges from branches such as medical, banking, tax, and mobile business, to working inareas such as CI/CD, container deployment, architecting server clusters, daily operations,and many other areas.I would like to thank my family for being patient, since my work started asa hobby and I spend a lot of time going in depth with new challenges.

Table of ContentsPrefaceSection 1: Understanding the Role of LinuxSystem Administrator1What Is the Role of a System Administrator?Where are systemadministrators in the real world? 4Wearing the administrator andengineering hats 6The difference between the role ofan administrator and the role ofan engineer 7Hats 7The wonderous variety of the role 12Understanding systems in thebusiness ecosystem 14Learning system administration 16Build a home lab Getting family and friends involved Start as a generalist and progressonto a specialist in the SystemAdministrator field Volunteer for non-profits or nonbusiness organizations Self-study Age does not matter Internships 171818192021Introducing the IT Professional 22The fallacy of success at any cost 25Summary 25162Choosing Your Distribution and Release ModelUnderstanding Linux inproduction Is Linux UNIX? 2829Linux licensing Key vendors and products 3033What about BSD? 33

viii Table of ContentsDebian Ubuntu IBM Red Hat Enterprise Linux (RHEL) RHEL alternatives Fedora OpenSUSE and SLES Digging into distribution history Other Linux distributions 3535363738383940The myth of popularity 41Using multiple distributions Making the choice 4243Releases and support: LTS,current, and rolling 45What does support mean? 47Release model: rapid release Release model: LTS Release and support scheduleinterplay: The overlap Release model: Rolling Why not just update the packagesmanually Choosing the release model for ourworkloads 4950Choosing your distribution 58Do not fear risk 59Summary 6052535456Section 2: Best Practices for LinuxTechnologies3System Storage Best PracticesExploring key factors in storage 64Cost 64Durability 65Availability 65Performance 66Scalability 67Capacity 68EXT4 77XFS 77ZFS 78BtrFS 79Clustered file systems 80Network filesystems 81Getting to know logical volumemanagement (LVM) 84Understanding block storage:Local and SAN 68Whatever happen to partitions 85Locally attached block storage Storage Area Networks (SAN) The terrible terminology of SAN 696969Utilizing RAID and RAIN 87RAID RAIN 8788Surveying filesystems andnetwork filesystems 74

Table of Contents ixLearning about replicated localstorage DRBD Gluster and CEPH Proprietary and third-party opensource solutions Virtualization abstraction of storage 9091929393Analyzing storage architecturesand risk 95General storage architectures Simple local storage: The brick RLS: The ultra-high reliability solution The lab environment: Remote sharedstandard storage The giant scale: Remote replicatedstorage Storage best practices Storage example Summary 9696991001011031041094Designing System Deployment ArchitecturesVirtualization 112Type 1 hypervisor 113Type 2 hypervisor 113Hypervisor types are confusing 115VMware ESXi 116Microsoft Hyper-V 116Xen 116KVM 117Is virtualization only for consolidation? 117Containerization Cloud and VPS Virtual Private Servers (VPS) On premises, hosted, andhybrid hosting Colocation 119122130133System Design Architecture Standalone server, aka the snowflake Simple does not necessarilymean simple Many to many servers and storage Viewing the world as a workload Layered high availability Reliability is relative Hyperconvergence Best practices in System DesignArchitecture Risk assessment and availability needs Workload interplay Defining high availability Summary 1361361381391401431441451461471491511541345Patch Management StrategiesBinary, source, and scriptsoftware deployments Compiled and interpreted software 158158Misleading use of source installation 160Patching theory and strategies 165The risk of delayed patching 166

x Table of ContentsAvoiding patches because of Windows 167Testing patches is rarely feasible 169Timeliness of patching 170Linux deployment andredeployment 177Rebooting servers 180Compilations for theadministrator Finding your green zone Avoiding planned downtime isplanning for unplanned downtime 173The compilation era 175Compilation by engineeringdepartment 176Summary 1811821846DatabasesSeparating a Database from aDBMS 189The Database The Database engine The Database management system 189190193Comparing relational andNoSQL databases 196Discovering common databaseson Linux 199Common relational databaseson Linux Drop In replacements Common NoSQL Database Products onLinux Document databases 200201204204Understanding databasereplication and data protectionconcepts 207Summary 212Section 3: Approaches to Effective SystemAdministration7Documentation, Monitoring, and Logging TechniquesModern documentation: Wiki,live docs, repos Repos Ticketing systems Approaching documentation 216220220221Tooling and impact Netdata Capacity planning 223224226It Is already designed when purchased 226

Table of Contents xiLog management and security 233Why central logging? 237Alerts and troubleshooting 242On-device and centralized alertingsystems 243Pushed and pulled alerts In house and hosted monitoring RMMs and monitoring Summary 2452472482498Improving Administration Maturation with Automationthrough Scripting and DevOpsThe GUI and the CLI: Administrationbest practices Consolidation and the age ofsqueezing systems Automation maturity Local and remote automation Command line Scheduled tasks Scripting PowerShell on Linux 253253257258259259260260Scripting combined with taskscheduling State management Infrastructure as code Platforms and systems Modern tools of automation Configuration management systems Version control systems Summary 2652652682682722732752789Backup and Disaster Recovery ApproachesAgents and crash consistency Locking mechanisms in Linux MySQL example with mysqldumputility Backup strategies &mechanisms Types of backups Snapshots, archives, backups,and disaster recovery Snapshots 280282287288289296296Archives Backups Disaster recovery Backups in a DevOps world Version control systems IT provides solutions, vendorssell components Triage concepts Summary 299302307309309312313315

xii Table of Contents10User and Access Management StrategiesLocal and remote users User managementmechanisms Using automation to turn local usesinto remote users The famous RDP exposure risk Are operating system logins relevantin the modern world? 318320320324326Remote access approaches 329How do I approach remote access 331SSH, key management, andjump boxes 332Do you still need both a networkedge firewall and an operatingsystem firewall? Does changing the default port ofSSH work? SSH key management Jump boxes Alternative remote accessapproaches Terminal servers and virtualdesktop infrastructure (VDI) Understanding terminal services andVDI conceptually Summary 33333533633733834134134411TroubleshootingThe high cost of disasteravoidance Sources of solutions There is no magic support Visualizing what IT handles and whatengineering handles IT vendor managements Triage skills and staff I can give status, or I can fix things 348348350353354355356IndexOther Books You May EnjoyStaffing for triage: The perceiver Logical approaches totroubleshooting Stories of troubleshooting Technical social media in problemsolving Investigating versus fixing Summary The postmortem 360362363365366369370

PrefaceLinux Administration Best Practices is a guide for understanding the context, decisionmaking, and ideologies behind one of the most critical functions in businessinfrastructure: systems. Systems, that is, operating system level management, remainsthe cornerstone of communications and infrastructure. Linux remains the most popularoperating system family of choice today and is only gaining more and more traction,making the need for well-trained, deeply knowledgable Linux administration teams thatmuch more important.Who this book is forThis book is intended for those IT professionals working with Linux or as systemadministrators who want to take their craft to the next level. This book is about bestpractices and so we approach the role and thinking of the system administrator ratherthan learning individual tasks. This book will challenge how you think and how youapproach system administration. This book will not teach you about the tasks of systemadministration, but it will teach you how to think like a career administrator.What this book coversChapter 1, What Is the Role of a System Administrator?, explains the actual role andmandate of the system administration function and how to apply this to your ownrole in your career and your organization.Chapter 2, Choosing Your Distribution and Release Model, goes through how to choosethe right Linux variation for your organization (Linux comes in many flavors and styles),understanding the importance of release models.Chapter 3, System Storage Best Practices, attempts to take you from newbie to masterregarding storage, which remains one of the least understood areas of systemadministration, taking a high-level approach.

xivPrefaceChapter 4, Designing System Deployment Architectures, breaks down assessing deploymentapproaches and when different models will work for you.Systems do not exist in a vacuum; they are deployed in conjunction with other systems,often needing to interoperate for functionality or redundancy. Combining systems inmeaningful ways is complex and can be counterintuitive.Chapter 5, Patch Management Strategies, looks at patching and updates, which mightsound mundane but are at the core of any system task list, and are often more complexthan is realized. Good patch management will help protect you and your organizationfrom disasters both accidental and malicious.Chapter 6, Databases, digs into database concepts and how they apply to the systemsrealm so that you can provide better support and guidance to your database users.Technically not part of the operating system, database management has historicallyfallen to system administrators.Chapter 7, Documentation, Monitoring, and Logging Techniques, looks at strategies forboth manual and automated processes for tracking the desired state and current state ofour systems.Chapter 8, Improving Administration Maturation with Automation through Scriptingand DevOps, looks at many different ways to approach automation considering practicalityand real-world needs in addition to perfect, theoretical approaches. Everyone talks aboutthe automation of system tasks, but many organizations fail to do it.Chapter 9, Backup and Disaster Recovery Approaches, goes far beyond conventionalwisdom and approaches disaster recovery with a fresh eye and modernism to provideways to make backups easier and more effective than they normally are. The single mostimportant task in system administration is protecting data.Chapter 10, User and Access Management Strategies, looks at best practices for maintainingusers, discusses decision making and user management approaches, and investigatesarchitectures for remote access to the operating system. What good is a system if no onecan access it?Chapter 11, Troubleshooting, looks at how taking a planned, intentional approach totroubleshooting improves resolution speed, reduces stress, and might just find problemsthat would have been kept hidden otherwise. Nothing is harder than figuring out whatto do when something is wrong and the pressure is on.

PrefacexvTo get the most out of this bookYou are expected to have general knowledge of Linux and operating systems. We assumeexperience in working with systems in a production environment and a business setting.This book covers high-level concepts rather than technical processes and so a workingknowledge of Linux and operating systems is beneficial.The software covered in this book are Linux-based operating systems such as Ubuntu,Fedora, Red Hat Enterprise Linux, and Suse.No running system is necessary to use this book. This book focuses on high-level conceptsand while knowledge of Linux and operating systems is very useful, there is no need to behands-on with a running system while reading this book.Download the color imagesWe also provide a PDF file that has color images of the screenshots/diagrams usedin this book. You can download it here: 92 ColorImages.pdf.Conventions usedThere are a number of text conventions used throughout this book.Code in text: Indicates code words in text, database table names, folder names, filenames,file extensions, pathnames, dummy URLs, user input, and Twitter handles. Here is anexample: "System is a reference to operating system and designates the scope of the role:managing the platform on which applications run."Any command-line input or output is written as follows: lvcreate -l 100%FREE -n lv data vg1Bold: Indicates a new term, an important word, or words that you see onscreen. Forexample, words in menus or dialog boxes appear in the text like this. Here is an example:"As we progress through our exploration of Linux System Administration, the idea of hatsand really digging into job roles and functions will become more and more clear."Tips or Important NotesAppear like this.

xviPrefaceGet in touchFeedback from our readers is always welcome.General feedback: If you have questions about any aspect of this book, mention the booktitle in the subject of your message and email us at customercare@packtpub.com.Errata: Although we have taken every care to ensure the accuracy of our content, mistakesdo happen. If you have found a mistake in this book, we would be grateful if you wouldreport this to us. Please visit www.packtpub.com/support/errata, selecting yourbook, clicking on the Errata Submission Form link, and entering the details.Piracy: If you come across any illegal copies of our works in any form on the Internet,we would be grateful if you would provide us with the location address or website name.Please contact us at copyright@packt.com with a link to the material.If you are interested in becoming an author: If there is a topic that you have expertisein and you are interested in either writing or contributing to a book, please visitauthors.packtpub.com.Share Your ThoughtsOnce you've read Linux Administration Best Practices, we'd love to hear your thoughts!Please click here to go straight to the Amazon review page for this book and shareyour feedback.Your review is important to us and the tech community and will help us make sure we'redelivering excellent quality content.

Section 1:Understanding theRole of Linux SystemAdministratorThe objective of Section 1 is to help you to comprehend the scope, responsibilities,role, and mandate of the System Administrator function. We take the reader past theconcept of tasks and really attempt to dig into the purpose and value of the role ata much deeper level.This part of the book comprises the following chapters: Chapter 1, What Is the Role of a System Administrator? Chapter 2, Choosing Your Distribution and Release Model

1What Is the Roleof a SystemAdministrator?Few things in our industry sound like they should be simpler to answer than this one,simple question: what is a system administrator? And yet, ask anyone and you'll get somewidely differing opinions. Everyone seems to have their own take on what the title or roleof System Administrator implies, including and possibly most varying in people who usethis title for themselves or from the companies that hand it out!Welcome to system administration and specifically Best Practices of Linux Administration.In this chapter we are going to dive into understanding the job, role, and functions of a realsystem administrator and try to understand how we, in that role, fit into an organization.In tackling this book, it is necessary both for myself to have some semblance of a clearcourse in writing, but also for you to understand if this book is for you, or to grasp thescope that I am attempting to cover, for me to clearly define what a system administratoris to me.Understanding exactly what is expected of a true system administrator will be thefoundation for applying that definition of the role to the upcoming best practices thatapply both to system administration generally and specifically to Linux administration.

4What Is the Role of a System Administrator?In this chapter we are going to cover the following main topics: Where are system administrators in the real world Wearing the administrator and engineering hats Understanding systems in the business ecosystem Learning system administration Introducing the IT professionalWhere are system administrators in thereal world?I think that one of the most challenging things about attempting to understand whata system administrator is comes from the fact that the title of system administrator isoften given out, willy nilly, by companies with little to no understanding of InformationTechnology (IT), systems, or administration and treat it like a general filler for IT rolesthat they do not understand or know how to name. It also has a strong tendency tobe given out in lieu of pay raises or promotions to entice junior staff to remain in anotherwise unrewarding job in the hopes that an impressive title will help them laterin their career, so much so that in the end, the number of people working as systemadministrators is a very small number of people compared to the number of people withthe title. In fact, it is no small stretch to guess that the average person with the title ofsystem administrator has never thought about the meaning of the title and may have littleinkling of what someone in that role would be expected to do.If we look solely by title, system administrators are everywhere. But they exist mostly atcompanies too small to have plausibly employed a system administrator at all. Systemsadministration as a dedicated job is nearly exclusive to large companies. Most companiesneed someone to do the tasks of system administration, but only as a part of, and oftenonly a small part of, their overall duties. It is the nature of IT that in small and mediumsized companies you typically have generalists who wear many hats and do every neededIT role while having little to no time to focus on any one specific function. Whereas inlarge enterprises you generally get focused roles, often grouped into focused departments,that do just a single IT role: such as system administrator. But even in some enterprisesyou find departments organized like separate, small businesses and still having generalistsdoing many different tasks rather than separating out duties to lots of different people.

Where are system administrators in the real world?5There is nothing wrong with this, of course. It is totally expected. It's much like how,as a homeowner, you will often do a lot of work on the house yourself, or you mighthire a handyman who can do pretty much whatever is needed. You might need someplumbing, painting, carpentry, wiring, or whatever done. Whether you do it yourself,or your handyman does, you do not refer to either of yourselves as plumbers, painters,carpenters, and others. You are just a handy person, or the person that you hired is. Youstill recognize that a dedicated, full time, focused plumber, painter, carpenter, or electricianis a specialized role. You might do all those tasks occasionally, you might even be good atit, but it's not the same as if that was your full-time career. If you decided to claim to bethese things to your friends, they would quickly call you out on the fact that you are quiteobviously not those things.System administrators are like plumbers. Everyone who owns a house does at least a littleplumbing. A handyman who does home maintenance full time might do a fair amount ofplumbing. But neither is a plumber. A very large housing development, or a constructioncrew might have a dedicated plumber on staff. Maybe even more than one. And nearlyevery homeowner must engage one from time to time. If you are me, regularly. Mostplumbers either work for large companies that have need of continuous plumbing servicesor they work for plumbing contracting firms and have the benefits of peers and mentorsto help them advance in their knowledge.Nearly every business no matter what size we are talking about needs systemadministration tasks done. For very small businesses it is not uncommon for these tasksto amount to no more than a few hours per year, and when needed the scheduling is oftenunpredictable with many hours needed all at once and large gaps of time in which nothingis needed. In large businesses, you might need tens of thousands of hours of systemadministration tasks per week and require entire departments of dedicated specialists.So just like plumbers, you find small businesses either hiring IT generalists (akin to thehomeowner's handyman) or outsourcing system administration tasks to an outsidefirm like an Managed Service Provider (commonly referred to as an MSP) or keepinga consultant on retainer; and you will find large companies typically hiring full timespecialist system administrators that do nothing else and work only for that firm.System administration tasks exist in every business, in every industry and create thefoundation of what I feel is one of the most rewarding roles within the IT field. Withsystem administration skills you can chart your own course to work in a large firm, be aconsultant, join a service provider, or enhance other skills to make yourself a better andmore advanced specialist. Without a firm foundation in system administration a generalistwill lack one of the most core skills and have little ability to advance even in the generalistranks. And at the top of the generalist field, true CIO roles primarily pull from those withextensive system administration comprehension.

6What Is the Role of a System Administrator?At this point we know what a system administrator is, where you will find them inbusinesses, and why you might want to pursue system administration either as yourcareer focus or as an enhancement to a career as a generalist. Now we can go into realdetail about what a system administrator really does!Wearing the administrator and engineeringhatsIn this section, we will explore two parts: How does administration and engineering differ. How to identify the role you are performing.The name system administrator itself should clue us in as to what the role should entail.The title is not meant to be confusing or obfuscated. Yet many people believe that it issome kind of trick. If you spend enough time working in the small business arena youmight even find that many people, people who are full time IT professionals, may noteven believe that true system administrators exist!System is a reference to operating system and designates the scope of the role: managingthe platform on which applications run. This differentiates the system administrator rolefrom, say, a database administrator (commonly called a DBA) who manages a databaseitself (which runs on top of an operating system), or an application administrator (whomanages specific applications on top of an operating system), or a platform administrator(who manages the hypervisor on which the operating system runs), or a networkadministrator (who manages the network itself.) Being called a system administratorshould imply that the focus primarily or nearly entirely of the person or role is centeredaround the care and feeding of operating systems. If your day isn't all about an operatingsystem, you aren't really a system administrator. Maybe system administration is part ofyour duties but being a system administrator is not the right title for you.Administrator tells us that this role is one that manages something. The direct alternativeto an administrator is an Engineer. An engineer plans and designs something; anadministrator runs and maintains something. I often refer to these roles as the A&E rolesand often the titles are used loosely and meaninglessly based on how the speaker thinksthat it will sound. But, when used accurately, they have very definite meanings and in eacharea within IT (systems, platforms, network, databases, applications, and others.) you haveboth working in concert with one another. Of course, it is exceptionally common to haveone human acting as both an engineer and an administrator within an organization, theroles have extensive overlap in skills and knowledge and necessarily must work in greatcooperation to be able to do what they do effectively.

Wearing the administrator and engineering hats7The difference between the role of an administratorand the role of an engineerThere is a key difference between the two roles, however, that impacts organizationsand practitioners in a very meaningful way, which is very important to discuss becauseotherwise, we are tempted to feel that separating the two roles is nothing more thanpedantic or a game of semantics. This difference is in how we measure performanceor success.An engineering role is measured by throughput or the total q

Scott Alan Miller is an information technology and software engineering industry veteran of 30 years, with more than a quarter of a century on UNIX and Linux. His experience has included companies of every size, in every region of the world, in nearly every industry. Scott has been a technician, lead, manager, educator, consultant, writer,