Transcription
Download from Wow! eBook www.wowebook.com Classic Shell ScriptingArnold Robbins and Nelson H. F. BeebeBeijing Cambridge Farnham Köln Sebastopol Tokyo
Classic Shell Scriptingby Arnold Robbins and Nelson H. F. BeebeCopyright 2005 O’Reilly Media, Inc. All rights reserved.Printed in the United States of America.Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.O’Reilly books may be purchased for educational, business, or sales promotional use. Online editionsare also available for most titles (safari.oreilly.com). For more information, contact our corporate/institutional sales department: (800) 998-9938 or corporate@oreilly.com.Editors:Tatiana ApandiAllison RandalProduction Editor:Adam WitwerCover Designer:Emma ColbyInterior Designer:David FutatoPrinting History:May 2005:First Edition.Nutshell Handbook, the Nutshell Handbook logo, and the O’Reilly logo are registered trademarks ofO’Reilly Media, Inc. Classic Shell Scripting, the image of a African tent tortoise, and related trade dressare trademarks of O’Reilly Media, Inc.Many of the designations used by manufacturers and sellers to distinguish their products are claimed astrademarks. Where those designations appear in this book, and O’Reilly Media, Inc. was aware of atrademark claim, the designations have been printed in caps or initial caps.While every precaution has been taken in the preparation of this book, the publisher and authorsassume no responsibility for errors or omissions, or for damages resulting from the use of theinformation contained herein.ISBN: 978-0-596-00595-5[LSI][2011-03-11]
Table of ContentsForeword . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ixPreface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi1. Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11.1 Unix History1.2 Software Tools Principles1.3 Summary1462. Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82.12.22.32.42.52.62.72.82.9Scripting Languages Versus Compiled LanguagesWhy Use a Shell Script?A Simple ScriptSelf-Contained Scripts: The #! First LineBasic Shell ConstructsAccessing Shell Script ArgumentsSimple Execution TracingInternationalization and LocalizationSummary8991012232425283. Searching and Substitutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303.13.23.33.4Searching for TextRegular ExpressionsWorking with FieldsSummary30315665iii
4. Text Processing Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 674.14.24.34.44.54.64.7Sorting TextRemoving DuplicatesReformatting ParagraphsCounting Lines, Words, and CharactersPrintingExtracting the First and Last LinesSummary677576777883865. Pipelines Can Do Amazing Things . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 875.15.25.35.45.55.6Extracting Data from Structured Text FilesStructured Data for the WebCheating at Word PuzzlesWord ListsTag ListsSummary87941001021051076. Variables, Making Decisions, and Repeating Actions . . . . . . . . . . . . . . . . . . 1096.16.26.36.46.56.6Variables and ArithmeticExit StatusesThe case 7. Input and Output, Files, and Command Evaluation . . . . . . . . . . . . . . . . . . . . 1407.17.27.37.47.57.67.77.87.97.10iv Standard Input, Output, and ErrorReading Lines with readMore About RedirectionsThe Full Story on printfTilde Expansion and WildcardsCommand SubstitutionQuotingEvaluation Order and evalBuilt-in CommandsSummaryTable of Contents140140143147152155161162168175
8. Production Scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1778.1 Path Searching8.2 Automating Software Builds8.3 Summary1771922229. Enough awk to Be Dangerous . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2239.19.29.39.49.59.69.79.89.99.109.11The awk Command LineThe awk Programming ModelProgram ElementsRecords and FieldsPatterns and ActionsOne-Line Programs in awkStatementsUser-Defined FunctionsString FunctionsNumeric 0. Working with Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26710.110.210.310.410.510.610.710.8Listing FilesUpdating Modification Times with touchCreating and Using Temporary FilesFinding FilesRunning Commands: xargsFilesystem Space InformationComparing FilesSummary26727327427929329529930711. Extended Example: Merging User Databases . . . . . . . . . . . . . . . . . . . . . . . . . 30811.111.211.311.411.511.6The ProblemThe Password FilesMerging Password FilesChanging File OwnershipOther Real-World IssuesSummary308309310317321323Table of Contents v
12. Spellchecking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32512.112.212.312.412.5The spell ProgramThe Original Unix Spellchecking PrototypeImproving ispell and aspellA Spellchecker in awkSummary32532632733135013. Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35213.113.213.313.413.513.613.713.8Process CreationProcess ListingProcess Control and DeletionProcess System-Call TracingProcess AccountingDelayed Scheduling of ProcessesThe /proc FilesystemSummary35335436036837237337837914. Shell Portability Issues and Extensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38114.114.214.314.414.514.614.714.8GotchasThe bash shopt CommandCommon ExtensionsDownload InformationOther Extended Bourne-Style ShellsShell VersionsShell Initialization and TerminationSummary38138538940240540540641215. Secure Shell Scripts: Getting Started . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41315.115.215.315.415.515.6vi Tips for Secure Shell ScriptsRestricted ShellTrojan HorsesSetuid Shell Scripts: A Bad Ideaksh93 and Privileged ModeSummaryTable of Contents413416418419421422
A. Writing Manual Pages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 423B. Files and Filesystems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 437C. Important Unix Commands . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 473Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 478Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 484Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 509Table of Contents vii
ForewordSurely I haven’t been doing shell scripting for 30 years?!? Well, now that I thinkabout it, I suppose I have, although it was only in a small way at first. (The earlyUnix shells, before the Bourne shell, were very primitive by modern standards, andwriting substantial scripts was difficult. Fortunately, things quickly got better.)In recent years, the shell has been neglected and underappreciated as a scripting language. But even though it was Unix’s first scripting language, it’s still one of the best.Its combination of extensibility and efficiency remains unique, and the improvements made to it over the years have kept it highly competitive with other scriptinglanguages that have gotten a lot more hype. GUIs are more fashionable than command-line shells as user interfaces these days, but scripting languages often providemost of the underpinnings for the fancy screen graphics, and the shell continues toexcel in that role.The shell’s dependence on other programs to do most of the work is arguably adefect, but also inarguably a strength: you get the concise notation of a scripting language plus the speed and efficiency of programs written in C (etc.). Using a common, general-purpose data representation—lines of text—in a large (and extensible)set of tools lets the scripting language plug the tools together in endless combinations. The result is far more flexibility and power than any monolithic software package with a built-in menu item for (supposedly) everything you might want. The earlysuccess of the shell in taking this approach reinforced the developing Unix philosophy of building specialized, single-purpose tools and plugging them together to dothe job. The philosophy in turn encouraged improvements in the shell to allow doingmore jobs that way.Shell scripts also have an advantage over C programs—and over some of the otherscripting languages too (naming no names!)—of generally being fairly easy to readand modify. Even people who are not C programmers, like a good many systemadministrators these days, typically feel comfortable with shell scripts. This makesshell scripting very important for extending user environments and for customizingsoftware packages.ixThis is the Title of the Book, eMatter Edition
Indeed, there’s a “wheel of reincarnation” here, which I’ve seen on several softwareprojects. The project puts simple shell scripts in key places, to make it easy for usersto customize aspects of the software. However, it’s so much easier for the project tosolve problems by working in those shell scripts than in the surrounding C code, thatthe scripts steadily get more complicated. Eventually they are too complicated for theusers to cope with easily (some of the scripts we wrote in the C News project werenotorious as stress tests for shells, never mind users!), and a new set of scripts has tobe provided for user customization For a long time, there’s been a conspicuous lack of a good book on shell scripting.Books on the Unix programming environment have touched on it, but only briefly,as one of several topics, and the better books are long out-of-date. There’s referencedocumentation for the various shells, but what’s wanted is a novice-friendly tutorial,covering the tools as well as the shell, introducing the concepts gently, offeringadvice on how to get the best results, and paying attention to practical issues likereadability. Preferably, it should also discuss how the various shells differ, instead oftrying to pretend that only one exists.This book delivers all that, and more. Here, at last, is an up-to-date and painlessintroduction to the first and best of the Unix scripting languages. It’s illustrated withrealistic examples that make useful tools in their own right. It covers the standardUnix tools well enough to get people started with them (and to make a useful reference for those who find the manual pages a bit forbidding). I’m particularly pleasedto see it including basic coverage of awk, a highly useful and unfairly neglected toolwhich excels in bridging gaps between other tools and in doing small programmingjobs easily and concisely.I recommend this book to anyone doing shell scripting or administering Unixderived systems. I learned things from it; I think you will too.—Henry SpencerSP Systemsx ForewordThis is the Title of the Book, eMatter Edition
PrefaceThe user or programmer new to Unix* is suddenly faced with a bewildering variety ofprograms, each of which often has multiple options. Questions such as “What purpose do they serve?” and “How do I use them?” spring to mind.This book’s job is to answer those questions. It teaches you how to combine theUnix tools, together with the standard shell, to get your job done. This is the art ofshell scripting. Shell scripting requires not just a knowledge of the shell language, butalso a knowledge of the individual Unix programs: why each one is there, and howto use them by themselves and in combination with the other programs.Why should you learn shell scripting? Because often, medium-size to large problemscan be decomposed into smaller pieces, each of which is amenable to being solvedwith one of the Unix tools. A shell script, when done well, can often solve a problemin a mere fraction of the time it would take to solve the same problem using a conventional programming language such as C or C . It is also possible to make shellscripts portable—i.e., usable across a range of Unix and POSIX-compliant systems,with little or no modification.When talking about Unix programs, we use the term tools deliberately. The Unixtoolbox approach to problem solving has long been known as the “Software Tools”philosophy.†A long-standing analogy summarizes this approach to problem solving. A SwissArmy knife is a useful thing to carry around in one’s pocket. It has several blades, ascrewdriver, a can opener, a toothpick, and so on. Larger models include more tools,such as a corkscrew or magnifying glass. However, there’s only so much you can dowith a Swiss Army knife. While it might be great for whittling or simple carving, you* Throughout this book, we use the term Unix to mean not only commercial variants of the original Unix system, such as Solaris, Mac OS X, and HP-UX, but also the freely available workalike systems, such as GNU/Linux and the various BSD systems: BSD/OS, NetBSD, FreeBSD, and OpenBSD.† This approach was popularized by the book Software Tools (Addison-Wesley).xiThis is the Title of the Book, eMatter EditionCopyright 2011 O’Reilly & Associates, Inc. All rights reserved.
wouldn’t use it, for example, to build a dog house or bird feeder. Instead, you wouldmove on to using specialized tools, such as a hammer, saw, clamp, or planer. So too,when solving programming problems, it’s better to use specialized software tools.Intended AudienceThis book is intended for computer users and software develo
This is the Title of the Book, eMatter Edition ix Foreword Surely I haven’t been doing shell scripting for 30 years?!? Well,now that I think about it,I suppose I have,although it was only in a small way at first. (The early Unix shells,before the Bourne shell,were very primitive
