Transcription
The definiTive guide To Linuxand unix sysTem Programming fffRead and write files efficientlyUse signals, clocks, and timersCreate processes and execute programsfffffWrite secure programsWrite multithreaded programs using POSIX threadsBuild and use shared librariesPerform interprocess communication using pipes,message queues, shared memory, and semaphoresWrite network applications with the sockets APIWhile The Linux Programming Interface covers a wealthof Linux-specific features, including epoll, inotify, andthe /proc file system, its emphasis on UNIX standards(POSIX.1-2001/SUSv3 and POSIX.1-2008/SUSv4)makes it equally valuable to programmers working onother UNIX platforms.The Linux Programming Interface is the most comprehensive single-volume work on the Linux and UNIXprogramming interface, and a book that’s destined tobecome a new classic.about the authorMichael Kerrisk (http://man7.org/) has been using and programming UNIX systemsfor more than 20 years, and has taught many week-long courses on UNIX systemprogramming. Since 2004, he has maintained the man-pages project, whichproduces the manual pages describing the Linux kernel and glibc programmingAPIs. He has written or cowritten more than 250 of the manual pages and is activelyinvolved in the testing and design review of new Linux kernel-userspace interfaces.Michael lives with his family in Munich, Germany.covers current uNiX standards (PosiX.1-2001/susv3 and PosiX.1-2008/susv4)T H E F I N E ST I N G E E K E N T E RTA I N M E N T w w w.nostarch.com 99.95 ( 114.95 CDN )KerrisKShelve In: linux/programmingISBN: 978-1-59327-220-35 999 59 781593 272203This logo applies only to the text stock.689145 72200The LinuxProgramminginTerfaceThe Linux Programming Interface is the definitive guideto the Linux and UNIX programming interface—theinterface employed by nearly every application thatruns on a Linux or UNIX system.In this authoritative work, Linux programmingexpert Michael Kerrisk provides detailed descriptionsof the system calls and library functions that you needin order to master the craft of system programming,and accompanies his explanations with clear, completeexample programs.You’ll find descriptions of over 500 system callsand library functions, and more than 200 example programs, 88 tables, and 115 diagrams. You’ll learn how to:0The LinuxProgramminginTerfaceA Linux and UNIX System Programming Handbook Michael KerrisK
PRAISE FOR THE LINUX PROGRAMMING INTERFACE“If I had to choose a single book to sit next to my machine when writingsoftware for Linux, this would be it.”—MARTIN LANDERS, SOFTWARE ENGINEER, GOOGLE“This book, with its detailed descriptions and examples, contains everythingyou need to understand the details and nuances of the low-level programmingAPIs in Linux . . . no matter what the level of reader, there will be somethingto be learnt from this book.”—MEL GORMAN, AUTHOR OF Understanding the Linux Virtual Memory Manager“Michael Kerrisk has not only written a great book about Linux programmingand how it relates to various standards, but has also taken care that bugs henoticed got fixed and the man pages were (greatly) improved. In all threeways, he has made Linux programming easier. The in-depth treatment oftopics in The Linux Programming Interface . . . makes it a must-have referencefor both new and experienced Linux programmers.”—ANDREAS JAEGER, PROGRAM MANAGER, OPENSUSE, NOVELL“Michael’s inexhaustible determination to get his information right, and toexpress it clearly and concisely, has resulted in a strong reference source forprogrammers. While this work is targeted at Linux programmers, it will be ofvalue to any programmer working in the UNIX/POSIX ecosystem.”—DAVID BUTENHOF, AUTHOR OF Programming with POSIX Threads ANDCONTRIBUTOR TO THE POSIX AND UNIX STANDARDS“ . . . a very thorough—yet easy to read—explanation of UNIX system andnetwork programming, with an emphasis on Linux systems. It’s certainly abook I’d recommend to anybody wanting to get into UNIX programming(in general) or to experienced UNIX programmers wanting to know ‘what’snew’ in the popular GNU/Linux system.”—FERNANDO GONT, NETWORK SECURITY RESEARCHER, IETF PARTICIPANT, ANDRFC AUTHOR
“ . . . encyclopedic in the breadth and depth of its coverage, and textbooklike in its wealth of worked examples and exercises. Each topic is clearlyand comprehensively covered, from theory to hands-on working code.Professionals, students, educators, this is the Linux/UNIX reference thatyou have been waiting for.”—ANTHONY ROBINS, ASSOCIATE PROFESSOR OF COMPUTER SCIENCE, THEUNIVERSITY OF OTAGO“I’ve been very impressed by the precision, the quality and the level of detailMichael Kerrisk put in his book. He is a great expert of Linux system callsand lets us share his knowledge and understanding of the Linux APIs.”—CHRISTOPHE BLAESS, AUTHOR OF Programmation système en C sous Linux“ . . . an essential resource for the serious or professional Linux and UNIXsystems programmer. Michael Kerrisk covers the use of all the key APIsacross both the Linux and UNIX system interfaces with clear descriptionsand tutorial examples and stresses the importance and benefits of followingstandards such as the Single UNIX Specification and POSIX 1003.1.”—ANDREW JOSEY, DIRECTOR, STANDARDS, THE OPEN GROUP, AND CHAIR OFTHE POSIX 1003.1 WORKING GROUP“What could be better than an encyclopedic reference to the Linux system,from the standpoint of the system programmer, written by none other thanthe maintainer of the man pages himself? The Linux Programming Interface iscomprehensive and detailed. I firmly expect it to become an indispensableaddition to my programming bookshelf.”—BILL GALLMEISTER, AUTHOR OF POSIX.4 Programmer’s Guide: Programming forthe Real World“ . . . the most complete and up-to-date book about Linux and UNIX systemprogramming. If you’re new to Linux system programming, if you’re a UNIXveteran focused on portability while interested in learning the Linux way,or if you’re simply looking for an excellent reference about the Linux programming interface, then Michael Kerrisk’s book is definitely the companionyou want on your bookshelf.”—LOÏC DOMAIGNÉ, CHIEF SOFTWARE ARCHITECT (EMBEDDED), CORPULS.COM
San Francisco
THE LINUX PROGRAMMING INTERFACE. Copyright 2010 by Michael Kerrisk.All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means, electronic ormechanical, including photocopying, recording, or by any information storage or retrieval system, without the priorwritten permission of the copyright owner and the publisher.14 13 12 11 10123456789ISBN-10: 1-59327-220-0ISBN-13: 978-1-59327-220-3Publisher: William PollockProduction Editor: Riley HoffmanCover Design: Octopod StudiosFront Cover Photo: Rob SuistedBack Cover Photo: Lynley CookCopyeditor: Marilyn SmithCompositor: Susan Glinert StevensProofreader: Linda SeifertFor technical reviewers, please refer to the author’s acknowledgementsFor information on book distributors or translations, please contact No Starch Press, Inc. directly:No Starch Press, Inc.38 Ringold Street, San Francisco, CA 94103phone: 415.863.9900; fax: 415.863.9950; info@nostarch.com; www.nostarch.comLibrary of Congress Cataloging-in-Publication DataKerrisk, Michael, 1961The Linux programming interface : a Linux and UNIX system programming handbook / by Michael Kerrisk.p. cm.Includes bibliographical references and index.ISBN-13: 978-1-59327-220-3 (hardcover)ISBN-10: 1-59327-220-0 (hardcover)1. Linux. 2. UNIX (Computer file) 3. Operating systems (Computers) I. Title.QA76.76.O63K496 2010005.4'32--dc222010001947No Starch Press and the No Starch Press logo are registered trademarks of No Starch Press, Inc. Other product andcompany names mentioned herein may be the trademarks of their respective owners. Rather than use a trademarksymbol with every occurrence of a trademarked name, we are using the names only in an editorial fashion and to thebenefit of the trademark owner, with no intention of infringement of the trademark. UNIX is a registered trademark of The Open Group.The information in this book is distributed on an “As Is” basis, without warranty. While every precaution has beentaken in the preparation of this work, neither the author nor No Starch Press, Inc. shall have any liability to anyperson or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by theinformation contained in it.This logo applies only to the text stock.
For Cecilia, who lights up my world.
BRIEF CONTENTSPreface . xxxiChapter 1: History and Standards . 1Chapter 2: Fundamental Concepts . 21Chapter 3: System Programming Concepts. 43Chapter 4: File I/O: The Universal I/O Model. 69Chapter 5: File I/O: Further Details . 89Chapter 6: Processes . 113Chapter 7: Memory Allocation. 139Chapter 8: Users and Groups . 153Chapter 9: Process Credentials . 167Chapter 10: Time. 185Chapter 11: System Limits and Options. 211Chapter 12: System and Process Information . 223Chapter 13: File I/O Buffering. 233Chapter 14: File Systems . 251Chapter 15: File Attributes . 279Chapter 16: Extended Attributes . 311Chapter 17: Access Control Lists . 319Chapter 18: Directories and Links . 339Chapter 19: Monitoring File Events . 375
Chapter 20: Signals: Fundamental Concepts . 387Chapter 21: Signals: Signal Handlers. 421Chapter 22: Signals: Advanced Features . 447Chapter 23: Timers and Sleeping. 479Chapter 24: Process Creation. 513Chapter 25: Process Termination. 531Chapter 26: Monitoring Child Processes . 541Chapter 27: Program Execution . 563Chapter 28: Process Creation and Program Execution in More Detail. 591Chapter 29: Threads: Introduction. 617Chapter 30: Threads: Thread Synchronization . 631Chapter 31: Threads: Thread Safety and Per-Thread Storage. 655Chapter 32: Threads: Thread Cancellation. 671Chapter 33: Threads: Further Details . 681Chapter 34: Process Groups, Sessions, and Job Control . 699Chapter 35: Process Priorities and Scheduling . 733Chapter 36: Process Resources . 753Chapter 37: Daemons . 767Chapter 38: Writing Secure Privileged Programs . 783Chapter 39: Capabilities . 797Chapter 40: Login Accounting . 817Chapter 41: Fundamentals of Shared Libraries . 833Chapter 42: Advanced Features of Shared Libraries. 859Chapter 43: Interprocess Communication Overview . 877Chapter 44: Pipes and FIFOs. 889Chapter 45: Introduction to System V IPC. 921Chapter 46: System V Message Queues . 937viiiBrief Contents
Chapter 47: System V Semaphores . 965Chapter 48: System V Shared Memory . 997Chapter 49: Memory Mappings . 1017Chapter 50: Virtual Memory Operations . 1045Chapter 51: Introduction to POSIX IPC. 1057Chapter 52: POSIX Message Queues . 1063Chapter 53: POSIX Semaphores . 1089Chapter 54: POSIX Shared Memory. 1107Chapter 55: File Locking . 1117Chapter 56: Sockets: Introduction . 1149Chapter 57: Sockets: UNIX Domain . 1165Chapter 58: Sockets: Fundamentals of TCP/IP Networks. 1179Chapter 59: Sockets: Internet Domains . 1197Chapter 60: Sockets: Server Design . 1239Chapter 61: Sockets: Advanced Topics. 1253Chapter 62: Terminals . 1289Chapter 63: Alternative I/O Models. 1325Chapter 64: Pseudoterminals. 1375Appendix A: Tracing System Calls . 1401Appendix B: Parsing Command-Line Options. 1405Appendix C: Casting the NULL Pointer . 1413Appendix D: Kernel Configuration . 1417Appendix E: Further Sources of Information . 1419Appendix F: Solutions to Selected Exercises. 1425Bibliography. 1437Index .1447B r i e f C on t e n t six
CONTENTS IN DETAILPREFA CExxxi1HISTORY AND STANDARDS1.11.2A Brief History of UNIX and C .2A Brief History of Linux .51.2.1The GNU Project .51.2.2The Linux Kernel .6Standardization . 101.3.1The C Programming Language . 101.3.2The First POSIX Standards. 111.3.3X/Open Company and The Open Group . 131.3.4SUSv3 and POSIX.1-2001 . 131.3.5SUSv4 and POSIX.1-2008 . 151.3.6UNIX Standards Timeline . 161.3.7Implementation Standards. 171.3.8Linux, Standards, and the Linux Standard Base. 18Summary. 191.31.412F UNDAMENTAL C ONCEP .152.162.172.182.192.20The Core Operating System: The Kernel .The Shell .Users and Groups .Single Directory Hierarchy, Directories, Links, and Files .File I/O Model .Programs.Processes .Memory Mappings .Static and Shared Libraries.Interprocess Communication and Synchronization .Signals.Threads.Process Groups and Shell Job Control.Sessions, Controlling Terminals, and Controlling Processes .Pseudoterminals .Date and Time .Client-Server Architecture .Realtime .The /proc File System.Summary.213S YSTEM PROGRAMM ING CO NCEP TS3.13.23.33.43.5System Calls .Library Functions .The Standard C Library; The GNU C Library ( glibc) .Handling Errors from System Calls and Library Functions.Notes on the Example Programs in This Book.3.5.1Command-Line Options and Arguments .3.5.2Common Functions and Header 464748505051
3.63.73.8Portability Issues.3.6.1Feature Test Macros .3.6.2System Data Types .3.6.3Miscellaneous Portability Issues.Summary.Exercise .4F ILE I/O: THE UN IVERSA L I/O MODEL4.14.24.3Overview .Universality of I/O .Opening a File: open() .4.3.1The open() flags Argument.4.3.2Errors from open() .4.3.3The creat() System Call .Reading from a File: read() .Writing to a File: write().Closing a File: close() .Changing the File Offset: lseek() .Operations Outside the Universal I/O Model: ioctl() .Summary.Exercises 98080818686875F ILE I/O: FURTHER 5.14Atomicity and Race Conditions . 90File Control Operations: fcntl() . 92Open File Status Flags . 93Relationship Between File Descriptors and Open Files . 94Duplicating File Descriptors . 96File I/O at a Specified Offset: pread() and pwrite() . 98Scatter-Gather I/O: readv() and writev() . 99Truncating a File: truncate() and ftruncate() . 103Nonblocking I/O . 103I/O on Large Files . 104The /dev/fd Directory . 107Creating Temporary Files . 108Summary. 109Exercises . esses and Programs.Process ID and Parent Process ID.Memory Layout of a Process .Virtual Memory Management .The Stack and Stack Frames .Command-Line Arguments (argc, argv) .Environment List .Performing a Nonlocal Goto: setjmp() and longjmp() .Summary.Exercises .xiiC on t e n t s i n D e t a i l11 3113114115118121122125131138138
7ME MORY ALLO CATIO N7.17.27.37.4Allocating Memory on the Heap .7.1.1Adjusting the Program Break: brk() and sbrk() .7.1.2Allocating Memory on the Heap: malloc() and free() .7.1.3Implementation of malloc() and free() .7.1.4Other Methods of Allocating Memory on the Heap .Allocating Memory on the Stack: alloca() .Summary.Exercises .8U S ER S AN D GR OU PS8.18.28.38.48.58.68.7The Password File: /etc/passwd .The Shadow Password File: /etc/shadow .The Group File: /etc/group .Retrieving User and Group Information .Password Encryption and User Authentication .Summary.Exercises .9PROCESS CREDENTIALS9.19.29.39.49.59.69.79.89.9Real User ID and Real Group ID.Effective User ID and Effective Group ID.Set-User-ID and Set-Group-ID Programs .Saved Set-User-ID and Saved Set-Group-ID .File-System User ID and File-System Group ID.Supplementary Group IDs .Retrieving and Modifying Process Credentials.9.7.1Retrieving and Modifying Real, Effective, and Saved Set IDs .9.7.2Retrieving and Modifying File-System IDs .
The Linux Programming Interface is the definitive guide to the Linux and UNIX programming interface—the interface employed by nearly every application that runs on a Linux or UNIX syste
