WIXLIB

CS110 Lecture 9: Pipes and InterprocessCommunication, Part 2CS110: Principles of Computer SystemsWinter 2021-2022Stanford UniversityInstructors: Nick Troccoli and Jerry CainIllustration courtesy of Roz Cyrus.PDF of this presentation1

CS110 Topic 2: How can our programcreate and interact with other programs?2

Learning About ProcessesCreatingprocesses andrunning otherprogramsLecture 6/7Inter-processcommunicationand PipesSignalsRace ConditionsThislectureLecture10/11Lecture11assign3: implement multiprocessing programs like "trace" (to trace anotherprogram's behavior) and "farm" (parallelize tasks)assign4: implement your own shell!3

Learning GoalsGet more practice creating and using pipesLearn about dup2 to create and manipulate file descriptorsUse pipes to redirect process input and output4

Lecture PlanReview: pipesRedirecting process I/OPractice: Implementing subprocessPractice: Implementing pipeline5

Lecture PlanReview: pipesRedirecting process I/OPractice: Implementing subprocessPractice: Implementing pipeline6

PipesA pipe is a set of two file descriptors representing a "virtual file" that can be written toand read fromIt's not actually a physical file on disk - we are just using files as an abstractionAny data you write to the write FD can be read from the read FDBecause file descriptors are duplicated on fork(), we can create pipes that are sharedacross processes!7

Illustration courtesy of Roz Cyrus.8

Key Idea: because the pipe filedescriptors are duplicated in the child,we need to close the 2 pipe ends in boththe parent and the child.9

Parent-Child Communication1 static const char * kPipeMessage "Hello, this message is coming through a pipe.";2 int main(int argc, char *argv[]) {int fds[2];3pipe(fds);4size t bytesSent strlen(kPipeMessage) 1;56pid t pidOrZero fork();7if (pidOrZero 0) {8// In the child, we only read from the pipe9close(fds[1]);10char buffer[bytesSent];11read(fds[0], buffer, sizeof(buffer));12close(fds[0]);13printf("Message from parent: %s\n", buffer);14return 0;15}1617// In the parent, we only write to the pipe (assume everything is written)18close(fds[0]);19write(fds[1], kPipeMessage, bytesSent);20close(fds[1]);21waitpid(pidOrZero, NULL, 0);22return 0;2324 }parent-child-pipe.cHere's an example programshowing how pipe worksacross processes (fullprogram link at bottom).10

continued.Illustrations courtesy of Roz Cyrus.11

continued.Illustrations courtesy of Roz Cyrus.12

continued.Illustrations courtesy of Roz Cyrus.13

continued.Illustrations courtesy of Roz Cyrus.14

continued.Illustrations courtesy of Roz Cyrus.15

continued.Illustrations courtesy of Roz Cyrus.16

continued.Illustrations courtesy of Roz Cyrus.17

PipesThis method of communication between processes relies on the fact that file descriptors areduplicated when forking.each process has its own copy of both file descriptors for the pipeboth processes could read or write to the pipe if they wanted.each process must therefore close both file descriptors for the pipe when finishedThis is the core idea behind how a shell can support piping between processes(e.g. cat file.txt uniq sort).18

Lecture PlanReview: pipesRedirecting process I/OPractice: Implementing subprocessPractice: Implementing pipeline19

Redirecting Process I/OEach process has the special file descriptors STDIN (0), STDOUT (1) and STDERR (2)Processes assume these indexes are for these methods of communication (e.g. printfalways outputs to file descriptor 1, STDOUT).01Terminal23FileIdea: what happens if we change FD 1 to point somewhere else?20

Redirecting Process I/OIdea: what happens if we change FD 1 to point somewhere else?1 int main() {2printf("This will print to the terminal\n");3close(STDOUT FILENO);45// fd will always be 16int fd open("myfile.txt", O WRONLY O CREAT O TRUNC, 0644);78printf("This will print to myfile.txt!\n");9close(fd);10return 0;11 }012Terminal21

Redirecting Process I/OIdea: what happens if we change FD 1 to point somewhere else?1 int main() {2printf("This will print to the terminal\n");3close(STDOUT FILENO);45// fd will always be 16int fd open("myfile.txt", O WRONLY O CREAT O TRUNC, 0644);78printf("This will print to myfile.txt!\n");9close(fd);10return 0;11 }012Terminal22

Redirecting Process I/OIdea: what happens if we change FD 1 to point somewhere else?1 int main() {2printf("This will print to the terminal\n");3close(STDOUT FILENO);45// fd will always be 16int fd open("myfile.txt", O WRONLY O CREAT O TRUNC, 0644);78printf("This will print to myfile.txt!\n");9close(fd);10return 0;11 }012Terminal myfile.txt23

Redirecting Process I/OIdea: what happens if we change FD 1 to point somewhere else?1 int main() {2printf("This will print to the terminal\n");3close(STDOUT FILENO);45// fd will always be 16int fd open("myfile.txt", O WRONLY O CREAT O TRUNC, 0644);78printf("This will print to myfile.txt!\n");9close(fd);10return 0;11 }012Terminal myfile.txt24

Redirecting Process I/OIdea: what happens if we change a special FD to point somewhere else?Could we do this with a pipe?Process 101Process 22Terminal0pipeREAD12pipeWRITEWhy would this be useful?25

Redirecting Process I/OI/O redirection and pipes allow us to handle piping in our shell: e.g. cat file.txt sort0cat12Terminal0pipeREADsort12pipeWRITEThis allows the shell to link together two distinct executables without them knowing.(How?)26

Redirecting Process I/OStepping stone: our first goal is to write a program that spawns another program and sends data toits STDIN.Our programsort012.4012TerminalpipeREADpipeWRITEThe sort executable has no idea its input is not coming from terminal entry!27

Redirecting Process I/OOur first goal is to write a program that spawns another program and sends data to its STDIN.1. Our program creates a pipe2. Our program spawns a child process3. That child process changes its STDIN to be the pipe read end (how?)4. That child process calls execvp to run the specified command5. The parent writes to the write end of the pipe, which appears to the child as its STDIN"Wait a minute.I thought execvp consumed the process? How do the file descriptors stickaround?"New insight: execvp consumes the process, but leaves the file descriptor table in tact!28

Redirecting Process I/OOne issue; how do we "connect" our pipe FDs to STDIN/STDOUT?dup2 makes a copy of a file descriptor entry and puts it in another file descriptor index. Ifthe second parameter is an already-open file descriptor, it is closed before being used.int dup2(int oldfd, int newfd);Example: we can use dup2 to copy the pipe read file descriptor into standard input!dup2(fds[0], STDIN FILENO);29

Redirecting Process I/Odup2 makes a copy of a file descriptor entry and puts it in another file descriptor index. Ifthe second parameter is an already-open file descriptor, it is closed before being used.int dup2(int oldfd, int newfd);Illustrations courtesy of Roz Cyrus.30

Lecture PlanReview: our first shellRunning in the backgroundIntroducing PipesWhat are pipes?Pipes between processesRedirecting process I/OPractice: Implementing subprocess31

subprocessTo practice this piping technique, let's implement a custom function called subprocess.subprocess t subprocess(char *command);subprocess is the same as mysystem, except it also sets up a pipe we can use to write tothe child process's STDIN.It returns a struct containing:the PID of the child processa file descriptor we can use to write to the child's STDIN32

Demo: subprocesssubprocess-soln.c33

Lecture PlanReview: pipesRedirecting process I/OPractice: Implementing subprocessPractice: Implementing pipeline34

PipelineI/O redirection and pipes allow us to handle piping in our shell: e.g. cat file.txt sort0cat12Terminal0pipeREADsort12pipeWRITEFinal task: write a program that spawns two child processes and connects the first child'sSTDOUT to the second child's STDIN.35

Redirecting Process I/OOur final goal is to write a program that spawns two other processes where one's output is theother's input. Both processes should run in parallel.1. Our program creates a pipe2. Our program spawns a child process3. That child process changes its STDIN to be the pipe read end4. That child process calls execvp to run the first specified command5. Our program spawns another child process6. That child process changes its STDOUT to be the pipe write end36

pipelineLet's implement a custom function called pipeline.void pipeline(char *argv1[], char *argv2[], pid t pids[]);pipeline is similar to subprocess, except it also spawns a second child and directs itsSTDOUT to write to the pipe. Both children should run in parallel.It doesn't return anything, but it writes the two children PIDs to the specified pids array37

Demo: pipelinepipeline-soln.c38

pipe2There were a lot of close() calls! Is there a way for any of them to be done automatically?int pipe2(int fds[], int flags);pipe2 is the same as pipe except it lets you customize the pipe with some optional flags.if flags is 0, it's the same as pipeif flags is O CLOEXEC, the pipe FDs will be automatically closed when the surroundingprocess calls execvp.39

pipeline1 void pipeline(char *argv1[], char *argv2[], pid t pids[]) {2int fds[2];3pipe(fds);45pids[0] fork();6if (pids[0] 0) {close(fds[0]);78dup2(fds[1], STDOUT FILENO);close(fds[1]);910execvp(argv1[0], argv1);11}1213close(fds[1]);1415pids[1] fork();16if (pids[1] 0) {17dup2(fds[0], STDIN FILENO);close(fds[0]);1819execvp(argv2[0], argv2);20}2122close(fds[0]);23 }The highlighted calls toclose() would no longer benecessary if we use pipe2 withO CLOEXEC because thesurrounding process for eachcalls execvp.Note that the parent must stillclose them because it doesn'tcall execvp.40

pipeline with pipe21 void pipeline(char *argv1[], char *argv2[], pid t pids[]) {2int fds[2];3pipe2(fds, O CLOEXEC);45pids[0] fork();6if (pids[0] 0) {7dup2(fds[1], STDOUT FILENO);8execvp(argv1[0], argv1);9}1011close(fds[1]);1213pids[1] fork();14if (pids[1] 0) {15dup2(fds[0], STDIN FILENO);16execvp(argv2[0], argv2);17}1819close(fds[0]);20 }This version of pipeline usespipe2 with O CLOEXEC.41

Pipes and I/O Redirection: Key TakeawaysPipes are sets of file descriptors that allow us to communicate across processes.Processes can share these file descriptors because they are copied on fork()File descriptors 0,1 and 2 are special and assumed to represent STDIN, STDOUT andSTDERRIf we change those file descriptors to point to other resources, we can redirectSTDIN/STDOUT/STDERR to be something else without the program knowing!Pipes are how terminal support for piping and redirection (command1 command2and command1 file.txt) are implemented!42

Lecture RecapReview: pipesRedirecting process I/OPractice: Implementing subprocessPractice: Implementing pipelineNext time: signals (another form of interprocess communication)43

Practice Problems44

A Publishing ErrorThe program below takes an arbitrary number of filenames as arguments and attempts to publishthe date and time. The desired behavior is shown at right:1234567891011121314static void publish(const char *name) {printf("Publishing date and time to file named \"%s\".\n", name);int outfile open(name, O WRONLY O CREAT O TRUNC, 0644);dup2(outfile, STDOUT FILENO);close(outfile);if (fork() 0) return;char *argv[] { "date", NULL };execvp(argv[0], argv);}int main(int argc, char *argv[]) {for (size t i 1; i argc; i ) publish(argv[i]);return 0;}12345myth62: ./publishPublishing date andPublishing date andPublishing date andPublishing date andone twotime totime totime totime tothree fourfile namedfile namedfile namedfile named"one"."two"."three"."four".However, the program is buggy!What text is actually printed tostandard output?What do each of the four filescontain?How can we fix the issue?publish.c45