WIXLIB

Shared storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .The consumer program is more complicated than necessary to highlight features of the locking API. In particular, theconsumer program first checks whether the file is exclusivelylocked and only then tries to gain a shared lock. The relevantcode is:cess involves reading or writing. As always, programmingcomes with tradeoffs. The next example has the upside ofIPC through shared memory, rather than shared files, with acorresponding boost in performance.lock.l type F WRLCK;Linux systems provide two separate APIs for shared memory: the legacy System V API and the more recent POSIXone. These APIs should never be mixed in a single application, however. A downside of the POSIX approach isthat features are still in development and dependent uponthe installed kernel version, which impacts code portability. For example, the POSIX API, by default, implementsshared memory as a memory-mapped file: for a sharedmemory segment, the system maintains a backing file withcorresponding contents. Shared memory under POSIXcan be configured without a backing file, but this may impact portability. My example uses the POSIX API with abacking file, which combines the benefits of memory access (speed) and file storage (persistence).The shared-memory example has two programs, namedmemwriter and memreader, and uses a semaphore to coordinate their access to the shared memory. Whenevershared memory comes into the picture with a writer, whetherin multi-processing or multi-threading, so does the risk of amemory-based race condition; hence, the semaphore is usedto coordinate (synchronize) access to the shared memory.fcntl(fd, F GETLK, &lock); /* sets lock.l type to F UNLCK if nowrite lock */if (lock.l type ! F UNLCK)report and exit("file is still write locked.");Shared memoryThe F GETLK operation specified in the fcntl call checksfor a lock, in this case, an exclusive lock given as F WRLCKin the first statement above. If the specified lock does notexist, then the fcntl call automatically changes the lock typefield to F UNLCK to indicate this fact. If the file is exclusivelylocked, the consumer terminates. (A more robust version ofthe program might have the consumer sleep a bit and tryagain several times.)If the file is not currently locked, then the consumer triesto gain a shared (read-only) lock (F RDLCK). To shortenthe program, the F GETLK call to fcntl could be droppedbecause the F RDLCK call would fail if a read-write lockalready were held by some other process. Recall that aread-only lock does prevent any other process from writingto the file, but allows other processes toread from the file. In short, a shared lock Example 3. Source code for the memwriter processcan be held by multiple processes. After/** Compilation: gcc -o memwriter memwriter.c -lrt -lpthreadgaining a shared lock, the consumer pro#include stdio.h gram reads the bytes one at a time from#include stdlib.h the file, prints the bytes to the standard#include sys/mman.h output, releases the lock, closes the file,#include sys/stat.h and terminates.#include fcntl.h Here is the output from the two pro#include unistd.h grams launched from the same terminal#include semaphore.h with % as the command line prompt:#include string.h #include "shmem.h"% ./producerProcess 29255 has written to data file.void report and exit(const char* msg) {perror(msg);% ./consumerNow is the winter of our discontentMade glorious summer by this sun of YorkIn this first code example, the data sharedthrough IPC is text: two lines from Shakespeare’s play Richard III. Yet, the sharedfile’s contents could be voluminous, arbitrary bytes (e.g., a digitized movie), whichmakes file sharing an impressively flexibleIPC mechanism. The downside is that fileaccess is relatively slow, whether the ac-8**/exit(-1);}int main() {int fd shm open(BackingFile,/* name from smem.h */O RDWR O CREAT, /* read/write, create if needed */AccessPerms);/* access permissions (0644) */if (fd 0) report and exit("Can't open shared mem segment.");ftruncate(fd, ByteSize); /* get the bytes */A guide to inter-process communication in Linux. CC BY-SA 4.0 . Opensource.com

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shared storageThe memwriter program should be started first in its ownterminal. The memreader program then can be started(within a dozen seconds) in its own terminal. The outputfrom the memreader is: The memwriter then calls the mmap function:caddr t memptr mmap(NULL,/* let system pick where toput segment */ByteSize,/* how many bytes */PROT READ PROT WRITE, /* accessThis is the way the world ends.protections */MAP SHARED, /* mapping visible to otherEach source file has documentation at the top explaining theprocesses */link flags to be included during compilation.fd,/* file descriptor */Let’s start with a review of how semaphores work as a0);/* offset: start at 1st byte */synchronization mechanism. A general semaphore also iscalled a counting semaphore, as it has a value (typicallyinitialized to zero) that can be incremented. Consider a to get a pointer to the shared memory. (The memreadershop that rents bicycles, with a hundred of them in stock,makes a similar call.) The pointer type caddr t startswith a program that clerks use to do the rentals. Everywith a c for calloc, a system function that initializes dytime a bike is rented, the semaphore is incremented bynamically allocated storage to zeroes. The memwriterone; when a bike is returned, the semaphore is decreuses the memptr for the later write operation, using themented by one. Rentals can continue until the value hitslibrary strcpy (string copy) function.100 but then must halt until at least onebike is returned, thereby decrementing Example 3. Source code for the memwriter process (continued)the semaphore to 99.caddr t memptr mmap(NULL,/* let system pick where to put segment */A binary semaphore is a special caseByteSize,/* how many bytes */requiring only two values: 0 and 1. InPROT READ PROT WRITE, /* access protections */this situation, a semaphore acts as aMAP SHARED, /* mapping visible to other processes */mutex: a mutual exclusion construct.fd,/* file descriptor */The shared-memory example uses a0);/* offset: start at 1st byte */semaphore as a mutex. When the semaif ((caddr t) -1 memptr) report and exit("Can't get segment.");phore’s value is 0, the memwriter alonecan access the shared memory. Afterfprintf(stderr, "shared mem address: %p [0.%d]\n", memptr, ByteSize - 1);writing, this process increments thefprintf(stderr, "backing file:/dev/shm%s\n", BackingFile );semaphore’s value, thereby allowing thememreader to read the shared memory/* semaphore code to lock the shared mem */(see Example 3).sem t* semptr sem open(SemaphoreName, /* name */Here’s an overview of how the memwritO CREAT,/* create the semaphore */er and memreader programs communiAccessPerms,/* protection perms */cate through shared memory:0);/* initial value */ The memwriter program, shown above,if(semptr (void*)-1)report and exit("sem open");calls the shm open function to geta file descriptor for the backing filestrcpy(memptr, MemContents); /* copy some ASCII bytes to the segment */that the system coordinates with theshared memory. At this point, no mem/* increment the semaphore so that memreader can read */ory has been allocated. The subseif (sem post(semptr) 0) report and exit("sem post");quent call to the misleadingly namedfunction ftruncate:sleep(12); /* give reader a chance */ftruncate(fd, ByteSize); /* get the bytes */ allocates ByteSize bytes, in this case,a modest 512 bytes. The memwriterand memreader programs access theshared memory only, not the backingfile. The system is responsible for synchronizing the shared memory and thebacking file./* clean up */munmap(memptr, ByteSize); /* unmap the storage */close(fd);sem close(semptr);shm unlink(BackingFile); /* unlink from the backing file */return 0;}A guide to inter-process communication in Linux. CC BY-SA 4.0 . Opensource.com9

Shared storage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A t this point, the memwriter is ready for writing, but it firstcreates a semaphore to ensure exclusive access to theshared memory. A race condition would occur if the memwriter were writing while the memreader was reading. If thecall to sem open succeeds:calls to shm open are similar in the two programs except thatthe memwriter program creates the shared memory, whereasthe memreader only accesses this already created memory:int fd shm open(BackingFile, O RDWR O CREAT, AccessPerms);/* memwriter */int fd shm open(BackingFile, O RDWR, AccessPerms);sem t* semptr sem open(SemaphoreName, /* name */O CREAT,/* create the semaphore */AccessPerms,/* protection perms */0);/* initial value */t hen the writing can proceed. The SemaphoreName (anyunique non-empty name will do) identifies the semaphorein both the memwriter and the memreader. The initial value of zero gives the semaphore’s creator, in this case, thememwriter, the right to proceed, in this case, to the writeoperation. After writing, the memwriter increments the semaphorevalue to 1:/* memreader */With a file descriptor in hand, the calls to mmap are thesame:ca ddr t memptr mmap(NULL, size, PROT READ PROT WRITE, MAPSHARED, fd, 0);The first argument to mmap is NULL, which means that thesystem determines where to allocate the memory in virtualaddress space. It’s possible (but tricky) to specify an addressExample 4. Source code for the memreader processif (sem post(semptr) 0) . ith a call to the sem post function. Inwcrementing the semaphore releases themutex lock and enables the memreaderto perform its read operation. For goodmeasure, the memwriter also unmaps theshared memory from the memwriter address space:munmap(memptr, ByteSize); /* unmap the storage * This bars the memwriter from further accessto the shared memory (see Example 4).In both the memwriter and memreader programs, the shared-memory functions of main interest are shm open and mmap: on success,the first call returns a file descriptor for the backing file, which the second call then uses to geta pointer to the shared memory segment. The/** Compilation: gcc -o memreader memreader.c -lrt -lpthread **/#include stdio.h #include stdlib.h #include sys/mman.h #include sys/stat.h #include fcntl.h #include unistd.h #include semaphore.h #include string.h #include "shmem.h"void report and exit(const char* msg) {perror(msg);exit(-1);}int main() {int fd shm open(BackingFile, O RDWR, AccessPerms);/* empty to begin */if (fd 0) report and exit("Can't get file descriptor.");/* get a pointer to memory */caddr t memptr mmap(NULL,ByteSize,/* let system pick where to put segment *//* how many bytes */PROT READ PROT WRITE, /* access protections */MAP SHARED, /* mapping visible to other processes */10fd,/* file descriptor */0);/* offset: start at 1st byte */A guide to inter-process communication in Linux. CC BY-SA 4.0 . Opensource.com

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Shared storageinstead. The MAP SHARED flag indicates that the allocatedmemory is shareable among processes, and the last argument (in this case, zero) means that the offset for the sharedmemory should be the first byte. The size argument specifies the number of bytes to be allocated (in this case, 512),and the protection argument indicates that the shared memory can be written and read.When the memwriter program executes successfully, thesystem creates and maintains the backing file; on my system, the file is /dev/shm/shMemEx, with shMemEx as myname (given in the header file shmem.h) for the shared storage. In the current version of the memwriter and memreaderprograms, the statement:The memreader, like the memwriter, accesses the semaphore through its name in a call to sem open. But the memreader then goes into a wait state until the memwriter increments the semaphore, whose initial value is 0:if (!sem wait(semptr)) { /* wait until semaphore ! 0 */Once the wait is over, the memreader reads the ASCII bytesfrom the shared memory, cleans up, and terminates.The shared-memory API includes operations explicitly tosynchronize the shared memory segment and the backingfile. These operations have been omitted from the exampleto reduce clutter and keep the focus on the memory-sharingand semaphore code.The memwriter and memreader programs are likely to exeshm unlink(BackingFile); /* removes backing file */cute without inducing a race condition even if the semaphorecode is removed: the memwriter creates the shared memoryremoves the backing file. If the unlink statement is omitted,segment and writes immediately to it; the memreader cannotthen the backing file persists after the program terminates.even access the shared memory until this hasbeen created. However, best practice requiresExample 4. Source code for the memreader process (continued)that shared-memory access is synchronizedif ((caddr t) -1 memptr) report and exit("Can't access segment.");whenever a write operation is in the mix, andthe semaphore API is important enough to be/* create a semaphore for mutual exclusion */highlighted in a code example.sem t* semptr sem open(SemaphoreName, /* name */O CREAT,/* create the semaphore */AccessPerms,/* protection perms */0);/* initial value */if (semptr (void*) -1) report and exit("sem open");/* use semaphore as a mutex (lock) by waiting for writer to increment it */if (!sem wait(semptr)) { /* wait until semaphore ! 0 */int i;for (i 0; i strlen(MemContents); i )write(STDOUT FILENO, memptr i, 1); /* one byte at a time */sem post(semptr);}/* cleanup */munmap(memptr, ByteSize);close(fd);sem close(semptr);Wrapping upThe shared-file and shared-memory examples show how processes can communicatethrough shared storage, files in one case andmemory segments in the other. The APIs forboth approaches are relatively straightforward. Do these approaches have a commondownside? Modern applications often deal withstreaming data, indeed, with massively largestreams of data. Neither the shared-file nor theshared-memory approaches are well suited formassive data streams. Channels of one type oranother are better suited. Part 2 thus introduces channels and message queues, again withcode examples in C.unlink(BackingFile);return 0;}A guide to inter-process communication in Linux. CC BY-SA 4.0 . Opensource.com11

Using pipes and message queues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Using pipes andmessage queuesLearn how processes synchronize with each other in Linux.This section turnsto pipes, whichare channelsthat connect processes for communication. A channel has awrite end for writing bytes, and a read end for reading thesebytes in FIFO (first in, firstout) order. In typical use, oneprocess writes to the channel, and a different processreads from this same channel. The bytes themselvesmight represent anything:numbers, employee records,digital movies, and so on.Pipes come in two flavors,named and unnamed, andcan be used either interactively from the command lineor within programs; examples are forthcoming. This sectionalso looks at memory queues, which have fallen out of fashion—but undeservedly so.The code examples in the first section acknowledged thethreat of race conditions (either file-based or memory-based)in IPC that uses shared storage. The question naturallyarises about safe concurrency for the channel-based IPC,which will be covered in this section. The code examplesfor pipes and memory queues use APIs with the POSIXstamp of approval, and a core goal of the POSIX standardsis thread-safety.Consider the man pages for the mq open [1] function,which belongs to the memory queue API. These pages include a section on Attributes [2] with this small table:Interfacemq open()AttributeThread safetyValueMT-SafeThe value MT-Safe (with MT for multi-threaded) means thatthe mq open function is thread-safe, which in turn impliesprocess-safe: A process executes in precisely the sense that12one of its threads executes, and if a race condition cannotarise among threads in the same process, such a conditioncannot arise among threads in different processes. The MTSafe attribute assures that a race condition does not arise ininvocations of mq open. Ingeneral, channel-based IPCis concurrent-safe, althougha cautionary note is raised inthe examples that follow.Unnamed pipesLet’s start with a contrivedcommand line example thatshows how unnamed pipeswork. On all modern systems, the vertical bar represents an unnamed pipe atthe command line. Assume % is the command line prompt,and consider this command:% sleep 5 echo "Hello, world!" ## writer to the left of ,reader to the rightThe sleep and echo utilities execute as separate processes,and the unnamed pipe allows them to communicate. However, the example is contrived in that no communication occurs. The greeting Hello, world! appears on the screen; then,after about five seconds, the command line prompt returns,indicating that both the sleep and echo processes have exited. What’s going on?In the vertical-bar syntax from the command line, the process to the left (sleep) is the writer, and the process to theright (echo) is the reader. By default, the reader blocks untilthere are bytes to read from the channel, and the writer—after writing its bytes—finishes up by sending an end-ofstream marker. (Even if the writer terminates prematurely, anend-of-stream marker is sent to the reader.) The unnamedpipe persists until both the writer and the reader terminate.A guide to inter-process communication in Linux. CC BY-SA 4.0 . Opensource.com

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Using pipes and message queuesparent in case of failure. In the pipeUN example, thecall is:In the contrived example, the sleep process does notwrite any bytes to the channel but does terminate afterabout five seconds, which sends an end-of-stream marker to the channel. In the meantime, the echo process immediately writes the greeting to the standard output (thescreen) because this process does not read any bytesfrom the channel, so it does no waiting. Once the sleepand echo processes terminate, the unnamed pipe—notused at all for communication—goes away and the command line prompt returns.Here is a more usef

Shared memory (with semaphores) Pipes (named and unnamed) Message queues Sockets Signals I’ll introduce you to some core concepts before moving on to the first two of these mech-anisms: shared files and shared memory. core concepts A process is a program in exe