WIXLIB

Device naming has been and will continue to be an issue. There are worldwidenamings in fiber channel, SCSI namingby probe order, and the same with IDEdevices. Ts’o mentioned that in somecases you will not be able to enumerateall devices at boot time, and sixteen bitsfor dev t (that holds the minor devicenumber) will not be enough.Jens Axbone mentioned that he had donesome tests with Andre Hedrick wherethey moved the io request locks withinthe device layer and got better performance. In the future, elevator scansshould only occur once, when eitherinserting or scanning, not twice as happens now. Because of writeback cachingon ATA, you have to do a write-backflush even if you disable the cache on thedrive. Don Duggans quipped, “Theplacebo bit.”Tweedie agreed that each driver shouldmaintain its own queue. Someone saidthat we build devices that look like 36logical devices but are really hundreds ofspindles. We would prefer that you candisable part of the elevator. Tweedieresponded, “That can be done, but youwill still want us to merge requests, justnot order them.”ADVANCED FILE SYSTEM INTEGRATIONStephen Lord, SGILord discussed some advanced featuresof XFS, suggesting that some of themcould be moved into the kernel as part ofthe VFS interface. The key ideas surrounded the notion of delayed writes.Instead of scheduling a write of dataimmediately to disk, in delayed writesonly the space for the data is reserved,and the actual write is done later. As mostprograms will continue to write data,June 2001 ;login:After some comments in the Q&A aboutthe need for Linux to scale larger, DaveMiller asked if it makes sense for anyoneto use 128 CPU systems. Lord answeredthat SGI’s customers were buying them.Miller suggested that the latency issuescould be reduced by using clusters ofeight CPU machines. Lord replied thatclusters can fail because of variable bandwidth needed by different parts of theapplication. Others seemed to think thatsmaller scale clusters might be the future.Miller brought up NUMA (non-uniformmemory access) architectures and wondered if it is necessary to scale up to somany CPUs.The discussion shifted back to talk aboutdelayed I/O and what happens if insufficient space is reserved for file metadata.Lord replied that SGI has been successfully doing this for years.I LLUMINATING THE N ETDRIVER API . . .ONE MORE TIMEJamal Hadi Salim, Znyx Networks (alsoRobert Olsson and Alexy Kutsnetsov)Hadi Salim exposed a serious problemwith existing Linux kernels: underextreme network loads, the system collapses. A comparison between a FreeBSDsystem and a Linux system showed thatthe BSD kernel could handle up to70kpps (kilopackets per second) but thatLinux folds at only 24kpps. Keeping inmind that a T1 filled to capacity with 64byte packets is about 25kpps, this is notgood. Furthermore, on SMP systems,performance is worse. This behavior alsostarves other network interfaces that arenot overloaded.When Andy Grover asked why BSD didso much better, someone replied, “Theylied!”solution that appears to provide a tenfold increase in performance. Their goalswere to reduce interrupts on overload,drop packets early on overload, removeor reduce unfairness, and balance latencyand throughput, without requiring specific hardware solutions (e.g., Tulip chip).CONFERENCE REPORTSdelayed writes cause disk writes tobecome batched, and temporary filesmay never need to be written to disk atall. tiguous on disk but discontinuous inmemory. “Each I/O merge is a scan of upto 8,192 requests!” Tweedie said. Scheduling could be made more fair by scheduling per spindle rather than for host busadapter.An obvious solution occurred to me (toobad no one asked me about this earlier).Each packet arriving at the networkinterface generates an interrupt, and it isthis interrupt processing that soon overwhelms the kernel. Serial card designersof the early nineties had already figuredthis out, and the fastest drivers usedpolling instead of interrupts. HadiSalim’s solution is similar in that interrupts are disabled when the loadincreases and only re-enabled when theDMA ring assigned to the device isempty. If more packets arrive when theDMA ring is full, they are dropped without any further processing. Using 2.4.0beta 10 they could get 200kpps peakusing commodity hardware that supportsDMA and polling.Larry McVoy, who seems to have workedfor all the UNIX workstation vendors atsome point, remarked, “Rob Warnick atSGI did an enormous amount of goodwork in this area, forced because they[SGI] had to work with slow processors.If you do this stuff right, you just pass thepackets right up, but if you get a blast,then you start to do interrupt collapsing.”When Miller remarked that this wouldclean up the drivers a lot, there was around of applause.There was more discussion about theeffect of this solution on servers (betterperformance), latency (less latency), andolder drivers. Hadi Salim said that thesystem reverts to old behavior when thedriver does not support polling. The session ended with everyone feeling goodabout this solution, which is likely to findits way into the 2.5 kernel.Hadi Salim, working with Olsson andKutsnetsov, had experimented with aLINUX 2.5 KERNEL DEVELOPERS SUMMIT 7

LINUX HOT PLUGJohannes Erdfelt, VA Linux; Greg KroahHartman, WireXErdfelt started off by saying that theydon’t have a solution to this problem. Hisown area of expertise is USB, particularlyhubs and hot plug needs to deal withSCSI, Firewire, PCMCIA/Cardbus, andhotswap PCI in addition to USB devices.The problems include name and file permissions: that is, the device name presented to the user should not change andneither should the permissions associated with that device. Since not all ofthese devices have anything resembling aserial number or UUID, identifying themcan be a problem.Other issues include what to do whenthere are multiple drivers (e.g., using aparallel port) and how to notify applications (hey, a joystick just appeared!).Existing solutions include cardmgr forPCMCIA (which works because it is notvery complicated [laughter] compared toUSB, scsidev, or devfs/devfsd). Using ascript, /sbin/hotplug, works but does nothelp with naming. At this point, Erdfeltasked for suggestions.Someone immediately asked why devfs(think procfs) doesn’t work, and Erdfeltanswered that it doesn’t solve the namingproblem. Ts’o pointed out that in thePCMCIA world, shell scripts could handle things pretty well, but the SCSIdevices would be more difficult. Linussaid that /sbin/hotplug worked well forhim. He thought that vendors could berelied on to provide scripts for all thehot-pluggable devices and that wouldsolve 99% of the problem. “A script is avalid answer for a geek’s machine; it canbe edited, and RedHat can do it for everypossible device.”Peter Anvin said that he is responsible forassigning device numbers. And rightnow, we are running out of characternumbers. He just assigned number 228out of 225, and suggested that the discus-8sion (or flame war) be done offline in the10 p.m. BoF. This did not stop the discussion, which continued until the end ofthe session. Anvin made an interestingpoint when he mentioned that the Japanese wanted Japanese device names ratherthan English ones.prompted concern from Jes Sorenson,who is apparently porting Linux to IA64.“Can I make it work without Python?” heasked. Many people pointed out that thework could be done on the cross-compiling system and that Python was notrequired on the target system.RECEPTIONThere was actually a fair amount ofresistance to this effort. Raymondpointed out that the new version uses40% less code, runs faster, and has features the old tool does not. At one point,the discussion veered into arguing aboutthe kernel symbol table. But it ended on agood note, with Raymond saying, “Downthe road, I want to make configuration soeasy that your Aunt Tilly could do it.”This was followed by enthusiasticapplause.The reception drew most of the attendees, even some people who had said earlier that they couldn’t stay. I wound uptalking to Peter Loscocco of the NSAabout his part in a project to add realsecurity to the Linux kernel and missedthe BoFs. Well, I could have attendedsince they went until midnight, but Ipacked it in. There were also BoFs afterthe conference finished. You can readabout the BoFs, and get another perspective on the conference, at LWN: http://lwn.net/2001/features/KernelSummit/ .KERNEL KBUILDKeith Owens, OC Software; and EricRaymondNine o’clock Saturday morning startedoff pretty quietly, no surprise there. Thetopic concerned a proposed replacementfor the kernel configuration tools in thepresent distribution. I was happy to hearthis, but not everyone else was. The present system presents a text-based menuthat does not let you go backward, and aGUI-based one that also fails to satisfyme.Raymond is not too fond of it either.“I’ve been examining the existing kernelconfiguration system, and I have aboutconcluded that the best favor we coulddo everybody involved with it is to take itout behind the barn and shoot it throughthe head.” Owens and Raymond set outto build a much improved system, andthey did.The new system (CML2) makes it impossible to generate an invalid configuration.The new system uses a Python engine toenforce a set of rules to do this. ThisFUTURE VM WORKRik van Riel, Conectiva S.A.Van Riel had a detailed presentation covering a lot of difficult topics in VM, hisown specialty. Occasionally he wound uplistening to the discussion that ragedabout contentious issues, such as pagesize.Van Riel began by discussing memorybalancing, deciding which pages to stealand when. The current implementationscans the process page tables and stealswhatever it can. Better methods wouldinvolve keeping track of working sets,providing processes with a definedamount of physical memory. Anotherapproach would use reverse mapping,keeping track of which processes have amapping from a virtual page to a physicalpage. Doing this would add eight bytes tothe page table entry, doubling its size.Once again, moving to a larger page sizewas suggested. Linus pointed out thatearly versions of his kernel did just that,but that it would make no sense to dothis “just for Oracle.” Van Riel explainedthat there would be benefits for otherusers, such as smaller page tables and areduction in page faults. Large pages areVol. 26, No. 3 ;login:

The Linux kernel still has some problemswith deadlocks: for example, it needs toallocate memory to free some pages.Thrashing can also occur; Van Riel suggested suspending processes when thesystem begins to thrash, in order to stabilize the load, and gradually releasing eachprocess as the system recovers.MANDATORY ACCESS CONTROLS / SE LINUXPeter Loscocco, NSAI thought that this proposal was wellreceived, although Loscocco told me laterthat he was disappointed that he didn’tget more support. You can get moredetails, or the code if you like, at http://www.nsa.gov/selinux/ . Loscoccoalso told me that in the eighties, he wasresponsible for the ARPANET IMP thatpassed through the NSA site, and thatstories about how the NSA was copyingall the data on the ARPANET were reallyfunny. Eventually, the NSF moved theIMP somewhere else because of difficulties in keeping it up and running. Butthat’s another story.SE Linux uses a large kernel patch to addMandatory Access Controls (MAC) to astandard Linux kernel. MAC means thatpermissions are no longer discretionary,and that only a security administratorcan change them. The changes to the kernel are pervasive and fine-grained, meaning that it is possible to tie downeverything.The design goals of SE Linux includedevelopment of:Separation policies to keep data separate from some other part of thesystem Containment policies to restrict Webserver access to authorized data Integrity policies to protect applications from modification Invocation policies to control thechain of processing The current implementation focuses onenforcement, not policy. Loscocco saidthat the sample implementation doescome with some policy models thatcould be used to nail down a Web server,for example.This session ended with Linus saying thathe liked the code he had seen, but he wasaware that there were many security projects in progress right now, and that hedid not want to have to choose betweenthem. If a function call could replaceeach section of code the NSA design hadadded, then it might be acceptable. People who did not want to use the securitycould replace this with a function thatsimply returns.ASYNCHRONOUS I/O FOR LINUXBen LaHaise, RedHat Canada Ltd.One important point is that SE Linux isnot an Orange Book system. The systemhas not been evaluated, although it hasAsynchronous I/O would allow programsto write data to some device and thencontinue without waiting for the operation to complete. LaHaise pointed outthat this would be useful in event-basedapplications when you want to avoidJune 2001 ;login:LINUX 2.5 KERNEL DEVELOPERS SUMMIT using thread-based I/O (8KB overheadper thread per each request, and therecould be tens or thousands of requests).It also makes efficient use of raw I/O, fitswell with zero copy, and has lower overhead for daemons than select/poll systemcalls.CONFERENCE REPORTSShared page tables would also help in thecase where processes share memory.Again, this is also something that Oracleasked for, but it would involve constraints (like having to be aligned on a4MB boundary on some architectures).Another layer of locking would also beneeded, leading to the possibility ofdeadlocks.evolved from earlier work such as DTOSwithin MACH, and Flask with Fluke (seea paper presented at a USENIX SecuritySymposium). SE Linux uses TypeEnforcement, which goes way beyondOrange Book designs. Type Enforcementincludes the usual subject (user) andobject (resource) found in earlier modelsbut adds the program to the equation.With Type Enforcement, you can specifythat a user can write to /etc/shadow butonly when running the passwd program. not suitable for all uses, so the idealwould be to mix large and small pages.Asynchronous I/O, as proposed byLaHaise, would add four new systemcalls:submit ios() allows a process to fireoff an asynchronous operation.io cancel() is for canceling outstanding operations.io getevents() gets information oncompleted operations.io wait() allows a process to wait forthe completion of a specific operation,essentially turning it synchronous.Linus, who obviously was already familiar with the proposal, interruptedLaHaise at this point: “You have alreadyadded four system calls, why have a newdevice?” The device in question would benamed /dev/aio and be used to arrangefor a section of mapped memory that thecalling process could use to detect I/Ocompletion. Linus was not at all happywith this use of mapped memory andcontinued to ask if LaHaise reallythought that mmap was necessary.Toward the end of his defense of mmap,Ulrich Drepper, defender of the systemcall API, asked, “Why do you want to addall this support to character devices? It iscompletely wrong to do this.” LaHaisepointed out that POSIX AIO doesn’t do avery good job of the semantics, and thathis approach will really make a differencewith sockets, particularly UDP. Drepperasked if you can have AIO for fsync(), andLaHaise responded that he had code thatcould currently do that.This project is not finished yet, and iscurrently about a 3000 line patch. Workto be done included adding networksockets, limiting memory pinning, andwriting some documentation.9

LINUX AND POWER MANAGEMENT, ACPIAndy Grover, IntelGrover started out by explaining the reasons why PC power management isimportant. He suggested some obviousthings, like green PCs and mobile PCs.When he mentioned that servers mightalso be able to do this to reduce powerwhen sitting idly in large clusters, Linusreacted. It was obvious this is somethinghe wanted.In the older scheme, currently supportedby the Linux kernel, APM handles powermanagement. The trouble with that isthat APM is an obsolete interface, andthat the BIOS keeps track of APM so thatit is not managed by the OS. The replacement for APM is the Advanced Configuration and Power Interface (ACPI),conceived by Intel, Microsoft, andToshiba, which allows OS-directed powermanagement.Grover went on to describe five softpower levels and four hard (device) levels, ranging from fully on to power off.Grover felt that implementing soft powerlevel three would be the best target for2.5. S3 permits idle devices to be disabledand has better power saving than levelsone or two and quicker wakeup thanlevel four (sleep). Enabling level three is aprerequisite for doing level four.Grover explained that implementing levelthree means that it must be possible toshut off devices, and then to turn themall on again while restoring sufficientstate for them to return to working condition. This implies having a device manager that knows enough about all thedevices to shut them down or reinitializethem.A lively discussion began at this point,with different people wondering how tomake this work. Linus pointed out thatthe PCI device has extra, unused fields inits structure that would allow it to beextended to handle any device, and buildan internal device tree. He had actuallyconsidered adding the changes in earlier,but the patches are very large, and addinglarge patches disturbs people. Ts’opointed out that 2.5 might be the idealtime to do this.The following Web sites have more info: http://developer.intel.com/technology/iapc/acpi ; http://phobos.fachschaften.tu-muenchen.de/acpi (forACPI 4 Linux)BITKEEPERLarry McVoy, BitMover Inc.Before he got started, McVoy suggestedthat we thank Ted Ts’o for making thedevelopers summit possible. Then, hehad a short rant about scaling up theLinux kernel, suggesting that SMP clusters, with one kernel monitoring everyfour kernels, was the way to go.McVoy was really there to talk about BitKeeper, a source code control system.Subversion, another CVS system, hadbeen the topic of a Friday night BoF,which McVoy could not attend. According to McVoy, as well as Victor Yodaiken,BitKeeper had the only GUI interfacethat would really work for kernel hackers.He went on to describe various features,as well as demonstrating them for thekernel hackers.“BitKeeper is a peer-to-peer system. Youget revision control files; we merge revision histories. We can sync sideways,rather than go up and then down, whichis what we did when I worked at Sun(and I wrote their systems). You have tokeep track of revisions, but we compressthem.”Listening to McVoy, and the reaction tohis demonstration, made it appear thatBitKeeper really would work well in theworld of Linux. There are some problems: for example, getting more checkedout than you want if several moduleshave been modified since the last timeyou peered. Raymond complained whenhe found that the only editor bindingsSummit group photo. See the Linux Weekly News for a color version with ids of the participants http://lwn.net/2001/features/KernelSummit/ [photo: J. Corbet/LWN.net]10Vol. 26, No. 3 ;login:

CONFERENCE REPORTSwere for vi/vim (Raymond is an emacsfan). In the end, McVoy said that Alan Coxsays we are about 75% to getting wto theway that Alan works now. BitKeeper is aproduct but is free for noncommercialuse. I enjoyed listening to McVoy, and itwas obvious that lots of attendees werewilling to listen to an experienced “oldtimer” (McVoy is about 40 years old).BUGTRACKING OPEN SESSIONTed Ts’o took the helm for this final official session. The focus was on bugtracking, which appears (to me) to be verylame at this point. The current mechanism is that someone posts a bug to thekernel developers mailing list, and thensomeone (usually Alan Cox) notices thebug, saves it, and perhaps dispatches it tothe person who manages that portion ofcode. Ts’o said he tried this for a whileand found it very difficult to do.Ts’o also made “a modest proposal.”Rather than have long periods betweenreleases, which leads to a last-minuterush of code submissions, he proposedthat a date be announced for a featurefreeze. While there was serious discussionof this, no date for 2.5 was announced.Suggestions for improving bugtrackingincluded using a database or tryingBugzilla. Another suggestion was to useRedHat’s sendbug script, which collectsinformation about the system andincludes it in the bug report, somethingthat turns out to be really importantwhen trying to support many CPU architectures, disk subsystems, etc.When Ts’o ended the discussion it was6:35 p.m. on Saturday night. The roombroke into six large groups who just kepton talking.Perhaps the workshop should have beenthree days long.June 2001 ;login:LINUX 2.5 KERNEL DEVELOPERS SUMMIT 11

Linus, Larsh, and Alan Cox. Linus sug-gested fast semaphores for scheduling rather than spin locks, and Cox suggested that the user space spin locks work like Mozilla."What Mozilla is doing is count-ing how many times you spin on locks before giving up, assuming that some 6 Vol. 26, No. 3;login: other process has been locked but is not .