Transcription
Practical Real-Time LinuxXenomai and PREEMPT RTArnout Vandecappellearnout@mind.be 2015 Essensium N.V.This work is licensed under aCreative Commons Attribution-ShareAlike 3.0Unported Licensehttp://www.mind.be/content/Presentation Real-Time Linux.odp
Adding controlto a high-pressure pumpPractical Real-Time LinuxArnout Vandecappellehttp://mind.be2
Adding controlto a high-pressure pump50μsPractical Real-Time LinuxArnout Vandecappellehttp://mind.be3
Practical Real-Time Linux Xenomai: separate RT and LinuxMotor control PREEMPT RT: native RT in LinuxGNSS receiver Conclusions and future directionsPractical Real-Time LinuxArnout Vandecappellehttp://mind.be4
XenomaiSeparate Real-Time and LinuxPractical Real-Time LinuxArnout Vandecappellehttp://mind.be5
Xenomai separates real-timefrom Linux scheduler50μsPractical Real-Time LinuxArnout Vandecappellehttp://mind.be6
Xenomai has a separatelow-latency schedulerXenomai threadsXenomai servicesLinux threadsLinux kernelXenomai schedulerAdeos I-Pipe HALPractical Real-Time LinuxArnout Vandecappellehttp://mind.be7
Xenomai threadsshadow Linux threadsXenomai threadsXenomai servicesLinux threadsLinux kernelXenomai schedulerAdeos I-Pipe HALPractical Real-Time LinuxArnout Vandecappellehttp://mind.be8
Xenomai thread switches tosecondary mode on Linux syscallXenomai threadXenomai servicesLinux threadLinux kernelXenomai schedulerPractical Real-Time LinuxArnout Vandecappellehttp://mind.be9
Xenomai thread switches tosecondary mode on Linux syscallXenomai threadXenomai servicesLinux threadLinux kernelXenomai schedulerPractical Real-Time LinuxArnout Vandecappellehttp://mind.be10
Avoid mode switchesto guarantee RT Xenomai does a lot out of the box by wrapping POSIX callsCheck .10.0NAMEROOT/0bench mainbench scopebench Eventbenchbench viewerIRQ42: pio0IRQ52: [timer] Using PTHREAD WARNSW: sends a SIGXCPU signal tion-to-xenomai-dual-kernelPractical Real-Time LinuxArnout Vandecappellehttp://mind.be11
CPU affinity and isolcpustrades off performance vs jitter100%75.00μs75%50%50.00μs25%0%Practical Real-Time Linux25.00μsArnout Vandecappellehttp://mind.be12
Xenomai separates real-timefrom Linux scheduler50μsPractical Real-Time LinuxArnout Vandecappellehttp://mind.be13
Implement RT driverswith RTDM framework Real-time drivers should work in Xenomai domainnot Linux domainNo access to Linux functions that schedule mode switch Real-Time Device Model provides–thread (task) ical Real-Time LinuxArnout Vandecappellehttp://mind.be14
Userspace APIfor RTDM drivers rt dev open(), rt dev write(), Avoid mode switch on device openread() and write() on RTDM deviceare handled by POSIX skin Makes code non-portablebut stub is anyway neededPractical Real-Time LinuxArnout Vandecappellehttp://mind.be15
Various problems encounteredwith Xenomaix86 SMI workaround overheatingMix of different skins (RTDM, native, posix) harder to maintainExtra code for simulation on non-RT PCNo valgrindPractical Real-Time LinuxArnout Vandecappellehttp://mind.be16
PREEMPT RTReal-Time in LinuxPractical Real-Time LinuxArnout Vandecappellehttp://mind.be17
GPS receiverMeasurementssampled @3kHzPractical Real-Time LinuxArnout Vandecappellehttp://mind.be18
GPS receiverwith connectivityMeasurementssampled @3kHzPractical Real-Time LinuxArnout Vandecappellehttp://mind.be19
PREEMPT RT is close to Linux http://git.kernel.org/cgit/linux/kernel/git/rt Pure kernel implementation, no API/ABI change On its way upstreamslowed down since 3.2 but should be picking up again Continuous testing in the OSADL QA ealtime-linux.0.html Main difference: every interrupt is a threadPractical Real-Time LinuxArnout Vandecappellehttp://mind.be20
PREEMPT RT removes almost alldisabling of interruptsSource: /Practical Real-Time LinuxArnout Vandecappellehttp://mind.be21
Debugging real-time issuesis more difficult with PREEMPT RT No atomic process listtop - 08:40:38 up 40 min,1 user,Cpu(s): 10.4%us,0.0%ni, 81.0%id, 8.6%sy,load average: 1.01, 1.02, 0.93PID USERPR S %CPU %MEM694 root-61 S 26.4 29.49:41.71 pollingThread593 root-51 S2:09.29 irq/288-pm wkup691 root-7 S5.3 29.42:04.05 thrDNPR737 root20 R5.30:00.07 top5.30.01.7TIME 0.0%wa,COMMANDNo “mode switches” to detect priority inversionPractical Real-Time LinuxArnout Vandecappellehttp://mind.be22
PriorityPriority inversion problemTimePractical Real-Time LinuxArnout Vandecappellehttp://mind.be23
PriorityPriority inversion problemPriority inversionTimePractical Real-Time LinuxArnout Vandecappellehttp://mind.be24
PriorityPriority inheritance solves inversionbut is expensiveTimeNormal mutex (95% uncontended):Priority Inheritance mutex:Practical Real-Time LinuxArnout Vandecappelleavg 600nsavg 16μshttp://mind.be25
Priority inversion issues encountered interrupts disable from RTEMS codehad to be replaced with PI-mutexopen() and socket() take lock on thefile descriptor table open all files at initialisation time or in other threadPractical Real-Time LinuxArnout Vandecappellehttp://mind.be26
Priority inversion issues encountered Memory manager lock–mlockall(MCL CURRENT MCL FUTURE)pre-fault stacks–pre-fault malloc()– Monitor page faultsPractical Real-Time LinuxArnout Vandecappellehttp://mind.be27
Priority inversion issues encountered fork() creates COW references of all pagesDifficult to discover–––Use vfork() fork() is implicit in system()Move hard real-time code to separate process a lot of workCreate system()-wrapper that calls separate processover D-BusNote: dbus-daemon does not listen on TCP port :-)Practical Real-Time LinuxArnout Vandecappellehttp://mind.be28
Real-time drivers All drivers are “real-time” Play with IRQ and kthread priority kworker is not associated with specific driver convert to kthreadtrackingThreadschedule work()[irq56]SPIPractical Real-Time LinuxArnout Vandecappellehttp://mind.be29
Real-time drivers All drivers are “real-time” Play with IRQ and kthread priority kworker is not associated with specific driver convert to kthreadtrackingThread[spi pump messages][irq56]SPIPractical Real-Time LinuxArnout Vandecappellehttp://mind.be30
Conclusionsand future directionsPractical Real-Time LinuxArnout Vandecappellehttp://mind.be31
Xenomai is converging with PREEMPT RT The Linux kernel running under Xenomaican be PREEMPT RT– Linux ROOT gets currently running thread's priorityXenomai 3.x offers dual-kernel and native option–Use same “Alchemy” APIon Xenomai and PREEMPT RT Xenomai latency is still significantly better RTnet only supports Xenomai (RTDM)Practical Real-Time LinuxArnout Vandecappellehttp://mind.be32
Xenomai or PREEMPT RT?Xenomai–––Tight latency requirementReal-time networking requirement (RTnet, Ethercat)Migrating existing RTOS applicationPREEMPT RT–––Use existing drivers from RT threadsRun same application on different platformsMigrating existing POSIX applicationPractical Real-Time LinuxArnout Vandecappellehttp://mind.be33
Practical Real-Time Linux Arnout Vandecappelle htt
