WIXLIB

Build Community AndroidDistributionandEnsure the QualityJim Huang ( 黃敬群 ) jserv@0xlab.org Developer & Co-Founder, 0xlabhttp://0xlab.org/Oct 28, 2011 / ELC Europe (Prague)

Rights to copy Copyright 2011 0xlabhttp://0xlab.org/contact@0xlab.orgAttribution – ShareAlike 3.0Corrections, suggestions, contributions andYou are freetranslations are welcome!to copy, distribute, display, and perform the workto make derivative worksLatest update: Oct 28, 2011to make commercial use of the workUnder the following conditionsAttribution. You must give the original author credit.Share Alike. If you alter, transform, or build upon this work, you may distribute theresulting work only under a license identical to this one.For any reuse or distribution, you must make clear to others the license terms of thiswork.Any of these conditions can be waived if you get permission from the copyright holder.Your fair use and other rights are in no way affected by the above.License text: lcode

0x1ab 162 16x10 11 427(founded on April 27, 2009)0xlab is another Hexspeak.

About Me(1) Come from Taiwan(2) Contributior of AndroidOpen Source Project (AOSP)(3) Developer, Linaro(4) Contributed to GNUClasspath / Kaffe, Linuxinternationalization (i18n),Openmoko

Commercial Partners of 0xlabARM / LinaroContribute to Linaro Android since the first line of codeAzureWaveBuild wireless networking & image processing solutionsMediatekAndroid based consumer productsOpen Embedded Software FoundationContribute to the reference ally, partners can benefit from opensource efforts and our experience.

Agenda(1) Build Android distribution(2) Lesson learned from AOSP(3) Ensure the Quality(4) Bring enhancements back toCommunity

Build Android Distributionbased on non typical open source projects

The reason why we built communityAndroid Distribution:Initially, we just wanted to enable wirelessconnectivity features on Android for our hardwarepartnters.But, we never thought that it was difficult to dothings efficiently.

0xdroid: enhanced Android distributionBeagleboardDevKit8000We suffered from performance and usability issues in AOSP. 0xdroidis basically the environment where we can develop and experiment.

What does 0xdroid deliver? Hardware enablement: Beagleboard (TI OMAP3),Pandaboard (TI OMAP4), Snowball (ST-EricssonUx500; on-going) Provide full source code for HAL Usability: software cursor, window manager fix,large screen tweaks, network connectivity fix Performance: ARM specific optimizations,graphics enhancement Features: Bluetooth HID (keyboard/mouse),external modem, 3D effects, customizedLauncher

But, 0xlab is not really making yetanother Android distribution. Wewish to help community.0xdroid is just a testbed (or reference implementation),and the valuable changes should be merged inupstream or other community projects.

Strategy and Policy open source efforts to improve AOSP We focus on small-but-important area of Android.– toolchain, libc, dynamic linker, skia, softwareGL, system libraries, HAL, UX Develop system utilities for Android– benchmark, black-box testing tool,validation infrastructure Feature driven development– Faster boot/startup time, Bluetooth profile,visual enhancements Submit and share changes to.– AOSP, CyanogenMod, Android-x86, andLinaro

Working Model by 0xlabRowboatCyanogenModAndroid-x86

Lesson Learned from AOSP

Let's go Upstream!Unfortunately, contributing to AOSPis an art .You never know how Google thinks of yourpatches exactly, even through Gerrit (code review system).

Problems We faced AOSP looks like “An Open Source Pretender" No public roadmap– Therefore, we ignore the modificationsagainst Android framework. The merged changes usually show up in next 1 or2 public release.– It is really hard to introduce/track therelevant changes. Not clear discussions on android-contrib mailinglist. Sometimes, you have to have privatecommunications to Google engineers. Version control / Code Review on invisiblerepositories (internal and far-away GIT tree)

Change represented in GerritgerritupstreamChange 123Patchset 1dev-bdev-aCommit cdeb34CreateLocalBranch% git checkout -b topic-branchPatchset 2Commit 41dbe5workGoogle provides the great codereview tool for AOSP, but.PushtoGerrit% git push gerrit \HEAD:refs/for/master

Flow of AOSP submitted ed!Push toGerritReviewapproved/submittedUpstream repoHere “upstream” means AOSP master and Google internal tree.

NoNo explicitexplicit changechange historyhistoryAndroidAndroid stylestyle openopen sourcesource modelmodelGap between Goolge internalGIT/Perforceand AOSP Git

After Gingerbread, it gets much clear fortoolchain partprebuilt/prebuilt/commit 19dcd0aaf7d08d57a4460229e43c45commitAuthor: JingJing YuYu jingyu@google.com jingyu@google.com Author:Date:Tue DecDec 1414 10:55:2310:55:23 20102010 0800 0800Date:TuePatched toolchaintoolchain toto fixfix aa fewfew gccgcc andand binutilsbinutils bugs.bugs.PatchedSources toto buildbuild thisthis toolchaintoolchain areare listedlisted onon arm eabi 4.4.3/SOURCESarm eabi 4.4.3/SOURCESSourceslinux x86/toolchain/arm eabi 4.4.3/SOURCESlinux x86/toolchain/arm eabi 4.4.3/SOURCESbuild/ syncedsynced totobuild/commit e8828f9458b1828a6f85e7791ae2aacommitAuthor: JimJim HuangHuang jserv@0xlab.org jserv@0xlab.org Author:Date:Fri AugAug 2020 23:30:3723:30:37 20102010 0800 0800Date:FriAnd rollbackrollback thethe followingfollowing 33 patches.patches.Andcommit 680529baca731c003bc58b68d72511commitAuthor: JingJing YuYu jingyu@google.com jingyu@google.com Author:Date:Thu AugAug 1212 15:52:1515:52:15 20102010 0700 0700Date:ThuAlthoughAlthough we can check gitgit log,log, wewestillstill havehave nono ideaidea whywhy they changedchanged

Observed AOSP Working Model “master” branch in Androidis the bridge between AOSP and Google internaltree. There are many contributions merged fromcompanies, organizations, and individuals. But noefficient code review available for non-existingrepositories. And, only few Google engineers doreview changes. Master branch the latest AOSP Partial changesby Google (bug-fixes from internal tree)– Not fully verified codebase. The best hints are the opinions written by Googleengineers inside Gerrit.– Send patches if possible

My interpretation ofAndroid:Hardware is Revolution;Sotware is basicallyEvolution;Android is Hardware-drivenSoftware Revolution

Functional View (Android 1.5)ApplicationsApplicationsGalleryPhoneWeb BrowserGoogle Maps・・・・・Android gerContentManagerView rLocationManagerSystem enCOREOpenCORESQLiteOpenGL ESOpenGL c libclibcbionicLinux KernelKernelLinuxNotificationManagerAndroid RuntimeRuntimeAndroidClass LibraryDalvik VirtualVirtual MachineMachineDalvikFreetype

Functional View (Android 2.3)ApplicationsApplicationsGalleryPhoneWeb BrowserGoogle Maps・・・・・Android gerContentManagerView rLocationManagerOpenGL ES 2.x accelerated.Drop 2D accelSurfaceFlingerSurfaceFlingerSystem LibraryLibrarySystemStageFrightSQLiteV8 bridgeGLES 2.0OpenGL ESOpenGL nic libclibcbionicSkia supports GPU backendIn Android ICSSMP fixesNotificationManagerRenderScriptAndroid RuntimeRuntimeAndroidClass LibraryDalvik VirtualVirtual MachineMachineDalvikSMP improvementsFreetypeJIT compilerLinux KernelKernelLinuxTheThe overalloverall designdesign isis consistant,consistant, butbut thethe currentcurrent modelmodelpreventsprevents fromfrom diversediverse communitycommunity contributions.contributions.

AOSP statistics for a.org(Dec rusted-logic.com7mc.pp.se7googlemail.comThe number are commits since Froyo release.However, the valuable changes from community such asCyanogenMod are usually absent due to long-time review process.

Ensure the Qualitywhen building custom Android distribution andmerging changes from community

Mission in our development:Improve UX in SoCUX User ExperienceSoC Integrated Computing Anywhere

Quality in custom Android Distribution 0xlab delivers the advantages of open sourcesoftware and development.– Quality relies on two factors: continuousdevelopment strong user feedback Several utilities are developed to ensure thequality and released as open source software.– 0xbench (Android benchmarking tool)– ASTER (Android System Testing Environment and Runtime)– LAVA (Linaro Automated Validation Architecture) In the meanwhile, performance is improved byseveral patches against essential components.

LAVA: Automated Validation Infrastructure for AndroidAndroid benchmark running on LAVA.Automated Validation flow includesfrom deploy, then reboot, testing,benchmark running, and result submit.Android support on VAAndroid related commands in LAVA:* deploy linaro android image* boot linaro android image* test android basic* test android monkey* test android 0xbench* submit results on host

0xbench:comprehensive open source benchmarksuite for Android A set of system utilities forAndroid to performcomprehensive systembenchmarking Dalvik VM performance OpenGL ES performance Android Graphics frameworkperformance I/O performance JavaScript engine performance Connectivity performance Micro-benchmark: stanard C library,system call, latency, Javainvocation, .

: 0xBench

Collect and Analyze results onserver-side

Android Functional Testing(1) stress test(2) Automated test

Stress Test According to CDD (Compatilbility DefinitionDocument), Device implementations MUST includethe Monkey framework, and make it available forapplications to use. monkey is a command that can directly talks toAndroid framework and emulate random user input.adb shell monkey p your.package.name v 500 Decide the percentage of touch events, keybordevents, etc., then run automatically.

ASTER: Automated Test Blackbox-test vs. Whitebox-test An easy to use automated testing tool.

Bring Enhancements back toCommunity

What do we deliver to communty? Patches merged in AOSP, CyanogenMod, andAndroid-x86 Implement 100% open source OpenGL ESadaptation based on Mesa/3D into Android– The world-first, important to Android-x86 Performance: ARM specific optimizations,graphics enhancement Features: Bluetooth HID (keyboard/mouse),external modem, 3D effects, customizedLauncher

Arithmetic on Nexus STune Dalvik VM performance (armv7)

2D on Nexus SApply extra performance tweaks against optimized build(NEON)

Benchmark: 2D (arm11-custom)

Benchmark: 3D (arm11-custom; no GPU)This explains that we have several system tools and development flowto help customers/community to verify the performance and improve.

Bionic libc Android C/C library 0xlab’s Optimizations (merged in Androidupstream)– Memory operations: Use ARMv6 unaligned access tooptimize usual cases– Endian/Data Type conversion: Use ARMv6 fast endianprimitives. Useful for TCP/IP (big endian / little endianconverting)– Various ARM optimized string operations memcpy, strcmp, strcpy, memset

Dynamic Linker Optimizations

Why and How? The major reason to optimize dynamic linker is tospeed up application startup time. Approaches: Implement GNU style hash support forbionic linker Prelinker improvements: incrementalglobal prelinking–reduce the number of ELF symbol lookupaggressively Changed parts– apriori, soslim, linker, elfcopy, elfutils

(normalized) Dynamic Link erbootanimation0

(normalized) Symbol Lookup ocmediaserver-0.2bootanimation0

DT GNU HASH: visible dynamic linking improvement Better hash function (few collisions) Drop unnecessary entry from hash Bloom filterlibc.soprintfvoid foo (){printf(“fooooo”);bar();}libfoo.sofoobarDT GNU HASHfoobarlibfoo.soDT HASHfoobarprintf

Symbols lookup#in ELFfail#gnu hash filtered by 342378(76%)H {x, y, z} hash functionsHash function may collision Bloom filter may got false positivesBit array

Bluetooth HID(Keyboard/Mouse)

UI customizations Provide several UI/Launcher combination forsmall and large screen devices. Sizing from HVGA, VGA, SVGA (Phone), to 720p/1080p (TV) Either modified Android Launcher or newreplacement Licensed under Apache Software License