Transcription
State of theOSU Open Source LabLance Albertson, Directorlance@osuosl.org / @ramerethhttps://osuosl.org @osuoslState of the OSU Open Source LabLance Albertson @ramereth
Session Overview1.2.3.4.5.OverviewOSL StudentsCurrent and New ServicesInfrastructure EnhancementsGoals for this yearhttps://osuosl.org @osuoslState of the OSU Open Source LabLance Albertson @ramereth
Overviewhttps://osuosl.org @osuoslState of the OSU Open Source LabLance Albertson @ramereth
Summary FOSS “Hosting Company”–––––Free or low cost hosting services for FOSS projectsCo-LocationVirtual Machines and other private cloud servicesAccess to a wide array of architecturesSoftware distribution and mirroring Mentor undergraduate students in DevOps–––Gain real-world experience on production systemsPast graduates include co-founders of CoreOSMany have key roles in high profile tech companies Staff––https://osuosl.org @osuoslDirector6-10 undergraduate studentsState of the OSU Open Source LabLance Albertson @ramereth
History Started in 2003––Scott Kveton & Jason McKerr co-foundersOSU Information Services Offered Co-Location hosting for FOSS––Gentoo, Debian, and Freenode were early projectsGrowth spread via word of mouth among projects kernel.org, ASF, Drupal, Linux Foundation Seed funding––OSU initial fundingGoogle & RealMedia initial sponsors Organizational Changes––Moved to College of Engineering in 2013Unit of the Center for Applied Systems & Software(CASS) https://osuosl.org @osuoslPartnered with another experiential program on campus whose focuswas software developmentand hardware testingState of the OSU Open Source LabLance Albertson @ramereth
Funding Model Corporate donations–––IBMGoogleFacebook Hosting contracts–––Linux FoundationDrupalOpen Source Robotics Foundation In-Kind donations––TDS (bandwidth for FTP mirroring)Intel, Facebook, Hudson Trading (hardware donations) Support from you! (Thank you!) No direct funding from OSU / State of Oregonhttps://osuosl.org @osuoslState of the OSU Open Source LabLance Albertson @ramereth
Our role in the FOSS Ecosystem Provide a neutral hosting facility–Foster relationships between FOSS and businesses Stable physical home for core FOSS––Free hostingFlexible to the needs of projects Access to unique hardware and architectures–OpenPOWER, ARM, RISC-V, MIPS, etc Compute and storage resources––Software mirroringCI/CD compute resources Systems Engineering expertise–Helping projects focus on improving their project Training the next generation of FOSS leadershttps://osuosl.org @osuoslState of the OSU Open Source LabLance Albertson @ramereth
New Projects in 2018/2019General Hosting MuttFreeBSD FoundationMX LinuxPentoo LinuxGNU RadioOhio LinuxFestLyXCVPM (Computer Vision PackageManager)Academic TorrentsPersonal Telco ProjectF-DroidOpenPOWER Hosting OpenPOWER Hosting s://osuosl.org @osuosl IstioTensorFlowLLVMDockerOpenCVTasseract OCRgcc-gfortranx265Apple SwiftLSBFedoraMozillaFFMpegGHCWebMNumPyGo LangEnvoyRocksDBState of the OSU Open Source LabLance Albertson @ramereth
Current Projects stool-NGF-DroidGrmlLFS (Linux From Scratch)Apache Software Foundation CVPMFedora LinuxGSB (Gnome.SlackBuild)libjpeg-turboApereoCygwinFedora RISCVHackagelibpngArch LinuxDarcsFFmpegHadoop SparklibvpxBearSSLDebian LinuxFinnixHaikuLineageOSBlockchainDebian RISCVFirefoxHarmony ChixBroDelugeFOSS TranslationHortonworks Data PlatformLinux Driver ProjectBuildbotDockerFreedesktop.orgICULinux FoundationBuildrootDrupalFreedroidInkscapeLinux FundBusyBoxEclipse CommunicationFrameworkFuntoo LinuxIstioLinux KernelGCC/ClangJawsLinux Plumbers ConferenceGCC Compile Farm ProjectJellyfishLinux Standard BaseGCC FortranJenkinsLLVMGentoo Linuxjuju-charmsLTTngGinga MiddlewareJupyter / Base-notebookLuneOSGnomeK-3DLyXGNU RadioKDEMageiaGo LanguageLAPACKMandriva UsersCactiCentOS LinuxCheckpoint-restoreCiviCRMCloud FoundryCPANCppcheckCRANEclipseEclipse OMREclipse OpenJ9Elastic BeatsElephant ShedElggeLinuxELRepohttps://osuosl.org @osuoslState of the OSU Open Source LabLance Albertson @ramereth
Current Projects (M-Z)Manjaro LinuxNTPSECOpenWRTReproducible buildsuClibcManulixNumPyOrientDBRobot Operating SystemVector LinuxMariaDBNvidia-dockerOSDVRPM FusionVideoLANMesosOcamlOSGeoRPMVSXSIMDMidnight CommanderOCMPA-RISCRTEMSWebDAVMonkey ServerOFTCParrotRustX265 HEVC EncoderMozdevOpen AdvicePentoo LinuxSahana FoundationX2goMozillaOpenBLASperfbookSalix eSQLSnowpatchYumMusicbrainzOpen HatchPowerPC NotebookSwiftZarafaMuttOpenIDPrestoTeaching Open SourceMX LinuxOpenJDKPrivlyTesseract-OCRMycroft hon Software Foundation The DocumentationFoundationPyTorchTobin's Linux kernelQEMUTravis-CIqiskit-sdk-pyTrinity Rescue KitRainmeterTrisquel LinuxReplicantUbuntuhttps://osuosl.org @osuoslState of the OSU Open Source LabLance Albertson @ramereth
OSL Studentshttps://osuosl.org @osuoslState of the OSU Open Source LabLance Albertson @ramereth
Alumni CoreOS–Alex Polvi, Brandon Phillips Linux Foundation–Eric Searcy, Rudy Grigar, Trevor Bramwell Microsoft–Sarah Cooley Many other companies such as:–https://osuosl.org @osuoslAWS, Tesla, Mozilla, Redhat, Puppet, Zillow, Apple,LinkedIn, Tag1 Consulting, etcState of the OSU Open Source LabLance Albertson @ramereth
Students Role Interact with FOSS projects on daily basis––Deploy websites, troubleshoot issuesOnboard new projects Chef cookbook creation and maintenance––Creating new wrapper cookbooks for servicesFixing and updating current cookbooks Hands-on experience––Installing and (un)racking new/old hardwareTroubleshooting real-world problems Support ticket queue rotation––https://osuosl.org @osuoslEach student is in charge of every ticket that comes induring that weekEnsures each student is knowledgeable about the all ofour systemsState of the OSU Open Source LabLance Albertson @ramereth
Hiring process Open book quiz––––Asks basic questions about LinuxSimple bash exercisesSimple Chef exerciseEnsures applicants are passionate enough to complete aquiz In person interview–Non technical questions (⅓) –Technical questions (⅔) https://osuosl.org @osuoslGet a better understanding of their personalitySee if they would make a good fit with the teamWide range of questions from simple to more complicatedDon’t expect the students to know all the answersTry and assess their ability to problem solve on the flyState of the OSU Open Source LabLance Albertson @ramereth
Onboarding process Walkthrough guide––––Account setupHow to contribute to internal documentationBasics of LinuxBasics of Chef Chef onboarding––––Exercise with a test cookbookWalks them through typical scenarios during cookbookdevelopmentCreate a PR and wait for feedbackSenior students provide feedback and work through Assign simple tasks initially–https://osuosl.org @osuoslWithin 2-3 months, add them to the ticket rotationscheduleState of the OSU Open Source LabLance Albertson @ramereth
Current and New Serviceshttps://osuosl.org @osuoslState of the OSU Open Source LabLance Albertson @ramereth
OSL Managed Platform Current and new systems–––CentOS 7/6 (servers)Debian 8/9 (for student and staff workstations)Managed via Chef Chef––Wrapper cookbooks using community cookbooksFull unit and integration testing –ChefSpec Test Kitchen InSpec Awesome!Jenkins pipeline to automate testing and deployment Legacy systems––https://osuosl.org @osuoslCentOS 6 or Gentoo LinuxManaged via CFEngine 2.xState of the OSU Open Source LabLance Albertson @ramereth
OSL Hardware Hardware Budget? In-Kind donations––Rely mostly on In-Kind donationsIntel (Meego) hardware (2012) –EMC (Neutrino) hardware (2016) –Arista 2x 1g/10g switches8 x 4-node compute nodesFacebook (2016) –Dell R610, R710 machines3 OCP Racks with total of 90 compute nodesHudson Trading (2018) “Pallets” of Arista 10g switches Wish list–––https://osuosl.org @osuosl1U/2U compute / storage nodes 3TB SATA HDD’s / SSD’s40g End-row switches / 1g ToR switchesState of the OSU Open Source LabLance Albertson @ramereth
Core Infrastructure Services Mailing list–––200 lists currently hostedMailman v2.x basedShared instance Email forwarding––Includes spam and virus filteringEmail store is available if required DNS Web application hosting Systems Engineering consulting for projectshttps://osuosl.org @osuoslState of the OSU Open Source LabLance Albertson @ramereth
Managed vs. Unmanaged Hosting Managed–––––Operating System and its updatesConfiguring and managing services (httpd, etc)Infrastructure designMonitoring and remediationAll aspects managed with Chef Unmanaged––https://osuosl.org @osuoslWe spin up a host and you manage all aspectsWe only require an account with full sudo fortroubleshooting and emergenciesState of the OSU Open Source LabLance Albertson @ramereth
Software Mirroring Three server cluster––Hosted in Corvallis, Chicago and New YorkSplit via a round-robin DNS Stats–––1.7Gbps daily average across all three nodes15 TiB capacity / currently using 12 TiB100 repositories/projects hosted Hardware Specs–––256G RAMIBM POWER8 8286-42AStorage ––https://osuosl.org @osuoslTiered SAS & SSD RAID610Gib uplinkDonated by IBM (Thank you!)State of the OSU Open Source LabLance Albertson @ramereth
Co-Location Hosting 300 Co-location hosts for projects Project Racks–Gentoo, Linux Foundation, Drupal, Apache SoftwareFoundation, etc Projects own their hardware and ship it Requirements–––––A reasonable need for having a physical server instead ofusing a virtual machineRack mountable server that includes railsOut-of-Band management via IPMI or serialBuilt by a vendorExceptions made for special architectures OSL Datacenter––https://osuosl.org @osuoslShared with UniversityOSL uses around 32 out of 70 racksState of the OSU Open Source LabLance Albertson @ramereth
Storage: Ceph Two clusters–––5 node cluster for OpenPOWER OpenStack only8 node cluster for x86 OpenStack and other OSL servicesBoth deployed in 2018 x86 Cluster Specs––––––Nautilus (12.x.x)100 TiB raw capacity52 x 2 TB SATA drives (6 or 8 per node)8 x 800 GB NVMe SSD drives (1 per node)Bluestore with SSD’s for metadata caching10g networking OpenPOWER Cluster Specs–––https://osuosl.org @osuosl295 TiB raw capacity40 x 8 TB SATA drives40g networkingState of the OSU Open Source LabLance Albertson @ramereth
Storage: Ceph Current uses––Block storage for OpenStackCephFS (replaced GlusterFS) Future plans––––https://osuosl.org @osuoslObject storageExpanding / Upgrading hardwareBlock storage for Ganeti (maybe?)Geo-Replication?State of the OSU Open Source LabLance Albertson @ramereth
Private Cloud PlatformsTale of two platforms Ganeti––––Stable and easy to maintainLittle to no public APIDeployed since 2009Poor at self service OpenStack––––https://osuosl.org @osuoslMostly stable and difficult to maintainExcellent public APIDeployed since 2013Awesome at self serviceState of the OSU Open Source LabLance Albertson @ramereth
Private Cloud: Ganeti Been using Ganeti since 2009–– Powered with KVM and DRBD on local storagePrimarily CLI driven, minimal GUI and no public APIProduction cluster–––––120 VMs ( 15 VMs per node)8 nodesStorage: 25 TB total / 10.6 TB usedRam: 500 GB total / 350 TB usedProjects: phpBB, Busybox/Buildroot, ROS, Jenkins, qemu, etc Project Clusters–PSF, OSGeo, CiviCRM, OSL Internal Current uses––https://osuosl.org @osuosl“Pet” long-running VMsImportant traditional servicesState of the OSU Open Source LabLance Albertson @ramereth
Private Cloud: OpenStack OSL’s adventure with OpenStack–Been using it internally since around 2013 ––Test-Kitchen (Integration testing with Chef)Created ppc64le based cluster firstx86 cluster opened for projects in 2018 Specs (x86 cluster)––Powered with KVM and Ceph for storage50 VMs on 8 compute nodes 128 GB RAM, 10G networking Projects–https://osuosl.org @osuoslAcademic Torrents, Ohio LinuxFest, GNU Radio, Foreman,glibc, SharkLinux, AnthonLinux, CVPM, Linux Foundation,GNOME, FreedroidRPG, GCCState of the OSU Open Source LabLance Albertson @ramereth
OpenPOWER Collaboration with IBM for over ten years FOSS access to ppc64/ppc64le architecture OpenStack powered cluster on POWER9/8––5 x POWER8 systems ( 225 VMs)3 x POWER9 systems ( 22 VMs) 100 projects using the cluster–Projects: –Docker, LLVM, Travis CI, Alpine, GHC, VideoLan, Go, Node.js,OpenJDK, LTTng, RockDB, Nvidia Docker, PostgreSQL, etcMany of the ppc64/ppc64le binaries you use were likelybuilt on this cluster! Funded and supported by IBM Also host bare metal machines for:–https://osuosl.org @osuoslGCC Compile Farm, Debian, FreeBSDState of the OSU Open Source LabLance Albertson @ramereth
OpenPOWER GPU GPU hardware access for FOSS projects––OSL OSU Center for Genome Research andBiocomputing (CGRB)CGRB manages the hardware How projects access the GPU hardware–Sun of Grid Engine –HPC scheduling software managed by the CGRBOSL managed Jenkins portal (https://powerci.osuosl.org) Uses Nvidia Docker to get access to the hardware Upcoming changes–Incorporating OpenStack Zun to provide shell access toGPU resources via containers https://osuosl.org @osuoslSharing via VM not feasible due to pci-passthru limitationsState of the OSU Open Source LabLance Albertson @ramereth
Other IBM resources IBM-Z (s390x)–––Two LPAR’s hosted by Marist College in NYJenkins CI Portal (https://ibmz-ci.osuosl.org)Docker images AIX––https://osuosl.org @osuoslProvide hosting but don’t manage the resourcesAccess to select FOSS projects for building/testing on AIXState of the OSU Open Source LabLance Albertson @ramereth
Infrastructure Enhancementshttps://osuosl.org @osuoslState of the OSU Open Source LabLance Albertson @ramereth
Major projects in the past year Chef upgrades and cleanupCFEngine to Chef migrationCephOpenStack upgradesOpen Compute compile farmIPv6Monitoring and metricsLetsEncrypthttps://osuosl.org @osuoslState of the OSU Open Source LabLance Albertson @ramereth
Chef projects 130 hosts managed Upgrades–––Chef 13 upgrade completed last yearChef 14 upgrade starting soonUpdated to latest community cookbooks Testing––Switched to using cookstyle instead of rspecStarting migration towards InSpec from ServerSpec Improvements–––https://osuosl.org @osuoslProper peer reviewBetter test coverageUsing better coding standardsState of the OSU Open Source LabLance Albertson @ramereth
CFEngine to Chef Migration 20 Remaining Legacy systems Started migration to Chef in 2013 Major services remaining that need to bemigrated–––––––Email relays (4)Mailman (1)Mysql cluster (2)PostgreSQL server (2)Misc managed project VMs ( 5-7)VPN/NAT gateway server (1)Other misc OSL hosts ( 2) Hope to finish the migration by the end of2019!!https://osuosl.org @osuoslState of the OSU Open Source LabLance Albertson @ramereth
OpenStack upgrades Software–––Completed upgrade to Ocata in December 2018Pike upgrade scheduled for the end of MarchPlan to continue until running latest stable release Storage–––Migrated from local storage to CephBig performance jump and also allowed for live migrationExpanded storage capability Upcoming changes–––https://osuosl.org @osuoslAdding support for IPv6Enabling additional services for K8s supportImprove DNS integrationState of the OSU Open Source LabLance Albertson @ramereth
Open Compute Compile Farm Hardware donated by Facebook Compute node specs:–140G RAM, single 3TB SATA disk, 10g NIC GCC Compile Farm–Connected with other projects who needed raw CPUpower Projects using this:–OpenWRT, LineageOS, VLC, Reproducible Builds, RISC-V(Debian/Fedora), F-Droid, GNOME, Freedesktop.org,Buildroot, OpenBSD, GCC Compile Farm 59 of 90 nodes allocatedhttps://osuosl.org @osuoslState of the OSU Open Source LabLance Albertson @ramereth
Open Compute Compile Farm OCP logistics–Rack height –Power –7ft (newer models are taller)Elevator and network cage issues277V 3-Phase AC / 48V DCNetwork switches with DC PSU’sFirmware upgrades BIOS chip removable and burnable HVAC limitations––Seeking 150k to upgrade coolingNeed your help to fund this! Networking logistics––https://osuosl.org @osuoslOn OSU network instead of OSL networkLimited IPv4 space - No IPv6 (yet)State of the OSU Open Source LabLance Albertson @ramereth
IPv6 Deployed in 2016–2605:bc80:3010::/48 OSL Public services using IPv6:–––DNSFTP MirroringShared websites (using our load balancer) Deployment–Static IP address assignment –Dual-Stack –https://osuosl.org @osuoslISP Recommended this configurationEach subnet gets their own /64Per-project /56 reservationsState of the OSU Open Source LabLance Albertson @ramereth
Monitoring and metrics Prometheus & Grafana––Deployed in Jan 2019https://dashboard.osuosl.org Setup exporters for the following:––node exporter (all the CentOS 7 systems)snmp exporter –Switches, PDU and Environmental sensorsopenstack exporter Working on deploying more exporters–––apache, haproxy, ipmi, etcIntegration with OpenStack Ceilometer / GnocchiCustom metrics (WIP) https://osuosl.org @osuoslFTP project disk usageDatabase disk usageBackup project disk usageState of the OSU Open Source LabLance Albertson @ramereth
LetsEncrypt Managed with Chef–Easy for single hosts HAProxy with failover––Currently using an NFS-based solutionWorks but not happy with the implementation LetsEncrypt support on ftp.osuosl.org?–––https://osuosl.org @osuoslNot yet (sadly)Currently has a wildcard *.osuosl.org certLooking at refactoring HAProxy solutionState of the OSU Open Source LabLance Albertson @ramereth
Other misc projects RANCID–Network switch configs in git 10g backend network upgrades––––– Ganeti DRBDCephOpenStackNew VPN serverBackup serverMigrated from djbdns to BindRazor Server (Netboot installations)Increased FTP storage capacityFirmware upgrades–https://osuosl.org @osuoslThank you Meltdown/Spectre/etcState of the OSU Open Source LabLance Albertson @ramereth
Goals for the next year Complete legacy system migrationsUpgrade OpenStack & Ceph to latest stableUpgrade to Chef 15Migrate Ganeti to CephDeploy a proper ELK stack–Integrate more metrics into Grafana Start replacing aging OSL Network core Start working on CentOS 8 migration? Migrate CentOS 6 hosts–Upgrade to 7 (or 8 when ready) Deploy an ARM OpenStack cluster?https://osuosl.org @osuoslState of the OSU Open Source LabLance Albertson @ramereth
Thank you!Lance Albertsonlance@osuosl.org / @ramerethhttps://osuosl.org @osuoslState of the OSU Open Source LabLance Albertson @ramereth
– EMC (Neutrino) hardware (2016) Arista 2x 1g/10g switches 8 x 4-node compute nodes – Facebook (2016) 3 OCP Racks with total of 90 compute nodes – Hudson Trading (2018) “Pallets” of Arista 10g switches Wish list – 1U/2U compute / storage nodes – 3TB SA
