Transcription
Getting Out of Quicksand, WithDevOps!Roman Pickl20.03.2019Join at Slido.com with#devops2019
Wait a moment? How high workload kills efficiency!How did I end up here?Roman Pickl (@rompic) Technical Project Manager @ Elektrobit Former CTO @ Fluidtime (now a Kapsch company) Loves CI/CD/DevOps Here to learn Elektrobit (EB) 20182
Wait a moment? How high workload kills efficiency!Andrew Shafer, DevOpsDays Cairo 2018https://youtu.be/BeNdum6u6eg?t 2081 Elektrobit (EB) 20183
Wait a moment? How high workload kills efficiency!Efficiency vs. EffectivenessEfficiency vs. EffectivenessTom DeMarco, 2001, Slack: Getting Past Burnout, Busywork, and the Myth of Total Efficiency Elektrobit (EB) 2018By Xwizard35 - Own work, CC BY-SA id 58355021
Wait a moment? How high workload kills efficiency!How the team currently looks like100 13090Symptoms of overload Decreased team morale Working long hours / when sick More frequent illness Unhealthy tasks queue Imbalanced metrics Elektrobit (EB) 20185
Wait a moment? How high workload kills efficiency!It hurts again and again Elektrobit (EB) 20186
Wait a moment? How high workload kills efficiency!No thanks, too busySource unknown Elektrobit (EB) 20187
Wait a moment? How high workload kills efficiency!What your manager thinks should be done100 Elektrobit (EB) 20181001002020From anotherteam with aproject onhold8
Wait a moment? How high workload kills efficiency!Queuing theory Different types of queues (e.g. G/G/1)Wait Time (%Busy)/(% Idle) Waiting Time:20– Service time1816– Variation in process and arrival14 Kingman Equation / approximationWait time– Utilization121086 Little’s Law:– Avg. Lead Time Avg. WiP / Avg. Throughput Focus on throughput rather than utilization4205% 10% 15% 20% 25% 30% 35% 40% 45% 50% 55% 60% 65% 70% 75% 80% 85% 90% 95%https://en.wikipedia.org/wiki/Kingman%27s formulahttps://less.works/less/principles/queueing theory.html Elektrobit (EB) 20189
Wait a moment? How high workload kills efficiency!Feels like quicksandThe more you fight it, the more it pulls you in.Source: nd Elektrobit (EB) 201810
Wait a moment? How high workload kills efficiency!What you think should be done: hire more people80808080“slack is the missing ingredient required for all change”Invest roughly a day per week to- Spur innovation- Rethink- Practice new ways- Master new skills- Improve efficiency.Scrum: sustainable pace3Ms 15% timeGoogle’s 20% timeHard limit on toil at Google SRE: 50%#2 SRE principle: SREs must have time to make tomorrowbetter than today Elektrobit (EB) 201811
Wait a moment? How high workload kills efficiency!What actually happens„People under time pressure don‘t think faster“ - Tim ListerYOU Elektrobit (EB) 201812
Wait a moment? How high workload kills efficiency!It sometimes takes a crisisKC Green Elektrobit (EB) 201813
Wait a moment? How high workload kills efficiency!Quantify the work: Activity Accounting2018 Accelerate State of DevOps Report: ps.html Elektrobit (EB) 201814
Wait a moment? How high workload kills efficiency!MeasureFlow time– Lead time– Cycle timeFrequency of deploymentMean time to restore servicesWiP is the biggest deterrent to flowWhat is the biggest bottleneck?Can we fix the bottleneck?Ask why (5 times),Hydrargyrum Elektrobit (EB) 201815
Wait a moment? How high workload kills efficiency!XKCD - Compilinghttps://xkcd.com/303/ Elektrobit (EB) 201816
Wait a moment? How high workload kills efficiency!Lightweight Capacity Management Tendency to do more planning / estimation and more control Accept uncertainty and appreciate flexibility Decreasing marginal utility of each hour spend planning Every hour spent planning is an hour not delivering (asresources often overlap) Lightweight capacity planning Just-in-time approach Prioritize e.g. by cost of delayA Practical Approach to Large-Scale Agile Development – Gruver, Young, FulghumPrinciples of Lean Product Managment by Jez Humble - https://youtu.be/cH6bnQzJojo?t 1083"A Practical Approach to Large Scale Agile Development" - Gary Gruver at Spark 2013https://www.youtube.com/watch?v 2QGYEwghRSM Elektrobit (EB) 201817
Wait a moment? How high workload kills efficiency!Do less (at the same time) Reduce WiP Reduce batch size Reduce variability / increase predictability Do less Self service platform Eliminate before you try to automate- Less is More Elektrobit (EB) 201818
Wait a moment? How high workload kills efficiency!Automate Make time to create automation Documenting the process and making it repeatable is the firststep Automate, eliminate or engineer out the drivers that are notkey to the value proposition– New work – hard to automate– Triage work – automate for consistency– Repetitive work – automate Fix the biggest bottleneck firstPicture: hen/ Elektrobit (EB) 201819
Wait a moment? How high workload kills efficiency!Left-Over PrincipleHireRareDocumentCreate ToolsManualEliminateEasyAutomateThe Practice of Cloud System Administration Elektrobit (EB) 2018DifficultAcquireFrequent20
Wait a moment? How high workload kills efficiency!A word of cautionThe paradox of automationDemarco T, Slack: Getting Past Burnout, Busywork and the Myth of Total Efficiency, page 108ffForsgren N., Humble J., Gene K. Accelerate p. 12 ffAccelerate State of DevOps Report 2018 p. 28ff Elektrobit (EB) 201821
Wait a moment? How high workload kills efficiency!Virtuous circle“Improving your software delivery effectiveness will improve your ability to work in smallbatches and incorporate customer feedback along the way”“Lean product management practices positively impact software delivery performance,stimulate a generative culture, and decrease burnout”“[ ] software delivery performance drives Lean product management practices”“[ ] it becomes a reciprocal model or, colloquially, a virtuous cycle” – Accelerate Elektrobit (EB) 201822
Wait a moment? How high workload kills efficiency!Improvement Katahttp://www-personal.umich.edu/ mrother/Homepage.html Elektrobit (EB) 201823
Wait a moment? How high workload kills efficiency!Prioritization of ImprovementHardPriorityPriority42Low ImpactThe Practice of Cloud System Administration Elektrobit (EB) 2018High ImpactPriorityPriority31Easy24
Wait a moment? How high workload kills efficiency!Virtuous CircleMore release / testautomation /improvementsFaster releases / moretime for new workMore frequent releases /smaller batches / less WiPMore time to automate /test / lower utilization /decreased burnout /higher job satisfactionFewer bugs per release /faster feedback/ higherconfidence / lowervariability and reworkFewer changes perrelease / lower lead timeBased on Virtuous Circle of CI/CD The Site Reliability Workbook Elektrobit (EB) 201825
Wait a moment? How high workload kills efficiency!How it ended up working:808080Spent 1 day a week to improve “something”Created self service pipelinesUsed Information radiatorsClear priorities and weekly goals Elektrobit (EB) 201826
Wait a moment? How high workload kills efficiency!ConclusionCrisis situations are opportunities for changeOften it takes fresh eyes or at least courage to see the problemsGet out of the quicksand, the sky is the limitQuantify the work and set goalsMeasureLess is moreAutomate the right thingsEstablish a continuous improvement processStart a virtuous circle Elektrobit (EB) 201827
Wait a moment? How high workload kills efficiency!Read these if you want to know more Elektrobit (EB) 201828
Wait a moment? How high workload kills efficiency!Dominica DeGrandis – Making work visibleWe don‘t let our servers get to 100%capacity utilization, so let‘s not do thatto ourselves Elektrobit (EB) 201829
Wait a moment? How high workload kills efficiency!Further references & information Daniel Vacanti: Little‘s (F)law: https://vimeo.com/52683659 Håkan Forss, Queueing theory in software development - ALEBathtub 2011-06-30, y-insoftware-development-ale-bathtub-2011-0630 GOTO 2015 Why Scaling Agile Doesn't Work Jez Humble - https://www.youtube.com/watch?v 2zYxWEZ0gYg&feature youtu.be&t 3003 (itoften takes a disaster for lasting change) John Cutler – Too Busy to Improve -Improve-for-Roman-e2ap3i 2018 Accelerate State of DevOps Report: ps.html Dominica DeGrandis - Making Work Visible: How to Unmask Capacity Killing WIP - https://www.youtube.com/watch?v KR7Y8IUgyyA Keynote: Continuous Delivery Sounds Great By Jez Humble @ Agile India 2017 https://www.youtube.com/watch?v SjVV3xuYKJs A Practical Approach to Large Scale Agile Development"- Gary Gruver at Spark 2013 https://www.youtube.com/watch?v 2QGYEwghRSM High-Performance Team, Management, and Leadership Behaviors and practices https://devopsresearch.com/assets/transformation practices.pdf John Cutler - This feels like going faster vs. this actually makes us faster 6895720449 John Cutler – Slack vs. Blocked 6c294938b1 Mike Rother - http://www-personal.umich.edu/ mrother/Homepage.html Elektrobit (EB) 201830
Don’t wait and get in touchRoman Pickl com
"Lean product management practices positively impact software delivery performance, stimulate a generative culture, and decrease burnout" "[ ] software delivery performance drives Lean product management practices" "[ ] it becomes a reciprocal model or, colloquially, a virtuous cycle" -Accelerate Virtuous circle Wait a moment?
