WIXLIB

Cloud innovation and automationDevOps, Shift-Left & Self-HealingA Performance Clinic with AndiAndreas GrabnerDevOps Activist @ Dynatrace#Perform2018

Performance Clinics & More: From AppMon to DynatracePerformance Clinics: .com/demoshttp://dynatrace.ai/coffeebreakPodcast: http://bit.ly/pureperf

Shift-Left: Break Pipeline EarlierDevPerf/TestActionable Feedback LoopsShift-Right: Tags, Deploys, EventsOpsBizPath to NoOps: Self-Healing,

Shift-Right: Tags, Deployments & EventsReleaseAutomationDynatrace AutomationAPI, CLI, Auto-Detectiondocker run –e DT TAGS BLUEdtcli tag srv CartServicev2 GREENdtcli evt push pg tomcat1desc JVMMemIncr hint 100MBdtcli evt push host .*demoversion 123 source {git commit}Dynatrace Smartscape

AWS DevOps Tutorial: Defining Tags via Environment Variables#1 Copies to EC2 & Extracts#2 Executes start server.sh:export DT TAGS APPLICATION NAME APPLICATION NAMEexport DT CUSTOM PROP "DEPLOYMENT ID DEPLOYMENT ID DEPLOYMENT GROUP NAME DEPLOYMENT GROUP NAMEAPPLICATION NAME APPLICATION NAME“export DT CLUSTER ID " DEPLOYMENT GROUP NAME APPLICATION NAME“pm2 start app.js & pm2start.log#2 DT CLUSTER ID#2 DT TAGS#2 DT CUSTOM PROP

AWS DevOps Tutorial: Defining Service Tags via Naming Rule#2 – Call the Tag “DeploymentGroup”#3 – For Services#4– Tag Value comes from our Process Environment Variable#5 – Only apply rule if env variable exists#1

AWS DevOps Tutorial: Push CodePipeline Deployment Events to Dynatrace CodeDeploy Information from same StageDynatrace Events REST APIpushDynatraceDeploymentEventPushes CodePipelineInfo to Dynatrace

AWS DevOps Tutorial: Push CodePipeline Deployment Events to DynatraceMonspec (Monitoring as Code)"SampleJSonService" : {"etype": "SERVICE","environments" : {"Staging" : {"tags" : [{"context": "CONTEXTLESS","key": "DeploymentGroup","value": "Staging"}]},"Production" : {"tags" : [{"context": "CONTEXTLESS","key": "DeploymentGroup","value": "Production"}]}},pushDynatraceDeploymentEventPulls Tag Information from MonspecDynatrace Events REST APIPushes Event for Entities with Tag DeploymentGroup:Staging

Shift-Left: Continuous Performance :PERFmyservice:GREEN

Continuous Performance: Inspired by Dynatrace Performance Architect Team“Performance Signature”for Build Nov 16“Performance Signature”for Build Nov 17“Multiple Metrics”compared to prevTimeframeSimple Regression Detectionper Metric“Performance Signature”for every Build

“Monspec”: Performance Signature as Code for Continuous Performance ValidationEntity: SampleJSonServiceEnvironment: StagingTag: DeploymentGroup:StagingEnvironment: ProductionTag: DeploymentGroup:ProductionMetrics: ResponseTime (Avg)Metrics: ResponseTime(Max) 10sMetrics: FailureRate(Avg)Metrics: ResponseCountFactor: 90% Less vs ProductionCompare: StagingToProductionSource: Staging; Compare: ProdFactor: Staging 10% SlowerCompare: StagingToProdYesterdaySource: Staging; Compare: ProdFactor: 10%, Timeshift: 86400sCompare: StagingToStagingLastHourSource: Staging; Compare: StagingFactor: 0; Timeshift: 3600sCompare: StageToStageYesterdaySource: Staging; Compare: StagingFactor: 0; Timeshift: 86400sPipeline Run: Validate Build with StagingToProductionEnvironment: StagingEnvironment: ProductionSource (Staging):1.64msCompare (Production): 1.41msThreshold ( 10%):Status:1.56msVIOLATION

Run #7Run #6Run #5Run #4Run #3Run #2Run #1“Monspec”: Continuous Build Validation in Action across Pipeline runs!

“Monitoring as Code”: A look into monspec.jsonmonspec.json{"SampleJSonService" : {"etype": "SERVICE", // "Options are SERVICE, APPLICATION, HOST,PROCESS GROUP INSTANCE"Entity:What Entity do we want to validate?"name": "SampleNodeJsService","environments" : {"Staging" : {"tags" : [{"context": "CONTEXTLESS","key": "DeploymentGroup","value": "Staging"}]Environments: Tell us how we can detect},"Production" : {these entities in Dynatrace"tags" : [{"context": "CONTEXTLESS","key": "DeploymentGroup","value": "Production"}]}},

“Monitoring as Code”: A look into monspec.jsonmonspec.json{"SampleJSonService" : {."comparisons" : [{"name" : "StagingToProduction","source" : "Staging","compare" : "Production","scalefactorperc" : {"default": 10,"com.dynatrace.builtin:service.requestspermin" : 90},"shiftcomparetimeframe" : 0,"shiftsourcetimeframe" : 0,},{"name" : "StagingToProductionYesterday","source" : "Staging","compare" : "Production","scalefactorperc" : {"default": 10,"com.dynatrace.builtin:service.requestspermin" : 90},"shiftsourcetimeframe" : 0,"shiftcomparetimeframe" : 86400}monspec.json{"SampleJSonService" : {."comparisons" : [{"name" : "StagingToStagingLastHour","source" : "Staging","compare" : "Staging","scalefactorperc" : { "default": 0},"shiftsourcetimeframe" : 0,"shiftcomparetimeframe" : 3600},{"name" : "ProductionToProductionLastH","source" : "Production","compare" : "Production","scalefactorperc" : { "default": 0},"shiftsourcetimeframe" : 0,"shiftcomparetimeframe" : 3600}],

“Monitoring as Code”: A look into monspec.jsonmonspec.json{"SampleJSonService" : {.Metrics: Which Metrics, Aggregation, Upper/Lower Boundaries"perfsignature" : [{"timeseries" : egate" : "avg", // min, max, avg, sum, median, count, percentile"validate" : "upper", // upper or lower// "upperlimit" : 100, // Optional: Can be used to define a FIXED THRESHOLD// "lowerlimit" : 50, // Optional: Can be used to define a FIXED THRESHOLD},{"timeseries" : egate" : “p90"},{"timeseries" : gate" : "avg"},{"timeseries" : gregate" : "count",“validate" : "lower"}],

AWS DevOps Tutorial: Integrate into PipelineMonspec from S3Staging: Register Build Validation!Adds ItemregisterDynatraceBuildValidationAdds Build Validation RequestBuild Validation Request Item-Pipeline InformationMonspecTimestamp TimeframeComparison Definition NameUpdates Build Validation Request-Approves/Rejects IF “In Progress” & Comment Field containsname of RegisterBuildValidation ActionUpdated WorkCloudWatch Events(e.g: 1min)Dynatrace Entities & Timeseries REST APIResolves Tags and gets list of EntitiesQueries Metrics for these Entities

AWS DevOps Tutorial: Automating Approval Process

AWS DevOps Tutorial: Build Validation ReportOpen in Browser: tionResults

Path to NoOps: Auto-Remediation, Self-Healing, ?Escalate at 2AM?Auto Mitigate!4321CPU Exhausted? Add a new service instance!2High Garbage Collection? Adjust/Revert Memory Settings!3Issue with BLUE only? Switch back to GREEN!4Hung threads? Restart Service!?Impact Mitigated?5Still ongoing? Initiate Rollback!?Still ongoing?Update Dev TicketsMark Bad Commits15Escalate

AWS DevOps Tutorial: Automate Rollback with AWS Lambda and AWS CodeDeployPush Deployment Information, e.g:CodeDeploy DeploymentIdUses Dynatrace Events APIto pull CUSTOM DEPLOYMENT eventsCalling Lambda via API GatewayRedeploy previousrevisionhandleDynatraceProblemNotification

Additional DevOps Tricks!

Service Naming Rules: Distinguish services between Staging and Production#2#3DEPLOYMENT GROUP NAME/Detected ServiceName#1Only when DEPLOYMENT GROUP NAME exists

Here is the result! Nicer Service Names!

Log Rule Events: Automated Problem Detection based on custom Log Messages

Log Rule Events: Automated Problem Detection based on custom Log Messages

Log Rule Events: Automated Problem Detection based on custom Log Messages

Log Rule Events: Automated Problem Detection based on custom Log Messages

Log Rule Events: Automated Problem Detection based on custom Log Messages

Actionable Feedback Loops: BusinessNew Requirement DefinitionLabels become Key User ActionSuccess CriteriaLive Data Queries

Thank you

AWS DevOps Tutorial: Push CodePipeline Deployment Events to Dynatrace pushDynatraceDeploymentEvent Pushes CodePipelineInfo to Dynatrace Dynatrace