Transcription
Cerner Case StudyHow Atlassian Data Center Helped One HealthcareIT Company Manage Scale
How Atlassian Data Center HelpedOne Healthcare IT CompanyManage ScaleWhen Vice President Brian Wallace and Knowledge Architect Mike Dammanfaced rapid scaling of their Atlassian tools at Cerner - the leading Americansupplier of information technology - they knew they would need a strategyto meet the needs of their growing teams.What do you do when your Jira Software instance grows from 10 to 10,000 users? This was what Brian Wallace and Mike Damman, Vice Presidentand Knowledge Architect at Cerner, the leading U.S. supplier of healthcareinformation technology, needed to answer in order to meet the needs oftheir growing teams.How do you guarantee reliability across such a large instance? What canbe done to mitigate the effects of downtime? These were some of thequestions I tackled with the Cerner team while we identified challenges andcame up with some great solutions.How to scale Jira Software to 10,000 users 2
Challenge 1: Scale Jira SoftwaregloballyCerner had three federated instances of Jira Software Server with thousandsof developers using each instance every hour of every day around the globe.Jira Software quickly became mission-critical and every minute of downtimeor performance degradation made it more difficult for Cerner teammembers to support their customers. They needed a solution that providedhigh availability.In the fall of 2015, Cerner chose to upgrade from Server to Data Centerso that they could cluster multiple active servers and provide users withuninterrupted access to Jira Software.This wasn’t just critical at the time, but knowing that they were going toadd several thousand more users in the coming year, they needed a solutionthat would scale with them.How to scale Jira Software to 10,000 users 3
Challenge 2: Risks to high availabilityUsing Zabbix and Splunk to monitor their Jira instances, Cerner was able toidentify one area that needed to be addressed immediately if they wantedto provide true high availability: REST API abuse. Their log analysis showedthat team members were using the REST API to get real-time statusupdates – so whether teams knew it or not, they were pinging Jira Softwareinstances every single second.Cerner didn’t want to restrict users from creating custom dashboards orself-serving the data they needed, but it was obvious that they had to dosomething different.“We wanted to be able to isolate REST calls to a single serverso that it didn’t have an impact on other users”MIKE DAMMANVice President and Knowledge Architect at CernerWith a multi-node cluster they could intelligently distribute traffic bydedicating one node solely to external REST API requests.Cerner also wanted to guarantee that all external requests went to thisdedicated node because having users manually change the domain to an IPaddress, or another domain, wasn’t reliable.That’s when they reached out to their Technical Account Manager (TAM) inAtlassian to help them come up with a better solution.How to scale Jira Software to 10,000 users 4
An Atlassian Technical Account Manager (TAM) is an experienced solutionsadvisor from Atlassian that unlocks the potential of your tools, improvesteam collaboration throughout your organization, and helps you operateand use our products effectively at e impact and rate of growth for external integrations (robot) differ fromhuman interactions and can add stress to individual nodes in the cluster,bringing down everyone else using that node.How to scale Jira Software to 10,000 users 5
Solution: Intelligently route trafficCerner needed the Data Center configuration to ensure all external RESTAPI requests were routed away from other traffic. They planned to have fournodes in their cluster behind a load balancer with each node performing thefollowing services:1. Node 1 – External REST API node2. Nodes 2 & 3 – Normal usage nodes3. Node 4 – Admin and power user node; not in the loadbalancer and only accessible by IP addressLoad Balancer1234How to scale Jira Software to 10,000 users 6
The Atlassian Technical Account Manager originally thought that the bestoption would be to use the load balancer to route all requests with ‘/rest’ tothe REST API node.However, he soon found the REST API was also being used throughout JiraSoftware, including the login page, so leaving it to ‘/rest’ would mean wewould still be mixing REST API traffic with normal usage. Cerner needed abetter solution.The TAM found that he could isolate REST API requests by looking for‘/rest’ in each request AND by looking at where the request originated usingthe HTTP referer header. If a person was trying to login to Jira Software orwas already using Jira Software, they would get directed to or remain ona Normal Usage node. Otherwise, if the person or bot was requesting theREST API, they would get directed to the REST API node.Atlassian TAM proposed the solution to Cerner and after a few rounds oftesting, they went live with Jira Software Data Center in October 2015.How to scale Jira Software to 10,000 users 7
The results: Performance at scaleWithin their first week of implementing the proposed Data Centerconfiguration, Cerner was seeing 4 times the amount of traffic on the RESTAPI node as on the other two nodes.Response times are faster, CPU utilization has decreased across theirnon-admin nodes compared to a single server instance, and they haven’tseen a single unplanned outage in 2016, all while scaling Jira Software tothousands of new users.How to scale Jira Software to 10,000 users 8
Response timeCerner needed to make sure that as they continued to add users thatapplication responses times maintained or improved. This re-architectingproved that Cerner was able to reduce their response time by nearly half,from 150ms to 80ms. Even at peak traffic times – looking at page loadsspecifically – response times remained steady.RESPONSE TIME and USAGEResponse timeGo live dayUsage (page loads)Jan 27, 2016Jan 28Jan 29Jan 30Jan 31Feb 1Feb 2AVERAGE RESPONSE TIME500ms400BEFORE DATA CENTER300AFTER DATA CENTER2001000Oct 6, 2016Oct 7Oct 8Oct 9Oct 19Oct 20Oct 21Oct 22How to scale Jira Software to 10,000 users 9
Unleash your team of teams withAtlassian. Scale with confidence,simplify complexity, and moveyour business forward withAtlassian Data Center - offeredfor Jira Software, Confluence,Bitbucket, Jira Service Desk, andCrowd.To learn more aboutAtlassian Data Center,contact your AtlassianSolution Partner today.Atlassian’s full set of enterpriseservices and strong partnernetwork aid and support yoursuccess as you scale.How to scale Jira Software to 10,000 users 10
Load Balancer. How to scale Jira Software to 10,000 users 7 The Atlassian Technical Account Manager originally thought that the best option would be to use the load balancer to route all requests wi
