Guide To Selecting A Workflow Orchestration Product - BMC

Transcription

PeerPaper ReportGuide to Selectinga Workflow OrchestrationProductBased on real user reviews of Control-M from BMC

ABSTRACTAs companies seek digital transformation, they are asking IT to implement increasinglycomplex workflows that span multiple platforms, data sources and developmentmethodologies. To handle these requirements, IT departments are turning to workfloworchestration tools. Though not new, these tools have grown more sophisticated inrecent years. Modern best practices and selection criteria are emerging. This paperexplores how IT managers can leverage these practices and selection tips to getthe best workflow orchestration results. It is based on reviews by IT Central Stationmembers, sharing their experience with Control-M, BMC’s workflow orchestrationproduct. 2019, IT Central StationGuide to Selecting a Workflow Orchestration Product

CONTENTSPage 1.IntroductionPage 2.A Brief Overview of Workflow OrchestrationPage 3.Workflow Orchestration Use CasesPage 4.Benefits of Workflow OrchestrationPage 5.Best Practices and Product Selection FactorsPay Attention to Non-Technical/Organizational IssuesTake Advantage of NotificationsWork Across PlatformsLeverage Functionality for Flexibility, Ease of Use and Self-ServiceConsolidate Solutions When NecessaryLook for Remote File Management CapabilitiesAutomate As Much As PossibleDo Maximal MonitoringSeek ReliabilityPage 10.Implementation TipsPage 1 1.Upgrade, Licensing and Pricing TipsPage 12.Conclusion 2019, IT Central StationGuide to Selecting a Workflow Orchestration Product

INTRODUCTIONBusiness initiatives like digital transformation invariably require multiple IT systems tooperate together in choreographed workflows. Advances in integration technology,such standards-based Application Programming Interfaces (APIs), make suchorchestration possible. Workflow orchestration tools establish and monitor theorchestration. Though not new, these tools have grown more sophisticated in recentyears. As the technology evolves, users are devising new best practices. The criteriafor what constitutes the right solution are also evolving. This paper explores how ITmanagers can leverage these practices and selection tips to get the best workfloworchestration results. It is based on reviews by members of IT Central Station sharingtheir experience with Control-M, BMC’s workflow orchestration product. 2019, IT Central StationGuide to Selecting a Workflow Orchestration Product1

A Brief Overview of WorkflowOrchestrationA notable reality in today’s IT world isthat no single system does everything.Vital business processes frequentlyrequire multiple systems to functionin coordinated ways. For example, torealize the goal of data visualization,it is usually necessary to orchestratethe operations of systems that ingestand store data, systems that analyzedata and those that do the actualvisualization and reporting. To the enduser, the results are seamless. But underthe surface, at least three separatesystems are inter-operating and handingoff data and procedure calls. Today, thisalmost always means the invocation ofAPIs. Figure 1 offers a simple referencearchitecture for the resulting process.WorkflowWorkflow Orchestration ToolSystemsComprisingthe WorkflowAPIAPIAPIAPISystem ASystem BSystem CSystem DFigure 1 - Workflow orchestration connects a workflow with the systems that complete its steps 2019, IT Central StationGuide to Selecting a Workflow Orchestration Product2

Workflow Orchestration Use CasesIT Central Station members are putting workfloworchestration solutions to work. For example,an IT staffer at a company with over 1,000employees uses Control-M “to automate batchprocessing, create automated workflows tosupport various applications, and integratevarious endpoints in the workflow to supportbusiness processes.” A database securityspecialist at a manufacturing company with morethan 10,000 employees uses Control-M for batchrecovery and job processing.An e-business engineer at a financial servicesfirm with over 1,000 employees explained howhis batch processing orchestration works onControl-M. He said, “We are receiving files fromanother system, then we use the File Watch Utility(because we have no view into the other systemand how it works). However, when files arrive oncertain servers, we’re able to pick them up andtrigger further downstream processes from them.”To get a sense of the scale of some of theseoperations, consider that a Control-M analyst at aretailer has a daily schedule that includes about4,000 jobs, with around 42,000 iterations ofjobs. A system admin and architect at a financialservices firm with over 1,000 employees sharedthat his team does up to a million jobs a day. Hecommented, “With Control-M, there is visibilityinto it. There are notifications when things gowrong. I don’t think our company could runwithout it.” This user is also putting Control-M intohis company’s DevOps workflow. 2019, IT Central StationGuide to Selecting a Workflow Orchestration Product3

Benefits of Workflow OrchestrationWorkflow orchestration confers anumber of technical and businessbenefits on organizations that adoptit. As the database security specialistat the manufacturing company shared,“Control-M has been there in back ofall automated work lists by providingconsistency, timely execution, andremoving chances of human errors.”The financial services system adminand architect felt that Control-M hadimproved his application reliabilityand Service Level Agreement (SLA)performance.A VP of Control-M scheduling at afinancial services firm with over 1,000employees noted, “It saves us time.Instead of 800 applications internallydoing their own thing, we centralizeit into one location where there arenotifications.” He added, “It saves us‘‘It saves us time.It saves us moneyfor suremoney for sure. We turned a companyof mom-and-pop little fiefdoms all overthe place into a company where thingsare centralized in one location.” 2019, IT Central StationGuide to Selecting a Workflow Orchestration Product4

Best Practices and ProductSelection FactorsThe IT field is no longer in the firstgeneration of workflow orchestrationtools. As the technology evolves, ITprofessionals are able to offer bestpractices and suggestions for whatnow comprises an effective solution.IT Central Station members havecommented on their experiences withControl-M in this regard. Their insightsinclude ideas like paying attention toorganizational issues, working acrossplatforms, consolidating solutions andmore. 2019, IT Central StationGuide to Selecting a Workflow Orchestration Product5

Pay Attention to Non-Technical/Organizational IssuesWorkflow orchestration does not exist in avacuum. To work effectively, it should beviewed in the broader organizational context.For instance, as a digital solutions manager at atransportation company with more than 10,000employees observed, “Some of our folks on thedistributed side of things need to learn how touse Control-M effectively in regards to outputwhen tasks or jobs fail. They need to give ussmarter outputs, so we can resolve things morequickly.” In this case, the two groups need tocollaborate better to make the most of theControl-M toolset.‘‘This feature allows developers tobetter understand the schedule, andallows them to correct any potentialissues in advance.Similarly, the IT staffer praised Control-M’s autoedit variable resolution for use in the planningstage of a workflow. He said, “This feature allowsdevelopers to better understand the schedule,and allows them to correct any potential issuesin advance.” Again, developers and IT staff dobetter when they can work together throughthe tool. A principal IT data research and mininganalyst at a healthcare company further notedthat it’s important to have “an advanced strategyaround how development [using Control-M] isgoing to take place.” As he put it, “If each team isdoing their own thing, it’s hard to manage it.” 2019, IT Central StationTake Advantage of NotificationsWorkflow orchestration solutions come withsophisticated notification capabilities. Thebest practice is to take full advantage of thisfeature. Doing so results in better awarenessof job status and greater overall IT productivity.Figure 2 highlights this practice, along with otherrecommended ways to get the most out of aworkflow orchestration tool. Other commentson notifications from IT Central Station membersincluded: “[Control-M’s] most valuable feature would bethe ability to detect and notify when a processhas not completed successfully.” - Automationspecialist at a financial services firm with over5,000 employees “Maintaining and monitoring of workloads havebeen and continue to be the most valuablefeature in our environment. There is a batchmonitoring tool called Batch Impact Manager,which proactively warns when processingis behind and SLAs are in jeopardy of beingmissed.” - Operations manager at a retailer withmore than 10,000 employees “If a job fails, that development team isnotified right away, which improves reliability.Previously, it was on the operators to notifythe developers that their job failed, erred, oraborted. Now, it’s all automated.” - Data centeroperations supervisor at a non-tech companywith over 1,000 employees “Because it’s a tool which allows us to doscheduled work, it allows for notificationswhen jobs aren’t running within that scheduledtime frame. This improves the opportunity tomeet SLAs.” - Manager of digital solutions at atransportation companyGuide to Selecting a Workflow Orchestration Product6

wWorkflow Orchestration ToolSystemsComprisingthe WorkflowAPIAPIAPIAPISystem ASystem BSystem CSystem DFigure 2 - Some of the best practices recommended by Control-M users include leveraging the toolset for monitoring,alerting, self-service and collaboration between multiple internal groupsWork Across PlatformsWorking across platforms emerged as a bestpractice in Control-M reviews. An operationsmanager at a tech services company with morethan 10,000 employees shared, “Cross-platformscheduling has transformed how we run ourestate workload. It is just so much more efficientnow as we can tie together all the workloadsacross the estate and make the whole processreactive to events.”‘‘.[its] most valuable feature wouldbe the ability to detect and notifywhen a process has not completedsuccessfully.”A senior operations analyst at an insurancecompany with over 1,000 employees alsocommented, “This product works with all theplatforms that we use today. We’re able to 2019, IT Central Stationcentralize our managing of job flows for all ourplatforms. That’s how it really helps us.” For thedatabase security specialist at the manufacturingcompany, the advantage of Control-M comesfrom its flexibility to schedule the jobs on variousOS-level platforms. In his case, usability wasenhanced by the tool’s agent-level architecture.Leverage Functionality forFlexibility, Ease of Use andSelf-ServiceUsability and self-service are two majorinnovations in the workflow orchestrationcategory. The technology isn’t (or shouldn’t be)the exclusive preserve of command line experts.IT Central Station members therefore recommendthat users of workflow orchestration tools takefull advantage of features for flexibility, ease ofuse and self-service. A head of IT procurementat a renewables and environment company wasGuide to Selecting a Workflow Orchestration Product7

pleased that Control-M lets his users access thesystem and schedule their own jobs, for example.A production engineer at a small tech servicescompany chose Control-M because it was userfriendly.Other comments in this vein included: “Unlike the batch controls of other solutions,BMC [Control-M] includes a graphical userinterface (GUI).” - Co-founder of a smallcompany “I love the usability. It works.” - Data centeroperations supervisor at a non-tech company “The most important features are that it is easyto use and graphical, since I’m a graphicalperson. This allows me to see it on the screen.I’ve used other scheduling tools, and theinformation wasn’t there. Being able to seethe jobs that connect to another job is reallyimportant to me. It is very flexible and dynamic.”- Control-M analyst at a retailerConsolidate Solutions WhenNecessaryGiven that the goal of workflow orchestrationis greater efficiency and productivity, itmakes sense that system users recommendconsolidating solutions when necessary—and when possible. As the retail operationsmanager explained, “When we first started usingControl-M, we had multiple scheduling solutionsacross several platforms. Going to Control-Mconsolidated all of them and allowed us reactivityacross all of them based on the completion ofprocessing.”The financial services system admin and architectoffered a good example of how consolidation canwork. He said, “Oracle database backup teams 2019, IT Central Stationused to do this all themselves via Cron. Now,they have automated Cron to Control-M for a lotof our database backups where they used to dothis outside of Control-M.” The VP of Control-Mscheduling had a similar approach to Oracledatabase backups.Look for Remote FileManagement CapabilitiesRemote file management is a feature to considerwhen selecting a workflow orchestration solution,according to Control-M users. This was on themind of an IT staffer at a healthcare company withover 1,000 employees, who praised Control-Mfor its ability to manage files remotely. He wasalso pleased with Control-M’s wide range ofconnectors.The production engineer laid out his remote filehandling use case by saying, “We have manyfeeds coming in from different companies whichare used by the business for various reasonsand we must collectively have a central point togather the files and feeds. We also use Control-M‘‘I love the usability. It works.for encryption, decryption, and sending dataacross to different business users that beginat a point of time, and making sure that we arenot missing unnecessarily.” He felt that ControlM’s file management feature was “a real help.”As he described, “We have a lot of businesswhich depends on feeds which, if not properlyprocessed, affects the stock exchange. So,Control-M acts as a mediator in between thatand provides it in a very efficient way. This hasreduced a lot of manual intervention required as abusiness.”Guide to Selecting a Workflow Orchestration Product8

Automate As Much As PossibleThe orchestration of workflows overlaps withworkflow automation. The two concepts arerelated, though different. Orchestration refersto the sequences of actions between systems.Automation in this context usually means therunning of the orchestrated sequences with littleor no user involvement, i.e. on a preset schedule.IT Central Station members recommend usingwhatever automation features are available tothe greatest extent possible. As the data centeroperations supervisor revealed, “Automation isits [Control-M’s] most valuable feature. It comesdown to if you schedule a job, then it runs onits own. You don’t need to have an operatormanually start a script, start a mainframe job,etc.” The e-business engineer also expressedadmiration for Control-M’s ability to go fromone system to another, triggering actions in theprocess.Other reviewers had comparable insights: “[The value comes from] the opportunity toautomate work so you have an audit trail,especially with governmental requirements in aregulated industry, such as the airline industry.It’s really important that we have that audit trail.”- Manager digital solutions at a transportationcompany “We are automating more things. I sometimeshear an application team say, ‘We arerunning this manually, and we want to make itautomated.’ I will make a few jobs to save themfrom doing what they are doing manually andautomate it. I am always looking for more thingsto automate.” - Control-M analyst at a retailer “We value Control-M mainly for the ability tocontrol multiple nodes in a coordinated manner. 2019, IT Central StationControl-M has the ability to really coordinateacross a lot of nodes. That’s the most valuablething.” - Team lead at a transportation companywith over 5,000 employeesDo Maximal MonitoringJob monitoring comes up as a factor in selectionof a workflow orchestration tool, as the financialservices system admin and architect noted.He said, “Monitoring is a valuable aspect of it.‘‘The monitoring tool is very good, andit is easy for expert and entry-levelusers to use on a short notice.The monitoring tool is very good, and it is easyfor expert and entry-level users to use on a shortnotice.” The operations manager at the retailerconcurred, saying, “Maintaining and monitoringof workloads have been and continue to be themost valuable features in our environment.” Tohim, these features were important because they“proactively warn when processing is behind andSLAs are in jeopardy of being missed.”Seek reliabilityControl-M users praised the solution’s reliability,which is another selection factor for a workfloworchestration tool. According to the insurancecompany senior operations analyst, “It’svery stable. It’s continuously running - we’rea 24/7 shop.” For the Control-M analyst atthe retailer, “Control-M is always running andnever breaking. I always tease server peopleabout rebooting, since my application is alwaysrunning.”Guide to Selecting a Workflow Orchestration Product9

Implementation TipsIT Central Station members alsooffered tips on the implementationof a workflow orchestration tool.A technical consultant at a small techservices company advised, “Make useof outsourced software solutions suchas Atgen for ensuring a stable setupprocess.” The Control-M analyst at theretailer did an upgrade, where BMChelped with implementation. He shared,“The upgrade process is great. They[BMC] have a whole department withtheir AMIGO program, where you canhave someone walk you through it.We have upgraded to 9.18. When wego to 9.19, it will be real quick. It shouldbe almost hands off from what Iunderstand.” 2019, IT Central Station‘‘The upgrade process is great. They[BMC] have a whole department withtheir AMIGO program, where you canhave someone walk you through it.Guide to Selecting a Workflow Orchestration Product10

Upgrade, Licensing and Pricing TipsControl-M users also shared insightsinto licensing and pricing. Thehealthcare principal, IT data researchand mining analyst provided a highlevel perspective on this subject. Hesaid, “My most important criteria whenselecting a vendor, in this case, sinceit’s a mature product, would be easeof migration.” The technical consultantat the small tech services companydiscussed how to save money byconsolidating to Control-M. Control-M’sefficiency enables users to do the sameamount of work with fewer jobs. As aresult, he offered the following savingstip: “Select more job-based pricingoptions as Control-M has less jobs tooffer.”For the production engineer at thesmall tech services company, the issuewas which controller modules will helptheir cost.” He further recommended,“First, check which modules arereally required and also what kind oflicensing makes sense for their [client’s]business. If it’s a very large enterprisethen it would be great to use a premiumenhance the work. He explained thatthe choice of modules will “ultimatelytransform their work into an automatedsolution, which in turn will reducebased license. If not, it’s better to usea job count based license. So that is apoint which they should check beforeimplementing.” 2019, IT Central StationGuide to Selecting a Workflow Orchestration Product11

CONCLUSIONIT departments today are developing new approaches to getting the most out ofworkflow orchestration solutions. The fundamental benefits of the technology arewell understood. Now, it’s about gaining maximum advantage from the solution thatpowers the orchestration. To this end, IT Central Station members offer insight

“Maintaining and monitoring of workloads have been and continue to be the most valuable feature in our environment. There is a batch monitoring tool called Batch Impact Manager, which proactively warns when processing is behind and SLAs are in jeopardy of being missed.” - Operati