Transcription
Software- und Organisations-ServiceOpen Source JobSchedulerJobSchedulerin a NutshellProduct OverviewSoftware- und Organisations-Service GmbH www.sos-berlin.com
2Open Source JobSchedulerIntroduction to JobSchedulerThe JobScheduler is a workload automation product that isused to launch scheduling objects, such as jobs, job chainsand orders, when time, file or calendar events occur.The JobScheduler provides solutions from simple to complexscheduling scenarios and is available with open source andcommercial licenses.The JobScheduler has been under continual developmentsince 2005 and has a mature status. The JobScheduler wasrecognized by Gartner Inc. with a placement in their MagicQuadrant for workload automation.
3JobScheduler worldwideOpen Source Users and Commercial CustomersAbout usSOS GmbH wasfounded in 1983, wespecialized in officeautomation, development of heterogeneoussoftware programs andthe provision ofservices and support.In 2005 SOS GmbHintroduced Dual Licensing (Open Source &commercial licensing)for JobScheduler andin 2011 for YADE Managed File Transfer.Today SOS GmbHcontinues with the ongoing development ofits open sourceproducts, the deliveryof services and supportto their customersworldwide. Open source users in more than 100 countries 191 commercial customers in 27 countries
4ContentsTopics Architecture Components Platforms Security Use Cases Cross-Platform Scheduling File Watching File Transfer Features Interfaces Key Features High Availability Agent Cluster Master Cluster
5ArchitectureJobScheduler ComponentsMore Information JobScheduler Master JobSchedulerUniversal Agent JobScheduler Master processes objects, such as jobs, job chains, orders and schedules Job Scheduler Universal Agent execute jobs on any machines in the networkUser Interfaces: JOC Cockpit JOE Graphical Editor User Interfaces JOC Cockpit: web-based interface for operating and monitoringJobScheduler in near real-time JOE: graphical editor for JobScheduler objects such as jobs, job chains,orders and schedules
6JOC Cockpit / Web Service / Master / AgentArchitecture: ComponentsJOC CockpitUser Interface for job control The JOC Cockpit is auser interface for jobcontrol with browsers Users access the Masterusing a Web Service thatperforms authenticationand authorization –optionally against anLDAP directoryInterfacesPowerShellCLILDAPDirectory The PowerShellCommand Line Interfaceand External Applicationsuse the same WebService for access to aJobScheduler Master Authorization is availablefor individual requests tothe JobScheduler MasterMaster / Agent The JobSchedulerMaster executes tasksand orchestrates Agents Agents are deployed ontop of existing serversrunning the programsand scripts that shouldbe scheduledProgramming InterfaceScripting InterfaceAuthentication and AuthorizationJobSchedulerMasterAgentsJOC CockpitUser InterfaceExternalApplicationsJobSchedulerWeb gentsDatabase AccessDatabase AccessDatabaseDatabase Access
7Platforms and DatabasesArchitecture: PlatformsMore Information Supported Platforms and Databases Supported Platforms Supported Databases JOC Cockpit & Web Service Windows and Linux JobScheduler Master Windows and Linux JobScheduler Universal Agent Windows and Linux Solaris, AIX, HP-UX, Mac OS and any other platform that supports a Java Virtual Machine Databases Oracle, SQL Server MariaDB, MySQL and PostgreSQL
8Supported PlatformsArchitecture: PlatformsCockpit / Web Service The JOC Cockpit andREST Web Service areavailable for Windowsand LinuxJobSchedulerJOC CockpitWindowsJobSchedulerWeb ServiceWindowsJobSchedulerJOC CockpitLinuxJobSchedulerWeb ServiceLinuxMaster / Agent JobScheduler Masteris available for Windowsand Linux JobScheduler Agentsare available for anyplatform that supportsa Java Virtual MachineDatabase accessJobSchedulerMasterWindowsDatabaseEnables job execution: on JobSchedulerMaster instances on JobSchedulerAgents for any platformJobSchedulerMasterLinuxWorks with supporteddatabases including: Oracle SQL Server MariaDB/MySQL PostgreSQL The JobSchedulerREST Web Serviceand Master use a database on any platformJobs Jobs can be executedlocally on the Master Jobs can be executedon any JobSchedulerUniversal XJobSchedulerAgentLinuxJobSchedulerAgentMac obSchedulerAgentAIXJobSchedulerAgentRaspberry PiDatabaseWorks with anyplatform that supportsa Java Virtual MachineJobSchedulerAgent. any platform
9JOC Cockpit / Web Service / Master / AgentArchitecture: SecurityUsers Network Zonewith restricted useraccess Limited access thatrequires authentication Any connection to aMaster is authenticated by the Web Service that can be configured to use LDAP Use of HTTPS forconnections can beenforcedNetwork Zone withoutuser access Master and Agentinstances are operated in this zone without direct user access The Master instancesare accessed exclusively by the WebService Agent instances areexclusively accessedby Master instancesNetwork Zonewith limiteduser accessHTTP / HTTPSJOC CockpitUser InterfaceHTTP / HTTPSPowerShellCLILDAPDirectoryLDAP proprietary protocolExternalApplicationsDatabase proprietary protocolJobSchedulerWeb ServiceNetwork Zonewithoutuser accessHTTP / HTTPSJobSchedulerMasterAgentsHTTP / HTTPSHTTP / HTTPSHTTP / ntsDatabase proprietary protocolDatabase proprietary protocolDatabaseDatabase proprietary protocol
10Samples of Real World Use CasesUse Cases Cross-Platform Scheduling Processing with Agents across Platforms Agentless Scheduling across Platforms File Watching Watching incoming files on a Master Remote File Watching with Agents File Transfer Server-2-Server file transfer with YADE Push Files to Internet via Jump Host Pull Files from Internet via Jump Host
11Processing with Agents across PlatformsUse Case: Cross-Platform SchedulingUse Case The customer operatesservers with Windows,Linux, Solaris and AIX Jobs with dependencieshave to be executed onthe different serversSolution Remote execution withUniversal Agents A JobScheduler Masterfor Windows optionallyexecutes jobs locally andorchestrates four Agents Job dependency isconfigured with Masterwho contacts Agents toexecute jobsBenefits JobScheduler covers allrequired platforms Job dependencies formulti-platform executioncan be configured Single point ofconfiguration & operation Zero configuration forUniversal Agents andeasy deployment to largeclustersMaster contacts Agents to execute jobsJobSchedulerMasterWindowsMaster collects log output and execution resultsSequence of jobs configured with the JobScheduler MasterJob ChainJob1Job gentAIX
12Agentless Scheduling across PlatformsUse Case: Cross-Platform SchedulingUse Case The customer operatesservers with Windows,Linux, Solaris and AIX Jobs with dependencieshave to be executed onthe different serversSolution A JobScheduler Masterfor Windows executesjobs locally and orchestrates execution on threeservers by SSH The job dependency isconfigured with theMaster who contactsAgents to execute jobsBenefits No software installationis required for jobexecution on existingSSH servers Job dependencies formulti-platform executioncan be configured Single point of configuration and operationJobSchedulerMasterWindowsMaster executes jobs by SSHJob2Master collects log output and execution resultsJob executedwith MasterSequence of jobs configured with the JobScheduler MasterJob ChainJob1Job ChainJob3SSH ServerLinuxJob4SSH ServerSolarisSSH ServerAIX
13Watching incoming files on a MasterUse Case: File WatchingUse Case Files have to bewatched on differentplat-forms, e.g. onSolaris Job starts shallbe caused on Windowsand Linux after arrival ofa trigger file on SolarisSolution Agents watch for incoming files and sendsignals to the Masteronce a file arrives The Master is configured to know whichworkflows are to followBenefits Remote file watchingwith Master & Agents isplatform independent All configuration for theworkflows are stored onthe Master, the Agentsexecute jobs with zeroconfiguration Remote file watchingcan be applied to anynumber of platforms andserversMaster receives Agentsignal to start job chainJobSchedulerMasterLinuxMaster contacts Agents toexecute jobs in a job chainJob1Agent signals arrival ofincoming trigger filesMaster collects log outputand execution ntSolarisWatching forincoming filesJobSchedulerAgentLinux
14Remote File Watching with AgentsUse Case: File WatchingUse Case Files have to be watchedon Solaris and sent to aLinux server. Aftertransfer the files areprocessed on Linux anda successor job onWindows is executedSolution Universal Agent watchesfor incoming files andsends signal to theMaster once a file arrives The files are directly sentto a Linux server with afile transfer job on Solaris The Master is configuredto know which workflowsare to followBenefits Remote file watchingwith Master and Agentsis platform independent Configuration for theworkflows are onlyrequired on the Master,the Universal Agentsexecute jobs with zeroconfigurationMaster receives Agentsignal to continue job chainAgent signals arrival andtransfer of incoming filesJob 1YADE ClientJITL JobJobSchedulerMasterLinuxMaster contacts Agents toexecute jobs in a job chainJob2Master collects log outputand execution resultsJobSchedulerAgentLinuxFiles tobe sendJob3JobSchedulerAgentSolarisWatching forincoming filesJobSchedulerAgentWindows
15Managed File TransferUse Case: File TransferMore Information JobScheduler Add-on YADE ManagedFileTransfer Managed File TransferJobScheduler provides different methods for managing the transfer of files JobScheduler Add-on YADE: JobScheduler component for Managed File Transfer YADE JITL job templates are pre-installed in the JobScheduler andcan be called directly from the JOC Cockpit YADE Managed File Transfer: YADE Command Line Client is operated independently fromJobScheduler (standalone) Command Line Interface YADE Client CLI is executed by shell jobs YADE Managed File Transfer with JobScheduler YADE includes the components YADE JITL Jobs: job interface integrated with JobScheduler YADE Background Service: collects transfer protocols and history and addsthem to the database YADE History Viewer: GUI to view history and transfer logs
16Server-to-Server File Transfer with YADEUse Case: File TransferUse Case Files shall be transferred by usingdifferent transferprotocols for datasources and targetsJobSchedulerMasterSolutionYADEJITLJob A virtual file systemapproach is used byYADE. This allows tocopy and move filesbetween sources andtargets with any givenprotocolServer BSFTPData TargetPull fileswith HTTPBenefits Any combination of filetransfer protocols ispossible, e.g. FTP,FTPS, SFTP, HTTP,WebDAV No touchdown of fileson JobSchedulerMaster Transfers areperformed in memory,resulting in veryeconomical use ofresourcesPush fileswith SFTPPull fileswith FTPServer AFTPData SourceServer CHTTPData SourcePush fileswith WebDAVServer DWebDAVData Target
17Push Files to Internet via Jump HostUse Case: File TransferUse Case Servers in the intranethave no internet access. Connection are onlyallowed from the intranetto the DMZ. From theDMZ transfers to theinternet are allowedSolution YADE Client in the Intranet contacts YADE forDMZ to initiate a securetransfer of files from theintranet to the internet. YADE Client in intranetsends files to YADE inDMZ for temporarystoring In a final step the YADEClient in DMZ pushes thefiles to the internetBenefits Files are stored temporarily in the DMZ and areremoved after transfer No credential data arestored in the DMZ All configuration itemsare stored in the intranetDMZJump HostIntranetSource HostJobSchedulerMasterInternetTarget HostYADE Clientfor DMZ1Push Files2Push FilesWebDAV, FTP,FTPS, SFTPServerYADE ClientJITL JobFiles are temporarily storedand removed after transferFiles tobe sendFiles havearrived
18Pull Files from Internet via Jump HostUse Case: File TransferUse Case Servers in the intranethave no internet access Connection are onlyallowed from the intranetto the DMZ. From theDMZ transfers to theinternet are allowedSolution The host in the DMZ actsas jump host for YADE YADE Client in intranetcontacts YADE in DMZto start a transfer and topull files from the internetto the DMZ YADE Client in intranetpulls the files from theYADE in DMZ to the finaldestination in the intranetBenefits Files are stored temporarily in the DMZ and areremoved after transfer No credential data arestored in the DMZ All configuration itemsare stored in the intranetDMZJump HostIntranetTarget HostJobSchedulerMasterInternetSource HostYADE Clientfor DMZ1ContactYADE DMZ2Initiatefile transferHTTP, WebDAV,FTP, FTPS, SFTPServerYADE ClientJITL Job34Pull filesfrom InternetPull filesfrom DMZFiles are temporarily storedand removed after transferFiles fortransfer
19Interfaces & Key FeaturesFeatures Interfaces User Interfaces Programming Interfaces Key Features Command Line OperationStart TimesLogging and ScriptingJobs, Job Chains and OrdersError Handling and Event HandlingCentral ConfigurationDirectory Monitoring and File Watchingmore
20JOC Cockpit and JOE Object EditorInterfaces and Key Features: User InterfacesMore Information JobScheduler OperationsCenter: JOC Cockpit JOE Object Editor JobScheduler Operations Center: JOC Cockpit Web-based GUI provides overview for all relevant information onobjects, starting and monitoring jobs, job chains and orders Complete overview for the JobScheduler Master status includingCluster, healthy and unhealthy Agents Automated updates in near real-time JOE Object Editor Manage JobScheduler Objects
21JOC Cockpit - DashboardInterfaces and Key Features: User InterfacesDashboard The Dashboard offersa comprehensiveoverview of mostrelevant information inthe form of widgets Information in theDashboard is updatedautomatically in nearreal-time The Dashboard showsthe JobSchedulerMaster statusincluding clusterinformation The Agent overviewshows healthy andunhealthy AgentClusters The Dashboard is astarting point tonavigate to objects ofinterest, e.g. failedorders, suspendedorders, late orders etc.
22JOE Object Editor - Job ManagementInterfaces and Key Features: User InterfacesJOE Job Editor JOE is used to create,configure and manageJobScheduler objectssuch as jobs, jobchains, orders,schedules, processclasses and locks JOE reads and writesXML files for objects toso-called hot folderswhere they are loadedby JobScheduler JOE can be operateddirectly in the scheduling environment or onany computer thatsupports a JavaVirtual Machine
23REST Web Service InterfaceInterfaces and Key Features: Programming InterfacesJobScheduler ProgrammingInterfaces JobScheduler RESTWeb Service PermissionsJOC Cockpit JOC Cockpit RESTWeb Service Permissions JobScheduler REST Web Service (XML) The JobScheduler comes with a powerful REST Web Service interfacefor access by external applications This interface is used for tasks such as adding orders, adding eventsand retrieving job and order status information Access to the REST Web Service is restricted by permissions JOC Cockpit REST Web Service (JSON) The JOC Cockpit comes with a REST Web Service interface for JSON This interface is used for access to the job inventory and the job historyas well as individual objects such as jobs, job chains and orders andprovides current status information Access to the REST Web Service is restricted by permissions
24Job Application Programming InterfaceInterfaces and Key Features: Programming InterfacesMore Information API Interface ReferenceDocumentation Areas of Operation The JobScheduler comes with an extensive set of objects and methodsfor implementation with jobs and monitors. Such implementations oftenmake use of the API Interface to check and manipulate JobSchedulerobjects, e.g. to add orders, to modify the order and job status The API is available for jobs and monitor scripts that implementconditions for job execution Language Support The API is available for scripted jobs, e.g. in the languages JavaScript,PowerShell, VBScript The API can be used from Java jobs
25Command Line InterfaceInterfaces and Key Features: Programming InterfacesMore Information Command LineOperation PowerShell CommandLine Interface Shell Command Line Operation The JobScheduler can be operated from the command line, allowing awide range of operations to be carried out by an external application: checking the JobScheduler status controlling the status of jobs, job chains and orders adding orders to job chains adding events for event handling PowerShell Command Line Interface The JobScheduler Command Line Interface (JCLI) can be used tocontrol JobScheduler instances (start, stop, status) and job-relatedobjects such as jobs, job chains, orders, tasks Areas of Operation include to work as a replacement for command scripts, e.g. to start JobSchedulerand to raise events provide bulk operations on jobs, job chains and orders schedule jobs and orders manage Agents, e.g. to check the Agent status
26Key Features (1/4)Interfaces and Key FeaturesMore Information Start Times Logging Scripting Start Times Time events are one way of starting jobs, job chains and orders, withStart Times being set for a predefined point in time such as time of day,weekday, day of month etc. Logging The JobScheduler creates a number of logs to provide specificinformation about jobs, job chains, orders, tasks and JobSchedulerinstallation and operation Scripting Scripting interface that allows execution of scripts in languages such asJavaScript, etc.
27Key Features (2/4)Interfaces and Key FeaturesMore Information Jobs, Job Chains andOrders Error Handling Event Handling Jobs, Job Chains and Orders The JobScheduler's unique job and order concept includes theorganization of jobs into job chains and the use of dependencies Jobs: basic unit for the processing of executable files Job Chains: assembly line on which job nodes are passed sequentially Orders: represent triggers that will cause a job chain to be started Error Handling A number of methods for error handling are available, e.g. stop a job,suspend an order, setback an order and more Event Handling Mechanism for implementing complex dependencies between jobs orbetween jobs and external events
28Key Features (3/4)Interfaces and Key FeaturesMore Information Central Configuration Directory Monitoringand File Watching Resource ContentionManager Central Configuration Central configuration allows the efficient distribution of configuration filesfrom a central source to distributed JobScheduler instances by use of aSupervisor JobScheduler Directory Monitoring and File Watching Two methods to start jobs and job chains automatically based on thearrival of incoming files Resource Contention Manager Process classes and locks are used to manage the use of resourcessuch as databases or printers
29Key Features (4/4)Interfaces and Key FeaturesMore Information Notifications Monitoring Notifications JobScheduler comes with its own mail client which it can use to sendnotifying e-mails in the event of, for example, jobs ending in error Localization Monitoring JobScheduler can be monitored by System Monitors. Such productsinclude e.g. HP OpenView , Microsoft SCOM , Nagios , op5 ,Opsview , Zabbix etc. Localization Language files are provided for the installation of JobScheduler and formost of its operating interfaces. The default language is English Installation Languages: Engl
SOS GmbH was founded in 1983, we specialized in office automation, develop-ment of heterogeneous software programs and the provision of services and support. In 2005 SOS GmbH introduced Dual Licen- . Job Scheduler Un
