Transcription
An IBM Proof of TechnologyWebSphere MQ Managed File TransferIntroduction toWebSphere MQ Managed File Transfer 2014 IBM Corporation
IBM SoftwareIBM statements regarding its plans, directions, and intent aresubject to change or withdrawal at IBM's sole discretion.Information regarding potential future products is intended tooutline our general product direction and it should not be reliedon in making a purchasing decision.The information mentioned regarding potential future productsis not a commitment, promise, or legal obligation to deliver anymaterial, code or functionality. Information about potential futureproducts may not be incorporated into any contract. Thedevelopment, release, and timing of any future features orfunctionality described for our products remains at our solediscretion.2WebSphere MQ Managed File Transfer Overview 2014 IBM Corporation
IBM SoftwareAgenda Common problems transferring file data Introduction to IBM WebSphere MQ Managed File Transfer IBM Managed File Transfer portfolio Key MQ Managed File Transfer concepts Usage scenarios for MQ Managed File Transfer3WebSphere MQ Managed File Transfer Overview 2014 IBM Corporation
IBM SoftwareIBM WebSphere MQ familyPortfolio of messaging capabilities optimized for a range of connectivity challengesWebSphere MQWebSphere MQ for z/OSfor mission critical datafor IBM System z investmentWebSphere MQ Managed File Transferfor managed file transferWebSphere MQ Adv. Message Securityfor maximum security4WebSphere MQ Telemetryfor sensors and devicesWebSphere MQ Low Latencyfor high speed deliveryWebSphere MQ Managed File Transfer Overview 2014 IBM Corporation
IBM SoftwareHow do most organizations move files today?Most organizations rely on a mix of “home-grown” code, legacy products and different technologies andeven different people! FTP– File Transfer Protocol (FTP) is usually combined with writingand maintaining home-grown code to address its limitations. Why is FTP use so widespread?––––FTP is widely available – Lowest common denominatorPromises a quick fix – repent at leisureSimple concepts – low technical skills needed to get startedFTP products seem “free,” simple, intuitive and ubiquitous. Legacy file transfer products––––A combination of products often used to provide silo solutionsOften based on proprietary versions of FTP protocolCannot transport other forms of data besides filesUsually well integrated with B2B but rarely able to work withthe rest of the IT infrastructure – especially with SOA People– From IT staff to business staff and even security personnel– Using a combination of email, fax, phone, mail, memorykeys, and more.5WebSphere MQ Managed File Transfer Overview 2014 IBM Corporation
IBM SoftwareShortcomings of basic FTPLimitedreliability Unreliable delivery – Lackingcheckpoint restart – Files canbe lost Transfers can terminate withoutnotification or any record –corrupt or partial files can beaccidentally used File data can be unusable aftertransfer – lack of character setconversionLimitedflexibility Changes to file transfers often require updates to manyFTP scripts that are typically scattered acrossmachines and require platform-specific skills to alter All resources usually have to be available concurrently Often only one FTP transfer can run at a time Typically transfers cannot be prioritized6WebSphere MQ Managed File Transfer OverviewLimitedsecurity Often usernames and passwordsare sent with file – as plain text! Privacy, authentication andencryption often not be available Non-repudiation often lackingLimited visibilityand traceability Transfers cannot be monitored and managedcentrally or remotely Logging capabilities may be limited and may onlyrecord transfers between directly connected systems Cannot track the entire journey of files – not just fromone machine to the next but from the start of itsjourney to its final destination 2014 IBM Corporation
IBM SoftwareWhat is Managed File Transfer?Reliable, controlled, auditable movement of files No agreed specification for managed file transfer products to certifyagainst General consensus that managed file transfer involves the following: Auditable Reliable Secure AutomatedWho transferred a file? Where? When? Was it this file?Automatic resumption of interrupted transfers. No partial file data left lying aroundLimits access to authorized users. Protects file data in transitDesigned for “lights out” operation Centralized Any file size Integrated Cost effective7Can be monitored and managed from one central locationImposes no practical limits on file sizes. Efficient regardless of file sizeIntegrates well with applications that typically perform file processingReuses existing skills and infrastructureWebSphere MQ Managed File Transfer Overview 2014 IBM Corporation
IBM SoftwareWhat is WebSphere MQ Managed File Transfer?Adds managed file transfer capabilities to WebSphere MQ WebSphere MQ Managed File TransferA Auditable Reliable Secure Automated Centralized Any file size Integrated Cost Effective8BCXYZFull logging and auditing of file transfers archive audit data to a databaseCheckpoint restart. Exploits solid reliability of WebSphere MQProtects file data in transit using SSL. Provides end-to-end encryption using AMSProviding scheduling and file watching capabilities for event-driven transfersProvides centralized monitoring and deployment of file transfer activitiesEfficiently handles anything from bytes to terabytesIntegrates with MB, WSRR, ITCAMs for Apps, DataPower Connect:DirectReuses investment in WebSphere MQ. Wide range of support (inc. z/OS and IBM System i )WebSphere MQ Managed File Transfer Overview 2014 IBM Corporation
IBM SoftwareA consolidated transport for both files and messages Traditional approaches to file transferresult in parallel infrastructures– One for files – typically built on FTP– One for application messaging – basedon WebSphere MQ, or similar High degree of duplication in creatingand maintaining the two infrastructuresFile transfersApplicationmessaging Managed File Transfer reuses the MQnetwork for managed file transfer andyields:– Operational savings and simplification– Reduced administration effort– Reduced skills requirements andmaintenance9WebSphere MQ Managed File Transfer OverviewConsolidated transportfor messages and files 2014 IBM Corporation
IBM SoftwareAgenda Common problems transferring file data Introduction to MQ Managed File Transfer IBM Managed File Transfer Portfolio Key MQ Managed File Transfer concepts Usage scenarios for MQ Managed File Transfer10WebSphere MQ Managed File Transfer OverviewYou are here 2014 IBM Corporation
IBM SoftwareWith Sterling Commerce, IBM offers comprehensive MFT CapabilitiesAddressing multiple use cases and scenarios for both internal and multi-enterprise file transfer WebSphere MQ Managed File Transfer providesfile transfer optimized for data delivery acrossWebSphere MQ networks Sterling Connect Direct provides peer-to-peerfile transfer optimized for data delivery within andbetween enterprises across Connect:Direct protocol Sterling File Gateway provides trading partneron-boarding, broad protocol support, managementand visibility Sterling File Gateway includes a plug-in for IBMWebSphere MQ Managed File TransferIBM statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion.Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision.The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality.Information about potential future products may not be incorporated into any contract.The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.11WebSphere MQ Managed File Transfer Overview 2014 IBM Corporation
IBM SoftwareIBM MFT visionAccelerate and simplify governance of the growing volume of business-critical data movement within andbeyond the enterprise with Smarter MFTMaximize the agility and performance of dynamic business networks by reducing the complexity, risk, and cost offile transferVisibilitySecurity and performance Single view of transfer activity Transaction and business monitoring Dashboards, analytics, and scorecards Assured delivery and high availability Protection of file data in transit and at rest Accelerated transport and low latencyUsability and managementUniversal Any transport, any protocol, and anypartner Global and cloud-enabled deployment Broad platform coverage and industrystandards Persona-based, easy-to-use interfaces Unified control/configuration ofinfrastructure Community on-boarding and coordinationConnectivity Integration with BPM to drive business processes Using ESBs to enable service orientation SOA registry/repository for lifecycle governanceIBM statements regarding its plans, directions, and intent are subject to change or withdrawal without notice at IBM’s sole discretion.Information regarding potential future products is intended to outline our general product direction and it should not be relied on in making a purchasing decision.The information mentioned regarding potential future products is not a commitment, promise, or legal obligation to deliver any material, code or functionality.Information about potential future products may not be incorporated into any contract.The development, release, and timing of any future features or functionality described for our products remains at our sole discretion.12WebSphere MQ Managed File Transfer Overview 2014 IBM Corporation
IBM SoftwareAgenda Common problems transferring file data Introduction to MQ Managed File Transfer IBM’s Managed File Transfer Portfolio– Introducing IBM Sterling Commerce products Key MQ Managed File Transfer concepts Usage scenarios for MQ Managed File Transfer13WebSphere MQ Managed File Transfer OverviewYou are here 2014 IBM Corporation
IBM SoftwareComponents of a typical WMQ MFT networkApplications exchanging file data Agents– AgentWebSphere MQ“Coordination”Queue ManagerA historical record of file transfersCoordination queue manager–Gathers together file transfer eventsCommands14AgentSend instructions to agentsLog database– AgentCommands– The endpoints for managed filetransfer operationsWebSphere MQ Managed File Transfer OverviewLogdatabase 2014 IBM Corporation
IBM SoftwareComponents of a typical WMQ MFT network agents Act as the end points for file transfersLong running MQ applications thattransfer files by splitting them into MQmessages– –AgentCan both send and receive multiplefiles at the same timeAgentWebSphere MQ“Coordination”Queue ManagerGenerate a log of file transfer activitieswhich is sent to the “coordinationqueue manager”This can be used for audit purposes.Associated with one particular queuemanager (either v6 or v7)–15AgentMulti-threaded file transfers– Efficient transfer protocol avoidsexcessive use of MQ log space ormessages building up on queues.Applications exchanging file dataAgent state on queuesWebSphere MQ Managed File Transfer OverviewCommandsLogdatabase 2014 IBM Corporation
IBM SoftwareWebSphere MQ Managed File Transfer agent architectureManaged File TransferagentJVMCommand Process threadsFile ChuckBufferPoolSchedulerTriggersThread PoolQueue Manager(WebSphere MQ V6 )COMMANDFile Xfre Ack threadThread PoolDATAFile Receiver threadsSchedulerTriggersFile Sender threadsCommand Process threadsFile Xfre Ack threadFile Receiver threadsFile Sender threadsFile ChuckBufferPoolManaged File TransferagentJVMREPLYQueue Manager(WebSphere MQ V6 )STATEDATAEVENTCOMMANDREPLYSTATEEVENT Multi-threaded architecture for performance and scalability Highly efficient WebSphere MQ pacing-based stream I/O with full in-flight checkpoint and restart Built-in scheduler and trigger support Bindings (server) and client support Uses WebSphere MQ Publish and Subscribe infrastructure for transfer logs and progress16WebSphere MQ Managed File Transfer Overview 2014 IBM Corporation
IBM SoftwareComponents of a typical WMQ MFT network – Commands 17Send instructions to agentsand display information aboutagent configuration– Via MQ messagesMany implementations ofcommands:– MQ Explorer plug-in– Command line programs– Open scripting language– JCL– Documented interface toprogramWebSphere MQ Managed File Transfer OverviewApplications exchanging file dataAgentAgentAgentWebSphere MQ“Coordination”Queue ManagerCommandsLogdatabase 2014 IBM Corporation
IBM SoftwareComponents of a typical WMQ MFT network – Log database Keeps a historical account oftransfers that have taken place– Implemented by the ‘databaselogger’ component whichconnects to the coordinationqueue manager–– AgentAgentAgentWebSphere MQ“Coordination”Queue ManagerStand-alone applicationOr JEE applicationQueryable via web gateway–18Who, where, when etc.Applications exchanging file dataAlso a documented interfaceWebSphere MQ Managed File Transfer OverviewCommandsLogDatabase 2014 IBM Corporation
IBM SoftwareComponents of a typical WMQ MFT network –Coordination Queue Manager Applications exchanging file dataGathers together information aboutevents in the file transfer networkAgent Can be made highly availableMessages stored forwardedWebSphere MQ“Coordination”Queue ManagerMQ v7 publish/subscribe– Allows multiple log databases,–command installsDocumented interfaceCommands19AgentNot a single point of failure–– AgentWebSphere MQ Managed File Transfer OverviewLogdatabase 2014 IBM Corporation
IBM SoftwareDeployment20WebSphere MQ Managed File Transfer Overview 2014 IBM Corporation
IBM SoftwareWebSphere MQ Managed File Transfer Agent using Server Connectionz/OSz/OSMQSeriesServerChannelsWebSphere MQFile TransferEdition AgentWebSphere MQQueue ManagerCSQ6WebSphere MQFile TransferEdition AgentWebSphere MQQueue ManagerCSQ1Sender/ReceiverWindowsLinuxUNIX LinuxMQSeriesClientChannelWebSphere MQFile TransferEdition Agent21Sender/ReceiverSender ReceiverSender/ReceiverWebSphere MQ Managed File Transfer OverviewWebSphere MQFile TransferEdition AgentWebSphere MQQueue ManagerLINUX01 2014 IBM Corporation
IBM SoftwareAgenda Common problems transferring file data Introduction to MQ Managed File Transfer IBM’s Managed File Transfer Portfolio– Introducing IBM Sterling Commerce products Key MQ Managed File Transfer concepts Usage scenarios for MQ Managed File Transfer22WebSphere MQ Managed File Transfer OverviewYou are here 2014 IBM Corporation
IBM SoftwareScenario: Painful FTP-based solutionFTP Pain point– Home-grown, FTP-based solution isunreliable and hard to manage. Managed File Transfer helps– Directory monitoring/scripting/JCLprovides integration with existing systemswithout the need for modifications– Protocol bridge allows a staged migrationfrom existing FTP/SFTP infrastructure– Integrates with existing infrastructure: IBM Message Broker IBM Sterling Connect:Direct– Combines with B2B products: DataPower XB60/XI50 IBM Sterling File Gateway– Secured by industry standard SSL orWebSphere MQ AMS23WebSphere MQ Managed File Transfer Overview 2014 IBM Corporation
IBM SoftwareExample usage of monitoring and program execution3. MFT transports fileto destination.ExistingApplication1. Application writesfile to file system.24WMQMFTAgent2. Agent monitors filesystem, spots arrival of fileand based on rules,transfers the file.WebSphere MQ Managed File Transfer OverviewFTP5. MFT can also start another applicationto process the file.WMQMFTAgentExistingApplication4. At destination, MQ MFTwrites file to file system. 2014 IBM Corporation
IBM SoftwareProtocol bridging agentsFTP Support for transferring files located on FTP and SFTP servers– The source or destination for a transfer can be an FTP or an SFTP server Enables incremental modernization of FTP-based, home-grown solutions– Provides auditability of transfers across FTP/SFTP to central audit log– Ensures reliability of transfers across FTP/SFTP with checkpoint restart Fully integrated into graphical, command line and XML scripting interfaces– Just looks like another MFT agent Files exchanged between MFT and FTP/SFTPAgentAgentFTP/SFTPAgentWebSphere here MQ Managed File Transfer Overview 2014 IBM Corporation
IBM SoftwarePart ofWMB7.0.0.1WebSphere Message Broker nodesFTPMessage BrokerExecution groupMessage E InputFTE Output FTE Input node– Build flows that accepts file transfers from the WMQ MFT network FTE Output node– Build flows that are designed to send a file across a WMQ MFT network When WMQ MFT nodes are used in a flow an MFT agent is automatically started in the MessageBroker execution group26WebSphere MQ Managed File Transfer Overview 2014 IBM Corporation
IBM SoftwareIntegration with IBM Sterling Connect:DirectAgentC:DnodeAgentAgentWebSphere MQC:DBridgeagentC:DnodeC:DAudit The Connect:Direct bridge capability supportsmanaged file transfers that span MFT andC:D with a joined up audit e MQ Managed File Transfer OverviewTrading partnerInside the MFT audit trail The audit information for each MFT transferreferences related C:D audit information. 2014 IBM Corporation
IBM SoftwareInteroperation with DataPower XB62 B2B applianceFTP Documented and tested configurations for integrating with DataPowerappliances–WebSphere DataPower XB60 B2B appliance – for B2B connectivity–WebSphere DataPower XI50 integration appliance – for ESBconnectivity Enables sending files to trading partners over a range of protocol transports–via DataPower appliances acting as B2B gatewaysMulti-protocol transfers to B2B trading partnersCompany AAgentAgentWebSphere MQAgentXB60WebSphere MQ Managed File Transfer OverviewSFTPAS2Company BCompany C etc etcAgentInternal Network28HTTPSAgentDMZInternetTrading partner 2014 IBM Corporation
IBM SoftwareSecuring file data with SSL and WMQ AMSAgentAgent29Svrconn conn re MQ Managed File Transfer OverviewFTP WMQ MFT supports transportlevel encryption using SSL.Agent Data is encrypted before it issent over a channel anddecrypted when it is received. When combined with WMQAdvanced Message SecurityAgent– Allows file data to beencrypted at the sourcesystem and only decryptedwhen it reaches thedestination system– Data is secure – even whenat rest on a queue 2014 IBM Corporation
IBM SoftwareScenario: Staged migration to messaging Pain point– Hard to migrate to an event-driven architecture, as lots of applications communicate bytransferring files Managed File Transfer helps– Deliver files as message payloads and vice versa– Monitor queues and transfer message payloads to filesWebSphere MQandManaged File Transfer30WebSphere MQ Managed File Transfer Overview 2014 IBM Corporation
IBM SoftwareOptions for converting data between files and messagesOne file to one messageWMQMFT One file becomes one message The file can be split basedonOne file to a group of messagesWMQMFT– Size– Binary delimiter– Regular expressionOne message to one fileWMQMFT One message becomes one fileA group of messages (or all messages on the queue) to one fileWMQMFT31WebSphere MQ Managed File Transfer Overview Optionally, a delimiter can beinserted between each messageused to compose the file. 2014 IBM Corporation
IBM SoftwareMonitoring queues for the arrival of messagesRemember, MFTcan monitor for files arriving. The WMQ MFT agent can monitorqueues for the arrival ofmessages, then perform anaction, such as transferring thepayload from the messages as afile. (as per the previous slide) Conditions that can be monitored–Queue not empty–Complete group of messages32WebSphere MQ Managed File Transfer OverviewExistingapplicationWMQMFTagentWell, it can also monitor formessages arriving on a queue.ExistingapplicationWMQMFTagent 2014 IBM Corporation
IBM SoftwareScenario: Web-based managed file transfers Pain points– Difficult to mix human-imitated file transfers with existing infrastructure for machine-tomachine managed file transfer– Managed file transfers to zero-install, small-footprint devices Managed File Transfer helps– A RESTful API for exchange files with an WMQ MFT network– Example web 2.0 applications to use as a starting pointAgentAgentAgentWebgatewayHTTPSWebSphere MQAgent33AgentWebSphere MQ Managed File Transfer OverviewHTTPHTTPS 2014 IBM Corporation
IBM SoftwareEnabling business users to upload files from a remote locationWeb gateway example, business scenario: In this example, usage scenario the web gateway allows a business user to upload a file (viathe company web portal) to a back-end system, where it can be processed.Head OfficeBusiness userReportBack-endsystemon-site withcustomerInternetLogs on tocompany web portal1. The business userlogs onto the companyweb portal using a webbrowser and isprompted to select afile to upload.34WebgatewayFile transported using HTTP2. The portaluses the RESTfulAPI provided bythe web gatewayto upload the fileusing HTTP.WebSphere MQ Managed File Transfer OverviewWebSphereMQ networkReport isprocessedand loadedinto database.File transported using WMQ MFT3. The webgateway transfersthe file, using WMQMFT, to a back-endsystem.4. At theback-end system,WMQ MFT startsa program toprocess the datafrom the file. 2014 IBM Corporation
IBM SoftwareEnabling researchers to pick up the results of a batch processWeb gateway example, business scenario: In this example, the web gateway enables a researcher to pick up files that have beenproduced (hours earlier) by batch processing, at a back-end system.Data CentreWebgatewayBack-endsystemWebSphereMQ networkProcessesresearch dataduring a 3 a.m. to 4 a.m.batch processing window.WANFile spacea per-user “in-tray” for filesFile transported using WMQ MFT1. A batch processrunning at the datacenter produces a set ofresults which it sends,using WMQ MFT, to theweb gateway.35ResultsResearchLab2. The webgateway systemplaces the data intoa file space, where itawaits collectionby the user.WebSphere MQ Managed File Transfer OverviewLogs into researchportal to collect resultsFile transported using HTTP3. The user logsin to the researchportal using her webbrowser and is showna list of files waitingfor her attention.4. The userselects a file todownload and theweb gatewaytransfers the fileto her computer. 2014 IBM Corporation
IBM SoftwareComponents used for uploading to a back-end systemWeb gateway:FileprocessedUserLogical path for file ntWeb gateway systemJEEapplicationserverWMQQueueManagerWebSphere MQ networkWeb browserMFTagentThe databaselogger could,alternatively,be run in theJEE container.36WebgatewayapplicationFile systemAudit informationWebSphere MQ Managed File Transfer gger 2014 IBM Corporation
IBM SoftwareComponents used for sending a file to an web-userWeb gateway:Back-endsystemFilespaceUserLogical path for file dataMFTAgentLogical path for file dataWeb gateway systemJEE applicationserverWebSphere MQ networkFrom theperspective of anMFT user, thetransfer to the webgateway system issimilar to any othertransfer.WMQ MFTadministrator37WebSphere MQ Managed File Transfer OverviewWMQQueueManagerMFTagentFile systemHTTP(S)Web browserWebgatewayapplicationDatabaseThe file systemand database areused together toimplement the‘file space.’ 2014 IBM Corporation
IBM SoftwareAgenda Common problems transferring file data Introduction to MQ Managed File Transfer IBM Managed File Transfer Portfolio– Introducing IBM Sterling Commerce products Key MQ Managed File Transfer concepts Usage scenarios for MQ Managed File Transfer Wrap-up–––38ResourcesPlease provide feedbackQuestionsWebSphere MQ Managed File Transfer OverviewYou are here 2014 IBM Corporation
IBM SoftwareResources WMQ V7.5 Information Center:– .jsp WMQ V7.5 Managed File Transfer Introduction:– /com.ibm.wmqfte.doc/wmqfte intro.htm Redbooks/Redguides/Redpapers:– Getting Started with WebSphere MQ Managed File Transfer V7 l– IBM WebSphere MQ Managed File Transfer Solution Overview l– Managed File Transfer for SOA using IBM WebSphere MQ Managed File Transfer l– B2B Enabled Managed File Transfer using WebSphere DataPower B2B Appliance XB60 andWebSphere MQ Managed File Transfer l– IBM Sterling Managed File Transfer Integration and WebSphere Connectivity for a MultiEnterprise Solution l?Open– Multi-Enterprise File Transfer with WebSphere Connectivity l?Open Trial download:– ansfer/ Early design program– Interested in participating in the development of future versions of MFT? Ask your local IBM representative to nominate you for the MFT EDP program.39WebSphere MQ Managed File Transfer Overview 2014 IBM Corporation
IBM SoftwareCopyright and Trademarks IBM Corporation 2014. All Rights Reserved.IBM, the IBM logo, and ibm.com are trademarks orregistered trademarks of International BusinessMachines Corp., registered in many jurisdictionsworldwide. Other product and service names might betrademarks of IBM or other companies. A current list ofIBM trademarks is available on the Web at “Copyrightand trademark information” atwww.ibm.com/legal/copytrade.shtml.40WebSphere MQ Managed File Transfer Overview 2014 IBM Corporation
IBM SoftwareITALIANHINDITRADITIONAL CHINESESPANISHFRENCHRUSSIANJAPANESETAMILBRAZILIAN PORTUGUESETHAISIMPLIFIED CHINESEGERMANARABICWe appreciate your feedback.Please complete the survey form to improve this educational event.41WebSphere MQ Managed File Transfer Overview 2014 IBM Corporation
IBM WebSphere MQ family WebSphere MQ WebSphere MQ for z/OS WebSphere MQ Managed File Transfer WebSphere MQ Adv. Message Security WebSphere MQ Telemetry WebSphere MQ Low Latency for mission critical data for IBM System z investment for managed file transfer for maximum security for sensors and devices for high speed delivery
