Connect 9.13.1 Installation Guide - Docs.precisely

Transcription

Connect 9.13.1 Installation GuideVersion 9.13.1

Table of Contents1 - Connect OverviewConnect Overview.52 - Installed Connect ETLComponentsInstall Connect ETL/Connect for Big Data.7Prerequisites.9Step-by-Step Installation.13Configure the Editor Runtime Service.38Apply a New License Key to an ExistingInstallation.43NoSQL se.86Teradata.86Vertica.87Other DBMSs.89Define ODBC Data Sources.935 - Connect to MessageQueues from ConnectIBM WebSphere MQ.963 - Run Connect6 - Connect to Salesforce fromConnectGraphical User Interfaces.48Connect Help.48Connect to Salesforce from Connect.1004 - Connect to Databases fromConnect7 - Connect to SAP fromConnectConnect to Databases from Connect.50Amazon Redshift.50Azure Synapse Analytics (formerly SQL ve data warehouses.62Apache Impala.71Microsoft SQL Server.74Netezza.75Connect to SAP from Connect.102Register Connect in SAP SLD.103Connect 9.13.1 Installation Guide8 - Connect to HDFS fromConnectConnect to HDFS from Connect.1052

9 - Connect to Connect:Directnodes from Connect15 - Connect InstallationComponent OptionsSecurity.107Install and Configure.107Connect installation component options.121Connect Portal and Management Service installationand configuration.122Connect Portal Copy Runtime Service install andconfiguration.130Connect CloudFSUtil configuration.13410 - Connect to CyberArkEnterprise Password VaultCyberArk Licenses.10911 - Connect to Protegrity DataSecurity GatewayConnect to Protegrity Data Security Gateway.11112 - Connect to QlikView dataeXchange files from QlikViewor Qlik SenseQlikView desktop installation overview.113Qlik Sense desktop installation overview.11313 - Connect to Tableau DataExtract files from TableauTableau desktop installation overview.11514 - Remove ConnectETL/Connect for Big Data fromYour SystemWindows Systems.117Unix Systems.117Connect for Big Data in a Hadoop Cluster.118Connect 9.13.1 Installation Guide3

1 - Connect OverviewIn this sectionConnect Overview.5Connect 9.13.1 Installation Guide

Connect OverviewConnect is a high-performance data transformation product. With Connect you can design, schedule,and control all your data transformations from a simple graphical interface on your Windows desktop.Data records can be input from many types of sources such as database tables, SAP systems,Salesforce.com objects, flat files, XML files, pipes, etc. The records can be aggregated, joined,sorted, merged, or just copied to the appropriate target(s). Before output, records can be filtered,reformatted, or otherwise transformed.Metadata, including record layouts, business rules, transformation definitions, run history and datastatistics, can be maintained either within a specific task or in a central repository. The effects ofmaking a change to your application can be analyzed through impact and lineage analysis.You can run your data transformations directly from your desktop, on any UNIX or Windows server,or schedule them for later execution, embed them in batch scripts, or invoke them from your ownprograms.Connect 9.13.1 Installation Guide5

2 - Installed Connect ETLComponentsIn this sectionInstall Connect ETL/Connect for Big Data.7Prerequisites.9Step-by-Step Installation.13Configure the Editor Runtime Service.38Apply a New License Key to an Existing Installation.43Connect 9.13.1 Installation Guide

Install Connect ETL/Connect for Big DataInstalled Connect components are dependent on your license key: Connect server license key installs components based on whether you select a Standard, Full,Classic, or Custom install. See Connect installation component options. Connect workstation license key installs the Job and Task Editors, Job and Task Editors, theConnect engine (dmxjob/dmexpress), and the service for Job and Task Editors, Connect EditorRuntime Service, dmxd.The version of Connect server software must be at least as high as the version of the Connect clientsoftware to develop jobs and connect to the server. Thus, when installing a new version of Connect,install the same release of Connect on your client and server machines. If you are upgrading andunable to install both client and server at the same time, upgrade the server prior to upgrading theclient.Connect for Big Data OverviewConnect for Big Data is the Hadoop-enabled edition of Connect, providing the following Hadoopfunctionality: ETL Processing in Hadoop – Develop a Connect for Big Data ETL application entirely in the ConnectGUI to run seamlessly in the Hadoop MapReduce framework, with no Pig, Hive, or Javaprogramming required. Currently, jobs can be run in either MapReduce or Spark. See the onlineConnect Help topic "Connect for Big Data”. Hadoop Sort Acceleration – Seamlessly replace the native sort within Hadoop MapReduceprocessing with the high-speed Connect engine sort, providing performance benefits withoutprogramming changes to existing MapReduce jobs. See the Connect for Big Data Sort User Guide,which is included in the Documentation folder under your Connect software installation directory. Apache Spark Integration – Use the Spark mainframe connector to transfer mainframe data toHDFS. See the online Connect Help topic “Spark Mainframe Connector”. Apache Sqoop Integration – Use the Sqoop mainframe import connector to transfer mainframedata into HDFS. See the online Connect Help topic "Sqoop Mainframe Import Connector”.Connect for Big Data RequirementsConnect for Big Data requires the following: Connect for Big Data EditionConnect 9.13.1 Installation Guide7

A supported Hadoop MapReduce and/or Spark distribution: MapReduce Cloudera CDH 5.x (5.2 and higher) – YARN (MRv2)Hortonworks Data Platform (HDP) 2.x (2.3 and higher) – YARNApache Hadoop 2.x (2.2 and higher) – YARNMapR, Community Edition and Enterprise Edition only (previously termed M5 and M7,respectively), 6.x – YARN Pivotal HD 3.0 – YARNConnect for Big Data is certified as ODPi (1.0 and higher) interoperable. Spark Spark on YARN on the following Hadoop distributions: Cloudera CDH 5.x (5.5 and higher) Hortonworks Data Platform (HDP) 2.3.4, 2.x (2.4 and higher) MapR 5.x (5.1 and higher), Community Edition and Enterprise Edition only (previously namedM5 and M7, respectively) Spark on Mesos 0.21.0 Spark Standalone 1.5.2 and higherConnect for Big Data Component Setup and OperationA Connect for Big Data setup consists of the following: Windows workstationConnect must be installed as described in Step-by-Step Installation, Windows Systems.Connect Job and Task Editors are used for MapReduce job development.MapReduce jobs are submitted to Hadoop via the ETL server from the Job Editor.Linux server (edge node)Connect must be installed as described in Step-by-Step Installation, UNIX Systems.The Hadoop client must be installed and configured to connect to the Hadoop cluster.The Editor Runtime Service, dmxd, must be running to respond to jobs run via the Windowsworkstation; it calls dmxjob with the /HADOOP option, which ultimately calls hadoop to submit jobsto the cluster. Hadoop cluster Connect must be installed without dmxd on all nodes in the Hadoop cluster as described inStep-by-Step Installation, Hadoop Cluster. Each mapper and reducer runs the map side or reduce side task(s), respectively.Connect 9.13.1 Installation Guide8

All file descriptors for sources, targets, and intermediate files are carefully connected so they fitinto the Hadoop MapReduce flow.PrerequisitesBefore you install Connect on your system, ensure that the following are available: Connect software: This is generally shipped downloaded from Precisely’s web site as aself-extracting executable file (Windows) or a tar file (UNIX). Connect license key: License keys are sent via e-mail as an attachment file calledDMExpressLicense.txt. If you need specific system information to obtain a license key, refer to thesection below on Getting Connect License Information.If you have a Connect server license key and plan to install Connect installation components, thetype of user required for your setup depends on whether you extend impersonation privileges. SeeConnect installation user setup considerations topic below. Operating system: Connect runs on the following operating systems, with the listed release beingthe minimum supported. Both 32 bit and 64 bit versions are supported, unless otherwise stated:AIX release 6.1 64-bit; HP-UX release 11.31 IA64 64-bit; Linux kernel version 2.6.18 to 2.6.31 withC library version 2.5 to 2.11 on Pentium-class x86 64 64-bit machines; Linux kernel version 2.6.16with C library version 2.4 on IBM System z 64-bit mainframes; SunOS 5.10 SPARC 64-bit; WindowsVista; Windows 7; Windows 8.x; Windows 10; and Windows Server 2008, 2012; and 2012 R2. Java version requirements: On Windows and UNIX/Linux systems, Connect requires Java runtimeversion 1.8 or higher unless you are only running Connect Sort, which does not use Java. Connectfor Big Data requires JDK 8. Communication security protocol: On Windows and UNIX/Linux systems, Connect supportsTransport Layer Security (TLS) up to and including TLS version 1.2. User rights: Sufficient privileges to install and start Windows Services for Windows platforms androot privileges to install and start UNIX daemons on UNIX platforms. An umask setting of 022 isrequired so that other users can run the installed executables. The installation procedure sets andresets umask if required. Pluggable Authentication Modules (PAM): If you want to use PAM for authentication on UNIXor Linux platforms, PAM must be installed and configured on the system. Database client software: If you want Connect to access data in database tables (either as datasource or target), then the appropriate database client software must be on the system andaccessible via the appropriate shared library or dynamic link library (dll) paths. For example, toaccess an Oracle database, Oracle Client must be installed on the system where you run Connect;to access a database via ODBC, an ODBC data source must be defined on the system where yourun Connect. For details on how to connect to a specific Database Management System (DBMS),refer to the section Connecting to Databases from Connect.Connect 9.13.1 Installation Guide9

Message queue client software: If you want Connect to access data in a message queue, thenthe appropriate message queue client software must be on the system and accessible via theappropriate shared library or dynamic link library (dll) paths. For example, to access an IBMWebSphere MQ queue, IBM WebSphere MQ client must be installed on the system where yourun Connect. For details on how to connect to a specific message queue type, refer to the sectionConnecting to Message Queues from Connect. SAP client software: If you want Connect to access data in an SAP system, then the appropriateSAP client software must be installed on the system where you run Connect and accessible viathe appropriate shared library or dynamic link library (dll) paths. For details on how to connect toan SAP system, refer to the section Connecting to SAP from Connect. Hadoop software: If you want Connect to access data in a Hadoop Distributed File System (HDFS),or you want to run Connect for Big Data ETL MapReduce jobs, then a Hadoop distribution configuredto access the cluster must be installed on the edge/ETL node from which you run Connect. Fordetails on how to connect to HDFS, refer to the section Connecting to HDFS from Connect. Connect:Direct software: If you want Connect to access data using a Connect:Direct connection,a Connect:Direct server and client (CLI/API) must be installed on the system where you run Connectand must be configured to access the required Connect:Direct nodes. For details on how to connectto a Connect:Direct node, refer to Connecting to Connect:Direct nodes from Connect. QlikView software: Connect supports QlikView data eXchange (QVX) files as targets. To accessQVX files as sources from QlikView or Qlik Sense, refer to Connecting to QlikView dataeXchange files from QlikView or Qlik Sense. Tableau software: Connect supports Tableau Data Extract (TDE) files as targets. To access TDEfiles as sources from Tableau, refer to Connecting to Tableau Data Extract files from Tableau.Connect installation user setup considerationsThe type of user required for your setup depends on whether you extend impersonation privileges: If you plan to use impersonation when running the Editor Runtime Service, dmxd, you must installas root. When running the Connect Portal Copy Runtime Service, dmxrund, considerations exist for thetype of user that installs components.User setup when running dmxrundIf you do not plan to use impersonation when running dmxrund, setup a non-administrative user toinstall and run on Windows or setup a service user to install and run on Linux.As the administrative user has impersonation privileges by default, setup a new user who does nothave administrative rights.Connect 9.13.1 Installation Guide10

Setup a non-administrative/service user for WindowsAs the administrative user has impersonation privileges by default, setup a new user who does nothave administrative rights.Setup a non-administrative/service user for LinuxTo install and run job requests without impersonation, create a service user, such as connectuser,and run the installation as connectuser.Setup impersonationIf you plan to use impersonation when running dmxrund, setup an impersonated user to install andrun on Linux.WindowsAs the administrative user has impersonation privileges by default, no setup is required.LinuxConnect installation impersonation considerations on Linux follow: No impersonation: Running jobs without impersonation does not require root access. Upon receiptof a job submission request from the dmxrund calls the Connect Engine, dmxdfnl, to run thesubmitted job as the service user, connectuser. Impersonation: Running jobs with impersonation requires root access to impersonate the specifieduser. While dmxrund never is granted root access, another installed component, dmxexecutor, canenable impersonation. When dmxrund detects that dmxexecutor is installed in the required directorywith the correct permissions, dmxrund calls dmxexecutor to impersonate the specific user that callsthe Connect engine, dmxdfnl, which runs the submitted jobs.To install and run job requests with impersonation, do the following: Create a service user, such as connectuser. Create a service group, dmexpress.Note: If you choose to change the name of the service group, you must update theSERVICE GROUP property of the Connect custom impersonation configurationproperties file. Add connectuser to the dmexpress service group. Run the installation as connectuser. Ensure that the following files are in the specified directories with the specified permissions:Connect 9.13.1 Installation Guide11

Directory and filePermissionsNotes Connect installation /bin/ -rwsr-x--dmxexecutorThe ‘s’ represents the set-user identification (setuid) bit and indicatesthat dmxexecutor is extended impersonation privileges to run submittedjobs as a specific user. Connect installation /conf/ -rwx-----dmxexecutor.confUpdates to dmxexecutor.conf are required only if you choose to customizethe impersonation.Setup Connect License InformationTo obtain a license key, you need the computer name, the hardware model, the number of processors,and the operating system of each system on which Connect is to run. You can gather the systeminformation by running the Connect License Information program.Windows SystemsYou can run the Connect License Information program in the following ways: From Precisely’s web site xe If Connect is installed, go to Programs, Connect from the Start menu and select License Information.The program prepares a license information document with the system information and then displaysit in a Notepad window. You can save the form (File, Save As) and e-mail it to Precisely or to yourlocal Connect sales agent. The information is then used to create your license key(s).UNIX SystemsYou can run the Connect License Information program in the following ways: From Precisely’s web site h If Connect is installed, go to the connect install /bin directory, where connect install denotesthe directory where Connect is installed, and type: ./licenseinfoThe program generates and displays a text file named PreciselyLicenseInfo.txt in the current directory.You can e-mail the file to Precisely or to your local Connect sales agent. The information is thenused to create your license key(s).Connect 9.13.1 Installation Guide12

Step-by-Step InstallationWindows Systems1. Make sure that any previous version of Connect has been removed (see Removing Connectfrom Your System later in this guide if necessary).2. You can also install Connect by running\Windows\x86\setup.exe for 32-bit Windows\Windows\x64\setup.exe for 64-bit Windows x64extracted from the downloaded executable directly or via Control Panel, Add/Remove Programs.3. You are prompted to either enter a license key or start a free trial. If you have selected to entera license key, you can type in the location of the DMExpressLicense.txt, or browse to it, whenprompted. You can also enter the license key manually.4. Read the terms of the Precisely License Agreement and confirm your acceptance of them.5. Review the product options, components and features that are enabled by your license key.6. If your license key is a Connect server license key, a menu displays from which you select from among the componentoptions: StandardFullClassicCustomFor information on these options, see Connect installation component options. Select an optionand make the appropriate selections. Connect workstation license key, no component options display for selection.You are eligible for the classic Connect ETL/Connect for Big Data installation, which installs the Joband Task Editors, Job and Task Editors, Connect engine (dmxjob/dmexpress), and the service forJob and Task Editors, Connect Editor Runtime Service, dmxd.Confirm the file folder into which you want to install Connect. The file folder is subsequently referredto as connect install .Select the program folder in which you want the Connect icons to appear.Connect 9.13.1 Installation Guide13

Review the Setup Information, choose back to change these options or install to complete installation.If your license key enables the Editor Runtime Service, select the configuration options for the Service.You can also configure the Editor Runtime Service later via Control Panel, Administrative Tools,Services.You may be prompted to choose whether to automatically run SyncSort jobs in Connect, eitherimmediately or after subsequent un-installation of SyncSort, depending on the presence of theSyncSort Conversion license option and an existing installation of SyncSort.Upon setup completion, a list of menu shortcuts display in the Connect program folder, which isavailable through the Windows Start menu.To run the Connect Portal web UI, you must configure the Connect Portal and ManagementService, DMXManager, including authentication. Then, start the Connect Portal and ManagementService via Control Panel, Administrative Tools, Services.To run copy projects in Connect Portal, start the Connect Portal Copy Runtime Service via ControlPanel, Administrative Tools, Services. See Connect Portal Copy Runtime Service installationand configuration for more details.To run CDC replication projects in Connect Portal, separately install the latest version of ConnectCDC. The Connect CDC Listener service starts automatically when you install Connect CDC, butthe listener service must be running to run CDC replication projects in Connect Portal.If you performed a full install, including the Job and Task Editors, the following menu shortcutsdisplay: Precisely Connect Apply a New License KeyConnect Application UpgradeConnect Global FindConnect HelpConnect Job EditorConnect PortalConnect ServerConnect Task EditorLicense InformationReference GuidesRelease NotesIf you performed a standard or classic install, with the Job and Task Editors but not the ConnectPortal and Management Service, the following menu shortcuts display: Precisely Connect Apply a New License Key Connect Application Upgrade Connect Global FindConnect 9.13.1 Installation Guide14

Connect HelpConnect Job EditorConnect ServerConnect Task EditorLicense InformationReference GuidesRelease NotesIf you installed the Connect Portal and Management Service only (custom install), the following menushortcuts display: Precisely Connect Apply a New License KeyConnect HelpConnect PortalLicense InformationReference GuidesRelease NotesIf you did not install the Job and Task Editors nor the Connect Portal and Management Service, thefollowing menu shortcuts display: Precisely Connect Apply a New License KeyConnect HelpConnect PortalLicense InformationReference GuidesRelease NotesIf you have ActiveX based SyncSort applications which you choose to run with Connect, and yousubsequently uninstall SyncSort, you may need to re-register the SyncSortX ActiveX control. Toregister the ActiveX control, open a command prompt and type the following command:regsvr32.exe connect install /Programs/SyncSortX.dllSilent InstallationSilent installation requires a silent setup file that can be recorded during an interactive installation.Installation steps may differ depending on product licensing, so changing the version of Connect oradding or removing packages may require re-recording the silent setup file.To record the installation optionsConnect 9.13.1 Installation Guide15

Open a command prompt; type the full path for the installation program followed by the options:–r –f1 silent setup file where silent setup file is the full path for the file to record the installation options. If youare installing from a downloaded image which is located in c:\downloads, you would type acommand like:C:\downloads\DMExpress 1-4 windows.exe–r –f1c:\temp\setup.issAn interactive installation starts and all the selected installation options are saved in the specifiedfile.To run the installation in silent modeOpen a command prompt; type the pathname of the install executable followed by the options:–s –f1 silent setup file -slog log file where silent setup file is the full path for the file that was previously used to record theinstallation options, and log file is the full path for the installation log file generated by silentinstallation. If you are installing from a downloaded image which is located inc:\downloads, you would type a command like:C:\downloads\DMExpress 1-4 windows.exe –s–f1c:\temp\setup.issIf you do not specify the –slog option, then setup generates a log of the silent installation, setup.log,in the folder from which the setup is run or in the folder where the specified silent setup file is located.Multiple command line options are separated with a space, but there should be no spaces inside acommand line option (for example, –slogc:\setup.log is valid, but –slog c:\setup.log isnot).Note: When running silent installation on a machine with User Account Control enabled, anadministrator command prompt or batch file can be used to avoid the initial prompt by theoperating system requesting elevated privileges. To start a Command Prompt withadministrative privileges, right-click the Command Prompt shortcut and select "Run asadministrator".UNIX SystemsPrerequisites for COBOL SupportConnect can be used to accelerate COBOL SORT and MERGE verbs or to process COBOL datafiles as source or target. In order to use these features, you must have a license to use the COBOLcompiler on the system where the Connect task runs.Connect 9.13.1 Installation Guide16

Micro Focus COBOL or Server ExpressThe following variables must be set prior to installation: the COBDIR and PATH variables must beset and exported to include the COBOL compiler, and the following environment variable for sharedlibraries must be set to include all the shareable libraries used by the compiler and exported on thecorresponding platform:AIXLIBPATHHP-UXSHLIB PATHLinuxLD LIBRARY PATHSolarisLD LIBRARY PATHAcuCorp’s ACUCOBOL-GT COBOL Development SystemSupport for ACUCOBOL-GT is available on the following UNIX platforms:Operating SystemArchitectureHP-UX64-bit for ItaniumAIX64-bit on PowerPCSunOS64-bit on SPARC processorsThe bit level of Connect must match that of the ACUCOBOL-GT installation.Before running the Connect install script, set the environment variable ACUCOBOL:export ACUCOBOL acucobol install dir where acucobol install dir is the location of your ACUCOBOL-GT installation. If theenvironment variable COBDIR is set, unset it:unset COBDIROnce Connect has been installed, additional steps need to be performed to enable support forAcuCOBOL. Please refer to the Connect online help topic “Installing support for AcuCOBOL.”COBOL-ITSupport for COBOL-IT line sequential files is available on the following UNIX platforms:Connect 9.13.1 Installation Guide17

Operating SystemArchitectureAIX64-bit on PowerPCLinux64-bit for Intel-compatible processorsThe bit level of Connect must match that of the COBOL-IT installation. The minimum supportedCOBOL-IT version is 3.7.Before running the Connect install script, do the following: Unset the environment variable COBDIR, if set:unset COBDIR Set the environment variable COBOLITDIR:export COBOLITDIR cobol-it install dir where cobol-it install dir is the location of your COBOL-IT installation.To configure COBOL-IT runtime environment variables, refer to the Connect online help topic,“Installing support for COBOL-IT.”Informix C-ISAM SupportIf you plan to use Connect to process Informix C-ISAM files, the environment variable INFORMIXDIRmust be set and exported prior to running the install script. The directory INFORMIXDIR/lib mustcontain the library libisam.a.Unikix VSAM SupportIf you plan to use Connect to process Unikix VSAM files, the environment variable UNIKIX must beset and exported prior to running the install script. The directory UNIKIX/lib must contain the librarylibbcisam.a.Interactive InstallationIf you are installing from a tar file that you downloaded from Precisely’s web site, extract the contentsof the tar file on your UNIX system using a command similar to:tar xvof DMEXPRESS.TARThis creates a directory dmexpress under the current directory.Connect 9.13.1 Installation Guide18

Log in as user root if you want to install or configure the Editor Runtime Service. The Editor RuntimeService allows you to submit tasks or jobs from the Connect Task Editor or Job Editor components,running on remote desktops, to execute on this Connect server.To install using downloaded software, navigate to the dmexpress directory created when you extractedthe contents of the tar file and then run the install program. For example,cd /usr/tmp/dmexpress./installDepending on your system and the licensed options, you may be asked several questions. Forexample, on platforms where both a 32-bit and a 64-bit version of Connect are available, you areasked to choose which one you would like to install.You are prompted to either enter a license key or start a free trial. If you have selected to enter alicense key, specify the location of the license key file, DMExpressLicense.txt, when prompted.Read the terms of the Precisely License Agreement and confirm your acceptance of them.Review the product options, components and features that are enabled by your license

HDFS. See the online Connect Help topic "Spark Mainframe Connector". Apache Sqoop Integration - Use the Sqoop mainframe import connector to transfer mainframe data into HDFS. See the online Connect Help topic "Sqoop Mainframe Import Connector". Connect for Big Data Requirements Connect for Big Data requires the following: