WIXLIB

Test report: Talend Enterprise Data Integration – Big Data EditionData processing goes bigDr. Götz GüttichTalend Enterprise Data Integration is a powerful set of tools to access, transform,move and synchronize data. With more than 450 connectors, it can shift data fromany number of different sources to just as many targets. The Big Data Edition ofthis solution is optimized to take advantage of Hadoop and itsdatabases or technologies such as HBase, HCatalog, HDFS, Hive, Oozieand Pig. IAIT decided to put Talend Enterprise Data Integration throughits paces and see how it stacks up in the real world.When IT specialists talk about‘big data’, they are usuallyreferring to data sets that are solarge and complex that they canno longer be processed withconventional data managementtools. These huge volumes ofdata are produced for a variety ofreasons. Streams of data can begenerated automatically (reports,logs, camera footage, etc.). Orthey could be the result ofdetailed analyses of customerbehavior (consumption data),scientific investigations (theLarge Hadron Collider is an aptexample) or the consolidation ofvarious data sources.These data repositories – whichtypically run into petabytes andexabytes – are hard to analyzebecause conventional databasesystems simply lack the muscle.Big data has to be analyzed inmassively parallel environmentswhere computing power isdistributed over thousands ofcomputers and the results aretransferred to a central location.The Hadoop open sourceplatform has emerged as thepreferred framework foranalyzing big data. Thisdistributed file system splits theinformation into several datablocks and distributes theseblocks across multiple systems inthe network (the Hadoop cluster).By distributing computing power,1Hadoop also ensures a highdegree of availability andredundancy. A "master node"handles file storage as well asrequests.

Talend Enterprise DataIntegrationHadoop is a very powerfulcomputing platform for workingwith big data. It can acceptexternal requests, distribute themto individual computers in thecluster and execute them inparallel on the individual nodes.The results are fed back to acentral location where they canthen be analyzed.However, to reap the benefits ofHadoop, data analysts need away to load data into Hadoop andsubsequently extract it from thisopen source system. This iswhere the Big Data Edition ofTalend Enterprise DataJaspersoft, SAP, Amazon RDSand Salesforce and supports avariety of databases such as DB2,Informix and – obviously –Hadoop.How does it work?Talend Enterprise DataIntegration is a code generator.All the user needs to do is definea data source – a CSV file or adatabase for instance – and thenspecify the operation to beperformed on that data. In thecase of a CSV file, for example,the user will specify details likeencoding method and fieldseparator, although the optionsavailable will of course varyfrom source to source. As soon asUsers are presented with a welcome screen when they launch TalendEnterprise Data IntegrationIntegration comes in. Aspreviously mentioned, TalendEnterprise Data Integration canread data from just about anysource, process the data asinstructed by the user and thenexport it. Taking a simple job asan example, this tool can readCSV files, select certain fieldslike name or address, and exportthe results to an Excel file. Butits capabilities don’t end there: Italso works with businessintelligence solutions likethe data source is set, the usercan link it to their workspace asan icon.Next, the user can define theirpreferred workflow. They canmanipulate the data in a numberof ways – filter, sort, replace,transform, split, merge andconvert. There is also a mapfunction for transforming data.This lets users select specific datafields, rearrange these fields,automatically insert additional2data like numbering and muchmore. Each of thesetransformation features comeswith its own icon which can bedragged and dropped to theuser’s workspace and configuredthere.Once the data source has beendefined and the user has specifiedhow the tool should process theinformation, the next task is todefine the export settings. Forthis, Talend offers variousconnectors to supported targetsystems such as Informix orHadoop. Talend Enterprise DataIntegration uses icons for theconnectors, too. Again, the usercan simply drag and drop theicons and configure them on theirworkspace. Here again, theconfiguration options depend onthe export medium. In the case ofan Excel spreadsheet, forexample, the user just needs toenter the export path.Lines are used to illustrate theflow of data between icons.Analytics users can usually justdrag these (sometimes they alsoneed to select certain connectiontypes from a menu). The job canbe started once all of these stepshave been completed. Firstly,Talend Enterprise DataIntegration creates the coderequired to execute the job. Thenit starts the job and transformsthe data. The generated code canbe in Java or SQL. Depending onthe technology used, thegenerated code can be in Java orSQL for Hadoop, MapReduce,Pig Latin, HiveQL and others.Since each job step is symbolizedby an icon – for which users onlyhave to specify the generalrequirements in order to generatethe code automatically – usersdon’t need programming skills to

run complex, code intensive dataprocessing jobs.The above example is a simpleone to illustrate the process, butTalend Enterprise Databetween the subscription andopen source products is extrasupport options (including SLAs)and additional features such asshared repositories, wizards,shared jobs, version control,When they use a CSV file as their source, users can configure all kinds ofparametersIntegration can handle muchmore complex tasks such asimporting data and then mappingspecific fields, transformingcertain data types and sorting themodified output prior to export.VersionsTalend software is available in anumber of editions, starting withfreely downloadable open sourceproducts Talend Open Studio forData Integration and TalendOpen Studio for Big Data. Theseare available on the vendor’swebsite and are free to downloadand use. The subscription basedproduct is called TalendEnterprise Data Integration and isavailable in four editions: Team,Professional, Cluster and BigData. The main differencereference projects, to mention buta few. The subscription versionsthemselves offer differentcombinations of features,including support for loadbalancing, high availability andHadoop. Talend is in fact one ofthe first vendors to provideHadoop support.The website provides acomparative matrix of theavailable software versions andtheir functions. For our test, welooked at the Big Data Edition ofTalend Enterprise DataIntegration, which requires asubscription. It is fair to pointout, however, that the opensource product Talend OpenStudio for Big Data has quite anextensive set of functions, and3would be perfectly suitable for allinternal data transformation jobs.Administrators looking to fast track import and export scriptwriting should look at what thesubscription products have tooffer.The testFor the test we used anenvironment with Hadoop 1.0.3running in a vSphere installationbased on IBM’s X Architecture.After installing Talend EnterpriseData Integration on a workstationoperating on the x64 edition ofWindows 7 Ultimate, we startedby importing data from a CSVfile. We transformed the data andexported it as an Excelspreadsheet to become familiarwith the solution’s functionality.We then set up a connection toour Hadoop system, imported thesame CSV data again and wroteit to Hadoop. Then, we re exported the data to check thateverything had worked properly.For the next step, we took100,000 or ten million records ofcompany data. We analyzed thisdata using Pig. Finally, we usedHive to access the data inHadoop and worked with theHBase database. We will coverthese terms in greater detail lateron. As for specifications, TalendEnterprise Data Integration runson Java 1.6 – the latest version.The vendor recommends the 64 bit version of Windows 7 as theoperating system and a standardworkstation with 4 GB ofmemory.Installing Talend EnterpriseData IntegrationInstalling Talend Enterprise DataIntegration is straightforward.The user just needs to make surethat their system runs a supported

version of Java and then unzipthe Talend files to a folder oftheir choice (e.g. c:\Talend).allow the user to defineautomated tasks like the splittingof fields.The next step is to launch theexecutable file. The system willinitially ask for a valid license (alicense key is, of course, notrequired in the open sourceeditions). Once the key has beenvalidated, the user will see ascreen with some general licenseterms. After the users click toconfirm, they can get started bycreating a repository (withworkspace) and setting up theirfirst project. When they open thisproject they will be presentedwith the development tool’swelcome screen, which guidesthe users through the first steps.The workspace is located in thetop center of the screen. Here, theuser can define jobs using icons.Working with TalendEnterprise Data IntegrationThe design environment is basedon the Eclipse platform. Arepository on the left allows theuser to define items like jobs,joblets and metadata. The jobsoutline the operations –represented by icons – to beperformed on the data. Themetadata can be used to set upfile, database and SAPconnections, schemas, etc. Andthe joblets create reusable dataintegration task logic that can befactored into new jobs on amodular basis.Performing an analysis job with Pig, running here on the Hadoop serverThe Code subfolder contains twointeresting additionalfunctionalities. Job scripts, on theone hand, are processdescriptions – i.e. codegeneration instructions – in XMLformat. Job scripts can provide afull description of the processes,so users can implement functionsfor the import of tabledescriptions, for example.Routines, on the other hand,Underneath, the context sensitiveconfiguration options can be setfor the selected icon. The optionsfor starting and debugging jobsand lists of errors, messages andinformation are also found here.A panel on the right of the screencontains a Palette with thevarious components in icon form.These include the import andexport connectors as well as thefunctions for editing data,executing commands and so on.Users can also integrate theirown code into the system at anytime. The Palette therefore holdsall of the components that can bedragged to the workspace anddropped.The first jobsFor the test, we carried out thefirst job at this point, importing aCSV file and then writing thedata to an Excel spreadsheet. Aswe have already basicallydescribed this job in the4introduction, we will move on tothe job of writing the data from aCSV file to Hadoop. For this job,our source was the CSV file thatwas pre defined under metadatawith its configuration parameterslike field separator and encoding.After dragging it to theworkspace, we defined an exporttarget. For this, we selected thetHDFSOutput component fromthe Big Data folder in the Paletteand dragged the icon next to oursource file. The HDFS in thename stands for HadoopDistributed File System.The next task was to configurethe output icon. After clicking theicon, we were able to enter therequired settings in theComponent tab below theworkspace. These settingsincluded Hadoop version, servername, user account, target folderand name of target file inHadoop. For our test, we againused a CSV file as our target.To finish, we had to create aconnection between the twoicons. We did this by right clicking the source icon anddragging a line as far as theHDFS icon.