Transcription
Hitachi Hyper Scale-Out Platform (HSP)Hortonworks Ambari VM Deployment GuideMK-95HSP017-0311 October 2016
2016 Hitachi, Ltd. All rights reserved.No part of this publication may be reproduced or transmitted in any form or by any means, electronic ormechanical, including photocopying and recording, or stored in a database or retrieval system forcommercial purposes without the express written permission of Hitachi, Ltd., or Hitachi Data SystemsCorporation (collectively, “Hitachi”). Licensee may make copies of the Materials provided that any suchcopy is: (i) created as an essential step in utilization of the Software as licensed and is used in no othermanner; or (ii) used for archival purposes. Licensee may not make any other copies of the Materials."Materials" mean text, data, photographs, graphics, audio, video and documents.Hitachi reserves the right to make changes to this Material at any time without notice and assumes noresponsibility for its use. The Materials contain the most current information available at the time ofpublication.Some of the features described in the Materials might not be currently available. Refer to the most recentproduct announcement for information about feature and product availability, or contact Hitachi DataSystems Corporation at https://support.hds.com/en us/contact-us.html.Notice: Hitachi products and services can be ordered only under the terms and conditions of the applicableHitachi agreements. The use of Hitachi products is governed by the terms of your agreements with HitachiData Systems Corporation.By using this software, you agree that you are responsible for:1) Acquiring the relevant consents as may be required under local privacy laws or otherwise fromauthorized employees and other individuals to access relevant data; and2) Verifying that data continues to be held, retrieved, deleted, or otherwise processed in accordance withrelevant laws.Notice on Export Controls. The technical data and technology inherent in this Document may besubject to U.S. export control laws, including the U.S. Export Administration Act and its associatedregulations, and may be subject to export or import regulations in other countries. Reader agrees tocomply strictly with all such regulations and acknowledges that Reader has the responsibility to obtainlicenses to export, re-export, or import the Document and any Compliant Products.Hitachi is a registered trademark of Hitachi, Ltd., in the United States and other countries.AIX, AS/400e, DB2, Domino, DS6000, DS8000, Enterprise Storage Server, eServer, FICON, FlashCopy,IBM, Lotus, MVS, OS/390, PowerPC, RS6000, S/390, System z9, System z10, Tivoli, z/OS, z9, z10, z13,z/VM, and z/VSE are registered trademarks or trademarks of International Business Machines Corporation.Active Directory, ActiveX, Bing, Excel, Hyper-V, Internet Explorer, the Internet Explorer logo, Microsoft,the Microsoft Corporate Logo, MS-DOS, Outlook, PowerPoint, SharePoint, Silverlight, SmartScreen, SQLServer, Visual Basic, Visual C , Visual Studio, Windows, the Windows logo, Windows Azure, WindowsPowerShell, Windows Server, the Windows start button, and Windows Vista are registered trademarks ortrademarks of Microsoft Corporation. Microsoft product screen shots are reprinted with permission fromMicrosoft Corporation.All other trademarks, service marks, and company names in this document or web site are properties oftheir respective owners.
ContentsPrefacevIntended audienceProduct versionRelease notesDocument revision levelDocument conventionsAccessing product documentationGetting helpCommentsDeploy the Ambari VMs on HSPviviviviviiviiviiiviii1Before you beginProcedureChange the ssh key (optional steps)Deploy the Ambari templateInstall script package22466Setup Ambari7Procedure8Configure the new cluster9Procedure10ContentsHortonworks Ambari VM Deployment Guideiii
ivContentsHortonworks Ambari VM Deployment Guide
PrefaceThis document describes and provides instructions for deploying the Ambari2.2.2 and HDP 2.4 VM template on Hitachi Hyper Scale-Out Platform (HSP).This Preface includes the following information: Intended audience Product version Document revision level Document conventions Accessing product documentation Getting help CommentsPrefaceHortonworks Ambari VM Deployment Guidev
Intended audienceIntended audienceThis document is intended for system administrators, Hitachi Data Systemsrepresentatives, and authorized service providers who need to deploy theAmbari 2.2.2 and HDP 2.4 VM template on Hitachi Hyper Scale-OutPlatform (HSP).Readers of this document should be familiar with the following: Ambari administration Linux operating system and working in a restricted shell environment Site-specific network informationProduct versionThis document applies to Hyper Scale-Out Platform release 1.2.0.037 orlater.Release notesThe release notes for this product are available on Hitachi Data SystemsSupport Connect: https://support.hds.com/en us/contact-us.html.Read the release notes before installing and using this product. They maycontain requirements or restrictions that are not fully described in thisdocument or updates or corrections to this document.Document revision levelRevisionviDateDescriptionMK-95HSP017-00April 2016Initial releaseMK-95HSP017-01May 201601, supersedes and replaces MK-95HSP01700MK-95HSP017-02June 201602, supersedes and replaces MK-95HSP01701MK-95HSP017-0311 October 201603, supersedes and replaces MK-95HSP01702PrefaceHortonworks Ambari VM Deployment Guide
Document conventionsDocument conventionsThis document uses the following typographic t in a window or dialog box, such as menus, menu options,buttons, and labels. For example:On the Add Pair dialog box, click OK.Text appearing on screen or entered by the user. For example:The -split option Italic in anglebracketsIndicates a variable, which is a placeholder for site- or installationspecific details that you need to provide. For example:copy source-file target-file monospaceIndicates text that is displayed on the screen or text that you needto enter. For example:# pairdisplay -g oradbAlso, the name of a directory, folder, or file. For example:The horcm.conf file This document uses the following icons to draw attention to information:IconMeaningDescriptionImportantCalls attention to information that is essential to thecompletion of a task.CautionWarns that failure to take or avoid a specified action couldresult in adverse conditions or consequences (for example,loss of access to data).Accessing product documentationProduct documentation is available on Hitachi Data Systems SupportConnect: https://knowledge.hds.com/Documents. Check this site for themost current documentation, including important updates that may havebeen made after the release of the product.PrefaceHortonworks Ambari VM Deployment Guidevii
Getting helpGetting helpHitachi Data Systems Support Portal is the destination for technicalsupport of products and solutions sold by Hitachi Data Systems. To contacttechnical support, log on to Hitachi Data Systems Support Connect forcontact information: https://support.hds.com/en us/contact-us.html.Hitachi Data Systems Community is a global online community for HDScustomers, partners, independent software vendors, employees, andprospects. It is the destination to get answers, discover insights, and makeconnections. Join the conversation today! Go to community.hds.com,register, and complete your profile.CommentsPlease send us your comments on this document:hsp.documentation.comments@hds.comInclude the document title and part number, including the revision (forexample, -01), and refer to specific sections and paragraphs wheneverpossible. All comments become the property of Hitachi Data Systems.Thank you!viiiPrefaceHortonworks Ambari VM Deployment Guide
1Deploy the Ambari VMs on HSPThis chapter describes the procedure for deploying the Ambari 2.2.2cluster. At the end of this procedure the VM instances will be deployed andready to be configured. Before you begin ProcedureDeploy the Ambari VMs on HSPHortonworks Ambari VM Deployment Guide1
Before you beginBefore you beginVerify that you have: An initialized HSP cluster running at least HSP 1.2.0.037. Ensure thatyour HSP cluster has its DNS domain name, DNS server and NTP serverconfigured correctly. Downloaded copy of the tgz file containing the vm-template for Ambarifrom Hortonworks. This file is available from Technical InformationService Center (TISC), and may be called something like CentOS-7.2HSP-1.2 r1.tgz Downloaded copy of the TGZ file containing the repository-vm forAmbari from Hortonworks. This file is available from TechnicalInformation Service Center (TISC), and may be called something likeRepo-with HDP-2.4 Ambari-2.2.2 r1.tgz Linux client.ProcedureFirst, add the vm-template to the HSP cluster.1. Extract the two tgz files so that you have an ISO file for the vm-templateand an ISO file for the repository VM template.2. Extract the qcow2 images from the ISO files. On the Linux client, mountthe ISO image as shown below:linux# mount –t iso9660 –o loop /tmp/CentOS-7.2 HSP-1.2 r1.iso/mnt/isolinux# mount –t iso9660 –o loop /tmp/Repo-with HDP-2.4 Ambari-2.2.2r1.iso /mnt/iso3. Mount a share from the HSP cluster on your Linux system:linux# mount HSP Cluster Virtual IP Address :/ myshare /mnt/hspWhere /myshare is the name of the available share.4. Copy the qcow2 images from the ISOs to the share:linux# cp /mnt/iso/CentOS-7.2 HSP-1.2 r1.qcow2 /mnt/hsplinux# cp /mnt/iso/Repo-with HDP-2.4 Ambari-2.2.2 r1.qcow2 /mnt/hsp2Deploy the Ambari VMs on HSPHortonworks Ambari VM Deployment Guide
ProcedureIf you will be powering on the template in sandbox mode (for example,to Change the ssh key (optional steps)) the CentOS qcow2 imageneeds to be writeable:linux# chmod 666 /mnt/hsp/CentOS-7.2 HSP-1.2 r1.qcow25. On the HSP cluster, add the vm-template and RepositoryVM-template:admin@Node003: hspadm vm-template add --name ambari template1 -image-format qcow2 --image-path myshare :/CentOS-7.2 HSP-1.2r1.qcow2 --use-address-pool Yadmin@Node003: hspadm vm-template add --name RepositoryVM template--image-format qcow2 --image-path myshare :/Repo-with HDP-2.4Ambari-2.2.2 r1.qcow26. Verify this was successful:admin@Node003: hspadm vm-template list7. Deploy a single instance of the Repository-VM:admin@Node003: hspadm vm-instance add --name RepositoryVM --vmtemplate RepositoryVM template --vm-size small --use-address-pool Y8. Verify this was successful and make a note of the IP address of theRepository-VM (in this example it is 10.129.71.30):admin@Node003: hspadm vm-instance listYou can also check that the repository is running by browsing to itsIP address.Deploy the Ambari VMs on HSPHortonworks Ambari VM Deployment Guide3
Procedure9. The Ambari installation process described later in this document utilizesan ssh private key for password-less ssh.You can either:¡Use the default ssh key which is included in the vm-template.Go to Deploy the Ambari template.or¡Change the ssh key.Go to Change the ssh key (optional steps)Change the ssh key (optional steps)If you want to change the ssh key, please follow the steps in this section.a. Deploy an Ambari VM instance in sandbox mode.admin@Node003: hspadm vm-instance add --name ambari sandbox -vm-template ambari template1 --is-sandbox Yb. Verify this is successful, and make a note of the IP address of theinstance (in this example the IP address is 10.129.71.20):admin@Node003: hspadm vm-instance listc. Log into the vm sandbox instance.ssh 10.129.71.20 (sandbox instance address)Username: rootPassword: smrace14Deploy the Ambari VMs on HSPHortonworks Ambari VM Deployment Guide
Procedured. Generate a new ssh key.First, look at the contents of the existing ssh private key.[root@ambari sandbox ]# cat .ssh/id rsaThe current ssh key is displayed.Now create the new ssh key.[root@ambari sandbox ]# ssh-keygenOutputGenerating public/private rsa key pair.Enter file in which to save the key (/root/.ssh/id rsa):/root/.ssh/id rsa already exists.Overwrite (y/n)? yEnter passphrase (empty for no passphrase):Enter same passphrase again:Your identification has been saved in /root/.ssh/id rsa.Your public key has been saved in /root/.ssh/id rsa.pub.The key fingerprint is: The new ssh key is displayed.e. Examine the contents of the new ssh private key to verify that it haschanged.[root@ambari sandbox ]# cat .ssh/id rsaYou can now use this new ssh key in your Ambari installation.f. Update authorized keys:[root@ambari sandbox ]# cat .ssh/id rsa.pub .ssh/authorizedkeysg. From the HSP cluster, power off the VM sandbox instance:admin@Node003: hspadm vm-instance shutdown --name ambarisandboxh. Go to Deploy the Ambari template below.Deploy the Ambari VMs on HSPHortonworks Ambari VM Deployment Guide5
ProcedureDeploy the Ambari template1. Deploy the vm-template. In this example we are creating a 5 nodeHadoop cluster.admin@Node003: hspadm vm-instance-group add -–name ambaridev -vm-template ambari template1 --vm-size medium -–num-instances 5 -use-address-pool Y2. Verify this was successful:admin@Node001: hspadm vm-instance list --template ambaritemplate1Install script packageHSP provides a script that automates much of the setup required byAmbari. For example the script sets up the /etc/hosts file so that all ofthe nodes can find each other. These scripts are packaged as an RPM whichcan be fetched from the Repository-VM. The node is not configured to usethe Repository VM yet so wget is used instead.1. SSH to any one of the Ambari VM instances, this will become the Ambariserver (in this example 10.129.71.21 is used):ssh 10.129.71.21 (VM instance address)Username: rootPassword: smrace12. Fetch the script RPM from the Repository VM (in this example theRepository VM IP address is 10.129.71.30):wget oarch.rpmVerify the RPM was successfully fetched to the current directory.3. Install the script package using yum:yum -y install AmbariPrep-2.2.2-1.noarch.rpm6Deploy the Ambari VMs on HSPHortonworks Ambari VM Deployment Guide
2Setup AmbariThis chapter describes the procedure for setting up Ambari. At the end ofthis procedure the cluster will be ready to be configured. ProcedureSetup AmbariHortonworks Ambari VM Deployment Guide7
ProcedureProcedure1. Log into the first vm-instance. This instance will become the Ambariserver.ssh 10.129.71.21Username: rootPassword: smrace12. Run the install, setup and start of Ambari server. In this example therepository VM IP address is 10.129.71.30, this will enable all of theAmbari VM instances to access the repository:/opt/escale/bin/ambariPrep.py --repo-host 10.129.71.30There are several options that can be given, these can be seen by usingthe –h option. The vm-instances are configured to inherit the domainname that is set on the HSP cluster.Note: the script creates the file system supplied via the –-fsname option. The script willfail if the filesystem was created prior.Example:[root@ambaridev-1 ]# /opt/escale/bin/ambariPrep.py -repo-host RepositoryVM IP Address --fsname ambaridevfs--domain gse.hds.com --user admin --pw admin password Upon completion of ambariPrep.py, ambari will have been installed,configured and started. Please note that it can take the Ambari serverup to 3 minutes to start responding on port 8080 for the GUI. You canfollow the progress on ambaridev-1 (i.e. the instance on which you ranambariPrep.py) by:[root@ambaridev-1 ]# cd /var/log/ambari-server[root@ambaridev-1 ]# tail –f ambari-server.logWait for text similar to:Example Output04 May 2016 16:16:16,003 INFO [main] AmbariServer:548 - *********Started Services **********8Setup AmbariHortonworks Ambari VM Deployment Guide
3Configure the new clusterThis chapter describes the procedure for configuring the new Ambaricluster. At the end of this procedure the Ambari cluster will be configuredfor use. ProcedureConfigure the new clusterHortonworks Ambari VM Deployment Guide9
ProcedureProcedure1. Open up a browser on a desktop that has access to the vm-instancerunning Ambari. This is the Ambari server vm-instance created in theprevious section.Connect to:http://10.129.71.21:8080The login will be:Username adminPassword admin10Configure the new clusterHortonworks Ambari VM Deployment Guide
Procedure2. The wizard screen will come up next. Click Launch install Wizard.3. Assign a name to the new Hadoop cluster. Click Next .Configure the new clusterHortonworks Ambari VM Deployment Guide11
Procedure4. Select the HDP stack version to load on the new cluster.Currently only HDP 2.4 is available with this template. Click Next .5. Install options. On this page we will need to gather information from theAmbari server.Ssh 10.129.71.21Username: rootPassword: smrace1First we need to gather the FQDNs of each of the vm-instances that willrun Ambari agents and be a part of the Hadoop cluster. TheambariPrep.py script has already setup the hosts file which is where theinformation can be pulled from.Example:[root@ambaridev-1 ]# grep gse /etc/hosts awk '{print idev-5.gse.hds.com12Configure the new clusterHortonworks Ambari VM Deployment Guide
ProcedurePaste the output into the Target hosts section.Second we will need to get the private ssh key from the Ambari server.This must be pasted into the text box below the radio button indicatingProvide your SSH Private Key. Please copy and paste from id rsa.For example:[root@ambaridev-1 ]# cat .ssh/id rsaOutput-----BEGIN RSA PRIVATE MR3EEH8Un8HXp6btbcuVe oIBAQCOhcqgJUBU7OJr8o WoxIWD7DOc0 jtJK4S Kvy0yaS8ApJhIZERhC W9nX2G/xbs0fJiwXfg6PJJZLr5VBrnpTP n6/vTGXesgNsHSXl/rCXUI1fJgjPfAoGBAN bt8g17 8QzcNq1om CmApE7m7H1XMnFOo4AyDtVtTFvwrc LuYW2FX6Kb6HCS8vt2wCBVY/gUhR FPGO9oR8RPn0W0pVfWLacc9ZKQNeKnCzsA0zhGzzUT ECgYEAyki BJtUR3sPwBlFgFLBENnvdkcyUFfSHsL77py8tUMBKwxeI ej8oF v/kdPe6 TIneQtzkEBnywqHpSCMyGsPmq2qexICrdaLc/1UxKcRr/1j baxJ9IMMI -----END RSA PRIVATE KEY----Important: This is the default ssh private key provided with the VM template. Theprocedure required to generate a new set of ssh keys is described in Change the ssh key(optional steps).Configure the new clusterHortonworks Ambari VM Deployment Guide13
Procedure6. Click Register and Confirm.7. All hosts should show success. Click Next .14Configure the new clusterHortonworks Ambari VM Deployment Guide
Procedure8. Choose Services.Install the following: Yarn Mapreduce2TezHiveHBasePigSqoopZookeeperAmbari MetricsEscaleFSSparkCurrently not in the repo: OozieFalconStormFlumeAccumuloAtlasKafk
2016Hitachi,Ltd.Allrightsreserved. Nopartofthispublicationmaybere
