Transcription
Isilon Advisor User GuideTHIS TECHNICAL DOCUMENT IS FOR INFORMATIONAL PURPOSES ONLY, AND MAY CONTAIN TYPOGRAPHICAL ERRORS ANDTECHNICAL INACCURACIES. THE CONTENT IS PROVIDED AS IS, WITHOUT EXPRESS OR IMPLIED WARRANTIES OF ANY KIND.Copyright 2009 - 2016 Dell Inc. All rights reserved. Dell and the Dell EMC logo are trademarks of Dell Inc. in the United States and/or otherjurisdictions. All other marks and names mentioned herein may be trademarks of their respective companies.Page 1
Isilon AdvisorTopicsWelcome to Isilon AdvisorRunning Isilon AdvisorLog Set Collection (Gather)Command Line Interface (CLI)User InterfaceSettingsCheck DescriptionsCluster ChecksCluster Configuration ChecksNode ChecksNode Status ChecksDisk ChecksRunning an AnalysisAnalysis ResultsExporting ResultsGenerating a ReportSharing and Receiving ResultsPrevious RunsUpgrade AdvisorFAQRequesting Access to the Log Processor (Internal Users only)SupportPage 2
Welcome to Isilon AdvisorDescriptionThe EMC Isilon Advisor is a smart, fast and effective application that enables customers to self-support common Isilon issues and acceleratetime-to-resolution. This is the same application used by EMC Isilon Technical Support Engineers and Field Representatives to resolve servicerequests. Use it to diagnose, troubleshoot and proactively avoid issues by analyzing the current health of your cluster and listing items that requireattention.Items that require attention can range from simple checks to critical alerts. They are listed in order of importance and color-coded for quick visualreference (red: critical, yellow: needs attention, green: no problem found). More importantly, Isilon Advisor provides links to documentation onhow to resolve the issues identified and if further assistance by EMC Support or Field is needed, a summary of each check can be extracted to aflat file and attached to a Service Request (SR). This information will aid the EMC team in resolving your issue faster.Isilon Advisor performs the following checks:Pre-upgradePost-upgradeGeneral HealthFirmwareCustom CheckFeaturesProactively identify issues and potential solutionsRun reports on 50 different checksTake preventative action on known issuesGain visibility into the health of your environment in minutesBenefitsAccelerate Time-to-ResolutionIA enables you to resolve documented known issues without engaging EMC. You can resolve issues immediately after beingdiagnosed therefore eliminating one business day or longer from the troubleshooting process.Proactively Avoid IssuesThe built-in Health Check feature diagnoses current issues and identifies potential ones. Leverage years of collective knowledgefrom EMC and routinely run the Health Check to ensure good hygiene.Reduce Capital and Operational ExpensesBy resolving issues early you reduce or eliminate future critical issues that may result in data loss, data unavailability anddowntime. Faster time-to-resolution equates to less time spent troubleshooting and faster restoration to peak performance andavailability.Quickly Assess Your EnvironmentThe Isilon Advisor provides visibility into the overall health of your environment within a matter of minutes, enabling you tosustain optimal cluster health with minimal overhead and time spent.Intended UsersIsilon Advisor is intended for Isilon customers, EMC personnel, and business partners.Page 3
Running Isilon AdvisorOverview and DownloadThe EMC Isilon Advisor is a smart, fast and effective application that enables customers to self-support common Isilon issues and acceleratetime-to-resolution. Use it to diagnose, troubleshoot and proactively avoid issues by analyzing the current health of your cluster and listing itemsthat require attention.See Isilon Advisor for additional information and a link to the download.Running the ApplicationIsilon Advisor is a self-contained single file executable and is not installed. Simply copy the appropriate executable from the zip file into thedirectory of your choice and execute it. This launches both a command window to run the web server and your default browser usinglocahost:{port} and the default ports of 8080, 8081 and 8088. Optionally, you can specify a specific port using the Command Line Interface (CLI).When executed for the first time, an IsilonAdvisor directory is created in one of the following locations:Windows: C:\Users\ username \IsilonAdvisor\Linux: /home/ username /IsilonAdvisor/ScriptingAs of version 2.3.0, scripting is supported through the Command Line Interface (CLI).Space ConsiderationsThe minimum free space required to run Isilon Advisor is 4GBIn addition, tgz files require that you must have enough space to uncompress the entire log set and still have free space leftLog sets can be gigabytes in size and are contained in a tarball which has to be unfurled, so plan accordingly for the necessary free space forboth the tarball and the unfurled data.Log SetsYou must use full log sets gathered with the isi gather command. For collection details, refer to Log Set Collection (Gather).OneFS SupportThe following OneFS versions are supported:7.1.x7.2.x8.0.xOperating System SupportThe following operating systems are supported:Windows 7 and 10Ubuntu 15.x or 16.x (Linux)Browser SupportThe following browsers are supported:Chrome 39 or laterFirefox 34 or laterSafari 7 or laterPage 4
Beware that unsupported browsers, such as Internet Explorer (IE) may cause erratic and undesirable behavior with inputs andformatting.Page 5
Log Set Collection (Gather)Where are my Isilon logs?Compressed log sets can be found at "/ifs/data/Isilon Support/pkg/"The log folder will have a naming convention of ecting LogsSee the below procedure for how to gather logs and configuration files from the cluster using the OneFS web administration interface and the isigather info command, or for a demonstration of these procedures, see the video below.ProcedureYou can gather logs using the OneFS web administration interface or the command-line interface.OneFS web administration interfaceOneFS 7.x1. Log in to the OneFS web administration interface.2. Click Help Diagnostics.3. Click Start Gather.It takes several minutes to gather the logs. When the process is complete, the logs are automatically uploaded to the Isilon FTP site. Thecompleted log gather is listed in the Archived Info Manager section.OneFS 6.0 - 6.51. Log in to the OneFS web administration interface.2. Click Help Diagnostics Gather Info.3. Click Start Gather.It takes several minutes to gather the logs. When the process is complete, the logs are automatically uploaded to the Isilon FTP site. Thecompleted log gather is listed in the Archived Info Manager section.Command-line interface1. Open an SSH connection on any node in the cluster and log on using the "root" account.2. Run the following command:isi gather infoIt takes several minutes to gather the logs. When the process is complete, the logs are automatically uploaded to the Isilon FTP site.NOTEIf problems are encountered during the upload process, the package will need to be sent manually to ftp.isilon.com.You can find the package in the following location, where timestamp is the date and time that the script was run:/ifs/data/Isilon Support/pkg/IsilonLogs-mycluster- timestamp .tgzFor detailed instructions for uploading files to ftp.isilon.com, see How to upload files to Isilon Technical Support, 16759.NOTEFor a list of the options that can be used with the isi gather info command, run the following command from the command line:isi gather info -hPage 6
Command Line Interface (CLI)Using the CLI enables users to create a script to run Isilon Advisor against multiple logs, automatically run a specific check against those logs,and export the results. It also enables a user to use an alternate port to run multiple instances of Isilon Advisor at the same time, or because thedefault ports 8080, 8081 and 8088 are already in lth Check", "Firmware Check", "PreUpgradeCheck", "PostUpgrade Check"YesThis is the check that the application will run.-outputPath to where the output should be placedYesThis is the folder where the results will be placed.-logsetPath to the log set or a folder nameYesSpecifies if a folder in which case all files located in that folder, or a singlefile in which case that single file will be processed.-exportResultsyes noNoIf set to yes, then the exported results will be triggered and placed in thespecified output folder-srNumber0 - 999999999NoOptional and contains the SR that is being used for this analysis.-helpN/ANoDisplays a list of available options.-port1024 - 65535NoThis is the port that the IsilonAdvisor will use. If no value is specified, theapplication will use 8080, 8081 or 8088.i.e. http://localhost:{port}ExamplesRun a health check and export results to a fileIsilonAdvisor 2.x.x os -check "Health Check" -output "c:\data\report output" -logset "c:\data\logs\logfile123.tgz" -exportResults yes -srNumber 12345678Runs the Health Check analysis on the log file "c:\data\logs\logfile123.tgz" and place the results and the exported results into the"c:\data\report output" folder.Run Isilon Advisor on an alternate portIsilonAdvisor 2.x.x os -port 45007Runs the IsilonAdvisor on port 45007 and start up the default browser on http:\\localhost:45007.Display the CLI parametersIsilonAdvisor 2.x.x os -helpDisplays the help information for the IsilonAdvisor CLI and exit.Validation1. If the required parameters passed are in error or not present, the application will not start in any form.2. If the port range is not valid the application will not start.3. If the port passed is already in use, the application will not start.Page 7
User InterfaceResponsive Web DesignThe user interface uses a responsive web design which is an approach aimed at providing an optimal viewing experience across multipleplatform/devices: desktop, mobile and tablet. It provides easy reading and navigation with a minimum of resizing, panning, and scrolling. Forexample, when the browser window changes from landscape to portrait, the form fields reflow to match the changes in dimensions.Navgation BarItemDescriptionHomeDisplay the home page with the inputs and available analyses.SettingsGlobal settings.PreviousRunsPrevious runs displays a list of the previous analyses that have been run. Selecting Results for a previous check displays theresults from that check without needing to re-analyze the logs.HelpDisplay the FAQs for answers to commonly asked questions.SupportHow to request support for bugs, questions, and enhancement requests.?Page level help for the displayed page.Selecting a Log SetEnter the Log Set Location* or use the browse button to navigate to it.*Internal users attempting to access a log set on the Log Processor will be prompted for their DELL EMC SSO credentials and must have beenapproved for access.See Requesting Access to the Log Processor (Internal Users only) for requesting access.Selecting an AnalysisClick the appropriate tab to navigate between analyses.Running an AnalysisSee Running an AnalysisPage 8
Reviewing ResultsAnalysis OverviewClick the " " or text to expand the overview and click the "-" or text again to close the overview.Cluster Results SummarySummarizes critical and warning items as well as installed patches for quick and easy reference.Open / Close AllClick Open All or Close All to open or close all of the analysis results.Page 9
Sort / Filter ByUse Sort By and Filter By to organize and filter the analysis results.Displaying DetailsFor any results row, click the " " or text to expand the results and display the details and click the "-" or text again will close the details.Copy and PasteCopying and pasting of details is not available via the right-click context menu so just use the standard Windows shortcut keys Ctrl-C for copy, Ctrl-V for paste.Page 10
SettingsGlobal SettingsSettingDescriptionReports FolderLocation for creation of the Isilon Advisor reports.The default location is in Reports subdirectory in the IsilonAdvisor directory.Windows: C:\Users\ username \IsilonAdvisor\Reports\Linux: /home/ username /IsilonAdvisor/Reports/Page 11
Check DescriptionsSee each type of check for the appropriate descriptions.Cluster ChecksCluster Configuration ChecksNode ChecksNode Status ChecksDisk ChecksPage 12
Cluster ChecksCheck NameDescriptionActiveUpgradeChecks for running upgrade processes on the cluster. During an upgrade, major changes or administrative tasks to the clustershould not be performed until upgrade is completed.Cluster HealthReports the overall health of the cluster per OneFS.Cluster NameReports the name of the cluster.Cluster SizeReports the total cluster size of the cluster per OneFS.Cluster TimeReports time drift, cluster time, and time sync. If the cluster time is off, or if nodes are reporting a time drift, authentication andsubsequent access to the cluster can be affected.Time Drift reports the largest amount of time between two nodes. The IA will report if the time drift exceeds 60 seconds.Cluster time is determined by nodes in quorum (50% 1 nodes) having the same time to the nearest minute. The IA willprovide a notification if the cluster time cannot be determined. This can happen if the time is not synchronized on at least50% 1 of the nodes.Time Sync shows the cluster's time sync configuration and will warn if the cluster is not configured to sync its time with anexternal NTP server.ContactInformationReports the primary and secondary contact information configured on the cluster to ensure notifications are reaching theproper personnel.EMCTechnicalAdvisories(ETA)Reports on conditions that meet active EMC Technical Advisories (ETAs). ETAs alert customers to potential data unavailableor data loss scenarios.InstalledPatchesReports installed patches and checks whether the cluster has any out-of-date patches installed.Job StatusChecks the status of all on-cluster maintenance jobs enabled on the cluster.NotificationReports how email notifications are configured on the cluster to ensure notifications are reaching the proper personnel.OneFSVersionReports the OneFS version that is running on the cluster.Open EventsReports all open events as seen by OneFSOpenSMMasterReports which node is the subnet master.Checks for InfiniBand switch configuration errors.Checks for misconfigured InfiniBand network settings.Storage PoolsReports the storage pool configuration and any issues.UpgradeServiceChecks the upgrade service status and reports any issues.ConfigurationSee Cluster Configuration Checks.Page 13
Cluster Configuration ChecksCheck NameDescriptionESRSReports whether ESRS is enabled or disabled.File SharingConfigurationReports isi auth output and checks for errors.Gateway PriorityChecks the priority on all routable gateways and flags any gateways that share the same priority level. When gatewaysshare the same priority level, the network could experience degraded performance.Global NamespaceAcceleration (GNA)Reports the Global Namespace Acceleration (GNA) status and hardware qualifications for enabling GNA. Flags anycustom sysctls, debug sysctls, and efs.lbm.leak freed blocks in sysctl.conf file.HDFS ConfigurationChecks to see if HDFS is enabled or disabled and reports status and configuration.Installed LicensesReports the license status of software modules for the cluster.Isi Snapshot D LogChecks for Snapshot errors.Isi Sync PolicyReports the SyncIQ policies configured on the cluster.NFS StatusLists all NFS exports that are configured on the cluster.Report Snapshot InfoReports the Snapshot details for the cluster.SmartConnectIQChecks and reports the configured subnets, pools, node members, and devices for SmartConnectIQ and AdvancedSmartConnectIQ.Service PrincipalNames (SPN) ListLists the service principal names (SPNs) for Kerberos authentication.SupportIQ StatusChecks whether SupportIQ is enabled or disabled.Page 14
Node ChecksCheckNameDescriptionBoot DriveCheckChecks SSD boot drives and partition mirroring status and reports any error conditions that could indicate a failing or failed bootdrive.NodeCountsReports mismatched nodes by counting each node type found in the cluster. If any single node type has less than three nodes,that node type will be flagged as being mismatched. Also flags nodes that are EOSL.NodeFirmwareChecks for firmware versions and flags any nodes below current firmware levels.Node HealthChecks the health of each node and flags any that nodes that report status other than OK.NodePartitionsFlags any nodes with any writeable partition at 92% full.Node SizeChecks for free space on HDDs and SSDs and reports any node with 10% or less space remaining.NodeUptimeChecks the node uptime and reports any nodes with an uptime of 180 days.NVRAMBatteryCheckChecks the NVRAM battery status to determine if the batteries are charged sufficiently to preserve the contents of the NVRAMand reports any issues.StatusSee Node Status Checks.Page 15
Node Status ChecksCheck NameDescriptionCTO ReportThe CTO reports the serial number, configuration number, product number, and the AsBuilt Record (ABR) for all nodes inthe cluster at the time the nodes were built. This helps to verify proper hardware in the event new hardware is needed.DMI LogFlags any nodes on which DIMM errors are present.Hardware StatusChecks for any devices that are reporting errors.Kernel OpenFilesReports a count of open files by node.Lwiod LogsReports errors in the Lwiod logs.NetstatConnectionsReports connection counts to and from each node and reports any listen queue overflow errors. This can be used toidentify performance issues or network hardware issues.NetworkInterface Card(NIC) StatusReports the NIC status of each node.Routing TablesDisplays each node's network routing table. This can be used to identify network issues.Var CrashReports the capacity of the /var/crash partition on the cluster.Page 16
Disk ChecksCheckNameDescriptionBay HealthReports drive bay health status of each node.DiskFirmwareChecks for firmware versions and flags any drives below current firmware levels.LoadChecks the disk load of all disks in each node. Heavy disk load will affect performance.IDI ErrorsReports any IDI errors found. IDI errors are Isilon Data Integrity errors and have the potential to indicate data loss orunavailability.Page 17
Running an AnalysisWorkflow1.2.3.4.Select a log set*Click an analysis typeFor a custom check, click the appropriate checksRun the analysisLog SetEnter the Log Set Location* or use the browse button to navigate to it.*Internal users attempting to access a log set on the Log Processor will be prompted for their DELL EMC SSO credentials and must have beenapproved for access.See Requesting Access to the Log Processor (Internal Users only) for requesting access.Analysis TypeClick the appropriate tab to select the type of analysis to run.AnalysisDescriptionPre-upgrade CheckDetermines whether or not the cluster is ready for an upgrade, and is based on the Health Check, but with tighterthresholds.Post-upgradeCheckDetermines if an upgrade is still in progress, and is based on the Health Check, but adds the upgrade process check.Health CheckChecks on the state of the cluster, including the nodes and disks.Firmware CheckChecks whether node and disk firmware are current.Custom CheckBased on the Health Check with some additional checks, and enables selection of any or all checks.Custom Check AnalysisWhen selecting a custom check, you can select which group(s) or specific check(s) you want to run. Just click the appropriate group(s), specificcheck(s) or Check All to run all of these checks.Page 18
Run the AnalysisClick on Run check type CheckPage 19
Analysis ResultsAll results are color-coded for quick visual reference (red: critical, yellow: needs attention, green: no problem found).Status SummaryA summary of the cluster, node and disk conditions is displayed and depicts the highest level risk found for each with a summary of the number ofcritical and warning items and the number of tests completed.StatusDescriptionCriticalItems that need to be investigated and addressed. These items indicate a condition that could potentially cause dataunavailability or data loss and should be addressed immediately.WarningIndicate a configuration or state that should be investigated to determine the appropriate action. This may or may not meanthat an action is required. This may be items that are not in accordance with engineering standards or best practices.InformationalItems that do not need to be investigated or addressed.OverviewThe analysis overview contains a label for the analysis type run and once expanded provides an overview of which log was processed, the clusterit came from and optionally a SR# (service request number) associated with this analysis. As Isilon Advisor (IA) analyzes the cluster and nodestatus based on known issues and best practices, the IA health status may not agree with the ISI (OneFS) health status. This is in part due to thefact that IA has more current information and thus more checks than OneFS.FieldDescriptionLog PathItems that need to be investigated and addressed. These items indicate a condition that could potentially cause dataunavailability or data loss and should be addressed immediately.SR#Indicates a setting is misconfigured or that the cluster is experiencing a problem. These items require attention andinvestigation as soon as possible.ClusterNameName of the cluster for the associated log.Log DateTimestamp from the associated log.Page 20
Cluster/NodeNameThe name of the cluster and node for the associated log.ISI StatusThe cluster or node health status as seen by OneFS.IA StatusThe cluster or node health status as seen by Isilon Advisor.CRIT - Items that need to be investigated and addressed. These items indicate a condition that could potentially causedata unavailability or data loss and should be addressed immediately.WARN - Indicates a setting is misconfigured or that the cluster is experiencing a problem. These items should beinvestigated as soon as possible.OK - Items that do not need to be investigated or addressed.Cluster Results SummarySummarized list of just the critical and warning items and the installed patches for a quick and easy reference.FieldDescriptionCheck NameName of the check that was run.ResultResult summary of the check that was run. For installed patches, a detailed list of the patches.KB / Docu LinkList of relevant reference materialsResultsThe initial view summarizes all results into brief, single line results.Page 21
DetailsTo view details for any result, click and expand a that result. When possible the details in red highlight the item found, and the links take you todocumentation with additional information about your item.Page 22
Exporting ResultsUse exported results to share Isilon Advisor results with others, such as support or coworkers. The exported results are contained in a single zipfile and can be opened in any of the supported browsers and does not require Isilon Advisor to be installed. Exported results are displayed usingthe same results view as seen in Isilon Advisor.1.2.3.4.Click the Export Results buttonSelect a folder for the exported resultsExport is saved to the folderCopy and send the zip file to the desired receiverBrowser SupportThe following browsers are supported:Chrome 39 or laterFirefox 34 or laterSafari 7 or laterBeware that unsupported browsers, such as Internet Explorer (IE) may cause erratic and undesirable behavior with inputs andformatting.Page 23
Generating a ReportUse reports to share Isilon Advisor results with others, such as support, coworkers or customers. Word reports contain both summary anddetailed level results, while PowerPoint reports contain only summary level results.1.2.3.4.Click the Generate Report buttonSelect a report typeSelect a folder for the reportReport is saved to the folderPage 24
Sharing and Receiving ResultsShare Exported ResultsThis method works well for sharing the results with anybody, regardless of whether or not they have Isilon Advisor installed, and presents thesame results view as seen in Isilon Advisor.Sharing1. See Exporting ResultsReceiving1. Unzip the zip file to the desired folder2. Open the unzipped folder3. Open the html fileShare Summary text fileThis method works good for sharing the results with EMC support or field personnel who are familiar with the technical aspects of an IsilonCluster, such as EMC Isilon Support.Sharing1. Navigate to the appropriate directory for the cluster2. Copy AnalysisType Check yyyymmdd-hhmmss.txt from the appropriate directory and send it to the appropriate receivera. Windows: C:\Users\ username \IsilonAdvisor\Reportsb. Linux: /home/ username /IsilonAdvisor/ReportsPage 25
Previous RunsPrevious runs displays a list of the previous analyses that have been run. Selecting Results for a previous check displays the results from thatcheck without needing to re-analyze the logs.Page 26
Upgrade AdvisorUse Upgrade Advisor to generate pre-upgrade, upgrade and post-upgrade steps with links to relevant documentation.1. Select the Current OneFS Version2. Select the Target OneFS Version3. Expand and review the resultsPage 27
FAQUse the FAQs for answers to commonly asked questions.Page 28
Requesting Access to the Log Processor (Internal Users only)Only required if you map the drive as HTTPS uses SSO validationPlease log a request using the account portal to be added to LPXUsers group.1.2.3.4.5.6.7.8.9.10.11.12.Go to http://accountportal.emc.comSelect ACTIONS, then Request Access from the dropdown menuFrom the User Detail tab, go to the Find Profile section and search for your profileSelect NextFrom the Access Catalog tab, select "Isilon – Log Processor (LP)" from the Filter By dropdown listSelect "Isilon – Log Processor (LP)" from the "Isilon – Log Processor (LP)" dropdown listFrom the PROPEL Role Dictionary section, select " " from the add columnEnter your justification and click OKSelect NextFrom the Verify request tab, select your access request and click SubmitAdditional information can be found hereYour request will be routed to the appropriate team for approvalPage 29
SupportGS Tools Support has updated our support processes to better serveour users.For GS Tools products only, you no longer have to create a Service Request (SR) simply use the"Contact Us" option on the NavBar and we will do that for you. Once the SR is created you willreceive a notification via email containing the SR number and a link to it. Please open the emailand add any files or screenshots to the SR and not to any of the emails.Contact Us is available from our GSTools Central navigation bar. Click onthe Call out icon to launch theContact Us form.Click on "Request Support"For "Request Support" thefollowing fields are requiredSubject (mandatory) be descriptive the tool orapplication nameEnvironment (optional) Browser versions addedautomatically, addadditional informationsuch as operatingsystem depending onyour application beingdesktop or web based.Contact Email (optional)- added automatically,can be changed ifrequired.Phone Number (mandatory) - enter a phonenumber including countrycode for our support staffto contact you ifrequired.Message (mandatory) enter as muchinformation as possibleto explain the issue youare having and anyPage 30
application information,versions, history, actionsetcClick SendConfirmation message will bereceived, please follow theinstructions in the message.Page 31
Welcome to Isilon Advisor Description The EMC Isilon Advisor is a smart, fast and effective application that enables customers to self-support common Isilon issues and accelerate time-to-resolution. This is the same application used by EMC Isilon Technical Support Engineers and Field Representatives to resolve service
