Transcription
Real-Time Monitoring Installation and Configuration, on page 1 Performance Monitoring , on page 1 Tools, on page 3Installation and ConfigurationThe Unified RTMT installer can be downloaded using Tools Plug-ins menu on the Cisco Unified ContactCenter Express Administration web interface. See “Cisco Unified Real-Time Monitoring Tool” section inCisco Unified Real-Time Monitoring Tool Administration Guide for Cisco Unified Contact Center Expressand Cisco Unified IP IVR for installation and configuration procedures, available sw/ps556/prod maintenance guides list.htmlPerformance MonitoringUnified CCX provides performance counters (called perfmon counters) for application performance monitoring.The perfmon counters help expose various performance values and enables to track application performancein real time.The perfmon counters contain counter-based information, such as the name and index of the counter, the scale,the type, subcounters to set when setting a counter, the current values, and a map containing counter instancedata. Each performance counter instance object contains instance-based data, like the instance ID and currentvalues.You can log perfmon counters locally on the computer and use the performance log viewer in Unified RTMTto display the perfmon CSV log files that you collected or the Real-time Information Server Data Collection(RISDC) perfmon logs. Choose System Performance on the Unified RTMT tool to view perfmon counters.Performance ObjectsUnified RTMT provides a set of default monitoring objects that assist you in monitoring the health of thesystem. Default objects include performance counters or critical event status for the system and other supportedservices.The system logs information every 10 seconds for predefined system counters.Real-Time Monitoring1
Real-Time MonitoringPerformance CountersPerformance CountersTo troubleshoot system performance problems, you add a counter (query) that is associated with the perfmonobject to the performance monitor, which displays a chart for the counter. Choose System Performance Open Performance Monitoring to add a new counter.For more information about monitoring objects and counters, see “Performance Monitoring” section in theCisco Unified Real-Time Monitoring Tool Administration Guide for Cisco Unified Contact Center Expressand Cisco Unified IP IVR, available sw/ps556/prod maintenance guides list.htmlPerformance Objects and Counters for Unified CCXFollowing are the Unified CCX application specific objects: Unified CCX database monitors Unified CCX engine JVM heap Intelligence center database performance Info Intelligence center JVM statistics Intelligence center system condition table Intelligence center thread pool section Intelligence center tomcat connector Reporting engine info Ramfs SchedulerInfoNoteExpand the objects in RTMT to display the counters. Right click on each counter and select CounterDescription for the description.Critical ServicesThe Critical Services monitoring function provides the name of the critical service, the status (whether theservice is up, down, activated, stopped by the administrator, starting, stopping, or in an unknown state), andthe elapsed time during which the services are functional on the system.NoteUnified RTMT does not display a partial running status of a service in Unified CCX. For example, it doesnot display a service as “running” under "Critical Services" if some of its subsystems are down. The partialstatus of the Unified CCX services will only be viewable from the Unified CCX Serviceability Administrationweb interface.Real-Time Monitoring2
Real-Time MonitoringToolsNoteYou can view and manage Cloud Connect services using Cloud Connect CLIs. For more information, seeCloud Connect section in the Command Line Interface chapter.ToolsUnified RTMT provides various tools to monitor and troubleshoot system issues. The following section brieflydescribes these tools.AlertsUnified CCX generates alert messages to notify the administrator when a predefined condition is met, suchas when an activated service fails to start. The system sends alerts as email or displays alerts as a popupmessage on RTMT.RTMT contains preconfigured and user-defined alerts that support alert modifications. Although you canperform configuration tasks for both types, you cannot delete preconfigured alerts (whereas you can add anddelete user-defined alerts). Predefined alerts are configured for perfmon counter value thresholds as wells asevent (alarms) notifications.Unified CCX AlertsThe following list contains preconfigured Unified CCX alerts:Table 1:Alert NameSyslog Alarm NameDescriptionDB CRA % Space UsedDB CRA % Space UsedThe percentage ofused space in theUnified CCXdatabase, db cra.The database,db cra, contains theUnified CCXhistorical andconfiguration data.DBReplicationStoppedDB REPLICATION STOPPEDUnified CCXDatabaseReplication hasbeen removed. Thistypically happenswhen thereplication queuesbecome full due tothe inability tocontact the othernode.Real-Time Monitoring3
Real-Time MonitoringUnified CCX AlertsHistoricalDataWrittenToFilesIntelligence CenterCUIC DATABASE UNAVAILABLECUIC DATABASE UNAVAILABLEThis alert occurswhen theIntelligence CenterCUIC DATABASE UNAVAILABLEevent getsgenerated. Thisindicates the systemdetected criticalerror with database.Intelligence CenterCUIC DB REPLICATION FAILEDCUIC DB REPLICATION FAILEDThis alert occurswhen theIntelligence CenterCUIC DB REPLICATION FAILEDevent getsgenerated. Thisindicates theDatabasereplication failed.Intelligence CenterCUIC REPORT EXECUTION FAILEDCUIC REPORT EXECUTION FAILEDThis alert occurswhen theIntelligence CenterCUIC DB REPLICATION FAILEDevent getsgenerated. Thisindicates that thereporting servercould not run areport. This couldbe because theassociateddatasource isoffline.Real-Time Monitoring4UCCX HISTORICAL DATA WRITTEN TO FILES Historical data isnot written to theUnified CCXdatabase and hasbeen written to thefile system. Pleaseverify the state ofthe Unified CCXdatabase.
Real-Time MonitoringUnified CCX AlertsIntelligence CenterCUIC UNRECOVERABLE ERRORCUIC UNRECOVERABLE ERRORThis alert occurswhen theIntelligence CenterCUCI UNRECOVERABLE ERRORevent getsgenerated. Thisindicates that thesystem has detectedan internal errorwithin ReportingServer which mayprevent it fromfunctioningcorrectly. Restartmay be required.CCXToCUICAdminSyncFailedUCCX TO CUIC SYNC FAILEDThis alert occurswhen the UnifiedCCX has failed tonotify CUIC on anyresource change.CCXToCUICCVDSyncFailedUCCX TO CUIC SYNC FAILEDThis alert occurswhen the UnifiedCCX has failed tonotify CUIC on anyresource change.CCXToCUICEngineSyncFailedUCCX TO CUIC SYNC FAILEDThis alert occurswhen the UnifiedCCX has failed tonotify CUIC on anyresource change.PurgeInvokedAUTO PURGE COMPLETEThis alert occurswhen the UnifiedCCX Auto Purginghas completed.UnifiedCCXEngineMemoryUsageHigh UnifiedCCXEngineMemoryUsageHighThis alert occurswhen thepercentage of JVMheap memory usedby Cisco UnifiedCCX Engineprocess is greaterthan the configuredthreshold value.Real-Time Monitoring5
Real-Time MonitoringUnified CCX AlertsEMAIL SERVER DOWNEMAIL SERVER DOWNThis alert occurswhen the emailserver is notreachable.EmailOAuthConnectionFailedEMAIL OAUTH CONNECTION FAILEDThis alert occurswhen CCP isunable to connect tothe email OAuthserver or get theaccess token.EmailAuthenticationFailedEMAIL AUTHENTICATION FAILEDThis alert occurswhen the emailcredentials arewrong.CCPTomcatServiceDownSS PARTIAL SERVICE CCP TOMCAT DOWN This alert occurswhen CCP Tomcatis not reachable.CCPXMPPServiceDownCCP XMPP SERVICE DOWNThis alert occurswhen the UnifiedCCX has failed tocontact CCPruntime server(XMPP).OutboundScheduledContactImportFailed OB SCHEDULED CONTACT IMPORT FAILED Scheduled importof outboundcontacts failed.OutboundContactImportSchedulingFailed OB CONTACT IMPORT SCHEDULING FAILED Scheduling ofoutbound importcontact failed.UserPasswordMismatchAcrossNodes UserPasswordMismatchAcrossNodesOne or more userpasswords are notsame across boththe nodes.ReasonCodesSyncRetryFailureReason Codes Syncfrom Finesse failedand reached toMaximum numberof retries.REASONCODE SYNC RETRY ERROREnsure that Finesseservice and UnifiedCCX database areactive.Real-Time Monitoring6
Real-Time MonitoringCisco Identity Service AlertsCCPCacheStatusFullCCP CACHE STATUS FULLUnable to cacheemails as disk spaceis low. No newemails will befetched.CCPCacheStatusReachedLowThreshold CCP CACHE STATUS REACHED LOW THRESHOLD Existing emailshave consumed aconsiderableamount of diskspace.CacheStatusOnlineCCP CACHE STATUS ONLINECCPSSLErrorSS PARTIAL SERVICE CCP SSL ERROR This alert occurswhen SSLConnectivity withCustomerCollaborationPlatform fails.NoteConsiderableamount of diskspace is nowavailable. Newemails would nowbe fetched.To view or edit values for any alert, right click on the alert and select Set Alert/Properties.Cisco Identity Service AlertsYou can view the Cisco Identity Service alerts from the Unified CCX pane.The following list contains preconfigured Cisco Identity Service alerts:Table 2:Alert NameSyslog Alarm NameDescriptionIdSInitializationFailureIDS INIT ERRORThis alert occurs when an error isencountered during IdS initialization.IDPMetaDataLoadErrorIDP META DATA LOAD ERROR This alert occurs when the trust could notbe established between IdS and IdP duringinitialization.SPMetaDataLoadErrorSP META DATA LOAD ERROR This alert occurs when SAML SP metadataInitialization fails.Real-Time Monitoring7
Real-Time MonitoringSyslog and AlertIDPMetaDataUpdateErrorIDP META DATA UPDATE ERROR This alert occurs when there is an errorupdating IdP metadata and propagatingacross the cluster.SPMetaDataUpdateErrorSP META DATA UPDATE ERROR This alert occurs when SAML SPcertificate regeneration fails.TokenMetaDataUpdateError TOKEN META DATA UPDATE ERROR This alert occurs when TOKEN Keystoreregeneration or update fails.IdSSecurityConfigNotPresent IDS SECURITY CONFIG NOT PRESENT This alert occurs when some IdS securityconfiguration files are not present on thesecondary node.IdSSecurityConfigPullFailure IDS SECURITY CONFIG PULL FAILURE This alert occurs when the security configcould not be pulled from the primary IdSnode.SAMLCertificateLoadFailed SAML CERTIFICATE LOAD FAILED This alert occurs when the system is unableto read the SAML SP certificate.IdSStateNotConfiguredSTATE NOT CONFIGUREDThis alert occurs when the trust betweenIdS node and IdP is yet to be establishedor when the IdS configuration could notbe synchronized from the master node.IdSStateOutOfServiceSTATE OUT OF SERVICEThis alert occurs whenever a system errorresults in the IdS Application failing tostart.NoteTo view or edit values for any alert, right-click the alert and select Set Alert/Properties.Syslog and AlertBelow are the set of syslog messages and alert which can be viewed from RTMT.Syslog Alarm NameDescriptionCONTM INIT FAILUREContainer Manager initialisation failedCONTM INIT HTTP FAILUREContainer Manager HTTP Server initialisation failedCONTM INIT PROVISIONING FAILUREContainer Manager fails to initialise the provisioningCONTAINER AUTO UPDATE FAILUREContainer Manager fails to update containers basedon the provisioningCONTAINER AUTO UPDATEContainer Manager failed to update containers basedon new provisioning, and failed to revert backcontainers based on existing provisioningRECOVERY FAILUREReal-Time Monitoring8
Real-Time MonitoringSyslog Support for Critical Cisco Finesse Log MessagesSyslog Alarm NameDescriptionCONTAINER AUTO UPDATE PERSISTContainer Manager failed to persist the newprovisioning after container auto updatePROVISIONING FAILURESyslog Support for Critical Cisco Finesse Log MessagesCisco Finesse generates syslogs for critical log messages. Use the following procedure to view the logs usingUnified RTMT.Before you beginDownload and install RTMT on a client computer from the following e, where FQDN is the Fully QualifiedDomain Name of the Finesse server.Step 1Log in to Unified RTMT using Finesse administrator credentials.Step 2In the tree hierarchy, select SysLog Viewer or choose System Tools SysLog Viewer Open SysLog Viewer.Step 3From the Select a Node drop-down list, choose the server where the logs that you want to view are stored.Step 4Under the Logs tab, select Application Logs CiscoSyslog to view and save the syslog file.When you double-click the CiscoSyslog message, the Show Detail dialog displays the syslog definition andrecommended actions in an adjacent pane.TipFor more information, see Cisco Unified Real-Time Monitoring Tool Administration Guide stem log messages generated by Cisco Finesse are also available under SysLog Viewer System Logs messages.The following are the different types of messages and corresponding descriptions that are captured in the SysLog Viewer System Logs messages. CTI SOCKET ERRORSystem has encountered an error connecting to the CTI server. CTI CONNECTION LOSTSystem has lost contact with the CTI server. CTI OPEN FAILURECTI Server rejected open request. CTI CONNECTION RETRIES EXCEEDEDSystem has failed to connect to CTI server in the allowed number of retries. CTI CONNECTION ESTABLISHEDSystem has successfully connected to CTI server.Real-Time Monitoring9
Real-Time MonitoringTraces and Logs SUBSYS INIT ERRORError initializing subsystem. UNABLE TO CONNECT TO XMPP SERVERUnable to connect xmpp server. DB SS CONNECTION CHECKThere was an error connecting to the database. cfservice CORE ERROR DB CONNECTIONUnable to connect to the Database. AWDB NOT ACCESSIBLEUnable to connect to AWDB server. VOS DB ADAPTER ERRORThere was an error on the VOS DB Adapter operation. FINESSE APP STARTUP ERRORError during Finesse Application Startup. OF STATE CHANGEDOF subsystem state successfully changed. CONNECTED TO XMPP SERVERSuccessfully connected to xmpp server. SSO API ERRORError processing REST API Request for SSO. API ERROR DETAILError processing REST API request. AWDB CONNECTION ERRORError while connecting the AWDB server. AWDB CONNECTION SWITCH SUCCESSAWDB server connection successfully switched.Traces and LogsThe trace and log central feature in RTMT allows you to configure on-demand trace collection for a specificdate range or an absolute time. You can collect trace files that contain search criteria that you specify andsave the trace collection criteria for later use, schedule one recurring trace collection and download the tracefiles to a SFTP or FTP server on your network, or collect a crash dump file.Real-Time Monitoring10
Real-Time MonitoringCloud Connect ServiceabilityAfter you collect the files, you can view them in the appropriate viewer within the RTMT. You can also viewtraces on the server without downloading the trace files by using the remote browse feature. You can openthe trace files by either selecting the internal viewer that is provided with RTMT or choosing an appropriateprogram as an external viewer.For more information about traces and logs, see “Tools for traces, logs, and plug-ins” in Cisco UnifiedReal-Time Monitoring Tool Administration Guide for Cisco Unified Contact Center Express and Cisco UnifiedIP IVR, available sw/voicesw/ps556/prod maintenance guides list.htmlCloud Connect ServiceabilityUse the Cisco Unified Real-Time Monitoring Tool (RTMT) to collect Cloud Connect logs.You can use the Cloud Connect CLIs to: List all the containers in your deployment. View details of the specified container in JSON format. Start the specified container. Stop the specified container. Generate the heap dump for the specified container that is running JVM. Generate the thread dump for the specified container that is running JVM. Set log level for the specified service. Update the Cloud Connect Cherry point connector configuration details. Download and install RTMT on a client computer.For more information on Cloud Connect CLIs, see Cloud Connect.CUCM Telephony Data MonitoringFollowing entities can be monitored using CUCM Telephony Data RTMT: Triggers Call Control Groups CTI portsTo access CUCM Telephony Data, click Cisco Unified CCX tab in RTMT.Triggers PageThe Triggers page displays the following information for the triggers that are configured for Unified CCX:Real-Time Monitoring11
Real-Time MonitoringCall Control Groups pageTable 3: Triggers Page OptionsCountersDescriptionTriggerDNThis field displays the directory number that isassociated with the trigger.Trigger StateThis field displays the state of the trigger, which canbe In Service, Out of Service, or Unknown.Application NameThis field displays the name of Unified CCXapplication that is associated with the trigger.Ready for CallThis field indicates whether the trigger is ready toaccept the call.CallControlGroup IDThis field displays the ID of the call control groupthat is associated with the trigger.Media Group IDThis field displays the ID of the media group that isassociated with the trigger.Last State Change TimeThis field displays the time of last state change forthe trigger.Recommended ActionThis field provides the reason the trigger state is Outof Service or Unknown and provides therecommended action to return the trigger state to InService.NoteThis field is populated only if the triggeris in Out of Service state or Unknown state.Call Control Groups pageThe Call Control Groups page provides the following information about the current Call Control Group thatis configured for Unified CCX:Table 4: Call Control Groups Page OptionsCountersDescriptionCallControlGroup IDThis field displays the ID that is associated with thecall control group.Group StateThis field displays the state of the call control group,which can be In Service, Partial Service, or Out ofService.Total PortsThis field displays the total number of CTI ports thatare configured for the call control group.InService PortsThis field displays the number of in-service CTI ports.Real-Time Monitoring12
Real-Time MonitoringCTI Ports PageCountersDescriptionOOS PortsThis field displays the number of out-of-service CTIports.CTI Ports PageThe CTI Ports page provides the following information about the current CTI ports that are configured forUnified CCX:Table 5: CTI Ports Page OptionsCountersDescriptionCTI Port DNThis field displays the directory number of the CTIport.CallControlGroup IDThis field displays the ID of call control group towhich the CTI port belongs.Port StateThis field displays the state of CTI port, which canbe In Service or Out of Service.CallIDThis field displays the call ID of the last call that isavailable on the CTI port before the port state changedto Out of Service.NoteLast State Change TimeThis field is populated only if the port stateis Out of Service.This field displays the last time when the CTI portstate changed.Summary PageThe Summary page provides the following information:Table 6: Summary Page OptionsCountersDescriptionOverall Telephony Subsystem StateThis field displays the state of the Unified CCXtelephony subsystem, which can be In Service, PartialService, or Out of Service.Call Control Groups In ServiceThis field displays the number of call control groupsthat are in service.Call Control Groups Out Of ServiceThis field displays the number of call control groupsthat are out of service.Call Control Groups In Partial ServiceThis field displays the number of call control groupsthat are in partial service.Real-Time Monitoring13
Real-Time MonitoringCisco Unified Analysis ManagerCountersDescriptionEnabled TriggersThis field displays the number of triggers that areassociated with valid call control group IDs.Disabled TriggersThis field displays the number of triggers that areassociated with invalid call control group IDs.Triggers With Config ErrorsThis field displays the number of triggers withconfiguration errors.NoteIn UCCX system, if we do not configure any Trigger and CTI Ports then CM Telephony displays Out ofService status. Similarly in IPIVR, if we do not configure ICM Subsystem then ICM Subsystem displays Outof Service status.Cisco Unified Analysis ManagerUse Cisco Unified Analysis Manager, a tool included with the Unified RTMT to perform troubleshootingoperations. Unified Analysis Manager also allows you to monitor various aspects of the devices added to thetool. Unified Analysis Manager is used to collect troubleshooting information from your system and analyzethe information. It can identify the supported Unified Communications (UC) products and applications thatyou have in your system and troubleshoot call failures across these UC applications, collecting trace and logfiles and other platform and configuration information. You can use this information to troubleshoot on yourown or send the information to Cisco Technical Assistance for analysis.Unified Analysis Manager for Unified CCXTo monitor and troubleshoot a Unified CCX-based solution with the help of Unified Analysis Manager, youmust connect to a Unified Communications Manager server and then add the Unified CCX nodes accordingly.You can add following nodes/servers for monitoring: Unified CCX node Call record serverConsider the following points while adding nodes/servers for monitoring: To add nodes/servers, ensure that you select Node Type as Unified CCX. To add a call record server, enter uccxsct in the JDBC User Name field.For detailed procedures to perform these actions, see “Cisco Unified Analysis Manager preferences”section in the Cisco Unified Real-Time Monitoring Tool Administration Guide for Cisco Unified ContactCenter Express and Cisco Unified IP IVR, available sw/ps556/prod maintenance guides list.htmlReal-Time Monitoring14
Real-TimeMonitoring InstallationandConfiguration,onpage1 . Cisco Unified Real-Time Monitoring Tool Administration Guide for Cisco Unified Contact Center Express . Cisco Unified Real-Time Monitoring Tool Administration Guide for Cisco Unified Contact Center Express
