EMC Control Center Troubleshooting

EMC ControlCenter
Version 5.0.3
Troubleshooting Guidelines
P/N 300-000-302
REV A02
6/17/02
This document presents descriptions and remedies to problems you

may encounter installing or using components of EMC ControlCenter
version 5.0.3. Topics include:
◆ Introduction............................................................................................2
◆ System Overview...................................................................................5
Procedures
◆ Troubleshooting Principles ................................................................12
◆ Diagnosis and Remedy.......................................................................15
Reference ....................................................................................................40
◆ Agent Names and Codes....................................................................41
◆ Port Settings .........................................................................................45
◆ Trace and Log Files..............................................................................45
◆ Error Messages.....................................................................................47
◆ Tools.......................................................................................................47
1
Introduction
Introduction
Troubleshooting involves diagnosing and often simultaneously
repairing a system malfunction. A range of behaviors, from
undesirable (for example, poor performance from traffic bottlenecks)
to nonperformance (for example, functional errors, data
inaccessibility, or corruption) may prompt you to act.
EMC ControlCenter™ aids in troubleshooting and fixing the storage
environment it oversees. The scope of this manual, however, is the
troubleshooting of ControlCenter software itself. While other
ControlCenter manuals cover the successful installation and
functioning of ControlCenter, this guide covers unexpected failures
that could disable normal ControlCenter operation, describing
symptoms that may include unusual behavior in the GUI.
This manual identifies categories of possible unacceptable behaviors

in ControlCenter, and recommended user reactions to them. It
attempts to identify occurrences by known symptoms. It does not
cover—but may provide pointers to—preventative actions if they
concern the same system failures.
Audience This document is intended for use by ControlCenter system

administrators.
About this Document This document provides the two basic types of diagnostic and
remedy information: procedures and references.
Procedures
Commonly encountered problems, along with their remedies, are
described with specific procedures. In certain cases, specific
remedies cannot be predefined, and a strategy or an escalation
path to EMC Customer Support is given.
See Troubleshooting Principles on page 12 and Diagnosis and Remedy
on page 15.
Reference
Tables and examples describing diagnostic objects such as logs,
initialization files, and the like are provided here.
See Reference on page 40.
2 EMC ControlCenter Version 5.0.3 Troubleshooting Guidelines

Introduction
Related To avoid problems, the user should adhere to specifications provided in the
Documentation EMC ControlCenter version 5.0.3 Installation and Configuration Guide, EMC
ControlCenter Version 5.0.3 Release Notes, EMC ControlCenter/Open Edition
Support Matrix, and EMC ControlCenter Performance and Scalability Guidelines.1
You should have the ControlCenter version 5.0.3 documentation set

when troubleshooting, or when preparing to call an EMC
representative or EMC Customer Support. It contains the following:
1. EMC ControlCenter version 5.0 Tutorial CD, P/N 053-000-939 Rev
A01, provides a tour of ControlCenter features.
2. Within the ControlCenter application, Online Help is provided
through the main menu Help, with submenus Contents, Index,
Search and Error Message Help. This is a searchable resource
with more detailed information than the print documentation.
3. The following printed documents are provided with version 5.0.3:
• EMC ControlCenter version 5.0 Introduction, P/N 300-000-295
Rev A01
• EMC ControlCenter version 5.0.3 Installation and Configuration
Guide, P/N 300-000-296 Rev A02
• EMC ControlCenter version 5.0.3 Installation Map, P/N
300-000-297 Rev A02
• EMC ControlCenter version 5.0 Quick Reference Guide, P/N
300-000-298 Rev A01
• EMC ControlCenter version 5.0.3 Release Notes, P/N 300-000-303
Rev A02
4. In addition to those listed above, the following documents are
provided in PDF format on EMC ControlCenter version 5.0.3
Documentation/Help CD, P/N 053-000-938 Rev A02:
• EMC ControlCenter version 5.0 User Guide, P/N 300-000-299
Rev A01
• EMC ControlCenter version 5.0 OS/390 Tape Cartridge Release
Notes, P/N 300-000-304 Rev A01
• EMC Symmetrix Storage Concepts Guide, P/N 300-000-306 Rev
A01
1. EMC ControlCenter/Open Edition Support Matrix and EMC ControlCenter

Performance and Scalability Guidelines are available on Powerlink
(http://powerlink.emc.com).
EMC ControlCenter Version 5.0.3 Troubleshooting Guidelines 3

Introduction
• EMC ControlCenter version 5.0 Integration Packages Product

Guide, P/N 300-000-300 Rev A01
• EMC ControlCenter version 5.0 Migration Guidelines, P/N
300-000-301 Rev A01
• EMC ControlCenter version 5.0.3 Troubleshooting Guidelines, P/N
300-000-302 Rev A01
Typographical Conventions
EMC uses the following type style conventions in this guide:
AVANT GARDE Keystrokes
Palatino, ◆ Dialog box, button, icon, and menu items in text

bold ◆ Selections you can make from the user interface,
including buttons, icons, options, and field
names
Palatino, ◆ New terms or unique word usage in text

italic ◆ Command line arguments when used in text
◆ Book titles
Courier, Arguments used in examples of command line
italic syntax.
Courier System prompts and displays and specific
filenames or complete paths. For example:
working root directory [/user/emc]:
c:\Program Files\EMC\Symapi\db
Courier, ◆ User entry. For example:

bold symmpoll -p
◆ Options in command line syntax

System Overview
System Overview
ControlCenter is a suite of applications used to manage the data
storage and related devices in an EMC Enterprise Storage Network
(ESN). A very brief overview of system components is provided here.
For a much more detailed overview on components and their context, consult
EMC ControlCenter Version 5.0 Introduction (P/N 300-000-295).
Components ControlCenter is a suite of interworking software components. Each

component is the subject of various tools that aid troubleshooting.
Console Tier Infrastructure Tier Agent Tier
Repos
Console(s)
omer Storage Environment
ECC-000042
Figure 1 ControlCenter Software Tiers
Infrastructure Tier The infrastructure manages and maintains the driving components of
ControlCenter, the ECC Server, Store, and Repository.
ECC Server
This is the primary driving component of ControlCenter. It performs
several management tasks among components.
Repository
This component is a relational database holding the current and
historical metadata for the ControlCenter Console and infrastructure,
as well as the extended storage environment.

System Overview
Store
The Store populates the Repository with data collected from the
storage environment by the agents.
Agent Tier Master Agent

A Master Agent runs on each host to manage individual agents on
that host.
Storage Agents
These components handle communication with specific storage
systems.
Includes agents for — Symmetrix®, Celerra™, CLARiiON®,
Compaq StorageWorks, Hitachi Data System (HDS), RVA/SVA,
and IBM ESS.
Host Agents
Each operating system class requires a host agent tailored to its
environment.
Includes agents for — Windows, Solaris, HP-UX, AIX, MVS, and
Novell.
Connectivity Agents
These agents gather information about the interconnections between
storage-system hosts, devices, and networks.
Includes agents for — SNMP, switches, and SDM.
Database Agents
Two agents provide for communication with databases.
Includes agents for — Oracle and DB2.
Backup Agents
These agents assist in backup operations.
Includes agents for — Tivoli Storage Manager for: AIX, HP-UX,
Windows NT, Windows 2000, and Solaris.
Tape Agent
Provides capability for reporting on and managing tape subsystems.
Includes agents for — OS/390 for StorageTek/STK, VTS/SMS,
CA1/TMC, and RMM/DFSMS.
Other Agents
Some agents are not specific to hosts or other devices, but act on
multiple devices and applications, or the system as a whole.

System Overview
Includes agents for — Integration Gateway and Workload

Analyzer (WLA) Archiver.
Console and Client The Console is the consolidated GUI for ControlCenter. It provides a
Tier window on the objects managed by application, their attributes, and
their relationships to each other. See Figure 2 on page 8.
Other client applications may be part of this tier.
Interfaces Command Line

ControlCenter processes produce command line output, which is
written to a corresponding output file. See Table 34 on page 46.
Installation Utility
Parts of the installation and configuration procedures are managed
through the Installation utility. This software is installed from the
ControlCenter CD set early in the installation procedure.
Console
After installation, normal ControlCenter operations are run through
the GUI provided by the Console. (Examine detail in Figure 2 on
page 8, or consult other ControlCenter manuals.)

System Overview
Figure 2 ControlCenter Console (Example View)

System Overview
Recognizing The following descriptions refer to features for dealing with problems
Problems in the detected by ControlCenter in the managed objects it is monitoring.
Monitored Storage Malfunctions in ControlCenter itself are the focus of this document,
System however; see the discussion starting with the following section,
Troubleshooting Principles on page 12.
EMC ControlCenter Version 5.0 Introduction (P/N 300-000-295) and EMC

ControlCenter Version 5.0 User Guide (P/N 300-000-299) provide more detailed
descriptions of ControlCenter components and their operation.
GUI Devices The GUI has several devices that will indicate error conditions, when
encountered, in the devices it monitors.
Window Panels
Tree Panel
This lower left-hand panel displays a structured list of storage
environment physical and logical objects. The list may be collapsed or
expanded in a manner similar to that in Windows Explorer.
A line-item object (for example, a physical device) may indicate a
problem by using a warning or error version of its miniature icon. See
Table 1 for descriptions of icon problem indicators.
Target Panel
As in the tree panel, warning and error icons indicate problems with
devices. The icons may be presented in a graphical layout or as part
of a table.
Icons
The Console uses small icons to indicate that particular devices are
part of the interworking storage network. These icons provide:
Device Recognition - A device that may be physically present but is
not recognized will not appear as a listed object in the tree or in
configuration dialog boxes.
Warnings and Errors Indication - When there is a problem with a
known device, the icon will change appearance to indicate that a
problem exists. See Table 1 for descriptions.

System Overview
Table 1 Error Indicators in Console Icons
Icon Interpretation
Red circle with cross:

Fatal or Critical alert exists in managed object
Yellow triangle with exclamation point:

Warning or Minor alert exists in managed object
Red arrow at lower right of folder:

Folder contains children (at any level below) that have alerts
Red circle (shown) or Yellow triangle:

Named folder’s children (at any level below) have alerts
Red circle or Yellow triangle (shown), with red arrow:

Managed object, which also acts as a folder, contains
children (at any level below) that have alerts
GUI Content Alerts

ControlCenter agents can be configured to monitor specific variables
for warning and error conditions.
A condition occurs ordinarily when a value exceeds a preset
threshold. When a warning or error condition occurs, an alert
message is sent by the monitoring agent to the ECC Server. (In certain
cases an autofix automatic recovery is performed.)
Activated alerts may be reviewed in the Console by monitoring
alerts. An alert toolbar appears at the right side of the task bar. It
shows number of critical errors as well as total number of alerts.
Clicking the alert button displays an alert panel in the target panel.
Figure 3 Alert Toolbar

System Overview
Activity Records Command History

From Console Monitoring view, Command History provides a table
containing details about the sessions and commands associated with
selected objects that were issued through the Console.
Trace and Log Files
System activity is recorded in some cases in trace (*.trc) files, and
in some cases to log (*.log) files. For more information on these
files, refer to Trace and Log Files on page 45.

Troubleshooting Principles
ControlCenter provides several reporting mechanisms and tools for
recording application events and alerting the user when problems
occur.
When ControlCenter Malfunctions
◆ Note the symptoms that prompted your attention. Record any information provided (pop-up
dialog boxes, warning icons, and so on).
◆ Use Online Help from the Console (if it is running) to interpret error messages.
◆ Find symptoms in diagnostic tree, and review the information provided in the various paths.
If symptoms are not identified in diagnostic tree—

◆ Identify tools available from Powerlink (http://powerlink.emc.com/).
If symptoms and solutions are not identified in Powerlink,

or you are otherwise unsure whether to take an action —
◆ Contact Customer Support.
• Before calling, have the following files available:
All server.trc.<n>, store.trc.<n>, consoleLaunch.log, console.trc,
masteragent.log, <agent>.log files. Refer to the Reference section for information on
locating these files.
Recognizing ControlCenter is a configuration, maintenance, monitoring, and

ControlCenter recovery system for a target storage environment. If ControlCenter
Problems has been configured correctly and is operating properly, then when
its managed objects produce state errors, ControlCenter agents will
send alarms as well as record passive indicators (such as error events)
to log files.
Monitoring The user should establish a schedule for regular system monitoring.
Through regular use, problem indicators are learned and problems
may be recognized as soon as possible after they occur.

Responding to The following is a general response sequence to symptom discovery.

Problems Use the itemization that follows as a checklist.
Symptom Identification
To identify the symptom of a ControlCenter problem:
1. Recognize the problem — Determine what specifically and
visibly indicates a problem (such as warning icon appears, alert
appears, expected icon is missing, pop-up error window). Note
what is visible without (at this point) manipulating the GUI.
• In the diagnostic tree, see Symptoms (white box at top), as
well as Procedure and Context (grey box below).
2. Interpret the Symptoms — Formulate an explanation (where,
when, and how it is occurring).
• In the diagnostic tree, see Suspected Diagnosis column.
• If there is no obvious interpretation, follow diagnostic tree
tests, in Item order.
Preparation
To prepare before acting:
3. Isolate the problem — Identify where (on which hosts, between
which connections) the problem has occurred or is occurring.
4. Protect the environment — If relevant and practical, physically
or logically isolate the problem from the rest of the symptom.
5. Gather information — At this point, look for additional
symptoms.
• In the diagnostic tree, follow Diagnosis and Remedy paths in
the order of numbered Suspected Diagnoses.
6. Decide if help is needed.
• In the diagnostic tree, Diagnosis and Remedy instructions may
suggest you attempt a fix, or may ask you to call Customer
Support.

Diagnosis
When the diagnostic tree (see discussion beginning at page 15) does
not address a problem:
7. Explore the symptoms — navigate through (drill down) the GUI
tree and select views for further error indicators; open your logs.
8. Organize evidence gathered from exploration, and hypothesize
diagnoses.
9. Formulate remedies based on your hypotheses.
Remedy
Following confirmation of a Suspected Diagnosis, steps will be
implied, or made explicit, in Diagnosis and Remedy Procedures to:
10. Perform remedies.
11. Test as indicated to confirm success.

Diagnosis and Remedy

Only a site system administrator or similarly qualified ControlCenter
user should perform diagnostic procedures.
Using the Tree The diagnostic tree is a set of tables, generally one for a particular
major symptom or group of symptoms indicating a malfunction.
Each table has three basic parts:
◆ At the top of a table (in a white box), symptoms are stated.
◆ Below the symptoms (in a shaded box), operator context is
identified.
◆ Below the symptoms and context, a set of numbered Suspected
Diagnoses is given, with corresponding Diagnostic and Remedy
Procedures.
Table 2 Diagnostic Tree Key
Symptoms
Procedure if applicable
Book
Chapter Symptoms describes what the user saw to prompt his or her reaction.
Procedure describes what the user was doing and/or the user’s objective.
Sections Context tells where in the product interface the user was working.
Context if applicable
Console
Tree panel
Menu Items
Final Menu Item
Item Suspected Diagnosis Diagnostic and Remedy Procedures
Reference Suspected Diagnosis is one (of possibly Diagnostic Procedure is the path the user should follow to confirm or deny a
Number for several) root causes for why the symptom particular Suspected Diagnosis. Includes any work needed prior to actual fixing
each occurred. procedures—protect environment, gather visible information.
suspected Remedy Procedure is the (additional path) path (if any) the user should follow to
diagnosis addressed a confirmed Suspected Diagnosis.

Tree Outline Table 3 gives an outline of topics for which troubleshooting

procedures are provided in the diagnostic tree (Tables 5 through 27)
on the following pages.
Table 3 Diagnostic Tree Outline

Context Symptom Page
Infrastructure Installation
Install
Installation Won’t Launch 18
Error Dialog Boxes: “InstallerPane”, “error code 2” 19
Error Dialog Box: “ECC uninstall directory” 20
Error Dialog Box: “Failed to launch”, “Access is denied” 21
Error Dialog Box: “no disk in drive” 22
CD Fails to Eject 22
Error: “Return code 255” 23
Repository Installation Slow 23
Repository Installation Hang at 1% 24
WLA PerformanceView License Key Not Valid 25
Uninstall
Error Dialog Boxes: “InstallerPane”, “error code 2” 26
Processes and Connectivity
ECC Server
Server-Database Connection Fails 27
Server-Database Connection Dropped 28
Store
Store-Database Connection Dropped 29
Console
Console Does Not Appear on Startup 1 30
Console Does Not Appear on Startup 2 31
Hosts
Host Icon Does Not Appear in Console 35
Agents
Agent Installation Cannot Be Performed on Console 32
Agent Startup Quits 33
Host Agent Device Icon Does Not Appear in Console 36
Console Error Dialog Box: “Agent not found”, 39
“Communication failed”

Table 3 Diagnostic Tree Outline (continued)

Context Symptom Page
Symmetrix systems
Symmetrix Icon Does Not Appear in Console 37
Symmetrix Icon Does Not Appear in Console; Agent 38
Error Message
Documentation Table 4 is intended to parallel the outline in Table 3. It identifies

References documentation to be consulted for further information about
component installation and operation.
Table 4 Topics Covered in Documentation

Scope Manual
Preparation
Networks EMC ControlCenter version 5.0.3 Installation and
Hosts Configuration Guide, Chapter 2, Chapter 3.
Environments
Installation
ECC Server EMC ControlCenter version 5.0.3 Installation and
Store Configuration Guide, Chapter 2.
Repository
Console EMC ControlCenter version 5.0.3 Installation and

Other ControlCenter Configuration Guide, Chapter 3.
clients
Agents EMC ControlCenter version 5.0.3 Installation and

Configuration Guide, Chapter 4, Chapter 6.
Startup
Maintenance
Repository EMC ControlCenter version 5.0.3 Installation and
Configuration Guide, Chapter 9.

Diagnostic Tree Refer to Table 3 on page 16 for a list of all symptoms.
Table 5 Installation Won’t Launch
Installation won’t launch

Procedure
Installation and Configuration Guide
Context
(Various)
1 CD-ROM drive in use. 1. Shut down any other running media players.
2. Eject CD 1.
3. Reinsert CD 1 and restart installation.
2 Autorun not enabled. You can launch the installation from CD 1 by command line:
<CD_drive>:\ECC.exe
3 Dirty or scratched CD-ROM. 1. Clean CD: Wipe with clean cloth from inner toward outer edge.
2. If CD still appears damaged, contact EMC for replacement.

Refer to Table 3 on page 16 for a list of all symptoms.
Table 6 Error Dialog Boxes: “InstallerPane”, “error code 2”
Error dialog boxes appear during installation

Procedure
Context
• Installation startup from CD 1
4 Installation target is nonwritable or 1. Close dialog boxes. Data entry screen “Choose installation directory”
nonexistent. should be available.
Refer to EMC ControlCenter version 5.0.3 Installation and

2. Enter valid, writable folder name for installation target.

3. Continue installation.

Table 7 Error Dialog Box: “ECC uninstall directory”
Error dialog box appears during install

Procedure
Context
• After an uninstall
• Installing infrastructure
• Installation startup
5 Although registry keys have been 1. Add registry key (REG_SZ):

removed, uninstall folder still exists.
HKEY_LOCAL_MACHINE
SOFTWARE
EMC Corporation
EMC CONTROL CENTER
5.0
INSTALL_ROOT
a. Open the Registry Editor with: Start, Run, regedit (or regedt32).
b. Drill down (as shown above), and select folder (5.0).
c. Create new key (right-click in right-hand pane, and select New, String
Value).
2. Set value of new key.

To do so, right-click name, select Modify, under Value Data in text box,
enter absolute pathname of installation directory.
3. Restart installation.


Table 8 Error Dialog Box: “Failed to launch”, “Access is denied”
Error dialog box appears during installation

Procedure
Context
• After an uninstall
• Installation startup from desktop or Start
menu
• After splash screen
6 One or more folders within the installation 1. Remove the two registry keys shown:
folder are missing or corrupt.
HKEY_LOCAL_MACHINE
SOFTWARE
EMC Corporation
EMC CONTROL CENTER
5.0
INSTALL_ROOT
LAST_CDDRIVE_USED
b. Drill down (as shown above), select, and delete.
2. Remove installation root folder.

3. Restart installation from CD 1.


Table 9 Error Dialog Box: “no disk in drive”
Error dialog box appears during installation

Procedure
Context
• CD-ROM tray is open
7 CD-ROM is expected in drive tray. 1. Insert next CD and close drive; dialog box should disappear.
2. Continue installation.
This error is harmless to software and
installation procedure. Refer to EMC ControlCenter version 5.0.3 Installation and
Table 10 CD Fails to Eject
CD does not eject automatically

Procedure
Context
• Auto-eject is enabled
8 Windows Explorer CD-ROM folder is open. 1. Close Windows Explorer, or just the CD-ROM folder within it.
9 Demo32.exe process is still running. 1. Open Task Manager, select Processes tab, select Image Name list.
2. Locate and select Demo32.exe and click End Process.

Table 11 Error: “Return code 255”
CD 2 installation aborts at 90% complete, with error: “Return code 255”

Procedure
Context
• Installing infrastructure: Repository
10 The Repository backup directory is

read-only or does not exist. Refer to EMC ControlCenter version 5.0.3 Installation and
1. Uninstall the Repository.

2. Create Repository backup directory.
3. Reinstall Repository.
Table 12 Repository Installation Slow
Repository installation seems takes longer than 35 minutes

Procedure
Context
11 Host may not be equipped according to Check that system configuration meets current specifications:
EMC specifications. • Refer to EMC ControlCenter/Open Edition Support Matrix dated
4/24/02 or later, available on EMC Powerlink.
• If unavailable, refer to EMC ControlCenter Performance and Scalability
Guidelines (P/N 300-000-431 Rev A04 or later), available on EMC
Powerlink.
12 Repository host is down. 1. Leave ControlCenter installation as is—do not yet take recovery
measures.
2. Check that Repository host is running. Reboot if necessary.
3. Wait for ControlCenter installation to recognize the Repository; then
continue.

Table 13 Repository Installation Hang at 1%
Installation of Repository hangs at 1% completed

Procedure
Installing and Configuring the Infrastructure
Installing the Repository
Context
13 Slow host processor. Installation can take several minutes to advance past 1%; please wait. The
process should not take longer than 35 minutes; else, refer to Table 12 on
page 23.
14 Oracle silent installation failed. Look for Oracle installation error messages in:
\\Program Files\Oracle\Inventory\logs\
InstallActions.log
and
\\Program Files\Oracle\Inventory\logs\
oraInstall.err
15 Windows version is neither of the Replace host or host operating system with the EMC-supported English or
supported English or Japanese versions. Japanese language Windows 2000.

Table 14 WLA PerformanceView License Key Not Valid
WLA PerformanceView license key not valid

Procedure
Installing and Configuring the Infrastructure
Installing Other Components
Context
• Installing Workload Analyzer
16 The host had a previous installation of 1. Delete any remaining previously installed folders.
ControlCenter, but was not fully cleaned of 2. Reinsert CD 1.
previous software.

Table 15 Error Dialog Boxes: “InstallerPane”, “error code 2”
Error dialog boxes appear during uninstall

Procedure
Context
• Launching uninstall from Add/Remove
Programs
Installation utility continues to function erroneously and will show the following dialog box if an uninstall is attempted:
17 Registry values 1. Exit the uninstall utility.

REPOSITORY_HOSTNAME, 2. Add registry keys (REG_SZ):
REPOSITORY_SERVER, or both, are HKEY_LOCAL_MACHINE
missing. SOFTWARE
EMC Corporation
EMC CONTROL CENTER
5.0
REPOSITORY_HOSTNAME
REPOSITORY_PORTNUMBER
b. Drill down (as shown above), and select folder (5.0).
c. Create new keys (right-click in right-hand panel, New, String Value).
3. Set values of new keys:
a. Right-click REPOSITORY_HOSTNAME, select Modify; under Value Data
in text box, enter Repository host name.
b. For REPOSITORY_PORTNUMBER, enter Oracle port (default = 1521).
4. Restart installation.


Table 16 Server-Database Connection Fails
Server does not appear to connect to database

Procedure
User Guide
Context
(Various)
Procedures at right apply to Items 18 and Preliminary steps:

19. Check for an error message to confirm the database is unavailable:
1. Open server.trc
2. Locate the following error message:
com.emc.ecc.common.EccuntimeException:Database
connection failed
3. If above error message is not found, try any other Suspected Diagnoses
available for the same symptoms.
Else, if above error message is found, continue with Item 18 and/or 19.
18 Account eccadmin is missing. Perform “Preliminary steps” 1 through 3 above.

(User currently logged in does not have 4. Check that account eccadmin has been created on ECC Server
sufficient privileges to start the services.) host. If not, create it.

5. Retry Console login.
19 Database failure. Perform “Preliminary steps” 1 through 3 above.

4. Check Services to ensure that at least the following are running (Status
shows Started):
OracleECCREP-HOMEAgent
OracleECCREP-HOMEDataGatherer
OracleECCREP-HOMETNSListener
OracleServicerambdb
a. From the Windows desktop, select Start, Settings, Control Panel,

Administrative Tools, Services.
b. For any service that is not running, right-click service name, and
select Start.
5. Retry Console login.

Table 17 Server-Database Connection Dropped
Server connection to database appears to have dropped

Procedure
User Guide
Context
(Various)
20 Database failure. 1. Check for an error message confirming the database is unavailable, and
respond:
a. Open server.trc
b. Locate the following error message string:
DatabaseMonitorPlugin: Database short term
failure
Else, if the above error message is found:
• Check Repository status using ControlCenter System Monitor:
a. Start System Monitor. (See ControlCenter System Monitor on
page 47.)
b. Under the General Status (or Server Status) tab, check whether
Repository is active or not.
• Alternatively, check status using Task Manager.
3. Check Services to ensure that at least the following are running (Status
shows Started):
OracleECCREP-HOMEAgent
OracleECCREP-HOMEDataGatherer
OracleECCREP-HOMETNSListener
OracleServicerambdb

b. For any service that is not running, right-click service name, and select
Start.

Table 18 Store-Database Connection Dropped
Store connection to database appears to have dropped

Procedure
User Guide
Context
(Various)
21 Connection failure. 1. Check for a database connection error message:

a. Open store.trc
b. Locate error message:
GotSQLException:Io exception:Connection reset
by peer:socket write error
Else, if the above error message is found:
• Check component status using ControlCenter System Monitor:
page 47.)
Store and Repository are active or not.
• Alternatively, check status using Task Manager.
3. From the Windows desktop, select Start, Settings, Control Panel,

If Store is not running, skip to step 4.
Else, if Store is currently running, prepare for a clean restart:
a. Locate Service EMC ControlCenter Store.
b. Stop Store: Right-click on service name, and select Stop.
c. Wait 2 minutes.
4. Start (restart) Store: Right-click service name, and select Start.

Table 19 Console Does Not Appear on Startup 1
Console window does not appear

Procedure
Installing the Console and Client Applications
Installing the Console
Context
Windows desktop: Start ECC Console icon
22 ECC Server is not running. 1. Check that Console logging is enabled. (See Console Logging in EMC
ControlCenter Version 5.0.3 Release Notes.)
2. Ping ECC Server host.
If host does not respond, try other remedies for same symptoms.
3. Check for an error message:
a. Open console.trc
b. Locate the following error message:
Could not connect to the ECC server; The EMC
ControlCenter server on <hostname> may not be
ready for logins
Else if message is found, check service status:
page 47.)
b. Under the General Status (or Server Status) tab, check
whether server is running (“active”) or not.
• Alternatively, check component status using Task Manager.
5. If server is not running, skip to step 6.
Else if server is running, prepare for a clean restart:
b. Locate service EMC ControlCenter Server.
c. Stop service: Right-click on service name, and select Stop.
6. Restore system:
a. Restart server: Right-click service name (see Step 5), and select
Start.
b. Restart Console: Click Start ECC Console icon.

Table 20 Console Does Not Appear on Startup 2
Console does not start up

Procedure
User Guide
Context
• Console startup from ECC Client Install
Configuration window
23 ECC Server is not running. 1. Check server status:

page 47.)
b. Under the General Status (or Server Status) tab, check
whether ECC Server is running (“active”) or not.
2. If ECC Server is running, skip to step 3.

Else, if ECC Server is not currently running, start it:
b. Locate service EMC ControlCenter Server.
c. Start service: Right-click on service name, and select Start.
3. Verify that server.trc is being updated:

Locate file detail for server.trc at:
<install_root>\ecc_inf\data\<hostname>\data\
4. If server.trc is being updated, skip to step 5.

Else, if server.trc is not being updated:
a. Stop ECC Server: From Services window, right-click service name,
and select Stop.
b. Open a command window.
c. To display any startup errors, start the ECC Server from the
command line, by executing:
<install_root>\ecc_inf\data\StartServer.bat
5. Retry Console startup.

Table 21 Agent Installation Cannot Be Performed on Console
Console popup dialog box shows:

Trouble communicating with master agent:
Cannot assign requested address: connect
Procedure
User Guide
Context
(Various)
24 Master Agent is down. 1. Check Master Agent status at its host.
2. If the Master Agent appears down, restart it:

Windows:
Windows Services at the agent host.
UNIX:
<install_root>/exec/start_master.csh

Table 22 Agent Startup Quits
Agent process startup fails

• Fails to establish server connection
• Causes printing of occasional error messages
• Hangs
Procedure
User Guide
Context
(Various)
25 No server is running. Start (restart) server:

Administrative Tools, Services
b. Right-click Service EMC ControlCenter Server, and select Start.

Table 22 Agent Startup Quits (continued)
Agent process startup fails

• Fails to establish server connection
• Causes printing of occasional error messages
• Hangs
Procedure
User Guide
Context
(Various)
26 No communication channel current For items 26 and 27:

between agent and ECC Server. 1. Open <agent_abbrev>_sst.log
2. Locate:
27 Wrong ECC Server host port.
Error caught in SST Function [init], Failed
connecting to Host (IP) – Port
3. If error message found, reset port.
28 No Store is running. 1. Open <agent_abbrev>_sst.log file.

2. Locate error message:
Error caught in SST Function [init], Received
error from server: Cannot contact store:
<store_name>
3. If error message is found, check services status:

page 47.)
Store is running (“active”) or not.
4. If Store is not currently running, continue to step 5.

Else if Store is currently running:
Prepare for restart:
Administrative Tools, Services
b. Right-click service EMC ControlCenter Store, and select Stop.
5. Start (restart) Store: Right-click service name (see step 4), and select
Start.

Table 23 Host Icon Does Not Appear in Console
Host instance icon does not appear

Procedure
Installing and Configuring Open Systems Agents
Remote Install via the Console
Context
Console tree panel:
Administration
Install
Hosts
29 Master Agent is not running. Check that the Master Agent is running.
Windows:
1. Open Task Manager.
2. Under Processes tab, attempt to locate mstragent.exe
UNIX:
1. Run ps -ef
2. Attempt to locate mstragent in output.
3. If process name does not appear, Master Agent is not running. Do the
following:
Windows:
• Restart Master Agent through Windows Services.
UNIX:
• Run <install_root>/exec/start_master.csh
30 Network error occurred. Check to see if a network error is reaching the ECC Server.
Windows, UNIX:
1. Open masteragent.log
2. Find any entries labeled network error
3. Ping IP address of device indicated in error log entry.

Table 24 Host Agent Device Icon Does Not Appear in Console
Host agent device icons do not appear

Procedure
Context
Console tree panel
31 MSR Agent or MNR Agent is not running. Check whether MNR or MSR Agent is running:
Windows:
2. Under Processes tab, attempt to locate mnragent.exe,
msragent.exe
UNIX:
1. Run ps -ef
2. Attempt to locate mnragent, msragent in output.
3. If the process does not appear, the agent is not running:

Restart the agent.
32 Network error occurred. Check whether a network error is reaching the ECC Server.
1. Open mnr_sst.log and msr_sst.log
2. Find any entries labeled “network error.”
3. Ping the IP address of device indicated in error item to see if it can be
reached.
4. If necessary, restart host or check network.
33 Store errors occurred. Check Store log store.trc for any SQL errors.

Table 25 Symmetrix Icon Does Not Appear in Console
Symmetrix icon not found

Procedure
Context
Console tree panel
34 Symmetrix agent is not running. Check that the Storage Agent for Symmetrix is running.
(installation problems) Windows:
2. Under Processes tab, attempt to locate egsagent.exe
UNIX:
1. Run ps -ef
2. Attempt to locate egsagent in output.
3. If process does not appear, Symmetrix agent is not running.

Restart Symmetrix agent through Console.
35 Network error occurred. Check to see if a network error is reaching the ECC Server.
1. Open egs_sst.log
2. Find any entries labeled network error.
3. Ping IP address of error device to see if it can be reached.
4. If necessary, restart host or check network.
36 Store errors. Check Store log store.trc for SQL errors.
37 Other errors. In general, check that logs show:

• Symmetrix storage array(s) present and discovered.
• AMPRIM message was transferred from server.
• Configuration is being sent to STOR.

Table 26 Symmetrix Icon Does Not Appear in Console; Agent Error Message
Symmetrix icon does not appear and

Agent continues printing error message
Procedure
User Guide
Context
(Various)
38 1. Open <agent_abbrev>_sst.log
2. Locate:
EGSAgent::AmPrimary __|Current Symms list
but without any Symmetrix storage arrays are listed.
3. If list is missing:
a. Wait until connection is established.
b. Break connection.

Table 27 Console Error Dialog Box: “Agent not found”, “Communication failed”
Console fails executing commands

• Console popup shows:
Agent not found
or Communication failed
Procedure
User Guide
Context
(Various)
39 Master Agent is down. 1. Check Master Agent status using the Console.
2. If the Master Agent appears down from the Console, check its status
locally, using the Task Manager at the agent host.
3. If the Master Agent locally appears down, restart it from Windows
Services at the agent host.
40 Agent is down. 1. Check Master Agent status, and restart if necessary, according to Item
38.
2. Check agent status using the Console.
3. If agent appears down, skip to step 6.
4. Check agent logs on agent host for failure indicators.
5. If no messages are found in agent logs, try a new Suspected Diagnosis,
or contact EMC.
6. Restart agent from Console.

Reference
Reference
The following tables identify names and locations for specific objects
in ControlCenter.
Contents
◆ Agent Names and Codes ................................................................... 41

◆ Port Settings......................................................................................... 45
◆ Trace and Log Files ............................................................................. 45
◆ Error Messages .................................................................................... 47
◆ Tools ...................................................................................................... 47

Reference
Agent Names and

Codes Three-letter codes are assigned for each of the agents. They are used
as prefix names, and as labels of folders in the ControlCenter
directory structure, as well as other reasons. The first and third letters
refer to EMC-internal identification, but uniquely identify the agent.
The second letter refers to the host operating system, as identified in
Table 28.
These codes are listed alphabetically in Table 29 and in Table 30 (the
latter ordered by the official “short name,” or a keyword or
description in italics).
Agent code names do not always carry over to agent executable names—in
some cases, generic agents are used. See Table 28 and Table 29 for full
comparison in EMC ControlCenter version 5.0.3.
Table 28 Agent Operating System Indicator
Middle Character
of Agent Code Operating System
A AIX
G (multiple or “generic”)
H HP-UX
M MVS
N Windows NT, Windows 2000
S Solaris
V Novell

Reference
Table 29 Agent Codes
Executable
Code Agent Name (if applicable)
CNC Storage Agent for Celerra

CNG Integration Gateway (Windows) cngagent.exe
CSG Integration Gateway (Solaris) csgagent
EGA Connectivity Agent for SNMP
EGF Connectivity Agent for SDM (generic)
EGN Connectivity Agent for Switches (generic)
EGO Database Agent for Oracle (generic)
EGS Storage Agent for Symmetrix (generic)
ENF Connectivity Agent for SDM (Windows) egfagent.exe
ENN Connectivity Agent for Switches (Windows) egnagent.exe
ENO Database Agent for Oracle (Windows) egoagent.exe
ENS Storage Agent for Symmetrix (Windows) egsagent.exe
ENW WLA Archiver (Windows) enwagent.exe
ESF Connectivity Agent for SDM (Solaris) egfagent
ESO Database Agent for Oracle (Solaris) egoagent
ESS Storage Agent for Symmetrix (Solaris) egsagent
MAR Storage Agent for AIX maragent
MAY Tivoli Storage Manager Agent (AIX) mayagent
MGA Master Agent (generic)
MGD Storage Agent for HDS (generic)
MGL Storage Agent for CLARiiON (generic)
MGQ Storage Agent for Compaq StorageWorks (generic)
MGY Backup Agent for TSM (Tivoli Storage Manager) (generic)
MHD Storage Agent for HDS (HP-UX) mhdagent
MHR Host Agent for HP-UX mhragent
MHY Tivoli Storage Manager Agent (HP-UX) mhyagent

Reference
Table 29 Agent Codes (continued)
Executable
Code Agent Name (if applicable)
MM2 Database Agent for DB2

MMC Storage Agent for IBM ESS
MMH Host Agent for MVS HSM
MMI Storage Agent for RVA/SVA
MML Logical Agent for MVS
MMN Master Agent (MVS)
MMP Physical Agent for MVS
MMS Host Agent for MVS SMS
MMT Tape Agent for MVS
MMY Backup Agent for TSM (Tivoli Storage Manager)
MND Storage Agent for HDS (Windows) mndagent.exe
MNL Storage Agent for CLARiiON (generic) mglagent.exe
MNQ Storage Agent for Compaq StorageWorks (Windows) mnqagent.exe
MNR Host Agent for Windows (Windows) mnragent.exe
MNY Tivoli Storage Manager Agent (Windows)
MSR Host Agent for Solaris msragent
MSY Tivoli Storage Manager Agent (Solaris) msyagent
MVR Host Agent for Novell mvragent.exe

Reference
Table 30 Agent Naming Index
Short Name or
Index-only item Code Full Name
AIX MAR Host Agent for AIX

Celerra CNC Storage Agent for Celerra
CLARiiON MGL Storage Agent for CLARiiON
DB2 MM2 Database Agent for DB2
Gateway CNG Integration Gateway
HDS MGD Storage Agent for HDS
Host MAR Host Agent for AIX
Host MHR Host Agent for HP-UX
Host MSR Host Agent for Solaris
Host MNR Host Agent for Windows
HP-UX MHR Host Agent for HP-UX
IBM ESS MMC Storage Agent for IBM ESS
Logical MML Logical Agent for MVS
Master MGA Master Agent (Windows, UNIX)
Master MMN Master Agent (MVS)
MVS DB2 MM2 Database Agent for DB2
MVS IBM ESS MMC Storage Agent for IBM ESS
MVS (Logical) MML Logical Agent for MVS
MVS (Physical) MMP Physical Agent for MVS
MVS (Tape Mgt) MMT Tape Agent for MVS
MVS HSM MMH Host Agent for MVS HSM
MVS SMS MMS Host Agent for MVS SMS
Novell MVR Host Agent for Novell
Oracle EGO Database Agent for Oracle
Physical MMP Physical Agent for MVS
RVA/SVA MMI Storage Agent for RVA/SVA
SDM EGF Connectivity Agent for SDM
SNMP Connectivity EGA Connectivity Agent for SNMP
Solaris MSR Host Agent for Solaris
StorageWorks MGQ Storage Agent for Compaq StorageWorks
Switch EGN Connectivity Agent for Switches
SYMAPI Server — SYMAPI Server
Symmetrix EGS Storage Agent for Symmetrix
Tape Management MMT Tape Agent for MVS
TSM MGY Backup Agent for TSM (Tivoli Storage Manager)
TSM MMY Backup Agent for TSM (Tivoli Storage Manager)
WLA Archiver ENW WLA Archiver

Reference
Port Settings Port settings are requested by the installation wizard, but may be
reset. Default listening port settings are given in Table 31.
Table 31 Defalt Listening Port Settings
Component Port Number
Server 5799
Enhydra Server 30000
Repository 1521
Master Agent 5798
Trace and Log Files Each component has at least one log file. Depending on its source
code origin (see Error Levels), a log can capture up to five or six
severity levels.
Error Levels
Log files for the Java-based ECC Server, Store, and Console can be set
to capture messages at one of the threshold levels shown in Table 32;
while log files for the C++ based agents can be set as in Table 33.
Logging can be adjusted from default levels by consulting with EMC
Customer Support.
Table 32 Logging Error Levels: ECC Server, Store, Console
Level Name and Indication
7 ERROR — failure
6 WARN — warning
5 INFO — information
4 CONFIG — static configuration message
3 FINE — tracing information
2 FINER — detailed tracing information
1 DEBUG — most detailed tracing information

Reference
Table 33 Logging Error Levels: Agents
Level Name and Indication
0 Fatal error
1 Warning
2 Information
3 Detailed information
4 Debugging (primarily for development and testing)
5 Debugging (primarily for development and testing)
Trace and Log Tables 34 and 35 show trace log files available and their locations.
Files and Locations
Table 34 Infrastructure Trace and Log Files on Infrastructure Host
Name Location Description

server.trc <install_root>\ecc_inf\data\<hostname>\data\ Server trace
store.trc <install_root>\ecc_inf\data\<hostname>\data\ Store trace
console.trc %USERPROFILE%\ecc\<infrastructure_host>\ Console trace
consoleLaunch.log %USERPROFILE%\ecc\<infrastructure_host>\data\ Console launch log
snmpcollector.log <install_root>\ecc_inf\data\<hostname>\data\ SNMP collector trace
Table 35 Trace and Log File Paths
Name Location Description

On infrastructure host(s):
masteragent.log <destination_directory>\exec Master Agent log
mga_console <destination_directory>\exec
On infrastructure host(s), in agent-specific
Example logs: directories:
egs_sst <destination_directory>\exec\EGS500
EGS500_<year-month-day-time> <destination_directory>\exec\EGS500
mnr_sst <destination_directory>\exec\MNR500
MNR500_<year-month-day-time> <destination_directory>\exec\MNR500

Reference
Error Messages Consult the online error Help for comprehensive error message
information.
Online Error Help Each Help message contains the message label, description, reason,
and action to take (if any). See Figure 4 on page 47.
Figure 4 Online Help Message Entry
Console
Two types of error messages are presented:
◆ Responses to unauthorized requests to delete a managed object
◆ Symmetrix error alert messages
Tools Software tools to help with ControlCenter troubleshooting are

identified below. These packages are available only from EMC
Powerlink.
ControlCenter System This utility, which is installed on an ECC Server host, provides status
Monitor information for ControlCenter components.
ControlCenter Log This utility, which runs on Windows platforms, is used to search logs
Viewer for patterns to assist log analysis, and to zip log files for delivery to
Customer Service.

Reference
Copyright © 2002 EMC Corporation. All Rights Reserved.

EMC believes the information in this publication is accurate as of its publication date. The information is subject to
change without notice.
THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION MAKES NO
REPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO THE INFORMATION IN THIS
PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED WARRANTIES OF MERCHANTABILITY OR
FITNESS FOR A PARTICULAR PURPOSE.
Use, copying, and distribution of any EMC software described in this publication requires an applicable software
license.
EMC and Symmetrix are registered trademarks, and Celerra, ControlCenter and Connectrix are trademarks of EMC
Corporation. All other trademarks used herein are the property of their respective owner.

EMC Control Center Troubleshooting

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

EMC Control Center Troubleshooting

Uploaded by

Copyright:

Available Formats

EMC ControlCenter

This document presents descriptions and remedies to problems you

This manual identifies categories of possible unacceptable behaviors

Audience This document is intended for use by ControlCenter system

2 EMC ControlCenter Version 5.0.3 Troubleshooting Guidelines

You should have the ControlCenter version 5.0.3 documentation set

1. EMC ControlCenter/Open Edition Support Matrix and EMC ControlCenter

EMC ControlCenter Version 5.0.3 Troubleshooting Guidelines 3

• EMC ControlCenter version 5.0 Integration Packages Product

AVANT GARDE Keystrokes

Palatino, ◆ Dialog box, button, icon, and menu items in text

Palatino, ◆ New terms or unique word usage in text

Courier, ◆ User entry. For example:

◆ Options in command line syntax

4 EMC ControlCenter Version 5.0.3 Troubleshooting Guidelines

Components ControlCenter is a suite of interworking software components. Each

Console Tier Infrastructure Tier Agent Tier

Figure 1 ControlCenter Software Tiers

EMC ControlCenter Version 5.0.3 Troubleshooting Guidelines 5

Agent Tier Master Agent

6 EMC ControlCenter Version 5.0.3 Troubleshooting Guidelines

Includes agents for — Integration Gateway and Workload

Interfaces Command Line

EMC ControlCenter Version 5.0.3 Troubleshooting Guidelines 7

Figure 2 ControlCenter Console (Example View)

8 EMC ControlCenter Version 5.0.3 Troubleshooting Guidelines

EMC ControlCenter Version 5.0 Introduction (P/N 300-000-295) and EMC

EMC ControlCenter Version 5.0.3 Troubleshooting Guidelines 9

Table 1 Error Indicators in Console Icons

Red circle with cross:

Yellow triangle with exclamation point:

Red arrow at lower right of folder:

Red circle (shown) or Yellow triangle:

Red circle or Yellow triangle (shown), with red arrow:

GUI Content Alerts

Figure 3 Alert Toolbar

10 EMC ControlCenter Version 5.0.3 Troubleshooting Guidelines

Activity Records Command History

EMC ControlCenter Version 5.0.3 Troubleshooting Guidelines 11

When ControlCenter Malfunctions

If symptoms are not identified in diagnostic tree—

If symptoms and solutions are not identified in Powerlink,

Recognizing ControlCenter is a configuration, maintenance, monitoring, and

12 EMC ControlCenter Version 5.0.3 Troubleshooting Guidelines

Responding to The following is a general response sequence to symptom discovery.

EMC ControlCenter Version 5.0.3 Troubleshooting Guidelines 13

14 EMC ControlCenter Version 5.0.3 Troubleshooting Guidelines

Diagnosis and Remedy

Table 2 Diagnostic Tree Key

Item Suspected Diagnosis Diagnostic and Remedy Procedures

EMC ControlCenter Version 5.0.3 Troubleshooting Guidelines 15

Tree Outline Table 3 gives an outline of topics for which troubleshooting

Table 3 Diagnostic Tree Outline

16 EMC ControlCenter Version 5.0.3 Troubleshooting Guidelines

Table 3 Diagnostic Tree Outline (continued)

Documentation Table 4 is intended to parallel the outline in Table 3. It identifies

Table 4 Topics Covered in Documentation

Console EMC ControlCenter version 5.0.3 Installation and

Agents EMC ControlCenter version 5.0.3 Installation and

EMC ControlCenter Version 5.0.3 Troubleshooting Guidelines 17

Diagnostic Tree Refer to Table 3 on page 16 for a list of all symptoms.

Table 5 Installation Won’t Launch

Installation won’t launch