Secure IED Management Case Studies
A. Salvador, Network Integration Specialist, SUBNET Solutions Inc., D. Mack, Substation
Integration Specialist, SUBNET Solutions Inc., and C. Carnegie, System Integration Specialist,
SUBNET Solutions Inc.,
Abstract--This paper presents a summary and analysis
deployment of a secure Intelligent Electronic Device (IED)
management system at two different utilities in North America.
Each of these utilities began their investigation into technologies
and methodologies for securing their system for North America
Reliability Corporation (NERC) Critical Infrastructure
Protection (CIP) standards. This report is intended for anyone
considering the implementation of an IED management system
Index TermsDevice Management, Interoperability, Legacy
Devices, Multi-Vendor.
I. INTRODUCTION
T HE two utilities featured in this paper had the same goals
when first initiating the their projects; to meet NERC CIP
[1] standards by implementing IED management systems that
assist with non-operational data collection, provide secure
remote access and increase system reliability. However, due to
old, yet reliable protection equipment, both utilities
encountered issues with integrating legacy devices with modern
IED management systems.
This paper summarizes the lifecycle of both projects,
including a description of the problems and challenges the
utilities faced during the implementation to meet their unique
challenges, while keeping in mind their expansion plans over
the next few years.
II. CASE STUDY #1
A utility was dealing with the problem of trying to manage
all of the distribution automation devices they have on their
network. They had too many devices to manage manually, and
access to device event data was non-existent or difficult, which
prevented them performing root-cause analysis of circuit
reliability. They had security and worker safety risks in
managing the device passwords, settings and configuration
changes. Because of this, the utility executed a project to
develop a solution that overcame these challenges.
A. Project Goals
The utilitys main goal is to meet NERC CIP standard by
implementing an IED management system that securely allows
personnel to access remote devices, manage its passwords,
obtain configuration files and retrieve event files without
physically accessing these devices. By allowing personnel to
remotely manage the devices, the utility expects to save labor
cost, equipment maintenance cost, and increase safety and
reliability within its system.
978-1-4673-8848-1/16/$31.00 2016 IEEE
B. Existing System and Issues
The utilitys generation plants consist of 41 dam sites, 30
hydro facilities, and 9 thermal sites. They have approximately
18,000 kilometers of transmission lines, 260 substations and
56,000 kilometers of distribution lines. The utility is providing
electricity to approximately 1.8 million customers in the region.
Due to the utilitys size and coverage area, approaching a
solution to manage all their remote devices, which is estimated
to be around 10,000 devices, is not a simple task. The utility is
not only faced with a quantity problem but also terrain issues.
Due to the mountain range nearby, certain substations and
devices are difficult to communicate with due to a complex
communication infrastructure as well as the possibility of bad
weather. Depending on the location of the devices, bandwidth
can be a scarce commodity, and only operational data is
collected over the communication lines. To obtain important
non-operational data such as event files and configuration files,
personnel are forced to physically access these remote devices,
which introduces expensive labor cost and safety issues due to
the terrain. Since non-operational data was so difficult to
collect, analysis of the data is intermittent and fails to produce
a clear picture of the system for data trending and predictive
maintenance.
Another problem the utility faces is the vast number of
brands of devices that are implemented in the field. Each vendor
provides devices that use a different protocol or protocol variant
and, most of the time, a proprietary configuration software
application that must be learned by the utilitys personnel that
are collecting the data.
C. Solution Design
To reduce labor cost and increase safety, the utility focused
on providing their distribution engineers a solution for
managing IEDs on their distribution network. The project scope
included reclosers, voltage regulators, switchgear relays and
capacitor bank controllers and allowed for future expansion to
substation and transmission equipment. The solution was
required to be device/vendor agnostic with the ability to secure
access to, and manage non-operational data of these IEDs
remotely. Due to the unforgiving terrain of the utilitys
distribution network, the solution needed to be robust enough
to handle low performance communications such as satellite
and cellular connected IEDs. As well, the solution would allow
them to become NERC CIP compliant.
The utility selected a software integration company as the
contractor to drive this project to completion. Together, they
designed a central IED management solution to overcome these
challenges. The platform is vendor agnostic, provides a means
for remotely maintaining non-operational IED data, and

is scalable for future expansion. It supports remotely connecting
to IEDs regardless of the communication protocol or
connection media that is utilized by the devices. This remote
connectivity provides users with the ability to configure and/or
retrieve data, as if the user is directly connected to the device.
Most importantly, it provides them a secure method of
accessing the remote IEDs for various management tasks.
Furthermore, the system is designed to significantly aid the
utility for compliance with guidelines, standards, and policies
for critical cyber assets as outlined within NERC CIP
requirements.
The solution is installed in the corporate local area network
(LAN) and natively integrated with the utilitys active
directory. The system is isolated in a demilitarized zone (DMZ)
utilizing firewalls with strict access rules. An example of the
rules on the firewall between Corporate LAN and the DMZ
include ports required to access the system with Microsoft
Remote Desktop client, authenticate with the Microsoft Active
Directory and send outbound email notifications. The firewall
between DMZ and remote device network is configured to only
allow access to the distribution IEDs intended to be managed
by the solution.
The solution is modular, allowing the utility to pick and
choose which modules were relevant to their system. The
Modules available and implemented by the utility were for
remote engineering access, password management,
configuration management and event file management.
The Remote engineering access module allows personnel at
the utility to remotely access the field device securely from the
centralized system. The module only displays the devices that
the user has permission to access, and other devices are
completely hidden and are inaccessible to the user. The system
is configured with pre-approved vendor applications and is
linked to the proper field devices. The user has the option of
what pre-approved applications they want to use to connect to
each device. Once connected to the field device, the system
monitors the users activity such as commands sent and/or
mouse clicks used within the system. If the user sends an un-
approved command or clicked on un-approved item, the system
intercepts the action and prevents the command or action being
sent to the field device. All remote connections made by the
user to field devices are logged and auditable for NERC CIP or
for internal purposes.
The Password management module allows the system to
manage the field devices passwords. Passwords are encrypted
and centrally stored within the system. The system
automatically accesses the device and changes its passwords
based on utilitys preference; for example, at a configurable
interval, such as once a year, or when an employee uses a
password and no longer needs the current password. The
Password management module assists the utilitys compliance
with NERC CIP where it requires that field devices have their
passwords changed to its highest complexity every 15 months.
Making password changes on all field devices manually can be
tedious and very expensive for the utility.
The Configuration management module allows the system to
automatically retrieve configuration files from field devices. It
compares any new incoming configuration file against a
previous configuration file that was retrieved from the same
device. If the module detects any differences, it can initiate
2
work flow events that can alert specific personnel allowing
them to examine the new configuration file, and determine if
new changes should be approved or denied. The new
configuration file can stay in the device or the old configuration
file can be pushed down to the device if the changes were
rejected. All versions of the configuration file are centrally
stored, archived and are accessible to utilitys personnel. For
devices that have low bandwidth communication, the
automated retrieval of configuration files can be disabled and
only on demand retrieval is allowed. This allows users to only
retrieve configuration files when required and communication
is reserved for operational data.
The Event file management module allows the system to
automatically retrieve new event files from field devices. When
a new event file comes into the system, the module can alert
subscribed personnel via email. Personnel can then analyze the
new event file and determine the importance of the event. All
event files are centrally stored and archived within the system
where personnel can access multiple files and analyze them for
predictive maintenance on the field device or its corresponding
equipment. Similar to configuration management, devices that
have low bandwidth communication, the automated retrieval of
configuration files can be disabled and only on demand retrieval
is allowed.
In addition to the modules that the IED Management system
provides, new software features and enhancements are required
to meet all of the project requirements. One of the key concepts
used to tackle the design of these new features is the process of
Agile Development. This iterative approach to the software
development process introduced two important benefits to the
project, evaluation and transparency.
Many times, high level requirements defined in a projects
scope of work are not enough to fully understand the expected
function or use case. To reduce wasted time and money, the
requirements are discussed and defined in detail with the utility.
After development of a function is complete, the utility is given
the opportunity to evaluate it. Going through this iterative
process several times throughout the project provides
transparency, demonstrates progress, and confirms the solution
for the utility. The Agile approach is a win-win scenario for
both the utility and the integration company. The utility
receives the functionalities they are looking for and the
integration company increased its products functionalities and
capabilities for other utilities to take advantage of.
In addition to the benefits of the agile approach mentioned
above, the process helps drive innovation for advancements in
vendor agnostic solutions, specifically, advancements in the
ability to manage and interoperate with closed and proprietary
IEDs. A key requirement for the utility is to automate time
consuming and inefficient tasks for their distribution teams.
Tasks such as collecting and organizing IED event data,
collecting and looking for out of band IED configuration
changes, and changing IED passwords. For some IEDs, these
tasks can easily be automated if they support open standard
protocols such as Modbus, SEL ASCII, and telnet. The utility
requires automation of these tasks for the several devices such
as the SEL 751 and Eaton Form 6. The SEL 751 provides well-
documented, open standard protocols and is easily automated
in the system through a protocol driver. This type of driver
automates tasks to the end device using the native

communication protocol. However, it was clear in the initial
investigations that the Eaton Form 6 was a closed system, and
did not support open standard protocols. Analysis of the
communication protocol showed messages were encrypted or
hashed. Users are forced to manually manage these IEDs with
vendor-specific software.
Lacking the ability to communicate with the Form 6 device
using an open standard, the concept of an Application Driver in
the IED management system was developed during the project.
This type of driver is based on a software wrapper developed to
remotely drive a vendor application to communicate with an
IED. It provides the ability to automatically mimic the mouse
clicks, keyboard entries, and file saves normally performed by
a human. Working with the utilitys distribution teams, several
workflows were created for all the mouse clicks and keyboard
entries for communicating with the Eaton Form 6 using the
ProView application. These workflows were turned into Jobs
within the system. These Jobs can be issued to a specific device
on demand by an authorized user or scheduled on a recurring
basis by the system. The jobs are generalized to be vendor
agnostic providing users with a standard mechanism for
interacting with all devices being managed by the system. Jobs
such as change password, get fault files, get configuration, get
SOE, and get data profiler can be issued against an Eaton or
SEL device. Jobs can also be disabled by users in the event that
a device is physically being worked on in the field. It is also the
responsibility of the system to determine if the device required
using an Application Driver or a Protocol Driver to complete
the job. The jobs would also only upload new information from
an IED by comparing it to the last known configuration, last
collected fault file, or last retrieved SOE. This comparison
becomes invaluable when configured on a scheduled basis, as
the system can then watch for out-of-bounds configuration
changes or event fault data for post-event analysis.
Further, the Application Driver can permanently hide or
restrict access to menus, buttons, and/or dialogs within the
vendor application. This concept is incorporated in the remote
engineering access module to improve security and reduce
human error. In the case of the Eaton ProView software, a user
is required to log into the application first, followed by also
logging into the end device. By utilizing the Application Driver
capabilities, a workflow was developed to automatically log
into the application and device, without any user interaction or
knowledge of the passwords. Following this workflow, the user
is then given control of the application to remotely
communicate with the IED. Extending this concept to security,
depending on the users authorized permissions, certain menus
and buttons in the application like Trip or Close can be disabled
throughout the remote engineering access session with the user.
This is important to the utility, as the ProView software
provides these buttons to the user, no matter what device access
level or application role was used to connect to the device.
The concept of logging is also an important requirement for
the utility. The goal was for the system to provide situational
awareness. In terms of individual devices, the concept of device
state was created to allow users to immediately understand if a
device was normal, disabled, tagged or decommissioned when
users navigated to devices in the system. All jobs issued, either
by a user or the system, are also displayed and logged for the
device. The jobs contain state information indicating if the job
3
completed successfully or if it failed for a specific reason.
Furthermore, all IED specific documents that are managed by
the solution, such as configuration files, point mappings,
drawings, stencils, etc. are versioned controlled. The version
history for each file indicates the modified date, modified user
and comments associated with the version. It also allowed users
to retrieve previous versions for recovery situations. In
addition, configuration files are integrated in document
workflow approval processes. For all files in the Working
library, users are required to exclusively check out the file
before they are allowed to edit them in the system. When a user
checks out a file, the system indicates which user is working on
the file. When a user checks in a file and attempts to approve it
to a major revision, other users with authorized approval
permissions are notified by email of the new configuration.
Only the users with the approval permissions can approve or
reject the file after verifying the changes.
III. CASE STUDY #2
A utility was able to cost-effectively implement a secure and
powerful relay and meter integration and management solution.
A key to this projects cost-effectiveness was that the utility
was able to avoid significant capital and labor costs associated
with relay firmware upgrades and/or relay replacement and
recommissioning. Instead, the utility was able to implement a
solution that met the objects listed above without changing any
of the relays they have deployed over the last two decades.
A. Project Goals
The utilitys main goal is to meet NERC CIP standard by
implementing additional security and record keeping for
protection relays and meters at minimum cost. However, the
utilitys system consists of old devices and contains schemas
that were designed before NERC CIP existed. The question of
How do you implement new functionality to existing
infrastructure at minimum cost? presented itself to the utility.
B. Existing System and Issues
The existing system can be described as a large coal
generation plant connected to a Bulk Electric System (BES) at
230kV supplying power to a large vehicle manufacturer. The
distribution grid for the system operates at 230kV, 120kV and
34.5kV. The plant generates 750 MW of power to supply the
vehicle manufacturer, but has sufficient capacity so that it can
sell some of the power to the state for other consumer purposes.
The protection system is comprised of GE Universal Relay
(UR) relays that have version 2.xx firmware. These relays have
been in place since the plant was commissioned and are
functioning as expected.
The secondary protection scheme is comprised of Alstom
Micom and Schweitzer Engineering Laboratories (SEL) relays.
The system metering is handled by Power Measurement
Laboratories model 7600 and 7650 revenue meters.
As part of the project to increase security and record keeping,
the utility deployed a new IED Management system and Human
Machine Interface (HMI) system. The IED Management
system allow the utilitys personnel to securely access the relay
and meters for maintenance and automatically manage the
devices passwords for NERC CIP compliance. The HMI
system allows the utility to have better visual knowledge of

their protection system. The HMI system provides real-time
visual displays, digital alarm panels and data trending. As part
of the IED management system, the system can also
automatically retrieve event files from the SEL and GE UR
relays.
The main issue that was observed during the IED
management system deployment was the advanced capabilities
of the newer Modbus protocol used for Event File Collection in
the GE UR relays caused the version 2.xx GE UR relay to go
into a restart sequence on every event file collection cycle. In
other words, during the event file collection sequence, the
command would begin power cycling every relay in the entire
system, one at a time. It was determined that the newer Modbus
commands produced this unexpected result with the older GE
UR relays. GE was unaware of the issue with the version 2.xx
firmware and recommended the replacement of the GE UR
relays with newer hardware and firmware version 7.xx. The
utility contemplated these two solutions for the project:
Replace all GE UR version 2.xx relays with version.
7.xx relays
Leave all GE UR version 2.xx relays in place and
modify the Modbus protocol used for Event File
Collection
C. Solution 1: Replace all GE UR version 2.xx relays with
version 7.xx relays.
This solution is to remove and replace all existing UR relays
with newer version of the relays. There are several problems
with this solution. The first issue is the cost. Thirty-seven relays
at a cost of $6000 per relay is approximately $222,000. This
significant number was not budgeted into the cost of the project.
The second issue is the manpower and outage time required to
replace every single relay. Allocating manpower resources and
scheduling time for systems to be down while relays are
replaced would have added a significant increase in the budget.
The third issue is the need to recommission every device and
connection because all the original wiring would have to be
disconnected and then reconnected to all thirty-seven relays.
Once again, there are significant time and financial costs
associated with the commissioning effort.
D. Solution 2: Leave all GE UR version 2.xx relays in place
and modify the Modbus protocol
This solution is leaving the current relays in place and have
them function as they had in the past. This was the best idea for
the utility, but the issue with restart sequence caused by event
file collection needed to be resolved. In discussions with the
IED management system vendor, they determined that it would
be possible to modify the Modbus protocol in the IED
management system to handle older GE UR relays.
After some testing with older GE UR relays, a modification
to the Modbus protocol was implemented and a hotfix for the
system was provided to the utility.
E. Solution Design
The final decision made by the utility was to implement
Solution 2. A hotfix was created by the integration company
and the utility was able to successfully retrieve event files from
the old GE UR relays without restart issues. Once the Modbus
protocol change was proven to be working as expected, the
utility realized they made the right decision. The cost to pay the
4
vendor to modify the Modbus protocol was a small fraction of
what it would have cost to replace all the GE UR relays.
In regards to labor cost, the implementation of the hotfix took
only a few hours, and minimal labor cost, to install and test the
functionality on the live system in comparison to the labor cost
associated with commissioning of new relays and its protection
schema. Event file collection for all GE UR relays was resumed
and the entire system performed as expected with no issues for
any relays.
IV. CONCLUSION
As part of any hardware vendor business model, hardware
vendors will always implement features that will encourage
utilities to purchase more hardware. As experienced in the first
case study, it is not uncommon for hardware vendors to
implement proprietary protocols within their devices and their
software application hoping the utility will keep their hardware
because they do not want to go through another process of
learning another vendors protocol or application. A proprietary
protocol encourages the utility to implement a single-vendor
system instead of implementing the best equipment available
on the market to do the job required regardless of its vendor. To
implement a highly reliable and secure system, the utility needs
to encourage multi-vendor systems. Multi-vendor systems will
force vendors to move to open standard applications which will
promote cost savings, personnel safety and system reliability.
The second case study experienced a different problem with
hardware vendors. With new functionalities being introduced to
the industry every year, its very easy for hardware vendors to
implement these new features in new hardware and not old
hardware; thereby enticing utilities to upgrade their relays and
meters. However, depending on the functionality, it can be
implemented somewhere else. Utilities need to analyze their
current system and determine if certain functionalities can be
implemented at a higher level which can be pushed to all their
hardware to reduce implementation cost as well as labor cost.
As stated in this case study, a simple hotfix to an upper level
system that was communicating to all the relays saved this
specific utility a large amount of money and labor.
Ultimately, the main goal for any utility is to make their
system more safe, reliable and cost effective. Some utilities lack
internal expertise or teams that understand IT technologies. For
some of these utilities, they are turning towards vendors to bring
that level of expertise and knowledge to secure and manage
their IEDs by bringing in proven IT technologies. Utilities need
to ensure vendors recommendations are objective and cost
effective and simply not recommendations that will require
more purchasing of hardware.
V. REFERENCES
Standards:
[1] North American Electric Reliability Corporation [Online].
Available:
http://www.nerc.com/pa/Stand/Pages/CIPStandards.aspx