You are on page 1of 7

Power communication networks

Migrating electrical
power network SCADA
systems to TCPAP and
Ethernet networking
Traditional SCADA systemsfor electricity utilities rely on data transmission over
fixed analogue circuits and modems. This method has been satisfactory over the
years but it is becoming obsolete and unsuited to todays requirements. The use of
TCPIIP technology can overcome the limitations of analogue communications,
allow the network to be moreflexible in terms of expansion and reconfiguration,
and have higher bandwidth potential whilst retaining the qualities essential for
SCADA operations.

by Kwok-Hong Mak and Barry Holland

S
audi Electricity Company-Eastern Control centre: The location of the
Region Branch (SEC-ERB) supplies SCADA master station, which requires a
electrical energy to business and telecommunication service to transport and
domestic customers in the Eastern deliver real-time data on the power network.
Region of the Kingdom of Saudi Arabia. The Functionality: SCADA systems require a
company has about 7000 employees and direct link from the master station to every
generates 8000 MW to 0.5 million customers. remote terminal unit (RTU). This link is
Parsons Brinckerhoff Ltd. (PB) was often provided by a radial branching
commissioned by SEC-ERB in February 2001 communication network with the hub at the
to carry out a technical review of its existing control centre.
supervisory control and data acquisition Connectivity: Most electricity utilities have
(SCADA) communication network and to all their grid substations and power plants
identify new communication concepu, that are served by RTUs. Secure redundant data
more flexible in terms of expansion and communication routing into the control
reconfiguration, and have higher bandwidth centre is often required.
potential whilst retaining the qualities essential Response time: Response times for data
for SCADA operations. collection are usually in the order of 1-55.
Five seconds is typically the maximum delay
Overview of communication networking before the information is presented at the
requirements control centre.
The main function of a SCADA system is to Availability: Because of the operational
provide accurate and up-to-date information nature of the data, a SCADA system should
on an electrical power network and to allow be continuously available and is often
system operators to respond to all power self-monitoring. Typical systems require
network conditions. In order to support availability of at least 99.995% in the
SCADA data transmission, a communication communication links between master
network is essential. SCADA communication station and RTUs.
networking requirements typically include: Bandwidth: SCADA systems transmit at low

POWER ENGINEERING JOURNAL DECEMBER 2002 305


Power communication networks

data rates that may range from 75biUs to versatile communication medium due to its
2400biUs. high-bandwidth capability and immunity
Environmental: Communication equipment from electromagnetic interference. The use
should he immune to severe electromagnetic of optical fibre as a digital communication
disturbances. Fibre-optic links are often medium has, become commonplace in most
specified as the only medium that will deliver electricity utilities as the benefits of
the required noise-immune bandwidth. implementing fibre-optic communication
systems and decreasing costs have become
In addition to the SCADA network, electricity more apparent.
utilities are likely to use a data network that is TCPIIP networking: Transmission control
based on officelocal area networks (LANs) and protocoVInternet protocol (TCPAP) suite
a corporate wide area network (WAN) for non and associated technologies have become
SCADA applications such as: the de facto world standard for data
networking. They arc used to deliver the
office automation and electronic messaging Internet and corporate intranets using
(e-mail) connectionless networking. TCPllP has not
distributed computer applications been ideally suited to the transport of
access to mainframe computers SCADA traffic because transmission times
intranet and Internet services. can vary and performance under extreme
conditions may not be predictable. However,
In most ulilities, corporate data network a great deal of development work has taken
services are provided by an information place to produce quality of service (QoS)
technology (IT) department over links pro- standards that will guarantee delivery of
vided by private and public telecommunication connection-oriented traffic over TCPllP
networks. Administrative data traffic requires a networks.
digital, high-speed backbone network that Emergence of QoS over TCPIIP: The technical
connects into most of the utilities major obstacle preventing the implementation of
offices. Many power system operators also SCADA on TCPAP networks has been the
operate private fixed and mobile voice absence of a standardised method for
communication networks. providing quality of service guarantees over
TCPAP data networks. QoS standards are
What is driving the need for change? central to the successful provision of TCPAP
Traditional SCADA networking is based o n service to networked applications that have
fixed voice grade circuits and modems thus differing requirements. For example,
guaranteeing that a communication path is SCADA data traffic requires very low and
there when required and that transmission consistent transmission delays but limited
delay and variations are very small. This bandwidth administrative data and e-mails
technology is becoming obsolete and unsuited require higher bandwidth but are far less
to the developing needs of todayk power critical of time delays and variations. QoS
system operations. technologies are designed to enable a
In recent years, telecommunication systems network to provide transport services
have undergone radical change prompted appropriate to the needs of each application.
mainly by the desire to increase system The lnternet Engineering Task Force
performance. New technologies are providing (IETF-a body which is responsible for
huge increases in performance at lower and developing specifications required for
lower unit cost. At the same time, deregulation interoperable implementation of the
and privatisation of the electricity industry Internet) has studied options for providing
coupled with the liheralisation of QoS on TCPAP networks. Two schemes are
the telecommunication market have imposed emerging- Diffserve and Intserve. It should
new requirements on the electrical power be noted that the two schemes are not
communication network. Key drivers for mutually exclusive:
change of traditional SCADA networking
include the following: (i) Diffserve (Differentiated Services RFC 2474)
is characterised by the [act that it is the network
Opticalfibre installations: Optical-fibre tech- that controls the quality of service.
nology is regarded as the most powerful and Differentiated services use multi-protocol label

306 POWER E N G I N E E R I N G JOURNAL DECEMBER 2002


Power communication networks

switching (MPLS) to deliver QoS. It proposes substation automation and remote manage-
three classes of data transfer: expedited, assured ment. The support of analogue modems is
and best efforts forwarding. MPLS Is a system becoming increasingly difficult. Their use is
that is supported by a number of major network already limited and will diminish further as the
equipment suppliers. Data frames are marked as world moves to digital communications.
near the network edge as possible with a Changing to TCPAP networking will enable
suitable label that indicates priority. This the management of SCADA networks to be
marking is applied by a device known as a label integrated into a system common to the
edge router. The core network label switch corporate data network. Reconfigurations will
routers can then treat the frames appropriately. he simplified to keyboard commands rather
MPLS can be used for setting u p virtual private than rewiring at multiple points. Bandwidth
networks (VPNs). VPNs enable the separation can be allocated as required and RTUs
of differing data traffic types on a single physical themselves remotely managed. The advantages
data network. Separation may be required for of TCPflP networking include:
security, management or billing purposes.
worldwide adoption (e.g. the Internet)
(ii) Intserve (Integrated Services RFC 1633) very well developed hardware and software
allows the end system or application to request market
a QoS from the network using resource simplicity and choice of application layer
reservation protocol (RSVP). The network protocols
must then set up facilities to provide the -,
inherent resilience of the IP routing concept
requested QoS. Intserve is an end system based strong network management, including
service. It proposes three levels of service: remote control and monitoring.
guaranteed, controlled and best efforts. In
practice, RSVP has been seen to be slow in Using TCPAP and the commonly associated
setting up connections and it may not scale Ethernet technology will give power system
well. RSVP gives control to end system operators access to a wide range of standards-
computers and, hence, can lead to many based inexpensive hardware and a large pool of
claimants for the highest priority trained staff.
TCPAP has not been widely introduced into
Whether using Diffserve or Intserve, QoS wide area SCADA communication systems to
cannot be guaranteed on Ethernet LANs which date. This is partly due to the replacement cycle
provide the last few metres of the connection to but also because of the non-deterministic
the SCADA measuring points and master nature of Ethernet and TCPAP communi-
systems. With 10 or 100MbiVs available on cations. The development and implementation
each Ethernet and SCADAS low bandwidth of QoS standards removes the risks associated
requirement this will not be a problem. with the non-deterministic nature of the
underlying connectionless networking. As
Why move SCADA networking IO TCPAP? TCPAP and Ethernet support is becoming
The TCPAP networking protocol mite was available from SCADA equipment manufac-
originally devised to support military research turers, the use of wide area TCPAP networking
projects in the USA. It went on to become the from master stations to RTUs is practicable.
standard used by the Internet. Partly due to its SCADA applications have historically used a
simplicity and its rapid standardisation very wide variety of application level protocols.
procedures TCPflP has become the de facto Two standards are emerging which are both
internacional standard. designed to operate successfully with TCPAP
Conventional SCADA network designs rely and Ethernet. They are DNP3 (Distributed
on the predictable nature of connection- Network Protocol 3) and IEC 60870-5-104.
oriented services using fixed audio bandwidth DNP3 is gaining acceptance in the North
links, analogue modems and specific protocols. American market whilst IEC 60870-5-104 is
Setting up and maintaining these network. calls favoured in Europe.
for specialised shills. Reconfigurations involve
hardware rewiring, are time consuming and Migration to a single integrated network
costly Bandwidth is limited to 3 kHz, which is TCPAP (with QoS) networking presents the
adequate for current RTUs hut potentially opportunity to migrate to a single network
limiting as businesses move towards the use of for both operational and non-operational

POWER ENGINEERING JOURNAL DECEMBER 2002 307


Power communication networks

1 Integrated TCPIIP
communication network

digital TCPilP
Communication network Communication

communication

requirements. Applications will include some encing and the use of remote video for training
or all of the following: purposes can he expected to grow in
importance.
Business data: The exchange of files and Voice communications: Migration of voice
e-mails between office locations for services onto the TCP/fP network can be
administrative purposes and access to achieved in stages:
mainframe computing, corporate intranet and (i) PABX (private automatic branch exchange)
the Internet. to PABX connection over the IP network.
SCADA data: SCADA equipment can he (ii) 1P telephony
connected to suitable L A N segments that are
linked by the corporate WAN (wide area Full 1P telephony is the stage at which the
network). A speed of 100Mbit/s is recom- telephone instrument is plugged into the LAN or
mended regardless of data requirements as the replaced by audio devices in the PC (personal
costs for 10 and 10OMbitls are only marginally computer). Deployment requires a complex set
different. Having made the Ethernet to up of call managen, 1P telephones and QoS, but
Ethernet TCPAP connection across the net- it promises to remove the need lor separate
work, it will be necessary to engineer adequate PABXs. Electricity utilities that have invested in
performance for the connection using QoS. PABX systems need not rush into IP telephony
QoS cannot he guaranteed on the Ethernet They should consider moving to IP telephony
LAN, so care should be taken not to allocate only as their installed PABX systems become due
other services o n the SCADA LAN% for replacement.
Video transmission: The use of closed-circuit Substation automation: Electricity utilities have
television (CCTV) for remote security moni- been considering a totally automation approach
toring of substations and power plants is being to substation integration with networking and
employed by many electricity utilities around intelligent electronic devices (IEDs) to manage
the world. Two options for putting these their power network. Substation automation is
services on the TCPflP network are simple likely to he closely integrated with SCADA and
Webcams or a full video monitoring service protection systems. Communication network
with remote access controls. Video confer- services will be vital to the integration of data

308 POWER ENGINEERING JOURNAL DECEMBER 2002


Power communication networks

acquisition, control and protection. TCP/IP mode) or gigabit Ethernet switches, routers
networking is ideally suited to the prolision of and the interconnecting data links. The core
appropriate networking services for this new network will transport data between points
r
application that will require high performance on the distribution layer. Full diversity of
and reliability routes between core layer switches must he
provided and data link speeds should be at
Electricity utilities may wish to develop TCP/IP least E l (2-048MbiUs). Core network
networking capability, such as shown in Fig. 1 , devices will be MPLS label switch routers.
to support their power system operation and (ii) Distribution layer: The distribution layer
administrative functions. consists of routers that are designed to
carry traffic in and out of the core network
Technical proposal for a TCPAP and in a resilient manner. The distribution layer
Ethernet networking routers will normally he located at major
Network topology: To make the best use of sites. Connection from the core to the
existing networking hardware and available distribution layer will be by digital data
bandwidth, and to meet the high availability links at speeds up to E l . Each distribution
requirements of a SCADA system, we have layer location will he connected to two
suggested that SEC-ERB follows a network separate core switches. Distribution layer
topology based on a layered approach with an routers will perform the label edge router
appropriate level of circuit and equipment function of MPLS. Most RTUs will be
diversity The network should consist of four connected to the distribution layer.
layers: (i) core, (ii) distribution, (iii) access and (iii) Access layer: The access layer is where
(iv) users, as shown in Fig. 2. most end user LANs are connected to the
network via access routers. Access routers
(i) Core network layer: The core network layer will be connected to the distribution layer ~ ~
ne,.,ork ~ ~ -
comprises A I M (asynchronous transfer using suitable network links; these will he topology

core I distribution I access

POWER ENGINEERING JOURNAL DECEMBER 2002 309


Power communication networks

64kbiVs but where video services are usually called a 'firewall' and will often
envisaged an E l circuit may be more incorporate a 'demilitansed zone' (DMZ).
appropriate. The access routers will be The firewall is used to implement policies on
situated in substation and office locations. which data can enter or leave the network.
(iv) Users: Users' LANs (such as PCs, file and SCADA traffic is effectively segregated
print servers) will be connected to the from other networked applications when it is
access layer directly or via local LAN carried on analogue circuits. When all traffic
switches. SCADA RTUs will use LANs that is carried on TCPAP, it will he necessary to
are directly connected to the distribution provide specific security controls to prevent
layer. SCADA master stations will have unauthorised staff from accessing data. The
diverse connections to two distribution simplest method for achieving segregation is
layer devices. to use the facilities of MPLS to build a VPN
for SCADA.
High availability design. High availability TCPAP addressing. The proposed change to
networking service can he provided to TCPAP networking for SCADA systems and
SCADA services. To give the best availability, possibly voice and video may have an impact
we have suggested that SEC-ERB follows on an organisation's TCPAP addressing plan.
design guidelines that minimise the effect The most flexible and secure implementation
of a single node or circuit outage. The can be achieved by using a private address
following guidelines were recommended: space as defined in RFC 1918. Use of the
Class A address specified there (10.0.0.0)
(i) Each core node should he connected to at will give the greatest address range. The use
least two others. of a private address space with a secure
(ii) Network connections from a core node gateway to any external connection will
should he independently routed from other enable operators to retain the use of IP V4
connections to the same node. and to defer changing to IP V6 until there are
(iii) Each distribution layer point should have clear benefits in doing so.
two independent routers and two indepen- Power system operators would be well
dent communication links back to the core advised to set aside blocks of the available
layer. Users LANs should not normally be (10.0.0.0) address space for each of the pro-
connected to distribution layer nodes. posed distribution layer locations. Addressing
(iv) SCADA master stations should be connected can then be allocated to the appropriate access
to the network via two independent diwihu- layer in contiguous groups. Whilst this is not
tion routers and two independent commun- essential, it will simplify the network
ication links to two different core nodes. summarisation tables in the routers and also
(v) SCADA measuring points should he simplify network management. We also
connected to two independent RTUs, each recommend that a TCP port is reserved for
linked to an independent distribution SCADA use to further simplify network
layer router with an independent management and commissioning.
communication link connected to different The use of MPLS VPNs provides a suitable
core nodes. It may be acceptable to connect method for adding 'other' users' to the
lower priority RTUs to access layer routers. network infrastructure and segregating their
traffic from each other. 'Other' users
QoS recommendation: We have recommended addressing plans can be accommodated
to SEC-ERB that it adopts the internationally without change as long as they remain within
standard based Diffserve and MPLS as the their own MPLS VPN.
method of providing QoS on its digital 8 Network management: A number of network

communications network. Many networks management systems are commercially


will be using proprietary inter-router available, including HP Openview, Aprisma
protocols (e.g. EIGRP). MPLS requires the Spectrum, IBM Tivoli, Nortel Optivity, CA
use of open shortest path first (OSPF), a Unicentre and several Cisco products. All
recognised international standard. these systems use simple network
Networks security: To manage the threat from management protocol (SNMP), the most
external sources most (if not all) operators of suitable protocol for managing TCPAP
private TCPAP networks use a secure gateway networks. Operators should consider using
to manage the connection. This gateway is SNMP to manage RTUs and manufacturers

310 POWER ENGINEERING JOURNAL DECEMBER 2002


Power communication networks

should be requested to supply suitable using MPLS VPNs in the short term and through
management information bases (MIBs) the development of full traffic engineering
The TCPflP network will be carrying services in the future. Entry into this field will
operationally critical SCADA data, so depend on market conditions.
electricity utilities must ensure that network
management support is available 24 hours a Conclusions
day, seven days a week. There is a significant benefit in migrating
Testing and staged implementation of digital SCADA systems to TCPAP and Ethernet
network: The implementation of QoS on networking. Many power system operators
TCPIIP networks is relatively new in the already have the technical infrastructure,
electricity industry It requires a well planned capability and capacity to develop a successful
and systematic sequence of testing, migration of SCADA networking to TCPllP In
installation and commissioning to migrate addition, TCPAP networking with QoS has the
operational and business requirements from technical capability to support other power
existing systems. This sequence is essential system operational requirements as well as
to maintaining continuity of service and business administration.
ensuring that the work has no adverse effect Electrical protection systems have critical
on the power system operators' business requirements regarding reliability and
administration or power system operation. performance of communication links, so we do
Electricity utilities are encouraged to test not recommend transfer of teleprotection
the performance of their detailed design signalling to TCPAP networks until such time
proposals to verify that the QoS config- that connectionless networks have matured
uration can deliver the required performance and can guarantee the required service for
characteristics and that they have full electrical protection systems at all times.
confidence in the concept before it is A technical report and a presentation were
implemented network wide. To achieve such given to SEC-ERB detailing our proposal for
confidence requires a proving period of SCADA network migration. SEC-ERB concurs
offline network testing and a staged with our recommendations and is currently
migration implementation. making arrangements to proceed with the first
phase of SCADA migration to TCPnP and
Potential business benefits of implementing Ethernet networking.
a single network Our review and recommendations were
Migration to a unified TCPllP network capable specific for an electricity utility, but the
of supporting all of a power system operators' principles proposed for migration to TCPllP
data, voice and video requirements will and Ethemet networking are equally applicable
simplify network management and enable to other industries employing SCADA systems.
economies of scale to be applied to the
communication network. Acknowledgments
A single network for all requirements will The authors wish to acknowledge the help of
remove the need for multiple network their colleagues in the preparation of this
management tools and specialised training and article and they thank the management of
knowledge by networking staff. It also allows SEC-ERB and Parsons Brinckerhoff Ltd. for
the standardisation of hardware and software. permission to publish this article.
Moving to an all TCPIIP network will enable
0 IEE: 2002
electricity utilities to select equipment from a Kwuk~HangMak is a senior engineer with Parsons
very wide range of compatible types. There is Brinckerhofl Ltd.. Amber Court. William Arms~rong
potential to use business leverage to obtain best Drive. Newcasrle upon Tyne NE4 7YQ, UK. He was Lhe
possible prices for this widely used technology project managerilead engineer lor the technical review
of SEC-ERBs SCADA communicarion migralion project.
The sale of network capacity in the form of
He has experience of many projects in the electricity
spare fibres has been available for some time, hut industry, ranging from studies to commissioning and
this represents the crudest and potentially least training of relecommunicaiion syrlems. Barry Holland
profitable option. Utilities will need to consider is a senior engineer Parsons Brinckerhoff Ltd.,
selling managed data services to third parties in Westbrook Mills. Godalming, Surrey GU7 2AZ. UK. He
was the project engineer for the technical review of
order to maximise profitability Demand for this
SEC~ERBsSCADA communication migration project.
type of service is limited almost exclusively to He has experience of TCPnP network design and
TCPAP networking. Resale can best be achieved operation that was gained in civil aviation.

POWER ENGINEERING JOURNAL DECEMBER 2002 31 1

You might also like