Khaparkheda

Thermal Power Station Automated PLC Control System (APCS):

Software Development & Failure Analysis

Introduction & Background
This paper is aimed at providing an account of development cycle of software called
APCS (Automated PLC Control System) at a thermal power generation plant in
Central India, eventual failure in deployment of the system and analysis of failure.
The information is obtained through interacting with two members who were part
of the program at aforementioned plant. It is confirmed that the shared information
is NOT classified business data or trade secrets. Khaparkheda Thermal Power
Station is coal based power plant run by Maharashtra State Power Generation
Company a semi autonomous electricity generation body under state Government
of Maharashtra, India. The plant has been operational since 1989 and has capacity to
generate 1340 MW of power.

Thermal Power Plant is typically located on a site of coal mines where huge feeders
feed pulverized coal to boiler, which converts steam that in turns runs the high
pressure, intermediate pressure and low pressure turbines. These turbines are
connected to generators that generate electricity that can be distributed further
through AC lines. Khaparkheda Thermal Power Plant, here onwards referred to as
KTPS, had a disjoint system of PLC (Programmable Logic Controllers) for each part
of the plant. In 1999, the management board decided to use modernize the plant by
developing automation software named APCS (Automated PLC Control System) that
could monitor various different units of the plant, provide controls to electrical &
mechanical engineers and report in case of emergencies. This project was initially
scheduled for 4 years. A team of 7 Computer Engineers, 5 Electrical Engineers, about
20 technicians and 5 rotating interns were allocated for this project. However, it
didnt meet its phase one target and overshot the budget, initially set at 60 million
rupees (~1.2million USD, 2004) and was ultimately abandoned in 2004.


Project Timeline and Phases

Goals of APCS
In this inter-disciplinary project, embedded C control software was written in
waterfall model for hardware that was partially new and mostly legacy. The initial
requirements dictated that the first two units of the plant patch the old software
while the deployment of third one use latest software. This needed parallel
development of same functionality of two separate platforms. Moreover, the need to
provide communication between the two systems posed another challenge. The
continuous 24x7 operation of thermal power plant, which cannot be turned off,
posed another set of software testing challenges. The basic goals of the software
were as below

1. Provide automated control of PLCs to 8000+ technicians on the plant
2. Monitor the throughput of the plant over entire year
3. Provide quantified data for the improvement of efficiency and reduction of
waste
4. Increase worker safety using automated controls of the software that could
be triggered remotely
5. Eliminate manual labor by improving communication between various
modules of the plant

Root cause analysis of the project failures

Scope creep
This was the biggest problem plaguing the project. As the new unit of 210MW was
being installed while the software was still being developed, the scope of project
was not finite and kept changing with new information. The management wanted to
include as many new features as possible without regard for reliability and
implementation of the software. The lack of computer science background among
senior management also hurt in deciding on prioritizing between features.

Code Bloat
The idea that the code can be simultaneously being written for both new and legacy
platforms caused problem with code management. The two-paced approach is not a
very good idea in any software development project where two separate modules
are supposed to interact with each other. One module development often turns into
a bottleneck by sheer difference in its goals although it may be mirroring the
functionality of other modules.

Development model mismatch
The plant in earlier projects has used single cycle waterfall model approach and
they preferred to use it for this project as well. In hindsight, this was a mistake too
as the inherent slow pace of waterfall model was difficult to course correct when
better tools were available in India starting from 2000s computer science field
boom. A partial agile method within overall waterfall might have served the project
better.

Poor change control
This was probably due to lack of foresight among senior management. Unable to
identify the apt change control software like perforce which was available at the
time of development turned fatal. Management preferred sticking to in-house
version of software version control software, which lacked powerful features for
collaborative and cautious development. This in turn contributed to more bugs and
extra time in debugging.

Cost Overrun
This was the mostly consequence of delay in completing the project. Being part of
semi-government body, the financial constraints to complete the project were
relatively low.

Obsolescence of technology
The ultimate death knell of the project turned out to be the fact during 5 years of
project development, the goals set out at the start could not be met and underlying
technology itself lied at the verge of obsolescence.

Conclusion

This software project was marred with various avoidable problems. Following
solutions could have helped resolve many of the issues

Inclusion of technical personnel in defining the requirements of the project instead
of management only approach would have helped chalk out the goals for the project.
Hiring third party experts and investment in gathering 10 year long vision for the
software would have helped too. The users of the software i.e. 8000+ technicians of
the plant would have benefited great deal with easy to use software for maintaining
the complex operation that running a thermo power plant is.