Information and Software Technology 48 (2006) 566577

www.elsevier.com/locate/infsof

The use and effects of an electronic process guide and experience

repository: a longitudinal study
Felicia Kurniawatia,b,*, Ross Jefferya,b,1
b

a
National ICT Australia Ltd, Locked Bag 9013, Alexandria NSW 1435, Australia
School of Computer Science and Engineering, K17, University of New South Wales, Sydney, NSW 2052, Australia

Received 27 January 2005; revised 2 June 2005; accepted 10 June 2005

Available online 19 July 2005

Abstract
This paper presents a consolidated view of two evaluations on the use of an electronic process guide and experience repository within a
small software development company. The use and effects of the tool were studied over a period of one and a half years, first for 6 months and
then 1 year after its installation, for another 5 months. The tool was used regularly and in a consistent manner in both studies but declining
usage was observed in the second study. The repository remained used to retrieve mostly examples and templates but the number of retrievals
of anecdotal experiences, such as lessons learned had noticeably increased. Similar benefits such as time saving and improved documentation
quality were observed in both studies, with additional benefits in the second study like improved project planning and cost estimation, and
easier negotiation and traceability of altered or new system requirements with clients. The initial load that users experienced in learning to
use the tool was not observed in the second study. The results show that tangible benefits can be realised quickly and continued to be
experienced, leading to users having higher morale and more confidence in executing their tasks.
q 2005 Elsevier B.V. All rights reserved.
Keywords: Software process improvement; Electronic process guide; Experience repository; Industry case study

1. Introduction
The introduction of a process guide provides a one-off
improvement opportunity through the benefits of declaring a
defined, systematic and repeatable approach to software
development. An electronic process guide offers several
advantages over a printed process handbook, including easy
access over the web for the most up-to-date version of the
guide, electronic search facilities and hyper-navigation to
ease browsing information. Experience management refers
to approaches to structure and store reusable experiences. It
aims to reduce the overhead of searching for information
that can support software development activities. Experience management also appears to be more effective when it
* Corresponding author. Address: School of Computer Science and
Engineering, K17, University of New South Wales, Sydney, NSW 2052,
Australia. Tel.: C61 2 8374 5516; fax: C61 2 8374 5520.
E-mail addresses: feliciak@cse.unsw.edu.au (F. Kurniawati), rossj@
cse.unsw.edu.au (R. Jeffery).
1
Tel.: C61 2 8374 5516.

0950-5849/$ - see front matter q 2005 Elsevier B.V. All rights reserved.
doi:10.1016/j.infsof.2005.06.002

is process centric, as reported in [25,31,37]. Thus, the two

concepts have a symbiotic relationship in that the process
guide is more useful when supported by experiences and the
experience base is more useful when it is process focused.
The electronic process guide/experience repository (EPG/
ER) is an implementation of this relationship and supports
users through provision of guidance that is supplemented by
task-specific experiences. The EPG/ER installed in Allette
Systems consists of a customised process model based on
the ISO 12207 Software Life Cycle Processes [13] standard
and an experience base that can be populated with
experiences that are structured according to the processes
in the process model. It was generated using WAGNER [5]
and then installed at the company. The company added their
own experiences into the repository and used it in a variety
of projects.
Allette Systems has around 15 staff members and mainly
implements and integrates publishing systems for clients to
create, manage and disseminate information [9]. It also does
web development and conversion of paper information to
electronic format. The applications built are often written in
the Java programming language. They also often use
eXtensible Stylesheet Language: Transformation (XSLT)

F. Kurniawati, R. Jeffery / Information and Software Technology 48 (2006) 566577

to process data in the eXtensible Markup Language (XML)

and Standard Generalised Markup Language (SGML)
formats. Allette Systems is suitable for this study because
it is a small to medium enterprise (SME) typical of the vast
bulk of the Australian IT industry. Hence, the results of this
study would be much more relevant and useful for the
Australian IT industry. The size of the projects that Allette
Systems are involved in varies from 1 week to 3 months
long. Each project typically has one to three development
staff members with a project manager. Sometimes, a project
manager may also be involved with development work.
This paper presents findings from two studies over a
period of one and a half years on the use of an EPG/ER in a
small software development company based in Sydney,
Australia. Tool usage and users feedback were first
monitored for a period of 26 weeks, and then a year later
for a period of 21 weeks. The studies determined how the
EPG/ER was used, the impacts that users experienced from
using the tool, and ways that the tool could be improved.
Each study has been separately published in [6, 7] and this
article is the first to provide a consolidated view of the
results from both studies and the relationships between
them. By combining the results from both studies, a more
thorough understanding of the use and effects of an EPG/ER
can be gained. It was found that in both studies, the tool was
used regularly and in a consistent manner and that benefits
such as improved project planning, project documentation
and increased morale were experienced from using the tool.
However, declining usage of the tool was observed based on
the slower rate of increase in the number of experience
entries and the lower number of hours of usage recorded.
Nonetheless, users retained a positive attitude towards the
tool. Retrievals of anecdotal experiences in the ER, such as
lessons learned have also noticeably increased in the second
study, suggesting that users have learned to reuse more
sophisticated forms of experiences. Users also no longer
reported the burden of the learning load in the second study.
The initial study showed that benefits were derived quickly
while the latter validated that it provided long-term benefits,
demonstrating that the usefulness of the tool was not limited
to only a short period of time.
The next section presents related work on electronic
process guides and experience repositories. Section 3 explains
the design and implementation of the EPG/ER and Section 4
states the research questions and describes the method used to
evaluate the EPG/ER. Section 5 presents the results from the
studies while Section 6 discusses the implications of these
results. Finally, Section 7 concludes the paper.

2. Background
2.1. Electronic process guide (EPG)
Kellner et al. [1] defined a process guide as a reference
document for a particular process. Paper-based process

567

guides are commonly used by larger organisations to

communicate their company-specific processes. These are
usually large, bulky, complex and expensive to produce and
maintain. Although organisations have started to distribute
their process handbooks in electronic formats such as pdf,
Postscript, or even html, these documents often still do not
take full advantage of web-based concepts such as hypernavigation. Based on his experience with paper-based
guides and industrial needs in the software engineering
domain, Kellner et al. defined the basic requirements and
design principles for an EPG:
R1 To provide a description of the process (in terms of a
process language (preferably graphical) describing
activities, artefacts, roles, agents, and resources as well
as the major relationships between them, i.e. product
flow, role assignment and decomposition,
R2 To make maximum use of diagrams, tables and
narrative to provide an effective user interface,
R3 To make extensive and effective use of hyperlinks to
support flexible navigation through the information
contained in the guide,
R4 To make it easy to access desired information (e.g.
frequently used information) very quickly,
R5 To facilitate orientation in the process by using
familiar structures for each page of the electronic
guide,
R6 To not overwhelm the user with too many overlapping
windows,
R7 To be implemented using commercial off-the-shelf
technology for www applications.
In the software engineering domain, several EPGs of
proprietary processes like RUP [13] and Mentor [14] have
been widely available for some time and tools such as
Spearmint [15], ARIS [16] and Adonis [17] have been
developed to allow quick and economical generation of
EPGs for tailored and/or company specific processes. More
recently, Dingsoyr and Moe [3] proposed the use of process
workshops as a tool to develop process guides for software
companies and presented their experience in implementing
this method in a Norwegian satellite software company.
Some studies of the use of EPGs are as follows. BeckerKornstaedt et al. [2] reported positive user feedback of an
EPG but did not provide further usage profiles or describe
how people utilise EPGs. This study motivated the
development of the Spearmint/EPG toolset. Scott et al.
[40] presented an in-depth analysis on the usage of
Spearmint EPGs and user opinions within Allette Systems,
but only considered data collected over a period of 8
months. Moe et al. [41] reported experience on the use of
process guides as software process improvement in a small
Norwegian company. It presented encouraging user feedback on an EPG after it had been in use for over a year but
did not provide actual usage patterns.

568

F. Kurniawati, R. Jeffery / Information and Software Technology 48 (2006) 566577

2.2. Experience repository (ER)

An application of knowledge management in the software engineering domain was the experience factory work
of Basili et al. [18] within NASA Goddards SEL was an
organisational approach for a large organisation with a high
proportion of quantitative data and a budget that allowed
infrastructure development. A smaller scale initiative was
Schneiders [19] in the Daimler-Chrysler research center
project, which collected mostly qualitative data by the use
of interviews, workshops and feedback forms. However,
there is no known data on the use of the created experience
base in both cases. Much research has also been done on the
development of tools to support experience storage and
retrieval such as [2023]. But again, not much data on the
usage effects of these tools are presented.
Other publications on organisations that have used
software engineering repositories present qualitative descriptions at high level of abstractions of the setup of and lessons
leaned from an experience repository project. For example,
Conradi and Dingsoyr [26] provided notes on experience
repositories in four organisations describing the type of
information stored, the technology used and high-level
descriptions of the companies experiences with the information stored. Lindvall et al. [27] outlined three case studies
describing some high level lessons learned and Brossler [28]
described experience usage in software development and
management, and some observed intangible benefits.
Maurer and Holzs [36] work on process centred
knowledge management inspired the implementation of
the experience repository as an extension to the EPG used in
the studies. In their approach, knowledge is linked to
development processes and thus processes are used as the
primary means of indexing information. The effectiveness
of structuring experience around the task at hand is also
supported by Ye and Fischer [24] who designed,
implemented and evaluated a system which delivered
information relevant to the task at hand and personalised
to the background knowledge of an individual developer.
Schenider and Hunnius [30] also identified user guidance as
one of the main quality aspects that determined the chances
for success of an experience repository.

3. Design and implementation of the EPG/ER

As part of an ongoing SPI effort in the Allette Systems
[4], a web-based EPG consisting of a process model
based on the ISO 12207 standard [12] was installed in
the company. The single process model consists of
the companys project management and software development processes, customised to suit small to medium sized
software development work that is typical for the company.
The ER was then implemented using the PageSeeder [8]
application as an extension to the companys EPG.
Although other open source content management tools

have been considered as an ER for the EPG, since

PageSeeder is a product that was developed within Allette
Systems, it was a logical choice for this organization to use
it as the experience repository component as the company
already had the skills and expertise to use and maintain
PageSeeder. We described our experience in investigating
the appropriateness of TikiWiki version 1.7.4 [33] and
Typo3 version 3.6 [34] for their use as alternative
knowledge functions for the EPG/ER in [35]. Experiences
stored in the repository are of four types: checklists,
examples, templates and generic experiences. The generic
experiences category allow entries such as anecdotes,
lessons learned, code fragments and links to other
information considered to be useful by the users. The
experiences are structured according to the processes. The
resulting tool is the EPG/ER, which supports users by
providing process guidance that is supplemented by taskspecific experiences. In this way, the process guide becomes
more useful as it is supported by experiences and the
experience base is more useful as it is process focused.
The EPG/ER was automatically generated using the
WAGNER [5] framework, which presented several advantages. For a large and complex process, manually coding an
EPG can be a long and difficult task and the maintenance of
the EPG can become very costly as processes change. Using
the framework, the generator needs only to be written once
and the quality of the EPG is guaranteed for subsequent
EPGs. Moreover when the processes change, only the
process specification needs to be altered and a new EPG can
be generated quickly and economically.
Fig. 1 shows the WAGNER framework. Unified Modeling
Language (UML) activity diagrams are used to graphically
specify the processes. These are then exported into XML
Metadata Interchange (XMI) format. An XSLT translator is
then used to transform the exported XMI using EXtensible
Stylesheet Language (XSL) and JavaScript scripts that define
the layout and functionality of EPGs produced. The UML
modelling tool used in WAGNER is Enterprise Architect [31]
and the XSLT translator used is Xalan-Java [32].
Fig. 2 shows an example page of the EPG/ER. Each page of
the EPG/ER consists of two frames. The diagrammatic
overview on the left-hand side displays the UML diagram of
the process that supports browsing and hyper-navigation of
the process. The description section on the right-hand side
displays the description of the currently displayed entity
and experiences relevant to the activity. To navigate through
the process model, a user simply selects the activity
he or she is interested in. For example, when a user clicks
on the Requirements and Architecture activity image,
sub-activities of this activity will be displayed on the diagrammatic overview section and textual information
regarding the Requirements and Architecture activity will
be shown on the right-hand side.
After each study, the EPG/ER was improved according to
suggestions made by users. The main difference between the
layout of the original and latest version of EPG/ER is the

F. Kurniawati, R. Jeffery / Information and Software Technology 48 (2006) 566577

569

Fig. 1. WAGNER framework for automatic generation of EPG/ERs.

replacement of the navigation tree frame with a hyperlink at

the top of the description section, which provides access
to the navigation tree. This was done because the presence of
the navigation tree frame was found to be redundant given the
ability to navigate using the diagrammatic overview. It also
provided more space for viewing the processes and their
textual information, thus eliminating the overcrowded effect
felt with the original EPG/ER. Furthermore, the EPG/ER
now has a homepage that provides a search facility for the

ER. Fig. 3 shows an example of the homepage. It contains

links to the search facility, activity page of the EPGs and
quicklink pages that acts as interfaces for each type of
experience. Each quicklink page contains an alphabetically
ordered list of a given type of experience in a given EPG.
The architecture of the EPG/ER is shown in Fig. 4. The
EPG/ER homepage lies at the top level of the architecture
and provides a single point of access for multiple EPG/ERs
installed in the company via a collapsible tree menu. This

Fig. 2. Example layout of an EPG/ER.

570

F. Kurniawati, R. Jeffery / Information and Software Technology 48 (2006) 566577

Fig. 3. Example of the EPG/ER homepage.

page allows a user to invoke the search facility and a

quicklink interface page. An entity page of a given EPG lies
on the second level. The description section of this page
contains information and links to ER entries that are

relevant to it. An experience entry of the ER lies on the

bottom level. Access to a given ER entry is possible through
the homepage or through viewing its associated EPG
activity.

Fig. 4. Architecture of the EPG/ER.

F. Kurniawati, R. Jeffery / Information and Software Technology 48 (2006) 566577

4. EPG/ER evaluation research method

The method we used does not attempt to separate the
EPG/ER and the process it contains but instead evaluates
them as a single piece of technology. It was designed to use
a mixture of qualitative and quantitative data to obtain
answers to the following research questions:
RQ1
How is the EPG/ER used?
RQ2
Has the way the EPG/ER is used changed over time?
RQ3
What are the effects of using the EPG/ER?
RQ4
Have the effects of using the EPG/ER changed over time?

4.1. Data collection

Four sources of data were used in both studies: EPG/ER
and PageSeeder server logs, user surveys and timesheets.
Although technology acceptance model (TAM) [10]
surveys were used in the first study, it was discovered
after that study, that they might not provide reliable source
of information to infer observations on the level of
acceptance. This is supported by [11], which considers the
application of TAM in explaining intranet usage in two
organisations and [25], which included a discussion of TAM
in the evaluation of a cost estimation model. Thus, TAM
surveys were not conducted in the second study and results
from the TAM surveys in the first study will not be
presented or discussed here. An earlier publication [6]

571

provides the results from the TAM surveys conducted in the

first study.
Fig. 5 shows a timeline of the EPG/ER usage data
collection throughout the two studies: from 14/05/02 to
22/11/02 and from 06/05/03 to 29/09/03. The gap in the
middle of the graphs from 23/11/02 to 05/05/03 represents
the time between the end of the first study and the start of the
second study. In the first study, the use of the EPG/ER was
monitored over 6 months from the day of the training
session. EPG/ER server logs and timesheets were collected
continuously over this period. PageSeeder server logs were
collected at the end of the study. A predictive TAM survey
was conducted immediately after the training session and a
follow-up TAM survey was done at the end of the study.
User surveys were carried out at 1, 3 and 6 months into the
study. In the second study, EPG/ER server logs were
collected continuously over 5 months and PageSeeder
server logs were collected at the end of the study. User
surveys were carried out every 4 weeks beginning on the
first day of the study and timesheets were collected at the
end of the study.
Users attended a training session before the EPG/ER was
made generally available at the start of the first study.
During the training, users were introduced by a researcher to
the principles of process engineering and process improvement. They were also instructed on the use of the tool and
were told about the research project and the data that would
be collected. They were then shown an overview of the
processes and experiences before being instructed on their
use by representatives of the company. The EPG/ER was
intended for voluntary use, thus users were encouraged not
to alter their normal usage behaviour and were assured of
confidentiality. Furthermore, a separate task was set up on
the companys time recording system such that users could

Fig. 5. The timeline of data collection over days.

572

F. Kurniawati, R. Jeffery / Information and Software Technology 48 (2006) 566577

Table 1
Data sources
Source

Type

Analysis

EPG/ER server logs

PageSeeder server logs

Quantitative
Quantitative
Qualitative
Quantitative
Qualitative
Qualitative

EPG/ER page accesses; experiences retrieved from the ER

Time when experience is entered into the ER
Type of experience entered into the ER
Sum of hours spent on using the EPG/ER
Purpose of time spent on using the EPG/ER
Purpose of using the EPG/ER; Benefits and detriments of using the EPG/ER;
Improvements to the EPG/ER

Timesheets
User surveys

use the tool without suffering penalty to their assigned tasks.

After the training session, the EPG/ER was made available
for general use and the company was left to use it as they
desired. Twelve users who made up the development and
project management staff in the company were involved in
this study. The other three staff members were administrative staff.
The second study used an improved version of the
EPG/ER, which was made available for use the next day
after its installation. There were ten users included in this
study as the other two who were involved in the initial study
had left the company. Since these users were already
involved in the first study and the changes made to the
EPG/ER were not too significant, users did not need to go
through another training session. However, to further ensure
unbiased data collected, they were again encouraged not to
alter their normal usage behaviours and assured of
confidentiality.

user surveys revealed the purpose of using the EPG/ER.

To address RQ3 and RQ4, user surveys were analysed to
provide user reported benefits and detriments that were
experienced. It also provided user suggested improvements to the tool.
Server logs were checked as soon as they were received
from the company to ensure that they contain the data
required before analysis is done using simple Excel scripts.
Automatic filtering and interpretation extracted only
relevant entries and identified the pages that were accessed
at a given point of time in the study and the person
associated with the IP addresses. When needed, timesheet
records and user survey results were also clarified as soon
as they were received. Moreover, survey results were
verified for non-responses and any misunderstandings of
questions.

4.2. Data analysis

5. Results

Table 1 shows the types of these data sources and the

analysis that were applied. It shows that the collected data is
either quantitative or qualitative and is concerned with
actual and user reported usages, user reported effects and
user suggested EPG/ER improvements.
To address RQ1 and RQ2, the server logs, timesheets
and user surveys were analysed to provide actual and
user reported usage data of the EPG/ER. EPG/ER server
logs provided EPG/ER page accesses and experiences
retrieved from the ER. It recorded the time a given page
in the EPG/ER was being accessed and also the IP
address that was accessing the page. PageSeeder server
logs recorded the time and the type of experiences
entered into the ER. Timesheets provided the time spent
and purpose of the time spent on using the tool while

Table 2 shows the relationship between the research

questions and the data used to answer the questions. RQ1
and RQ2 are addressed in Section 5.15.5. RQ3 and RQ4
are addressed in Section 5.6.
5.1. Page accesses to the EPG/ER
Several observations from the studies indicate consistent
and continuing usage of the EPG/ER. These indicators of
EPG/ER usage are charged on a daily basis as opposed to a
weekly basis in our previous papers. The EPG logs revealed
Fig. 6, which shows the number of page accesses over the
time of both studies. It shows declining but continuing usage
of the EPG/ER.

Table 2
Research questions
Question

Evaluation

RQ1. How is the EPG/ER used

RQ2. Has the way the EPG/ER is used changed over time?

EPG/ER page accesses

Experiences retrieved from the ER; no. of PageSeeder entries; purpose of
using the EPG/ER; time spent on using the EPG/ER
User reported benefits and detriments

RQ3. What are the effects of using the EPG/ER?

RQ4. Have the effects of using the EPG/ER changed over time?

F. Kurniawati, R. Jeffery / Information and Software Technology 48 (2006) 566577

573

Fig. 6. EPG/ER page accesses over days.

5.2. Number of experience entries in the ER

The PageSeeder logs revealed the growth of the experience
repository during the periods of time studied. This is illustrated
in Fig. 7, showing a steady increase in the number of entries in
the repository. The steep climb to 96 at the beginning of the
first study is due to two experts seeding the experience
repository before it was made available for general use.
At the end of the first study, the repository was heavily
populated with code examples (83 of 276), followed by
document examples (44 of 276), templates (21 of 276), 19
(of 276) anecdotal experiences and 12 (of 326) checklists.
At the end of the second study, there were 113 (of 326) code
examples, 52 (of 326) document examples, 26 (of 326)

templates, 19 (of 326) anecdotal experiences and 12 (of 326)

checklists in the repository. The remaining were comments
made on existing entries in the repository.
5.3. Experiences retrieved from the ER
Table 3 shows the structure of experience repository
usage in both studies by listing the number of retrievals for
each experience type. Code examples in the repository were
the most extensively accessed experience type during the
first study. Although few anecdotal experiences and
checklists are in the repository, they still received a notable
amount of interest from users. It was not possible to identify
the most accessed experience in the repository, however

Fig. 7. Experience repository growth over days.

574

F. Kurniawati, R. Jeffery / Information and Software Technology 48 (2006) 566577

Table 3
Experience repository usage
Experience type

No. of retrievals
(study 1)

No. of retrievals
(study 2)

Examples
Templates
Checklists
Experiences
Total

167
71
16
36
290

118
50
3
57
228

the data collected from the first study indicated that the
examples interface page (similar to the examples
quicklink page for the current EPG/ER) was the second
most accessed page, after the homepage. The templates
interface page was third, experiences fifth and checklists
sixth.
Concrete artefacts appear still to be the type of entries
that are extensively used in the organisation in the second
study. Code examples remain the type of experience that
users find most beneficial to share and use, while templates
still received considerable interest. We also observed that
anecdotal experiences have not only continued to be a
popular type of experience entry in this organisation, but
also that the number of retrievals of this type of experience
was the only one that has increased in the second study.
5.4. Time spent on using the EPG/ER
The timesheets showed that users spent an average of 0.
208 h per week per person (65 h over 26 weeks and 12
users) using and contributing to the EPG/ER during the first
study and an average of 0.076 h per week (16 hours over 21
weeks and 10 users) during the second study. These
numbers are likely to be less than the actual time because
despite of users report in the regular user surveys of having
used the EPG/ER for small amounts of time, usages were
sometimes not recorded on the timesheets.

a learning platform for easy sharing of experiences and

problem solutions, which helps promote a culture of
sharing information. The explicitly defined processes and
checklists assisted project management tasks and the
conduct of the study had increased awareness of the need
to improve process practices within the company. Users also
reported that a noticeable amount of time and effort was
required to learn to use the tool and that the amount of time
needed to enter experiences can sometimes be an unwanted
distraction from immediate tasks.
The benefits experienced earlier were still reported in the
user surveys during the second study. They further added
statements that using the tool has improved project
planning, cost estimation and eased negotiation and
traceability of altered or new system requirements with
clients. This is due to the processes in a project being
explicitly represented and better provision of documentation
supports, such as templates and examples. A project
manager reported that using the documentation templates
also picked up errors, omissions and misunderstandings in
the clients functional specifications and thus, shares more
responsibility of project success with client. It also helped
increase awareness of milestones and force review of key
documents and schedules. These ongoing benefits appeared
to have provided, as said by a project manager overwhelming confidence in our ability to plan and execute the
project. The previous negative effects of using the tool were
not reported here, however, a user suggested that the tool
might not be worthwhile for small-scale projects of 2 days
work.

6. Discussion
Implications on RQ1 and RQ2 are addressed in Section
6.1. Implications on RQ3 and RQ4 are addressed in Section
6.2. User suggested improvements to the EPG/ER are
addressed in Section 6.3.

5.5. Purpose of using the EPG/ER

6.1. Implications on EPG/ER usage over time
In the earlier study, users reported that the EPG/ER is
used to assist management, programming and documentation tasks. Templates and other supporting documents are
often used to define task lists for projects and guide the next
step in the process. Examples of code fragments are also
often referred to during system development. Similar results
are observed in the later study with the addition of users
reporting its use in estimating costs, the collection and
documentation of system requirements and writing of test
plans.
5.6. Benefits and detriments of using the EPG/ER
In the earlier study, users reported in the user surveys that
the templates and examples were time saving and improved
the quality of documentation, that the tool provided

It is not surprising that there is less usage of the EPG/ER

in the second study than in the first, because software
organisations usually use technologies and develop applications that are similar to their previous projects.
Furthermore, given a stable workforce, apparent usage of
the repository will decline over time, as most experiences
that are particularly useful would already be available in the
repository and after the first study, users would already be
familiar with the content. Users have also informally
mentioned having copied some of the repository content
to their local drives and only using the repository to retrieve
updated versions. Furthermore, with the search facility,
which was provided to users in the second study, it was
possible to directly access a process page containing the
given experience entry without having to browse other

F. Kurniawati, R. Jeffery / Information and Software Technology 48 (2006) 566577

process pages. Hence this may also contribute to the lower

number of page accesses observed in the second study.
On the other hand, this pattern of declining usage may
indicate the EPG/ER is on its way to becoming obsolete.
Perhaps, the processes in actual use have changed over time
and thus the modeled processes require updating to reflect
this. Also, it is possible that more active support from the
management is needed. As an empirical study of the usage
of an informal experience repository in a medium size
software development company [29] noted, formal structures, techniques and procedures are generally overrated
while the power of social aspects is underestimated.
However, users did not suggest anything along this line,
whether formally or informally, and retained their positive
attitude towards the tool even after the second study. Thus it
seems more likely that this decline is attributed to users
having become familiar with the content of the process
guide and repository.
Results on the type of experiences accessed over time
suggested that the repository was used mostly as a vehicle to
reuse code examples. This indicates that they were
perceived as being particularly valuable to the users.
Development artefacts from previous projects also appear
to be useful for this organisation. This implies that concrete
artefacts remain to provide the best return on investment for
reuse in this organisation. It also appears that given more
time to use the repository, users have somehow learned to
distil anecdotal experiences for reuse. In spite of the same
number of anecdotal experiences available in the repository,
the number of retrievals of this type of experience has
noticeably increased in the second study. We suggest that
there is an opportunity for an explicit and lightweight
method of eliciting experiences in the organisation such that
the repository would continue to grow and serve as a
platform for learning.
The reported purposes for which experiences are used for
highlighted that the tool is more heavily used during the early
and middle stages of a project lifecycle. We found that this is
due to the straightforward nature of the processes involved in
the project closure. The process model included three types
of processes for project closure: update of the experience
repository, billing and update of the organisations cost
estimation tool. Based on the observed growth of the
repository, we know that the organisation is indeed updating
the repository. Also, to bill for a project, the organisation
does not require information from the EPG/ER. The
organisation is also not actively using their cost estimation
tool. It may be worth investigating whether there are other
project closure activities that are actually performed by staff
members but are not currently included in the process model.
6.2. Implications on the effects experienced from using the
EPG/ER
The reported effects showed that in spite of the initial
load that users experienced in having to learn to use the tool,

575

the usage of the EPG/ER brought about many more benefits

than detriments. Benefits experienced were mainly from
having intuitively structured experiences that served as
supporting artefacts for the users in executing their work. A
highly usable EPG/ER encouraged good documentation
practices that in turn resulted in improved traceability of
changes in the system being developed. It is important to
note that not only short-term benefits were derived from
using the tools and that compounded benefits led to users
having higher morale and confidence in doing their work.
6.3. Improvements made to the EPG/ER
Various suggestions to improve the EPG/ER were
implemented after each study. After the first study, the
look and feel of the tool was improved and a search facility
for the repository was provided. In the second study, we
discovered that users would prefer to have the tool
integrated with the companys other project management
support tools, as it would make it easier to use. User survey
results also revealed that users felt that it would be useful to
have a more straightforward way to define project costs and
tasks. Interestingly, in spite of low number of retrievals of
checklists, users suggested that it would also be useful to
augment the repository with experiences, specifically
checklists.
Following users suggestions, the EPG/ER is now
integrated loosely with the companys own in-house project
management and timesheet recording system. Each project
task can now be explicitly linked to a given process in the
EPG/ER and users are able to view this association and
conveniently access the knowledge already available for
that process. By doing so, we hope to have provided an even
more effective structure for users to search for relevant
information that are needed to complete a task.

7. Conclusion
Both studies provide insights into the short and long-term
use and effects of a software engineering process guide and
experience repository in a small software organisation. They
demonstrate that tangible benefits can be realized quickly
and that the EPG/ER remained useful with benefits accruing
over time. The ways in which the tool is being used were
similar across the two studies. Despite declining usage,
users retain a positive attitude towards the tool and no
longer reported the burden of learning load. Further, users
appeared to have learned to reuse more sophisticated form
of experiences.
The results suggests that it may be worthwhile for users
to perform regular checks on the conformity of the modelled
processes to the actual processes in use in order to reveal
any need for update of the process model and process areas
that can be further improved. The work already done Cook
et al. [38] on measuring the correspondence of change

576

F. Kurniawati, R. Jeffery / Information and Software Technology 48 (2006) 566577

request processes to their model and Lowe et al. [39] on

hypermedia development assessment method may provide
some hints in doing this for the software development and
management processes. Social aspects such as having active
encouragement and reminders to use the tool from management or a process champion should also be considered, to
ensure that staff members do not become complacent about
using the tool, thus hindering further process improvements.
Furthermore, given the interest in more sophisticated forms
of experiences such as checklists and anecdotal experiences,
development a more formal yet still lightweight method of
experience elicitation suited to a small software organisation use may be valuable. This is because the method will
be able to provide guidance and structure that can assist
users in creating more experiences for the repository, which
in turn, leads to continual process improvement in the
company.
The results from the studies validated the effectiveness of
the tool as an SPI tool by bringing about not only benefits
such as improved documentation and release of experts
from guidance of novices, but also a bolstered confidence in
the organisation to plan and execute software projects
because of the ongoing benefits gained. These results serve
as supporting evidence that user guidance is an important
attribute for effective experience repositories [30] and that
processes provide a natural and logical structure for people
to consider their work [37]. It suggests that users are guided
efficiently to the available knowledge needed for performing specific tasks instead of having to find relevant
information on their own. Further, in spite of increased
workload due to the introduction of a new tool, the users will
benefit in the long run.

Acknowledgements
This work was made possible by funding from an ARC
Linkage Grant, Allette Systems, and NICTA that is funded
by the Australian Research Council and Department of
Communications, Information Technology and the Arts.
Much of the earlier work on this project was led by Dr
Louise Scott.

References
[1] M.I. Kellner, U. Becker-Kornstaedt, W.E. Riddle, J. Tomal, M.
Verlage, Process guides: effective guidance for process participants,
Proceedings of the Fifth International Conference on the Software
Process (1998) 1125.
[2] U. Becker-Kornstaedt, M. Verlage, The V-Modell guide: experience
with a web-based approach for process support, Proceedings of
Software Technology and Engineering Practice (1999) 161168.
[3] T. Dingsoyr, N.B. Moe, The process workshop: a tool to define
electronic process guides in small software companies, Proceedings of
Australian Software Engineering Conference (2004) 350357.

[4] L. Scott, R. Jeffery, L. Carvalho, J. DAmbra, P. Rutherford, Practical

software process improvementthe IMPACT approach, Proceedings
of the Australian Software Engineering Conference (2001) 182189.
[5] L. Scott, F. Kurniawati, WAGNERWeb-Based Process Guide and
Experience Repository, Centre for Advanced Software Engineering
Research (CAESER) Report No. 02/1, www.caeser.unsw.edu.au,
2002.
[6] L. Scott, R. Jeffery, The anatomy of an experience repository,
Proceedings of the 2003 International Symposium on Empirical
Software Engineering 2003 (2003) 162171.
[7] F. Kurniawati, R. Jeffery, The long-term effects of an EPG/ER in a
small software organisation, Proceedings of the 2004 Australian
Software Engineering Conference (2004) 128136.
[8] Pageseeder, 2001, Weborganic systems: www.pageseeder.com.
[9] Allette Systems Pty Ltd: www.allette.com.au.
[10] F.D. Davis, Perceived usefulness, perceived ease of use, and user
acceptance of information technology, MIS Quarterly (1989) 319
339.
[11] R.P. Horton, T. Buck, P.E. Waterson, C.W. Clegg, Explaining intranet
use with the technology acceptance model, Journal of Information
Technology16(2001) 237249.
[12] ISO, ISO/IEC 12207:1995 Information TechnologySoftware Life
Cycle Processes. 1995, International Standards Organisation.
[13] P. Kruchten, Rational Unified ProcessAn Introduction, AddisonWesley, Reading, MA, 2000.
[14] Object-Oriented, Managing Successful Software Projects with
Process MeNtOR, Object Oriented Pty Ltd, 1998.
[15] U. Becker-Kornstaedt, D. Hammann, R. Kempkens, P. Roesch, M.
Verlage, J. Zettel, Support for the process engineer: the spearmint
approach to software process definition and process guidance,
Proceedings of the 11th Conference on Advanced Information
Systems Engineering CaiSE99 (1999) 119133.
[16] ARIS, 2000: www.ids-scheer.de.
[17] Adonis, 2001: www.boc.at.
[18] G. Caldiera, H.D. Rombach, The experience factory in: J. Marciniak
(Ed.), Encyclopedia of Software Engineering, Wiley, New York, 1998
pp. 469476.
[19] K. Schneider, LIDs: a light-weight approach to experience elicitation
and reuse, Proceedings of the Second International Conference on
Product Focused Software Process Improvement (2000) 407424.
[20] K. Althoff, A. Birk, S. Hartkopf, W. Muller, M. Nick, D. Surmann, C.
Tautz, Systematic population, utilization and maintenance of a
repository for comprehensive reuse, Proceedings of the 11th
International Conference on Software Engineering and Knowledge
Engineering (1999) 2550.
[21] F. Houdek, H. Kempter, Quality patternsan approach to packaging
software engineering experience, Proceedings of the 1997 Symposium on Software Reusability 22 (1997) 8188.
[22] S. Henninger, J. Schlabach, A tool for managing software development
knowledge in: F. Bomarius, S. Komi-Sirvio (Eds.), Proceedings of
Product Focused Software Process Improvement: Third International
Conference vol. LNCS 2188, Springer, Berlin, 2001, pp. 182195.
[23] B. Lewis, On-Demand KM: A Two-Tier Architecture, IT Professional
4 (2002) 2733.
[24] Y. Ye, G. Fischer, Supporting reuse by delivering task-relevant and
personalized information, Proceedings of 2002 International Conference on Software Engineering (2002) 513523.
[25] J. Keung, R. Jeffery, B. Kitchenham, The challenge of introducing a
new software cost estimation technology into a small software
organisation, Proceedings of the Australian Software Engineering
Conference (2004) 5259.
[26] R. Conradi, T. Dingsoyr, Software experiences bases: a consolidated
evaluation and status report, Proceedings from the Second International Conference on product focused Software Process Improvement (PROFES 2000) LNCS 1840 (2000) 391406.
[27] M. Lindvall, M. Frey, P. Costa, R. Tesoriero, Lessons learned about
Structuring and Describing Experience for Three Experience Bases,

F. Kurniawati, R. Jeffery / Information and Software Technology 48 (2006) 566577

[28]

[29]

[30]

[31]
[32]
[33]
[34]
[35]

Proceedings of the Third International Workshop. Advances in

Learning Software Organisations (LSO 2001, 2001), pp. 106119.
P. Brossler, Knowledge management at a software engineering
companyan experience report, Proceedings of the Workshop on
Learning Software Organisations (1999) 7786.
T. Dingsoyr, E. Royrvik, An empirical study of an informal
knowledge repository in a medium-sized software consulting
company, Proceedings of the 25th International Conference on
Software Engineering (2003) 8492.
K. Schneider, J. von Hunnius, Effective experience repositories for
software engineering, Proceedings of the 25th International Conference on Software Engineering (2003) 534539.
Enterprise Architect 2003: http://www.sparxsystems.com.au/.
Xalan-Java. 2003: http://xml.apache.org/xalan-j/index.html.
TikiWiki 2002: http://tikiwiki.org.
Typo3: http://typo3.com.
F. Kurniawati, A. Kitchenham, R. Jeffery, Implementing a Process
Model-Based Software Development Support environment: comparing an open source component with a proprietary tool, The 5th
International Workshop on Software Process Simulation and
Modeling (2004).

577

[36] F. Maurer and H. Holz, Process-oriented Knowledge Management

for Learning Software Organisation, Proceedings of the 12th
Knowledge Acquisition Workshop, 1999.
[37] H. Holz, A. Konnecker, F. Maurer, Task-specific knowledge
management in a process-centered SEE, Proceedings of the Third
International Workshop on Advances in Learning Software Organisations (2001) 163177.
[38] J.E. Cook, A.L. Wolf, Software process validation: quantitatively
measuring the correspondence of a process to a model, ACM
Transactions on Software Engineering and Methodology 8 (2) (1999)
147176.
[39] D.B. Lowe, A.J. Bucknell, R.G. Webby, Improving hypermedia
development: A reference model-based process assessment method,
Proceedings of the 10th ACM Conference on Hypertext and
Hypermedia (1999) 139146.
[40] L. Scott, L. Carvalho, R. Jeffery, J. DAmbra, U. BeckerKornstaedt, Understanding the use of an electronic process
guide, Journal of Information and Software Technology 44 (10)
(2002) 601616.
[41] N.B. Moe, T. Dingsoyr, T. Dyba, T. Johansen, Process guides as
software process improvement in a small company, EuroSPI (2002).