Business-process-driven

gray-box SOA testing

&

Challenges are emerging in testing service-oriented architecture (SOA) systems.

Current testing is not sufficient to deal with the new requirements arising from several
SOA features such as composition, loose coupling, and code without a graphical user
interface. The most critical architecture information of an SOA solution is actually how
services are composed and interact with each other. This paper proposes a gray-box
testing approach, that is, an approach that involves having access to internal workings,
data structures, and algorithms when designing the test cases but tests at the user
Z. J. Li
level as a black box, that is, by applying inputs and observing outputs. This approach
H. F. Tan
leverages business processes and the underlying SOA layered architecture to better
H. H. Liu
test SOA solutions. A commonly used language to model business processes is BPEL
J. Zhu
(Business Process Execution Language), which is the focus of the approach described
N. M. Mitsumori
in this paper. Among the layered artifacts, the business process view represents the
global behavior of the SOA system and thus is a good candidate as supplemental
architectural information to the functional requirement or specification in test-case
design and generation. This approach has three key enablers: test-path exploration,
trace analysis, and regression test selection. BPELTester is an innovative tool that
implements this method. It has been piloted in several projects and the initial pilot
results are presented in this paper.

INTRODUCTION
Service-oriented architecture (SOA) provides a
service ecosystem in which composite business-
centric services can be assembled on the fly as
requirements change. Testing of the new compo-
nents is similar to traditional application testing, but
testing the assembled processes requires new and
more extensive testing approaches.
Problem definition
Black-box testing treats the software as if one had no
understanding of internal behavior. It tests the
functionality according to the requirements. Thus,
the tester inputs data and sees only the output from
the test object. Current black-box testing at the
graphical user interface (GUI) level is not sufficient
to deal with the new requirements arising from
several SOA features such as composition, loose
coupling, and GUI-less code.
ÓCopyright 2008 by International Business Machines Corporation. Copying in
printed form for private use is permitted without payment of royalty provided
that (1) each reproduction is done without alteration and (2) the Journal
reference and IBM copyright notice are included on the first page. The title
and abstract, but no other portions, of this paper may be copied or distributed
royalty free without further permission by computer-based and other
information-service systems. Permission to republish any other portion of the
paper must be obtained from the Editor. 0018-8670/08/$5.00 Ó 2008 IBM

IBM SYSTEMS JOURNAL, VOL 47, NO 3, 2008 LI ET AL. 457

 Composition of coarse-grained services introduces
test-coverage challenges. Test coverage is a well-
accepted aspect of testing discipline. It directly
relates to the risk-management objectives of the
projects being tested. It is also the metric that
dictates the overall test activity plan. So how is
coverage different in SOA? SOA coverage must
consider how each element operates within a
process, not just how an element operates in
isolation. It is reported in Reference 1 that Theresa
Lanowitz, lead analyst for application testing for
Gartner, Inc., observed that traditional coverage
ratios become diluted when considering the process
as a whole. For example, if traditional coverage is
set at 80 percent for each part of a three-part
process, the overall coverage equates to only about
50 percent (i.e., 80 percent 3 80 percent 3 80
percent). This coverage level is more unacceptable
in the SOA context because the composed process
will be used by many more users in a much more
unpredictable way compared with traditional com-
ponent-based software. Therefore, additional tech-
niques are needed not only to improve the coverage
of all hidden elements, but also to raise the
assurance level by covering the end-to-end process
itself.
The constant change that occurs in a loose-coupling
SOA environment can imperil system stability.
Changes are driven by market pressures, customer
demands, regulation, reorganization, and techno-
logical evolution. In an SOA environment, a change
in one part of a business system is no longer an
isolated alteration that has no impact on external
parties. One now has to determine if a change affects
anything else in the series of connections. To
mitigate the risk, there is a need to identify all test
cases that should be rerun in order to ensure that the
change does not have a negative impact on the
originally working functions (i.e., regression test
selection) and to ensure that the test cases evolve
with system changes.
With loose coupling, people can develop and
maintain their own blocks independently without
interfering with others’ blocks, but this indepen-
dence sits on the syntactical match, not a semantic
match. As long as a service does not change its
interface, the consumer will not notice an imple-
mentation change. However, there is a potential risk
that the internal functional logic might have
458 LI ET AL.
changed, and those changes will affect the con-
sumer’s function. This is why loose coupling makes
regression test selection more important.
Since many services are GUI-less, there is a need for
test-case generation from non-GUI artifacts such as
WSDL (Web Services Description Language) and
BPEL (Business Process Execution Language). [By
2
BPEL, WSDL, and XSD (XML Schema Definition) in
this paper, we are referring to services or functions
created in these languages.] Such test cases have
advantages over GUI-specific test cases in that they
can persist through GUI changes and can be easily
automated.
Related work
In current SOA testing research and in the product
marketplace, a lot of work is focused on testing Web
3
services, although that is only a small piece of SOA.
In most Web-service testing research, the test data
or test sequence is generated based on the interface
specifications. They belong to black-box testing.
Due to the important role of BPEL in service
composition, BPEL testing has also become a focus
for research in recent years. One author of this paper
proposed a general test framework for BPEL unit
4
testing in 2005, proposed a Java** test framework
5
for BPEL unit testing in 2006, and implemented a
6
test-generation algorithm that forms the base of the
test-path exploration function of the BPELTester,
which is introduced in this paper. The test-genera-
tion algorithm in Reference 6 is only a preliminary
implementation, with a simple model, a simple
searching algorithm, and the omission of some
advanced BPEL features.
From another perspective, due to the dynamics of
SOA, testing that is based on static analysis is
insufficient to ensure the coverage of an SOA
7
system. Canfora and Di Penta proposed the idea
that testing and runtime monitoring techniques
should be combined to improve the quality of an
SOA system.
Because SOA conforms to open standards, service
interaction information can be captured in the
middleware without modifying the applications
either before or after deployment and even when the
environment includes a mixture of programming
IBM SYSTEMS JOURNAL, VOL 47, NO 3, 2008
languages and operation platforms. This monitoring
capability at the middleware layer makes it easy to
log the Web-service interactions as a trace. Also,
there are tools to capture and visualize the execution
8
of Web services. Web Services Navigator is
designed to help users understand the dynamic
behavior of a distributed SOA system by visualizing
the execution of Web-services-based applications.
However, there are no methods to correlate trans-
action trace with business process. In this paper, we
report a trace-analysis framework that can perform
the correlation.
9 Regression test selection identifies the impacted test
In the regression testing area, Liu et al. proposed a
method to select regression test cases based on the
comparison of the old and new BPEL processes. Its
limitation is that it cannot identify the new-
structural tests (test cases associated with the new
process structure). To overcome this limitation, in
this paper we propose a new regression test
selection method based on the direct comparison of
the two sets of test cases that are generated when
the test-path exploration function is used for both
the old BPEL process and for the new BPEL process.
Furthermore, a minimization algorithm is proposed
to select a minimum set of test cases that can cover
all of the activities impacted by a process change.
Brief summary of the new solution
This paper proposes a gray-box testing approach
that leverages BPEL as the underlying SOA layered
architecture to better test SOA solutions by meeting
three challenges: ensuring proper coverage for
composition, ensuring proper regression testing for
loose-coupling SOA, and doing test generation for
GUI-less code. This approach has three key en-
ablers: test-path exploration, trace analysis, and
regression test selection.
Test-path exploration automatically finds the typical
execution scenarios that correspond to end-to-end
business transactions (and the critical test data) and
ensures the proper coverage of implemented Web
services and internal system integration scenarios.
These test scenarios can be used in both process unit
testing and service and process integration testing,
and they can be executed in various SOA test
automation tools, including IBM WebSphere* Inte-
gration Developer (WID) and IBM Rational* Tester
2
for SOA Quality. In this paper, enhancements to the
original model and to the path-searching algorithm
IBM SYSTEMS JOURNAL, VOL 47, NO 3, 2008
are introduced. These enhancements aim to provide
a greater capability for test generation that can be
applied to real, complex BPEL processes.
Trace analysis maps Web-service invocation records
that are collected in a distributed environment to
test paths inside BPEL processes. It validates these
traces, provides insight into the business importance
and relevance of the test cases, and helps improve
test coverage by introducing newly discovered
runtime process execution patterns.
cases when Web-service implementations or busi-
ness processes are changed. The impacted test cases
can then be selected to be run or rerun in order to
test the changed application.
The left side of Figure 1 shows the SOA solution
development process and associated main artifacts.
In the business-level specification, we have process
models, requirements, and functional design. In the
IT-level (information-technology level) implemen-
tation, we have a realized SOA architecture, service
implementations, BPEL process implementations,
and other applications. Then the implementation
artifacts are deployed to the test environment for
system testing. After that, they are delivered to the
production environment.
The right side of Figure 1 shows a scenario of how
our technique works. The test-related artifacts are
positioned according to the development layers to
which they belong or correspond. For example,
BPEL belongs to the IT level.
Test-path exploration and trace analysis work in a
complementary way. Test cases group 1 is designed
directly from requirements and represents tradi-
tional specification-based test cases. This group is
already present in many projects. Test cases group 2
is designed from BPEL processes and represents test
cases derived using the technique proposed here. By
running group 1 test cases, collecting traces,
mapping them back to BPEL paths, and comparing
such paths to autogenerated test paths, we can
identify the differences in test cases and improve the
coverage by combining the two groups of test cases.
In addition to test traces, this kind of trace analysis
also works for production traces.
LI ET AL. 459
 Business-level specification

Business process Requirement Test Test cases group 1

model design design

Test cases group 2

Tools
IBM WebSphere Business Integration Modeler
IBM Rational Software Modeler

Build
IBM Rational RequisitePro*

IT-level implementation Test path Test

Revise
SOA architecture refinement test cases run

External Automated Manual

activities activities activities Path update
BPEL test paths Updated BPEL test paths

Service BPEL Application Web Test path Regression Test path

exploration test selection revision
Tools
IBM Rational Software Archtect Process update
IBM WebSphere Integration Developer BPEL Updated BPEL
Deploy

Test and production environment Trace

analysis
Automatic
verification
Packaged IBM WebSphere
application Application Servers

- Link to BPEL test paths

Legacy - Evaluate test coverage
application - Debug problems Transaction traces
- Assign test weight
Tools - Add new test paths
IBM Web Services Navigator

Figure 1
Gray-box SOA testing method

Regression test selection works in response to Web
service or business process changes. When business
processes are updated, we need to identify affected
test paths. Based on this, test cases can be revised
more easily. We also need to remove obsolete test
paths and add new test paths.
The next section of this paper elaborates the test-
path exploration technique. The following sections
discuss trace analysis, regression test selection, and
BPELTester implementation. We then conclude the
paper and forecast future work.
TEST-PATH EXPLORATION
Like other programs, a BPEL process contains
different execution paths that represent different
Web-service interactions (transaction patterns) be-
tween SOA system components. It is ideal to
exercise all the possible runs of a program to detect
hidden bugs in testing.
Test generation for both sequential and concurrent
10–12
programs has a long research history. BPEL has
unique features in both syntax (e.g., flow with
activity synchronization, join condition) and se-
mantics (e.g., multiple-choice workflow pattern,
dead path elimination) that need special treatment.
In an earlier work, we implemented a graph-search-
6
based approach to BPEL test-case generation. We
first transformed BPEL to BPEL flow graph (BFG),
which is an intermediary model extended from
control flow graph (CFG). Then we generated
concurrent test paths based on BFG and generated
test data for each path.

460 LI ET AL. IBM SYSTEMS JOURNAL, VOL 47, NO 3, 2008

 EventHandlers CompensationHandler BFGNamedElement
+name : String BFGLink
+status : ExecutiveStatus
+ID : Integer +linkStatus : Boolean = true
+type : String +transitionCondition : String
+nameMapping : String
children +links

<<enumeration>>
ScopeTree ActivityDomain BFGNode ExecuteStatus
father +scopeName : String +inputAction : String +Executed
BFGProcess
+outputAction : String +NotExecuted
+nodes
+internalAction : String +name : String +DPE
+Ready

CatchAllNode CatchNode BFGActivitiyNode BFGFlowNode

+faultName : String +isStartNode : Boolean
+faultVariable : String +isFinishNode : Boolean
+faultMessageType : String

CompensationNode BFGDisjoinedNode BFGUnitedNode

+scopeName : String

BFGForkNode BFGConditionNode BFGJoinNode BFGMergeNode

+joinCondition : String

Figure 2
BFG metamodel

BFG Assign data calculation information is kept in the

The full BFG is defined in a Unified Modeling
Language metamodel in Figure 2. Table 1 summa-
rizes its major elements.
A BPEL process will be translated to an internal
representation of type BFGProcess, which is com-
posed of nodes (type BFGNode) and links (type
BFGLink). Each link has a linkStatus and a
transitionCondition attribute. The linkStatus is used
to mark whether a link is on a DPE (dead path
elimination) path. The transitionCondition holds
various BPEL conditions (each switch case, while,
or link can have a condition). There are two types of
BFG nodes: BFGActivityNode and BFGFlowNode.
BFGActivityNode represents basic BPEL activities,
including Invoke, Receive, Reply, and OnMessage,
which are instructions for Web-service interaction,
and Assign, Wait, and Terminate, which are internal
activities. The input message information is kept in
the inputAction attribute of the BFGNode, and the
internalAction attribute. The actions are used to
perform symbolic execution when calculating test
data. BFGFlowNode represents structured BPEL
activities, including Flow, Switch, While, and Pick,
which introduce control flow structures. As intro-
duced in Reference 6, Flow will be transformed to a
BFGCondition node and a set of BFGForkNode to
implement its multiple-choice workflow pattern.
Switch, While, and Pick will be transformed to a
BFGCondition node.
The ScopeTree class is used to keep the nested BPEL
scope structure and provides navigation between
scopes to correlate BPEL fault handlers and com-
pensation handlers to faults and to compensate for
actions that may occur during normal processing.
All the BPEL Catch and CatchAll activities are
attached under a related ScopeTree node. The
behaviors inside the BPEL Catch and CatchAll and
compensationHandlers will be transformed to

IBM SYSTEMS JOURNAL, VOL 47, NO 3, 2008 LI ET AL. 461

 Table 1 BFG elements

Type Description

BFGNamedElement Structure collecting common attributes, such as name, type, and identification

BFGProcess The top-level structure representing a BPEL process, composed of BFGLink and BFGNode

BFGLink Structure that connects BFGNode

BFGNode Structure collecting common node attributes. A set of actions record the data calculations
(assignments) in the node

BFGActivityNode Structure for basic BPEL activities

BFGFlowNode Structure for structured BPEL activities

BFGDisjointNode Structure for fork and switch activities

BFGUnitedNode Structure for join and merge activities

BFGForkNode Structure for pure concurrency

BFGConditionNode Structure for exclusive choice

BFGJoinNode Structure paring with BFGForkNode

BFGMergeNode Structure paring with BFGConditionNode

CompensateNode A special BPEL basic activity that must have a scopeName attribute

ScopeTree Structure for BPEL scope hierarchy

ActivityDomain Structure for organizing fault-handling logic

CatchNode Structure for Catch activity inside a faultHandler

CatchAllNode Structure for CatchAll activity inside a faultHandler

CompensationHandler CompensationHandler

EventHandlers EventHandlers

BFGNode and BFGLink elements in exactly the same
way as described in the previous paragraph. BFG
CatchNode, CatchAllNode, and CompensationHan-
dler all have a startNode and an endNode attribute
to point to the contained behaviors.
Based on this model, BPEL fault-handling logic can
be supported. During the transformation of BPEL to
BFG for an Invoke activity, if it can throw a fault, the
algorithm will search the ScopeTree, find the Catch
or CatchAll that captures the fault, and make a
connection from the Invoke activity to the matched
Catch or CatchAll node. Here we must handle the
situation in which a fault is thrown inside a
faultHandler. The BPEL specification says that such
a fault should not be captured by the scope to which
the faultHandler belongs (in order to avoid loops in
the fault-handling logic). The fault-catch matching
algorithm must be able to support this requirement;
otherwise the test generation will get into infinite
loops.
Another complexity of the BPEL fault-handling logic
exists in the fact that after a fault is thrown, all other
activities inside the scope need to be terminated.
BPEL can express concurrent behaviors, so the exact
execution path is indeterministic when a fault
occurs. Suppose we have two threads: When one
thread throws a fault, the other thread may complete
one, two, or more activities depending on runtime
scheduling and the speed of peer services. It is
inefficient to enumerate all these varieties. Our
approach is to find the longest possible execution
path. To achieve this, the path-searching algorithm
starts from the fault point and tracks down the path
that is followed when the Invoke gets a normal

462 LI ET AL. IBM SYSTEMS JOURNAL, VOL 47, NO 3, 2008

(LoanRequest.amount,0) Receive Loan Request
LoanRequest.amount<10000
(AssessResult.risk,'high') Invoke Assessor
AssessResult.risk = 'low'
Assign Approval
Invoke Approver
Reply Loan
Figure 3
Test path, test-path feasibility, and test data
response. All the nodes that are in the fault
propagation path (also in the fault scope) will be
terminated; the other nodes not affected by the fault
will be executed. This is implemented as a special
kind of DPE.
Using the Eclipse Modeling Framework (EMF), the
BFG model can be transformed to Java classes that
provide application programming interfaces to
create, modify, and store BFG models transformed
from BPEL processes. During the transformation,
data logic must also be processed. BPEL variables
are filled in a variable table with their possible value
domains. In addition, WSDL and XML Schema
Definition (XSD) information referenced by the
BPEL process should also be parsed to obtain the
Web-service definitions (operations, messages, and
data types).
Test-path feasibility and test-data generation
A test path consists of activities that start from the
process entrance (typically a Receive activity) and
move forward until a process exit (typically a
Reply), but not all the test paths identified as such
IBM SYSTEMS JOURNAL, VOL 47, NO 3, 2008
LoanRequest.amount> =10000
AssessResult.risk! = 'low'
Sequence
Assign Init Assignment: LoanResponse.result:='approve failed',
FaultCount.count:= 0
While While condition: LoanResponse.result='approve failed'
&& FaultCount.count<2
(LoanResponse.result,'success')
Structured
Links activity
Links with transition condition
Assign activity
Sequence connection
Test path Web service activity
are actually feasible. A test path is feasible if there
exists at least one set of test data that satisfies all the
conditions (described as logic expressions) in this
path. We can use satisfiability and constraint solving
14,15
tools to solve the Boolean expressions and the
13
formulas of inequalities and equalities derived from
each test path. If no solution is found, the path is
infeasible and filtered; otherwise the
path is feasible, and a solution is also at hand.
Typically, the data constraints and the solution
cover only part of the required data, composed of
some fields of an input message. The rest of the
input message fields can be filled manually or by
using automatic data generation tools.
Figure 3 gives an example BPEL process and a test
path with test data. The arrows pointing to BPEL
activities show process input locations where some
test data has been generated and listed in a format
similar to (LoanRequest.amount,0). For this exam-
ple, we can also tell that the while loop must be
executed (i.e., cannot be skipped), because its
condition is definitely to hold after the Assign Init
LI ET AL. 463
 activity. In other words, any test path that skips the
While loop is infeasible.
Optimized test-path-searching algorithm
The original test-path-searching algorithm in Refer-
ence 6 iterates two steps: find a test path and solve
the path constraints. In this way, it exercises all
possible test paths (both feasible and infeasible). It
does not leverage any feasibility information that
has been obtained from the earlier test paths. An
improvement here can be explained with an
example. Suppose test path 1 is infeasible, and it can
be determined that it has become infeasible at the
point Node1, which is a BFGCondition node where
the partial constraint at this point is already
unsatisfiable. With this knowledge, we no longer
need to explore the other branches of Node1. In this
spirit, a branch-cut mechanism is implemented as
an improvement to the original algorithm.
The key data structure of this algorithm is a branch
tree. It is dynamically constructed along with the
path searching. Whenever a BFGCondition node is
encountered, N children tree nodes are added to the
current tree node that the current test path contains,
where N is the number of branches of the
BFGCondition node. Note that the path-searching
algorithm is a depth-first searching one, so the
branch tree will be constructed in a similar way. It
can be seen that the branch tree is an exact tree
representation of all test paths found so far: A path
from the root to each leaf node is a complete or
incomplete test path.
Initially, all the branch nodes are in ‘‘Unknown’’
status. Whenever a complete test path is found and
the feasibility status is determined, the branch tree is
updated. If the test path is feasible, all the tree nodes
in the path are set to ‘‘Feasible’’ status. If the test
path is infeasible, the path constraint is backtracked
until a ‘‘Feasible’’ node is encountered or the
constraint becomes satisfiable for the current node
or the root node is encountered. Then the node
preceding the current node along the backtracking
path will be the cut point. All the test paths under
this node no longer need to be explored.
This branch-cut algorithm improves the test-gener-
ation performance significantly with very little
overhead.
Test coverage
The following two coverage goals can be selected
before test generation.
464 LI ET AL.
 All-path coverage—Enumerate all possible execu-
tion paths. However, if a process contains a while
structure, each repetition will produce a new
execution path, and unlimited repetition will
produce infinite execution paths. In order for the
test-generation algorithm to complete in a rea-
sonably short time, we chose to set an upper limit
on the repetition number for each while structure
via user configuration. So a more precise name
would be limited all-path coverage.
 Branch coverage—Try to cover each branch at
least once. BPEL branches include: switch(case/
otherwise), pick(onMessage/onAlarm), invoke
(normal/fault), while(enter/skip), link(true/false).
(Depending on the data condition, not all the
branches could be covered.)
After test generation, there are other goals that can
be achieved by selecting (automatically) a subset of
test paths from all the paths generated:
 Branch coverage—This makes sense only when
all-path coverage is selected before test genera-
tion.
 Normal coverage—All test paths that contain no
faults.
 WS fault coverage—All test paths that contain a
Web-service fault (due either to a fault response to
an Invoke activity or a Throw activity).
 Service coverage—A minimal set of test paths that
can cover all services.
 Customized operation coverage—This goal is
measured in terms of Web service operations.
Users can select a set of Web-service operations
and the tool will automatically select the test paths
that can cover any or all of the chosen operations;
the tool can also select a minimal set of test paths
that can cover all the chosen operations.
We also provide exact coverage rate for the
following coverage goals:
Branch coverage, covRate ¼ (#branch covered)/(#all
branches).
Service coverage, covRate ¼ (#service covered)/(#all
services).
Operation coverage, covRate ¼ (#operation cov-
ered)/(#all operations).
WS fault coverage, covRate ¼ (#fault covered)/(#all
faults).
For each category, all the uncovered items are also
listed as references for users to design further test
cases.
IBM SYSTEMS JOURNAL, VOL 47, NO 3, 2008
This flexible coverage capability is important to
ensure sufficient SOA business-process or service-
interaction coverage and at the same time allows
users to do customization according to their specific
needs and constraints.
TRACE ANALYSIS
Trace analysis takes traces collected in a testing or
production environment, identifies the transactions,
and maps each transaction to the test paths of the
related BPEL processes. It can be used to build the
linkage between test cases (as well as normal
production uses) and BPEL process execution paths.
Based on this linkage, the following functions can be
provided:
1. Runtime validation using the implementation
information: A process execution path gives
information about how the services are com-
posed to provide another service, and a transac-
tion trace gives information about how the
services interact with message exchanges. Thus
we can validate the correctness of a business
process path execution by combining them.
2. Test-path coverage analysis for a set of test cases
in terms of the BPEL process structure coverage.
3. Finding all test cases that need to be updated with
the system evolution: Changes that can affect a
test path will also affect the related test cases.
4. Identifying the real usage profile of a business
process in production: Traces will reveal the real
executed paths and their occurrence frequencies.
This information is useful to find uncovered
paths for testing by comparing it with the testing
profile and to prioritize test cases for the next
rounds of testing based on the occurrence
frequencies of the related test paths.
The transaction trace is translated to an intermedi-
ary model that is independent of any transaction
trace format. It is composed of a set of service
invocation events. Each event has three parts: port,
operation name, and the direction of the message.
The BPEL process is translated into a forest model (a
set of trees in which there is a scope tree
representing the nested BPEL scopes and pointing to
many other trees of the behaviors inside the scopes).
It uses a tree structure to keep the control flow
information, where activities related to service
invocations appear as leaves in the model. The
correlator travels the forest model to map each event
of a transaction to one leaf in the forest and then
IBM SYSTEMS JOURNAL, VOL 47, NO 3, 2008
finds a path to link all those leaves together to form
a path.
In the following sections, we explain the imple-
mentation details of the BPEL-to-forest model
translation and the correlator.
BPEL-to-forest-model translator
The forest model separates the scopes, activities,
and links, and uses a parent-child relationship to
describe the nested control flow structure in the
BPEL process. It can be defined as: forest ¼ hT, pi,
where T is the set of the trees related to the process.
Each tree represents the behavior inside a specific
scope; p is the tree representation of all the scopes
defined in the process.
Each node of the scope tree has links to several
behavior trees: normal activity tree, eventHandlers
tree, faultHandler tree, and compensationHandler
tree if they exist in the process definition.
Figure 4 shows how the BPEL control flow structure
is represented in a tree model. Each activity in a
structured activity is a child node of its immediately
enclosing structured activity. The links information
in a flow is represented in an Adjacency Matrix
where the rows and columns are activities, and the
link information is stored in the intersection of its
source and target activity.
The translator converts a BPEL process definition to
a forest model by traveling all the elements. The
translation rules are as follows:
1. One activity is translated to a behavior tree,
which is a child of its immediately enclosing
activity.
2. One scope is translated to a scope behavior tree
like other types of activity, and it is also
translated to a scope tree, which is the child of its
immediately enclosing scope.
3. Various handlers are translated to appropriate
behavior trees.
Correlator
This component finds execution paths of the BPEL
processes for a specific transaction. The transaction
trace is an ordered set of service invocation events.
The correlator is a finite-state-machine (FSM)
generator. The states of the FSM are called match
points. Each match point includes a source events
LI ET AL. 465
 A Sequence AND Split Flow

Sequence B A B C Flow A (. . .) B A ... C

C AND Join

XOR Split Switch While

Switch or pick A (. . .) B A ... B While A A

XOR Join Loop

Figure 4
Mapping from the BPEL structure to the tree model

set (a set of the unmatched events of the transac-
tion), a matched leaves set (a set of matched leaves
that have been mapped to the events of the
transaction), and a target leaves set (a set of leaves
in the forest that are possible to match the remaining
unmatched events in the transaction). Each transi-
tion is a transfer from the current match point to
another match point by matching the next un-
matched event in the source events set to one of the
leaves in the target leaves set. In the initial state, the
matched leaves set is empty, the target leaves set
includes all the leaf nodes, and the source events set
includes all the events of the transaction.
Beginning at the initial state, all the transitions and
states are constructed until all the final states are
found. For each state, we match the first one or two
events in the source events set to all the possible
leaves in the target leaves set according to the
service port, the operation name, and the direction
of the message of the events. Table 2 shows the
mapping rules. For example, the first rule states:
When a message belongs to a service provided by
the process (service owner), and the direction of the
message is from partner to process (message
direction), the event will correspond to a Receive or
OnMessage BPEL activity.
For each found mapping, a new state is generated.
For the new state, the leaf is added to the matched
leaves set, the events are removed from the source
events set, and all impossible leaves in the target
leaves set are removed. Impossible leaves include
the following:
 Leaves that are in the switch or pick branches that
are not in the path.
 Leaves that are in the dead path.
 Leaves that are in the tree of the left siblings of the
children of the matched sequence.

Table 2 Rules for mapping trace events to BPEL activity

Service owner Message direction Operation type BPEL activity

Process partner ! process Any Receive or OnMessage

Process process ! partner 2-way Reply

Partner process ! partner 2-way Invoke, Catch

partner ! process

Partner process ! partner 1-way Invoke

466 LI ET AL. IBM SYSTEMS JOURNAL, VOL 47, NO 3, 2008

If the source events set is empty and there are no
leaves in the target leaves set, a final state is reached
and one path is found.
To form a path from a set of leaves, the correlator
travels the forest model to allocate all the ancestors
of each identified leaf. Those ancestors and leaves
are kept in a set named visited set. We travel the
BPEL process definition and identify whether the
current node is in the visited set: if yes, the node is
kept; otherwise the node is deleted. All kept nodes
make the path for the current transaction.
REGRESSION TEST SELECTION
The objective of business process regression test
selection is to identify the impact of business
process changes on test cases and then take proper
test-case update actions accordingly and determine
the subset of test cases to run or rerun.
9 decision points and branches, so by comparing the
Liu et al. proposed a method to select regression
test cases based on the comparison of the old and
new BPEL processes. The advantage of this method
is that only the relationship between the existing test
cases of the old process and the BPEL activities
should be known in advance, and no assumption of
automatic test generation capability is made. How-
ever, it cannot identify the new structural cases,
which need to be designed in one way or another.
To overcome this limitation, in this paper we
propose a new regression test selection method
based on the direct comparison of the two sets of
test cases that are generated when the test-path
exploration function is used for both the old BPEL
process and for the new BPEL process. Furthermore,
a minimization algorithm is proposed to select a
minimal set of test cases that can cover all activities
impacted by the process changes. These have been
implemented in the BPELTester tool.
Test-path classification
We classify the test paths for the original process
and the new process into four categories:
1. Reusable paths—Not impacted by the process
changes, representing test paths that are common
to the old and new processes.
2. Modified paths—Impacted in the sense that
conditional branches on the test paths are the
same but activity attributes or nonconditional
activities are somehow changed, representing old
and new test paths that have minor differences.
IBM SYSTEMS JOURNAL, VOL 47, NO 3, 2008
3. Obsolete paths—Only valid for the old process;
representing old test paths that do not exist for
the new process.
4. New-structural paths—Only valid for the new
process; representing new test paths having no
correspondence in the old process.
Test selection based on test paths comparison
After a business process is changed, new test paths
can be generated using test-path exploration. Use M
for the modified paths in the old path set, M 0 for the
modified paths in the new path set, R for the
reusable paths, D for the obsolete paths, N for the
new-structural paths, P for the old path set, and P 0
for the new path set. It is obvious that the following
are true: P ¼ R þ M þ D and P 0 ¼ R þ M 0 þ N.
It is clear that by exactly comparing the paths in P
and P 0 , R could be obtained, which exists both in P
and P 0 . Test paths in M and M 0 have the same
decision points and branches, we can obtain M and
M 0 , and then D and N can also be easily identified.
Test-selection minimization algorithm
In regression testing, the modified and new-struc-
tural test paths should be rerun for the new process.
Sometimes, instead of rerunning all of them, one
may want to further select a subset of these test
paths that has a lower level of test coverage,
covering all the impacted activities by the changes.
For this purpose, a test-selection minimization
algorithm is implemented.
First, the impacted activities in the new process
should be identified. For a nonconditional activity, if
it is added into the new process, modified, or deleted
from the old process, the added activity, modified
activity, and previous activity of the deleted activity
in the new process are impacted activities. For a
conditional activity, Table 3 is used to identify the
impacted activities (not including the changed
activity itself). In this table, the change actions
include addition, modification, and deletion.
Let xi be the existence indicator of test path i. Let xi ¼
1 indicate that test path i is selected, and xi ¼ 0
indicate the opposite. The expression CA(xi ¼ 1)
represents the activities on test path i, and CA(xi ¼
0) ¼ U. IA represents all the impacted activities in
the new process. Then the constraint of the
optimization problem can be expressed by
CAðxi Þ [ CAðx2 Þ [    [ CAðxn Þ  IA: ð1Þ
LI ET AL. 467
 Table 3 Impacted activity table
Activity type Impacted activity
Case Otherwise
Otherwise none
onMessage none
onAlarm none
Catch CatchAll
CatchAll none
Link other links that have the same source
activity with this link
Further, taking into consideration the possibility
that test paths could have different priorities we can
use the objective function
Fðx1 ; x2 ; . . . ; xn Þ ¼ f ðw1 Þx1 þ f ðw2 Þx2 þ   
þ f ðwn Þxn
wherein wi represents the priority of test path i, and
f(wi) is a monotonically decreasing function of wi. A
simple function can be f(wi) ¼ 1/wi.
With the above definitions, it is obvious that the
test-selection minimization problem becomes an
optimization problem, with Equation 1 as the
objective function and Equation 2 as the constraint.
In the BPELTester implementation, for simplicity we
set all wi to 1 (meaning that all the test paths have
the same priority). In this case, Equation 2 becomes
Fðx1 ; x2 ; . . . ; xn Þ ¼ x1 þ x2 þ    þ xn :
In general, the optimization problem is NP-hard.
BPELTester uses a dynamic programming algorithm
to find a solution.
BPELTESTER IMPLEMENTATION TOOL
The proposed approach for SOA testing has been
implemented in an Eclipse-based tool named
BPELTester. With a BPEL process and related
artifacts (e.g., WSDL and XSD) as input, this tool can
generate all the execution paths with the test-path
exploration function. On the other hand, with test
cases (directly designed from business require-
ments) run and traces logged, or traces collected
from real production environments, BPELTester can
analyze the traces and map the contained transac-
tions to the execution paths of the business process
with the trace analysis function. When the business
468 LI ET AL.
process definition or the service is changed,
BPELTester can identify the changed parts and the
impacts to test cases.
Tool architecture
Figure 5 shows the tool architecture. There are three
layers: user interface (UI), model, and, at the back
end, algorithm implementation.
The UI layer provides functions to import the
required BPEL, WSDL, XSD, and trace files, to
visualize the BPEL process, to display all paths and
their details, and to show the delta of two BPEL
processes and the impact to the test paths when a
process is changed. The UI details are shown in
Figure 6.
The model layer includes two common models
(among others) that are shared by the three
technical pieces. One is the BPEL model and the
ð2Þ other is the test-path model. In our tool implemen-
tation, the Eclipse model generator is used to
generate a hierarchy of Java classes from the XSD
specification of Web Services BPEL (WS-BPEL).
These classes represent the abstract syntax of the
language and parse the BPEL program to a set of
instances of those classes.
There are three back-end algorithm implementation
modules. Test path exploration and trace analysis
transform the BPEL model to the required interme-
diary model and follow the described approaches to
generate test paths. Regression test selection ana-
lyzes the two BPEL models to obtain the delta and
compares the two sets of test-path models to classify
them into the four categories enumerated in the
section ‘‘Test-path classification.’’
Graphical interface
The BPELTester graphical interface offers eight
views. The package explorer view lists all the
created projects and their files. The process editor
view visualizes the process and highlights the
selected path and the delta between the old and new
changed process in regression test selection. There is
a path outline view that lists all paths for the process
displayed in the process graph view. When users
select a path in the outline view, three views (path
attribute, branch, and coverage) show the detail
information of this path. The message view shows
the message content of the selected BPEL activity.
IBM SYSTEMS JOURNAL, VOL 47, NO 3, 2008
User interface Model Alogrithm implementation

Test-path exploration
Package explorer Selected WID BFG
artifacts Transform Generate
New views

Path attribute BPEL/WSDL/XSD

Branch
Trace Trace analysis
Coverage Forest
Transform Correlate

Message
Test paths Trace model
Parse
Path comparison

Regression BPEL changes Regression test selection

Compare BPEL/WSDL Compare paths

Path outline
Path changes
Process editor
Classify test cases

Figure 5
BPELTester architecture

Finally, the regression view shows the result of the
test-path classification after the process is changed.
PRELIMINARY EXPERIMENTAL RESULTS
BPELTester has been applied in BPEL processes of
varying complexity measured roughly by the total
number of activities and branches. These processes
are from various industry projects. Here we take 12
processes whose activity numbers are smaller than
100 and, in Figure 7, show their complexity and the
test-path distribution in different categories. Four
categories are listed and compared: all paths,
feasible paths, branch coverage paths, and service
coverage paths. Branch coverage paths constitute a
subset of feasible paths, which constitute a subset of
all paths. Service coverage paths are selected from
feasible paths.
Some processes use advanced BPEL features of fault
handling, and the implemented test-generation
algorithm has proved that it supports these features
very successfully. Figure 7 shows the following
points:
For most processes (9 out of a total of 12), feasible
paths account for a large portion of all paths that are
found by exhaustive searching. However, these are
mostly small processes whose branch number is less
than 30. For big processes whose branch number
exceeds 50, the chance is that a significant portion of
the test paths will be infeasible. This fact is
illustrated by process 12, which has 1301 paths in
total, but only 205 feasible paths. We need to
perform more experiments to identify the charac-
teristics of big processes. Also, as the test-generation
time is linear to the number of test paths searched,
this tells us that a more efficient generation
algorithm is needed to avoid the performance
problem for big processes. In fact, we tried a BPEL
process with more than 100,000 paths but only a
tiny portion of them were feasible. With the original
6
algorithm, test generation for this process reported
that it was out of memory before completion. Via the
branch-cut algorithm, the infeasible paths can be
removed as early as possible during the searching
procedure, and it requires only several minutes to
complete.
It is a clear fact that the branch coverage path
number is very small for all the processes, including
the complex ones. These paths are typically the
longest ones in the process. We think that branch

IBM SYSTEMS JOURNAL, VOL 47, NO 3, 2008 LI ET AL. 469

 Figure 6
BPELTester user interface
coverage is not adequate for business-process
testing as it cannot check the interactions among a
lot of upstream and downstream branches. Howev-
er, if the user wants to achieve only the branch
coverage, BPELTester can help filter out a minimal
number of test paths for the branch coverage goal. It
is a similar case for service coverage.
Test-generation time is another important metric to
evaluate a test-generation tool. Based on the
preliminary result collected from more than 10 BPEL
processes whose activity and branch numbers were
all under 100 (the upper bound was 88 for activities
and 73 for branches), the time maximum is 266.844
470 LI ET AL.
seconds (less than 5 minutes) on a desktop PC with
an Intel Pentium** processor running at 2.79 GHz
with 1.98 GB of memory. However, we cannot make
any deterministic statement on the relationship
between test-generation time and process complex-
ity. Among the many factors are: activity number,
branch number, branching structure, condition
number and complexity, and data calculation
number and complexity. More experiments are
needed to establish a clear relationship.
Transformation to execution formats
The resulting test paths can be mapped to three
kinds of test cases.
IBM SYSTEMS JOURNAL, VOL 47, NO 3, 2008
1. BPEL process unit test cases—The goal is to test a 100
single process in isolation from its dependent
services. This type is supported in the WID 80
integration test client now and can be categorized

Path percentage
as ‘‘service component testing.’’ 60
2. Service integration and composition testing—The
goal is to test the service composition capability 40
based on business scenarios. This type is now
supported in the IBM Rational Tester for SOA
20
Quality and can be categorized as ‘‘service
functional testing.’’
0
3. Business process end-to-end testing—The goal is Process 1 2 3 4 5 6 7 8 9 10 11 12
to test the end-to-end business process. This tests
the business processes and their dependent Activities 22 16 15 51 18 32 33 41 53 88 32 78
services. It is preferred that internal interactions
Branches 4 12 8 8 6 19 20 34 28 50 54 73
are checked following a gray-box testing style.
This type can be done from the GUI level or at the Service 1 1 1 2 1 2 1 2 2 3 1 2

service-invocation level, as supported by the IBM Branch 2 3 2 2 1 7 4 6 4 3 5 13

Rational Functional Tester. Feasible 2 5 4 5 2 12 12 17 10 264 481 205
All 3 6 4 5 6 17 29 29 46 312 490 1301
We have implemented transformation types 1 and 3,
Service coverage paths
and transformation is ongoing for type 2. Branch coverage paths
Feasible paths
All paths
CONCLUSIONS AND FUTURE WORK
As more enterprises are implementing SOA, some Figure 7
challenges of testing SOA systems are emerging. Test-path distribution in categories
Current black-box GUI-level testing is not sufficient
to deal with the new requirements arising from
several SOA features. This paper proposed a gray- ness-level process models (e.g., models supported
16
box testing approach that leverages BPEL business in IBM WebSphere Business Modeler ) to gener-
processes and the underlying SOA-layered architec- ate business-level test cases, and this has been put
ture to better test SOA solutions. This approach has into our research plan.
three key enablers: test-path exploration, trace  The trace analysis should also go beyond the
analysis, and regression test selection. analysis of BPEL execution traces to end-to-end
business-transaction tracking and generate test
These technologies have been implemented in IBM cases at various levels. This is part of our ongoing
BPELTester. Initial experimental results have vali- research, as is the regression test-selection tech-
dated our observations in SOA testing and also raise nique.
the following new requirements:

 IBM BPEL extensions, like Java snippets, need to

be handled in order to improve the preciseness of
the generated test paths. IBM BPEL allows one to
embed any Java program snippet that can describe
complex control logic and data manipulations. A
test-generation algorithm should be able to pro-
*Trademark, service mark, or registered trademark of
International Business Machines Corporation in the United
States, other countries, or both.
**Trademark, service mark, or registered trademark of Sun
Microsystems, Inc., or Intel Corporation in the United States,
other countries, or both.

cess BPEL and Java logic in an integrated manner.

 In addition to BPEL, other SOA architecture
information should be leveraged so that a broader
CITED REFERENCES
scope of SOA behavior can be tested. There is a 1. Solstice Software, Inc.,‘‘Assuring Quality Business Pro-
suggestion that we apply the technique to busi- cesses through SOA Testing,’’ White Paper, http://www.

IBM SYSTEMS JOURNAL, VOL 47, NO 3, 2008 LI ET AL. 471

 spectrum-systems.com/white_papers/wp_assuring_
quality_solstice.pdf.
2. Rational Tester for SOA Quality, IBM Corporation, http://
www-306.ibm.com/software/awdtools/tester/soa/index.
html.
3. J. Offutt and W. Xu, ‘‘Generating Test Cases for Web
Services Using Data Perturbation,’’ ACM SIGSOFT Soft-
ware Engineering Notes 29, No. 5, 1–10 (2004).
4. Z. J. Li, W. Sun, Z. B. Jiang, and X. Zhang, ‘‘BPEL4WS
Unit Resting: Framework and Implementation,’’ Proceed-
ings of the IEEE International Conference on Web Services,
Orlando, FL, 2005, pp. 103–110.
5. Z. J. Li and W. Sun, ‘‘BPEL-Unit: JUnit for BPEL
Processes,’’ Proceedings of the 4th International Confer-
ence on Service Oriented Computing, Chicago, 2006, pp.
415–426.
6. Y. Yuan, Z. J. Li, and W. Sun, ‘‘A Graph-Search Based
Approach to BPEL4WS Test Generation,’’ Proceedings of
the International Conference on Software Engineering
Advances, Tahiti, French Polynesia, 2006, pp. 14–23.
7. G. Canfora and M. Di Penta, ‘‘Testing Services and
Service-centric Systems: Challenges and Opportunities,’’
IT Professional 8, No. 2, 10–17 (2006).
8. W. De Pauw, M. Lei, E. Pring, L. Villard, M. Arnold, and
J. F. Morar, ‘‘Web Services Navigator: Visualizing the
Execution of Web Services,’’ IBM Systems Journal 44, No.
4, 821–845 (2005).
9. H. Liu, Z. J. Li, J. Zhu, and H. F. Tan, ‘‘Business Process
Regression Testing,’’ Proceedings of the International
Conference on Service Oriented Computing, Vienna,
Austria, 2007, pp. 157–168.
10. T. Katayama, E. Itoh, Z. Furukawa, and K. Ushjima,
‘‘Test-case Generation for Concurrent Programs with the
Testing Criteria Using Interaction Sequences,’’ Proceed-
ings of the 6th Asia Pacific Software Engineering
Conference, Takamatsu, Japan, 1999, pp. 590–597.
11. R.-D. Yang and C.-G. Chung, ‘‘Path Analysis Approach to
Concurrent Program Testing,’’ Information and Software
Technology 34, No. 1, 43–56 (1992).
12. R. N. Taylor, D. L. Levine, and C. D. Kelly, ‘‘Structural
Testing of Concurrent Programs,’’ IEEE Transactions on
Software Engineering 18, No. 3, 206–215 (1992).
13. F. Budinsky, D. Steinberg, E. Merks, R. Ellersick, and T. J.
Grose, Eclipse Modeling Framework, Addison-Wesley
Professional, Boston, 2003.
14. N. Eén and N. Sörensson, The MiniSat Page, http://
minisat.se/Links.html.
15. J. Forrest and R. Lougee-Heimer, CBC User Guide, http://
www.coin-or.org/Cbc/.
16. WebSphere Business Modeler, IBM Corporation, http://
www-306.ibm.com/software/integration/wbimodeler/.
Accepted for publication February 5, 2008.
Published online July 30, 2008.
Zhong Jie Li
IBM Research Division, IBM China Research Laboratory, 2F,
Diamond Building, ZGC Software Park, Haidian District,
Beijing, P.R.C. 100094 (lizhongj@cn.ibm.com). Dr. Li holds
B.S., M.S., and Ph.D. degrees in computer science from
Tsinghua University, Beijing. He joined the IBM Research
Division in 2004 and has worked in the Service Building
Technologies department. He has worked on applying
patterns for e-business (P4eb) and on SOA testing. His
472 LI ET AL.
interests include various aspects of software engineering,
automation of test generation, and Internet protocol testing.
Hua Fang Tan
IBM Research Division, IBM China Research Laboratory, 2F,
Diamond Building, ZGC Software Park, Haidian District,
Beijing, P.R.C. 100094 (tanhuaf@cn.ibm.com). Dr. Tan holds
a B.S. degree in computer science from the Beijing Institute of
Technology, China, and a Ph.D. degree in distributed
computing from the Institute of Computing Technology, China
Academy of Science. She joined the IBM Research Division in
2005 and has worked on applying runtime trace to support
and enhance SOA testing. Her interests include runtime
monitoring, dynamical analysis, and selective regression
testing.
He Hui Liu
IBM Research Division, IBM China Research Laboratory, 2F,
Diamond Building, ZGC Software Park, Haidian District,
Beijing, P.R.C. 100094 (hehuiliu@cn.ibm.com). Mr. Liu holds
B.S and M.S degrees in computer science from Xidian
University. He joined the IBM Research Division in 2006 and
has been working on SOA testing research.
Jun Zhu
IBM Research Division, IBM China Research Laboratory, 2F,
Diamond Building, ZGC Software Park, Haidian District,
Beijing, P.R.C. 100094 (zhujun@cn.ibm.com). Mr. Zhu holds
B.S. and M.S. degrees in computer science from Shanghai
JiaoTong University, China. He is a Ph.D. candidate in Peking
University. He works on technologies to provide Web-
delivered business services to small and medium business. His
current research interests include software and solution
engineering, SOA and modeling, and testing and industry
solutions. He is a member of the ACM.
Naomi M. Mitsumori
IBM Global Services, 4400 North First Street, Suite 100,
San Jose, CA 95134 (naomi@us.ibm.com). Ms. Mitsumori
holds a B.S. degree in chemical engineering from the
University of California, Berkeley, and an M.S. degree in
computer science from George Mason University. She has
worked with internal and external clients on projects ranging
from life-cycle testing (specializing in test design, automation,
and performance) to database application design and
programming. She designed the technical infrastructure of the
IBM Certification Test Center and the IBM Common
Development Test Laboratory and defined and implemented
the IBM Business Transformation and Chief Information Office
Testing Standards and Guidelines. Additionally, she has
established and trained test engineers in China, India, Mexico,
Japan, Australia, and North America. Currently, as an IBM
Distinguished Engineer and the Application Services CTO of
Test, her activities include the definition and rollout of the
IBM testing services offerings with a global IBM team. &
IBM SYSTEMS JOURNAL, VOL 47, NO 3, 2008