See discussions, stats, and author profiles for this publication at: https://www.researchgate.

net/publication/261128348

Testing as a Service (TaaS) on Clouds

Conference Paper · March 2013

DOI: 10.1109/SOSE.2013.66

CITATIONS READS
30 3,447

4 authors, including:

Jerry Gao Xiaoying Bai

San Jose State University Tsinghua University
218 PUBLICATIONS   2,841 CITATIONS    89 PUBLICATIONS   1,679 CITATIONS   

SEE PROFILE SEE PROFILE

Wei-Tek Tsai
Arizona State University
462 PUBLICATIONS   7,179 CITATIONS   

SEE PROFILE

Some of the authors of this publication are also working on these related projects:

Mobile TaaS View project

Component Testing View project

All content following this page was uploaded by Jerry Gao on 18 March 2015.

The user has requested enhancement of the downloaded file.

 2013 IEEE Seventh International Symposium on Service-Oriented System Engineering

Testing as a Service (TaaS) on Clouds

Jerry Gao1,2, Xiaoying Bai2, and Wei-Tek Tsai2,3

1
San Jose State University, USA, 2Tsinghua University, China, and 3Arizona State University, USA

Tadahiro Uehara
Fujitsu Laboratories Ltd., Japan
Contact email: jerry.gao@sjsu.edu,

ABSTRACT o What are the major differences between conventional software

Cloud computing leads an opportunity in offering testing as a
service (TaaS) for SaaS, clouds, and cloud-based applications. This
brings new business opportunities, challenges, and demands in
innovative service models, testing techniques, QoS standards, and
requirements. This paper provides a comprehensive tutorial on
testing as a service in a cloud environment. It answers the common
questions raised by engineers and managers, and provides clear
conceptual discussions about testing as a service (TaaS), including
its scope, objectives, motivations and values, distinct features,
required techniques, as well as testing environments. It not only
presents a classification of different types of testing services in
TaaS, but also offers a clear comparative view and perspectives
between conventional software testing service and cloud-based
testing as a service. In addition, it examines underlying issues,
challenges, and emergent needs.
testing service and TaaS in a cloud infrastructure?
o What are the special issues, challenges, and needs?
o What are the current TaaS practice, tools, and major players?
o What are the required TaaS infrastructures, frameworks, and
tools?
This paper is written to provide a tutorial on Testing as a
Service by covering its essential concepts and providing in-depth
discussion in different perspectives. The paper introduces and
clarifies testing services in a cloud environment, including its scope,
objectives, distinct requirements, features, and benefits. In addition,
the paper classifies various testing services for SaaS and cloud-
based applications, and addresses new issues, challenges, and
needs. Moreover, it presents a comparative view between
conventional software testing services and TaaS in a cloud
infrastructure. Furthermore, the paper also reviews several major
players and their products using a comparative view.

Keywords - cloud computing, cloud testing, testing as a service The rest of this paper is structured as follows. The next section
(TaaS), software testing, testing service and delivery. presents our view of testing as a service and related concepts,
including TaaS scope, objectives, motivations, and benefits,
1. INTRODUCTION different testing forms and environments. Section 3 is devoted to
the discussion of different types of testing services. It focuses on the
Cloud computing is changing the way of management and
new requirements and distinct features, and highlights the major
delivery of computing systems, networks, and software, but it also
differences between TaaS and conventional software testing
brings new business opportunities and challenges in IT industry and
services. Section 4 examines the primary issues, challenges, and
software industry. One of them is software testing. A new term,
needs, particularly in TaaS infrastructures. Section 5 reviews the
known as testing as a service (TaaS), is becoming a hot subject in
existing research work and emergent technologies.. Finally, the
different research communities, as well as cloud computing and IT
conclusion remarks are given in Section 7.
businesses.
TaaS in a cloud infrastructures is considered as a new business
2. UNDERSTANDING TESTING AS A SERVICE (TAAS)
and service model, in which a TaaS provider undertakes software
testing project activities and tasks for a under-test web-based 2.1. What is Testing as a Service (TaaS)?
software (or application system) in a cloud infrastructure, and
The term of “Testing as a Service (TaaS)” was initially introduced
delivers them as a service for customers.
and coined by Tieto (http://www.tieto.com/) in Denmark in 2009,
Although there are many published papers discussing cloud
and its TaaS solution was nominated by IBM for the Software
architectures, technologies, and models, design, and management,
Innovation Award 2009. Recently, TaaS has received lots of
cloud testing and TaaS are still new research subjects in both cloud
attention in both academic and industry communities due to its
computing community and software engineering community. As
advantage in its scalable testing environment, cost reduction, utility-
more internet-based applications are moved into the cloud world, based service models, and on-demand testing services.
and more SaaS applications and cloud-based systems developed,
there will be a strong testing service demand in cloud environments
In the past few years, there are a number of definitions about TaaS
and infrastructures. This implies that many test engineers and
in a cloud infrastructure. Here is one example.
quality assurance managers have to understand the new issues,
needs, and challenges in testing as a service on clouds.
“Testing-as-a-Service” delivers application quality
Although there is a number of published papers addressing
management solutions in a flexible service model that
TaaS subject [ ][ ][ ][ ], we are lack of a comprehensive tutorial on
accelerates the implementation of your quality center of
TaaS in cloud infrastructures, and need more clear understanding
excellence.” Testing-as-a-Service involves the on-demand test
and in-depth discussion. Here are some typical questions about
execution of well-defined suites of test material, generally on
TaaS.
an outsourced basis. (by Qutesys at http://www.qutesys.com/).
o What is testing as a service in the cloud world? What are its
scope, motivations, requirements, and unique features?
Other definitions emphasize different perspectives of TaaS. For
o What are the testing forms, environments, and provided
example, George Candea et al. in [7] identified three kinds of
diverse services?
testing service below.

978-0-7695-4944-6/12 $26.00 © 2012 IEEE 212

DOI 10.1109/SOSE.2013.66
- A public certification service, akin to Underwriters Labs, that
independently assesses the reliability, safety, and security of
software;
- A "home edition" on-demand testing service for consumers to
verify the programs they are about to install on their PC or
mobile device; and
- A "programmer's sidekick" enabling developers to thoroughly
and promptly test their code with minimal upfront resource
investment.
As pointed out by Tauhida Parveen and Scott Tilley in [4], “TaaS is
automated software testing as a cloud-based service”. This suggests
that the focus of test automation is not only test automation tools
and solutions, but automatic test services. In [14], the authors view
Testing-as-a-Service as a new model to provide testing capabilities
to end users.
In [21], we defined testing as a service (TaaS) in a cloud
infrastructure as follows.
o TaaS provides an on-demand service model for software
testing, in which on-demand test requests in software
validation processes are processed in a cloud-based scalable
testing environment based on pre-defined service-level-
agreements (SLAs).
o TaaS on clouds offers a new business model for software
testing based on utility billing. This allows end users to obtain
different types of testing services using the pay-as-you-test
model to achieve cost-sharing and cost-reduction.
2.2. Why is TaaS Important?
Many IT and software vendors and businesses like to find out why
TaaS is important, and where and when they need TaaS. Here is a
list of primary reasons.
1. Cost-reduction and resource-sharing in test processes
TaaS in a cloud infrastructure enables IT and SaaS vendors to
set up a virtual testing laboratory on a private (or public) cloud
without purchasing computing resources and testing resources
(i.e. test tool licenses). This reduces the front costs and
increases resource-sharing and utilization. For a large IT
company, the cloud-based TaaS provides a cost-effective way
to support many production lines which require diverse
computing resources and test tools.
2. Scalable test environments with virtualization [18]
Due to the elastic nature of clouds, cloud-based TaaS are able
to provide a scalable test environment for both online
applications and SaaS applications through automatic
provision and de-provision based on virtual and physical
computing resources. This feature addresses the strong demand
of SaaS scalability measurement and performance/load testing
by supporting large-scale test simulation and execution [19].
3. On-demand automated testing service in 365/7/24 [4][21]
TaaS is able to provide on-demand automated testing services
to respond diverse online test services from IT and SaaS
vendors at anytime and anywhere.
4. Pay as you testing at any time
Unlike conventional software testing services, in TaaS,
customers and clients are charged for their received
services based on a pre-defined utility model and cost
metrics. They will be charged in a pay-as-you-test
approach. This approach provides more flexibility for
213
TaaS customers to meet dynamic business and service
needs.
5. Multi-tenant based testing services [20]
Since multi-tenancy is a special feature of SaaS
applications, TaaS must support multi-tenant based
testing services according to the given QoS requirements
and service level agreements in functions, performance,
and security.
6. Quality certification by third parties
Easily leverage scalable cloud system infrastructure to
test and evaluate system (SaaS/Cloud/Application)
performance and scalability.
According to [13], IBM reported the experience on cloud
testing in small business division, where a flexible and cost-efficient
cloud-based development and testing environment is implemented,
and cloud testing has demonstrated the following major benefits.
• Reduce its capital and licensing expenses as much as 50% to
75% using virtualized resources.
• Reduce operating and labor costs as much as 30% to 50% by
automating development and testing resource provisioning and
configuration.
• Shorten its development and testing setup time from weeks to
minutes.
• Improve product quality and reduce the detected defects by as
much as 15% to 30%.
• Help to accelerate cloud computing initiatives with
IBMCloudBurst™ implemented through QuickStart services.
Testing Services for
TaaS over Applications or SaaS
Clouds Over Clouds
TaaS on Testing Service for
Cloud-Based Software
Clouds
(or SaaS) on Clouds
TaaS for Web Testing Web-
App.on Based
Clouds Applications on
Clouds
Figure 1. Different Levels of Testing as a Service (TaaS)
2.3 Testing Environments and Forms Supporting TaaS
There are three different forms of testing as service (TaaS) in a
cloud environment. Each of them has different focuses and
objectives.
o TaaS for web-based software on clouds – In this form, web-
based software is deployed on a cloud, and validated using the
provided test services given by TaaS vendors. The major
objective here is to take the advantage of large-scale test
simulations and elastic computing resources on a cloud.
o TaaS on clouds – In this form, cloud-based applications and
SaaS systems are deployed, executed, and validated on a cloud
for software validation on a cloud infrastructure or a cloud
 platform. Unlike the previous form, TaaS on clouds must
address and validate the scalability and multi-tenancy of SaaS
systems. In addition, a SaaS system usually is developed based
on a specific cloud’s technologies and its services, SaaS
connectivity protocols and APIs must be validated too.
o TaaS over clouds – In this form, SaaS applications crossing
hybrid clouds are deployed and validated based on different
clouds (such as private clouds, and public clouds). In a hybrid
cloud infrastructure, diverse on-demand test services are
provided and delivered by a TaaS vendor. A typical SaaS
system crossing multiple clouds usually is structured with
components, service software, and servers deployed crossing
over several clouds. These components and severs are
communicated, connected, and interacted following different
connectivity protocols.
Figure 2 Different Test Environments [18]
As shown in Figure 2, there are three types of cloud-based test
environments [18].
• A private/public cloud test environment, in which vendors
deploy SaaS applications SaaS in a private (or public) cloud
to validate their quality.
• A cloud-based enterprise test environment, in which
application vendors deploy web-based applications in a cloud
to validate their quality in a cloud infrastrutcure. As shown in
Figure 3(b), this cloud-based enterprise test environment is
useful to test cloud-based enterprise service programs where
• A hybrid cloud test environment, in which vendors deploy
cloud-based applications on a hybrid cloud infrastructure to
check their quality.
2.4. The Scope and Classification of Testing as a Service
Figure 3 shows the scope of testing as a service (TaaS), which
includes the following eight types of testing services.
• Scalable test environment service – This allows clients to
configure and set up their desirable virtual test laboratory
environment by requesting and selecting diverse computing
resources and licensed tools for software testing.
• Multi-tenant test modeling and adequacy service – This offers
test modeling tools and adequate test coverage analysis
techniques to support engineers to conduct SaaS testing based
214
on multi-tenanted SaaS requirements in functions, security,
and performance.
• Digital test management service – This supports test project
management, test-ware management, and detected problem
management. Here, test-ware includes test suites with test
cases, test data and scripts, and test results. All these test-wares
are managed and grouped based on the types of testing. In
addition, they can be uploaded, downloaded, transferred, and
migrated upon user requests.
• On-demand automated test and control service - This offers
on-demand automated test execution and control services
according to requested testing task schedules for a test project.
• Test solution integration and composition service – This
offers and delivers the diverse systematic test solutions based
on well-defined test models, methods and algorithms to
achieve well-known test criteria. For any selected test solution,
TaaS vendors should provide solution adoption, selection,
generation, integration, and composition services. A test
solution here refers to test modeling tools, test methods and
algorithms, test-ware generation and management technology.
• Test tracking and monitor service – This allows test
engineers to track and monitor diverse program behaviors at
different levels in/on/over clouds during a test process.
• Large-scale test simulation service - This supports test
engineers to set up the desirable a test simulation environment
with selected facilitates (such as tools) to fulfill different types
of simulation needs. Typical types of simulation needs include:
communication traffic and load simulation, SaaS/cloud
connectivity simulation, GUI-based user scenario simulation,
and domain process-based simulation.
• TaaS contracting and billing service – This provides end
users with TaaS service management in accounts, contracts,
and billing reports. With this service, TaaS vendors are able to
offer customers with selectable testing service contracts based
on pre-defined pricing models and billing metrics.
Scalable Test Multi-tenant Test
Test Tracking Modeling and
Environment Service
& Monitoring Adequacy Service
Digital Test
On-Demand
Management Testing as a Automated
Service Service (TaaS) Test Service
Large-Scale Test Test Solution
Simulation Service Integration
TaaS Contracting & Composition
and Billing Service
Figure 3 The Scope of Testing as a Service (TaaS)
In short, testing as a service (TaaS) in a cloud infrastructure refers
to the delivery of diverse on-demand automated test executions and
control services, and test project management services to clients
based on well-defined service level agreements (SLAs) and pre-
defined QoS standards. by leveraging cloud technologies and
solutions. It has four major objectives. Figure 4 shows a detailed
classification of TaaS services.
 Figure 4 A Classification of Different TaaS Services
2.5. What are New Features in TaaS?
Unlike conventional web-based software testing, TaaS on a cloud
infrastructure has several unique features and distinct requirements.
Several published papers have discussed a number of new features
and requirements in different perspectives. Here are some
examples:
• Cloud-based testing environment [18][20]], in which diverse
computing resources and test-wares can be selected,
configured, and provisioned (or de-provisioned) dynamically.
• SLA-based software testing [18], in which well-defined QoS
service-level agreements are the essential base for testing
services.
• Utility billing and price models [18][20], in which the pre-
defined utility model, billing approach, and cost metrics are
used for pay-as-you-test.
• On-demand testing service [18][20], in which various on-
demand service requests will be issued and processed in a
testing service SaaS from a TaaS vendor.
• Large-scale test simulation [18][19][20], in which large-scale
test simulation is required to support systeem-level testing in
system scalability, performance, load, reliability, and
availability.
The detailed discussions can be found in the related references.
Here we discussed the other new characteristics in testing as a
service (TaaS).
• Both computing resources and testin ng resources are
commodes – Here the testing resources refer to various well-
defined testing techniques, tools, and solutions which are
215
provided by test solution providers as commodes, and they are
shared and utilized by end users with service charges.
• Testing SaaS actually is testing SaaS with its databases. –
Since SaaS programs commonly support multi-tenancy at
different level, this suggests that any SaaS application must be
validated for various tenants with their supported data
contexts. In other words, for each tenant, SaaS software must
be tested with its own meta-database scheme, and its dynamic
data store with configured and customized features and data.
• Testing for multi-tenancy of SaaS applications – The multi-
tenancy of SaaS applications demands TaaS vendors to offer
tenant-based validation services to support function testing,
QoS validation, as well as security assurance based on tenant-
based requirements and service-level agreements.
• Intelligent on-demand test automation services – On-demand
testing is one of the major services in TaaS. Although this
provides a distinct advantage to support dynamic test
operations and executions to deal with large-scale auto-testing
and simulation, it requires more intelligent control in automatic
test scenario selection, test script control, and bug reporting
since auto-testing must cope with testing failures, program
exceptions and bugs.
• Continuous validation and re-testing – Supporting online
access of SaaS applications with multi-tenancy in 365/24/7
requires continuous validation and regression test techniques.
This suggests that system function validation and QoS testing
need to be performed while SaaS operates.
• Test solution integration and comp m osition service – An ideal
TaaS p rovider should be able to allow engineers (or users) to
composite and integrate the selected test solutions and tools
from a TaaS environment to form a desirable test environment.
Hence, test solution integration and composition becomes a
very important service for TaaS.
3. TAAS COMMUNITY, PROCESS, AND ROAD MAP b) Test solution vendors – they provide diverse test tools, test
project and process management solutions.
Testing as a service (TaaS) refers to providing static/dynamic on- c) Test-ware vendors – they use in-house, free-lance, or third-
demand testing services in/on/over clouds for the third-party is at party engineers to provide test-ware design and generation
any time and all time (365/7/24). One of the primary objectives is to service for a contracted test project, including test scripts, test
reduce the IT budget of businesses to focus on their core businesses cases, documents.
by outsource software testing tasks to a third party using TaaS d) Testing service vendors - they provide, manage, and control
service model. TaaS involves the on-demand test execution of well- TaaS infrastructures, platforms, and deliver testing services
defined suites of test material, generally on an outsourced basis. The based on provided resources, test tools, and test-wares from
execution can be performed either on client site or remotely from other vendors.
the outsourced providers test lab/facilities. As shown in Figure 5,
the work-flow of TaaS includes the following major TaaS service Vendors may set up three kinds of TaaS clouds.
capabilities. - Private TaaS cloud, which provides and deliver diverse
dedicated testing services on a private cloud to limited users
for private businesses and enterprises,
- Public TaaS cloud, which provides and delivers various
QoS REQ testing services on a public cloud to public users for software
Manageme testing and quality engineering education, training, and
certification purposes.
TaaS Pricing Test Envir. - Community TaaS cloud - which provides and delivers various
and Billing Service testing services on a community cloud. A typical example will
be a research TaaS community cloud supporting for TaaS
TaaS Process research, result validation and case studies.
Management
Tracking & Test Solution Multi-tenant Test Adequacy
Monitor Service Service Service

On-Demand Test On-Demand Automated

Test Service Simulation Testing Service

Figure 5 A TaaS Workflow Example [18]

Large-Scale Test Simulation
Service

Digital Test Management

TaaS
Testing as a Service
provider
Service (TaaS)
TaaS end-users
(Test Engineer)
Scalable Test Environment
Service

Figure 7 A Road Map to Achieve Test as a Service on Clouds

Test-Ware TaaS end-users
Provider (QoS Engineer)
A Road Map for Testing as a Service (TaaS)
Test Solution
TaaS end-users
Provider As shown in Figure 7, a road map to achieve TaaS maturity
(Test Manager)
includes the following steps:
• Step #1: Scalable test environment service – The objective of
Figure 6 User Perspectives of A TaaS Cloud for SaaS
the first step is to support the dynamic creation and
deployment of a scalable test environment by leveraging
As shown in Figure 6, vendors can set a TaaS community as a cloud computing resources on a selected cloud infrastructure. Here,
to deliver various testing services for different types of users. There TaaS users are offered with selection and configuration
are three types of TaaS client users. They are: (a) test managers, (b) services of different computing platforms and resources to
test engineers or QA engineers, and (c) end users of SaaS. Each allow them to form desirable cloud-based test environments.
TaaS cloud involves three types of vendors. Both licensed and free test tools and solutions are also
a) Cloud vendors – they provide the required networks, such as provided as selective resources for clients.
computing and storage resources, and internet infrastructures.

216
 TABLE 1 A Comparison between TaaS and Conventional Software Testing Services

Service Types Conventional Software Testing Services Expected TaaS Services from Vendors

Primary Objectives - Labor intensive testing services to reduce engineering - Solution-based services to reduce resource and testing
costs costs for test projects
- Validate software quality according to given - Assure SaaS quality against expected system function
requirements requirements and SLAs
Testing Focuses - System functions and system QoS attributes, such as - SaaS service functions, and special features, such as
performances, reliability, availability, vertical multi-tenancy, customization, and configurability
scalability, security, and so on. - SaaS QoS attributes, such as scalability and reliability
- SaaS connectivity, interoperability and portability
QoS requirement - Apply ad-hoc or enterprise-oriented QoS requirements, - Modeling QoS requirement with selectable test models
modeling and standards, and specifications with QoS modeling services
specification service - Perform ad-hoc test modeling analysis
Service business - A project-based business model with testing contracts - A service-based transaction model based on a defined
cost model and service-level-agreement
Service delivery - Manual service delivery following a pre-defined contract - On-demand dynamic test service delivery in 356/7/24

Test process support - Manually perform a pre-selected ad-hoc or enterprise- - Select, apply, and perform a well-defined test process
oriented test process with a selected test process tool offered by a TaaS vendor and supported in a tool
Testing Execution - Offline testing in a test lab. before product delivery - Offline testing in a private cloud-based test environment
- On-demand test execution in a cloud-based virtual test
environment
- Continuous testing for SaaS in/on/over clouds
Test modeling - Perform ad-hoc test modeling and analysis - Perform diverse test modeling & analysis using tools
Test edition and - Manual and semi-automatic test edition and generation - Semi-automatic test edition and generation using
generation using licensed or in-house tools selective test generation tools based on desirable test
models
Test environment - Manually configure and set-up a test laboratory and - Systematic configure and set-up scalable test
environment with physical computing resources environment based on virtual and physical computing
resources
- Dynamic provision and de-provision of computing
resources
Test tool - Manual test tool configuration and deployment of - On-demand selection, configuration and deployment of
limited licenses highly shared tools provided by TaaS vendors
Test composition and - Ad-hoc test editing, integration, and composition using - Semi-automatic test edition, integration, and
integration limited tools composition using selective tools
Testing Techniques - Apply selected well-known white-box and black-box - All existing testing techniques for software
testing techniques at the component level (or unit level) - Required innovative continuous testing techniques
and the system level - New testing solutions to deal with SaaS multi-tenancy
and elasticity
Test-ware - Manual management of test scripts and test suites (test - Systematical test-ware service management for test
management cases/data) following enterprise-based templates using a scripts/test suites (test cases/data) using selective tools
selected test tool - Test suite migration, download and upload services
Software tracking & - Ad-hoc or enterprise-oriented application-based program - Selective, customizable and configurable tracking and
monitoring tracking and monitoring for application behaviors, monitoring services for SaaS, including end-to-end
messages, and API interactions transactions, connectivity, data accesses, and user
operations in a cloud infrastructure
Test simulation - Generate and perform limited test simulation in a pre- - Perform and apply customizable and configurable large-
fixed test environment using selected simulation tools scale data load, traffic load, and user accesses with test
simulation services
Solution integration - Ad-hoc manual integration and composition - Systematic integration and composition
and composition
Test service cost - Computing hardware, tools, and engineering costs - Based on a pre-defined service-level-agreement (SLA)
- Pay-as-you-test service costs
Test adequacy analysis - Apply selected test adequacy and measure test coverage - Apply selective and desirable test adequacy criteria and
and measurement analysis using a license (or open source) receive the coverage analysis with pre-selected
solutions

217
• Step #2: Digital test management service – The
second step focuses on offering essential test
management services for clients, including test project
management, test process management, and test-ware
management solutions and services for TaaS clients. In
addition to conventional test management solutions,
TaaS vendors must provide test migration on/over
clouds, test- ware uploading and downloading on
clouds.
• Step #3: Large-scale test simulation service – In this
step, TaaS vendors must offer large-scale test
simulation, management, and control service for SaaS
clients and test engineers to support them in SaaS
performance validation, QoS evaluation, and scalability
analysis.
• Step #4: On-demand automated test service – The
fourth step needs to deal with test automation needs by
offering on-demand automatic test execution, test
control, test failure processing, and bug reporting. To
achieve this, both pre-scheduled and dynamic test
requests should be processed timely and responded
with quality results.
• Step #5: Multi-tenanted test adequacy services – The
final step pays a special attention to deliver automatic
test adequacy analysis for each SaaS tenant. With this
service, TaaS is able to dynamically monitor and
evaluate the quality of its underlying test services for
each testing projects.
4. CONVENTIONAL TESTING SERVICE V.S.
TAAS
Table 1 provides a comparative view from service
perspectives between conventional software testing services
and testing as a service on clouds. The distinct differences
are summarized as follows.
- Primary objectives – Conventional testing service is
labor intensive service. Its primary objective is to
reduce engineering costs in a test project. However, the
major objective of TaaS in cloud is to reduce and share
the cost of computing resources and speed-up the
project cycle by leveraging a cloud infrastructure to
provide a scalable test environment to support the need
in testing SaaS scalability.
- Testing objectives and focuses - Similar to
conventional software testing, software program
structures, functions, behaviors, and interfaces, and
QoS requirements must be validated. However, SaaS
testing has new focuses on SaaS elasticity, multi-
tenancy, connectivity, configurability, customization,
and interoperability. Unlike conventional software
testing, SaaS testing must be conducted to confirm the
posted SLAs from its vendor.
- Testing environment and execution - Instead of using
a fixed testing environment, SaaS testing requires a
cloud-based scalable testing environment with
provisioned computing, and diverse selectable and
configurable test tools. As we known, most test
executions in software testing are carried out in an
offline mode. However, test executions in TaaS should
be performed in both offline and online modes.
- Testing techniques - Although current testing
techniques are applicable to TaaS, engineers require
new testing methods to cope with special SaaS
218
features, such as multi-tenancy, horizontal scalability,
customization, and connectivity. In addition, effective
SaaS testing requires innovative continuous testing
approaches and solutions to support SaaS frequent
upgrading.
- Systematic tool composition and integration – In
current practice, engineers need to manually integrate
various in-house tools with existing licensed tools
together to form a desirable system test bed or/and
environment. However, in TaaS, users expect TaaS
vendors provide some easy integration and
composition solutions and tools to support engineering
operations for on-demand testing tasks as well as
continuous testing operations.
- Large-scale test simulation on clouds – Today, it is
common practice to construct some test simulators to
support software testing at the system level. Due to
SaaS scalability its test simulation must be performed
at the large-scale. This implies that a well-defined
cloud-based test simulation platform and infrastructure
with tools are required.
- Service delivery and business model – Unlike current
practice, test services are delivered as a labor-intensive
engineering service based on a project contract
between a client and a service provider. In TaaS,
different on-demand service solutions are provided by
vendors to support various engineering activities in
SaaS testing. In TaaS, all services are delivered and
charged as service transactions based on a pre-defined
pay-as-you-test billing model and service-level-
agreements (SLAs).
- Test supporting services – Comparing with
conventional testing service, TaaS could provide a rich
set of testing services to support engineers. These
services can be classified into the following classes:
• Class #1 – Test environment services – This
includes test environment construction and
configuration, tool selection, configuration and
deployment, test solution composition and
integration.
• Class #2 - Pre-testing services - This includes
QoS modeling, test planning, modeling and
analysis, and contracting.
• Class #3 – Test management services - This
includes test process support, test-ware
management, test coverage analysis and
measurement.
• Class #4 – Test supporting services – This
includes test editing and generation, on-demand
test execution, program tracking and monitoring,
and test simulation.
5. ISSUES, CHALLENGES AND NEEDS
Due to the new features of SaaS, there are a number of
emergent needs in testing as a service (TaaS). In [18][20],
we have identified a number of special issues, needs and
challenges in cloud testing.
Here we highlight some primary issues and challenges, as
well as essential needs.
 shows that different test projects may be supported by a
virtual test laboratory and accessed by different user groups,
including test and Q&A engineers, as well as test project
managers, and users. Comparing with a conventional test
laboratory, a virtual test laboratory has several distinct
advantages:
- On-demand establishment and deployment
- Dynamic provision of computing resources and tools
- Highly scalable in horizontal and vertical configured
- Highly resource sharing and tool license sharing

However, with our experience in [21], we found a couple of

challenges and issues in computing resource allocation and
management.

Issue #1: What are the cost-effective resource allocations

and management strategies that support on-demanded
virtual test laboratory construction and deployment in a
TaaS infrastructure?

The complexity of this issue is caused by dynamic provision

Figure 8. A Cloud-Based Infrastructure for SaaS Virtual requirements for creating and deploying diverse virtual test
Test Laboratory
laboratories with dynamic computing resource needs and
tool license demands. Here are some typical factors to be
5.1. Challenges and Needs in Building Cloud-Based
TaaS clouds concerned.
- Both static and dynamic test project schedules must be
There are a number of issues, challenges, and needs in considered
building cloud-based TaaS clouds. This section discusses - If third-party clouds are used, then the usage cost and
and summaries some important ones. business model becomes a critical factor in resource
allocation and management
(a) Issues and needs for scalable test environments - Both computing resources and licensed tools need to be
managed and allocated
- Dynamic fulfillment of computing resource needs for
each virtual test laboratory
CP
EC Instance - Market-driven policy and cost-based approach must be
System
Memor Virtual adopted and balanced.
Machine Admin.

This suggests that there is a need for dynamic cost-effective

Test Lab. methods for virtual test laboratory construction, deployment
Storag and resource allocation and management in TaaS.
Project
Manager
(b) Issues and needs for well-defined TaaS
Softwar infrastructure
Virtual Virtual
Test Lab. Test Lab. Issue #2: What is a well-defined cloud-based TaaS
infrastructure for SaaS validation and evaluation?
Test Tools User
Clearly, there are different ways to design cloud-based TaaS
Test Project infrastructure. The key requirements and design concerns
for a TaaS infrastructure can be listed here.
- Elasticity and scalability
- Standard interfaces and connections to test tools and
third-party solutions
- Cloud technology independency and standards
Figure 9. Different Users in SaaS Virtual Test Labortary - Easy connectivity to other clouds
- Easy SaaS deployment, execution and validation
As indicated in [21], there is a strong demand to provide a - Cost-driven design in TaaS infrastructure, for example,
cloud-based TaaS infrastructure by leveraging cloud design for energy saving
technologies to provide resource provision and de-provision
for virtual test laboratory in order to support SaaS QoS A typical TaaS infrastructure which includes three parts:
validation. - A SaaS test environment (or SaaS test platform),
As shown in Figure 8, a virtual test laboratory can be which is used to support the deployment, execution,
created or deleted with a dynamic client request to obtain and validation of an under-test SaaS system on a
necessary computing resources and tools to form a desirable selected cloud. In addition, it consists of a set of
test laboratory based on a cloud infrastructure. Figure 9

219
 supporting test frameworks, such as program tracking - How to deal with test failures or detected bugs during
framework, and test framework, and so on. large-scale test execution in an automatic manner?
- TaaS agent layer, which enables large-scale test During large-scale test execution, any test failure
simulation, tracking, and on-demand test automation. usually causes the stop of automatic test execution
- A cloud-based TaaS server which includes the because an incorrect operation outcome (or an
following components. inappropriate program update) always leads to
a) Business engine – it supports TaaS service incorrect program states and invalidated system
contracting, billing, and user account and conditions.
management.
Performance/Load
b) Tool manager - it configures and controls a set of Test Agent
tools supporting test and problem management as GUI Test
Simulation Connectivity
well as test process management for each test Agent Simulation
project.
Agent
c) Simulation engine – it supports large-scale test
simulation by working with different test
simulation agents, such as GUI test simulation SaaS
agent, and load test simulation agent.
d) Test tracking engine – it supports and controls
different test tracking capability for under-test
SaaS program behaviors and performance. SaaS Test Simulation Third-Party
e) On-demand test engine – it support on-demand System
test automation, including test selection,
scheduling, execution, reporting as well as test
failure processing.
TaaS-
f) Test repostory manager – it supports mult-tenant
Unit Test Simulation
based test-ware management, migration, and Engine
deployment. Multi-Tenant
Test DB
As the advance of cloud testing and TaaS technology, more
TaaS infrastructures are needed to support diverse needs
Figure 10. Test Simulation Infrastructure in TaaS
from different vendors, such as cloud vendors, SaaS
vendors, and TaaS vendors.
GUI Test
Test Simulation Needs in a TaaS cloud Tracking Agent
Performance/Load
To support effectively support SaaS testing on TaaS clouds, Test Tracking Agent
engineers need several types of test simulations. These
include: a) load simulation for interaction traffic messages; Connectivity
b) user access simulation; c) connectivity simulation; and d) Tracking
third-party system simulation. Figure 10 shows an example
of test simulation infrastructure in a TaaS infrastructure.

Test Tracking Needs in TaaS

SaaS

Similar to conventional software testing, supporting SaaS

validation in TaaS must provide well-defined SaaS program
tracking capability to support: a) error tracking, b) system Test Tracking
and user interaction tracking, c) system performance and Third-Party
QoS tracking, and d) SaaS connectivity tracking. Unlike System
conventional program tracking, software tracking in TaaS
must address SaaS multi-tenancy by supporting tenant-
oriented program tracking. Hence, a systematic multi-tenant
based program tracking solution should be provided in a TaaS -Tracking
TaaS infrastructure. Figure 11 shows an example of Engine
software tracking structure in a TaaS infrastructure. Multi-Tenant
Tracking Repository
On-demand automated testing issues and needs in TaaS

To support on-demand automated testing services, TaaS
vendors must provide both pre-scheduled and on-demand
automatic testing execution services. Since SaaS test
automation must deal with large-scale test simulation and
test execution, hence engineers must encountered two
critical issues:
Figure 11 A Software Tracking in a TaaS Infrastructure
Hence, engineers need intelligent test execution solutions
and innovative auto-test infrastructure to support and
process any test failures during automatic test execution.

220
This implies that automatic test scripts must be designed and
developed to make sure that automatic test scripts are auto
re-settable, auto-recoverable, and auto-clearable. In addition,
the auto test runner must provide some intelligent test
engine which controls and manages large-scale auto-test
execution.
- How to provide automatic test solutions to address
SaaS elastic scalability?
W.T. Tsai, et al. in [10] points out the importance of
scalability testing for SaaS systems on clouds.
Supporting test automation for SaaS program, TaaS
vendors must provide test automation solutions with
elastic scalability. This implies a TaaS infrastructure
must be designed to support scalable test automation,
control, and reporting. For example, an auto-test engine
must be able to deal with one or many SaaS instances
at the same time to process a large-scale auto tests.
- How to manage tenant-based test repository and
control tenant-oriented test execution to address SaaS
multi-tenancy?
Due to SaaS multi-tenancy, test design and
management must be done according to tenant-based
QoS requirements in functions, behaviors,
performance, and security. This demands TaaS vendors
to provide tenant-based test repository management
service, and support tenant-based test execution and
control.
5.2 Technical Challenges in TaaS
#1: Continuous validation and regression testing - The
multi-tenancy feature of SaaS brings new challenges and
requirements for SaaS testing on clouds. As indicated in
[20], engineers need to test the multi-tenancy of SaaS. In
[18], we pointed out the challenge of continuous validation
and regression testing for cloud-based applications.
Whenever software changes for SaaS due to bug-fixing,
feature enhancement, and adding new tenants, engineers
must re-test SaaS to confirm its quality. However, according
to [18] it is still an open question about how to conduct and
validate SaaS during software service time using a dynamic
approach.
#2: Testing Solution Integration and Composition - In a
TaaS infrastructure, engineers must deal with the integration
of different testing tools and solutions during runtime.
Clearly, current software research results did not provide
adequate automatic approaches to supporting test solution
integration and composition as a service due to the
following facts pointed in [18].
- There is a lack of well-defined quality standards for
SaaS connectivity protocols and interaction interfaces
to facilitate SaaS integration and connectivity
validation.
- There is a lack of cost-effective integration and
composition service solutions and frameworks to
facilitate test tool integration, assembly, and
composition in clouds, on clouds, and even over
clouds.
#3: Technology lock-in – This is a typical concern and issue
in building a cloud-based TaaS environment and
infrastructure for SaaS programs. Since many existing cloud
technologies used some enterprise-specific or proprietary
221
tools, languages and APIs, it is important for engineers to
consider how to avoid technology lock-in when building
TaaS infrastructures.
5.3. Business and Standard Challenges
• Lack of connectivity standards – Although many
existing SaaS systems and cloud technologies have
provided well-defined connectivity APIs, there is a
lack of well-defined standards in SaaS connectivity.
This causes more difficult to build a cloud-based TaaS
infrastructure for SaaS to adapt to diverse SaaS
connectivity interfaces and interactions.
• Lack of test tool standards – Although there are many
existing software test tools, there is a lack of tool
connectivity standards in software test automation.
This becomes a huddle in supporting tool integration
and composition in TaaS infrastructures.
• Lack of well-defined cost models and billing metrics –
Although a few TaaS of vendors have used certain
simple cost model and billing metrics for their
provided testing services, they are very primitive and
lack of well-defined billing metrics and theoretic
foundation. Hence, more well-defined cost models and
rational billing metrics are needed to support TaaS.
7. RELATED WORK
Since 2010, researchers began to pay attention to testing as
a service in a cloud environment. A number of published
papers have addressed this subject in different perspectives.
Several papers covered some concepts about testing
services. For example, Ashfague Ahmed [1] focuses some
basic concepts about software testing as service. In this
book, he explains how the normal functions of
manufacturing can be applied to commoditize the software
testing service to achieve consistent quality across all
software projects. Leo Vander Aalst in [8] presents his
definition, process, infrastructure and some experience
results on software testing as a service. In 2009, Y. Yang at
al. [11] discusses that software testing can be
conceptualized as a service rather than being viewed as a
sequential line of responsibility in software development.
In their view, TaaS has two key aspects: (1) a service to
developers, and (2) a service to end users. Leah Muthoni
Riungu, et al. in [3] reports their qualitative study to show
an increasing demand for software testing as an online
service on clouds. In this paper, authors present their
survey results based on the interviews with managers from
eleven organizations. In 2010, G. Candea and his co-
authors in [7] discuss three application forms for TaaS.
They are: a) a “programmer’s sidekick” enabling
developers to thoroughly and promptly test their code with
minimal upfront resource investment; b) a “home edition”
on-demand testing service for consumers. In 2011, a TaaS
workflow process is introduced in [18]. In addition, the
paper also reviews and presents the major players in testing
as a service and production lines.
There are a number of publications discussing TaaS tools
and solutions. For example, Yulian Liu at al. in [14][15]
presents a TaaS reference architecture based on ontology,
process automation and SOA techniques. The proposed
cloud-based TaaS framework consists of 5-layers: a) test
service tenant and contributor layer, b) test task
management layer, c) testing resource management layer,
d) test layer, and testing database layer. Their research
results focus on TaaS task management, scheduling, and
processing. In addition, Xiaoying Bai, et al. in [20] presents
a review on test service tools. The paper provides a
comprehensive review about recent research results on
testing service and tools. It covers several important issues
in TaaS. One of them is how design TaaS infrastructure,
test platforms, and environment. Moreover, it also covers
some testing service tools on clouds. For example, R.
Buyya, et al. in [5] introduces an extensible simulation
toolkit (known as CloudSim), which enables modeling and
simulation of Cloud computing environments for
computing resources in data centers. To address test
execution issues in cloud testing, L. Ciortea et al. in [17]
presents Cloud9, which is the first symbolic execution
engine that scales to large clusters of machines to support
automated testing of real software. Its major focus is how
to use symbolic execution to support on-demand software
testing service.
In addition, several published papers addressing
performance testing and scalability evaluation on clouds.
For instance, W. T. Tsai et al. in [10] presents their
insightful discussion on SaaS scalability testing including
its importance and issues. Jerry Gao, et al. in [19] discuss
SaaS performance testing and scalability on clouds. In this
paper, they present a graph-based model to define a set of
performance and scalability evaluation metrics for SaaS
systems on clouds. Moreover, a systematic cloud-based
SaaS performance validation approach is presented based
on Amazon EC2 cloud technology with a case study report.
In [21], Yong Chen et al. in [16] propose their scalability
testing and analysis system (called STAS), in which they
introduced a scalability metric (known as isospeed-e) to
assist system scalability measure and analysis..
8. CONCLOUSION AND FUTURE WORK
Testing as a service (TaaS) is becoming a hot research topic
in both cloud computing and software engineering research
communities. As the advance of cloud technology and
testing as services, more research results are needed to
address the open issues and challenges on TaaS
infrastructures, techniques, and automation solutions. More
innovative testing techniques and solutions, and QoS
standards are needed to support on-demand testing services
in a scalable cloud infrastructure, for example, SaaS testing
adequacy and standards for multi-tenancy.
This paper provides a comprehensive review and in-depth
tutorial on cloud-based TaaS for SaaS applications. It offers
essential tutorial concepts on TaaS definitions, scope,
motivations and benefits, as well as classified testing
services and test environments. In addition, it discusses in
details about TaaS requirements, distinct features, issues,
challenges, and needs. Moreover, it highlights the major
differences between conventional software testing and
cloud-based TaaS.
222
ACKNOWLEDGMENT
This project is sponsored U.S. National Science Foundation
project DUE 0942453 and National Science Foundation
China (No. 61073003), National Basic Research Program of
China (No. 2011CB302505), the Open Fund of the State
Key Laboratory of Software Development Environment
(No. SKLSDE-2009KF-2-0X), and Research Grant from
Fujitsu Laboratory.
REFERENCES
1. Ashfague Ahmed, Software Testing as a Service,
Auerbach Publications, 1 edition, September, 2009.
2. T. Vengattaraman, P. Dhavachelvan, R. Baskaran,
“Model of Cloud Based Application Environment for
Software Testing”, (IJCSIS) International Journal of
Computer Science and Information Security, Vol. 7,
No. 3, 2010.
3. Leah Muthoni Riungu, Ossi Taipale, Kari Smolander,
"Software Testing as an Online Service: Observations
from Practice," Third International Conference on
Software Testing, Verification, and Validation
Workshops (ICSTW), pp.418-423, 2010.
4. Tauhida Parveen, Scott Tilley, "When to Migrate
Software Testing to the Cloud?" Third International
Conference on Software Testing, Verification, and
Validation Workshops (ICSTW), pp.424-427, 2010.
5. R. Buyya, et al., “Modeling and simulation of scalable
Cloud computing environments and the CloudSim
toolkit: Challenges and opportunities”, Proceedings of
International Conference on the High Performance
Computing & Simulation (HPCS 09).
6. Liu Gu and Shing-Chi Chenug, “Constructing and
testing privacy-aware services in a cloud computing
environment: challenges and opportunities”
Proceedings of the First Asia-Pacific Symposium on
Internetware, Beijing, China, 2009.
7. George Candea, Stefan Bucur, and Zamfir Cristian,
“Automated Software Testing as a Service (TaaS)”,
2010. Proceedings of the 1st ACM Symposium on
Cloud Computing, 2010.
8. Leo.van der Aalst, “Software testing as a service
(STaaS),” Internet:
http://www.tmap.net/Images/Paper%20STaaS_tcm8-
47910.pdf, [May 06, 2009].
9. G. Goth, ““Googling” test practices? Web giant’s
culture encourages process improvement”, IEEE
Software, 25, 2, pp. 92-94, 2008.
10. W.T. Tsai, Yu Hang, and Qihong Shao, “Testing the
Scalability of SaaS Applications”, Proceedings of
IEEE International Conference on Service-Oriented
Computing and Applications (SOCA), 2011.
11. Y. Yang, C. Onita, J. Dhaliwal, X. Zhang,
“TESTQUAL: conceptualizing software testing as a
service,” Proc. 15th Americas conf. on information
systems, 6-9.08, San Francisco, California, USA,
paper 608, 2009.
12. N. Yigitbasi, et al., “C-Meter: A framework for
Performance Analysis for Computing clouds “,
Proceedings of 9th IEEE/ACM International
Symposium on Cluster Computing and the Grid
(CCGRID '09).
13. “IBM Smart Business Development and Test Cloud”,
IBM Global Technology Services, 2010. URL:
http://www935.ibm.com/services/us/index.wss/offerin
g/midware/a1030965
14. Yulian Liu, “Testing as a Service over Cloud”,
Proceedings of the Fifth IEEE International
 Symposium on Service Oriented System Engineering,
2010.
15. Lian Yu, Shuang Su, Jing Zhao, et al, “Performing
Unit Testing Based on Testing as a Service (TaaS)
Approach”, Proceedings of International Conference
on Service Science (ICSS) 2008.
16. Chen Y, Sun X. STAS: A Scalability Testing and
Analysis System. 2006 IEEE International Conference
on Cluster Computing. 2006:1-10. Available
at:http://ieeexplore.ieee.org/
17. Liviu Ciortea, et al, “Cloud9: a software testing
service”, ACM SIGOPS Operating Systems Review,
Volume 43 Issue 4, January, 2010.
18. Jerry Gao, Xiaoying Bai, W. T. Tsai, “Cloud-Testing -
Issues, Challenges, Needs and Practice”, published by
Software Engineering: An International Journal
(SEIJ), Vol. 1, September, 2011.
19. Jerry Gao, Xiaoying Bai and W. T. Tsai, “SaaS
Performance and Scalability Evaluation in Clouds”,
Proceedings of the Sixth IEEE International
Symposium on Service-Oriented System Engineering
(SOSE2011), Irvine, California, USA, 2011.
20. Xiaoying Bai, Muyang Li, Bin Chen, Jerry Gao and
Wei-Tek Tsai, ” A Survey of Cloud Testing Tools “,
Proceedings of The Sixth IEEE International
Symposium on Service-Oriented System Engineering
(SOSE2011), Irvine, California, USA, 2011.

223

View publication stats