Risk Based Testing

-Why we need RBT ?

-Types of risks
-Managing risks
-Methods of evaluation & risk analysis
-Costs and benefits

Ladislau Szilagyi
www.EuroQST.ro
 Definitions (ISTQB glossary)
Risk = a factor that could result in future negative
consequences; usually expressed as impact and
likelihood.
Risk based testing = a testing strategy intended
to reduce the level of product risks and inform the
stakeholders of their status, starting in the initial
stages of a project. It involves the identification of
product risks and the use of risk levels to guide the
test process.
Safety = the capability of the software product to
achieve acceptable levels of risk of harm to
people, business, software, property or
environment in a specified context of use.
 Risk outcome

Risk = negative outcome? not always

Positive risks are opportunities, desired by both
the Project Manager and the stakeholders, and
may positively affect the project ( such as
increasing the ROI or finishing the project ahead
of time ).
But, positive risk may generate negative risks
( finishing a part of the project way before
schedule will create a lot of slack, as other
resources are not scheduled to work on the
project until much later ).
 Risk types

Product risk
Functional
Non-functional

Project risk
External
Organizational
Technical
 Risk types

Project risk categories

External:
service provider related issues;
client related issues;

 Risk types

Organizational:
skill and staff shortages;
personal and training issues;
political issues, such as:
problems with testers communicating their needs and
test results;
failure to follow up on information found in testing and
reviews (e.g. not improving development and testing
practices).
improper attitude toward or expectations of testing
(e.g. not appreciating the value of finding defects
during testing).

 Risk types

Technical:
problems in defining the right requirements;
the extent that requirements can be met given
existing constraints;
the quality of the design, code and tests.
 Risk dimensions

Likelihood = the estimated probability that a risk

will become an actual outcome or event

Impact = the damage that will be caused if the

risk become an actual outcome or event
 Risk dimensions
Risk dimensions are depending on the context:

Automotive:
Exposure (the relative expected frequency of the
operational conditions in which the damage can possibly
happen)
Control (the relative likelihood that the user can act to
prevent the damage)
Severity of the damage

Avionics:
Threat
Vulnerability
Consequences
 Risk factors

Technical
Complexity of technology and teams
Personnel and training issues among the
business analysts, designers, and programmers
Conflict within the team
Contractual problems with suppliers
Geographical distribution of the development
organization
Legacy versus new approaches

 Risk factors

Tools and technology

Bad managerial or technical leadership
Time, resource and management pressure
Lack of earlier quality assurance
High change rates
High earlier defect rates
Interfacing and integration issues
 Risk factors

Business
Frequency of use of the affected feature
Damage to image
Loss of business
Potential financial, ecological or social losses or
liability
Civil or criminal legal sanctions
Loss of license
Lack of reasonable workarounds
Visibility of failure leading to negative publicity
 Risk options

Ignore
Assume
Delegate
Mitigate
Contingency planning
 RBT activities

Establish the context

Risk identification
Risk analysis
Risk assessment
Risks sorting
Risk mitigation
Risk contingency
Risk reporting
 Establishing the context

Study the business domain

Identify the stakeholders

Define a framework

Planning of the remaining activities

 Product Risk Identification

Risk statement

Something may fail in some way due to some

circumstances

Something The component or feature where the

problem could occur

Fail in some way What potential failure

Some circumstances The reasons or vulnerabilities,

why we are concerned
 Product Risk Identification techniques

Expert interviews
Project matrix
Independent assessment
Use of risk templates
Lessons learned
Risk workshops
Brainstorming
Checklists
 Product Risk Analysis techniques

Informal fast, cheap, but not accurate

Pragmatic Risk Analysis and Management (PRAM)
Systematic Software Testing (SST)
Product Risk Management (PRisMa)

High
1
Probability

3
4

Low
Low High
Consequence
 Product Risk Analysis techniques

Formal accurate, but takes time, expensive

Cost-of-exposure
Quality function deployment
Fault tree analysis
Hazard analysis
Failure Mode Effect Analysis
FMEA - Example of formal Risk Analysis technique
Failure Mode Effect Analysis
Define the System
Identify Potential Failure Modes & Their Causes
Evaluate the Effects on the System of Each Failure Mode
Identify Failure Detection Methods
Identify Corrective Measures for Failure Modes

3 factors:
Severity = The criticality of the effects of bugs in this failure mode,
should any exist, from 1 (most damaging) to 5 (least damaging),
Likelihood = The probability ofand extent of impact associated
withbugs included in this failure mode, from 1 (most probable) to
5 (least probable).
Priority = The importance of fixing bugs in this failure mode, should
any exist, based primarily on the ability of the delivered system to
meet customer needs, though also on logistical project issues,
regulatory or standards compliance, or other business
considerations, from 1 (most important to fix) to 5 (least important to
fix).
Product Risk Analysis techniques
 Product Risk Mitigation techniques

Non-testing related
Testing related
Choosing an appropriate test design technique
Reviews & inspection
Reviews of test design
Level of independence
Most experienced person
The way re-testing is performed
Regression testing
RBT activities
 Project Risk Mitigation techniques

Non-testing related
Testing related
Early preparation of test ware
Pre-testing test equipment
Pre-testing earlier versions of the product
Tougher entry criteria
Requirements for testability
Participation in reviews of earlier project results
Participation in problem and change management
Monitoring of the testing progress and quality
Questions?