White Paper:

The Seven Habits of

Highly Effective Testing
Organizations: Redux

Sponsored by MKS
Written by Lee Copeland
lee@sqe.com
The Seven Habits of Highly Effective Testing Organizations: Redux

2 | S t i c k y M i n d s . c o m W h i t e P a p e r



In 2001, I wrote an article for StickyMinds.com called The Seven Habits of Highly Effective
Testing Organizations. It was a compilation of industry best practices that I used in my
consulting work. I recently reviewed what I wrote eight years ago, reconsidered my positions,
factored in the advances in our industry, and revised my seven habits. Ive kept six, but
refocused the seventh.


1. A Separate Software Quality Assurance Organization

The software quality assurance (SQA) function defines effective and efficient processes
and standards for the entire software development process, including, but not limited
to, testing. SQA defines the organizations quality goals, the methods for reaching those
goals, and the various roles, responsibilities, and processes of an organization-wide
quality system. Typically, those processes include static testing (reviews, inspections,
and walkthroughs) and dynamic testing (unit, integration, system, acceptance,
regression, etc.). In addition, SQA audits the development and testing products and
processes during and after the project and offers suggestions for improvement.
The focus on the key aspects of quality assuranceeffective and efficient processes and
standardsis just as important today as it was eight years ago. Unfortunately, even after this
time, the number of organizations committed to these ideas remains very small. Few
organizations specify quality goals for their products and even fewer adopt methods to meet
those goals.
However, a number of changes have been adopted over the past eight years. First, the
emergence and adoption of agile methods has brought better development and testing
processes to many organizations even without an SQA function to drive that adoption. This
speaks volumes to the effectiveness of many SQA groups, when a few developers can institute
major process changes very quickly after years of ineffective efforts by SQA organizations.
One important advance has been the substantial commitment to testing, both at the unit and
acceptance levels. The processes of test-driven development (TDD) and acceptance test-driven
development (ATDD) have made testing the in thing to do during development, not that
thing that no one liked to do at the end of development.
Another advance that has come though the adoption of agile methods is the abandonment of
the creation of Big Requirements Up Front and Big Design Up Front. The agile development
process has implemented the Shewhart/Deming cycle of Plan-Do-Check-Act in an iterative and
incremental fashion, defining requirements and design, implementing these requirements, and
then testing them in small chunks.

The Seven Habits of Highly Effective Testing Organizations: Redux

3 | S t i c k y M i n d s . c o m W h i t e P a p e r

2. A Distinct Software Testing Organization

Software testing is a recognized technical specialty within the software development
process. By designating individuals as testers, we can focus their activities on the quality
aspects of software. In a recent survey of attendees at the STAREAST testing conference,
90 percent of all organizations responding have a formal testing group. The placement
of this group varies within the organization. While the majority of testing groups are
included within the development organization, some testing groups are part of quality
assurance, and in other organizations, the testing group reports elsewhere.

Eight years ago, I envisioned the testing group distinct from other groups, such as
developers. Today, Ive abandoned the idea of the necessity of a distinct testing group.
Agile methods have shown that integrating testers with developers can be more
effective than throwing code over the wall to a testing organization. In hindsight, I
should have stated the need for a distinct testing function. It is the function that is
important, not where it appears on the organization chart. Still, we need team members
who have the ability to test wellto perform effective test analysis, design,
implementation, and execution.

What has not changed is the need for people who are skilled at and enjoy testing; the
necessity for skill building, typically through classes, conferences, and workshops; and
the need for automation tools to support the testing effort. For organizations in which
developers are being asked to take on increased responsibility for testing, these are
particularly important.

In some organizations, restrictions imposed from the outside will require a formal testing
group on the organization chart. If that is your situation, dont fight itput the box on the
organization chart. But remember, the box does not reduce the need for great people
supported by valuable automation tools.

3. Risk-based Testing

In the original version of the article, I wrote that the major components of risk-based
testing are:
Risk analysisIdentify the possible risks that a project faces, estimate the loss to
the organization if that risk occurs, choose risk mitigation strategies and
calculate their cost, compare the cost of the loss to the cost of risk mitigation,
then choose appropriately.
Test planning and estimationCreate testing strategies based on the risks that
have been identified and choose sets of test cases based on those strategies.
The Seven Habits of Highly Effective Testing Organizations: Redux

4 | S t i c k y M i n d s . c o m W h i t e P a p e r

Test trackingTrack the number of tests that have been defined, implemented,
and executed; track the number of tests that have passed or failed; examine the
test cases to determine the level of coverage they provide; and evaluate the
efficiency and effectiveness of the testing process.
Analysis of testings contributionDetermine the return on investment that the
test group generates by discovering defects before they impact the
organizations customers.
This is timeless advice, although today I would say testing rather than test group in
determining the ROI. Unfortunately, in the organizations I visit, I see little improvement in risk
analysis, formal test planning, and the analysis of the contribution that testing makes, both to
products and organizations. Still, we dont identify well what is most important to test, we dont
design test cases effectively and efficiently, and we cant quantify the benefits we provide
through testing. No wonder testing is often not more highly thought of.


4. A Defined and Integrated Testing Process

A formally defined and integrated testing process includes these basic elements:
Static testing of the work products of development, including requirements,
design, code, test strategies, plans, procedures, and data
Dynamic testing at the unit, integration, system, acceptance, and regression test
levels
A managed testing environment
The preservation of test artifacts including test strategies, test cases, test
procedures, and test data specifically for reuse on subsequent versions of a
single application or sharing between applications
More timeless advice. I cant add to this. Still, we face two problems: (1) many organizations are
not aware of the processes and tools used in formal testing, and (2) organizations continue to
respond that they do not have the time and resources to do better testing. I continue to be
both amazed and discouraged by the number of test professionals who do not know the basic
rudiments of their profession. One of my favorite interview questions as a test consultant is,
What is your favorite book on software testing? The most common response is, Ive never
read a book on software testing.


5. Visible Defect Tracking and Reporting

Effective defect tracking requires the creation and maintenance of a database describing key
aspects of each defect. Each recorded defect should include a unique tracking number, a
description, the cause, who detected it, how it was detected, the cost of finding it, the
The Seven Habits of Highly Effective Testing Organizations: Redux

5 | S t i c k y M i n d s . c o m W h i t e P a p e r

estimated cost to the organization if the defect had been delivered to the customer, and the
person assigned to repair it. Allow all stakeholders to examine this data. Look for defect
patterns or trends that indicate that the defects are not merely random but are the result of a
systemic problem within the development process.
Of all the seven habits, this one is subject to the most discussion. With the rise of agile
methods, a new approach to handling defects has emerged. Rather than log defects in a
database, many organizations have adopted a fix-and-forget approach. Ron Jeffries proposes
a four-step process: (1) write a test showing the defect exists, (2) add that test to the test suite,
(3) modify the system until the test passes, (4) doneno other action is necessary. Not only has
the defect been repaired, but a test now exists in case of any regression failure. Mary
Poppendieck stated, In other words, stop worrying about the defect database and start
worrying about why you are STILL creating code where defects are discovered a significant
amount of time after the code has been written.
Still, I like the idea of tracking defects. As famous software tester Yogi Berra once said, You can
see a lot by looking. And looking for patterns in defects, their causes, their detection, and their
solution can yield valuable information we can use to improve our development and testing
processes.


6. Change Management

Change management is an umbrella concept that includes three basic areas:
Requirements managementRequirements are reviewed to determine whether
they are feasible and appropriate, clearly stated, consistent with each other and
with the system as a whole, and testable.
Release managementRather than develop each change to a system
individually, a number of changes are collected together into a release. When
the release is defined, all the changes are designed, coded, and tested together.
When complete, the release is moved into production. Release management
establishes a common expectation regarding the length of time required to
implement a change. It provides a mechanism to evaluate and integrate changes
proposed by multiple sources within the organization, and it spreads the testing
overhead over a larger base, yielding a more efficient process.
Configuration managementDefines baselines, controls changes, and reports
the status of application software configurations and the hardware on which
they run. A library of current and past versions is maintained.
The three components of change managementrequirements management, release
management, and configuration managementcontinue to be important to producing quality
software. The agile processes support these components, but in a different way than classic
waterfall methods.
The Seven Habits of Highly Effective Testing Organizations: Redux

6 | S t i c k y M i n d s . c o m W h i t e P a p e r

In agile development, requirements are typically defined and prioritized by the product owner.
As individual requirements are selected for development, they are described in detail, reviewed
by the team, developed, and tested.
Release management in agile is done at a number of levelsrelease, sprint, and daily.
Configuration management is even more important in agile because of the rapid work effort.


7.a (Eight Years Ago) Visible Project Planning and Tracking

Effective project planning requires a defined software lifecycle with stages of manageable size.
The activities within each project are based on that lifecycle. Each stage should be planned,
estimated, and scheduled before work is done. This formal plan should be visible to all
stakeholders in the project. As the development progresses, the actual activities should be
compared to the plan. This comparison should also be visible to all concerned. Variations
between planned and actual activities should be investigated. Differences may signal a
misunderstanding about the size or scope of the project, a lack of skills or resources, or that the
plan was faulty in some way.
While this may sound rather waterfall-ish, it applies equally to agile methods. Processes
such as Scrum and Extreme Programming are defined software lifecycles. Their
iterations are planned, estimated, and scheduled before the work is done. The use of
backlogs prioritized by the product owner make plans visible. The use of burn-down and
burn-up charts make progress visible. The most effective charts show progress in terms
of deliverable functionality that is useful to the customer rather than effort expended.

Planning has been defined simply as figuring out what to do next. Most of us would admit
that to be effective and efficient, planning is important. But when and how should that planning
be done? Must it be done so very early in the project, when our knowledge is typically at its
minimum? Waterfall approaches call for Big Planning Up Front, while Scrum calls for planning
for releases, sprints, and days. Both call for planning of specific types at specific times.

Over the years, Ive developed a more general model, based on an analogy of the planning of
plays in (American) football. When are football plays planned? Our first thought might be in
the huddle before each play, but the following table shows more possibilities:

Before the game begins (the first ten plays are scripted and executed
without regard to their success or failure)
Before each play (in the huddle, based on an overall game plan, field
position, strengths and weaknesses, and player experience)
At the line of scrimmage (depending on the defensive setup)
Start of a play (play actionrun or pass depending on the defense)
During the play (scramble when all else has failed)
The Seven Habits of Highly Effective Testing Organizations: Redux

7 | S t i c k y M i n d s . c o m W h i t e P a p e r


Each one of these planning approaches is used by professional and collegiate teams. A general
rule is that we should plan as much as we can (based on the knowledge available), when we can
(based on the time available), but not before.


7.b (Today) Visible Test Planning and Tracking

When I wrote the article eight years ago, I assumed that everyone would understand
that project planning implied test planning. Apparently, in many organizations, this is
not the case. I often see detailed plans for development and make-it-up-as-we-go
plans for testing.
The most important work product of test planning is the test plan. By test plan I dont mean a
list of test cases. Test plans focus on higher level strategic decisions that should be made before
testing begins. The IEEE 829 Standard for Software Test Documentation lists key elements of
such a plan:
Scope of the test effort
Schedule
Resources required
Responsibilities
Test levels
Test tools, techniques, methods, and metrics
Risks and prioritization
Defect resolution and reporting
Test coverage requirements
Pass/fail criteria
Environmental needs
While the creation of the test plan is usually the responsibility of the test team, effective testing
requires that this plan be communicated to, understood by, and supported by both the
stakeholders and the entire project team. Without their support, testing will generally be
ineffective.


Whats New and Important Since Eight Years Ago
I believe that there have been a number of innovations over the past eight years that are:
affecting the testing community for good:
Agile development methods have raised the stature of testing to a level never seen
before.
The Seven Habits of Highly Effective Testing Organizations: Redux

8 | S t i c k y M i n d s . c o m W h i t e P a p e r

The rise of the context-driven school of testing reminds us that testing is to serve the
projects stakeholders, not to dictate to developers and managers. It reminds us that a
one-process-fits-all approach is ineffectivedifferent projects require different
missions, strategies, techniques, and tools.
The advent of specialties in testingfunctional, performance, security, usability,
reliability, conformance, and a host of others. Not only is the technical scope of testing
increasing, we realize we must specialize to improve our own competencies.
The publication of really good books on testingmanagement, techniques, and
automation.
The creation of small testing workshops with a specialized focus and participatory style.
These tend to be invitation-only, limited to fifteen to twenty participants. All
participants make presentations, and the learnings from the conference are collected,
published, and made broadly available.
American journalist A.J. Liebling once wrote, Freedom of the press is guaranteed only
to those who own one. Today, with the ubiquitous Internet, we all have access to a
press. These presses host dozens of excellent testing blogs and open source testing
training.
As organizations have responded to the markets demand for higher quality products,
and as agile has freed testing from the end of the development process, testing has
broken out of its silo to become a vital force throughout the development life cycle. In
addition, complete application lifecycle management solutions are now available that
integrate test management into the overall development process yielding greater
transparency and productivity.

What Do the Next Eight Years Look Like?

Nobel Prize winning physicist Niels Bohr (or perhaps it was Yankee baseball player Yogi Berra,
the sources are not consistent) said, Prediction is difficult, especially about the future. And so
it isand so I wont make anything but the most obvious predictions.
Virtualization will become ubiquitous, providing multiple machines on a single server.
Using virtualization, multiple operating system/browser/applications can be created and
stored and then loaded and run to provide different testing environments. When a
defect is observed, the entire machine state can be saved for later investigation.
Cloud computing will provide immense resources for testers that are unavailable now.
Imagine having thousands of servers at your command, at a very low cost, and only for
the time you require. No longer will limitations on machines managed by your
organization restrict the amount of testing that can be performed. Imagine the ability to
create thousands of users for performance testing. Imagine what you could do if you
had unlimited hardware resources at your disposal.
The Seven Habits of Highly Effective Testing Organizations: Redux

9 | S t i c k y M i n d s . c o m W h i t e P a p e r

In the futureregarding the benefits of testingsome organizations will get it, but most wont.
Some will understand the great value that quality assurance and quality control activities add,
will implement them, and will reap the rewards. Most will just slouch along, pennywise and
pound foolish, hoping for the best. I hope you will get it.














Lee Copelands update of his StickyMinds.com original column, The Seven Habits of Highly
Effective Testing Organizations is sponsored by MKS.
Breaking out of the silo of test management can elevate visibility of testing as a part of the
software development lifecycle and also improve the accountability of testing groups. The
emergence of complete application lifecycle management solutions like MKS Integrity, where
test management functionality can be integrated into the development lifecycle, achieves this
by delivering greater transparency and productivity. Visit http://www.mks.com/products/test or
email info@mks.com for more information.