Fallacy of Certifications

Murali Chemuturi

Introduction

The definition of quality (Fitness for Use) itself leaves a bit to be desired. It leaves the
terms “Fitness” and “Use” open to interpretation. Thus we see a plethora of products
claiming to be of “Quality” without even adding adjectives like “good”, or “best”. Still the
term “quality” itself implies great quality in the minds of people in general. In
manufacturing and other engineering fields, industry associations, Governments, and
Armed Forces brought out standards and conforming to those standards ensured “Good
Quality” deliverables.

When it came to software, the picture of standardization is not that bright. True, IEEE
(Institution of Electrical and Electronic Engineers, USA) brought out some standards and
termed them as Software Engineering Standards – not “Software (Quality) Standards”.
These standards are more in the nature of guidelines rather than standards in the strict
sense of the word “standard”. They are open to interpretation and adaptation.

Somehow, it came to be believed that adhering to a defined process would ensure

software quality and ISO (International Standards Organization) brought out 9000 series
of standards. This is followed by SEI (Software Engineering Institute, Carnegie Mellon
University, USA) brought out CMM and CMMI (Capabilities Maturity Model Integrated).
CMMI itself is twofold – CMMI for Development and CMMI for Acquisition.

Many organizations wishing to outsource their development work started insisting on

certification, especially of CMMI. A certificate enables opening of doors for bidding and
lack of it, closed the doors. The vendors started getting the coveted certificate either by
hook or by crook, just to be in the race. And understandably certification organizations
sprouted like mushrooms. Now plenty of development organizations got certified – at
least on ISO and in many cases both ISO and CMMI.

We also have TMM (Testing Maturity Model), People Capability Maturity Model (PCMM),
Software Engineering Capability Maturity Model (SE-CMM), IT Service CMM and so on -
Lee Copeland lists 34 maturity models in his article “The Maturity Maturity Model (M3)”
on Stickyminds.com web site.

ISO 9000 series of standards started out focusing on Quality with QMS (Quality
Management System) as the main document and Quality Policy as the backbone for the
organizational processes.

But perhaps due to pressure from industry, these process standards diluted into
organizational processes, shifting the focus from quality to organizational vision, and
goals etc. CMMI goes one step further stating that the process should be to achieve
business goals. The quality of deliverables has clearly taken backseat.

I had occasion to be associated with some certified organizations, as a consultant or as

a member of the audit team or as an employee. A significant number of those do not
adhere to their own defined process. I was horrified to find one ISO 9000 certified
organization in which the MR (Management Representative) did not read the process
documentation. I noticed that the quality head of a CMM level 5 organization, doesn’t
know how to open the URL for the organizational process. I observed a CMM level 4
certified organization that does not collect nor maintain any metrics. In yet in another
organization that is certified for ISO 9000 and aiming for CMMI level 3, I heard the CEO
stating that he does not want any managers in his organization – I was dumbstruck
wondering who would manage their software projects, if everybody were a coder.

Organizations unearthed the loopholes in the models, consultants advise how to cook
the books to get the certificate and appraisers are available who would certify for a fee. I
posed a question for the CMMI appraisers group on Yahoo if they ever refused a
certificate – only one or two replied in the affirmative – rest all maintained a dignified
silence.

The time has come to develop a different paradigm for quality in software, a paradigm
that is focused on the quality of the deliverable than on the organization.

Let us examine some of the criticisms of these Maturity Models (MMs).

One criticism of all these MMs is that they emphasize on the organizational business
objectives, but not on Product (or Deliverable if you prefer) Quality! The confidence that
a process driven organization delivers Quality is misplaced. One – the process itself
may be flawed. Two – each process has loopholes. Three – the people are focused on
conforming to process, more than achieving excellence in quality. Four – management
of the organization focuses on delivering and selling than on quality. The poor quality
head, if there is one, is there to coordinate with certifying agencies and he has no control
whatsoever on the product quality. In many organizations, I saw that the person holding
the post of quality head (under many other nomenclatures like SEPG Head, Quality
Coordinator, Quality Manager, Director Quality and so on) is not really qualified or
experienced for holding that post or possess much knowledge about the quality
concepts and tools.

MMs focused less on development of software and ensuring that quality is built-in but
focused more on support processes. CMMI has more (8) specific goals for Project
Management (Project Planning, Integrated Project Management, Risk Management,
Configuration Management, Project Monitoring and Control, Quantitative Project
Management, Supplier Agreement Management, and Requirements Management)
where as it has less (3) specific goals for Quality (Process and Product Quality
Assurance, Validation and Verification). It has only three specific goals (Product
Integration, Requirements Development and Technical Solution) for development of
software. It has two specific goals (Organizational Process Focus and Organizational
Process Development) for organizational process definition and four specific goals
(Causal Analysis and Resolution, Decision Analysis and Resolution, Measurement and
Analysis, and organizational Process Performance) for measurement and analysis. The
remaining two specific goals are Organizational Innovation and Deployment and
Organizational Training. Thus the focus on Quality is too diluted – three out of 22!

Even then, the MMs do not insist that their model must be implemented in to-to. They
accept “largely implemented” as adequate for giving a certificate. The model itself is not
tightly defined and is made so flexible that the practices are open to any interpretation.
Some allow “Alternative Practices” in place of the practices defined in the model. This
allows the organization to do what they want and still get certified as conforming to the
model!
Another criticism of the MMs is that they do not define any quality thresholds for
achieving the certification. Conformance to self-defined process is adequate. Say the
standard for an electrical equipment would define the insulation resistance in quantitative
terms so that human beings do not get an electric shock by handling that equipment. But
no software engineering standard defines what should be the defect density for, let us
say, a financial application!

Another criticism of MMs is that they do not specify the number of years the organization
needs to be in operation before they can be mature enough to be ready for certification.
That way, even a one-year-old organization can get certified. There are single person
organizations that are certified!

Another criticism about the MMs is that they do not specify any quality objectives to be
achieved for obtaining the certificate. Mere conformance or showing evidence of
conformance in just six projects or less gets the organization certified – no need to
demonstrate achievement of quality!

The owners of the MMs do not maintain the actual performance of the organizations
after certification. ISO specifies surveillance audit that is cursory but CMMI does not.
Whether the quality has improved, or any reduction in complaints – the owners of MMs
do not keep track of.

Let us examine some loopholes in these certifications.

Pecuniary consideration

Certification agencies charge a high fee – ($200 per hour is perhaps an indicator and the
appraisal period ranges from 2 days to 3 weeks). Suppose an appraiser rejects
certification to one of his clients, do you think that appraiser would get calls for
appraising from any other organization? Therefore, the best that an appraiser would do
is to cancel the assignment if he is dissatisfied with the preparation of the organization.
Who would take the risk of being branded as too strict an appraiser? The organization
that offers certification easily is the one most sought after.

Another issue is that the certificate to issue certificates is being issued too easily. Pass
an exam and you get it or attend a training program and you get it. Go forth and multiply
certificates.

Besides, these certifying organizations are profit-pursuing business organizations that

have expenses to meet, targets to achieve, and growth to be aimed at. That is the
reason why we see a plethora of certificates being issued.

Method of appraisal

The appraisal process itself is under criticism – the appraiser looks at the evidence
presented to it. It is more like conformance audit – not an investigative audit. What is the
guarantee that the evidence is not cooked up to suit the requirements of the appraiser?
If financial accounting books (that are subject to statutory independent audit) are being
cooked, why can’t certification evidence be? I have had a call to cook the evidence for a
certification audit and the organization head told me, with a straight face, that the
certification agency knows this and is a willing partner. Of course, I told them to go look
for somebody else. In many cases, the appraiser-organization itself happens to be both
the process consultant as well as the appraiser for the organization. Surprisingly, neither
the models nor the appraisers see any conflict of interest in such an engagement.

Appraisal invariable goes by sampling. Sample gives a good picture of the universe
when –

1. The universe is homogenous

2. The sample is picked up randomly

The appraiser doesn’t ensure both the above pre-requisites before beginning the
appraisal!

In software organizations, both are false. The population is not homogenous.

1. All projects are similar perhaps, but not identical

2. All project managers are not uniformly qualified / trained nor have similar
experience

There is no guideline when 100% project appraisal becomes mandatory.

Sample selection

In none of the appraisals that I witnessed, either from close quarters or from a distance
did select candidate projects thru any random sampling technique. The appraisers
accepted the projects offered by the organizations. Two projects for ISO and six projects
for CMMI – that is the norm followed by the organizations. The development
organizations term projects as CMMI-Projects and non-CMMI Projects (alternatively ISO
Project and non-ISO Projects). The appraiser assumes that all other projects are
identical to the projects presented and accords certificate, which these organizations
flaunt to induce customers.

Redressal Mechanisms

A customer or any concerned person has no place to complain if they have something to
report on any certified organization. The email ids of officials responsible for looking into
complaints about organizations (that are certified and using the certificate to obtain
business) are not publicly available.

Nor those officials take up suo-moto action on erring organizations. CMM was retired
more than two years ago and still, there are many organizations claiming CMM level on
their web sites!

Post-certification reporting requirements

There are none! The certified organizations need not submit compliance reports to any
one! ISO mandates surveillance audit twice a year and the NCRs (Non-conformance
Reports) raised by the auditors can be cleared by next audit and in the meanwhile the
certificate is not suspended. No certificate is ever revoked – at least not to my
knowledge or publicized.
CMMI requires re-appraisal once in three years – the certified organization can exploit
the certificate for three years fully.

Public limited companies (in other words certified by the Registrar of Companies or an
equivalent authority who issues a certificate of incorporation) are mandated to publicize
their financial achievements audited by an independent auditor. Should the certifying
models not mandate such a requirement?

There is no requirement in any of the models that the certified organization must make
available their quality data like sigma level, defect density or the NCRs raised by
auditors or Opportunities for Improvement pointed out by appraisers, on the web site of
either the organization or that of the model owner. The organizations do not display their
quality performance on their web sites.

Any revocations so far?

I searched the web sites of SEI and ISO to locate the list of organizations for which they
rejected / revoked / cancelled the certificate. I could find none – nor could locate a link
on Google search. This can mean that they have not rejected / revoked / cancelled even
one certificate or they are keeping such a list confidential. My assumption is that they did
not rejected / revoked / canceled a single certificate. Public display of such a list would
go a long way in improving the credibility of the certification.

Auditing the auditors

The way, the certifying agencies are audited, leaves much to be desired. Model owners
audit periodically the certifying agencies – not the organizations that are certified by the
certifying agency! This again is a conformance audit not an investigative audit. So the
surveillance on certifiers is also lax.

Final Words

The objectives of model definition and certification are not for ensuring quality.
Possession of a certificate by an organization does not guarantee quality of deliverable.

Satyam Computers has all the certificates from all types of certifying agencies; still those
certificates did not prevent the Chairman from committing a massive self-confessed
fraud! If the certified processes are working well, how could this happen?

World Bank has banned organizations like Satyam and Wipro (both of whom have the
highest levels of certification) from carrying our any work for the bank. If these
organizations are treating an organization like World Bank in this manner what is the
level of quality they are giving to other customers who do not have the same clout as the
World Bank?

What makes me wonder is that neither the ISO nor the SEI revoked the certificates for
these two organizations! Does this tell us something about the credibility of the
certificate?
Clearly, the certification failed. I know that I will be ridiculed for this. Some kid has to
shout – that the King is naked!

**************************************
About the Author – Murali Chemuturi is a Fellow of Industrial Engineering and an MBA
and has over 30 years of corporate experience and about 8 years of consulting
experience. He can be reached on murali@chemuturi.com or thru his web site
http://www.chemuturi.com . Your feedback is welcome and appreciated - it will be
responded to in 24 hours.
**************************************