Experiences With CMM and ISO 9001 Benchmarks

by William Decker, Joe Haskell, and Frank McGarry Computer Sciences Corporation (CSC)

1.

Introduction and Purpose

T e use o! industry benc marks to measure process maturity and process compliance as increased si"ni!icantly in recent years# T$o $idely applied benc marks are t e Capability Maturity Model (CMM) (%e!erence &) o! t e So!t$are 'n"ineerin" (nstitute (S'() and a set o! )uality standards de*eloped by t e (nternational Standards +r"ani,ation ((S+) (%e!erence -)# .lt ou" t ere is disa"reement re"ardin" t e *alue and application o! t ese benc marks, many Go*ernment or"ani,ations are usin" t em to identi!y and select )uali!ied contractors# For e/ample, 0.S. as adopted (S+ 1222 as an internal re)uirement (%e!erence 3) and many o! 0.S.4s !uture contracts $ill likely re)uire compliance $it an applicable (S+ standard# Similarly, ot er Go*ernment or"ani,ations re)uire a speci!ied CMM maturity le*el as a prere)uisite !or submittin" a proposal# Compliance $it industry benc marks is becomin" increasin"ly important to or"ani,ations seekin" "o*ernment contracts# CSC4s S'.S Center pro*ides products and ser*ices to 0.S. at t e Goddard Space Fli" t Center in Greenbelt, Maryland# T e $ork is per!ormed under t e C0M+S contract# To comply $it 0.S.4s direction, CSC obtained (S+ 122& re"istration in May &115# To demonstrate so!t$are process maturity, CSC under$ent a So!t$are Capability '*aluation (SC') in 0o*ember &115# T is paper pro*ides an o*er*ie$ o! t e e/periences o! t e S'.S Center in pursuin" (S+ re"istration and CMM compliance# Comparisons o! t e CMM key process areas (67.s) to t e -2 elements o! (S+ 122& a*e been per!ormed by ot ers (%e!erences 8 and 9) and are not t e sub:ect o! t is paper# %at er, t is paper pro*ides a i" ;le*el comparison o! t e (S+ and CMM assessment processes, cost, and *alue as percei*ed by t e participants# Compre ensi*e lessons learned reports $ere prepared by t e S'.S Center !or use by ot er CSC or"ani,ations in pursuin" (S+ re"istration and CMM compliance#

2.

Profile of the Organization

T e cost and cycle time !or ac ie*in" compliance $it an industry benc mark are dependent upon t e or"ani,ation4s startin" point# . mature or"ani,ation $ill !ocus on ad:ustment o! e/istin" processes rat er t an process de!inition# (n contrast, a less;mature or"ani,ation $ill initially !ocus on "ettin" key processes de!ined, documented, and deployed# T e cost and cycle time needed !or ac ie*in" compliance $it a "i*en CMM le*el or an (S+ standard are muc less !or an or"ani,ation t at as a solid process !oundation upon $ ic to build# T is paper is based on t e e/periences o! CSC4s S'.S Center# . brie! description o! t e or"ani,ation is needed in order to place its e/periences in conte/t# Hence, an o*er*ie$ o! t e S'.S Center is pro*ided in Sections -#& t rou" -#8#

2.1

Size of SE S Center and !"pe of Work #erformed

T e S'.S Center consists o! appro/imately <22 indi*iduals pro*idin" products and ser*ices to 0.S.# T e acti*ities included systems en"ineerin", so!t$are de*elopment and maintenance, system inte"ration, and system operations# .ppro/imately 82 percent o! t e $ork is so!t$are de*elopment and maintenance# T e $ork includes mission desi"n, control center de*elopment, ad*anced systems e*aluation, data processin", and !li" t dynamics mission operations# Fi!teen to -2 pro:ects $ere acti*e durin" t e &115 assessments# T e si,e o! t ese pro:ects *aried !rom 9 to &92 personnel# 2.2 #rocess Impro$ement Strate%"

.s s o$n in Fi"ure &, CSC be"an so!t$are process sel!;assessments and SC's in &11&# =et$een &11& and &11>, or"ani,ational policies, process documentation and support tools $ere en anced based on e/perience# .s a result, by &11> mature processes $ere routinely in use t rou" out t e S'.S Center# Ho$e*er, alt ou" process impro*ement $as on"oin", a process impro*ement plan (7(7) $as not $ritten until &11>#

1990

1991

199%

199$

199#

199"

199!

199

199&

&

S S S

SCE ISO 9001 registration audit (R), surveillance audits (S) Software process self assessments and software process audits 100$9&1%'(001
Figure 1. SEAS Center Benchmar ing !istor" T e 7(7 is based on t e ?uality (mpro*ement 7aradi"m# T e &11> 7(7 included si/ "oals !or impro*ement in products and ser*ices and t$o "oals related to process maturity# T e t$o process;related "oals $ere to (&) ac ie*e re"istration a"ainst (S+ 122& by +ctober &115 and (-) demonstrate compliance $it CMM @e*el 3 as determined by a SC' durin" &115# 2.' #rocess (oc)mentation

(n &11&, t e !ollo$in" process;related documentation and tools ad been in use !or se*eral yearsA

T$enty 7ro"ram +!!ice Directi*es de!inin" minimum re)uirements !or pro:ect per!ormance .n earned;*alue reportin" system t at ad been in place !or < years, producin" mont ly cost and sc edule *ariances .n &&;c apter, 922;pa"e system de*elopment met odolo"y Se*enty;!i*e standards and procedures Barious andbooks and "uidebooks to supplement t e ot er documents Barious pro:ect;le*el documents to address pro:ect;speci!ic issues

=et$een &11& and May &11>, t ese documentation and tools $ere impro*ed to address t e c an"in" needs o! t e or"ani,ation# .s a result, at t e time o! t e &115 (S+ re"istration assessment and SC', t e !ollo$in" process;related documentation and tools $ere in routine useA T$enty 7ro"ram +!!ice Directi*es, updated to address $eaknesses !ound durin" internal assessments .n earned;*alue reportin" system in routine use !or appro/imately &9 years . &-;c apter, 992;pa"e system de*elopment met odolo"y, en anced since &11& to address c an"e re)uests submitted by S'.S Center personnel &2& standards and procedures Barious andbooks and "uidebooks to supplement ot er process;related documents 7ro:ect;le*el process documents to address pro:ect;speci!ic issues

Wit t is process documentation in routine use in May &11>, t e S'.S Center establis ed its "oal o! ac ie*in" (S+ re"istration and CMM compliance by late &115# 2.* #rocess Mat)rit" as Meas)red +" Ind)str" Benchmarks

. istory o! t e S'.S Center benc markin" acti*ities is s o$n in Table &# . &11& SC' produced a CMM @e*el & ratin" !or t e S'.S Center, e*en t ou" documented processes $ere routinely used# =y February &11>, all process and process compliance issues ad been addressed and all but one o! t e @e*el -;3 67. $ere satis!ied as determined by a SC'# .n SC' conducted in late &115 !ound !ull compliance $it all @e*el -;3 67., as $ell as compliance $it se*eral o! t e @e*el 8;9 67.# T e (S+ re"istration assessment $as conducted in May &115# Durin" t e assessment, &8 minor noncon!ormities $ere identi!ied, eac o! $ ic $as promptly eliminated# T e (S+ re"istrar recommended re"istration at t e end o! t e assessment $eek# . subse)uent sur*eillance assessment $as success!ully conducted in 0o*ember &115#

#a$le 1. Summar" of Benchmar ing Acti%ities

SCE ,10-91. % mont*s &"0 staff(*ours /inimal SCE ,2-9/. # mont*s 1&00 staff(*ours 0one ISO ,0-91. 1% mont*s $#00 staff(*ours %00 *ours Consultant Internal auditor training )reregistration assessment )erform gap 4evelop anal1sis implementation plan -se lessons learned from 1991 -se e.ternal SCE e.perts 3ocus on 5rain staff deplo1ment 3ocus on deplo1ment 1$ of 1& 6)2s ISO registration satisfied ac*ieved SCE ,11-91 to 1-92. % mont*s &00 staff(*ours 0one ISO ,11-91. ! mont*s "00 staff(*ours 0one

)reparation time Organi+ational effort, -se of e.ternal consultants and training

)reparation strateg1

)erform S)2s

Complete action items )rovide awareness seminars -se internal assessments

Continue process improvement initiatives 3ocus on management review, internal audits, and corrective actions

Result

3ew 6)2s satisfied

C// 7evel $ ac*ieved8 7evel #(" 594 1:9&

ISO registration maintained

, In addition to routine improvement activities

&.

Soft'are Capa$ilit" E%aluation and (egistration )sing ISO *++1

+r"ani,ations considerin" pursuin" (S+ re"istration andCor CMM compliance are usually interested in a comparison o! t ese benc marks# Typical )uestions include t e !ollo$in"A Ho$ are t e t$o benc marks similar or di!!erentD For eac , $ at is t e cost o! t e e!!ortD (s pursuit o! (S+ re"istration andCor CMM compliance o! *alue to t e or"ani,ationD

T ese )uestions are addressed in Sections 3#& t rou" 3#9# '.1 ssessment Simi3arities Selection o! representati*e pro:ectsA . recommendation $as made by t e S'.S CenterE t e pro:ect selection $as made by assessment team (For bot types o! assessment, t e S'.S Center identi!ied pro:ects t at, as a "roup, $ere representati*e o! t e $ork per!ormed and co*ered t e assessment criteria# T e assessment team made t e !inal selection o! pro:ects, takin" into account t e S'.S Center input#) 0umber o! pro:ects selected by assessment team as representati*e o! S'.S CenterA 8 0umber o! S'.S Center personnel inter*ie$ed durin" t e assessmentA 92 to >2 Duration o! t e assessmentA & $eek Focus o! assessment on process de!inition, process compliance, and process impro*ement (ifferences a*e primary di!!erences as

=ased on t e e/perience o! t e S'.S Center, t e assessments a*e t e !ollo$in" similaritiesA

'.2

=ased on t e e/perience o! t e S'.S Center, t e assessments identi!ied in Table -#

T ere are clear di!!erences in preparin" !or a SC' as opposed to (S+ re"istration# For e/ample, a )uality manual documentin" t e or"ani,ation4s strate"y !or ac ie*in" )uality is re)uired !or (S+ re"istration# (n producin" its )uality manual, t e S'.S Center updated its policies and processes to re!lect or"ani,ation "oals and actual business practices# +n t e ot er and, t e SC'4s !ocus on so!t$are de*elopment capability resulted in a critical e*aluation o! t e ade)uacy and e!!iciency o! so!t$are de*elopment practices# T ere $ere also ma:or di!!erences in t e preassessment documentation re)uirements o! t e SC' Team and (S+ %e"istration Team# T ese are identi!ied and discussed# T ere are si"ni!icant di!!erences in o$ t e t$o assessments are conducted# For e/ample, SC' inter*ie$s $ere conducted in "roups o! !unctional area representati*es !rom multiple pro:ects# T us, t e assessment team could determine $ et er indi*idual pro:ects !ollo$ed consistent

#a$le 2. ,ifferences Found in C-- and ISO (egistration

SCE )rogram management, software pro;ects, and support organi+ations /aterials re<uested in advance =1 evaluation Completed organi+ation and pro;ect profiles, team <uestionnaires, >e1 software documents, and map of documents to C// Interview format @roups of functional area representatives from eac* participating pro;ect interviewed =1 entire SCE team Si+e of evaluation team Si. people Evidence re<uested 2=out 100 software(related wor> products8 evidence t1picall1 gat*ered following t*e interview Result of assessment C// maturit1 level8 strengt*s, wea>nesses, and improvement initiatives defined in compre*ensive report Organi+ation elements t*at participated ISO 9001 ssessment )rogram management, all pro;ects, and support organi+ations ?ualit1 /anagement S1stem /anual, definition of scope of registration )ro;ect representatives interviewed individuall1 or in a group 5wo people 2=out 0 wor> products8 evidence t1picall1 e.amined during t*e interview Registered or not registered8 wea>nesses documented

so!t$are de*elopment processes# (n contrast, t e (S+ %e"istrar inter*ie$ed pro:ect members as a "roup to determine $ et er personnel !rom t e "i*en pro:ect !unctioned as an inte"rated team# T ere are ad*anta"es to eac inter*ie$ !ormatE t ese are identi!ied and e/plored# T ere $ere di!!erences in t e post;assessment acti*ities# For t e SC', process $eaknesses $ere identi!ied in a Final Findin"s %eport, and a compre ensi*e action plan $as implemented to address eac !indin"# For t e (S+ re"istration assessment, process $eaknesses $ere identi!ied as t e assessment took place (an e/c an"e o! *ie$s), and correcti*e action $as implemented early# (nterestin"ly, t e S'.S Center !ound t at bot t e SC' and (S+ re"istration assessment tended to identi!y symptoms rat er t an real problems# T e S'.S Center strate"y !or de!inin" and addressin" real problems !rom assessments is discussed# '.'
'.'.1

Or%anization Effort and !ime to #repare

Soft4are Capa+i3it" E$a3)ation

.s s o$n in Table -, S'.S Center personnel spent &<22 sta!!; ours on t e SC' conducted in February &11># Most o! t is time $as spent (denti!yin" "aps in t e or"ani,ation4s documented processes, relati*e to CMM @e*els -;3 Fpdatin" documentation to impro*e processes and !ill t e "aps Deployin" t e re*ised processes (primarily trainin") 7er!ormin" administrati*e $ork, suc as completin" t e pro:ect pro!iles and CMM )uestionnaires re)uired by t e SC' Team 7articipatin" in SC' inter*ie$s

+nly <22 sta!!; ours $ere re)uired to prepare !or t e SC' conducted in 0o*ember &115# =y t at time, S'.S Center personnel $ere con!ident t at all CMM;speci!ied processes $ere documented and used# Hence, t e SC' took place $it minimal impact on t e S'.S Center#
'.'.2 ISO &e%istration

.s s o$n in Table -, S'.S Center personnel spent 3822 sta!!; ours on t e (S+ re"istration e!!ort conducted in May &115# Most o! t is time $as spent Gainin" an understandin" o! (S+ 122& and preparin" an implementation plan 7er!ormin" a "ap analysis based on (S+ 122& Fpdatin" documentation to impro*e processes and !ill t e "aps (included $ritin" a )uality manual as re)uired by (S+) Deployin" t e re*ised processes and support tools (@otus 0otes deployed to support internal communications and document control) 7er!ormin" administrati*e $ork, suc as selectin" a (S+ re"istrar (t ird;party e/ternal assessor)

7articipatin" in t e (S+ inter*ie$s

T e (S+ re"istration e!!ort in May &115 $as success!ul# Ho$e*er, (S+ re)uires periodic sur*eillance assessments to ensure process impro*ement is continuin"# +nly 922 sta!!; ours $ere re)uired to prepare !or t e sur*eillance assessment conducted in 0o*ember &115# =y t at time, S'.S Center personnel $ere con!ident t at all outstandin" issues !rom t e re"istration assessment ad been ade)uately addressed# T e sur*eillance assessment took place $it minimal impact on t e S'.S Center# '.* Cons)3tants and !rainers

Many consultants and trainers are a*ailable to assist an or"ani,ation ac ie*e (S+ re"istration andCor CMM compliance# %e"ardin" suc ser*ices, t e S'.S Center !ound t e !ollo$in"A
'.*.1

Typical cost o! suc ser*ices is G&59 per our, plus e/penses# Companies and indi*iduals pro*idin" suc ser*ices seem $ell;)uali!ied and competent (it is a *ery competiti*e industry)# Suc ser*ices can be *ery $ort $ ile to establis credibility and minimi,e t e possibility o! an unsuccess!ul e!!ort#
Soft4are Capa+i3it" E$a3)ation

T e S'.S Center did not use consultants to prepare !or its SC'# Ho$e*er, se*eral S'.S Center personnel attended trainin" in t e CMM t at $as pro*ided by its parent CSC di*ision#
'.*.2 Internationa3 Standards Or%anization

T e S'.S Center did use consultants and trainers to prepare !or (S+ re"istration# Speci!ically, assistance $as pro*ided to t e S'.S Center as !ollo$sA .n indi*idual !rom an e/ternal CSC di*ision $as assi"ned to t e S'.S Center !or se*eral $eeks at t e start o! t e e!!ort# T is indi*idual pro*ided ad*ice to t e S'.S Center (S+ (mplementation Team re"ardin" implementation strate"y# T is indi*idual also pro*ided orientation and (S+ a$areness trainin" !or S'.S Center personnel# (& sta!!;mont ) .n e/ternal trainer $as ired to pro*ide trainin" o! S'.S Center internal auditors# T is indi*idual also pro*ided ad*ice re"ardin" ot er (S+;re)uired acti*ities# (- sta!!;$eeks) T e t$o;member (S+ re"istration team conducted a prere"istration assessment to !amiliari,e S'.S Center personnel $it t e assessment process and identi!y any ma:or $eaknesses in t e S'.S Center ?uality Mana"ement System# (&#9 sta!!;$eeks) T e t$o;member (S+ re"istration team conducted t e re"istration assessment# (- sta!!; $eeks) 5a3)e Based on S)r$e" of #articipants

'.0

T e percei*ed *alue o! t e (S+ re"istration and SC' e/periences $as measured by sur*eys o! participants#

(mmediately !ollo$in" (S+ a$areness trainin" (& year prior to t e re"istration assessment) S'.S Center personnel $ere asked $ et er t ey !elt t e e!!ort $ould impro*e t e S'.S Center# Ten speci!ic issues $ere addressed (e#"#, impro*ed processes, more customer con!idence, less paper$ork)# Follo$in" t e re"istration acti*ity, t e same personnel $ere t en asked $ et er t ey !elt t e e!!ort did impro*e t e S'.S Center# .s s o$n in Fi"ure -, t e (S+ e/perience si"ni!icantly e/ceeded e/pectations#
ISO 9001 #E&CE#!IO 6
100A 90A &0A 0A se3ected "es !0A "0A #0A $0A %0A 10A 0A 1 % $ # " ! 7)estion n)m +er & 9 10 1st <uestionnaire %nd <uestionnaire

1( Improve processes %( Increase customer confidence $( Improve efficienc1 #( Reduce paperwor> "( Reduce defects

!( Improve teamwor> ( Improve communication &( Belp identif1 pro=lems earlier 9( Belp win contracts 10(Improve understanding of processes

E.pected from ISO (":9!) Results from ISO (":9 )

100$9&1%'(00%

Figure 2. ISO (esults E.ceeded E.pectations T e same )uestions $ere asked o! t e participants in t e 0o*ember &115 SC'# T e (S+ e/perience $as rated i" er t an t e SC' on < o! t e &2 )uestions# . comparison o! t e ans$ers to !our o! t e )uestions is s o$n in Fi"ure 3# Comments re"ardin" t ese !our ans$ers !ollo$sA (ncreased Customer Con!idenceA (S+ H <-I, SC' H 8-I# T is likely re!lects t e !act t at t e customer (0.S.) as made (S+ compliance a priority, $it little *alue placed on CMM# %educed 7aper$orkA (S+ H -<I, SC' H -5I# S'.S Center personnel !ound neit er benc mark ad muc impact on t e ma"nitude o! process;related documentation and paper$ork associated $it process use#

1008 908 208 108

Selected Des &% !

9& &#

/08 008 *08 '08 208 108 08

Increased 1st 9tr Customer Confidence (?%) Reduced 2nd 9tr )aperwor> (?#) #% % %&

"1

C SCE (11:9 )

Reduced 'rd 9tr 4efects (?")

Belp *th 'in 9tr Contracts (?9)

C ISO (":9 )

100$9&1%'(00$

Figure &. ISO *++1 Scored !igher than C-- on / Out of 1+ 0uestions %educed De!ectsA (S+ H >5I, SC' H 9&I# T e relati*e lo$ scores !or eac benc mark indicates minimal problems $it product de!ects prior to benc markin" acti*ities# (0.S.4s stated priority !or t e S'.S Center is cost and cycle time reductionE t e )uality o! S'.S Center products as ne*er been an issue#) Help Win ContractsA SC' H 1<I, (S+ H <8I# Go*ernment a"encies ot er t an 0.S. are increasin"ly usin" t e CMM to identi!y and select contractors, ence t e relati*ely i" score !or SC'# 0ote t at S'.S Center personnel stron"ly belie*e compliance $it bot industry benc marks is important in obtainin" ne$ $ork#

S'.S Center personnel $ o participated in bot assessments $ere askedA J(n your opinion, $ ic industry benc mark caused "reater impro*ement#K .s s o$n in Fi"ure 8, t ese indi*iduals !ound (S+ 122& to a*e a "reater positi*e impact# Ho$e*er, it is important to note t at t e (S+ 122& e!!ort ad an impact on t e $ ole or"ani,ation (rat er t an :ust so!t$are de*elopers), and t at t e e!!ort applied to (S+ re"istration $as almost t$ice as "reat as t e e!!ort applied to CMM compliance# Finally, as s o$n in Fi"ure 9, bot benc marks $ere *ie$ed !a*orably by participants $ o $ere asked to rate t e e/perience as (&) $ell $ort t e e!!ort, (-) o! mar"inal *alue, or (3) o! little or no *alue# For bot benc marks, appro/imately <2 percent o! t e participants rated t e acti*ity as J$ell $ort t e e!!ort#K T e percenta"e o! !a*orable ratin"s by personnel not directly in*ol*ed in t e benc markin" acti*ities $as some$ at less#

100 90 20 10 #ercent /0 00 *0 '0 20 10 0 *1 2 *0

In "o)r opinion: 4hich ind)str" +enchmark ca)sed %reater impro$ement; 9)estionnaire distri+)ted to SCE participants: pro%ram mana%ement: and 7)a3it" ass)rance ,*2 responses.

<

+o)t the same

< CMM ,1200 = 200 staff>ho)rs. < ISO 9001 ,'*00 = 000 staff>ho)rs.
100$9&1%'(00#

Figure 1. ISO *++1 had 2reater Impact than C--3 $ut also had 2reater Effort Applied

1 100 1 1 20 1
/0 1

2 11

12

' 12

21 '0

/2 21 // 02
C 7ittle or no value C /arginal value C 'ell wort* t*e effort
6ot Inter$ie4ed Inter$ie4ed 6ot Inter$ie4ed Inter$ie4ed
100$9&1%'(00"

#ercent

1
*0

0 0

19

20

0 0
0

0
ISO 9001 CMM

Figure 4. Both Benchmar s 5ere 6ie'ed Fa%ora$l"3 Especiall" $" Participants

1.

7oticea$le Impacts on SEAS Center

+pportunities !or or"ani,ational impro*ement as a result o! a SC' or (S+ %e"istration depend on t e type o! or"ani,ation and its maturity# For t e S'.S Center, many impro*ements resulted !rom t e SC' and (S+ %e"istration acti*ity# T ese impro*ements include t e !ollo$in"A 'liminated ambi"uities in or"ani,ation roles and responsibilities ((S+) Clearly de!ined current *ersion o! key documents, alon" $it location, o$ner, and c an"e process (SC') (mplemented control, update, and distribution o! key documentation t rou" t e use o! electronic libraries ((S+) Structured senior mana"ement re*ie$s to identi!y problems and address critical needs o! t e or"ani,ation (primarily SC') Fpdated policies and processes to ensure e!!ecti*eness and ali"nment $it or"ani,ational "oals (prompted primarily by (S+) Fp"raded t e trainin" pro"ram to address or"ani,ational needs (primarily SC')

4.

(ecommendations to Other Organizations

=ased on t e S'.S Center e/perience, t e !ollo$in" recommendations s ould be considered by or"ani,ations settin" out in pursuit o! (S+ re"istration andCor CMM complianceA Demonstrate commitment o! senior mana"ementE i! not a i" priority, t e e!!ort $ill !ail# 'stablis an internal "roup responsible !or interpretin" t e benc mark and pro*idin" "uidance to t e rest o! t e or"ani,ationE do not re)uire all personnel to be $ell *ersed in CMM or (S+ standards# (mplement processes and procedures to impro*e t e or"ani,ation, rat er t an :ust to !it t e industry benc mark# Do not use anot er or"ani,ation4s processesE build on $ at e/ists and use e/periences o! or"ani,ations doin" similar types o! $ork# (Documents do not c an"e t e or"ani,ation4s culture#) Focus on compliance $it t e "oals o! t e benc mark, rat er t an details# Spend at least t$ice as muc e!!ort in deployin" rat er t an documentin" processes# Conduct internal audits and track action items to closure#

8.

Ac no'ledgments

To be supplied#

9.
1. 2.

(eferences
Charles V. Beber, et al., CMU/SEI-93-TR-24, ESC-TR-93-177, Capability Maturity Model for Software, Version 1.1, February 1993 American National Standard. Quality Systems Model for Quality Assurance in Design, Development, Production, Installation, and Servicing, prepared by the American Society of Quality Control Standards Committee for American National Standards Committee Z-1 on Quality Assurance, August 1, 1994 NASA Management Instruction 1270 M. Paulk, How ISO 9001 Compares With the CMM, IEEE Software, January 1995, pp. 74-82 F. Coalier, How ISO 9001 Fits Into the Software World, IEEE Software, January 1995, pp. 98-100

3. 4. 5.