Professional Documents
Culture Documents
Chris Miller
Office of the Deputy Director, Software Engineering and System Assurance SYSTEMS & SOFTWARE ENGINEERING
Office of the Deputy Under Secretary of Defense for Acquisition and Technology US Department of Defense
October 2008
Version 2.0
Industrial Programs
Slide 2
Slide 3
Slide 4
Development -
Software Engineering and Systems Assurance (SSA) initiatives Program feasibility analysis using estimation models Software Resources and Data Report: Feasibility Study Revision of MIL-HDBK-881A to improve software guidance Integration of software metrics with EVM to assess consistency of estimates
Systems & Software Engineering Chris Miller, October 2008
Decision Support Reviews NunnPSRs/NARs completed: 57 AOTRs completed: 13 Nunn-McCurdy Certification: 13 Participation on Service-led IRTs: 3 Technical Assessments: 13
OTRR 9% Pre-MS C 19% McCurdy 13% DAE Review 5% Other 25% Pre-MS B 26% Pre-MS A 3%
Service-Managed Acquisitions
Air Force 33% Agencies 9%
Land 13%
Army 27%
Opportunity for SSA to support program decision making by providing software estimates
Estimation activities aimed at gauging overall program feasibility and quantifying magnitude of top program risks Focus on support for engineering vs. budgeting decisions
Program A
10
Program A
918 5375 62
1,204 7147 68
1590 9571 74
Result:
Analysis revealed existing acquisition strategy was not feasible Service added more schedule to acquisition strategy DUSD(A&T) estimates change in acquisition strategy saved $5 billion
Systems & Software Engineering Chris Miller, October 2008
11
Program B
Major subcontractor with significant software content did not use any parametric estimating tool
Firm Fixed Price subcontract
At one point there was a substantial conflict on estimates between major subcontractor and prime, related to requirements
One portion of software grew from ~250KSLOC to 863KSLOC (3.5x), reflecting shall not degrade current capability requirement
Increase in software engineering staff by 57 percent (40 people) over 9 month period
Based on schedule, contractor should be drawing down software staff
12
Program B
Changes in SLOC estimates
CSCI 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 D C E F G H H I G K L C M C 65,000 33,000 15,000 114,000 26,000 6,000 2,000 33,000 15,000 23,000 7,000 1,000 NA NA 752,000 283,255 373,587 7,879 218,609 25,622 16,580 34,806 42,355 126,238 26,404 12,500 29,121 100 2,600 2,078,969 25,467 100,129 7,879 218,609 34,544 16,580 34,806 42,355 119,626 26,404 12,500 29,121 1,000 2,600 1,413,521 18,367 68,129 2,000 16,409 8,922 15,580 17,206 31,655 6,433 2,604 1,000 25,621 1,000 400 7,100 32,000 5,879 202,200 25,622 1,000 17,600 10,700 113,193 23,800 11,500 3,500 Unavail 2,200 82% 99% 100% 99% 100% 100% 100% 99% 100% 91% 100% 100% 100% 81% 60% 99% 100% 95% 100% 100% 100% 97% 100% 91% 100% 100% 100% 43% C 325,000 415,249 401,732 131,921 269,811 98% 98% Supplier A B A Original 40,000 27,000 20,000 NM Review 374,739 50,363 38,962 2 wks. later 250,844 50,363 38,962 New 201,144 40,363 38,962 Reuse 49,700 10,000 0 Complete % 75% 90% 71% Test coverage to-date 68% 85% 60% DR's/Problem reports 47 0 5 1 5 0 0 0 1 0 2 2 0 1 0 1 1 7 0 0
13
Staff Months
800
100 KSLOC (Optimal Schedule)
600 400
50 KSLOC (Optimal Schedule) 150 KSLOC (Nominal) 100 KSLOC (Nominal) 50 KSLOC (Nominal)
200 0 0 5 10 15 Months
20
25
30
14
Program B
Recommendations
PMO reach a decision on unstable requirements to prevent further code growth in or out Program office bring in parametric estimating consultant to review contractors estimates for most volatile software components
Result
Acquisition Decision Memorandum requires Service to conduct software review
15
Prioritized Risks
Program Context
Risk Management
Insight Needed
16
BAC = $1420
$1,400 $1,200
$1,051 $1,145 $1,251 $1,318 $1,370
$1,420 $1,471
$1,559
$1,579
DOLLARS ($K)
$1,000
$792
EV Metrics Calculated Schedule Range: 09/08/08 - 10/12/08 Calculated Estim ate at Com plete Range: $1,797K - $2,515K Cum ulative CPI = 0.79 SW Metrics Calculated Schedule Range: 05/14/08 - 06/18/08 Calculated Estim ate at Com plete Range: $1,570K - $1,710K
$291
$338
$200
Data Date
$0 Oct07
Nov07
Dec07
Jan08
Feb08
M ar08
Apr08
M ay08
Jun08
Jul08
Aug08
Sep08
Oct08
Nov08
Dec08
TIME (Months)
BAC BCWS (PV) BCWP (EV) ACWP (AC) ETC
Acquisition Oversight confidence increases as ETCs overlap i.e., as multiple measures indicate to same conclusion
Systems & Software Engineering Chris Miller, October 2008
Impossible Zone
Analysis of CSCI data: Apparent Maximum Size of 200 KSLOC (per CSCI) Apparent Minimum Schedule (Impossible Zone) Environment & Complexity are primary drivers of variation Apparent Maximum Schedule of 50 months
Source: Long, L. G. et al, Aerospace Corp Report,2004
Systems & Software Engineering Chris Miller, October 2008
Notable changes:
Replace 'material item' with ' acquisition program' Add words to make 'artifacts' equal to 'products' Include words that make 'product-oriented' and 'DoDAF architecture' views interchangeable WRT to creating a WBS
Results
Compiled a Comment matrix Drafted a new Appendix B (for software)
During final review and clean up, the working group was made aware of 90s effort (draft MIL-HDBK-171)
Systems & Software Engineering Chris Miller, October 2008
The Software Cost Control working group goal is to provide an community-coordinated set of recommendations Reached out to industry members of NDIA to review the suggested changes
Ensure recommended changes are improvements from industry perspective Obtain additional examples to include in Appendix B NDIA feedback is due on September 19th
Submission of suggested changes to PA&E ARA will take place after collection of NDIA feedback and consolidate into a single baseline
Systems & Software Engineering Chris Miller, October 2008
Questions/Discussion
Contact Information: Chris Miller SSE/SSA Support Software Engineering & Systems Assurance ODUSD(A&T) Systems & Software Engineering Christopher.miller.ctr@osd.mil
24