State of Tennessee TEDS Assessment Report

State of Tennessee
TEDS Assessment Report
January 9th, 2015
Contents
Overview
TEDS Risk Portfolio
Risk Ranking Scales
TEDS Potential Alternatives
10
TEDS Potential Alternatives High Level Timelines
12
Detailed Findings
14
Software Development Life Cycle (SDLC) Findings
15
Architecture Findings
22
Project Management Office Findings
28
Operational & Organizational Readiness Findings
34
The State acknowledges and agrees that any advice, recommendations, information, Deliverables or other work product provided to State by KPMG LLP is intended for State's sole benefit and KPMG LLP does not
authorize any other party to rely upon such advice, recommendations, information, Deliverables or other work product and any such reliance shall be at such party's sole risk. 2014 KPMG LLP, a Delaware limited
liability partnership and the U.S. member firm of the KPMG network of independent member firms affiliated with KPMG International Cooperative (KPMG International), a Swiss entity. All rights reserved.
Overview
Per the contract Assessment of the State's Tennessee Eligibility Determination System (TEDS) Project
contract #31865-00381 KPMG, was engaged to perform an objective review to assist the State in
assessing the current Tennessee Eligibility Determination System (TEDS) project being delivered by
Northrop Grumman Corporation (NG).
The Assessment Report contains a TEDS alternatives analysis which identifies TEDS go forward project
options and alternatives for consideration by HCFA executive leadership. The TEDS potential
alternatives are based upon the included TEDS Risk Portfolio and Detailed Findings with consideration
for the project schedule, costs, benefits, resources, and contract implications.
TEDS Risk Portfolio

The following summary findings reflect the analysis from KPMGs assessment of the TEDS project.
Definitions of impact probability and time criticality can be found on slides 7, 8, and 9.
Impact
Probability
Time
Criticality
Requirements Quality & Completeness - Key requirements types were missed or

defined imprecisely, creating potential gaps in TEDS solution.
High
Almost
Certain
Immediate
Requirements Traceability - TEDS requirements are incompletely or ineffectively

traced, resulting in potential gaps in solution.
Moderate
Almost
Certain
Immediate
Resources - TEDS projects lacks adequate Medicaid eligibility domain resources, and
Moderate
roles were not defined to support success.
Almost
Certain
Immediate
Software Development Life Cycle (SDLC)
Observation
Development - TEDS code is highly customized, complex, and may face performance
and quality issues (NG documents indicate 50-70% customization vs. original
projections of 15%).
High
Likely
Short Term
Testing Testing approach to TEDS is insufficient to ensure a successful solution.
High
Almost
Certain
Short Term
Almost
Certain
Short Term
Quality - Quality Management has had insufficient impact on ensuring project success. Moderate
TEDS Risk Portfolio

Impact
Probability
Time
Criticality
Business Operating Model - A documented business operating model for Medicaid

Member Services is currently incomplete for the TEDS future state. The system
processes are not at a sufficient level of detail to support business operations.
High
Almost
Certain
Immediate
Fit Gap Analysis - The fit-gap of TEDS (based on the Avance product) to State
requirements and approach to estimate the DDI effort cannot be validated.
High
Almost
Certain
Immediate
Capacity Plan - A documented capacity plan was not available for review. The system
infrastructure could be prone to crashes or significant degradation in performance upon
stress.
High
Almost
Certain
Immediate
System Architecture At least one component of the system architecture will be at

end of life within the next year. No new releases available to add or change
functionality or address issues of any kind in version 7.1.X, including security.
Defects with system software will not be resolved in a reasonable timeframe.
High
Almost
Certain
Immediate
Likely
Long Term
Architecture
Observation
Eligibility Modernization Program (EMP) Architecture Governance - A documented

Moderate
EMP Architecture Governance Approach and Plan does not exist.
TEDS Risk Portfolio

Impact
Probability
Time
Criticality
Schedule Management - The management of the project has been unsuccessful in

managing an event driven milestone based project schedule.
High
Almost
Certain
Immediate
Project Schedule - Change Requests and gap activities, rooted in insufficiencies

relative to requirements management and project estimation, coupled with an
aggressive project schedule, contributed to an unreliable project schedule.
High
Almost
Certain
Immediate
Resource Management Knowledgeable resources available for critical tasks are

limited and resources are not adequately represented and managed in the Integrated
Master Schedule (IMS).
Moderate
Almost
Certain
Long Term
Project Reporting - The project schedule status reporting, for approximately the first
ten months of the project, was not adequate.
Moderate
Almost
Certain
Long Term
Project Management Office (PMO)
Observation
Risk & Issue Management - The execution of the risk and issue management
processes was not adequate.
Low
Somewhat
Long Term
Likely
TEDS Risk Portfolio

Impact
Probability
Time
Criticality
Organizational Readiness - Organizational Readiness was not sufficiently planned

for, leading to a gap in overall organizational management for the TEDS effort.
High
Likely
Long Term
Organizational Change Management (OCM) / Training OCM / Training approach

for the TEDS application is insufficient to allow State employees to fully perform job
functions.
High
Almost
Certain
Short Term
Service Management - TEDS Operation & Maintenance Plan lacks sufficient definition
and detail to service management activities.
High
Almost
Certain
Short Term
Demand Management - TEDS Operation & Maintenance Plan lacks sufficient

definition and detail of an integrated demand management process.
High
Almost
Certain
Short Term
Governance - TEDS Operation & Maintenance Plan lacks sufficient definition and
detail of IT governance activities.
Moderate
Almost
Certain
Short Term
Communication Management - Communication planning for the TEDS project does

not engage all project stakeholders.
Moderate
Almost
Certain
Long Term
Operational & Organizational Readiness
Observation
Knowledge Transfer - Although informal knowledge transfer activities through ad hoc

project meetings, the project has not identified / followed a defined methodology to help
ensure appropriate attendance / outcomes.
Low
Somewhat
Long Term
Likely
Risk Ranking Scales: Impact
Project has the potential to be shut down until properly mitigated.

High
Long-term negative impact on administration / operations (e.g., total system failure, failed implementation).
Litigation, increased liability.
Considerable impact on project.

Moderate
Short mid-term negative impact on administration / operations (e.g., loss of data, project delays,
inefficiency / ineffectiveness).
Fallout from risk will have negative stakeholder / public perception / reaction for TennCare.
Impact on project is manageable but detrimental.
Low
Project is impacted but can continue to operate.

Short-term negative impact; impact on administration / operations.
Risk is manageable.
Risk Ranking Scales: Probability
Almost
Certain
Likely
Somewhat
Likely
Event is expected to occur in most scenarios.

More than 90% probability (e.g., snow in Chicago in January and February).
Event will probably occur in most circumstances.

More than 50% probability and up to 90% certainty (e.g., snow in Chicago in December).
Event should occur at some time.

More than 25% and up to 50% probability (e.g., snow in Chicago in November).
Risk Ranking Scales: Time Criticality
Immediate
Impact and probability are such that this risk needs to be addressed within 30 days.
Short Term
Impact and probability are such that this risk needs to be addressed within 90 days.
Long Term
Impact and probability are such that this risk needs to be addressed within 90+ days.
TEDS Potential
Alternatives
Pros
Resource
TEDS Potential Alternatives

New SI with NG Software
Sole Source
Leverage some current efforts and

investments to date.
Secure rapid procurement of

an operational system.
Potential deployment within Enhanced

Federal Funding Participation (EFFP)
timeframes.
Provide resources with

domain and product
knowledge.
Inability to obtain resources with domain

/ product knowledge, perpetuates
unproductive outcomes.
Reduce risk by leveraging a

working solution.
Costly effort with low probability of

success to remediate requirements
quality, traceability, design and testing
issues.
Cons
Restart with Procurement of a New SI Vendor
Restart with
Northrop Grumman (NG)
Capacity and performance planning was

not adequately accounted for in the
initial solution architecture, resulting in
high likelihood of costly performance
issue remediation.
Inability to accurately estimate and
manage project schedule / tasks,
resulting in low confidence of effective
future schedule development and
management.
Many of the existing NG workforce
would likely not be retained.
Competitive negotiation
Secure rapid procurement Provide clear TN

business and IT vision
of an operational system.
defined for selected
vendor.
domain and product
knowledge.
Reduce risk by leveraging
a working solution.
Immediate need to remediate
unsupported software components and

to account for operations & maintenance
activities.
Secure a vendor with

proven / recent
implementation
execution.
Secure a vendor with

proven / recent
domain and product
implementation execution.
knowledge.
Secure product
Secure product confidence.
Secure cost competitive
confidence.
Improve CMS confidence.
option.
Improve CMS confidence.
Improve stakeholder
Secure product
Improve stakeholder
confidence.
confidence.
confidence.
Improve CMS
Secure cost competitive
confidence.
option.
Improve stakeholder
Provide clear TN business
confidence.
and IT vision defined for
Extended timeline for
Modification required to meet
selected vendor.
TN to move away from
TN business and IT vision.
Additional funding
contingency operations.
Additional funding required
required on behalf of CMS
Additional funding
on behalf of CMS / State on
/ State on Tennessee.
required on behalf of
Tennessee.
Potential public
CMS / State of
Potential public perception
perception risk of limited
Tennessee.
risk of no competition.
competition.
Secure a vendor with proven

/ recent implementation
execution.
RFP
Leverage existing
hardware and software
assets.
Highest complexity.
High probability of
extended timeline.
Low confidence in quality
and performance of
software.
High risk premium paid by
the State of Tennessee in
part due to costly effort
with low probability of
success to remediate
requirements quality,
traceability, design and
testing.
High operating costs.
Low likelihood of NG
product qualified resources
available.
Immediate need to
remediate unsupported
software components and
to account for operations &
maintenance activities.
Stakeholder skepticism.
Stakeholder skepticism.
NG Contractual
Implication
Amend existing NG
Contract.
Termination of existing NG
Contract.
Termination of existing
NG Contract.
Termination of existing
NG Contract.
Termination / renegotiation
of existing NG contract(s).
11
TEDS Potential Alternatives High Level Timeline
Ent. Align
Critical Pre-Requisites for

following options
Calendar Year
Quarter
Business Operating Model
2014
Q4
2015
Q1
Q2
PMO Start
Q3
Q4
EFFP Ends
Business Requirement Definition

System Requirement Specifications
Competitive
Negotiation
Sole Source
NG Restart
Restart
CAP / Resource Mod.
Potential Re-scope
CMS Funding & Contract Modifications
Potential Restart
MES Market Assessment
CMS Funding Updates
Contract Negotiations
Sole Source Start
CMS Funding Updates
Competitive Negotiation & Proposal
New Vendor Start
RFP
CMS Funding Updates

RFP & Proposal
New Vendor Start
12
TEDS Potential Alternatives High Level Timeline

Calendar Year
Quarter
2015
Q3
2016
Q4
Q1
Q2
Q3
Q4
MES Market Assessment

Business Requirement Definition
NG Restart
Release 1
Sole Source
Release 1
Competitive
Negotiation
System Requirement Specifications
Release 1
Release 2
Operations & Maintenance
Release 2
Release 2
RFP
Release 1
Release 2
13
Detailed Findings
Software
Development Life
Cycle (SDLC)
Findings
SDLC Requirements Quality & Completeness

Impact
High
Probability
Almost Certain
Time Criticality
Immediate
Key requirements types were missed or defined imprecisely, creating potential gaps in TEDS solution.
Observations
Risks
1. Requirements approach was oriented around

Avance configuration requirements, rather than
State of Tennessee needs.
1. Missed requirements likely leave solution unable to support

Tennessees anticipated business model, resulting in higher labor
costs for TennCare.
2. NG risk based process evaluation document

notes gaps in multiple requirements types,
including reporting, performance, interfaces,
conversion, hardware, backup and recovery and
security.
2. Key requirements will not be developed because they were not

identified, leading to gaps in functionality and or reporting.
3. No operational requirements defined to support

the TennCare eligibility continuum.
4. NG approach to State requirements relied
heavily on items as stated in contract, without
elaboration to support design, development and
testing activities.
5. Roles and responsibilities for requirements
management are poorly defined, leading to
confusion across the team per NG document.
6. Tracking of requirements through discrete
requirements traceability matrices and similar
sign-off approach limited visibility to missing
categories of requirements.
3. Gaps in security, reporting, or other areas of solution.

4. Inability to complete operational readiness testing (ORT) and / or
customer service impacts if solution prevents effective delivery of
services to Medicaid clients.
Recommendations
1. State should define business processes to support TennCares
Medicaid eligibility continuum and development of a revised,
comprehensive State requirements set.
2. State should assess current and planned system functionality
against revised comprehensive State requirements set.
3. Northrop Grumman should develop an action plan to address
requirements gaps identified through comparison of planned
functionality to the revised, comprehensive State requirements set.
16
SDLC Requirements Traceability

Impact
Moderate
Probability
Almost Certain
Time Criticality
Immediate
TEDS requirements are incompletely or ineffectively traced, resulting in potential gaps in solution.
Observations
Risks
1. NG has acknowledged, via requirements audit

and risk based process evaluation, multiple gaps
in traceability. It is unclear whether a full impact
assessment has occurred on development that
occurred prior to the audit.
1. Improperly traced requirements may lead to gaps in meeting State

of Tennessee needs.
2. Per NG, traceability relationships across levels

have been forced. Independent Verification and
Validation (IV&V) sampling of 40 items found 6070% exception rate when tracking logical
linkages from requirements through test
verification procedures.
3. IV&V review of relationships between
requirements and test verification procedures
(TVPs) has identified forced linkages not
supported by logical relationships.
2. Unlinked requirements may have been left out of solution, leading

to reporting, security and functionality gaps.
3. System Integration Test results may overstate completion of
solution if tests do not test the requirement effectively.
4. Requirements, as managed in Contour, could have been
accidentally or intentionally changed or deleted by NG team
members due to lack of access controls through much of project.
Recommendations
1. NG should conduct an impact assessment on requirements with
traceability issues identified, and provide to State for review.
4. Requirements included in original designs and

marked as development complete have
subsequently been identified by Northrop
Grumman as change requests.
2. State should identify resources to engage in a detailed review and

validation of traceability.
5. Contour access permissions were not limited as

of May 2014, creating control gap in changes to
requirements.
4. State should review TVPs to confirm that they are testing

requirement completion effectively, or to update as needed.
3. NG should secure additional resources to support necessary

remediation of forced relationships.
17
SDLC Resources
Impact
Moderate
Probability
Almost Certain
Time Criticality
Immediate
TEDS projects lacks adequate Medicaid eligibility domain resources, and roles were not defined to
support success.
Observations
Risks
1. High levels of turnover among Business Analysts

(BAs) in early stages of project, and BAs lacked
Medicaid experience.
1. Key Medicaid requirements are likely to be missed, such as

reporting.
2. Limited pool of resources with Avance product

knowledge.
3. NG Requirements Manager, per NG document,
saw role as extending no further than managing
data in Contour rather than ensuring effective
requirements identification.
4. Current business architect primarily brings
product knowledge, with some knowledge of
Montana and Wyoming Medicaid systems, but
little ACA experience.
5. No effective NG mitigation strategy in place to
close Medicaid and ACA knowledge gap. Onetime training offered, without clear follow-up.
6. State project team had limited resources,
creating capacity challenges and contributing to
missed or unclear requirements and limited
review of traceability mappings and
configuration.
2. Incomplete design, configuration and testing exists due to

mistranslation of requirements.
3. Late identification of errors in traceability and configuration
decisions, contributing to delays in schedule to remediate.
4. Missed requirements and design gaps leading to solution gaps
because of limited representation of operational, security and other
key functional and technical perspectives in requirements and
design phases.
Recommendations
1. NG should secure a Medicaid / ACA eligibility resource for full-time
assignment to TEDS team, responsible for CMS compliance
alignment.
2. NG should review role definitions and training programs to focus
team members orientation around project success while
maintaining compliance with NG project procedures.
3. State should develop a more robust project team structure
representative of all necessary organizational functions and levels,
and with sufficient capacity to support detailed validation activities.
18
SDLC Development
Impact
High
Probability
Likely
Time Criticality
Short Term
TEDS code is highly customized, complex, and may face performance and quality issues.
Observations
Risks
1. TEDS is highly customized; NGs Action Item

Log indicates 50-70% customization vs. original
projections of 15%.
1. Delays in development due to complexity of code from high level of

customization needed.
2. Resolution time for critical blockers has

increased, potentially reflecting growing code
complexity.
3. Original Avance code was not evaluated for
quality prior to deployment for TEDS. NG
document notes concerns about circular
references in code. Quality issues are being
addressed on an ad hoc basis.
4. No scalability assessment conducted on Avance
core baseline prior to implementation to validate
performance (e.g., response time or availability)
against larger state population. NG has
acknowledged finding issues with scaling.
5. High rates of peer review failures without
apparent root cause analysis. Peer review
metrics were not tracked until August 2013 and a
standard tool for peer reviews was not in place
until January 2014.
2. Significantly increased effort to maintain system (e.g., patches or

upgrades) in the future due to increased complexity from
customizations.
3. Potential unstable performance and resulting public relations
impacts if code quality issues are not fully addressed, in turn
causing adverse impacts on service delivery and / or user
experience.
4. Potential performance issues or substantial hardware costs if code
scalability concerns cannot be addressed.
Recommendations
1. State should engage a third party to conduct detailed code
assessment to determine code quality, consistency, efficiency.
2. NG should provide evidence, to be reviewed by State,
demonstrating that all customizations are sufficiently documented.
3. State should assess potential long-term maintenance / staffing
costs based on increased level of customization.
4. NG should provide scalability results to date, outstanding analysis
items, and proposed remediation plan for State review.
19
SDLC Testing
Impact
High
Probability
Almost Certain
Time Criticality
Short Term
Testing Approach to TEDS is insufficient to ensure a successful solution.

Observations
Risks
1. Test verification procedures have not been

validated for relevance to requirements, and in
some cases are not effective in testing the
requirements.
1. Test results may overstate solution completion.
2. UAT is finding high failure rate when repeating

SIT tests using test verification procedures,
raising questions about reliability of SIT results.
3. Same tests are used for System Integration
Testing (SIT) and User Acceptance Testing
(UAT). NG has not consistently followed defined
testing process in addressing defects.
4. Performance testing not yet completed against
full code and business processes.
2. Lack of true UAT may result in gaps in completeness and

effectiveness of testing.
3. Limited testing could result in unidentified errors that do not
manifest until production, with large volumes of live data, resulting
in adverse customer experience and adverse reputational impacts.
4. Increased CMS scrutiny, requiring TennCare time and resources to
manage response.
Recommendations
1. State should identify resources to review and validate test
verification procedures.
2. State should identify appropriate resources to develop and validate
UAT scenarios and scripts, based on State-defined requirements.
3. NG and State should develop performance targets and ensure
completion of performance testing to validate adequate
performance levels.
20
SDLC Quality
Impact
Moderate
Probability
Almost Certain
Time Criticality
Short Term
Quality Management has had insufficient impact on ensuring project success.

Observations
Risks
1. Significant emphasis on process e.g., work

products rather than outcomes, such as a
working solution.
1. Quality program, while ensuring completion of work products, does

not ensure solution quality.
2. Major risks not included in risk register, including

original schedule aggressiveness, domain
knowledge, requirements participation.
3. Quality program assumes process compliance,
without timely validation to confirm compliance.
2. Risks become issues because of late identification, impacting

project schedule and quality.
3. Quality does not improve, despite metrics and identification of risks.
4. Continued delays and quality issues prevent timely go-live.
4. Lessons learned / action items do not

necessarily resolve issue. e.g., domain
knowledge gap addressed through one-time nonmandatory training.
Recommendations
5. Unclear whether quality assurance findings are

consistently incorporated into project.
2. NG should ensure that the quality management program reviews

the effectiveness of risk management activities.
1. NG should review / update the quality management program to

increase emphasis on outcomes rather than process adherence.
3. NG should provide evidence around execution of quality controls.

State should review controls evidence and identify exceptions or
design gaps.
4. State should designate key resource(s) to review Mission
Assurance results, escalate within State and coordinate tracking
and resolution of quality assurance / mission assurance issues.
21
Architecture
Findings
Architecture Business Operating Model

Impact
High
Probability
Almost Certain
Time Criticality
Immediate
A documented business operating model for Medicaid Member Services is currently incomplete for the
TEDS future state. The system processes are not at a sufficient level of detail to support business
operations.
Observations
Risks
1. Business operating model and processes are incomplete

and hence they do not align to CMS Eligibility and
Enrollment flows in support of Medicaid business
operations and requirements.
1. State cannot adequately plan training, operational

readiness and user acceptance testing due to lack of
available business process flows which should align with
system processes.
2. System documentation lacks sufficient detail to effectively

align to TennCares future business processes.
2. State will not have clarity in terms of automated / manual

steps needed to perform organizational change
management and sustain business operations.
Recommendations
1. Perform gap analysis between the business processes
and the system to ascertain manual steps to perform
business operations.
23
Architecture Fit GAP Analysis

Impact
High
Probability
Almost Certain
Time Criticality
Immediate
The fit-gap of TEDS (based on the Avance product) to State requirements and approach to estimate the
DDI effort cannot be validated.
Observations
Risks
1. The detailed fit-gap of TEDS (based on the

Avance product) to Tennessee (TN)
requirements before and after requirement
sessions is unavailable for review. TEDS is
highly customized; NGs Action Item Log
indicates 50-70% customization vs. original
projections of 15%. However, there is no
evidence of systematic quantification of
customization effort.
1. Effort estimates may be inaccurate for the gaps in the system that
need to be enhanced to meet TN requirements. This may result in
schedule slippage especially at late stages of the project as well as
need to have significant float in the schedule to account for the
unpredictability.
2. A documented estimation methodology utilizing

the fit-gap as input is unavailable. The approach
described verbally is an acceptable industry
standard practice however no work products
were available to review and validate the
approach.
2. The degree of customization required to meet TN needs might be

significant in which case the product significantly deviates from the
baseline. This will cause product upgrades to be very difficult and
an increased complexity in maintenance tasks due to the high
degree of customization.
Recommendations
1. NG should document the fit-gap and estimation approach which
should be validated by the State.
2. NG should provide evidence of fit-gap and corresponding effort
estimates used to develop the schedule.
3. NG should track progress in order to validate estimates to identify
deltas in order to improve schedule predictability.
4. NG should track change request in the requirements in order to
categorize fit-gap appropriately.
24
Architecture Capacity Plan

Impact
High
Probability
Almost Certain
Time Criticality
Immediate
A documented capacity plan was not available for review. The system infrastructure could be prone to
crashes or significant degradation in performance upon stress.
Observations
Risks
1. Documented capacity plan which contains

metrics for estimating the space, computer
hardware, software and connection infrastructure
resources does not exist. The overall approach
addressing capacity planning has been
verbalized in meetings and pending submission
to the State.
1. Potential degradation of system performance upon go-live.
2. Transaction numbers have not been finalized /

agreed upon in order to develop appropriate
system throughput needs.
2. Additional infrastructure procurement may be required.
Recommendations
1. Confirm transaction assumptions and validate with other States of
similar size.
2. State needs to consider documenting response time (performance)
SLAs in addition to the uptime SLAs reflective of the system
architecture.
3. NG should develop a capacity plan mapping the infrastructure
needs of the TennCare enterprise.
4. Plan for a performance test that 1.x times max load, extended load
test at max load (>24 hours) and a break test to validate the system
can handle the stress.
25
Architecture System Architecture

Impact
High
Probability
Almost Certain
Time Criticality
Immediate
At least one component of the system architecture will be at end of life within the next year and defects
with system software will not be resolved in a reasonable timeframe.
Observations
Risks
1. IBM WebSphere ILOG JRules V7.1.X will end support

September 30, 2015 and extended support will have to be
purchased from IBM to a maximum of 3 additional years
available.
1. JRules extended support is only as of the end of life date.

Extended support does not include any net new
enhancements post end of life, and issues resolution will
not be timely.
2. No new releases available to add or change functionality
or address issues of any kind in version 7.1.X, including
security.
3. State will have to migrate to a new/different rules engine
within the first 3 years of O&M to prevent any significant
maintenance challenges.
Recommendations
1. Develop a migration plan to move away from the end of
life rules engine.
26
Architecture EMP Architecture Governance

Impact
Moderate
Probability
Likely
Time Criticality
Long Term
A documented Eligibility Modernization Program (EMP) Architecture Governance Approach and Plan
does not exist.
Observations
Risks
1. Documented Architecture Governance Approach and

Plan does not exist.
1. Without an Architectural Governance plan / Architecture

Review Board, there is not a documented process for
managing / prioritizing shared business / technical architecture
components.
2. Current governance processes focus solely on

infrastructure and are conducted on an ad-hoc basis.
2. Missing and duplicate business / technical capabilities across

the organization resulting in higher total operating and system
cost of ownership.
Recommendations
1. State should consider enhancing an EMP architecture
governance plan including but not limited to current and future
state blueprint, demand management, solution and technical
architecture review governance process, and application
portfolio rationalization.
27
Project Management
Office Findings
PMO Schedule Management

Impact
High
Probability
Almost Certain
Time Criticality
Immediate
The management of the project has been unsuccessful in managing an event driven milestone based
project schedule.
Observations
Risks
1. The baselined Project Integrated Master

Schedule (IMS), updated August 1, 2014,
has not been formally approved by the
State. The last baselined IMS was approved
by the State in January 2014.
1. Without an approved baseline, appropriate variance and trend analysis

can not be adequately performed and stakeholders will not have timely
access to accurate project status information.
2. Continuing changes to the projects scope

and requirements have impacted the project
schedule.
3. Task estimations in the IMS lack sufficient
applicable methodology relative to Medicaid
eligibility determination and Tennessee
specific requirements.
4. A high volume of Change Requests (CRs),
many of which are directly linked to
insufficiencies in requirements management,
have contributed to project delays.
Additionally, CRs continue to be a concern
as the ACA requirements evolve.
2. Without the ability to stop / delay additional CRs, Release 1.0 will
continue to be delayed.
3. Further delays in the completion of the project will occur if estimates are
not accurate or if the remaining work and associated timelines are not
fully incorporated in the current draft of the IMS.
Recommendations
1. The State should approve a baselined IMS to manage and control the
project activities.
2. NG and the State should monitor and closely manage the process for
the State and NG to jointly analyze CRs and their impact to resources,
scope and schedule prior to the CRs being approved and incorporated
into the IMS.
3. NG and the State should agree to freeze changes in scope due to CRs
and deliver the initial release of the TEDS system. Additional CRs
should be planned to be incorporated in a future release (post go-live).
4. NG should continue to monitor the IMS on a regular basis and take
appropriate action to closely manage the critical path.
29
PMO Project Schedule

Impact
High
Probability
Almost Certain
Time Criticality
Immediate
Change Requests and gap activities, rooted in insufficiencies relative to requirements management
and project estimation, coupled with an aggressive project schedule, contributed to an unreliable
project schedule.
Observations
Risks
1. Significant number of change requests

(CRs) can be directly linked to insufficiencies
in requirements management.
1. Under reporting of task slippage and delayed task completion to senior

Eligibility Modernization Program (EMP) management and stakeholders
does not provide the proper alerts for timely management action.
2. The initial NG schedule was compressed in

order to meet ACA deadlines, using
analogous estimating from prior projects as
the model for TEDS.
2. Exclusion of CRs and gap activities in the latest version of the IMS do
not ensure that the task will be completed as intended.
3. The August 2014 IV&V Monthly Executive

Status Report states that there was not
aggressive mitigation and monitoring of task
delays and target date completion.
Recommendations

Status Report states that must have CRs
that require approval and closure, continue
to remain open on the IMS Punch List. CRs
are incorporated in October 10, 2014 version
of IMS which is currently unapproved by the
State.
5. Significant items, including but not limited to
OCM and O&M, are missing from the project
schedule.
3. Unrepresented tasks have a negative impact on the project schedule.
1. NG should baseline and the State should approve the IMS to potentially
enable better tracking of project progress and forecasting resource
needs and timing.
2. NG should develop a process to identify and redeploy available
resources to behind schedule tasks as soon as potential task completion
delays are identified.
3. NG should continue to monitor task progress within the IMS on a regular
basis to support timely identification, mitigation and communication of
potential task slippage.
4. NG should continue to publish and provide up-to-date versions of the
IMS to the State on a regular basis to support visibility on upcoming
work and status. NG should continue to manage, monitor a short-term
(3 4 week) set of tasks by resource.
30
PMO Resource Management

Impact
Moderate
Probability
Almost Certain
Time Criticality
Long Term
Knowledgeable resources available for critical tasks are limited and resources are not adequately
represented and managed in the Integrated Master Schedule (IMS).
Observations
Risks
1. NGs resource management has been

ineffective.
1. Dependence on a limited group of adequately knowledgeable and

experienced State resources for project-critical activities increases the risk
of task slippage through bottlenecking or potential unavailability.
2. State Subject Matter Professional (SMP)

resources to help define requirements,
participate in design, and otherwise
complete the project continue to have
limited availability. These SMPs hold
significant leadership positions at
TennCare and have many other competing
priorities, particularly operating under a
mitigation plan that was necessary due to
delays in the system implementation of
TEDS.
Status Report states that NG made a
number of management changes,
replacing team managers who have been
on the project for an extended period of
time.
4. NG reported significant turnover of
Business Analysts (BA) and other
positions critical to the requirements and
design processes for TEDS.
2. High NG turnover results in a short term decrease in progress.

Changes in NG project management results in interruptions to the
intellectual continuity and overall project approach.
Lack of Avance product knowledge, as well as high turnover in NGs
BA population, contributes to increased learning curves and decreased
continuity, causing task delays and rework.
Recommendations
1. Enhance the State project team with additional members who are
knowledgeable in State business processes and requirements and can
work collaboratively with NG in defining business processes, defining
requirements and performing testing. These resources should have
authority to make binding decisions on behalf of the State.
2. NG should secure, assign and manage adequate (quality and quantity)
resources to critical tasks to support the mitigation of slippage and
potential risks to task completion.
31
PMO Project Reporting

Impact
Moderate
Probability
Almost Certain
Time Criticality
Long Term
The project schedule status reporting, for approximately the first ten months of the project, was not
adequate.
Observations
Risks
1. The weekly and monthly status reports

throughout the first several months of the
projects did not provide adequate
information on project performance.
1. Inadequate status reporting contributes to:
2. The State issued Control Letter 1 in May

2013 to require detailed performance data in
weekly status reports. However, the detailed
performance data, such as Schedule
Performance Index (SPI) did not appear in
weekly status reports until 8/13/2013.
3. Monthly status reporting from NG indicated
that project schedule performance was
green (less than 5% variance from plan)
from 2/6/2013 to 9/9/2013. However,
weekly status report performance data for
month of August 2013 indicated that actual
performance was greater than 10% variance
from plan. NG did not report a red status
for project scheduling until 10/7/2013, less
than three months before contracted go-live.
Diminished ability for the State to make tactical and / or strategic

decisions due to lack of timely information, leading to ineffective
responses to schedule delays.
Increased impact of identified and unidentified risks.
2. Without an approved baseline, appropriate schedule variance and trend
analysis can not be adequately performed.
3. Incorporating the SPI data without proper context does not provide the
appropriate alerts to senior management.
4. Inaccurate status reporting diminishes stakeholder confidence in the
established project management and communication processes.
Recommendations
1. NG should ensure that project schedule management processes are
being executed as documented.
2. NG should encourage timely State feedback and comments stemming
from status reports from project stakeholders to support communication
that addresses effective response mitigation steps.
32
PMO Risk & Issue Management

Impact
Low
Probability
Somewhat Likely
Time Criticality
Long Term
The execution of the risk and issue management processes was not adequate.
Observations
Risks
1. A number of significant risks were not

documented in the initial Risk Register at the
onset of the project, including:
1. Improperly or incompletely captured risks have not been adequately

identified, escalated, managed and addressed, which may impact
quality, accuracy and mitigation action.
Overall schedule feasibility;

Requirements sufficiency;
NG resource domain knowledge; and
Level and knowledge of State resources.
Status Report states that little progress has
been made in the resolution of open risks,
issues, action items and parking lot items.
3. Appropriate risk probabilities are not
consistently documented in the shared Risk
Register in accordance with the established
Risk Management Plan.
2. Unresolved / undocumented risks may become issues and impact

project milestones, schedule and quality.
3. Poor assessment of risk probability impacts managements ability to
appropriately mitigate risks and issues.
4. Lack of systematically recording progress and updates within the Risk
Register reduces the ability to effectively monitor and report the current
status of identified risks.
Recommendations
1. NG should execute and adhere to the established Risk Management
Plan processes to effectively identify, analyze, vet and document risks
and develop strategies to mitigate or avoid those risks.
2. NG should continually update, monitor and report risks on a weekly
basis to ensure that the appropriate processes and actions are taken
for all identified risks.
33
Operational &
Organizational
Readiness Findings
Operational & Organizational Readiness Organizational Readiness

Impact
High
Probability
Likely
Time Criticality
Long Term
Organizational Readiness was not sufficiently planned for, leading to a gap in overall organizational
management for the TEDS effort.
Observations
Risks
1. An Organizational Change Plan Strategy, or

similar document, has not yet been created to
support development of the TennCare futurestate organization under TEDS.
1. The organization may be inadequately staffed to support the

business processes of the new system, leading to an inability to
process applications efficiently.
Contractual obligations shifted from NG to

the State for ownership of the Organizational
Change materials.
Organization change was considered and
planned for but the State was overly focused
on development.
2. A thorough stakeholder analysis was not
conducted at the onset of the project, leading to
the potential for inadequate internal TennCare
and other external State stakeholder
involvement.
External State stakeholders may not have
been engaged as agents of change.
Internal TennCare stakeholders may not
have a clear understanding of post-TEDS
roles and responsibilities.
Continued on following page.
2. The system may exclude important concerns of external State

stakeholders, whose needs were not considered in the planning
process.
3. Potential misalignment between the future state organization and
TEDS functional needs may lead to communication gaps, as changeoriented communications may not be disseminated to the appropriate
individuals or roles.
4. Limited integration of end-to-end business processes with the TEDS
application may lead to a gap in business activities, creating a risk in
processing eligibility determinations.
5. Limited integration of individual job functions and roles in TEDS may
create gaps in operational tasks, leading to a business risk as the
agency processes eligibility determinations.
Recommendations
1. State should develop an Organizational Change Plan, including an
approach to managing an actionable communication plan,
organizational development and workforce transition plans.
35
Operational & Organizational Readiness Organizational Readiness

Impact
High
Probability
Likely
Time Criticality
Long Term
Organizational Readiness was not sufficiently planned for, leading to a gap in overall organizational
management for the TEDS effort.
Observations (continued from previous page)
Recommendations (continued from previous page)
3. An thorough analysis of job role changes and

responsibilities in the future state has not yet
been conducted, leading to a potential gap in
understanding of how job roles will change or
work performed will be impacted.
2. State should reassess external State stakeholder concerns to help

ensure stakeholder needs are still fully considered.
The State focused on the development of the

system that the roles would be supporting
prior to defining the specific roles and
responsibilities required to perform the task.
3. State should conduct a gap analysis to identify the differences and

impacts between current job functions and anticipated future job
functions.
4. State should identify the end-to-end business processes, including
roles pertaining to each activity, that are expected to take place in
the future state integrated with TEDS.
4. Business process documentation has not yet

been developed for the future state, leading to
a gap in understanding of how job processes
(e.g., hand-offs) will change.
The State focused on developing system
and policy training materials prior to
developing business process documentation,
since the processes were uncertain without
sufficient opportunity to fully learn the new
TEDS functionality.
36
Operational & Organizational Readiness OCM / Training

Impact
High
Probability
Almost Certain
Time Criticality
Short Term
OCM / Training approach for the TEDS application is insufficient to allow State employees to fully
perform job functions.
Key Observations
Risks
1. The modules within the TEDS Performance

Support Center represent the flow of a single
scenario for one role.
1. Users may be unable to fully perform responsibilities due to limited

TEDS exposure on a range of case scenarios.
Training curriculum, as of May 2014, did not

cover the full TEDS application due to the
incomplete development of TEDS, which
resulted in incomplete materials associated
with specific roles within TEDS.
2. Level of completion varies between Participant
Guide and Facilitator Guide. Participant Guide
requires significant additional detail prior to
Release 1.
Training materials provided within the
Facilitator Guide have all modules complete
with the exception of the Client Web Portal,
whereas the Participant Guide remains largely
incomplete.
2. Coordination between the expected go-live date and the timeframe

for development of training materials may lead to an education gap,
followed by a negative stakeholder perception for both State and
NG.
3. Insufficient level of detail of the training materials may prevent State
users from carrying out their job responsibilities in a timely or
accurate manner without significant additional training, overuse of
the help desk, or a drain on other resources (e.g., power users).
Recommendations
1. NG and the State should consider developing additional role and
scenario based training lessons to provide facilitators with a better
understanding specific to the training they will deliver to end-users.
2. NG to consider updating the Participant Guide so it reflects training
materials within the Facilitator Guide.
37
Operational & Organizational Readiness OCM / Training

Impact
High
Probability
Almost Certain
Time Criticality
Short Term
OCM / Training approach for the TEDS application is insufficient to allow State employees to fully
perform job functions.
Key Observations (continued from previous page)
3. Curriculum groups and roles were defined within the Training

Plan, however there was no indication they were used to design
role and scenario based training materials in the TEDS
Performance and Support Center.
3. NG to consider creating a document, with a mapping

of the roles, scenarios and required trainings, to
better define the State roles required to attend within
each training module.
Six curriculum groups were defined to contain roles for which

the curriculum would be divided. It appears training for all but
one of these groups are yet to be developed.
4. Train-the-trainer sessions to-date were not effective, due to the
low-level of TEDS application maturity and low attendance by
State personnel.
4. NG should still deliver a complete and

comprehensive final train-the-trainer session, and
include participants from other agencies.
5. State to consider developing a process to evaluate
the efficacy of the training materials for successful
job / task completion post-training sessions.
TEDS application presented in the Train-the-trainer (TTT)

sessions could only demonstrate defined-path scenarios (i.e.
If a trainer was asked to demonstrate an activity, the system
was not capable to demonstrate the function or would crash).
TTT sessions have not yet included other agencies.
5. Neither NG nor the State have plans to evaluate the efficacy of
train-the-trainer or end-user training sessions / materials.
The content of the Likert questions within the Training
Evaluation is not designed to test whether the TEDS training
is adequate for the trainee to perform their job function.
38
Operations and Maintenance Plan Service Management

Impact
High
Probability
Almost Certain
Time Criticality
Short Term
TEDS Operation & Maintenance Plan lacks sufficient definition and detail to service management
activities.
Observation
Risks
1. Review of the TEDS Operations and

Maintenance (O&M) Plan revealed insufficient
detail in the definition of O&M processes in the
following areas:
Lack of industry aligned, and insufficiently documented service

management processes will lead to:
No reference to industry aligned service

management processes / tools.
No evidence of detailed documentation to
support proposed processes.
No evidence of context of the integration

of different service management
processes.
Insufficient detail and alignment to an

O&M benefit measurement methodology
(performance measures).
1. Reactive, poor and / or inadequate response to production issues.

2. Lack of visibility to system performance.
3. Difficulty in identifying, sourcing and maintaining service
management tools and resources.
4. Inadequate performance against service level agreements.
5. Negative customers experience due to inadequate service levels.
Recommendations
1. Conduct comprehensive review of operations and maintenance
requirements, leveraging available industry best practices.
2. Define and document service level agreements.
3. Define processes owners and assign task specific responsibilities
for Operations and Management service management activities.
39
Operations and Maintenance Plan Demand Management

Impact
High
Probability
Almost Certain
Time Criticality
Short Term
TEDS Operation & Maintenance Plan lacks sufficient definition and detail of an integrated demand
management process.
Observation
Risks

detail to implement a comprehensive demand
management process:
Lack of industry aligned, and insufficiently documented demand

management strategy will lead to:
Proposed processes focus largely on

engineering tasks and lack integration of
voice of the customer.
No evidence of documentation regarding
types of change, release cycles and
prioritization criteria.
No evidence of a plan to identify
upcoming business requests and priorities
for future projects / initiatives.
1. Inefficiency identifying and reacting to future demand everything

is a priority but nothing gets done.
2. Difficulty translating change requests into value propositions.
3. Increase in O&M costs with lack of cross functional prioritization
criteria.
4. Insufficient ability to respond effectively and in timely manner to
changes in federal policy may result in system malfunctions.
Recommendations
No evidence of a plan to identify and react

to public policy changes (impact analysis
and integration with existing demand
management function).
1. Develop a comprehensive demand management process

incorporating State resources and processes.
No evidence of appropriate staffing plan

(number of FTEs, skill level, knowledge
management).
3. Develop human capital management strategy to address future

resource allocations and staffing needs.
2. Create business case template and estimating model to prioritize

approve or cancel requests.
4. Develop knowledge management plan to ensure that resources

adequately trained and job responsibilities can be shared.
40
Operations and Maintenance Plan Governance

Impact
Moderate
Probability
Almost Certain
Time Criticality
Short Term
TEDS Operation & Maintenance Plan lacks sufficient definition and detail of IT governance activities.
Observation
Risks

detail to implement an effective IT Governance
program for the O&M phase in the following
area(s):
Lack of industry aligned, and insufficiently documented IT governance

strategy will lead to:
Incomplete plan for mechanisms to

support IT-related decision making and
the enforcement of those decisions.
No clear indication of roles,

responsibilities and accountability for IT
operations.
No evidence provided of appropriate NG

human capital management strategy for
post go-live phase (number of FTEs, skill
level, knowledge management).
No evidence of clear and approved plan

or procedures for specific transition tasks
to follow up with transition decisions.
1. Ineffective decisions approval and execution process.

2. Inefficient allocation of resources.
3. Inconsistent risk mitigation process and oversight of compliance
regulatory mandates.
4. Insufficient ability to respond effectively and in timely manner to
transition decisions.
Recommendations
1. Establish responsibilities and accountability for making key
decisions within the organization.
2. Develop and continually update an IT organizational model
(determining the decisions making process, alignment with key
organizational stakeholders, etc.).
3. Develop human capital management strategy for post Go-Live
activities.
4. Develop transition strategy for post Go-Live activities.
41
Operational & Organizational Readiness Communications Management

Impact
Moderate
Probability
Almost Certain
Time Criticality
Long Term
Communication planning for the TEDS project does not engage all project stakeholders.
Observations
Risks
1. Communications planning documentation does not

fully describe and account for the needs of external
stakeholders (i.e., DOH, DCS, DHS, OIR and DOC).
1. Exclusion of certain stakeholder needs may adversely impact

the systems ability to meet all stakeholder needs, creating a
reputational risk or risk of negative perception and resistance
to new system operations.
The existing communications management

documentation focuses on NG and TennCare
stakeholders.
2. The State provides frequent, direct written and verbal
feedback to NG (executive and project leadership)
but does not consistently respond to TEDS(NG)
satisfaction surveys, which diminishes the survey
effectiveness.
NG requests feedback from State project
stakeholders on a monthly basis, but response
levels have varied and have been zero in recent
months.
2. Decreased lines of communications may create an ownership

or accountability risk between Northrop Grumman and the
State.
3. Delayed or non-execution of triggered communications events
may lead to resistance to change from all project stakeholders
who have not been adequately prepared for major
organizational and system changes.
Recommendations
1. State should conduct a more thorough and formal stakeholder
analysis and develop an improved and detailed actionable
communication plan.
2. State should respond to existing TEDS customer satisfaction
surveys and utilize as a communications mechanism between
NG and the State to help resolve issues identified in the survey
responses.
42
Operational & Organizational Readiness Communications Management

Impact
Moderate
Probability
Almost Certain
Time Criticality
Long Term
Communication planning for the TEDS project does not engage all project stakeholders.
Observations (continued from previous page)
3. Communications, both recurring and triggered, are

not actively tracked using the communications matrix,
which leads to gaps and lack of execution of
engaging all stakeholders.
3. State should assign a designated State TEDS communications

lead and update the Communications Matrix with recurring and
triggered communications, as they are executed.
Recurring meetings take place, based on

documentation reviews; however, responsibilities
for triggered communications are not established in
many cases.
4. No designated State role is responsible for
coordinating and executing a comprehensive
communication plan; rather, those in project
governance roles are also responsible for executing
upon each communication event.
State project governance roles maintain multiple
levels of responsibility related to the
communication plan.
43
Operational & Organizational Readiness Knowledge Transfer

Impact
Low
Probability
Somewhat Likely
Time Criticality
Long Term
Although informal knowledge transfer activities through ad hoc project meetings, the project has not
identified / followed a defined methodology to help ensure appropriate attendance / outcomes.
Observations
Risks
1. NG and the State regularly transfer knowledge

through project briefings, stakeholder meetings,
facilitated group sessions and ad hoc working
groups. However, these events are considered
regular project activities and are not structured to
help ensure that knowledge transfer is being
performed with all of the expected State
personnel.
1. A lack of attention to knowledge transfer may lead to gaps in

system, organizational, or operational knowledge as the project
transitions to TEDS operations.
A specific Knowledge Transfer Plan has not

been developed for this project and a focus on
knowledge transfer was not being addressed
until NG and the State began the System
Implementation Phase and planning for
Operations and Maintenance had began.
Recommendations
1. NG and the State to identify State roles critical to the remaining
project phases (e.g., testing, system implementation and ongoing
operations and maintenance) and require the roles to attend all
appropriate meetings and project activities to help ensure projectspecific knowledge is adequately transferred.
2. TEDS project team may want to consider developing a plan prior to
the operations phase with the expectations of specific knowledge
transfer activities that will be performed. This plan should be tiered
and prioritized into short-, medium- and long-term needs.
44
2014 KPMG LLP, a Delaware limited liability

partnership and the U.S. member firm of the KPMG
network of independent member firms affiliated with
KPMG International Cooperative (KPMG
International), a Swiss entity. All rights reserved.
NDPPS 256039
The KPMG name, logo and cutting through
complexity are registered trademarks or trademarks
of KPMG International.

State of Tennessee TEDS Assessment Report

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

State of Tennessee TEDS Assessment Report

Uploaded by

Copyright:

Available Formats

State of Tennessee

TEDS Assessment Report

January 9th, 2015

TEDS Risk Portfolio

Risk Ranking Scales

TEDS Potential Alternatives

TEDS Potential Alternatives High Level Timelines

Software Development Life Cycle (SDLC) Findings

Project Management Office Findings

Operational & Organizational Readiness Findings

TEDS Risk Portfolio

Requirements Quality & Completeness - Key requirements types were missed or

Requirements Traceability - TEDS requirements are incompletely or ineffectively

Software Development Life Cycle (SDLC)

Testing Testing approach to TEDS is insufficient to ensure a successful solution.

TEDS Risk Portfolio

Business Operating Model - A documented business operating model for Medicaid

System Architecture At least one component of the system architecture will be at

Eligibility Modernization Program (EMP) Architecture Governance - A documented

TEDS Risk Portfolio

Schedule Management - The management of the project has been unsuccessful in

Project Schedule - Change Requests and gap activities, rooted in insufficiencies

Resource Management Knowledgeable resources available for critical tasks are

Project Management Office (PMO)

TEDS Risk Portfolio

Organizational Readiness - Organizational Readiness was not sufficiently planned

Organizational Change Management (OCM) / Training OCM / Training approach

Demand Management - TEDS Operation & Maintenance Plan lacks sufficient

Communication Management - Communication planning for the TEDS project does

Operational & Organizational Readiness

Knowledge Transfer - Although informal knowledge transfer activities through ad hoc

Risk Ranking Scales: Impact

Project has the potential to be shut down until properly mitigated.

Considerable impact on project.

Project is impacted but can continue to operate.

Risk Ranking Scales: Probability

Event is expected to occur in most scenarios.

Event will probably occur in most circumstances.

Event should occur at some time.

Risk Ranking Scales: Time Criticality

TEDS Potential Alternatives

Leverage some current efforts and

Secure rapid procurement of

Potential deployment within Enhanced

Provide resources with

Inability to obtain resources with domain

Reduce risk by leveraging a

Costly effort with low probability of

Restart with Procurement of a New SI Vendor

Capacity and performance planning was

Secure rapid procurement Provide clear TN

Immediate need to remediate

unsupported software components and

Secure a vendor with

Secure a vendor with

Secure a vendor with proven

TEDS Potential Alternatives High Level Timeline

Critical Pre-Requisites for

Business Operating Model

Business Requirement Definition

CMS Funding Updates

TEDS Potential Alternatives High Level Timeline

MES Market Assessment

System Requirement Specifications