V&V 50 Draft Minutes – November 7-8, 2016 Page 1

DRAFT Minutes V&V 50

Verification and Validation in Computational
Modeling of Advanced Manufacturing
November 7-8, 2016

GE Global Research Center, Niskayuna, NY

One Research Circle
Niskayuna, NY 12309
Steinmetz Hall

Monday November 7, 2016 - Steinmetz Hall

8:30am CALL TO ORDER – Rachuri

WELCOME FROM GE GLOBAL RESEARCH - Christine Furstoss, Vice President and Technical
Director, Manufacturing & Materials GE Global Research – See Appendix A for slides
WELCOME FROM THE DOE - Dr. Mark Johnson, A brief overview of DOE Advanced
Manufacturing Office

8:50am INTRODUCTION OF PARTICIPANTS – All (13 members present - insuffient for a Quorum) -
See Attendance List Appendix B

9:05am AGENDA ADJUSTMENTS – (add brief introductions of working groups, to provide first-time
attendees background and context for deciding best way to participate in breakout groups)

9:15am CHAIR REPORT, COMMITTEE OVERVIEW, MEMBERSHIP UPDATE – Rachuri, Heller

• Membership Ballot update – ballots on contributing members will proceed to the V&V
standards committee
• Interested Members who have submitted paperwork need to first attend 2 face-to-face
meetings and submit ASME personnel forms
• Yucheng Liu serve from V&V 50 on the Symposium Organizing Committee
• Ade Makinde and Meisam Salahshoor liaisons to AMSC Additive Standard Collaborative

9:30am Introduction of working groups

• V&V Interactions with the Model Life Cycle
• V&V UQ for Data-centric/driven Modeling in Advanced Manufacturing
• V&V for Additive Manufacturing
• Terminology, Concepts, Relationships and Taxonomy for VVUQ in Advanced Manufacturing
• VVUQ for Subtractive/Other Manufacturing

10:00am ASME Procedures: Types of Documents and their Development

References and Resources – See Appendix C for document names and PDF document inserts.
V&V 50 Draft Minutes – November 7-8, 2016 Page 2

Notes on Discussion:
• Need to have the V&V standards committee provide an overview of V&V standards, how
they interrelate, and context for V&V 50 document development.
• Types of draft documents – Guide and Technical Report seem to be most suited to the work
we are doing.

10:45am Discussion of Use Case Analysis of V&V.

Notes on Discussion:
• Use cases illustrate the process; not the technology or the example. Methodology of how to
apply V&V standards in Advanced Manufacturing is the point of the use cases.
• Specifically, in 1) physics-based models, 2) data driven and hybrid models.
• Currently no template we’ve agreed upon. Ronay and Rumi using one format; Sudarsan
shared a couple (see May 2016 meeting minutes Appendix H, Appendix I and Appendix K,
posted on
https://cstools.asme.org/csconnect/CommitteePages.cfm?Committee=101978604&Action=
48062&ChooseTemplate=10288)
• V&V10.1 - An Illustration of the Concepts of Verification and Validation in Computational
Solid Mechanics (see Appendix D for Executive Summary, and available on C&S Connect at
https://cstools.asme.org/csconnect/CommitteePublications.cfm?PublicationID=2508&Com
mittee=100079931&NoToolbar=yes, login first)
• V&V10 working on a revision to their standard, which will address UQ more thoroughly.
• 10.1 is a narrative use case; could also have a code-based one, implementable.
• CAUTION – introducing use cases prematurly can cramp the process; don’t build the process
of applying V&V standards to advanced manufacturing around use cases. First the process,
then the use case illustrating it. (i.e don’t build a standard by examples)
• Don’t include use cases in the body of a guidance document; rather as an appendix or
supplement.
~3:00pm FOCUS AREA TEAM BREAKOUT SESSIONS (revised from draft agenda)
• 3-4 pm Terminology, Concepts, Relationships and Taxonomy for VVUQ in Advanced
Manufacturing. Room CR5 – See Appendix E, also posted in C&S Connect for presentation
o Requests – 1) suggest any additional standard references that need to be included;
and 2) review the definitions
• 4-5 V&V Interactions with the Model Life Cycle. – See Appendix F, available on C&S Connect
at
https://cstools.asme.org/csconnect/CommitteePublications.cfm?PublicationID=2508&Com
mittee=100079931&NoToolbar=yes, login first)
• 5-6 pm
o V&V for Additive Manufacturing. See Appendix G for example document shared.
o V&V UQ for Data-centric/driven Modeling in Advanced Manufacturing. – see
Appendix H for draft document shared. 2 outputs identified. 1) Technical Report. 2)
analysis of adopting CRISP DM for advanced Manufacturing.

6:00pm Adjourn Day 1

 V&V 50 Draft Minutes – November 7-8, 2016 Page 3

Tuesday November 8, 2016 – Room CR5

8:00am CALL TO ORDER – Rachuri

8:15am FOCUS AREA TEAM REPORT OUT

8:25 – Subtractive Manufacturing – Huijuan Dai
• Will provide use cases to the Model Life Cycle group and others as needed. What
are the expectations for this group? Not clear that it needs to be a separate group.
8:15-8:45 - Terminology, Concepts, Relationships and Taxonomy for VVUQ in Advanced
Manufacturing. (no additional report)

8:45-9:15 - V&V for Additive Manufacturing. – See Appendix G

• Consensus is that Additive does not have to be a separate working group.
9:15-9:45 - V&V UQ for Data-centric/driven Modeling in Advanced Manufacturing. See
Appendix H
9:45-10:15 - V&V Interactions with the Model Life Cycle. (no additional report)

Notes on Discussion
Restructure V&V50 Working Groups, and have new members join individual working groups, rather than
the main subcommittee (allows for quorums at meetings, and for focused emails and communications
within each working group)
Sending a note to all V&V50 members about the formation of the following Working Groups, and ask for
their interest in participating on them.
After informing entire subcommittee membership, initiate a ballot in C&S Connect to form the following
Working Groups under V&V50, and to add the following members to those groups.
V&V50 Working groups (5) could potentially have a session at V&V Symposium.
1. Terminology, Concepts, Relationships and Taxonomy for VVUQ in Advanced Manufacturing.
o Chair – Sankaran (Maha) Mahadevan
o Interested members: Gaurav Ameta, Tina Lee, Sanjay Jain, Sudarsan Rachuri
• V&V Interactions with the Model Life Cycle
o Chair – Joe Hightower
o Gordon Shao, Eric Sawyer, Rumi Ghosh, Aaron Bernreuther, Mark Benedict, Jeffrey
Fong, Bill Schindel
• VVUQ Challenges and Methods in Systems of Models
o Chair / Vice-Chair– Barron Bichon / Ade Makinde
o Eric Sawyer, Mahesh Mani, Rish Sant, Huijuan Dai, Matteo Bellucci, Maha, Suresh
Sundarraj (confirm?)
• VVUQ Methods in Data-driven and Hybrid models
o Chair / Vice-Chair – Rumi Ghosh / Ronay Ak
o Heather Reed, Gaurav Ameta, Tina Lee, Gordon Shao, Vincent Paquit, Joe Hightower,
Jeffrey Fong, Sagar Kamarthi
• VVUQ Applications in Process Technologies
o Chair / Vice-Chair - Huijuan/Ade
 V&V 50 Draft Minutes – November 7-8, 2016 Page 4

o Rish Sant, Gaurav Ameta, Mahesh Mani, Sudarsan Rachuri, Nathan Andrews (confirm),
Matteo Bellucci, Sagar Kamarthi (confirm), Maha Mahadevan, Mark Benedict

11:15am ANY OTHER BUSINESS – All

• ISEE Big Data conference December 2016 - http://cci.drexel.edu/bigdata/bigdata2016/
• AIAA Science and Technology Forum (AIAA SciTech). http://www.aiaa-scitech.org/
o Of particular interest to our committee might be the Non-Deterministic Approaches
Conference that is held as part of this forum. (Barron is conference chair this year.)
o Multiple sessions on UQ and V&V, as well as special sessions this year on approaches for
ICME models and on the DARPA EQUiPS program.
• ASTM Smart Manufacturing Journal http://www.astm.org/SSMS/ - Sudarsan Editor-in-Chief
• Next meeting: Tuesday, May 2, 8:00am-5:00pm prior to V&V Symposium, 2017. Las Vegas, NV,
Westin Hotel (UNLESS the group decides to have at another time to allow members to attend
the courses that will be offered on May 1-2

11:25am ACTION ITEMS

1. Request V&V standards committee to provide an overview at the V&V Symposium meetings,
and to consider a committee meeting schedule that allows committee members to attend one
of the workshops. – MH – UPDATE – Request sent 11/16/16
2. Send a note to all V&V50 members about the formation of the following Working Groups, and
ask for their interest in participating on them. - MH
3. Follow up on formation of Working Groups (set up ballots in C&S Connect) - MH
4. Send a Doodle poll third week of January 18-19, 2017 to schedule next telecon. – MH - done
5. Send out presentations from February 2016 meeting on V&V10 and V&V20 standards – MH
6. Share V&V40 draft with V&V50 – MH – Update – All members have access to ALL C&S drafts by
doing a quick search by Committee or keyword (V&V 40).
7. Obtain and share current draft of V&V10 update – MB
8. Forward suggestion to add an author index to the V&V Symposium program – MH – Update –
Request sent 11/16/16

11:30am ADJOURNMENT – Rachuri

Respectfully submitted,

Marian Heller
Staff Secretary, V&V 50
212-591-8514
hellerme@asme.org
 V&V 50 Draft Minutes – November 7-8, 2016 Page 5

Appendices:

• Appendix A –
• Appendix B - Attendance List
• Appendix C –
• Appendix D –
• Appendix E –
• Appendix F –
• Appendix G –
• Appendix H –
• Appendix –
• Appendix A –
• Appendix A –
• Appendix A –
• Appendix A –
•

Appendix B – Attendance List

Members Present
Ronay Ak, NIST Sanjay Jain (WebEx)
Matteo Bellucci Tina Lee (present 11/7, WebEx 11/8)
Mark Benedict, Vice Chair Sankaran Mahadevan
Barron Bichon Ade Makinde
Huijuan Dai Sudarsan Rachuri, Chair
Rumi Ghosh (WebEx) Heather Reed
Johnathan Goodsell (WebEx) Gordon Shao
Marian Heller, Secretary Simin Zhou (representing Rafi Haludeen)
Joe Hightower

Members Absent
Ravi Aglave Rafi Haludeen (represented by Simin Zhou)
Nathan Andrews Yucheng Liu
Hessam Babaee John Matlik
Ilias Bilionis Aditya Newalkar
Jeffrey Bischoff Soundar Srinivasan
James Coburn Suresh Sundarraj
Bala Deshpande Ben Thacker
Chris Freitas Cumali Semetay
Ghanshyam Gaudani Sean Whitsitt, Cont. Member

Visitors Present Dean Bartles

Richard Arthur Roger Burkhart (WebEx)
 V&V 50 Computer Modeling for Advanced Manufacturing Draft Page 6
Minutes – November 7-8, 2016

Jeffrey Fong Rish Sant

Mark Johnson (WebEx) Eric Sawyer
Salgar Kamarthi Bill Schindel
Mahesh Mani Steve Weinman
Vincent Pacquit Lang Yuan

Appendix C – References for Types of Documents and How Standards Committees Work (available for
download on C&S Connect, V&V50 committee page  Presentation and Resource Documents 
November 2016 – Shared Documents.)

• ASME Codes & Standards Development Committee Procedures with S&T Supplemental (see
PDF inserts below)
o Sections S6.4-S6.6 (Technical Reports, Guides and Draft standards for Trial Use
o Appendix IV: Draft Preparation, Dating of Referenced Standards, and Incorporation
of Copyrighted Material
o Appendix IX: Draft Standards for Trial Use
o Appendix X: Procedure for Development of ASME Technical Reports
• Committee Handbook –
o Section 6 – How Does the Work Get Done? (see PDF insert below)
• S&C Volunteer Training Modules - These cover the general categories of Administrative,
Process, and Legal. They are accessible at the Codes and Standards Web site at
http://cstools.asme.org/TrainingModules.cfm

ASME Codes & Standards Development Committee Procedures with S&T Supplemental Sections S6.4-
S6.6 (Technical Reports, Guides and Draft standards for Trial Use)
V&V 50 Computer Modeling for Advanced Manufacturing Draft Page 7
Minutes – November 7-8, 2016
 V&V 50 Computer Modeling for Advanced Manufacturing Draft Page 8
Minutes – November 7-8, 2016

ASME Codes & Standards Development Committee Procedures with S&T Supplemental, Appendix IV:
Draft Preparation, Dating of Referenced Standards, and Incorporation of Copyrighted
Material
 V&V 50 Computer Modeling for Advanced Manufacturing Draft Page 9
Minutes – November 7-8, 2016

ASME Codes & Standards Development Committee Procedures with S&T Supplemental, Appendix IX:
Draft Standards for Trial Use and Appendix X: Procedure for Development of ASME Technical Reports
 V&V 50 Computer Modeling for Advanced Manufacturing Draft Page 10
Minutes – November 7-8, 2016

Committee Handbook Section 6 – How Does the Work Get Done?

 V&V 50 Computer Modeling for Advanced Manufacturing Draft Page 11
Minutes – November 7-8, 2016

Appendix D – V&V10.1 Executive Summary (this is not in the same format as the actual document)

AN ILLUSTRATION OF THE CONCEPTS OF VERIFICATION AND VALIDATION IN COMPUTATIONAL SOLID

MECHANICS

1 EXECUTIVE SUMMARY

This Standard describes a simple example of verification and validation (V&V) to illustrate some of the
key concepts and procedures presented in V&V 10. The example is an elastic, tapered, cantilever, box
beam under nonuniform static loading. The validation problem entails a uniform loading over half the
length of the beam. The response of interest is the tip deflection. The validation test plan and the
metrics and accuracy requirements for comparing the calculated responses with measurements are
specified in the V&V Plan, which is developed in the first phase of the V&V program. In setting validation
requirements and establishing a budget for the V&V program, the V&V Plan considers the level of risk in
using the model for its intended purpose. Successfully meeting the V&V requirements means that the
computational model for the tapered beam has been validated for the intended use discussed in this
document, viz., predicting the response of a tapered beam tested in the laboratory.

To encompass as much of the general V&V process as possible in this example, a computational model
was developed specifically for the tapered beam problem, even though it is more likely that a general-
purpose finite-element code would be used in practice. The conceptual model is a Bernoulli–Euler beam
for which the governing equations are solved with the finite-element method. The computational model
was verified (checked for proper programming of the mathematical model and the solution procedure)
by comparing computed values of tip displacement with an analytical solution to a relevant but simpler
problem. A mesh refinement study initially revealed that the model did not converge at the expected
theoretical rate. Further diagnosis revealed a programming error, correction of which led to the proper
convergence rate. Knowing the allowable error due to lack of convergence then allowed an appropriate
level of mesh refinement to be selected.

For validation (comparing with experimental results), 10 virtual trials of the same test were performed
to quantify the distribution of results due to unintended variations in material properties, construction
of the test specimens, and test execution. Other virtual tests were conducted to characterize
uncertainties in selected model input parameters, namely rotational support stiffness and elastic
modulus. 1

The following two validation approaches were considered:

(a) a case where uncertainty data were not available and obtained instead from subject matter
experts.
(b) a case where uncertainty data were available from repeat tests and calculations.

In both cases the same metric was employed to demonstrate the use of uncertainty information in the
model test comparison. The validation metric is a measure of the relative error between the calculated
and measured tip deflection of the beam.
 V&V 50 Computer Modeling for Advanced Manufacturing Draft Page 12
Minutes – November 7-8, 2016

The same model was used in both validation cases. In each case, the metric was compared to an
accuracy requirement of 10%. In both cases the model was validated successfully. Had the validation
been unsuccessful, it would have been necessary to correct any model deficiencies, collect additional or
improved experimental data, or relax the validation requirement.

Appendix E: V&V 50 Task Group on Terminology, Concepts, Relationships and Taxonomy for VVUQ in
Advanced Manufacturing

Group Members: Sankaran Mahadevan (lead), Johnathan Goodsell, Sanjay Jain, Tina Lee, Gaurav Ameta

Approach for this task

In each topic listed below, the approach will be as follows:
(1) survey the definitions in existing V&V standards (e.g., ASME, IEEE, AIAA, ISO, DoD etc.)
(2) explore applicability of existing definitions to advanced manufacturing
(3) suggest adaptations/extensions of existing definitions to advanced manufacturing
(4) suggest definitions of new concepts unique to advanced manufacturing

Outline of Topics
1. Verification
• Definition
• Code verification
• Solution verification
• Error estimation
• Accuracy requirements
2. Validation
• Definition
• System response quantities (SRQ)
• Validation domain
• Application domain
• Accuracy requirements
• Validation metrics
• Validation hierarchy
3. Calibration
• Model inputs, model parameters, model discrepancy
• Physics-based models
• Data-driven models
• Data types (precise vs. imprecise; ordered vs. unordered; sparse data; big data;
qualitative data; text, voice and image data)
• Fusion of heterogeneous data
4. Uncertainty quantification
• Aleatory sources
• Epistemic sources  data, models
• Uncertainty aggregation (roll-up)
• Predictive capability assessment
• Quantification of Margins and Uncertainty (QMU)
5. Additional topics?
 V&V 50 Computer Modeling for Advanced Manufacturing Draft Page 13
Minutes – November 7-8, 2016

Draft Group Descriptions

Terminology, Concepts, Relationships and Taxonomy for VVUQ in Advanced Manufacturing. For
topics in VVUQ in advanced Manufacturing, the approach will be as follows:
(1) survey the definitions in existing V&V standards (e.g., ASME, IEEE, NASA, AIAA, ISO, DoD etc.)
(2) explore applicability of existing definitions to advanced manufacturing
(3) suggest adaptations/extensions of existing definitions to advanced manufacturing
(4) suggest definitions of new concepts unique to advanced manufacturing

V&V for Additive Manufacturing is a focused topic area to explore V&V issues in a context. It is on a
scale that this nascent group can engage with and make progress on. There is real opportunity to
potentially engage with the software industry that is creating the first AM commercial models and help
define best practices based on lessons learned in prior ASME V&V efforts.

V&V UQ for Data-centric/driven Modeling in Advanced Manufacturing. Focuses on understanding

VVUQ issues/problems the manufacturing industry tackles for data centric/driven modeling in advanced
manufacturing. Initial work is focused on collecting use cases with focus on data driven modeling and
the open research questions that are needed to be answered in this context. The main focus is to
understand the manufacturing system from observed data.

V&V Interactions with the Model Life Cycle. Treating the model as a product consumed in operational
use, the work will focus on understanding a variety of use cases, model types, and the model life cycle.
They will develop a document to provide guidance and methods across the life cycle, accounting for the
variety of use cases and variety of models.