KEYWORDS: Data modeling, Process modeling, Modeling and simulation, Calibration, Databases, Performance modeling, Defense and security, Software, Software development, Defense systems
This paper presents a method that integrates Modeling and Simulation (M&S) Verification and Validation (V&V) as part
of the M&S design. Experience indicates that in the past, very few models were developed with V&V as part of the
design process. Formal V&V was usually done after the model had been released to the user community and was being
used by major programs to support major decisions. This has changed in recent years as declining resources have
resulted in a growing reliance on M&S. As awareness of the issues and the risks involved has increased, Department of
Defense (DoD) policies have been written that require V&V to be implemented as part of the M&S design, development,
and acquisition process. Many things can go wrong when a model is not carefully verified and validated. Not only does
lack of V&V make a model difficult, if not impossible, to use, but the model may fail to support its intended use. V&V
reduces the risks of developing an M&S that does not meet requirements or of using an inappropriate simulation to
support a decision. While risks can never be eliminated entirely, they can be quantified in a way that they provide the
decision maker with an indication of how high the cost of using an erroneous M&S result can be. This approach
therefore allows optimal decisions to be made. This paper gives a description of a methodology for implementing V&V
processes and documentation into the M&S design process.
The Department of Defense (DOD) is exercising a risk-based process for verifying, validating and accrediting models
and simulations (M&S) used in system acquisition. Test and laboratory facilities can potentially have even greater
potential negative consequences to a program than M&S if there are errors present in the test and analysis results, since
test results are usually considered closer to the "truth" than M&S results. This paper will discuss how the risk-based
M&S verification, validation and accreditation (VV&A) process is being applied to test and laboratory facilities, issues
associated with this different application of the process, and thoughts on the broader applicability of risk-based VV&A
beyond the current application.
To bring reality into models and simulations (M&S), the Department of Defense (DOD) combines constructive M&S
with real equipment operated by humans in field environments. When such a live, virtual, and constructive distributed
environment (LVC-DE) is assembled, there exist ample opportunities for success or failure depending on many issues.
Each M&S tool, along with the means used to connect it to the others, must be examined independently. The combined
M&S, the interfaces, and the data they exchange must be tested to confirm that the entire system is interoperable and is
achieving its intended goals. Verification and Validation (V&V) is responsible for systematically investigating, creating,
and documenting the artifacts needed to assess the credibility of such LVC-DE. The ultimate goal for V&V is to
evaluate the capability, the accuracy and the usability of such LVC-DE.
The Battlespace Modeling and Simulation V&V Branch has extensive experience performing V&V of LVC-DEs. In a
recent project, the task consisted of conducting V&V of the LVC-DE, the supporting infrastructure, and the legacy M&S
tools. From a V&V perspective, many things were done correctly; however, several adjustments were necessary to
improve the credibility of the LVC-DE. This paper will discuss lessons learned during the implementation and provide
recommendations for future LVC-DE applications.
KEYWORDS: Standards development, Error analysis, Data modeling, Process modeling, Systems modeling, Modeling and simulation, Samarium, Systems engineering, Defense and security, Decision support systems
This paper presents a risk-based Verification, Validation, and Accreditation (VV&A) process for Models and
Simulations (M&S). Recently, the emphasis on M&S used to support Department of Defense (DoD) acquisition has been
based on the level of resources allocated to establishing the credibility of the M&S on the risks associated with the
decision being supported by the M&S. In addition, DoD VV&A regulations recommend tailoring the V&V process to
allow efficient use of resources. However, one problem is that no methodology is specified for such tailoring processes.
The BMV&V has developed a risk-based process that implements tailoring of the VV&A activities based on risk. Our
process incorporates MIL-STD 3022 for new M&S. For legacy M&S, the process starts by first assessing the current risk
level of the M&S based on the credibility attributes of the M&S as defined through its Capability, Accuracy and
Usability, relative to the articulated Intended Use Statement (IUS). If the risk is low, the M&S is credible for application,
and no further V&V is required. If the risk is medium or high, the Accreditation Authority determines whether the M&S
can be accepted as-is or if the risk should be mitigated. If the Accreditation Authority is willing to accept the risks, then a
Conditional Accreditation is made. If the risks associated with using the M&S as-is are deemed too high to accept, then a
Risk Mitigation/Accreditation Plan is developed to guide the process. The implementation of such a risk mitigation plan
is finally documented through an Accreditation Support Package.
KEYWORDS: Data modeling, Error analysis, Defense and security, Modeling and simulation, Samarium, Safety, Coastal modeling, Systems modeling, Process modeling, Document management
This paper presents a new methodology for assessing the risks associated with the level of Verification,
Validation and Accreditation (VV&A) of a given Model and/or Simulation when used in support of major
decisions. As stated by DoD Instruction 5000.61 , "It is DoD policy that: Models and Simulations (M&S)
used to support major DoD decision-making organizations and processes shall be accredited for that
specific purpose by the DoD Component M&S Application Sponsor." This Instruction applies to "All
models and simulations developed, used, or managed by the DoD Components after the effective date of
this Instruction." The requirements cited above have set the need for VV&A of M&S at the forefront of
concerns for DoD and DoD-Components acquisition personnel. When an acquisition program involves a
large number of models, cost associated with VV&A can become enormous. There is a need therefore to
have a systematic approach for assessing and prioritizing the risks associated with the level to which
individual models have been verified, validated, and accredited. To provide decision makers with a
judicious way for determining the risks associated with using a given M&S, and the extent to which VV&A
work will be needed to meet these requirements, we have developed a methodology for assessing the risks
associated with the level of VV&A of a given M&S when used to support decision-making. This approach
parallels the formal DoD Risk Assessment procedure, but with application to the use of M&S, as it relates
to VV&A. Risks associated with the levels of VV&A are evaluated and assessed based on the following
criteria:
- Likelihood of the M&S being inaccurate and/or inappropriate for the application.
- Consequences of the M&S being inaccurate and/or inappropriate for the
application.
- Importance of the (acquisition) decision supported by the M&S.
- Level of Reliance of the (acquisition) decision on the M&S.
The assessment results are used to classify individual models into three risk categories
(Red for High Risk, Yellow for Medium Risk, and Green for Low Risk). All models
classified as High Risk, are subjected to further analysis and recommendations made as to
further work needed to reduce the risk to an acceptable level through formal VV&A.
Models assessed as Medium Risk are also further evaluated and recommendations made
with regards to the need for further risk mitigation. Models rated Low Risk are either not
of primary importance in supporting the (acquisition) decisions being made, or they may
have been assessed to have been satisfactorily verified, validated, and possibly accredited
to an acceptable level.
Conference Committee Involvement (7)
Modeling and Simulation for Defense Systems and Applications X
21 April 2015 | Baltimore, MD, United States
Modeling and Simulation for Defense Systems and Applications IX
6 May 2014 | Baltimore, MD, United States
Modeling and Simulation for Defense Systems and Applications VIII
30 April 2013 | Baltimore, Maryland, United States
Modeling and Simulation for Defense Systems and Applications VII
24 April 2012 | Baltimore, Maryland, United States
Modeling and Simulation for Defense Systems and Applications VI
26 April 2011 | Orlando, Florida, United States
Modeling and Simulation for Defense Systems and Applications V
6 April 2010 | Orlando, Florida, United States
Modeling and Simulation for Military Operations IV
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.