trade studies (or trade-off studies)

자료: http://www.robustdecisions.com/tradestudies.pdf

• 자료 제목: Trade Studies with Uncertain Information 
• 지은이: Dr. David G. Ullman(President, Robust Decisions Inc.), Brian P. Spiegel(Aerospace Electronic Systems, Honeywell)
• Sixteenth Annual International Symposium of the International Council On Systems Engineering (INCOSE)
• 날짜: 8 - 14 July 2006 

※ 메모: A trade study is the activity of a multidisciplinary team to identify the most balanced technical solutions among a set of proposed viable solutions (FAA 2004). These viable solutions are judged by their satisfaction of a series of measures or cost functions. These measures describe the desirable characteristics of a solution. They may be conflicting or even mutually exclusive. Trade studies, often called trade-off studies, are commonly used in the design of aerospace and automotive vehicles and the software selection process (Phillips et al 2002) to find the configuration that best meets conflicting performance requirements.

***

Abstract

During every stage of the design process, designers trade off performance, cost, and risk
in an evolutionary process whose goal is to find a satisfactory solution. This paper
explores a recent method to manage the trade study process especially when uncertainty
is pervasive and decisions are a mix of quantitative and qualitative information. We
believe that it is possible to support a trade study process that is sensitive to the
uncertainties in evolving system information, a key ingredient in managing risk,
robustness, changes and spiral development. In this paper we explore what is needed to
support such activities. To do so we follow an example as it gets increasingly complex
and realistic. As the issues addressed increase in computational need, we make use of
Accord, a decision support system base on Bayesian Team Support methods.

Introduction

With the increasing demand for complex and interrelated systems comes the challenges
of managing the decisions being made by a team of collaborating experts, each working
on a piece of the puzzle, and all vying for their share of the scarce resources. In early
stage design, this process is especially challenging as there is limited knowledge,
uncertainties are high, and the decisions made have far reaching effects on the directions
pursued thereafter, and hence the affordability, reliability/safety and effectiveness of the
final product. It is clearly more viable and less expensive to refine a design at the time
that it is being conceived. Therefore efforts towards making good decisions at this stage
have high payoffs.
During every stage of the design process designers trade off performance, cost,
and risk in an evolutionary process whose goal is to find a satisfactory solution. This
paper explores a recent method to manage the trade study process especially when
uncertainty is pervasive and decisions are a mix of quantitative and qualitative
information. We believe that it is possible to support a trade study process that is
sensitive to the uncertainties in evolving system information, a key ingredient in
managing risk, robustness, changes and spiral development.
In this paper we explore what is needed to support such activities. To do so we
follow an example as it gets increasingly complex and realistic. As the issues addressed
increase in computational need, we make use of Accord, a decision support system based
on Bayesian Team Support methods
What are Trade Studies
A trade study is the activity of a multidisciplinary team to identify the most balanced
technical solutions among a set of proposed viable solutions (FAA 2004). These viable
solutions are judged by their satisfaction of a series of measures or cost functions. These
measures describe the desirable characteristics of a solution. They may be conflicting or
even mutually exclusive. Trade studies, often called trade-off studies, are commonly used
in the design of aerospace and automotive vehicles and the software selection process
(Phillips et al 2002) to find the configuration that best meets conflicting performance
requirements.
The measures are dependent on variables that characterize the different potential
solutions. If the system can be characterized by a set of equations, we can write the
definition of the trade study problem as: Find the set of variables, xi that give the best
overall satisfaction to the measures:

T1 = f(x1, x2, x3…..)
T2 = f(x1, x2, x3…..)
T3 = f(x1, x2, x3…..)
TN = f(x1, x2, x3…..)

Where Tj is a target value and f(…) denotes some functional relationship among the
variables. Further, the equality between the target and the function may be a richer
relationship, as will be developed below. If the equations are linear, as in the production
volume example used as a starting point below, then this problem is solvable using linear
programming techniques. Generally, one or more of the targets is not fixed at a specific
value and it is desired to make these T values as large or small as possible. These are
generally referred to as cost functions and the other measures are treated as constraints.
If the situation was as described above formal optimization or linear programming
methods would work and there would be no need for this paper. However, in practice
needed information is:
• Uncertain - to be detailed below
• Evolving - new information is being developed that affects the trades
• Both qualitative and quantitative - at Honeywell the most important trade studies
have predominantly qualitative information
• Comes from conflicting sources - in systems engineering, many people have
some of the information needed; no one person has it all.
• The best choice comes from a team, building a shared mental model of the
situation.

Trade studies are essentially decision-making exercises - choose an optional concept or
course of action from a discrete or continuous set of viable alternatives. In the FAA
Systems Handbook (FAA 2004) the decision analysis matrix (aka Pugh's method) is
suggested to support the activities, but this method can not support uncertainty, a mix of
quantitative and qualitative information, or teams. To manage uncertainty, the authors
suggest supplementing point estimates of the outcome variables for each alternative with
computed or estimated uncertainty ranges. The Standard Approach to Trade Studies
(Felix 2004), an INCOSE paper from 2004 suggests a similar approach.
The NASA Systems Engineering Handbook (NASA 1995) suggests using multiattribute
utility theoretic (MAUT) or the Analytic Hierarchy Process (AHP). But, these
too are not good with uncertainty, mixed information and teams. The authors suggest
using probability based methods to maximize utility when uncertainty predominates, but
give little detail on how to approach this.
Another approach to supporting trade study information is to use the Bayesian
Team Support (BTS) methods. These methods were designed to manage the types of
information itemized in the list above. In this paper we will introduce BTS and apply it
to a trade study example to explore its applicability.
The Effect of Uncertainty on Trade Studies
What makes early system design and trade studies most challenging is that much of the
critical information is uncertain, evolving, and may be lacking in fidelity. Further, with
team members from many disciplines and with different values about what is important,
information may be conflicting. These terms, “uncertain”, “evolving”, “fidelity” and
“conflicting” permeate this paper and thus need clarification.
There are two types of uncertainty. The first, variability (i.e., stochastic
uncertainty, irreducible uncertainty, or common cause variability) is the result of the fact
that a system can behave in random ways. The weather will change, material properties
are variable, and there will always be chip junctions failures. In general, even though
some portion of variation can be controlled (e.g., insulation from weather changes) there
is always variation that is either uncontrollable or too expensive or difficult to warrant
controlling.
The second type of uncertainty results from the lack of knowledge about a
system (i.e., subjective uncertainty or state of knowledge uncertainty). It is a property of
the team members’ cumulative experience and the amount of time they have spent on the
current or similar concepts. Both types of uncertainty are direct causes of risk - as, in a
world with no variability and perfect knowledge, there would be no risk.
Typically, probability theory has been used to characterize both types of
uncertainty. Variability is usually analyzed using the frequentist approach associated with
traditional probability theory. However, traditional probability theory is not capable of
capturing lack of knowledge uncertainty, which, in early design is a large cause of risk.
One method for managing lack of knowledge uncertainty is Bayesian methods as will be
discussed later.
During the design process information is evolving. It begins with customers’
criteria and matures to the final drawings, specifications and code. Through this
development, the trade offs and risks are changing as the systems evolve. Managing this
evolution is crucial in systems as changes in one system will affect others. Sometimes
these interactions are missed leading to rework, compromised performance or system
failure.
As part of design activities, experts run simulations to predict performance and
cost. Early in the design process these simulations are at low levels of fidelity, some
possibly qualitative. Fidelity is the degree to which a model or simulation reproduces the
state and behavior of a real world object. To increase fidelity requires increased
refinement and increased costs to the project. Generally, with increased fidelity comes
increased knowledge, but not necessarily so as it is possible to use a high fidelity
simulation to model garbage and thus do nothing to reduce uncertainty. Often,
especially in early trade studies, there are no formal simulations and all or most of the
evaluations are qualitative. These evaluations are no less valid than detailed simulations.
In fact, it has been argued that gut-feel is the key to good decisions (Klein 1996, Gladwell
2005).
Finally, the team members’ interpretation of the available information may be
conflicting. Conflicting interpretations occur naturally due to differences in background,
role in the project, interpretation of the information, expertise, and problem solving style.
Conflicts are not good or bad, just different interpretations of the available information.
Traditional solution methods can not take these uncertainties into account. If they are
small compared to the actual values then these methods can be used assuming the
uncertainties exist to find a solution and then the take into account the uncertainties using
sensitivity analysis. However, if the uncertainties are significant, another philosophy
needs to be followed.
Details on Bayesian Team Support
Bayesian decision theory has its roots in the work of an obscure 18th century cleric (Rev.
Bayes) who worried about how to combine evidence in legal matters. However, its
modern form traces to the work of John Von Neumann, mathematician and computer
pioneer, in the 1940s; and J. Savage in the 1950s. In Savage’s formulation (Savage
1955), a decision problem has three elements: (1) beliefs about the world; (2) a set of
action alternatives; and (3) preferences over the possible outcomes of alternate actions.
Given a problem description, the theory prescribes that the optimal action to choose is the
alternative that Maximizes the Subjective Expected Utility (MSEU). Bayesian decision
theory excels in situations characterized by uncertainty and risk, situations where the
available information is imprecise, incomplete, and even inconsistent, and in which
outcomes can be uncertain and the decision-maker’s attitude towards them can vary
widely. Bayesian decision analysis can indicate not only the best alternative to pursue,
given the current problem description, but also whether a problem is ripe for deciding
and, if not, how to proceed to reach that stage.
As classical statistics revolutionized the discovery of knowledge in the early 20th
century, so Bayesian decision theory is revolutionizing the application of knowledge in
the 21st. This revolution is already underway. Microsoft, for instance, is investing
heavily in the use of Bayesian methods that improve the filtering and management of
information, daily barraging PC users. (생략/abbr......)