Analyzing Uncertainty and Model Assumptions

Analyzing Uncertainty

Uncertainty analysis investigates the uncertainty of variables that are used in decision-making problems in which observations and models represent the knowledge base. In other words, uncertainty analysis aims to make a technical contribution to decision-making through the quantification of uncertainties in the relevant variables.

In physical experiments uncertainty analysis, or experimental uncertainty assessment, deals with assessing the uncertainty in a measurement. An experiment designed to determine an effect, demonstrate a law, or estimate the numerical value of a physical variable will be affected by errors due to instrumentation, methodology, presence of confounding effects and so on. Experimental uncertainty estimates are needed to assess the confidence in the results. A related field is design of experiments.

Likewise in numerical experiments and modeling uncertainty analysis draws upon a number of techniques for determining the reliability of model predictions, accounting for various sources of uncertainty in model input and design. A related field is sensitivity analysis.

A calibrated parameter does not necessarily represent reality, as reality is much more complex. Any prediction has its own complexities of reality that cannot be represented uniquely in the calibrated model; therefore, there is a potential error. Such error must be accounted for when making management decisions on the basis of model outcomes.

Analyzing Assumptions

Statistics, like all mathematical disciplines, does not infer valid conclusions from nothing. Inferring interesting conclusions about real statistical populations almost always requires some background assumptions. Those assumptions must be made carefully, because incorrect assumptions can generate wildly inaccurate conclusions.

Here are some examples of statistical assumptions.

• Independence of observations from each other (this assumption is an especially common error).
• Independence of observational error from potential confounding effects.
• Exact or approximate normality of observations.
• Linearity of graded responses to quantitative stimuli, e.g. in linear regression.

Classes of assumptions – There are two approaches to statistical inference: model-based inference and design-based inference. Both approaches rely on some statistical model to represent the data-generating process. In the model-based approach, the model is taken to be initially unknown, and one of the goals is to select an appropriate model for inference. In the design-based approach, the model is taken to be known, and one of the goals is to ensure that the sample data are selected randomly enough for inference.

Statistical assumptions can be put into two classes, depending upon which approach to inference is used.

• Model-based assumptions. These include the following three types:
  • Distributional assumptions. Where a statistical model involves terms relating to random errors, assumptions may be made about the probability distribution of these errors. In some cases, the distributional assumption relates to the observations themselves.
  • Structural assumptions. Statistical relationships between variables are often modelled by equating one variable to a function of another (or several others), plus a random error. Models often involve making a structural assumption about the form of the functional relationship, e.g. as in linear regression. This can be generalised to models involving relationships between underlying unobserved latent variables.
  • Cross-variation assumptions. These assumptions involve the joint probability distributions of either the observations themselves or the random errors in a model. Simple models may include the assumption that observations or errors are statistically independent.
• Design-based assumptions. These relate to the way observations have been gathered, and often involve an assumption of randomization during sampling.

The model-based approach is much the most commonly used in statistical inference; the design-based approach is used mainly with survey sampling. With the model-based approach, all the assumptions are effectively encoded in the model.

Checking assumptions

Given that the validity of any conclusion drawn from a statistical inference depends on the validity of the assumptions made, it is clearly important that those assumptions should be reviewed at some stage. Some instances—for example where data are lacking—may require that researchers judge whether an assumption is reasonable. Researchers can expand this somewhat to consider what effect a departure from the assumptions might produce. Where more extensive data are available, various types of procedures for statistical model validation are available—e.g. for regression model validation.