March 2006 Doc.: IEEE 802.

11-06/0333r0

Introduction to Measurement Uncertainty

Date: 2006-3-02
Authors:
Name Company Address Phone email
Dr. Michael D. Foegelle ETS-Lindgren 1301 Arrow Point Drive (512) 531-6444 Foegelle@ets-lindgren.com
Cedar Park, TX 78613

Notice: This document has been prepared to assist IEEE 802.11. It is offered as a basis for discussion and is not binding on the contributing individual(s) or organization(s). The material in
this document is subject to change in form and content after further study. The contributor(s) reserve(s) the right to add, amend or withdraw material contained herein.

Release: The contributor grants a free, irrevocable license to the IEEE to incorporate material contained in this contribution, and any modifications thereof, in the creation of an IEEE
Standards publication; to copyright in the IEEE’s name any IEEE Standards publication even though it may include portions of this contribution; and at the IEEE’s sole discretion to permit
others to reproduce in whole or in part the resulting IEEE Standards publication. The contributor also acknowledges and accepts that this contribution may be made public by IEEE 802.11.

Patent Policy and Procedures: The contributor is familiar with the IEEE 802 Patent Policy and Procedures <http:// ieee802.org/guides/bylaws/sb-bylaws.pdf>, including the statement
"IEEE standards may include the known use of patent(s), including patent applications, provided the IEEE receives assurance from the patent holder or applicant with respect to patents
essential for compliance with both mandatory and optional portions of the standard." Early disclosure to the Working Group of patent information that might be relevant to the standard is
essential to reduce the possibility for delays in the development process and increase the likelihood that the draft publication will be approved for publication. Please notify the Chair
<stuart.kerry@philips.com> as early as possible, in written or electronic form, if patented technology (or technology under patent application) might be incorporated into a draft standard being
developed within the IEEE 802.11 Working Group. If you have questions, contact the IEEE Patent Committee Administrator at <patcom@ieee.org>.

Submission Slide 1 Dr. Michael D. Foegelle, ETS-Lindgren

March 2006 Doc.: IEEE 802.11-06/0333r0

Abstract

This presentation introduces the common industry

concept of Measurement Uncertainty to represent the
quality of a measurement.
Other common terms such as accuracy, precision, error,
repeatability, and reliability are defined and their
relationship to measurement uncertainty is shown.
Basic directions on calculating uncertainty and an
example are included.

Submission Slide 2 Dr. Michael D. Foegelle, ETS-Lindgren

March 2006 Doc.: IEEE 802.11-06/0333r0

Overview

• Definitions
• Measurement Uncertainty
– Type A Evaluations
– Type B Evaluations
– Putting It All Together – RSS
– Reporting Uncertainty
– Special Cases
• Example Uncertainty Budget
• Summary
• References

Submission Slide 3 Dr. Michael D. Foegelle, ETS-Lindgren

March 2006 Doc.: IEEE 802.11-06/0333r0

Definitions

• Error – The deviation of a measured result from the

correct or accepted value of the quantity being
measured.

Error

• There are two basic types of errors, random and

systematic.
Submission Slide 4 Dr. Michael D. Foegelle, ETS-Lindgren
March 2006 Doc.: IEEE 802.11-06/0333r0

Definitions

• Random Errors – cause the measured result to deviate

randomly from the correct value. The distribution of
multiple measurements with only random error
contributions will be centered around the correct value.

• Some Examples
– Noise (random noise)
– Careless measurements
– Low resolution instruments
– Dropped digits Random Errors

Submission Slide 5 Dr. Michael D. Foegelle, ETS-Lindgren

March 2006 Doc.: IEEE 802.11-06/0333r0

Definitions

• Systematic Errors – cause the measured result to deviate

by a fixed amount in one direction from the correct
value. The distribution of multiple measurements with
systematic error contributions will be centered some
fixed value away from the correct value.

• Some Examples:
– Mis-calibrated instrument
– Unaccounted cable loss

Systematic Errors

Submission Slide 6 Dr. Michael D. Foegelle, ETS-Lindgren

March 2006 Doc.: IEEE 802.11-06/0333r0

Definitions
• Measurements typically contain some combination of
random and systematic errors.
• Precision is an indication of the level of random error.
• Accuracy is an indication of the level of systematic error.
• Accuracy and precision are typically qualitative terms.

Low Precision Low Precision High Precision High Precision

Low Accuracy High Accuracy Low Accuracy High Accuracy
Submission Slide 7 Dr. Michael D. Foegelle, ETS-Lindgren
March 2006 Doc.: IEEE 802.11-06/0333r0

Definitions
• Measurement Uncertainty combines these concepts into a
single quantitative value representing the total expected
deviation of a measurement from the actual value being
measured.
– Includes a statistical confidence in the resulting uncertainty.
– Contains contributions from all components of the measurement
system, requiring an understanding of the expected statistical
distribution of these contributions.
– By definition, measurement uncertainty does not typically contain
contributions due to the variability of the DUT.
• The “correct” value of a measurement is the value generated by the DUT
at the time it is tested.
• Variability of the DUT cannot be pre-determined.
• Still, the uncertainty of a particular measurement result will include this
variability.
Submission Slide 8 Dr. Michael D. Foegelle, ETS-Lindgren
March 2006 Doc.: IEEE 802.11-06/0333r0

Definitions

• Repeatability refers to the ability to perform the same

measurement on the same DUT under the same test
conditions and get the same result over time.
• By repeating the test setup between measurements of a
stable DUT, a statistical determination of System
Repeatability can be made. This is simply the level of
random error (precision) of the entire system, including
the contribution of the test operator, setup, etc.

Submission Slide 9 Dr. Michael D. Foegelle, ETS-Lindgren

March 2006 Doc.: IEEE 802.11-06/0333r0

Definitions

• Reproducibility typically refers to the stability of the

DUT and the ability to reproduce the same
measurement result over time using a system with a
high level of repeatability.
• More generally, it refers to achieving the same
measurement result under varied conditions.
– Different test equipment
– Different DUT
– Different Operator
– Different location/test lab

Submission Slide 10 Dr. Michael D. Foegelle, ETS-Lindgren

March 2006 Doc.: IEEE 802.11-06/0333r0

Definitions

• Reliability refers to producing the same result in

statistical trials. This would typically refer to the
stability of the DUT, and has connotations of
operational reliability of the DUT.
• Correction - value added algebraically to the
uncorrected result of a measurement to compensate for
systematic error.
• Correction Factor - numerical factor by which the
uncorrected result of a measurement is multiplied to
compensate for systematic error.
• Resolution – indicates numerical uncertainty of test
equipment readout. Actual uncertainty may be larger.
Submission Slide 11 Dr. Michael D. Foegelle, ETS-Lindgren
March 2006 Doc.: IEEE 802.11-06/0333r0

Measurement Uncertainty

• A measurement uncertainty represents a statistical level

encompassing the remaining unknown error in a
measurement.
• If the actual value of an error is known, then it is not
part of the measurement uncertainty. Rather, it should
be used to correct the measurement result.
• The methods for determining a measurement
uncertainty have been divided into two generic classes:
– Type A evaluation produces a statistically determined
uncertainty based on a normal distribution.
– Type B evaluation represents uncertainties determined
by any other means.
Submission Slide 12 Dr. Michael D. Foegelle, ETS-Lindgren
March 2006 Doc.: IEEE 802.11-06/0333r0

Type A Evaluations

• Uncertainties are determined through Type A evaluation

by performing repeated measurements and determining
the statistical distribution of the results.
• This approach works primarily for random
contributions.
– Repeated measurements with systematic deviations from a known
correct value gives an error value that should be corrected for.
• However, when evaluating the resulting measurement,
the effect of many systematic uncertainties combine with
random uncertainties in such a way that their effect can
be determined statistically.
– Eg. A systematic offset in temperature can cause an increase in the
random thermal noise in the measurement result.
Submission Slide 13 Dr. Michael D. Foegelle, ETS-Lindgren
March 2006 Doc.: IEEE 802.11-06/0333r0

Type A Evaluations

• Type A evaluation is based on the standard deviation of

repeat measurements, which for
_ n measurements with
results qk and average value q, is approximated by:
n
1
s(qk )   k
(n  1) k 1
( q  q ) 2

• The standard uncertainty contribution ui of a single

measurement qk is given by:
ui  s ( q k )
• If n measurements are averaged together, this becomes:
s ( qk )
ui  s(q ) 
n
Submission Slide 14 Dr. Michael D. Foegelle, ETS-Lindgren
March 2006 Doc.: IEEE 802.11-06/0333r0

Type B Evaluations

• For cases where Type A evaluation is unavailable or

impractical, and to cover contributions not included in
the Type A analysis, a Type B analysis is used.
– Determine potential contributions to the total meas. uncertainty.
– Determine the uncertainty value for each contribution.
• Type A evaluation.
• Manufacturer’s datasheet.
• Estimate a limit value.
Note: Contribution must be in terms of the variation in the measured
quantity, not the influence quantity.
– For each contribution, choose expected statistical distribution and
determine its standard uncertainty.
– Combine resulting uis and calculate the expanded uncertainty.

Submission Slide 15 Dr. Michael D. Foegelle, ETS-Lindgren

March 2006 Doc.: IEEE 802.11-06/0333r0

Type B Evaluations

• There are a number of common distributions for

99.7%
uncertainty contributions:
• Normal distribution: 68%

Ui
ui 
k
where Ui is the expanded
uncertainty of the
contribution and k is the
coverage factor (k = 2
for 95% confidence).
• Examples: -4s -3s -2s -1s 0 1s 2s 3s 4s

– Results of Type A evaluations

– expanded uncertainties of components 95%
Submission Slide 16 Dr. Michael D. Foegelle, ETS-Lindgren
March 2006 Doc.: IEEE 802.11-06/0333r0

Type B Evaluations

• Rectangular distribution – measurement result has an

equal probability of being anywhere within the range of
–ai to ai.
ai
ui 
3
• Examples:
– Equipment manufacturer ±
accuracy values (not from
standard uncertainty budget)
– Equipment resolution limits. -2 a i -ai 0 ai 2ai

– Any term where only maximal

100%
range or error is known.
Submission Slide 17 Dr. Michael D. Foegelle, ETS-Lindgren
March 2006 Doc.: IEEE 802.11-06/0333r0

Type B Evaluations

• U-shaped distribution –
measurement result has a higher
likelihood of being some value -2 ai -ai 0 ai 2ai

above or below the median than

being at the median.
ai
ui 
2
-2 ai -ai 0 ai 2ai

• Examples:
– Mismatch (VSWR)
– Distribution of a sine wave
– 5% Resistors (Culling)
Submission Slide 18 Dr. Michael D. Foegelle, ETS-Lindgren
March 2006 Doc.: IEEE 802.11-06/0333r0

Type B Evaluations

• Triangular distribution – non-normal distribution with

linear fall-off from maximum to zero.
ai
ui 
6

• Examples:
– Alternate to rectangular or
normal distribution when
distribution is known to
peak at center and has a -2 ai -ai 0 ai 2ai

known maximum
expected value. 100%

Submission Slide 19 Dr. Michael D. Foegelle, ETS-Lindgren

 March 2006 Doc.: IEEE 802.11-06/0333r0

Type B Evaluations

• Another Look

99.7%

68%
Normal Distribution Rectangular Distribution U-Shaped Distribution Triangular Distribution

-4 -3 -2 -1 0 1 2 3  ai

4-2 -ai 0 ai 2a-2
i ai -ai 0 ai 2a-2
i ai -ai 0 ai 2ai

Submission
95% 100% Slide 20 Dr. Michael D. Foegelle, ETS-Lindgren
100%
March 2006 Doc.: IEEE 802.11-06/0333r0

Putting It All Together - RSS

• Once standard uncertainties have been determined for

all components, including any Type A analysis, they are
combined into a total standard uncertainty (the
combined standard uncertainty, uc), for the resultant
measurement quantity using the root sum of squares
method:
N
uc   ui
2

i 1

where N is the number of standard uncertainty

components in the Type B analysis.
• The combined standard uncertainty is assumed to have
a normal distribution.
Submission Slide 21 Dr. Michael D. Foegelle, ETS-Lindgren
March 2006 Doc.: IEEE 802.11-06/0333r0

Reporting Uncertainty

• The standard uncertainty is the common term used for

calculations. It represents a ±1s span (~68%) of a
normal distribution.
• Typically, measurement uncertainties are expressed as
an Expanded Uncertainty, U = k uc, where k is the
coverage factor.
• A coverage factor of k=2 is typically used, representing
a 95% confidence that the measured value is within the
specified measurement uncertainty.
• Reporting of expanded uncertainties must include both
the uncertainty value and either the coverage factor or
confidence interval in order to assure proper use.
Submission Slide 22 Dr. Michael D. Foegelle, ETS-Lindgren
 March 2006 Doc.: IEEE 802.11-06/0333r0

Special Cases

• For Type A analyses with only a small number of

samples, the standard coverage factor is insufficient to
ensure that the expanded uncertainty covers the
expected confidence interval. Must use variable kp.
N-1 1 2 3 4 5 6 7 8 10 20 50 
kp 14.0 4.53 3.31 2.87 2.65 2.52 2.43 2.37 2.28 2.13 2.05 2.00

• RSS math works for values in dB! However,

distribution of a linear value may change when
converted to dB.
– Uncertainties typically always determined in measurement output
units.
Submission Slide 23 Dr. Michael D. Foegelle, ETS-Lindgren
March 2006 Doc.: IEEE 802.11-06/0333r0

Special Cases

• Not all distributions are symmetrical!

– Can develop asymmetrical uncertainties (+X/-Y) treating
asymmetric inputs separately.
– Can separate random portion of uncertainty from systematic
portion and apply a systematic error correction to measurement.
(Convert asymmetric uncertainty to symmetric uncertainty.)
error correction = (X+Y)/2, U = (X-Y)/2

Submission Slide 24 Dr. Michael D. Foegelle, ETS-Lindgren

March 2006 Doc.: IEEE 802.11-06/0333r0

Example Uncertainty Budget

Contribution Source Value Unit Distribution u_j (dB)
Mismatch: Transmit Side 0.00 dB U-Shaped 0.00
Analyzer Output Port Source Reflectivity Manufacturer -35.00 dB
Analyzer Output Port VSWR 1.04

• Antenna Input Port VSWR (1775-2000)

Antenna Input Port Reflectivity
Measured 1.45
-14.72 dB
Cable Loss (S21 & S12) Measured 8.00 dB

Mismatch: Receive Side 0.01 dB U-Shaped 0.00

Analyzer Input Port Load Reflectivity Manufacturer -42.00 dB
Analyzer Input Port VSWR 1.02
Antenna Output Port VSWR Measured 1.35
Antenna Output Port Reflectivity -16.54 dB
Cable Loss (S21 & S12) Measured 3.00 dB

Network Analyzer Measurement Uncertainty Manufacturer 0.40 dB Rectangular 0.23

(Full Two-Port Calibration, 50 dB path loss, Wide Dynamic Range device)

Transmit Cable Loss Variation Measured 0.05 dB Rectangular 0.03

(Due to flexing, etc.)

Mounting Accuracy: Reference Antenna Calculated 0.00 Rectangular 0.00

Antenna Mounting (PLS Laser Aligned & Custom Mounts) 0.13 inches
Range length 14.50 feet

Reference Antenna Gain Uncertainty Manufacturer 0.22 dB Normal 0.11

Miscellaneous Uncertainty CTIA 2.1 G.13 0.20 dB Normal 0.10

Total Uncertainty, u_c Type B RSS 0.28

Expanded Uncertainty, U k=2 0.55
Validity Range: 1775-2000 MHz
Submission Slide 25 Dr. Michael D. Foegelle, ETS-Lindgren
March 2006 Doc.: IEEE 802.11-06/0333r0

Summary

• This presentation gives common definitions for various

terms that have been used and misused in the TGT
draft.
• The concept of measurement uncertainty has been
introduced as the industry standard replacement for
terms such as accuracy, precision, repeatability, etc.
• Basic information has been given for a general
knowledge of the concepts and components of
measurement uncertainty.
• This document is not intended as a reference! Please
refer to the published documents referenced here.
Submission Slide 26 Dr. Michael D. Foegelle, ETS-Lindgren
March 2006 Doc.: IEEE 802.11-06/0333r0

References
• 1. NIST Technical Note 1297-1994, “Guidelines
for Evaluating and Expressing the Uncertainty of
NIST Measurement Results”, Barry N. Taylor and
Chris E. Kuyatt.
• 2. NIS-81, “The Treatment of Uncertainty in EMC
Measurements”, NAMAS
• 3. ISO/IEC Guide 17025, “General requirements
for the competence of testing and calibration
laboratories.”

Submission Slide 27 Dr. Michael D. Foegelle, ETS-Lindgren