"Significantly-Out-Of-Tolerance" (SOOT) Conditions

While you may have never heard this term before, you may encounter it IF you're a supplier to Collins Aerospace OR Pratt & Whitney.

The Collins "AEROSPACE SUPPLIER QUALITY REQUIREMENTS" (ASQR-01, Rev. 15), includes the following section:

7.1.5.1.2.1.1 Monitoring and Measuring Resources - General Significant-Out-Of-Tolerance (SOOT) condition is defined as Measuring and Test Equipment (M&TE) out-of-tolerance condition that either (a) exceeds 25% of the product tolerance (the maximum acceptable level of deviation from a product’s specification, measurements or standards), or (b) when measured error of the M&TE is greater than two times the calibration tolerance (the maximum acceptable deviation between the known standard and the calibrated device) when product tolerance is not known. When M&TE is identified as SOOT, the supplier shall:
▪ Document the SOOT results in calibration records.
▪ Assess impact of the SOOT condition on product inspected with the SOOT M&TE since its last non-SOOT calibration.
▪ Notify P&W within 24 hours if it is determined that impacted parts could have been shipped (see section 10.2.3)
▪ Ensure corrective action measures to eliminate the SOOT condition and minimize product risk. Such measures may be included but are not limited to:
* Adjustment or repair or replacement of M&TE
* Review of future use of similar M&TE
* Review of conditions that may have caused SOOT (e.g. improper usage, storage, or maintenance)
* Over inspection of potentially impacted production hardware

Rule (a):
The Measuring and Test Equipment (M&TE) out-of-tolerance condition exceeds 25% of the product tolerance (the maximum acceptable level of deviation from a product’s specification, measurements or standards).

Suppose you're measuring the length of a part that should be 100mm with an acceptable error (tolerance) of ±2 mm (meaning the measurement can be between 98.0 mm and 102.0 mm).

25% of the product tolerance - 2 mm ÷ 4 = 0.5 mm (or 2 mm x 0.25 = 0.5 mm

Calibration of Measuring Device
The measuring device was calibrated at 100.0 mm and found to read (display) 100.6 mm.

This indicates a bias error of +0.6 mm.

Since 0.6 mm > 0.5 mm, this is more than 25% of the tolerance of the product, so it's a SOOT condition for the measuring device.

Measurements
Suppose the product was measured, and the measurement device indicated it was 98.5 mm.

Since we know that the measurement device is reading with a bias of +0.6 mm, then the measured value “could” actually be 97.9 mm… which would be out of tolerance.

98.5 mm (reading) - 0.6 mm (the bias) = 97.9 mm (actual measured value)

I say “could” because the measuring device itself has a tolerance that is unknown. The unknown calibration tolerance of the measuring device doesn't impact the fact that the bias correction shows the product is out of tolerance.

Rule (b):
The Measuring and Test Equipment (M&TE) error is greater than twice the calibration tolerance when product tolerance isn't known.
Example:
Keeping with the above example, suppose your measuring device has a calibration tolerance of ±0.5 mm (meaning the device could be off by ±0.5 mm).

Twice the calibration tolerance = 2 × 0.5 mm = 1.0 mm

Calibration of Measuring Device
Assume that the measuring device was calibrated at 100.0 mm and found to read 98.9 mm (1.1 mm low).

This means the device underestimates measurements by 1.1 mm (since 100.0 mm - 98.9 mm = 1.1 mm).

Bias: -1.1 mm (the device's readings are 1.1 mm less than the actual measurement).

Measurements

When taking a measurement, the measuring device reads: 101.0 mm

However, the “actual” measured value must include this bias: Actual = Displayed Value + Bias

This means: 101.0 mm + 1.1 mm = 102.1 mm

Conclusion: 102.1 mm > 102 mm, so the part is out of tolerance

This means that whatever reading your measuring device displays has a -1.1 mm bias (indicating the actual measurement value is +1.1 mm greater than indicated).

So if your device displays a measurement of 101.0 mm, it's true reading would be 102.1 mm (101.0 + 1.1 mm)!

Since the bias (1.1 mm) this is more than twice the calibration tolerance (± 1.0 mm) this is a SOOT condition.

AS9100C (Issued 1999) had included the following requirement:

7.6 Control of Monitoring and Measuring Equipment
In addition, the organization shall assess and record the validity of the previous measuring results when the equipment is found not to conform to requirements. The organization shall take appropriate action on the equipment and any product affected.

After I personally issued a MAJOR nonconformity to a major aerospace company for not following this requirement, this entire paragraph was removed from AS9100D (1999-11). Was this coincidental? I doubt it.

The removal of this requirements was a step BACKWARD for AS9100… and Collins not only recognized this, they took it a step further by defining the word “significant”.

The oldest standard I’ve seen use the term “Significantly Out of Tolerance” (SOOT), was MIL-STD 45662A, “CALIBRATION SYSTEMS REQUIREMENTS” (dated 1 August 1988). It appeared in sec. 5.6, “Out-of-tolerance-conditions”, which stated:

If any M&TE or measurement standard is found to be significantly out-of-tolerance during the calibration process, the contractor’s calibration system shall provide for the notification to the respective user and designated contractor quality element of the out-of-tolerance condition with associated measurement data so that appropriate action can be taken.

While “significant” was undefined, MIL-STD 45662A, sec. 5.2 stated:

Unless otherwise specified in the contract requirements, the collective uncertainty of the measurement standards shall not exceed 25 percent of the acceptable tolerance for each characteristic being calibrated.

So it was generally interpreted to mean that anywhere that a 4:1 “Test Accuracy Ration” (TAR) was not maintained was SOOT.

MIL-STD 45662A was WIDELY used from 1980 until it was cancelled on Feb. 27, 1995. And the U.S. DoD then adopted ANSI/NCSL Z540.1:1994.

The SOOT concept was included in ANSI/NCSL Z540.1:1994, “Calibration Laboratories and Measuring and Test Equipment - General Requirements”, to assure the DoD that Z540.1 was an acceptable replacement for MIL-STD 45662A (I have first hand knowledge of this because I was on the committee that developed Z540.1). The SOOT concept was included in ANSI/NCSL Z540.1:1994 (R2002), sec. 13.6, which states:

13.6 The laboratory shall notify customers promptly, in writing, of:
a). any event such as the identification of defective calibration equipment that casts doubt on the validity of results given in any calibration report or certificate, or amendment to a report or certificate. such notification shall quantify the magnitude of error created in the calibration results.
b). any customer's measuring and test equipment found significantly out-of-tolerance during the calibration / verification process. Measurement data shall be reported so that appropriate action can be taken.

The NCSL web site states that ANSI/NCSL Z540.1:1994 (R2002) was withdrawn as an active standard (July 2007) and superseded by ANS/ISO/IEC 17025:2005 for part 1 and ANSI/NCSL Z540.3-2006 for part 2.

The ONLY other standards where I’ve the SOOT concept appear was in ANSI/ASQC Standard M1-1987, “Calibration Systems”, which states:

3.12 Out-of-Tolerance Notification
When calibration ensembles are found to be out of tolerance, customer organizations using instruments calibrated by the calibration ensembles shall be notified, in a timely manner, of such out-of-tolerance conditions if they may have had a significant effect on the customer instruments.
Comments:
There are various methods for implementing the requirement: mutual agreement between the customer and the calibration organization that notification will be initiated only when a specified degree of out of tolerance occurs, notification will be effected when any out-of-tolerance condition is detected, notification will be effected only for certain items (criticality of application), an analysis performed indicating no measurements made were traceable to the out-of-tolerance parameter, etc. Effort should be directed to minimize adverse impact to the customer when a known out-of-tolerance condition was detected in the calibration organization.

It should be noted that the SOOT concept was NOT included in ANSI/ASQC Standard M1-1996 or ANSI/NCSL Z540.3-2006, “Requirements for the Calibration of Measuring and Test Equipment”.

BTW, ANSI/NCSL Z540.3-2006 (R2013) was withdrawn as an active standard (October 2020) and superseded by ISO/IEC 17025:2017.

The SOOT concept was NEVER included in ISO Guide 25, ISO 17025, ISO 10012-1 or in the JCGM 200, “International vocabulary of metrology – Basic and general concepts and associated terms (VIM)”

So the term actually only “survived” until July 2007… and the word “significantly” was never clearly defined in this context.

Ultimately, the SOOT concept was limited to U.S. Metrology Labs.