====== "Significantly-Out-Of-Tolerance" (SOOT) Conditions ====== While you may have never heard this term before, you may encounter it IF you're a supplier to Collins Aerospace **OR** Pratt & Whitney. The Collins [[https://prd-sc102-cdn.rtx.com/-/media/rtx/s/suppliers/asqr-01/asqr01.pdf?rev=315a3ca505784208afafed3b139144b2|"AEROSPACE SUPPLIER QUALITY REQUIREMENTS" (ASQR-01, Rev. 15)]], includes the following section: \\
// 7.1.5.1.2.1.1 Monitoring and Measuring Resources - General Significant-Out-Of-Tolerance (SOOT) condition is defined as Measuring and Test Equipment (M&TE) out-of-tolerance condition that either (a) exceeds 25% of the product tolerance (the maximum acceptable level of deviation from a product’s specification, measurements or standards), **or** (b) when measured error of the M&TE is greater than two times the calibration tolerance (the maximum acceptable deviation between the known standard and the calibrated device) when product tolerance is not known. When M&TE is identified as SOOT, the supplier shall: \\ ▪ Document the SOOT results in calibration records. \\ ▪ Assess impact of the SOOT condition on product inspected with the SOOT M&TE since its last non-SOOT calibration. \\ ▪ Notify P&W within 24 hours if it is determined that impacted parts could have been shipped (see section 10.2.3) \\ ▪ Ensure corrective action measures to eliminate the SOOT condition and minimize product risk. Such measures may be included but are not limited to: \\ * Adjustment or repair or replacement of M&TE \\ * Review of future use of similar M&TE \\ * Review of conditions that may have caused SOOT (e.g. improper usage, storage, or maintenance) \\ * Over inspection of potentially impacted production hardware \\ //
===== What does this mean? ===== **Rule (a):** \\ The Measuring and Test Equipment (M&TE) out-of-tolerance condition exceeds 25% of the product tolerance (the maximum acceptable level of deviation from a product’s specification, measurements or standards). \\ This SOOT criteria focuses on the Accuracy Ratio as it relates to the product... without consideration to the accuracy of the measuring device. __Example:__ \\ Suppose you're measuring the length of a part that should be 100mm with an acceptable error (tolerance) of ±2 mm (meaning the measurement can be between 98.0 mm and 102.0 mm). 25% of the product tolerance - 2 mm ÷ 4 = 0.5 mm (or 2 mm x 0.25 = 0.5 mm). __Calibration of Measuring Device__ \\ The measuring device was calibrated at 100.0 mm and found to read (display) 100.6 mm. This indicates a bias error of +0.6 mm. Since 0.6 mm > 0.5 mm, this is more than 25% of the tolerance of the product, so it's a SOOT condition for the measuring device. __Measurements__ \\ Suppose the product was measured, and the measurement device indicated it was 98.5 mm. Since we know that the measurement device is reading with a bias of +0.6 mm, then the measured value "//could//" actually be 97.9 mm... which would be out of tolerance. 98.5 mm (reading) - 0.6 mm (the bias) = 97.9 mm (actual measured value) I say "//could//" because the measuring device itself has a tolerance that is unknown. The unknown calibration tolerance of the measuring device doesn't impact the fact that the bias correction shows the product is out of tolerance. In other words, a minimum 4:1 Accuracy Ratio was not maintained between the measuring device and the product being measured. ---- **Rule (b):** \\ The Measuring and Test Equipment (M&TE) error is greater than twice the calibration tolerance when product tolerance isn't known. \\ This SOOT criteria focuses on the Accuracy Ratio as it relates to the measuring device... without consideration to the tolerances associated with the product. __Example:__ \\ Keeping with the above example, suppose your measuring device has a calibration tolerance of ±0.5 mm (meaning the device could be off by ±0.5 mm). This provides a 4:1 Accuracy Ratio with the produce tolerance being measured.\\ Twice the calibration tolerance = 2 × 0.5 mm = ±1.0 mm \\ __Calibration of Measuring Device__ \\ Assume that the measuring device was calibrated at 100.0 mm and found to read 98.9 mm (1.1 mm low). This means the measuring device error was found to be greater than twice the calibration tolerance. Conclusion: \\ This somewhat arbitrary approach toward defining a SOOT condition has little to do with determining the IMPACT of the Out-Of-Tolerance (OOT) condition. \\ If we know the product tolerance, and determine that it had been a 4:1 Accuracy Ration, then halving the calibration accuracy of the measuring device (which is what doubling the error does), means that a ≤2:1 Accuracy Ratio exists. That would indeed be significant. \\ HOWEVER, what if the measuring device was 10 times more accurate than the product tolerance? \\ In that case, an error of twice the calibration tolerance would have simply reduced the Accuracy Ratio from 10:1 to 5:1. Since an Accuracy Ration would have still been greater than 4:1, the impact would have been insignificant. ===== Why was this added to their requirements? ===== AS9100C (Issued 1999) had included the following requirement:
//7.6 Control of Monitoring and Measuring Equipment \\ In addition, the organization shall assess and __record__ the validity of the previous measuring results when the equipment is found not to conform to requirements. The organization shall take appropriate action on the equipment and any product affected.//
Under this version of AS9100, while performing an audit on behalf of a Certification Body at a major aerospace company involved with the development of AS9100, I had personally issued a MAJOR nonconformity for __not__ retaining these records. I seriously doubt that it was coincidental that the requirement for retaining these records was eliminated from AS9100:2016 (Rev. D). AS90100:2016 (Rev. D) changed this requirement to read: \\
7.1.5.2 Measurement Traceability \\ The organization shall determine if the validity of previous measurement results has been adversely affected when measuring equipment is found to be unfit for its intended purpose, and shall take appropriate action as necessary.
This removed the requirement to retain a record of the nonconforming (e.g., Out-Of-Tolerance) condition. The removal of this requirements was a step BACKWARD for AS9100... and Collins not only recognized this, they took it a step further by defining the word "significant". ===== Origin of SOOT ===== The oldest standard I’ve seen use the term “Significantly Out of Tolerance” (SOOT), was MIL-STD 45662A, "CALIBRATION SYSTEMS REQUIREMENTS" (dated 1 August 1988). It appeared in sec. 5.6, “Out-of-tolerance-conditions”, which stated: \\
//If any M&TE or measurement standard is found to be significantly out-of-tolerance during the calibration process, the contractor’s calibration system shall provide for the notification to the respective user and designated contractor quality element of the out-of-tolerance condition with associated measurement data so that appropriate action can be taken.//
While “significant” was undefined, MIL-STD 45662A, sec. 5.2 stated: \\
//Unless otherwise specified in the contract requirements, the collective uncertainty of the measurement standards shall not exceed 25 percent of the acceptable tolerance for each characteristic being calibrated.//
So it was generally interpreted to mean that anywhere that a 4:1 "Test Accuracy Ration" (TAR) was not maintained was SOOT. The SOOT concept did **NOT** appear in MIL-STD 45662, "CALIBRATION SYSTEMS REQUIREMENTS" (dated 10 June 1980); which superseded MIL-C-45662A dated 9 February 1962. MIL-STD 45662A was WIDELY used from 1980 until it was cancelled on Feb. 27, 1995. And the U.S. DoD then adopted ANSI/NCSL Z540.1:1994. The SOOT concept was included in ANSI/NCSL Z540.1:1994, "Calibration Laboratories and Measuring and Test Equipment - General Requirements”, to assure the DoD that Z540.1 was an acceptable replacement for MIL-STD 45662A (I have first hand knowledge of this because I was on the committee that developed Z540.1). The SOOT concept was included in ANSI/NCSL Z540.1:1994 (R2002), sec. 13.6, which states: \\
//13.6 The laboratory shall notify customers promptly, in writing, of: \\ a). any event such as the identification of defective calibration equipment that casts doubt on the validity of results given in any calibration report or certificate, or amendment to a report or certificate. such notification shall quantify the magnitude of error created in the calibration results. \\ b). any customer's measuring and test equipment found significantly out-of-tolerance during the calibration / verification process. Measurement data shall be reported so that appropriate action can be taken.//
The NCSL web site states that ANSI/NCSL Z540.1:1994 (R2002) was withdrawn as an active standard (July 2007) and superseded by ANS/ISO/IEC 17025:2005 for part 1 and ANSI/NCSL Z540.3-2006 for part 2. The ONLY other standards where I’ve the SOOT concept appear was in ANSI/ASQC Standard M1-1987, “Calibration Systems”, which states: \\
//3.12 Out-of-Tolerance Notification \\ When calibration ensembles are found to be out of tolerance, customer organizations using instruments calibrated by the calibration ensembles shall be notified, in a timely manner, of such out-of-tolerance conditions if they may have had a significant effect on the customer instruments. \\ Comments: \\ There are various methods for implementing the requirement: mutual agreement between the customer and the calibration organization that notification will be initiated only when a specified degree of out of tolerance occurs, notification will be effected when any out-of-tolerance condition is detected, notification will be effected only for certain items (criticality of application), an analysis performed indicating no measurements made were traceable to the out-of-tolerance parameter, etc. Effort should be directed to minimize adverse impact to the customer when a known out-of-tolerance condition was detected in the calibration organization.//
It should be noted that the SOOT concept was **NOT** included in ANSI/ASQC Standard M1-1996 or ANSI/NCSL Z540.3-2006, “Requirements for the Calibration of Measuring and Test Equipment”. BTW, ANSI/NCSL Z540.3-2006 (R2013) was withdrawn as an active standard (October 2020) and superseded by ISO/IEC 17025:2017. The SOOT concept was NEVER included in ISO Guide 25, ISO 17025, ISO 10012-1 or in the JCGM 200, "International vocabulary of metrology – Basic and general concepts and associated terms (VIM)" So the term actually only “survived" until July 2007… and the word “significantly” was never clearly defined in this context. Ultimately, the SOOT concept was limited to U.S. Metrology Labs.