Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
articles:soot [2025/07/12 16:29] – [Why was this added to their requirements?] rrandallarticles:soot [2025/07/26 21:48] (current) – [What does this mean?] rrandall
Line 1: Line 1:
 +====== "Significantly-Out-Of-Tolerance" (SOOT) Conditions ======
 +
 +While you may have never heard this term before, you may encounter it IF you're a supplier to Collins Aerospace **OR** Pratt & Whitney.
 +
 +The Collins [[https://prd-sc102-cdn.rtx.com/-/media/rtx/s/suppliers/asqr-01/asqr01.pdf?rev=315a3ca505784208afafed3b139144b2|"AEROSPACE SUPPLIER QUALITY REQUIREMENTS" (ASQR-01, Rev. 15)]], includes the following section: \\ 
 +<blockquote>// 7.1.5.1.2.1.1 Monitoring and Measuring Resources - General Significant-Out-Of-Tolerance (SOOT) condition is defined as Measuring and Test Equipment (M&TE) out-of-tolerance condition that either (a) exceeds 25% of the product tolerance (the maximum acceptable level of deviation from a product’s specification, measurements or standards), **or** (b) when measured error of the M&TE is greater than two times the calibration tolerance (the maximum acceptable deviation between the known standard and the calibrated device) when product tolerance is not known. When M&TE is identified as SOOT, the supplier shall: \\ 
 + ▪ Document the SOOT results in calibration records. \\ 
 + ▪ Assess impact of the SOOT condition on product inspected with the SOOT M&TE since its last non-SOOT calibration. \\ 
 + ▪ Notify P&W within 24 hours if it is determined that impacted parts could have been shipped (see section 10.2.3) \\ 
 + ▪ Ensure corrective action measures to eliminate the SOOT condition and minimize product risk. Such measures may be included but are not limited to: \\ 
 +  *    Adjustment or repair or replacement of M&TE \\ 
 +  *    Review of future use of similar M&TE \\ 
 +  *    Review of conditions that may have caused SOOT (e.g. improper usage, storage, or maintenance) \\ 
 +  *    Over inspection of potentially impacted production hardware \\ //  
 +</blockquote> 
 +
 +===== What does this mean? =====
 +
 +**Rule (a):** \\ 
 +The Measuring and Test Equipment (M&TE) out-of-tolerance condition exceeds 25% of the product tolerance (the maximum acceptable level of deviation from a product’s specification, measurements or standards). \\ 
 +
 +<wrap em>This SOOT criteria focuses on the Accuracy Ratio as it relates to the product... without consideration to the accuracy of the measuring device.</wrap>
 +
 +__Example:__ \\ 
 +Suppose you're measuring the length of a part that should be 100mm with an acceptable error (tolerance) of ±2 mm (meaning the measurement can be between 98.0 mm and 102.0 mm). 
 +
 +25% of the product tolerance - 2 mm ÷ 4 = 0.5 mm (or 2 mm x 0.25 = 0.5 mm).
 +
 +__Calibration of Measuring Device__ \\ 
 +The measuring device was calibrated at 100.0 mm and found to read (display) 100.6 mm.
 +
 +This indicates a bias error of +0.6 mm.
 +
 +Since 0.6 mm > 0.5 mm, this is more than 25% of the tolerance of the product, so it's a SOOT condition for the measuring device.
 +
 +__Measurements__ \\
 +Suppose the product was measured, and the measurement device indicated it was 98.5 mm. 
 +
 +Since we know that the measurement device is reading with a bias of +0.6 mm, then the measured value "//could//" actually be 97.9 mm... which would be out of tolerance. 
 +
 +98.5 mm (reading) - 0.6 mm (the bias) = 97.9 mm (actual measured value)
 +
 +I say "//could//" because the measuring device itself has a tolerance that is unknown. The unknown calibration tolerance of the measuring device doesn't impact the fact that the bias correction shows the product is out of tolerance.
 +
 +<WRAP center round info 80%>
 +In other words, a minimum 4:1 Accuracy Ratio was not maintained between the measuring device and the product being measured.
 +</WRAP>
 +
 +----
 +
 +**Rule (b):** \\ 
 +The Measuring and Test Equipment (M&TE) error is greater than twice the calibration tolerance when product tolerance isn't known. \\
 +
 +<wrap em>This SOOT criteria focuses on the Accuracy Ratio as it relates to the measuring device... without consideration to the tolerances associated with the product.</wrap>
 +
 +__Example:__ \\ 
 +Keeping with the above example, suppose your measuring device has a calibration tolerance of ±0.5 mm (meaning the device could be off by ±0.5 mm). This provides a 4:1 Accuracy Ratio with the produce tolerance being measured.\\ 
 +
 +Twice the calibration tolerance = 2 × 0.5 mm = ±1.0 mm \\ 
 +
 +__Calibration of Measuring Device__ \\ 
 +Assume that the measuring device was calibrated at 100.0 mm and found to read 98.9 mm (1.1 mm low).
 +
 +This means the measuring device error was found to be greater than twice the calibration tolerance.
 +
 +Conclusion: \\
 +This somewhat arbitrary approach toward defining a SOOT condition has little to do with determining the IMPACT of the Out-Of-Tolerance (OOT) condition. \\
 +
 +If we know the product tolerance, and determine that it had been a 4:1 Accuracy Ration, then halving the calibration accuracy of the measuring device (which is what doubling the error does), means that a ≤2:1 Accuracy Ratio exists. That would indeed be significant. \\ 
 +
 +<wrap em>HOWEVER</wrap>, what if the measuring device was 10 times more accurate than the product tolerance? \\
 +
 +In that case, an error of twice the calibration tolerance would have simply reduced the Accuracy Ratio from 10:1 to 5:1. Since an Accuracy Ration would have still been greater than 4:1, the impact would have been insignificant. 
 +
 +===== Why was this added to their requirements? =====
 +
 +AS9100C (Issued 1999) had included the following requirement:
 +<blockquote>//7.6 Control of Monitoring and Measuring Equipment \\ 
 +In addition, the organization shall assess and __record__ the validity of the previous measuring results when the equipment is
 +found not to conform to requirements. The organization shall take appropriate action on the equipment and any product
 +affected.//</blockquote>
 +
 +Under this version of AS9100, while performing an audit on behalf of a Certification Body at a major aerospace company involved with the development of AS9100, I had personally issued a MAJOR nonconformity for __not__ retaining these records. I seriously doubt that it was coincidental that the requirement for retaining these records was eliminated from AS9100:2016 (Rev. D).
 +
 +AS90100:2016 (Rev. D) changed this requirement to read: \\ 
 +<blockquote>7.1.5.2 Measurement Traceability \\ 
 +The organization shall determine if the validity of previous measurement results has been adversely affected when
 +measuring equipment is found to be unfit for its intended purpose, and shall take appropriate action as necessary.</blockquote>
 +
 +This removed the requirement to retain a record of the nonconforming (e.g., Out-Of-Tolerance) condition.
 +
 +The removal of this requirements was a step BACKWARD for AS9100... and Collins not only recognized this, they took it a step further by defining the word "significant".
 +===== Origin of SOOT =====
 +
 +The oldest standard I’ve seen use the term “Significantly Out of Tolerance” (SOOT), was MIL-STD 45662A, "CALIBRATION SYSTEMS REQUIREMENTS" (dated 1 August 1988). It appeared in sec. 5.6, “Out-of-tolerance-conditions”, which stated: \\ 
 +<blockquote>//If any M&TE or measurement standard is found to be significantly out-of-tolerance during the calibration process, the contractor’s calibration system shall provide for the notification to the respective user and designated contractor quality element of the out-of-tolerance condition with associated measurement data so that appropriate action can be taken.//</blockquote>
 +
 +While “significant” was undefined, MIL-STD 45662A, sec. 5.2 stated: \\ 
 +<blockquote>//Unless otherwise specified in the contract requirements, the collective uncertainty of the measurement standards shall not exceed 25 percent of the acceptable tolerance for each characteristic being calibrated.//</blockquote>
 +
 +So it was generally interpreted to mean that anywhere that a 4:1 "Test Accuracy Ration" (TAR) was not maintained was SOOT. 
 +
 +<WRAP center round info 80%>
 +The SOOT concept did **NOT** appear in MIL-STD 45662, "CALIBRATION SYSTEMS REQUIREMENTS" (dated 10 June 1980); which superseded MIL-C-45662A dated 9 February 1962. 
 +</WRAP>
 +
 +MIL-STD 45662A was WIDELY used from 1980 until it was cancelled on Feb. 27, 1995. And the U.S. DoD then adopted ANSI/NCSL Z540.1:1994.
 +
 +The SOOT concept was included in ANSI/NCSL Z540.1:1994, "Calibration Laboratories and Measuring and Test Equipment - General Requirements”, to assure the DoD that Z540.1 was an acceptable replacement for MIL-STD 45662A (I have first hand knowledge of this because I was on the committee that developed Z540.1). The SOOT concept was included in ANSI/NCSL Z540.1:1994 (R2002), sec. 13.6, which states: \\ 
 +<blockquote>//13.6 The laboratory shall notify customers promptly, in writing, of: \\ 
 +a). any event such as the identification of defective calibration equipment that casts doubt on the validity of results given in any calibration report or certificate, or amendment to a report or certificate. such notification shall quantify the magnitude of error created in the calibration results. \\ 
 +b). any customer's measuring and test equipment found significantly out-of-tolerance during the calibration / verification process. Measurement data shall be reported so that appropriate action can be taken.//</blockquote>
 +
 +
 +The NCSL web site states that ANSI/NCSL Z540.1:1994 (R2002) was withdrawn as an active standard (July 2007) and superseded by ANS/ISO/IEC 17025:2005 for part 1 and ANSI/NCSL Z540.3-2006 for part 2.
 +
 +The ONLY other standards where I’ve the SOOT concept appear was in ANSI/ASQC Standard M1-1987, “Calibration Systems”, which states: \\ 
 +<blockquote>//3.12 Out-of-Tolerance Notification \\ 
 +When calibration ensembles are found to be out of tolerance, customer organizations using instruments calibrated by the calibration ensembles shall be notified, in a timely manner, of such out-of-tolerance conditions if they may have had a significant effect on the customer instruments. \\ 
 +Comments: \\ 
 +There are various methods for implementing the requirement: mutual agreement between the customer and the calibration organization that notification will be initiated only when a specified degree of out of tolerance occurs, notification will be effected when any out-of-tolerance condition is detected, notification will be effected only for certain items (criticality of application), an analysis performed indicating no measurements made were traceable to the out-of-tolerance parameter, etc. Effort should be directed to minimize adverse impact to the customer when a known out-of-tolerance condition was detected in the calibration organization.//</blockquote>
 +
 +
 +It should be noted that the SOOT concept was **NOT** included in ANSI/ASQC Standard M1-1996 or ANSI/NCSL Z540.3-2006, “Requirements for the Calibration of Measuring and Test Equipment”. 
 +
 +BTW, ANSI/NCSL Z540.3-2006 (R2013) was withdrawn as an active standard (October 2020) and superseded by ISO/IEC 17025:2017.
 +
 +
 +The SOOT concept was NEVER included in ISO Guide 25, ISO 17025, ISO 10012-1 or in the JCGM 200, "International vocabulary of metrology – Basic and general concepts and associated terms (VIM)"
 +
 +So the term actually only “survived" until July 2007… and the word “significantly” was never clearly defined in this context.
 +
 +Ultimately, the SOOT concept was limited to U.S. Metrology Labs.
 +