====== Understanding Quality KPIs & Objectives ====== {{keywords>Quality Objectives}} Before discussing "KPIs and Objectives" (Goals/Targets), we must first recognize and understand the hierarchical relationship between the Quality Policy, KPIs, and Quality Objectives. - A “Quality Policy” establishes the intentions and direction of an organization (as formally expressed by its top management) related to quality. - KPIs are “measures associated with goals or targets showing how well an organization is achieving its objectives”. - A “Quality Objective” is a "result (goal/target) to be achieved" related to quality. {{ :articles:pyramid_policy-kpis-objectives.png?nolink |}} “Quality Objectives” are the specific “Goals” / “Targets” established within a KPI. For example, if “% of Rework” were a KPI, the objective might be “≤0.2% of items produced”. Most companies associate their "Quality Objectives" with "KPIs" (Key Performance Indicators) because managers are more familiar and comfortable with that term. However, managers tend to interpret KPIs as broadly encompassing many different aspects of the business (e.g., financial performance, health, safety, environmental). In reality, a more "specific" term would be "__Quality Performance Indicators__". While the use of this term would "remind" management that these performance indicators must remain within the scope of the "Quality Management System", this would lead to confusion when an external auditor asks to see your KPIs. ===== The Quality Policy ===== The second requirement in ISO 9001 / AS91xx, sec. 5.2.1 is to "//establish, implement, and maintain a quality policy that provides a __framework__ for setting quality objectives//". \\
5.2.1 Establishing the Quality Policy \\ Top management shall establish, implement, and maintain a quality policy that: \\ a. is appropriate to the purpose and context of the organization and supports its strategic direction; \\ b. provides a __framework__ for setting quality objectives; \\ c. includes a commitment to satisfy applicable requirements; \\ d. includes a commitment to continual improvement of the quality management system.
This firmly establishes the "Quality Policy" at the top of the hierarchy pyramid. ISO doesn't care that having a "//Quality Policy//" is directly contrary to [[https://deming.org/eliminate-slogans-exhortations-and-targets/|point 10]] ("//Eliminate slogans, exhortations...//") of [[https://deming.org/explore/fourteen-points/|Demings 14 Points]]. The requirement for a "Quality Policy" is driven by Philip Crosby's failed "Zero Defects" motivational management philosophy... and has absolutely no influence whatsoever on the level of quality that a company produces. The "Quality Policy" is effectively a non-value-added form of "virtue signaling". However, this is something that many 3rd Party Auditors will ask both management and employees about. Since the "Quality Policy" will likely never be seriously "critiqued" after it is established, the goal should be to simply satisfy the ISO 9001 / AS91xx requirement by creating a short, simple "Quality Policy". For example: \\ "//__(Insert Company Name)__ is committed to satisfying all applicable requirements; and continual improvement of the quality management system.//” ===== KPIs (Key Performance Indicators) ===== While neither ISO 9001 nor AS91xx specifically mention KPIs (Key Performance Indicators), these must be innately identified in order to establish objectives. Clearly, you must know what you're going to measure before you can identify a goal/target. However, AS9101 does define a KPI.
AS9101 (Rev. G) - 3.2 Key Performance Indicator (KPI) \\ Measures associated with __goals or targets__ showing how well an organization is achieving its __objectives__ or critical success factors. KPIs are used to objectively define a __quantifiable and measurable__ indication of performance.
ISO 9001 & AS9100, sec. 6.2 are identical in stating:
6.2.1 The organization shall establish quality objectives at relevant functions, levels, and __processes__ needed for the quality management system.
While many ISO 9001 auditors are flexible or lax in how they interpret this requirement for a quality objective to be established for EACH process, AS9101 requires AS91xx Auditors to complete a “Process Effectiveness Assessment Report” (PEAR) during each assessment.
AS9101 (Rev. G) - 5.3.7 Process Results \\ 5.3.7.1 The audit team shall record measures, targets, and values of KPIs related to each audited operational process (see 9100-series standards clause 8) on the PEAR (see Form 3, Section 2), taking into account the confidentiality of information (see ISO/IEC 17021-1 clause 8.4 requirements). \\ NOTE: Upon mutual agreement between the organization and the CB, other processes can be recorded on a PEAR.
A PEAR form is required to be completed for each “operational process” (e.g., Contracts, Programs, Engineering, Procurement, Production). While the PEAR form includes fields for the Auditor to record data relating to procedures reviewed, records examined, people interviewed, etc., it also has a specific section for the auditor to record the KPIs for the processes, the “Objectives” (e.g., “Goal/Target”) associated with each KPI, and the actual data (i.e., value) for the period audited (usually the previous 12 months). {{ :articles:as9101g_form_3_sec._2.png.png?nolink&800 |}} The “Comments” field is for the auditor to either note that the “objective” was met OR what actions are being taken to meet it. While there is no requirement for the "actions" to be recorded (documented), both ISO 9001 / AS91xx auditors //should// ask how 6.2.2 a thru e are being addressed.
6.2.2 When planning how to achieve its quality objectives, the organization shall determine: \\ a. what will be done; \\ b. what resources will be required; \\ c. who will be responsible; \\ d. when it will be completed; \\ e. how the results will be evaluated.
These questions are straightforward... and if management cannot answer each of them, then a nonconformity is justified. In fact, AS91xx specifically requires Auditors to issue an NCR when “Quality Objectives” are not being met, and no action is being taken.
AS9101 (Rev. G) - 5.3.7 Process Results (Cont.) \\ 5.3.7.2 \\ The audit team shall issue an NCR against the relevant 9100-series standard clause, when the process is not delivering the planned results __and appropriate action is not being taken__. \\ NOTE 1: The NCR may be issued against 9100-series standards clause 4.4.1.c and/or 4.4.1.g, if the nonconformity is related to the effective operation and control of the process. \\ NOTE 2: Nonconformities identified against 9100-series standards clauses 4.4.1.c and/or 4.4.1.g, resulting from multiple PEARs, may be combined into a single NCR.
The “action” is not specifically required to be a “corrective action”. An acceptable “action” could be an “improvement” activity. ===== Quality Objectives (Goals/Targets) ===== The first requirement in sec. 6.2.1a is that "The __quality objectives__ shall be consistent with the __quality policy__". \\
//6.2 Quality Objectives and Planning to Achieve Them \\ 6.2.1 The organization shall establish __quality objectives__ at relevant __functions, levels, and processes__ needed for the quality management system. \\ The __quality objectives__ shall: \\ a. be __consistent with the quality policy__; \\ b. be __measurable__; \\ c. take into account __applicable requirements__; \\ d. be __relevant to conformity of products and services__ and to enhancement of customer satisfaction; \\ e. be monitored; \\ f. be communicated; \\ g. be updated, as appropriate. The organization shall maintain documented information on the quality objectives.//
ISO 9000:2015 defines a "Quality Objective" as:
3.7.2 Quality Objective \\ objective (result to be achieved) related to quality.
A “Quality Objective” may be expressed as a “Goal” or “Target” (as these are considered synonyms). While management may establish many important goals/targets, ISO 9001 & AS91xx are ONLY concerned with “quality” objectives. All other objectives should be considered (by the auditors) to be “out of scope”. ISO also doesn't care that establishing "//Quality Objectives//" is directly contrary to Point 11b. "//Eliminate management by objective...//" of [[https://deming.org/explore/fourteen-points/|Demings 14 Points]]. Therefore, a discussion of this would be a moot point for any company seeking or maintaining ISO 9001/AS 91xx registration. To summarize, the organization (business) is required to identify **__measurable__** quality objectives in each **__process__** that are __**relevant to the conformity of products and services**__, and relate to the fulfillment of __**applicable requirements**__. Establish “realistic” objectives. Once a KPI is identified, it should be measured and analyzed to assess the current process performance before establishing an objective to strive toward. These analyses often reveal inherent process limitations (e.g., capability, capacity) that must be escalated to management in order to be overcome. In addition to the below "Common KPIs & Quality Objectives", the [[https://iaqg.org/|IAQG (International Aerospace Quality Group)]] has published the {{ :articles:scmh-7.11.2-key-performance-indicators-kpi-metrics-and-definitions-rev-c-dated-19jun2020.pdf |"SCMH-7.11.2-Key Performance Indicators KPI Metrics and Definitions" (Rev. C, Dated-19JUN2020)}}, as a part of their [[https://iaqg.org/tools/scmh/|SCMH (Supply Chain Management Handbook)]]; which is freely available upon obtaining a free user name and login. \\ \\ While I personally disagree with several of their suggested KPIs (as some could actually incentivize poor quality - e.g., establishing "//Time to Root Cause//" & "//Corrective Action Timeliness//" as KPIs //could// prioritize timeliness over effectiveness, ignoring the existence of "//common cause//" variations promotes futility (inefficiency) through attempting to treat all nonconformities as being the result of "//assignable cause//" process variations, using the term "//internal customer//" dilutes the importance of the //actual// customer invoking requirements and purchasing the products/services), the IAQG list does provide some potentially good ideas - and may inspire other, better ideas. ===== Common KPIs and Quality Objectives ===== The below examples have been accepted by various AS91xx auditors. The “Suggested Goals / Targets” are only provided as a starting point. As data is gathered, these goals/targets should be adjusted based upon (1) the observed capabilities of each process (Cpk) and (2) the value each metric provides. A key consideration in analyzing these KPIs should be “data collection ability” and “data integrity”. While I would ordinarily caution "//Beware the sirens call to mediocrity//", in a strange twist, the [[articles:ocap|OCAP]] (required for all AS91xx companies) actually promotes mediocrity by linking the company's performance to a "risk factor" - used to determine whether the company is a High, Medium, or Low risk! This discourages the use of "//stretch goals//" because the [[articles:ocap|OCAP]] will "punish" these companies with a 'High" risk rating (which is used to adjust the audit time by ±10%). While your "true" goal will often be 100% compliance... these goals are established for "certification" purposes. If you don't know the difference, read: [[articles:qms_vs_iso9001_cert_mgmnt_system|"How a QMS is different from a "Certified Management System"]] To put the suggested goals/targets in perspective: 99.80% is approx. 4.38 Sigma Short Term (ZST) – or 2,000 DPMO (Defects Per Million Opportunities). That is the average % products delivered “defect-free” from a “typical” machining “Job Shop” (producing high variation, low volume lots at inconsistent rates). 98.0% On-Time Delivery (OTD) is approx. 3.55 Sigma Short Term (ZST) – or 20,000 DPMO. This % is typical of many manufacturing facilities that rely upon raw material suppliers and freight carriers who function outside of their control. ^ Process ^ KPI ^ Suggested \\ Goal / Target ^ | Contracts | Proposals submitted prior to deadline (typically established by the customer) | 98% | | ::: | Contract / Customer Purchase Order revisions requested after acceptance (due to review errors) | ≤0.25% of opportunities | | Project | Project Milestones achieved on-time | 98% | |::: | Project Performance | SAT or better (Defined in [[https://www.acquisition.gov/far/subpart-42.15|FAR Subpart 42.15 - Contractor Performance Information]]) | |::: | Project Deliverables (e.g., [[https://www.smalltofeds.com/2007/07/contract-data-requirements-list-cdrl.html|CDRLs]]) delivered on-time | 98% | | Engineering | Engineering Milestones achieved on-time | 98% | |::: | % of ECOs (Engineering Change Order) due to "design” errors (e.g., poor manufacturability) | ≤2% of ECOs submitted | |::: | % of ECOs (Engineering Change Order) submitted citing "drawing” errors | ≤2% of ECOs submitted | |::: | First Article Inspection (FAI) errors (a reflection of design for manufacturability) | ≤0.25% of opportunities | |::: | Software – Defects (bugs) released (usually ranked) | Varies by Rank | | Procurement | Purchase Order Accuracy | ≥99.8% | |::: | % of nonconforming products from suppliers (e.g., Dispositioned as Return To Vendor (RTV), scrap) | ≤0.25% of line items | |::: | % of errors from suppliers (e.g., incorrect or missing documentation, shipped to the wrong address) | ≤0.25% of line items | |::: | **% On-Time Delivery from Suppliers** see Note 1 | ≥98% | | Production / Manufacturing | Production / Manufacturing Schedule met | ≥99% | |::: | % First Pass Yield (FPY) | ≥99.8% | |::: | Rolled Throughput Yield (RTY) | ≥98% | |::: | % of Rework | ≤0.2% | |::: | % of Repair | ≤0.2% | |::: | % of Scrap | ≤0.25% | |::: | % of Customer Returns (Warranty) | ≤0.2% of opportunities | |::: | % of Customer Returns (non-warranty) | ≤0.2% of opportunities | |::: | % Product Recalls (Notifications sent to customers) | ≤0.1% of opportunities | |::: | **% On-Time Delivery** see Note 2 | ≥98% | Note 1: \\ AS 9100, sec. 8.4.1.1 \\ //The organization shall: \\ c. periodically __review external provider performance__ including __process, product and service conformity, and on-time delivery performance__;// [The above would include applicable SDRL (Subcontract Data Requirements List) items] Note 2: \\ AS 9100, sec. 5.1.2 Customer Focus \\ //Top management shall demonstrate leadership and commitment with respect to customer focus by ensuring that: \\ d. __product and service conformity and on-time delivery performance__ are measured and appropriate action is taken if planned results are not, or will not be, achieved.// [These metrics are also required to be monitored and reviewed in AS 9100, sections: 9.1.2 & 9.3.2] The IAQG "[[https://scmh.iaqg.org/|Supply Chain Management Handbook]]", Section 7.11.2 "Key Performance Indicators-KPI Metrics and Definitions" consists of a comprehensive list of possible KPIs that a company can choose to use. Access to the "[[https://scmh.iaqg.org/|Supply Chain Management Handbook]]" requires a User Name and Password Login - Free. ===== Management Reviews ===== ISO 9001 & AS9101 also requires:
9.3.2 Management review inputs \\ The management review shall be planned and carried out taking into consideration: \\ c) information on the performance and effectiveness of the quality management system, including __trends__ in: \\ 2) __the extent to which quality objectives__ have been met;
The vast majority of auditors will expect you to have supporting objective evidence of quality objectives having been reviewed during management reviews (ISO 9001 / AS 91xx, sec. 9.3.3). And as per ISO 9001 / AS91xx, sec. 9.3.2c, it must reflect "__trends__" (e.g., bar charts, line charts, Cpk control charts). Normally, the best approach is to have this data displayed in a Cpk chart with control limits serving as the "objectives" (i.e., the objective is to be within established control limits). A mistake many companies make is to associate "improvement" goals with "quality objectives". Improvement goals established without understanding the associated KPI Cpk are often unrealistic. {{page>wiki:pathforward}}