Quality Metrics Support Improved Quality Across the Value Network

10 years ago, Congress provided FDA the statutory authority to conduct risk-based inspection, with the promulgation of the FDA Safety and Innovation Act (FDASIA). The act also provided the agency the authority to request information in lieu of an inspection setting the stage for the Quality Metrics (QM) Initiative.

According to Susan de Mars, FDA Senior advisor with the Office of Regulatory Operations and Policy at the time, described how FDASIA “… allows us to request records that we can get during an inspection in advance of or in lieu of a physical inspection.”

“This means that we can replace an expensive on-site inspection with this kind of records review. We can also use this authority to collect information and then based on that target where we need to do the on-site inspections” added de Mars.

Supporting Quality Across the Value Network:

Quality Metrics offer a variety of benefits supporting quality improvements across the Value Network:

They provide FDA an objective means to measure, evaluate, and monitor quality across the product and process life cycle to proactively identify and mitigate risks.
They are used throughout the drug and biological product industry to monitor quality control systems and processes and drive continuous improvement efforts in manufacturing.
They are important because failure to update and innovate manufacturing practices and lack of operational reliability can lead to quality problems.

While industry and FDA recognize that the minimum standard for ensuring product safety and effectiveness is compliance with current good manufacturing practice (CGMP) (21 CFR parts 210 and 211 for drug products and the International Conference on Harmonisation guidance for industry entitled “Q7 Good Manufacturing Practice Guidance for Active Pharmaceutical Ingredients”) compliance does not necessarily translate into consistent control over manufacturing performance and quality. This requires a focus on continuous improvement and the implementation of an effective Pharmaceutical Quality System (PQS).

Quality Metrics can be used to support a variety of Quality initiatives including:

Using of Quality Metrics data to support a manufacturer’s ability to develop an effective PQS that ensures CGMP compliance and supply chain robustness.
Gaining insights into manufacturing performance and allow opportunities to identify innovative manufacturing practices.
Providing oversight of outsourced activities and material suppliers as well as monitoring of supply chain activities to minimize disruptions.

FDA will assess information submitted to the QM Reporting Program to:

Determine an establishment’s manufacturing quality and reliability quantitatively and objectively.
Adapt a comprehensive quality surveillance program from this data.
Identify products and processes at risk of future quality assurance issues.

To support FDA’s transition of a risk-based approach, the Agency proposed to develop and implement a QM Reporting Program to support its quality surveillance activities. FDA intends to analyze the quality metrics data submitted by industry to:

Obtain a more quantitative and objective measure of manufacturing quality and reliability at an establishment
integrate the metrics and resulting analysis into FDA’s comprehensive quality surveillance program
apply the results of the analysis to assist in identifying products at risk for quality problems like shortages and recalls.

Proposed Changes to QM Reporting Program

According to the FDA, the Agency is considering modifications to the QM Reporting Program based on stakeholder feedback which would allow different industry sectors to use different quality metrics. Specifically, the FDA would identify critical practice areas for ensuring sustainable product quality and availability. Manufacturers would be permitted to select practice area metric(s) which enable continual improvement opportunities. Further, instead of following strict guidelines for the metrics, the reporting establishments would inform the FDA how their reported metrics were calculated. Overall, this would provide manufacturers with necessary flexibility by reducing standardization of quality metrics and definitions across sectors.

Through the collective feedback gathered from pilot participants, FDA has identified the following four general practice areas as appropriate at this time for the QM Reporting Program:

Manufacturing Process Performance,
PQS Effectiveness,
Laboratory Performance,
Supply Chain Robustness.

Examples of quality metrics associated with each practice include the following:

1. Manufacturing Process Performance

Process Capability/Performance Indices (Cpk/Ppk): A measure that compares the output of a process to the specification limits and can be calculated as a proportion (e.g., total number of attributes with Ppk greater than 1.33 divided by total number of attributes where Ppk is used). It is important to consider standard deviation measurements using a reasonable sample size.
LAR: A measure of the proportion of lots that were accepted in a given time period. Examples of inputs that can be used to calculate LAR include lots completed, lots dispositioned, lots attempted, lots rejected, lots released, lots approved, abandoned lots, and parallel/backup lots.
Right-First-Time Rate: A measure of the proportion of lots manufactured without the occurrence of a non-conformance. Examples of inputs that can be used to calculate a right-first-time rate include number of deviations, lots dispositioned, lots attempted, number of nonconformances, and lots approved in the first pass.
Lot Release Cycle Time: A measure of the amount of time it takes for the lot disposition process. Lot release cycle time can be calculated with an appropriate unit of measurement such as number of hours or days.

2. PQS Effectiveness

CAPA Effectiveness: A measure of the proportion of CAPA plan implemented and deemed effective (i.e., effectiveness verifications closed as effective). Examples of inputs that can be used to calculate CAPA effectiveness include number of CAPAs initiated, CAPAs closed on time, CAPAs closed as “effective,” overdue CAPAs, and CAPAs resulting in retraining.
Repeat Deviation Rate: A measure of the proportion of recurring deviation measures. Examples of inputs that can be used to calculate repeat deviation rate include total number of deviations and number of deviations with the same assignable root cause.
Change Control Effectiveness: A measure of timeliness and effectiveness of implemented changes to GMP facilities, systems, equipment, or processes. Examples of inputs that can be used to calculate this metric include on-time closure of the change, total number of late effectiveness checks, total number of changes initiated, number of changes that are initiated reactively versus proactively, and total number of changes deemed effective.
Overall Equipment Effectiveness: A measure of operating productivity, utilizing planned production time. Overall equipment effectiveness can be calculated using inputs related to availability (e.g., planned production time, operating time), performance (e.g., production capacity), and quality (e.g., production output that does not result in acceptable product).
Unplanned Maintenance: A measure of the proportion of maintenance time that was not planned or scheduled. Examples of inputs that can be used to calculate this metric include total maintenance hours and planned maintenance hours.

3. Laboratory Performance

Adherence to Lead Time: A measure of the proportion of tests in the laboratory that are completed on time according to schedule requirements. Adherence to lead time can be calculated, for example, by tracking initiation and testing turnover time in release and stability tests (i.e., the number of days between the start date and completion date for quality control (QC)); tracking data review and documentation; tracking final result reporting prior to batch disposition; or comparing QC testing completion date against the target date.
Right-First-Time Rate: A measure of the proportion of tests conducted without the occurrence of a deviation. Right-first-time rate as a metric for laboratory performance can be calculated, for example, by tracking the invalid assay rate, the number of assays invalidated due to human errors, or CGMP documentation errors during review.
Invalidated Out-of-Specification Rates (IOOSR): A measure that indicates a laboratory’s ability to accurately perform tests. Examples of inputs that can be used to calculate this metric include total number of tests conducted and total number of out-of-specification results invalidated due to an aberration of the measurement process.
Calibration Timeliness: A measure of a laboratory’s adherence to inspecting, calibrating, and testing equipment for its intended purposes as planned. This metric can be measured by tracking calibration criteria and schedules.

4. Supply Chain Robustness

On-Time In-Full (OTIF): A measure of the extent to which shipments are delivered to their destination containing the correct quantity and according to the schedule specified in the order. This metric can be calculated using inputs such as the number of orders shipped, number of past due orders, or number of orders shipped within tolerance.
Fill Rate: A measure that quantifies orders shipped as a percentage of the total demand for a given period. Examples of inputs that can be used to calculate this metric include total number of orders shipped, the number of orders placed, and the number of orders received.
Disposition On-Time: A measure of the proportion of lots in which the disposition was carried out on time. Examples of inputs that can be used to calculate this metric include the total number of lots dispositioned and the total number of lots dispositioned on time.
Days of Inventory On-Hand: A measure of how a company utilizes the average inventory available. It is the number of days that inventory remains in stock.

Conclusion

Following the authority provided by Congress a decade ago through FDASIA, the FDA has continued to pursue improvements in product quality through inspections and record analysis. This directive has culminated in the FDA’s Quality Metrics (QM) Initiative, industry greater flexibility in reporting quality assurance metrics.
Given that the majority of participants in FDA’s QM pilot programs preferred to report data at an establishment level, the Agency is considering an approach for aggregating and reporting quality metrics data at the establishment level, with the option to segment by manufacturing train, product type, or product level ( e.g., application number or product family).

Once the data are submitted, FDA intends to analyze it with statistical and machine learning methods to provide useful insights for inspection resource allocation. Examples include examination of product trends and clusters; exploratory and time-series analyses for signal identification, thereby monitoring the health of the establishment over time; and utilizing quality metrics data as an input into machine learning models to assist in determining an establishment’s overall PQS effectiveness.

FDA is seeking comment from stakeholders on the agency’s proposed direction for its QM Reporting Program. Comments can be made on the docket at https://www.federalregister.gov/documents/2022/03/09/2022-04972/food-and-drug-administration-quality-metrics-reporting-program-establishment-of-a-public-docket?utm_medium=email&utm_source=govdelivery#open-comment

Axendia will continue to monitor and report on developments in FDA’s Quality Metrics Initiative.

Contact Research@axendia.com to schedule an Analyst Inquiry on this topic.

Supporting Quality Across the Value Network:

Proposed Changes to QM Reporting Program

Conclusion

Related Reading: