Analysis of the effects of operational excellence implementation on inspection outcomes in the pharmaceutical industry: an empirical study

Goal: The goal of this paper is to test the possible connection between pharmaceutical manufacturing plants’ Operational Excellence implementation and regulatory inspection outcomes. 
Methodology: This paper uses logistic regression models on a unique dataset compiled from proprietary operations datasets and published FDA inspection outcomes. 
Results: The findings show that sites with advanced Operational Excellence implementation are more likely to receive favorable inspection outcomes and more so when specifically focusing on Total Quality Management implementation as a subset. A similar trend exists for the Quality Control laboratories and their compliance deficiencies when analyzed separately. 
Limitations of the investigation: The limitations of this research mainly lie in the limited size and composition of the available dataset. European manufacturing sites and quality control labs are overrepresented and the sample size for quality control labs is rather small. 
Practical Implication: This paper can help industrial managers and regulatory officials to better direct their resources to manufacturing sites and QC laboratories that are at a higher risk of quality non-compliance. 
Originality/value: It is a long-standing maxim in literature and practice that manufacturing sites shall build improvement capabilities to reach Operational Excellence which includes superior product quality. Quality Management is particularly important in the pharmaceutical industry and often evolves on its own in the organizations. Pharmaceutical production plants are inspected regarding their quality processes and systems by regulatory authorities. However, there is lacking published evidence of the interplay of Operational Excellence practices and regulatory inspection outcomes.


INTRODUCTION
Pharmaceutical drug shortages and product recalls occur frequently and affect companies, markets and patients continuously. Quality issues account for 62% of the these shortages as documented by the FDA (FDA -CDER, 2019). In recent years, the U.S. market also faced rising numbers of drug recalls (Hall et al., 2016). This development explicitly also applies to Class 1 recalls, which categorizes violative drugs with the most severe health consequences (O'Connor, Yu and Lee, 2016). In addition to the risk to patients, such recalls also have come with notable negative economic effect on the affected companies (Shah, Ball and Netessine, 2017). In conclusion, the pharmaceutical industry, regulators and patients suffer from quality deficiencies.
This trend persists even though manufacturers and regulators are still on an ongoing quest to improve Quality Risk Management and develop the industry towards Operational Excellence (Friedli et al., 2013;Waldron, 2017). Quality certainly is a key part of manufacturing excellence in its traditional sense (Voss et al., 1995;Friedli et al., 2018). However, this understanding is not as clear for its interpretation in the pharmaceutical industry. On the one hand, Operational Excellence (OPEX) or Lean Production initiatives in the companies often evolved out of earlier 6σ programs that directly addressed quality (Friedli et al., 2010;Nenni, Giustiniano and Pirolo, 2014). Yet, on the other hand, Quality and Operational Excellence are often separated organizationally and culturally, which can potentially be attributed to the different disciplines of scientists and industrialists being in charge of the programs. There also is a regulatory requirement of having an independent quality function. (Colledani et al., 2014;Eger et al., 2018;Friedli et al., 2018).
Scholars early on highlighted that excellence can only be achieved over a longer period of building and sustaining improvement capabilities (Hayes and Wheelwright, 1984). Operational Excellence therefore is the organizations' sustained adoption of Lean Production, which specifically requires stable, unceasing dedication (Pettersen, 2009). This includes the employees' absorption of the Lean philosophy and their steady empowerment to becoming change agents in the implementation of lean practices (Womack, Jones and Roos., 1990). Only then can they maintain an unbroken motivation to continuously improve their workplace, processes and products (Zhang, L.;Narkhede, B. E.;Chaple, 2017). This process demands the top management to devote their time to Lean and show their dedication (Jackson, 1996;Zhang, L.;Narkhede, B. E.;Chaple, 2017). From the FDA perspective, the agency encourages companies to foster Operational Excellence since late 2001 (Friedli et al., 2013). Based on the regulators perspective and competition in the industry, pharmaceutical companies need to build Operational Excellence and Quality Management capabilities to improve their performance. However, the integration of a quality focus into overall excellence programs exists only partially.
Since the release of the ICH Q9 guidelines, the pharmaceutical industry invested great efforts into the adoption of Quality Risk Management as a crucial component of Quality Management and Operational Excellence (International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use, 2005;Waldron, 2017). While this discourse is closely related to and often driven by practitioners, it often focusses a very detailed level often discussing individual tools, such as FMEAs (O'Donnell and Greene, 2007;Claycamp, 2012;Fahmy et al., 2012). This research thereby offers limited insight on managerial or plant level. On the other hand, researchers have analyzed regulatory inspection outcomes in various contexts, but not with regard to Operational Excellence implementation (Table 1). Gray et al. (2011, p. 739) define quality risk "as the propensity of a manufacturing establishment to fail to comply with good manufacturing practices" (GMP), which is evaluated in the quality surveillance inspections performed by the U.S. Food and Drug Administration (FDA). The GMP guidelines document "minimum requirements for the methods, facilities, and controls used in manufacturing, processing, and packing of a drug product" (Center for Drug Evaluation and Research, 2018). This motivates an in-detail study of Quality Risk on a manufacturing plant level using FDA inspection outcomes as dependent variable.
Quality Control (QC) Labs are an essential part of and sometimes bottleneck in the pharmaceutical value chain (Köhler, 2019). The test results of drug products, substances and components are the basis for the release of only high-quality drugs into the market. Timeliness and reliability of such tests as well as diligent handling of their results therefore are a critical prerequisite for the compliant supply of pharmaceuticals. This fundamental importance and the problems linked to it are well known to the FDA. QC Labs were the target of a large share of warning letters to industry in recent years (The FDA Group, 2019). At the same time, companies strengthen their efforts to implement Operational Excellence practices in their QC labs (Köhler, 2019).
In conclusion, the industry aims to improve their Operational Excellence implementation and Quality Management practices as encouraged by the regulator. At the same time, the connection between such practices and quality failures is vague. The aim of this paper therefore is to identify relationships between OPEX programs and adverse inspection outcomes. The study thereby distinguishes between OPEX implementation in the overall manufacturing site and the quality control labs specifically. The results help industry practitioners to better direct their inspection readiness programs or internal quality audits. At the same time, it can help regulatory authorities improve their risk assessments of manufacturing facilities they oversee.
To assess the hypotheses, logistic and linear regression analyses were chosen. Inspection outcomes are operationalized in two ways. The publicly available data on inspection final decision is a categorical variable. For QC labs, the number of QC lab-relevant citations in a citation are used as an additional, non-categorical numerical variable. The data on Operational Excellence implementation stems from two large surveys for pharmaceutical manufacturing sites and labs, respectively.
The paper continues with a theoretical perspective on Quality Risk management in the pharmaceutical industry and a review of related literature analyzing inspection outcomes. It then introduces the research design focusing on the empirical basis of the work, posed hypotheses and testing methodology. The results are presented in the following chapter and divided into an overall site and QC lab section. Lastly, conclusions are drawn and implications for further research and industry practitioners are given.

THEORETICAL FRAMEWORK
The introduction already outlined the general development of Operational Excellence and Quality Management in the pharmaceutical industry. The following section provides more detail on the increased focus on quality risk in the industry. After that, an overview of existing empirical research on inspection outcome is given to summarize the current state of knowledge and outline the research gap.

Quality Risk Management in the Pharmaceutical Industry
The desire to eliminate the chance of failures and non-quality in products and processes rather than correcting them has been part of industrial engineering for long. Quality Management techniques have turned to preventive approaches several decades ago (Pfeifer, 2002). Similarly, in the Lean Production or Operational Excellence sphere preventive methods such as Poka Yoke also are an important component (Ohno, 1988).
The combination of the terms 'risk' and 'quality management' however is rather new. The evidence of adverse quality events persisting and increasing even though industries mature in Quality Management and Operational Excellence, could be a motivator for the specific risk perspective (Bates et al., 2006). ISO recognized the need to foster risk-based thinking beyond selective preventive methods in their renewed 9001 standard (International Organization for Standardization, 2016). Since then, the topic received strongly increased attention also in the academic literature (Martins et al., 2019;Rampini et al., 2019).
Contributions on Quality Risk Management also exist outside the ISO 9001 context. While there are a few examples from diverse backgrounds, such as higher education, civil engineering and agriculture (Raban and Turner, 2006;Borodin et al., 2014;Qing et al., 2014), the vast majority stems from the domain of pharmaceuticals. Since the release of the guidelines for Quality Risk Management by an international consortium of regulators and industry (International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use, 2005), it became very popular in the industry (Waldron, 2017).
The statute suggests the application of a quality risk perspective to all stages of a drug's lifecycle and all steps of the value chain, as well as inspections. Applying the second core principle to the field of inspections, ICH states that "Risk ranking and filtering can be used to prioritize manufacturing sites for inspection/audit by regulators or industry" (International Conference on Harmonisation of Technical Requirements for Registration of Pharmaceuticals for Human Use, 2005, p.14). Building on that, one decision framework suggests determining inspection frequency based on product complexity, criticality and past regulatory compliance. However, that model still relies purely on quantitative assessments (Pharmaceutical Inspection Convention 2012).

Quality Risk studies using regulatory events
Starting in 2011, several articles addressing FDA inspection outcomes and product recalls were published using different sets of publicly available data that could be connected to the inspection outcomes. In most publications inspection outcomes are used as dependent variable (DV) with several examples shown in Table 1. However, there is also a related case that assigns it to be the independent variable (IV, see Ball et al., 2017).
The findings include that offshore manufacturing locations and language differences between the company headquarter and the plant increase quality risk. Other risk factors include lower education levels and absence of other manufacturing facilities in the vicinity. Confirmed in two studies, longer time spans since last inspection also increase the risk of adverse inspection outcomes, which is in line with the FDA's own approach (Gray et al., 2011;Anand et al., 2012;Gray and Massimino, 2014).
Hints towards a deterioration of quality compliance can also be shown when analyzing the relationship of inspection outcomes and ISO 9000 certifications. Companies that recently were certified show better inspection outcomes with that effect then decaying over time. The positive effect is particularly strong for companies that pursued certification early, possibly demonstrating a proactive approach to improving their Quality Management system (Gray et al., 2015a).
Another previously studied phenomenon is the relationship between brand manufacturers and contract manufacturers (CM). While CM typically operate at a higher risk, that risk particularly increases if the CM operates with high process and/or product heterogeneity. That risk can be reduced by closer collaboration between the buying and supplying company, as well as closer monitoring by the regulator (Handley and Gray, 2015;Gray et al., 2016).
Last to be spotlighted is an unpublished study by Seiss (2018) that is the product of an FDA research contract. In that study, the author applies predictive models to structural data of manufacturing plants, and past regulatory compliance to predict subsequent inspection outcomes. The data can produce models with some predictive power on whether or not actions were indicated. The predictive performance increases slightly when adding additional financial information to the dataset.
The literature was compiled through a Scopus and Google Scholar search for literature that connects Operational Excellence (also Lean, Production Systems, Quality Management) and regulatory inspection outcomes (also quality compliance, quality risk, process compliance. Since the implementation of Operational Excellence in the pharmaceutical industry is relatively new, the search corridor was limited to the past 15 years. Individual sources then were selected based on whether they use inspection outcomes in the study. The selection of relevant studies of FDA inspection outcomes is displayed in Table 1.

Article Results
Gray et al. (2011) Used individually in regression models, regional high school completion quota, industry agglomeration, proximity to headquarter, and mainland location reduce quality risk Anand et al. (2012) Good states of process entropy deteriorate over time; mergers negatively affect entropy whereas acquisitions do positively Gray and Massimino (2014) Time since last inspection and language difference clearly harm process compliance; no clear picture for cultural differences, but similar power distance at HQ and site has positive impact Gray et al. (2015a) Colocation improves quality conformance regardless of access to information/communication technology or time; colocation especially important for large companies and complex products Gray et al. (2015b) ISO 9000 certification has a positive effect on process compliance if obtained early; the effect decays over time Gray et al. (2016) Contract Manufacturers (CM) operate at higher risk, but more experience reduces risk; higher inspection intensity reduces risk particularly for CM Altamuro et al. (2017) Weak compliance culture harms operational and financial non-compliance; contemporary financial and operational non-compliance correlate Ball et al. (2017) Inspection outcomes does predict future recall severity; site-specific experience of the investigator does harm that relationship Seiss (2018) Quality compliance history does somewhat predict future inspection outcomes; financial data slightly improves the predictions In conclusion, quality risk in pharmaceutical production so far has been studied on two levels. There is a detailed perspective on methods and tools for handling and reducing quality risks in the process. Secondly, there are econometric studies that evaluate various context factors in relation to quality risk at the plant level. However, there is a lack of literature addressing operational characteristics or OPEX implementation in the context of plant inspection outcomes.

RESEARCH DESIGN
This chapter begins with a description of the available data for the study and the pairing of different data sources. Operational Excellence implementation databases and FDA inspection outcome data then are introduced in more detail. Thirdly, the hypotheses are posed in conjunction with the theoretical background and the available data. Lastly, the operationalization of data and statistical test types are introduced.

Empirical set-up
The proposed analyses build on two benchmarking studies that consists of more than 100 survey items related to OPEX implementation ("enablers") and published FDA inspection outcomes. The two benchmarking surveys differ in scope with one targeting the overall manufacturing site and the second one being designated to quality operations, particularly the quality control (QC) labs. The data for both surveys is gathered on a standardized fivepoint Likert scale and answered by a group of operations and quality representatives of the respective establishments. The participants are tasked to give an answer that is representative to either the previous year or twelve months rolling period.
The comparison between the surveys and FDA databases was done using company name, site address and year as key criteria for a match. The data reported into the benchmarking surveys typically refers to a site's operational performance related to the full calendar year immediately prior to when the benchmark is undertaken. Therefore, the criterion "year" seeks to compare the benchmarking survey with the last available FDA inspection conducted before the benchmark for that given organization. Inspection outcomes for more than three years prior to the benchmark were not considered. The cutoff is a compromise between timeliness and data availability. The authors acknowledge that this bears certain limitations, because it includes data pairs with an already considerable time gap of three years. Also, the included inspection could be followed by changes in the site, particularly based on CAPAs (Corrective and Preventive Actions) after adverse inspection outcomes. However, CAPAs following an inspection typically specifically target compliance and not necessarily have an impact on Operational Excellence programs or enabler implementation. Furthermore, sites might reassign resources from OPEX and continuous improvement to CAPA and remediation activities. Therefore, the analysis is approach is reasonable with the limitations being kept in mind.  Regarding the QC lab survey, the questions are sorted into three clusters: Maintenance & Quality System (MaintQualSys), Planning & Steering (PlanSteer), and the Management System (MgmtSys). The datasets of 95 quality control labs gathered for calendar years 2015 to 2019 are included in this analysis. They follow a similar geographical distribution as the manufacturing plants described above, but largely represent different establishments with a small overlap. 57 datasets were matched with FDA inspection outcomes and all of these showed sufficient data to be included in the analysis.

Operational Excellence implementation databases
The correlation chart for QC specific enablers is displayed as Figure 2. The enabler groups in both surveys are notably correlated among each other. While this hinders a causal inference between a potential change in implementation level and resulting variance in 'goodness' of inspection result, it allows predictive statements on the current implementation state.

FDA inspection outcome data
Within the inspection outcomes, NAI means that the inspection concluded favorably with 'No Actions Indicated', meaning that no violations were documented. Inspections concluded 'Voluntary Actions Indicated' -VAI are intermediate outcomes with some violations found. Highly adverse inspections result in with the agency assigning 'Official Actions Indicated' OAI. The resulting data from the matching shows fewer OAI final decisions than the overall population ( Figure 3), but the distribution frequency is consistent with the overall population of all FDA inspection outcomes when evaluated per Pearson's Chi-squared (χ 2 = 2.73) test. In literature, several operationalizations of the inspection outcomes have been used over time. The most prominent version developed by Gray et al. (2011) is a scale that uses the final decision following the inspection, as well as the issuance of the FDA form 483. This form is the formal way of communicating deficiencies in the plant. Through a Delphi process with an expert panel, these inspection outcomes were transformed onto a scale from 0 to 3.5. However, only three of these outcomes occur frequently. Typically, without a form 483 there are no actions indicated (NAI) and otherwise either voluntary (VAI) or official actions (OAI) are ruled (Gray et al., 2011). Operationalizations based directly on the final decision (NAI / VAI / OAI) are also popular. While OAI certainly is a highly adverse inspection outcome and rare, VAI is quite common and less critical. To overcome the challenge of small samples skewing our analyses, the dependent variable is simplified to the binary categories: 1 = NAI; 0 = {OAI, VAI}. This means that VAI and OAI are combined even though there are distinct. The authors are aware of this limitation of the analysis approach and it shall be pronounced here. Nevertheless, OAI and VAI both are notably more different from the ideal inspection outcome NAI. Also this approach was chosen in alignment to a recent study describing predictive models for risk based inspection planning (Seiss, 2018).
In addition to the final decision, the FDA in some cases also publishes the specific issues that were found in inspections as 'citations'. While the data is much more limited, these citations allow a closer look at the actual deficiencies regarding quality compliance at the particular sites. Of all the matched inspection outcomes, the citations were available for only 16 of them. Each inspection dataset is equipped one or several citations so 57 citations were available in total. These were examined for whether or not individual citations are directed at deficiencies in QC labs. The examination was done in independent reviews of two researchers and then aggregated. Weaker partial and direct relations were distinguished. In conclusion, only the minority of citations connect to QC labs as shown in Figure 4.

Lab relevance of inspection outcomes
Lab unrelated Partially lab-related Directly lab-related The weighted sum of citations related to the lab per inspection is used as an alternative target variable for analyses with the QC data. To obtain that metric, directly lab-related citations were weighted at 1, partially lab-related citations at 0.5, and then counted per inspection. The resulting distribution is shown in Figure 5.

Hypotheses
Since the manufacturing sites build capabilities by advancing in lean implementation, this set of cumulative performance advantages should also enable them to achieve better results in quality compliance. The first hypothesis states that plants which are further advanced in their OPEX implementation and therefore reached a high maturity in their improvement journey are more likely to receive favorable inspection outcomes (Hypothesis 1a). Similar results are expected for establishments with QC labs that are advanced in their OPEX implementation (Hypothesis 1b). The Quality System can evolve unrelated to the overall Operational Excellence, which is particularly true for the pharmaceutical industry (cf. Introduction). Accordingly, a stronger relationship for Total Quality Management implementation, and the Maintenance & Quality System in the labs, is expected respectively (Hypotheses 2a, 2b). Lastly, the third hypothesis proposes that there is a negative link between the labs' implementation of 'Maintenance and Quality Systems' practices and the number of lab-related citations issued in an inspection (Hypothesis 3). Therefore, more advanced quality practices in QC labs should result in fewer GMP violations documented by the regulator.

Methodology
In order to test the connection of the enablers on the inspection outcome, logistic regression models were chosen. For the site and QC lab analyses, each first model uses the overall OPEX enabler implementation as only independent variable. The second hypothesis is tested using a model with the practice categories as IVs. These IVs are chosen in alignment with the five groups of enablers in the study that are: TPM, TQM, JIT, EMS, and BE for the overall manufacturing site analyses. Analogously for the QC labs, there are three categories. Any significant coefficients are tested again in an individual model without the other coefficients. Results are evaluated by the coefficient strength and significance of the independent variables and the fit of the overall model through its achieved reduction in deviance.
For the QC labs there is a second step of analyses using the alternative target variable ( Figure 5). The weighted number of inspection citations for the QC labs is a near normal distributed variable (Shapiro-Wilk normality test p-value = 0.514). It is analyzed in context to enabler implementation through a linear regression. Due to the low number of observations, only a model with a single input at a time is feasible. Therefore, any predictor is analyzed in a separate model. For the sake of completeness, the same approach is taken for all control variables. Figure 6 illustrates all analyses steps with their scope, the boxes showing the dependent variables and the different independent variables tested. Several models with nonsignificant results are not included as a table, all corresponding tables are referenced next to the analyses.

RESULTS
This section contains the model summary tables and a brief description of their contents. It is structured to start with the analyses of the overall manufacturing sites and the QC lab specific results follow as second part. To assess sensitivity, all logistic regression models were tested with and without control variables. Throughout all steps, only minimal differences exist regarding the coefficients of interest. To simplify the document, only the models with control variables are shown.

Overall manufacturing plants
The output of the first regression model is shown in Table 2. It is found that overall OPEX implementation is positively associated with favorable inspection result. That coefficient shows up to be significant at a 5% level. The location in a low-cost labor country appears to have a strong negative effect on compliance evaluation, but is not significant at a 5% level. The model fit is evaluated using the change in deviance between the Null-model and the hypothesis model using the overall enabler implementation. Using the Chi-squaredcharacteristic, the deviance change (76.028-72.167) at one reduced degree of freedom is significant at a 5% level (p = 0.0494). The model thereby arguably fits sufficiently well and supports the first hypothesis. The model output for the second hypothesis is shown in Table 3. It is found that TQM implementation is linked strongly positive to favorable inspection outcome with a significance level of 5%. There is a negative link between the 'Basic Elements'-enablers, low-cost location and inspection outcomes that are close to a significant level. The effect for BE could be an effect of the collinearity, but remains undetermined at the time. Offshore or low cost locations have already been found to be of higher risk in previous research (Gray et al., 2011). Other Operational Excellence practices and control variables do not show any notable relationship with inspection outcomes. The model fit is again assumed good with the Chi-squaredcharacteristic of the deviance change resulting in a p-value <1% (p = 0.0068). The estimate of the coefficient for TQM implementation is about 50% higher than for overall Operational Excellence implementation above.
Since the model is burdened with a relatively high number of coefficients compared to the available sample size, the non-significant coefficients are removed for an adapted model. Tested individually, the effect of Total Quality Management implementation remains the same, but shows up on an even higher level of significance with p < 1%. The control variables do not show as significant coefficients. The overall model fit also remains at the same high level. The analyses thereby support hypothesis 2a and the positive effect particularly of TQM implementation on inspection outcomes.

Quality control labs
The output of the two logistic regression models for QC labs is shown in Table 4. It is found that overall OPEX enabler implementation, as well as the three enabler categories do not seem to have an impact on observed inspection outcomes. Hypotheses 1b and 2b therefore must be rejected. On the other hand, there is an observation of establishments with larger QC labs being met with adverse inspection outcomes slightly more often. The control variable is a significant at the 5% level coefficient in both models. The model fit is significant at the 5% level for the first model (p = 0.037), bit slightly worse for the second model (p = 0.076). The lack of a statistical link between the Operational Excellence implementation in QC labs specifically and overall site inspection outcomes is understandable. The inspections have a much larger scope than only the QC labs themselves. Hypothesis 3 therefore aims to analyze the connection between the 'Maintenance & Quality System' enablers in the lab and the inspection citations directed at it. The corresponding analysis results are displayed in Table 5. The analysis shows a notably significant fit of the overall model and negative impact of "Maintenance and Quality System" enablers on the weighted lab-related citations in FDA inspections. With the significance level of the coefficient and the overall model, it is reasonable to assume a link between the maturity of the labs and their evaluation by the FDA. The Rsquared values are reasonably good in the context of complex socio-technical systems that strongly depend on human behavior (Frost, 2019). Nevertheless, a relatively large part of the variation in the compliance evaluation of QC Labs is not determined by the tested variable, but other factors. Finally, the relatively low number of underlying observations is highlighted here as a limitation to the robustness of that result. In conclusion, the analysis provides limited support for hypothesis 3.

CONCLUSION
Considering to the results obtained, sites that have a greater level of implementation of their overall Operational Excellence plans and programs have better inspection outcomes. This relationship is stronger and highly significant for enablers specifically associated with the quality systems as measured in the enabler category Total Quality Management TQM. These results shall serve as motivation to industry and regulators to take OPEX and TQM maturity into account in inspection planning and preparation.
No link between Operational Excellence implementation in quality control labs and site inspection outcomes could be established. However, there tends to be a link between labspecific enabler implementation and the deficiencies regulators find in them. These findings are based on a relatively low number of observations and should be tested for their robustness once more data is available. The FDA has dedicated a lot of attention to QC labs in their recent activities (The FDA Group, 2019). However, this finding shall motivate regulators to keep the overall scope of manufacturing sites in mind and to not narrow their focus. Industrial managers must be aware of the importance an Operational Excellence implementation has for the quality compliance evaluation, but refrain from silo optimization.
From an academic perspective, this offers an additional perspective into the firm and its relation to regulators. Future research could provide further detail to such production principles or investigate possible links between operational performance and inspection outcomes. Furthermore, additionally focusing not only enablers or capabilities, but also on operational performance in the context of regulatory inspection outcomes of manufacturing sites could show to be promising to identify leading indicators for quality risks.