Why Internal QMS Audits Fail
– WHITE PAPER –
Why Internal QMS Audits Fail
Safis Solutions, LLC
Medical device companies are required to perform internal audits of their quality management system (QMS). The quality management system elements requiring verification are dependent on the standards and regulations that apply. Typically, multiple standards and regulations apply simultaneously.
Companies spend considerable time, effort and money performing audits of their QMS, yet they are often surprised and disappointed by an independent third party auditor’s findings of noncompliance. Although there can be many reasons for this, it is often the result of one, two or all three of the following scenarios.
1. Small Audit Teams or Dedicated Auditors: when small audit teams exist or auditors are dedicated to individual sites or individual QMS elements, audit bias can develop. Repeat auditing of the same QMS elements or the same sites by the same auditor eventually results in the auditor making assumptions about where the issues will be relative to physical location or QMS element. An auditor’s ability to anticipate problem areas based on history or other indicators is very beneficial, however when it results in some elements not being audited time after time, it creates an environment where evolving issues go undetected. This is a concern because the assumptions and bias is typically unintentional and ultimately sets us up for failure.
2. Company Growth: as companies grow in size and scope of activities, it becomes increasingly difficult to perform assessments of the full QMS in a single audit. Single audits become impractical, inefficient and ineffective. This typically results in companies changing their approach to a series of smaller scope audits to assess multiple sites and the QMS elements individually or in what are deemed to be logical groupings. In a QMS, the relationship of the individual elements to each other should be thought of as supplier and customer relationships. This supplier / customer relationship involves data & information and depending on the QMS elements, involves physical materials and products. An inadequate assessment of this supplier / customer relationship within the QMS is a common result of using the multiple audit approach. By not adequately assessing linkages and communication between individual processes and QMS elements, we can miss identifying compliance issues.
3. Pre-audit Research: most internal audits are preceded by research of procedures, processes, training, internal nonconformance history, complaints and other information relevant to the area to be audited. Pre-audit research is very beneficial, however as indicated in problem statement # 1 above, an auditor’s ability to anticipate problem areas based on history or other indicators can result in some areas not being audited and it creates an environment where concerns go undetected. As in # 1 above, this is where assumptions and unintentional bias ultimately set us up for failure.
Proven solutions do exist for all three problem scenarios stated above. The good news is that none of the individual solutions will negatively impact the ability to implement any of the others. These are discussed below individually followed by a summary discussion.
1. Small Audit Teams or Dedicated Auditors: having small audit teams is not an uncommon problem for those tasked with assessing the adequacy of a company’s QMS state of compliance.
A. The obvious recommendation for resolving this concern is to hire additional auditors. For equally obvious reasons this is not always possible and it may not always be necessary. Undertaking an honest assessment of the scope of what needs to be audited, the experience and qualifications of the existing audit team and the compliance track record of the company’s QMS will be necessary. A determination can then be made on the need for additional auditors.
B. A common practice in many companies is to take employees from various non-audit departments and have them audit. There are many benefits to this since it helps educate them on the regulatory requirements and fills a resource need without major expense. If properly done, this is a good practice. However, when others are chosen with little to no compliance or auditing background and there is little to no understanding of the QMS elements being audited, many issues can easily be overlooked. If / when this option is used, it is critical to provide the necessary up front education and audit oversite. This increases the baseline knowledge of the short term auditors and saves time for the experienced auditors while allowing the two to discuss and review the audit approach and logic before and during the audit as well as during creation of the audit report.
C. An action that can be taken immediately is to ensure that everyone on the audit team is trained on and knowledgeable of as many of the regulations and standards that apply to the company as practical. By increasing the baseline knowledge of all auditors, the ability for them to audit all physical locations and all QMS elements increases (increased audit team flexibility). This enables the company to rotate experienced auditors through different areas and eliminates, or at least reduces the concern for the audit bias that comes with the same person auditing the same area over and over again. With the scenario of having multiple sites where the company dedicates auditors to individual sites, the increased baseline knowledge of all auditors makes this inter-site rotation a much more feasible option.
2. Company Growth: the necessary action to be taken to address the concerns associated with smaller scope audits and the associated potential for inadequate assessment of the supplier / customer relationship of the QMS elements is to ensure the audits being performed clearly include assessments of the linkages and communication between the processes / QMS elements. This requires a concerted effort to stay away from assumptions about what those linkages and communications are. For example, well known, typical linkages checked between QMS elements include the following:
A. Design output and manufacturing specifications
B. Process changes and training
C. Inspection results and the nonconformance process
D. Field actions and the CAPA system
However, we also need to understand the not so well known relationships between processes / QSM elements such as the following. Some of the suggestions in this section involve understanding additional information. This can be accomplished during the audit and at times may include upfront research. Where upfront research is involved, also refer to section 3 below for suggestions.
A. Complete and accurate ID and traceability information received from the prior process / element and sent to the next process / element: auditing the full traceability information chain from incoming to distribution defeats the purpose of the smaller, limited scope audit. However, verifying the information coming into the process / element being audited and the information being sent to the next process / element is complete and accurate and most importantly does not leave a traceability gap is necessary.
B. Clear documentation of acceptance status for materials, components & products: understanding what has been on hold, been reworked or flagged for any other potential problem provides insight into areas that may require a more detailed assessment (e.g. validation, training, etc.)
C. Environmental and contamination controls from process to process: auditors typically assess these controls for the areas where product is processed, however it is critical to ensure these controls are assessed for storage and transfer activities as well. For purposes of this discussion, storage and transfer includes transfer activities and any storage between the prior operations and the next operations, not just within the activities of the QMS element being audited. Keep in mind that prior and next operations may include transfers to and from other facilities.
D. In-process nonconformance and scrap trends vs. customer complaint trends: when auditing internal processes, understanding what is experienced post-distribution provides insight into potential internal problem areas that can be audited. Likewise, understanding internal scrap and nonconformance trends can provide insight into the adequacy of complaint investigations and evaluations.
E. Any other potential areas of information and material transfer from design through post market activities: As stated above, auditing all the information throughout the entire QMS defeats the purpose and benefit of performing the smaller scope audits, however having an understanding of the key problem areas is beneficial.
3. Pre-audit Research: in order to address the risk of inadequate auditing due to bias from pre-audit research there are key concepts to keep in mind including the fact that pre-audit research is typically an audit of history and like any audit, relies on sampling. This means that it is a snapshot in time and can be misleading relative to current activities. Therefore:
A. If research did not find any issues, it does not mean they do not exist today.
B. If research finds issues in a given area, it does not mean they remain there today.
C. If research identifies or implies the causes of issues, it does not automatically mean it is a root cause or that the cause might not have changed.
This does not mean that pre-audit research is not valuable, in fact it is very valuable. This however does mean that we should use the research as an indication of potential compliant or potential non-compliant areas to help us determine where to begin the audit and where to focus our initial efforts. There is a risk – we rely too heavily on this insight and assume it is reliable enough to ignore some elements of a QMS, some company processes or some procedures. In the end, the auditors must determine how to rely on the research. Too much dependence and we can be inefficient. Too little dependence and we miss the benefits of the gained insight. This is the never ending quandary faced by auditors. It gets easier with time and experience but it is never a guarantee. Use common sense and avoid assumptions.
Auditing is not an exact science. The risk of missing the identification of an out of compliance issue is influenced by sound audit practices including auditor knowledge, auditor experience, understanding of the overall activities that take place and knowing which regulations / standards apply. The key problem scenarios discussed in this White Paper are all scenarios that have been shown to negatively impact the effectiveness of audits. The solutions discussed for each have been shown to be effective in eliminating, or reducing those negative consequences. However, we must always rely on sound auditing practices and avoid assumptions as much as possible. Auditing is sampling. Remember to sample all areas but focus where history tells us there are, or could be issues