Name: Internal auditing tips: from beginner to expert
Uploaded: 2025-05-22T15:26:03.886Z
Duration: 1 h 1 min 4 s
Description: Internal auditing tips: from beginner to expert

Transcript for "Internal auditing tips: from beginner to expert": Hi, everyone. Welcome to our internal auditing webinar. I'm just gonna give everyone a few moments to enter, and then we'll get started. So welcome, everyone. We are thrilled to have you join us for our fourth webinar in conjunction with Medical Device HQ. Our previous webinar is focused on risk management and software as a medical device. They were extremely successful, and I'm delighted to see participants who are at those webinars along with new faces here today. We'll start by discussing how you can interact with us over the next hour. So to communicate during the webinar, please use the q and a box. Your questions will be selected for response as you enter them into the chat. In case we disappear due to any technical issues, stay where you are, and we'll be back as soon as possible. Hopefully, that won't be happening today. So shortly, we're gonna run through the entire auditing process with a deep dive into topics such as the requirements for a medical device internal auditor, should medical device organizations conform to ISO 19 o 11, how to write nonconformities, and what is a risk based approach in auditing. After that, you'll have the opportunity to ask Peter anything you like on internal auditing. Please use that q and a box that I just mentioned. We encourage you to post questions while Peter is presenting as he responds to them as he discusses the various internal auditing topics. Please note that this webinar will focus on the topic of internal auditing in the medical device industry and is not on how the internal how internal audits are managed in Qualio itself. So I'd ask that you keep your questions as generic as possible so that everybody attending benefits from the responses. Also, because we received so many questions in advance, we may not be able to get through all of them in the one hour time frame. But fear not, Peter will be writing a blog post, to cover any unanswered questions, and you'll get a link to that in the near future. So now to the introductions. I'm Lola, a senior quality success manager at Qualio and a lead auditor for ISO thirteen forty five, nine thousand and one, and twenty seven zero one. Before joining Qualio, I worked in quality and compliance roles in large multinationals and start ups across Europe and North America. I use my prior experience in EQMS implementation to work closely with customers in Qualio so that they achieve their goals and bring their product to the market efficiently. I'm delighted to be joined again today by Peter Sebelius, founder and CEO of Medical Device HQ, which I'm gonna tell you more about in the next slide. Peter is the head of delegation in the ISO committee developing the next edition of ISO 19 o 11, which is a standard on auditing. And he's also one of the author authors of the upcoming ISO TS twin twenty three four eighty five, also known as the ISO thirteen four eighty five handbook. Peter is known for his clear explanations and practical guidance. His engaging approach and deep industry knowledge make him a highly regarded trainer, and we're so happy to have him with us today. Before we dive into the into today's session, let me give you a quick introduction into into medical device HQ and how it can make you more confident in the development of your medical device. So Medical Device HQ offers trainings and expert resources to help teams bring medical devices to the market efficiently and with confidence. What makes their courses stand out is that they're created by ISO and IEC standards committee members, and there's a very strong focus on practical application. They also offer fantastic resources in the form of articles and YouTube videos, and I highly recommend checking out the Medical Device HQ YouTube channel. A link should pop up on your screen for that now. Their training cover a wide range of topics ranging from internal auditing, risk management, and software development, all of which I've taken myself and can confirm are excellent. And there's also many other key areas which are offered. You can take them online in blended formats with live sessions and now excitingly also through your company's LMS. We'll include a link to all the courses offered by Medical Device HQ in a follow-up email after the webinar. So without further further ado, I'll pass you to Peter. And thank you so much, Lola, for the introduction. I'm gonna start by saying there is so much to be said about this area. So you might be might find that I'm going through this, quite fast. I'm actually gonna do that, but don't don't worry because, after this webinar, there will be a recording of of this presentation or webinar that will be shared with you. So if I'm going too fast in one or two places, you can always go back and watch it again. That's one of the advantages of having that kind of recording. So, I would like to start by putting some different types of audits on the map for you, particularly you who are beginners, so that you know what I mean and what the webinar applies to and not. So on the highest level, audits can be divided into internal and external audits. Now in this webinar, I will be focusing on internal or first party audits, but more or less everything I will say is also applicable to second party audits or supply chain audits, to use a different term. So the principles will apply if you perform an audit of a potential or existing supplier. Now in the medical device industry, we love standards, don't we, particularly people in QA and RA? And I bet there is a lot of you, on this webinar. So let's bring up a few standards to consider in this context. Now I will be assuming that you, are performing first or second party audits and that the audit criteria or the requirements that you will be comparing with includes the ISO thirteen forty five standard. And when that's the case, also, ISO forty nine zero one on risk management and ISO 9,000 on quality management fundamentals and vocabulary are applicable. Why? Because they are normative references in the ISO thirteen forty five. So to audit an ISO thirteen forty five quality management system, you need to have some understanding of these standards as well. Now the next two standards to consider are ISO 19,011, which is relating to auditing. Also, this standard is referenced in the ISO 13.5, although only in a note field. If the audit is using remote auditing methods, then consider using ISO TS 17,012. Now ISO 17,021, and they are really similar. I know. Easy to get mixed up. But the the 17,021 dash one on the right hand side is only applicable to organizations that perform third party audits. So your notified body or certification body would work according to that standard. You don't need to do that for first or second party audits. Now those were the most important standards that you need to know or be aware of when auditing. I can look at the top of this table on screen. You can see that I've divided it into audit criteria and how to audit. Now, audit criteria is defined as a set of requirements used as a reference against which objective evidence is compared. Thus, if you audit an ISO 13.5 based system, that standard should be the audit criteria. But it could be many other things as well. It could be a supplier agreement. It could be the MDR or the QMSR or any other standard. And this division that I made between audit criteria and how to audit makes sense also when we discuss the competence of auditors. So I will let that be a segue to that area. Peter, if I could just hop in there. Before you talk about auditor competencies, we had a question from one of our participants which ties in nicely with all of the applicable standards that you just discussed. How can an internal auditor ensure that every clause from an applicable standard or every requirement for for a specific regulatory region is covered? And is that even possible? Yep. So that's a great question. The I would say the typical answer to that question is to prepare a really good audit checklist. Now the problem with audit checklist is, it would contain all the typical clauses from a standard and then questions that you're gonna be addressing or asking during the audit. Now the problem with that, it's a little bit of a double edged sword because you do ensure that you ask, for every clause, you ask a question and check up on it. But you also get very, like, it's like panel vision. You're only looking at one clause, so you might miss, all sorts of nonconformities or conformities for that matter that is passing you by when you're focusing on one thing. So I would say use an audit question checklist, but use it with caution and prepare well. That is that is the the best way, I would say, to ensure that you cover it every all simply. Yeah. And you can also yeah. And you can combine different regulations and standards as well, by all means. It just takes a little bit extra work, obviously. Great. I'll pass back to you so to talk about, auditor competencies. Thank you very much, Lola. So, there is a sub clause in ISO 13.5 that relates to human resources, and it says that personnel performing work affecting product quality shall be competent on the basis of appropriate education, training skills, and experience. Now auditing is one of the areas where external third party auditors typically will look for course certificates, as records of being trained in this area of auditing. There are some other areas as well. You will find risk management. They typically look for that. Sometimes usability engineering. The p if you have a PRRC, yes, they expect records of training in that field as well. And auditing is one. So what competencies should you have if you conduct audits? Well, firstly, when you're an auditor, you need to have knowledge and skills in the audit process, simply how it it's done and the audit principles that are behind the audit process. And this goes back to the ISO 19,011 that we looked at before. But you also need to have an understanding of management systems in general, and it's both general as well as sector specific. So you need to understand ISO thirteen four eighty five and what the requirements in that standard mean. So if you're auditing an ISO thirteen four eighty five quality management system, a course certificate like this from, from Medical Device HQ as it so happens on internal auditing will be great because it includes the ISO thirteen four eighty five. But this is not just a discrete, advertisement for my training course. But if you also want to audit with the MDR as audit criteria, then you should add evidence of knowledge in that regulation as well. So before you start auditing, make sure your CV and training courses are relevant, for the audits that you conduct. And I believe there was one question. Lola, correct me if I'm wrong. Like, what do you need to know to audit a sterilization process and things like that? I don't have the exact answer, but you need to have proper training. And And if that's from university or from, business to business professional training, it could be either either of those options, I guess. But you need to be confident. Otherwise, you wouldn't know what you're doing simply. Peter, do you think it would be okay to bring in subject matter experts at certain times like that? Absolutely. You can do that. You can bring in a technical expert that supports the auditor in the process. Yes. But it's also a little bit of a balance. I would say, the auditor, him or herself, needs to know a little bit about the area anyhow. But, the technical support. So the next knowledge area, to understand is the organization's context. If you have never set foot in industry, let's say you were coming straight from university, you will struggle with auditing because what you see will not make that much sense to you. So you need to have an understanding of organizations and in general and how they work. And lastly, you also need to have knowledge about regulatory requirements. Now I'm not saying you need to know every single regulation in detail, but remember that the ISO thirteen forty five refers to regulatory requirements in general. Thus, if, for example, GDPR applies because the auditee is processing personal data, you need to have some basic understanding of the applicability of GDPR. So some knowledge about regulations is a must, but there is more to it. You should also have some personal characteristics and, you could say a little bit more soft skills alongside with these things. Now you should be ethical, and you should be open minded and diplomatic and observant. There are some more things, but we don't have the time today to go through everything. But I think you I think you've noticed, that not everyone is equipped equally when it comes to, in in this example, how observant they are. Because we don't some people pay lots of attention to details, others don't. And when you are auditing, you need to be able to pay attention to details. And I can tell you one example of this just to illustrate what I mean. Now I asked my 11 year son, or or 11 year old son, Theodore, if his room was clean. And he said, yes. I'd clean it. It's perfectly clean, and this is what it looked like. And it's it's like a museum of chaos. And if you can't see that this is a mess, you should maybe not be an auditor. Now I'm not saying Theo is gonna be an auditor, but I'm sure, you've seen these people that are just like, how could you sign that thing when page two was missing? Some people simply don't react. So be mindful of that. You should have that personal characteristic or trait to be successful in this area. Now I am pretty successful at finding things or quite observant. I once ended up writing 13 nonconformities in two hours. Now having said that, it's important to remember that auditing is not about just finding nonconformities. It's equally important to also find conformity, and that is something I'm gonna be coming back to in a while. Peter, in terms of personal behaviors, we had quite a few, questions that were submitted from attendees who said their colleagues don't view them as an auditor because they work with them on a daily basis outside of audits. And on top of that, sometimes their colleagues tend to get defensive in internal audits. You know, they become then the auditor is focusing on their mistakes when the actual intent is to focus on auditing the company's QMS. Have you been in similar situations? And if so, what advice would you have for our attendees? Absolutely. Yes. I think every auditor has been in a situation where the audit team becomes defensive. It's a natural human instinct. The the people that are being audited are living, breathing, and and and thinking about their QMS the whole time. Well, not everyone, but many of them are working constantly on this. And then someone comes in and starts poking at it saying it might not be good. That's obviously, something that could trigger emotions. And, when I did my, lead auditor training many, many years ago, I my instructor, he said, he had a metaphor for this. He said, being an auditor, it's like you meet your best friend who's just got a new baby, and they're walking around in the park with their baby prom or trailer. I don't know the English word, but they're walking around with their with their child. And then you look into to the the prom and say, oh, that's a very ugly child you have. And and that will obviously trigger a lot of emotions. And and that's what we're doing the whole time as an auditor. We are poking at these things and these mistakes. So, obviously and and there's a reason why number three is diplomatic. You need to be very mindful of this and try to be a pleasant person and, not intimidate people, simply. And, still, a lot of people will be really nervous. Another story of this is a large team of auditors came into a factory, and they approached, the person in production, that got so nervous from seeing all the auditors group around her that she fainted at the spot, just fell to the floor. People get nervous, some get defensive, and it's completely natural. And it's the auditor's job to be as, you know, emotionally intelligent as possible to avoid making this appointment. Sorry. That was a long answer to a question, but it's perfectly normal if it happens. No. Thank you for going through those examples. It's great to have those kind of real world, discussions about how auditors make people feel. Yep. Yep. I think it is too. So take that with you, that those stories. So so, if you aim to conform to ISO 13.5 or QMSR or most other medical device standards and regulations, you simply, have to perform audits. And you can see that in this, quote or or, screen capture from the ISO 13.5, which says that the organization shall conduct internal audits at planned intervals. So it is required. There's more text. I'm not gonna read it all, but I'm gonna continue to the next part here, which says that an audit program shall be planned taking into consideration the status and importance of the processes and areas to be audited as well as the results of previous orders. Now I think so far, it's quite easy to understand. But the next sentence highlights something which I think is pretty important, and that's the difference in the level of detail between the ISO 19,011 and ISO thirteen forty five. Because in ISO thirteen forty five, the audit program is only required to include audit criteria, scope, interval, and methods as you can see in the standard. Now this is significantly less information than what ISO 19,011 recommends that you should include in the audit program. So from one point of view, it's maybe not that strange because ISO 19,011 is much more comprehensive for everything relating to audits. So what you have in ISO thirteen four eighty five is the minimum requirement. ISO 19,011 will include much more guidance. And, therefore, I think we should use the ISO 19,011. But if you don't believe me, the ISO 19,011 is actually also mentioned in the note in sub clause eight two four. So also, the authors of ISO thirteen forty five says that you should at least look at the ISO 19,011. Now let me bring up a few things you need to know about internal audits that are not mentioned in neither the ISO thirteen forty five nor the ISO nine two thousand eleven, but these things are nevertheless expected. And I believe there was a question about this, but we might come back to that, Lola. So one of those things is that organizations should not have less than one internal audit per year. And when we talk about the audit program, there is another expectation. The whole QMS must be covered by internal audits every year. Right? If you don't live up to these two unofficial requirements, you may get nonconformances from third party auditors. So and I believe there was this question about what is the minimum required. Right? There was. Yeah. We did get a question about the general frequency of internal audits. I can speak for myself. I've worked in a small company that had limited resources, and we had an audit program that, it ensured the entire QMS was audited but over a two to three year basis. So as the requirements that you mentioned are unofficial, is it still acceptable if a company just simply doesn't have the resources to perform an audit of the whole system every year? I would like to turn the question around and say, why don't you just audit the whole system but very quickly on the parts that are not so important to avoid any potential discussions with a regulatory body that would say you need to cover it? I mean, we should keep in mind there's nothing saying that you need to do it for x number of days or anything like that. So I would I would like to turn it I always do the whole system every year. But if you are in a early start up and it's mostly design and development going on, you might not spend more than a few minutes reading through the complaints procedure, and then it's done. You don't need to look more at it because you don't have any products on the market. And I would do the same there, for as long as it takes. And then, eventually, you start spending more time on the things that you and the processes that you actually use. Okay. And we had a question come in that asked, like, is it better to have a department based audit audit schedule or process based? Oh, yeah. Good question. So this is, this is a tricky one, how you do this in the best way. I tend to believe it's best to do it based on processes, but that requires that you have a, good definitional processes in your organization, which procedures support the various processes. If you don't have that, it seems to gonna be very difficult. So I I I can't give a particular answer to it because it it really depends on the context of the organization and how it's structured. So you need to just make sure you can, from an administrative point of view, ensure that everything is covered. Great. Thanks, Peter. Okay. So thanks for the good questions. So the audit process is made up of three steps. You should prepare the audit. This is something that is often neglected a little bit, And then you should conduct the audit activities, and lastly, prepare and distribute the audit report. Those are the three steps. But there's more, things, happening both before and after. There there should be an audit program just like we read in the ISO thirteen forty five. And if, there are nonconformities, there will be audit follow-up and corrections, corrective actions, and so forth. So it's a little bit more than this. Now before we jump into a more granular flowchart of of this process, I'm gonna bring up an area that is very often misunderstood, and that is, about the risk based approach. So when you work in the medical device industry, we mostly refer to risk as it's defined in the ISO 14,901, that it's about the combination of probability of occurrence of harm and the severity of harm. But did you know that there are and this is a little bit, it's actually quite funny. There are 41 more definitions of risk in ISO standards alone. So 42 definitions all in all. And that obviously creates confusion. One of the most relevant alternative definitions is from the ISO 31,000, and that is that the risk is the effect of uncertainty and objectives. And there is a huge discussion whether this is a good definition or not, mainly because it includes opportunities or positive risk, whatever that is. Now those that are not happy with it, they say there is no such thing as positive risk. So a risk based approach in ISO 31,000 context also includes, for example, exploiting opportunities. And what could that be? Well, it could be that, you, manage to do two audits with one trip only. Instead of going twice, you do two sites at the same with the same trip. That's usually not how we think of it. You can tell the difference. This is completely business oriented, whereas we tend to think about harm and and patient safety or people or safety to people. So it's quite different. But ISO thirteen forty five does include an element of risk as it's defined in ISO 31,000, and that is, the risk of regulatory nonconformities. So it's not only about harm. So one important thing to say, you don't need to have a documented risk assessment for your processes according to ISO thirteen four eighty five. This is a common misconception. As long as the auditee or you or whoever it is, can explain how you've taken risk into account when you determine and set up your processes, you are meeting the requirements of ISO thirteen four eighty five. So that extra risk analysis or risk assessment at the end of every procedure is not required. You can have it, but it's not a requirement. Now it's easy to think that you should spend more time or focus the audit on areas that would make the biggest impact on safety on the safety of the device or the for the patient, for example, production or product release compared to areas that are only indirectly impacting product safety, for example, management review or document control. And sometimes that is correct, but it all comes down to the audit objectives. So we looked at different types of audits before, just at the start. And I I, didn't comment on this last column when we looked at this slide before. But if you take a look at the typical audit objectives and you're going to determine the continuing suitability, adequacy, and effectiveness of the quality management system, then I would argue that you cannot focus more on specific areas. So at least not ignore certain areas because then you have not answered the question if the quality management system as a whole is effective. However, if you are going to identify opportunities for the improvement of the management system, you could focus on areas where you think you will find the biggest improvement. Thus, you can use the risk based approach to spend more time on areas based on the magnitude of risk that would, be the result of failing in a particular area, and base it on the definition in ISO fourteen ninety seven one. So the short answer to the question on how to use the risk based approach is to use the medical device industry, definition of risk, which is about probability of occurrence of harm and the severity of harm, and include the risk of regulatory nonconformities. And then you need to look at your audit objectives to determine if you can focus on various areas or if you should be spending equal amounts of time and effort on all the areas. So it simply not becomes a biased result, that instead should match up to the audit objectives. And I I regret to say that if this is confusing to you, it's perfectly natural. Everyone is confused about this because of the many definitions of of risk in these various ISO standards. So if it's a little bit abstract to you, you're in the right place, I would say. I hope that's reassuring. Peter, before you jump to the next, before you jump to the next topic, not directly related to risk, but we have three questions coming in about training of internal auditors. For example, could a certified lead auditor train a team member to the level of a lead auditor, or do you officially need a certification? And do all auditors required to be sort of are they required to be certified? Are they are there other ways that they could become, lead auditors? So there is nothing in ISO 13.5 that says that training should take place in in a particular way. So, you could most certainly be trained by, internally. But what what I, as an auditor, would like to see is obviously that that person who provides the training is competent enough to do it. And let me give you an example. If it's read and understood of the internal auditing procedure and and they sign off, yes. I read it and I understand. I would not at all accept that. You cannot learn this this topic through, reading a procedure. So it's there is no, yes, no, black, white answer. It all comes down to the the what competence is is should be achieved and the training that went into it and and the people behind the training. Great. Thanks, Megan. Thank you. Thank you. So, Nees, now I'm jumping into the more granular audit process, and we're gonna have a look at this one and go through this and a few more, during this webinar. Now you can see on the left hand side that I've, included who is responsible for which task. So you can see it in these swim lanes. The first step in preparing the audit is to appoint an auditor. And as the swim lane points out, this should be done by the individual responsible for the audit program or the audit client if there isn't an audit program. And the next step is to define the audit objectives, scope, and criteria. This should be done by the individual responsible for the audit program again. However, in my experience, this task will also very often involve the auditor. In fact, sometimes an auditor might have both roles, and be the individual responsible for managing the audit program as well as performing the audits. And that's that's fine. That's fine. Now if you are the auditor, this is where you start. You should reach out to the auditee and go over the details relating to the audit and get feedback from the auditee. And based on how that goes, you will determine if the audit is feasible with reference to the audit objectives. Now the feasibility is very often ignored in the medical device industry. You know, if I'm to be cynical, I would say that some medical device organization, they only include the bare minimum in their procedures to meet ISO thirteen forty five. And there is no mention of feasibility in ISO thirteen forty five. Therefore, it's left out. However, determining feasibility is really important. As you can imagine, if you don't get the sufficient time to perform the audit, which I think a few reported, that they had issues with before this webinar, or the auditee is not cooperating, the audit is not feasible. The whole point is that you should not run headfirst into an audit that already before it starts is doomed to not achieve its objectives. Now if the audit is feasible, you can continue and request quality documents from the auditee. And, thankfully, most of the times, you will find that it is feasible, but it happens that it's not. Now if it's an internal audit, there might not be that much of requesting because everything might be accessible to you on your computer or in your document archive. If it's a second party audit, you're more likely to have to request the documentation. What quality documents are we talking about? Well, it depends on the audit scope, but typically, it would be the auditee's quality manual and standard operating procedures or, the SOPs that are relevant to the audit scope. It could also be records. Then you should read up on the documents you have, that you have received and create the audit schedule and complete the audit plan based on the information you have, which includes the quality documents that I just mentioned. Now what if the ODT does not provide any documents beforehand and said, look. They're all confidential, which could happen. Well, then you simply need to take that into account when you create the audit schedule. Thus, you will have to spend a lot more time on-site or virtually with the ODT. So, Peter, because you mentioned time on-site here and one of the hot topics that came in, had several questions associated with this was time management. So how do you conduct an audit in a reasonable time frame and ensure the right level of attention is given to the different processes and procedures? Well, I would say the more skilled auditing you are, the better you will be at this. I mean, that is the art of auditing. Right? Knowing how much time need to be spent and trying to get that sufficient time to do it. Again, if you don't get the sufficient time, it's not feasible. You should push back and escalate it and say, look. These are not the right preconditions. And even though I didn't have it on my list, integrity is also one of the, audit principles that the auditor should have the integrity to say, no. I'm not gonna do it. This is bad. It's not gonna work. That's great advice. Thank you very much. I know it's true, but it's also difficult. I know that. So my apologies. Yeah. But it's really in the audit principle that you should have that integrity to say no. Yes. So I hope you can, can accept that, tough answer to that question. Now the auditor and the auditee should agree on the plan, and the audit plan is actually, that's one I I should say, one of the templates that we will be sharing with you after this webinar. You will receive two templates. And if, there will be a link sent to you in the email after the, webinar. And if you become a subscriber of Medical Device HQ newsletter, you get this premium template that we usually charge money for for free together with an opening meeting checklist, which is the next thing that we're gonna be look at looking at. But first, let me get a few terms correct. So an audit plan is for one particular audit. And once the audit activities have been conducted, there will be a report of the audit. Now let's let's be careful here because if we refer to internal audits, it's likely that you will have more than one audit to cover the whole QMS. Thus, there will be several plans and several reports. And where do you plan these ones? Well, that's in the audit program. And why do I bring this up? Well, because quite a few people mix them up. They say, audit plan instead of audit program and and use the same term for both concepts. And as always, I say this in probably every webinar and presentation and course I have, that if there is a good and well defined term in a standard, use it and don't come up with your own variations of definitions. So audit program is the defined arrangement per set of one or more audits planned for a specific time frame and directed towards a specific purpose. The audit plan is for one audit. And then what about audit schedule? Well, audit schedule is one section in the audit plan, which defines how much time you spend on each area when you start and finish and when you have lunch and so forth. So I hope by now, don't we? As we've gone through this, that you will always use the right term for these different things. Now let's go back to conducting the audit activities. So the first thing, that happens here, in the, this part or or phase is to conduct the opening meeting. And this is a meeting between the, auditor and the auditee's representatives to get things started. Typically, it takes between fifteen and thirty minutes, and I recommend creating an agenda for the opening meeting so that you if you're the auditor, don't forget to include, all the things that you should say. Now if you if you want, again, to have that list, made for you, we will be sharing a checklist for that, particular the topics like a checklist for opening meetings that will be available to you after this webinar. And and then it's time to start collecting all evidence audit evidence. Many people would say, oh, I conduct the audit. That's where you start here. But, again, ISO 19,011 would refer to this as collecting the audit evidence. How should you determine then if and the document if the audit evidence conforms to the audit criteria or not? And that's what I'm gonna be bringing up soon. But I had a feeling, Lola, did you have a question for me here? You know what? I did, but we actually answered it before, and it was about auditing areas that, we don't have, you don't have, you know, enough expertise in. But another question came in that I think would be really good at this point. Someone wants to know, wouldn't you define the audit objective scope and criteria first? As one must know which qualifications an auditor must demonstrably meet before being appointed? So do that before you appoint the the lead auditor. That is true, but that comes from the audit program, which happens before preparing the audit. So that belongs in the green box before that we looked at. So if we go I'm I'm not gonna do remind on all the slides, but there was the audit program, and then you start preparing the audit. So the objectives and scope, is an input to that step. So it's absolutely correct. Perfect. Thank you. Yeah. So let me then say what you should do and maybe, also, what you should not do when you collect the audit evidence. Now, firstly, don't jump to conclusions. And this is based on my experience from many, many junior auditors and people that have taken my courses in this area. You should remember that the burden of proof that there is a nonconformity is on you, the auditor. You have to show if the auditing is conforming or not conforming. You cannot say, for example, I did not see a particular record. That doesn't mean that it's a nonconformity. You have to go the extra mile to confirm with the auditing that the record is nonexistent. You simply have to do that if it's something that is missing, I should at least say. And secondly, as an auditor, you should not have opinions about how the auditee is running its company or structuring its QMS. Remember, it says in the ISO 13.5 that the outline of the standard is not, meant to guide you in any way on the structure of your QMS. And if you have, previously implemented 10 or 20 or 50 QMSs, you might not have implemented them in the same way as the auditee, and that doesn't mean that the auditee is wrong. This goes back to being open minded. Accept that the auditee might have their own way of doing things. As long as they meet the requirements, there are no more conformities. And this is a very, very common mistake. And while we're at it, you should not audit your own work. Yes. In small organizations, it can be a challenge to find an, independent enough auditor. But auditors should also be trained to realize that complete independence is very difficult to achieve in a small organization. If you cannot achieve it, to a sufficient degree, then you should use external auditors to come in and do the audits. And, yes, external consultants can do internal audits as well. There's no problem for that. That's actually probably been one of, the most common questions that I've seen come in just, in the last few minutes and also before is in terms of not auditing your own work. I suppose the advice that people want for those small companies with limited resources. So the the external auditors or consultants would would be your your top one there. Yeah. Yeah. It would be. Yes. It would be. That but, again, in a small organization, you can't create that complete independence. So, you need to it's you need to use some good judgment there simply on on when you need to have someone external versus when you when you can do it yourself. And if you're in, like, in a startup environment where there are plenty of startups, you could also consider swapping internal auditors with each other. So if you have two competent people, you could just, let them audit each other. Just close an NDA and and swap the hours with each other. That might be one way you could do it for start ups. That's a great suggestion. So, I'm gonna illustrate the first two don'ts here on my bullet point list with examples. And there is a requirement to have a documented procedure for design and development in ISO thirteen forty five. And the auditor asks for the design and development procedure, and the auditee responds, we don't have a procedure with that name. Now the auditor jumps to conclusions by assuming that this response means that there is no documented procedure for design and development. But what the auditor should have done is to continue and added a question that would establish without a doubt that the auditee does not meet the requirement of having a document procedure on the design and development by specifically referring to the sub clause seven three one. If they did, then maybe the auditee would have said, well, we do have procedures for that purpose. It's just that they happen to have a different name than the auditors thought they would have. So I hope you can see how the auditor now jumped to conclusions just assuming that the procedure had a particular name. Again, the burden of proof when writing a nonconformity is on the auditor, and the auditor has to provide objective evidence that the requirement has not been fulfilled, which if there wasn't a procedure in this case, could be that the auditee's representative confirms that no document procedure as required by sub clause seven three one has been established. That would be the evidence needed. Again, it's not enough to just say, I didn't see the procedure relating to design and development because maybe you just didn't look in the right place. Now let's switch to being objective. The auditee notices that the product realization procedure or sorry. The auditor notices that the product realization procedure, includes both design as well as production and thinks that this is odd. And I I could agree that's quite unusual. The auditee states that, well, the merging of these two areas in one procedure works well for the auditee, but the auditor thinks it will be better if the procedure was split in two and writes a nonconformity. Well, firstly, this is not a nonconformity. The auditor should not have opinions on how the auditee structures its QMS as long as it meets the requirements. Now if the auditor is concerned about the merging of the two areas, he or she would have to look for evidence that this has actually led to real issues or nonconformities. So one way to ensure you are objective when auditing is to use a special way of writing the nonconformities. You could say it's a method or a pattern that will help you, ensure that you, don't make up subjective nonconformities. But before we get to that, why do I say nonconformity and not audit finding? We got a few references in the questions before the webinar to audit findings. Well, the term audit finding is used in the pharmaceutical industry. And in that context, it means nonconformity. In the medical device industry, we can also use the term audit finding, but then it refers to either a conformity or a nonconformity. So an audit finding is not necessarily a nonconformity. It's just the conclusion of the evaluation of the audit evidence. And here is another interesting point, at least for people who are really nerdy. When it's a standard, then you conform or don't conform. If it's a regulation, then you comply or you don't comply. That's you should not say that you comply with ISO 13.5. You conform to the requirements of ISO thirteen forty five, but you comply with the NDR. Now you can be a pain in the neck for those that don't know the differences between these terms. Good luck with that. Now when you find audit evidence that does not meet a requirement or audit criteria, you just copy the requirement from the ISO 13.5, when that's the criteria. Then you remove any irrelevant text and just negate the sentence, which is done by adding various forms of has not or not. And then you have a description of the nonconformity. So let's take a few examples. We'll start with five two in ISO thirteen forty five, which says that the top management shall ensure that customer requirements and applicable regulatory requirements are determined and met. During an audit, you learn that the auditee is developing a software that will process personal data for EU customers. The management representative confirms that they have not considered the general data protection regulation or short GDPR, which they really should if they're processing personal data. And this means that the applicable regulatory requirements have not been determined. Now when you write the nonconfirmity, use the requirement text. I copied it down to the track changes role. I removed the unnecessary text and negate the sentence. So I removed the section number. I replaced the shall with a has not and added d in ensure. But then the next edit is really important, and it's relating to the removal of customer requirements and and met from the nonconformity. Now why do I do that? Well, so far, the only information the auditor has at this time is that the auditee has not considered GDPR. So the auditor cannot say that GDPR requirements haven't been met. Now it's tempting to think that the auditee doesn't meet GDPR requirements. It's maybe even likely that they don't. But with this information only, you don't know that. Remember, you have the burden of proof. The auditee might, for example, have implemented more rigorous data protection procedures that would meet or exceed GDPR requirements. Again, very unlikely. But you should not assume that they don't meet GDPR requirements even though they did not consider them. So the final description of the nonconformity is that top management has not ensured that applicable requirements are determined. No more, no less than that. Now if you would like to go down the road and say they haven't met GDPR requirements, you could start inquiring that and find evidence of them not doing that. And then you could add that again to the nonconformity. But at this time, you don't know that. So that was the first example. Let's take one more. Here, I will start with the audit evidence. The quality manual version three point zero states that all members of staff must wash their hands before entering the canteen. Now when attending lunch in the canteen, John Smith was observed not washing his hands before entry. Not good. Now if you read ISO thirteen four eight five, there aren't any requirements relating to washing hands before entering the canteen. So the first question is, is it really nonconformity if there is no requirement in the standard? Well, the answer is yes. The quality management system states that certain things must be done, and John Smith didn't wash his hands. But there is no specific subclothes for washing hands. So what sub clause should you use then as the audit criteria? Well, you should refer to four one three, which says that for each quality management system process, the organization shall, and I jump directly to the c bullet here, implement actions necessary to achieve planned results and maintain the effectiveness of these processes. Now when the organization is not conforming to its own procedures, but there is no specific requirement in the standard relating to that step or what they're not conforming to, this would be the relevant or the criteria to refer to in a lot of cases because the organization, has not implemented the actions necessary to achieve the planned results. If they had, John Smith would have washed his hands before entering the campaign. So using the same methodology as before, we get the following nonconformity. The organization has not implemented actions necessary to achieve planned results. Sticking to this type or or this way of documenting the nonconformity is key to getting it right. Now if you can't write a nonconformity in this way following this pattern, maybe it shouldn't be a nonconformity. Maybe it's a subjective, observation that you've made, which we should avoid. So after having collected the audit evidence, it's time to determine the audit conclusions. This is where the audit, audit or audit or audit team get their notes in order and make conclusions on what they've seen. Meaning, this is mostly about deciding what should be reported as nonconformities during the closing meeting. The closing meeting will often have the same participants as the open meeting. There will be a short presentation of the results of the audit. What everyone wants to know is, of course, if there are any nonconformities. But keep in mind, and this is important, there should ideally not be any surprises regarding nonconformities at the closing meeting. Any discovered nonconformities should have been communicated during the previous steps. So don't come up like, oh, and there are four things I never mentioned to you. That's not good practice. Then again, once the closing meeting is done, the auditor leaves the auditee and starts completing approving the audit report. A piece of advice here, don't do a student syndrome or procrastination or anything like that here because the longer you wait with completing the report, the less sense your notes will make to you. So make sure you get this started as quickly as you can. The more you've typed in during the actual audit, the better it is. And then distribute it to the stakeholders as quickly as you can. And that's where the audit has been completed. But, again, the overall process doesn't end here because there if there were nonconformities, there must be follow-up and the auditee would implement corrections and corrective actions. But those activities are formally outside the audit. So that was the audit process on a fairly high level. Does it always look like this? Not really, but it's good if it does because what I presented is well aligned with ISO 19,011. And ISO 19,011 does represent best practices in auditing. And remember, it is even referenced in the ISO thirteen four eighty five. Yes. So, Peter, we had a question that came in about what the best practices for acting on any recommendations, minor nonconformances, or major nonconformances that might come from an audit. Like, what should be documented as a kappa, an n c, or what shouldn't be actioned at all? So that's a good question. I would argue that if it is a well, let me put it this way. If it is a nonconformity, then you need to be really careful if it's not gonna, lead to a corrective action. The way ISO 13.5 is, written, more or less all nonconformity should result in corrective action. Now there are exceptions to that. You might already have the corrective action ongoing in the same area or you do see it as a single one off, which is, with a very low frequency or something like that. But, generally speaking, if it's a nonconformity, it should be a corrective action that follows. And I would say corrective action, not preventive because if there's been a nonconformity, I consider them to be corrective actions. And yeah. And there is that question of minor and major. Keep in mind that that's not a requirement to have in ISO thirteen twenty five. You may want to have them, but those are actually coming from the ISO 17,021 dash one that I brought up at the early, early in this presentation for third party, certification bodies where you, where you have those minor and major grading, but not in ISO thirteen forty five. Yeah. Did that answer the question, Yolanda? That answers the question. I know we're we've just come to the end of the presentation that you gave, and we have ten minutes left for questions. And I have a lot of questions that have come in, so I think we're gonna fill that ten minutes. So, to everybody who's participating, I'll do my best to get through them. As I mentioned, if not, we'll make sure we get you, the feedback that you're looking for one way or another. May I just say may I just say one thing, Jennifer? Because I would love to connect with people that have listened to this web webinar. If they're interested in internal auditing as I am, I would love to have them in my network on LinkedIn. So while we while we, address questions, I put up a small QR code, which I hope you can see and just click on your phone and connect with me. I will be very happy about that. I hope it works the way it's supposed to. And there will be the links as well for the YouTube channel, which I recommend seeing. There is one or two videos also on this. So now now I'm I'm, I'm I said what I wanted to do. Now You're fine. I'm just the questions now. Great. Okay. Well, the first two are gonna be relevant to AI. So given the world that we live in, is seeing a massive surge in AI adoption, What are your thoughts on AI being used for compliance monitoring and to aid in the internal audit process? Yes. So firstly, don't trust it. Use it as a tool, but don't trust it. That's very important. For example, if you if you let Chunky Petit, talk about risk. It's, and medical device risk management is definitely gonna mix up ISO fourteen nine so one with FMEA, which a lot of people are now at the last starting to become aware of. It's not the same thing. So it simply, replicates a lot of the problems that are out there or the misunderstandings. But as a tool, yes. It's great. I, for example, use it when I write, conformities. Now conformity is actually if you see notified bodies, audits reports, they write a lot about, like, evidence that the oddity is conforming. They spend more time on that than nonconformity, ideally. And I always find that I'm not a native English speaker, so I always struggle a little bit about writing that in a nice way. So then I found that I can easily put in, like, which documents, and and I say they seem to comply pretty good with you know, I use the wrong words and anything. I say, could you write this so it sounds like a good conclusion? And, poof, I get a nice text that sounds great. Obviously, again, you need to check it and review it carefully so you can stand by it. But, but it does work. And you can also ask Chatipati to review procedures and say, is there something missing? Again, don't replace it with reading yourself. I would start by reading it myself and basically say, did I miss something? So I think one should be really, really careful that it is a tool that can be used. But then you also need to check, obviously, if it's a second party audit, are you allowed to upload the auditee's procedures to Chattingly Petit? Are you even allowed to upload your own procedures to Chattingly Petit? An important question. Again, then there are other AI tools or or large language models out there, but you need to check it in every instance of this. Thanks, Peter. I agree with you on that. Stephanie, it's a very exciting time. I you know, AI won't replace humans for auditing, but there's definitely ways that we can use those tools to, speed audits up, make them more efficient. So I'm excited to see what's what's coming in the near future with us. The second AI question that we got from somebody, was their product is software as medical device that also incorporates AI. While we understand it's challenging to find someone with experience in both areas, would you still recommend that the external auditor have relevant experience in both? I would recommend it. Yes. Would I require it? I'm not sure. I don't know. That depends on, a little bit on the classification of the product and the risks involved and how the AI is actually working. I would say that. So I I can't give a definitive answer on that. No. I I appreciate, that answer. And, yeah, it's it's hard to find someone with ex experience in both, but it's always good to see, you know, whoever's performing your audience is constantly trying to, develop professionally in areas that maybe they weren't originally trained in. So hope that answers your questions to the participant that put that in. Okay. Some other interesting questions that have come in. Is there any requirement regarding how you should record evidence? Well, you are required to record it, and it should be as objective as possible. So it's like, think of it as something coming from the court that the the oddity, with the name or employee number x and y zed was going through the cleaner without cleaner gowns at 11:22AM or whatever you write up there. It should, it should be as clear and as objective as possible. Great. Someone else has asked if there's a delay in the distribution of the auto report, should that be raised as a kappa? Well, it could be. It could be. Yes. Because if, it it depends on what your procedures say. If you say that the the audit report should be sent, within, I don't know, three weeks or something or be finished within three weeks and it ends up being six weeks, then you have that clause for one three that the organization did not achieve the planned results. It achieved something else. So that is, by definition, a a nonconformity. Fantastic. On to the next one then. We have one attendee who noted that they are often harder on themselves during internal audits than they need to be. So they said that they try to mirror the same level of stringency to that of a notified body auditor. What do you say to that? Is it possible to be harder on yourself than the notified body would be on you? And is that a bad thing? Well, the in the perfect world, you you would just have, like, either you conform or you don't, and all auditors would agree on that. Right? That would be the perfect world. In real life, it's not like that. There is a little bit of wiggle room, or subjectivity always even though we should try to avoid it as much as possible. So I think one should try to have the same level as everyone else. Now what is that same level? That's hard to tell. But I also I can also say that a lot of notified body auditors actually, I would I would say and I'm I'm I'm sorry if I if I I am saying this in a bad way, but, I wish sometimes that more auditees pushed back on some, notified body auditors because they are very used to, getting their will through. Like, people assume that they know it all and that that they have the right interpretation of the requirements. But a lot of them are new. A lot of them have forgotten a few things. And and then I think one should say, well, we do consider that the requirement has been met because a, b, and c. We we don't think this is a nonconformity. And that's something that too few companies do, I think. That's some really valuable advice. So I'm gonna push on to another question that we actually got a lot of presubmissions in, not directly related to internal auditing, but centered around leadership commitment. How do you get leadership to understand the importance of internal audits? Yeah. So so this is, this is a good question. In lots of organization, management sees QA and quality management as an overhead. It's something that is necessary for compliance, but with very little value added. I regret to say that, but that is my experience. And I think what the question we should all ask ourselves in the first place when we work with QA and RA, does the quality management system, our internal audits, bring value? Are they efficient? Meaning, do do we waste resources while we do them, or do they bring value, a positive return on that investment that we do in them? And if it does, we should explain that to those that are paying for us to do it and market it, basically, and say, this makes perfect sense. Look at all the big improvements we've done that we have we have saved, patients' health and money in the long run because we did all these things. Now if you say, now our internal audit, they don't bring value. Obviously, management is not gonna be committed to it. They don't want to spend money on it because it doesn't add value. But if it does, we would not want to invest in it. So I think the first question is really to say, is this thing worth it, the time and effort? Does it do what it's supposed to? And if it does, then promote it internally as much as you can. Great. Thanks, Peter. An interesting quick question here on how an internal auditor shouldn't jump to the issues that they already know about. So if you know, for example, where the weak points in your processes are, should you audit those straight away or could you be missing potentially bigger issues by not dedicating enough time to reviewing a particular process as a whole? Well, I would say yes and yes. But the because I believe there were two questions. If you know that there is a problem, I think you should write it up as a nonconformity to get management's attention and say, look. This is something we need to work on. How how deep do you need to dig? Well, it depends. I mean, root cause analysis should do the digging actually as well. So from that point of view, you don't need to go that deep. But, but I would definitely bring it up without just ignoring everything around it, for sure. So, Yeah. I hope that answers the question. Do bring them up. I know and this is oh, this is so cynical. I know some auditees, QA managers that have deliberately pointed out mistakes in their QMS to the notified body auditor because they haven't been able to convince management to spend time and money to fix the issue. So then they just give it to the order and say, can you write this up? Because then it needs to be fixed. Speaking of management extension. Strategic strategic planning there. Great. We you know what? We're in the last minute. We've got fifty seconds left. So, Peter, I'm gonna wrap the questions up here. But as I as I mentioned to the participants, we'll figure out a way to get the, unanswered questions answered. So thank you so much everyone for joining us. Thanks for participating, and I hope you've all learned something new today. The session is recorded, and it'll be emailed to you soon with lots of extra useful resources including the link to Medical Device HQ, a short survey to get your feedback on how you you found the session this session, which would be really appreciated. I want to thank Peter so much for his time and answering your questions. Peter, in the last couple of seconds, anything you'd like to add? No. No. Big thanks to you, Lola, for hosting this, and thanks to everyone who joined in. It's a super interesting and very important area. So thanks for spending the time with us. Great. See you, everyone. See you. Bye bye.