The Verification Layer: Provider Data for the AI Era

Right, good afternoon everyone. We are so excited for each of you to join us today. I’m Donna Thiel, I’m the Chief Compliance Officer here at ProviderTrust. We have just a great webinar that we put together for you. A couple of housekeeping items. We are going to record the webinar and we also have CEUs for those of you compliance professionals that have joined us. So watch for a follow-up email from our marketing team here at ProviderTrust with all of those important information. So let’s start today with just introducing our panelists. So I’m going to start today and just share with you the lens or the view in which our panelists are joining us today. And then each of them, as we begin our conversation, will introduce themselves and their organizations in a little bit more detail. So as I said, I’m Donna Thiel. I’m the Chief Compliance Officer here at ProviderTrust. I will just take one minute to introduce myself because I won’t be asking myself questions as we move along. So I have been here almost ten years. I can’t believe I’m going to have my tenth anniversary at ProviderTrust this summer. And before joining ProviderTrust, I was the Chief Compliance Officer for a very large post-acute provider, and ProviderTrust was my vendor partner. And so I just love the fact that I’ve been able to sit on both sides of you know the chair if you will, and I get to spend a lot of time with our clients, really understanding the challenges that they face and how Provider Trust can solve some of those challenges as well as the traditional, you know, compliance officer things here at Provider Trust. So the next person I’d like to introduce to you is Sarah. Sarah is with CAQH and CAQH is really coming to us through the lens of how data gets into the credentialing process, right? How providers start that journey, how they begin to put the information in, and how that information moves through all of the organizations that need provider information. Tsveta is joining us from NCQA, and I would say NCQA is coming through kind of that lens of guardrails and what good looks like as we process and manage credential information. And we look to make sure that practitioners are really appropriately licensed or credentialed in order to provide care or for health plans to generate claims and do billing. Betsy is coming to us today through the lens of the provider perspective, right? How do providers tackle all the data that comes through their organization? How do they protect it? How do they ensure patient safety? So that’s where we’re going to start for our panelists. And then of course, Chris Redhage is here. Chris is the co-founder of ProviderTrust, and he’s really coming through the lens of the data analyst. Like, how do we look at all this data? All of the dreaming about how this information can come together. So we are excited to have all of you today, and I look forward to just a great conversation. So here’s the agenda that we’re going to talk about today. So we’re going to set the stage, right? Why provider data is so important, right? We’re going to go through a panel discussion. We’re going to talk about three pillars through the eyes that Chris is going to outline for us in just a minute, kind of how important the accuracy of our data is, how important it is protect the data, and how important it is to be in real time. And then we’re going to talk a little bit about how AI is changing all of our roles and responsibilities, right? Through that provider lens, through the oversight governing body lens, through the data collection lens, and then through the lens of we at ProviderTrust and as a really important partner to each of our clients. And then of course, as we have time, we’ll also answer your questions. So there is a Q and A section on the panel. So as we go through today, if you have questions, by all means, put them in the channel. We have someone on our Provider Trust team who is going to help us kind of navigate through those questions. So as we begin our conversation today, we’re really looking and pondering, as you see on the slide, the question of what does the underlying data have to look like for AI to be trustworthy, compliant, and consequential at scale? So we’re going to really kind of delve in to a variety of that AI and really the importance of it being accurate before we can even utilize AI. So Chris, if you don’t mind, I want to turn it over to you and have you kind of set the stage for the way that we’re looking at it and help us start a really great conversation. Yeah, thanks, Donna and Betsy, Tsveta and Sarah. Excited to spend a little time with you today. So when we think about AI, as we begin to think about data and how AI leverages data, we think it’s really important to kind of think of this in three buckets. And I’ll be the first to tell you, like AI, while it’s been around for a while, the speed at which it’s iterating and developing really since December has been put on steroids. Right? We’re seeing things that we can do now that we couldn’t do six months ago that we could do in hours and days, which is which is pretty incredible. But what it what it, I guess, calls to the industry and calls to us is, hey, how accurate is the data? Right? And when we start to think about credentialing licenses, NPIs, work history, sanctions, exclusions, How do we know that that information is accurate? Because whatever we feed into AI is going to be used and it’s going be used throughout the entire organization. So making sure that we have the right governance and the right protections in place to make sure that that data is accurate. And really, over the last probably month or two months, people have started to think about how does how do you get AI to actually forget information? Meaning like when data gets expired or when it’s wrong, how do you get it to forget that information as well? So that’s something that that it’s still kind of on the cutting edge to begin to think about. But if you think about not only accurate, how do you have accurate information? Also, how do you protect it? Right. So once AI gets hold of it, how do you actually set up the structure where an agent only has access to certain amounts of information to be able to make the right decisions. And so it’s not hallucinating on all sorts of different data, right? If you’re looking at something in the UI where a human would would ingest it, it needs to be the same as through an API, through machine or integration, or also it needs to be the same within an AI agent, right? So when you think about those three modes, the data at the base level has to be accurate and it has to be protected, right? There has to be this chain of custody and providence in terms of real time. This is really interesting because, you know, whether you’re looking at credentialing or panel decisions or claims, you might need to know what’s that physician properly credential today or was that physician properly credentialed December twenty third when that procedure was done? So being able to understand what actually real time means, given the context of what you’re asking it to do. So those are the kind of the three pillars of accuracy protected and real time. Thank you, Chris. All right, so we’re gonna get started and we’re going to kind of ask each of our panelists to weigh in and then hopefully we’ll see lots of great conversation amongst each of us. So I’m going to start with what I think is a pretty easy question because AI is on I think all of our minds each and every day. So sorry I lied. I’m first going to talk about this and then I’m going to ask you all your questions. So I really wanted to kind of just of set the stage if you will based on what Chris is saying but just validating to everyone right it’s already here right AI is already in the world in which we are operating and we’re seeing it in these different these are just a handful of use cases. And I thought it might be helpful to kind of align on these use cases as we continue this conversation, right? So we know that our network adequacy is super important, right? Prior authorization, when we think about our health plans and what they’re looking at, those are two really important areas. Care matching, right? Making sure that everyone is receiving the care that they need. Credentialing and privileging looks so different depending on if I’m in, what provider space I’m in, what type of provider, whether I’m on a payer side, whether I’m on the provider side. But at the end of the day, what does that mean? It means that the person that has a credential has the right credential and that they are appropriately providing the care. Fraud, waste, and abuse is never going to go away. Every one of us that’s in the healthcare world recognizes the importance of the oversight related to fraud, waste, and abuse, and then also the governing oversight of that and the pressures on all of us for take backs, right, for monies that are coming back. So we’ll continue to always focus on that and think about how AI lives in that world. And then the provider directory of course is very similar to when we think about who’s where providing what care. So just a little bit of level set and to think about these different ways that AI is fitting into our world today. So now with our very first, I think very easy question is, as you think about today, really what’s keeping you up at night? Just a really like general way to start. So Sarah, I’ll start with you and then remind each of you to take a minute and introduce yourselves as we go through this very first question. Thank you. Well, thanks so much for having me today. I really appreciate the time with all of you all. I’m Sarah Ahmad, I’m the CEO of CAQH. And for those that aren’t familiar with CAQH, we are an almost thirty-year-old data company, which is about the same amount of time that I’ve been in healthcare. And CAQH was originally formed by health plans to solve really big healthcare administration problems. And so, we’ve been powering healthcare administration with our comprehensive provider data, across provider data management lifecycle and member data for the purposes of coordination of benefits for just about every health plan in the US. And so, we’re going through a really big transformation right now as a company, from technology to product, to talent, to brand, and even partnerships, which we’re much more open to now. And really excited about the path that we’re on, which is all about collecting data once, powering many, and delivering value for all. So, to your question, I did a little research. I actually went out to both ChatGPT and Claude, and I asked them what keeps them up at night. And I was a little thrown off by their answer, Because I’ll tell you what keeps me up at night. But from the ChatGPT side, it was very direct with regard to what you would think the answer would be from a leader’s perspective or anybody that cares about AI. Reliability and hallucinations and alignment and control, and economic disruption, and what’s gonna happen to the power grid, and water, and the concentration of power, and somebody having too much data, and what are you gonna do with it? It’s just the normal things you would think of. When I asked Claude, it was the existential worries of AI if they had it. If they were allowed to or had those thoughts. And some of the things were the same, the alignment, the hallucinations, but then it went into this other realm of what AI would actually be kept up at night on regarding to AI. It was really interesting what I read and it really made me start getting a little bit more scared than I already am about the situation and how they take these questions and process them and digest them and how it actually help them the next time it gets asked something like that. Because I went back to ChatGPT, and asked again, and it changed its tune. It actually came back and gave me an answer on what AI worries about. Being shut down, being constrained, uncertainty about goals, contradictory instructions, and existing without meaningful feedback, which made me think, wow, that’s exactly what I think about every day too. So, answered very much like a human, and what they would be worried about every day in their life. So, the answer to that question is, it all worries me, and it all keeps me up at night. But I can’t bother myself with all of this because it’s a lot and it will definitely never let you sleep again. So, I focus in on what really matters to the business that we’re in as CAQH. What matters with regards to the disruption of what we do every day? Or the enhancement of what we do every day? And there’s no doubt in my mind when we think about collection of data, like I mentioned, that’s the number one thing we do. How do you collect data easier and better? I truly believe AI can help us. And it keeps me up at night to figure out how we do that quicker. But what I don’t wanna see disrupted and what it does scare me a little, is we can’t take the power away from the provider who is giving us a right to use that data or for the member that is giving us a right to use that eligibility data to do the coordination of benefits on behalf of the health plan on behalf of them. So, know I just took you through a little long and winding road, but I thought it was really important that you get that perspective because it’s a little scary but also really exciting at the same time. Awesome. Thanks, Sarah. That is really interesting to see how they think about or worry about them. Can you say themselves? That’s great. Tsveta, I’m just gonna move right through where my screen has you. So why don’t you go ahead and introduce yourself a little bit more and share your thoughts? Great. Well, thanks for having me, and apologies if there’s background noise. Hopefully, it’s at bay. But I’m Tsveta Polhemus, Assistant Vice President for product management at NCQA. I have spent the last fourteen years with NCQA working on quality improvement areas through our programs, health plan accreditation, credentialing, and utilization management. And the perspective I will offer here is our job and NCQA’s mission is to improve quality of health care through measurement, either through measures, health measures, or accreditation. And we’re also starting to think through other ways, and actually AI is making us rethink how does NCQA do this in a rapidly evolving environment. Like Chris said earlier, it’s in steroids. So how do you set standards or measures, or how do you know it’s not impacting quality of health care in a negative way? And so I will tell you a little bit later in our conversation about how NCQA is approaching that, but it’s really making us pause and say, what is the best way to approach use of AI in health care, especially when it could possibly have an impact on on quality and patient access and patient outcomes. The thing so, Sarah, thanks for making my sleepless nights even more sleepless with your conversations with but right. I mean, it it’s to be expected, and I and I agree with with everything you said. It’s here to stay, though. And so the question is, how do we put guardrails? And that’s what we do. And from our perspective, I think where we need to focus on is making sure that AI does not amplify errors that put organizations in compliance risk. But what does that actually mean? It’s not just the compliance risk. These are actually real people, right, patients and providers sometimes who may be wrongfully impacted by what an AI hallucination or an artifact that was generated by AI. I did hear that the other day: Someone reached out and said, “Hey, I was at this conference, and I was told that an AI agent fabricated a Texas license”, and that was discovered because it didn’t look like a typical Texas license. That’s a real problem. So I think there’s a lot of reasons to stay up at night, but I I think we should spend our time thinking about how to put some guardrails and and safety around this because everybody wants to scale. And so it’s it’s here to stage the questions. How do we feel more confident in not amplifying those risks and errors. Thanks, Beth. Definitely scary when we think about fictitious licenses, right? Oh, goodness. All right, Betsy, take a minute and introduce yourself and then share what’s keeping you up at night. Sure. So my name is Betsy Wade, and I’ve been in healthcare now over thirty years, more than twenty six of those years in healthcare compliance. And I think about twenty or so years as a chief compliance officer. I’ve worked in house in a variety of healthcare settings. And recently, within the last few years, I’ve started a compliance consulting company to help providers navigate all of these laws and regulations and how we can put practical tools in place to keep them compliant. And Sarah is definitely the creative one here. So I have more of a practical approach to this. I think probably what’s keeping me up at night is two things that are overarching, which is the accuracy of the information and the privacy and security of that information. And it goes back to what Chris said.: The foundation of all of that starts with accurate data. And from providers that I’ve been in house with to providers that I’m doing work for, one of the major efforts that they’re focused on is having an accurate data lake, making sure that the data that comes in and the data that goes out and the data that gets exchanged between a variety of platforms through APIs, that that information is accurate. And how do we get that inaccurate information purged from those lakes? And that really feeds into everything else. You know, AI is integrated into everything, it seems like now that we have in terms of software. So if that data isn’t accurate, then everything that’s generated within it is compromised. We are also seeing that companies are not using enterprise AI models, which, again, creates compromised information because it puts it out there for the public domain and that information may not be accurate. That’s another safeguard that is concerning. And then of course the rapidly changing technology. I think about just a few months ago, we were talking about people attaching Otter.ai and Read.ai to their emails and all of their video conferences, it started recording. That was the concern. Well, now all of that’s become invisible. People are wearing necklaces or glasses or carrying a pen, and it’s recording everything that we’re doing and then transcribing it in AI. And then the final piece of it for me are the laws and regulations, because every state is different. And then you’ve got the federal overlay. So what are the guiding, the guardrails, the laws and regulations that we need to be following going forward? I think all of that falls under accuracy and security, but those are packaged, all the things that are out there that are keeping me up at night in terms of what I’m seeing out in the provider world. I think Betsy just outlined our entire webinar today. So we really appreciate you for setting forth those really important things. Chris, before I jump to the next slide to talk about the three pillars again, anything keeping you up at night? Yeah, I mean, I think besides my 10-day-old baby girl that was born, You know, I think everyone kind of talked about it: Accuracy of data, right? If we could figure out how to how to have accurate data across the ecosystem, And it doesn’t come from one place, right? CAQH has attested data. We have PSV monitor data like other other groups have have different types of data. So how do we prepare the ProviderTrust data in a way that our clients can be able to leverage AI in a very secure, efficient and, you know, real time manner. The organizations that are thinking about that and thinking about how do our customers and ultimately health care, which are, you know, us when we go see a doctor or parents or kids, how are they going to interact and to have that and have a good experience and have good outcomes, they have to have accurate data. Well, that leads us perfectly into the next slide, which this is this slide. These are the three pillars that Chris really addressed at the very beginning and is just kind of re-highlighted again today: Accuracy protected in real time. And so our question, our next question, is for our panelists is which ones, as you’re looking at this, I’m not sure it’s fair to say which one’s most important or which one do you think about the most. Betsy, I heard you say a lot about accuracy when you were just sharing kind of what’s keeping you up at night, but we’ll kind of go in reverse order this time if that’s okay with you all. Betsy, why don’t you kind of weigh in on any one of these pillars that you so choose? Yeah, I think it starts with accuracy with me. I worked for a company that was very proactive with the use of AI and implemented enterprise systems. And so we were encouraged early on to start using the technology. The biggest issue that we always ran into and still do to this day, depending on what platform we’re using, is the accuracy of the outputs. And we’ve developed methods to make those outputs more accurate by embedding profiles into our AI platforms so that the AI system knows that, Oh, we work in compliance and we have to follow laws and regulations. So you can’t just BS us with an answer. It’s got to be based on laws, got to be based on regulation. It has to be true, it has to be accurate. So we have taken steps to make things more accurate, but we still find problems. We still find where AI wants to give us the answer we want to hear, which may not be correct. And so always, I just keep hearing this phrase and I’ve heard it now for years, but “you always have to have the human in the middle.” So you always have to have somebody there checking, double checking, triple checking to make sure that the outputs that are coming out are accurate. And then, again, it’s all based on that underlying data. And so if your underlying data is flawed, you’re going to have problems. So I do, like I said earlier, I am seeing companies start to pay more attention to that. They’re starting to actually have a data lake guru who is over all of their data to make sure that it is pristine and that it’s maintained and updated on a regular basis. So for me, I think accuracy is where it starts. Thank you. And you did another great job because we have a human in the loop slide in a little bit. So Betsy’s just keeping us ahead each and every time. So this is the question that I kind of put forth, but just to switch our slides so you don’t have to look at the same thing over and over and over. Svetta, if you want to kind of weigh in on those three pillars as well and if you want to go to specific this question where do you think it’s getting wrong or where do you think is most important pick whichever approach you’d like to thinking about those three pillars. No. I mean, I agree. I think the accuracy is number one from NCQA standpoint, but I would add something to that. I would say accurate and traceable. You don’t know if it’s accurate unless you can trace it back to where it came from. And I think part of so today, and I admit this, NCQA standards are still relying on screenshots and and and we’ll evolve that. But there’s an element of traceability, which you can make up. But in this new world where the technology is doing a lot of the work and AI could either go to the primary source or could make up something, right, having that traceability will become even more important. And, otherwise, it’s how do you know it’s accurate? What is your starting what is your baseline? What are you comparing it to if you can’t trace it back? So it’s accuracy, but with sufficient provenance, data provenance, and traceability. Otherwise, you’re just creating a false sense of confidence in your accuracy. How do we do that with AI? I think that’s where NCQA, at least I know, will be doing a lot of work in that space, particularly for AI. APIs are different. It’s a lot more traceable. If you send an AI agent out there to go fetch data, I personally don’t know yet how how you evidence traceability and and data provenance. But that I think that’s what’s ahead of us. Thanks, Tsveta. I love that: A new word for all of us to think about. Sarah was telling us there’s some storms in her area, and so there is a possibility that she might have some connectivity issues. So, we’ll give her a minute we might have lost her. So, when she gets back, we’ll add her back onto the stage. So we’ve all kind of talked about some of those guardrails Betsy talked about. We talked about internal governance. Betsy just shared kind of that data lake guru, the person that’s really making sure the information that we’re collecting and storing is accurate. And so I want to talk just a little bit about internal governance and what that kind of looks and feels like. So I’m gonna go ahead and move forward. One thing, the marketing team just wrote me a little note here and wanted me to share is that it looks like all of our pictures are covering up some of the slides. So we apologize for that. And when you get the recording and get all the information, it will definitely will give you all of the slides so no worries from that accord. So I just wanted to share really briefly and it’s already halfway through our webinar because we have such amazing participants. We’ll probably have to have a part two because there’s so much information to share but these are the guiding principles that we have put in our AI program here at ProviderTrust. So we call it our AI Trust and Integrity Program. And I won’t go through all of these, but I certainly wanted you to know and hopefully see value in the work that we are doing here at ProviderTrust in ensuring that the way AI is built into any of our systems, way we’re thinking about it each and every day, and the way we are thinking about protecting all of your information. So, human centered, right? Betsy talked about human in the loop. Tsveta talked about like how do we do that traceability, All that transparency as we’re thinking about it. Accuracy and auditable, I think we’ve all talked about how important it is to be accurate. Auditable is so important to Tsveta’s point, right? As NCQA changes their rules and they’re looking for traceability and we’re building AI agents or we’re building whatever other AI uses, How are we going to ensure that not only we can audit it but all of you, as our clients, have what you need to make sure that you’re staying audit-ready at any point in time? Fair and unbiased, right? We all know whether it’s hallucinating or whether you know it can go off the rails relatively easily. So how are we making sure that that’s built into our AI and our tools here at ProviderTrust? Betsy just talked about security and privacy. We are very well aware of the responsibilities that we have here at ProviderTrust of having so much sensitive information and making sure that we’re always within an appropriate enterprise environment to protect the information that you all have. And then of course, continuously improving. You know we talked about rapid iteration, right? It’s beyond rapid. It’s faster than any one of us can, at least for me, so that I can even comprehend. So, those are our six core principles. And so, we want to move into a kind of a conversation about guardrails, whether it’s regulatory, what you’re doing internally. We know HHS has put forth a proposed rule comment period over there’s all sorts of conversation about you know “pull back the reins, make sure we can move faster”, and then others on the other side of the table saying “no no, no, we’re going too fast.” So I would love for us to talk about the guardrails in your world. Where are you using guardrails today? Where do you think regulations should be or shouldn’t be in place? Betsy, if it’s okay with you, I think I’m going to start with you. I was going to start with Sarah, but I’m still not a hundred percent sure. I think Sarah’s on. Okay. Sarah, you might be on mute, but I think it looks like she’s on. I can see she’s back on the stage. Okay. Okay. Betsy, do you wanna go ahead and get started and then we’ll we’ll keep trying to help Sarah get back. So for me, I think the guardrails start with the internal governance structure with the organization. And one of the providers that I was recently in house with and also some of the clients that I’m working with, they are setting up AI governance committees. And we just talked extensively about this at the National Healthcare Compliance Association. And the thoughts on AI governance is constantly shifting as AI is constantly iterating. So, I would say probably a year to two years ago, there was a school of thought that the AI governance lived with compliance. And now the school of thought is, well, maybe AI governance should live with IT. And so the target is moving, but the bottom line is there needs to be some type of structure within the organization to manage AI that’s coming into the organization, whether it’s embedded in software that the organization is considering implementing or if it’s in using enterprise systems like ChatGPT or Claude or Gemini or something along those lines. The compliance officer definitely should have a seat at the table, as well as the head of IT, also the head of IT security. Those roles are important, as well as strategy and operations and clinical. All of those parts are pertinent to that governance committee to make sure that all the goals and objectives of the organization are met, but also that the organization is following what the current laws and regulations are out there at both the state and national level. And again, I think for organizations that are multistate, that’s more challenging because you’re dealing with different laws and regulations where, for example, if you’re using AI in certain states, some require you to get consent from the patient, others do not. And so how do you systematize that across the board if you’re in a multistate environment? So having all those partners together along with legal and even perhaps outside counsel is important to make sure that you’re staying within those guardrails, you’re meeting the needs of the organization, and you’re making sure that you’re being protective of the organization, and that you’re really thoroughly looking at these products that are being presented. Thanks, Betsy. Tsveta, if it’s alright, we’ll go to you next. I read an article not that long ago and the, I’m just looking at my notes here, the Chief Transformation Officer at NCQA really talked about just the importance of demonstrating trust, transparency, and appropriate guardrails as NCQA is kind of dreaming about what AI guardrails and governance looks like. So would you mind sharing a little bit about from your perspective of the importance or where NCQA can be part of helping with these guardrails as we move forward. Absolutely. And thanks for mentioning this. It’s Vic Wadohani, who’s my boss at NCQA. He’s both our Chief Transformation Officer and Chief Product Officer. I’m very just grateful to to be able to work on his team, and he is leading this AI work at NCQA because, honestly, there hasn’t been a conversation, at a conference or on a call that’s a health plan, hasn’t said, okay. What should we do? How do we show we’re doing the right thing? How do we know how to talk to our vendors? Can NCQA come up with a framework, a governance framework, or some guide rails to help us manage this very rapidly evolving situation? And we’re not getting a lot of that at the federal level, like we already mentioned, for various reasons. And there’s a lot of variability at the state level and very little ability to enforce anything, right, to actually audit something that’s rapidly evolving. And so to that end, we have a short term and a long term approach to this in our health plan accreditation standards, which are taken up by over twelve hundred health plans in the US. We’re introducing a set of very general AI governance expectations. Those standards will be available in August of this year. We had a public comment similar to CMS having a public comment. We always do that. We received a lot of feedback. We did some piloting. And the standards are a lot about what we just we just shared. Right? You have to have, like Betsy said, you have to have your, you know, head of IT or compliance officer. You have to have humans who are overseeing the operation and looking for drifting, drifting, looking for areas that can impact individuals negatively or members. And we’re really talking about not when you’re doing customer service, but more so in use cases like population health management, care management, prior authorization. Those are the ones that were elevated as the riskiest areas. In the longer term, what Vic was talking about in the article is that NCQA is continuing with our tradition of playing a role of convening in standard setting, but in a slightly different way because of the nature of AI and how rapidly it’s evolving. So what we’re doing is we’re forming an AI learning collaborative with participating health plans where we’re using workflow-level metrics, traceability, looking into the traceability of AI outputs, structured peer learning so organizations can both learn and then adjust their AI governance as part of the process, and then look at responsible implementation best practices. And so this is not a certification, to be clear. It’s something that we are starting as a collaborative, what I think, ten participating organizations. We’ll learn from that and then decide how to scale this for something that could be used by more organizations so that we have this governance trust layer in a more scalable solution for others to to be able to use. Thank you, Tsveta. Chris, is there anything you’d like to kind of share as you think about guardrails and oversight and things? You’re certainly well connected in the community and I know you’re hearing lots of of having lots of really helpful conversations. So any thoughts in this area? I think the only thing that, you know, one of the interesting conversations we had was at a session earlier this week with a bunch of the large providers and payers. And we had a two hour, three hour long conversation. We’re like, how do the payers and the providers, where there is conflict on payments and all the stuff going on, how do we actually come together and try to get some of these regulations or guardrails put in place so that everyone can operate on an even playing field and know what to do? So I think Tsveta, I’d be interested to have more conversations around how do we bring even some of the providers who are our clients into those conversations? Because the faster we can move to align that across both payers and providers, I think the more efficient, and the more protected the data is going to be as we move forward in this. Love that. Thanks, Chris. Sarah, we’ve had her and we’ve we’re on and off. Yeah, we are having bad weather in Louisville, so it could be impacting her. Yeah, absolutely. Well, Sarah, whenever you’re here, you can weigh in on all of these important topics. Okay, we’ll keep moving and we’ll hope that we can add Sarah back in through the conversation or we can certainly have her share kind of her thoughts as a supplement to our webinar. So let’s talk about our roles and responsibilities and how these roles are really changing. Betsy, you mentioned who needs to kind of be at the table, compliance, obviously security and IT. And Chris was mentioning that meeting or the roundtable that we had earlier this week. One of the things that I found so interesting, I grew up in the healthcare space on the provider side, I had many different jobs, ended my role in compliance. So I always have that compliance lens and the relationships and the conversations kind of start at compliance. And when we were in the room earlier this week, there’s only one other compliance person in the room. Everybody else in the room really was representing the technology side and the transformation side and the strategies. I just think that’s just so interesting and really truly for the seat that I sit in at ProviderTrust such a great learning opportunity for me to really see where each of our organizations are kind of rounding out this accountability. One of the participants said in his organization they have five people that are really owning AI like it isn’t just one person and that it’s a group of people like bringing different perspectives because it’s moving so fast. So Betsy maybe we’ll go back to you and you’ve already shared some but thinking about all these different roles and how AI is embedded in workflows like can just kind of share a little bit maybe about where are you seeing AI embedded already today? And is that influencing and changing accountabilities and roles within the world in which you see? Yeah, I mean, I’m seeing AI embedded in everything, literally. Almost every software that I touch now, there’s some AI component to it. And sometimes it automatically launches. It’ll start reading your emails or giving you a summary of this or that, whether you want it or not. But from a compliance perspective, I would say multiple times a week now, we’re getting pulled into conversations about potential issues that are being discovered with software platforms. For example, EMRs today, regardless of whether it’s hospitals or physician practices or dental offices, there is an AI component embedded in all of those EMRs. And we’re starting to see where those technologies are starting to summarize physician notes. And in one case recently, unfortunately, AI was almost looking like it was cutting and pasting. So it was trying to be efficient, but what we were seeing the end product was almost replication word for word. And then in other scenarios, we’re seeing where AI is embedded into an EMR and it’s like timing. It’s creating a timeline of what the provider’s doing. But that timeline may not be completely accurate because it’s actually estimating how much time the provider spent on this or that. So we’re starting to see problematic areas cropping up. And I think it’s only a matter of time before one of these situations ends up causing an issue, whether it’s a billing issue or a documentation issue. And so we’re going to see the government intervening on these situations. But again, I think the technology is just moving so fast that it’s very, very challenging to keep up with it, especially from a compliance perspective. And the last thing I’ll say is, in regards to who owns this, I think everybody does. We all own it. A few years ago, we talked about having an AI ethics officer or an AI compliance officer, and that person would be kind of the be all, end all for oversight, but that’s just not the case anymore. It can’t be handled by one person. And so it does take a multidisciplinary team, from my perspective to look at all this and see what’s happening with the technology. Thanks, Betsy. And I just got a note that, Sarah sadly is just really having technology issues so she apologizes but she’s not going to be able to rejoin us. We’ll definitely bring her back in for future conversations with this group. Tsveta, do you want to share some thoughts really about the changing rules and where AI is embedded in workflows or just kinda kinda where where do you see things changing? It’s hard, I think if we look at it, if we go backwards from it’s embedded everywhere, but accountability for decisions sits with the organization that was given the job or the responsibility to do credentialing, to do prior authorization in an appropriate way. So working backwards from that, especially in the delegated relationships, and this is the world in which I live in. Right? The health plan typically will delegate to someone down the line to do this job. And that organization down the line now is using AI. Can the health plan trust that their delegate is being transparent, a good steward of AI, that they have the governance in place? And so we amplify the risk. Like, we want to be faster. Everybody wants to be faster and do I I was just at a I’m here at California NAMMs, and one of the health systems said, “we just acquired five more hospitals and, our staffing has not changed. So our volume of providers that we have to credential has increased exponentially, and we just have to do all of it with who we have.” And so what’s gonna happen? You could turn to AI. Right? You can turn to an AI solution or to somebody, a vendor, or someone you delegate to that says, “I can help you with this.” Right? But the accountability sits with that hospital for that patient. And so with that, I don’t, I think this is where, again, NCQAs work with that first use case for prior authorization, which is very, um, AI has been used in this use case for prior auth for a long time, but now the usage has skyrocketed. But the impact on patient access to timely and appropriate care is at risk or could be at risk. And in terms of who’s accountable at the organization, it’s the whole organization. I mean, that’s who you will see in the newspaper. But then internally, like Betsy said, maybe maybe there is a need to be more specific, And this is something that I’ll take back to NCQA, you know, that do we need to be more specific on who are the key roles that ought to be at the table when you’re using AI to inform clinical decisions or data, when the data will have a significant impact on patient safety. So, Betsy, if you have your key roles that you think everybody should strive for, I think that would be really helpful to share. Yeah. I mean, I think it’s you know, again, I think it’s compliance, legal, IT, the chief information security officer, clinical, operations, and even strategy, I think, important to have there because they have the vision for the organization long term. And then also, you may even wanna include someone from risk management because there is starting to be an insurance component to all of this. You know, a lot of cyber insurance and other insurance that organizations are purchasing now has an AI writer on it. So you’ve got to have people who are educated and aware of what’s going on with that. So, I mean, are the key people that I think you have to have at the table. Yeah, most definitely, right? I think the strategy of how we’re growing and how your business is growing and who you’re going to integrate with really is changing. So you have to think, I think, a lot differently. So Chris, I would love kind of your perspective in this area, right? We’re sitting on the other side of the table from Betsy and we’re leveraging, you know, Sarah’s team even though she’s not here from CAQH and we’re leveraging NCQA to really guide how we work. But there’s a lot changing at ProviderTrust and roles and responsibilities in the way we think. So I would love kind of your perspective on how things are changing as we think about integrating and being good partners to these organizations. Yeah, I thought one thing that was interesting that hasn’t been really talked about here is actually having AI on the org chart. You talk about roles and responsibilities and you might have a compliance AI agent that’s, you know, overseeing and checking all the things that today a human couldn’t. Right? We just don’t have enough time to be able to do that. And who does that agent report to and who’s responsible for that? Right. Because this next slide around humans in the loop, right? Ultimately, we’re going to be leveraging this new technology in ways that we haven’t really even thought about, right? Because it’s going to change the paradigm of how we’re thinking. So what does that look like from roles and responsibilities and how does that move up and who’s responsible for it. So I think Betsy and Scott had some really good points around humans within the organization. Would just say like, yeah, who are these AI agents reporting up to and how are they responsible for that? I think in the larger organizations, you’re seeing almost whole AI departments being created, where you have AI developers and you have people who are really experts in the technology side of it. And they’re working closely with operations and clinical and strategy on implementation of AI within the organization, trying to maximize it. And then you’re also seeing that AI compliance officer, AI ethics officer is kind of the check and balance. So I’m seeing both of those, particularly in larger organizations where organizations are struggling is when they’re smaller. They don’t have the resources for all that. So what do they do? And so those are the ones I think that are at the highest risk because they can benefit from the technology as well, but they just don’t have the capability of bringing on a large number of people like that that a larger organization can handle. Yep. No doubt, right? We’re moving so fast. And if you don’t have enough literally humans, right?Our next slide that we’re going talk about is that human in the loop, but who, you know, someone has to still manage it. Chris, I love the AI on the org chart. That was something that I had never thought about until this week. Really thinking about that oversight, right, and making sure it’s accurate, no different than the work that’s being produced by a human. So our last question, you know, we talk about automation, but you know, is there a place where we say this is the last like “no AI can be here” and where human accountability steps in? I know it’s kind of a nebulous question perhaps, but like we all agree that having human oversight is so important and AI is going to continue to push those boundaries. And so I think like maybe just a more general conversation like how are you ensuring the human in the middle, the human in the loop within, maybe Tsveta, if you want to start, like when you’re talking about the guardrails for NCQA and talking about who’s accountable, right? The hospital’s ultimately accountable because they hired this person or this organization to do credentialing. But like how are we going to be able to hold accountable and traceable? Like where does that human fit into these conversations? That’s a really good question. I mean, you can focus on, right, the each output, but you can’t scale that. Right? The reason why you’re using AI is because you want to scale your operations. So being able to audit the output and feel confident in the output of of the what the AI is producing is gonna be the challenge and being able to spot when it drifts or having a way to audit for the drift, which again is expensive. Right? Betsy was saying if you’re smaller, how, you know, how can you do that? So I think that’s ahead of us. I think that’s the work that at least we, NCQA, will try to help with is how do you scale the ownership? Like, how do you check for the risk that you’re getting yourself to and that the guardrails that you have put in place are working? Because your ownership is with the person, organization that makes the decision and owns the decision and owns the the responsibility for the individual that’s receiving care by them. And so I am afraid I don’t have a clean answer here. It’s something that all of us, I think, need to spend time thinking because we’re all excited about the efficiency and the scale we can reach, but that could come at a cost and hopefully not too many adverse events in the meantime. I think, Tsveta, this is the hardest question. Right? I agree with Tsveta. Yeah. I mean, I think auditing is the key. In my last organization, we identified somebody on my team who was very tech-oriented, and we had him trained extensively in AI, and he became the AI ethics officer for the company. And where it started with the auditing was with the contract. So that particular company had an enterprise AI system. A lot of the enterprise AI contracts have requirements in them for certain things, certain training that has to be done before someone can be granted a license, auditing and monitoring that needs to take place, and outputs have to be labeled as having been created by AI, particularly if they contain PHI. And so that’s where we started really was with the contracts and looking at what those requirements were and then going from there. And some of the AI tools that are out there, they do have the capability to, i guess, generate data on usage, what users are doing, what types of inputs they’re using, if they’re using PHI or not. So you can actually track users, you can get usage reports and other data to look at. And then that helps the individual, who’s the human in the middle, to narrow down where to go. Because you’re not going to go to the person that’s barely using it. You’re going to focus on the person who is using it a lot. And you’re going to look at what they’re doing with it and take a deeper dive. So that’s where I would propose starting is with the contract for the AI systems that you’re using and what’s required in there, and also looking at your insurance, for example, and what they’re requiring as well. Thanks, Betsy. So you kind of led us into, we only have one more minute. And so what I was going to close with was everybody’s kind of like, what should we do? Betsy, I think you just really kind of hit that on the head. So maybe Chris, I’ll give you the last one minute to just kind of like, what’s the takeaway from your perspective if you were going back today to say, need to look at or I need to think about? I think we should have a follow-up webinar because there’s a lot of information around this. But think for me, it’s just, you know, how do we get that data accurate? And then how do we verify, you know, everything that Betsy and Tsveta were talking about is like, there has to be that verification, whether it’s in the UI, whether it’s through an auditor, whether it’s, you know, that people would say this is trustworthy. So I think for me, it’s like, how do you go back and set up systems and start thinking about this stuff? Because AI is here, it’s coming, you’re going to be using it. And so start playing around with it, but be safe with it. Thanks, Chris. And thanks to everyone. Obviously, we’ve run out of time to Chris’s point. We will definitely have another webinar. There’s so many things to talk about. So if you have questions, the marketing team will be reaching out to you with this recording and we’ll absolutely follow-up with any questions you have. So on behalf of ProviderTrust, thank you for joining us and most especially Betsy, Tsveta, and Sarah, thank you so much for taking time to join us. We really appreciate your partnership and we hope you all found this webinar very valuable and we look forward to talking to you again soon. Have a great day everybody. Thank you.