Artificial Intelligence in Healthcare: A Change Management Problem

This report is based on a webinar presented by Jason Jones, PhD, Chief Data Scientist at Health Catalyst February 27, 2019, titled, “3 Perspectives to Better Apply Predictive & Prescriptive Models in Healthcare.”

Artificial intelligence (AI) is a hot topic in health and healthcare today. Most of the focus has been on the technical challenges of building and deploying predictive and prescriptive models. While there is always room for improvement, algorithms have advanced to a point that allows us to focus on the bigger challenge: change management. By acknowledging that change is hard, “tooling up,” and maintaining a human-centered perspective, we can achieve better adoption and results with AI.

Predictive and prescriptive models can enhance health and healthcare, which will ultimately help improve Quadruple Aim outcomes: population health, patient experience, reduced cost, and positive provider work life. However, successful adoption of predictive and prescriptive models heavily depends upon behavior change, which requires more than technical accuracy.

While predictive algorithms abound, tools to facilitate change management remain scarce. This article will help clinical and operational leaders obtain value from predictive and prescriptive models using three perspectives: functional, contextual, and operational. Investing time and effort to ensure these three levels of model understanding is necessary for broad-scale AI adoption.

Predictive Versus Prescriptive Models

Healthcare leaders currently delving into AI must understand what predictive and prescriptive models are and how they differ:

• With a predictive model, data scientists try to anticipate a person’s behavior or a circumstance before it happens—ideally, with enough time to take action—but it does not necessarily tell data scientists what action to take. For instance, a predictive model might reveal a patient’s prognosis and a 30 percent chance of readmission ;within 30 days (but not how to manage the patient’s risk).
• A prescriptive model goes one step further—not only does the model tell data scientists what is going to happen, it also recommends the specific course of action the health system should take in response. In this readmission example, not only would a prescriptive model reveal the patient’s prognosis and chances of being readmitted but also recommend the patient be scheduled for a follow-up phone call one day after discharge and a multidisciplinary visit three days after discharge.

Healthcare leaders and data scientists have many predictive and prescriptive modeling resources available, including some free clinical prediction models. In fact, these resources teach healthcare professionals that developing predictive models is no longer a question of technical aptitude but whether a health system is prepared to deploy them.

A Framework for AI Change Management in Healthcare

The first question clinical and operational leaders need to answer is, “What are we trying to achieve?” Algorithms and predictive models are commodities employed to achieve specific outcomes. A famous quote by musician George Harrison applies to this concept: “If you don’t know where you’re going, any road will take you there.” The same can be said about predictive and prescriptive models. It’s important to know the desired outcome before setting out to build a predictive model.

Healthcare leaders often disagree over the best course of action to affect change within an organization. To successfully adopt AI technologies, frontline leaders can use a framework for change management that sets expectations and facilitates productive dialogues to encourage consensus and buy-in. The framework includes three levels of understanding:

1. Functional: Does the model make sense? If a patient has multiple comorbid conditions, does it make more sense that he’s going to be readmitted, or that he’s taken care of because he’s receiving care from a group of specialists? The functional level of understanding ensures data scientists and healthcare leaders recognize the relationship between the input feature or parameter and what they’re trying to predict or suggesting as an action item.
2. Contextual: How does it fit into the workflow? A healthcare analytics team has built the world’s most accurate predictive model, but the team doesn’t know what to do with the results. Analytics teams can’t lose sight of the fact that the results of this prediction will drive patient action. It is imperative that the team understands who the patient is, what she’s going to do with the prediction, and when it would make sense for her to do it.
3. Operational: What benefits and risks are traded? Healthcare professionals and clinicians make very few choices that are risk-free and don’t require some level of trade-off. How can analytics teams understand those trade-offs and make the best decisions for the organization? And how can the organization as a whole make those trade-offs with eyes wide open and accurately measure if they’re achieving success?

Three Levels of Predictive Model Understanding

Once healthcare leaders have the three-level framework for change management, they can begin to deploy predictive and prescriptive models. These three levels of understanding create the framework for change management that enables successful deployment and adoption of predictive and prescriptive models. Attaining functional, contextual, and operational understanding helps ensure that teams don’t lose track of the problem they’re trying to solve and bolsters their confidence within the organization to move decision makers toward agreement.

Three Example Predictive Models to Demonstrate Understanding

To demonstrate, this article will examine three examples of how teams can move through the three levels of understanding.

#1: Functional Model Understanding

In the first example, an analytics teams wants to build a predictive model that determines which patients in the emergency department (ED) are at greatest risk for progressing to severe sepsis or septic shock within 24 hours.

In functional model understanding, the team might begin by asking questions to achieve a basic understanding of how the model should work:

• Does a patient’s risk for severe sepsis or septic shock increase when his temperature is low or high?
• If the patient’s temperature is low, should his risk go up or down?
• If the patient’s temperature is high, should his risk up or down?
• If so, by how much?

The analytics team needs help answering these questions; they need input from clinicians to build a model that makes sense. Figure 1 illustrates a sample model for predicting whether a patient will progress to severe sepsis or septic shock within a 24-hour period, using six key parameters. In this example model, the patient’s risk of developing sever sepsis or septic shock moves up or down based on how the parameters change.

Clinician input and understanding are key to getting the healthcare organization to embrace and adopt this model down the road. Data analysts and data scientists need to make the predictive model transparent, so clinicians understand how it works, and, in turn, support the model—a critical step for adoption.

George Box, a British statistician, famously said, “All models are wrong. Some models are useful.” The team is trying to build a useful model, and the utility will not just be how well it predicts the event, but what somebody does with the newly discovered information. Obtaining functional model understanding is the first step.

Contextual Model Understanding

Taking the Sepsis example above, obtaining contextual model understanding requires the team to ask a different set of questions:

• Are we finding the right patients at the right time?
• Who should take action?

Once the team has created the predictive model and gains a functional understanding, they need to analyze the model in a contextual sense, starting with whether the model is identifying the right patients at the right time. If the model identifies a patient who is at risk for severe sepsis or septic shock but fails to identify him until after he’s in critical care, the model is not useful.

The next question the team might ask for contextual understanding is, “Who’s going to take action?” Should a phlebotomist draw a lab with an automated order? Should the patient receive fluids earlier? Figure 2 depicts an example I-Chart that provides details leading up to an incident that can be used for contextual model understanding.

To obtain contextual model understanding, the team would look at patients who may be at risk for severe sepsis or septic shock. The I-Chart provides additional information about diagnoses, comorbidity, and recent clinical encounters (e.g., whether the patient had recently been in an ambulatory setting). The chart shows the first 24 hours, from the time the patient checks into the ED, and the six parameters identified above by the analytics team in functional model understanding. This allows the team to understand the complete context for this patient in relationship to the predictive model. With this information, and the questions they ask in the proper context, the team can make adjustments to improve the model’s utility and further gain clinician buy-in.

Operational Understanding

Using a different example to demonstrate operational model understanding, the team wants to identify patients who will develop congenital heart disease (CHD). Unlike the sepsis example, the team is not limited to events in a specific set of hours or days but decades worth of information. In building this model, the team will want to consider the following questions:

• What percentage of patients will the model capture?
• If the patient is prescribed a statin, what is the chance she will develop CHD?

To build a useful model, the team will not only need to think about which patients might develop heart disease over the next decade but also what to do with that information. One possible action is to prescribe a statin for the patient. Prescribing a statin is a relatively low-intensity action but also comes with possible side effects and associated costs. Operational model understanding compels teams to examine the cause and effects and weigh the tradeoffs that often come with medical decision making.

If the team’s goals for this model are to capture 50 percent of patients who develop CHD in the next 10 years and, for those patients identified, have 50 percent of statin prescriptions associated with CHD, how can they set expectations appropriately for the tradeoffs they’re making? Can they gain clinician and leadership buy-in for a model that would benefit 50 percent of identified patients? If not, how can they modify the model to get the buy-in they need for adoption? This type of organizational dialogue and value balancing facilitates operational model understanding.

Supporting Leadership Decision Making

To successfully adopt predictive and prescriptive models, most healthcare organizations will do the heavy lifting in change management and efforts to overcome organizational inertia. The analytics teams can leverage three specific tool sets to facilitate the change management process: functional understanding, contextual understanding, and operational understanding. With these three tools, teams are more likely to successfully deploy predictive and prescriptive models.

An important part of the success of this model is ongoing collaboration, along with the goal of creating something that everyone (from data scientists to clinicians) can understand, trust, and utilize. Successful deployment ultimately depends on the usefulness and level of adoption of the model.

Additional Reading

Would you like to learn more about this topic? Here are some articles we suggest:

1. Meaningful Machine Learning Visualizations for Clinical Users: A Framework
2. Healthcare Data Management: Three Principles of Using Data to Its Full Potential
3. Machine Learning in Healthcare: What C-Suite Executives Must Know to Use it Effectively in Their Organizations
4. A New Era of Personalized Medicine: The Power of Analytics and AI
5. Machine Learning, Predictive Analytics, and Process Redesign Reduces Readmission Rates by 50 Percent

---

PowerPoint Slides

Would you like to use or share these concepts? Download the presentation highlighting the key main points.

Click Here to Download the Slides

Artificial Intelligence in Healthcare: A Change Management Problem from Health Catalyst