How to Use AI in Healthcare to Unlock Four Secrets in Your Data

Healthcare has focused its development of artificial intelligence or augmented intelligence (AI) on narrow, transactional use cases, such as readmission risk, deterioration, sepsis detection, and smoking cessation likelihood. These important, transactional use cases happen one patient and provider at a time.

Just as importantly, health systems must focus on broader use cases like management and strategic analyses. An error in interpreting performance at the management or strategic level impacts many patients and providers, not just one patient and their care team as in the case of narrow, transactional use cases.

Surprisingly, only 4% of analysts, department leaders, and even board members can correctly identify all four of the following points in a line chart: baseline, extreme points, persistent shift, and forecast. Regrettably, these four points remain a secret to most analysts and leaders, even though these points unlock the meaning of the data. In response to these findings, Health Catalyst urges health systems to focus significantly more development of AI in healthcare on empowering broader, strategic use cases for leaders to reduce the risk of incorrect analytic interpretation.

Is your organization’s leadership part of the 4% that can correctly identify these four points in data presented over time? If not, your data and analytics may be doing more harm than good. Take this interactive assessment now to find out where you stand.

AI in Healthcare and the Four Secrets in Healthcare Data

Today’s health system leaders strive to be data driven. They understand that across industries, organizations that use data to make decisions are two times more likely to hit their business goals and grow at an average annual rate of 30%. 

To become data-driven, healthcare organizations must uncover meaningful insights from their data. Correct analysis and interpretation will reveal valuable insights or secrets hidden in the data health systems already capture. For example, leaders should analyze data collected over time to determine whether performance is increasing, decreasing, or remaining steady. Specifically, leaders should attempt to uncover four valuable secrets in that data—a baseline, extreme points, and persistent shifts in performance, and use that information to forecast future performance.

How All Leaders Use Data

Leaders can use the four points shown in Figure 1 to make data-driven decisions:

• Baseline—to decide whether they should intervene to improve performance.  Further, leaders use baselines to decide the appropriate components of SMART (specific, measurable, attainable, relevant, time-based) goals.
• Extreme point—to determine what factors caused isolated extreme performance (points outside of normal variation) to decide whether to replicate (positive performance) or prevent (negative performance) those factors as a means of performance improvement.
• Persistent shift—to determine whether performance has substantively changed or if a recent change is simply within normal variation. They decide whether to intervene when performance shifts negatively. They celebrate or double down when performance shifts positively.
• Forecast—to decide whether future performance requires intervention. They may not intervene in poor performance that is forecasted to improve. Conversely, leaders may intervene in good performance that is forecasted to decline.

How Quality Leaders Use Data

Data is used to improve performance throughout healthcare organizations. Chief quality officers (CQOs) and quality directors want to use data to make decisions about care delivery for septic patients. They could use the four data points in a myriad of ways:

• Baseline—to ­understand the baseline performance and variation to set a SMART goal such as, “improve sepsis mortality from 15 to 12% within one year.”
• Extreme point—to conduct root-cause analyses (RCAs) on extreme points in sepsis mortality to determine whether patients’ health or another factor caused a temporary drop in performance.
• Persistent shift—to determine whether sepsis mortality has meaningfully changed. They then continue to monitor for persistent shifts to determine whether an improvement initiative reduced sepsis mortality rates. 
• Forecast­­—to predict future performance and determine whether to focus improvement efforts on sepsis mortality or another clinical quality measure. If the forecasted performance for sepsis mortality is the same, the quality leader may focus improvement efforts on another clinical quality measure that is rapidly declining.

The Challenge of Unlocking the Four Secrets in Your Data

Health systems invest millions of dollars into data and analytics—this includes investments in people, processes, and technologies—to become data-driven organizations. Even with those investments, business and clinical leaders can still draw incorrect conclusions from correct data and analytics. Worse, business and clinical leaders don’t know how often they draw incorrect conclusions from analytic visualizations.

Consider a data visualization of sepsis mortality like that in Figure 2:

Note that the four points shown in Figure 1 are more difficult to determine in Figure 2. This is the line chart Health Catalyst uses extensively to measure the accuracy of analytic interpretations. Surprisingly, only 4% of analysts, department leaders, and even board members correctly determine all four points: the baseline, extreme points, persistent shifts, and forecast. Unfortunately, misidentifying these four data points can lead to incorrect conclusions and, therefore, negative consequences. Let’s look at each.

The Problems with Misidentifying Baselines

Identifying baselines is not easy. In the sepsis mortality line chart, the baseline could be interpreted as 9%, 13%, or somewhere in between.

If leaders incorrectly identify the baseline as better than it really is, they may not set a much-needed goal to improve that measure’s performance. Alternatively, SMART (Specific, Measurable, Attainable, Relevant, Timebound) goals can be misdirected without a clear baseline.

Use case – Without a clear baseline, quality leaders may accidentally set a goal with a target rate for sepsis mortality that falls within the range of normal variation. While that goal will certainly be attainable, it is not meaningful. Worse, leaders may falsely conclude that the “improvement” is due to targeted interventions that, in reality, did nothing to improve performance.

The Problem with Misidentifying Extreme Points

Identifying extreme points is complicated. Notice in the line chart below that some extreme points are obvious, while others are not. The Month End Date 2016-02-29 is likely an extreme point. But what about 2018-02-28?

Misidentifying extreme points wastes time and opportunity.

• Wasted time: When leaders misidentify points within normal variation as extreme points, they waste time and resources conducting RCAs on those points.
• Wasted opportunity: When leaders fail to identify extreme points and don’t conduct RCAs on those points, they miss the opportunity to prevent negative performance or replicate the positive performance in the future.

Even worse, should leaders misidentify an extreme point of positive performance, conduct an RCA, and replicate the cause, all the effort to replicate the cause is worthless because the performance was within normal variation.

Use case – Without clear identification of extreme points, quality leaders may not recognize their sepsis mortality rate significantly worsened during a period with high rates of respiratory illness. They will not realize high rates of respiratory illness stretched their emergency department (ED) staff and disrupted sepsis care delivery. Without that knowledge, they will not set a goal to mitigate the impact of surges in the ED.

The Problem with Misidentifying Persistent Shifts

Identifying persistent shifts in performance is also challenging. Notice in the sepsis mortality chart below that the presence or absence of a persistent shift is unclear.

More unclear is whether multiple persistent shifts have occurred:

• Did the performance start to improve on 2019-02-28?
• Did the performance start to decline on 2020-02-29?
• Or are both points within normal variation, meaning both 2019-02-28 and 2020-02-29?

The time and opportunity costs of misidentifying persistent shifts are similar to the cost of misidentifying extreme points.

• Wasted time: When leaders misidentify performance variation as a persistent shift, they waste time and resources conducting RCAs on those points.
• Wasted opportunity: When leaders fail to identify persistent shifts, they miss the opportunity to conduct RCAs aimed at reversing the cause of declining performance.

Even worse, should leaders misidentify variation as a persistent shift, conduct an RCA, and intervene, all the effort to reverse the cause is worthless.

Use case – Without clearly identifying persistent shifts in performance, quality leaders will not know whether the start of a labor shortage in critical care staff worsened sepsis mortality rates.

The Problem with Misidentifying Forecasts

Performance data does not typically include a forecast for future performance. However, to aid in decision making about interventions for improving sepsis mortality, leaders must consider what the performance rate will be in the future if nothing changes.

Forecasting performance isn’t easy. In the sepsis mortality line chart, performance ranged from 11 to 19% from August 31, 2020, to March 31, 2021. Further, performance declined three months in a row (October through December) but improved during the last two months (February and March).

Consequently, predicting sepsis mortality rates during the next six months won’t be easy. Questions include the following:

• Will performance continue to improve to 11% in six months?
• Will performance remain at about 15% during the next six months?
• Will performance worsen to 21% in six months?

A leader must correctly predict performance to know if they must intervene to improve performance. Even more difficult, a group of stakeholders must agree on the predicted performance to know whether they must intervene to improve performance. When stakeholders can’t agree on predicted performance, they can’t agree on whether to intervene to improve performance. One leader may confidently declare that performance will improve only to find that performance worsens.

Use case – A quality leader may incorrectly decide to shift their focus to performance in other areas based on an incorrect forecast that sepsis mortality rates will continue to improve.

The Solution for Unlocking the Four Secrets in Your Data: AI in Healthcare

Given the level of complexity in data interpretation—even when using visualizations—it’s no wonder only 4% of healthcare analytics users can correctly interpret the four key points. Fortunately, augmented intelligence (AI) can help healthcare leaders optimize their analytics for a myriad of strategic use cases, including the interpretation of performance data over time.

The use of AI in healthcare to strengthen traditional analytics has been termed augmented analytics. Gartner defines augmented analytics as, “the use of enabling technologies such as machine learning and AI to assist with data preparation, insight generation and insight explanation to augment how people explore and analyze data in analytics and BI platforms.”

Configuration of Advanced Statistical Methods

Organizations must develop AI in healthcare that automatically and correctly applies advanced statistical methods for use with performance measurements over time (time series data). Accordingly, data science teams must develop AI that applies methods like statistical process control (SPC), changepoint detection, and forecasting:

• SPC charts enable decision-makers to identify baselines and extreme points.
• Changepoint detection enables decision-makers to identify persistent shifts in performance.
• Time series forecasting enables decision-makers to predict performance during the next six months.

Automatic Configuration of Advanced Statistical Methods

Organizations must develop AI in healthcare that applies advanced statistical methods automatically because data science resources are limited. Most data science teams do not have time to manually apply advanced statistical methods for all charts that present performance over time—not with the volume of charts consumed every day across healthcare organizations by clinical, operational, financial, and population health leaders. Additionally, most business intelligence analysts do not have the knowledge base to apply advanced statistical techniques. Consequently, data science teams must create AI that automatically applies advanced statistical methods that guide leaders when interpreting performance over time.

Correct Configuration of Advanced Statistical Methods

To develop AI in healthcare that automatically applies advanced statistical methods for time series data, data science teams must develop AI that correctly applies SPC charts (with changepoint detection), and time series forecasting based on the variables below. Specifically, the AI must determine the appropriate techniques for different types of healthcare measures, including the values in each measure. Incorrect configuration may lead to incorrect conclusions and flawed decision making.

Correct Configuration of SPC Charts

AI in healthcare must automatically and correctly apply SPC charts. To apply SPC charts correctly, the AI must be able to identify the measure type and distribution of the time series data first. Then, the AI must correctly apply the appropriate chart type, center method, and confidence interval calculation. Below, Table 1 illustrates the common mapping between some measure types, chart types, and confidence interval calculations.

• Important: The AI must choose the best configuration of variables below, evaluate the performance, sensitivity, and specificity, and automatically reapply if necessary.
  

Correct Configuration of Changepoint Methods

AI in healthcare must automatically and correctly apply changepoint methods. The correct changepoint detection method is based on data type (closely related to measure type), length (number of data points), and period (e.g., daily, biweekly, monthly data). Consequently, the AI must automatically identify type, length, and period, then correctly apply changepoint methods such as Strucchange, BCP, E.Agglo, and E.Divisive.

• Important: The AI must choose the best changepoint method, evaluate the performance, sensitivity, and specificity, and automatically reapply using another method if necessary.

Correct Configuration of Forecasting Methods

AI in healthcare must automatically and correctly apply forecasting methods. The correct forecast method is based on length (number of data points) and period (e.g., daily, biweekly, monthly data). Consequently, the AI must automatically identify the length and period of the data and then correctly apply forecasting methods such as exponential smoothing, Tbats, and regression.

• Important: The AI must choose the best forecasting method, evaluate the performance, sensitivity, and specificity, and automatically reapply using another method if possible.

Getting Started

The scope of work described above is significant. Regrettably, most data science teams won’t have the time required to build algorithms that automatically and correctly apply advanced statistical methods for time series data.

However, without advanced statistical techniques, analysts, department leaders, and even board members can do more harm than good when they incorrectly interpret time series data. Fortunately for healthcare leaders, Health Catalyst has already built the algorithms described above that automatically apply SPC, changepoint detection, and forecasting.

The Health Catalyst Healthcare.AI Solution

Health Catalyst built its Healthcare.AI product suite to automatically apply advanced statistical methods into analytic outputs and visualizations. In other words, the Healthcare.AI product suite provides augmented analytics for analysts and leaders.

The Healthcare.AI product suite automatically, correctly, and transparently (not a black box) applies advanced statistical and machine learning methods in common analytic tools such as Power BI, Qlik, Tableau, as well as analytics applications provided by Health Catalyst. The Healthcare.AI product suite applies advanced statistical methods such as SPC+, Forecast, Power, Time Series+, Forest Plot+, Risk Adjustment, and Peer Finding.

The Time Series+ capabilities in Healthcare.AI apply to time series data. The “+” in “Time Series+” refers to the addition of SPC (with changepoint detection) and forecasting. The next two versions of the line chart (from Figure 2, above) compare the visualization of sepsis mortality rates with and without the Time Series+ capabilities in the Healthcare.AI product suite.

Without the Time Series+ Capabilities in Healthcare.AI

With the Time Series+ Capabilities in Healthcare.AI

Notice in the line chart above that the Time Series+ capabilities in Healthcare.AI clearly identify the baseline, extreme points, persistent shift, and forecast:

• Baseline—the horizontal purple line clearly identifies the baseline performance as 11%.
• Extreme points—the red points clearly identify seven extreme points, meaning those outside the range of normal variation.
• Persistent shifts—the vertical yellow line clearly identifies that a persistent shift occurred in October 2020 when the average sepsis mortality rate increased from 11 to 16%.
• Forecast—the sequence of six purple dots clearly identifies the forecasted performance as 15% ​for six months into the future.

Using the Time Series+ capabilities in Healthcare.AI, analysts, leaders, and board members correctly identify baselines, extreme points, persistent shifts, and forecasted performance. Correct identification of those four points empowers leaders to accomplish the following: 

• Set attainable SMART goals that they specify correctly and measure accurately such as “improve sepsis mortality from 15 to 12% in one year” (baselines).
• Determine the need to improve sepsis care delivery because sepsis mortality rates are 13%, not 11% (baselines).
• Capitalize on the opportunity to prepare for ED surges that temporarily worsen the sepsis mortality rate (extreme points).
• Celebrate the success of continuous improvement efforts in sepsis care delivery because mortality rates improved from 15 to 12% and the performance shift was not part of normal variation (persistent shift).
• Confidently decide to focus improvement initiatives on other clinical quality measures that recently declined (persistent shift) because sepsis mortality, while not at the 11% target, is improving and will be at 11% within 3 months (forecast).

In addition to the decisions listed above, analysts and department leaders use the Healthcare.AI product suite to make many more data-driven decisions. That’s the value of developing AI for strategic use cases in addition to transactional use cases like readmission risk, deterioration, sepsis detection, and smoking cessation. Advanced statistical techniques and AI ensure leaders do not do more harm than good with their data and analytics.

Conclusion

The process of transforming data into measurable value has many potential points of failure. Interpreting common reports and analytics is just one potential point of failure. Because there are many points of failure in transforming data into measurable value, Health Catalyst provides solutions for interpreting analytic outputs as well as solutions that eliminate other points of failure across the complete healthcare data value chain. The healthcare data value chain illustrates the building blocks required to transform raw data into measurable value (Figure 3).

While this article focuses on analytics as a common point of failure in transforming data into measurable value, AI depends on good data. Consequently, Health Catalyst urges health systems to think upstream and downstream from the point of analytics in the data value chain:

• Upstream – data acquisition and data enhancement.
• Downstream – interventions and best practices as well as expertise and improvement.

Upstream, health systems must think about points of failure such as data governance, data correctness, and data completeness—significant points of failure in the process of transforming data into measurable value. Downstream, health systems must think about points of failure such as interventions and change management. As you work to transform more data into more measurable value, remember these points of failure. And remember that Health Catalyst is the only company exclusively focused on healthcare that provides solutions proven to integrate and optimize the complete data value chain.

Healthcare leaders committed to making their organizations data driven by optimizing the complete healthcare data value chain can start by taking the analytics interpretation online assessment to understand their capabilities around interpreting analytics outputs. They can also access more knowledge through Health Catalyst thought leadership on AI and the Healthcare.AI product page. Finally, healthcare leaders can contact a Health Catalyst representative to discuss how to optimize the healthcare data value chain to transform more data into more measurable value.

Additional Reading

Would you like to learn more about this topic? Here are some articles we suggest:

• Five Ways Healthcare AI Gives You Superpowers
• Expanding AI in Healthcare: Introducing the New Healthcare.AI™ by Health Catalyst
• Three Reasons Augmented Intelligence Is the Future of AI in Healthcare
• Artificial Intelligence and Machine Learning in Healthcare: Four Real-World Improvements
• Your Guide to Augmented Intelligence in Healthcare: Three Hows for AI Success