Analytics Improves Abstraction Quality, Efficiency, and Inter-rater Reliability

THE CHALLENGE

Banner Health participates in The American College of Surgeons National Surgical Quality Improvement Program (NSQIP), engaging Health Catalysts Tech-Enabled Managed Services to collect data on more than 150 variables for thousands of patients. The organization uses the data to measure performance against national standards to improve processes and patient outcomes, making abstraction accuracy critical. However, confirming high inter-rater reliability (IRR) was challenging, as assessment methods and audit tools relied upon burdensome, error-prone, manual processes that weren’t scalable.

THE PROJECT

Banner Health and Health Catalyst established a goal to ensure IRR greater than 90 percent. Leveraging the Health Catalyst® Data Operating System (DOS™) platform, the organization implemented an abstraction IRR analytics application.

Leaders randomly select IRR cases from each registry, measure, or data element utilizing available cases for the entire sample population and assign the cases for review. Leaders can also select and assign specific cases. Each abstractor reviews the same case, entering results into the web-based application. IRR results are visualized in the analytics application and reviewed by the abstraction team and leader. The team discusses any differences, using the data to identify opportunities for improvement and engaging with the appropriate agency as needed to clarify standards.

THE RESULT

Banner Health and Health Catalyst now have a standard, scalable IRR process for NSQIP, and every registry, measure, or abstracted data element of interest. The organization improved IRR efficiency, effectively visualizing the IRR rate and number of abstracted and matched values. Banner Health trusts the abstracted data accurately represents performance and can confidently use the high-quality data for improvement.

• 93 percent inter-rater reliability for all registries—more than 23K abstracted variables.
• 100 percent of abstractors receive peer review and feedback through the IRR process.
• Scalable, efficient, accurate IRR process that can be applied to every registry.

“The IRR analytics application further increases our confidence in the high-quality data abstracted by Health Catalyst, enabling us to use the data for both reporting and improvement.”

Nirav Patel, MD, FACS, Medical Director of Surgical and Procedural Services, Banner Health

ADDITIONAL READING

• Clinical Chart Abstraction Improves Pre-Op Pulmonary Function Testing
• New Analytic Insights Improves Mortality O/E
• Tech-Enabled Managed Services: Smartsourcing Clinical Data Abstraction Reduces Costs and Improves Value
• Analytics Driven Improvement Efforts Reduces Surgical Supply Costs by $3.2M
• Analytics Accelerates COVID-19 Elective Surgery Restart