Many healthcare systems across the world are making use of digital patient records and other routinely collected administrative data to monitor, regulate and, ultimately, improve hospital quality and safety.

However, organisations seeking to exploit such ‘big data’ are encountering persistent difficulties in making their systems work. As researchers from the Department of Geography have shown, the effectiveness of big data is all in the way you collect and use it.

The faulty smoke alarm

The Care Quality Commission (CQC), the independent regulator of all health and social care services in England, has become a global leader in developing statistical surveillance systems that use big data to detect substandard healthcare quality.

Between 2005 and 2009, hundreds of patients at the Mid-Staffordshire NHS Foundation Trust died needlessly and many more suffered violations of their dignity. A public inquiry into the scandal in 2013 concluded that it was essential that the CQC conduct “risk-related monitoring” of healthcare providers “in as near real-time possible."

In response, the CQC created a big data monitoring system – known as Intelligent Monitoring – to calculate a numerical risk score (aka a ‘smoke alarm’) for poor or declining quality of care in hospitals. Based on the predictions of this system, the CQC then sent out on-site inspection teams to the locations thought mostly likely to be delivering poor care.

However, researchers at King’s Department of Geography discovered that the CQC’s Intelligent Monitoring system could not predict the subsequent findings of the CQC’s on-site inspection teams. In fact, the system’s predictions were incorrect more often than not. This meant that the inspection teams were not being sent where they were most needed.

The research team attributed the failure to a series of challenges that had beset attempts by the CQC to identify the indicators needed to predict which hospitals were most likely to provide poor quality healthcare. These challenges included a lack of professional, political and wider societal agreement on the meaning of ‘healthcare quality’. It also included a difficulty in measuring ‘risks’ to quality in real-time, with such a vast range of healthcare services – provided across multiple locations and within each NHS trust.

Shaping understanding and demonstrating international relevance

Influenced by King’s research, the CQC has since redesigned the way it uses big data to detect poor quality hospital care. Their new statistical surveillance system, called Insight, addresses the problems originally identified in the previous system, helping to better direct the on-site inspection teams nationwide.

Subsequent research by the team has demonstrated the relevance of their findings for many international healthcare regulators using similar statistical surveillance systems.

The research submitted by Professor Demeritt and Professor Rothstein shows that fundamental differences in the way healthcare quality indicators are constructed, measured and used could be impeding international efforts to benchmark quality and identify best practice.