Predictive risk: to app or to automate?

Blog post

Published: 25/10/2012

The purpose of predictive risk modelling is to segment a given population on the basis of their risk of experiencing a particular outcome, for example an emergency hospital admission (Billings and others, 2006).

This is often used for case finding, where appropriate prevention techniques are matched to each risk stratum. High risk patients will be a small minority of the total population, and the form of the intervention will change depending on the level of risk.

The highest risk patients might receive a personal intervention, such as being assigned a community matron or admitted to a virtual ward. The high to medium risk segment might receive telephone follow-up.

At some point, the number of people in the segment will be so large and the individual risk so low that population-based prevention will be more appropriate (and cost effective) than personally-targeted interventions.

One of the reasons for this multi-level approach is that there are so few high risk patients that to focus on them alone significantly limits the progress that can be made in reducing emergency admissions overall – an issue raised in a recent British Medical Journal (BMJ) paper.

When using predictive risk there is a tendency to focus on the highest risk people because they have the most reliable predictions, and if your intervention costs hundreds of pounds per patient, you need to be confident you’re targeting the right people.

However, there are two ways of thinking about the accuracy of these models: how likely people predicted as high risk are to experience the outcome (known as the positive predictive value or PPV), and the proportion of all the people having the outcome that the model is able to identify (known as the sensitivity).

Sensitivity is often sacrificed to increase the PPV; that is, models are built to produce very reliable predictions for a small number of people.

We faced a similar issue when we developed the PARR-30 predictive risk algorithm to detect patients at risk of emergency readmission within 30 days of discharge (full findings available in a BMJ Open paper).

One thing we focused on was constructing a model that could be run quickly and easily at the bedside, possibly as an app for a smart phone or tablet computer.

When we developed the model it had an impressive PPV, which held its own against other international 30 day readmission models such as LACE (Kansagara and others, 2011), despite using fewer variables than most.

We tested the model with NHS trusts in two different ways. One used the model as a prototype app and the other implemented it as part of their hospital information system. We found that the app worked pretty well – the information it needed was easily available and doctors found it easy to use (completing the form in an average of 100 seconds per patient).

However, very few of their patients were categorised as high risk. And that made us wonder whether an app is an effective way to go for a predictive risk model, even if it has a strong PPV. No matter how quick the app is to use, the clinician will spend a lot of time filling it in for low risk patients (and little result).

In this context, building the model into the information systems and automatically alerting the hospital staff only when a patient is high risk seems a more sensible way of applying the model.

By doing this, doctors can easily be prompted to target the more intense interventions towards the highest risk patients (where there is greatest need, which justifies greater investment).

For now, medium risk patients need effective but less costly support, which recognises their lower likelihood of readmission and the extra ambiguity in targeting individuals.

For the future, the challenge for researchers is to develop predictive risk models that retain high PPVs and have increased sensitivity.

Higher sensitivity means more patients correctly identified as high risk, which can be converted into greater reductions in emergency admission and readmissions.

Suggested citation

Blunt I (2012) ‘Predictive risk: to app or to automate?’. Nuffield Trust comment, 25 October 2012. https://www.nuffieldtrust.org.uk/news-item/predictive-risk-to-app-or-to-automate

Comments