How has Covid-19 changed the evaluation landscape?

The Covid-19 pandemic has led to significant changes in the way that health care services are delivered – and these changes have been delivered at a remarkable pace. Many services, for instance, have of necessity used digital alternatives to face-to-face consultations. Such changes may bring benefits, but also costs. Evaluating the effects, the cost effectiveness and patient and staff experience that keeps pace with these changes will be vital.

Rapid evaluations are often adopted in health care services as a quicker alternative to full-scale research evaluations that can often take years to complete. These faster assessments are crucial for ensuring that ongoing changes to health care services are supported by evidence, as well as allowing timely decision-making for policy and practice. The pace of service changes during the pandemic, and the huge desire for quicker learning, have highlighted the need for these rapid evaluations, with evaluators coming under increasing pressure to produce timely findings to support decisions.

But Covid-19 has precipitated some significant changes in the research landscape – affecting not only demand for evaluations but also the supply.

A different world

So how can we possibly make sense of this ever-changing landscape? The ‘Rapid evaluation in health care conference’ will this year be taking place virtually from tomorrow (Tuesday 26^th) until Thursday (28^th). The conference, run by the Nuffield Trust and the Health Foundation in collaboration with rapid evaluation groups NIHR RSET and NIHR BRACE, brings together analysts, service users, commissioners of evaluations and evaluators in health care to consider how Covid has affected the evaluation landscape, to explore what good practice in a rapid, rigorous evaluation looks like, and to share lessons learned from recent and ongoing evaluations.

And there are a huge number of issues to discuss. Covid-related research has understandably been prioritised by commissioners and evaluators. This has created a crowded field, with opportunity costs, with many non-Covid evaluations being paused or stopped altogether.

Balancing priorities

The pandemic also poses significant practical challenges to conducting evaluations. Protecting the time and capacity of already overwhelmed and exhausted NHS and care staff should be a vital priority when planning and designing evaluations.

There is also a greater need for evaluators to consider what might realistically be achieved in a given timescale. This includes decisions about prioritising specific aspects of an evaluation, such as focusing on routinely available data or narrowing the scope of work, as well as the implications of data collection – such as the relative costs and benefits of collecting qualitative and quantitative data quickly.

Given the demand for rapid learning, there is a risk that this will limit opportunities for evaluators to engage with patients and the public. It is therefore crucial that evaluators reflect on ways to continue to collaborate with users and patients, so that their views continue to help shape the research.

Speedy assessments have benefits, but costs can include forgoing some aspects of the quality of evaluations. So what trade-off are we happy to accept between the quality of a study and the need for timely evidence?

Change compounding change

It is essential that commissioners and evaluators seek to fully understand how the pandemic has changed the evaluation landscape. And the research community must work together to share strategies and best practices for how evaluations can be better designed and conducted, to ensure that the rapid changes in health and care brought about by Covid are evidenced, and decisions relating to policy and practice are driven by high-quality, timely evidence.

Despite all this, there will be life after the pandemic. Commissioners of evaluations and researchers will need to turn their thoughts to other areas that have been deprioritised and neglected during the crisis. There will of course be many things we have learned during Covid-19 that we can apply in the future. This week’s conference is the ideal forum for driving this agenda forward.

Join us each morning from Tuesday to Thursday (January 26-28) for the third annual rapid evaluation in health care conference, which is a virtual event organised by the Nuffield Trust and the Health Foundation in collaboration with NIHR RSET and NIHR BRACE. To register and for more details on the programme, please visit this page.