Mixed-method evaluation of implementing artificial intelligence in chest diagnostics for lung disease: Phase 1 findings

Previous research has suggested that using AI to support diagnoses of lung diseases and cancer using x-ray or CT scans could improve speed and accuracy of diagnosis. There are also claims it could reduce pressure on teams and reduce overall health care costs. Yet there is very limited evidence based on real-world use of AI in this context and on its effects on staff and patient experience, clinical effectiveness and costs.

NHS England launched its Artificial Intelligence Diagnostic Fund (AIDF) in 2023 to pilot the use of AI in chest diagnostics. This evaluation, which forms a part of the AIDF programme, aims to:

• Evaluate the deployment, procurement, and early implementation of AI for chest diagnostics (x-ray or CT)
• Understand what factors influence the above processes
• Identify settings and data sources for future longer-term evaluations. 

Read the slide set for the full detail - please note that these findings have not yet been peer reviewed. 

Read the slide-set summary

Read the review paper

Read the blog

Phase 1 lessons are summarised below. Phase 2 is now underway - read more here.

Emerging lessons

Overall lessons for implementing AI

1. Plan in sufficient time to implement AI pathways, allowing for procurement and preparation processes, or otherwise build in capacity (e.g. staffing, time) to enable the required procurement and preparation steps to be expedited and completed rapidly. There may be tradeoffs with time if services are adapting to use of AI and/or laying foundation for future AI implementation and sustainability.
2. Engage with key stakeholders early at a range of levels (national, network, trust, IT, supplier, evaluator) to support procurement, preparation and implementation Ensure clear communication and collaboration processes.
3. The services that support procurement, preparation and implementation need sufficient resourcing for dedicated project management, clinical time and evaluation capacity.
4. Network-level support is key in supporting trusts to implement these tools (e.g. with project management, keeping momentum and facilitating shared learning across networks).
5. Gaps in representation of the patient voice, communication with patients, patient experience and issues of equality, diversity and inclusion must be addressed and considered when developing and implementing AI services.

Procurement

1. Dedicated expertise is needed within teams to coordinate the process of procurement.
2. Communities of practice (facilitated by national teams) can support local teams to progress with procurement processes by sharing learning.
3. AI suppliers can be used to showcase the functionality of AI tools during the procurement process, to avoid uncertainty in clinical teams over exactly what AI can achieve.
4. While NHS trusts have autonomy in procuring services, there may be value in considering national procurement of tools. This may free up network capacity to facilitate local deployment.

Preparation for deployment of AI 

1. Governance processes took time and were challenging for networks and trusts. Therefore, thinking about whether further support can be provided to complete governance processes, or whether governance processes can be simplified, may be helpful. Some possible solutions include:
   • National level provision of expert informed guidance on who is responsible, how to implement AI, and how to mitigate challenges
   • Having clear points of contact
   • Having clear timelines
   • Specific AI resource templates
   • Having support from suppliers (e.g. involvement in planning meetings etc).

Staff experience

1. In local services, accessible clinical champions are needed to drive AI forward, support staff and mitigate staff worries/concerns
2. Education and knowledge gaps need to be addressed by providing sufficient training and education to staff using AI
3. Open discussions and early engagement with on the ground staff to understand and reduce any concerns or worries.

Monitoring and evaluation

Gaps in evidence must be addressed to better inform the developing of services, e.g.

• Evaluations of real-world pathways of AI implementation at various stages.
• Better data infrastructure and capacity within local and national teams (data collection and staff).
• Consideration of evaluation plans more broadly.
   

Further research is required to evaluate AI in real-world settings. The second phase of this RSET evaluation will look at the use of AI tools in radiology (specifically chest diagnostics), considering:

• The implementation, adaptation and sustainability of AI tools
• The real-world impact of AI on care delivery, effectiveness, and cost-effectiveness of care
• Patient and carer experiences of receiving care facilitated by AI chest tools