Risks and responsible use of AI in medicines information – Anuj Sunder

This presentation focused on the risks associated with AI use in medicines information (MI) services and the need for a structured, responsible use framework. It emphasised that while AI offers opportunities to support MI workflows, it should augment, not replace, professional judgement.

Real-world examples demonstrated limitations of AI tools, including inaccurate or inconsistent answers, hallucinated references which appear plausible, and reliance on weak evidence, all of which have potential to cause patient harm if not critically reviewed.

The speaker highlighted the importance of AI literacy, understanding how models work, and recognising key risks such as bias, hallucinations, and over-reliance. A practical framework for use stratification was presented, distinguishing between low-risk administrative tasks, cognitive augmentation, and high-risk clinical decision-making where AI use should be restricted.

Risk mitigation strategies included critical appraisal of outputs, verification against primary sources, maintaining human-in-the-loop oversight, and establishing governance, audit, and error-reporting processes to ensure patient safety and professional accountability. Anuj shared an overview of a MI AI training programme to educate MI teams and support induction plans for pharmacists.

The UKMi has published two position statements on use of AI in answering medicine related queries for healthcare professionals and specifically for medicines information staff. Ongoing UK research was also highlighted, evaluating whether large language models can safely support MI queries, with the aim of informing future national guidance.

Overall, the presentation stressed that safe AI adoption in MI requires clear governance, training, and continual evaluation to ensure patient safety and professional accountability.

AI utilisation in healthcare: a cancer pharmacy perspective – Bastiaan Buijtenhuijs

This presentation outlined the growing role of AI within cancer services, highlighting how increasing workload pressures (volume, complexity, scope, and operational responsibility) are driving the need for technological support in oncology pharmacy.

Examples of current AI applications were provided across diagnostics (e.g. improved cancer detection and triaging), and pharmacy practice, including aseptic preparation, workflow optimisation, and clinical decision support. In systemic anti-cancer therapy (SACT), AI was shown to support toxicity prediction (identifying high risk patients, preventing toxicity and enhancing safety checks), dose optimisation, and prescription verification, improving safety and enabling more personalised care.

A key focus was the development of AI-supported prescription risk stratification tools, enabling prioritisation and potential automation of routine verification tasks to reduce cognitive burden and optimise workforce deployment.

The presentation also highlighted national leadership activity, including the BOPA AI Cancer Advisory Group and BOPA LLM Taskforce which is working to guide safe and effective AI implementation across oncology pharmacy.

Overall, the session emphasised that AI is already being deployed in cancer care and has significant potential to enhance safety, efficiency, and workforce sustainability, provided implementation is supported by appropriate governance and multidisciplinary collaboration.

Ambient voice AI and the future of pharmacy practice – Dr Yasmin Karsan

This presentation explored the emerging use of ambient voice AI to transform clinical documentation and workflows in pharmacy practice. The technology uses natural language processing to passively capture conversations, generate structured clinical records, and reduce reliance on manual data entry.

Significant efficiency gains were highlighted, including reductions in documentation time and consultation length, alongside increased patient-facing time and improved quality and completeness of clinical records. Applications include medicines reconciliation, consultation documentation, discharge summaries, and audit/data capture.

However, the presentation emphasised important risks, including hallucination errors, data privacy concerns, compounding health inequity for under-represented populations with lower transcription accuracy, and clinician over-reliance on AI outputs. Robust governance, explicit patient consent, and continued clinical oversight were identified as essential.

Implementation considerations included phased adoption, integration with existing systems, and alignment with regulatory frameworks such as GDPR, DTAC, and MHRA guidance.

Overall, the session concluded that ambient voice AI offers substantial potential to improve efficiency, patient safety, and workforce experience, but must be deployed within strong governance frameworks, with pharmacists retaining full accountability for clinical outputs.

Discussion summary after morning session:

1. Strategic opportunity

• AVT and related AI tools were seen as potentially very useful, with benefits that were relatively easy for clinicians to recognise.
• Improved efficiency was seen as a big potential benefit, through reduced administrative burden, better support through documentation and saving of time for higher-value clinical work.
• Presenting AI as an enabler of better practice would be preferable to be adopting the technology for its own sake.

2. Evidence and use cases

• There is a limited pharmacy-specific evidence base, especially on medicines-related outcomes, medication safety and its impact on pharmacy consultations.
• Further research is needed on priority pharmacy use cases, including medicines histories, medication reviews and support for complex decision-making.
• To identify where AI could add meaningful value we need to look at real service problems.

3. Clinical quality and consultation impact

• Can AVT reliably capture medicines information when clinicians speak naturally rather than reading out prescribing details verbatim?
• Could variation in consultation style affect the quality and consistency of AI-generated outputs?
• Could use of AVT alter clinician behaviour, consultation dynamics and the quality of interaction with patients over time?
• What would the impact be of AI-supported summarisation on note quality, clinical reasoning and cognitive workload?

4. Safety, governance and assurance

• There were broad concerns about governance, including underlying data sources, model design, handling of patient-identifiable data, third-party processing and data hosting arrangements.
• More transparency is needed, including strong clinical safety controls and clearer assurances on how supplier products operate in practice.
• Current supplier approval routes and regulatory classifications must provide sufficient assurance for safe deployment.

5. Standardisation, workforce and professional practice

• If AI-generated documentation becomes overly standardised it could reduce nuance and create systematic omissions that would be difficult to detect.
• AI could assist the pharmacy workforce by taking on lower-value tasks, freeing staff for more complex clinical roles. However, the profession should actively shape adoption to avoid unintended and unwanted long-term effects on roles and practice.
• Professional messaging should focus on improving care, solving practical problems and raising standards.

6. Equity, regulation and next steps

• Will these technologies promote digital inclusion across diverse patient groups, including those with different accents, languages and knowledge of technology?
• Will existing regulations be sufficient to address equity, safety and performance in real-world settings?
• Suggested next steps include: stronger assurance requirements, further evaluation in pharmacy settings and creation of safe testing environments to explore practical use cases.

Unintended consequences of technology for pharmacy practice – Angela Burgin

This presentation explored the unintended impacts of digital systems—and emerging AI—on pharmacy practice, drawing on research into the shift from paper-based prescribing to electronic prescribing and medicines administration (EPMA) and electronic patient records (EPR). The speaker highlighted that while digital tools bring clear benefits, they have also introduced significant changes to ways of working, often occurring faster than the workforce, professional frameworks, and regulation can adapt.

Key unintended consequences identified included reduced physical presence of pharmacists in clinical settings, increased reliance on documentation rather than real-time collaboration, and diminished patient interaction and opportunities for teaching prescribers. These shifts have contributed to more isolated working practices and changes in communication within multidisciplinary teams.

The presentation also emphasised changes to professional role and identity, with pharmacists increasingly seen as system users or “processors of information,” rather than proactive clinical contributors. Technology-driven workflows, including dashboards and KPIs, were noted to risk narrowing how pharmacy value is defined and measured. These outcomes were explained as natural responses to the introduction of technology—particularly increased remote access, availability of data, and organisational drivers—rather than isolated issues.

Looking ahead to AI adoption, the speaker recommended a more intentional approach, including designing meaningful pharmacist touchpoints in care, strengthening digital and relational skills, and systematically identifying unintended consequences post-implementation. Greater clarity on the value and contribution of pharmacy practice was also highlighted as essential.

Overall, the presentation stressed the importance of learning from prior digital transformation to ensure AI is implemented in ways that protect patient safety, workforce wellbeing, and the professional contribution of pharmacy.

The AI Commission with a focus on governance and regulation – Dr Mani Hussain

Mani Hussain introduced the National Commission for AI, hosted by the MHRA and established in September 2025. The Commission brings together around 60–65 experts from AI, technology, regulation, healthcare, and industry to advise on a new regulatory framework for AI in healthcare, with the aim of ensuring technologies are safe, effective, and do not harm patients or the public.

He summarised findings from engagement sessions with over 30 professional bodies, including pharmacy representatives, exploring views on the use of AI in healthcare. Several key themes emerged:

• Quality, safety and effectiveness: Professionals stressed the need for real-world evidence and ongoing monitoring to ensure AI performs safely in actual care settings. Concerns were raised about bias in training data, representativeness, and the risk of hallucinations or misleading outputs.
• Clinical judgement and autonomy: There was broad support for AI as a decision-support tool, provided that professionals remain the final decision-makers. Concerns included the risk of deskilling, future use of more autonomous systems, and unresolved questions around liability.
• Patient-clinician relationship: AI tools used by patients may increase access to information and support more informed discussions, but they may also increase workload where professionals need to correct inaccurate information. Some benefits were noted, such as ambient voice technologies that reduce administrative burden.
• Workforce implications: Staff uncertainty and anxiety about the impact of AI on roles were highlighted. Concerns focused particularly on job security, changing skill requirements, and who is responsible when AI-supported decisions go wrong.
• Governance and safe deployment: Participants emphasised the need for transparency, multidisciplinary involvement in adoption decisions, and clear governance arrangements. Concerns were raised about “black box” decision-making.
• Post-market monitoring: There was support for stronger systems to monitor AI once deployed, potentially through more real-time digital feedback mechanisms.

Cross-cutting themes included automation bias, psychological safety, equity and fairness, and the challenge of fragmented development across the NHS.

Overall message: AI offers significant potential benefits for patients, professionals, and providers, but more work is needed to develop a regulatory framework that protects safety without stifling innovation. A report from the AI Commission is expected in June.

Discussion summary after afternoon session:

The group looked at opportunities and challenges of digital technology and artificial intelligence in pharmacy practice, including the impact of existing digital systems on clinical roles, the future role of prescribing pharmacists and how AI could support safer, more efficient and more patient-centred care. Concerns were raised around implementation, governance, trust and the practical support needed for adoption across pharmacy settings.

1. Impact of technology on clinical practice:

What are the unintended consequences of digital systems in hospital pharmacy? Pharmacist participation might be reduced in ward rounds and multidisciplinary teams. The foundation training year and the expansion of prescribing capability could provide opportunities to strengthen patient-facing and clinically integrated roles.

2. Prescribing capability and professional confidence:

Prescribing qualifications can significantly increase pharmacists’ confidence, autonomy and ability to make timely treatment changes in digital prescribing systems. At the same time, newly qualified prescribers will need appropriate supervision, governance and support in practice to develop safely.

3. Governance, data protection and cyber security:

Are healthcare providers over-reliant on large overseas technology providers? And will confidential patient data be properly handled? There is a clear need for GDPR compliance, secure data processing, clear assurances on the intended use of data and robust safeguards for AI systems, including protection against misuse or malicious prompting.

4. Implementation and service redesign:

Technology by itself does not deliver meaningful transformation. There are various examples of good local practice but the wider impact has been limited because implementation too often focuses on introducing systems rather than redesigning workflows with the participation and cooperation of those who do the work.

5. AI, automation and risk appetite:

Pharmacy may be more risk-averse than other specialities in adopting AI and automation. An example from dermatology (skin cancer detection) suggests that, where evidence is strong and safeguards are in place, autonomous or semi-autonomous tools may help manage demand, reduce backlog and release clinicians to focus on patients with more complex needs.

6. Practical use cases and value:

AI can solve real problems in pharmacy practice, especially repetitive and administrative tasks that reduce time for patient care. AI can help in areas such as medication reviews, information gathering and clinical note preparation – provided evidence demonstrates it can save time , reduce workloads, is cost effective and/or leads to real quality improvements.

Future adoption of AI in pharmacy should be problem-led, evidence-based and tailored to setting. Hospital pharmacy, community pharmacy and other sectors face different challenges, and future work should reflect those differences more explicitly. Adoption of AI should support professional judgement and patient-centred care, rather than simply introducing new technology for its own sake.

There are several areas where the Royal College of Pharmacy could provide practical support:

• Sharing evidence and case studies (with both positive and negative outcomes.
• Helping to build AI literacy through continuing professional development and education.
• Convening a community of practice for those applying AI in pharmacy.
• Providing practical guidance on how to assess tools, understand risk, and make informed decisions about adoption.
• Gathering views, including public perspectives, to build trust and confidence in the responsible use of AI.

There was cautious optimism about the role of AI and digital technology in pharmacy. Although there are clear opportunities to improve efficiency, release professional time and strengthen patient care, this will depend on strong governance, trustworthy evidence, thoughtful implementation and practical support for the workforce.

The next phase of work must be more practical, setting-specific and focused on the real challenges faced in practice.

Closing remarks – Darren Powell

AI adoption in pharmacy and healthcare is already well under way, but progress remains uneven and concentrated mainly in administrative rather than clinical use. The discussion highlighted growing interest and significant activity across the sector, but also pointed to persistent barriers, including limited confidence, variable capability, gaps in training, and uncertainty around governance and accountability. Participants noted that while examples of AI training and implementation exist, these are not yet consistent across the profession, underscoring the need for a more coordinated approach.

A central theme was the importance of shaping AI as a support to, rather than a replacement for, professional judgement and patient care. Attendees discussed AI’s potential to act as a partner to clinicians through functions such as workflow support, safety checks, and enhanced decision-making, while also recognising emerging risks including automation bias, new accountability challenges, and the need for effective mitigation and post-market surveillance. There was particular concern that current efficiency gains are not yet clearly translating into improved patient outcomes, and that AI may either strengthen or weaken the patient-clinician relationship depending on how it is implemented.

The discussion concluded with a clear message that the profession should actively design its AI future rather than drift into it. Participants emphasised the value of bringing together emerging position statements and learning across organisations to avoid duplication and build shared understanding. The next steps will include two further workshops, a short report from each session, and a final review of where the key gaps lie, what the Royal College of Pharmacy can do to support the profession, and how it can work with wider stakeholders across the pharmacy landscape.

In attendance:

Name	Title	Organisation
Aditya Aggarwal	Portfolio Pharmacist, Policy and Healthtech advisor	NHS Pharmacy
Amna Khan Patel	CPhO Clinical Fellow	NHS England
Anika Puri	Clinical Fellow	Care Quality Commission
Angela Burgin	Advanced Clinical Pharmacist for Digital Medicines I Clinical Safety Officer	Leeds Teaching Hospitals NHS Trust
Anuj Sunder	Vice Chair UKMi Tech and Innovation Group	UK Medicines Information
Aurora Todisco	Patient representative	Coalition for Personalised Care
Bastiaan Buijtenhuijs	Chair, AI group & Head of Product at iQ HealthTech	British Oncology Pharmacy Association
Dan Al-Thion	Community Pharmacy IT Policy Manager	Community Pharmacy England
Darren Powell	Chair	RCPharm Digital Pharmacy Expert Advisory Group
Fiona McIntyre	Policy and Practice Lead, Scotland	Royal College of Pharmacy
Heidi Wright	Practice and Policy Lead, England	Royal College of Pharmacy
Helena Young	Senior learning and development pharmacist	Centre for Postgraduate Pharmacy Education
John Freestone	Co-founder	Digipharma
Manir Hussain	Deputy Director – Provider Assurance	National Commission into the Regulation of AI in Healthcare | MHRA
Marina Khan	Clinical Fellow	NHS England
Paula Higginson	Head of learning development	Centre for Postgraduate Pharmacy Education
Rahul Singal	Chief Pharmacy and Medicines Information Officer	NHS England
Raymond McGregor	Lead Pharmacist – Pre-Assessment Clinics & Orthopaedics	NHS Golden Jubilee, Scotland
Roz Gittins	Chief Pharmacy Officer and Deputy Registrar	General Pharmaceutical Council
Shalini Gujral	Group Chief Pharmacist	Phoenix
Sheetal Ladva	CPhO Clinical Fellow	Royal College of Pharmacy/MacMillan Cancer Support
Tariq Muhammed	Chief Executive Officer	Titan PMR Ltd
Vicky Damani	Clinical Product Lead	First databank
Yasmin Karson	Pharmacy AI Consultant	Digital Clinical Safety Karsons Pharmacy