Medical literature monitoring (MLM) for adverse drug reactions is a critical component of the pharmacovigilance process and a regulatory requirement for all medicinal products on the market.
Currently, MLM is a high-volume task that demands considerable time from pharmacovigilance specialists. Despite the extensive effort, only a small percentage of screened articles lead to valid individual case safety reports (ICSRs) relevant to authorization holders. Moreover, the amount of real-world data that companies must review is rapidly increasing, with literature output growing by approximately 8% annually.
Leveraging AI in Medical Literature Screening
These challenges highlight the potential for automating workflows through artificial intelligence (AI) and natural language processing (NLP) technologies.
For instance, AI can prioritize retrieved articles, ensuring the most critical and potentially serious ones are reviewed first. NLP techniques can identify relevant sections of text, helping to highlight important details such as medication or patient mentions, thus saving reviewer time.
However, the true potential of AI extends beyond these initial applications. Given the growing volume of scientific literature, it is crucial to rethink how AI is used in MLM to enable companies to search more comprehensively, identify events more quickly, and allow pharmacovigilance specialists to focus on higher-value activities.
Reframing AI's Role in MLM
The question arises: Can we harness recent advances in machine learning and NLP to effectively filter out irrelevant articles without compromising result quality?
Using AI as a filter means moving beyond incremental improvements that still require specialists to review each incoming article. The goal is to decouple human effort from the sheer volume of articles, considering that the rate of ICSRs from search hits is very low.
The diagram below illustrates a simplified MLM process with an AI filter to eliminate irrelevant articles early in the workflow, before they are reviewed by a specialist:
AI as a Pre-Screening Filter
Our aim is to reduce the volume of incoming abstracts before they undergo the first review by a specialist. This approach raises several design questions for our AI system:
1. Incomplete Information in Abstracts: Adverse reactions might be implied and only fully described in the full article.
2. Causality Determination: Establishing a link between an event and a specific product often requires the full article.
Despite these limitations, it is feasible to filter out irrelevant articles that show no evidence of an adverse event. This is the primary objective of a specialist performing abstract screening, which AI models can emulate. We aim to train models to detect suspected adverse events, defined as:
- Explicit or implied adverse drug reactions in the abstract or title.
- Suspected adverse events linked to any drug mentioned in the abstract.
An article flagged as a suspected adverse event might not ultimately be an ICSR for a particular product. However, this approach minimizes the risk of missing real events and enables the development of robust, drug-agnostic models. This methodology is used to train models for the Datacred's Crypta AI solution.
Training and Evaluation
With the assistance of pharmacovigilance experts, we prepared a dataset of abstracts from biomedical literature covering a wide range of products and created labeling guidelines based on the criteria above. We used this dataset to train a deep learning model, leveraging SciSpacy pre-trained embeddings, which utilize biomedical text as their source.
For evaluation, an independent, gold-standard dataset of screening results would be ideal, especially if it includes multiple products. Fortunately, the EMA conducts literature screening for several products and provides screening results. We are grateful to the EMA for sharing a dataset for our evaluation.
Once trained, our models predicted suspected adverse events on the EMA dataset, excluding any examples present in our training set to avoid contamination. This experiment is detailed in our recently published study in the DIA Regulatory Science Forum.
Simulating Safe Threshold Levels
In literature screening, it's crucial to avoid missing potential events. The AI models should retrieve nearly all articles that could be ICSR candidates while filtering out irrelevant ones. To achieve this, we simulated outcomes to evaluate the volume of irrelevant articles filtered for a target recall of 95%. Different recall values can be adjusted based on specific MLM process needs.
We calibrated our model using historical data to issue predictions for the desired recall. The chart below shows the trend of percentage articles saved versus desired recall for each month tested.
Results indicated that calibrating prediction thresholds to maintain a target recall was effective, achieving savings of over 40% (up to 49%).
Next Steps
Our benchmark AI model, trained using the suspect adverse event approach, has shown promising results, filtering over 40% of inbound articles while maintaining 95% recall. We continuously refine our models and data labeling processes based on experimental feedback. As our labeled data grows and AI models are further optimized for this task, we anticipate further improvements in efficiency and result quality.
We recognize the importance of safe and trustworthy AI in drug safety and continue to adapt our approach in line with emerging regulatory guidance for pharmacovigilance.
About Crypta
Crypta is an all-in-one literature monitoring solution designed for pharmacovigilance teams. Its adaptable workflow, integrated scientific database, and innovative AI productivity features provide quick, cost-effective, and fully traceable results for any screening requirements.
Interested to know more about Crypta, book a demo
Comments