Adaptive AI algorithms, often referred to as “black box AI”, change as they learn, and it is difficult to explain how they are making their decisions. This presents a challenge for regulators all over the world, not only of medical devices, but in many other AI tools that carry risk. To protect patient safety, it will be necessary to understand if the algorithm has changed significantly since it was approved either due to the change in the actual algorithm logic, features or because of data drift (which could mean that the model is out-of-date in light of new data). If it has, both manufacturers and regulators need to know whether it remains safe and fit for purpose and if instructions on use need to be modified.
This project combines a clinical and regulatory stream with a data science stream. The clinical and regulatory stream, through workshops with experts, will develop a position on what is a significant change in an algorithm’s performance from a clinical and regulatory perspective, referring to real, disguised examples and simulated examples.
The data science stream will use Concept Drift Detection to identify changes in Predictive Clinical Models due to effects of updating with new data, and to detect when models have become out-of-date.
This will be based on using COVID-19 primary care data:
- It will adopt a moving window approach to simulate changes in data as more is collected over time.
- It will explore / develop “concept drift” metrics to score how well a model fits current data, how this has changed, and how performance is affected.
- It will explore methods to update models given new data and assess
- It will compare metrics / update methods from (2) and (3) on Deep Learning Models / Bayesian Models / Tree-Based Models
Over the next two years the regulatory framework for Software as a Medical Device (SaMD) and AI Software as a Medical Device (AIaMD) will be re-defined in the UK by MHRA. The framework, which is a Ministerial priority, will make best use of the most up-to-date thinking. A methodology to identify a significant change in an adaptive AI algorithm will be an important element of the framework which will bring AI software medical devices safely to market.