WHAT ARE QUALITY ASSURANCE LABS?

Quality Assurance Labs will rigorously evaluate AI models across three phases of the algorithm lifecycle:

• Pre-deployment
• Implementation
• Post-deployment and monitoring

These labs focus on ensuring that AI models meet high standards for accuracy, reliability, and safety before they are deployed in clinical settings. They provide an independent assessment to verify that AI tools function as intended and do not pose risks to patients. 

Quality assurance labs will play a critical role in building trust in AI technologies by providing transparency and accountability in their evaluation processes, so that AI users and beneficiaries will have more information about how this technology works in practice. 

The labs are envisioned as independent entities tasked with rigorously testing AI health tools to ensure their safety, effectiveness, and fairness before, during and after deployment in healthcare. These labs are not part of any government regulatory process and are purely led by the private sector.

CHAI will certify these quality assurance labs, and will be a key factor in ensuring the trustworthiness of labs evaluating AI models. This certification process will assess several critical areas. Firstly, it will confirm that labs have access to high-quality data that is complete, diverse, longitudinal, and protects user privacy. Second, it will ensure labs possess the necessary expertise and capabilities to evaluate models and the claims they make about their intended uses. The range of potential evaluations is vast, so the certification framework will encompass a comprehensive list of capabilities. Finally, CHAI certification will verify that labs can safeguard a model vendor’s intellectual property through methods like confidential computing.