Objective: Develop a Implementation Guide (IG) that provides best practice guidance and a testing & evaluation (T&E) framework for evaluators to leverage when assessing the responsible use of predictive AI solutions in healthcare across the entire AI lifecycle.
T&E Framework
Metrics, Measures, and Methods per CHAI Principle & Considerations
Testing & Evaluation (T&E) Objective:
- Determine how to practically evaluate CHAI Assurance Standards (AS) Guide principles, for a sepsis use case, by defining metrics, measures, and methods across the AI Lifecycle
- Describe how to apply those metrics, measures, and methods using a real-world Predictive AI use case (sepsis)
- Usefulness
- Fairness, Equity, and Bias Management
- Transparency
- Safety
- Privacy and Security
BP Framework
Best Practice Guidance per CHAI Principle & Considerations
Best Practice (BP) Objective: Review a set of best practice guidance (e.g., business, data, and security requirements, etc.) within the context of a sepsis use case
- Business Requirements
- Data Requirements
- Technical Requirements
- Security Requirements
Co-Chairs
Sonya Makhni
Mayo Clinic Platform
Raj Ratwani
MEDSTAR
Geralyn Miller
Microsoft
Han-Chin Ching
AWS
Marzyeh Ghassemi
MIT
Initial Members
View All
MEMBERS
ORGANIZATIONS
| Michelle Un | CVS |
| Jessica Handley | Medstar |
| Bernard Bizzo | Mass Gen |
| Saad Alam | AdventHealth |
| Benjamin Rader |
Boston Children’s
|
| Jon McManus | Sharp |
| Ram Rimal | UNC Health |
| Melissa Davis | Yale |
| Girish Nadkarni | Mt. Sinai |
| Naveen Raja | UC Riverside |
| Rachel Tripp | Mayo |
| Chaitanya Shivade | Amazon |
| Olga Matlin | Amazon |
| Chris Lindsell | Duke |
| Keith Morse | Stanford |
| Amelia Sattler | Stanford |
| [email protected] | Unifi AI |
| Micah Seller | Intel |
| Sarah Gebauer |
RAND Corporation
|
| Niels Olson |
College of American Pathologists
|
| Eric Henry |
King and Spalding
|
| Gary Weissman | Penn Medicine |
| Walter Wiggins |
Greensboro Radiology
|
| Jon Ng | Iterative Health |
| Andy Beck | Path AI |
| Dereck Paul | Glass Health |
| Daniel Nadler | Open Evidence |
| Brett Moran | Parkland |
| Demetri Giannikopoulos | Aidoc |
| Kristyn Looney | IU Health |
| Laura Patton | Oracle |
| Tristan Naumann | Microsoft |
| Erik Duhaime | Centaur Labs |
| Kadija Ferryman | Johns Hopkins |
| Laks Meyyappan | CVS |
| Allan Fong | Medstar |
| Sayon Dutta | Mass Gen |
| Saad Alam | AdventHealth |
| William La Cava |
Boston Children’s
|
| Jon McManus | Sharp |
| Rachini Moosavi | UNC Health |
| Tristan Markwell | Providence |
| Daniella Meeker | Yale |
| Matthew Solomon |
Kaiser Permanente
|
| Arasha Kia | Mt. Sinai |
| Jason Adams | UC Davis |
| Ricardo Henao | Duke |
| Priya Desai | Stanford |
| Mary Richards |
Amputee Coalition
|
| Enes Hosgor | Gesund AI |
| Stephen Konya | ONC |
| Keith Morse | Stanford |
| Pramod Misra | Unifi AI |
| Colleen Houlahan | Cleveland Clinic |
| Atul Kanvinde | Shepard Center |
| Jansen Seheult |
College of American Pathologists
|
| Divya Chhabra | Dosis |
| Nita Farahany | Duke |
| Mozziyar Etemadi | Northwestern |
| Christos Davatzikos | Penn Medicine |
| Andrei Petrus | Lucem Health |
| Dom Pimenta | Tortus |
| Subbu Venkatraman | Eko Health |
| Chris Mansi | Viz AI |
| Jim Min | Cleerly |
| James Leo | MemorialCare |
| Elizabeth Johnson |
Montana State University
|
| Craig Norquist | Honor Health |
| Demetri Giannikopoulos | Aidoc |
| Laura Patton | Oracle |
Phases of Work
| Phase One | |
|---|---|
| Week of August 26th: Usefullnes, Usability & Efficacy | – Review metrics, measures, methods collected from WG members to date and obtain feedback, identify gaps, discuss details to include in Technical Implementation Guide (Use Case Overview and Testing Metrics/Measures) |
| Week of September 2nd: Fairness, Equity & Bias Management and Safety & Reliability | – Review feedback from previous meeting and incorporate into draft technical implementation guide
– Review metrics, measures, methods collected from WG members to date and obtain feedback, identify gaps, discuss details to |
| Week of September 9th: Transparency, Intelligibility & Accountability | – Review feedback from previous meeting and incorporate into draft technical implementation guide
– Review metrics, measures, methods collected from WG members to date and obtain feedback, identify gaps, discuss details to |
| Week of September 16th: Security & Privacy | – Review feedback from previous meeting and incorporate into draft technical implementation guide
– Review metrics, measures, methods collected from WG members to date and obtain feedback, identify gaps, discuss details to |
| Week of September 23rd: Align on drafted set of best practices and methods/metrics, and discuss next steps for HLTH Conference | – Work Group present findings, cross-pollinate, obtain feedback, identify gaps, ensure alignment |
| Week of September 30th: Review Period | – CHAI Board Member review |
| Week of October 7th: Review Period | -Work Group Member review |
| Week of October 14th: Final Work Group edits | – Prepare initial version of Technical Implementation Guide and HLTH presentation slides
– Present at HLTH Conference on October 19th |
| Phase Two | |
|---|---|
| November – December 2024 | – Continue above approach to complete first version of Technical Implementation Guide |
| Phase Three | |
|---|---|
| Spring 2025 | – Obtain feedback from CHAI Member community across Technical Implementation Guide
– Incorporate feedback and share with public |
