AI/ML Bias

Date: October 14, 2021

Time: 21:00 - 22:00 UTC

Meeting Chairs: Yann Joly, Edward Dove

Objective: to conduct a cross work stream discussion on AI/Bias and discuss the opportunities here for GA4GH standards, including identifying key, specific areas to address

Minutes:

Welcome, Introduction

• Introduction of Tina Hernandez-Boussard
• AI and Bias – focus on AI models evaluation, how we can make suggestions as an organization and what are issues with models evaluation
• Can discuss how we can move this forward in GA4GH

The current landscape of AI/ML Bias - Tina

• Where is state of art today, where can we take this, where can we go
• AI applications in health care: in biomed research, translational research, medical practice
• We’ve seen a lot of information that algorithms we develop are biased and don’t perform fairly across populations

• Examples: heart failure and male symptoms – AI tools deployed in triage – but female and male heart system very different, used male in training data

• Very accurate for males but failed to identify women with symptoms
• Data used to train models biased so models don’t perform appropriately across systems

• AI determined dermatology: to identify cancer lesions on skin

• Algorithms work well on patients with fair skin, not on those with darker skin for identifying lesions and lesion location
• AI driven algorithms in health care system biased, speak to trust of patients in system

• Similar with COVID-19 and Data technology

• Disproportionate effect on minorities
• Flurry of AI tools published in literature, very poorly designed, based on small set of data, many articles retracted, ethical concerns
• Promoted issue of: Can we trust AI models being deployed in health care

• Genomic databases

• Can look at databases across research community, racial bias in sample collection – these samples are center for precision medicine
• Precision medicine relies on labeling variants as pathogenic, relying on databases, but databases heavily reflect European ancestry (81%)
• Missing population specific pathogenic information
• Research more likely to use data from European ancestry and genomic test results may be more challenging or unattainable

AI lifecycle

1. Data acquisition - data de-identification process used in model development
2. Models developed and trained on data they receive – whole idea is model optimization
3. Model evaluation – generally has focused on model performance – don’t see a lot of work and standard on model evaluation across populations
4. Model deployment – even less data to understand biases in AI systems
5. Data creation

Where are biases in AI lifecycle?

• Data acquisition: Data enter system at the beginning, acquired from specific populations or modalities – breaks down trust

• Gaps in data modalities data representation, secondary use. Risk and liability of re-using the data can lead to lots of misunderstanding

• Model Development: Models learn from data we give them – if we give biased data they will perpetuate it
• Model deployment – bias in where model deployed and what population benefits
• Model evaluation - we don’t see many standards for model evaluation across populations
• Data created by cycle often skewed and unbalanced, leads to participation bias and trust

Mitigating Biases

• Need understanding of importance of biases and how to mitigate them
• Need broad stakeholder engagement from the beginning of question being designed, data acquirement, and how we think about model deployment, and how we share info with community

Important types of biases in data

• Historical bias in our healthcare system
• Representation bias - not all populations are reprsented
• Measurement – sometimes don’t have actual variable we want to measure, use proxies
• Aggregation - group certain populations together due to lack of data for one population

Model evaluations

• Where we see most work – most literature focused on performance evaluation
• Look at class imbalance
• Algorithmic fairness – methods allow us to evaluate performance of algorithm across populations such as anti-classification, classification parity, and calibration
• Group performance
• Need to make sure models actually ensure health outcomes across all populations

Building trust through transparency

• Report of population from which data acquired
• Model design and development
• Model evaluation and validation

• If we develop a model working well in one population, is it transferable to other populations?

Discussion: Opportunities for AI/ML Bias Standards in GA4GH

Q/ Engagement: what might be useful and feasible models for stakeholder engagement in AI/ML in genomics? AI development teams, community groups impacted?

THB: People are identifying that without diversity, models are not moving forward or being applied to right populations

Opportunity for GA4GH to determine what are right stakeholders to be at table

Stakeholders all across sectors: computer scientists, legal, bioethicists, hospital admin etc. – spectrum is broad

YJ: Community increasingly aware of problem of lack of diversity, but struggling with what to do – this is where we need to work the most on finding solutions to tackle from policy perspectives

SK: We are only seeing certain groups reap the benefits and if there is more focus at the start of research development will see change

AL: So much attention on diversity – how will GA4GH uniquely contribute

Almost all model performance depends on data

Is there stuff around meta data where GA4GH could have impact? We need more focus on other types of diversity in the datasets.

YJ:Canada recently had report on data reporting in COVID-19 where Canada scored poorly – sometimes data is so sensitive to take we don’t take it at all

THB: Meta data is very important to understand how to use it, secondary use

What is minimum set of info we need to capture around datasets to ensure proper use of data and aware of potential biases – just because biased doesn’t mean we can’t use it – we need to properly report it and be transparent of potential biases

Q/ Does it make sense to have model evaluation both before and post deployment of model?

THB: Yes

Important to continue to evaluate once deployed because data change

i.e. COVID19 – if we deploy a model how long will it be accurate with variant and treatments changes – need to make sure they are still reaching level of performance intended to, determine when we need to retrain data

Q/ How best can we develop needed capacity in AI/Bias/ethics in Genomics at diff levels – what is the basic standard principles we should be bound to to check we are on the right track and to be as transparent and impactful as ethically possible

THB: No current standard in this area – there are basic agreements but is wild west

Opportunity for community to think of standards to move forward with

List of minimum standards must come from community

Want standard on representations of data, operation of models, deployment, post-deployment evaluations, how to acquire data, what info to share

YJ & AL: We need to think of levels of data to be collected – whole dataset levels and whole human levels

Q/ How can GA can best address these issues – would an assessment tool for people to base AI tools on be helpful?

TD: What might be some useful and feasible models for stakeholder engagement in AI/ML in genomics? AI development teams surveying individual and community groups who might be impacted by the (automated) decision/algorithm? Ad hoc stakeholder groups? Is there sufficient nudge from industry or regulators (or others) to make this happen?

T H-B: stakeholder engagement is a key component to move convo forward. Without it, the models are not moving forward or working or useful.  Great opportunity for GA4GH to think about who needs to be at the table. Talking about stakeholders across sectors - legal, bioethivists, end users, hospital administrators, computer scientists. Spectrum is very broad

Yann/Lewis: May want to start from concrete project/use case to work with

If had case could look at details to understand and point out why this is important

AL: how can GA4GH uniquely contribute here. Is there stuff around with the metadata where GA4GH can have an impact on.

T H-B: metadata is so important - what was collected, why, what was the setting. So important to understand how we can use it, and for secondary use. What is the minimum set of info we need to capture around these datasets for appropriate use and reuse of the data? Just because it’s biased doesn’t mean we can’t us it. We need to properly report the data and what biases there are, and we cannot do it without metadata

YP: Does it makes sense to have model evaluation both before and post deployment of the model?

T H-B: really important to continually evaluate the model because data changes. We can develop an AI model for COVID. But it may not be accurate the next day. Things change rapidly and we need to ensure they are still reaching a high level of performance. Need to know when you need to re-train the data, get more data, evaluate the model with new drugs, treatment, etc. Need to focus on the post-deployment of data. That’s where we identify societal harms, where performance is not accurate across populations

JK: How best we develop the necessary capacity in AI/ Bias/Ethics in Genomics at different level?

T H-B: what are the standards we want to move forward with and how to hold people up to those standards? But it has to come from the community. Community effort for it to be accepted and adopted. Standards around representation of the data. Evaluation of the model. How and when it should be deployed. And Post-deployment evaluation. How you’re going to acquire the data. What info needs to be shared with people giving the samples? We need to identify what these standards should be

AL: Two levels of more non-genetic data are in play: at the whole dataset level (as Tina was mentioning) and at the whole human level (as Yann was mentioning). More of both needed, and careful thinking about appropriate aggregation

M R-A: How do you think GA4GH can best address all these issues? Do you think an assessment tool on addressing the level of bias at these levels would be a good idea? How can we narrow this scope to be useful in a GA4GH tool?

AL: Maili, would be helpful to have a concrete project to work with. There are then umpteen ethics frameworks one can attempt to apply. This could reveal gaps ga4gh could fill.

T H-B: Exactly - we need to understand does this AI model make a difference - what is the impact to the patient, population, etc..

Questions for discussion presented by THB

What are liabilities of data sharing?

SK: Interesting in terms of minority communities

Biggest issue when data shared, incentives unethical to promote sharing, and research from data doesn’t often come back to them – AI involves huge environment and doesn’t come back to communities

How do you really ensure people giving their data reaping their benefits?

YJ: main EDI group has also pointed that out as topic to follow up on – on GA radar

SK: When thinking of type of data received from people – also important to record data on types of environments healthcare is going from – related to metadata – environmental data increasingly important

THB: Health care setting represents 10% of health of individuals, mainly environment and social factors – so capturing genomic information essential, but also need to think of social and behavioural characteristics that contribute to well-being – thinking of how to incorporate metadata into models

Transparency and evaluation standards: what at minimum standards to be proposing – what minimal info need to go with reporting of AI models?

Population, evaluation characteristics, also need to know actual question AI model designed to answer, and who is intended user? In which population are they most appropriately applicable

YJ:In genomics paying more attention to metadata- how to you see doing this in context for large scale genomics and health research differ from AI models?

THB: A lot to add – when we think of samples and information as a whole is not representative of all populations – when we use information need to understand population it comes from and where we will apply it – genomics far ahead of clinical AI research – need to think of how to move to next level of ensuring what we identify in data sets are applicable across populations or to which populations

SK: Evaluation – when we do evaluations there is a failure to understand that the data we’re comparing against is likely biased – we assume human generated data always correct and what we should be aiming for – there isn’t concrete way forward to correct for bias

Other thoughts:

THB: We assume that human made labels are appropriate labels to use (i.e. race) – but we now have people with genome testing who have genetic history – is there an opportunity to look at that and look at how we’re classifying into racial categories – is it appropriate to use these to guide us?

ED: We could start with survey of GA4GH members to see what AI using or developing – see if these could be used as case study

Alice: Work on social studies on racial identity after genetic tests, work on self-identified measures of race vs self-identified ancestry – get more detailed info with latter

Biggest issue is ancestry is a proxy – what ancestry is is a reference point where we cut off what percentage on average you share with reference panel for predesignated group – what does it tell you other than ancestry - reinforces racialized thinking

Don’t have reference panel from parts of the world that have never been sampled

Genetic ancestry just as problematic, if not more because idea of subjectivity masks the bias involved

THB: Think about are there standards we need to push forward regarding data collection – this something for GA4GH to think about

What are the minimum standards?

NEXT STEPS:

• Surveying GA4GH community to discover what areas of AI/ML need addressed etc.
• Form an across workstream groups of interested parties to address this topic as part of GA4GH roadmap – need a variety expertise to work on this question

• Anyone interested in workgroup send an email to REWS group email address (rews-coordinator@ga4gh.org) – would require meeting on maybe monthly basis and would mostly involve getting your thoughts on how we can take these questions forward