AGENDA

REWS Diversity in Datasets Meeting 4
8th September 2022 @ 00:00 UTC, 01:00 BST OR 7th September 2022 @ 8pm EST, 5pm PT, 10am AEST, 8am Hong Kong

(dial in numbers below)

Zoom Recording [password: 14wGDwk%]

Zoom Details:

Join Zoom Meeting

https://us02web.zoom.us/j/85634858979?pwd=NlNDMHdhTExCZFJDeWxubll6ZEtqQT09

Meeting ID: 856 3485 8979

Passcode: 611022

Attendees

1. Tina Hernandez-Boussard
2. Anna Lewis
3. Gabriela Chavarría
4. Henriette Raventós
5. Hanshi Liu
6. Lindsay Smith
7. Beatrice Kaiser
8. Bimal Chaudhari
9. Norbert Tavares
10. Ainsley Newson
11. Donrich Thalder

Minutes

AL: start by going through the background document. Focusing on items a-e:

1. Calls for diversity in genomic datasets are frequent and well motivated,
2. But that what diversity in genomic datasets actually entails is underspecified,
3. It is not hard to understand why the field has a lack of diverse data,
4. Genomics should learn from other fields who are also grappling with diversity in datasets,
5. But there are nonetheless some special features of genomic data that merit special consideration in this context
6. The question of diversity in genomic datasets has some international components, particularly because of the ways genomic data is shared across borders.

Can see the mission statement of GA4GH - “a policy-framing and technical standards-setting organization, seeking to enable responsible genomic data sharing within a human rights framework.” - that’s the core of it, very relevant for this policy, this data sharing across borders.

Any comments or questions about this background doc. Does it capture what we’ve discussed so far and do a good job of setting up the background. Important to point out that it is underspecified - their field assumes we know exactly what it is we are trying to achieve

THB: agree, we also use different terms depending on the hat we’re wearing, important to specify key terms and say how we’re defining these - depending on ethics genomics etc have different meanings

AL: not just giving definition of a term, but saying how it gets operationalized in different topics - need to say what type of diversity is actually needed.

Bimal made a great point that underrepresented in biomedical research isn't the same as underrepresented in genomics research

First meeting talked about relationship between representation and diversity

THB: in informatics 2 ways to think about this question - can think about when we build the datasets what presentation in terms of matching the exact population or in terms of each group having each distribution in the population - both have different pros and cons

Will post the reference to this

AL: diversity does not equal representativeness in genomics

Any comments on this background

We will recirculate the link

Our main task to think about different frameworks we can borrow from when thinking about what needs to go into our policy - Tina will speak to paper she has up in medical informatics space, will brainstorm different things we can borrow from to set up for discussion at next meeting

THB: we've been thinking about an ethical approach - my work is in medical informatics, lots of AI - paper thinking about holistic approach to AI which is a cyclical program - data acquisition and model development first 2 most relevant to this team

Thinking about data creation and origination - where data captured have pros and cons when we think about representativeness - need to think about sources in terms of what does that population represent - organization of datasets important, disclosure needs to be more transparent

Next piece is data we use for our studies, is great opportunity to improve our standards for what we report on - are trying to make datasets fit question were asking rather than making sure the data can answer our question -are legal and reg mandates that say what information we can share and how - there are financial agreements at play in how we can share information - so we do a lot of consolidation and aggregation, information gets lost - the way we have aggregated or redacted data may not carry forward, limits ability to ask questions and address data diversity

When we think about capturing data there are 4 types of biases: historical (in medical records, there no matter how data processed, affect the dat awe capture i.e. in crim justice system African Americans more likely to be convicted of similar crimes regardless of level of offense so AI models developed using this data reinforce biases), representation bias (when data sample does not represent the population AI will be applied to)

HL: is historical bias more general, not specifically genomics

THB: yes - we see this in medical where people were denied treatment, will have to think about in genomics

AL: want to see if there are useful frameworks we can borrow and build on - we might find these “buckets” useful for our discussion

THB: measurement bias important when we make proxies

Aggregation bias when we group populations together and we assume they have underlying similarity

Mitigation strategies: population representative datasets have group proportions match population proportions

Other way is proportions from each group, equal proportions

AL: we’re probably going to have to break things down to particular use cases since genomics so broad but representation bias will be important for polygenic score but not so much for other scenarios so maybe today we can think of different context we can consider

THB: agreed.

When we think about building and collecting these data have to think across the whole ecosystem, are limited in thinking about team diversity - have to think about all the stakeholders we need to consider to having a broad inclusion of stakeholders key to how to mitigate and be thoughtful about data being collected and how we use it

Level of generalizability could be useful in genomics - for engineer need to make sure experiments generalizable internally, for temporal want to make sure algorithms robust over time as med info changes constantly, geographic want to make sure across populations, centers, care studies, and across domains - are different ways we can think about how we might want to use something like this

Other important thing - this is a cycle, once were developing these AI models we’re generating sata from the info we have, learn biases, related to data we capture - important to think of how each phase affect the next

Take home point: these are the 5 stages we talk about - points regarding transparency, regulatory expansion, stakeholder involvement, evaluation metrics

NT: incomplete data jumps out to me - in the document there is no mention of metadata - this is a key aspect

THB: you're right, paper in chat about metadata and data sharing, metadata really how we understand what data re and how we build models using the data

AL: yes need to cover metadata.

Diagram from other group - they break down their recommendations into different stages of the research process - have recommendations about population descriptors, organize recommendations under terminology - we could do a similar thing - terminology, analysis, reporting, under each they have some recommendations,

Here is another set of buckets from GA4GH - (see diagram in background doc) other parts of GA4GH outside of REWS about data standards - we could use this diagram as a starting point

LS: background on diagram: essential was intended to show all the GA4GH published standards and categorizing them as part of a lifecycle process - this is one potential lifecycle, this one about starting with a population, with a data donor and putting them through consent, research ethics to data custodian, then standards split to “techy” how to analyze retrieve datasets etc and other is how researchers or other clinicians can access info that goes into those datasets i.e. passports or data access review standard policy and then thinking about how all of that can go back to inform human health and medicine - diagram to show how standards fit in life cycle

AL: from this there's a piece about who are the data donors, then jumps to data acquisition, getting data donor into database, then analyzing datasets and retrieving datasets

LS: discovery services are ways of creating registries of data, are similar in philosophy to BEACON

AL: we could take pieces from here that map onto the GA4GH data life cycle - everybody has a slightly diff take on the data lifecycle - we could define key points of data life cycle for our purposes

In terms of scope would be helpful to think about what kind of contexts are we supporting - we’re got diagnostic yield of analysis of NGS data, clinical deployment of PGS

AN: screening healthy people, newborns or adults in Australia - there is a research study looking at 3 conditions - there is interest in screening but not a lot currently

Have a screening program, grants that are under embargo to do modeling for how newborn screening might be used - but not necessarily the Genomics England model, this is to use genomics as an adjunct in newborn screening

Public health or precision public health should be added

Offer to connect with individuals working in Indigenous engagement in Australia at the moment, lots of current work bringing issues to the surface and trying to discuss them frankly

AL: Asking for use cases not part of the list

NT: these are all very clinical and missing research aspect of things

AL: trying to develop understanding of traits

HR: in research what's important for diversity is genes by environment, lots of studies in our case have included LAtin American people living in the US and they assume that's the result, miss whole environment aspect, should be considered, and push that research in these populations should be done in their local settings

AL: fits in understanding of traits

Does this need to come up in other types of research, isn't just understanding a phenotype according to a  geneticist, more like cellular level phenotypes, need to have a think about what is missing

NT: i assumed you meant phenotype by traits, but could encompass all, cellular to macro phenotype, would it be out of scope to think of engineering, way off in the future but make people uncomfortable to think about - genetic engineering, or perturbation which happens now

AN: rare disease may be good use case, big driver for why we want data across borders, usually happens in research context because of lack of funding, could be a way to pin research, an area where out of necessity there is a lot of codesign/engagement

AL: solving diagnostic cases crosses clinic/research divide but are always looking at same thing

HL: when you talked about diversity in DTC - what was the general idea about that?

AL: probably no, but we could consider what are diversity in datasets considerations

HL: paper on DTCs collaborating with research institutions - that data and the diversity implications of it will find its way into research in general

AL: all interconnected, what are the use cases - the use case would be returning someone’s ancestry results, which is probably outside of our scope but we can think about that a bit more

BC: I agree with Hanshi’s point, think DTC’s as a data source are a source of diversity in data, may be that ancestry isn't the use case, but that data source will probably have to be addressed because is a data source for the other use cases were interested in

AL: where this connects to data lifecycle - may be case that data availability would be a piece of the data life cycle and DTC could be source to consider maybe for some cases more than others

AN: on data life cycle - important to present as a cycle

NT: reference genome

AL: the reference genome kind of detects how we define the diversity in our data, but reference has its own problem

BC: own use case within itself

AL: for next time will make table of use cases and different points of data lifecycle, we may have to choose where we pick out battles, may put our principles i.e. when calling for diversity, diversity to what end, think of how they apply to different points in the data lifecycle - since were giving a tool for how to think about diversity don't have to show every use case

BC: how do we fill cells of table - 3rd dimension sources of bias - at the intersection, in the cells the considerations of relevance to the use case at that part of the cycle

LS: REWS meeting next week Thursday the 15th at 8am EST - everyone should have the invitation for that

AL: next meeting will be back to a time inconvenient for Australia

Next Steps

• AL will put together a table showing the use case and different points of the data lifecycle