AGENDA

REWS Diversity in Datasets Meeting 8
23rd January 2023 @ 4pm UTC

Zoom Recording

In preparation for the meeting please read the following:

• Overarching document of work to date

• Background document 
• Rare disease use case

• PGS Use Case

Zoom Details:

Join Zoom Meeting

https://us02web.zoom.us/j/4426465151?pwd=VXp4djBjRmN4YWw4cGNZUkQrSGlJdz09

Meeting ID: 442 646 5151

Passcode: GA4GH

Attendees:

1. Lindsay Smith
2. Maili Raven-Adams
3. Alessia Costa
4. Donrich Thaldar
5. Yasmeen Kurdi
6. Sarion Bowers
7. Henriette Raventos
8. Hanshi Liu
9. Beatrice Kaiser
10. Anna Lewis
11. Graciela Moya
12. Sarah Nelson
13. Tina Hernandez-Boussard
14. Judit Kumuthini
15. Gabriela Chavarria

Minutes:

Start today with a recap of polygenic scores. Hoping by April Connect we can have a first draft of the policy ready for discussion. By September, hope to have a finalised draft.

AL: Trying to get beyond just a call for more diverse data. Specifying what types of data we need and why, spell out what diversity really means in the context of genetics and genomics. We don’t think there is one overall answer, we think it will depend on the use case. We first discussed interpretation of variants associated with rare disease. Second use case is to explore PGS.

The first use case, the answer to the question diversity to what end is progress towards equity in receiving an accurate genetic diagnosis. Equity = equal likelihood for all humans that a variant or “allele” causative of an underlying rare disease is correctly identified as such. Main insights: key role of filtering frequency, the need to define suitable groups. Would really appreciate any comments on our document.

Second use case. Here the answer to the question diversity of what end is progress towards equity in receiving a clinically useful PGS. Equity = comparably predictive PGS, where predictive performance is assessed along with relevant dimension of difference (and acknowledge that this definition is limited). Main insights: the focus on “ancestral diversity” is problematic - “ancestry” is ambiguous, not clear if genetic or self-identified, and things other than genetic ancestry matter.

Need to focus on multiple dimensions of difference. Shouldn’t bucket people into categories. Other dimensions of difference might matter, like do the scores perform equally well based on sex, or socioeconomic status, or air quality. They key thing here is that depending on the trait, different dimensions might be important.

SN: Do we have a need to further specify what comparably predictive means?

AL: Great question. There’s the thing you care to quantify, and sometimes there’s different ways to quantify. I was thinking mostly in terms of r squared. Is it ok to just be focused on that way of capturing things? Do we want to give any bounds? To the first point on the r squared, will see if I can find people to ask.

SB: How do we address historic datasets in this? If we have massive amounts of data PGS are based off of, how to use the recommendations.

AL: What our plan/structure is to do diversity to what end, and go through each process of the data lifecycle and the implications of that. At least for assessing the performance, don’t need it in the original data, but in the validation.

GM: Trying to separate the populations or looking at some global population?

AL: The question of which populations. To assess predictive performance, need to define some group of people, how to select those groups. Could do on self-reported traits, use genetic ancestry (continuous space), could compare biological males and females. Many different ways, the dominant paradigm has been in terms of ancestry and the issue is that it’s not really clear what it means.

THB: We need consensus among the group that we’re hitting the right use cases, feel free to stop us or raise your hand so we can get a nice conversation going.

Someone brought up the word “predictive” - is that the right word in that sentence. It’s a loaded word. When we work across disciplines, people have different meanings for different words.

AL: Do you think there are misinterpretations we’re setting ourselves up for?

THB: Coming from a methodological field. It’s just a loaded word.

AL: Call out what we mean in the footnote.

JK: You mentioned that validation data does not need to be as large as testing data. Aspect of introducing simulated data. Have to keep on doing different things to see where things work, where to increase granularity, dimensionality, etc.

AL: It’s certainly the case where in an ideal world, one can have an enormously large dataset, one on the number of people, the breadth of data, and the amount of data you have on each person. There is usually a tradeoff between those two things. Getting deeper data is more expensive per individual, you either get bigger breadth or bigger depth. What we’re saying here is that when it comes to the validation, shouldn’t sacrifice the depth of data, because you ought to assess how well the score performs. Can help you to see how we’re underperforming in one group and need for more data.

To follow up with more detail on literature.

Open Qs:

Multiple dimensions: what to do about correlations in the data? (eg. between geography and genetic ancestry?)

Types of data: GWAS, validation, what else? PRS development

What are implications for data sharing?

What to say about “post publication” eg. clinical reports?

AL: There’s validation data, but one of the issues is that people develop scores in different ways.

SN: One thing that comes to mind, is that it can come from individual data or summary stats, which can be limited in certain ways.

THB: The metadata associated with validation data is key. We need to think about what it is we need to know about the subpopulations. I think we need to know the metadata to appropriately define subpopulations.

AL: I agree to think about it, but it would be good to know specifically what we need.

MR: Yes, perhaps a good idea to explore - and/or perhaps ask researchers, for example, to be transparent about their method of validation - and some of this metadata.

AL: Anything to assess how good a proxy is.

JK: About summary stats - the reference datasets for GWAS affects the sizes themselves. How would we take that into account.

AL: What type of reference data?

JK: Like the reference genome.

AL: Could connect with the pangenome folks who are building a larger reference set.

HR: Even if we include zip code or something as a proxy, that only works for some locations. Important to include examples of data that would improve how you would interpret the results. Dimensions are different depending on the condition.

AL: Could you add that example to the doc?

AL: Important to talk about the clinical side of things. What are the implications for clinical reporting. Let’s say that we’ve fallen short of comparable predictiveness.

AC: I can see how it can relate to counselling and engagement with the patient. Should publicly funded healthcare system be using a tooling that is not broadly applicable.

AL: Links back to Sarah’s comment about how comparably predictive do things need to be.

HR: I have a problem with how useful in general PGS are at the clinical setting. How useful and how important for the healthcare system depending on the healthcare system you have. For costa rica, which has public healthcare, if you compare the advantage of using the score vs a good family history, it makes no sense. Very different funding.

MRA: Maybe we need another box about actual use in the clinic for both use cases.

AL: Agree, should add a box. Maybe it’s just called clinical reporting. It’s such a narrow definition of equity we’re giving. Limited evidence it actually helps anybody. Have to walk to the line to provide useful contributions to this debate. Need to point out that equity is so much broader than performance by different types of data.

SB: Wondering why the focus on equity if we’re using such a narrow lens, is it maybe better to be thinking about other concepts.

AL: Generalizability, representativeness and diversity. Putting the value on equity, which is related most to generalizability. Real attention on how we should be defining the different groups. Maili, should integrate some of those pieces into the doc.

AL: Closing thoughts. What do you hope we will discuss next month.

(from chat)

HL: I like the idea about integrating environmental factors and examples of what that would look like. Maybe even the challenges of doing that or case studies of projects that have done that.

AC: Draft of policy to allow enough time for comments ahead of April meeting

SB: I would be quite interested in an outline of a policy - what are all the elements that will be covered.

AL: The PGS use case doc, would love more comments on that.