GA4GH REWS: AI/ML Bias
2021-09-09
- Quick round table introduction
- Tina will explain the work she is doing
- Discuss how this initiative can fit in GA4GH
Tina - focused on using clinical and health data on using AI to improve healthcare delivery and outcomes
Where Tina is on her work:
We’ve seen a lot of work on AI and healthcare and a lot of focus is on bias in the modeling development and data training. But there is a spectrum of areas to think about - what does it mean for the data to be biased, how should it be collected and used so it’s not biased. Need standards and additional ways to guide people. It’s a wild west, people are getting data any way they can, these aren’t regulated, and have no way of evaluating if they are biased, and/or if they improve care for certain individuals. We put out a couple of papers about standards in transparency. We don’t know how tools are trained or how their data is trained. Good opportunity to start developing these standards and the min information we need to know about a model to make it useful. When these tools and AI products come on board, there’s an opportunity to develop standards around the metadata, how where they developed, what populations are they coming from, what evaluation metrics are being used, how is the data captured and stored, and patient's consent. Right now, in all of those areas, there are no standards on how to capture that. There is also nothing on development bias, on auditing AI once it goes into the healthcare system (data drift concept), on what are the potential hams on certain populations and whose job it is to monitor this. There is a wide landscape to develop.
There is a great opportunity here to take a step back and look at the picture as a whole. From the very start, we need to ID what type of data it is and how it is being collected, how was the model design created, what is the endgame of the use of that data. We have often seen products that were developed in health care for purpose A but end up being used for purpose B.
https://pubmed.ncbi.nlm.nih.gov/32594179/
AoU - when it comes to transparency and fairness, there is a hyper focus on the model itself, but in reality there is the entire environment where the clinicians influence it, the data itself also influences it. We need to take the lens a bit further up and look at the entire ecosystem rather than the metrics and the performance of the model itself.
The research and clinical settings are now merging, we should look at both, and not necessarily distinctively. Ex. patient from Stanford, data gets into an electronic database automatically. It’s a continuous flow from clinical to research and research to clinical, there is no distinction anymore in that area. More and more use of data such as patient generating data: mobile devices, adverse events, this data also comes into play for both research and clinical care.
MZ: You mentioned standards, would be of interest to know who is the audience or who would benefit. At some point there is a role for the research ethics board or IRB to evaluate. If they don’t have the tools there is a gap that we need to fill. If we create a tool that would go on the market, we need to guide regulators. A lot of these things are unregulated, then you also have to consider the developers playing a role in making things better or clearer. Important for us to determine the spectrum, the actors and where to start.
AI is not regulated by the FDA because AI is continuously learning and evolving. Health care systems have developed clinical decision making processes. But, here, there is an opportunity to develop these standards for developers, before they even approach the health care system, so they develop tools with a minimum of required standards. If we start at the developers and data acquisition people site, there would already be a minimum of standards and a minimum of fairness/equity/transparency integrated in the tools. Then, the tools developed down the pipeline would at least start with these minimum requirements. More to do starting early on in the chain of creation.
YJ: risk predictions models are reviewed by regulators in the US. How much of that actually dig deep into AI bias? Probably not much.
SK: I went through the approval process for an AI tool, and the biggest gap I saw was the lack of scrutiny in ground truth data. There is a blind trust in the model that is presented, not tested against anything when applying for a grant. It reflects our own biases. It shows how hard it is to train models and evaluate them.
TD: Thinking from the perspective of the global alliance. One idea that comes to mind is an ad hoc working group that brings together multidisciplinary expertise. This initiative is not only a matter of REWS, we need cross disciplinary expertise, e.g., technical knowledge is out of REWS’ expertise. Trying to map the different stakeholders is an important exercise. It helps us ID potential risks and liabilities and how they ought to be addressed. Standardization is likely much more technical as opposed to rules and ethical principles. In terms of rules, the artificial intelligence app in the EU - legal obligations that developers and users ought to imbed. THere are already documents out there, a number of non-exhaustive principles were developed, we will not reinvent the wheel, we should take and adapt what already exists, and offer our own expertise to personalize it to our particular context.
There are a lot of frameworks and guidance coming out. A lot of it is focused on model performance, not the type of data, how it is collected, how it is stored, who’s capturing it and what is the intended use of that initial capture. Data governance is a huge issue that is inadequately addressed and highlighted. The EU is the first group to think about informing patients on how their data is being used in AI. That’s the first step and we should build on this.
Like the idea of a cross-stream group that would be assembled. One of the key issues is one of transparency, if we could already get the info out there on how that data is collected, have transparent disclosure, think of a policy on minimal requirements of disclosure on data requirements/collection, what kind of measures should we adopt to make sure there is no bias. As we form a group, we need to formulate a research question, and would need to have an idea that can take off fairly quickly.
We need to obtain more data and metadata on how these tools are developed. Our model won’t be enforced if the developers and agencies won’t adopt it. Also, there are not only doctors to take into consideration, but the software developers as well.
This initiative should fall very well in the open access mission of GA4GH.
ED: Not just a REWS initiative, these issues transcend regulatory and ethics. Need to reach out to technical WS colleagues. However, REWS could formulate some key principles of disclosure. Identify a few (3) questions that are particular to genomic and health data sharing (research and clinical). Can transform the questions into outputs, which might be standards, or a policy or a commentary on law.
Moving forward: envisioning a workshop led by Tina at Connect, where we can have ample time for questions from everyone. Maybe no need for a broad discussion at this point, but we should at least get some input from a variety of experts. Need to be advertised at the plenary. Having some ownership/leadership from REWS would help get this going. There is surely a connexion that we can exploit between this project and the GA4GH EDI initiative.