GA4GH Computable Cohort Representation Hackathon
This document is to assist with the planning and execution of the GA4GH Computable Cohort Representation Hackathon, kicking off on April 21 2022 and running throughout May. In this document you will find an overview of the hackathon’s goals and key resources, as well as space for taking notes on the 5 hackathon topics. If you have any questions or would like to participate in a topic meeting, please don’t hesitate to reach out to lindsay.smith@ga4gh.org.
We will explore 1 topic per week for 5 weeks. Each meeting will be 90 minutes.
Overview
- The question(s) this group is trying to address: How do we represent a set of patient’s info that is queryable from different systems? What are the pain points? How do we formulate these queries?
- MIACC group - working to define the sets of attributes required to describe a cohort consistently. Eg. in a database what columns do we have?
- PHENO group - working to capture the deep phenotypic characteristics as standard definition to execute against a baseline population to extract counts and statistics of the subjects within and across datasets. Getting more granular in terms of the details we can get.
Goals
- Crosswalk a chosen phenotype (Asthma) to determine the ideal representation of that phenotype
- Specify the phenotype in an executable format (CQL, etc.)
- Test the standard across multiple implementations in a federated manner (Data Connect, Beacon, FHIR)
- Data Access processes - Not Hacking - Accessathon?
Resources
- Includes links to: Jeremy’s branch of the hackathon github, Ian’s fork, dbGap asthma datasets, Example FHIR scripts, Example DataConnect scripts
Table of Contents
Topic Sessions:
Week 3: Data Connect / Phenopackets-FHIR
Week 4: Phenotype Matching Tools / MIACC-like queries
Week 5: CQL use of FHIR / Use of Terminology Servers and Mapping
Week 1: Access and DARathon
DATE: Friday May 6 @ 9 - 10:30am ET / 2 - 3:30pm BST / 11pm - 12:30am AEST
Materials/ Pre-Work:
Action Items:
- Ian and Sushee[a][b]l to submit access request to HDR UK asthma dataset
- Ian to look at some of the EGA datasets as well
- Ian to share access request, put it into Google Drive (protected)
- Lindsay to feed some questions/pain points to DURI
Minutes:
IF: What I wanted to do was query data, and access got in the way of that. Been looking at some of the Synthea data, and we can do something with synthetic but it doesn’t always come up to the mark.
[Share screen on dbgap]
Complicated user interface, have to go through a number of steps. The key document that I had to provide was this, the research use statement. One of things we might DARathon is how we put a research use statement together that would be approved. This is a bit different from being a researcher, we are planning to assist in providing associations. What we have might work for others also. Many datasets have been approved the day after access request. Ones approved were the cancer datasets, the asthma ones are still under review. The rejected ones were ones I had applied to awhile ago.
PvD: It’s like a ticketing system
SV: HDR UK runs an access system for the UK. Some use Salesforce for tracking. I have two questions: 1) how consistent is the set of questions they ask, things like who your collaborators are, what data fields do you need across other databases and 2) when you put in the application, what were the questions on the other side of the table, what do they need to approve or reject.
KH: In terms of what we look for, some important things is to make sure the data use limitations are going to be abided by. Make sure the research use statement addresses Data Use Limitations (Disease specific research, Publication required, etc). If see there is an IRB (institutional review board) attachment needed make sure there is an IRB approval. Making sure the people applying are who they said they are. Make sure they have a research question, methods, and that they’re using the data consistently within the Data Use Limitations.
SV: For cohort representation research, one of nuances is that it’s not Ian doing the research
IF: Yes agree it’s like a ticketing system. We’ve seen instances where people have used Jira for this kind of thing, the HDR UK use of Salesforce is another example. It’s always a balance between configuring an off-the-shelf package to the specifics of your workflow, or developing one from scratch.
KH: Not uncommon to see people developing tools
SV: What were the specific criteria?
KH: if you’re talking about taking the data and putting it somewhere else, that’s not acceptable.
SV: But if I’m providing a high level service on top? I can provide a query on top
KH: You’d have to talk about with the database managers
IF: I’ll show how I tried to head them off.
KH: Every collaborator that is going to access the data, needs separate data access requests
SV: We can’t name our collaborators, because anyone can query
IF: I have no collaborators on my application. Maybe we all want to work off those same datasets. You all might be collaborators, you would need to put your own application in. Look at if that document works for other people. For dbGaP I can add downloaders to my “project” without them having to put in their own application, but they can only be within my institution. What those ‘downloaders’ do with the data becomes my responsibility; it’s too much of an ask take that responsibility for those outside one’s organization.
SV: A delegation of your rights within NCI. We would be in this list perhaps
IF: Because you’re not in the same institution, I can’t.
Back to the question of consistency. Even within this list, things are not consistent, visual parsing to do. The asthma ones are being done at NHLBI where Karen is now. Even within NIH there is diversity about how these groups operate. Is a data access committee for EGA data going to look at this the same way as NIH or NHBLI? I have one data point for that - previously for a pancreatic data set, I applied to EGA. I sent the exact same research use statement and it was approved. No questions were asked, it happened quite quickly.
SV: Imagine you submitted an access request to EGA first, then took that approval and went to NCI. To the DAC, would that been seen as something favourable.
KH: In my opinion, i think probably it wouldn’t sway one way or the other, have to look objectively. Data use limitations might be different
IF: The statement says something like “this research is not specific to this disease”. They bring much more of an administrative eye than a scientific eye.
SV: Are there datasets that require multiple data access committees simultaneously?
KH: Probably not, but not sure
IF: My request went to multiple DACs, but no communication between them
SV: UK context. I go the GEL and say I want access to asthma datasets. I get approved. Then I get approval to the EHRs. I can now re-identify the patients, because the access committees aren’t talking to each other. I’m coming from a secondary use perspective. Two sides: how to standardize the set of questions, how to show how trustworthy you and your research is. Then, what is the selection or decision criteria. Downstream, what is the impact on other data custodians.
IF: All good points. We want to find a computable cohort representation. In FASP, we’re trying to avoid analysis paralysis. The things you just mentioned could get in the way, but FASP’s role is that it can help identify issues when we bring GA4GH components together to do a real task, but FASP can’t fix everything. We need to send that back to the right workspace. Should be sending these data access questions to DURI - e.g. are DACs applying the same criteria?
SV: Agree. Outcome could be to start with your template. Feed that into DURI and say, here is two applications we could standardize.
IF: Some of the EGA studies I’ve already applied to va dbGap. My request is going to be - here is my research use statement.
PvD: From what you’ve shown, is that you get access, but sometimes you only get access with some restrictions. How is that handled.
IF: Handling the restrictions is the DAC’s business. My end of the deal is when they come back to me and ask to fulfill the requirements they set.
Very similar granularity for EGA as is in dbGap. For the pancreatic EGA dataset, the request was an email to the researcher themself. Difference in policy from EGA and dbGap. I’ve heard it asked whether that difference is a good thing or not. A question like that I would punt to DURI to think about.
SV: There will always be differences in data access committees. They all require the basics of who needs access, for what purpose, what are the methods.
Once you get to a dataset in the HDR Gateway, you get the high-level data. If you want to request access, it will tell you how. Simple form. As a more complicated example - can go to the HDR UK Papers test set.
IF: Susheel, you mentioned the fax problem. Mapping that to what we’ve been looking at, what are you faxing?
SV: Still fill out the online form. Once approved, I have to fax in the data sharing agreement. Me as a researcher faxing. If I have 50 collaborators, they all have to do the same thing.
IF: I’m intrigued by this one dataset in HDR about genetics of asthma. Likely relevant to our hackathon.
SV: Brings you to a normal registration form. What’s important is what can we standardize, and what is different between the different custodians. For cohort discovery, the place where computable cohort rep could be attached as part of the application.
(IF: Somewhere in the above Susheel had indicated that you would have to indicate, at a fairly granular level, the specific data you wanted to access. It sounded like this could be at the level of individual fields. I was picking that point up in the following)
IF: It sounds like I might have to submit to a DAC every query I want to do.
PvD: Could have the situation where you have a dataset, but requesting access for the data in the dataset.Custodian gives you only the dataset that matches that definition.
SV: When I fill in my form, I attached my cohort definition, the SQL for example.
IF: Given the process people go through for cohort building. I tried to address this in my request, I specifically mentioned FEV1 and FVC to indicate that I was looking at disease (asthma) specific data. This was high level though, and only meant to be illustrative of the kinds of variable I wanted to access. I’m not sure I could anticipate, in advance when submitting a request, whether I could list all the specific variables I’d end up wanting to analyze.
I’m getting the impression that DACs are mostly administrative, they may not even know what FEV1 or FVC are and why they matter. In a way, this not expressed in query language. What I would not want to happen is for them to say ok, this person needs access to these two variables, we’ll only give him access to those. Did the same with the cancer section.
SV: If you submitted this to a DAC in the UK, they'll deny. They want to know what columns. Only get access to specific parts of the dataset. Eg. say I want demographics, this is the inclusion/exclusion criteria.
PvD: Ian said he wrote this without knowing what’s in the dataset.
SV: Don’t know how many items are filled. Justifying why we need computable cohort definition. Our dictionary is horrible.
PvD: Do you ask in the data access request what variables? Default its te entire dataset?
VL: All or nothing
SV: Genomics England does something similar to that. Like a reading library where you can browse all the books and take your pick.
IF: We’re operating in different cultures and legal jurisdictions. Isn’t some of this dictated for you? GDPR - is that where it comes from?
SV: No
The EU has a strong information rights regime. That’s a good thing in the sense that the GDPR makes researcher think through why you need access to the data.
PvD: I think that most of things happen when you capture and store the data. How to make sure that data is not identifiable. When you request data, most of those things are taken care of.
SV: Depends on the type of data, whether in hospital settings or EHR.
PvD: Here, you cannot capture social security numbers. In other EU countries they are able to do that.
KH: For this, what if you pick one major database, would it be possible to put in data request to get all of the fields to do this feasibility study?
SV: I can try! I will take Ian’s research use statement and apply for access and see what happens.
IF: We should continue to share information. We want to be looking at the same thing.
SV: That dataset already has a service that sits on top of it, but it’s not standardized.
VL: It’s a the same process where people have to fill out a form with a set of questions, tick the box. Same process as you showed Ian. Contact person you have to reach out to. All or nothing scenario. In terms of phenotypic data and cohort building, some metadata you can’t even ask for from the custodian. Very manual in a sense. At the same time, wanted to highlight that there is a different type of granularity on the data itself, whether we can even touch that.
IF: Karen asked the question - let’s find a big database with asthma info from different countries. Question is, what would be the big dataset in Australia?
VL: Right now, the answer is no, we don’t have it yet. Most AUS data ends up in the EGA, including our center. Might have something like GEL soon.
Question for DURI about sharing access requests
https://www.ncbi.nlm.nih.gov/books/NBK570242/#DAreq_ControlledAcc.how_does_the_dbgap_d
Summary: Reviewed the data access process for different databases (eg. dbGap, HDR UK, EGA). End goal to submit an access request for Asthma datasets in each, Ian shared the research use statement he prepared as an exemplar. Questions about the data access process will be forwarded to DURI for consideration.
Post-meeting Activity:
Email response from OH:
> What would be the big database in Australia – that we might be going to look for asthma subject? If so, how’s the process for the data access request (i.e. re: whether using “Research Use Statement”, any specific variables (phenotype/meta, genes, …) need to put in the request, “all or nothing” model or kind of data access policy, etc).
That database does not exist (yet). Australia doesn’t have a centralised resource such as dbGap or a Local/Federated EGA. Some of this work is in exploration, but for the most part research groups here submit their final data to EBI/EGA for lack of an alternative. There’s also no central registry of these data sets so any data access requests are handled by the respective groups and their institutes.
We (Australian Genomics, Australian BioCommons and others) are trying to change this but it’s all very much in the pilot phase. If the data involves any of the pilot flagships of AG there’s a process in place that I can walk you through, but it’s currently still paper- and email-driven despite the machine-readable consent and DUO codes available for most.
I realise this doesn’t help in the context of the hackathon and FASP but that’s where we’re at currently.
Week 2: Cohort Beacon
DATE: Friday May 13 @ 9:30 -11:00am ET / 2:30 - 5:00pm BST / 11:30pm - 1:00am (Thurs) AEST.
Materials/ Pre-Work:
We have a Beacon v2 with one of the CINECA dataset (CEUK1), we plan to load the other ones, but not there yet.
It is available here as API: https://dev.ega-archive.org/beacon-apis/cineca/cohorts/
And we have a mockup server with a UI here: Beacon Training UI - Cohorts (ega-archive.org)
Action Items:
- Lindsay - add everyone to Cohort rep slack channel
- Lindsay - Check that the Data Connect team is in next weeks invitation
- Jordi, Gordon & David - explore the idea of a CINECA implementers meeting
- Lindsay - Check in with Comms re: v2 dissemination
Minutes:
IF: Will we get a sense of how to set up a Beacon server?
JR: Yes!
SV: I’m more interested in how the server talks to the underlying databases.
JR: Beacon is a specification, a protocol. JSON schema based. What we provide is the blueprints, not the house. The Beacon has two big blocks, the API that includes the request and response, and then we have the model, part of the entry types we are having in the beacon.
What are the implementation options?
Option A: can implement the API in whatever language you choose with an internal database.
Option B: MongoDB we have at CRG. Has one implementation of the API, Python -> Django. Stores the data in the same language, no transformation from the content of the database to the Beacon.
Option C: Less realistic, because it’s trying to implement everything.
DB: In these reference implementations, do you provide an ingestion mechanism?
JR: Yes, next slides. We recommend that you have a beacon layer on top of your own API. If you API is similar to Beacon 1, the relation is pretty easy. IF the model is completely different, will need to go through multiple tables.
SV: I have a HAPI FHIR server. It’s got a CQL endpoint. So that server will exist on its own, and the Beacon API will sit in front accepting the user query. I would need to write another layer either in Beacon or CQL to transmit the query that comes in.
JR: If you get a token, it is your implementation
SV: We need a middle layer for the transformation up and down. If there is an ingest script I can just export
JR: We don’t recommend that, won’t get the Beacon updated. Of course, sometimes that is the way to go. Transforming the Beacon queries is cumbersome.If you have nothing because yo’re just testing, you download the reference implementation and populate it. Typical containers.
IF: I think one thing as we go through this series, we should keep things open. Can see how you would layer this. We’re mentally architecting this in a way. Can see that Beacon could sit on top of a FHIR search engine as it could CQL. I still need to learn the benefit of CQL. My general understanding is that it help you paper over the cracks of difference between FHIR servers. As we think about these concerns and how we modularize them, will sort out how that work. We all know that we need to get to the representation to how we implement that as a query. I have a feeling that we’re not going to be able to think about cohort rep without thinking how the “paper over the cracks model” gets provided. I.e. what is it doing functionally
SV: Last meeting, we discussed standardizing how that query standardizing would work. CQL defines functions and could use to define inclusion/exclusion. Imbedded in every FHIR server, so build into most health care systems.
MC: Don’t quite understand Option 2?
JR: Beacon API will say give me the list of individuals. Can translate that into Seql. If you get the query list of individuals, need to translate into which FHIR. Easier to get information from API to API, then to CQL or SQL. Someone needs to do the translation.
MC: I see - option2 is just using your existing API to answer Beacon API queries. got it, thanks!
SV: To paraphrase in my head, Option A is much more about taking the dta, convert?
JR: No, create implementation from scratch.
SV: Sorry, A is create your own Beacon API on top.
JR: C is not a option to implement, but ake something that exists and deploy it
SV: For that, there is another backend process where you convert the data in mongoDB
JR: Yes. 4 pieces - tools for loading data, mongoDB, the query engine and an example dataset
IF: For loading - what are you handling on your loaders
JR: Small facilities that want something quick and dirty to prove the concept on a local machine.Using a CSV do a quick validation, then convert to BFF (beacon friendly form). This is for shallow things.
IF: What I was doing was looking at the xml files for that dataset and how to parse and load them. The CINECA UK one.
SV: This format is specific to cohort beacon type analysis. Can you load anything else besides these example sheets? Need to have the data in some form so the schema is equivalent.
JR: Eventually converts to JSON documents according to Beacon JSON schema.
SV: Where in this excel is the phenotypic information?
JR: Not the best example, would go into individual
IF: What I’m trying to do on my other screen is create the Cineca UK table. Can you send it? How to parse the xml.
JR: It’s been done, in the github somewhere. I can find it
https://github.com/mrueda/Beacon2/tree/main/CINECA_synthetic_cohort_EUROPE_UK1
https://github.com/ga4gh-beacon
IF: Trying to wrap my head around how to use the vcf. A vcf is essentially a table in which you can put in anything you like. What you put in the table section, you define the format and say what the fields are. My question is, because I wonder if you’re taking the tabular information out of the vcf and putting it into a ‘proper’ table, a database table you can query.
JR: This is one toy. This works with vcf for this dataset, anything else will probably give you issues. My premise is that you have the data ready and you have your format, table, etc. with the amount of data you are managing. Managing based on the volume and quality of data they have.
IF: One of the most interesting things about Beacon is that you decided on a model. Would like to understand the model. The richness of Beacon version 2 is that people can have different models.
JR: Goal of Beacon is for as much data as possible to be queried. Viable as much as the models are similar. Need to define what you will accept as queries.
IF: You will get to showing us what that representation looks like.
SV: What you’ve standardized is almost a pseudo-CQL query language.
JR: We standardized a simple model
SV: The model is based on the query, but the query is standardized
IF: They have focused on a specific use case.
JR: We provide one model, the clinical genomics diagnosis domain + cohorts, and then the protocol - how you query and how you respond. Define a query language and model whatever you want with that. If tomorrow, you want a Beacon on images, you define it.
SV: Imagine you have a DICOM image files to query, the query has to change and accommodate extra bits of metadata.
JR: Model, not the query.
DB: Basically, its not because I implement Beacon and you implement yours, it doesn’t mean we can talk to each other. We need to agree on the queries and answers.
IF: Most important thing - success of Beacon 1 was that there were so many of them. Goal that you can go and query 1000s of Beacons. It’s that there are all these Beacons standardized on the same model.
JR: It is a discovery mechanism, not a query mechanism. Can have different networks of Beacons - some use OMOP, or FHIR.
IF: Beacon gives us the Swan on top of the water, not the paddling underneath.
SV: Who maintains the governance of agreeing to the model
JR: Every community. Need to ask what the need is, likely need to combine a solution that is not aligned with yours. It gets interesting when you say I need to talk to someone speaking a different language. Other one is when you don't have a query language. Beacon is to make different solutions talk to each other.
DB: What’s trying to be build in CINECA will require each member of CINECA using the same querying language and set of parameters.
JR: Yes, aiming for the same model
IF: CINECA Beacon. There may be 6 or 30. All are using the same model. And that’s the swan. For another community, they might have different model or swan.
DB: This should be made clear in the CINECA network
JR: Different when you have an establish community, this is still new. If you light a beacon, light both - the one you share with your neighbors and the Beacon model. Thats our recommendation.
IF: What i’ve seen in the H3Africa dataset. On the spreadsheet, on the left is the model and the right is where they’ve mapped to that. Not clear if everyone has signed up for the model. Need to deal with those realities.
JR: That’s one of the reasons we don’t want people to transform the data. If you only want to implement individuals, thats fine. Implement the things that make sense for you. Have a model, but don’t need to implement all of it. Allows you to do things incrementally.
Can get three levels of answers - boolean, count, and all the data. Showing individuals with all of the different attributes. Can change the ontology terms, if clever enough, we can match to each other. This is not in the database, only in the Beacon layer. In the Beacon layer you implement that mapping.
IF: What I want to do is try this out.
JR: In sql, need to say column, gender, person = male
IF: I still feel like I’m looking at the stuff under the water. How to say im interested to find people of a specific gender at a specific age range. What’s the best view you can give us of how simple it is to express that query of gender and age/
JR: Here’s an example with females with asthma. If you look at the implementation, this is not part of the specification. Sending http request with this structure. Have which API version, and the filters. A POST with the request. Both blocks should abide to the schema for Beacon to accept as a proper request.
IF: When you showed us that query, which dataset was that querying against?
JR: The UK set.
IF: Maybe we’ll get whether we can look at that dataset in FHIR. What you’re querying is that field. We actually want to query on a lab result for creatinine. This is going to depend on if they’re present in the dataset. 2 questions here: if you’re working off datasets that have that information in them, and if they were, can you query them in the beacon.
JR: Depends on the implementation.
JR: Can see the filtering terms that Beacon understands, in this case for cohorts. Can see datasets, individuals, datasets. In our implementation, this will look inside the mongoDB and find anything that matches that string, it will return it back. Will depend on the implementation. Officially, it should depend on the filtering terms. There are 3 types: ontology, medical, and the fuzzy search.
SV: For Beacon I can see this working. But when a researcher thinks about this, they think of things like negations queries, relationship, or temporal (eg. had asthma, and then came into the hospital with cancer). That temporal relationship between different events, and also relations, are very important. CQL can help us, because I don’t have to remember the filtering terms, even if the underlying was coding in SNOMED, it will work it out because it has the mapping. All I have to do is remember asthma
IF: There is one thing here at the back of my mind - bringing realistic scenarios of what happens in healthcare. Timescale can be short or long. In one hospital visit, they might measure the same parameter before given a drug. Testing responsiveness to that drug. Is the patient responded as not. Question is - do you capture the raw data or the summary, and you often have to capture both. With what you’re looking at with Beacons, we’ve been looking at research datasets, rather than healthcare. Will probably end up coming back to this. What’s the substrate.
JR: UK dataset is from regular healthcare practices. Should look like something realistic enough. We asked clinical partners what they need from a clinical beacon, they wanted something much more shallow.
IF: It’s the “I’ll look at it” that’s important, what’s the it. Just asking for all the data. Not only do they want to discover the cohort, they want all the rich data that goes with it.
SV: Maybe the hybrid approach is what we need, start simple and add. Don’t want the query interface with Beacon to get to that complexity. But also need a way to override and ask for very specific queries, especially if you want temporal queries.
JR: For that kind of scenario, we allow Beacon to tell you we have something that might interest you, and give you a link to a more complex place. Not trying to solve that with Beacon.
IF: If we go back to the initial discussions in the Fall, Melissa said it’s one thats statistical described. Maybe change computable to computed. You’ve pre computed the numbers. Not the same as searching creatinine values. Is there is a need to standardize around a model where a cohort is summarized like you’re showing.
JR: Not clear on how to jump from one place to another.
IF: Avoid the pain, just wrap up what we can do.
IF: Would you have in your CanDIG datasets, asthma patients? If they are like the UK datasets, made of up of a fairly broad population. There are going to be asthma patients in it.
DB: We do have asthma patients. They will be annotated with an ontology. If its annotated with SNOMED term, and then run a query using another tag, it won’t be found.
IF: There’s a whole lot more questions for us to explore. Interesting that we’ve got this different datasets and different technologies, gives us much better exposure to what the problem is.
IF: Would like to run queries on an interface Jordi has shown us. Also interested in the dataset. Would really like to know how this works from the CanDIG end of CINECA.
GK: Are there plans for an implementer meeting?
JR: Trying to get implementers into the Beacon call we are having every month. So far, we have no plan. Understand its needed, but implementers are all at different speeds. In CINECA, would make sense to bring the cohorts together. Can plan for something there. Starting to draft a plan for dissemination in Global Alliance - Lindsay can check in with Comms.
IF: Implementathon, practically get people who are doing these things together. Clearly a need to do it comprehensively in CINECA. Maybe the follow up is to trigger what Gordon mentioned.
IF: Next week we’re talking about Data Connect. Relates to this in that we looked at 2 of the CINECA datasets. Just looked at the UK sets. What I hope we can do next week is illustrate how Connect works and use the CINECA dataset to illustrate that. More important thing we’re looking at is understanding the data.
VL: I have attempted Beacon v2 model in Gen3 -- basically graph kind of data store underneath. Still WIP and very experimental. Sharing links here
https://umccr.github.io/umccr-dictionary/
https://github.com/umccr/umccr-dictionary#beacon
Summary: Jordi presented the Beacon v2 server populated with CINECA (UK) data, and the 3 different options of how someone could set up a Beacon service. Attendees discussed how Beacon could be used to discover cohorts of interest.
Week 3: Data Connect / Phenopackets-FHIR
DATE: Thursday May 19 @ 9 - 10:30am ET / 2 - 3:30pm BST / 11pm - 12:30am AEST
Materials/ Pre-Work:
Data Connect Trino implementation
See https://github.com/ianfore/data-connect-trino?organization=ianfore&organization=ianfore
Represent any dbGap data dictionary as json schema
See asthma example - with TopMed studies.
Three asthma datasets from the lists identified by Marion and Ian are in Data Connect and the hackathon Github here https://github.com/ianfore/cohort-rep-hackathon/tree/jba-test/output/DataSynthesizer
Other datasets from the list can be generated and added. Best version of the list at the following link.
Scroll to the bottom for the most readable list.
The first parts of the notebook show how the list was assembled from the FHIR server.
Simulacrum Patient table also loaded to Data Connect
Schema here
Capability of Data Connect to represent Cineca synthetic data tested - see notebook
Phenopackets-FHIR IG Repo + Docs Build
Action Items:
- Ian will post a selection of the things shown, and others including Phenopackets in the repo.
- Lindsay to check that the FHIR-Phenopackets folks are invited to the week 5 meeting
Minutes:
IF: Rather than focusing on the technology, we should focus on the datasets.
JJ: The computable, is not also about finding the cohort to start with?
IF: Yes, original intent is that the cohort part is machine readable. Highlights that we’re thinking about computability at two levels.
What I’m going to show here are a few things that relate to that comparison with Synthea. When we’ve been talking about Data Connect, been trying to drive what it does while understanding the problem. Don’t have the dataset Jordi shared in Data Connect yet. Some of those datasets also have different characteristics, what is the origin of those datasets. Gives a different substrate to work off of if you have a dataset generating from healthcare. This asthma focus that we have for our hackathon, how is that representing. This is a work in progress.
PR: How to get access to any of this data?
IF: https://github.com/ga4gh/cohort-rep-hackathon
IF: The focus we should make on Data Connect is not on tables, but about describing data and schemas. Phenopackets in Data Connect is a nested structure. One of the most important things is describing datasets in a way they are usable and reusable. Underneath we can have many different kinds of data stores, like graph databases, FHIR, phenopackets, google sheets, relational databases, etc. Provides an API to connect to those. Means we can use these different sources and aggregate from different tools.
You can put any table up, but the reaction to that is it’s not harmonized or using semantic terminologies. Yes, we want that, but that’s not where everyone is. So there are different entry points.
PvD: The architecture, the data stores, are those already implemented?
IF: Relational databases, CSVs, Phenopackets, FHIR have all been done. Not sure about vcfs. There’s a table in a vcf.
AK [from chat]: https://github.com/elimuinformatics/vcf2fhir - representing vcf using FHIR
PvD: If we are going to have cohort definitions in Phenopackets or FHIR, is Data Connect the place where you have the schema mapping?
IF: Yes, we have examples.
We used the CINECA data in the demo in Montreal. It’s in the recording. dbGap has these xml data dictionaries. Door 2 is you have a schema, it’s still unharmonized data, but at least the columns are more defined. This schema shows the age range, sex defined as a string.
In the CINECA dataset, would find a csv and the column headings, as well as a data dictionary in excel. Imported that into big query. We turned that excel sheet into a dictionary.
CINECA dataset - collab between Canada, Europe and Africa, has 4 synthetic datasets they created. They had described a minimal metadata model for the project. H3Africa is mapped their synthetic data to that model.
IF: We’re enabling is enabling pulling that data into a dataframe, where its easy for a data scientist to do that transformation. They then know what those codes mean.
PR: How would data connect work to connect dbgap with CINECA study?
IF: Will show you two things. This data dictionary is out of the cancer research data commons, which uses dbgap studies.
There’s a NCIT code for biological sex, I can take that and look up on identifiers.org. When we’re talking about data, I need to know what the 1s and 2s in it mean. NCIt will define the concept of male and female. What this template is doing was referring to the data element ID.
JJ: What are these caDSR things? Are these links to NCIt?
IF: Some python code extracts the links to NCIt. The codes in the achema are to caDSR.
PR: Let’s say I have a dataset and have a German way of defining sex, can I use this system, or do I need to make a map? What does the API do?
IF: The API will tell you how the data is defined. Data Connect is not going to set out to fix every problem.
PR: Do I need to map my data to this standard?
IF: You can do whatever you want to do?
PvD: Not mapping to data connect, just whatever data source you want to. Can we use data connect as this middleware to facilitate.
IF: Don’t expect that any technology is going to give it to us off the shelf.
Can use that API to get out of that repo to find the individual values. What this is not showing is that yes, it links to a NCIt code. This is the ID of that value set. The two are cross referenced.
PR: This is a particular code for a certain dataset. Does data connect offer ways of linking to recommend code, or does it transform souce data?
IF: It will give you the ability to connect to those definitions.
PR: If someone gives me an excel table, I don’t need the API to connect.
IF: The added value is that it’s a common API. If I have the spreadsheet, it is establishing the standard by which we communicate that connection. If I have got two data sources, and both have gender, if all refer to the same cde, I know that I can harmonize those columns from the three sources.
PR: As an user, I need to decide?
IF: If everything has been done right, it can be called whatever, if the yhave the code I know what it is. See that they point to two different elements.
If you saw your column uses this data element, will not take on trust they mean the same thing.
PR: If I have some particular study that has some explicit or implicit schema, how do I connect that to Data Connect?
JJ [in chat]: https://linkml.io/
I think LinkML might be a useful way of defining and mapping schema elements
IF: Went to dbgap and looked at what datasets have asthma. These studies were independently run. What is the config file you have you create that you were referring to.
PR: These are the files you get directly from dbgap, so in general these will not be using the concepts you mentioned.
IF: We’re trying to fix that is that point
PR: How do I tell the system to interpret it in that way?
IF: Someone needs to do the legwork of the annotation. The format that helps
JJ: does it matter to have to use the cde?
IF: Can use any metadata repository you want. We can point to schemablocks, we can point to phenopackets.
IF: Going to add a line of code an make a raw data connect call. Brings back this definition. It called this code from the data dictionary. The punchline is about mappings.
PR: Would be useful for somebody to basically go through these 15 thousands studies?
IF: TopMed did that. The problem, is that they set out to do that and only go so far
JJ: The dbgap has a specified schema for the data that needs to go in there, and some groups have provided mappings?
IF: They conform to a model. Everyone has to complete the data dictionary when you submit to dbgap.
JJ: The choice of terminologies go in there could be whatever the submitter wanted to put in it
IF: yes, but NIH is also working on that.
We can provide the template and say represent it in this template. Giving the format for the data dictionary.
JJ: Much like a phenopacket.
IF: The punchline. If no mapping existed, here’s how I do the transform. I manually wrote that. Because Data Connect provides me that information, its relatively easy.
PR: Do we make decisions as to the target terminology?
IF: Not ruling out any of those possibilities. This example is a data scientist wanting to work in Python.
We probably need a standard for mappings, shouldn’t be a part of the data connect API and is probably needed across several work streams.
AK [in chat]: https://www.hl7.org/fhir/mapping-language.html
PR: It’s been hard to maintain efforts like this in bioinformatics. Would be nice for GA4GH to look into this, it’s something everyone needs.
FJ [in chat]: I think a mapping API should be the interface to existing mapping knowledge bases. e.g. https://mondo.monarchinitiative.org/
IF: Coming back to my mappings I was creating. Don’t know if they are authoritative, but trying to use authoritative terminology.
We can do numeric mappings.
Ian will post a selection of the things shown, and others including Phenopackets in the repo.
Lindsay to check that the FHIR-Phenopackets folks are invited to the last week meeting
Summary:
Ian walked through the Data Connect server he set up, with dbgap data dictionary examples. All of the content we didn’t have time for can be found here: https://github.com/ga4gh/cohort-rep-hackathon/tree/main/notebooks
Week 4: Phenotype Matching Tools / MIACC-like queries
DATE: Tuesday May 24 @ 9 -10:30am ET / 2 - 3:30pm BST / 11pm - 12:30am AEST - POSTPONED
Materials/ Pre-Work:
Action Items:
Minutes:
Week 5: CQL use of FHIR / Use of Terminology Servers and Mapping
DATE: Wednesday June 1st @ 4 - 5:30pm ET / 9 - 10:30pm BST / 6am - 7:30am (Thurs) AEST
Materials/ Pre-Work:
CQL & FHIR
Detailed instructions on how to set up the FHIR server with Synthea data locally can be found in the readme of our GitHub repo. A cloud server has been set up at https://cohort.ga4gh-demo.org/.
Terminology servers and mapping
Asthma mapping examples added
Phenopackets FHIR IG
Hackathon slides
Here
Action Items:
- Philip will create an asthma phenotype definition in CQL and run this on the FHIR server with Synthea data
- Will also look into how the FHIR server behaves as a concept map
- Lindsay (and all) to start review all of the content and collate ideas
Minutes:
[kick off with round of intros]
IF: Shouldn’t feel like we cram everything in today. Find time to fill in missing gaps. Phenopackets not related to FHIR that we need to talk about.
PvD: CQL = clinical quality language. HL7 standard that provides tools to describe clinical language. Can define things like codesystems, value sets, individual codes. Can write the statements where in this case, they said we have some observations (eg. smoking status) and looking for observations where the status isn’t amended.
IF: When you have a status as final or amended, are we talking about the standard is in final state, or the process of however that is obtained for an individual patient?
PvD: This specific case, the status of a condition is the status of the result value.
IF: Take that on advisance, that what its saying
PvD: What CQL can do, can translate into FHIR API call. Can store the CQL library in the FHIR library.
FJ: That get request, the code is LOINC, but when you submit this, what’s the response? The definitions?
PvD: Would get something like what's on the right.
IF: This comes to the question of where have we got to. How would we take that CQL representation of a query, and run it against several FHIR servers.
PvD:Found this paper, they took phenotype definitions from Phekb and represented them in CQL and FHIR and made them available to the public. Can see the criteria from PheKb where all patients have a particular set of ICD codes, excludes others, and give more information on whats in the dsm4. They took this narrative and then defined in CQL. From the definition, defined the valuesets that contain all of the different codes from the definition. If you were to run on a FHIR server, the idea is that you get all of the patients that conform to that definition.
JG: How would you execute on the server?
PvD: If you use an implementation like HAPI, then CQL is already supported. Can transform into a more machine readable format of CQL. Not sure if you would do it via the restAPI or via the library.
JG: My understanding is that we have this library that contains the CQL, and you also need to define a measure, because CQL is measure level centric. Then you get back a measure report which is the calculation of the measure, might have a numerator and denominator. Is this more of a patient selection cohort type case.
IF: There’s something interesting about what you’ve said - this is a thing, a measure, what does that mean in Australia or in your experience? In the US, clinical quality has a specific significance and I don’t know if that has broad application.
JG: In Australia, we haven’t achieve as high of a level of digitization as the US. Similar concept, but probably more clinical in nature, maybe like number of adverse events per number of prescriptions. And might be used for reporting to the federal health department to monitor performance.
IF: Very similar to what it is in the US. Clinical quality measures came out of the affordable care act. Not sure who will be looking at these kinds of reports. Business focused thing.
PvD: Originally, it would be or quality measures, and I think in this paper they’re talking about phenotype definitions in OHDSI or i2b2. But they also say its technology agnostic. I’m sure its also used for other things, but not originally what it was used for.
IF: With the examples show so far, you’ve shown a query. We want to run against different FHIR servers, how things get coded in different servers is different. This would only work if Autism had been coded or followed the same model for this query. CQL gives some ability to paper over the gaps, is that right?
PvD: As any other FHIR implementation, it conforms to some profiles or resources so can only reference to those base resources. They took those existing phenotype definitions and translated into CQL, and I don’t think they had to use any exotic FHIR extensions. In this paper, they also mention the common data model harmonization project, and they mapped different data models to each other. If we could use some of these mapping, and we have a FHIR server with FHIR data, and an OMOP server with OMOP data - in theory, we have this confirming to the FHIR model, take the omop to fhir mapping, use the same CQL query for both the omop and fhir server. Would like to try this in practice. Means we would have some cohort definition that could be used across servers.
IF: Who runs some of these mappings?
JG: Hosted by the FDA, but in collaboration with NIH and those groups.
IF: Static mapping (this term maps to that term), not computational mapping. There’s some computation to create the mapping, but after that, it’s just data.
PvD: Sure they made them manually, data model mappings. Can go to the terminology mappings as well. Would be nice to use the same definition to query another database. But could run into trouble with the codes. Let’s say there is an ICD 9 code and I use a SNOMED, then this query is not going to work.
IF: This is a problem that is somewhat independent of the technology. FHIR has this capability, resource called ConceptMap - the ones you showed, were they expressed there?
PvD: ConceptMap is terminology to terminology, not sure if you can do data model mappingsfor that, i think something called FHIR Mapping Language can be used.
AK: There is a mapping language (https://www.hl7.org/fhir/R4/mapping-language.html), but not sure if it’s completely automated or can run over servers.
IF: Because it’s using some FHIR-related server for the mapping, when youre looking at CQL, is it baked into CQL to make use of those mappings in that format.
PvD: I don’t think so. Say you execute this on a FHIR server (which we should do), let’s say it has a conceptmap that maps this ICD code to this snomed code. Would be interested in testing whether if we used the LOINC code, will it pull out the semantically equivalent SNOMED codes.
If that works, we can work with different data models,so lets say FHIR and OMOP, and we could potentially also work with different codes. We can provide mappings and computationally store that mapping in a FHIR server, and that means we have a CQL phenotype definition that uses these codes.
JG: I think there’s functions in FHIRpath and by extension CQL that can do concept translation. So it’s not always straightforward for a server to translate code without any context because concept maps are use case specific. Can write in your expressions, translate this code to this other code system, using this particular conceptmap.
PvD: Would have to point of conceptmap, and the server should be able to interpret that.
JG: Could be multiple different ways of doing that, multiple maps.
IF: When we talk about cohort building, ETL that works for one thing doesn’t necessarily work for another. Need to be able to drive from picking the mapping that works for your purpose.
JG: I’d be suspicious of any ConceptMap that’s general in nature.
IF: Concept is a function in CQL?
PvD: In FHIR.
IF: Wanted to ask Gilberto, how would this compare and how would one bring to bear the mapping from the NCI structure?
G: We have a number of mappings, not sure of the mapping. I can look them up.
FJ (in chat): MONDO maps ICD, SNOMED CT, PhenoDB, MedDRA, and others - https://mondo.monarchinitiative.org/
PvD: Practical question related to that - I can imagine it would be very hard to determine what mappings are useful to have. Let’s assume someone has to create those mappings manually. Maybe on a technical level we can say yes we can support those mappings, but do we do it in practice
IF: We should try, experiment in this hackathon
G: https://nciterms.nci.nih.gov/ncitbrowser/pages/mapping_search.jsf?nav_type=mappings&b=0&m=0
JG: Can be difficult to map between terminologies, and can be imprecise. ICD is closed, SNOMED is open world ontology. So trying to go from ICD to SNOMED, because trying to convert this category into all possible concepts that could fall within it.
G: We have a number of terminologies in the NCI term browser. The ones on the bottom, most of those are manually curated. We can candidate matches, its not perfect, but we go from there. The maps are very simple.
IF: There are mappings as resources that exist that are created by a lot of different people. There should be a marketplace for mappings. If you were in CQL or Beacon and needed to be doing mappings, is someone else's mappings readable and usable.
Placeholder for what are the one or more standards that mappings exist in, and what is desirable. There’s going to be a mapping of mappings.
PvD: Isn’t that also what Data Connect offers?
IF: Not really, I can illustrate
PvD: UMLS Server has an API. If you search for Asthma, shows a number of codes that all have the same meaning.
AK: Is UMLS linked to PheKb?
PvD: Not sure
IF: We started talking about Data Connect in session 3, and ended up in same spot in talking about mappings. I wrote a function to get a decode template, which is decoding the values but bringing it back as JSON dictionary. Use a data frame code to transform into something human readable. Was going to recode the smoking concept.
PvD: So links them together by the ID
IF: Would be relatively easy to create whatever mapping I want. Point is we can go and save them.
Going to create a mapping where I translate 0 to unknown and store in my database. We need repositories like this at the fingertips of people building cohorts.
PvD: depending on what stage you introduce those mappings, if you know you have a mapping between two codes, could include directly into CQL definition of the phenotype. Wouldn’t need a mapping. Need to be really sure your mapping is correct, or you’re changing the phenotype definition.
IF: Which hat are you wearing when you’re doing that. Providing mappings for other people is a different level.
PvD: I think for what we are trying to do, our job ends at we have a way of also defining mappings in a computable way. Responsibility of the person doing the computation that the mappings are correct.
IF: Ability to access and make use of mappings in a machine readable way is very important.
JG: Planning on demonstrating this over a particular flavor of FHIR, like an IG? Do you say in an IG that you should code up your conditions in this subset of terminologies.
PvD: Let’s say you have a dataset that conforms to Phenopacket, but then you take an asthma definition from PheKb and end up with different codes and elements. Not sure how to solve that. Would have to rewrite your definition according to the Phenopackets IG. Then maybe use FHIR mapping language. Interesting problem.
JG: Conceptually, can either put the burden of transformation on the producer or consumer of the data.
IF: When you get that community to agree, becomes a cohort for the study. In the first use. Then there is the secondary use case, where you want to build cohorts across diverse datasets.
PvD: Have a visual on the different technologies available now.
Summary: Philip Van Damme introduced Clinical Quality Language (CQL) and how it can be used on a FHIR server to return cohort of patients that conform to a definition. There was interest in testing whether we can use the same CQL query in different servers (eg. OMOP and FHIR) and if we used the one code (eg. LOINC), will it pull out the semantically equivalent other codes (eg. from SNOMED)
Summary & Retro
[a]@forei@mail.nih.gov @susheel.varma@hdruk.ac.uk Just tagging as a reminder!
[b]Done - submitted request for Genetics of Asthma Severity & Phenotypes dataset.
SUPP-1NNN Cohort Discovery Access Request | UserId: secret | Environment: prod
Also looked at the EGA datasets - still trying to identify a useful dataset.
[c]@Peter.Robinson@jax.org @j.jacobsen@qmul.ac.uk FYI - there is a Data Connect + Phenopacket example we didn't get to today, found at https://github.com/ga4gh/cohort-rep-hackathon/blob/main/notebooks/Phenopacket%20examples.ipynb