GA4GH Data Connect API

Minutes and Actions 2024-2025

These are the minutes for the GA4GH Data Connect (formerly Search API) Sub-Group of the Discovery Work Stream. For further information, please visit the GA4GH Work Stream page.

For 2020 and prior please see the separate Discovery Networks the minutes/agenda for Discovery Networks.

Table of Contents:

Start-Up Guide to Data Connect API

Meeting Protocols

List of Meetings

2025-09-24

2025-08-27 - cancelled

2025-07-25 - Cancelled

2025-05-28

2025-02-26

2024-11-27: Roadmap review

2024-10-30: GA4GH tech team overview

2024-09-25

2024-08-28

2024-07-31

2024-06-26

2024-05-29: HDRUK, Data Connect and Passports

2024-03-27: Canceled

2024-02-28

2024-01-31: Use case presentation

2024-01-10: Roadmap review and refresh

Start-Up Guide to Data Connect API

Any questions or issues? Contact miro@dnastack.com / jonathan@dnastack.com 

Meeting Protocols

• Please note that by participating in meetings, attendees agree to adhere to the GA4GH Standards of Professional Conduct.
• Meetings may be recorded for note-taking purposes. Recordings will be deleted within three months of the meeting taking place.
• Dates should be specified in the international format yyyy-mm-dd

Copy/paste the below, add date and topic and refresh the table of contents at the top of the page

List of Meetings

Also available: 2018 Meeting Minutes, 2019 Meeting Minutes, 2020 Meeting Minutes, 2021-2023 Meeting minutes

From 2020-02, meetings are in conjunction with Discovery Networks, where required.

2025-05-28

Chair: Jonathan Fuerth, Miro Cupak

Attendees “Name (Affiliation)”: Beatrice Amos, Marshall Thompson, Nara Sobreira, Ian Fore, José Ma Fernández, Robin Kageyama, Ben Kamphaus, Marc Perry, Michael Sierk, Takudzwa N Musarurwa, Augusto Rendon, Kathy Reinold, Heather Creasy

Link to meeting recording: https://us02web.zoom.us/rec/share/TjImhbwDR5I3mp3HIhpx_27bHEORV9RadXsgNl1Y0VuJcCXSaZdpPr-552Ev2iW9.e9xWEX7DEW7Zz3Bn?startTime=1748447868000 

Meeting summary:

Intro:
Jonathan gave a brief recap of the two informal ‘lean coffee’ format sessions held at the Boston Connect meeting, where participants discussed the future of the Data Connect standard and its intersections with other GA4GH standards. The key action item - synthesising input from those sessions - remains outstanding, and Jonathan noted plans to revisit it in a future meeting.

Marshall’s presentation:

Marshall Thompson recently transitioned from President to CTO of the Rare Cancer Research Foundation (RCRF), allowing him to focus more on technical and infrastructure-related work. RCRF, founded 13 years ago by a patient with uveal melanoma, aims to address the lack of research infrastructure in the rare cancer space.

Historically, the foundation has focused on enabling patient-donated living tissue to reach researchers, helping to build research models and test hypotheses.

Marshall emphasised the need for more research and innovation in the space to expand treatment options for patients and concluded by expressing his interest in discussion and exploring possible overlaps with the work of the GA4GH Data Connect group.

Marshall described ongoing efforts to address the challenges of working with rare cancers, which, though individually uncommon, collectively account for about a third of all cancer diagnoses. He emphasised the need for systemic infrastructure to improve research, diagnosis, and treatment, advocating for a shift from a “vicious cycle” of fragmented data and limited progress to a “virtuous cycle” where shared infrastructure and data drive continuous improvement. Central to this is the creation of an open-source, integrated data ecosystem where patients, clinicians, and researchers all contribute and benefit.

Marshall detailed how his team collects patient samples and sequences tumors, feeding data into a commons that supports a learning health system. The key challenge is managing heterogenous data in a way that doesn’t assume the kinds of questions researchers will ask in the future. To address this, his team designed data tools that embrace change and evolution in data models. He criticized traditional bioinformatics workflows - cobbling together spreadsheets, ad hoc R scripts, and manual data harmonization - as inefficient and fragile.

Instead, they developed a tool called Pret, a data-driven compiler that allows domain experts (like scientists) to define ETL (Extract, Transform, Load) processes with minimal coding, using human-readable mappings from diverse data silos into a canonical schema. Pret enforces referential integrity and other validation checks at the point of ingestion, ensuring that the ingested data is robust and ready for analysis or reuse. Ultimately, Marshall argued that such approaches free up researchers from tedious data-wrangling, allowing them to focus on real scientific questions and innovations for patient care.

Marshall introduced pattern.org, the tissue donation program underlying his vision, which is to build open, accessible systems that can reveal meaningful patterns across seemingly unrelated diseases and data types - particularly to help rare cancer research, but with applicability well beyond.

–

Jonathan praised the work as addressing long-standing Data Connect goals - specifically the need to reduce redundant effort when reconciling disparate datasets and terminologies. He noted that the system, which enables domain experts to define mappings, could help share this work across projects.

Marshall elaborated, explaining that their schema is not canonical but highly adaptable. They support different formats like EDN (used in Clojure), YAML, and JSON schemas. These schemas describe how data attributes relate - for instance, how measurements refer to cell populations or other entities. Because the schema itself is data, it can be used to build tools that generate or validate these structures visually and programmatically - such as Blockly-like drag-and-drop interfaces that emit structured YAML or JSON.

Ben added technical depth, explaining that storing both data and its schema together in a sparse representation allows inference over the relational graph. This supports context-aware autocomplete, schema validation in editors like VS Code, and even projection into formats like tables for tools such as Trino. He emphasized that the system’s underlying architecture enables flexibility in building rich interfaces and supporting multiple forms of data interaction without rigid limitations.

Ian, formerly with the NCI and involved in projects like TCGA, raised a broader question about where Marshall’s system fits within the wider biomedical data ecosystem. He acknowledged the foundational role of large-scale initiatives like TCGA while pointing out that no one entity builds an ecosystem - we all exist within it. He asked how Marshall’s system addresses integration with and use of external data, such as TCGA.

Marshall and Ben affirmed that this is a key goal. While legal and access restrictions prevent open redistribution of TCGA data, their tools can ingest and work with such data where permitted. The strength of the system lies in semantically mapping datasets - creating meaningful links across different resources. This can unify disparate schemas not by enforcing a universal one, but by enabling mappings between them, allowing localised flexibility while supporting interoperability through common frameworks. Marshall emphasised that this federated approach can scale across diseases, data commons, and communities.

–

Jonathan noted a shared understanding of both the problem and solution space, and raised the possibility of aligning data model annotations within the JSON schema used by Data Connect.

Marshall responded positively, referencing prior work using Trino (a distributed SQL engine) and suggesting that similar schema descriptors could likely be generated programmatically for Data Connect, as was done for Trino. This led to the discovery of strong alignment between the two teams' technical efforts - Jonathan mentioned that his team at DNAstack had been working with Trino extensively, even developing custom connectors.

Marshall expressed a particular interest in improving accessibility for non-technical users by better integrating data into familiar visualisation tools.

Jonathan highlighted a feature called "questions" built on Data Connect, which allows non-SQL users to run templated queries for consistent data exploration.

Together, they discussed the potential of using concept mapping metadata to help users generate new, useful queries that align with broader analytical definitions.

Taku asked whether there were APIs connecting services like Trino and Arapython to the underlying data stores.

Marshall and Ben Kamphaus explained that various APIs exist, including a JSON query API and native system APIs depending on the use case. They emphasised their flexibility in projecting data into various formats, including potential future GraphQL endpoints.

2025-02-26

Chair: Jonathan Fuerth, Miro Cupak

Attendees “Name (Affiliation)”: Beatrice Amos, Jaclyn Estrin, Jose M Fernandez (Barcelona Supercomputing Centre), Marc Perry

Link to meeting recording: https://us02web.zoom.us/rec/share/h7gmsGq2OsnrxQkzFqMNvf4leTri9-RDPoWCpSNKsHpqkdYtopOGI_CfO7wy3Kq7.Rl90-i8O_opgwiLH?startTime=1740589193000 

Meeting summary:

Implementations:
The plan for this meeting has been to gather feedback from Genomics England's implementation experience to help future adopters of GA4GH standards. But some key contributors seemed to have moved on, so next steps are unclear.  

Miro and Jonathan mentioned reaching out to Augusto, who is leading a driver project and will present at GA4GH Connect.

Jacqueline expressed enthusiasm for crafting a high-level story on the impact of implementations while also supporting a more technical blog post.

Finalising plans for the upcoming GA4GH Connect sessions:

Jacqueline will be attending the Connect meeting for the first one or two days and encouraged attendees to connect informally if needed.

Please review the session agendas, check for scheduling conflicts, and consider any pre-Connect tasks, such as reaching out to potential contributors. Miro outlined his goals for the Data Connect and Service Registry sessions, which included gathering community feedback and prioritising future work. He noted that the Service Registry session would be more of a reboot, while Data Connect had ongoing active development.

The group discussed various interactive engagement tools for their session, such as Sli.do, The team considered a structure that includes a welcome session, context setting, a roundtable discussion on use cases, and a voting mechanism to prioritise ideas. Miro emphasised keeping the session informal and accessible while ensuring key stakeholders are aware and able to join. Beatrice raised the idea of inviting Data Connect implementers to contribute insights, which Miro acknowledged but aimed to keep discussions unbiased. The team agreed on a flexible approach, adjusting plans based on participation levels.

2024-11-27: Roadmap review

Chair: Jonathan Fuerth, Miro Cupak

Attendees “Name (Affiliation)”: Beatrice Amos, Nara Sobreira, Leon Funnell, Jonny Blanksby, Marc Perry, Nara Sobreira, Stojan Jovanovic, Ian Fore

Link to meeting recording: https://us02web.zoom.us/rec/share/5tgtRbWx-lva5ClwrhwlH4jjrmcxcwNGIW0nK_1DeREtr_wBeGgXZ2n9eZpPp842.IrZF03yz5MGH3vPP 

Meeting summary:

Data Connect meeting summary:

Reminder: upcoming GA4GH Connect meeting in Boston, scheduled for April 1st - 4th, 2025. Proposal for a more interactive, work-focused event aimed at progressing standards development, differing from prior presentation-heavy sessions.

GEL (Genomics England), as an official GA4GH Driver Project automatically invited - how does this group feel such meetings could best support their needs, particularly in driving standards forward?

JonnyB acknowledged GEL’s early stage of involvement, stating their limited capacity to shape broader discussions about Data Connect’s direction but expressing interest in understanding the rationale behind specific features, such as formatting and pagination.

He emphasised the need for GEL to gather more user feedback and implementation experience before contributing significantly. He also raised the possibility of minor adjustments to standards based on practical challenges.

Pagination discussion:

Leon (contractor working on the Data Connect API implementation using a Dremio backend for GeL) described their focus on parsing SQL queries and implementing cursor-based pagination, with plans to potentially support offset-based pagination in the future. He suggested standardising pagination extensions to prevent arbitrary naming and ensure industry consistency.

JonathanF responded by explaining the design choices in their implementation, emphasising minimal barriers for implementation while maintaining performance, such as avoiding the need for full-result buffering. He supported the idea of optional capabilities for servers to indicate readiness for advanced features like random-access pagination.

Leon highlighted that implementation-specific capabilities, like performing a `SELECT COUNT` without significant penalties, could enable use cases such as downloading entire datasets for offline use. He emphasised the importance of prioritising these features and adhering to standard verbs.

Corey (GeL) added a use case involving downloading filtered query results as complete files (e.g., CSV, TSV, or VCF formats) for local analysis or pipeline inputs. He proposed extending the Data Connect API to support such asynchronous operations, where large queries are processed in the background, generating downloadable files.

Jonathan acknowledged the need for such capabilities and shared their experience of using Python generators to handle paginated results efficiently while avoiding high memory usage. He also discussed the potential for asynchronous callbacks to manage exceptionally large datasets.

Corey elaborated on GeL's current functionality, which provides various file formats for download, and suggested integrating similar features into the Data Connect API.

Leon noted that extending the API may necessitate updates to client implementations to support these enhanced features.

2024-10-30: GA4GH tech team overview

Chair: Jonathan Fuerth, Miro Cupak

Attendees “Name (Affiliation)”: Jimmy Payyappilly, Beatrice Amos, Marc Perry, Takudzwa Musarurwa, Jose M Fernandez, Mike Warfe, Nara Sobreira, Heather Creasy

Link to meeting recording: https://us02web.zoom.us/rec/share/uXchHZXIDlj0D-5-7l5qUaq3XlpIPRCcl8Is6UHQzUdIemj7ZCvndGPN0H1o6PR1.G24z3llyDX8hnSmT 

Meeting summary:

Jimmy, the GA4GH tech team lead, joined our meeting to discuss their ongoing initiatives and potential collaborations. Jimmy provided an overview of three major tech team projects:

1. Internal Implementation and Starter Kits: The team has developed a system for managing closed-access datasets, successfully tested with the 1000 Genomes Project, and is expanding to datasets like UK Biobank.
2. Harmonised Test Bed Infrastructure: A centralised registry validates and displays compliance with GA4GH standards. Current focus is on web-based standards like Refget, RNAget, and DRS, with plans to extend to non-web standards and private test bed environments.
3. DOI Minting for GA4GH Artefacts: In collaboration with Crossref, the tech team is enabling DOI assignments for standards and related artefacts, aiming for a user-friendly submission system by the next Connect meeting.

Jimmy also elaborated on plans for

• improving compliance suites,
• supporting GA4GH-verified implementations, and
• fostering collaboration with workstreams.

He emphasised a flexible approach, inviting input from product teams to integrate their needs into the tech team’s roadmap. Discussions with Miro and others addressed the practicalities of integrating external implementations into the test bed using secure, automated mechanisms.

—

Nara suggested a review of the roadmap as a potential topic. She proposed discussing priorities for the Genius project and identifying tasks to recommend to Jimmy for prioritisation. Miro and Beatrice agreed that reviewing the roadmap could be useful but noted that the current stable state of the standards, with limited feedback from implementers, presents fewer new development opportunities. Beatrice emphasised bandwidth constraints in the team for any new features, though additional support from Jimmy might be explored through a prioritisation process.

Nara also asked about the necessity and frequency of meetings, suggesting that maintaining contact was valuable even if immediate agenda items were sparse. She shared insights from a recent NCPI meeting where NIH participants discussed standards such as FHIR and GA4GH APIs, highlighting opportunities to promote Data Connect adoption in broader contexts. She mentioned challenges faced during the Matchmaker Exchange project and suggested leveraging user feedback to drive improvements and uptake of Data Connect.

2024-09-25

Chair: Jonathan Fuerth, Miro Cupak

Attendees “Name (Affiliation)”: Beatrice Amos, Marc Perry

Meeting summary:

Meeting summary:
The main focus  of this meeting was planning next steps for Data Connect and the Discovery workstream, post-Plenary. Part of what we’d like to do is focus on improving communication, and coordination. We also discussed ways on how to gather feedback from the broader community, the need for better organisation and advocacy for less prioritised standards, and lessons learned from the recent GA4GH Plenary, where booth setups were considered less effective.

Key points included future outreach efforts, addressing workstream interoperability, and potential restructuring to ensure balanced attention across various standards. We also touched on how the team could improve collaboration and effectiveness moving forward.

Connect feedback:
Plenary - a lot of emphasis on equity, inclusion, and indigenous genomics. BA mentioned the importance of face-to-face meetings in fostering more meaningful interactions. The next plenary was announced for Uppsala, Sweden, in September 2025.

Future of Data Connect and GA4GH standards:
BA raised the question of how to manage "successful" standards like Data Connect, which have been widely adopted but don't require constant updates. The group agreed that such standards could be considered "stable" rather than in "maintenance mode." Miro noted that while there is interest in further developing these standards, bandwidth and resources are limited. We discussed how to maintain community engagement, support Driver Projects, and ensure the active use of these standards without the need for frequent meetings.

Marc added that, as a "metadata Nazi," his role often involves enforcing metadata standards across projects, highlighting the ongoing challenge of ensuring consistent use across different initiatives.

BA emphasised the importance of engagement, including surveys and targeted outreach, and shared her recent discussions with discovery workstream leads about their roles and the need for better communication. While Beacon has strong momentum due to funding, other standards within the Discovery workstream need more advocacy. Miro echoed these concerns, suggesting a reorganisation or restructuring to better support the other standards.

2024-08-28

Chair: Jonathan Fuerth, Miro Cupak

Attendees “Name (Affiliation)”: Beatrice Amos, Marc Perry (UCSC), Heather Creasy, Mike Warfe, Mike Conway, Ayush Garg

Meeting summary:

Please subscribe to the GA4GH calendar: https://www.ga4gh.org/what-we-do/calendar/ to stay updated on meetings.

Logistical and planning issues for Connect:

Miro, Ian, and Jonathan, all core contributors, will be unable to attend Connect in person. Beatrice to check with Amreen about Plan B.

The value of showcasing Data Connect implementations to a new audience at the Plenary booth without key contributors present prob not effective.

The roadmap includes three key objectives:

1. increasing adoption of Data Connect,
2. improving interoperability with other G4GH standards and external ontologies, and
3. releasing version 2 of Data Connect with enhanced features like Beacon integration and Schema Registry connection.

Need feedback to refine this roadmap, ensuring it addresses current and future needs.

There is an ongoing need for community engagement and adoption strategies across various GA4GH products. Perhaps the tech team could focus on developing metrics to track adoption. BA check with Jimmy^[a]. 

Interoperability with non-G4GH standards like HL7 FHIR very important. We also need to strengthen collaborations with Driver Projects, like the Autism Sharing Initiative and Genomics England.

We need to better track and support implementation efforts.

2024-07-31

Chair: Jonathan Fuerth, Miro Cupak, Ian Fore

Attendees “Name (Affiliation)”: Marc Perry (UCSC), Kathy Reinold, Heather Creasy, Mike Warfe, Mike Conway

Apologies: Beatrice Amos (vacation)

Link to meeting recording: https://us02web.zoom.us/rec/share/Qlkm7AUocxr6jv5Gf0RCs6KcU07-UmEvrYkPDadnnxx-SPsgrZfezhHskStDzI-o.WChddyWxtMy7BHs1 

Meeting summary:

Connect Session Planning:

Kathy mentioned that most team members would not attend in person, but a few representatives would be present. Miro will attend and DaMaSC will have multiple volunteers present to help out.

Miro will create one-pagers to showcase their projects, particularly focusing on Data Connect and related initiatives.

Kathy mentioned the formation of two focus groups within the DaMaSC team to accelerate their initiatives.

Miro mentioned the NIH grant that needs a roadmap formulation across work streams and subgroups, with a tight deadline before the upcoming connect meeting.

The team discussed the implementation challenges and requirements for foreign keys in Data Connect, highlighting the need for referencing standards that are not always defined in JSON Schema. They brainstormed potential solutions for referencing various data elements, ontologies, and units of measure that may fall outside the scope of JSON Schema. The discussion also touched on specific use cases from the Cancer Data Aggregator and the necessity of capturing all requirements via CADSR, including external reference pointers to other ontologies.

The team brainstormed use cases and external groups to engage for the Data Connect session, including suggestions to involve the Australian Data Commons, NCPI, and African groups. They also discussed technical aspects, such as the need for column definitions to reference external standards, the usage of JSON schema, and handling units of measure. The importance of clear data definitions to simplify data harmonisation was highlighted.

Will create a shared document for session proposals and the roadmap.

2024-06-26

Chair: Jonathan Fuerth, Miro Cupak

Attendees “Name (Affiliation)”: Beatrice Amos, Augusto Rendon (Genomics England), Ian Fore, Mike Conway, Jose M Fernandez (Barcelona Supercomputing Centre), Marc Perry (UCSC), Nara Sobreira

Link to meeting recording: https://us02web.zoom.us/rec/share/k4FIojbxau6C-L2EjnEYyhk8BA2n9wnWcsVnAYgr3IAJD9_KJXn1zs0PCULB8nLy.cgD7Ksh4Di7T_dmp (Start at ~11.45 after intros)

Meeting summary:

Connect session proposal discussion:

Two proposals from Data Connect:
1) The first proposal will focus on the often overlooked aspects of the service registry and service info, which have not changed in six years despite significant adoption. The goal is to survey current usage and gather community feedback to potentially update these standards with new fields.

2) The second proposal will address the updated connector, aiming to understand its adoption across different groups and gather feedback for future improvements and roadmap planning. Miro will be looking for presenters and participants for Connect.

Ian asked whether there is an existing prioritised list of enhancements and features or if a new one needs to be created. Miro mentioned that a consolidation of GitHub issues was done a few months ago, providing a starting list that includes planned items and ideas from past discussions, such as incorporating GA4GH Passport functionalities into Data Connect.

Ian asks if the list is ready for review or if more items need to be gathered. Miro said that while they have an initial list, they are open to new suggestions. The goal is to refine the list and potentially create a concrete plan for a revised version of Data Connect, depending on community interest and available bandwidth, particularly noting that DNAstack’s resources are limited.

Miro would like to expand the list of active contributors and we probably need to clarify what makes someone an active contributor, whether through feature development, implementing Data Connect, or addressing specific problems. Need input and wider usage from the community.

Augusto (Genomics England) explained that their team has not yet committed to implementing Data Connect on a large scale as it is a new initiative for them. Currently, architects and other relevant team members are evaluating Data Connect to ensure it meets their requirements. He proposed that they could provide feedback based on their findings.

Augusto noted that Data Connect appears generic enough to handle their backend queries effectively, which is advantageous. They aim to avoid dependency on a third-party API like Dremio's, which they use as a SQL engine on their data lake. Instead, they prefer to utilise Data Connect, allowing them to standardise their API usage and communicate this standard to vendors and users.

While initial impressions are positive, Augusto indicated that a more detailed analysis is needed to confirm Data Connect’s suitability for their purposes.

Ian commented on Miro’s session proposal about the service registry. The service registry is not well-known within GA4GH, even though it is frequently needed to identify existing GA4GH implementations. Many people are unaware of the registry's existence, leading to underutilization.

He said that awareness and use of the registry would increase if more people knew about it. While it might not be ideal to enforce mandatory registration, there should be a motivation for users to register their services. GA4GH should promote the registry more actively and consider making it more prescriptive.

The Connect meeting could be an ideal platform to raise awareness, but the session should be positioned as a strategic priority for the entire GA4GH community, rather than just a specific work stream topic. This broader exposure would help integrate the registry’s use into the GA4GH ecosystem more effectively.

Miro shared that the GA4GH tech team had received feedback from a community member facing issues with the Service Registry reference implementation. This led to a conversation about whether the GA4GH tech team should take over its maintenance, an idea Miro supported. He noted that DNA Stack extensively uses the Service Registry and has a private fork with substantial updates. Miro proposed integrating these updates into the public reference implementation to benefit the broader community.

Over time, the repos have fallen out of sync, and Jonathan said that updating the reference implementation with DNA Stack’s enhancements would be beneficial.

—----

Jonathan demos a new feature that was recently launched, built on Data Connect. It might align with future standards, even though it currently doesn't combine with any standards. The feature addresses challenges faced when scaling Data Connect usage, especially with large datasets like whole genome sequence variants. Users often run expensive queries or struggle with complex queries. The new feature, called "questions," simplifies the querying process for large tables.

He showed the more advanced "questions" feature, which enables grouping and aggregation beyond the capabilities of the basic interface. By adding a question to the collection, he showed how to group data by municipality and sum totals of cases, making complex queries more accessible.

This feature simplifies the process of writing complex queries, enabling users to filter, join tables, and perform aggregations without needing advanced SQL knowledge. It provides an easy-to-use interface for retrieving accurate data and mentioned a common use case: sharing aggregated data, such as at the county level, where row-level data cannot be freely shared due to privacy concerns. The "questions" feature helps in providing secure, aggregated data access without exposing individual records.

The ease of saving parameterized queries using the new feature:

Jonathan says these queries can be named and stored in a repository, a bit like prepared statements. When executed, users provide arguments, and the data returns in a consistent format as defined by Data Connect.

The idea of having a shared query definition across different data sources was discussed. This would allow similar logical results to be generated even if the underlying schemas differ, such as querying for all paediatric cases. While the formal interface for these parameterized queries has not been defined, there is significant value in standardising parameters and result formats.

Augusto mentioned GE - the goal is to move from large, monolithic tools managed by a central team to community-managed, reusable queries, facilitating low-code application development.

Miro touched on issues related to re-identification attacks and the introduction of query restrictions. The Beacon team had implemented protections, such as limiting queries around specific genomic locations to a maximum of five, to mitigate these risks.

2024-05-29: HDRUK, Data Connect and Passports

Chair: Jonathan Fuerth, Miro Cupak

Attendees “Name (Affiliation)”: Beatrice Amos, Mike Warfe, Matt Barlin, Heather Creasy (NCI), Jose M Fernandez (Barcelona Supercomputing Centre), Martin Kuba (ELIXIR-CZ, Masaryk University in Brno), Binam Bajracharya (UChicago) Marc Perry (UCSC), Rachel Tesfaye, Loki Sinclair, Marin Kuba, Alastair Thomson (ARPA-H), Andrew Russette, Augusto Rendon (Genomics England), Binam Bajracharya, Corina Antonescu, Ann Novakowski (sage Bionetworks), Heather Creasy, Heidi Sofia, Jaime Delgado (UPC Barcelona TECH), Jessica Seegobin, Jonathan Lawson, Marc Perry (UCSC), Keren, Marion Shadbolt (CanDIG-UHN), Mike Warfe(NCI), Nara Sobreira, Renan Martin, Taku, Tom Conner (Broad), Corina Antonescu, Phil Q

2024-03-27: Canceled

Chair: Jonathan Fuerth, Miro Cupak

Apologies: Beatrice Amos

2024-02-28

Chair: Jonathan Fuerth

Attendees “Name (Affiliation)”: Beatrice Amos, Mike Warfe, Matt Barlin, Heather Creasy (NCI), Jose M Fernandez (Barcelona Supercomputing Centre)

Apologies: Miro Cupak

Link to meeting recording: https://us02web.zoom.us/rec/share/8ECNzf62xV_k-1arYxJndAizNtoQflrYIGNAg_K2Yb5F1WloUtjcrvBkhbB9Pe3O.iMAgCOQMERRfAYE6 

Meeting summary:

Isaac mentioned various initiatives and projects related to Data connect and authentication: the development of Passport support for Data Connects, funded by the UK Research Council. There was an active project to deploy a proof of concept (POC) by March 2025, focusing on federated cohorts and using GPU standards.

The discussion also touched on the implementation of authentication and authorization within Data Connect systems. Next meeting - will compare notes on implementation strategies and potentially updating specifications to provide more guidance on this aspect. BA to add to next meeting agenda.

JF asked about the necessity and implementation of authentication at various levels of data access.

A mention of authentication as optional, following tiered access recommendations, including public, registered, and access-controlled tiers. JF would like to delve deeper into Passports and authorization topics in future meetings.

Isaac explained HDR implementation -  data access tiers, authentication, and authorization processes within a national-level effort led by HDR UK. There was discussion on how this initiative could alleviate concerns, particularly among legal teams, regarding adopting new authorization methods for accessing protected data. The involvement of NHS England in adopting similar solutions is a positive development.

Jonathan said one notable gap on the roadmap was the need for clearer specifications regarding authentication and authorization within Data Connect systems.

Mike mentioned federated analytics and federated queries, with an example on different approaches being explored, including aligning data schemas, federated learning, and query templates across multiple sites. These approaches aim to address challenges related to data interoperability and privacy preservation in federated settings.

2024-01-31: Use case presentation

Chair: Miro Cupak, Jonathan Fuerth

Presenter: Takudzwa Nyasha Musarurwa

Attendees “Name (Affiliation)”: Nara Sobreira, Beatrice Amos, Tim Beck (Nottingham, ELIXIR-UK), Jose M Fernandez (Barcelona Supercomputing Centre),Nara Sobreira, Taku, Heather Creasy (NCI), Kathy Reinold, Heidi Sofia, Barlin, Mike Conway (NIH/NIEHS), Mike Warfe, Alexander Kanitz, Ian Fore

Link to meeting recording: https://us02web.zoom.us/rec/share/LyQkhJZGcPVR7AQTU2CEQ35yNTEJFqvOTb1EUd5J2MlDKR3xyztYGWIQsChdbWc_.TcKCJ-5TUikVXpZi 

Meeting summary:

Taku introduced a pilot platform developed for open data science. He demonstrated a proof of concept for data sharing and computation across various African facilities. The platform aims to implementGA4GH standards, utilizing 1000 genomes data.

The process involves splitting trim files into four bases distributed across nodes in South Africa, Mali, and Uganda. Users can query data from any node and create a Data Connect. The Data Connect was set up on the ellipse node, containing information about files in other nodes. Users receive IDs for the files, submit them to the WES, and process the data through a pipeline, generating a multi QC report.

The initial steps included requesting IDs, identifying nodes hosting the data, and obtaining a passport token for data access. The token, along with user ID and available visas, is sent to the password broker. This token allows access to data protected by passport, ensuring restricted access.

After obtaining the passport token, users submit their IDs to the SFA repository, receiving the location of the data objects. The process continues with performing computations, obtaining output locations, and finally sending output locations to the WES Server for result combination into a large QC report.

The presented architecture is part of a pilot project, with plans to migrate to services running on gen 3, DNA stack, and the interior for a more production-ready setup. The goal is to enhance the platform's functionality and readiness for broader usage beyond the pilot stage.

Live demo

2024-01-10: Roadmap review and refresh

Chair: Miro Cupak, Jonathan Fuerth

Attendees “Name (Affiliation)”: Beatrice Amos, Jose M Fernandez (Barcelona Supercomputing Centre), Ian Fore, Nara Sobreira, Taku, Heather Creasy (NCI), Kathy Reinold

Link to meeting recording: https://us02web.zoom.us/rec/share/ErTzqtN1z4OLXHcXe0n19X0zcwzuilWPNqK5mrC-gnRXEft5rAtijEG8QEuXD9go.ExrD8oPTY8puUTlz 

Meeting summary:
Ian emphasised the importance of targeted discussions and specific examples rather than general calls, suggesting a more focused approach for future sessions, particularly related to DRS IDs and Data Connect integration. He suggests the need for more concrete examples and resolutions rather than general discussions. How do we proceed with specific issues and how do we address them within the workstream or do we do this through broader discussions?

In chat:
Jonathan Fuerth:

Open Data Connect tie-ins/collaborations with other work streams: Data Connect as search for DRS (Cloud), Data Connect’s data model/schema repository needs (DaMaSC), Data Connect getting more specific about Passport authentication/authorization (DURI)

Data Connect has been mentioned in 7 4 open DRS issues: https://github.com/ga4gh/data-repository-service-schemas/issues?q=is%3Aissue+is%3Aopen+data+connect - could we pick one and propose a session to make progress (maybe even resolution?) on one?

Taku: https://github.com/elwazi/elwazi-pilot-node-install 

Discussion on the connection between Data Connect and Elwazi project. Taku/Ian delved into his practical example - discussed the use of phenotype data and its connection to DRS IDs in data queries.

Ian/Taku discussed identifying specific issues, seeking ways to link various data points, and considering a standardized format to enhance data retrieval efficiency between different projects.

Cohort building and standards alignment:

• Cohort building and discovery workstreams overlap, need one place to pull together overlapping topics.
• Initiative aims to connect sessions and brainstorm ideas for the coming year with Ian, Francis and other group members.

#END