GA4GH Future of VCF

Minutes and Actions 2021

These are the minutes for the File Formats meetings, a subgroup of the GA4GH Large Scale Genomics Work Stream. For further information, please visit the GA4GH Work Stream page.

Table of Contents

Meeting Protocols

Agendas, Minutes and Actions

2021-12-08: VRS/VCF Alignment

2021-11-29: VCF Scalability

2021-11-01: VCF Scalability

2021-10-04: VCF Scalability

2021-08-09: VCF Scalability

2021-07-12: VCF Scalability

2021-06-30: VCF/VRS Alignment Meeting

2015-06-14: VCF Scalability Reboot

2021-05-18: Fifth Meeting 2021

2021-04-15: Fourth Meeting 2021

2021-03-31: VRS/VCF Alignment Meeting Cancelled

2021-03-23: Third Meeting 2021

2021-02-22: Second Meeting 2021

2021-02-03: VRS/VCF Alignment Meeting

2021-01-25: First Meeting 2021

2020-01-06: Joint VRS/VCF Meeting #4

2021-MM-DD: Template

Previous Meeting Minutes

Meeting Protocols

• Please note that by participating in meetings, attendees agree to adhere to the GA4GH Standards of Professional Conduct.
• Meetings may be recorded for note-taking purposes. Recordings will be All deleted within three months of the meeting taking place.
• Dates should be specified in the international format yyyy-mm-dd

Agendas, Minutes and Actions

2021-12-08: VRS/VCF Alignment

Chair: Daniel Cameron, Larry Babb, Alex Wagner

Attendees “Name (Affiliation)”: Rob Davies (Sanger), James Stevenson (Nationwide Childrens), Tim Cezard (EVA), Anders Leung (Broad), Neerjah Skantharajah (GA4GH),

Meeting recording: https://us02web.zoom.us/rec/share/sLAAqIIC-ukS8yQGxzFmuPHH48__HdJnB9t1dfSbjYzvYBZ6Czf3qbDu5ijyHMD_.SWq0uCtoLv_GAzx8

2021-11-29: VCF Scalability

Chair: Albert Smith

Attendees “Name (Affiliation)”: Anders Leung, Tim Poterba, Daniel Cameron, Robert Davies, Gisli Masson,  Sarah Hunt, Tim Cezard

Meeting Recording

2021-11-01: VCF Scalability

Chair: Albert Smith

Attendees “Name (Affiliation)”: Jonathan LeFaive, Louis Bergelson, Gisli Masson, Olen Schulz-Trieglaff, Tim Cezard, Robert Davies, Tom White, Yash Puligundla, Chris Chang, Lee P, Tim Poterba, Anders Leung, Roman Valls Guimera

Meeting Recording (to be linked)

Minutes

AS: Want to get our bearings on where we are in terms of the landscape analysis. Next step we want to get to is in terms of benchmarking. It was pointed out during Connect that there are things on the landscape analysis that can’t be benchmarked. So I want to get to the point of what and how we are going to benchmark.

Lee P: The scalability piece is rather interesting. We have a lot of customers asking how do we store VCF files and how do we query them interactively on the cloud. How are they retrievable in a scalable way? That’s very interesting to our clients.

AS: How did you get to know about us/

LP: I usually go to the cloud WS ones, but it just so happens that I was able to attend this one today. I’m representing a couple of different hats from the Amazon side. We’ve been engaged with GA4GH largely from the Cloud WS, but we’re starting to fan out a bit more into the different storage formats.

AS: Do you have any recommendations that you give to your customers?

LP: From the VCF and variant store side, we’re still collecting information. A lot of the customers I talk to are stuck in the phase of trying to make sense of their VCF in the first place.

AS:

LP: Is VCF around here to stay or is it going to be transformed into something different?

AS: You’ve asked the big question. It’s to be determined. What we are actually trying to accomplish and our goals will hopefully be helped along through the landscape analysis. We haven’t determined how many issues we want to solve. Some of the parameters we set up are not all mutually compatible and there are trade offs. It’s still a big question. We have to get to hte testing and figuring it out. We only have two or three ideas on the table that are closer successors to VCF. We will come back to that.

TP: To ad don, in some respects VCF is already a dead format. The groups doing the larger scale analysis have not been using VCF or at least have not been using it in a way that it’s meant to be used. One of the purpose of this groups to come to an agreement of how to represent data and how to share it between ourselves.

LP: You mention the folks who haven’t been using VCF are using an alternative. What is that?

TP: Depends on the group, but for example, SVCR. All of Us is another good example. They are creating their own variant store on top of Google Big Query.

AS: If we’re going to head anywhere as a group, having some sort of representation that can be transferred is something that we might be able to accomplish.

Landscape Analysis Review

AS: We have prioritized the use cases What does the group/community think we should focus on? We could probably reorganize this so we can insert a statement of purpose for the group.

TP: In favour of moving forward with some concrete benchmarking task. Even though we won’t all agree on what to benchmark, once we make it concrete people will be more vocal. Even proposing something wrong that is concrete will generate more discussion.

AS: In what sense do you mean defining what the benchmarking are?

TP: Starting with the datasets are more important than starting with the metrics going out. We may not get to a point where we all have the ability to generate metrics for the dataset. Things are very abstract at this point.

AS: Personally, I would recommend that Oliver and Thomas have been big advocates of having simulated data. Tom, i believe you’re connected to what Jerome might be able to do.

TW: Yeah, I’m familiar with the repo that Jerome created.

AS: Has he created a repo for general simulation, or intended with simulation of data that can be used in the context of this group?

TW: I think it simulates genotypes using his research and the other fields it gerneates in a very simple minded way. There’s more discussion there, but basically it can expert VCF files

RD: Does it simulate like... values?

AS: I think there was some interested or desire in having a repo for this group. We probably should start and make some parameters. Making some simulated datasets. Don’t know what that might be at the moment. We either need to pull this in or start simulating some datasets right there. Also need to speak to Jerome. There’s also been some talk about working with the UK Biobank data. It was a bottleneck when we talked about it before, but now not so much.

GM: On november 11th there would be 150K whole genomes available. We generate the VCF… 3terrabytes.

AS: What does decode do internally for representation and querying.

M: We generate those data and then use it as a starting point for our … pipeline. Where we input it into remaining cohorts. Then we do the actual association that’s done using a … format. We created our own format. Before beacon came out. Basically a haplotype compressed format. We managed to get the imputed data down to 10 terr or so. Then we won’t really need to look at VCF much more. This is write once read once.

AS: From the perspective Is there interested If there was a better standards, would you be comfortable working with that or do you think internal advantages are better than what the community might be able to come up with.

We do transfer data into Bgen format. Thenw e run association on that. To what extent, there is information lost when doing that. Generally at this sale, 12 people haven’t been looking all that muhc at those data. Not on large scale.

AS: There will 150K whole genomes available at UK biobank. There is an application associatedwith this group. I don’t know if anyone knows about this. I have an account and have logged into the workspace but I’ve never attempted to do anything. So I don’t know the full overhead of what goes into that.

LP: My only comment is that I know of it but I dn’t have may details specific to it. But I could connect interested parties to the right people at amazon.

AS: If this group was trying to do benchmarking ont he scale of what’s going on, would there be support from Amazon to support computer storage?

LP: There's a possibility but it’s a question of what kind of support. Our easier mechanism are to provide AWS credits. If there’s interest in that I can connect you to the right folks. Just need to figure out what the use case is.

AS: I don’t know if I have a specific proposal, but picking specifically what to benchmark we should make a repo and start a parameter for simulated dataset. I would also like to push GA4GH to make sure that the group has access to all the exomes and whole genome sequences data form UK Biobank.

TP: In what form are they available?

GM: It’s VCF

TP: Is it multi-allelic with full schema? Assuming not, so this might factor into benchmarking. If we’re using data that’s already been scaled down, that might not be the best simulation target.

GM: i think they are passing on the VCFs as we return them. Cannot exactly tell you what fields there are, but we’re trying to be standard. We binarize most markers...not sure will get back to you.

AS: Since it will be available soon, we can look. The current app doesn’t cover that data. We’d have to get our app extended to that data. GA4GH may need to pay for that. If we could have a report at our next meeting on what’s actually there, we can discuss that. Will be a to do for myself to look at what’s going to be at UK BioBank. Need to have a good report of exactly what is there. LB has advocated for “how do you merge VCFs” as an important use case, and we haven’t touched that.

LB: Agree that merging is an important consideration.

GM: I’m not sure if it’s mentioned in this doc (landscape) I noticed that when looking at the data on DNA Nexxus, one thing that Uk Biobank has to do is to have this sample ID different between users. This is probably a bit complicated with VCF. Not sure how they hacked that.

AS: the workaround for that is for GA4GH we have a single ID.

GM: Understand for the project, but for VCF in general. If one wants to replace IDs then it’s an issue. It’s inconvenient for BAM,CRAM and VCF files.

AS: Right. Think that’s one of the reasons that they’re preferable to V-GeN.

LP: What’s the rate of decay that we’re looking at here? Is VCF going to be gone tomorrow or is VCF going to still be around because it’s a widely known interchange format.

AS: It will never die. VCF has certain attractiveness in terms of having a full matrix and being able to read it. It’s quite useful in some ways. And there are many toolchains to support it.

RD: how big a file do you need to start seeing these scaling effects?

TP: You don’t see this atr early sample sizes. You might even see the opposite effect. It only includes sites of variation so once you pick up all the common variations (around 1000/2000 samples) you start to see it.

GM: Can reveal that after 150000 sample we are up to **** variants.

AS: Similar to our experience in TOPMed.

AS: Next time I think we need to talk about what we’re going to benchmark. Which will include a better report on simulated data and what’s there as well as what is/can be available through UK Biobank. That’s where we’ll take the discussion. We’ll organize so that we stick to the high priority items so that we can move towards testing.

LP: From the sponsorship perspective, it would be helpful to know how much you think you need. If you ask AWS to sponsor for sompute/storage we will ask how much is feasible for you. How much data are you looking to store and how much compute do you think you need?

AS: We probably can talk about this going forward.

2021-10-04: VCF Scalability

Chair: Albert Smith

Attendees “Name (Affiliation)”: Tim Poterba (Broad / Hail), Tim Cezard (EMBL-EBI), Susan Fairley (GA4GH), James Bonfield (Sanger), Anders Leung, Christopher Chang, Jerome Kelleher, Jonathon LeFaive, Robert Davies (Sanger), Tom White, Gisli Masson, Lee Lichtenstein

Meeting Recording

Meeting Transcript 

2021-08-09: VCF Scalability

Chair: Albert Smith

Attendees “Name (Affiliation)”: Alex Stuckey, Angela Page, Christopher Chang, Gisli Masson, James Bonfield (Sanger), Jerome Kelleher, John Marshall, Lee Lichtenstein, Ole Schulz-Trieglaff, Oliver Hofmann, Rob Davies, Susan Fairley, Tim Cezard, Tim Poterba, Tom White, Bimal Chuadhari, Roman Valls Guimera

Meeting Recording

Minutes

Benchmarking - Jerome Kelleher (slides to be linked)

• Use simple simulations as a way of giving rough ideas of how these methods work in best case scenarios.
• AS: What sort of field and assumptions do you have here?
• JK: Very clean genotype data. Best possible case scenario for genotype compression. This is the best case scenario. Can’t get any cleaner genotype data than this.
• Have only done 10Mb here because the variant dimension is not very interesting. Files aren’t too unmanageable.
• Also simulated some GQ data. Model where you choose one digit number - makes the data incompressible because it’s random.
• Simulations shown were basic, but can be made more sophisticated through population structure, imposing genotype error, adding other pre-genotype data
• Good thing to do would be to use a combination of evaluation metrics
• Example of a good combination of metrics: Can we access the variant data sequentially? Can I sequentially iterate through all the data haplotypes?

Q&A

AS: Since you’re working with simulated data, have you done a similar benchmark with any real data? How well do you think your model is capturing anything that’s there?

JK: What this is at the moment is a first approximation of LD and haplotype structure. If you just generate genotype data where all your sides are independent won’t get any information on LD. Will be incompressible across sites. Capturing a very good first approximation. Not capturing any sense of errors*. If our data is all error, what’s the point in storing it. Have to assume that the majority of what we’re storing is useful information. Imposing some sort of error process on top of data should give some idea of reality.

AS: How are you handling/what’s the effect of missing data?

JK: Encode missing data as -1.

OH: One way of sharing this would be through sharing the model. Tim, from the last call you mentioned AD and**

TP: Solution that can tolerate output of any of the * would be ideal.

OH: Would be good to define minimal sets you want to have in there.

AS: Point of getting all the fields and capturing is important. Huge place for evaluating through simulated data, but at some point we will have to **

JL: have you compared certain metrics from simulated against real data? Are you training your model on real data in order to generate these?

JK: Method is coalescent** recombination**. Theoretical model about how populations evolve. What I’m generating at the moment is the simplest form of how humans would grow* in this framework. What would be the simplest model that you think would be useful?

JL: Have you actually looked that your output holds up and is consistent with what you see in real data.

JK: Personally haven’t, but a large number of people have. Parameters that I’m estimating were calculated by people. Sure you could find discrepancies but the broad brushstrokes would be more or less correct.

OH (chat): I am not sure that errors in the model (or changes, additions) would result in a fold-change of the compression level. So simple would be better.

AS (chat): Agree, my thoughts in this vein making sure we don’t have inadvertent assumptions

JK (chat): Code here: https://github.com/jeromekelleher/ga4gh-variant-sim

JB (chat): I think it depends on how advanced the compression system is.  If it's just chucking data blocks into zstd then I agree.  If it's doing something very specific such as constructing trees and assigning individuals to branches (imputation style) then it could be significant.

TP (chat): One comment from our experience working with 20-150K WGS datasets at the Broad - the GT field isn’t a significant fraction of the data footprint, so compressing the GT field, even to zero, doesn’t solve the core scaling issues with VCF. We haven’t done this experiment since gnomAD v2 (20K WGS), but storing just the hard calls in VCF form was about 2-3% of the file size of the full schema, meaning the remaining 97-98% can be blamed on the quality fields like AD,GQ,PL

VCF and Zarr - Tom White (slides)

• Zarr is a format for the storage of chunked, compressed, N-dimensional arrays
• Linked paper talks about how if you store large data sets in the cloud, they are available for analysis by anyone
• Naming of file is index to chunk structure
• Each array is defined as having its own compressor and filters. Filters applied before compression takes place Can have a series of filters. Because each chunk is a separate file, they are compressed separately to other chunks.
• Zarr default compressor - meta-compressor
• Does blocking and compression itself
• Arrays can be written in hierarchical groups with each array havings it’s own type, shape, chunking and compression settings
• There are a multitude of backend stores
• How does VCF data map to Zarr? Pioneered by Alistair Miles. In the sgkit project, we have refined the process a bit. Store each field in a separate Zarr array. Single group is a single VCF file. For each of the different types of field map those to different shapes of Zarr array.
• Have a few benchmarks run on 1000 genomes data on chr22
• Numbers are comparable with numbers Jerome just showed us.
• Advantages to this format is that it is partitioned by variant and sample.

Q&A

AS: Lot of things great from an analytical perspective. But is analyzing samples quickly part of our mission?

JB: Looking at data compression size as well as access pattern. The size aspect has two components. Some of these fields are not linear, they grow exponentially. Can see that with good compression algorithm...if we get large thing back in memory that could be an issue. Is it possible to access as sparse array**

TW: It’s not. When you decompress a Zarr chunk it will be the size of its dimensions. Is a Zarr v3 that will have capability of plugging in that in a more natural way, but for now you’d have to add it on yourself.

TW: LA is a rep that would work with any format so it would plug into this kind of format.

JL: Within the grid, is there an option to specify for array whether the data is stored**

Could you elaborate on the meta-compressor?

TW: The shuffle will shuffle the order of the bits so you store the most significant, the second most significant etc.**

**

AS: When we talk about compression it’s important to think about all the fields. Comes back to simulated and keeping that in mind. Have a little bit of caution/though we shouldn't just optimize to gstk.

RD: Need to think about what is the most common use case for VCF files.

JK: Agree but it does exclude the things that we can’t do with VCF.

EG: What are the downstream applications of the tags that are being kept on the VCF. Is the PL field often used in downstream use?

AS: Important question. In our own work with TOPMed deal with that issue by removing some of the fields. But there are potential use cases where people do care about these things.

EG: Probably need some kind of confidence estimate. Should be careful about the downstream utility.

Chat:

RVG: https://aws.amazon.com/blogs/aws/s3-glacier-select/ … Zarr does not seem to be listed/supported nor does in AWS Athena (Parquet, ORC, JSON are). Why is Zarr considered a “HDF5 for the cloud” given those “cloud native” integration gaps? Benchmarks showing on how Parquet does not escale?

JB: The design of Zarr sounds good though.  Has some similarities (because it's all obvious frankly) with CRAM 3.1 in filters + codecs.  Even the striping shuffle :)

JK: Columnar data for the win @James!

LL: Does Zarr offer 8-bit width?

JK: Yeah, supports all numpy data types (8, 16, 32, 64) integer + floats

JB: Column store loses compression between fields, eg when we always have A=foo when B=bar for columns A and B, but that's not so common and we get selective access very efficiently if we do columnar access.

JK: Another nice property is we don't need to make multiple copies of the file - not need for a GT-only version, because you only pull down data you are interested in.

JB: I think we need to dedicate time to actual research too.  Eg maybe we discover PL takes up 50% of our total storage, and that it is needed, but how many significant bits does it need?  Can we quantise it?  Lots of questions, but first we need the data encoded of course.

EG: it's crucial to understand how useful these entities are. also, will they be used in a future setting where many genomes are just represented as whole assemblies?

OH: Tim - to a point. If we have some callers that rely on fields that absolutely do not compress maybe we need to reconsider those callers.

TW: To answer Jonathon’s question: Zarr does allow column-major/minor representations (see order at https://zarr.readthedocs.io/en/stable/spec/v2.html)

TP: @Oliver - The proposal I presented last time (local alleles and reference blocks) can represent any schema from input GVCFs with linear scaling in the number of samples. I don’t think there’s any field which simply ‘does not compress’, and I am convinced there are solutions which do not sacrifice flexibility.

AS: We use the tags for filtering, then generally throw all that away can convert to another format

OH: @tim All the better - but thinking of the likes of FASTQ data switching to 4 quality values and we all just adjusted.

CC: re: sample-extraction benchmarks, we might also want to distinguish between extracting e.g. 1000 samples scattered across the full range of indices, vs. 1000 consecutive samples since they’re from the same subgroup and the samples were sorted on that dimension.

TP: I definitely think there are discussions worth having re: standardizing on fields that compress well, but I think this is a secondary concern after settling on a solution with linear scaling. Because once we do that, we’re really discussing constant factors.

JK: Great point @Christopher - would this be covered by the sequential sample access + random access benchmark I suggested?

OST: I think the problem is also that many variant callers tuned to provide the maximum information for single variants or small cohorts. Not all of these fields will be useful or reasonable to keep at 1M samples

CC: @Jerome Kelleher, it probably is, I guess I’m just trying to be precise about what that benchmark should cover.

2021-07-12: VCF Scalability

Chair: Albert Smith

Attendees “Name (Affiliation)”: Susan Fairley (GA4GH), Alex Wagner, Oliver Hofmann (Uni Melbourne), Lee Lichtenstein (Broad), Jonathan LeFaive, Kirill Tsukanov, Leonhard Gruenschloss (Centre for Population Genomics), Patrick Cummings (Broad / Hail), Michael Franklin (Centre for Population Genomics), Rob Davies (Sanger), Tim Cezard, Tim Poterba (Broad / Hail), Tom White, Christopher Chan, Gisli Masson, James Bonfield (Sanger), Neerjah Skantharajah (GA4GH),

Minutes

SAV File Format

• Savvy Paper: https://doi.org/10.1093/bioinformatics/btab378

Presentation:

• SAV stands for Sparse Allele Vectors
• V1 was a proprietary file format that no one was used to
• V1 only supports GT and HDS (Haplotype Dosage) FORMAT fields
• Version 2 extends BCF spec and support everything that BCF and VCF supports
• BCF typed value vs SAV typed value
• Positional Burrows-Wheeler Transform

• Initially proposed for binary data, but will work for any data with repeated values in columns*

• Sparse vectors are efficient both in terms of size and speed
• PBWT produces smaller files at the expense of slower de(serialization)
• Applying PBWT only to common variants provides a compromise
• S1R - R-tree index for genomic sequences

• Bottom up layout
• Fast/low memory
• Modified virtual offset
•  Can do slice queries in addition to genomic queries
• Supports any sort order (e.g., reverse sort)

• Encoding Phase Information
• BCF combines phase and genotypic information

• (allele +1) <<1 phased
• Adds unnecessary processing in cases when phase is irrelevant
• Removes sparsity (i.e., zeros)

• File level phasing status with phasing header

• ##phasing=[partial I full I none]

• Separate FORMAT field (PH) to store phase for each genotype
• “Number” Field → few types of data that have numbers that are not defined/specified in VCF spec
• Use case for “Number”? Is it only for documentation for human readers? Or are there other use cases?
• Discretization

• Phred-scaled integer*
• Linearly scaled integer*

Questions/Comments:

• AS: How much compression can the SAV format achieve?

• Comparisons done between 2000, 20,000 and 200,000 (deserialization/minute)

• OH: What fields do we want to have in VCF?
• AS: Case to be made to simulate with whatever data and benchmark against real data. Topic we can explore next month.
• RD: The number field is needed to find out how many values you have in your vector. Looks up number of genotypes to tell you what number you should have.
• JF: With VCF and BCF you can figure that out in the header.
• TW: If you’re reading just some of the samples without decompressing all the data?

• JF: Yes. In the case of a full BCF you can more or less discard the extra data. With sparse vector since you're only parsing non zero values, subsetting out a single sample is really fast. When talking about dense datasets, will encounter same problems as with VCF or BCF.

Scalable Variant Call Representation

• Slides: https://drive.google.com/file/d/173eaasA3sFykLYKFtpRBubFRYrloLRgW/edit 
• SVCR is a proposed representation to replace project VCFs
• Representation is the schema and what the fields mean, Format is the layout of data as bytes
• VCF: connotes both representation and format

• Example: VCF (text) and BCF (binary) are two formats for the same representation

• #1 reason for linear scaling - sparsity. No record duplication!
• “Alt” field same as project VCF
• New Field - LA (Local Alleles)
• In order to interpret LGT, must refer to LA
• LA is important for fields like AD (allelic depth) and PL (phred-scaled genotype likelihoods)
• #2 reason for linear scaling: local alleles + local information. Same number of records and same size
• SVCR is a lossless transformation of input GVCF

• It is possible to recover the input GVCFs
• Also possible to merge two SVCRs together into another SVCR. No need to return a full joint calling pipeline when new samples arrive

Questions and Comments:

• AS: When you have overlapping alleles, how do you deal with that?
• TP: Same way they’re handled in GVCFs. Rare that position of polymorphism would change from VCF to full GVCF.
• JB: I really like it - makes a lot of sense. Plus it's great to explain it this was - it's orthogonal to the file format. (CC +1)
• AS: Can you comment on what you are doing with HAIL? Having ways of passing data from one study to another - how might you accomplish that?
• TP: Don’t expect that these numbers will change depending on what properties are used. Could create a VCF 4.x that includes this representation. What would need to be included is definition of LA**
• OH: Worthwhile to take another look at this.
• RD: Seems to be some work on this here: https://github.com/samtools/hts-specs/pull/434

General Discussion

• AS: Discussion point for next meeting - benchmarking
• JB: Building a *simulator* rather than *simulated data set* may be valuable, with md5um on generated output, as distributing code is much better than distributing 100s of TB of data

2021-06-30: VCF/VRS Alignment Meeting

Chair: Daniel Cameron and Alex Wagner

Attendees “Name (Affiliation)”: Jonathan LeFaive, Rob Davies, Tim Hefferon, Andy Yates, John Marshall

Recording:

https://us02web.zoom.us/rec/share/fzHAupfpij9ojRpbc_TWwfzt8EUrUJqd_2CbISLhc7DzMmKWAtIq9jr8FNJzE5ty.VsT60iqIFmFiHsrq Passcode: P0Z=DzRx

Minutes:

Sequence Collections (slides here)

• Aims to unambiguously represent collections of reference sequences
• What’s driving use cases?
• How do you handle sequence naming?

• You can’t give service 38 and expect 38 genome to come back, it would be checksum based identifier. It’s a misnomer of how use cases are set up.
• Sequence names themselves part of collection names of region form part of specification.

• Issue 8 details what is being considered:

• Refget checksum
• Sequence length
• Assigned name
• Topology
• “Priority”- designation of molecule type
• Alphabet used
• Molecule designation

• How do we think this will work?

• We want this specification to be extensible and allow any number of attributes to be associated together
• We wanted to be able to interrogate various different parts of the collection to satisfy the second use-case
• Order of sequences is important and can affect downstream analysis
• Some attributes could be considered optional or globally assumed
• There’s no reason the spec should assume collection type

• This solution structure could be used for any ordered system of attributes

• Intersections

• VCF: Round trip sequence verification/validation and caching headers
• VRS: sequence molecule designation and grabbing checksums for genome

• Meetings are every fortnightat 3pm BST
• Alex: discussion of different attributes you want to include. Molecular designation intended to be included. For genomic DNA is there any notion of sequence and reverse complement sequence? Any conventions on how people should use sequences?

• nothing  at the moment. Only one way to digest. If there are reverse complements, there would be another sequence payload that would understand that but nothing built in to say this reverse complement is this checksum.

• Double stranded or single stranded? It could but I haven't thought of it yet. Still human-centric.
• Thinking avour VRS v2.0. Underlying sequences, alphabet agnostic, problematic when we think of complex things. Only makes sense for certain molecular types. Transcript location for example, typically contain sequence and structure on top of sequence. Moving into fusions, a model like this will be useful for downstream uses cases and not treating as agnostic sequence but rich concept.

• Will need to go back to original refget spec, needs to be coded there. That would be simple reverse lookup. Definitely possible.
• Nested payloads, each array with associated, if you want to add another element. If all you need to do is present array length end but if you need to rewrite entire collection that’s not great. That’s how we looked at it to be additive.

• Current model for metadata payloads insufficient, goal is to let sequence collections to hold information. Problem here is it’s string digest which is fine if just about DNA but wrong when talking about other things. Need another payload that describes molecules that you found sequence checksum in. but name and identifier given will have to be linked to the instance of the molecule found (slight change).
• Plan to revise structure of objects in refget associated with what we’re doing in sequence collections? Yes, v.2 in refget.
• How does API interact with SAM and VCF? They specify all fields in headers, a collapsed reference is backward compatible breaking change.

• That’s the intersection we need to talk about. We need to address it in multiple areas. Tooling compatibility, if they don’t understand header payload then tools will fail.
• Actual design of APIs, how do you specify format you want. Headers are calculated on the fly.
• Don’t know best way best caching mechanism
• Backward incompatibility - you can still leave SQ headers in SAM files and content headers in VCF, but specify additional header for genome. Doesn’t hold a lot of value over having contigs there.
• John: in a SAM context, intention was that replace headers to refer to one header that would be incompatible with old implementations.

• Spec needs to maintain header values, would you expect VCF header to come back or BAM, or SAM? Or different data structure that is used to interpret same information from.

• Daniel: API native format which would be converted to equivalent SAM or VCF header. Simpler for implementation
• Rob: go with whatever API says.
• Can’t we be as loose with the format, we don’t have level of induction between major consumers and API

• Best place to discuss intersections?

• Reverse lookup group will do most of these changes (in refget). Haven’t had much engagement from GA4GH other than EBI and Sanger. Isn’t much appetite for this. Tim Cezard part of discussions, we can bring changes to that meeting. And bring it back here as well as touchpoint, how refget metadata endpoint will change.

VRS-VCF Translator

• Purpose to enable translation from VCF files to VRS objects and VRS objects back to VCF records
• Adds bidirectional VCF/VRS translation tools, SNVs and Indels only
• Supports multiallele records
• Resolves genotyping variation sets for now
• Re-normalizes (full-justify <_> left shift)
• Narrowly scope, could benefit additional eyes. Check out PR: https://github.com/ga4gh/vrs-python/pull/55
• Number of test cases from VCF providers, they’re stored in that PR.
• Challenge: VCF has a lot of associated sample leve and metadata that VRS isn’t equip to handle. Supports dictionary for additional attributes but can’t take everything in VCF and turn into VRS object, some of it falls in VA (separate scope). Custom fields also make it complicated.

How to specify trans phasing?

• Can and how do we represent in spec?
• How do we genotype weird things. Eg. 2 copies of blue, end up with breakpoint from blue to yellow. Whole genome duplication, then we get a SNP for each copy. In terms of phasing and genotyping, how do we describe genotype of blue/yellow breakpoint and Red SNP.
• Questions will be addressed soon in VRS. particularly with building genotype class as well as work in fusions and SVs. immediate term goals. Initial thought, for VRS we want to create structure that defines novel sequences and be able to add variation on derived novel sequences. Not philosophy to date. In haplotypes, we collect set of alleles for reference, we say where incision occurs, simplify yby calling insertion event same thing and downstream allele as same thing. Large scale - different challenge. Not as close or similar a change. Push for model where we can define structural rearrangement and use that as base or reference.
• Start an issue on hts-specs to get discussion from wider group.

2015-06-14: VCF Scalability Reboot

Chair: Albert Smith

Attendees “Name (Affiliation)”: Oliver Hofmann (AGHA), Gavin Band, Rob Davies, Daniel Cameron, Lee Lichtenstein (Broad), John Marshall, James Bonfield, Kristian Cibulskis (Broad), Ole Schulz-Trieglaff, Susan Fairley (GA4GH), Thomas Keane, Tim Poterba (Broad / Hail), Jonathan LeFaive, Jerome Kelleher, Gisli Masson, Christopher Chang, Tony Cox

Meeting Recording

Minutes

AS: VCF not optimized for population-scale variation. The solutions are often localized. It’s becoming infeasible for transfer between systems. Community needs better methods to exchange data. Goal is to have a format that is universally supported and that scales to thousands of genomes. And relatively easy to scale to other formats. There are lots of other problems/questions that come out of these larger overarching goals. There are potential implementatio

ns out there already, but how do we evaluate and what methods do we use? Multiple concerns of looking at this issue. Different people will see this problem in different ways. Want to continue to engage key stakeholders. Previously, there were a lot of good ideas on the table but never got to benchmarking. Got focused on getting access to a common dataset. Became trapped and things stalled out. Want to gather input, create a problem statement, landscape potential solutions and agree on a high level strategy. Hope to develop a standard that can be certified by GA4GH. Requirements for GA4GH means no IP and completely open. Round table of introductions.

GM: Have a similar problem. Large cohorts and we are sharing those. For example, have  cohort with 150,000 variants and I think the VCF..340 terbaytes, not convenient to move around. Foresee, having a data set that is 6 times larger.

RD: look afte VCF .Responsible for some of the code that needs to read and interoperate with htis format.

JB: Spec maintainer for SAM and CRAM, at Sanger. Was briefly involved in VCF at the start, but haven’t been since.

OH: University of Melbourne, coordinating LSG with Thomas.

GB: Researcher at university of Oxford. Tool developer. Also designed the VGEN? Format. Has to handle genotypes and simplified it by throwing away info you can’t get in VCF format.

TP: software developer, interested in this problem since they are facing it everyday at Broad. Interested in talking about not just formats, but also representation.

KC: From Broad, on cancer informatics and cloud computing side. Part of All of Us project. Question I have is How do we get data from our solution into other solutions without going through VCF files?

OS: Work for Illumina, one of the developers working on DRAGEN Hit the problem of VCF formats very frequently.

SF: GA4GH’s recent CSO.

LL: From Broad Institute. Part of the All of Us genomics. Don’t have a good mechanism to..

JK: group leader at big data institute at oxford. GA4GH veteran. Genomics API. Consumer of large scale data, need to be able to access data in both dimensions efficiently. Alos work on simulation methods. Producing representation data of dominimon sample scale.

CC: Software Engineer, been maintaining plank software tool set. Working on solutions to subset of problems we’ve been discussing today.

DC: Tool developer in structural variant space. Main role is in VCF 4.4 structural rep. Interested in how the future of vcf defines scalability and how well structural variants get represented if at all.

JM: at the university of Glasgow working in cancer clinical trials. VCF is adequate for our purposes as only working with one sample at a time. Will bring expertise in formats and specifications. Need to focus on what we are representing as opposed to formats. Need to think more about what we’re representing that is more scalable.

JL: University of Michigan, tool developer of library that support VCF.

MF: Software engineer

TC: Based in Cambridge. Adding population scale.. features to DRAGEN platform

KG: Bioinformatician

JK: What were the formats that were put forward last time?

AS: From memory, Mike Lin has worked on Sparse VCF. We also had the sav format. HAIL team at various levels with input from the Broad. Data bricks was interested in joining but they never fully got engaged.

JB(chat): From what I recall, Sparse VCF, Hail, Tachyon, Pil, maybe more. Tachyon/Pil were Marcus Klarqvist. One thing which came up originally, involving representation, was which FORMAT fields we really need. Some are very easy to represent efficiently (GT) and others are costly (some of the PL data).

OH: Testing on fairly small datasets. Mike Lin’s app had a slightly larger dataset but came to a halt because we were trying to decide which one was the right one. See what we can do with simulated data sets.

TP: From April 8th doc, started making a doc on potential scope and questions to consider.

AS: Perhaps this should be our seed document.

TK (chat): I think there was a plan for a VCF 5. My memory is that the Broad/Hail team were keen on this? I think there was even a draft spec?

RD: In response to Thomas’ note, I think there was something like Mike’s spVCF but there was also local likelihoods mentioned and that's a pull request on VCF at the moment.

OH: There were two parallel streams. What kind of changes could be made to VCF 5 to buy us time and maintain spec...and in parallel, what does a new data set and file format look like.

TK (chat): What are the use-cases we want to solve: 1. efficient storage for large populations 2. Analysis/rapid access workflows 3. Backwards compatibility with vcf 3. Extendible custom annotations

AS: Is VCF 5 where we are headed? Starter doc has the beginning of a problem statement. What are the ideas on the table? Tim brought up the issue of representation. If you were talking about something that would be sharing All of Us data do you have ideas there or are you at the table to see what we’re talking about. How might we leverage your expertise?

TP: We do have a concrete proposal which is a different representation from the dense project VCF. Internally, we have been using a matrix where the entries are records that are present in sample GBCFs. You don’t get a reference genotype that is not in an original reference block declaration and you get all the variant calls. This is the sparse matrix. To analyze this you need to do forward scanning densification. Never want to store that because it will be hundreds of terabytes. Can write up exactly what we’ve been using and send it out to the group.  Something that is lossless with respect to input of GBCF, which is nice because you can take the two datasets and merge them into one. Would only require small changes to current VCF spec.

OH: How does this work for someone who doesn’t have your tooling?

TP: Nothing restricting us from exporting this to a variety of formats, including VCF. The limitation is that others aren’t using this conceptual representation yet. The question is how do we represent cohort-level sequencing information? If one group is dealing with Pls and another is using local likelihoods that’s going to be a point where the tooling the groups have built up will differ.

AS: Get people to present where they stand as well as some of their ideas. Tim, would you be prepared to present some of your ideas?

TP: Could definitely do it next month.

AS: Muse on what we’re talking about in this conversation to frame what you might say.

JL: Everyone’s perspectives will be difficult to summarize in a few sentences. To piggyback off of the last comments that were made, a round of presentations in the coming meetings would be helpful to get everyone on the same page. Haven’t really been involved. Would be good for me and everyone else to understand where everyone else is coming from and get an idea of all the ideas and trouble points that people are facing. Everyone here isn’t on the same in terms of what is out there and what is available.

JL: If the format is to be universal, it needs to have a good and thorough specification so it can be implemented in other tools. That will be a very important goal and action item to make this universal. Will be important to flesh out the specification.

AS: Are you alluding to a space where present specifications aren’t sufficient?

JL: It’s one thing to define things in meetings like this. If any format coming out of this is to be universally implemented, then having a specification that is owned by some entity is going to be necessary for it to be universally adaptable.

OH: Key point here will also be not just having new tools...Also need some form of backward compatibility or on-the-fly conversion to enable community to move forward

JB: Interchange format of on disk/permanent format? On interchange, probably want something that’s quick to read and write.

??:Interchange and compute need ot be the same thing. Nede to be able to compute directly onto the representation.

OH: Would be good to have. Could ship a couple of terabytes, but do it once and hten it’s updated. Are you saying to make it a requirements.

JK: The scale of data we are dealing with is growing so quickly that unless we are dealing with 10 million genome sequences…

AS: Start with whatever representation you have, then how you actually slice that is a major

OH: No disagreeing but would make it harder to get everyone to agree on something

JK: Agree, just adding that perspectivecomponent of what you want to do. If you want to do a more detailed analysis of what you want to do with compute… but if you want to go there those are the two dimensions that are important.

JK: couple of fundamental operations that oculd be supported fundamentally is a godo thing to think about

GB: As a tool developer, doesn’t really care. Would want a C library.. Want to get the genotype for a sample at a particular location .Might be higher level representations as well that could be abstract enough to put into a file format. Give me this region of 10000 samples from nomad, that’s more of an API thing than a file format thing.

KC: Thinking more about access patterns than APIs. slicing, aggregating etc. with file formats feels like it’s locked in together. To break those two apart would be nice to allow for specialized implementations.

TP: Still haven’t decided what is the data model that is being put into the format. One concrete questions that everyone who has assembled cohorts.. What happens to the PL field? The representation has broken down and we need a new way to represent that information.

AS: Great point, let’s take that up next month. What are our action items. Would like to have a couple people talk about their experiences. Tim will be one presenter. Since some people are newer, to understand a little better things that were said in discussion. Can circulate minutes. Two 15 minutes would be fantastic.

JL: I can present

PL will come up no matter which way you talk about things. Also need to flesh out the problem statement. How do we do that? Is this something we develop overtime? Should we have something more formal by the next meeting?

OH: If everyone has time to look at the starter doc, feel free to add things or remove things you feel are out of scope.

GB: Isn’t the next version of the reference going to be a pan genome. How does that intersect with all this?

SF: HPRC they’re coming to the end of their project. At the moment it’s fair to say they are still developing their file format. They are progressing. At the moment they are primarily focused on how do you build a pan genome incorporating.. Still looking at construction methods, how you would go about benchmarking..planning alpha release in next few months with publication. I think it may be a little ways out before that comes to fruition. Lots of things happening in the space at the moment. Recent work in GA4GH re: VRS and phenopackets. Also trying to stay in touch with HPRC. On top of that there’s also a cohort representation group getting up and running. Variation space is extremely busy at the moment. Breaking it up into smaller problems that makes it more practical to address.

2021-05-18: Fifth Meeting 2021

Chair: Daniel Cameron

Attendees “Name (Affiliation)”: Albert Smith, Mike Lin, Susan Fairley (GA4GH), Tim Cezard, Tim Poterba, Rob Davis, Thomas Keane, Louis Bergelson, Angela Page (GA4GH/Broad), Daniel Cameron (WEHI), Zhenyu Zhang (UChicago/GDC), Neerjah Skantharajah (GA4GH/OICR), Stephanie Gogarten (TopMED)

2021-04-15: Fourth Meeting 2021

Chair: Daniel Cameron

Attendees “Name (Affiliation)”: Daniel Cameron, Rob Davies (Sanger), Angela Page, Susan Fairley, Thomas Keane, Oliver Hoffmann, Tim Hefferon, Kirill Tsukanov

Meeting Transcript

2021-03-31: VRS/VCF Alignment Meeting Cancelled

2021-03-23: Third Meeting 2021

DC: Is zoom failing for everyone, not just me?

Minutes:

RD: We could start up an alternate one….

Chair: Daniel Cameron

Attendees “Name (Affiliation)”: Daniel Cameron, Rob Davies (Sanger), Alexander Senf, aleung (Broad), Albert Smith (Michigan), Tim Cezard, Lee Lichtenstein (Broad), Tim Poterba

2021-02-22: Second Meeting 2021

Chair: Daniel Cameron

Attendees “Name (Affiliation)”: Daniel Cameron, Kirill Tsukanov, Rob Davies (Sanger), Alexander Senf, aleung (Broad), Albert Smith, Tim Cezard

2021-02-03: VRS/VCF Alignment Meeting

Chair: Daniel Cameron, Alex Wagner

Attendees “Name (Affiliation)”: Bob Freimuth, Rob Davies, Tim Cezard, Tim Hefferon, Justina Chung

2021-01-25: First Meeting 2021

Chair: Daniel Cameron

Attendees “Name (Affiliation)”: Daniel Cameron, Kirill Tsukanov, Rob Davies, Albert Smith, aleung (Broad)

2020-01-06: Joint VRS/VCF Meeting #4

Chair: Daniel Cameron, Alex Wagner

Attendees “Name (Affiliation)”:  

2021-MM-DD: Template

Chair:

Attendees “Name (Affiliation)”:

Previous Meeting Minutes

2018 meetings minutes were mixed with the File Formats 2018 minutes. File Formats 2018

2019 Minutes available here

2020 Minutes available here

Further discussions also on the File Formats Minutes