GA4GH File Formats

Minutes and Actions 2021

These are the minutes for the File Formats meetings, a subgroup of the GA4GH Large Scale Genomics Work Stream. For further information, please visit the GA4GH Work Stream page.

Table of Contents

Meeting Protocols

Agendas, Minutes and Actions

2021-04-29

2021-04-01

2021-02-04

2021-01-07: First Meeting 2021

2021-MM-DD: Template

Previous Meeting Minutes

Meeting Protocols

• Please note that by participating in meetings, attendees agree to adhere to the GA4GH Standards of Professional Conduct.
• Meetings may be recorded for note-taking purposes. Recordings will be All deleted within three months of the meeting taking place.
• Dates should be specified in the international format yyyy-mm-dd

Agendas, Minutes and Actions

2022-01-06
Chair: James Bonfield
Attendees “Name (Affiliation)”: Rob Davies (Sanger), Louis Bergelson (Broad), Lee P. (AWS), John Marshall, Daniel Cameron, James Bonfield (Sanger)

2021-12-09
Chair: James Bonfield
Attendees “Name (Affiliation)”: Daniel Cameron, Anders Leung, Jonathan LeFaive, Robert Davies, Tim Cezard, Reggan Thomas, Yash Puligundla, James Bonfield (Sanger), Michael Hoffman (University Health Network/University of Toronto), Louis Bergelson,

JB: Should probably merge #1 and #2.

#559

This is assigned to Louis. Any update?

LB: I’ve been distracted by everything else and haven’t been able to make an update.

JB: If you think it’ll be a temporary thing it’s fine. If you’re struggling and need someone else to pick it up let’s have that conversation.

LB: It would be great to have more people maintaining VCF.

#558

Our decision here was to remove it from 4.4. LB you were going to make a PR to do that until the time comes when we can do it properly.

LB: Still need to do that.

#554

This was a glossary. Again don’t think there’s been any update

TC: Doesn’t feel very high priority on my list.

JB: Things like this can come down to anyone. Not necessarily the person ti’s assigned to.

#295

LB: Hasn’t been any decision made there. We need to choose something.

MD5 Calculation

Our guess is to liftover language from other specs to be consistent. Maybe that’s best if john or myself copy it from **

DC: Makes no sense to reference the sam*

In reality 90% of the MD5s that get written literally copy whatever the md5 field is in the input to whatever the call was in the VCF.

LB: Probably copied from the reference somewhere. Might make sense to pull the hashing stuff out into its own document.

JB: maybe but it may make the spec harder to read. It’s only one paragraph

TC: The point of this PR was for filling in something that was missing. Now it’s just adding something that is consistent with SAM. I agree that there should be a reference to refget as opposed to a reference to SAM. This one is about correcting the example.

#579

JB: Clarifying the use of threadscale. Requires someone who is more versed with the VCF world to double check that I’ve gotten it right. This needs review.

#567

JB: This goes back to ROb’s tidying up of the VCF header. It was the last thing that is still to do on the list of things.

RD: It’s extremely minor.

#553

SV Type. Punted a bit because we haven’t had anyone looking at it. It’s a 4.4 thing. Maybe we should amend this and make the distinction between 4.4 and 4.3,

From the PR it looks like all the commands have been addressed. So we probably should just make a second PR saying that it has been addressed.

JB: Daniel, you should look at it and see if ** has addressed the issue.

DC: I think it’s ready to merge, but I don't have merge permissions.

JB: Maybe we could try alternating meeting times.

DC: What’s our schedule for 4.4? I was thinking of making structural annotation variant ** 4.4 compatible. I have some time to do work between now and Christmas.

JB: We do need more maintainers. Can speak to Thomas and Oliver about this.

DC: The only impediment to doing that is the timing of the meetings.

TC: In the meantime, while Kieran is not here, I’m happy to help.

JB: Don’t need to be formally recognized as a reviewer, but would be helpful to have people looking at the PRs.

Base modification spec clarification

JB: John reviewed the changes and added a few minor comments which I just merged. I”m making another PR to migrate from half lower case half upper case to a formal tag. There’s continuous integration and testing that John has been doing.

#589

A backburner thing. What we’ve been discussing comes under that as well.

#591

This is all about VCF and the definition of its dictionary. Needs some better documentation. Essentially there is an internal hash that goes on ordering numbers. Peter has described it in the reply, but it should be in the spec. That’s something that should be done.

LB: There’s a lot of underspecified things in VCF like that.

Break ends

DC: I don’t think there’s actually an issue on this thing. Your ref can be whatever it is. The break end will be the break end next to it.

JB: Should we close this?

DC: Yes, I think it’s perfectly valid.

JB: I’ll add a comment onto this and close it as something that isn’t a good idea but is permitted by the spec.

#469

JB: Seems to have fizzled out.

DC: Current state is waiting to be merged.

JB: Can someone have a check through and click the merge button? Being 4.4 it’s probably good to have things done sooner rather than later.

TC: Someone needs to dismiss the review.

LB: We can merge it with the review outstanding. I’ll merge it.

#590

(Assigned to TC, but only because he brought it up. He is not the one taking it on)

JB: Another one of the many things needing clarifying

DC: Technically doesn’t need clarifying. We have a freeform header in our reference.

#593

JB: This hasn’t been updated since September.

LB: We have a new issue that’s related to this. The spec allows for many different representations of ** some of the things you can represent in VCF don’t have a definite analog in BCF. Should probably clarify how tools should handle when things are missing.

#605

JB: Not causing any issues with rendering right?

PL Ordering

JB: Would be nice to have a person that’s knowledgeable about VCF answer this.

LB: We’ve talked about having a per genotype phase probability field but we don’t have one. Is that what’s needed here? There was a question from Dan Cook about adding a new number value that is a per haplotype value which is what I think they need here. It’s by allele. Not by allele by haplotype. The PL fields draw probability of allele but not probability of phasing of the haplotypes. You can have the same allele and multiple haplotypes.

JB: If someone could answer that, even if it’s that they don’t have a good answer that would be great.

New and Updated items

#611

JB: There’s CSI for VCF and BCF and they have different format blocks. John replied and at some point he was talking about improving it. Can ask him about that in the next meeting.

JB: Info field with dot. Think this is valid. There’s a distinction between dot saying there’s not value v.s. Not specified at all. Not sure what we do about these but my gut feeling is that it’s already been answered and this should be closed. ButI’ll leave it to John

DC: +1 A field with just a dot is fine.

SAM/BAM/CRAM

JB: Yash is actively working on CRAM 3.1 with htsjdk. Looks like CRAM 3.1 is moving forward. May need improvements on the spec.

LB: Anyone interested in talking about how to represent missing fields in BCF?

JB: Probably better off talking to Peter.

BED

MH: This is a clever bug. Can I just fix this? I don’t have the permissions to change the issue metadata. I can reply at some point to James.

VCF 4.4 Timeline

Usually our criteria requires an implementation right?  4.4 is a bit of a weird one because there’s no actual syntax issues. Htsjdk would be able to read a 4.4. We potentially already have an implementation?

JB: Ideally we should have someone using those fields. Similar scenario to base modification. Before you sign it off as this is an official implementation…

LB: Do you have downstream code that deals with event types. Is there code that we usefully shared with the world that deals with this?

DC: Yeah I have code to deal with break end format.

TC: don’t we need to first finalize the PR and merge them and then remove the draft from 4.4 so we can tell people implementing it that it’s not a draft anymore? I thought the intention was so when the spec goes live we have a reference implementation of sorts.

JB: We can say it’s in development status. When we merge everything and have it ready it can go out for comment.

DC: The PRs have been stable for a year now. Assuming we merge everything together I think we’re at the point where we’re at a stable spec. Will update structural annotation* R package so it can read VCF 4.4.

DC: I have a PR for the SAM and VCF strip specs.

JB: Some of this comes down to practicality. Say you realign a read and change it’s cigar tag you might know the other end has an MC tag it will now be incorrect.

DC: It will of a make position column at least.

JB: Are you saying probably what we need is better metadata info in the header (to say I am fully complete) and if ** then you remove that tag. And you have another tool that when you run it, it*

2021-11-11
Chair: James Bonfield
Attendees “Name (Affiliation)”: James Bonfield (Sanger), Michael Hoffman (University Health Network/University of Toronto), Anders Leung, Tim Cezard, Lee P. (AWS), Robert Davies, John Marshall

Meeting Recording

2021-09-16
Chair: James Bonfield
Attendees “Name (Affiliation)”: Alex Stuckey, John Marshall, Yash Puligundla, Anders Leung (GA4GH), Jonathan LeFaive

Meeting Recording

2021-08-19
Chair: James Bonfield
Attendees “Name (Affiliation)”: Rob Davies (Sanger), John Marshall, Louis Begelson, Tim Cezard, Alex Stuckey

Meeting Recording

2021-07-22
Chair: James Bonfield
Attendees “Name (Affiliation)”: John Marshall, Jonathan LeFaive

Minutes

#539

JM: Both JM and JB to contemplate. Still to be worked on.

#522

JM: Can take off for next time.

#418

JB: Agreed last time that we would announce on the discussion or PR that we would merge it in 3 weeks. It is now merged. Now in as a draft tag. Leave in draft status for a month to discuss at this meeting later on. Since then there have been a couple queries raised. Will discuss later in meeting.

#531

JB: Will review what JM has done and hopefully we can merge next time.

JM: removed the word guidance because I wanted to acknowledge but not over state and so that it read better.

#433

JB: Carrying on with this. Happy that it’s a speed up on most platforms. Order 1 is a bit more tricky.

JB: In your trial implementation are you depending on the compilers or depending on the intrinsics?

JM: Compilers. **Can’t get it as good as manual implementations. Probably not a good starting point. What I have is code that checks the cpu and ** the appropriate function. Bit of a worry because it may not work as well on java. Questionable as to what the effect is. CRAM encoding and decoding maybe up to 20% speed increase.

JM: Have announcement at top of rolling meeting notes or set up emails. Things don’t necessarily have actions on them but good to have discussion points.

JB: Can probably drop from actions for next time.

#554

Not sure we agreed whether it belongs in spec or wiki. Go into spec (JB, JM +1)

#579

JB: In VCF spec some of the definitions say this is the phred-scale then use the notion that low numbers are accurate and high are inaccurate which seems like a reverse phred-scale. Made that change but need to verify with someone who knows VCF that my interpretation is correct.

JM: having issues with SAM Alignment mapping quality.

JB: Amended wording slightly of definition of phred-scale. To be explicit that we are talking about probability of error.

#567

Will assign someone next time

#553

KT has been working on it.

VCF/BCF

JM: How do we keep this wagon progressing. #413 was a PR 2 years ago where someone said how do I use this pedigree thing in the spec? I fixed minor typos. Commit #2 was adding a section about other ways in which pedigrees are used. Mixed up in the same PR fixes to errors and also new work. Focus was on larger pedigree discussion than trivial changes. Begs for process discussion.

JB: Maybe we should add to the agenda how long each PR has been open. If you have a joint one, see how long it lasts.

SAM/BAM/CRAM

JB: Has a tool which is looking for methylation around CPG islands* methylation annotations for some particular regions and not other regions. When he does analysis he wants to know should he include lack of modification at particular site as indication that he didn't call it or look at it. Proposed three different solutions. One extra piece of info. Implicit vs explicit. Implicit - calling base here, here, and here have been methylated and implicit assumption that others have not been. Explicit - here, here, and here and methylates, others are not, and everywhere else we haven’t looked. Maybe look for all methylated bases, but look for some other methylation pipeline*

JM: Immediate thought is that if you don't have methylation tags then that has to mean unanalyzed. That can’t mean not methylated.

JB: That’s a distinction I hadn’t thought of

JM: Maybe that’s a third category (no tag). Maybe within a single read he has analyzed and unanalyzed regions. Obviously an important point that we need to make somewhere.

In the explicit world you would write out “this base has not been modified” with an associated confidence call. What you’re really backing up is this is the probability of being modified.

Need to be explicit. If there’s no tag at all it means we haven’t analyzed it.

JM: Probably another question for Jared. Are there different regions within a single read?

Louis to look at #518

GA4GH Connect

JB: Future of VCF is something that could go there. Ideal place for other working groups to see if the use cases are complete.

JB: Round table with Thomas and Oliver and us giving an update.

JM: may or may not be appropriate for connect but some kind of organizational birds of a feather. Things that we never have time to discuss here.Larger meta meeting to discuss how we organize our work and keep it on track.

JB: Could be something we do at Connect.

LB: Maybe we could have the meeting within file formats but across all file formats groups.

LB: Meeting with refget. Reference sequence dictionary identifiers. Reference sequence people. How can we reference their spec.

RD: Growing quite a lot.

JB: Want to make their references collection a part of all file formats in bioinformatics.

JM: Issue in sequence collections repository. How do we embed sequence specifications in other formats?

2021-06-17
Chair: James Bonfield
Attendees “Name (Affiliation)”: Rob Davies, John Marshall, Louis Begelson,

Meeting Recording

Minutes

JB: Previous action #539 - have completed this. Did we decide who was going to write the text? JM: Table this for next time
JM: 522 - only thing remaining is does someone want to produce a** explaining what’s going on
JB:418 -  IDB? have now officially released it. People are actively using it. Probably merge as is with warning that now is time to complain because it will become a non draft in**timeline for this?
JM: next meeting to do the merge
RD: This will be merged in this form unless anyone says otherwise in the next three weeks

JM: 531 - Will make comments.

LB: 1387 - haven’t progressed on this. Will send an email to see who is working on gnomad, and report back next meeting.
JB: 559 - More or less agreed that it means unspecified and you shouldn’t assume either way.

JM: Comes down to similar sort of thing of when someone raises an issue saying they don’t understand what this is trying to tell me, we need to clarify it. Need to capture what aspect of it was unclear.

LB: Should make sure that it lines up with how it was defined in ft field. Assign self as VCF maintainer.

RD: 558 - we now support in htslib

JB: it’s in 4.3 that you have this ontology syntax (square brackets). HTS JDK** doesn’t support this, so that shows how rarely it’s been used. 558 links to 1240. Same as it was in VCF spec.

RD: There are also links to spec files.

JB: Should probably remove this from 4.4.

LB: have an outstanding issue to redefine this explanation. Makes sense to cull it from 4.4. Since no one is using it.

RD: ordering issue as well. E.g. Should meta line appear before line that uses it.

JB: Make sure to remove from 4.4. If we feel the urge we can go back and visit it properly.

LB: Also might make sense to rename it from meta to something else. Meta is a loaded term that doesn’t describe what it is.

JB: You are currently implementing 4.3 for htsjdk**?
LB: Yes

TC: Are we allowing in the header to have additional lines that do not contain any random line that will be passed on in the header?

JM: Spec does say something along the lines of unstructured headers

RD: Problem with meta thing is it uses entirely different syntax.

TC: If someone writes meta in 4.4., then that’s fine but it's part of the rest of the thing that’s allowed but it’s not part of the spec.If it’s still backwards compatible, it’s fine.

JB: Can probably say that they're not used. You have to be able to accept them, but**

JM: relevant section is 1.4 in VCF 4.3. Says it must be well formed, but doesn’t explain what well formed is. But reasonably well explained.

JB: Do we have any of this in test data?

RD: Yes there are lots of quote fails files

JB: But this wouldn't be a failure, it would be a grey…

TC: File data is strict about anything that doesn’t look like a properly formatted header line. All tests from VCF are coming from that validation tool. This said, it does accept that meta** line**

JB: 443: **Had some debate on this last time. CRAM 3.1 has a different**, hasn’t been adopted. Had an exploration about whether we can do better loop unrolling.**Can’t do manual vectorization. Makes it language, and maybe cpu, specific. Faster on arm**, but not as much as intel equivalent. It wasn’t a detrimental effect. Possibly useful defining.**Still unsure whether this is a good thing. It’s ongoing. Improves speed, and marginally improves compression

JB: previous action item #9 - has been merged.

JB: Did go through looking at updated item. Not much to say on SAM, BAM, CRAM. Will be VCF focused. There’s a couple questions that need VCF maintainer to answer. Someone wanted a glossary, this is a great idea (LB +1)

LB: Do we want to be linking to other random, not well defined specs?

JB: Don’t have to endorse them, but hing like gVCF is fine..can link to documentation and have a caveat that these are variants that are not necessarily endorsed by GA4GH

LB: Is this for the spec or for some auxiliary documentation?

TC: Glossary of concepts vs this kind of thing. Something was started between during alignment between VRS and VCF.

JB: There is the wiki idea. Might be nice to have as an appendix at the end of the document.

TC: *To reply to get info on what would be good in the glossary.

LB: Do we support string arrays in VCF?

JB: If it’s a string it’s kind of debatable.

LB: Will reply to that, but wil first make sure not wrong about separators.

JB: Need to get more people involved in spec world. One of the key things need to do is engage*

TC: 566 - asked Kirill? if he could look at it, but he was on leave. Louis if you can take a look at it and tell me if anything bothers you. Thing that bothered me the most is the example showing the ** as the same

JB: Phred  scale - thought the high numbers were the good one, but they weren’t. If someone who understands the distinction between likelihood and probability can review it...maybe change it to have to normalize it. 553 - Daniel added this last time and we never got around to discussing it.

LB: 519 - decided to close this because it wasn’t worth the changes

JB: Do you want to add a comment to suggest that we close it so people actively participating can agree or disagree?

LB: Will close it.

RD: 567 - that’s the PR that was stated by Peter. It was eventually merged.

JM: Represents two remaining questions that weren’t answered before. Been going on for two years, related to issue that were going on for 5 years.

LB: Keep open for now.

JM (chat): https://github.com/samtools/hts-specs/issues/577

TC: will answer that

LB: think we need to stretch VCF to be viable if not optimal. May end up in a world where we have a data model that is basically VCF with changes for reducing connection between things, but people implement their own format depending on how they want to access it.

JB: Could benefit from a layered file format. Wouldn’t need to pay penalty from pulling out others.

2021-05-27

Chair: James Bonfield

Attendees “Name (Affiliation)”: Michael Hoffman, James Bonfieled, Susan Fairley, Timothee Cezard, John Marshall, Mamana Mbiyavanga

Meeting Recording

2021-04-29

Chair: James Bonfield

Attendees “Name (Affiliation)”: Kirill Tsukanov (EMBL-EBI), Tim Cezard (EMBL-EBI), Rob Davies (Sanger), John Marshall (University of Glasgow), Yosr Hamdi (H3Africa), Daniel Cameron (WEHI), Louis Bergelson (Broad), Susan Fairley (GA4GH), Michael Hoffman (Toronto)

Meeting Transcript

2021-04-01

Chair: Louis Bergelson (Broad)

Attendees “Name (Affiliation)”: John Marshall (University of Glasgow), Kirill Tsukanov (EMBL-EBI), Tim Cezard (EMBL-EBI), Anders Leung (GA4GH)

Meeting Transcript

2021-02-04

Chair: James Bonfield

Attendees “Name (Affiliation)”: Rob Davies (Sanger), John Marshall (University of Glasgow), Kirill Tsukanov (EMBL-EBI), Tim Cezard (EMBL-EBI), Angela Page (GA4GH), Louis Bergelson (Broad), Anders Leung (GA4GH)

Previous actions

1. Resolved main chunk, but still to do item for improving computing intersections, esp linear index: we need a better schematic something we can work from, something like that would be useful

1. JM: my comment reflecting what Yossi thought was missing in previous PR;  there is a three-way tree texture that doesn’t tell you a lot of what you need to know, picture probably is a big part of improving this, focusing on linear index
2. How much detail do we need to go into given this is at a level not many are implementing; a couple years ago many issues came up with indexes bc many people started implementing;
3. JB: will try to come up with better picture

2. TLEN - what length means when you have non-primary alignments, don’t want to shake it all up and propose new thing, too many people in different directions; bite sized chunks are not going well either

1. JM to produce an image showing what is being discussed, and re-review anything in the text

3. VCF header length, was for LB to re-review, give to someone else to re-review? Initial writer was Peter

1. KT: could take a look at it eventually, not before mid-feburary; look at history of discussion
2. JM” collection of minor corrections, originally to make more detailed VCF description of what BCF header looks like and describe fields in more detail; has accreted 3-4 other minor corrections as well
3. RD: if you read current spec, you can actually implement VCF
4. KT: make it aligned with how it actually already works / is implemented, not changing the spec
5. JB: if anything is contentious, acceptable to fork into separate bits, some merge, some continue discussion on
6. JM: way to get this going is to do each thing it does at a time; end of day, if review, look at overall change and read through that, rather than the entire list of iterations
7. RD: same change on three diff files, now four files - four sets of files
8. JB: maybe that needs to be fixed!
9. LB to review PR this month, should only take 10 minutes

4. Base modifications

1. No updates, ground to a halt really need someone other than JB to make some decisions
2. JM to have another look, but thinks it’s in good shape; JB to help with buggy versions to explore with
3. JB: discussion about what should be done re: delta, whether we count all bases or ind’l type of base, trade off between ease of implementation and size on disk, natural bent is to go down compression route, someone less biased should consider best trade off in complexity vs. size

5. Update readme and be more inclusive with various things

1. JB: addressed JM’s comments and merged; 5.3.1 affects more than just CRAM so left it open, but comments addressed; should we contact others like Andy Yates? In theory repo is used by several groups beyond this meeting, but they’re minor
2. JM to go over readme, JB to give Andy and Mike heads up
3. Barely acknowledges GA4GH as it stands, repo predates it

VCF #1387

• JM: bc htsjdk works best with vcf 4.2, various folks including gnomad work from that spec; put out 3.1 revision last october, with several tags in it, eg., AC-non_v2-XX… hyphens not valid in 4.3 - they haven’t realized this bc spec they were working from didn’t mention it, or they didn’t read the spec, bc most people shouldn’t have to in theory, htsjdk hasn’t caught up with htslib so there were no warnings saying anything was wrong
• Imagine there was no awareness of a problem. To a great extent no problem, vcf tools works with these things, but one person found vcf tools query command didn’t work, parser interpreted as subtraction. Major implications.
• In some ways not a big problem, in other ways will only get worse, eg., running query over gnomad data
• How did this happen? We thought we were done and dusted with 4.3 five years ago, now thinking about 4.4 and people are still using 4.2
• VCF maintainers: do we need to open a dialogue with gnomad people? Make them aware there’s a problem here, can they adjust their tags?
• JB: do we know whether initial reporter of problem is user or maintainer on gnomad? Expect just user…
• Spoke with Peter to get his views, two ways to interpret, make change for 4.3. 4.2 are …
• Sam: spec was rubbish, so we tightened the spec, that was an amendment back dated, ban them now before they get used - this was peter’s view, never intention to have math characters should not be part of IDs never explicitly written, assumed IDs would be same as in any programming language, incorrect assumption; he edited the latest spec
• Both are valid ways to go, for vcf maintainers to decide - is this a 4.3 only thing, be more careful with parser, or retrospectively ban?
• KT to write to gnomad
• Latest gnomad version is 3, is the problem from gnomad version 2 or 3? 3.1 - latest version

• Immediate soln is for gnomad to fix that in their data

• Part of the problem, from beginning, confused about idea of changing, 4.3 released 5 years ago, confusing notion of changing spec that already released years ago, this is another line of discussion how this should be handled, spec of 4.2 as released more than 5 years ago, formally those characters must be supported, in practice we know they won’t be supported just for vcf 4.2
• Spec needs to be amended in some way even though I oppose changing anything retrospectively, but update to include that remark, not just 4.2 but all versions, 4.1 “version of this spec up to this date did not specify xyz, in practice…” explaining situation to avoid that happening in future
• Problem is on hstjdk side, can’t write vcf 4.3 - this is maybe part of the reason why people still using 4.2 - haven’t been following dev of htsjdk - why is this the case?
• JM: reason they haven’t worked on it in the past, not something Broad needed, it is being worked on now; got a CZI 1-year grant for funding to work on things like this in htsjdk, understanding about 6 months into that funding period
• Look at their draft PRs there’s one that is a draft of adding / generalizing the support for these things, plugin idea of diff plugins to support diff versions (December 2020)
• Haven’t looked at differences in htsjdk code, or if it’s just the framework
• Htsjdk will be able to read all versions,...can interoperate with file that says 4.3 at the top, but doesn’t implement this...in particular
• JB: normal syntax appeared in 4.3 draft in 2014, day one; black hole before that
• JM: these are living documents, all diff kinds of changes you can make to a spec, typos fixes, clarifications, loosen or tighten restrictions; most of changes have been made to 4.3 since 2015 have been bug fixes and clarifications, and odd loosening; people careful not to tightening
• Tightenings were starting to occur, so that’s why number was bumped to 4.3
• VCF is in a diff situation than SAM, sam just has one spec doc, say what diff between levels are, VCF older specs are out in wild and appear supported
• Added subtitle this week saying you should be looking at 4.3, add “need to be more careful about your info keys”
• Your info key “should” follow this, different from “must” but gives hint - real tragedy is that they’re looking at 4.2 at all
• KT: once spec is release a certain version inevitable there will be errata, corrections of errors, not changing anything per se but clarifying what was meant; that will always be necessary
• But also, it might make sense from some point in the future, to make sure those corrections stand out, so they are separate from original text of doc, “this paragraph was added on X date,” so folks know not in original version
• Important because thinking about versioning, digital archeologists in the future, presumably vcf files generated now may persist for maybe decades, not inconceivable that someone 10-20 years from now will want to implement 4.3
• Really important for person trying to decipher old file, to be able to read exact same version of the spec as by the person who generated it at the time
• JM: can do that by reading history of spec repo; also that’s why we have an appendix in SAM specs, saying on this date we added or tightened, changes to formatted as interpreted listed with date
• Vcf 4.3 has that in change log in index at end, but maintainers decided not to do that prior to 4.3, extent to which it might be useful to backfill that
• JB: if you want to be ultra cautious, depends on how good software is, may wish to have version number at top of file with commit hash, but that’s a software thing not a spec thing
• KT: at least there is some way, if you really want to find out, but in actual text of spec, if this was not original release, I think it should stand out
• JB: agree, depends on the type of correction - typos don’t need that, line must be drawn
• Worth considering not just in appendix, but footnotes
• KT: for this case with gnomad and hyphens, in that scenario that would be such a correction, edit and highlight with date, clarifying what is at issue (for 4.2 and maybe 4.1
• Other can of worms: how long are we planning to do this anyway, 3 draft versions and will likely proliferate at some rate
• JB: this is what I Was getting at with comment that JM merged, notice at top of doc; for 4.2 clear to say, this is not latest version of spec, please see new one, for 4.1 could be more aggressive, this is not current version and may not be supported by some software; historical doc but useful to warn people
• JM: in future of VCF meetings, this has been discussed, 4.1 will be retired when 4.4 is ratified (or a similar roadmap)
• KT: want fixed number of releases and clear status of each
• JB: up to software to decide when to retire support for a spec
• KT: currently spec makes effort not to mention any implementation details, but this might be useful as well, we have this feature, but not yet supported in xyz implementation
• JB: we have the vcf test files and the cram/bam/sam test files, pr in hts specs need to look at soon

• Set of text of files within validation suite, shouldn’t be specs to say what supports them but a separate doc to say what software supports them
• Somewhere down the road, decent enough corpus of validation tools, then can list pieces of software that do it

• LB to get in touch with them to say this is not standard
• JB: Was this just 3.1?
• JM: Believe not there in gnomad 2, not sure about 3.0

VCF/BCF

• JB: Anders is updating vcf and htsjdk, could potentially update BCF
• LB: do we know if anyone uses BCF?
• JB: Sanger does, bc so much better, order of magnitude faster, only reason to use VCF is bc of software
• LB: htsjdk has minimal bcf support, should move that forward

SAM/BAM/CRAM

• No new items

Test Data

• JB: just test data, not changing spec, whether needs much review or just bite bullet and ensure file names and directories sound,
• JM to get on and have a look and make some comments
• Some thoughts about directory structure, implementation oriented vs. domain oriented, area being exercised, which seems like natural organization for readers, maybe just a renaming; probably just merge, not normative
• JB: one person found a bug in CRAM implementation by using CRAM test data

Other business

• TC: GA4GH connect, refget thinking of having sessions to talk about sequence collections, wondering if people from file formats interested in attending to give requirements in what should go in seq collections, what should include for id’ing seq collection
• JB: good idea, has impact on headers for SAM and VCF
• LB: interested in attending, yossi has strong feelings
• TC: has been part of discussions for a long time
• JB: make sure it doesn’t clash with timing
• TC: is there anything related to BED files, these are also the people we’d want to include, GFF out of the question…
• JB: Aaron Quinlan and Michael Hoffman interested in becoming maintainer of BED; there was something recently from someone involved on google list,
• Any file using chromosomes and positions should have metadata for which assembly it’s using, been a problem for BED doesn’t have well-defined header syntax, wanted in V2, get BED as stands as standard, next version to be more robust
• KT: lsg / ff outreach - gnomad issue is
• AP to send email to FF list about outreach
• Where we’re not experts and need those folks to get more involved, broader community view on what should be done, rather than these two companies, sample size of 2
• Rarely the case but ability

2021-01-07: First Meeting 2021

Chair: James Bonfield

Attendees “Name (Affiliation)”: Rob Davies (Sanger), John Marshall (University of Glasgow), Kirill Tsukanov (EBI),

2021-MM-DD: Template

Chair: James Bonfield

Attendees “Name (Affiliation)”:

Previous Meeting Minutes

File Formats 2020

File Formats 2019

File Formats 2018

File Formats 2017