GA4GH File Formats  

2017 | 2018 | 2019  | 2020 | 2021 | 2022 | 2023 | 2024 | 2025 

Minutes and Actions 2026

These are the minutes for the File Formats meetings, a subgroup of the GA4GH Genomics Knowledge Standards (GKS) Work Stream. For further information, please visit the GA4GH Work Stream page.

Meetings are announced on the email list.  

Meeting Protocols

• Please note that by participating in meetings, attendees agree to adhere to the GA4GH Standards of Professional Conduct.
• Meetings are recorded for note-taking purposes. All recordings are deleted within three months of the meeting taking place.
• Dates should be specified in the international format yyyy-mm-dd

Meetings

The meeting takes place every 4 weeks on Tuesdays at 9 -10 pm UK time  

For the most up-to-date information, please check the GA4GH calendar.

Zoom Meeting Link: https://zoom.us/j/7082943722

Meeting ID: 708 294 3722

Passcode: GA4GH

Table of Contents

2026-11-03

2026-10-06

2026-08-11

2026-07-07

2026-06-16

2026-05-12

2026-04-21

2026-03-17

2026-02-24

2026-01-20

2025-12-02

Long-running TO-DO list (please offer to help if you are able):

Agendas, Minutes and Actions

2026-11-03

Chair: John Marshall

Attendees: James Bonfield, Melissa Cline, Rob Davies, Tim Fennell, Michael Macias, Daniel Puthawala, Beatrice Amos

Link to meeting recording: coming soon!

Meeting summary:

2026-10-06

Chair: John Marshall

Attendees: James Bonfield, Melissa Cline, Rob Davies, Tim Fennell, Michael Macias, Daniel Puthawala, Beatrice Amos

Link to meeting recording: https://us02web.zoom.us/rec/share/-8e8IL4PSw_5hNcKlCsoXMkYzMxcGeDFh6fkXs1Lbkj-wFJryCnHECoHqyTr5T6g.hgXIdmhClEzq1ZNU 

Meeting summary: (AI generated - check)

1. MAPQ and placeholder values for unmapped reads (issue #753; James's PR, recorded as #727 in the August minutes)

✉️ Shared by Rob in chat: https://sourceforge.net/p/samtools/mailman/message/24773233/ 

✉️  James: MAPQ=255 was discussed a bit before. Found it here: https://github.com/samtools/hts-specs/issues/864 

Background

• James's pull request adds recommended-practice text with default values for fields that have no meaning on unmapped reads. Integers default to 0 and strings to *. At the August meeting the group had asked for the text to cover all such fields, not just MAPQ, and this version does that.
• The aim is predictable round-tripping. CRAM does not store MAPQ or CIGAR for unmapped reads, so a decoder has to invent a value when converting back to SAM/BAM. The recommendation tells implementers what value to use so users don't see surprising changes.
• Daniel noted that in practice, unmapped reads are usually placed at their mate's position rather than given all-zero fields. James and John said this is already covered by a caveat paragraph in the PR and elsewhere in the recommended practices.

Why Michael raised it

• Noodles writes MAPQ 255 for unmapped reads. Other writers use 0, and the spec has never said which is correct.
• The spec defines 255 as "mapping quality is not available". Michael argued that, read literally, 255 is the right value when MAPQ is absent, for example when decoding CRAM to SAM. One of the definitions in the spec must therefore be wrong or ambiguous.

Discussion

• John said the 0 / * convention is "folklore" inherited from samtools and HTSJDK, and the PR spells that folklore out. He agreed that Noodles' use of 255 is a reasonable reading of the spec text taken at face value. That sentence was written about 16 years ago and has not been revisited.
• James and Tim recalled the history: 255 was originally meant for reads an aligner had placed but not yet scored. An early aligner (possibly Bowtie) set 255 on everything. That meant "mapped, quality not computed", which is different from "not mapped".
• James pointed out a second conflict. The spec also says that when the unmapped flag (bit 0x4) is set, no assumptions can be made about MAPQ. His reading is that the flag takes precedence, so neither 255 nor 0 is meaningful for an unmapped read. BAM is the outlier because a typed value has to be written. Tim added that this depends on the language: Rust can represent "none", but C has to assign a value.
• Rob found the 2010 SourceForge mailing-list thread. It describes unmapped reads under the then-current spec as RNAME *, POS 0, MAPQ 0, CIGAR *.

Related issue: reads placed in sequence missing from the reference (raised by Daniel)

• Daniel asked for an approved way to encode reads that a pangenome-aware aligner places inside an insertion relative to GRCh38. He first raised this on the same issue in 2024.
• Every current option causes problems:

• Marked as unmapped, the read is moved back to its mate's position by fixmate.
• Marked as mapped with an all-insertion CIGAR (e.g. 150I), there is no anchoring base for POS, and Noodles rejects the record. Daniel said Noodles is technically correct, so this is a gap in the spec.

• Daniel's team works on an aligner that produces these reads, and he has no clean representation to recommend to them.
• Tim agreed a legal representation is needed and noted that GATK's indel realigner has emitted similar records for years. He and James see this as a separate issue from MAPQ on truly unplaced reads. James said a proper pangenome alignment format is the long-term answer and something he would like to see supported.

Outcome

• Discussion tabled until the next meeting.
• Tim asked the group to decide on the principle behind this and other edge cases. Is the goal to write the established folklore into the spec unambiguously, or to accept that the spec text is ambiguous and live up to what it actually says? John agreed. The spec began as a distillation of the folklore, but any change must also respect implementations that followed the written text.

2. Accepting U (RNA) in SEQ and its BAM encoding (issue #801)

Background

• Michael raised this one for James's attention. HTSlib now converts U in a sequence to T when encoding BAM. James confirmed he made that change, after James Ferguson suggested it in late 2024.
• The spec has never been updated to match, so HTSlib is now technically non-conformant. Michael pointed to SAM spec §4.2.3 (BAM sequence encoding): the last sentence of its first paragraph says all other characters map to N, so U should become N, not T.

Discussion

• Spec status: John noted that the SEQ regular expression in the SAM fields table is permissive (any letter), so U was arguably valid in SAM. §4.2.3 was added later and was probably meant to describe the conversion rather than restrict it. It isn't marked as non-normative, though, so he agreed it has to change.
• John's draft: he has a draft that tightens the SEQ regular expression, following an earlier issue asking which letters are valid. The group had previously agreed that arbitrary letters such as Z should not appear in SEQ.
• Root cause (James): the spec was written for DNA only, and BAM's 4-bit encoding has no separate code for U. Recording that a read is RNA would need an aux tag or, better, a read-group (RG) level marker, since RNA versus DNA is normally a library property.
• Restoring U from BAM (open question, John): should it be possible to tell which reads should decode back to U rather than T, for example via a header or tag? This needs reopening.
• Mixed U and T in one read: James said the issue thread covers whether a single read could contain both. This is believed impossible today but conceivable with future library preps; such synthetic segments would normally be split off (e.g. as barcodes) before writing SAM. Michael Hoffman suggested on the issue handling U through IUPAC codes or base modifications. James felt this was the wrong fit, because U in RNA is a base, not a modification.
• HTSJDK: Tim supported the change. His recent sweep of HTSJDK history found exactly this gap with HTSlib, and the next HTSJDK release will also convert U to T.

Outcome

• The group agreed to make a minimal spec change now: allow U, treated as equivalent to T, in §4.2.3, and mention it in the SEQ field description. This brings the spec in line with HTSlib and the upcoming HTSJDK release.
• Restoring U on decoding, and any header or tag to flag RNA reads, is deferred to the issue thread.

3. VCF/BCF new and updated items

3a. Recent VCF example issues raised by "eCrum" (name given as Elias Crum; check the GitHub handle)

• John pointed out a small batch of recently raised VCF spec issues that need responses. A couple already have replies.
• On review in the meeting they look like problems with the spec's examples rather than its normative text. They cover local alleles, GT and a tandem-repeat example.
• Daniel's first impression of the local-allele one is that the example is correct. GT is not defined as a local-allele field: it refers to all alleles, and only the local-allele fields are subset. He was unsure what the reported problem is; for example, chr20 position 10 does have two alleles (T and G).
• Daniel and John both said day-job pressures have left them behind on reviews. John noted he also has a backlog of hts-specs PRs to review and write.

3b. Meaning of Number=G in INFO definitions (issue #272)

• John wants to revive issue #272, originally raised by Kevin Jacobs, one of the early maintainers. John recently ran into it in pysam.
• The problem: G means "one value per possible genotype", which only makes sense for per-sample FORMAT fields. The spec still allows it for INFO fields, where its meaning is undefined.
• Spec history: Daniel first thought he had already restricted this to FORMAT. He then clarified that the fields he explicitly scoped to FORMAT in VCF 4.5 were the new local-allele ones (LPL, LAA, LGL etc.). INFO Number=G predates that work and is still open.
• The thread has two positions:

• Deprecate it, because its meaning in INFO is unclear.
• Keep it (Petr Danecek's view): it lets bcftools merge handle such records sensibly, and it could be defined in terms of ploidy. Petr's 2018 pull request (#309) tries to clarify this, but John finds its wording hard to follow.

• pysam: John will probably treat INFO Number=G as equivalent to Number=. (any number of values). In effect that deprecates it without removing it.

4. AOB

4a. Upcoming GA4GH meetings and attendance

• John reviewed the agenda items Beatrice had added about future meetings.
• April Connect (Hinxton, 27–30 April 2027) and the 15th Plenary (Cape Town, September 2027): John would like to attend but is unlikely to travel from New Zealand unless funded. He expects to join online.
• Daniel also expects to attend online. He will try to get funding but thinks it unlikely.

4b. A file formats "Ask Me Anything" session

• Daniel suggested File Formats is probably the best group in GA4GH to run an AMA for a wider audience. It would cover:

• Recurring questions that often turn out to be resolved years ago.
• Real gaps the community spots, which Michael often raises.
• The history of why the specs read as they do. They aren't written like an IETF RFC, with no capitalised MUST/SHOULD, which confused him at first coming from a computer science background.

• John strongly agreed.

Consolidated actions from 6 October 2026

Suggested additions to next meeting's agenda: MAPQ/unmapped reads (#753/#727), INFO Number=G (#272/#309), and the AMA proposal.

2026-08-11

Chair: John Marshall

Attendees: James Bonfield, Melissa Cline, Rob Davies, Lee Lichenstein, Michael Macias

Actions:

• John: Ensure there is a link to the BZST draft spec repository in the meeting minutes. (Update: link is https://github.com/tfenne/bzst/tree/main/spec )
• James: Draft text for the SAM spec to specify default values (e.g. 0 for integers, '*' for strings) for fields that have no meaning for unmapped reads, based on the existing list in the spec.
• James: Repurpose or amend the existing pull request (PR 727) to include the new recommendation for unmapped reads, or create a new commit for it.
• John: Create an issue in the spec repository to address the ambiguity in the text about mapping quality value 255.
• Melissa: Keep the HTS spec team posted on any outreach or communication needs from the GA4GH driver projects.
• The group: Consider revisiting the meeting schedule to optimize for regular attendee time zones.

Link to meeting recording: https://us02web.zoom.us/rec/share/gMRV-G932iAeoCBEod9ovvPpUKlcbs_bsI2YR1gYSiVXL2Xc_qTo88djQ91CUl6w.NyKtO1Alg_zcXzRd 

Meeting summary:

1. CRAM 3.1 status and related implementations

Lee asked whether updated CRAM support had been discussed at the previous meeting (when Yossi and Tim were present).

James clarified that the CRAM-related discussion last month had been relatively minor. It focused mainly on tidying documentation around LTF8, which is now available as a pull request reviewed by Yossi. Implementation work has helped restore good interoperability across tools, and no major blockers were identified. The group is essentially back to a position where CRAM 3.1 support is viable.

Implementation updates noted:

• Picard has released version 3.5.0 with CRAM 3.1 support.
• IGV continues to be actively maintained.
• Lee reported that GATK support for CRAM 3.1 is expected imminently (targeted for the following Monday). He has already followed up with the relevant colleague.

✅ Action: None required at this stage; status noted as positive.

2. BST / BZST (modern compression work)

James provided a brief update on the ongoing work to modernise the compression layer (previously discussed in the context of Zstandard / BGZF2).

Tim has developed BZST, a derivation of the earlier BGZF2 proposal. It currently provides a clean random-access container format with an uncompressed-to-compressed index; genomic data handling is intentionally kept as a separate layer. James has updated his branch to read and write the format and has confirmed random-access functionality. Tim also has a Rust implementation, so basic interoperability between the two codebases has already been achieved. Minor tweaks to the specification continue. A draft specification exists (rendered more cleanly than the previous Markdown version) and a link will be added to the minutes. Tim is currently on holiday, so further discussion will continue on his return.

3. Outstanding pull requests

Molecular annotation tags

A newer pull request on molecular annotation was noted. It appears related to an earlier proposal received a few months ago (likely from the same Santa Cruz group, though a different author). The new submission places the material in the draft tags section and has shifted to 1-based coordinates (the earlier version had used 0-based, which the group had found less suitable for human-readable tagged fields). The group agreed this merits a proper review in due course.

Capital letters in modified-base subfields

James’s recent pull request adding support for capital letters in the relevant subfield (for example, allowing “C” to indicate a modified cytosine of any description) was reviewed. It simply brings the regular expression into line with what the rest of the specification already permits. The change was agreed to be a useful clarification and was merged during the meeting.

Lee reiterated that GATK CRAM 3.1 support remains on track and mentioned that John Smith may join future calls as they work to keep GATK more closely aligned with the specifications.

4. Mapping quality and unmapped-read fields (Issue / PR #727)

Michael Macias raised the long-standing issue and associated pull request (#727) concerning the recommended practice for MAPQ values on unmapped reads.

Key points from the discussion:

• The current recommended-practice text states that “no alignment should be assigned mapping quality 255.” There is ambiguity about whether this means 255 should never be used for a calculated quality, or whether it is the appropriate “unavailable” value.
• Many tools set MAPQ to 0 for unmapped reads.
• CRAM does not store MAPQ (or CIGAR) for unmapped reads, so the value is regenerated by the decoder (typically as 0).
• Historical behaviour in BWA can leave residual values in MAPQ, CIGAR, NM, MD and similar fields even after a read is marked unmapped.
• STAR currently writes MAPQ 255 for uniquely mapped reads (for Cufflinks compatibility), which sits at odds with the existing recommendation.

The group agreed that the existing text about MAPQ 255 needs clarification, and a separate issue will be opened for that.

For the current pull request, the preferred direction is to expand the recommendation under the unmapped-reads section of the specification: when the specification states that fields have “no meaning,” recommend clear default/placeholder values (typically 0 for integers and “*” for strings). Care is needed with flags (for example, the reverse-strand flag) and with fields such as RNAME/POS that already have defined conventions for unmapped reads.

James will draft expanded text covering the relevant fields for review at a future meeting. The existing PR can be updated or superseded by this broader approach.

🔶  Actions:

• Open a separate issue to clarify the wording around MAPQ 255.
• James to draft expanded recommended-practice text for unmapped-read fields.

5. Support from other Work Streams and meeting scheduling

Melissa asked whether there was anything the other Work Stream leads could do to assist. James noted that there are currently no topics stuck awaiting broader vendor or community input. He recalled earlier experiences (for example with the base-modification/methylation tags) where progress was slow and feedback only arrived after a decision had been reached. Should a similarly complex item arise in future, GA4GH outreach channels and Driver Project connections would be valuable. Melissa invited the group to keep her posted and indicated she would try to join future calls when the timing is more convenient for the West Coast.

A short discussion followed on the alternating meeting schedule. The current pattern was originally intended to cover a wider range of time zones, yet the regular attendees have remained relatively stable (UK, North America, Australia and New Zealand). The alternating arrangement means that John and Daniel are able to attend only every other meeting. Michael confirmed that either slot works well for him.

Suggestions included keeping most meetings at the later (more Europe- and UK-friendly) time and occasionally testing the earlier slot to check for additional participants. The group acknowledged the inherent trade-offs between consistency and experimentation and agreed it is worth revisiting the pattern in light of current participation.

(AI generated)

2026-07-07

Chair: James Bonfield, Daniel Cameron

Attendees: Yossi Farjoun, Michael Macias, Rob Davies, Tim Fennell

Link to meeting recording: https://us02web.zoom.us/rec/share/NviEq9RC9Dxc0hA2MAzegflkbwidWdXvDYo_In3r45zJO9aV62LI2rg0znotOOuo.bWAQeLKw8n_jxEia 

Meeting summary:

New actions:

• James: Touch base with Tim offline to resurrect and move forward the BGZF2 proposal, aiming to discuss before the plenary meeting.
• Beatrice: Confirm whether the upcoming Connect meeting will be hybrid or virtual, to determine the best format for session chairing.
• Yossi: Review James's new PR regarding ITF-8 and LTF-8 in CRAM to ensure it clarifies the ambiguity.
• James: Update the VCF/BAM labels and status in the issue tracker.
• Tim: Compile the strict SAM spec draft into a PDF, review it, and assess the level of effort to build a validator, providing feedback on the spec's format for automation.
• James: Ping Shane McCarthy (Tree of Life) regarding the implementation for splitting long chromosomes into sub-chromosomes.
• Tim: Draft a V2 proposal for BGZF, focusing on the compression format, to be discussed in a future meeting.

Preparations for the upcoming Connect meeting in Singapore:

The proposal was to revisit BGZF2. James acknowledged that the work had been dormant for several years but agreed it was worth reviving, as there are meaningful improvements that could benefit the wider community. Beatrice recommended that any session chair attend in person if possible, although a fully virtual session (not at Connect) would be preferable if no one could be present onsite. James agreed to reconnect with Tim after the meeting to restart work on BGZF2.

Review of outstanding action items:

James reported progress on several specification updates, including a pull request addressing the handling of ITF-8/LTF-8 encoding in CRAM. After revisiting an issue originally raised by Yossi, James discovered that the higher four bits of a five-byte ITF-8 value are not consistently zero across implementations. HTSJDK historically writes non-zero values due to what appears to be a long-standing implementation bug, while decoders ignore these bits. Because many existing CRAM files contain these values, the group agreed the specification should explicitly state that the upper four bits are undefined rather than requiring them to be zero, ensuring validators do not incorrectly reject valid files. James also clarified the LTF-8 specification, and Yossi agreed to review the proposed changes.

Metadata for oversized chromosomes and related specification updates

James also updated the group on a pull request resolving ambiguity in the MM tag regular expression. The issue had been discussed previously but was inadvertently forgotten until recently. The proposed fix is straightforward, and James asked for reviewers to examine it. He additionally highlighted an existing HTSLib proof-of-concept implementation addressing the problem of representing chromosomes larger than the specification's size limit. The proposal introduces metadata describing how oversized chromosomes are split into overlapping sub-chromosomes and enables range queries against the original chromosome coordinates by transparently translating requests to the appropriate sub-chromosome ranges. James encouraged others to review the implementation and consider similar support in other tools, noting that this would formalise practices already used by the community.

Extending support to VCF and other formats
The discussion went on with implications for related formats. Yossi asked whether supporting oversized chromosomes would also require changes to VCF. James confirmed that corresponding updates would be needed, and Daniel agreed that VCF could simply adopt the same approach as SAM. James noted he intended to coordinate with Shane McCarthy and others who have encountered these limitations in practice. Finally, the group reviewed Daniel's outstanding action items, with Daniel reporting that there had been no progress since the previous meeting.

Exploring AI-assisted specification validation

The group continued discussing the idea of using AI to support the development of a stricter validation framework for SAM/BAM specifications. Tim felt that producing an initial prototype would not require a great deal of work, explaining that the greater challenge would be agreeing on what constituted "strict" compliance. He suggested a possible tiered approach, ranging from basic conformance with the specification to progressively stricter levels that captured common community expectations and best practices. Daniel noted that the current draft proposal already took a different approach by defining a set of independent validation checks rather than hierarchical levels, and encouraged Tim to review the draft before proceeding.

Developing a reference strict validator

Daniel's draft specification has been submitted as a pull request but intentionally left unmerged until the validation approach had been refined. Tim agreed to take the next step by reviewing the draft, assessing how suitable it was for automation, and estimating the effort required to build a validator. He proposed identifying which checks could be implemented easily and which would require substantially more work, allowing the group to prioritise future development. Daniel also requested feedback on whether the draft specification itself was structured in a way that would facilitate automated validation, so that any necessary improvements could be incorporated before it progressed further.

Formal grammar and semantic validation

The discussion broadened to the relationship between formal grammars and semantic validation. James referred to earlier work by John on developing a formal grammar for the SAM specification, observing that a computer-readable grammar would provide a much firmer foundation for automated tooling than relying on natural language alone. Tim argued that clearly written, unambiguous English statements might be sufficient for expressing validation rules, particularly if multiple AI systems consistently interpreted them in the same way. James concluded that the two approaches addressed different problems: a grammar defines the syntax of the format, while a strict validator enforces its semantics and consistency rules, such as ensuring that related fields agree with one another. The group agreed that both components would ultimately be needed, with syntax validation determining whether a file is structurally valid before semantic checks are applied.

Current state of validation tools

James noted that, at present, even syntax validation relies largely on existing software implementations rather than a formal specification. Although parsers such as Noodles are considered among the strictest available, they do not replace a dedicated validator capable of reporting compliance with the specification.

Next-generation block compression: BGZF2 and Zstandard

Following the formal close of the agenda, the group held an extended discussion on the future of block compression. James and Tim outlined their differing perspectives. Tim favoured a minimal, highly adoptable change: replacing BGZF's 64K GZIP blocks with 64K Zstandard blocks to achieve smaller, faster files while retaining the existing BAI and CSI indices. He suggested that rolling out read support across the community several months ahead of write support could establish a broad base of compatible consumers before new files appeared in the wild, and expressed a hope that a backward-compatible BAM variant supporting either GZIP or Zstandard framing might be adopted with relatively little friction.

James argued for a more ambitious approach. He noted that the 64K block limit significantly harms compression for data types that exceed it, such as long reads and multi-sample VCF records, and that a format should be able to scale to arbitrary block sizes. He also detailed long-standing shortcomings in the current indices, which conflate genomic coordinates with file offsets and uncompressed with compressed offsets, making it difficult to randomly chunk unaligned data for parallel processing. Rob added that the CSI index cannot look up a single chromosome without reading the entire index, and observed that many performance problems stem from the index formats. The group aired several further grievances, including that CSI's auxiliary data is undocumented and differs between BCF and VCF.

Redesign versus incremental change

A key point of consensus emerged: because adopting a new compressor such as Zstandard requires a new software stack regardless, this represents the natural opportunity to also address the deeper issues rather than deferring them to a later migration. James argued that the software, not the format, ultimately drives adoption - as demonstrated by CRAM - and that the expertise needed to make these changes was present in the group. Tim raised the possibility of resolving several representation issues that affect BAM but not SAM (such as nibble encoding, long-chromosome support, and CIGAR field limits) as part of the same revision, effectively producing a "BAM v2." It was acknowledged that this would require versioning changes and could generate substantial user-support challenges and compatibility "foot guns." Rob noted that a comprehensive proposal of this kind had been put forward previously and met with pushback, so the earlier discussion is worth revisiting.

OpenZL and the case for future-proofing

James shared benchmarks on OpenZL, a newer programmable compression library from the author of Zstandard. Using a custom, column-aware codec, OpenZL produced files considerably smaller than BGZF2 at speeds comparable to BGZF level 1, and in trained/optimized mode showed dramatic improvements - though the training is still buggy and the format remains experimental. This prompted the question of whether it is worth committing to BGZF2 now, or designing the format so that OpenZL could slot in as it matures, since it also produces Zstandard-style frames. James further noted that a registered Zstandard-compressed VCF would reframe comparisons with alternatives such as Zarr, shifting the question from how those alternatives compare to raw VCF to how they compare to a modern compressed baseline.

Next steps

Tim agreed to draft a version 2 of the BGZF2 compression format, focusing on the compression layer itself, and to pick up the discussion of its impact on the higher-level formats at the next meeting. James noted he would be away the following week but was open to progressing the work thereafter.

(Zoom AI generated)

2026-06-16

Chair: James Bonfield, John Marshall, Daniel Cameron

Attendees: Beatrice Amos, Emma Carlson, Tim Fennell, Melissa Cline, Larry Babb, Michael Macias, Robert Davies, Yossi Farjoun

Link to meeting recording: https://us02web.zoom.us/rec/share/qYz1rbb2EhgXarnbndzMaR-TCQ61373h5heojRqPshZRIp3cxSQeFPXAj0Mmk9lt.G6ujIu5tWSCAcKAz 

Meeting summary: (Part aided by AI)

Review of previous action items:
1. GitHub Issue #855 Closed 🎉

• James confirmed that GitHub issue #855 has now been closed.
• The issue was effectively resolved through issue #831, which had already been merged.
• The remaining concern was the lack of a regenerated PDF, which John has now produced.
• There is some ambiguity in Yossi's final comment as to whether any secondary issue remains unresolved; James will review and confirm.

2. PDF Generation and Automation Discussion

• John raised the question of whether specification PDFs should be generated automatically.
• Historically, PDF publication was kept manual because:

• Human review was considered valuable.
• Multiple changes could be batched together before publishing.
• There was reluctance to have automated systems committing directly to the repository.

• The group noted that:

• CI/CD practices are now more mature and widely accepted.
• GitHub Actions are generally trusted more than third-party CI services used previously.
• Current PDFs for file format specifications are up to date, but the Crypt4GH PDF is outdated.

• Security considerations were discussed:

• Robert cautioned that GitHub Actions can still be misconfigured and create significant security vulnerabilities.
• John explained that current workflows avoid exposing credentials to pull-request-supplied code.
• Since outputs are primarily PDFs, the attack surface is relatively small.

• Potential action item: John may investigate adding an automated scheduled workflow to regenerate PDFs.

3. Review of Outstanding Action Items

• Most previously assigned actions showed no significant progress.
• Items with no movement included:

• BCF documentation work.
• Draft rationale PR work.
• SAM grammar work.

• The SAM grammar topic was deferred because it relates to a later agenda item.
• A separate backlog review meeting regarding Sam's outstanding items has not yet occurred.

4. Issue #631 – Character Array Ambiguity

• Discussion focused on ambiguity in the specifications regarding character fields.
• Example:

• If a field contains "AB", is that:

• One string "AB"?
• Or two characters "A","B"?

• Daniel noted the specification is unclear.
• Michael pointed out that HTSlib does not actually implement a distinct "character array" type and treats these values as strings.
• The issue remains unresolved because expected behaviour has not been formally defined.

5. Sequence Collections and Header Integration

• Ongoing discussion about aligning sequence collections with SAM/VCF headers.
• James previously documented proposed approaches but no further progress has been made.
• More discussion with the sequence collections group is likely required because expectations may not be technically achievable.
• No owner has been assigned to advance the work.

6. GVCF Specification Corrections

• Minor specification corrections remain open:

• Typos.
• Delimiter inconsistencies (commas vs semicolons).
• Missing symbolic * allele examples.

• Daniel intends to re-review and merge the associated PR before the next meeting unless objections arise.

7. Test Data Status (VCF, BCF, BAM)

• VCF 4.5 test data is still missing.
• Additional BAM test data is also needed.
• BAM testing is difficult because:

• The format is binary.
• There are many subtle edge cases.

• Examples of useful BAM test cases:

• Empty BGZF blocks that resemble EOF markers.
• Uncompressed BAM files (UBAM).
• Header inconsistencies.

• Existing BAM test coverage is currently minimal.
• James suggested importing existing HTSlib test files into the specification test suite.

8. Symbolic Alleles and SVLEN Clarification

• Robert raised an issue concerning calculation of variant lengths (SVLEN / related fields), especially for symbolic alleles such as <DEL>.
• Concern:

• Symbolic deletions and explicitly represented deletions should yield consistent length calculations.
• Current specification wording can lead to off-by-one misunderstandings.

• The group concluded:

• The specification may be ambiguous rather than incorrect.
• Additional examples and clarification are needed.
• Existing implementation data appears to support the current interpretation, but clearer guidance for implementers is required.

• Daniel also noted:

• Remaining VCF 4.5 clarification work around SVLEN, END, and deprecated fields.

9. Long Chromosome Support []

• James has begun implementing a proposal for handling chromosomes longer than current BAM limits.
• The preferred approach is a metadata-based solution, where:

• Large chromosomes are split into segments.
• Metadata records the relationship between segments and the original chromosome.
• Tools can reconstruct coordinates and perform range queries across segments.

• Advantages:

• Minimal changes to existing formats.
• Better compatibility with existing tooling.

• Challenges:

• Full transparency is difficult.
• Secondary structures such as SA tags would still need careful handling.
• Indexing and coordinate management become complicated.

• The proposal is currently at the prototype stage and requires further work.

10. Relationship to Sequence Collections

• The group briefly discussed whether sequence collections already address long chromosome handling.
• Preliminary conclusion:

• Sequence collections appear largely agnostic to chromosome size limits.
• SAM text format itself has few inherent limits; most restrictions arise from BAM encoding.
• HTSlib already uses larger internal data types than the BAM specification formally defines.

11. Alternative Approaches Considered

• Tim asked whether the solution used for oversized CIGAR strings (storing true values in auxiliary tags) could be adapted for long chromosomes.
• James explained that chromosome coordinates are fundamentally different because:

• They affect sorting.
• They affect indexing.
• They affect querying.

• Therefore, a simple "store the real value in a tag" approach is unlikely to work cleanly.
• Consensus was that the metadata-based chromosome segmentation approach remains the most practical path forward.

12. New Issues and Actions

• The MM ambiguity code issue remains unresolved and appears to have fallen off the radar.
• James took an action to investigate the issue further and determine whether the regular expression used in the specification is problematic.

13. Discussion of “Strict” SAM/VCF Specifications

• Daniel revisited a long-standing proposal for strict SAM and VCF specifications.
• The idea is to define a layer of validation beyond basic specification compliance, capturing:

• Data inconsistencies.
• Missing relationships between records.
• Situations likely to break downstream tools.

• Example:

• A read's mate information (MC tag, mate position, chromosome) should be consistent with the actual mate record.
• Existing specifications often do not explicitly require this consistency.

• Daniel had previously drafted:

• A catalogue of possible validation errors.
• Standardized error codes and descriptions.
• Semantic validation rules that go beyond syntax checking.

• Progress stalled because there was no validator implementation to make the specification practically useful.

14. Grammar vs Semantic Validation

• James linked the discussion to the ongoing work on a formal machine-readable grammar.
• He distinguished:

• Syntax validation → handled by a grammar/parser.
• Semantic validation → consistency checks across records and fields.

• The group agreed:

• A formal grammar is a prerequisite.
• Semantic validation can be layered on top afterwards.

• Daniel noted that his strict specification work assumes syntax validation already exists and focuses entirely on semantics.

15. Scope and Philosophy of “Strict” Validation

A key discussion centred on what "strict" should mean.

James' concerns:

• Some checks are genuine consistency checks.
• Others reflect assumptions required by specific tools rather than requirements of the format itself.
• Example:

• Missing mate records in a regionally subset BAM file should not necessarily make the file invalid.

Daniel's position:

• Strict validation should not redefine validity.
• Instead, it should report every potentially problematic condition.
• Different users and tools can decide which checks matter to them.
• Missing mates, inconsistent tags, absent SA tags, clipping inconsistencies, etc., should all be detectable and reported separately.

Emerging consensus:

• Validation should be configurable.
• Different checks can be enabled or disabled depending on use case.
• The validator should distinguish:

• Spec violations.
• Consistency issues.
• Tool-specific expectations.

16. GA4GH / GKS Support Discussion

Melissa asked whether the work could be supported through the broader GA4GH/GKS infrastructure.

Discussion points:

• A public validator would be a valuable community resource.
• GKS is considering broader implementation-resource hosting and web presence.
• Existing validators (e.g. EMBL's VCF validator) could potentially be surfaced through such infrastructure.

However:

• Hosting is not the primary problem.
• The main challenge is obtaining resources to:

• Define the rules.
• Build and maintain a reference validator.
• Keep it aligned with evolving specifications.

Daniel emphasized that the File Formats group operates largely through volunteer effort and lacks dedicated implementation funding.

17. Possibility of Funding and Community Support

• Daniel suggested possible future support mechanisms:

• Funding agencies (NIH, European funders, etc.).
• GA4GH advocacy.
• Dedicated reference implementation funding.

• Melissa acknowledged the need but indicated that no such resources are currently available.
• She viewed the discussion as useful input for future planning within GKS.

18. Feasibility of Building a Validator

• Tim argued that modern AI-assisted development could make building the validator itself relatively straightforward.
• In his view, the harder problem is agreeing on:

• What constitutes a strict specification.
• Which validations should be included.

• He questioned whether a strict standard would be useful if the vast majority of existing files failed it.

19. Validation vs Repair

James highlighted an important distinction:

1. Detecting problems (validation)
2. Fixing problems (repair tools)

Examples already exist:

• SAMtools fixmate
• Similar correction tools in other ecosystems

His view:

• A high failure rate is not necessarily problematic if:

• Errors can be automatically repaired.
• The validator clearly identifies the issue.

• The real problem is ambiguity, where the correct fix is not obvious.

20. Why a Validator Matters

Daniel argued that strict validation is particularly important for VCF and structural variant data because:

• Many widely used tools produce non-compliant output.
• Multiple incompatible "dialects" of VCF exist in practice.
• Developers who faithfully implement the specification may obtain incorrect results when processing these files.

Benefits of a validator:

• Expose common specification violations.
• Improve interoperability.
• Help tool developers understand exactly where outputs diverge from the standards.

James added that a GA4GH-endorsed validator could improve ecosystem quality by giving developers an authoritative way to test compliance.

21. Lessons from Existing Validators

Tim noted that validation tools already exist (e.g. Picard's ValidateSamFile) and have historically generated complaints because they sometimes enforced requirements that were not actually specified.

James agreed, citing examples where validators effectively required optional fields or tags.

This reinforced the need to clearly separate:

• Core specification requirements.
• Strict consistency checks.
• Tool-specific recommendations.

22. Tentative Offer to Build a Validator

• Tim offered to build a validator if the group can agree on:

• A well-defined strict specification.
• A clear set of validation rules.

• He believes implementation is likely easier than achieving consensus.
• Daniel responded that:

• The validator should expose all possible checks.
• Users could choose which checks matter for their workflows.

• James concluded that while some validations will remain debatable, many are uncontroversial and could form the foundation of an initial implementation.

(Zoom AI generated)

2026-05-12

Chair: James Bonfield

Attendees: Ligen, Rob Davies, Michael Macias, William, Xuxiali, Schichen Zhu, Wunan, Bitao, Akash Rampersad, Wallace Minxian Wang, Lee Lichenstein

Meeting recording: https://us02web.zoom.us/rec/share/5g6O8TKfkU2hOmRC2fw7vTvshlpb9cyz26FAzuwnDoIg-hcL9Ra_V0JaDS1YdxWa.uGCOdD90ngYzGjLx

Meeting minutes:

PBGZ tool: https://github.com/PBGZip/pbgz

New Tool Introduction: PBGZ is a genomic compression tool focused on efficient large-scale sequencing data handling and usability.

• Benchmark Results: PBGZ ranked second in the 2021 data compression competition, proving strong compression and runtime efficiency.
• Data Volume Challenges: DNBSEQ-10 generates over 14 terabases daily; China National Center holds over 100 petabytes of sequencing data.
• Design Features: PBGZ compresses FASTQ data in independent streams, ensuring robustness and random access with block sizes of 32 megabytes.
• Compression Performance: PBGZ achieves 2.4% compression on NOAA STAC data, outperforming gzip and ORA in speed and size reduction.
• Future Developments: Plans include extending PBGZ to SAM, BAM, and enhancing modular, stage-specific compression for various genomic data types.

Notes

Genomic Data Compression Innovation

The team introduced a new genomic compression tool, PBGZ, designed to efficiently handle large-scale sequencing data with a focus on balancing compression ratio, speed, and practical usability (19:15).

• Benchmark success confirms balanced design goals: PBGZ ranked second in the 2021 global data compression competition's balanced category, showing strong compression and runtime efficiency, aligning with its goal to support large genomic data storage and reuse.

• The collaboration with Professor Mingxian Wang’s team underpins robust algorithm and infrastructure development.
• The design targets real-world needs beyond file size, including reliable storage and efficient data reuse in workflows.

• Application context highlights massive data scale and cost shifts: Platforms like DNBSEQ-10 generate over 14 terabases per 24 hours with annual capacity exceeding 28,000 genomes, while reagent costs dropped to around $1 per gigabase, shifting bottlenecks to storage (21:01).

• China National Center for Bioinformation holds over 100 petabytes of raw sequencing data, creating urgent storage and archive cost pressures.
• PBGZ aims to provide a compact archival layer that preserves raw read-level evidence, essential for large cohort projects retaining multiple data processing stages.

• Design philosophy emphasizes workflow, reference, and structural awareness: PBGZ aims to be aware of genomic workflow stages, reference genome provenance, and platform-specific data structures to avoid risks like pipeline ambiguity and reference misuse (23:01).

• This approach contrasts with generic batch compressors by tailoring compression to the biological and computational context.
• The tool’s workflow awareness improves data reuse and reliability for downstream genomic analysis.

• Technical architecture favors stream separation and block-based compression: PBGZ separates FASTQ data into logical streams (ID, base, comment, quality), compressing each independently using an optimized FC codec derived from BSC (26:27).

• Block sizes are roughly 32 megabytes, allowing parallel compression, random access, and block-level fault isolation, improving robustness.
• The format supports concatenation of compressed files and deterministic MD5 checksums after parallel compression, enabling workflow reproducibility and distributed processing (32:21).

Performance Metrics and Deployment

PBGZ demonstrates strong compression ratios and throughput while supporting flexible reference handling and practical workflow integration (30:31).

• Compression ratios outperform gzip and rival ORA: On NOAA STAC data, PBGZ compresses to 2.4% of original size compared to 17% for GZ and 2.7% for ORA, showing significant savings (32:21).

• On T7 dataset, PBGZ achieves 17% compression versus GZ’s 35.51%, highlighting consistent size reduction across datasets.
• Compression speed reaches up to 265.84 megabytes per second on an 8-core i9 desktop, ensuring practicality for large data volumes.

• Lightweight deployment with flexible reference genome usage: PBGZ packages compressed data with optional embedded references, avoiding the need to bundle large fixed dictionaries like ORA’s 600 MB dictionary (34:30).

• Supports multi-species genomes and customized references, increasing adaptability.
• Runs efficiently on both x86 and ARMv8 architectures, broadening system compatibility.

• Advanced data access features support workflow needs: PBGZ allows random decompression at block level, so users avoid decompressing full files; corrupted blocks do not compromise entire files (34:30).

• Supports streaming workflows with dynamic meta offset set to zero when seeking is unavailable.
• Integration with command-line pipelines and remote data transfer workflows enhances usability.

Strategic Positioning and Future Roadmap

PBGZ targets early sequencing workflow stages, complementing mature tools like CRAM used post-alignment, with plans to extend compression to downstream genomic data formats (29:30).

• Non-competitive, complementary positioning with CRAM: PBGZ focuses on raw FASTQ compression before alignment, while CRAM compresses aligned SAM/BAM data, addressing different workflow layers (29:30).

• This segmentation allows tailored compression approaches optimized for data structure and usage at each stage.
• Future development plans include extending PBGZ coverage to SAM, BAM, and other genomic data objects.

• Ongoing research and open-source availability: PBGZ is not yet production ready but is openly available on GitHub, encouraging community exploration and feedback (47:56).

• The team welcomes comparisons with dedicated FASTQ compressors and further benchmarking.
• Continued improvements will focus on robustness, compatibility, and workflow integration.

• Long-term vision embraces modular, stage-specific compression: Recognizing diverse cell types and sequencing platforms, PBGZ development prioritizes adapting compression techniques to each genomic data stage and structure (44:00).

• This approach aims to improve reproducibility, storage efficiency, and data integrity across complex genomic pipelines.

Operational Updates and Project Coordination

The meeting reviewed pending tasks and progress on specifications and documentation related to genomic data formats, highlighting slow recent activity but some forward movement (52:30).

• BCF documentation updates pending review: Clarifications on phase encoding and GQ fields are mostly drafted but await review from key contributors like Yossi and Peter Danichek (53:44).

• Some issues, such as grammar and string versus dot notation, remain unresolved due to absent maintainers.

• Support for long chromosomes in BAM/CRAM advancing: A new nomenclature approach enables querying chromosomes over 2 gigabases by splitting them while preserving coordinate continuity (55:50).

• Early HDSLib implementation can query these split chromosomes transparently, though further work is needed for tools like mpileup.

• Low recent activity on issues and pull requests: April saw no significant updates besides minor comments and fixes, signaling a quiet development period (57:38).

• Some fix-related PRs are close to resolution, with maintainers planning to close or verify.

• GA4GH plenary and conference participation discussed: Upcoming Singapore event includes plenary and connect sessions; attendance from the China team confirmed (59:56).

• The group considered potential topics like compression techniques, VCF futures, and sequence collections but noted the need for more implementation examples before deeper discussion.

Technical Exploration of BGZF2 and Related Tools

James Bonfield shared work-in-progress on BGZF2, a block compression format replacing deflate with zstd, aiming to support long read data and large multi-sample VCFs with enhanced speed and size (01:04:00).

• BGZF2 offers improved speed and compression over original BGZF: It uses larger block sizes and standard libraries like zstd, making it easier to implement and faster for long-read compression.

• It includes an improved self-indexing scheme within the zstd stream, simplifying random access indexing.

• Exploring OpenZL as a future codec alternative: OpenZL, by the same authors as zstd, offers better compression ratios but is still immature and has issues with standard description language support (01:06:40).

• Current challenges include incomplete ITF data description models and lack of dynamic tagging needed for BAM auxiliary data.

• Partial BAM compression success but auxiliary data remains a hurdle: Attempts to compress BAM with OpenZL are hampered by the inability to tag fields with specific codecs, limiting full implementation (01:08:17).

• Work continues on raw serial compression and testing of various codec profiles.

• Potential future role as a replacement for Tabix: BGZF2 and related codec work may excel in compressing and indexing tab-delimited genomic data like VCF and BCF, improving speed and ease of use (01:09:52).

• The work is ongoing with no immediate production release planned.

Community Engagement and Next Steps

The meeting concluded with acknowledgments and a request for Wallace to share presentation materials for broader dissemination (01:09:52).

• Presentation materials to be shared via email for inclusion in meeting minutes and wider community access.
• The team expressed interest in further exploring PBGZ and BGZF2 capabilities, encouraging collaboration and testing.
• The meeting closed with thanks, signaling openness for ongoing dialogue and incremental progress on compression technologies.

Action items

Wallace and team from China National Center for Bioinformation

• Share the GitHub repository link and presentation slides of the PBGZ compression tool with meeting organizers for wider dissemination and archival in meeting records (01:09:52)

James Bonfield

• Verify and close the issue related to BCF documentation on phase encoding and coordinate review with Yossi and Peter Danichek (53:44)
• Continue development and testing on handling long chromosomes split in BAM/CRAM formats, including multi-part chromosome querying and refinements (55:50)
• Circulate details and updates on BGZF2 format and OpenZstd experimentation to the community for evaluation and feedback (01:05:02)

Meeting organizers

• Consider and propose topics for the upcoming GA4GH plenary and connect sessions, focusing on compression improvements, VCF future, BGZF2 updates, and sequence collection implementations (59:56)

2026-04-21

Chair: John Marshall

Attendees: Rob Davies, James Bonfield, John Marshall, Tim Fennel, Michael Macias

Apologies: Reggan Thomas (GA4GH), Daniel Cameron

Meeting recording: https://us02web.zoom.us/rec/share/PxNVPtDT-poYM4xCr_oqAxeazpGon-MRVilEvkELyEOYQ2cy3U6T27W7YwNpHCKJ.FmdRLwrDqQjIXUGu

Meeting minutes:

• Connect Meeting Engagement: Limited participation due to time zone differences; key sessions like Community showcase noted for collaboration opportunities.
• AI Workstream Interest: Approximately 50 contributions noted; notable idea on AI-generated testing highlighted but needed follow-up on outcomes.
• CRAM 3.1 Progress: Achieved near parity in read speed to HTSLib; writing performance advice suggests uncompressed BAM for speed.
• VCF Character Type Issues: Proposal to deprecate ‘character’ type for clarity; need consensus on standards and specs discussed.
• Long Chromosomes Support: Issues with chromosomes over 2 Gb; proposed tagging SQ lines for split chromosome management discussed.
• CRAM 3.1 Adoption Challenges: Lack of release tagging on GitHub slowed adoption; communication improvements needed for better trust and integration.

Conference and Community Engagement

The team reflected on the recent Connect meeting with mixed participation and insights, highlighting opportunities for improving engagement and visibility.

• Limited participation and varied interest at Connect meeting impacted by time zone differences and agenda relevance (02:28)

• Attendees noted low turnout and limited excitement for some agenda items
• The Community of Interest showcase was identified as a key session for collaboration
• Some attendees missed sessions due to inconvenient timing or lack of appeal

• AI workstream launch generated significant interest with around 50 contributions on AI applications in genomics workflows (09:36)

• Andy contributed a notable idea on writing tight specifications to enable AI-generated test data and code
• The AI spreadsheet captured diverse use cases but lacked follow-up on detailed outcomes

• Beacon network and GIF project sessions highlighted integration efforts and community interaction (11:19)

• “Ask me Anything” sessions, like the one for crypt4gh, were successful in uncovering user questions and increasing visibility
• It was proposed to include such open sessions in future agendas to boost engagement and knowledge sharing

• Next Connect meeting planned in Kingston with better time zone alignment offers hope for increased attendance and participation (13:04)

• Travel budget constraints and fuel prices affect plans, but the location is more accessible for some team members

HTSJDK and CRAM 3.1 Development Progress

Significant progress has been made on HTS JDK integration of CRAM 3.1 with ongoing validation and performance improvements.

• Functional CRAM 3.1 read and write code achieved in HTS JDK with near parity in read speed to HTSLib (16:15)

• Read code performance is now within 20-30% slower than HTSLib, a major improvement from previous fivefold slowdown
• Writing performance remains slower; current advice is to write uncompressed BAM streams where speed is critical
• Large-scale validation using AWS instances processing several terabytes is underway to ensure code quality and correctness

• Implementation aligned closely with HTSLib behavior to resolve spec ambiguities and undocumented optimizations (17:48)

• Differences in insert size handling and serialization of NM/MD tags were clarified and aligned with HTSLib standards
• Spec inconsistencies are minimal; updates to the spec are planned if ambiguities arise

• Noodles Rust implementation supports CRAM 3.1 codecs with strong performance, but HTS Codex remains faster due to low-level optimizations (21:15)

• Rust codecs cover rans, adaptive arithmetic, FTZ comp, and name tokenizer
• HTS Codex’s handwritten assembly maintains a performance edge, particularly for rans n16 codec

• Cram 4 format considered nonviable for standardization due to complexity and limited adoption (23:13)

• Cram 4 includes unused features that add complexity and have caused security flaws
• Recommendation is to deprioritize Cram 4 and focus on improving Cram 3.x series

• Exploration of new compression codecs such as C Standard and libbsc suggested to optimize performance and compression ratio for long reads and large VCFs (26:28)

• Libbsc praised for speed and compression improvements over bzip2, though Apache license may cause integration issues
• ZSTD favored over gzip for handling large-scale genomic data compression challenges

VCF Character Type and BCF Encoding Issues

The group identified unresolved ambiguities in handling character arrays in VCF and BCF formats, proposing to clarify or deprecate ambiguous types.

• Significant confusion exists around the ‘character’ data type in VCF and its encoding in BCF due to vague spec definitions (32:03)

• Htslib treats character arrays as strings, while HTS JDK attempts to split on commas, causing inconsistent parsing
• The spec lacks clear guidance on comma separation or escaping for character arrays, unlike other list types

• Potential solution to deprecate the ‘character’ type in favor of strings was discussed to simplify parsing and avoid ambiguity (37:10)

• The ‘character’ type appears unused in official metadata but may exist in third-party annotations
• Deprecation would reduce confusion and align with the specification’s emphasis on comma-separated lists

• Plan to prioritize this issue in upcoming meetings to achieve consensus and spec updates (37:10)

• Without the VCF spec maintainer present, the team committed to advancing this topic collaboratively
• Defining clear semantics for character arrays and escaping mechanisms is considered feasible and necessary

Handling of Long Chromosomes and Sequence Collections

The meeting revisited challenges with supporting chromosomes longer than 2 gigabases and managing large sample sets in genomic data formats.

• Long chromosome support remains problematic due to tool and spec limitations, with pragmatic workarounds favored (41:02)

• BAM format and many tools cannot natively handle chromosomes exceeding 2 Gb
• Current practice involves splitting chromosomes into smaller parts with aliasing to preserve query functionality

• Proposed enhancements include tagging SQ lines to record split chromosomes and offsets to manage reads spanning boundaries (44:10)

• Similar approaches to circular sequence handling discussed to maintain data integrity and usability
• Draft implementations are needed to test and refine the approach before formal adoption

• Sequence collections do not currently support grouping split sequences with metadata to denote relationships (46:30)

• Nested collections are possible but lack explicit metadata for chromosome splitting
• Handling millions of samples and contigs creates practical challenges, with some data rejected by repositories as a result

• The Tree of Life project’s use case highlighted limitations and informed pragmatic decisions to avoid complex fixes (41:02)

• Sanger Institute’s experience showed that optional software tools do not handle long chromosomes well
• Splitting and aliasing remain the most feasible solutions currently in use

Software Release and Adoption Challenges

There are ongoing challenges in software release management and adoption impacting the use of updated CRAM support in production.

• HTS JDK released CRAM 3.1 read support months ago but did not tag the release on GitHub, limiting visibility and uptake (50:00)

• Lack of formal tagging delayed inclusion in package managers like Conda and Debian, slowing downstream adoption
• Maven Central artifacts were published and signed but not widely acknowledged outside JVM ecosystems

• ENA Read Tools recently updated to use HTS JDK 4.3 with CRAM 3.1 read support, though writing support remains incomplete (53:19)

• The writing codec was included but not fully wired into pipelines, leaving read support as the primary functional feature
• The update was quietly merged without a formal release, causing some confusion about availability

• Internal pipeline upgrades to CRAM 3.1 face resistance due to legacy workflows and concerns over storage cost (56:34)

• Some teams prefer older CRAM 3.0 versions despite known security vulnerabilities
• Data release groups are starting to report successful use of CRAM 3.1 in production pipelines

• The team emphasized the importance of clearer release tagging and communication to improve adoption and trust (50:00)

• Better release management would facilitate wider use and integration into downstream projects

Process Improvements and Pending Issues

The group touched on ongoing maintenance, API improvements, and prioritization of outstanding issues to enhance project stability.

• John acknowledged backlog on acslib improvements and documentation, with efforts focused on releasing PISAM updates soon (58:36)

• API changes such as adding destination buffer length fields are being considered to improve safety and performance
• The release is overdue but prioritized over other open-source tasks due to community demand

• Discussion around the 'multiple characters not a string' issue highlighted the need to raise its priority and take decisive action (30:17)

• This longstanding problem has seen no movement since last May and requires coordinated resolution
• The team agreed to include it in upcoming agendas to ensure progress

• Meeting minutes and AI-generated transcripts remain inconsistent, causing difficulties in tracking decisions and actions (41:02)

• Some meetings lacked transcripts due to technical issues or not being activated
• Improving note-taking reliability is important for transparency and follow-up

• Future file formats meetings planned with hopes of better attendance and clearer agendas to address pending technical challenges (01:00:29)

• The team expressed willingness to keep discussions ongoing and engage additional members when possible

Action items

Rob

• Add discussion on VCF character array type to the agenda for upcoming meetings to work towards detailed resolution of specification ambiguities (38:52)

Tim

• Continue development and validation of the HTS JDK development branch to ensure CRAM 3.1 compliance and improve performance against HTSlib benchmarks (14:44)

John

• Complete outstanding pull requests and finalize documentation to enable the forthcoming PISAM release (58:36)

Developer team

• Draft an implementation proposal for handling very long chromosomes by splitting them and using command line aliasing, building on prior exploratory work (46:30)

Rob

• Coordinate review and prioritization of stalled issues including the 'multiple characters not a string' problem and chromosome length support in specifications and implementations (29:00)

Participants

• Consider organizing more community engagement sessions similar to 'Ask Me Anything' to increase visibility and address common questions on file formats and standards (11:30)

2026-03-17

Chair: James Bonfield

Attendees: Michael Macias , James Bonfield, Rob Davies

Meeting recording: https://us02web.zoom.us/rec/share/ZeZ6l6C0sIStl7jv7rfXyT3R0jGH_Awwi6LQuhuSqwchoBvuayWsvqqYf_jlySIx.Pze2JDWwFyaxk9QI

Meeting minutes:

2026-02-24

Chair: Daniel CameronJohn Marshall

Attendees: Reggan Thomas, Michael Macias , James Bonfield, Rob Davies, Yossi Farjoun

Meeting recording: https://us02web.zoom.us/rec/share/MGIdpTYGLJcutc3VbORkgI2JagEIyWhQznB4tM-UaOlAFUa5769wFhr3apfzndds.vc_GOu4Xu_1giM4Q

Meeting minutes:

Rob Davies:        https://samtools.github.io/hts-specs/BEDv1.pdf#section.5

Summary:

GQ Specification Update: Clarified GQ field now pertains to genotype quality; no formal errata issued to maintain spec clarity.

End Field Deprecation: The "End" field is technically deprecated; a workaround will be provided to ensure backward compatibility.

Local Allele Float Fixes: Minor updates to LPL, LGL, and LGQ definitions merged; considered unnecessary for errata.

Circular Genome Challenges: Circular genomes complicate indexing; proposed solutions involve splitting reads for backward compatibility in streaming.

Funding Issues: Current volunteer effort equates to about 4 hours monthly, impacting spec maintenance and critical updates.

BED File Clarifications: BED files only contain coordinate lines; track lines are out of spec for BED and need filtering.

Notes

Specification Clarifications and Updates

The group focused on refining key specification details to improve clarity and maintain backward compatibility.

GQ Specification Clarification and Errata Decision 

• Daniel updated the GQ field definition to clarify it applies to genotype quality, not variant presence.
• The group agreed this is a clarification, not a formal errata, aiming to reduce ambiguity without rewriting the spec.
• They discussed practical deviations from the spec by popular callers like GATK and Dragon, accepting slight differences in implementation.
• The decision reflects a preference for terseness in the spec to avoid opening complex debates while maintaining practical usability.

End Field Deprecation and Backward Compatibility 

• Daniel noted the technical deprecation of the "End" field in version 4.5 and its potential to cause misinterpretation of older records.
• Proposed a workaround to avoid GVCF blocks and symbolic structural variants in the same record to maintain backward compatibility.
• The group plans to take more time to phrase this carefully to clearly warn users of potential issues.

A pragmatic approach is prioritized to balance technical correctness and user impact.

Fixes to Local Allele Float Definitions

• Daniel merged a PR correcting the LPL, LGL, and LGQ definitions to floats, aligning them with their non-local counterparts.
• The change was considered minor, and the group decided errata was unnecessary for this update.
• This fix was prompted by the first real implementation encounter revealing the spec error.
• Example and Formatting Corrections in BCF 4.5
• Daniel identified and raised an issue (791) for malformed examples in the BCF 4.5 spec, including incorrect use of semicolons.
• This discovery came from reviewing section 5.5 and highlighted the need for more thorough example validation.
• The issue is being addressed to improve spec accuracy and developer reference.

Circular Genomes and Indexing Challenges

Discussion revealed technical difficulties and proposed solutions for supporting circular genomes in indexing and streaming.

Indexing and Streaming Circular Genomes Issues 

• The group recognized that circular genomes cause reads extending beyond reference ends, complicating indexing and streaming.
• CRAM format does not natively support circular references, often truncating or encoding reads verbatim without circular awareness.
• Yossi suggested splitting reads overlapping the genome end into supplementary alignments as a backward-compatible workaround.
• This approach requires resorting data but simplifies streaming and indexing logic for circular references.

Pileup and Read Ordering Considerations 

• They noted pileup tools need reads overlapping the start and end of circular genomes to function correctly.
• The index lookup for the start region must also query the end region to capture overlapping reads.
• Streaming pileup without an index is more complex and out of scope for current spec changes.
• The group agreed that splitting reads is a pragmatic solution to ensure streaming compatibility.

Handling NM Tag for Split Reads

• Debate arose over whether NM (edit distance) should count soft-clipped bases in split reads across circular genomes.
• The consensus was that NM counts only aligned bases, ignoring soft clips, so NM should be zero for perfect matches despite splitting.
• This subtlety emphasizes the need for practical implementation experience before finalizing spec language.

LSG Group Structure and Strategic Direction

The group engaged in a candid discussion about LSG’s current status, challenges, and potential future organizational changes.

Concerns Over LSG Engagement and Effectiveness

• The group acknowledged declining participation and less available time from members over recent years.
• John reflected on the early days with more active Broad participants compared to the current small, fatigued core.
• Yossi and others expressed a need for revitalization or structural change to maintain relevance and productivity.
• Options considered included increased coordination with GA4GH or restructuring.

Potential for LSG to Join Other GA4GH Work Streams

• They debated whether to retire LSG and merge its work into other GA4GH streams.
• Concerns were raised about losing practical focus since LSG handles heavily used real-world file formats.
• Tight technical coupling between formats like SAM, VCF, and CRAM complicates splitting responsibility.
• The group favored maintaining LSG’s independence to preserve pragmatic, implementation-grounded work.

Need for Regular Larger Meetings and Task Representation

• Proposal to hold quarterly broader LSG meetings (“LSG connect”) to share ideas and boost community energy.
• Discussion on appointing a second LSG representative to the GA4GH Task Working Group to improve influence.
• Emphasis on recruiting beyond format-specific leads to encompass broader LSG interests.
• These steps aim to strengthen coordination and visibility within GA4GH.

Funding and Maintenance Challenges

• The group lamented the lack of funding and driver projects specifically supporting LSG’s spec maintenance work.
• Current volunteer effort equates to about 4 hours per month, limiting progress on critical updates and reference implementations.
• John highlighted the absence of a GA4GH-wide maintenance model for specs post-driver projects.
• This gap threatens sustainability and timely evolution of crucial standards.

Future of VCF and Format Evolution

Discussion covered the stalled progress of future VCF format development and the need for scalable binary alternatives.

Status of Future VCF Initiative and BCF Replacement

• The Future of VCF group succeeded in clarifying local allele fields and fixing legacy issues.
• However, the larger goal to replace BCF with a scalable binary format stalled due to lack of resources.
• External projects like Savvy and Czar developed alternative formats offering better efficiency and scalability.
• The need for a community-driven, efficient binary format remains urgent, but stalled without funding.

Community Frustration and External Solutions 

• External groups have independently developed replacements and approached GA4GH for adoption.
• The current lack of progress in Future of VCF risks fragmentation and lost opportunity for standardization.
• The group recognizes this as a significant technical and organizational challenge requiring serious investment.

Emerging Needs Around Graph-Based Formats

• Recognition that graph genome formats are another growing area requiring standardization efforts.
• No current capacity to address this due to limited volunteer time and funding.
• This represents a future challenge for the group’s scope and resources.

HTSJDK Release and CRAM 3.1 Support

The team discussed the stalled official release of HTSJDK supporting CRAM 3.1 and its impact on downstream users.

Release Tagging Without Official Publication

• James explained that HTSJDK 3.4 has a tagged release with CRAM 3.1 reading support but no formal GitHub release.
• This prevents package managers like Conda and Debian from picking it up, causing distribution issues.
• The lack of official release creates confusion and slows adoption.

Validation and Rollback Issues at EBI 

• Andy Yates and Dave Jackson reported EBI rolled back CRAM 3.1 adoption due to HTSJDK issues.
• Andy validated round-trip reading with the tagged release and found it satisfactory, though writing tests remain pending.
• The problem stems from HTSJDK’s maintenance status and slow release processes.

Community Leverage and Governance Challenges

• The group has some ownership leverage within Fulcrum but faces limitations due to Broad’s reluctance to hand over control.
• The desire is to transition HTSJDK to a community-maintained project with clearer governance.
• This is complicated by Broad’s interest in retaining prestige and control over HSJDK and Picard.

BED File Format and Specification Maintenance

The group addressed ongoing issues with BED file format parsing and the challenges of maintaining less active specs.

BED vs BED Graph and Track Lines Distinction

• The group clarified that BED files strictly contain coordinate lines, while track lines belong to the BED graph extension.
• Track lines, although widely used, are officially out of spec for BED files.
• Tools parsing BED files with track lines should treat them as extensions or filter them out.

Spec Maintenance Gaps and Updating Plans

• James acknowledged outstanding minor fixes and typos in the BED spec that have not yet been addressed.
• The lack of active maintenance reflects broader challenges in sustaining less popular specifications.
• The group plans to consolidate these fixes before the next meeting to improve the spec’s accuracy and usability.

Action items

Daniel

Continue refining wording around the deprecated 'end' field in 4.5 spec with appropriate warnings and backward compatibility advice

Errata the LPL, LGL, and LGQ type corrections in 4.5 spec following confirmation of a small but needed specification fix

Update the BCF 4.5 specification example to fix semicolon and length field formatting errors, tracking via issue 791

Complete minor fixes to the BED specification ahead of the next meeting to improve clarity and consistency

Meeting participants

Gain practical implementation experience with circular genome indexing and pile-up handling to inform specification updates

Meeting organizers

Plan and confirm upcoming meeting schedule considering participant time zones and potential overlap with GA4GH Connect

Leads

Organize a larger, cross-product team meeting (quarterly or similar cadence) or GKS Connect event to improve engagement and coordination across GA4GH

Participants in LSG and TASC

Identify a second representative from LSG to join the Task committee to improve representation and involvement

Daniel and LSG representatives

Engage in discussions with GA4GH leadership about establishing a sustainable funding and maintenance model for HGS specs outside driver projects

James and relevant developers

Coordinate with HTSJDK repository owners (Chris and others) to publish an official HTSJDK 3.4 release that fully supports CRAM 3.1 for wider adoption

Picard contributors

Treat BED track lines as extensions outside official BED specifications; preprocess files to remove track lines if necessary

2026-01-20

Chair: James Bonfield

Attendees: Daniel CameronRob Davies, Yossi FarjounMichael Macias

Meeting recording: https://us02web.zoom.us/rec/share/BePsMQ1fDNZy17I9HkBqSuACSwBUgsetlP4jQj6HiSbRj-TlxOtqL6v_1rVXs7Mt.7O60d3SbcLy308yq

Meeting minutes:

The meeting discussed the challenges and updates related to VCF and BCF formats. Key points included the need for better documentation on Bayes encoding in BCF, the ambiguity of GQ field interpretation, and the potential for circular chromosome alignment issues. The team debated the best practices for handling circular chromosomes, including using supplementary alignments and smart indexing. They also reviewed the status of HTS JDK, which is in survival mode, and the need for a new release. Additionally, the team discussed the implications of missing values in SV claims and the importance of clear data provenance.

Action Items

• Draft a concrete proposal clarifying the meaning and computation of GQ (genotype quality) and present it for review
• Add verbiage to the draft/spec (provide written text to clarify the referenced item as discussed)
• Respond to Adam and formally reply about ALT symbolic allele handling and the validity of missing values for SV/DUP/DEL (advise that missing is not valid and provide recommended alternatives)
• Cut and publish the HTS JDK release (formalize who is responsible for cutting releases and push the tagged release)
• Add and document the proposed idea about representing sequence collections (provide the suggested SQ/HD-line approach and comments for the spec)
• Add a comment to the circular-chromosome/indexing PR or documentation clarifying indexing/query implications for circular references

Discussion on VCF and BCF Formats

•    • James Bonfield and Yossi discuss the evolution and current state of VCF and BCF formats.    • Rob Davies mentions that BCF is still popular for its niche uses.    • Daniel Cameron explains the issues with custom fields in VCF and the advantages of BCF.    • James Bonfield and Yossi discuss the future of BCF and the need for better support for large multi-sample cohorts.  

Review of Previous Actions and New Issues

•    • James Bonfield reviews the previous actions and mentions that most have not changed.    • Speaker 3 mentions a few changes, including the need to expand BCF documentation on Bayes encoding.    • James Bonfield and Daniel Cameron discuss the ambiguity of GQ field in VCF and its interpretation.    • Yossi explains the Bayesian model for genotype quality and the need for clear documentation.  

Discussion on Sequence Collection and Indexing

•    • James Bonfield and Yossi discuss the need for sequence collection in BCF and SAM.    • Rob Davies mentions the challenges of merging BAM files with different sequence collections.    • James Bonfield suggests using HD lines or RG lines for sequence collection.    • Yossi proposes using SQ lines for sequence collection and the challenges of indexing circular chromosomes.  

Circular Chromosomes and Indexing Challenges

•    • Yossi raises the issue of circular chromosomes and indexing challenges.    • Daniel Cameron suggests using supplementary alignments to handle circular chromosomes.    • Yossi discusses the position of mates in circular chromosomes and the need for smart indexing.    • James Bonfield and Rob Davies discuss the potential solutions for indexing circular chromosomes.  

Discussion on BTF and SV Claims

•    James Bonfield and Daniel Cameron discuss the validity of missing values in SV claims.    • Daniel Cameron explains the intentional exclusion of certain values to avoid ambiguity.    • Yossi suggests using symbolic alleles to handle non-standardized SV types.    • James Bonfield agrees that the current approach is logical and practical.  

AOB and Future Plans 

•    James Bonfield mentions the upcoming GA4GH Connect in April and the need for session proposals.    • Rob Davies asks about the status of HTS JDK, and Yossi confirms it is in survival mode.    • James Bonfield requests the release of HDS JDK 3.1, and Yossi agrees to handle it.    • The meeting concludes with a brief discussion on the challenges of maintaining code bases and the importance of clear documentation.  

2025-12-02

Chair: James Bonfield

Attendees: 

Meeting recording: https://us02web.zoom.us/rec/share/jVT2TsTGTHpbbAeHErj3o3hLlMndy6zJI9k4WfkRqzLnr7To-iAdvIThxVhthWp0.W95prvyUE-Z5_BDJ?startTime=1764691177000

Meeting minutes:

1. Introductions

• Mitchell introduced himself as a new Assistant Professor at the University of Utah, with prior work at the University of Washington developing computational tools for Fiber-Seq / FiberSeek.
• Attendees introduced their roles across Sanger, St. Jude, GA4GH Secretariat, and tooling ecosystems (e.g., Noodles).

2. Proposal Overview

• Mitchell proposed a new tagging approach to annotate sequence features on individual long-read molecules, independent of alignment.
• Intended use cases include chromatin accessibility, nucleosome positioning, and regulatory feature inference.
• Proposal includes multiple tags to encode:

• Annotation start positions (per molecule, original orientation)
• Annotation lengths
• Optional quality values
• Optional annotation names

• Motivation includes enabling standardized visualization (e.g., IGV) and broader tool interoperability.

3. Key Discussion Points

• Early Engagement: Group agreed it was appropriate to engage GA4GH early before implementations become entrenched.
• Implementation Expectations: GA4GH typically prefers seeing multiple independent implementations or examples before standardization.
• Coordinate System: Discussion on zero-based vs one-based coordinates; alignment with existing SAM conventions remains a concern.
• Data Volume & Compression:

• Annotations may range from ~1–500 per read.
• Separating starts, lengths, and qualities may improve columnar compression, but this should be validated empirically.

• Quality Values:

• Strong preference for optional quality fields to avoid fabricated values.
• Missing/absent quality indicators favored over mandatory placeholders.

• Orientation & Robustness:

• Agreement that annotations should be relative to original read orientation to survive pre-alignment processing.
• Discussion highlighted lessons learned from base-mod tags regarding sequence edits and coordinate drift.

• Existing Formats:

• Related but limited existing tags (e.g., PT) were discussed; not considered a good fit for this use case.

• Tag Naming & Governance:

• Draft or “beta” tags (e.g., mixed-case or experimental naming) recommended prior to formal specification.

• Tooling Support:

• No immediate requirement for HTSLib parsers, but tooling support strongly influences adoption.
• Rust/Noodles and IGV integration were discussed as likely future paths.

4. Commitments & Signals of Interest

• PacBio has indicated willingness to support IGV visualization if the format matures.
• Interest expressed in potential Rust-based parsing and eventual inclusion in tooling ecosystems if standardized.

5. Action Items

• Mitchell to:

• Generate example datasets using the proposed encoding.
• Test and compare file size/compression impacts of different tag configurations.
• Experiment with the format using custom or draft tags before formal proposal.

• Group to:

• Review examples and reconvene once implementations and benchmarks are available.
• Consider feedback from additional stakeholders and tooling maintainers.

6. Next Steps

• Proposal to mature through experimentation and limited adoption.
• Mitchell to return with updated examples, metrics, and potential PR when ready.