[a]Maybe this doc needs a version? And possibly any badge issues or used needs a version, too? Just in case the attributes get stricter (or less strict) over time, it might potentially be important to know whether an implementation is GA4GH Implementation 1.0.0 or GA4GH Implementation 5.3.1

1 total reaction

Oliver Hofmann reacted with 👍🏻 at 01/03/2024, 02:04 UTC

[b]Agreed. If approved it will be versioned appropriately. For the moment it's still a WIP

[c]Really like this.

[d]Thanks anonymous, but could you mention who you are please? Just so we know where our comments have come from. Thank you

[e]Link?

[f]From the list below, we are not only listing compatible services around the globe, but stuff that could be deployed elsewhere.

E.g. If UKBioBank implements a Beacon, but does not provide the software for it. Would it be entitled to be a GA4GH Implementation?

Am I right?

There is a reason for that approach?

[g]To answer the question no. If UKBioBank are not open source it's not a GA4GH Implementation.

See "Implementations versus a GA4GH Implementation"

[h]But to avoid confusion, it might still make sense to clarify/highlight somewhere that only codebases/repos not individual _deployments_ can be GA4GH Implementations (which, e.g., is different from the resources listed in a GA4GH Service Registry API implementation, which would list individual deployments/instances)

[i]Agreed I think this is where the major confusion has come from in the document is the difference between

- An open implementation

- A closed implementation

- A deployment of one of these items

[j]I think this misses a review / maintenance step. How frequently are acknowledged implementations reviewed to see if they still meet these requirements, or that they have kept up with changes to the underlying standards?

[k]Fair point. What would you consider to be a fair cadence?

[l]Perhaps a GA4GH Implementation would denote only a particular commit/tag/release? That would be quite strict, but also the safest.

[m]Yes good point it should be asserted for a software version + standard version e.g. refget-server-perl all versions are only compliant with refget v1.x

[n]We now have a 'remove' step so mostly good. Maybe mention who does the reviewing - i.e., who checks if a given reference implementation is still considered to be current.

[o]Do we want to set-up a form?

[p]Eventually yes, but at the moment no. I want to get the idea approved by the wider GA4GH which then lets us put more effort behind it

[q]Ideally those should be mailto: links?

[r]Agreed; maybe as email aliases, e.g. CSO@ga4gh.org so that these can stay current.

[s]👍

[t]How is this proved and to whom? What is the process and what should a developer expect from GA4GH? 

Some guidance &or case examples after "GA4GH Implementation attributes" would be helpful.

[u]Agreed this is a component that we have missed & flagged partly by Stephanie Li

[v]I am also wondering how this could be done without too much overhead in the beginning, i.e., in a kinda "good faith" situation where we almost all know each other. While still allowing us to go to mostly/fully automated stricter checks later, if/once the necessary workflows/services/infrastructure become available (e.g., automated compliance certification, interop matrix etc).

For the former case, I'm fine with this doc as it is. But for the latter case, I'm afraid it may not be structured enough.

Still, it's better to have something and let it grow organically, IMO. But perhaps it's a good idea to clearly state that no claims whatsoever can be made against GA4GH or the service developers based on the GA4GH Implementation badge.

[w]Good faith should get us quite far and then automation can be built much later on

[x]Personally, I would prefer to only have specific versions of a service be GA4GH Implementations. The next commit could blow it all up, and that could be very hard for users to understand.

Rather than allowing implementers to add the shield to a README unconditionally, we could ask that it is clear that only one or some commits/tags/releases have been verified as a GA4GH Implementation, e.g., in a table of releases, on a release page, or by (somehow) including the tag or 7-char commit SHA in the badge and linking to the corresponding commit etc.

[y]Good point like it

[z]Could describe how this interacts with/links with the GA4GH registry https://www.ga4gh.org/product/service-registry/ (i.e. I think this catalogue is a strict superset of what would be in the GA4GH registry but I'm not sure)

[aa]Service registry is indeed linked to this work and we will provide more information on the links

[ab]I'm not sure I understand you Denis - what is the catalogue that would be a strict superset of what would be in which GA4GH registry?

Note that (from my understanding) the catalog of implementations lists code bases, while a service registry lists deployments. So I don't think they are dealing with the same type of resource.

Also, note that we (ELIXIR Cloud) have an implementation of the GA4GH Service Registry API in which we add any service that we consider part of the federated ELIXIR Cloud network. As far as services that implement GA4GH standards are involved, the entries in there would be strict subset of a hypothetical all-knowing centralized GA4GH service registry. However, in practice there will likely never be such a single registry with all service deployments everywhere.

[ac]Hah, to be clear I don't understand this section so I'm paraphrasing in the hopes of a correction or clarification.

To elaborate on my guess, this section does seem confusing to me. The text before this section talks about implementations but this section talks about "A demonstration deployment must be registered with the proposed compliance test infrastructure" and then "based on our testbed reporting infrastructure."

So I was assuming that this section talked about deployments of implementations.

So what's the relationship between this catalogue and the registry https://github.com/ga4gh/ga4gh-registry

is what I'm asking.

Getting off-topic, I kinda assumed that https://registry.ga4gh.org/v1/services would eventually become that hypothetical all-knowing service registry though I totally understand it may not get there.

[ad]Good points. Let's see what Andy says, as I'm also only working based off my own projections of what I'm getting from this.

And I think you are right about being GA4GH"s centralized service registry being located at https://registry.ga4gh.org/v1/services. But as you say, this will never list all services. Apart from the fact that people will forget (or simply not know about) registering their deployments there, there will be many cases where people just don't find it desirable or useful to list services there. For example, I might be running a big company and deploy a network of WES and TES and DRS services internally, together with a service registry to be able to discover them, all strictly for internal use.

[ae]Yeah so we need to split this out more explicitly. We've confused concepts here for sure

[af]Above? Below in the table? There should be a link, as it is ambiguous. Especially if it is followed by a list of things that it does NOT mean

[ag]Ah yeah that's a good point. Will restructure

[ah]This is important in any comms

[ai]Agreed

[aj]My gut feeling is that external "readers" would simplify it and, eventually, understanding it as we don't want. 

Myself, as non-native English, am reading "GA4GH Implementation" as "Implemented by GA4GH". It doesn't sound to me as "GA4GH compatible" or "GA4GH verified".

[ak]Good point Jordi. We need to stay away from any language that indicates GA4GH is offering product accreditation, which is why we went towards implementation. But also the point here is the these items are not meant to be Implemented by GA4GH. We should reconsider the name if that is a connotation to anyone

[al]I also find "GA4GH Implementation" a bit confusing. Isn't "reference implementation" the more conventional designation for well documented, best practices but possibly limited (and buggy!) implementations of a standard?

FWIW, in the badge, where "GA4GH" and "Implementation" are separated, I find it less confusing/ambiguous.

[am]"GA4GH Reference Implementation" crystalizes the concept for me

[an]I mentioned this earlier that I wanted to stay away from the use of reference in the name. There's nothing that says these have to be reference implementations at all. Taking the definition from https://en.wikipedia.org/wiki/Reference_implementation the line I believe clashes with this concept is

The reference implementation often accompanies a technical standard, and demonstrates what should be considered the "correct" behavior of any other implementation of it.

A standard could have multiple ga4gh implementations, but this suggests there is a singular reference implementation. Being open source doesn't necessarily mean mean you have built software in a way that is clear but you probably would do for a reference implementation

[ao]Fair point, makes sense to me.

[ap]GA4GH approved implementations? Endorsed? All not ideal.

[aq]Does *not* mean that GA4GH *endorses* the GA4GH Implementation's security and compliance…

But a GA4GH Implementation must meet basic expectations of security and compliance

[ar]Not only support. Also the meeting of the attributes cannot be guaranteed beyond the point where it was reviewed and found to meet the criteria.

[as]Good point

[at]I struggle with this. How do we sign off on an implementation meeting 'minimal requirements' when we are not willing to confirm that testing has been performed here?

[au]There is a difference between testing of integration i.e. can two components talk to one and another, and testing that an implementation is compliant against its compliance kit.

Integration is the key word

1 total reaction

Alex Kanitz reacted with 👍 at 29/02/2024, 21:25 UTC

[av]other implementations of the same standard? does integration mean using a GA4GH standard to exchange information with a separate entity?

[aw]Integration would indicate the ability of an implementation of a GA4GH standard to be dropped into a system as is.

e.g. two implementations of refget exist. A software environment is using refget like Ensembl. If both implementations pass their compliance kits, then they should be drop in replacements for each other. No downstream client or system should need to change.

In this case exchange between GA4GH standards or with other information systems doesn't matter.

[ax]I follow now - thanks!

[ay]This is a major concern for us. I'd also make it clear that support is limited to the current features of the implementation (and not all optional bits).

[az]We have to accept that support is hard for some groups, and there might be ways of mitigating this with help from GA4GH or another mechanism. What we don't want is implementations becoming dead and incapable of responding to reasonable requests

[ba]I'm also struggling with this.

To me, the term "Support" implies an active response from the implementer. However, what is described below are just "passive" options.

[bb]The majority of work though is best effort. You're right the options below are passive, but without these options I fear nothing would qualify as we would set too high expectations

1 total reaction

Alex Kanitz reacted with 👍 at 29/02/2024, 21:29 UTC

[bc]Agreeing with Andy, we will probably need to keep the bar rather low here. And perhaps by issueing GA4GH Implementation badges that are tied to _specific commits/tags/releases_ of a service, this is not too much of an issue, as it doesn't give the impression that a whole project is bestowed that honor.

[bd]Thanks and agreed on the versions

[be]Maybe consider some kind of self-policing requirement or defining 'Implementation' such that the tracker/catalogue has to point the specific version of the implementation, i.e. every new version also has to comply.

[bf]Good point.

[bg]I found this confusing; registering all GA4GH Implementations to create a catalog of all GA4GH Implementations, in which GA4GH Implementations will be flagged? If the catalog is only GA4GH Implementations, wouldn't everything be flagged?

[bh]Agreed this is not clear. The idea is the larger catalogue tracks everything but that GA4GH Implementations are flagged in it.

[bi]I guess this is the same confusion that Jordi was mentioning earlier...

[bj]Agreed

[bk]Creating this path is important. We want to support GA4GH developed standards in Epic and use them to integrate with other genomic data consumers and producers but would never be able to meet many of the requirements outlined (I doubt many industry implementations would be interested in / able to do so).

[bl]Agreed we want to catalogue as many implementations as possible, but also flag those that conform tot he criteria. We do hope that some industry implementations could be open source but understand they will not be the majority

[bm]It would be great to discuss what it means for an industry implementation to be open source. For example:

We publish specifications, testing tools, and validators for APIs and interfaces that interoperate with Epic software on open.epic.com today.

We commit support to our customers and their trading partners and vendors via existing agreements. 

Most of the implementation code we need to write is specific to our system, so it wouldn't be useful for anyone else for genomic data exchange purposes. Publishing that code would increase security and privacy risk for our customers, though.

[bn]This is Catherine :)

[bo]Certainly we need to explore this properly and thank you

[bp]FWIW, what I would expect from an open source implementation of a GA4GH API is that the entire code necessary to deploy a working version of the service implementing the API is available for anyone to read. So either a microservice, or in the case of a monolith, probably the majority or even the entire code base.

But of course, that's my own expectation/interpretation. And perhaps it's not all that important. What really matter is that I can rely on implementations A, B  and C to be interoperable inasmuch as the common standard they implement prescribes. If that expectation is fulfilled and I can find instructions on how to use A, B and C, then it is up to me to figure out which I want to use. Maybe I don't want to use C because not _all_ of the codebase is available for me to read, or maybe I don't want to use B because I need to pay for it. But that's not something that we necessarily need to classify - that's really up to the user.

So from what I am reading, for your case, what's more interesting is compliance testing and showing up as a compliant implementation that people would want to consider. And you probably don't want to show up as a GA4GH Implementation as defined here - that is, sort of a well documented, best practice reference implementation - and a potential building block to build more complex systems with.

[bq]I follow. It's true - we wouldn't be a reference implementation. It's not that we want people to use our implementation. Instead, we want to promote adoption of the GA4GH standard in question by "advertising" that it can be used to interact or transact with our products.

2 total reactions

Alex Kanitz reacted with 👍 at 01/03/2024, 00:39 UTC

Oliver Hofmann reacted with 👍🏻 at 08/03/2024, 04:00 UTC

[br]We also wanted to avoid the use of the phrase reference implementations. I look at some implementations like https://github.com/DNAstack/data-connect-trino and have to say I don't think it's a reference implementation at all, but it more than likely covers the attributes below

[bs]I think I'm finally with you now, Andy! Thanks.

[bt]Absolutely needs a link if specifically mentioned as an example here

[bu]👍

[bv]doesn't mean approved

[bw]IMO, this should be a separate attribute, given that standard compliance is absolutely critical

[bx]This would restrict a number of software from becoming GA4GH Implementations because of this lack, but this should hopefully incentivize the process

[by]Speaking of that - for the last two years, a Google Summer of Code contributor for GA4GH has been working on a generic OpenAPI test runner and test spec. So far we have a set of TES compliance tests for that, and we are working on WES now. In principle, it should be easy to add tests for additional specs. However, the project needs broader support, now that the contributor moved on (or funding to keep him). Especially because some important changes (e.g., the full separation of the runner/spec from the test suite) are still in the middle of completion.

We could discuss this in TASC or elsewhere if you like, Andy. The project is here: https://github.com/elixir-cloud-aai/openapi-test-runner

[bz]Who runs the validation testing tools?  Is this self-reporting?  Run from the service registry?  I think it's critical to have some level of testing to get a badge otherwise I'm concerned that implementations that are not compatible with the specs could actually still get a badge...

[ca]@jimmy.payyappilly@ga4gh.org can point you to the questions about the conformance testing infrastructure

[cb]Perhaps another metric to list here is code coverage?

[cc]I stepped away from code coverage as I was unsure of the right metric to use. Would it be lines, function, statement, edge or condition coverage?

I'd appreciate a way of formulating the use of testing methodologies like code coverage, cyclomatic complexity or similar. 

Maybe it just needs adding to the last point which points towards CI & liniting