Difference: DocStdRFC2 (3 vs. 4)

IVOA Document Standards RFC (Version 1.2)

Discussion about any of the comments or responses should be conducted on the Standing Committee on Standards & Processes mailing list, stdproc@ivoa.net.

For reference, here is a link to the first RFC (Version 1.1)

The comments (M. Deimletner, M. Taylor) posted on the page open for comments on V1.2 WD are copied below.

Comments

Minimal implementation rules

I'm not sure if this is one of these "not again" topics, but I'd really like so see something on the implementation side. Let me dream for a moment: What if the document said something like "A PR describing protocols, formats or similar computer processible artefacts can only become REC if (a) at least two independent implementations [fantasy: one of them in a C-linkable language; this would almost immediately allow bindings for basically all major languages] are available and (b) a validation suite is provided that allows implementors to verify the conformity of their implementation [for what it's worth]".

As a matter of fact, I think the validation suite would be worth more than all words and probably even formal specifications, since the behaviour defined by it is behaviour clients can rely on.

Now, I admit that if something like this were to become part of the standards process, more diligence is needed as to the wording, and certain details would have to be worked out (if I had my way, I'd require the implementations to be published under a DFSG-free licence, for starters). But I'm convinced the VO would be more fun to work and more robust with if rules like these were adopted. -- MarkusDemleitner - 17 Feb 2009

Me too

I welcome the introduction since v1.0 of a reference to validation tools in the WD->PR promotion rules (section 2.1). However, the language is pretty pusillanimous:

"Preferably, the WG should be able to demonstrate two interoperable implementations of each feature, and validation tools should be available. If the chair of the WG believes that broader review is critical, the chair may advance the document to Proposed Recommendation even without adequate implementation experience. In this case the document status section should indicate why the chair promoted the document directly to PR."

I have two issues with this text.

1. The language here makes it sound like two interoperable implementations and the validation tools are very much an optional extra. I would like the document authors to justify why these should not be made a requirement, or at the very least a condition for which explicit exemption (perhaps by the TCG) is required.
2. The phrasing seems disingenuous to me. The document may be advanced to PR if the chair believes (and how this 'belief' is to be assessed is another matter) that broader review is critical. That sounds just about plausible, since PR means a requirement for feedback from other WGs. However, the requirement for two implementations and validation tools is not reiterated at the PR->REC boundary. So there's nothing in the DocStd document to say that a document can't go from WD to REC with inadequate implementation experience simply because the WG chair believes it needed broader review while it was still a WD. If that is really what's meant (and I hope it is not) then it ought to be made explicit; if not, the implementation requirements should be reiterated more strongly in section 2.2.

I prepared the above comments before reading Markus's contribution above. Emboldened by him, I'll state it a little more strongly: why on earth shouldn't two interoperating implementations plus comprehensive validation suite be an absolute requirement? In my experience, writing the implementation is a breeze compared to thrashing out the standard. Unless of course the standard is so written as to be difficult or impossible to implement. Writing and testing an implementation is absolutely the best way to find out whether this is the case or not.

-- MarkTaylor - 18 Feb 2009

Having slept on it I think maybe I should backtrack slightly - if there is a comprehensive validation suite, I think that a single implementation is probably sufficient. Two independently written bits of interoperating software is a good test of a standard - either two implementations or an implementation and a validation suite (which is a kind of implementation in any case). Having a validation suite is a massive help when it comes to ensuring that later implementations are correct.

-- MarkTaylor - 19 Feb 2009