IVOA Web>IvoaVOTable>RfcPage (2009-11-10, RobSeaman)

VOTable1.2: Request for Comments

An ammended version ~~PR-VOTable-1.2-20090929.pdf~~ PR-VOTable-1.2-20091102.pdf was produced following your comments on this RFC page; the sections concerned are

4.3 (xtype): no value specified any more

4.5 (UCD): recommendation to use UCD1+ but explicit acceptation of UCD1

4.8 (INFO): no sub-element any more (compatibility with previous versions)

8 (Differences with previous version) corrected accordingly

Appendix B: a remark was added about additional attribute; the elementFormDefault is qualified following discussions on the TAP list on this topic.

It is hoped that this last version is acceptable; thanks for your interest! (FrancoisOchsenbein 2009-09-29 + 2009-09-30 + 2009-11-02)

The proposed recommendation is now open for review by the TCG, from 6 October 2009. It is hoped that the TCG review will be achieved before the next Interop meeting, i.e. before 2009-11-06

This document will act as RFC centre for the ~~VOTable-1.2-20090710~~ VOTable-1.2-20090929 Proposed Recommendation.

Review period: 2009 July 10 – ~~2009 August 21~~ 2009 Sep. 9 (09/09/09)

In order to add a comment to the document, please edit this page and add your comment to the list below (include your WikiName so authors can contact you for further information). When the author(s) of the document have considered the comment, they will provide a response after the comment.

Discussions about any of the comments or responses should be conducted on the VOTable mailing list, votable@ivoa.net.

Comments from the community

Sample comment by WikiName
- Response (by WikiName)

From MarkTaylor

I will flag up here a couple of issues that I've noted in earlier drafts of this document. Francois and I disagree on them; I will concede if other reviewers of the document don't regard them as problematic.

As I and others have argued elsewhere (http://www.ivoa.net/forum/votable/0903/date.htm) the changes to the INFO element may cause backward compatibility issues. These issues are probably not too serious, but may cause annoyance, and I'm not convinced that justification for the change has been demonstrated.
As I have argued elsewhere (http://www.ivoa.net/forum/votable/0907/date.htm) I do not believe that the introduction of the spurious "encoding" attribute on the TD element is justified.

A related item which I've only noticed more recently, in the course of implementation is:

The new "ID" attribute on the TR element has apparently been introduced for the same reasons as the TD "encoding" attribute, and I'm against it for the same reasons. However, it is even more problematic, since correctly processing an xs:ID-type attribute is expensive because of the need to maintain crossreferences (I can explain in detail on request). There would be major impacts on STIL's efficiency in VOTable streaming if it took this ID attribute seriously.

There are two other changes from VOTable 1.1 which I have queried in earlier drafts (http://www.ivoa.net/forum/votable/0903/0999.htm) and which have not been addressed or discussed further:

The TABLE element now permits multiple DATA elements. I can't see this is a good idea - I presume it's a mistake in the schema. Should be corrected.
The VOTABLE and RESOURCE elements now permit a GROUP to appear as a direct child. This is probably reasonable in order to group PARAMs which are already permitted there, but this change should be documented in section 8.

Section 8 "Differences between 1.1 and 1.2" is useful but unfortunately incomplete. The changes since v1.1 of the VOTable Recommendation that I've spotted which are not documented here are (as elaborated on above):

New "encoding" attribute on TD element
New "ID" attribute on TR element
TABLE element permits multiple DATA children
GROUP element permitted as direct child of VOTABLE and RESOURCE

At least the first three of these I'd like to see changed, as noted above. However, if they are not, they should be recorded in section 8, preferably with reasons. Otherwise, the only way that VOTable authors and implementors, and reviewers of the document, are likely to spot them is by very close reading of the schema text itself, which is hard work.

Regarding the new placement of INFO elements (post-operational diagnostics) - I don't have any objection to this in principle, but is there a good reason to allow INFOs at the end of all the elements DATA, TABLE, RESOURCE and VOTABLE? Since an INFO can appear after a DATA, it seems unnecessary complication to allow it as the last element of DATA as well. It turns out that implementing this presents some difficulties in STIL. If there's good reason for it, I will work through them, but if there's no good reason for it I'd suggest to remove that possibility, so that the INFO can appear only at the end of TABLE, RESOURCE and VOTABLE.

There is also a small number of typos and minor items:

Section 3: "Each Resource element contains one or more TABLE elements" - should be "... zero or more ...".
Section 4.1: "elment" should be "element".
Section 4.3: "expresses a spatial of temporal coordinate" should be "... spatial or temporal ...".
Section 4.7: The new paragraph beginning "section 6 indicates ...": I don't believe that a null scalar "char" value can be represented with an empty TD element. Also, use in this paragraph of the phrase "default null value" seems a bit confusing to me; I suggest instead the phrase "representation of the null value" or "null representation".
Section 5.2: The example here contains a PARAM element with text content; this is not permitted.

-- Answer by FrancoisOchsenbein:

Thank you Mark for the detailed comments. I try to summarize in a few items the problems and issues:

The compatibility issue of the INFO tag: This point has been discussed and re-discussed for 2 years now; it was quite generally considered as minor by the various IVOA components -- it was e.g. straightforward to adapt Aladin and Savot. The structured <INFO> has definitive advantages for extensibility (see the continuation of the thread of your message, e.g. http://ivoa.net/forum/votable/0903/1004.htm)

The discussion by my count consists of one person (Francois) who has voiced support for the change and five (Alberto, Doug, Mark, Ray, Roy) who have at various times voiced opposition.

The compatibility issue has less to do with implementation changes in software (granted, these are not very onerous) than the implications for existing standards.

Regarding extensibility: the VOTable section in the current draft IVOA TCG Roadmap says "Once VOTable1.2 is accepted as an IVOA standard, we would consider that the VOTable WG could switch its status into a maintenance mode, i.e. its activities would essentially consist in a follow-up of IVOA activities to ensure that no conflict between the VOTable standard and the other IVOA standards would emerge". In this context, it sounds like the expectation is of only minor future changes, so I don't understand why it's important to build in the possibility for future extensibility, at some cost of backward compatibility, now. -- MarkTaylor - 24 Sep 2009

The schema vs document The XML schema does not represent the definition of the standard --- the document does define the actual standard. Therefore using the attribute "encoding" for TD or "ID" for TR is not legal according to the standard in its 1.2 specification (if present these attribute can be ignored). But failing to use the published XML schema to generate code for application resulted in a considerable trouble and I want to avoid as much as possible such difficulties; isn't it our role to facilitate the usage of our standards as much as possible ?
A note in section 8 (differences between 1.1 and 1.2) about this feature is not really appropriate; a note presently exists as a comment in the XML schema; if there is a strong feeling that this should be more visible, a paragraph at the beginning of Appendix B would seem more appropriate there than in the list of modifications.

TABLE with multiple DATA children could be fixed easily in the xml schema (after the appropriate tests)

GROUP as child of VOTABLE/RESOURCE: since the GROUP replaces the COOSYS, it looks natural to allow the GROUP where the COOSYS was allowed. This feature could be added in section 8, agreed.

Section 4.7: not sure to understand what you mean by I don't believe that a null scalar "char" value can be represented with an empty TD element -- what's wrong? Floating-point and character data have both a natural null representation (nan and NUL).

AlbertoMicol - 04 Sep 2009

I fully support Mark's comment on the INFO backward compatibility issues. It is an unnecessary burden for many of us, without a clear immediate advantage (as I understood it, it is only done to allow future extensibility). Especially important given that SIA 1.0 and SSA 1.04 relies on the INFO tag to provide QUERY_STATUS information and SERVICE_PROTOCOL versions; examples quoted from PR-SIA-1.0-20090521 and SSA-20080201 (REC):

<INFO name="QUERY_STATUS" value="OK">  Successful Search</INFO>
<INFO name="QUERY_STATUS" value="ERROR">DEC out of range: DEC=91</INFO> 
<INFO name="SERVICE_PROTOCOL" value="1.02">SSAP</INFO>

Hence, those protocols should be ammended, and all existing SIAP and SSAP services would need an upgrade, unless IVOA accept and handle multiple versions of those protocols requiring different votable versions (quite an unnecessary complication and a burden for applications developers).

-- Answer by FrancoisOchsenbein

SIA (as well as the ConeSearch) are not good examples because these recommendations correspond to protocols used since years but missing official recommendations until recently; and during the RFC period it was argued that nothing could be modified because these conventions are in use (BTW this concerns also the UCD1+ usage)

From MarkusDemleitner

First off, +1 from me on the "first three issues" comment by Mark. But I have another one: I'm unhappy about the inclusion of the sample document with embedded STC in 3.1. In effect, it raises that note to a recommendation status, and I think it's too early for that since there's been very little discussion about it. So, I'd like to propose that the sample document is removed.

Background: The note has some issues I've sent to Francois, but mainly I think it's wrong to clobber the column's utype attributes for something so common -- it would mean that specific data models could not assign utypes of their own to columns carrying STC information (or clients would have to know all these data models to know what STC roles are behind these specific utypes). Also, libraries have a much easier time if STC info remains confined to one element rather than it being all over the VOTable. So, my suggestion was to reference the the columns from the AstroCoords group rather than the other way round, something like

<GROUP utype="stc:AstroCoords.Position2D.Value2.C1" ref="RA1"/>

But well, that's something to be discussed elsewhere. I just think we shouldn't promote the note to de-facto recommendation just yet.

-- Answer by FrancoisOchsenbein

I don't think this standard means a de-facto recommendation for the note; the STC is an example of how we can assign very accurate meaning to the data included in a VOTable. Your suggestion doesn't describe the coordinate system, epoch and/or equinox -- which is really important for the data consumer.

-- Remark by MarkusDemleitner

Just to clarify: My suggestion is to have the stc:AstroCoordSystem groups more or less like they are now (though I think they could be flattened, i.e. the PARAMs could be direct children of that group); so that would remain. In stc:AstroCoords, however, instead of untyped FIELDrefs, you would have groups like the one above; this would free up ref and utype on the FIELDs themselve for less generic data models. Anyway, I'd still prefer if there were some language in the recommendation to the effect that the STC example is an example for utypes etc. rather than some demonstration for correct STC notation.

From NormanGray

MIME Types The document proposes using application/x-votable+xml for the VOTable MIME type: I don't believe this is appropriate for a standard that is intended to be taken seriously. RFC 4288, sect 3.4 says:

   These types are unregistered, experimental, and for use only with the active
   agreement of the parties exchanging them. [...] it should rarely, if ever, be
   necessary to use unregistered experimental types.  Therefore, use of
   both "x-" and "x." forms is discouraged.

Registering a separate media sub-type, as the FITS community did, may be appropriate (I've argued for this in the past, but acknowledge that this is potentially a non-trivial amount of work). If this really is too much work, then a 'vendor tree' registration (see sect 3.2) such as application/vnd.ivoa.votable+xml might be appropriate, not least because I think it has lighter registration requirements. Sect 3.2 says While public exposure and review of media types to be registered in the vendor tree is not required, using the ietf-types@iana.org mailing list for review is strongly encouraged to improve the quality of those specifications, and this might be worth following up.

utypes: There is not yet any utypes standard document; is it appropriate to refer to utypes in an IVOA standard before utypes are themselves standardised? (unless standardisation of the current pre-draft is regarded as a foregone conclusion)

Code generation: for what it's worth, I also agree with Mark's remarks above about adding gratuitous attributes in order to work around claimed deficiencies in code-generation software. (i) there still doesn't appear to have been any convincing demonstration that this is a real problem (so it's sounding like the 'XML parsers reorder elements' myth), and (ii) if code-generation software produces wrong outputs for correct inputs, then ... use different software.

-- NormanGray 2009 Sep 6

-- Answer by FrancoisOchsenbein

The mime type was discussed several times and the terms of the document reflects, I think, the consensus which was reached. Trying to register a mime-type is a long-term enterprise; and personnally I prefer the term 'experimental mime type' rather than a 'vendor tree' wink

And yes an official document on utypes is missing (started by JonathanMcDowell, a long time ago; I don't know what's its current status).

From PatrickDowler

xtype: While xtype is useful in services like TAP, the constants (STC-S and iso8601) are not that useful. One could use xtype="stc:Region" in place of STC-S and have the same meaning, with the additional clarity that stc:Region is a type while STC-S is a format (note: stc:Region has an implied STC-S format, but in services the allowed formats should be specified by the service; I am not sure if this is really the best choice from stc). Similarly, one could just use xtype="stc:ISOTime" directly. While the iso8601 xtype could be used in tap, the STC-S value does not distinguish between regions and points. In working on another problem, I came to the conclusion that point would be better handled by something like

xtype="stc:Position2D" datatype="double" arraysize="2" ref="the_coordsys"

and then the content is the usual comma-delimited pair of numeric values, and the ref connects the values to a common coordinate system. It is possible that the stc xtype(s) should have more extensive context that just the class names shown above... a detail.

Given that this can/should be done via STC (or other specifications that define datatypes), I think the pre-defined constants should be removed from the VOTable spec. They will just provide an additional choice and confusion for service implementors and application writers and STC already provides types for these two values.

-- PatrickDowler 2099/09/09

-- Answer by FrancoisOchsenbein

The proposed changes on the xtype attribute do not reflect the discussions about the necessity of introducing this new attribute for the TAP needs during the last Interop meeting. In your example, a utype would do exactly the same thing as your proposal, with a just a few characters more:

utype="stc:AstroCoords.Position2D.Value2" datatype="double" 
  arraysize="2" ref="the_coordsys"

therefore I don't see the necessity for a xtype any more ? My understanding was the xtype is required in order to express the quantities used in ADQL queries -- not stc definitions.

From RayPlante

These comments are restricted to typos and minor presentation items:

S3.2, p.10, "...a TABLE and the corresponding /TABLE tags,...": I didn't really get that this meant the opening and closing tags until I saw it done again in section 8. Can we change this to "...a TABLE tag and its corresponding closing tag, ..."?

S3.3, p10, "meaningfull": should be "meaningful"

S3.3, p10, "...to make ue of...": should be "to make use of"

S4.3, p13, "...requires data="char"...": should be "...requires datatype="char"..."

-- Answer by FrancoisOchsenbein: Thanks, will be fixed

Comments from the TCG during the normal RFC period (2009 July 10 -- 2009 August 21 September 09)

Applications (Tom McGlynn, Mark Taylor)

Data Access Layer (Keith Noddle, Jesus Salgado)

Data Model (Mireille Louys, AnitaRichards)

Grid&Web Sevices (Matthew Graham, Paul Harrison)

Registry (Ray Plante, Aurelien Stebe)

I have two TCG-level concerns:

I concur with the above RFC comments regarding the compatibility issues raised by the change to the INFO tag definition. This change means that SIA cannot use VOTable 1.2. My impression is that this change is motivated mainly for purposes of consistency and with possible future use, rather than a demonstrated problem experienced today. I would recommend reverting to the old definition.

In section 4.5, I think it would important to have an explicit statement indicating that the use of any UCD standard (e.g. v1.0 or v1.1) is allowed. This reflects current practice.

-- Answer by FrancoisOchsenbein

for the INFO tag, the answer is above. Reverting to the previous INFO definition means many changes in the document / schema. I don't know whether another RFC round would then be required ? (I would definitively prefer to avoid this!)

RayPlante comments: I don't think the necessary changes would be so great as to require another RFC; the revision would simply be in response to the RFC. In particular:

in the document, replace the wording in s4.8, 1st paragraph with the wording in v1.1, s2, paragraph 4 ("An informative parameter (INFO) is a restricted form of the PARAM....")

while I haven't considered all the implications, it seems that we can keep the change that allows INFO's use in additional locations.

in the schema, uncomment out the "OLD INFO definition" and comment out the new one.
We can pass these changes by all who commented on this topic; that, I think would suffice.
Again, I point out that this is not needed to solve current problems but rather causes problems of backward compatibility with other standards. Thus, if this is a really important evolution for the future, then perhaps it can tabled for a future when we are less dependent on ConeSearch and SIA.

the decision for UCD1+ was already effective for VOTable-1.1 -- difficult to revert to V1.0

RayPlante comments: I may have missed this, but I do not see a definitive statement in v1.1 that mandates the use of UCD1+. What I see in section 4.4 is:

a vague statement that UCDs are evolving

examples using UCD1+
If we assume that VOTable indeed requires UCD1+, does that mean that VOTable v1.1 cannot be used with Cone Search and SIA? In fact, many implementations of these services use VOTable v1.1.

RayPlante, in response to revision 20090929: I approve these revisions. Thanks!

Semantics (Sebastien Derriere, Norman Gray)

Seeing the remark from the Registry WG on section 4.5, if it is stated that any UCD standard is allowed, can we at least state that the ones standardized by IVOA are preferred?

RayPlante comments: Yes, that is what I'm suggesting.

Other than that, I approve the document. SebastienDerriere

-- Answer by FrancoisOchsenbein : as specified above the choice for UCD1+ is already effective.

VOEvent (Rob Seaman, Alasdair Allan)

VO Query Language (Pedro Osuna, Yuji Shirasaki)

VOTable (Francois Ochsenbein)

Standard and Processes (Francoise Genova)

Astro RG (Masatoshi Ohishi)

Data Curation & Preservation (Bob Hanisch)

Many expert eyes have gone over this document carefully, so I only focused on the changes between V1.1 and V1.2 that are noted in Section 8. These all seem in accord with the recent discussions.

Theory (Herve Wozniak, Claudio Gheller)

TCG (ChristopheArviset, Severin Gaudet)

TCG Review (from 06/10/2009 to 06/11/2009)

During the TCG review, Working and Interest Group chairs should add their comments under their name:

Applications (Tom McGlynn, Mark Taylor)

Thank you for reconsidering the content of the INFO element; the changes made to the draft which restore backward compatibility with previous versions of the standard remove the most serious objection to the text.

I remain unconvinced that that there is good motivation for the spurious TR/ID and TD/encoding attributes; in the first place I haven't seen persuasive evidence that these attributes do actually fix any code generation issues (I think it's also possible that they will introduce new ones), and in the second place while I agree that in case of conflict the document text should override the schema, I don't feel that this is a good reason to allow uncontrolled introduction of misleading clutter to the schema. However, with this reservation, the Apps WG approves the document.

-- MarkTaylor - 01 Oct 2009

Data Access Layer (Keith Noddle, Jesus Salgado)

From the DAL perspective I am happy to approve the document. I would however like to see resolution - or at least a plan for the resolution - of the stated issues alongside the final submission. I firmly believe that we need to get standards to a point where they are sufficiently fit for purpose (Good Enough) and then release them. In the final analysis, it is only in the heat of widespread development and deployment that the flaws and opportunities will be found and solutions produced.

-- KeithNoddle - 05 Nov 2009