IVOA Web>SimpleDALRegExt10RFC (2013-09-19, SeverinGaudet)

Request for Comment: SimpleDALRegExt v1.0

This document serves as the RFC center for the Proposed Recommendation entitled "Describing Simple Data Access Services, Version 1.0". The version reviewed during the RFC can be found at http://www.ivoa.net/Documents/SimpleDALRegExt/20120517/. The version revised for TCG Review can be found at http://www.ivoa.net/Documents/SimpleDALRegExt/20121116/.

RFC Review period: May 23, 2012 - June 27, 2012 Completed
TCG Review period: 4 Dec 2012 - 4 Jan 2013
Exec Approved for REC:

Attention TCG: if you prefer printing out a PDF version for review, consider this abbreviated version. It omits the inclusion of the appendices listing the full XML schemas, resulting in 50% fewer pages.

To add a comment on the document, please edit this page and add your comment to the list below in the format used for the example (include your WikiName so that authors can contact you for further information). When the author(s) of the document have considered the comment, they will provide a response after the comment.

Additional discussion about any of the comments or responses can be conducted on the Resource Registry mailing list, registry@ivoa.net. However, please be sure to enter your initial comments here for full consideration in any future revisions of this document

Notes on Implementations and Validaters

The SimpleDALRegExt page includes several compliant sample VOResource instances that use the extension schemas defined in the spec.. Instances can be validated with an XML Schema validater such as the one provided with Junx (which is based on Xerxes).

Previous versions of the extensions for ConeSearch, SIA, and SSA, which the PR versions are backward-compatible with, have been in production use by IVOA-compliant registries for several years.

Comments from the community

Sample comment by WikiName
- Response (by WikiName)

Comments from MarkTaylor

A couple of comments about the schema:

Data types: Several quantities are declared as xs:int type, for instance the sia:SIACapRestriction maxFileSize and maxRecords elements. Is this intentional? xs:int is a 32-bit quantity, which at least for maxFileSize seems like an unnecessary/unwise restriction. xs:positiveInteger or xs:unsignedLong might be more appropriate. Elsewhere xs:float and xs:double are used in different places, e.g. siaSkyPos uses xs:double and sia:SkySize uses xs:float. Is there a rationale for which floating point type is used where?
- RayPlante responds:I agree with moving to xs:positiveInteger and xs:double for these numeric items; will change in next version. I note that this can be considered a backward-compatible change.
Required elements: Most of the limit-type elements (maxRecords, maxFileSize) are declared as required. Is this a good idea? Some services may have no hard restriction on the number or size of data returned, or may not know at capability specification time what such limits are. In that case services will have to make something up, e.g. 999999999, which may make clients think that a particular limit exists when it doesn't really. Any reason not to make most of these optional? slap:maxRecords seems particularly anomalous: the table in sec 3.4.2 notes that this element is required, and also that if it is not specified there is no predefined hard limit.
- *RayPlante responds:* If I were to try to recall the rational, I would bet it would be historical. Nevertheless, I agree, based on our experience now, that making these optional is better than 9999999 for the reasons Mark describe; will change in next version. I note that this can be considered a backward-compatible change.

And some minor comments on the text, mostly typos:

Abstract: "describe a services" -> "describe a service"
Abstract: fundemental -> fundamental
Abstract: Stadard -> Standard
Sec. 1: "... form a query ... and send it to all konwn services which support that protocol". For cone search at least this is rather a bad idea and not to be encouraged (tools like AstroScope provided, then deprecated this facility), since it submits far too many queries to be scalable or useful. This point could be rephrased to discuss sending the same queries to a moderately-sized group of compliant services matching some additional selection criteria.
Sec. 1: "can be search in one query" -> "can be searched in one query"
Sec. 1: "supports one of the protocol" -> "supports using one of the protocols"
Fig. 1 caption: "SimpleDALExt" -> "SimpleDALRegExt" (twice)
Sec. 2: Missing close parenthesis following "extension schema [VDS]"
Sec. 2: "unambiguosly" -> "unambiguously"
Sec. 3: "each extension schema have" -> "each extension schema has"
Sec. 3: "fixes the value its standardID" -> "fixes the value of its standardID"
Sec 3.1.2: "cs:ConeSearch type is vr:Capability sub-type" -> "cs:ConeSearch type is a vr:Capability sub-type"
Sec 3.1.2: "The custom metadata that the sia:ConeSearch type provides" -> "... cs:ConeSearch ..."
Sec 3.1.2 metadata elements table: the Comments written under the testQuery element should be in the verbosity section instead
Sec 3.1.2.1: "at least matched record" -> "at least one matched record"
Sec 3.2.2: "cs:SIACapRestriction" -> "sia:SIACapRestriction"
Sec 3.2.2.2: "axes may not particularly parallel" -> "axes may not be particularly parallel"
Sec 3.2.2.2: "latitdude" -> "latitude"
Sec 3.3.2: "is vr:Capability sub-type" -> "is the vr:Capability sub-type"
Sec 3.3.3: "for compliance final SSA Recommendation" -> "for compliance with the final SSA Recommendation"
Sec 3.4.1: "The namespace prefix, sia" -> "The namespace prefix, slap"
Sec 3.4.2: "The sia:SimpleLineAccess type" -> "The slap:SimpleLineAccess type"
- *RayPlante responds:* All good catches; will change.

-- MarkTaylor - 19 Jun 2012

Comments from TheresaDower

(pardon any major formatting issues; twiki is not being kind to my preview/editing attempts)

I will admit I am unsure how much this document is supposed to be an administrative collation of the existing standards and how much we intend to change each of them with an eye on backward compatibility. This may or may not be the stage at which to address inconsistencies between the standards, but I can't think of a better one. (It may also be well past time, as I note we've already gone over the RFC date but are requesting more feedback.)

For example, the Cone Search test query is required. It is not required for any of the other service types. ( Sec 3.2.1:) In practice, I have seen registered cone searches handle this by populating testQuery with bogus values, leading to non-responsive test queries. In practice, I've also seen validators fail non-Cone Search services 100% fullstop for having no optional testQuery.
- *RayPlante responds:* This is good feedback based on experience; will make this optional
Likewise, SSA's ( Sec 3.3.2)) maxFileSize is optional, SIA's is required, and it's often populated with an intentionally meaningless value of 0.
- *RayPlante responds:* Agreed; will change as per above response to MarkTaylor.
I know there have been and continue to be good reasons for encouraging data/service providers to provide more and more metadata as the standards grow in complexity, but if there is a time to look back and make a consolidated standards document more internally consistent between standards, this is it.
- *RayPlante responds:* We'll have to strike a balance here. We can certainly provide consistency with regard to what's required as recommended in these comments. However, an important aim of this standard has been expediency: to provide endorsement for practices that are already in place. This implies preferring backward compatible changes. Further consolidation that might be useful (like refactoring to collect together common metadata) would open up questions that are not easily/quickly resolved (like, is ConeSearch still encouraged, and do we need to tweak the underlying protocol documents.). If we pass on these larger questions, some inelegance will inevitably remain.
These comments are obviously related to MarkTaylor's general ones about the creeping number of required fields. Many of these values are difficult or impossible for service providers to calculate for some catalogs and thus many of these fields get filled with bogus information, which is less than useless to validators and client software. (If I have understood RandyThompson's offline comments correctly, Sec 3.2.2: sia:MaxImageSize/Extent stand out as a particularly difficult example for large catalogs with variably rotated images from different instruments.) Even if we don't want to tackle every field in each standard right now, it would be good to look at some of these and either make them optional or provide guideline default values in more cases than we already do.

-- TheresaDower - 23 Jul 2012

Comments from Randy Thompson

Comments and questions (as sent to Ray last month)

* General:

Why is "test query" required for cone search services and optional for others? I think many people would like this required for all services.
Why is maxFileSize required for SIA and optional for SSA?
- *RayPlante responds:* will make optional as per above response to TheresaDower, MarkTaylor

* SIAP:

maxImageSize - I see MaxImageFileSize listed in 6.2 of the SIAP document and later in Appendix B as MaxImageSize. Are these all the same value? I guess this is really a SIAP document issue.
- *RayPlante responds:* The error is in the SIA document, specifically in Appendix B. The purpose of this appendix is to show the registration terms identified in the SIA 6.2 get represented in VOResource; the names in column 1 of the table in the appendix do not necessarily match what's actually used in as shown in column 3. The error is that "MaxFileSize" in column 1 should read "MaxFileSize". (Admittedly, they all probably should say "MaxImageFileSize".)

ImageFormats - the MIME types for provided image formats is listed as required for registering SIAP services (section 6.2 of SIAP V1.0 doc.) but I don't see it mentioned in this document.
- *RayPlante responds:* Formats should be specified via the VODataService element, "format". (SIA Appendix B should say this but does not.) I'll add a note to this effect.

ImageSize 3.2.2 - I've always assumed that maxImageSize was simply the maximum image size in pixels along the x and y axis, so registering a service describing an archive of 10,000, 500 x 200 pixel images would just have an image size of 500 x 200. However the definition seems to require that the sizes be given along the axes of RA and Dec which means the orientation of the images becomes significant and the calculation more involved. Plus the orientation may vary from image to image making the definition more confusing. For example, if the orientation for the above image archive can vary such that the long axes is parallel to the RA axis in one observation and parallel to the Dec axes in another, what is the value of maxImageSize?

Related to this, it might help to reword this sentence in 3.2.2.2:

   "The sky coordinate axes may not particularly parallel with the image axes
   (e.g. due to rotation or projection effects); in this case, the
   longitudinal side should be considered the side of the image that it
   cuts the smallest angle with."

Why not just define maxImageSize as the x,y dimensions in pixels of the largest image that can be requested? I suspect this is what most data providers do anyway.

- *RayPlante responds:* I think the text you quoted reflects my struggle trying to interpret the original text from SIA S6.2, which is terse. I agree that your interpretation--where which number is long and which is lat is unimportant--makes more sense. The best fix would be to not use "long" and "lat", which would be a non-backward compatible change.
  
  I note that non-backward compatitble changes are not out of scope. In the coming months registry maintainers will need to upgrade many of these records due to changes in VODataService; we could fold in changes in these schemas at the same time. I should send a note to the list about this.

RA,Dec vs Long,Lat - When a metadata value specifically requires RA and Dec, I'm confused why they're labelled as Long and Lat. It would certainly make sense if galactic or ecliptic coordinates were allowed, but if RA and Dec are required, why not call them RA and Dec?
- *RayPlante responds:* made moot by previous response.

maxImageExtent,maxImageSize,maxFileSize,maxRecords 3.2.2 - I'm surprised to see these all listed as required even though its stated as such in the SIAP document (section 6.2). I believe these have never been enforced in the VAO.
- *RayPlante responds:* see previous response re: required metadata.

* SSAP:

* supportedFrame 3.3.2 - this is listed as required, but I don't recall seeing it used in any registered services. Perhaps it is considered optional if only ICRS is supported?

- *RayPlante responds:* This point was a point of discussion at the Pune interop and it was concluded that making this required made things a bit more future proof.

* ProtoSpectralAccess 3.3.3 - We have several registered services supporting both SSAP V1.0 and its predecessor (frequently referred to as version 0.4). We list them under separate capabilities within one resource as was discussed with VO staff a few years ago. In section 3.3.3 it states:

"A VOResource resource description must not include both a
    ssap:SimpleSpectralAccess capability and a
    ssap:ProtoSpectralAccess capability that describe the same
    service base URL, as given by the <interface>'S<accessURL>."

I'm not sure what

"<interface>'S<accessURL>"

means, but does this sentence imply our services are no longer allowed?

- *RayPlante responds:* That snippet might be hard to parse: it means the accessURL within the interface element; I'll re-word. The intention is not to disallow services of type ProtoSpectralAccess. This stipulation just reflects the fact that you cannot have an SSAP service, as identified by its accessURL, that is simultaneously compliant with both versions.

-- RandyThompson - 05 Aug 2012

Comments from TCG members during the TCG Review Period: 04/Dec/2012 – 04/Jan/2013

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or not the Standard.

IG chairs or vice chairs are also encouraged to do the same, althought their inputs are not compulsory.

TCG Chair & Vice Chair ( _Séverin Gaudet, Matthew Graham )

Approved -- SeverinGaudet - 2013-09-19

Applications Working Group ( _Mark Taylor, Pierre Fernique )

Looks good, just a few errors/queries, mostly in section 3.3.2:

The ssap:CreationType type has as one of its enumeration options the value specialExtraction . I think that should read spectralExtraction .
The SSA creationType element documentation refers to SSA section 2.5.1; that should read 2.5.2.
The SSA supportedFrame element documentation refers to the IRCS system; should read ICRS.
The SSA maxFileSize element documentation Value Type entry is blank for some reason.
The ssap:PosParam type contains long and lat elements which are declared as optional (minOccurs="0"). Not sure if there is some good reason for that, but I can't see why you'd want a position with either or both coordinates missing (sec 3.3.2.2).
In sec 1 it says "the ability to create image cut-outs is irrelevent only to the Simple Cone Search protocol". The word "only" has been introduced since the last version, I don't think it should be there.

Once those are considered, Apps recommends acceptance. -- MarkTaylor - 2012-12-07

From RayPlante: all good catches; a draft with these corrections can be found on the SimpleDALRegExt page.

Data Access Layer Working Group ( Patrick Dowler, Mike Fitzpatrick )

1. Is there a way to specify allowed enumerated values in a capability custom <param> element? Specifically, if a service has some custom params with a small number of allowed values, this can say be expressed in the VOTable metadata response. Is that the only way or can they also put it in the registry record?

2. For SIA services, the maxImageExtent and skySize should specify units are in degrees.

3. Why is there no 'complianceLevel' for SIA? At least minimal/full values that say whether the 'should' is implemented or not.

4. Under SSA, 'maxFileSize' says "maximum image size" in the semantic meaning

Comment: I was able to implement VOSICapabilities for the CADC SIA service using this spec; the only problem was finding the schema file (at the time) so I could perform validation. http://www.cadc.hia.nrc.gc.ca/sia/capabilities

Clarification or explanation to improve the document is a sufficient response.

Approved

From RayPlante:

Note that describing parameters is done via vs:ParamHTTP which is defined by the VODataService extension, not this one. Strictly using this, it is not possible to show allowed values; however, it might be possible with an extension on the datatype class. Also, the vs:InputParam type also allows for arbitrary attributes (i.e. from beyond the VODataService schema); though admittedly, this would not be an optimal mechanism for listing allowed values.

note added.

This is mainly historical. The desire for the SCS and SIA extensions was to keep them as close to the versions that have been in use in practice successfully for several years. IMHO, I do not find complianceLevel particularly helpful in practice as it is too broadly defined. For example, several of the optional parameters are only applicable to cutout services (and I doubt any such service supports them all), so imageServiceType is a better indicator of whether it might take such parameters. The "query" level essentially admits that it is non-compliant; thus, the service should not be marked with this capability.

currently says, "maximum image file size"

A draft with these corrections can be found on the SimpleDALRegExt page.

-- PatrickDowler - 2013-05-17

Data Model Working Group ( _Jesus Salgado, Omar Laurino )

Approved

Some small comments (with response from RayPlante):

Introduction: typo -> whereever (RayPlante: thanks)

Section 2: This is not probably applicable to this document (more to DataService) but, in order to substitute the current SSAP approach for theoretical services with a VOResource approach, ParamHTTP (BaseParam) should allow for min/max values and vocabulary restrictions on the values (the idea behind TSAP and S3 is to allow the creation of an input form on the fly). Is this possible?

RayPlante: The generation of input forms was in part the motivation behind the vs:ParamHTTP from the VODataService schema (see reply to Pat's comments above), but there was reluctance at that time to bake too much in without a present driver. I think that a replacement for this type could accommodate this without too much disruption. (But as you note, this is outside the context of this spec.)

OK. We will take note of this as a possible enhacement of a future updated version of VODataService, then -- JesusSalgado - 2013-09-12

3.2.2.1 SkySize: In general, the allowed values are not to clear for some types. For example, SkySize seems to be in degrees but we only can know it by reading the help (as far as I know). It could be useful to allow for element restrictions (minInclussive, maxInclussive) that would help the validation. This is applicable to many other types like, e.g. SkyPos

RayPlante: At first impression, I thought that adding these restrictions would be fairly straight-forward and non-disruptive--which I think is largely true--however, as I got into, I saw not only the changes to the schema, but also all of the verbiage changes required in the spec document. My worry is the possibility of introducing more errors into the document, requiring more careful review of the revision. Given that we are trying to wrap this up (after a longer review period than intended), I'm inclinded not to make this change as it would not likely buy us much benefit compared to the risk of introducing errors. What's your feeling? Can we pass on this?

I note that we explicitly state in the VOResource spec that the schema is not the sole standard of validity and that validaters need to go beyond the schema to fully validate a resource record. This choice was made explicitly to help cut down on complexity within the schema documents.

I understand the risk so we can pass on this for this version -- JesusSalgado - 2013-09-12

Appendix: There are references to some xml namespaces not present in the location (like

http://www.ivoa.net/xml/VOMetadata/v0.1

It would be good to correct this in order to ensure a proper parsing.

RayPlante: I'm glad you noticed this--you actually looked at the details The VOMetadata namespace is something I introduced many years ago assist in the in-line documentation of schemas. It has no standard schema of its own, nor is one required to parse the IVOA schemas and instance documents nor to validate them. (It provides mark-up at the very top of the schema file for identifying some information not easily accessible else where; this mark-up allows a style-sheet to incorporate that info into, say, an HTML rendering of the schema; I use it to autogenerate some of the tables in the spec.) As this is not critical to IVOA operations, I can happily remove its use; however, history has shown this, I think, to be benign.

What's your preference: okay to leave in or take it out?

Not sure. If we leave then, the unresolved link should be "solved" at certain point. If we remove it, we can loose its possible use. I let you to decide what is the best approach. It is not a showstopper for me anyway -- JesusSalgado - 2013-09-12

A draft with corrections can be found on the SimpleDALRegExt page.

-- JesusSalgado - 2013-06-10

Grid & Web Services Working Group ( Andreas Wicenec, Andre Schaaff )

Nothing to add to the already existing comments. Go ahead.

Registry Working Group ( _Gretchen Greene, Pierre Le Sidaner)

In the SSA extension, "specialExtraction" should be "spectralExtraction", as specified in the SSA spec. (Mark mentioned this.)
All but the ConeSearch had some (at least subtle) non-backward-compatible changes. Thus, the version numbers should be incremented to v1.1; otherwise, some currently compliant records will become uncompliant. SIA has already been incremented as such.
The documentation for the SSA extension should note that MaxAperture measures a diameter and that the maximum value is 360 deg.

Following these corrections RWG approves. -- GretchenGreene - 2012-12-14

From RayPlante: a draft with these corrections can be found on the SimpleDALRegExt page.

Semantics Working Group ( _Norman Gray, Mireille Louys)

The Semantics WG approves this document.

Sub-editing: p2, The expansion of the 'XML' acronym appears, from the XML standard, to be "Extensible Markup Language", not "eXtensible Markup Language"

From RayPlante: so corrected (see pending draft).