Summary of Registry Issues Discussed at the May 2007 InterOp

During the Registry sessions, we took a census of the assembly regarding a number of proposals. This page summarizes the proposals and reports the consensus of the group in attendence. Since a number of active working group members couldn't be at the meeting, we want to give folks an additional opportunity to offer comments and register their preferences before we take any decisions as final.

Please review these items and add any comments you have, preferably by 8 June.

Index of Issues

Minor RI standard issues
Missing Agreements
Addressing the Graininess Issue
- 9. Develop VOSI 1.1 to get more metadata from service
- 10. VODataService: pointing to table metadata

Minor RI standard issues

1. VOResource root element in harvesting Identify response

Problem: The RI spec requires that the element enclosing VOResource records returned through the search interface and the harvesting ListRecords operation to be <ri:Resource>. The VOResource record returned via the harvesting Identify operation does not have this restriction (RI section 3.1.5).
Proposal: Add same requirement on the VOResource root element to RI section 3.1.5.
Consensus of attendees: YES
Additional Comments

add your comments here

2. Should record requirements apply to deleted records

Problem: There are records that have been created in error but which have propogated to other registries; thus, they have been marked as deleted. Often such records will be end up being non-compliant as a VOResource record; making it compliant would require editing the record (on behalf of the publisher). Should we relax the compliance requirements for records marked deleted, or allow registry providers to edit the records appropriately?
Question: Should record requirements apply to deleted records?
Consensus of attendees: YES. Registry curators can add place-holding values as needed so that they validate.
Additional Comments

add your comments here

3. Require the use of =xsi:schemaLocation in harvesting interface

Problem: The use of xsi:schemaLocation is required in the response from the search interface; however, that same requirement is not stated for the harvesting interface.
Proposal: Make the use of xsi:schemaLocation required in the harvesting interface in the same way as in the search interface.
Consensus of attendees: YES.
Additional Comments

add your comments here

4. Require use of namespace prefixes in `xsi:type` values

Problem: Whether you are using the ADQL search method or XQuery, it is somewhat difficult to do precise matching on xsi:type values unless the use of namespace prefixes are guaranteed. Consider two possible values, xsi:type='vr:Service' and xsi:type='vs:DataService'. To match either in an ADQL search, you can provide the constraint, xsi:type like '%Service'. To match only the generic service, you could say xsi:type like '%:Service' but only if you could be certain that a prefix will always be used. In XML Schema, the prefix is not required if the type is from the default namespace.
Proposal: Require the use of namespace prefixes in xsi:type values.
Concensus of attendees: TABLED. While it was recognized that this presented a problem for clients, it was not clear whether this would be easy to control by registry implementations. Prefixes are often handled automatically by the XML processing tools, and it's not clear that explicit control would always be possible.

Additional Comments

If we do not reach any concensus on this, I think we should put it in only as a recommendation, but not a requirement.

-- RayPlante - 24 May 2007

add your comments here

Although xsi:type is of xml type QName (http://www.w3.org/TR/xmlschema-1/#xsi_type) the xslt 1.0 and xpath 1.0 specifications only allowed for a data model where attributes have string values - the version 2.0 xslt and xpath seem to have a more sophisticated data model (http://www.w3.org/TR/xpath-datamodel/#AttributeNode) that allows for a "typed-value", so perhaps the problem will 'go away' for specifying xqueries for more modern tools, as it should be possible to specify the namespace in the attribute value in the same sort of way as for element names. (Though I note that Saxon, for instance, only fullly supports the "schema related" parts of xslt 2.0 in the commercial version). So in practical terms this problem remains even for client writers of xquery, and certainly anyway for writers of adql queries.

I think that this should be a requirement (or dropped entirely) as if the namespace prefix remains only a recommendation then client writers still have to write queries that will accomodate attribute values with or without prefixes to be able to query the registries that are not following the recommendation. Additionally there would have to be a way for the registry to declare that is was following the recommendation before the client could know that it was able to use the simplified form of the query.

-- PaulHarrison - 29 May 2007

5. Location of namespace declarations

Problem: Cutting out VOResource records out of a Harvesting response can be a little tricky because one has to make sure all the namespace declarations are captured. Particularly tricky is the appearance of a prefix within an xsi:type attribute value, because most XML parsers tend to ignore the prefix, treating the whole value as an opaque string. If we required that all namespace declarations used by the VOResource record appear within the VOResource portion of the OAI response (and not in the OAI envelope), it would be possible to lexically (i.e. not using an XML parser) cut out the VOResource record at the start and end of the <ri:Resource> element.
Proposal: Require that all namespace declarations used by the VOResource record appear within the VOResource portion of the OAI response.
Consensus of attendees: NO. Like in (3) above, namespace declarations are often controled transparently by XML tools, and so it might be hard to control where they get placed.
Additional Comments

add your comments here

Missing Agreements

I noticed that we neglected to integrate several agreements from the Victoria meeting into our standards, so we will need to do this.

6. Searchability of core VOResource metadata

Problem: It was AGREED in Victoria that all searchable registries must support constraint-based searching against, at a minimum, all of the metadata terms in the VOResource schema.
Proposal: Update RI to reflect this agreement.
Consensus of attendees: YES.
Additional Comments

add your comments here

7. Returning full VOResource records

Problem: It was AGREED in Victoria that a registry must return the full VOResource record as it was originally constructed or as harvested from another registry. RI does not specify this. In Beijing, we discussed this in the context of the graininess issue, and we thought that this requirement should be relaxed a bit, so as to allow extra fine grained information obtained by a means other than harvesting (e.g. directly from the service) to be returned to end users via the search interface.
Proposal: Add requirement to RI that the original VOResource record must be returned in its original state via the harvesting interface; however, a registry may add additional information in the VOResource records returned via the search interface.
Consensus of attendees: YES.
Additional Comments

add your comments here

8. VODataService: additional metadata to be added

Problem

In Victoria, we agreed to add three additional (optional) metadata for describing data collections and services that operate on them:

coverage.objectCount (defined by the RM)
number of table rows
utype in a column description

This requires the VODataService schema to be changed. Given that these are backward-compatible changes for document instances--that is, existing documents are still valid--we could issue a new version of the schema without changing the namespace (i.e. incrementing the version to 1.01). On the other hand, some client software may require updates which may argue for changing the namespace (i.e. incrementing the version to 1.10).

Proposal: Issue a new version (1.01) of the VODataService schema to add these metadata, keeping the namespace as http://www.ivoa.net/xml/VODataService/v1.0.
Alternate Proposal: Issue a new version (1.10) of the VODataService schema to add these metadata, changing the namespace to http://www.ivoa.net/xml/VODataService/v1.1.
Consensus of attendees: no decision taken.
Additional Comments

add your comments here

Originally did not have much of a problem with the Proposal, but thinking a little more one small problem here in Astrogrid is we have released upgraded a few registries already and the XML must validate to schema to be allowed into the registry, if a user comes in and writes XML to the 'new' namespace schema whereby placing a few of those optional elements it will reject his entry because it is invalid (the registry knows about the 'old' schema). I suspect I will need to upgrade these registries anyways for a few minor fixes in OAI so maybe it is not a huge issue, but I think it is more wise to do the Alternate proposal and use a different namespace since this 1.0 schema is already public. Also note I am thinking this might break one of your rules you mentioned on Nov 21 'Version Numbers on XMLSchemata' see this url: http://www.ivoa.net/forum/registry/0611/1760.htm KevinBenson

Addressing the Graininess Issue

After much discussion of the issue, we gravitated toward some concensus on some underlying points:

obtaining more of the detailed metadata directly the service is desirable.
if information can be obtained from the service, it does not need to be exchanged on the harvesting stream.
moving detailed metadata from the harvesting stream to the service allows registries to decide which information they wish to index and pass on to users.

In a related development, the VO Support Interfaces (VOSI) specification was altered to replace getRegistration()/getMetadata() with a getCapabilities() method that only returns a capability element. A major motivation is the rational that it would easier for a publisher to use a registry's publishing interface to create the core VOResource metadata. The metadata that are easiest to serve from the service itself are the ones that could be autogenerated from the implementation itself.

9. Develop VOSI 1.1 to get more metadata from service

Problem: Some projects are building user capabilities around detailed information served by a so-called "fine-grained" registry. However, the cost of curating this information is shared by all registries.
Proposal: After the completion of the upgrade to RI v1.0, we form a tiger team to work on a version 1.1 of the VOSI standard that would allow more detailed information to be served directly by services. With this in place, only slim VOResource records would be exchanged on the harvesting stream; this team would develop either new conventions or changes to RI that would make this happen.
Consensus of attendees: YES.
Additional Comments

add your comments here

10. VODataService: pointing to table metadata

Problem: A suggestion (by Ray) that preceded the above proposal was to add a metadatum to VODataService that allows one to point to a table description via a URL in lieu of inlining the information directly. It was re-raised (not by Ray) as a simple intermediate solution to the graininess problem that could be put into place now before the forming of the VOSI tiger team. In particular, we could provide broad support for this pointer for our current standard catalog services (SIA, CS, SkyNode) without any extra work by publishers, by pointing to proxy service. For example, a special service that understands SIA could translate FORMAT=METADATA results into a standard format.

Finally, a variation on this proposal was made to provide the link using XLink (just as is currently done with STC). Thus, it would not be necessary to update the VODataService schema. If this schema were to be updated anyway (because of 8. above), some xlink-related constraints could be made that would simplify its use. In any event, using XLink would avoid having to add a new metadatum that would likely be eventually deprecated by a VOSI 1.1 (as described in 9.).

Proposal: Define a convention for using XLink to reference a remote description a set of tables. Dereferencing the link would return the description rooted by the VODataService <catalog> element which could be inserted into the source VOResource record. If proposal 8. is accepted, backward-compatible XLink-related additions could be made to the VODataService schema to ease the use of XLink for this purpose.
Consensus of attendees: unclear, no decision taken.
Additional Comments

add your comments here