ObsCore v1.1: Proposed Recommendation: Request for Comments

Errata page ObsCore-1_1-Erratum-1

This page contains former public discussion of the Obscore 1.1 Proposed Recommendation.

Latest version for the specification : http://www.ivoa.net/documents/ObsCore/20170509/REC-ObsCore-v1.1-20170509.pdf

In order for you to easily check the differences add-ons/changes from previous version , a colored version is provided here

Reference Interoperable Implementations

Two ObsTAP services are currently serving ObsTAP 1.1.

• CADC: http://www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/tap/
• XCatDB: http://xcatdb.unistra.fr/3xmmdr5/tap
• Compare table updates by browsing the TAP table definitions:ivoa.obscore compliant with V1.0 and ivoa.obcore1.1 compliant with V1.1.

The following links provide direct access through TapHandle.

• CADC: http://saada.unistra.fr/taphandle?url=http%3A//www.cadc-ccda.hia-iha.nrc-cnrc.gc.ca/tap/
• XCatDB: http://saada.unistra.fr/taphandle?url=http%3A//xcatdb.unistra.fr/3xmmdr5/tap/

Others applications can be used to connect to these services (e.g. TopCat)

Validators

STILTS taplint can validate ObsCore. A version that can validate ObsCore 1.1 is available at ftp://andromeda.star.bris.ac.uk/pub/star/stilts/pre/stilts.jar. Run java -jar stilts.jar taplint tapurl=http://... and look at the results of the OBS stage (you can minimally use stages='CAP TMS OBS' if you don't care about the rest of the validation). (Taplint updated for PR-ObsCore-20160923 -- MarkTaylor - 2016-09-29)

Note: At time of writing (18 Apr 2016) the reference implementations listed above do not pass this validation.

-- MarkTaylor - 2016-04-18

http://xcatdb.unistra.fr/3xmdr5/tap has been updated to pass the Taplint validation.

-- LaurentMichel - 2016-04-27

The CADC implementation has been updated but still has two errors from taplint:

1. s_region has no listed unit; in my opinion this is correct since the VOTable field for this is currently datatype="char" arraysize="*" and units are not appropriate. I think this was a bug in ObsCore-1.0 and ObsCore-1.1 should be fixed to be more careful with s_region. It also has to be more carefully written to permit use of new ADQL xtypes (polygon) which would have a different datatype and units. As an involved party I will submit some new text to the editor.

2. dataproduct_type "catalog" is in use pending a decision on what new value should be added to the list.

There are also some warnings about other things the CADC TAP service does that are not ObsCore specific.

-- PatrickDowler - 2016-05-18

Comments from the IVOA Community during RFC period: 01 April 2016 - 15 May 2016

In order to add a comment to the document, please edit this page and add your comment to the list below in the format used for the example (include your Wiki Name so that authors can contact you for further information). When the author(s) of the document have considered the comment, they will provide a response after the comment.

Comments from MarkTaylor

There is a problem with the registration of ObsCore 1.1 services. Section 5 of the PR says this:

Since ObsCore-1.1 is a superset of 1.0, TAP services that support ObsCore-1.1 also support ObsCore-1.0 and should include both 'dataModel' elements, e.g.:

   <dataModel ivo-id="ivo://ivoa.net/std/ObsCore/v1.0">ObsCore-1.0</dataModel>
   <dataModel ivo-id="ivo://ivoa.net/std/ObsCore/v1.1">ObsCore-1.1</dataModel>

But since some of the field metadata has been changed between ObsCore 1.0 and 1.1 (by my count: 8 mandatory field Utypes, 1 mandatory field UCD and 8 optional field Utypes), ObsCore 1.1 is not a superset of ObsCore 1.0. Any service attempting to implement both will have to choose which version of the UCDs and Utypes to use (e.g. is the Utype for dataproduct_type "obscore:Obs.dataProductType" or "obscore:ObsDataset.dataProductType"?), it can't comply with both simultaneously. This anomaly has to be resolved in some way in the text.

-- Anser by editor

This is correct. The text has been changed accordingly and will not allow the two data models tags. ---- MireilleLouys - 2016-06-02

Also some minor comments on the detail:

• Sec 1: the paragraph "...an initiative of the IVOA Take Up Committee ... all archived astronomical observations" is included twice in the Introduction. changed
• Figure 1: there are a couple of standards not appearing in the architecture diagram that maybe should be there. <nop>DALI is mentioned in figure 1 caption, and <nop>DatasetDM is mentioned in section 3.3.3. changed
• Sec 4.4: could usefully make an explicit reference to sec 3.3.3. changed
• Sec 6: Some of the FIELDs have @unit="sec", which is not VOUnit compliant, it should be @unit="s". Plus some other incorrect units and UCDs you can discover by running the taplint validator mentioned above on the service. There are also one or two (cut'n'paste?) typos that make this text badly-formed XML ("< ?xml", "/FIELD>", incorrect double-quote character). changed
• Sec A: I don't understand some of the notation in this section, e.g. "AND=SERVICE REQ", "(SERVICE REQ+NEEDS ANOTHER SERVICE (CATALOG)". Did I miss something? text changed to be more explicit
• Sec A.1: "em_max >> 2.48E10" should read "em_max > 2.48E10". changed
• Sec A.1.6: "AND | (t_max + t_min)/2 – user_date | < 1 d" - is this valid <nop>ADQL?
  • PR-ObsCore-v1.1-20160709: still not addressed
  • Adql parser fails: currently tested on ref implementation for the correct syntax, -- MireilleLouys - 2016-08-26
  • changed in the new release using abs() and the defined default units -- MireilleLouys - 2016-09-12
• Table 5: There's a slight inconsistency in the Type column of this table: some enumeration columns are listed with types and others not, e.g. dataproduct_type has "enum string" while data_rights just says "enum". changed
• Sec B.6.3.1: "A format like MJD is useful for easy calculations and preferred for the Observation Core components model.". This makes it sound like use of MJD is optional here, but from sec 4.14 I understand that MJD is mandatory for t_min and t_max. Reword? changed ---- MireilleLouys - 2016-06-02
  • PR-ObsCore-v1.1-20160709: it has been reworded to "these should be stored in MJD format..." which still sounds advisory rather than compulsory. Shouldn't it read "must"?
  • yes changed, thanks -- MireilleLouys - 2016-08-26

-- MarkTaylor - 2016-04-15

The reference implementations currently do not pass validation (see above). This has to be fixed. -- MarkTaylor - 2016-04-18

Xcatdb reference implementation is fixed (see above) -- LaurentMichel - 2016-05-03

Comments from PatrickDowler

The CADC implementation of ObsCore-1.1 (referenced above) makes use of a non-standard dataproduct_type value: catalog. I would like to see this value added to the list of allowed values.

-- PatrickDowler - 2016-04-15

See author response below

-- LaurentMichel - 2016-04-27

Comments from JamesDempsey

The CASDA implementation of ObsCore v1.0 includes catalogue data products - the ASKAP processing pipeline will produce one or more catalog files for each image/cube. These currently have a blank dataproduct_type in accordance with the ObsCore spec. We have had numerous questions about this omission in our workshops as there is an expectation that they should have a type value. Thus we would like to see a 'catalog' value added to the allowed types.

-- JamesDempsey - 2016-04-15

We replied to this issue on the mailing list. In conclusion, the catalogue in a broad sense can also be an agregation of tables or a compilation of multiple source lists. On the contrary a source list is well identified as a list of sources detected in one observation that fits better your use case. Thus we propose to add sourcelist as a new dataproduct_type.

-- LaurentMichel - 2016-04-27

A new dataproduct _type has been added for "analysis data product" and should cover this use-case . Updated in last PR document

-- MireilleLouys - 2016-08-26

we finally opted for dataproduct_type='measurements' to cover the case of extraction sources related to another dataset described in Obscore. the general case for catalogs was not planned and is not covered for this specification.

-- MireilleLouys - 2016-09-23

Comments from TomMcGlynn

These comments are mostly on the understandability of the standard. So this isn't particularly associated with 1.1 versus 1.0 and I doubt that you will wish to address these comments in this version but hope it might have some effect on future versions. These are from my personal perspective of looking at the need to implement this standard in the future and trying to see if I can understand what it implies. If I were (and I will be) trying to implement the protocol, then it takes a while to understand where the pieces of this standard that matter are located.

Sections 1 and 2 are the usual filler and can be easily ignored. I think that what I really care about in 3.2, 3.3, 4 and appendix C and these are pretty clear. Part 5 is needed for entry into the registry and I find it confusing, but that's par for the course. I remain deeply baffled by section 3.1. I think it does a brilliant job of obfuscating a simple standard. I remain similarly unclear of what the role Appendix B is. If you want to note that there is something more complex on which this standard is based, do it by reference please rather than importing the complexity into this document.

Section 3.1 is the UML description for the model, a requirement for the IVOA data models standards. Details on the way parts of a datamodel are re-used need to be explained carefully. Section 4 describes the TAP implementation of the mandatory data model fields only. Section B describes all data model items including optional ones and provides some usage context and examples. This B Appendix can be considered as a Glossary for all data model items. -- MireilleLouys - 2016-07-09

I would suggest that section 6 be retained at most as an appendix. Agreed, changed -- MireilleLouys - 2016-07-09

Appendix C should be included in the normative text. Appendix A seems fine (though I don't know why it sometimes gives ADQL translations and sometimes not).

On a slightly more substantive note, I am not clear what role UCDs play? Since we specify column names and utypes for all columns are UCDs given simply for completeness or do they play some actual role in the way users interact with this table? I.e., what breaks if we were to omit or change a UCD for one of these columns?

-- TomMcGlynn - 2016-05-10

Late Comments from FrancoisBonnarel

Here are some comments and suggestions for typos correction and the like ...

• Page 7, list of acronyms. "!ObsTAP TAP interface to Observation Data Model" ----> "!ObsTAP TAP interface to Observation Core components Data Model" changed
• Page 8 "Such information will be prototyped as new features in DAL protocols...." ---> "Such information will be tackled as new features in future versions of DAL protocols...." changed
• Page 10 "The described datasets can be directly downloaded, or IVOA Data Access Layer (DAL) protocols such as for accessing images (SIA).." ---> "The described datasets can be directly downloaded, or IVOA Data Access Layer (DAL) protocols such as for accessing images (SODA).." changed
• Page 11 Introduction of section 3: there is a conceptual problem here; Observation model referred here (basically = result) is a little different of the concept described in 3.3.3. Maybe we should say there the concept has evolved ? The general Observation discussion is in the scope of the more general Dataset Metadata DM. Sorting out precisely the concept of an Observation as experiment or result is context dependant and will be illustrated later as in the CTA DM for instance.
• Page 12 New efforts on the provenance model may be inconsistent with the Provenance package drawn on Figure 2. If possible this should be made consistent. This reflects the simple ideas for Provenance considered in Obscore and not the full Provenance DM currently under definition
  
• Page 16 SSA protocol : the author list should give "& al". Tody and Dolensky alone are the editors. changed
• Page 18 ASDM (archive science data model) should be added to the list of achronyms of page 7. maybe with a reference added in acronyms with Fits
• Page 23 "called DataLinking, currently in development in the DM and DAL Working group" ---> to be replaced by "called DataLinking (reference to 2015-06-17 spec) changed
• Page 26 4.19 Observable axis description (o_ucd) I think event lists are datasets where "observable" is not applicable so the corresponding field should be NULL. Can we add this ? changed
• Page 30+31 Example : is it really necessary to let the "free" fields ? And the service descriptor for the datalink service after the main resource ? discussion to happen for a better illustration of a serialisation
• Page 33 : Example service : I am not really understanding what is on the main page and how I should use this apart from reading what is in "Protocol & Languages" page extensively. A revision of the introductory page is probably needed (I am ready to help). A possibility to get responses from real services would be well appreciated. Current improvement on the way
  
• Page 33 to 40 : use cases; AS already stated by Mark TAYLOR. Anything related to service functionalities ora capabilities (LIST = SERVICE REQ) should be removed. When available the SELECT should be given ONLY. changed
• Maybe some use cases expressed with abreviation such as FREQ or STOKES should be rephrased. These are just illustrations of the way a user would express Use-cases, not a predefined language
• page 44 : data product subtypes. If it is a field with free syntax contrary to dataproduct_type its should be said explicitly. changed
• page 46 : Creation date . Remove the reference to SIAV2.0 This protocol is no more using time stamps explicitly. Done
• page 46 : Publisher Dataset ID . I think "It may differ" has to be replaced by "It is generally different". And the parenthesis has to be removed. The main reason why it is different is that creator and publisher will differ in general. changed
• page 48 : B.6.1.1 Utypes in char are at the moment "Char.SpatialAxis.numBins2.I1" and "Char.SpatialAxis.numBins2.I2" Do you want to change this in Char2 ?
• page 50 : B;6.2.2 add "corresponding to utype Char.SpatialAxis.calibrationStatus" after "This attribute of the spectral axis"
• page 51 : B6.2.4 a) ---> fix "Full Width at half maximum(FWHM)" changed
• page 52 : B.6.3.4 ---> this parameter corresponding to utype "Char.TimeAxis.calibrationStatus"
• page 52 : B.6.4.1 complete the first paragraph this way: "This field corresponds to utype "Char.ObservableAxis.ucd".
• page 53 : B.6.4.2 add : "The corresponding utype is Char.ObservableAxis.calibrationStatus".

A general comment : All Utypes are presented in the main data model table, Table 5 beginning of Appendix B. These are eluded in the description of data model elements when the building logic for these strings are easy to derive from the "ClassName.attributeName" pattern.

-- FrancoisBonnarel 2016-06-01

Two more things from validation discussion with PierreLesidaner:

• In Obscore Table 1 we have "AstroCoordArea" type for the s_region. This is valid because we wanted to have STC-S there. This type is actually not a VOTABLE datattype but an xtype. In further tables we have datatype in tha TAP_SCHEMA to be ADQL:REGION. How does all this fit together. I think we want this field to be reused for selection in UPLOAD context ? Changed s_region type to 'string' and left the implementation details to the protocol version shipping the data model serialisation , TAP 1.0 or TAP1.1.
  
• s_resolution has type "float" in table 1 but has datatype ADQL:DOUBLE in other tables.The VOtable datatype is also double in the implementations. Changed homogeneized all definitions to double -- MireilleLouys - 2016-06-10

-- FrancoisBonnarel 2016-06-03

Comments from TCG member during the TCG Review Period: Extended to 18 November 2016

WG chairs or vice chairs must read the Document, provide comments if any and formally indicate if they approve or not the Standard.

IG chairs or vice chairs are also encouraged to do the same, although their inputs are not compulsory.

TCG Chair & Vice Chair ( _Matthew Graham, Pat Dowler)

Approved.

-- PatrickDowler - 2016-11-30

Applications Working Group ( _Pierre Fernique, Tom Donaldson )

I approve of this document. -- PierreFernique - 2016-12-02

Data Access Layer Working Group ( François Bonnarel, Marco Molinaro )

Valid and useful for DAL WG. I definitely think that the elements number along each axis and the tackling of velocity cubes is a real progress compared with 1.0 from the DAL perspective (ObsTAP and SIAV2.0 dealing with data cubes), as well as some cleaning in the Relationship to DatasetMetadata and Cube data model.

-- FrancoisBonnarel 2016-06-01

Data Model Working Group ( _Mark Cresitello Dittmar, Laurent Michel )

Approved

-- LaurentMichel - 2016-12-06

Grid & Web Services Working Group ( Brian Major, Giuliano Taffoni )

I approve of this document.

-- BrianMajor - 2016-11-21

Registry Working Group ( _Markus Demleitner, Theresa Dower )

I approve of the Oct 12, 2016 version of the ObsCore 1.1 document. Prior concerns have been addressed and the registry dataModel change is good.

-- TheresaDower - 2016-10-12

Semantics Working Group ( _Mireille Louys, Alberto Accomazzi )

The working group chair and vice-chair approve this document.

-- AlbertoAccomazzi - 2016-12-01

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick )

Data Curation & Preservation Interest Group ( Françoise Genova )

Knowledge Discovery in Databases Interest Group ( Kai Lars Polsterer )

Theory Interest Group ( _Franck Le Petit )

<!--
* Set ALLOWTOPICRENAME = TWikiAdminGroup
-->