(r15) ObsTAPdraftDiscussion < IVOA

IVOA Web>IvoaDataModel>ObsDMCoreComponents>ObsTAPdraftDiscussion (revision 15) (raw view)~~EditAttach~~
   +Discussion on the IVOA WD on !ObsTAP and the !ObsCore data model

   * List of current releases during the WG review
      * release v1.0-20110228: 
[[http://www.ivoa.net/internal/IVOA/ObsTAPdraftDiscussion/WD-ObsCore-v1.0-20110228-1pdf.docx][docx]]
[[http://www.ivoa.net/internal/IVOA/ObsTAPdraftDiscussion/WD-ObsCore-v1.0-20110228-1pdf.pdf][pdf]] Current draft with pointers to topics to discuss 

The authors could not agree on some important points; others were still
being worked on at the time the current draft of the specification
was released. It was therefore decided to post them on the twiki
(see pages below) for an open discussion. We need to finalize these
soon and we ask for your help. Please find below the list of topics
with indicated the two different perspectives among the co-authors.

%RED% Please provide your comments %ENDCOLOR% not in this page, but in each of the topic pages listed below: 

   1. %RED%dataproduct_type:%ENDCOLOR% 'other' and subtype. [[DataProduct][Click here for the discussion page]]
      * *Option 1.A*: dataproduct_type should allow 7 well-defined types (see list below) plus 'other' to allow *extensibility* to allow other science data products which do not fit into any existing category to be represented.
         * image
         * cube
         * spectrum
         * sed
         * timeseries
         * visibility
         * event
         * other
      * The additional and optional 'subtype' attribute would allow the data provider to specify in more detail, in data-provider specific terms, what each data product is (subtype would be free format).  Without the 'other' category there is no way to expose archive science data products which do not fit into any of the predefined categories, e.g., to fully represent all archive science data products or to prototype extension of the predefined types to new types of data.  This would not interfere with global data discovery where *one-and-the-same-query* is posed to many services (since the standardized categories are still there and would be used where applicable) but would allow more focused queries to be posed to specific archives, allowing the VO to be used for science archive data access as well as global data discovery.  In addition, the 'subtype' field would allow users to understand in more collection-specific terms what each data product actually is.
      * *Option 1.B*: It shall contain the first 7 types, but shall not contain 'other' nor a free format subtype because (1) this was not part of the requirements for !ObsTAP (not a single use case covered this) (2) 'other' -along with a free format subtype- would open up the possibility for data providers to come up with their own schemas, therefore defeating the main idea of !ObsTAP, that is, the possibility to send *one-and-the-same-query* to all obstap services. _[Note: the proponents of having an 'other' category to permit extension do not think this possible abuse would actually be a problem since the subtype attribute already provides sufficient means to more fully specify the scientific type of a data product.]_ 'other' could correspond to different types in different archive implementations.
      * In either case if other types are needed we could eventually amend the document and extend the list of standard data product types without breaking existing services.   
   1. %RED%Best UTYPE string-format:%ENDCOLOR% for humans (!camelCase), for DB systems (lowercase) [[FormatforUtype][Click here for the discussion page]]
      * *Option 2.A*: !UTYPEs shall be treated as case-insensitive strings, favouring !camelCase for readibility.  This continues the current situation in VO where Utypes are already specified to be case-insensitive; doing otherwise would invalidate standards which have been in place for several years, including both existing standards documents and implementations.  Scientifically it is clearly safer to consider (for example) <nop>DataID.CreatorDID and <nop>DataId.CreatorDid (or various other permutations) to refer to the same data model attribute.  In no case would we actually have tokens such as these which differ only in case, so performing a case insensitive comparision is clearly the safest approach.  Given that Utypes are used all the way up into science applications code, and may pass through many layers of software before being used by a service, it is difficult to guarantee that case will be preserved, hence a case-insensitive comparision is wisest.
      * *Option 2.B*: !UTYPEs shall be lowercase to avoid confusion, to avoid extra code to handle all possible cases in clients, !DBMSes, etc.. A case-insensitive string cannot be searched efficiently in all !DBMSes. E.g. the clause WHERE LOWER(utype)='abc.def' forces a table scan (no index can generally be used). _[Note: some of us do not agree with this assertion; performing efficient searches on case insensitive strings is very doable, and in fact is done every day.  Although the support for case-insensitive searches in DBMSes is variable, we have not yet found a case where it cannot be done efficiently, nor should this technical issue be the driver when scientific analyis will be at risk if data model elements cannot be identified merely due to a user confusion over upper or lower case terms.]_
   1. %RED%obs_title data model field:%ENDCOLOR% free format? [[FreeFormat][Click here for the discussion page]]
      * *Option 3.A*: obs_title shall be an optional data model field, a free format string that allows data providers to describe any given dataset in more details than otherwise possible, e.g.:
         * dataproduct_type='image'
         * obs_title='Stokes I continuum image at 1420 Mhz'
      * Without this it could be very difficult for a human user to understand what a specific data product is.  While the quantitative fields are essential for global discovery there is still a need for a human looking at the results of a search to understand what a specific data product is.  Furthermore obs_title is already a mandatory field for all existing DAL interfaces, e.g., SIA, SSA.  If not a mandatory field of <nop>ObsCore it should at least be an optional field.  This field has long been free format in the other DAL interfaces (and in FITS before that) with no problem; it does not matter since it is merely a descriptive field intended for human consumption.  Note also that even within the TAP_SCHEMA all schema elements (schemas, tables, columns, etc.) have a description field which is free format and this has never posed a problem.
      * *Option 3.B*: Not clear what the requirements are, and a completely free format field could be an issue. There is no time now to study this. Let's leave it optional until we clarify requirements and usage. [Note: the proponents of obs_title already agreed that it could be optional.  The main thing is to get it into the output metadata so that the user can understand what a specific data product is.].
   1. %RED%access_format%ENDCOLOR% [[AccessFormat][Click here for the discussion page]]
      * *This is still being defined.*
   1. %RED%Usage of obs_publisher_did%ENDCOLOR% [[PubDataLink][Click here for the discussion page]]
      * *This attribute,* and the general issue of dataset identifiers is well defined and has already been in use in existing standards for several years (e.g., ADS dataset identifiers, IVOA dataset identifiers as defined by the Registry, publisher dataset identifiers and others as used in SSA and Spectrum).  However some of the authors felt that existing discussion of usage was needed.  One serious issue is that the standard currently specifies that obs_publisher_did cannot be used in a WHERE clause, which defeats the whole idea of a dataset identifier being used to persistently identify and later retrieve or access specific datasets.
   1. %RED%Discovery of Polarization data%ENDCOLOR% [[IVOA.Polarization][Click here for the discussion page]]
      * About the importance of *polarization data*
   1. =em_min= and =em_max=: in the current !ObsTAP proposal they are allowed to be NULL. What does it entail? [[IVOA.ObsTAPdraftDiscussion.EmMinMax][Click here for the discussion page]]
   1. Suggestions for the draft SugText
   1. Suggestions for new UCDs to support !ObsTAP: [[IVOA.ObsTAPdraftDiscussion.NewUCDs][Click here for the discussion page]]

and feel free to add a new item when needed. Thanks. 








<br/>
<!--
      * Set ALLOWTOPICRENAME = %MAINWEB%.TWikiAdminGroup
-->
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
docx	WD-ObsCore-v1.0-20110228-1pdf.docx	r1	manage	348.2 K	2011-02-28 - 20:28	MireilleLouys	ObsTAP draft current version
pdf	WD-ObsCore-v1.0-20110228-1pdf.pdf	r1	manage	1716.2 K	2011-02-28 - 20:29	MireilleLouys	pdf version
Topic revision: r15 - 2011-03-07 - JuanDeDiosSantanderVela
IVOA
Log in or Register
IVOA.net
Wiki Home
WebChanges
WebTopicList
WebStatistics
Twiki Meta & Help
IVOA
Know
Main
Sandbox
TWiki
TWiki intro
TWiki tutorial
User registration
Notify me
Working Groups
Interest Groups
Time Domain
Committees
Stds&Procs
www.ivoa.net
Documents
Events
Members
XML Schema