Image Data Model Effort

The Image Data Model, currently under development, will describe multdimensional image and image cube datasets and instances. It will serve as the basis for the description of Image datasets in archives and VO queries, as well as the basis for Image data access protocols such as SIAV2 and related access methods such as siav2.accessData.

ImageDM-ObsCore Convergence

It was agreed in the fall 2013 interop to do whatever is necessary to bring the Image data model into convergence with ObsCore (e.g., ObsTAP), in particular making data models such as ImageDM and SpectralDM extensions of ObsCore, all sharing a common core data model. We are fast-tracking an updated "stable" version of both the ImageDM and SIAV2 drafts that will make a first cut at this convergence, sufficient to support ongoing development for the Cube initiative, while further development of the working drafts goes forward. During prototyping the stable version of ImageDM will be used for prototype development while the full standard evolves (i.e., there will be two versions of ImageDM until eventually the stable version is retired and replaced by the final standard). An initial analysis of what is required has been performed by a subset of the ImageDM author group (FB, ML, MCD, DT) and the results are summarized here:

• "Stable" version of ImageDM as of 17 Nov
• ImageDM spreadsheet as of 25 Dec (full data model, ObsCore/Char2 integration; minor tweaks)
  • PDF version of data model spreadsheet (25 Dec)

• Initial analysis of ImageDM-ObsCore convergence (19-25 Oct)
  • Spreadsheet summarizing the ImageDM and ObsCore differences (DT)
  • ImageDM and ObsCore Utypes vs Char2 (FB)
• Char Utypes updated (ML,FB 8 Nov - new)

The consensus at this point appears to be that we finalize a set of Characterisation V2 Utypes, use these for both ImageDM and SpectralDM, and eventually update the older ObsCore/ObsTAP document to comply, however what is eventually done will depend upon a more detailed analysis still to be performed. This will take a while, but all we need for ongoing Cube development is the updated ImageDM and SIAV2 drafts plus a summary of the revised Utypes planned for ObsCore.

Further updates will be posted here as this effort progresses. Discussion should be carried out on the DM mailing list.

Discussion Points:

This effort carries on from the above draft (Nov. 2013).

The following is a list of topics to be discussed regarding the draft and will guide the future evolution of the document. It is a dynamic list, topics will be added as needed. Discussion should be carried out on the DM mailing list. Here we will summarize the topics, provide links to the relevant threads and note the resolution. It would be greatly appreciated if we can keep discussion topics within the relevant thread.

1. Relation to ObsCore

Carries on from the Nov. draft, formalizing the relation to the ObsCore 1.0. This is a fairly broad topic, and will undoubtedly spawn other threads. The goal of this topic is to explicitely illustrate the Observation model used by the Image and Spectral models and how that relates to the existing ObsCore model. The result of this effort may lead to a recommendation to update that standard, but should not require it.

Threads:

• Revised ImageDM-ObsCore architecture

Links:

• UML diagram (draft)
• UML diagram (20131117) - Should accurately reflect the current relation to baseline further work.

2. Observation Relation to Dataset

This topic spawned from "Relation to ObsCore". This topic is very "ObsCore" centric, but since the Observation/Dataset relation is central to the Image and Spectral Model structure, it has significant ramifications on their design.

It is clear that there is a relationship between "Observation" and a more generic "Dataset". This "Dataset" would contain elements such as the dataProductType, and dataProductSubtype, presumably others. This object has not been formally defined.

In ObsCore, there is an implied relationship for Observation as an Extension of Dataset in the location of these attributes. So, I have always interpreted that Observation "is" a Dataset. This is reflected in my choice of the name "ObservationDataset" in the left hand package of my diagram. It implies that it is a Dataset extended for Observation purposes. Recent discussion brings this relationship into question, with assertions that an Observation can be associated with 0 or more Datasets. "

The goal of this thread is to gain concensus on what this relationship currently is, and how it should be represented in the Observation model used by the Image/Spectral models.

Threads:

• Observation relation to Dataset
• ObsCore - Dataset earlier brief thread

Links:

• UML diagram (draft) - Separation of Observation and Dataset

Resolution:

The Spectral/Image/Cube model work will show "Observation" as a type of Experiment which is associated with 0:* "Dataset"s. The Dataset will be generic, with ObsDataset extension to include metadata from the Observation model.

This includes elements such as Target, Proposal, ObsConfig.. etc.

3. Mapping

Major dopic for discussion of the Mapping object, its definition and purpose.

Threads:

• ImageDM - Mapping object

Links:

• Mapping compared to STC. Shows step-by-step transition of Mapping to STC Frames, Systems, Transforms.

Resolution:

The Image/Cube model will be refactored to distribute the Mapping information to the corresponding STC based structure for coordinate systems and frames.

The current STC model does not accomodate the Linear + Projection model for the coordinate system transforms, so adjustments will be made in that area.

IF access protocols (SIAP2) requires a Mapping object to encapsulate the transform information in a query response, it should be defined there, with a 'mapping' to indicate how to populate that structure from the model components (as outlined in the PDF file above).

4 . Provenance

Hopefully a very minor thread. There is an ambiguity in the ObsCore document, where the Utypes imply a "Provenance" container class in the model which is not reflected in the UML diagrams.

The goal of this thread is merely to decide if this is a mistake in the diagram which should be 'fixed' or not.

Thread:

• ImageDM: Provenance
• - same thread.. got split

Resolution:

The concensus seems to be that there is no logical head for Provenance at this time, and the model is not defined enough at this point to presume structure, so we will not include a Provenance class at this time.

5. Provenance: new elements in ImageDM.

The image model adds new elements under the Provenance umbarella (Bandpass, DataSource, ObsData). There is question regarding where/how these should be included.

Threads:

• new Provenance items

Resolution:

This thread has had no discussion.. At present, these items are to be modeled under the Observation section, brought to the Dataset via the ObsConfig container.

6. Derived object.

Minor topic regarding the need and placement of the Derived class.

Threads:

• ImageDM - Derived object

Resolution:

Yes, Derived should be included.. attached to Dataset as 'information derived from other model elements.'

Unified Models:

The next stage involves folding the above discussion points into a new set of model diagrams which are consistent with each other, and correctly model the object relationships.

Models involved:

1. Models involved:

+ Observation/Dataset: High level metadata for generic Dataset, plus Observation (hypothetical) experiment model, from which certain metadata is referenced as 'provenance' in an ObservationDataset.

+ NDCube: Specification of SparseCube and NDImage Datasets.

+ STC: STC-1.33 model (my interpretation of schema) provides baseline for STC elements. STCMod diagrams illustrate recommended modifications to that model to support NDCube.

+ Characterisation: Char-1.13 model elements used in NDCube, expanded to enough detail to verify consistency in axis detail between CharAxis and DataAxis coordinates.

+ ivoa Types: For this project, each of these models are defined using a common set of basic datatypes. We use the ivoa types model from the VO-DML project in order to fill our need, and facilitate collaboration with efforts to port the models into that framework.

2. SVN Repository:

The model documents (Modelio save set and XMI exports), along with generated diagrams are stored in the Volute repository .

3. Ongoing Discussions:

Discussion threads on model diagrams:

• Change Notices: part I
• Enumerated axes
• Target problem: move AstroTarget from Spectral to ObsDS
• Base Datatypes: add Time Quantiity, URL?, part II
• WCS: WCS distribution within STCMod
• Uncertainty: Uncertainty tree connection to Char-Accuracy
• NDImageDataset: review comments
• Dataset datatypes: review comments
• Obs-DS overview: review comments

Second Draft:

The above model diagrams were presented at the IVOA interop in Madrid, May-2014. A link to the presentation PDF is here.

The requirements for moving forward from that point included:

• define CubeDM as extension of a common baseline Observation-Dataset model usable for Cube, ObsCore, Spectral, and future models.
• define and incorporate a 'simpler' STC-2 model which meets requirements to support Cube model and compatible with official STC-2 development being done in parallel by Arnold.
• Goal: RFC ready standard by October interop.

Revised CubeDM document(s) were delivered to the IVOA DM working group Sept. 30. (see announcement)

The documents and Modelio UML project have been updated in the Volute repository. As explained in the announcement, this delivery split the models into 2 documents to better separate the areas of concern for Dataset Metadata, and particular Dataset types (e.g. NDCube). Pointers to the document PDFs are:

• WD-DatasetDM-1.0-20140930.pdf
• WD-CubeDM-1.0-20140930.pdf

Discussion Threads:

Current Drafts

Latest version of the draft:

• WD-DatasetDM-1.0-20140930.pdf
• WD-CubeDM-1.0-20140930.pdf

Previous versions:

• Image DM v1.0-20131117
• Image data model spreadsheet - 20130812
• Image DM v1.0-20130812 (draft for Sep2013 interop)

Some related documents can be found here:

• DAL Image Access discussions and documents

In particular, there is a new draft of the SIAV2 image access specification which is sync-ed up with the current version of the ImageDM.

The VAO Cube whitepaper relevant use cases and analysis addressing the problem of modeling and accessing multidimensional image data in the VO:

• VAO Cube whitepaper

This includes a section on the ImageDM, mainly addressing issues pertaining to the image cube use cases.

=