TWiki> IVOA Web>IVOADataModel>CubesDM (2018-03-30, MarkCresitelloDittmar) EditAttach

N-Dimensional Cube model version 1.0

Goal

The Cube Data Model, currently under development, will describe multdimensional image and cube dataset instances.

Much of the initial work for the cube model was done under the title of ImageDM. Please see the <a target="_self" href="ImageDM" title="ImageDM twiki">ImageDM twiki</a> for information on that effort. Since Nov. 2013, the work proceeded under this title. The primary difference being to express the Cube model as an extension of a more generic observation dataset. As such, this model is dependent on supporting component models DatasetMetadata, and STC2.

Participants

domain experts: DougTody (retired)

data modelers: MarkCresitelloDittmar

contributors: MarkCresitelloDittmar (editor); FrancoisBonnarel, JoseEnriqueRuiz, OmarLaurino, MireilleLouys, ArnoldRots, JesusSalgado, DougTody

Use Cases

The Cube model includes the following use cases:

1) Simple 2-D pixelated image. (the most common astronomical image type)

2) A single image cube which is sparse in either the spectral, spatial, or temporal plane. eg: JCMT spectral data cubes are sparse in the spatial plane.

3) Multiple sub-array cubes are often produced by instruments having multiple detectors. These cubes contain multiple cube segments which combine to form a complete, potentially sparse, cube. eg: spectral data cube with multiple spectral bands. eg: CCD mosaic where detectors may not be aligned (requiring different WCS)

4) Very large cubes. Cube data may be dispersed among multiple storage containers. The user-facing instance must hide this underlying complexity, providing the appearance of a single cube.

5) Interactive cube visualization and analysis. This application use case involves interpreting the cube instance, visualizing the content (possibly at various resolutions), interactively performing various operations on the cube ( slice, dice, 2-D projections, spectral extraction, etc )

Cube Classifications:

The above use cases outline the following classifications of cube data which are to be covered by this model:

1) Simple image: A single filled or mostly filled N-Dimensional image array with associated metadta describing the dataset, its WCS coordinate systems, etc.

2) Simple sparse image: A single N-Dimensional array as in #1, however, large portions of the image may be sparse (have no data samples)

3) Multiple subarray image: An image dataset containing multiple subarrays, ie: N-Dimensional image data arrays within the coverage of the overall image dataset. The subarrays may differ in size, resolution, coverage, or other characteristics. The overall dataset containing the subarrays may be sparse. The overall dataset does not have an explicit geometry or sampling, only coverage.

4) Large cube: Very large cubes may be represented as either a simple image or multiple subarray type. The cube model does not impose size limitations, so only important factor here, is that the cube description must be independent of the physical storage structure.

5) Wide-field survey: This is essentially the same as #3, except no subarrays (no explicitly pixelated data). Only automated virtual data generation techniques may be used to access the data, with sub-regions being computed on-the-fly and returned to a client. The mechanism for generating the cube is not relevant to its description, so this class is covered by #1 or #3.

6) Sparse data cube: Sparse data are commonly used for higher-dimensional cubes, and are typically sparse along one or more axes. eg; a multi-band image with only a few spectral coordinates (bands). Event data is an example of a data cube which is sparse in all measurement axes.

Requirements

  • Structure
    • The model shall be vo-dml compliant, producing a validated vo-dml XML description.
    • shall produce documentation in standard pdf format, and as vo-dml HTML
    • shall re-use, or refer to, dependent models. Additional descriptive or context specific information may be included in the specification, but the element definitions and core descriptions remain with the originating document.
  • Scope
    • This version of the model will focus on the first and last cube classification ( Simple Pixelated Image and Event list specifically ). These cover the extremes of the classes, and the expectation is that the Simple sparse image and multiple sub-array are minor variations in multiplicity from these other forms. NOTE: While classes 2 & 3 are not primary targets for this version, they should be taken into consideration during the modeling to help ensure compatibility.
    • This is an N-Dimensional model, where each dimension corresponds to data in some physical domain. The list of possible domain candidates is quite large, so for this version of the model, we will limit the required scope to include the following domains which are frequently included in astronomical data cubes, and in the high-priority specialized models (TimeSeries, Spectral, etc ) which are expected to re-use the same family of modeled objects as this model. Domains: Spatial, Spectral, Temporal, Polarization.
  • Content
    • Dataset
      • The Cube Dataset(s) defined by the model shall be defined as extensions of a generic observation dataset (DatasetDM). This generic dataset defines all high-level metadata required to support generic dataset discovery within the VO.
      • Additional metadata which are determined to be of a sufficiently general nature, shall be added to the content of the DatasetDM.
      • Additional metadata which are specific to data cubes/images shall be included in the extension objects defined in this model.
    • Pixelated Image Cube
      • shall define the structure and content of a pixelated image dataset. The specification shall include
        • number of image axes
        • size of each image axis
      • shall define the relation between the pixelated image axes and the corresponding physical domain axes. eg: pixel axes (#1,#2) => detector axes ( x, y )
        • typically this is defined as a Transform which defines the mapping from one "set of coordinate axes" (Frame) to another "set of coordinate axes" (Frame)
        • must support the industry standard relation specifications
          • FITS WCS Transform spec. (projections)
          • Linear
          • Matrix
      • shall support the specification of one 'set of physical domain axes' (Frame) as a mathematical function of another. Similar to the above, this related two physical frames.
        • must support the same standard relation specifications as above
      • shall support the identification of pixelated image data values as belonging to a particular physical domain, complete with units and associated Frame specification.
      • shall support the specification of the physical domain coordinate system(s) associated with the cube data and metadata.
        • may include Frame specifications for any of the supported physical domains
        • any particular physical domain may only appear once within a coordinate system specification.
        • may include Frame specifications for domains not included in the image data space. ( eg: a Time frame in a simple 2-D spatial image )
    • Sparse Cube (Events)
      • shall support the specification of the physical domain coordinate system(s) associated with the cube data and metadata. (as for Pixelated cube)
      • shall support the specification of one 'set of physical domain axes' (Frame) as a mathematical function of another. (as for Pixelated cube)
      • shall support the identification of data values as belonging to a particular physical domain, complete with units and associated Frame specification.
      • shall support the association of data values with their corresponding errors.
        • multiple error sources may be associated with the same data value. (eg: systematic and statistical errors)

Documents

Latest document:

The latest released document can be found on the IVOA Documents and Standards page.

Volute:

The volute repository holds all revisions of the document, as well as the source open office document, diagrams, etc. here

UML Model:

The model is being developed in UML, using Modelio-3.0. The model project can be obtained as a zip file in the volute repository, here.

We also provide an export of the UML specification in XMI format (version 2.4.1), which is compatible with the vo-dml xslt scripts for generating the vo-dml XML representation.

vo-dml:

VO-DML XML serialization of the model and corresponding HTML page are here

Discussions

Much of the initial work for the cube model was done under the title of ImageDM. These discussion threads are recorded in the ImageDM twiki.

Since 2014, the Cube model work has proceded under this title. Relevant discussion threads are recorded below:

Implementations

Instances of Cube require the use of several contributing models. The list below provides pointers to the relevant model documentation for the models used in the example serialization set. These files will be transferred to the ivoa repository as the models become REC.

Example Files:

We have generated a series of example serialization files for this model work. Four are real-world data files, and others which are focused on certain model elements or components not represented by the real-world files. We provide the link to the files here.. there you can find a 00README file which describes the content.

Edit | Attach | Watch | Print version | History: r8 < r7 < r6 < r5 < r4 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r8 - 2018-03-30 - MarkCresitelloDittmar
 
This site is powered by the TWiki collaboration platform Powered by Perl This site is powered by the TWiki collaboration platformCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback