Survey among Spectral Data Providers and Consumers

At the IVOA meeting in Cambridge in May, one of the highest priority projects identified for the DAL working group was to define a Simple Spectral Access (SSA) protocol for accessing 1D spectra and SEDs.

To help plan this effort a little survey was conducted among potential data providers and consumers. The results are summarized below. Of course, the summary is only as complete as the response we've received. Thanks a lot to all who took the time to answer to our questions!

More feedback is always welcome. Regards,
DougTody, MarkusDolensky

1a. Data Provider

• ESA ISO Science Data Archive (Pedro Osuna Alcalaya)
• ESA XMM-Newton Science Data Archive (Pedro Osuna Alcalaya)
• ESO/ST-ECF Archive (Pirenne, Wicenec, Dolensky)
• HEASARC (Tom McGlynn)
• Hyperleda (Phillipe Prugniel)
• INTA/LAEFF (Enrique Solano)
• NAOJ/SMOKA Science Archive (Satoshi Honda)
• NCSA (Ray Plante)
• NOAO (Mike Fitzpatrick)
• NOAO and CFLIB project (Francisco Valdes)
• OHP (Phillipe Prugniel)
• SDSS (Stephen Kent, Alex Szalay)
  *In brackets name of person who provided input

1b. Data Consumer

• MIDAS-MOS (Andreas Wicenec, ESO)
• NOAO/IRAF-Specplot (Frank Valdes, NOAO)
• Hyperleda/Pleinpot (Philippe Prugniel, CRAL)
• Specview (Ivo Busko, STScI)
  *In brackets name of person who provided input

About Data Providers

2. Spectral Data Collections

- Sloan Digital Sky Survey (SDSS), Data Release 1
186.000 objects, 3800-9200 Å, 1300 square degrees, resolution 1800

- NCSA Astronomy Digital Image Library (NCSA BIMA Data Archive)

- Variety of spectral data sources from X-ray and gamma-ray missions, including both relatively high and low resolution spectral data (HEASARC)

- Indo-U.S. Library of Coude Feed Stellar Spectra (NOAO & CFLIB)
1300 spectra, 3465-9469 Å @ 0.44A/pix

- at NOAO:

• High Resolution Atlas of Arcturus Spectra - 0.92-5.36 microns
• High Resolution Atlas of Arcturus Spectra - 3727-9300 Å
• Library of High Resolution Infrared Stellar Spectra in K-Band
• Library of Medium Resolution Infrared Stellar Spectra
• Coude Feed Spectral Library
• Library of Stellar Spectra
• Optical Spectrophotometric Atlas of SN 1987A
• Atlases of ThAR, FeAr, and HeNeAr comparison spectra

- at ESO/ST-ECF as of June 26, 03

- XMM-Newton 1D Scpectral Data
XMM/RGS, 20000 spectra (expected), 0.35-2.5 keV, resolution 200-800
XMM/EPIC, 1.5 mio. spectra (expected), 0.35-15 keV, resolution 20-50

- ISO 1D Spectral Data
10000 files, 2.4-197 micron, resolution 40-30000

- IUE Newly-Extracted Spectra (INES)
110000 spectra, 9500 objects, 1150-1980 Å & 1850-3350 Å resolution 1.676 & 2.669 Å/pix

- at NAOJ/SMOKA as of Aug. 25, 03:
(National Astronomical Observatory of Japan / Subaru Mitaka Okayama Kiso Archive)

OAO : 188cm telescope at Okayama Astronomical Observatory
Subaru : 8.2m optical-infrared telescope at the summit of Mauna Kea

- Hyperleda
10000 reference spectra of stars and galaxies, resolution 300 - 1000000, 1300 Å - 2.5 microns total volume 5 GB, collected from literature and reformatted

- Elodie
25000 extracted echelle spectra, resolution 42000, 4000 - 6800 Å on the fly processing options available: wavelength resampling, flux calibration etc. total volume 50 - 100 GB

3. Characteristics of Data

• number: ~100 K - 1 Mio. spectra/data provider; often only several 100s in very specific catalogs
• size: few KBs - few MBs/spectrum
• wavelength/energy ranges: gamma, X-ray, UV, optical, IR, radio
• resolution: 20 - 1.000.000
• linear and non-linear/irregularly sampling (for instance in energy domain)
• often with WCS
• stellar parameters: e.g.: Teff, log(g), [Fe/H]
• noise
• masks (coded aperture mask data, INTEGRAL)
• observation dependent transfer matrix to convert counts to phys. units
• spectral features (SLOAN): emission, absorption lines and parameters, emission redshift params. cross-correlation parameters
• time resolved spectral information (XTE, BATSE, Swift, HETE)

Columns/data items:

• wavelength (better fravergy: frequency, wavelength, energy)
• flux (magnitude, flux, flux density, counts)
• flux error
• quality flag (quality of what exactly?)
• variance arrays
• photon events
• antenna temperature vs. frequency (e.g. SWAS spectra)

4. Current Storage Format

• ASCII tables (catalogs, often highly processed data)
• Database tables (e.g. Sloan)
• FITS binary tables
• FITS images (often used for raw data (CCD images) or stacks thereof; radio spectral image cubes)

5. Is the data available online?

Yes. Data are generally on-line, but proprietary periods and restricted access depending on nationality may apply.

About Data Consumers

6. Name of application, package, demo, etc.

• aXe
• MIDAS-MOS
• NOAO/IRAF-Specplot
• Pleinpot
• Specview

Tools for SEDs:

• NED (on-line)
• AVO Prototype SED utility

7. Summarize capabilities of software

• reading various instrument dependent FITS and ASCII formats
• overplotting of multiple arrays
• various display options (labels, zoom, colors, log/linear scaling, ...)
• physical unit conversion
• model fitting capability
• classification against standards
• rebinning
• quality filters

see, for instance http://www.stsci.edu/resources/software_hardware/specview/what_is_specview

8. Desired characteristics of input data

• As a minimum, data should consist of (x,y) pairs of wavelength-flux values expressed in wavelength/frequency/energy units, and spectral flux density units respectively. The units information should be present as well, in header keywords or table column descriptors or any other suitable form.
• Additional information that could be included with the above, (if available) is a third value associated with each (x,y) pair representing the measurement error in flux density. Also, it won't hurt if obvious pieces of information such as the object name, are provided as well.
• Uncalibrated data (such as in counts/s units) can also be ingested but not much else can be done with it besides plain plotting.
• FITS WCS; useful for well sampled rasters

9. Desired input data format or formats (e.g. graphics, FITS, XML)

FITS (images, tables), XML, and plain ASCII are all supported by Specview at this point. It is fairly easy to write ingestor modules for almost any conceivable spectral format that uses one of those media.

10. General Comments

An interesting service would be to generate 1-D spectra from cubes on the fly. In addition to the usual search parameters, users would have to provide a spatial resolution. This service would be analogous to an image cutout service in that the output is generated to user specs on-the-fly from the underlying data. (Ray Plante)

Even at this level of detail it may be inappropriate to discuss the HEASARC monolithically. The characteristics of individual missions/instruments varies enormously. (Tom McGlynn)

1D and 2D currently have more applications than 3D. FITS binary tables format and support under development. Will be evolved to support VO spectral standards. (Francisco Valdes)

[...] the different storage formats and associated data tables would provide for some interesting use cases. Something like a cutout service might be appropriate for some of the data, the challenging thing for us however to put all these into an SSA is simply the variety of local data access methods required. (Mike Fitzpatrick)