IVOA Grid & Web Services: VOSpace

Contents

---

Overview

Users want to be able to easily publish and share their own data as well as have data reside close to computation nodes when large-scale processing or dynamic caches for intermediate data products in workflows are involved. Three modes of data transfer need to be supported: client-server (the client pushes data to, or pulls data from, the server), server-client (the server pushes data to, or pulls data from, the client) and server-server (one server pushes data to, or pulls data from, another server in response to a client request) also known as third-party transfer.

There are many solutions already to these problems so instead of inventing our own we specify a lightweight interface, VOSpace, that sits on top of existing storage solutions, providing a common interface to an underlying heterogeneity of implementations from local file systems to bulk data grids. VOSpace can distinguish between unstructured data, which is just a collection of bytes, and structured data where there is knowledge about how the bytes are arranged and so additional operations are possible. Individual VOSpace instances can also be federated into a peer network so that the user just interacts with a single data space.

VOSpace 1.x specifications relate to a SOAP-based web service.

VOSpace 1.0

VOSpace 1.0 defines an unconnected flat file space and is now an IVOA Recommendation:

Specification

• Service specification
• WSDL for 1.0 service
• Schema for 1.0 service
• VOSpace usage note v0.1

VOSpace 1.1

VOSpace 1.1 extends the existing 1.0 specification to support containers, links between individual VOSpace instances, third party APIs and a "find" mechanism.

Specification

• Service specification

• VOSpace 1.1-rc1 wsdl
• VOSpace 1.1-rc1 schema

• VOSpace 1.1-rc3 wsdl
• VOSpace 1.1-rc3 schema

Standard URIs

The VOSpace specification refers to a number of standard URIs for properties, views and protocols.

The following URIs will be used to represent the standard protocols:

• ivo://ivoa.net/vospace/core#httpget Will be used as the protocol URI for a HTTP GET transfer.
• ivo://ivoa.net/vospace/core#httpput Will be used as the protocol URI for a HTTP PUT transfer.

The following URIs will be used to represent the standard views:

• ivo://ivoa.net/vospace/core#anyview Will be used as the view URI to indicate that a service will accept any view for an import operation.
• ivo://ivoa.net/vospace/core#binaryview Will be used as the view URI to import or export data as a binary file.
• ivo://ivoa.net/vospace/core#defaultview Will be used by a client to indicate that the service should choose the most appropriate view for a data export.

In order to register VOSpace services in an IVOA registry, the service capabilities need to identified using a standard URI.

The following URIs will be used to represent the service capabilities:

• ivo://ivoa.net/vospace/core#vospace-1.0 Will be used as the capability URI for a VOSpace 1.0 service.
• ivo://ivoa.net/vospace/core#vospace-1.1 Will be used as the capability URI for a VOSpace 1.1 service.
• ivo://ivoa.net/vospace/core#vospace-2.0 Will be used as the capability URI for a VOSpace 2.0 service.

If a service implementation supports more than one version of the VOSpace API, then these capability URIs can also be used within a VOSpace service to identify different VOSpace capabilities for a node.

One use case for this would be a VOSpace-1.1 client talking to a service that implements both VOSpace-1.0 and VOSpace-1.1, where the client is acting on behalf of a third party agent that only understands VOSpace-1.0. In which case, the client can use the information in the VOSpace-1.0 capability to direct the third party agent to the VOSpace-1.0 endpoint.

Discussion

• Discussion page for 1.1 specification
• Discussion page for property registration
• Discussion page for data formats

History

• Old discussion page for VOSpace 1.1 features

VOSpace 2.0

VOSpace 2.0 defines a RESTful interface.

Specification

The proposed recommendation is available here.

Discussion

• Historical discussion page for specification

VOSpace 2.1

VOSpace 2.1 contains minor additions and corrections to the RESTful version 2.0.

Specification

The current Working draft is available here.

Discussion

• Discussion page for specification

Custom URIs

The VOSpace specification allows 3rd parties to define custom properties, protocols and views simply by choosing a unique URI to represent them.

The following URLs link to some examples of custom properties, protocols and views.

• VOSpacePublicShare A transport protocol for publicly sharing node content.

Note - this list is for information only, and is not intended to be a definitive list of approved properties, protocols or views.

It is possibly to use custom properties, protocols and views in VOSpace messages that are not in this list. However, if you want other people to use your custom URIs, then adding them to this list will enable people to find out about them.

VOSpace 2.2 or 3.0

A list of future potential VOSpace changes, enhancements and discussion items are listed here:

• New parameter on getNode the sets the sort field on the node children? Support reverse sorting too?
• Change StandardIDs to match the pattern of TAP 1.1:
  • Make use of the UWSRegExt interface type fields: type="uws:Sync" and type="uws:Async"
  • Single standardID for the specification
• Remove the internal-facing VOSpace standard capabilities, as noted by DaveMorris here: VOSpace 2.1 RFC
• Changes from François Bonnarel from the 2.1 RFC VOSpace 2.1 RFC
  • A model diagram like the one in fig 2 with all the classes, not only "node". Relationships should be shown but no need to do a full UML diagram.
  • A couple of diagrams showing where Web service operations occur (VOSPACE to client. Client to VOSPACE. Inside a VOSPACE. From: VOspace to another One etc...)
• Ability to sort on alternate columns in both ascending and descending order (eg lastModfied, contentLength). CADC has implemented this.
• In the transfer object, the 'direction' can conflict with the protocol URI. For example, the direction can be 'pullFromVoSpace' and the protocol can be 'HTTP-PUT'. This could be cleaned up to remove error cases.
• Section 4, Access Control: Version 3.0 should state access control policies at the Node level, or should this be considered orthogonal? * Custom views: views that clients create and use to do work/data reduction at the physical storage location.
• JPEG2000 Interactive Protocol (JPIP) and VOSpace? A number of articles on Astronomy and Computing
• move /views and /properties to a registry extension?
• From Pierre Fernique on the VOSpace 2.1 RFC page:
  • There are many OPTIONAL elements in the this standard. It is not so good for the interoperability (and concretelly we have very few VOSPace clients). I would suggest in the next standard version to avoid (if possible) too many optional features (either remove it, or impose it)
  • The auto generation of a unique URI is an important function. Presently it is implemented (optional) as a kind of "hack" via a reserved suffix keywork ".auto" to the URI. I am thinking that this important feature could be implemented in a better way, as a creatorNode parameter or something like this.

Implementations

group	version	endpoint	VOSpace root	Notes
Caltech	v1.1	testnvo.cacr.caltech.edu	vos://nvo.caltech!vospace/	HTTP transports, supports structured nodes and archive formats (tar, tar.gz)
ESO	v2.0	vops1.hq.eso.org	vos://org.eso!vospacetest/	HTTP transports
Astrogrid
CADC	v2.0	http://www.canfar.phys.uvic.ca/vospace	vos://cadc.nrc.ca!vospace	v2.1 in progress

Mailing list

The mailing list for discussion of the VOSpace specifications and related issues is: grid@ivoa.net which is archived at: http://mail.ivoa.net/pipermail/grid

VOSpace 1.0 history

Discussion

• Discussion page for property registration
• Resource schema (old version)
• Example document (old version)

The discussion pages that lead up to the V1.0 working draft are available here :

• Discussion page for the WSDL and XML schema (closed)
• Discussion page for specification (closed)
• Mailing list archive