AstroGrid FileManager

Dave Morris

AstroGrid

What Is FileManager ?

• A service that stores metadata about data in FileStore(s)
• Provides a virtual view of files distributed across multiple store locations
• Uses SOAP for control messages, but data is transferred by direct connection to the FileStore(s)

Hierachical view

• FileManager metadata is represented as a tree of nodes
• Individual nodes can be accessed via a root node and path
• {home}/solar-project/results/query-7.vot

Node identifiers

• Each node has a unique identifier, issued by the FileManager service
• Identifiers are based on the service IVO identifier, plus a unique node number
• ivo://ast.cam.ac.uk/filemanager#1234-5678
• Identifiers are globally unique
• The service identifier is unique within the registry
• ivo://ast.cam.ac.uk/filemanager
• The node number is unique within the service
• 1234-5678

Child nodes

• Child nodes can be addresses using their unique identifier
• ivo://ast.cam.ac.uk/filemanager#9674-124b
• Or, using their parent node and a path
• ivo://ast.cam.ac.uk/filemanager#1234-5678/solar-project/results/query-7.vot

Community accounts

• FileManager trees are tied to a Community account
• Each Community account has a separate root node in a FileManager
• The location of the FileManager root node is a property of the Community account
• Daves account in Cambridge Community
• ivo://ast.cam.ac.uk/dave
• FileManager root for this account is
• ivo://ast.cam.ac.uk/filemanager#1234-5678
• Node identifiers are for internal use, users should never have to see them
• Users should see file paths like {home}/solar-project/results/query-7.vot

Resolving data location

• Users see file paths like {home}/solar-project/results/query-7.vot
• If I am logged in as Dave at Cambridge
• ivo://ast.cam.ac.uk/dave
• {home} should automagically be resolved to
• ivo://ast.cam.ac.uk/filemanager#1234-5678
• Long identifier for the node is {home} plus the path
• ivo://ast.cam.ac.uk/filemanager#1234-5678/solar-project/results/query-7.vot
• Metadata for the node indicates the data is stored in FileStore at Leicester
• ivo://star.le.ac.uk/filestore#ab56-9a55

Client access

• Client software interacts with FileManager, which contacts FileStore(s) on their behalf
• A client would not normally interact directly with a FileStore

Creating new file node

• Client contacts FileManager and requests a new data node
• Client must provide
• The identifier of the parent node
• e.g. {home}/solar-project/results
• The name of the node
• e.g. query-8.vot
• Client may provide additional properties describing the node
• MIME type
• The preferred FileStore to use
• FileManager allocates a new (empty) node for the data
• FileManager replies to client with
• The identifier for the new node
• ivo://ast.cam.ac.uk/filemanager#9674-124b
• The array of properties associated with the node

Import init (short form)

Client has data it wants to send

• Client contacts FileManager and creates a new data node
• Client contacts FileManager requests an import by calling importInit()
• Client specifies the node identifier
• ivo://ast.cam.ac.uk/filemanager#1234-5678/solar-project/results/query-8.vot
• FileManager chooses preferred FileStore and calls importInit()
• FileStore creates an empty container and replies to FileManager with
• The identifier for the new container
• ivo://star.le.ac.uk/filestore#ab56-9a55
• The location to send the data to
• http://store.star.le.ac.uk/path/file
• FileManager keeps FileStore identifier and passes URL to the client
• Client uses the URL to send the data to the FileStore

Import init (async)

Client has data it wants to send

• Client contacts FileManager and creates a new data (file) node
• Client contacts FileManager requests an import by calling importInit()
• Client must provide the node identifier
• ivo://ast.cam.ac.uk/filemanager#9674-124b
• or
• ivo://ast.cam.ac.uk/filemanager#1234-5678/solar-project/results/query-8.vot
• FileManager chooses preferred FileStore and calls importInit()
• Preferred FileStore can be based on
• Properties of the node
• Properties of the parent node **
• Properties of the account **
• Default for this service
• ....

Import init (async)

• ....
• FileManager sends FileStore its own identifier as callback address **
• FileStore creates an empty container and replies to FileManager with
• The identifier for the new container
• ivo://star.le.ac.uk/filestore#ab56-9a55
• The location to send the data to, and the protocol and method to use
• http://store.star.le.ac.uk/path/file
• HTTP
• PUT
• FileManager stores the FileStore container identifier in the node properties
• filestore.location = ivo://star.le.ac.uk/filestore#ab56-9a55
• ....

Import init (async)

• FileManager replies to client with
• The location to send the data to, and the protocol and method to use
• http://store.star.le.ac.uk/path/file
• HTTP
• PUT
• ---- SOAP call completed ----
• Client uses the URL to send the data to the FileStore
• FileStore notifies FileManager when transfer has completed **
• FileManager updates node properties to include results from FileStore
• Content size
• Modified date

Import data (sync)

Client wants to store data from an external location

• Client contacts FileManager and requests an import by calling importData()
• Client specifies
• The node identifier
• ivo://ast.cam.ac.uk/filemanager#1234-5678/solar-project/results/query-8.vot
• Where to fetch the data
• http://www.ast.cam.ac/path/file
• HTTP
• GET
• FileManager passes the request to the preferred FileStore
• FileStore creates a new container and fetches the data from the URL
• FileStore replies to FileManager with
• The identifier for the new container
• Additional properties describing the data it recieved
• FileManager updates the node properties with response from FileStore
• FileManager replies to client