Searched: \.*
Results from IVOA web retrieved at 13:35 (GMT)
-- KeithNoddle, May 2009
The DAL roadmap is simplified to include the capabilities: DataLink, SIA query, SIA metadata (get-gory-details), and accessData (from DAL) as well as dependencies on the DM-WG for ObsCore and imageDM. We expect several versions of SIA-2.x and AccessData-1.x to be produced before all use cases are satisfied.
* Extension of the UCD List for planetary data:
There are requirements from the Planetary community to use and extend the IVOA UCD list for using these semantic tags while mapping their metadata.
Some concepts , current in Planetary science are missing .
An effort to propose new possible UCD strings and validate them in the IVOA UCD maintenance group is taking place .
Proposed list by May Interop 2014
The DAL roadmap is simplified to include the capabilities: DataLink, SIA query, SIA metadata (get-gory-details), and accessData (from DAL) as well as dependencies on the DM-WG for ObsCore and imageDM. We expect several versions of SIA-2.x and AccessData-1.x to be produced before all use cases are satisfied.
* Extension of the UCD List for planetary data:
There are requirements from the Planetary community to use and extend the IVOA UCD list for using these semantic tags while mapping their metadata.
Some concepts , current in Planetary science are missing .
An effort to propose new possible UCD strings and validate them in the IVOA UCD maintenance group is taking place .
Proposed list by May Interop 2014
→ call to read and comment the VODML mapping document JSAMP
→ Support for https → try and see if a few standard adjustements will be required HiPS
→ Study a new capability entry in the registry ?
* "promote" HiPS IVOA note as a Working draft
* Study a new capability entry in the registry ?
VOTable 1.4
* still awaiting VODML serialization decision before starting a new cycle JSAMP * Support for https → try and see if a few standard adjustements will be required
Addressing RFC comments, final update expected by end of June. Should be ready for TCG approval by end of July.
VODML-1.0: Cape Town discussions indicate the document is PR ready once a 'How To Get Started' style help document is completed. This document is scheduled for completion by end of June. PR period should start in July. Target REC by Trieste.
STC-2.0: WD feedback is strongly on the side of needing to split the model to a core package with secondary extensions for higher level packages (like Region and Area). Project scope to be reviewed Jun/Jul, with support of Dataset/NDCube models to be the focus. Need to schedule an external review panel to evaluate model and provide recommendations.
DatasetMetadata-1.0: Model is stable, with user implementations/serializations in both VODML Mapping and utype conventions. Focus will be on organizing external interoperable usage of the model in a SIAV2 thead. Target is REC ready by Trieste.
NDCube-1.0: Similar to DatasetMetadata, working draft is stable. User implementations to be packaged and uploaded to twiki. Focus will be on organizing external interoperable usage of the model in a SIAV2 thread. Target is REC ready by Trieste.
Provenance-1.0: The Dataset/Observation/Provenance models are coming toward a common framework. For Trieste, the goals will be to:
a) fanalize the vodml-compliant data model so that it works with only minor modifications for all our use cases.
b) deliver working draft model document; including minimum requirements; serialization strategy; and access to Provenance data.
c) work relations between SimDM and ProvDM
SourceDM-1.0: Team is to organize project goals and scope, summarized to twiki pages. Project should broaden use-case example set beyond Gaia. At present, there is no tageted status for Trieste.
<!--
* Set ALLOWTOPICRENAME = TWikiAdminGroup
-->
* Set ALLOWTOPICRENAME = TWikiAdminGroup
-->
STC in the Registry: Work in prototypes will continue based on a recent STC in the Registry note introducing the concept of MOCs in the registry. We will coordinate with DM on updates to or removal of existing STC registry extensions based on DM's standards reorganization and future compatibility. (Nov '18: ongoing. MOCs in the registry introduced, needs revisited next meeting, waiting on DM RFC)
Discovering Dependent Resources: A note in progress addresses the problem of discovering data collections that are published through other services, the prime use case so far being TAP tables. A path forward involving creating a new registry extension allowing both tables and capabilities was discussed in Victoria, this work will be continued and prototypes developed; work on the main document is postponed until this is resolved.
Standards records: We will continue to work with the chairs of the other IVOA working groups to ensure that they register their standards as envisioned by StandardsRegExt (cf. WriteAStandardsRecord). (Worked with Time Domain IG & DAL for revisiting VOEvent registration)
Maintenance: We will keep making sure the registry system runs; in particular, we will address anomalies diagnosed by ESAVO especially including inconsistencies in the Registry of Registries. (Validation run, operations meeting held, anomalies addressed with RofR and Euro-VO as an example, work ongoing)
Time domain IG members working with ZTF and LSST to develop brokers and filters: Lessons learned from ZTF and LSST will inform future developments of VOEvent Feedback from SSIG, investigate and report on how to proceed on VOEvent to meet their requirements Time Series: DAL: TimeSeries extension for DAL protocols as a single spec: “TimeSeries discovery and access protocol a set of extensions to DAL protocol for TimeSeries” Obscore extension (Time sampling…) Corresponding PARAMETERS in SIAV2-like interface SODA extension parameters for data selection Access or DataLinks tags in source-driven context DM: Representation (requires modelling and serialisation) Work close to Gaia, ZTF, LSST to meet their deadlines Deliver different versions on our path towards ultimate goal Present serialisations —> encourage applications to test —> collect feedback Steps towards a TimeSeries DM:
- 2.1 - Feature complete - validation being finalised
- 2.2
- Vector extension
- move from BNF to PEG for language specification
ADQL User Defined Functions
- ADQL astrometry library
- GAVO and Vizier implementation of coordinate transformation
- Functions added in pgsphere
TAP
- Allow ROT13 for encoding of ADQL query strings? SIAP-2.0 -> Data Access Protocol v1
- Expanding to cover spectra and potentially measurement sets and catalogues
- One shot cutouts from SIA2
- Renaming SIA2 to reflect expanded role
- Work continues in new GitHub repo: https://github.com/ivoa-std/DAP
- New WD by May interop SODA-1.1
- Errata for MOC parameter, rebin/project, pixel cutout
- One shot cutouts from SIA2
- Cross-join of parameters
- Metdata
- Select output by dataproduct_type DataLink 1.1
- Working draft out
- Needs implementations before RFC DALI-1.2
- multipolygon still pending
- otherwise ready for PRec+RFC (Q1-2023) SSAP
- Minor revision to
- make coverage information non mandatory
- Explain SSA utype prefix
- Longer term function is covered by DAP + SODA ConeSearch -1.1 [in connection with TD IG]
- work will progress to close issues
- a new WD before May 2023 Interop ? LineTAP
- Intended to replace SLAP
- WD before May 2023 Interop ? ProvTAP
- Continuing work towards mature WD status TAPRegExt [in connection with Registry WD]
- WD release in Q2-2023 ObjVisSAP
- PR-1.0 nearly there
- implementations ready Following Radio IG, TD IG activities for requirements
- refer to the roadmaps of these IGs for details
- Complete the REC step
- Providing a PyVO implementation MANGO (the missing piece in between MIVOT, PhotDM and MCT)
- REC step Field of view DM
- having a draft Observation Proposal DM
- having a draft CubeDM
- having drafts (including DatasetMetadata) + reference implementations Transform (3rd lettre of MCT)
- having a draft+ reference implementations Spectrum DM 2.1
- REC step
ADQL
LineTAP
https://github.com/ivoa-std/SSO/issues
Also, reformat of GitHub repository. Cross-science platforms execution project The GWS WG is going to start a project to identify the main changes and new standards to allow remote execution on different astronomical science platforms.
https://github.com/ivoa/ExecutionPlannerNote
This note should be either converted into a standard or, at least, upgraded making it a more powerful interface. To do that, it is necessary to standardise:
Characterisation of software (in terms not only of software dependencies but, also, dependencies that could affect execution like memory, requirements on computing architecture (e.g. CPUs and GPUs), data location, etc. Software should be exposed in a portable format
Metadata characterisation of the science platform node is also a requirement
Access to this metadata should be also characterised as per the definition of a software discovery service and a science platform resources discovery service. The execution planner could use metadata of both types to decide the best place to execute a certain workflow.
For the software characterisation and discovery, other CI/CD solutions could be expanded and already have the connections between code and containers. For the science platform node service, we need to have a dynamic service to capture the real-time status of the resources.
The execution planner and the science platform discovery service could be integrated into the same service and the software discovery looks to be a different one. Endorsement of a standard for workflow serialisation Identify one standard for workflow serialisation (e.g. CWL) and endorse it for the IVOA. This format should have software to parse and decompose a workflow in smaller executable elements in different languages if possible. Execution interface Identify an execution interface that could abstract the internal execution interfaces of the different science platforms. Probably something based on IVOA UWS could be used to control execution Credentials Delegation Review of related standards for authentication/authorisation combined with the execution interface. This could be a problem for a general approach as some computing centres have their local access policies For this exercise, SKA Regional Centre Network members are available to collaborate, lead and implement although it would be also needed to have members of other projects that are implementing (or already have) running science platforms to create another independent implementation (candidates Vera Rubin, ESA, CADC, Gaia UK,...)
* Set ALLOWTOPICRENAME = TWikiAdminGroup
-->
Converging to "Distributed Services and Protocols" - proposed by Dave Morris
<!--
* Set ALLOWTOPICRENAME = TWikiAdminGroup
-->
* Set ALLOWTOPICRENAME = TWikiAdminGroup
-->
ADQL defines an SQL-like grammar adapted for astronomical purpose. It is especially used by the TAP (Table Access Protocol) standard. Latest version of ADQL can be found at: Here is a diff view of all the modifications done on the above PR document since the beginning of this RFC period:
Substantive
For instance, coordinate system used in VizieR tables depends of each catalogues.
The coord syst is important in case of crossmatch (from an other table or from an UPLOAD) If the parameter is removed, it requires;
<!-- Set ALLOWTOPICRENAME =<span class="WYSIWYG_PROTECTED"> TWikiAdminGroup </span> -->
is present in the
I.e., the limited ISO 8601 format only, as defined for FITS and in STC. VOQL-TEG Answer: RDBMSs implicitly convert strings to internal (datetime or timestamp) form using a variety of techniques; ISO8601 format is an acceptable format already. As with other string representations, it is a service capability (possibly mandatory) to understand specific formats. The PR introduces a "geometry" data type, but it really is a hodgepodge of stings - essentially, a geometry data type is defined, if I understand it correctly, as a string that contains a function call, with a variety of functions and parameters that seem to be reinventing STC. If one wants to introduce a new data type, there is a much more elegant and ready-made option: an STC-S string. I would be more inclined to call it an STC data type, since the term geometry data type is misleadingly narrow, but that can be discussed. VOQL-TEG Answer: As described in the attached presentation (and above) STC can be used within the REGION construct; it is up to a service to specify that it understands STC and they can support as little or much as is feasible. The ADQL language itself remains independent of any specific string representation or version thereof. In the Interop discussions it was explicitly stated that we expect TAP services to use standards (for region and the coordinate system arguments). The text within the PR will be expanded to make this clear. Having defined such a data type, one can use the to-be-developed STC library to verify its validity and manipulate (interpret) the string, and define a set of STC core functions to provide the necessary operations, such as intersection, union, contained_in, etc. The STC-S string has the added advantage that it can also handle coordinates, so that problem would be resolved, too. As it is, the PR basically develops its own STC model, through an ad-hoc set of functions and parameters. This, I think, is undesirable. If the IVOA has an accepted standard for certain structures, one should use it, rather than roll one's own. The reason is a very practical one: it encapsulates the model issues in that particular standard, allowing its related library to handle the details in a transparent and uniform way across applications, allowing models to evolve as necessary without impacting the users. If ADQL and SIAP and SSAP and Registry all would start using their own definitions of, say, Circle we create chaos for the user. I have heard many complain that incorporating STC greatly complicates the systems into which it is to be integrated. That is nonsense. I have said it numerous times and will say it again: There is nothing wrong with applications recognizing only a limited subset of STC structures and coordinate systems - returning an error if they encounter something they cannot handle. That is the way STC was incorporated into VOEvent - and it works. It's perfectly acceptable to accept only simple strings and a limited set of shapes and coordinate systems. It is not hard to parse those strings and the enormous advantage is that (a) one is compliant with an IVOA standard and (b) one is ready to expand functionality to include more cases, shapes, and coordinate systems at any time, without any changes to the interface, just by adding code to the server software. Aside from the fact that it is not a good idea to develop a new STC model for ADQL, there are a number of specific problems with the way it has been done, as well. The only coordinate system flavor recognized in the PR is 2-D spherical. Coordinates are specifically assumed to consist of a longitude and latitude. This may seem sensible, but it means that one is painting oneself into a corner for future extensions. VOQL-TEG Answer: The defined geometry types are 2-D and the LONGITUDE/LATITUDE functions unnecessarily restrict it to spherical coordinates (see above for more about LONG/LAT). VOQL-TEG Answer: The PR was arrived at by balancing usefulness and complexity and the TEG feels we have found the best comprimise. We have considered future extensibility and found that it is plausible to make only modest extensions beyond the current PR. It is feasible to add 1-D regions (intervals) and 2-D ellipse in a way that is consistent with the PR. After that, one would have to break with SQL to add much value. Another issue is that, again, if I understand it correctly, the parameters in the function calls may be valued or consist of column references. The problem is that there is no way to handle the coordinate system if the coordinate elements consist of references. I cannot say "function ('ICRS', table.ra, table.dec)" because I do not necessarily know whether the table contains ICRS positions - nor should I have to. VOQL-TEG Answer: The first argument should be another column in the table, or the whole construct (a POINT) should be taken from a single column of the table. See attached presentation. If I ask whether the positions in a table lie within a certain region that I define in a certain coordinate system, the server (knowing what its own coordinate system is, one would hope) should perform the necessary transformations. The PR specifies that when a server cannot perform a necessary transformation, it should return a NULL. I question whether this is a sensible thing: it is indistinguishable from "I have no data". I would much rather see a standardized warning returned.
We should also consider the fact that an abuse acces to a service could be somewhere in a workflow (a simple pipeline or a more complex execution plan). In this case the user will probably not know that he is guilty. For him, the execution of his query could be considered as slow (if a delay is used) or not possible(if a redirection is used) .
* Collaborative Page for AccessData This page is to capture details of discussions around the AccessData capability being deveveloped for the Multi-dimensional data access CSP project. Primary discussion occurs on the dal@ivoa.net mailing list. Summaries from meetings, intermediate documents, and important links/resources can be added here. It is important to note that the grand plan for AccessData is to produce incremental versions of the document starting with basic features and adding more complex things later -- once the basics are implemented. Features must be prototyped and proven before they should be added to the specification. * Documents The initial draft document is WD-AccessData-1.0-20140312 and is available on this page only (see below) * Issues
If you would like to to participate, please add your prefix to the table.
Original text by -- DaveMorris with additions and suggestions from -- MarkusDemleitner
Alasdair Allan
eSTAR Project
School of Physics
University of Exeter
Stocker Road
Exeter
EX4 4QL
Tel. +44-1392-264160
Email aa@astro.ex.ac.uk
Web http://www.astro.ex.ac.uk/people/aa/
Related topics 
Results from IVOA web retrieved at 13:35 (GMT)
DAL Sessions
May 2009 Interop Meeting (back to main program page)DAL Sessions: Preliminary Schedule
| TAP | Monday, May 25, 14.0015.30 | Théâtre |
|---|---|---|
| Speaker | Title | File |
| KeithNoddle | Introduction | |
| GerardLemson | The SimDB perspective on TAP and DAL in general | ppt |
| Pierre Le Sidaner & Mathieu Hirtzig | Implementation of TAP in planetary data | |
| DougTody | Cross matching and Data Discovery with TAP (also PQL update) | |
| PatrickDowler | TAP V0.5 | |
| KeithNoddle | Discussion of outstanding TAP issues | |
| S*AP | Tuesday, May 26, 16.0017.30 | Amphithéâtre |
| Speaker | Title | File |
| JesusSalgado | Introduction, progress review and key issues | |
| JesusSalgado | Simple Line Access Protocol (SLAP) 0.9 WD | |
| FrancoisBonnarel | SIAv2 and Aladin/TOPCAT interaction | |
| DougTody | SIAv2 interface and issues | |
| DougTody | Generic dataset (GDS) developments | |
| FrancoisBonnarel | GDS link with Observation | |
DAL WG Summary slides
-- KeithNoddle, May 2009
IVOA Roadmap for 2013B
This outlines the roadmap for development activities by the various IVOA working and interest groups in 2013/14 between the Hawaii and Madrid Interops.Applications
- MOC 1.0: Following discussions between authors a PR has been submitted for publication in January 2014, and the RFC is running from 7 February to 7 March. If all goes smoothly, we hope to achieve REC status near the May 2014 interop.
- Code Repositories: Following a request from the Exec, the Apps WG will start a discussion on use within the IVOA of code repositories beyond those at the project level. The Exec made this request in the context of the discussion in the Time Domain focus session at the Heidelberg (May 2013) interop in which Mario Juric (LSST) expressed his disappointment at not finding a one-stop shop for IVOA-blessed implementations of VO standards, specifically in Python. We will consider the current status and future possibilities for the IVOA and its member projects in interacting with existing repositories (notably Astropy) or setting up new ones, taking into account the desirability, costs and benefits of the various options.
Data Access Layer
- pre-Hawaii Interop status:
- DALI-1.0 TCG review almost complete, new PR doc with all changes to date has been submitted
- WD-DataLink-1.0 is late (to be submitted asap, it is missing 2 sections), final issues to be discussed at Interop
- WD-STC-S.1.0 was posted in Sept, session at Interop
- no work on TSA-1.0
- cube access prototypes to be discussed at Interop
- DataLink-1.0
- WD in October 2013
- WG review January 204
- PR document: February 2014
- RFC in March 2014
- TCG review in April 204
- Recommendation: May 2013
- STC-S-1.0
- this has been deemed not needed for SIA-2.0 and AccessData-1.0; these will adopt simple geometry value(s) without metadata
- SIA-2.0
- AccessData-1.0
- WD after WD-SIA stabilises
- implementations need to be done
- more discussions in May 2014
- TSA-1.0 -- small standard that re-uses SSA-1.1 with the time series DM produced by DM-WG (presumably SimpleTimeSeriesDM)
- WD: TBD
- WG review:
- RFC period:
- TCG review:
- Recommendation: April 2014
- discovery of data cubes:
- SIA query interface (SIA query capability)
- TAP + ObsCore
- download of complete data cubes:
- direct link in query response for single-file case
- DataLink
- get additional metadata (SIA metadata capability)
- return instance of ImageDM, format TBD
- needed to perform some of the more advanced data access operations
- data access (AccessData capability)
- extract subsection of cube (doable without detailed metadata)
- flattening along one or more axes (sum or avg) (req: metadata)
- rebinning along one or more axes (req: metadata)
- precision pixel-level access (req: metadata)
The DAL roadmap is simplified to include the capabilities: DataLink, SIA query, SIA metadata (get-gory-details), and accessData (from DAL) as well as dependencies on the DM-WG for ObsCore and imageDM. We expect several versions of SIA-2.x and AccessData-1.x to be produced before all use cases are satisfied.
Data Model
PhotDMv1-1 1.0 TCG review started on July 2012. Work stopped due to coupling with utypes Tiger Team work as PhotDMv1-1 was intended to be in line with utypes recommendation. Process of adaptation to the ongoing discussions within Tiger Team. It is decided (12/05/2013 TCG meeting) to decouple both activities, update PhotDMv1-1 with the comments obtained during TCG review and encourage TCG members to comment on it (to be done before the end of Spring 2013 interop) PhotDMv1-1 will be adapted for 2013 Autumn interop following VO-DML approach Spectral DM 2.0 "Trying to be final" version released to the working group in March 2013. No major comments received. To be presented to WG during Spring 2013 interop. Implementation pending to formally start RFC process Characterization DM 2.0 Stable version released to the working group in September 2012. No major comments received. Implementation to be presented during Spring 2013 interop. Waiting green-light to start RFC process Image DM Draft to be presented during Spring 2013 interop. Wider list of authors to be identified VO-DML Draft to be presented during Spring 2013 interop. Prototypes and comments to be received between Spring and Autumn 2013 interopsGrid and Web Sevices
* PDL:- May 2012: first WD: PDL-0.1-WD-20120516
- May 2013: revised WD in 1.0 and status discussion in Heidelberg GWS session 1
- August 2013 : RFC
- March 2014 : end of the TCG Review
- Spring 2014: REC?
- Comments and suggestions from the community to prepare the VOSpace WD 2.1 for Madrid
- REC ? : probably in Fall 2014 or Spring 2015
- Comments and suggestions from the community to prepare the WD UWS 1.1 for Madrid
- REC ? : probably in Fall 2014 or Spring 2015
Registry
SimpleDALRegExt 1.0 SimpleDALRegExt has become a REC. VOEventRegExt In the context of VOEvent 2.0, a working draft of the VOEventRegExt is currently being finalized for metadata inputs with the VOEvent mailing list group. Matthew Graham is coordinating this effort. The RWG will participate in the RFC process to evaluate the specification in context of the registry standards integration. Registry Interface and RegTAP The RegistryInterface 2.0 (RI-2) working draft has been distributed to the RWG for review and is pending revisions for progression to the next phase for RFC. Our RWG goal for this standard is to move forward into the REC process following review and coordination during the May 2014 interop meeting. RI-2 will retain the standard OAI protocol for Publishing Registry and harvesting from the Publishing registry into the Fully Searchable Registry. This RI-2 standard will deprecate the the earlier SOAP interface support and provide 2 forms of web based REST interfaces, APIs, for the application client community: a TAP protocol based on the RegTAP relational registry model standard, and a SimpleRegREST interface. The later is the key to completion of this updated standard registry interface definition. A consolidated approach based on the integration of the Simple REST prototype via the EuroVO registry and VAO registry REST interface will be proposed. This standard RFC process is a high priority topic of discussion for the May 2014 interoperability meetings with direct impact to IVOA science application tool development that has registry interface dependency. An underlying Registry relational data model, RegTAP, has been developed which describes the core registry metadata to be exchanged using the RI 2.0 standard interface. This document is currently beginning the RFC phase. Registry Metadata for Footprints The RWG will continue to have discussions and evolve the current standards infrastructure in support of metadata for footprint descriptions, i.e. spatial regions of coverage. The advancement in the MOC and STC/s standards definition will be evaluated in the context of footprint representation for support of applications in development of data discovery. Future coordination between IVOA groups will involve potential development for spatial indexing in support of application integration with registry collections. Registry Operations and Curation The Registry WG continues to drive regular discussions during Interoperability meetings about IVOA Registries operations, including curation of VO Resources and validation of DAL services. The Registry WG is pending effort to develop a coordinated plan accepted by TCG and proposed to the Exec for developing the procedures and processes of curating nonresponsive or non-valid registered DAL services. This also includes consistency in Registry managed authorities and harvested resource metadata across the IVOA registry network. -- GretchenGreene - 12 February 2014 last updateSemantics
The WG will continue to contribute to other WGs and IGs as appropriate.* Extension of the UCD List for planetary data:
There are requirements from the Planetary community to use and extend the IVOA UCD list for using these semantic tags while mapping their metadata.
Some concepts , current in Planetary science are missing .
An effort to propose new possible UCD strings and validate them in the IVOA UCD maintenance group is taking place .
Proposed list by May Interop 2014
Data Curation & Preservation
The IG will continue discussions on general topics of interest to the community. Some of the topics we expect to focus on during the next 12 months include:- The Unified Astronomy Thesaurus
- Review of current effort described online at http://astrothesaurus.org
- Participation of IVOA members in merging / mapping similar vocabularies (e.g. the Paris Observatory's Etymological Dictionary: http://dictionary.obspm.fr/)
- Drafting editors to support the development and review of the thesaurus
- The Research Data Alliance
- Follow developments of the RDA: http://rd-alliance.org/
- Participate in working groups on persistent identifiers and data citation
Knowledge Discovery in Databases
Theory
The main goal of the Theory I.G. is to propose an access protocol to discover and retrieve simulated data. From last InterOp, several implementations have been tested. We also had a few teleconferences with some members of the Theory I.G. Search for simulations Some technical difficulties have been identified. The most important of them, is that, contrary to an observationnal data, a simulated data may require several thousands quantities to be described in a way allowing us to answer our usecases. Because RDBMS are not efficient with tables with more than thousands columns, the access protocol for theoretical data should not rely on specific technologies (as TAP/RDBMS). During the second Theory session in Heidelberg InterOp we will discuss this important topic. A possible solution, would be to define a few APIs with a main one, the SEARCH/CUTOUT function, to discover interesting simulations. The query language would be a subset of ADQL. On the other hand, the exchange format will have to be defined. We will discuss the oportunity to use the serialization of SimDM in VO-Tables following the proposal of the utype tiger team. Extraction of data Concerning the extraction of data, the situation is more simple. Services will provide a Datalink document to describe the possible operations on data. This will be presented and discussed in the common DAL / Theory session about datalink. Goals of this InterOp for the Theory I.G. are : 1) conclude on a solution to discover simulations that is independent on the technology used to store metadata, 2) define the large picture about the exchange format document in the communication client / server during the search phase of simulations, 3) advance on the syntax of datalink document.Education
The newly formed Edu IG is planning to discuss its Roadmap during the Heidelberg 2013 Interop. The basis for the discussion are the Terms of Reference that can be read in the IG's web page together with the Rationale. At the Heidelberg Interop, the Edu IG will have both a morning and an afternoon session on Thursday May 16. At least one of the two sessions will be devoted to discussions of the Roadmap and its implementation. Contributions from anybody interested will be most welcome. -- MassimoRamellaINAF - 2013-03-25Time Domain
The TDIG is working on a note endorsing SimpleTimeSeries as an IVOA-approved format for time series data and also waiting for feedback from the Registry WG on VOEventRegExt (see above). We anticipate participation in the multi-dimensional data discussions. We will issue an update note covering the VOEvent Transport Protocol, and then push that towwards becoming an IVOA-approved standard. -- MatthewGraham - 2013-05-09 -- JohnSwinbank - 2014-02-11Standard and Processes
No evolution of the standards forseen in 2013.Science Priorities
Science priority areas are: Multi-dimensional data, and Time Domain Astronomy. Focus sessions at the May 2013 interop meeting will engage projects and surveys that produce and use multi-dimensional data (including radio astronomy, IFU, high energy and simulation data) and Time Domain data (time series, light curves, transient event reports) in a discussion on the role of VO technologies in these areas. Multi-d and Time Series use cases are being collected as part of this effort.IVOA Roadmap for 2013A
This outlines the roadmap for development activities by the various IVOA working and interest groups in 2013 between the Heidelberg and Hawaii Interops.Applications
- VOTable 1.3:
- Mar 2013: PR: VOTable-1.3-PR-20130315
- May 2013: TCG review (ends 29 May 2013); no major disagreements at RFC stage
- Sep 2013: REC.
- MOC 1.0:
- Feb 2013: first WD (internal)
- Mar 2013: Revised WD: MOC-1.0-WD-20130314
- May 2013: Discussed at Heidelberg Interop in joint Apps/Registry session
- Sep 2013: Revised WD: MOC-1.0-WD-20130910
- Sep 2013: Discuss implementation experience at Hawaii Interop
- Fall 2013: PR?
- Early 2014: REC?
Data Access Layer
- DALI-1.0:
- May 22, 2013: revised PR document after RFC period
- TCG review: May 24 - June 24, 2013
- Recommendation: summer 2013
- DataLink-1.0
- WD in June 2013
- WG review July 2013
- PR document: August 2013
- RFC period August - September 2013
- TCG review: October 2013
- Recommendation: late 2013
- STC-S-1.0
- WD in July 2013
- implementations before Hawaii Interop
- WG review October 2013
- PR document: October 2013
- RFC period: November 2013
- TCG review: January 2014
- Recommendation: March 2014
- TSA-1.0 -- small standard that re-uses SSA-1.1 with the time series DM produced by DM-WG (presumably SimpleTimeSeriesDM)
- WD: TBD
- WG review:
- RFC period:
- TCG review:
- Recommendation: April 2014
- discovery of data cubes: TAP + ObsCore
- download of complete data cubes: DataLink
- extract subsection of cube using astronomical coordinates (STC-S)
- flattening along one or more axes (sum or avg)
- rebinning along one or more axes
- precision pixel-level access
Data Model
PhotDMv1-1 1.0 TCG review started on July 2012. Work stopped due to coupling with utypes Tiger Team work as PhotDMv1-1 was intended to be in line with utypes recommendation. Process of adaptation to the ongoing discussions within Tiger Team. It is decided (12/05/2013 TCG meeting) to decouple both activities, update PhotDMv1-1 with the comments obtained during TCG review and encourage TCG members to comment on it (to be done before the end of Spring 2013 interop) PhotDMv1-1 will be adapted for 2013 Autumn interop following VO-DML approach Spectral DM 2.0 "Trying to be final" version released to the working group in March 2013. No major comments received. To be presented to WG during Spring 2013 interop. Implementation pending to formally start RFC process Characterization DM 2.0 Stable version released to the working group in September 2012. No major comments received. Implementation to be presented during Spring 2013 interop. Waiting green-light to start RFC process Image DM Draft to be presented during Spring 2013 interop. Wider list of authors to be identified VO-DML Draft to be presented during Spring 2013 interop. Prototypes and comments to be received between Spring and Autumn 2013 interopsGrid and Web Sevices
* PDL:-
- May 2012: first WD: PDL-0.1-WD-20120516
- May 2013: revised WD in 1.0 and status discussion in Heidelberg GWS session 1
- Summer 2013: PR ?
- Fall 2013: REC?
Registry
SimpleDALRegExt 1.0 SimpleDALRegExt is currently in the TCG review. The RWG has incorporated several comments and feedback from the RFC period into the latest version and is still soliciting inputs from the TCG members. The official TCG review period was scheduled to complete January 2013, yet has been extended to collect all the working group inputs through the time of this roadmap draft. The RWG anticipates completion of the TCG review and discussion at the May interop meeting to finalize draft recommendation to the Exec for REC approval. StandardsRegExt 1.0 Following the standard nomenclature for Registry Extension standard, the StandardsRegExt has become a REC. IVOA standards will be instantiated in this registry extension metadata format and populated in the Registry of Registries central repository as reference implementations. VOEventRegExt In the context of VOEvent 2.0, a working draft of the VOEventRegExt was distributed to the RWG for comments and feedback on the standard description. This registry extension will provide VOEvent server and stream discovery. Further work is required to define the capability metadata for this new extension. Registry Interface and RegTAP The RegistryInterface (RI) 2.0 working draft has been distributed to the RWG for review. This standard is an technology upgrade from the previous version to include REST web interface for the registry search API. The RWG has agreed to replace the current RI 1.0 SOAP web based protocol using a TAP based service and support a simple registry resource model interface with specifications still in development. This standard is a high priority topic of discussion for the 2013 interoperability meetings with direct impact to IVOA science application tool development that has registry interface dependency. An underlying Registry relational data model, RegTAP, has been developed which describes the core registry metadata to be exchanged using the RI 2.0 standard interface. This document is in circulation as a RWG working draft. Other Registry Development The Registry WG continues to drive regular discussions during Interoperability meetings about IVOA Registries operations, including curation of VO Resources and validation of DAL services. The Registry WG is pending effort to develop a coordinated plan accepted by TCG and proposed to the Exec for developing the procedures and processes of curating nonresponsive or non-valid registered DAL services. This also includes consistency in Registry managed authorities and harvested resource metadata across the IVOA registry network. -- GretchenGreene - 19 March 2013Semantics
The Semantics WG is participating in ongoing thesaurus/vocabulary efforts in collaboration with the DCP and Theory groups. The WG will continue to contribute to other WGs and IGs as appropriate.Data Curation & Preservation
The IG will continue discussions on general topics of interest to the community. Some of the topics we expect to focus on during the next 12 months include:- The Unified Astronomy Thesaurus
- Review of current effort described online at http://astrothesaurus.org
- Participation of IVOA members in merging / mapping similar vocabularies (e.g. the Paris Observatory's Etymological Dictionary: http://dictionary.obspm.fr/)
- Drafting editors to support the development and review of the thesaurus
- The Research Data Alliance
- Follow developments of the RDA: http://rd-alliance.org/
- Participate in working groups on persistent identifiers and data citation
Knowledge Discovery in Databases
Theory
The main goal of the Theory I.G. is to propose an access protocol to discover and retrieve simulated data. From last InterOp, several implementations have been tested. We also had a few teleconferences with some members of the Theory I.G. Search for simulations Some technical difficulties have been identified. The most important of them, is that, contrary to an observationnal data, a simulated data may require several thousands quantities to be described in a way allowing us to answer our usecases. Because RDBMS are not efficient with tables with more than thousands columns, the access protocol for theoretical data should not rely on specific technologies (as TAP/RDBMS). During the second Theory session in Heidelberg InterOp we will discuss this important topic. A possible solution, would be to define a few APIs with a main one, the SEARCH/CUTOUT function, to discover interesting simulations. The query language would be a subset of ADQL. On the other hand, the exchange format will have to be defined. We will discuss the oportunity to use the serialization of SimDM in VO-Tables following the proposal of the utype tiger team. Extraction of data Concerning the extraction of data, the situation is more simple. Services will provide a Datalink document to describe the possible operations on data. This will be presented and discussed in the common DAL / Theory session about datalink. Goals of this InterOp for the Theory I.G. are : 1) conclude on a solution to discover simulations that is independent on the technology used to store metadata, 2) define the large picture about the exchange format document in the communication client / server during the search phase of simulations, 3) advance on the syntax of datalink document.Education
The newly formed Edu IG is planning to discuss its Roadmap during the Heidelberg 2013 Interop. The basis for the discussion are the Terms of Reference that can be read in the IG's web page together with the Rationale. At the Heidelberg Interop, the Edu IG will have both a morning and an afternoon session on Thursday May 16. At least one of the two sessions will be devoted to discussions of the Roadmap and its implementation. Contributions from anybody interested will be most welcome. -- MassimoRamellaINAF - 2013-03-25Time Domain
The newly formed TDIG is working on a note endorsing SimpleTimeSeries as an IVOA-approved format for time series data and also waiting for feedback from the Registry WG on VOEventRegExt (see above). We anticipate participation in the multi-dimensional data discussions. We will issue an update note covering the VOEvent Transport Protocol, and then push that towwards becoming an IVOA-approved standard. -- MatthewGraham - 2013-05-09 -- JohnSwinbank - 2013-06-01Standard and Processes
No evolution of the standards forseen in 2013.Science Priorities
Science priority areas are: Multi-dimensional data, and Time Domain Astronomy. Focus sessions at the May 2013 interop meeting will engage projects and surveys that produce and use multi-dimensional data (including radio astronomy, IFU, high energy and simulation data) and Time Domain data (time series, light curves, transient event reports) in a discussion on the role of VO technologies in these areas. Multi-d and Time Series use cases are being collected as part of this effort.IVOA Roadmap for 2014A
This outlines the roadmap for development activities by the various IVOA working and interest groups in 2014 between the Madrid and Banff Interops.Applications
- MOC 1.0: Following discussions between authors a PR has been submitted for publication in January 2014, and the RFC is running from 7 February to 7 March. If all goes smoothly, we hope to achieve REC status near the May 2014 interop.
- Code Repositories: Following a request from the Exec, the Apps WG will start a discussion on use within the IVOA of code repositories beyond those at the project level. The Exec made this request in the context of the discussion in the Time Domain focus session at the Heidelberg (May 2013) interop in which Mario Juric (LSST) expressed his disappointment at not finding a one-stop shop for IVOA-blessed implementations of VO standards, specifically in Python. We will consider the current status and future possibilities for the IVOA and its member projects in interacting with existing repositories (notably Astropy) or setting up new ones, taking into account the desirability, costs and benefits of the various options.
Data Access Layer
- pre-Hawaii Interop status:
- DALI-1.0 TCG review almost complete, new PR doc with all changes to date has been submitted
- WD-DataLink-1.0 is late (to be submitted asap, it is missing 2 sections), final issues to be discussed at Interop
- WD-STC-S.1.0 was posted in Sept, session at Interop
- no work on TSA-1.0
- cube access prototypes to be discussed at Interop
- DataLink-1.0
- WD in October 2013
- WG review January 204
- PR document: February 2014
- RFC in March 2014
- TCG review in April 204
- Recommendation: May 2013
- STC-S-1.0
- this has been deemed not needed for SIA-2.0 and AccessData-1.0; these will adopt simple geometry value(s) without metadata
- SIA-2.0
- AccessData-1.0
- WD after WD-SIA stabilises
- implementations need to be done
- more discussions in May 2014
- TSA-1.0 -- small standard that re-uses SSA-1.1 with the time series DM produced by DM-WG (presumably SimpleTimeSeriesDM)
- WD: TBD
- WG review:
- RFC period:
- TCG review:
- Recommendation: April 2014
- discovery of data cubes:
- SIA query interface (SIA query capability)
- TAP + ObsCore
- download of complete data cubes:
- direct link in query response for single-file case
- DataLink
- get additional metadata (SIA metadata capability)
- return instance of ImageDM, format TBD
- needed to perform some of the more advanced data access operations
- data access (AccessData capability)
- extract subsection of cube (doable without detailed metadata)
- flattening along one or more axes (sum or avg) (req: metadata)
- rebinning along one or more axes (req: metadata)
- precision pixel-level access (req: metadata)
The DAL roadmap is simplified to include the capabilities: DataLink, SIA query, SIA metadata (get-gory-details), and accessData (from DAL) as well as dependencies on the DM-WG for ObsCore and imageDM. We expect several versions of SIA-2.x and AccessData-1.x to be produced before all use cases are satisfied.
Data Model
PhotDMv1-1 1.0 TCG was recommendation on October 2013 PhotDMv1-1 will be adapted at certain point (October 2014 interop?) following VO-DML approach Spectral DM 2.0 "Trying to be final" version released to the working group in March 2013. No major comments received. To be presented to WG during Spring 2013 interop. Implementation pending to formally start RFC process, RFC process started May 2014 Characterization DM 2.0 Stable version released to the working group in September 2012. No major comments received. Implementation to be presented during Spring 2013 interop. Changes done to be presented during Spring 2014 (technical support) Image DM Presented during Spring 2013 interop. Connections with DAL. RFC to be started close to 2014 October interop VO-DML Draft presented during Spring 2013 interop. Prototypes and comments received between Spring nd Autumn 2013 interops Stable/final version distributed to WG before Spring 2014 interopGrid and Web Sevices
* PDL:- May 2012: first WD: PDL-0.1-WD-20120516
- May 2013: revised WD in 1.0 and status discussion in Heidelberg GWS session 1
- August 2013 : RFC
- March 2014 : end of the TCG Review
- Spring 2014: REC?
- Comments and suggestions from the community to prepare the VOSpace WD 2.1 for Madrid
- REC ? : probably in Fall 2014 or Spring 2015
- Comments and suggestions from the community to prepare the WD UWS 1.1 for Madrid
- REC ? : probably in Fall 2014 or Spring 2015
Registry
RegTAP We will integrate the feedback from RFC into the RegTAP PR and circulate it for TCG review. Although the list of changes after RFC was fairly substantial, most of the changes were actually clarifications for fringe cases. We hope to have RegTAP ready for REC by Banff. There are now about four implementations of RegTAP. We plan to have them all pass the validation suite by Banff. Registry Interface and RegTAP With RegTAP becoming REC, the Registry Interfaces document will need revision. There is a working draft on volute that retains the standard OAI protocol for harvesting from publishing registries, but it will remove the search capabilities and instead refer to standards like RegTAP and a planned simple REST registry interface. VOEventRegExt While the actual authoring a registry extension for VOEvents, VOEventRegExt, is in the hands of GWS, RWG will continue to work closely with the authors. An underlying Registry relational data model, RegTAP, has been developed which describes the core registry metadata to be exchanged using the RI 2.0 standard interface. This document is currently beginning the RFC phase. Registry Metadata for Footprints The RWG will continue to have discussions and evolve the current standards infrastructure in support of metadata for footprint descriptions, i.e. spatial regions of coverage. The advancement in the MOC standards definition will be evaluated in the context of footprint representation for support of applications in development of data discovery. Future coordination between IVOA groups will involve potential development for spatial indexing in support of application integration with registry collections. Identifiers We plan to issue an essentially minor update of the Identifiers recommendation; it will remove the obsolte XML form of identifiers and add text on the recommended forms for standard and dataset identifiers. Registry Operations and Curation The Registry WG continues to drive regular discussions during Interoperability meetings about IVOA Registries operations, including curation of VO Resources and validation of DAL services. The Registry WG is pending effort to develop a coordinated plan accepted by TCG and proposed to the Exec for developing the procedures and processes of curating nonresponsive or non-valid registered DAL services. This also includes consistency in Registry managed authorities and harvested resource metadata across the IVOA registry network. The RWG also keeps monitoring the health of the registry system as a whole, working with the operators of the publishing registries to resolve problems as they crop up. -- MarkusDemleitner - 2014-06-26Semantics
The WG will continue to contribute to other WGs and IGs as appropriate.* Extension of the UCD List for planetary data:
There are requirements from the Planetary community to use and extend the IVOA UCD list for using these semantic tags while mapping their metadata.
Some concepts , current in Planetary science are missing .
An effort to propose new possible UCD strings and validate them in the IVOA UCD maintenance group is taking place .
Proposed list by May Interop 2014
Data Curation & Preservation
The IG will continue discussions on general topics of interest to the community. Some of the topics we expect to focus on during the next 12 months include:- The Unified Astronomy Thesaurus
- Review of current effort described online at http://astrothesaurus.org
- Participation of IVOA members in merging / mapping similar vocabularies (e.g. the Paris Observatory's Etymological Dictionary: http://dictionary.obspm.fr/)
- Drafting editors to support the development and review of the thesaurus
- The Research Data Alliance
- Follow developments of the RDA: http://rd-alliance.org/
- Participate in working groups on persistent identifiers and data citation
Knowledge Discovery in Databases
Theory
The main goal of the Theory I.G. is to propose an access protocol to discover and retrieve simulated data. From last InterOp, several implementations have been tested. We also had a few teleconferences with some members of the Theory I.G. Search for simulations Some technical difficulties have been identified. The most important of them, is that, contrary to an observationnal data, a simulated data may require several thousands quantities to be described in a way allowing us to answer our usecases. Because RDBMS are not efficient with tables with more than thousands columns, the access protocol for theoretical data should not rely on specific technologies (as TAP/RDBMS). During the second Theory session in Heidelberg InterOp we will discuss this important topic. A possible solution, would be to define a few APIs with a main one, the SEARCH/CUTOUT function, to discover interesting simulations. The query language would be a subset of ADQL. On the other hand, the exchange format will have to be defined. We will discuss the oportunity to use the serialization of SimDM in VO-Tables following the proposal of the utype tiger team. Extraction of data Concerning the extraction of data, the situation is more simple. Services will provide a Datalink document to describe the possible operations on data. This will be presented and discussed in the common DAL / Theory session about datalink. Goals of this InterOp for the Theory I.G. are : 1) conclude on a solution to discover simulations that is independent on the technology used to store metadata, 2) define the large picture about the exchange format document in the communication client / server during the search phase of simulations, 3) advance on the syntax of datalink document. Status after mai InterOp SimDAL has been presented by David Languignon at the may InterOp at ESAC and has been discussed during two sessions. A general agreement has been found on what has been presented : http://wiki.ivoa.net/internal/IVOA/InterOpMay2014Theory/InterOpESAC14_Theory_SimDAL.pdf Now, planning is : 1) discuss with Registry W.G. to identify possible overlaps / complementarities between SimDAL repository and registries 2) try to prepare a first version of a WD for next InterOp on some parts of the SimDAL standard (depends on the available time of David Languignon) 3) continue implementations in France, Spain, Italy on various services.Education
Edu IG prompted the development of VESPA (VO Educational Service Publisher and Archive), a web procedure offering managers of educational telescopes the possibility of an easy publication of their data in the VO. At the moment VESPA is an advanced prototype which we plan to develop into an official first version before the next interop meeting. In order to do so we need to test the application with a real interested user. The user has already been identified, but the user requires VESPA to implement SSAP beside the already implemented SIAP. A related task for Edu IG is to propose solutions and desirables for the curation of educational resources inside or along standard registries. Most of the relevant aspects of this task are the subject of a Note available in the Edu IG page Ideas. Education is a field where the localization of documents in different languages is essential. There is no immediate solution to this problem but we plan to start developing ideas and strategies in order to tackle it. The already mentioned Note on Educational Resources already considers the existence of multilingual documents and their hierarchy. Edu IG continues to collect desiderata and ideas from educators and the general public and to propose them to VO, possibly leading to an even wider diffusion of VO resources toward communities other than professional astronomers. -- MassimoRamellaINAF - 2014-05-18Time Domain
A note endorsing the use of SimpleTimeSeries for representing time series data within the context of the VO was issued in May 2014. When the new version of the Document Standards defines the appropriate procedure, this note will be put forward for approval by the TCG. An advanced working draft of VOEventRegExt was issued in May 2014. At the May 2014 InterOp, the TDIG agreed to advance this to PR status (as soon as a few minor formatting issues are resolved). We have now received additional feedback from Registry WG members, and are considering a further revision of this document. A PR is expected to follow shortly. An advanced draft of the VOEvent Transport Protocol was issued in May 2014. This draft will undergo a further round of discussion within the TDIG before advancing to PR. We expect it to be ready for PR mid-2014. -- JohnSwinbank - 2014-06-16Standard and Processes
A new version of the Document Standards is foreseen in 2014 to:- implement a new kind of Notes which will be endorsed by the TCG following a procedure that will be defined in the document
- implement the possibility of publishing Errata, and the procedure which should be followed
Science Priorities
Science priority areas are: Multi-dimensional data, and Time Domain Astronomy. Focus sessions at the May 2013 interop meeting will engage projects and surveys that produce and use multi-dimensional data (including radio astronomy, IFU, high energy and simulation data) and Time Domain data (time series, light curves, transient event reports) in a discussion on the role of VO technologies in these areas. Multi-d and Time Series use cases are being collected as part of this effort.IVOA Roadmap for 2014B
This outlines the roadmap for development activities by the various IVOA working and interest groups in 2014 between the Banff and Sesto Interops.Applications
3 VOTable actions :- Unicode issue : Get a consensus (mailing list should be appropriate)
- VODML serialization : Collaborate with DM WG to propose a VOTable schema update to support VODML serialization based on new XML tags (proposal 4).
- Coordinate specifications : Clean the current bad situation by an announcement on the mailing list of the proposal 2+4 : un-deprecate COOSYS + Introduce VODML when it will be ready(deprecate STC note))
Data Access Layer
DataLink TCG review period finished. Last PR version in completion (2015/02/05). Almost ready for recommendation. SIAV2 A lot of usefull comments during TCG review. Current work on document, interface implementation and validation (2015/02/05) AccessData New draft taking into account Calgary/Banff compromise in progress (2015/02) TAP Draft for TAP1.0 errata available since December 2014 TAP1.1 draft in progress (2015/02) ADQL Draft for ADQL2.0 errata available since December 2014 ADQL2.1 draft in progress (2015/02)Data Model
Spectral DM 2.0 RFC period finished. General consensus for approval (Applications pending) of the document but better level of implementation pending to send document to Exec for formal approval. Maybe, Applications group could help on this (?) Cubes DM This effort includes two documents: Dataset DM and Cube DM. Full information, links to the documents, and history are provided in the ImageDM page. The documents are stable. Waiting for implementations to start formal RFC. VO-DML Stable version. RFC could start if the implementations are considered completeGrid and Web Sevices
UWS A new release of the UWS 1.1 working daft will be provided soon. At least two implementations of this release are starting. For UWS 1.1 a reasonable aim is to convert the document into a PR and to start the RFC around April 2015. VOSpace We are continuing the discussions by mail to find a consensus. The aim is to provide at least a new WD of the version 2.1 before the Sesto interop. We start also the writing of a document to help people to build a VOSpace. The work around VOSpace 3.0 is also going on, in parallel with the update to the version 2.1 SSO and authentication We start a new WD of the SSO document to take into account the evolutions (e.g. ). A note around Authorization (to clear its use in the IVOA) is also due for the Sesto interop.Registry
RegTAP We'll provide a 1.0 version for Exec approval in November. No major issues turned up. Identifiers We plan to issue an essentially minor (but major for technical reasons) update of the Identifiers recommendation; it will remove the obsolte XML form of identifiers and add text on the recommended forms for standard and dataset identifiers, together with a thorough treatment on the whys and hows. ETA is November. Registry Interface and RegTAP With RegTAP becoming REC, the Registry Interfaces document will need revision. There is a working draft on volute that retains the standard OAI protocol for harvesting from publishing registries, but it will remove the search capabilities and instead refer to standards like RegTAP and a planned keyword search interface. We'll migrate this to ivoatex and provide an internal working draft by December. Keyword Search The Paris group continues to develop their prototype for a keyword-based search interface to the Registry. We will work towards making an internal working draft laying out the interface. VOEventRegExt While the actual authoring a registry extension for VOEvents, VOEventRegExt, is in the hands of GWS, RWG will continue to work closely with the authors. Registry Metadata for Footprints The RWG will continue to have discussions and evolve the current standards infrastructure in support of metadata for footprint descriptions, i.e. spatial regions of coverage. The advancement in the MOC standards definition will be evaluated in the context of footprint representation for support of applications in development of data discovery. What we would eventually like to see is a facility for all-VO searches over footprints and other STC coverages within the Registry. Alternative Identifier Schemes For reasons lying in their design, IVORNs are not ideal for use in data citations. It would, however, be very desirable if with registration came citeability. We will therefore investigate ways in which registered services could be assigned DOIs in some controlled fashion. Parallel to this, we need to investigate ways in which alternative identifier schemes can be integrated into the Registry to allow managing the mapping between IVORNs and non-IVRON aliases. Registry Operations and Curation The RWG also keeps monitoring the health of the registry system as a whole, working with the operators of the publishing registries to resolve problems as they crop up; as resources permit, we also advise data providers on improving their registry records. And of course, the RWG is the framework in which the various operators of, in particular, searchable registries provide their services. -- MarkusDemleitner - 2014-06-26Semantics
The Planetary and Solar science communities proposed, together with the IVOA Semantics Working group, to extend the thesaurus of UCD strings. A list of new UCD terms has been discussed in the WG ( see WD at http://wiki.ivoa.net/internal/IVOA/PlanetaryUCD/WD-UCDlist-1.24-20140901.pdf ) . Tests are going on to validate the proposed kewwords. The CDS UCD assignating tool will be updated to support tagging of planetary data and solar data . Data for planetary and solar observations will be gathered in order to form an extensive test bench. Due to delays in the extension mechanism and tests, we have shifted the previous schedule to:- Draft to be updated for 1st April 2015
- Tests feb -march 2015
- Proposed rec planned for 1 June 2015 (?) and disccused at June Interop meeting.
Data Curation & Preservation
Knowledge Discovery in Databases
Theory
The main goal of the Theory I.G. is to finalized SimDAL A meeting has been organized in Paris in december 2014 with the participation of D. Languignon, H. Wozniak, C.-R. Para, M. Molinaro, M. Sanguillon, B. Debray, B. Godard, F. Le Petit. This meeting (in replacement of Banff Theory sessions) allowed to precise some technical points. At this meeting, a draft of SimDAL written by D. Languignon and F. Le Petit has been presented and discussed. Several milestones have been identified. The plans for the coming months are :- Discussion with DAL to avoid overlaps between Access Data and SimDAL [done]
- Take into account comments in the document
- Present a W.D. at the next InterOp.
Education
Time Domain
The SimpleTimeSeries Endorsed Note will be put to the RFC period following the Banff meeting. The VOEventRegExt document will be revised based on comments in the session and a new draft put out to the IG once all comments and concerns by the Registry group have been addressed. Similarly, the VOEvent Transport Protocol will be recirculated for final comments before being put up for RFC. The intent is that all three documents will be in RFC or ready for approval by the Sesto meeting. Members of TDIG will attend the Hotwired IV meeting in Santa Barbara next May and will report back to the IVOA in Sesto.Standard and Processes
Science Priorities
IVOA Roadmap for 2015A
This outlines the roadmap for development activities by the various IVOA working and interest groups in 2015 between the Sesto and Sydney Interops.Applications
VOTable 1.4 → awaiting VODML serialization decision before starting a new cycle→ call to read and comment the VODML mapping document JSAMP
→ Support for https → try and see if a few standard adjustements will be required HiPS
→ Study a new capability entry in the registry ?
Data Access Layer
SIA-2.0- Hopefully Recommendation before Sydney Interop rewritten
- WD rework during summer
- RFC by Sydney Interop
- WD before Sydney Interop
- RFC around end of 2015
- minor impact on TAPRegExt
- WD before Sydney Interop
- RFC around end of 2015
- links to UWS-1.1, ADQL-2.1
- impacts VOSI, TAPRegExt, DALI
- see Theory IG roadmap
- WD soon. RFC before end 2015
- see TD IG roadmap
- WD soon
Data Model
SpectralDM 2.0 Document stable, waiting for one pending implementation already provided Somes comments received fue to this implemenation. Discussions to be taken to either update the spec covering some of the suggestions or close SpectralDM 2.0 (and wait for recommendation) and include proposals into a possible SpectralDM 2.1 (draft for Sydney) ObsCore 1.1 Stable draft provided in Sesto VO/DML Document stable with enough implementations PR to be released after Sesto if agreement is obtained during Sesto (presentation of the status to be done during this interop) VO/DML mapping Working in a public draft in coordination with Apps. Not clear schedule (maybe PR for Sydney) SourceDM Initial discussions. First WD to be presented in Sydney CubeDM 1.0, DataSetDM 1.0 Parallel activities. Documents stable. Waiting to have implementations in coordination with SIAv2 DAL spec If no major problems are found before Sydney, maybe we could have a PR version in Sydney for both specs It makes use of a more or less stable version of STC 2.0 model STC 2.0 Preliminary stable UML provided. Spec still in a preliminary editorial phase. WD to be presented during interop in spring 2016Grid and Web Sevices
- VOSpace 2.1: RFC before Sydney
- UWS 1.1: RFC in July 2015
- SSO 2.0: RFC before Sydney
- New VOSI version:
- If there are no other additions besides the scalable tables rework, a VOSI 1.1 WD will be ready for Sydney
- If there are other additions, a VOSI 2.0 WD will commence but will not be ready for Sydney most likely
- IVOA Strong note on XML Schema versioning ready before Sydney
- Draft document on authorization before Sydney
- If time/resources allow: Note on "How to implement a VOSpace"
Registry
- Registering TAP tables (and other dataset-within-service-like things): We will prepare a note on the general principle ("add an -aux to the capability standard id and use a plain vr:Capability record") and work with DAL to have that included for TAP 1.1. Prototyping with VizieR should commence soon, prototype at the GAVO DC already exists.
- Registry searchable by spatial constraints: Development of a MOC-capable postgres extension will start.
- Registry Interfaces: Version 1.1 (on harvesting and being a well-behaved registry, with the old client interfaces removed) of the REC should be RFC-ready by Sydney
- Identifiers: Version 2 will go into RFC real soon now.
- Mapper between ivoid and DOI -- we'll explore, presumably via Datacite.
- Continued maintenance of Registry contents
- RofR is going to move from Urbana to Cambridge, MA.
Semantics
Data Curation & Preservation
Education
Knowledge Discovery in Databases
Operations
Theory
Simulation Data Access Layer SimDAL document as been approved by DAL as a DAL W.D. Actions up to the end of the year are : 1 - Take into account suggestions in the SimDAL WD and publish it on the dedicated IVOA webpage under DAL 2 - Implementation of (at least) two reference implementations 3 - development of a client / validatorTime Domain
VOEvent Transport Protocol There's agreement within the IG that this is broadly ready to move forward from Working Draft. Will need to be "sponsored" by a WG (likely DAL). VOEvent Registry Extension Revision required following comments. Pending manpower; hopefully addressed this year. VOEventContainer Currently on hold pending manpower and refined understanding of requirements. Time series representation Enrique Solano has demonstrated that neither of the two approaches currently suggested for representing time series in the VO -- SimpleTimeSeries and Spectral Data Model v2 -- address all potential use cases. Working towards a comprehensive solution is a priority. Pending manpower to make progress. Community engagement Time domain has been identified as a priority by the IVOA CSP. Community engagement is key to this, and will require support from the larger IVOA machinery.Standard and Processes
Science Priorities
IVOA Roadmap for 2015B
This outlines the roadmap for development activities by the various IVOA working and interest groups in 2015/16 between the Sydney and Cape Town Interops.Applications
HiPS - Hierarchical Progressive Survey* "promote" HiPS IVOA note as a Working draft
* Study a new capability entry in the registry ?
VOTable 1.4
* still awaiting VODML serialization decision before starting a new cycle JSAMP * Support for https → try and see if a few standard adjustements will be required
Data Access Layer
- TAP-1.1
- continue progressing the internal WD to full WD
- connections to DALI-1.1, TAPRegExt-1.1, VOSI-1.1, ADQL-2.1 ongoing revisions
- TAPRegExt-1.1
- WD ready since November 2015
- Recommendation track will reflect the TAP-1.1 one
- Registry WG effort
- ADQL-2.1
- continue progressing the internal WD to full WD
- ancillary wiki pages to promote DBMS back end support
- SimDAL-1.0
- endorsement from Theory-IG
- evaluate connections to DAL and Registry standards
- VTP-1.0
- PR before end march 2016
- SIAV-2.0
- Became a recommendation on december 23rd
- SODA-1.0
- Working draft published on December 21st
- Lively discussion on the DAL list
- Request for implementations and prototypes by DAL WG Chair
- DALI-1.1
- WD published on October 27th 2015
Data Model
ObsCore-1.1: New WD expected Jan. addressing final questions, should be PR ready. Targeting REC ready for Cape Town. VODML-1.0: Stable WD expected Jan. with 3 model implementations (STC2, DatasetMetadata, NDCube). Should be PR ready in Feb. Targeting REC ready for Cape Town. STC-2.0: Modeling efforts continue, mainly around vodml compliance. Will be focusing on getting Working Draft delivered for Feb 2016. The working group has been able to review the prototype model in DatasetMetadata, but the advent of an independent document may generate more feedback. Targeting PR ready by Cape Town. DatasetMetadata-1.0: Model WD is stable, with minor vodml compliance items to be resolved early 2016. The path from there depends on the level of feedback on the STC2 working draft. Options include proceding to PR with embedded prototype STC2, or removing that section and going forward with reference to STC2 WD. In either event, there is a question about what constitutes an acceptable implementation of this model in particular. We will be addressing this question early in 2016 in order to meet the target of PR readiness for Cape Town. NDCube-1.0: Similar to DatasetMetadata, this Working Draft has been fairly stable, with minor vodml compliance item to be resolved early in 2016. We expect some changes will be necessary as we work implementations between now and Cape Town. We have volunteers for producing examples, but may have trouble defining the serialization spec as these models fall between the 'old-style' and 'vo-dml' paradigms. Targeting PR ready for Cape Town. Spectral-2.0: Passed RFC period, TCG review held pending DM Chair recommendation. SourceDM-1.0: Status was presented in Sydney, sparking some good debate. Pre-WD efforts should continue, with update provided in Cape Town. At present, there is no targeted status for the Cape Town interop.Grid and Web Sevices
- VOSI 1.1 - Implementation and Working Draft before the new year.
- UWS 1.1 to become a proposed recommendation in the next month.
- SSO 2.0 to become a proposed recommendation early 2016
- VOSpace 2.1 working draft at least one month before Cape Town.
- XML Schema Versioning Policy to become endorsed note before Cape Town.
Registry
Registering TAP tables (and other dataset-within-service-like things): While GloTS works wonderfully for TAP in TOPCAT, we will nevertheless publish a note on the auxiliary capabilities, as that will provide more uniformity in discovery as well as metadata richness. We will then try to convince the major TAP providers to include the new capabilities in their records. Client support is already there in TOPCAT. If takeup is sufficient, we would go for a WD with this by Cape Town. Registry searchable by spatial constraints: We will continue to lay the software foundations in pgSphere (i.e., indexable MOCs). The existing footprintURL mechanism to distribute footprints will be good enough for prototyping. Registry Interfaces: Work on version 1.1 (on harvesting and being a well-behaved registry, with the old client interfaces removed) will resume once the new RofR operators (CfA) have some experience in operating it so we can properly assess what RofR material should be in RI 1.1 Identifiers: Version 2 is in TCG review and should pass the Exec before the end of 2015. DOI work: We will continue to talk with Datacite to evaluate requirements to keep/bring the datacite metadata and VOResource in sync in the common universe of discourse. This will probably lead to early drafts of VOResource 1.1 by Cape Town. TAPRegExt: 1.1 will go to a public WD soon -- the main question there is whether to factor out the parts relevant to UWS and have a separate capability for standalone UWS services. Work on Registry extensions for VOEvent services, HiPS, or VOSpace will continue or start depending on input by the respective WGs. We will continue maintenance of the Registry system and contents, as well as our validation operations.Semantics
Vocabularies / Thesauri / Ontologies
Coordinate and/or influence the development of knowledge bases which support IVOA standards and curation initiatives, in particular- The Unified Astronomy Thesaurus (UAT)
- Taxonomy of Facilities / Telescopes / Instruments
- UCDs : update for a better coverage on Planetary data
Liaise with DC&P
Provide a forum for the joint discussion of standards development outside of the IVOA which influence curation and preservation activities, in particular:- Semantics terms defined in the DAL specification for DataLink
- The Research Data Alliance
Data Curation & Preservation
Among the current topics of interest :- Digital Object Identifiers (DOIs)
- Following the activities of the Research Data Alliance to identify topics and outcomes of interest for the IVOA community and encourage people to join the RDA activities in domains where the IVOA communities can share expertise and lessons learnt.
Education
Important points for EduIG are:- citizen science as a learning experience for volunteers
- multilinguality
- publication into VO of data of educational observatories (e.g. VAPE)
- registry and repository for educational resources
- development and sharing of materials and practices to be used in schools
Knowledge Discovery in Databases
The main focus of the KDD interest group will be on redefining its charta. Main questions for 2016 are:- how to ensure a simple data access required for machine learning algorithms
- how to deal with structural/infrastructural challenges when storing and exchanging code
- how to standardize the execution of code close to the data
- how to provide explorative support and data visualization
Operations
Key areas for the operations interest group for early 2016 include:- the development of common framework for evaluating the consistency of VO services with the standards they purport to implement. This includes a common definition of the requirements for VO services and a consistent measurement of the comliance to these requirements.
- ensuring that the recently agreed upon procedures for ridding the registries of obsolete services are working
- minimizing the downtime for VO services
Theory
Main goal of Theory I.G. is to build the reference implementations for SimDAL and finalize the standard. The roadmap is the following- Build two implementations of the three parts of SimDAL
- Build a client / validator
- Update the SimDAL document
- Present these implementations at Cape Town InterOp.
Time Domain
Working with DAL to bring VOEvent Transport Protocol to Recommendation status. Still expecting engagement from CSP, Exec etc re plans for community engagement since the time domain has been recognized as a priority area for the VO.Standard and Processes
Working on an updated version of the Document Standards, expected for 2016:- Implementation of Errata and Endorsed Notes
- Implementation of the RFC process as discussed during the Sydney Interop, with a 6 week RFC period including both Community and TCG comments.
Science Priorities
The renewed CSP Terms of Reference support the stronger emphasis in IVOA on engagement with the large astronomy projects, and on guiding the priority area developments via use cases and requirements. In this period the CSP has lead the organisation of the “Interoperability of data from major astronomy projects” at the May 2016 IVOA Interoperability Meeting.IVOA Roadmap for 2016A
This outlines the roadmap for development activities by the various IVOA working and interest groups in 2016 between the Cape Town and Trieste Interops.Applications
- HiPS 1.0 WD
- VOTable
- Awaiting DM WG proposal about VO-DML serialization
Data Access Layer
- TAP-1.1
- WD published on April 28th 2016.
- waiting for DALI PR to go to PR.
- TAPRegExt-1.1
- WD ready since November 2015
- Recommendation track will reflect the TAP-1.1 one
- Registry WG effort
- ADQL-2.1
- WD published on May the 2nd 2016.
- Waiting for DALI PR to go to PR.
- ancillary wiki pages to promote DBMS back end support
- SimDAL-1.0
- Reference implementations
- PR since August 24th 2016. RFC comments
- VTP-1.0
- PR published on May the 6th 2016. Lack of RFC comments
- SIAV-2.0
- Became a recommendation on december 23rd 2015. New implementations came out or are in preaparation
- SODA-1.0
- PR rewritten by editors and authors published Septembre 19th 2016. RFC start before TRieste
- DALI-1.1
- new WD published on April 15th 2016. Maturity close to PR. Xtypes discussions
Data Model
ObsCore-1.1:Addressing RFC comments, final update expected by end of June. Should be ready for TCG approval by end of July.
VODML-1.0: Cape Town discussions indicate the document is PR ready once a 'How To Get Started' style help document is completed. This document is scheduled for completion by end of June. PR period should start in July. Target REC by Trieste.
STC-2.0: WD feedback is strongly on the side of needing to split the model to a core package with secondary extensions for higher level packages (like Region and Area). Project scope to be reviewed Jun/Jul, with support of Dataset/NDCube models to be the focus. Need to schedule an external review panel to evaluate model and provide recommendations.
DatasetMetadata-1.0: Model is stable, with user implementations/serializations in both VODML Mapping and utype conventions. Focus will be on organizing external interoperable usage of the model in a SIAV2 thead. Target is REC ready by Trieste.
NDCube-1.0: Similar to DatasetMetadata, working draft is stable. User implementations to be packaged and uploaded to twiki. Focus will be on organizing external interoperable usage of the model in a SIAV2 thread. Target is REC ready by Trieste.
Provenance-1.0: The Dataset/Observation/Provenance models are coming toward a common framework. For Trieste, the goals will be to:
a) fanalize the vodml-compliant data model so that it works with only minor modifications for all our use cases.
b) deliver working draft model document; including minimum requirements; serialization strategy; and access to Provenance data.
c) work relations between SimDM and ProvDM
SourceDM-1.0: Team is to organize project goals and scope, summarized to twiki pages. Project should broaden use-case example set beyond Gaia. At present, there is no tageted status for Trieste.
Grid and Web Sevices
Single Sign-On Profile (SSO) 2.0: Minor updates to are needed on the PR to address feedback received in Cape Town. Specifically, a few more authentication types (authTypes) and their identifiers need to be mentioned and standardized. There should be a mention of the use of "groups" for authorization in the document. It was mentioned that an IVOA note with examples on how to use the various authMethods described in the SSO 2.0 specification would be useful. VOSpace 2.1: Feedback continues to come in from the authors. Once these has been addressed the document will go through the six-week review period before becoming a proposed recommendation. VO Support Interfaces (VOSI) 1.1: Minor changes to be made before this can be made a recommendation. These should be addressed before the meeting in Trieste. UWS 1.1: The document has been sent to the TCG chairs to executive review. The GWS working group is encouraging groups to publish interoperable implementations of the various standards. As per the recommendations from the Focus Sessions held in Cape Town, the GWS working group will work with the Knowledge Discovery Interest Group to examine use cases, technologies, and possible standards in regards to the question of "code to data".Registry
VOResource 1.1: A large number of minor issues are being fixed in this first overhaul of VOResource in 10 years. The main new feature is a capability to represent alternative identifiers (DOIs, ORCIDs, etc). Until Trieste, we want to reach consensus on the remaining open questions, with implementations, also in RegTAP following after Trieste. SimpleDALRegExt 1.1: The adaption of SimpleDALRegExt to new DAL protocols (in particular SIAP 2.0) should see PR soon and should be ready for REC by Trieste. Registry Interfaces 1.1: RFC soon [Theresa?] TAPRegExt 1.1: This will be kept in lockstep to the progress of the TAP/ADQL updates. Registering TAP tables: We are encouraging takeup of the Discovering Data Collections Note. Until Trieste, we in particular hope for more feedback from the CDS. Since they would like to link tables and (cone search) services, there is a (weak) link with VOResource 1.1. We believe such links could be done based on record-local tags. Spatial searches in the Registry: We will continue to lay the software foundations in pgSphere (i.e., indexable MOCs). The existing footprintURL mechanism to distribute footprints will be good enough for prototyping. Standards records: We would like to finally have StandardsRegExt records for all IVOA recommendation, and we will work with the WG chairs to have them generated. A page at http://wiki.ivoa.net/twiki/bin/view/IVOA/WriteAStandardsRecord has been set up to guide standards authors. We will continue maintenance of the Registry system and contents, as well as our validation operations.Semantics
Vocabularies / Thesauri / Ontologies
In continuation of previous roadmap: Coordinate and/or influence the development of knowledge bases which support IVOA standards and curation initiatives, in particular- The Unified Astronomy Thesaurus (UAT)
- Taxonomy of Facilities / Telescopes / Instruments
- UCDs : update for a better coverage on Planetary data, revise assignation tools accordingly
- Provide support for externalizing controlled list and vocabularies to be used in IVOA standards (e.g VOResource, DataLink)
- Reach consensus and document process through which Vocabularies are curated and published
Liaise with DC&P
Provide a forum for the joint discussion of standards development outside of the IVOA which influence curation and preservation activities, in particular:- DataCite
- The Research Data Alliance
Data Curation & Preservation
The IVOA Data Curation and Preservation Interest Group continues to be the place to discuss topics such as data citation, DOIs, Certification, and to pass information about the activities of the Research Data Alliance relevant to the astronomical community.Education
The Education interest group is working on a repository for documents related to education together with its implications for the Registry. On a different line, the Education interest group is considering the possible benefits for the public of citizen science projects, in particular their use as pedagogical projects aimed at the discovery of astronomy in classrooms.Knowledge Discovery in Databases
The Knowledge Discovery group keeps on working on redefining its charta. Main questions for 2016 are:- how to ensure a simple data access required for machine learning algorithms -> HiPS as a possible access tool for spacially structured data
- how to deal with structural/infrastructural challenges when storing and exchanging code
- how to standardize the execution of code close to the data
- how to provide explorative support and data visualization
- perhaps, providing a registry of codes and tools?
Operations
Provide feedback on the operational implementation of VO standards including the level to which operational implementations pass validation services. Work to promote the development and use of validation services for VO standards. Continue to work to unify the reporting of validation errors.Theory
SimDAL 1.0- RFC process.
Time Domain
Immediately following the Stellenbosch InterOp, VOEvent Transport Protocol 2.0 moved to RFC. We expect to devote the bulk of effort in this period to responding comments and, hopefully, pushing VTP towards full recommendation status. Based on discussions at Stellenbosch, we will continue informally exploring prospects for time series representation within the VO. We will seek to engage with the large facilities represented at the Stellenbosh "focus sessions", and explore how we can best help address their needs. As and when the promised (Ciardi's talk) LSSTC funded alert system study takes place, we will actively engage with this effort.Standard and Processes
An update of the IVOA Standard Documents, initially proposed a V1.2, is now discussed a V2.0. It implements the Endorsed Note path and Errata for IVOA documents. The evolution was discussed in Stelleboch in the TCG meeting. RFC before Trieste meeting with the aim to conclude the new version before the end of the year.Science Priorities
Following the May 2016 focus sessions on “Interoperability of data from major astronomy projects” the CSP plans to collect more structured feedback from large projects in the form of a survey. The priority areas remain Multi-dimensional data and Time domain astronomy. The final stages of the development of standards to meet the minimal requirements for multi-d data access are to be followed closely, and their completion will be an important milestone.<!--
* Set ALLOWTOPICRENAME = TWikiAdminGroup
-->
IVOA Roadmap for 2016B
This outlines the roadmap for development activities by the various IVOA working and interest groups in 2016/17 between the Trieste and Shanghai Interops.Applications
- HiPS standards => from WD to PR
- VOTable 1.3 Errata (COOSYS)
- Awaiting VODML mapping in VOTable proposal (from DM WG)
Data Access Layer
- TAP-1.1
- new WD according to Trieste decisions preapared for PR(will come after Shanghaï)
- ADQL-2.1
- BNF and PEG grammar discussion
- UDF and coordinate system changes
- New 2.1 WD version in preparation for RFC
- SimDAL-1.0
- REC since march 20th
- VTP-1.0
- REC since march 20th
- SODA-1.0
- Last modification from RFC. PR pushed to the Exec in Shaghaï
- DALI-1.1
- Last modifications during RFC . PR pushed to the Exec in Shanghaï
Data Model
ObsCore-1.1- in RFC, should be REC by Shanghai
- standard
- is in RFC, expect some iteration.
- goal is to be in TCG review by Shanghai
- mapping
- evaluating new annotation syntax,
- goal is to have a new WD, perhaps in PR by Shanghai
- restructure and minimize content to NDCube use-case requirements
- goal is to have a generally accepted working draft, implemented in NDCube/Dataset models and ready for RFC by Shanghai
- keep updated to STC amd VO-DML developments
- implement examples in VOTable
- work as use-case for VO-DML mapping implementation
- goal is to have fully vetted/validated examples by Shanghai, with the documents ready for RFC pending completion/readiness of dependent models.
- goals
- publish Working Draft on IVOA Documents webpage
- VO-DML compliant representation of the model
- work out relations between SimDM and ProvenanceDM
- new section with expanded implementation notes for use cases (with used classes and attributes)
- regroup to get project working toward a working draft.
Grid and Web Sevices
During the interop in Trieste, the Grid and Web Services working group came to an agreement to support a standard means of authorization through group membership. Such a standard will enable interoperable authorization among IVOA services. It is to be determined whether a note documenting the endorsement and use of an external stardard, or an IVOA standard itself is the best way to incorporate group authorization into the IVOA. A draft of such a proposal will be presented in Shanghai. A new VOSpace 2.1 Working Draft to be ready before Shanghai. As per the discussions in the first GWS Trieste session, it will include mention of the Public Share Protocol. This protocol may be use to obtain long-lived URLs for publishing papers and DOIs. "Code to data" was a topic of much discussion in Trieste. The GWS working group would like to pair with the DAL working group to work on a simple IVOA approach on processing data using client submitted code. Grid and Web Services Standards statuses:- The SSO 2.0 final Proposed Recommendation will go to Exec by the end of 2016.
- The VOSI 1.1 final Proposed Recommendation will go to Exec by the end of 2016.
Registry
VOResource 1.1: There are still several open questions (e.g., representation of alternate identifier schemes) that we want to discuss on the mailing list before issuing a PR, together with a preliminary mapping into a RegTAP service (which would start as a proprietary extension before putting this extension on the standards trail). We will try to encourage more data providers to add DOIs or ORCIDs to their registry records. RFC could then start after Shanghai. Registry Interfaces 1.1 is essentially pending a clarification for having standalone registry records for RegTAP-based searchable registries. We expect to work this point out very soon. It will, however, depend on the Discovering Data Collections Note. We will, therefore, probably synchronise PR of RI 1.1 with the endorsement process of this Note. That, in turn, we want to start in November 2016. Standards records: We will continue to work with the chairs of the other IVOA working groups to ensure that they register their standards as envisioned by StandardsRegExt (cf. WriteAStandardsRecord). TAPRegExt 1.1 will be kept in lockstep with TAP 1.1; since the latter is expected to go PR this year, so will TAPRegExt 1.1. SimpleDALRegExt 1.1 has been fully reviewed in Trieste and should be approved by the Exec later this year.Semantics
Data Curation & Preservation
The Data Curation & Preservation IG will continue to follow relevant topics, in particular the status and activities of the Research Data Alliance and the implementation of DOIs.Education
Knowledge Discovery in Databases
Operations
The Operations Group continues to work to ensure effective operations of VO services. Two developments in the underlying network infrastructure may need to be addressed operationally. Some sites in the United States have been required to transfer to using secure HTTP (i.e., HTTPS) and this can invalidate existing entries. Since the use of HTTPS is expected to spread concerns and experiences will be addressed in Shanghai and future interops. The use of IP/V6 internet addresses is also becoming more widespread but there has been little investigation of the consquences of this new protocol. The results from validations and monitoring of VO service will continue to be published at web sites and discussed at interoperability meetings and the group will continue to work to circulate operational best practices and lessons learned from large and small VO providers.Theory
Time Domain
Standard and Processes
The aim is to finalize Document Standards V2.0 at the Shanghai Interoperability meeting at the latest. The RFC, including the request for comments from the TCG, was initially open 24 September 2016 - 5 November 2016.Science Priorities
Current list of Science Priorities being followed up are:- Time Series, time-domain astronomy
- Multi-dimensional data
- Multi-messenger astronomy
* Set ALLOWTOPICRENAME = TWikiAdminGroup
-->
IVOA Roadmap for 2017A
This outlines the roadmap for development activities by the various IVOA working and interest groups in 2017 between the Shanghai and Santiago Interops.Applications
- HiPS
- VOTable
- Usage: Encourage the App members to apply Erratum 1.0 (re-introduce coordinate description asap)
- Standard: Awaiting VODML VOTable serialization consensus (see DM roadmap)
- Authentication
- Continue to discuss on the requirements and impacts on App clients (ex: Aladin VOSpace access...)
Data Access Layer
SCS 1.1 :- internal WD for beginning of summer (M.Molinaro)
- internal WD for end of September (N.Moreau)
- Summary (F.Bonnarel) ,
- Decision in Santiago, implementation or endorsed note
- ADQL 2.1 WD for July and PR by September (D.Morris)
- ADQL PEG grammar in ADQL 2.2 draft later (W.Landry, D.Morris)
- Summary on the DAL future page (G.Mantelet), now
- MOC prototyping wiki collaboration (D.Morris), now
- Create a Task group with TDIG and DM for requirements and design
- Encourage prototypes development
- First conclusions by Santiago
- PR by end of June.
Data Model
VO-DML-1.0- REC by end of summer
- WD distributed to working group by early June
- work example serializations in application, come to working group concensus on syntax, and elements included in 1.0 version
- PR ready by September
- Keep updated to any changes in STC or VO-DML
- update examples in support of mapping evaluation effort
- add validation to model capability to example generation script
- otherwise.. considered ready for PR/RFC upon completion of STC
- evaluate models using cube example set and mapping syntax
- fold in working group comments
- goal is to have a generally accepted working draft, implemented in NDCube/Dataset models and ready for RFC by Santiago
- generate vo-dml compliant model
- no specified goals at this point
- continue work with TDIG and DAL to gather requirements for supporting time domain data.
Grid and Web Services
Priority 1:- Remote Execution (Code to Data) -- The GWS Working Group will make an effort to make progress around the idea of executing code near to the data. We are aiming to have prototypes ready for Santiago. These will allow for a working draft to be developed.
- Recommendation for Federated Authentication -- An IVOA recommendation on how to integrate with federated authentications systems and identity providers.
- Group Membership Service Working Draft -- Continue with the momentum from Shanghai on the mailing list and have a working draft ready by Santiago. Most of the key concepts have been settled.
- VOSpace 2.1 – RFC period will be extended to allow for more feedback. It is uncertain at this point whether this version will need more work.
- Note on how to implement a very simple VOSpace
- Note on pattern of making VO Service calls
- Note giving examples on how to use the SecurityMethods described in SSO 2.0
Registry
Registry Interfaces 1.1, which in particular lets RegTAP operators register their services as registries (registration as TAP services with RegTAP data in them, of course, can continue). That, however, builds on the Discovering Dependent Resources note, so that would probably have to be endorsed before RI 1.1 can actually become REC. VOResource 1.1: Should go to RFC in the next few weeks. We are still soliciting takeup of the new features. To make this more attractive, the RofR will investigate installing the new schema so registries already using new features do not become invalid. RegTAP 1.1: This puts new VOResource 1.1 features into RegTAP. We have first implementations of a proposed extension of the schema. A working draft should be published July-ish. Discovering Dependent Resources: This is a proposed endorsed note trying to solve the problem of properly discovering data collections that are published through other services, the prime use case so far being TAP tables. The central concept here are auxiliary capabilities, essentially saying "this data can be accessed via TAP/SIAP/whatever in a services with the following access URL". There is some takeup of the proposed scheme, and TOPCAT already supports discovery through it. To make it a valid replacement for current workarounds (i.e., GloTS), more takeup is necessary; various registry operators have pledged to work towards it. Standards records: We will continue to work with the chairs of the other IVOA working groups to ensure that they register their standards as envisioned by StandardsRegExt (cf. WriteAStandardsRecord). TAPRegExt 1.1 will be kept in lockstep with TAP 1.1. Maintenance: We will keep making sure the registry system runs; in particular, the anomalies diagnosed by the ESA registry have been diagnosed and, where there is a potential for later functionality problems, will be addressed in the coming months.Semantics
Data Curation & Preservation
The Data Curation \& Preservation IG will continue to be the place where liaison between the IVOA and the Research Data Alliance RDA is established, with regular discussion of RDA activities, their relevance to the IVOA, and the IVOA activities relevant to the RDA (eg Provenance). This is a standing item for the IG session at each IVOA Interoperability meeting. The DataCP IG also hosts discussions on topics such as DOI usage in astronomy.Education
Knowledge Discovery in Databases
We intent to build a task force that identifies key activities / challenges to focus on In addition important knowledge discovery uses cases should be identified and collected to be reported at the next interop meeting. If possible, prototype implementations of services should be developed and presented that cover some of the discussed topics at the Shanghai interop.Operations
Theory
Work in the "Implemementation Note" for SimDAL. We intend to describe several implementations, done by different groups, about different types of simulations. It is expected to have the note finished by Spring, 2018.Time Domain
Top priority for TDIG is the support the ongoing effort in the DAL and
DM working groups to standardize a data model and discovery & access
mechanisms for time series data within the context of the VO. TDIG
will focus on developing and curating scientific use cases and
engaging with the working groups to assist in developing and
evaluating prototype solutions.
In addition, we will begin to draft a modest revision to the VOEvent
to address non-urgent issues which have arisen over the six years
since VOEvent 2.0 was standardized. This will include clearer
definition of the semantics of the IVOID/IVORN in the context of
VOEvent and more powerful and expressive description of space-time
coordinates (e.g. by integrating STC2).
Finally, we will continue to work with the wider astronomical
community, and in particular major facilities, to assess our longer
term goal of evolving the VOEvent standard and associated
infrastructure to address the needs of future high-volume event
producers.
Standard and Processes
Follow-up of the Document Standards V2.0: real-life adjustment of the standardisation, endorsed notes and errata processes.Science Priorities
IVOA Roadmap for 2017B
This outlines the roadmap for development activities by the various IVOA working and interest groups in 2017/2018 between the Santiago and Victoria Interops.Applications
- HiPS
- VOTable
- Erratum 2: "arraysize=1" : discussion, redaction
- Standard: Awaiting VODML VOTable serialization consensus (see DM roadmap)
Data Access Layer
- TAP 1.1
- New PR by end of November
- RFC before Christmas
- REC possibly before Spring interop
- ADQL 2.1
- New version of the WD in November
- PR should follow directly from it
- Datatype Consistency check among ADQL, TAP, DALI
- Find a volunteers team ASAP
- SCS 1.1
- lets' go on with discussion of WD in end 2017, early 2018
- New WD in early 2018, if enough participated, PR after Spring interop
- SLAP 1.1
- Publish the current WD
- Revisit the DALI compliancy and upgrade rationale for a new WD in early 2018
- PROVDAL 1.0
- WD end 2017/early 2018
- PR before spring interop
- PROVTAP 1.0
- WD end 2017/early 2018
- reference implementation to be worked out for Spring interop
- DataLink discovery modifications
- DAL chair note to be published early december
- Endorsed note early 2018 ?
- DAL multi-D protocols (ObsCore, SIAV2, DataLink, SODA) upgrades or extensions : generic and TimeSeries-oriented
- DAL chair and co-chair note end 2017
- Follow-up to be coordinated with TDIG and DM WG.
- Errors detected by Validators in DAL services .
- DAL chair and co chair to revisit this with Ops and validator operators and report at spring TCG teleconf
Data Model
Grid and Web Services
Below is the 6-month roadmap for the Grid and Web Services Working Group after a successful and productive interop meeting in Santiago. The work related to UWS is the outcome of a resolution of a Registry WG issue of discovering services with multiple interfaces.- VOSpace 2.1
- To adopt new UWS sync and async types in interfaces for transfers
- Make clear in the standard capabilities the use of VOSpace standardIDs
- Add standard registry records (2.0 & 2.1)
- Goal: RFC / TCG approval before 2018
- New UWS standard registry records for interface types (uws:sync, uws:async)
- Complete KDD/GWS Machine Learning experiment for code-to-data
- more needed! (see below)
- Group Membership Service Working Draft
- Contributors needed for:
- Interoperable code-to-data prototypes
- Recommendation for Federated Authentication — An IVOA recommendation on how to integrate with federated authentication systems
- Note giving examples on how to use the SecurityMethods described in SSO 2.1
- Into the VO: Probabilistic Kernel Density Classification?
- If time permits:
- Note on the proper pattern of making VO Service calls
Registry
- Registry Interfaces 1.1, which in particular lets RegTAP operators register their services as registries, is ready for exec review now and should become REC within 2017.
- VOResource 1.1 (among other things, enabling DOI usage in the Registry): Has been in RFC since June; we will include some new changes to securityMethod (now 1:1 mapping to interface and allowing extensions through other schemas), so REC will probably have to wait until early 2018. Meanwhile, we will encourage takeup of the new features and terms among the authors of resource records.
- RegTAP 1.1, putting the new VOResource 1.1 features into RegTAP, will see a PR right after VOResource 1.1 becomes REC; prototype implementations already exist at GAVO and partially at ESAVO. The main change will forseeably be support for services requiring authentication; that will, in particular, require changes in common client queries to RegTAP services once these become more common.
- STC in the Registry: We will continue to roll out prototypes using the current footprint URI mechanism and, based on those, try a simple extension scheme for coverages on the axes in ObsCore. Whether or not this will be included in the VODataService schema or we will provide a standalone REC remains to be seen.
- Discovering Dependent Resources: This is a proposed endorsed note trying to solve the problem of properly discovering data collections that are published through other services, the prime use case so far being TAP tables. We still solicit takeup by operators of major data centers, with VizieR expected to publish complying resource records soon (cf. HowToEnableTableDiscovery). This also will help requirements and recommendations coming from Apps ("Aladin 10").
- Standards records: We will continue to work with the chairs of the other IVOA working groups to ensure that they register their standards as envisioned by StandardsRegExt (cf. WriteAStandardsRecord).
- TAPRegExt 1.1 will not be a dependency of TAP 1.1 any more, per agreements in Santiago. We will therefore probably delay a new PR until possible further requirements from ADQL materialise.
- Interest in developing a resource extension for VOEvent components has resurfaced; we will work towards having a revised working draft by Victoria (May 2018).
- Maintenance: We will keep making sure the registry system runs; in particular, we will address the remaining serious anomalies diagnosed by ESAVO and the grievances brought forth by the solar system IG.
Semantics
Data Curation & Preservation
The Data Curation \& Preservation IG will continue to be the place where liaison between the IVOA and the Research Data Alliance RDA is established, with regular discussion of RDA activities, their relevance to the IVOA, and the IVOA activities relevant to the RDA (eg Provenance). This is a standing item for the IG session at each IVOA Interoperability meeting. The DataCP IG also hosts discussions on topics such as DOI usage in astronomy.Education
Knowledge Discovery
Operations
Monitoring activities at VO-Paris, ESA and the HEASARC will continue, though efforts will be somewhat lessened at the HEASARC. A monthly mailing giving a summary of the validation and operational state of the VO will be started. The Ops group will encourage the development of validation capabilities in the DataLink and VOEvent. The implementation of standalone validators for the simple DAL services would be valuable to developers who currently must deploy a service before it can be validated by the existing web based frameworks. The Ops group will also continue to explore if and how we might extract and deseminate usage information to understand which protocols are enabling our communities. Understanding the differences in usage patterns may be very helpful in guiding both the development and deployment of standards.Theory
Time Domain
Standard and Processes
Science Priorities
IVOA Roadmap for 2018A
This outlines the roadmap for development activities by the various IVOA working and interest groups in 2018 between the Victoria and College Park Interops.Applications WG
Python- Engage with Astropy community
- Possible Astroquery VO module
- Work towards VODML-Mapping consesnsus.
- Determine if there are other driving factors for a new VOTable version.
- Encourage application support for client authentication, working with Grid and Web Services WG to outline the scope and details.
- Continue to encourage HiPS providers to declare their services in the VO registry according to the HiPS Standard (section 5.3).
Data Access Layer WG
(roadmap updated after 2018.07.05 TCG TConf)- Expected progress by Fall 2018 Interop (College Park)
- TAP-1.1: new PR with RFC extension, waiting UWSRegExt Note to be published
- ADQL-2.1: pending 1 missing implementation, new PR and RFC
- SLAP-2.0: missing SSLDM revision (source retrieval issues), otherwise near to PR
- DALI-1.2: gathering input at DALI-1.1-Next
- polygon issue (see mail thread) needs to be solved
- Time Series discovery and access Note to cover TDIG DAL roadmap
- Additional efforts to follow through and beyond next Interop
- ObsLocTAP & ObsVisSAP (were OLAP & OVAP) Observation Locator and Object Visibility protocols: WDs
- ProvTAP & ProvSAP Provenance access protocols: WDs
- SIA-2.1 consolidate feedback, possible revision start
- DataLink-1.1 consolidate feedback, possible revision start
- SIA & DataLink feedback will be gathered in a Note (by end of Summer)
- Still on the roadmap, but waiting for more feedback
- SODA
Data Model WG
Fairly packed roadmap as we try to close up current projects and start up the next phase. Current model efforts:- VO-DML
- REC approved in Victoria
- upload final REC version of documents ( July 2018 )
- upload new registry records for document and IVOA data model ( July 2018 )
- REC approved in Victoria
- "STC" component models (coords, meas, trans)
- document current versions of models; most content to be extracted from last model drop by Arnold ( July 2018 )
- fold in 'missing' elements identified in Victoria
- coords - SpaceReferenceFrame to enumeration to vocabulary to better support SSIG domain ( Aug 2018 )
- trans - restore transform operations needed by LSST, implemented by AST library ( Aug 2018 )
- coords/meas - restore basic Velocity ( Aug 2018 ); discuss/add 'ProperMotion', 'RadialVelocity', 'UpperLimit' ( Aug+ 2018 )
- with concensus on model content, move to PR/RFC process ( Sep 2018 )
- Dataset Metadata
- no contention, ready to move forward when measurement model dependency is satisfied. ( Sep 2018 )
- NDCube
- no open issues, ready to move forward when dependencies are satisfied... most likely AFTER "STC" and Dataset are completed ( Oct+ 2018 )
- Provenance
- resolve open questions; get concensus on content; migrate to RFC process ( Oct 2018 )
- TimeSeries
- continue working with TDIG to flesh out requirements for time domain data models and prototype/develop models as time and resources permit
- PhotDMv1-1
- Generate VO-DML compliant model; it is not yet clear if this is a major version update or minor.. the original model had gone far in this direction.
- preliminary work has been done in Volute in preparation for Victoria demos
- pick up from there with 'official' model project; modeler=Jesus, domain=??, tech=??
- Generate VO-DML compliant model; it is not yet clear if this is a major version update or minor.. the original model had gone far in this direction.
- Characterisation
- Migrate Characterisation to a VO-DML compliant model; this is a major project, at least in part motivated by time domain project needs
- modeler=Francois/Mireille?, domain=??, tech=??
- SourceDM
Grid and Web Services WG
- Authentication and Authorization
- Work with Applications WG on client authentication support
- Completion of the Group Membership Service (GMS) working draft
- Prototypes and implementations of GMS
- TAP 1.1 VOSI capabilities support
- Publication of the UWSRegExt 1.0 Note and XSD?
- Schema versioning: work out issues with versioning attribute context
- Encourage working group to collaborate on Science Platform development
- Continue with GWS telecons
Registry WG
RegTAP 1.1, including new VOResource 1.1 features, can now go through RFC; prototype implementations exist at GAVO and partially at ESAVO, and are planned at MAST (USVAO). The main change is in describing services requiring authentication, which will eventually require changes in some client queries to RegTAP services if these become more common. (Authentication changes decided at Nov '18 interop, work can proceed, use case/examples will need to be edited in the RegTAP document.)STC in the Registry: Work in prototypes will continue based on a recent STC in the Registry note introducing the concept of MOCs in the registry. We will coordinate with DM on updates to or removal of existing STC registry extensions based on DM's standards reorganization and future compatibility. (Nov '18: ongoing. MOCs in the registry introduced, needs revisited next meeting, waiting on DM RFC)
Discovering Dependent Resources: A note in progress addresses the problem of discovering data collections that are published through other services, the prime use case so far being TAP tables. A path forward involving creating a new registry extension allowing both tables and capabilities was discussed in Victoria, this work will be continued and prototypes developed; work on the main document is postponed until this is resolved.
Standards records: We will continue to work with the chairs of the other IVOA working groups to ensure that they register their standards as envisioned by StandardsRegExt (cf. WriteAStandardsRecord). (Worked with Time Domain IG & DAL for revisiting VOEvent registration)
Maintenance: We will keep making sure the registry system runs; in particular, we will address anomalies diagnosed by ESAVO especially including inconsistencies in the Registry of Registries. (Validation run, operations meeting held, anomalies addressed with RofR and Euro-VO as an example, work ongoing)
Semantics WG
Maintenance of UCDs : Improve the revision loop proposed. Adjust the Maintenance for UCD WD with the lessons learned. By end June 2018 Work out the process for sharing the progress on UCD terms revision with the community. By end July 2018 Complete the revision of terms asked by the planetary science on the UCDList_1-5_RFM web page. Propose face to face meetings to speed up the process. By nov 2018 Visibility of IVOA vocabularies Setup of a vocabulary page at the top the http://ivoa.net/documents front page as collaboration effort with the media group and IVOA document coordinator. A prototype page gives an idea of this prototyped page http://wiki.ivoa.net/twiki/bin/view/IVOA/VocabularyTestPage Organisation of the Vocabulary repository By nov 2018 Nomenclature for names for Observatories, instruments, missions. Collaboration Baptiste Cecconi (Lesia, Paris) - University of Graz - Fuzzy logic tools applied to instruments telescopes list cross match . Description of Space missions from Vespa General thesaurus building to be continued with ADS, AAS, astronomical journals, librarians and documentalists involved. Design of a metadata description based on the reuse of the PDS 4 Information model to categorize the various items. By nov 2018Data Curation & Preservation IG
The Data Curation & Preservation IG will continue to be the place where liaison between the IVOA and the Research Data Alliance RDA is established, with regular discussion of RDA activities, their relevance to the IVOA, and the IVOA activities relevant to the RDA (e.g. Provenance). This is a standing item for the IG session at each IVOA Interoperability meeting. The IG also hosts discussions on other topics of interest for Data Curation & Preservation. DOIs have been one of the important topics addressed in the past. It is proposed to have it as a topic again for the next DataCP meeting in November 2018, to discuss requirements, implementation, lessons learnt, granularity, etc.Education IG
- EduIG and Media Session at the Victoria Interop
- Several talks were given by Giulia Iafrate and Chenzhou Cui on VO Education at the XXXth IAU GA FM14 and BMs.
- Tutorial at the 2018 ADASS on "A comprehensive use case scenario of VO standards and protocols"
- The propolsal was accepted by ADASS POC - updated by Chenzhou on Sep. 12, 2018
Knowledge Discovery IG
Operations IG
Operations will continue its normal activities:- monitor VO services, expect to present weather reports at the next Interop
- monitor availability of validator software for existing and emerging VO standards, as summarised on the IvoaValidatorsSummary wiki page
Solar System IG
VOEvent: prototyping for planetary and space-weather sciences.- Finalizing VOEvent services in Planetary Sciences and Space Weather
- Serving VOEvent catalogs for a posteriori studies, using EPN-TAP
- review of the EPNcore currently used to adjust the model before proposing an IVOA standard.
- interoperability with NASA/PDS information models and local data dictionaries
- Review of solar and planetary reference frames, as a list (with synonyms)
- Review of new services
Theory IG
Work in the "Implemementation Note" for SimDAL. We intend to describe several implementations, done by different groups, about different types of simulations.Time Domain IG
AlertsTime domain IG members working with ZTF and LSST to develop brokers and filters: Lessons learned from ZTF and LSST will inform future developments of VOEvent Feedback from SSIG, investigate and report on how to proceed on VOEvent to meet their requirements Time Series: DAL: TimeSeries extension for DAL protocols as a single spec: “TimeSeries discovery and access protocol a set of extensions to DAL protocol for TimeSeries” Obscore extension (Time sampling…) Corresponding PARAMETERS in SIAV2-like interface SODA extension parameters for data selection Access or DataLinks tags in source-driven context DM: Representation (requires modelling and serialisation) Work close to Gaia, ZTF, LSST to meet their deadlines Deliver different versions on our path towards ultimate goal Present serialisations —> encourage applications to test —> collect feedback Steps towards a TimeSeries DM:
- PhotDMv1-1 --> DMLize
- STC --> check which parts are a must
- Characterisation --> DMLize
- Following Time Series minimal requirements
Standard and Processes
Science Priorities
-->IVOA Roadmap for 2018B
This outlines the roadmap for development activities by the various IVOA working and interest groups in 2018 between the College Park and Paris Interops.
Applications WG
Python
- Continue to engage with Astropy community
- Offer support for VOTable in Astropy core
- Astroquery VO registry module
VOTable Standards
- Shepherd TIMESYS element proposal through the change process.
- Changes need to include the 3 existing errata.
- Work towards VODML-Mapping consesnsus.
- Determine if there are other driving factors for a new VOTable version.
Authentication
- Encourage application support for client authentication, working with Grid and Web Services WG to outline the scope and details.
HiPS
- Continue to encourage HiPS providers to declare their services in the VO registry according to the <a target="_blank" href="http://ivoa.net/documents/HiPS/index.html" title="HiPS Standard">HiPS Standard</a>(section 5.3).
Data Access Layer WG
- TAP-1.1
- New PR reflecting decision
- Alert community and update RFC accordingly
- Update reference implementation and validator
- Go REC and put TAPRegExt in RFC
- New PR reflecting decision
- SLAP-2.0
- Release official WD, then wait “VO-DML” SSLDM-2.0
- ObsLocTAP & ObjVisSAP
- Progress WD
- ADQL-2.1
- Continue 2nd reference implementation
- Tackle REGION issue
- DataLink-1.1
- Start revision on (subset?) of identified features
- Services weather report SHOULD warnings: take a decision
- Continue effort on polygon winding interoperability
Data Model WG
Priority model efforts:- Errata:
- Coords: Astronomical Coordinates and Coordinate Systems (Nov 2018)
- migrate RefFrame, RefPosition, TimeStamp enumerations to vocabularies
- complete WD documentation, and distribute for review
- Meas: Measurements and Errors (Dec 2018)
- restore basic Velocity, correct Correlation Matrix
- complete WD documentation, and distribute for review
- Trans: WCS Transforms (Dec 2018)
- restore transform operations needed by LSST, implemented by AST library
- complete WD documentation, and distribute for review
- WG concensus on coord, meas, trans model content, move to PR/RFC process (Jan-May 2019)
- Provenance ( May 2019 )
- review/address RFC comments (Nov 2018)
- assess impact of compromise model on implementations (Dec 2018)
- update model/document accordingly (Dec/Jan 2018)
- update implementations (Jan 2018)
- revised model through RFC (Jan/Feb 2018)
- work ProvSAP/ProvTAP (DAL projects)
- Dataset Metadata
- no contention, ready to move forward when measurement model dependency is satisfied. ( May 2019 )
- NDCube
- no open issues, ready to move forward when dependencies are satisfied... most likely AFTER "STC" and Dataset are completed ( May 2019? )
- SSDL-2.0
- get volute workspace setup ( Jan 2019 )
- vo-dml compliant model representation - WD ( May 2019 )
- TimeSeries
- continue working with TDIG to flesh out requirements for time domain data models and prototype/develop models as time and resources permit
- PhotDMv1-1
- Generate VO-DML compliant model; it is not yet clear if this is a major version update or minor.. the original model had gone far in this direction.
- preliminary work has been done in Volute in preparation for Victoria demos
- pick up from there with 'official' model project; modeler=Jesus, domain=??, tech=??
- Generate VO-DML compliant model; it is not yet clear if this is a major version update or minor.. the original model had gone far in this direction.
- Characterisation
- Migrate Characterisation to a VO-DML compliant model; this is a major project, at least in part motivated by time domain project needs
- modeler=Francois/Mireille?, domain=??, tech=??
- SourceDM
Grid and Web Services WG
For Paris:
- VOSI best practices note
- Science Platforms: call for standardization proposals on:
- How to lookup registry capabilities for a platform
- New capabilities applicable to science platforms
- Container execution? Input Params? Output location?
- Group Membership Services -- feedback from community, new working draft
- XML Schema Versioning -- address known issues
- Bring 2 VOSI erratum to TCG
- Hold 2 or 3 GWS telecons
For consideration:
- Work with DAL on potential VOSI-based table uploads
- Extend Credential Delegation Protocol to include OAuth 2.0
- See if ICRAR pyvospace implementation experience can improve VOSpace standard
Registry WG
- RegTAP 1.1: Incorporate changes from Authentication/Authorization re TAP 1.1 and endpoints, including example queries
- Work on reference implementation at MAST/NAVO
- Continue work on VODataService 1.2 regarding MOCs in the Registry, present May 2019
- Discovering Dependent Resources: Incorporate changes required by Authentication / Authorization re: TAP.
- Standards Records: We will continue to work with the chairs of the other IVOA working groups to ensure that they register their standards as envisioned by StandardsRegExt
- VOEventRegExt: A new approach to registering event streams is being developed within TDIG. Registry will contribute to this work, preliminary effort to be presented May 2019
- Operations: We will keep making sure the registry system runs; in particular, we will address anomalies diagnosed by ESAVO especially including inconsistencies in the Registry of Registries
- Registry will review validation done at the RofR level based on standards which have since been updated.
- Registry will develop VOResource errata regarding # in identifiers, which causes validation failures
- Registry maintainers will work on resolving OAI and Resource validation errors in their publishing registry implementations
Semantics WG
Vocabularies
A new page is set to gather vocabularies thanks to Giulia I. and Markus D. ivoa.net/rdf. with new vocabularies are proposed. We need to check how these are used and can be updated following the evolution of the IVOA protocols and services. Bring the Theory SKOS concept vocabulary into the ivoa repository and allow easy reference with the LUTH SKOS services. UCD available in RDF was asked for during ADASS meeting. A short meeting and a few testing iterations are planned this spring with ARI/Markus Demleitner and CDS /M.Louys to propose a UCD /RDF representation and a maintenance process. More visibility on the vocabulary pages (main page of Ivoa document repository ? ) This can be part of the new ivoa site design. and deiscussed together with the Media group and Giulia. A discussion to come up with a revision Note about the way vocabularies may be used in the VO eco system. (Proposed by Markus D. )Instrument and Telescope Nomenclature
Automated xmatch tools for list of names have been tested in Strasbourg in November 2018 (Coll. Graz, CDS, LESIA) Iteration on the PDS4 Information model helped to provide a simple description of links and classes relevant to build up a telescope list and to bind instruments description and features to it . ( CDS/ LESIA ) . We got a positive feedback from Greg Schwarz (AAS journals) and Marion Schmitz (NED). A new curated list by Marion Schmitz to be circulated and tested this spring to help confirm or update this information representation. Prototype for a telescope and instrument database planned at CDS as an Internship project.Interactions with other DataModel projects
We plan to keep track of the terms defined in data models currently proposed , SimulationDM revision, TimeDomain DM , IVOA Provenance DM in collaboration with these projects.Data Curation & Preservation IG
- IVOA Note about (PID)DOIs
- Discussion and editorial team is constituted
- First outcome will be presented in Paris
Education IG
- organize an EduIG session and invite plenary talk at the Interop in Paris
- renew application for IAU DAEPO Working Group
Knowledge Discovery IG
For the meeting in Paris:- organize a KDIG session and invite talks
- present a first prototype of anotating structural properties in projected spaces
- foster the discussion on bulk data-retrieval and on site data processing by bringing code to the data,
Operations IG
Solar System IG
Theory IG
Time Domain IG
- Status of model:
- PhotDMv1-1 is in the process of being DMLized
- Characterisation needs to be DMLized
- Status of serializations
- VODML-Lite & utype
- For the next interop concentrate on the practical usage by clients
- A proposal for a TIMESYS element in VOTable:
- Work with Apps to add element to VOTable
- Work with Semantics to set-up the needed vocabulary
- Work with data providers towards implementations
- VOEvent
- Take input from SSIG into account
Standard and Processes
Science Priorities
IVOA Roadmap for 2019A
This outlines the roadmap for development activities by the various IVOA working and interest groups in 2019 between the Paris and Groningan Interops.
Applications WG
Standards
- VOTable 1.4: Complete RFC period and recommend (well) before Groningen Interop
- MOC 1.1: Complete RFC period and recommend (well) before Groningen Interop
- VOTable 1.5: Move VOTable standards development to ivoadoc in github, and consider other possible changes.
- MOC 2.0 (probably): Work towards standardizing Space-Time MOCs (twiki page TBD).
Python
- Continue to engage with Astropy community
- Support open development in PyVO
- Maintain votable parsing in Astropy core
- Includes supporting VOTable 1.4
HiPS
- Continue to encourage HiPS providers to declare their services in the VO registry according to the HiPS Standard.
- Discuss metadata and integration with other standards.
Data Access Layer WG
- TAP-1.1 expected REC in summer
- ADQL-2.1 waiting 2nd implementation to enter RFC (fall?)
- DataLink 1.1 revision (internalWD)
- DALI1.2 revision,pending some discussion
- ObjVisSAP / ObsLocTAP 1.0 WD before Groningen, PR in 2020
- Some “format” discussion,content to stay as is
- SLAP-2.0 & ProvTAP,dependencies on DM
Data Model WG
Provenance
- We hope to complete the process by the end of the Summer.
- The PR process should be started before the end of July
Meas/Cood/Transf
- Ongoing discussions with Mark on the schedule. The process must speed up once Provenance is OK.
- Models depending on STC are still pedning
VODML Adaptation"> PhotDMv1-1 VODML Adaptation
- Working with JS and ML to complete the job before Grôningen
Source DM
- We are working on an note (DM+DAL+Apps) giving some precise guidelines for using a component based model
TimeSeries
- see TDIG
Grid and Web Services WG
Authentication and Authorization- New SSO version, Working draft by October 2019:
- Describe approach for obtaining token credentials (SSO)
- Describe approach for validating token credentials (SSO)
- Define an IVOA "Realm" and how it determines the scope of services for which a token is applicable. (SSO)
- Implement OAuth in TAP
- Another Group Membership Service Working Draft with definition of User Identities (GMS 1.0)
- Encouraged groups to use Kubernetes. Data centres will be running services as containers. Natural step to run user code in containers. Standards after more experience.
- Encouraging remote executing prototypes with containers. What infrastructure and information did you need? What problems did you face?
- Describe POSIX access to a platform. New VOResource Interface Type?
- Encourage collaboration. Share knowledge, code and resources with similar institutes. Organic convergence.
Registry WG
Operations:- Validation project: at least one telecon with RofR and registry maintainers (Notes at RegistryTelecon2019A)
- Validation project: bring remaining full registries to full validation compliance (aside from any remaining standards and RofR errors)
- Formally write and propose errata for # in identifiers, OR update RofR to newer VOResource standard to not need it.
- Contribute to TDIG work on registering event streams as needed
- Continue work on VODataService 1.2
- RegTAP11RFC
Semantics WG
Standards Track
- The UCD Maintenance standard is currently in late RFC and was pending the solution of some minor technicalities. Since these have been solved during the Paris interop, it should go to Exec for their next meeting.
- We will produce a first working draft for "Vocabularies in the VO 2.0" around August 2019; this will, much more than its predecessors, address consensus vocabularies, in particular as regards their maintenance and evolution.
- Update vocabulary maintenance software in lockstep with Vocabularies 2.0 document; this, in particular, concerns support for vocabularies over classes, which we (probably) need for the VOTable-attached vocabularies; also, draft terms (as opposed to draft vocabularies) will need software support.
- Guide Timescale and Refpos vocabularies through the VOTable 1.4 RFC process.
- Work on metadata of the theory vocabularies.
- Perform the first “vocabulary enhancement proposals” (as per Vocabularies 2.0), presumably on Datalink terms.
- Observation facility vocabulary: test metadata and classes with the current TAP prototype, insert more lists, define or reuse a name resolver.
- UCD maintenance procedure: After completing the RFC process , put it in action for the current RFM pages.
Data Curation & Preservation IG
- DOI Survey to continue
- DOI Note draft to publish before Groningen and to discuss during the DCP session
Education IG
Knowledge Discovery IG
Operations IG
Continue normal activities:- monitor VO services, expect to present weather reports at the next Interop
- monitor availability of validators for existing and new standards (and versions), recorded at IvoaValidatorsSummary
- provide a forum for discussion of operational issues at and between interops
- encourage more use of Ops mailing list for topics of potential interest?
- try to provide better feedback between validation activities and WGs
Solar System IG
- Draft EPNcore / EPN-TAP version (with DAL?)
- List of reference frame / coordinate systems (with Semantics)
Theory IG
- Want to create SimDM 1.1: port to VO-DML (basically done) and fix vocabulary links.
- Discussion about maintenance of semantic vocabularies used in SimDM. Some can be shared (say astroobjects), but we should be able to create some just for theory. It will take too long to push through long acceptance process.
- How SimDAL services can be registered in the Registry?
- Any application able to use a SimDAL service?
Time Domain
- TimeSeriesDM:
- Explore how the current status of the data model is able to describe ZTF-DR1 products (Objects Table, lightcurves, single-epoch images, quality flags) * Work with Apps on ST-MOC:
- ST-MOCs
- Work with data providers to precompute STMOCs: VizieR catalogs and HiPS, and for solar system body ephemeris - work close to SSIG for that.
- Ingest in CDS MocServer: clients can resource by Space-Time.
- Create library in python and add to mocpy.
- Do we need a standard IVOA 2.0 MOC for SMOC, TMOC and STMOC?
- VO registry coverage by STMOC? Work with registry for that.
- Work with Semantics to improve discovery and client use of:
- VOEvents services and streams (What and where)
- Time Series as distributed by datalink (data product (sub)type)
- Get the refposition case of topocenter started for TIMESYS in VOTable1.4
Standard and Processes
Science Priorities
IVOA Roadmap for 2019B
This outlines the roadmap for development activities by the various IVOA working and interest groups in 2019 between the Groningan and Sydney Interops.
Applications WG
Standards
- VOTable 1.5: Move VOTable standards development to ivoadoc in github, and consider other possible changes.
- MOC 2.0 (probably): Work towards standardizing Space-Time MOCs (twiki page TBD).
Python
- Continue to engage with Astropy community
- Support open development in PyVO
- Maintain votable parsing in Astropy core
- Includes supporting VOTable 1.4
HiPS
- Continue to encourage HiPS providers to declare their services in the VO registry according to the HiPS Standard.
- Discuss metadata and integration with other standards.
Data Access Layer WG
Ordered by rough priority/maturity level. Timelines sketched where possible.- ADQL-2.1
- last issues to close
- PR & RFC by end 2019
- REC by/in Sydney (May 2020)
- DALI-1.2
- DataLink-1.1
- WD by end 2019/start 2020
- changes/feature roadmap assessment along the way
- SLAP-2.0
- PR soon after Groningen Interop (Oct./Nov. 2019)
- RFC by end 2019: pending 2nd implementation engagement
- follow SSLDM-2.0 recmmendation track
- ObsLocTAP & ObjVisSAP
- WD in 1.0 by Sydney
- ProvTAP
- expected to progress as proper WD
- ConeSearch-1.1
- restart revision
- Time requirements addition
- WD should appear by end 2019 or Q1 2020
Data Model WG ((2019A))
Provenance
- We hope to complete the process by the end of the Summer.
- The PR process should be started before the end of July
Meas/Cood/Transf
- Ongoing discussions with Mark on the schedule. The process must speed up once Provenance is OK.
- Models depending on STC are still pedning
VODML Adaptation"> PhotDMv1-1 VODML Adaptation
- Working with JS and ML to complete the job before Grôningen
Source DM
- We are working on an note (DM+DAL+Apps) giving some precise guidelines for using a component based model
TimeSeries
- see TDIG
Grid and Web Services WG
Authentication and Authorization- New SSO version, Working draft by October 2019:
- Describe approach for obtaining token credentials (SSO)
- Describe approach for validating token credentials (SSO)
- Define an IVOA "Realm" and how it determines the scope of services for which a token is applicable. (SSO)
- Implement OAuth in TAP
- Another Group Membership Service Working Draft with definition of User Identities (GMS 1.0)
- Encouraged groups to use Kubernetes. Data centres will be running services as containers. Natural step to run user code in containers. Standards after more experience.
- Encouraging remote executing prototypes with containers. What infrastructure and information did you need? What problems did you face?
- Describe POSIX access to a platform. New VOResource Interface Type?
- Encourage collaboration. Share knowledge, code and resources with similar institutes. Organic convergence.
Registry WG
Operations:- Validation project: Re-validate registries and bring remaining full/searchable registries to full validation compliance, given 2019A improvements to RofR's validator
- Validation project: at least one telecon with RofR and registry operators/software maintainers for remaining work given 2019A RofR changes
- Explore methods for announcing server software in resources as done with user-agent information client-side
- Support IVOA registry inclusion with EUDAT/EOSC/ESCAPE projects
- Encourage takeup of altIdentifier information, especially DOIs, across publishing registries
- Continue work on VODataService 1.2, especially regarding MOCs in the registry
- Explore registration for HiPS nodes
Semantics WG
Since VOTable 1.4 is now REC, we will remove the preliminary status on the associated vocabularies (timescale and refpos) shortly after the Groningen Interop. We will make sure all Semantics standards are properly registered as we update the record of UCD maintenance. The first updated UCD list according to the new process should be ready for the next TCG meeting. As it is going to be published, we will have to figure out how to represent the current UCD list in the document repository, somehow clearly indicating that people should no longer consult the REC. The work on Vocabularies in the VO 2 will go on. Soon after Groningen, we will try to run the first one or two VEPs and see how discussions will go. Provided the process proves workable, we might go for PR with Vocabularies in the VO 2.0 after Sydney. One vocabulary that might need to be discussed outside of any current standard is one of astronomical object types, usable in SSA's and Obscore's target_class and at the same time on Simbad and SimDB and ideally as synchronised with UAT as possible. We will try to engage the interested communities in such work.Data Curation & Preservation IG
- DOI Survey to continue
- DOI Note draft to publish before Groningen and to discuss during the DCP session
Education IG
Knowledge Discovery IG ((2019A))
Operations IG
- Service monitoring:
- Validator curation:
- Monitor availability of validator software for existing and new standards and versions (IvoaValidatorsSummary)
- Identifying Server Software:
- Experiment with HTTP headers to allow clients to identify VO software stack
- Keep an eye on Science Platform operational issues
- Provide a forum for discussion of operational issues at and between interops
Solar System IG
- Draft EPNcore / EPN-TAP version (with DAL?)
- List of reference frame / coordinate systems (with Semantics)
Theory IG
- Want to create SimDM 1.1: port to VO-DML (basically done) and fix vocabulary links.
- Discussion about maintenance of semantic vocabularies used in SimDM. Some can be shared (say astroobjects), but we should be able to create some just for theory. It will take too long to push through long acceptance process.
- How SimDAL services can be registered in the Registry?
- Any application able to use a SimDAL service?
Time Domain
- TimeSeriesDM Worked on a Note based on ZTF and Gaia data examples
- https://wiki.ivoa.net/twiki/bin/view/IVOA/TimeSeries2020
- Next steps: send email to TDIG, migrate to the IVOA GitHub repo
- Aim: WD by May
- ST-MOC next steps:
- Work towards standard. There should be a draft in May. P. Fernique working on that.
- Create library in python and add to mocpy. (done)
- VO registry coverage by STMOC? Work with registry for that. Still to be done
- Extension of Cone Search:
- Worked on some parts of the text to add an interval of time as a search parameter
- Need to figure things out on GitHub
- VOEvent:
- Updates by SSIG will lead to VOEvent2.1
Standard and Processes
Science Priorities
IVOA Roadmap for 2020A
This outlines the roadmap for development activities by the various IVOA working and interest groups in 2020 between the May and Oct Interops.Applications_WG
Standards
- VOTable 1.5
- Would love to demonstrate that a new minor version can be turned around fairly quickly.
- Begin discussion of existing issues (https://github.com/ivoa-std/VOTable/issues).
- Continue improving contributors' comfort level with the change process within github.
- MOC 2.0: Work towards standardizing Space-Time MOCs (twiki page TBD).
Python
- Continue to engage with Astropy community
- Support open development in PyVO
- Maintain votable parsing in Astropy core
- Includes supporting VOTable 1.4
HiPS
- With other WGs, discuss metadata and integration with other standards.
- Continue to encourage HiPS providers to declare their services in the VO registry according to the HiPS Standard.
Data Access Layer WG
- DataLink-1.1
- WD in progress
- a few issues to solve
- RFC6570 for templated endpoints to be further investigated
- RFC by Fall or end 2020
- ADQL-2.1
- still a few issues to go
- UDF Catalogue EN to connect to
- PR, RFC by Fall 2020
- REC by end 2020
- still a few issues to go
- DALI-1.2
- ConeSearch-1.1 [in connection with TD IG]
- ObsLocTAP recommendation path is now under DM WG
- PR-1.0 available and RFC started
- ObjVisSAP
- WD-1.0 available
- follow up for PR in Fall 2020?
- ProvTAP / ProvSAP
- TAPRegExt [in connection with Registry WD]
- started move to GitHub repo
- revision in view of the "caproles" discussion
- SIAP-2.0 / SODA-1.0
- gathering feedback also from PyVO implementations
- SLAP-2.0
- this specification is now stopped waiting for a possibly different solution
- if VAMDC XSAMS to VO modelling works it will be dropped and fit in a *TAP solution
- else it will be resumed and follow SSLDM-2.0
- this specification is now stopped waiting for a possibly different solution
Data Model WG
-
MCT:
- Meas/Coords ready for RFC2
- Transf ready to go in RFC once Coord is frozen
- Source Model (CAB-MSD)
- Ongoing work in the model spec
- Ongoing work in the mapping spec
- Both model and mapping to be tested on more data (Vizier Chandra XMM)
- PhotDMv1-1
- Must be modified to be serialized in VODML
- SSLDM2
- RFC process: model to be dropped if a TAP/CAMDC based solution can work.
Grid_and_Web_Services_WG
Auth and Authz: * New SSO version, Working draft by October 2020:-
- Describe approach for obtaining token credentials (SSO)
- Describe approach for validating token credentials (SSO)
- Define an IVOA "Realm" and how it determines the scope of services for which a token is applicable. (SSO)
- Implement OAuth in TAP
- Group Membership Service move to PR and it has two reference implementations: CADC and IA2
- Credential Delegation protocol: introduce delegation based on token.
- Encouraged groups to use Kubernetes. Data centres will be running services as containers. Natural step to run user code in containers. Standards after more experience.
- Encouraging remote executing prototypes with containers. What infrastructure and information did you need? What problems did you face?
- Describe POSIX access to a platform. New VOResource Interface Type?
- Encourage collaboration. Share knowledge, code and resources with similar institutes. Organic convergence.
Registry WG
Operations:- Ongoing validation: check all registry contents for standards validation and consistency
- Encourage takeup of altIdentifier information, especially DOIs, and consistent facility/instrument metadata, across publishing registries
- Facilitate creation and updates of VOResource versions of standards records as we update the GitHub repository, and provide them to the Registry of Registries
- Continue work on VODataService 1.2, especially regarding MOCs in the registry
- Explore registration for HiPS nodes
- "Caproles" revisions: collaborate with DAL on TAPRegExt revisions, make supporting changes in base Registry standards as required
Semantics WG
In Semantics, we will push ahead Vocabularies in the VO 2.0, aiming for RFC around the Granada interop. We believe the document now is essentially sound, and so the main challenge will be to gain sufficient implemenation uptake. We will try to engage authors of VOTable and VOResource validators, to extend the pyVO datalink client to take into account the term hierarchy, and to perhaps provide an ADQL user defined function that will allow the use of simple semantics in query expansion. We will also work with the Registry WG on resolving the open question of what "use the Unified Astronomy Thesaurus" means for VOResource 1.1, aming for an Erratum to VOResource and perhaps some sort of agreement between the operators of RegTAP services on how to speed up the adoption of UAT use in the Registry. Our ongoing activities include assistence in producing Vocabularies 2-style vocabulary enhancement proposals and moderating discussions, and maintaining the UCD controlled vocabulary.Data Curation Preservation IG
Education IG
- IAU 367: Education and Heritage in the Era of Big Data in Astronomy
- Release of the ESASky in Chinese
Knowledge Discovery IG
Operations IG
- Service monitoring:
- Validator curation:
- Monitor availability of validator software for existing and new standards and versions (IvoaValidatorsSummary)
- Keep an eye on Science Platform operational issues
- Provide a forum for discussion of operational issues at and between interops
Radio IG
- 2nd telecon planned for Sept/Oct (following first one in eary July) to discuss an implementation note for radio data in the VO, a possible VO workshop for software developers from radio astronomy archives, and common needs of the various types of radio data in the VO environment (visibility data, single dish data, pulsar observations etc...Obviously one of these common needs is related to provenance).
- K. Lutz (CDS) is leading a proposal for a radio data BoF session at ADASS.
Solar System IG
- Publish EPN-TAP working draft
Theory IG
Time Domain
- ConeSearch-1.1: see DAL
- TimeSeries: Need to make few modifications in particular to relax a parameter (optional instead of mandatory). Get it promoted to an endorsed note and hosted in the ivoa-std repository.
- ST-MOC next steps: working on the draft towards std -> should be exposed sometime before next interop.
- VOEvent: Navigate Baptiste's changes to VOEvent through the review process and get it accepted as a new version.
Standard and Processes
Science Priorities
IVOA Roadmap for 2020B
This outlines the roadmap for development activities by the various IVOA working and interest groups in 2020/2021 between the Nov and May Interops.Applications WG
Standards
- MOC 2.0: Work towards standardizing Space-Time MOCs.
- Working Draft published Nov 2020 (https://www.ivoa.net/documents/MOC/20201112/WD-MOC-2.0-20201112.pdf)
- Aiming for PR prior to May 2021 Interop. Need:
- Two interoperable implementations of each feature
- Validation tool(s)
- Ivoatex version in github
- VOTable 1.5
- Would love to demonstrate that a new minor version can be turned around fairly quickly.
- Stimulate diiscussion of existing issues (https://github.com/ivoa-std/VOTable/issues).
- Continue improving contributors' comfort level with the change process within github.
- Clean up applications WG wiki pages.
- Remove stale links
- Ensure guidance on current work is up to date
Python
- Continue to engage with Astropy community
- Support open development in PyVO
- Maintain votable parsing in Astropy core
- Includes supporting VOTable 1.4
HiPS
- With other WGs, discuss metadata and integration with other standards.
- Continue to encourage HiPS providers to declare their services in the VO registry according to the HiPS Standard.
Data Access Layer WG
standards on the recommendation path- DataLink-1.1
- WD progressing, new WD release (Q1-2021) after closing a few Pull Requests
- this might set the line for v.1.1
- further issues/features identified already for subsequent revision
- WD progressing, new WD release (Q1-2021) after closing a few Pull Requests
- ADQL-2.1
- UDF Catalogue EN pending
- issues progressing, a few to close before next Proposed REC (Q1 2021)
- REC by/at Spring Interop
- DALI-1.2
- multipolygon still pending
- otherwise ready for PRec+RFC (Q1-2021)
- ConeSearch-1.1 [in connection with TD IG]
- work will progress to close issues
- a new WD before Spring 2021 Interop
- ObjVisSAP
- PR-1.0 nearly there (Q1-2021 at max)
- implementations ready, RFC & REC should be possible by/at Spring Interop
- ProvTAP
- continues work towards mature WD status (by Spring Interop)
- TAPRegExt [in connection with Registry WD]
- started move to GitHub repo
- WD release in Q1/Q2-2021
- SIAP-2.0 / SODA-1.0
- moving to GitHub repo (partly already there, including issues)
- work on new WDs based on existing feedback (WDs by 2021 Spring Interop)
- EPN-TAP [DM WG connection]
- WD-2.0 ready for the community
- progress depending on feedback
- ObsLocTAP [managed by DM WG]
- RFC for PR-1.0 should finish by end 2020, REC by begining of 2021
- SLAP-2.0 -> LineTAP solution
- preliminary internal (to authors) WD
- progress expected to be reported at Spring 2021 Interop
- ProvSAP (idle)
- Following Radio IG, TD IG activities for requirements
- refer to the roadmaps of these IGs for details
- Spectroscopy data access could potentially lead new revisions
- file-oriented catalog data access: start of discussion
Data Model WG
Ongoing Standards- Measurements to REC ( RFC page)
- Coordinates to REC ( RFC page)
- ObsLocTap to REC ( RFC page)
- Transform to RFC ( RFC page)
- Model for source data Mango to RFC ( Github page)
- Votable Annotation ModelInstToVot ( Github page)
- Running a workshop to build a concensus on the model usage in the VO. The outcomes of this workshop might impact ongoing standards.
Grid and Web Services WG
Science Platforms
- Towards a Theory IG and GWS workshop on science platforms at Interop "gemini" 2021
- Encourage collaboration. Share knowledge, code and resources with similar institutes. Organic convergence.
- Identify possible ideas for standards on SP (e.g. common libraries to read/write data)
- Describe POSIX access to a platform. New VOResource Interface Type?
- How can we remote execute codes on data? What infrastructure and information did you need? What problems did you face?
- Encouraged groups to use Kubernetes. Data centres will be running services as containers. Natural step to run user code in containers. Standards after more experience.
Authentication and Authorization
- Group Membership Service move to PR
- it has two reference implementations: CADC and IA2
- we need a validator
- SSO version 1.2 needs to be discussed
- Describe approach for obtaining token credentials (SSO)
- Describe approach for validating token credentials (SSO)
- Define an IVOA "Realm" and how it determines the scope of services for which a token is applicable. (SSO)
- Credential Delegation protocol under discussion: introduce delegation based on token.
Registry WG
Operations:- Ongoing validation: check all registry contents for standards validation and consistency
- Encourage takeup of altIdentifier information, especially DOIs, and consistent facility/instrument metadata, across publishing registries
- Facilitate creation and updates of VOResource versions of standards records as we update the GitHub repository, and provide them to the Registry of Registries
- VODataService 1.2 through PR
- SimpleDALRegExt 1.2: make PR, identify roadblocks with SSAP time series and work on additional reference implementations as required
Semantics WG
We will move Vocabularies in the VO 2 to PR in early January, followed by RFC. Probably in parallel, we will hold an RFC for the related PEN on the adoption of the UAT. Together with Registry, we will then try to work out what needs to be done in terms of updates to VOResource 1.1 in order to clarify the use of the UAT in the Registry; at this point, we believe a largely explanatory erratum should do. We will continue to maintain the vocabulary repository with possible RFCs; in particular, we expect the vocabularies on messengers and on data product types to move ahead with the definining standards (VODataService 1.2 and SimpleDALRegExt, respectively). We would still like to move ahead with a vocabulary of object types, although no concrete plans exist at this point.Data Curation Preservation IG
Education IG
- Initiation of the partnership between the IVOA and IAU OAD
- Co-organization of the IAUS367, “Education and Heritage in the Era of Big Data in Astronomy”
Knowledge Discovery IG
Operations IG
- Tackle service validation failures:
- Target: significantly improve service compliance reports compiled by Euro-VO/PADC by May 2021 Interop
- Involve: validator operators + authors, service operators + authors
- Joint activity with DAL
- SoftID Note
- Operational identification of (client/server) operational software in the VO
- Note currently undergoing review
- Continuing activities:
- Monitor VO health
- Curate list of validation tools (especially for new/revised standards)
- Provide forum for discussion of operational activities
- Participate in and give feedback on standards development in support of upcoming operational missions
Radio IG
Continuing work on implementation note, content gathering mostly complete, but needs to be tidied up.Solar System IG
Theory IG
Time Domain IG
- ConeSearch-1.1 [in connection with DAL WG]
- Work will progress to close issues
- A new WD before Spring 2021 Interop
- MOC-2.0 [in connection with Apps WG]
- Need to adapt the 2 implementations to recent modifications
- In WD, aimed for a RFC before Spring 2021 Interop
- ObsCore extension for timeseries [in connection with DM WG]
- Have a running meeting on this topic.
- Draft a note to expose the elements before Spring 2021 Interop
Standard and Processes
Science Priorities
IVOA Roadmap for 2021A
This outlines the roadmap for development activities by the various IVOA working and interest groups in 2021 between the May and Oct/Nov Interops.Applications WG
- MOC 2.0: Work towards standardizing Space-Time MOCs.
- Working Draft published May 2021
- Moving to RFC Summer 2021
- Waiting on 2nd implementation
- VOTable 1.5
- Would love to demonstrate that a new minor version can be turned around fairly quickly.
- Stimulate diiscussion of existing issues (https://github.com/ivoa-std/VOTable/issues).
- Continue improving contributors' comfort level with the change process within github.
- Clean up applications WG wiki pages.
- Remove stale links
- Ensure guidance on current work is up to date
- Migrate HiPS and SAMP documents to Github
Python
- Continue to engage with Astropy community
- Support open development in PyVO
- Maintain votable parsing in Astropy core
- Includes supporting VOTable 1.4
Data Access Layer WG
standards on the recommendation path- DataLink-1.1
- WD progressing, new WD release (Q3-2021) after closing a few Pull Requests
- this might set the line for v.1.1
- RFC by Autumn Interop
- further issues/features identified already for subsequent revision
- WD progressing, new WD release (Q3-2021) after closing a few Pull Requests
- ADQL-2.1
- Ready for Proposed Rec (Q2 2021)
- RFC opening soon
- REC by Autumn Interop
- DALI-1.2
- multipolygon still pending
- otherwise ready for PRec+RFC (Q3-2021)
- ConeSearch-1.1 [in connection with TD IG]
- work will progress to close issues
- a new WD before Autumn 2021 Interop
- ObjVisSAP
- PR-1.0 nearly there
- implementations ready, RFC should be possible by Autumn Interop
- ProvTAP
- continues work towards mature WD status (by Autumn Interop)
- TAPRegExt [in connection with Registry WD]
- Now in GitHub repo
- WD release in Q3/Q4-2021
- SIAP-2.0 / SODA-1.0
- Now in GitHub repo
- work on new WDs based on existing feedback (WDs by 2021 Autumn Interop)
- EPN-TAP [DM WG connection]
- Validator being prepared
- Some tidy-up expected after validator work
- Should move to RFC in Q3 2021
- ObsLocTAP [managed by DM WG]
- RFC for PR-1.0 period closed. Should move to REC by end June
- Proposed for endorement at IAU General Assembly in August 2021
- SLAP-2.0 -> LineTAP solution
- preliminary internal (to authors) WD
- progress expected to be reported at Autumn 2021 Interop
- SSAP -
- Spectroscopy data access could potentially lead to new minor revisions in advance of any integration with a more general SIA2
- ProvSAP (idle)
- Following Radio IG, TD IG activities for requirements
- refer to the roadmaps of these IGs for details
- file-oriented catalog data access: start of discussion on Parquet
Data Model WG
- DM workshop conclusions
- ObsLocTap in REC (Done)
- Obscore evolution (TD, Radio)
- EPN-TAP (see DAL)
Grid and Web Services WG
Authentication and Authorization
- Group Membership Service moves to PR
- it has two reference implementations: CADC and IA2
- SSO version 1.2 is under discussion
- Describe the approach for obtaining token credentials (SSO)
- Describe the approach for validating token credentials (SSO)
- Define an IVOA "Realm" and how it determines the scope of services for which a token is applicable. (SSO)
- Credential Delegation Protocol under discussion: introduce delegation based on tokens.
VOSpace
VOSpace 2.1: an update of the standard is under discussion with some minor (and major) proposed changes.Science Platforms
- After theTheory IG and GWS workshop on science platforms at Interop 2021
- we had a large participation from scientists not related to IVOA
Registry WG
Operations and Software:- Ongoing validation: continually validate registry records across the IVOA and provide feedback to validators, registry maintainers, and standards authors/editors as relevant
- Support and Maintenance: Support the Registry of Registries, provide testing and feedback for new publishing registries that come online and major upgrades to existing registries, especially the changes to the VizieR registry scheduled for this timeframe.
- pyvo and regsearch: provide expertise for expanding the registry searching capabilities of the pyvo python client.
- VODataService 1.2 through PR
- SimpleDALRegExt 1.2: make PR, identify roadblocks with SSAP time series and work on additional reference implementations as require
- Facilitate creation and updates of VOResource versions of standards records as we update the GitHub repository, and provide them to the Registry of Registries
Semantics WG
The Semantics WG hopes to get the UCD list 1.4 approved by the TCG early in the semester; it has been ready content-wise for quite a while. We will then start work on the UCD list 1.5, for which some statistical terms are already in the pipeline. For SimpleDALRegExt 1.3, we will try to get the http://www.ivoa.net/rdf/refframe and http://www.ivoa.net/rdf/product-type vocabularies into shape. The first is also necessary for Coords in DM, for the second we will take care it will remain topical and useful for its original source in Obscore. For VODataService 1.2, we will accompany the acceptance process for the http://www.ivoa.net/rdf/messenger vocabulary, working with the SSIG to see if additional messengers from the are already required. With Vocabulares 2.0 now passed, we will work with Registry to get the note on adopting the UAT as an IVOA vocabulary endorsed and then produce an Erratum to VOResource 1.1, explaining how to actually apply it in VO resource records. We hope to publish a first draft of a vocabulary of object types in June. This will reflect the current state of SIMBAD's work on updating their object classes. Somewhat more remote is a vocabulary of instruments and facilities (where it is not unlikely that we will find that this goes beyond what vocabularies should reasonably do). If it is feasible, it would provide interoperable identifiers for such entities (e.g., for the VOResource elements and the respective fields in SSAP, SIAP, or Obscore), as well as pointers to names and identifiers them in legacy lists.Data Curation Preservation IG
Education IG
* Organise newcomers session for next Interop * Organise newcomers session for Southern Spring Interop * Start regular online meetings for advanced VO usersKnowledge Discovery IG
Operations IG
- Software Identification:
- SoftID Note was published as NOTE-softid-1.0 just after May Interop
- Encourage and monitor service and client takeup
- Tackle non-compliant services:
- Support service operators to use validation and improve services
- Update validators as required
- Chase up common/persistent issues
- Provide feedback on standards
- Continue to work with DAL to feed back operational experience to standards update/development
- Ongoing activities:
- Monitor VO health (conformance, reliability, performance)
- Curate IvoaValidatorsSummary
- Provide forum for ongoing discussions
- Schedule occasional telecons as required. Possible topics:
- Non-compliant service review and activities
- DAL/Ops standards feedback (and what to do about it)
- Validator developments
Radio IG
- Implementation Note posted to GitHub
- completion by fall interop ?
- ObsCore extension for radio data under discussion
- WD proposed by fall interop
- new media type proposal for radio data formats
- to be discussed
- discovery of single dish and time/frequency data (PSR or whatever) * better strategy by fall interop
Solar System IG
- Existing List of Objectives
- Identified the top two priority objectives for immediate action
- Assess and assign priorities to all remaining objectives
- Priority 1 Objective: Identifying observation facilities (ground and space). This is a complicated problem requiring integration and maintenance of data from multiple sources. The short-term goal is to produce a "usable product" as soon as possible, even if that is just a massive spreadsheet initially. Ultimately, this needs to be a reference service. By the November Interop:
- Scope the problem
- Draft the short-term design
- Determine contacts for identification authorities
- Priority 2 Objective: Standard List of Coordinate Systems and Reference Frames. By Nov. Interop:
- Evaluate the terminology and capabilities from the NASA/NAIF SPICE system that can be applied to the IVOA case.
- Determine timeline to meet implementation with STC-2.
Theory IG
Follow up Theory/GWS workshop on Cosmological Simulations on Science Platforms:- Plan at least one more workshop before next interop (Nov 2021), similar format as the one during interop May 2021
- Get feedback and input on possibility for standardizing access methods for cosmological simulations. Possibilities are
- Work with GWS, giving examples for how to do code-to-data for simulations on science platforms
Time Domain IG
On coverages- MOC2.0 WD: 2 implementations and validator - RFC (TD & Apps)
- Info collected under: https://wiki.ivoa.net/twiki/bin/view/IVOA/MocInfo?
- Everyone is welcome to comment under https://github.com/ivoa-std/MOC
- Use of Time-Frequency Catatalogs for by solar & planetary radio science - evaluate/ investigate further on the extension of MOC to add frequency / energy / wavelength (TDIG & SS & Radio IGs)
- FRBCat and TNS - follow-up on the possibility of adding a TAP service (TD & Radio IGs)
- Data Central’s Data Aggregation Service to quickly inspect FRB candidates - Follow-up on time series explorer (TD + DAL)
- Astro-COLIBRI - follow-up on usage of IVOA stds where possible, e.g ObsLocTAP, ObjVisSAP
- ObsLocTAP - follow-up on status
- Note under: https://github.com/ivoa/TimeSeriesDiscoveryAndAccess
- Update to include the SSA - keeping in mind the std is likely to evolve in the near future (TDIG & DAL)
Standard and Processes
Science Priorities
IVOA Roadmap for 2021B
This outlines the roadmap for development activities by the various IVOA working and interest groups between the Nov 2021 and May 2022 Interops.Applications WG
Python
- Support open development in PyVO
- Convene coordination meetings for maintainers, contributors and anyone interested.
- Help get TAPPlus into PyVO
- Maintain votable parsing in Astropy core
Standards and Other Business
- MOC 2.0: Work towards standardizing Space-Time MOCs.
- Complete RFC period and move to REC
- Migrate HiPS and SAMP documents to Github
- Clean up applications WG wiki pages.
- Remove stale links and make sure all information is current
- VOTable 1.5
- Stimulate discussion of existing issues (https://github.com/ivoa-std/VOTable/issues).
Data Access Layer WG
Standards on the recommendation path
- DataLink-1.1
- WD released Nov 2021
- this might set the line for v.1.1
- RFC by Spring Interop
- WD released Nov 2021
- ADQL-2.1
- Ready for Proposed Rec (Q1 2022)
- Validation required before RFC
- RFC by Spring Interop
- DALI-1.2
- multipolygon still pending
- otherwise ready for PRec+RFC (Q2-2022)
- ConeSearch-1.1 [in connection with TD IG]
- work will progress to close issues
- a new WD before Spring 2022 Interop
- ObjVisSAP
- PR-1.0 nearly there
- implementations ready
- ProvTAP
- continues work towards mature WD status
- TAPRegExt [in connection with Registry WD]
- WD release in Q1/Q2-2022
- SIAP-2.0 / SODA-1.0
- Exploring general simple data discovery (e.g. adding spectra)
- work on new WDs based on existing feedback (WDs by Spring Interop)
- EPN-TAP [DM WG connection]
- RFC open
- Aiming for REC in Q1-2022
Other standards & activities
- SLAP-2.0 -> LineTAP solution
- preliminary internal (to authors) WD
- progress expected to be reported at Spring 2022 Interop
- SSAP
- Spectroscopy data access could potentially lead to new minor revisions in advance of any integration with a more general SIA2
- Following Radio IG, TD IG activities for requirements
- refer to the roadmaps of these IGs for details
- file-oriented catalog data access: start of discussion on Parquet
Data Model WG
- DM workshop follow-up
- Obscore evolution (TD, Radio)
- EPN-TAP (see DAL)
Grid and Web Services WG
Authentication and Authorization
- Group Membership Service is in RFC we expect to conclude for the end of the year.
- it has two reference implementations: CADC and IA2
- SSO version 1.2 is under discussion we have a group working on implementations and testing.
- Describe the approach for obtaining token credentials (SSO)
- Describe the approach for validating token credentials (SSO)
- Define an IVOA "Realm" and how it determines the scope of services for which a token is applicable. (SSO)
- Credential Delegation Protocol: we should introduce delegation based on tokens. We need an editor.
VOSpace
VOSpace 2.1: an update of the standard is under discussion with some minor (and major) proposed changes.Science Platforms
- The Execution Planner interface aims to provide a simple way to discover and access computing services. we are working on a Note.
Registry WG
- Complete SimpleDalRegExt 1.2 RFC and approve to REC if possible (https://wiki.ivoa.net/twiki/bin/view/IVOA/SimpleDALRegExtV12RFC)
- Explore registration of documentation and tutorial notebooks: there is an existing Note about this, which could be endorsed or converted to a proposed recommendation, given takeup
- pyvo coordination: expand registry search for simplified data discovery options and test backward compatibility of search changes, not just for errors but for general returned results(see https://wiki.ivoa.net/twiki/bin/view/IVOA/IVOANov3RegistryEtherpad)
- Ongoing validation and curation efforts among registry maintainers, resource curators, and validator and standards authors
- Explore space-time coverage that was deferred from VODataService 1.2, approved in 2021A.
Semantics WG
On the UCD end of things, we are hoping to have a PEN for version 1.5 of the UCD list ready for TCG review by early 2022. This will include a few statistical concepts, and possibly MOCs, projection and a means to identify oscillation periods from other periodic phenomena; there is also a generic concept for PIDs considered. See https://wiki.ivoa.net/twiki/bin/view/IVOA/UCDList_1-5_RFM for details. Also regarding UCDs, we will prepare two Errata (to SSAP and SLAP) regarding malformed UCDs used in these standards. For SimpleDALRegExt RFC, the vocabulary http://www.ivoa.net/rdf/refframe is under review; we believe it is largely in shape. This cannot be said for http://www.ivoa.net/rdf/product-type, which is up for adoption by Datalink. We will provide a new prototype based on SKOS before the end of 2021 and evaluate whether it better fits the use cases envisioned. We have also produced a draft of a vocabulary of object types at http://www.ivoa.net/rdf/object-type. There is no fixed schedule on which this should be evaluated and developed, except that having it for the next revision of obscore would clearly be very useful. Also, we should revive the process of getting the note on the adoption of the UAT as an IVOA vocabulary endorsed by the TCG.Data Curation Preservation IG
Education IG
Knowledge Discovery IG
Operations IG
- Tackle non-compliant services
- Review specific items noted in Nov 2021 Euro-VO Registry Weather Report
- Chase up common/persistent issues
- Support service operators to use validation and improve services
- Develop/maintain/curate validation tools
- Ensure validators are available for all standards (esp. during REC process)
- Pursue automatic invocation capability for validation tools
- Understand discrepancies between different validators
- Work with other WGs:
- DAL: monitor follow up of issues reported May 2021 (in progress)
- Semantics: monitor UCD/VOUnit compliance changes since report Nov 2021
- Ongoing activities:
- Monitor VO health (conformance, reliability, performance)
- Provide forum for discussion of operational issues
Radio IG
- exposing Radio Astronomy Implementation review note
- continue discussion on ObsCore extension for visibility data and push it as EndorsedNote proposal or WD
- Running meeting with TDIG on discovery/acces to pulsar/time domain data in radio
- Running meeting on discovery/access for single dish data
Solar System IG
The objectives from the previous roadmap carry over:- Priority 1 Objective: Identifying observation facilities (ground and space). This is a complicated problem requiring integration and maintenance of data from multiple sources. The short-term goal is to produce a "usable product" as soon as possible, even if that is just a massive spreadsheet initially. Ultimately, this needs to be a reference service.
- Scope the problem
- Draft the short-term design
- Determine contacts for identification authorities
- Priority 2 Objective: Standard List of Coordinate Systems and Reference Frames.
- Evaluate the terminology and capabilities from the NASA/NAIF SPICE system that can be applied to the IVOA case.
- Determine timeline to meet implementation with http://www.ivoa.net/xml/STC/stc v1.30.xsd STC v1.30.xsd Errata: STC 1.33 Errata Goal History ...">STC-2.
- Remaining List of Objectives
- Identified the top two priority objectives for immediate action
- Assess and assign priorities to all remaining objectives
Theory IG
- SimDM migrationo to 1.1: small URL changes and VO-DML representation
- SimDM implementation efforts in Australia, JHU, and exercising Paul Harrison's VO-DML-based code.
Time Domain IG
Standard and Processes
Science Priorities
Keep up with community feedback Participate to the running meetings Evaluate the need of SLAP V2 Collect use cases for LineTAP Follow up on Data formats - parquet filesIVOA Roadmap for 2022A
This outlines the roadmap for development activities by the various IVOA working and interest groups between the Apr 2022 and Oct 2022 Interops.Applications WG
Python
- Support open development in PyVO
- Continue coordination meetings for maintainers, contributors and anyone interested.
- Add prototype features to PyVO: TAPPlus, New SSO
- Maintain votable parsing in Astropy core
Standards and Other Business
- Frequency or Energy inside MOC, MOC 2.0?
- In coordination with other groups, explore support for JSON. in IVOA
- Migrate HiPS and SAMP documents to Github
- Clean up applications WG wiki pages.
- Remove stale links and make sure all information is current
- VOTable 1.5
- Stimulate discussion of existing issues (VOTable issues)
- Science Platforms using VO-Standards (CTA, LSST, SKA ...)
Data Access Layer WG
- EPN-TAP [DM WG connection]
- RFC open
- Resolving remaining issues on MOC/ST-MOC support, extensions and vocabularies
- Aiming for REC in Q3-2022
- SIAP-2.0 / SODA-1.0
- Expanding to cover spectra and potentially measurement sets and catalogues
- One shot cutouts from SIA2
- Renaming SIA2 to reflect expanded role
- PRs open now for feedback
- New WD by Oct/Nov interop
- DataLink -1.1
- WD released Nov 2021
- this might set the line for v.1.1
- RFC by Oct/Nov Interop
- WD released Nov 2021
- LineTAP
- Intended to replace SLAP
- WD end of Q2-2022
- ADQL-2.1
- Ready for Proposed Rec (Q1 2022)
- Validation required before RFC
- RFC by Oct/Nov Interop
- ProvTAP
- Continuing work towards mature WD status
- Example implementation at CDS for Hubble HiPS
- DALI-1.2
- multipolygon still pending
- otherwise ready for PRec+RFC (Q4-2022)
- ConeSearch -1.1 [in connection with TD IG]
- work will progress to close issues
- a new WD before Oct/Nov 2022 Interop ?
- TAPRegExt [in connection with Registry WD]
- WD release in Q3/Q4-2022
- ObjVisSAP
- PR-1.0 nearly there
- implementations ready
- SSAP
- Spectroscopy data access could potentially lead to new minor revisions in advance of any integration with a more general SIA2
- Following Radio IG, TD IG activities for requirements
- refer to the roadmaps of these IGs for details
- file-oriented catalog data access: start of discussion on Parquet
Data Model WG
- DM workshop follow-up
- Meas/Coord-> Rec
- Mapping syntax common proposal
- -> RFC
- Work on libraries in progress
- MANGO to be updated
- Resume working on Cube/MetaData
- PhotDM REC
- Obscore evolution (TD, Radio)TBC
Grid and Web Services WG
- SSO - working towards next WD
- JSON-ification - small team of interested parties to look at what is easy and what is hard
- ExecPlanner - working towards first WD
- Science Platforms - continue collaboration with Theory IG, working towards 'hello world' prototypes.
Registry WG
- Expand registry search in pyvo for simplified data discovery options as proposed in pyVO PR #289 and test backward compatibility of search changes, document with new example notebook(s)
- Start implementing space-time coverage in registries as described in VODataService -1.2, pending approval of the PRs: DALI 1.2 (for MOC serialization) and RegTAP 1.2 (for the stc_spatial table)
- Investigate better inclusion of authoring data in the Registry resources (especially for tables) in order to improve automatic citation
- Coordinate with Ops IG to continue ongoing validation and curation efforts among registry maintainers, resource curators, and validator and standards authors
- Investigate registering more documentation and tutorial notebooks. Explore takeup and usefulness
Semantics WG
Semantics has quite a few documents in the pipeline at this point. First, we will rather quickly move the Note endorsing the Unified Astronomy Thesaurus for IVOA adoption to the TCG. Then, there are two maintenance releases for recommendations that ought to be ready for Working Draft early in this cycle: VOUnits 1.1 (minor updates, e.g., for missing units) and VocInVO 2.1 for the new github procedures. We expect that both of them can move on to PR by the southern spring interop. Progress with the UCD list 1.5 is harder to predict, but we expect to produce a PEN before the summer break. Several minor activities are expected, including pre-review changes to product-type in the context of the datalink review, the production of an erratum to the UCD List for SSAP (alternatively, the change could also go in via UCDList 1.5), or working on an interface between Wikidata and the IVOA for the Faciliites list. Finally, we will continue to process VEPs in the context of maintaining the vocabulary repository. By this activity's nature, we cannot predict at this point how much effort will be required for that.Data Curation Preservation IG
- Citation and basic Provenance in the VO - to specify Data Origin in VO output
- Good practices to provide Data in the VO
- Share Data Center experiences and report other sciences frameworks and networks
Education IG
2022- Improving the newcomer's session with regard of feedback. Eventual adaption of the Session to in person or hybrid meetings.
- Moving forward with the Edu note in Registry to make tutorial findable in the registry
- Updating the IVOA webpage in regard of VO publishing topics
- Finding a solution for putting votut-latex currently hosted on SVN at GAVO to ivoa github (Not as trivial as it sounds)2023/2024
- Presenting VO-publishing at AAS meeting (when in person meetings are possible)
- Organizing one week workshop/hackathon on VO publishing with the goal that participants will have a usable VO service running at the end of the workshop (when in person meetings are possible)
Knowledge Discovery IG
- ML-proofing existing and future science platforms:
- Are existing astronomy science platforms compatible with ML methods?
- Investigate whether science platforms can access tabular and non-tabular data through VO interfaces.
- Building libraries of well-established pre-trained models and integrating them in science platforms.
- Collecting resources for the "ready to ML in astronomy" kit
- Library of datasets: collecting data for standard DM application for testing and benchmarking of KD models ("Iris datasets" for astronomy)
- Looking at different "astronomy data challenges": can they represent the starting point for the library of astronomical datasets?
- Collecting and describing methods for standardization/normalization of data used in astronomy, with reference data to test different implementations of the methods
Operations IG
- Monitor VO health
- Coordinate weather reports (service compliance)
- Monitor metadata compliance (UCDs/VOUnits)
- Address non-compliancy
- Identify common/persistent errors
- Support service operators to improve service compliance
- Work with implementors on software-related issues (SoftID Note should help)
- Work with WGs on standards-related issues
- Validation
- Ensure validators are available for all IVOA standards where applicable
- Maintain list of validators (IvoaValidatorsSummary)
- Encourage use of validators
- on deployed services
- during standard development
- Authentication standards for VO API usage
- promote discussion among providers and standards groups regarding fairness of access and load management - authenticated access as a means to manage usage and trace/throttle/block API abusers
- Communication
- Provide forum for discussion of operational issues
Radio IG
- upgrading Radio Astronomy Implementation review note before next interop
- complete discussion on ObsCore extension for visibility data and publish it as a WD. Check consistency for optical interferometru
- Push the writing of a note for mapping pulsar or time dependant data to Obscore in collaboration with TDIG
- Demonstrate VO capabilities to MeerKAt and ThunderKat teams in collaboration with TDIG
- Push the writing of a note for mapping single dish (and beam forming ?) data to ObsCore
Solar System IG
- Top priority is getting EPN-TAP across the finish line.
- Observation facility codes are of high importance to both planetary and astrophysical users. We will pursue the Wikidata solution, which is very promising. PDS SBN has a vested interest in a good solution for this identification problem, and has offered support as needed for hosting, editing, testing, etc.
- There is untapped planetary data buried amongst astrophysical archives around the world. We will undertake an effort to get IVOA access for planetary data on the radar for these archives, starting with North America. To this end, we are planning at least one session at the fall Interop for which we invite representatives from astrophysics archives to present summaries of their planetary holdings and any plans they might have for implementing support for planetary IVOA queries.
Theory IG
Follow up April 2022 interop results, in a telecon before the summer:- work with volunteering science platforms to get simulation data stored. (GL, BM, DM, SO'T, ET)
- get code close to the data for their analysis
- get a SimDB running and register the simulation data sets on science platforms (PH)
Time Domain IG
- get regular group telecons restarted to gather project interests for the coming semester and beyond
- ObsCore:
- continue working with RIG to define mapping of radio and time dependent data to ObsCore
- TimeSeries data and discovery:
- work with DM to promote the development of the NDCube model, which forms the core for expressing TimeSeries and Spectral data.
Standard and Processes
- Transfer the IVOA assets in FAlRSharing
- Standard docs are missing DOIs in ADS over the past few years - work with ADS to get them assigned. (JE contacted AA - will fix)
Science Priorities
- Keep up with community feedback
- Participate to the running meetings
- After the interop: LineTAP
- Follow up on Data formats - JSON
- Publish in the VO: define the next steps
IVOA Roadmap for 2022B
This outlines the roadmap for development activities by the various IVOA working and interest groups between the Oct 2022 and May 2023 Interops.Applications WG
Standards and Related- In coordination with other groups, explore support for newer technologies in IVOA Standards
- Evolution of MOC adding frequency axe to MOC
- Following integration of VO-Standards into science plateform
- Migrate HiPS and SAMP documents to Github
- Clean up applications WG wiki pages.
- Remove stale links and make sure all information is current
- Support open development in PyVO
- Continue coordination meetings for maintainers, contributors and anyone interested.
- Maintain votable parsing in Astropy core
Data Access Layer WG
ADQL- 2.1 - Feature complete - validation being finalised
- 2.2
- Vector extension
- move from BNF to PEG for language specification
ADQL User Defined Functions
- ADQL astrometry library
- GAVO and Vizier implementation of coordinate transformation
- Functions added in pgsphere
TAP
- Allow ROT13 for encoding of ADQL query strings? SIAP-2.0 -> Data Access Protocol v1
- Expanding to cover spectra and potentially measurement sets and catalogues
- One shot cutouts from SIA2
- Renaming SIA2 to reflect expanded role
- Work continues in new GitHub repo: https://github.com/ivoa-std/DAP
- New WD by May interop SODA-1.1
- Errata for MOC parameter, rebin/project, pixel cutout
- One shot cutouts from SIA2
- Cross-join of parameters
- Metdata
- Select output by dataproduct_type DataLink 1.1
- Working draft out
- Needs implementations before RFC DALI-1.2
- multipolygon still pending
- otherwise ready for PRec+RFC (Q1-2023) SSAP
- Minor revision to
- make coverage information non mandatory
- Explain SSA utype prefix
- Longer term function is covered by DAP + SODA ConeSearch -1.1 [in connection with TD IG]
- work will progress to close issues
- a new WD before May 2023 Interop ? LineTAP
- Intended to replace SLAP
- WD before May 2023 Interop ? ProvTAP
- Continuing work towards mature WD status TAPRegExt [in connection with Registry WD]
- WD release in Q2-2023 ObjVisSAP
- PR-1.0 nearly there
- implementations ready Following Radio IG, TD IG activities for requirements
- refer to the roadmaps of these IGs for details
Data Model WG
MIVOT (model mapping syntax for VOTables)- Complete the REC step
- Providing a PyVO implementation MANGO (the missing piece in between MIVOT, PhotDM and MCT)
- REC step Field of view DM
- having a draft Observation Proposal DM
- having a draft CubeDM
- having drafts (including DatasetMetadata) + reference implementations Transform (3rd lettre of MCT)
- having a draft+ reference implementations Spectrum DM 2.1
- REC step
Grid and Web Services WG
- SSO - working towards next WD based on tokens and delegation.
- ExecPlanner - working towards first WD, splitting the note into three parts:
- The core ExedPlanner interface
- A related UWS interface for executing tasks
- A basic task description language
- Science Platforms - continue collaboration with Theory IG and KDD. What are the IVOA services to build a SP? Do we need some new standardization? What kind of interoperability are we looking for?
- VO in the cloud
- Looking at how we can publish data in Parquet format via S3 services.
- This is turning out to be much harder than expected.
- Looking at how we can publish data in Parquet format via S3 services.
- JSON-ification - small team of interested parties to look at what is easy and what is hard. Is it possible to move from XML to JSON? what is the effort needed?
- VOSpace - new usecases that also involves theory data and simulation archives will provide new requirements.
- CredDel - fundamental in the services orchestration should go for tokens
- New requirement for Resouce description in particular comuting and storage resources
Registry
- Continue support for pyvo and RegTAP compliance
- Continue development of new pyvo Registry search API (released on Sep 26, 2022 for pyvo 1.4)
- Start implementing space-time coverage in registries as described in VODataService -1.2, pending approval of the PRs: DALI 1.2 (for MOC serialization) and RegTAP 1.2 (for the stc_spatial table)
- Investigate better inclusion of authoring data in the Registry resources (especially for tables) in order to improve automatic citation: participate in discussions about DCP’s note “Data Origin in the VO”
- Coordinate with Ops IG to continue ongoing validation and curation efforts among registry maintainers, resource curators, and validator and standards authors
- Investigate registering more documentation and tutorial notebooks. Explore takeup and usefulness
- Discuss whether/how does cloud information belong in the registry and how to register information about multiple locations for finding data e.g., on-prem or on S3...
- Standards being worked on:
- StandardsRegExt -1.1
- in progress, started in Nov 2022 from 1.0
- contributors welcome: https://github.com/ivoa-std/StandardsRegExt
- DocRegExt (NEW) - https://github.com/ivoa-std/DocRegExt
- from Note "Educational Resources in the Virtual Observatory"
- support tutorials in several languages
- notebooks?
- VoResource -1.2
- in progress
- implement UAT-in-the-VO
- generalize altIdentifiers => DOIs
- RegTAP 1.2
- WD https://ivoa.net/documents/RegTAP/20220519/
- includes VODataService -1.2’s field for space/time/spectrum coverage
- interop issue bringing instances up-to-date
- StandardsRegExt -1.1
Semantics WG
On the REC track, Semantics still has the RFC for Vocabularies 2.1 (on github procedures) open. We will try to gather the missing WG reviews during November and thus hope we can pass the document on to the Exec in December; we do not see much potential for controversy in the new regulations. We also hope to release a Proposed Recommendation for VOUnits 1.1 before the end of 2022 and aim for RFC in January 2023. Given that the update addresses the difficult question of empty and missing units, that RFC could be more involved. The UCD list 1.5 should be available for TCG review right after the Interop. We hope that it will be endorsed at the next TCG meeting, or perhaps the one after that. Once that is done, the cycle for the UCD list 1.6 will begin in earnest. This will, in particular, comprise UCDs useful to annotate columns containing metadata on ambient conditions for observations. Beyond the Semantics WG's documents, we intend to drive along a new release of VOResource with the main goal of clarifying how the UAT is intended to be used within VOResource. Unless the Registry WG has additional items in this update, we expect this will be another easy update. Outside of the standards track, we still work on the product-type vocabulary that needs to be ready for Datalink 1.1. We also work on the facility nomenclature and hope to produce a note discussion the intended practices between the IVOA and wikidata before the next Interop. Finally, we will continue to process VEPs in the context of maintaining the vocabulary repository. By this activity's nature, we cannot predict at this point how much effort will be required for that.Data Curation Preservation IG
2023:- DOI : implementation, citation and good practices
- Data Origin in the VO: meta-data required to provide sufficient traceability to end-users in order to improve the understanding of the resultsets and enabling its reuse and its citation.
- Providing FAIR data
- Data preservation, TRUST principles and certification
- Data curation and Data Models
Education IG
2022- Improving the newcomer's session with regard of feedback. Eventual adaption of the Session to in person or hybrid meetings.
- Moving forward with the Edu note in Registry to make tutorial findable in the registry
- Updating the IVOA webpage in regard of VO publishing topics
- Finding a solution for putting votut-latex currently hosted on SVN at GAVO to ivoa github (Not as trivial as it sounds)
- Presenting VO-publishing at AAS meeting (when in person meetings are possible)
- Organizing one week workshop/hackathon on VO publishing with the goal that participants will have a usable VO service running at the end of the workshop (when in person meetings are possible)
Knowledge Discovery IG
- ML-proofing existing and future science platforms:
- Are existing astronomy science platforms compatible with ML methods?
- Investigate whether science platforms can access tabular and non-tabular data through VO interfaces.
- Building libraries of well-established pre-trained models and integrating them in science platforms.
- Collecting resources for the "ready to ML in astronomy" kit
- Library of datasets: collecting data for standard DM application for testing and benchmarking of KD models ("Iris datasets" for astronomy)
- Looking at different "astronomy data challenges": can they represent the starting point for the library of astronomical datasets?
- Collecting and describing methods for standardization/normalization of data used in astronomy, with reference data to test different implementations of the methods
Operations IG
Ongoing activity:- Monitor VO health
- Coordinate "weather reports" (service compliance)
- Address non-compliancy:
- Identify common/persistent errors
- Support/encourage service operators to improve service compliance
- Work with WGs on standards-related issues
- Validation:
- Ensure validators are available (IvoaValidatorsSummary)
- Encourage use of validators
- on deployed services
- during standard development
- Communication
- Provide forum for discussion of operational issues
- Host/invite "site reports"
- Work with DAL on standards issues:
- ConeSearch: UCD1+
- DALI: Sexagesimal markup standards
- SSA: A couple of Errata suggested
- Push for service metadata improvements:
- VOUnits, UCDs
- Upcoming themes:
- Operations in the cloud
- Authentication as an operational necessity
Radio IG
-
- Pushing Radio data extension (inteferometry/visibility and single dish) to Working draft status
- Update implementation review note
- work out the Pulsar/FRB radio data in the VO note
- Continue work on python notebooks for finding and downloading radio data through the VO.
Solar System IG
- Continue development on Observation Facility Codes
- Open Science initiatives are putting pressure on astrophysical observatories to accommodate planetary uses. We will highlight success stories in Interop meetings to encourage progress.
- Continue development on Observation Facility Codes
- Get some prototype panetary reference frames incorporated into the Reference Frames vocabulary
- Ensure convergence of the EPN-TAP dataproduct_type with the Data Product Type vocabulary.
- Support PDS/SBN efforts to exercise EPN-TAP and planetary additions to related protocols by implementing IVOA services on small data sets from several different planetary disciplines.
Theory IG
- Establish register of publicly available simulation datasets, and their current FAIRness
- Review SimDM, including how easy is it to implement it on existing systems
- Investigate the possibility of a TAP service for simulations, incorporating SimDM or otherwise
- Continue collaboration with GWS on Science Platforms and their role in simulation generation and distribution
Time Domain IG
- Monitor and support DM efforts towards representing TimeDomain data with the Cube model.
- Continue working with Radio IG on FRB project.
Standard and Processes
Science Priorities
Terms : https://wiki.ivoa.net/twiki/bin/view/IVOA/IvoaSciencePriorities- Motivate further discussions on Science platforms and the role of the IVOA
- Follow-up on RFE of Spectrum DM
- Advocate for DataLink usage
- First discussions on a High Energy Interest Group
- Representation of the IVOA at different meetings:
- Discussions on the evolution of the community on Alerts
- Discussions on scientific analysis usage of HiPS
IVOA Roadmap for 2023A
This outlines the roadmap for development activities by the various IVOA working and interest groups between the May 2023 and Nov 2023 Interops.Applications WG
Standards and Related- Support VOTable 1.5 (possibly get to RFC)
- Evolution of MOC adding frequency axis to MOC
- Following integration of VO-Standards and comon applications into science platform
- Migrate HiPS and SAMP documents to Github
- Clean up applications WG wiki pages.
- Remove stale links and make sure all information is current
- Support open development in PyVO
- Maintenance, new features? Organize meetings with developers, mainteners, users
- Maintain votable parsing in Astropy core
Data Access Layer WG
Primary:ADQL
- 2.1 RFC - complete and get to REC
- 2.2 - PEG, INTERVAL support, MOC support, Array support, string functions, OFFSET
- RFC open - complete and get to REC
- PRec + RFC mid 2023
- User managed persistent tables
- Experiment with Open API spec for these changes
- RESPONSEFORMAT, MAXREC
- Allow services to reject unknown parameters
- Modernize required UCDs (UCD 1.0-> UCD 1+)
- PRec by next interop
- Proceed with renaming, expansion of supported data product types
- Miscellaneous changes
- Moving to WD status
- return extended metadata
- New transformation types
- Moving to WD status
LineTAP
- Client and server implementations progressing
- Raise PRec once implementations ready
- Examine renaming to avoid radio confusion
- Restart effort to complete the standard and get to PRec
- Issue errata for minor issues already noted in github
- Can functions be reconciled between GAVO and OAJ?
- Continuing work towards mature WD status
Data Model WG
Primary: Spectrum 1.2 RFE:- confirm implementations are sufficient for RFC; possibly drop 'relorder' attribute
- proceed to RFC
- implement python workflow scripts as jupyter notebooks
- proceed to RFC
- working implementations
- meeting in September
- goal: RFC ready for next interop
- reducing content/scope; working implementation
- goal: concrete WD for next interop
- revise model per discussion in Bologna interop
- define and execute appropriate implementations (in context of TimeSeries, Cube, Mango)
- goal: concrete WD for next interop
- Meeting June 28
- evaluate and report on compatiblity of current models (eg: Cube) with High Energy data.
- evaluate and report on possible ObsCore extension, following Radio example.
- identify workflow threads to exercise models in this context
- finalize Note, see if this can become a WD
- as 'view of Provenance' similar to ObsCore
- integrate toolkit usage into model repositories
- utilize to generate code package of vo-dml compliant models.
- work on Dataset, and High Energy should define additional workflow threads which exercise this model in other contexts.
- goal: implement any defined workflow cases
- VO-DML compliant version of Characterisation model
- goal: WD by Spring 2024
- formalizing model basis for Aladin FoV format used by STScI and others
- goal: no stated goals this semester
- continue discussions with CADC about how this model fits into the IVOA ecosystem, if at all.
Grid and Web Services WG
Pending upgrades Pending Update SSO 2.1, including issues at:https://github.com/ivoa-std/SSO/issues
Also, reformat of GitHub repository. Cross-science platforms execution project The GWS WG is going to start a project to identify the main changes and new standards to allow remote execution on different astronomical science platforms.
- discover computing services, answering the question "what computing services are available and suitable to run my task?"
- access computing services, answering the question "how can I run my task on the selected computing service?"
https://github.com/ivoa/ExecutionPlannerNote
This note should be either converted into a standard or, at least, upgraded making it a more powerful interface. To do that, it is necessary to standardise:
Characterisation of software (in terms not only of software dependencies but, also, dependencies that could affect execution like memory, requirements on computing architecture (e.g. CPUs and GPUs), data location, etc. Software should be exposed in a portable format
Metadata characterisation of the science platform node is also a requirement
Access to this metadata should be also characterised as per the definition of a software discovery service and a science platform resources discovery service. The execution planner could use metadata of both types to decide the best place to execute a certain workflow.
For the software characterisation and discovery, other CI/CD solutions could be expanded and already have the connections between code and containers. For the science platform node service, we need to have a dynamic service to capture the real-time status of the resources.
The execution planner and the science platform discovery service could be integrated into the same service and the software discovery looks to be a different one. Endorsement of a standard for workflow serialisation Identify one standard for workflow serialisation (e.g. CWL) and endorse it for the IVOA. This format should have software to parse and decompose a workflow in smaller executable elements in different languages if possible. Execution interface Identify an execution interface that could abstract the internal execution interfaces of the different science platforms. Probably something based on IVOA UWS could be used to control execution Credentials Delegation Review of related standards for authentication/authorisation combined with the execution interface. This could be a problem for a general approach as some computing centres have their local access policies For this exercise, SKA Regional Centre Network members are available to collaborate, lead and implement although it would be also needed to have members of other projects that are implementing (or already have) running science platforms to create another independent implementation (candidates Vera Rubin, ESA, CADC, Gaia UK,...)
Plan for 2023:
- Update Execution Planner note to convert it to a formal draft, including a more complex definition
- Draft for November meeting
- Study on workflow standards
- Draft for November meeting
Registry WG
- Continue developing and maintaining the new pyvo Registry search API (released in 2022 for pyvo 1.4)
- Announce and perform deprecation of the Registry Interface 1.0 old client search functionality on clients and servers
- Standards being worked on:
- VOResource-1.2 (editor: MD)
- https://github.com/ivoa-std/VOResource
- Status: ready to go for PR
- StandardsRegExt -1.1 (editor: RS)
- https://github.com/ivoa-std/StandardsRegExt
- Status: ready to go for PR
- DocRegExt -1.0 (editor: MD)
- https://github.com/ivoa-std/DocRegExt
- Goal: registering educational resources: tutorials, use cases, courses
- Status: currently WD
- VOResource-1.2 (editor: MD)
- Facilitate discovery of data collections by product type (eg “time series”)
- Bring ObsCore dataproduct_types into the VODataService Registry records via update of VODataService
- Explore the possibility to add a mandatory tableset for CatalogService
- Curation and validation of the Registry
- Coordinate with Ops IG to continue ongoing validation and curation efforts among registry maintainers, resource curators, and validator and standards authors
- Implement actions from last Registry curation task force meeting and organize follow-up meetings
- Explore feasibility of a VOResource linter using schematron or/and checks with in Python (xpath & libxml2) to handle constraints (eg author names syntax)
- Create Registry WG + EDU IG collaboration to help/teach data publishers register their services
- Organize brainstorming meeting with Rubin Obs
- Continue discussing whether/how does cloud information belong in the registry and how to register information about multiple locations for finding data e.g., on-prem or on S3...
Semantics WG
- VOUnits 1.1 is PR since May 2023. Given that the update addresses the difficult question of empty and missing units, that RFC could be more involved.
- The UCD list 1.6 should be available for TCG review right after the Interop. We hope that it will be endorsed at the next TCG meeting, or perhaps the one after that.
- Once that is done, the cycle for the UCD list 1.7 will begin in earnest. This will, in particular, comprise UCDs useful to annotate columns containing metadata on ambient conditions for observations.
- Beyond the Semantics WG's documents, we intend to drive along a new release of VOResource with the main goal of clarifying how the UAT is intended to be used within VOResource. Unless the Registry WG has additional items in this update, we expect this will be another easy update.
- Outside of the standards track, we still work on the product-type vocabulary that needs to be ready for Datalink 1.1. We also work on the facility nomenclature and hope to produce a note discussion the intended practices between the IVOA and wikidata before the next Interop. Linking with semantic ecosystem of other communities, and making our vocabularies more interoperable with the rest of the science community.
- Finally, we will continue to process VEPs in the context of maintaining the vocabulary repository. By this activity's nature, we cannot predict at this point how much effort will be required for that.
Data Curation Preservation IG
- Good practices to provide Data in the VO -
- Provide a DOI reference note - (to improve Data citation)
- Promote Data Origin (see http://ivoa.net/documents/DataOrigin/) to working Group (DALI?) in order to have a basic Provenance in the VO
Education IG
2023- Improving the newcomer's session with regard of feedback from the Bologna Interop meeting. Suggestion is to have a rather technical Session online a week before the Interop meetings, and a rather organisational/social Session at the in person meetings covering topics like * How to join WGs (turn up to the Session, subscribe to mail list...) * How to join a mail list * Explain how the working groups work together at the example of a specific topic of the running interop. * One question raised was that the established IVOA people seem to be friends, which makes it harder for newcomers to fit in. This might be influenced by Bologna being the first in person meeting after 3.5 years. Yet, maybe it's possible that we have a handful of volunteers who would attend the newcomers session and introduce themselves
- Moving forward with the Edu note in Registry to make tutorial findable in the registry
- Updating the IVOA webpage in regard of VO publishing topics
- Finding a solution for putting votut-latex currently hosted on SVN at GAVO to ivoa github (Not as trivial as it sounds)
- Presenting VO-publishing at AAS meeting (when in person meetings are possible)
- Organizing one week workshop/hackathon on VO publishing with the goal that participants will have a usable VO service running at the end of the workshop (when in person meetings are possible also see RegistryWG)
Knowledge Discovery IG
- ML-proofing existing and future science platforms:
- Are existing astronomy science platforms compatible with ML methods?
- Investigate whether science platforms can access tabular and non-tabular data through VO interfaces.
- Building libraries of well-established pre-trained models and integrating them in science platforms.
- Designing a questionnaire to collect user requirements for science platforms to support ML methods and see how it fits into the plan of science platform from the developers' and relevant service providers' side.
- Artificial Intelligence and Large Language Models in the VO
- Coordinating an IVOA, inter-group "tiger team" that will scope the current and future potential impact of AI on VO.
- Collecting use cases and requirements for the integration of commercial and/or ad hoc LLMs models in IVOA-relevant topics.
- Investigating best practices to integrate LLMs with VO tools for a fusion of domain knowledge and data.
Operations IG
Ongoing activity:- Monitor VO health
- Coordinate "weather reports" (service compliance)
- Address non-compliancy:
- Identify common/persistent errors
- Support/encourage service operators to improve service compliance
- Work with WGs on standards-related issues
- Validation:
- Ensure validators are available (IvoaValidatorsSummary)
- Encourage use of validators
- on deployed services
- during standard development
- Communication
- Provide forum for discussion of operational issues
- Host/invite "site reports"
- Work with DAL on standards issues:
- ConeSearch: UCD1+
- DALI: Sexagesimal markup standards
- SSA: A couple of Errata suggested
- Push for service metadata improvements:
- VOUnits, UCDs
- Upcoming themes:
- Operations in the cloud
- Authentication as an operational necessity
Radio IG
- Pushing ObsCore Radio data extension (interferometry/visibility and single dish) forwards by solving issues and pushing implementations
- Update implementation review note with recent progress and additional implementations (projects such as NRAO, GBT, SKA, IRAM, SpanishVO, etc...)
- Upgrade the Pulsar and FRB Radio Data Discovery and Access note
- Continue work on integrating radio services and software in the VO and specifically in platforms (notebooks, clusters, DataLakes) for finding, downloading and processing radio data through the VO.
Solar System IG
- Continue development on Observation Facility Codes
- Get some prototype planetary reference frames incorporated into the Reference Frames vocabulary
- This tutorial explains how the Navigation and Ancillary Information Facility (NAIF) software defines (and uses) reference frames and coordinate systems: NAIF Frames and Coordinate Systems Tutorial
- The list of IAU defined body-fixed frames is included in the NAIF software documentation on frames, as an appendix: direct link to "frames.html" second appendix
- Ensure convergence of the EPN-TAP dataproduct_type with the Data Product Type vocabulary.
- Support PDS/SBN efforts to exercise EPN-TAP and planetary additions to related protocols by implementing IVOA services on small data sets from several different planetary disciplines.
Theory IG
Time Domain IG
Priority in this first semester is to review status of recommendation level and working draft level documents of relevance for time domain, and decide on priorities based on current status of the field of time domain astrophysics. In particular, upcoming surveys (LSST, LIGO) and the increase number of transient alerts across wavelengths (including high energy transients, of relevance in multi-messenger science), as well as the emergence of spectral time-domain surveys, such SDSS-V, set the following priorities:
-
VOEvent 2.1 working draft: new version coordinated by DAL, but the TD IG should follow and read the draft to provide feedback. The current, stable version of VOEvent is at https://www.ivoa.net/documents/VOEvent/20110711/
-
Evaluate if new version of VOEvent requires updates to the VOEvent Transport Protocol => Protocol for publishing VOEvent => https://www.ivoa.net/documents/VOEventTransport/20170320/index.html
-
Pick up discussion on Time Series Cube Data Model (2019), to get status, establish if it leads to a recommendation, etc. Does the datacube only have a time dimension, or also a spectral dimension? Of relevance for upcoming spectral time domain surveys. We need to ensue that it also fits high energy data (i.e. event files) => https://www.ivoa.net/documents/Notes/CubeDM/20190318/index.html
-
Status of (S)TMOC => Space and Time coverage => Does this require updates? Is it currently being used? https://www.ivoa.net/documents/MOC/20220727/index.html
-
Cone search service that allows for time constraints
-
Other, dormant documents: VOEventRegExt => Extension of the VO registry to describe a service providing VOEvents (dormant still 2014 ); STC-S => Space Time coordinate linear definitions (dormant still 2013)
Standard and Processes
<!--* Set ALLOWTOPICRENAME = TWikiAdminGroup
-->
IVOA Roadmap for 2023B
This outlines the roadmap for development activities by the various IVOA working and interest groups between the Nov 2023 and May 2024 Interops.Applications WG
Standards and Related:- Get VOTable 1.5 to RFC
- Evolution of MOC adding frequency axis to MOC
- Follow integration of VO-Standards and comon applications into science platform
- Migrate HiPS and SAMP documents to Github
- Clean up applications WG wiki pages.
- Support open development in PyVO (release 1.5)
- Support for MIVOT integration in PyVO
- Maintenance, new features? Organize meetings with developers, mainteners, users
- Maintain votable parsing in Astropy core (add support for VOTable 1.5)
Data Access Layer WG
Primary: ADQL- 2.2 - PEG, INTERVAL support, MOC support, Array support, string functions, OFFSET
- PRec by Sydney
- User managed persistent tables
- Experiment with Open API spec for these changes
- RESPONSEFORMAT, MAXREC
- Allow services to reject unknown parameters
- Modernize required UCDs (UCD 1.0-> UCD 1+)
- WD by Sydney (PR pending validators and implementations)
- Proceed with renaming, expansion of supported data product types
- Miscellaneous changes
- WD by Sydney
- return extended metadata
- New transformation types
- WD by Sydney
- Client and server implementations progressing
- Raise PRec once implementations ready
- Work restarting
- New WD and perhaps PRec by Sydney
- Issue errata for minor issues already noted in github
- Can functions be reconciled between GAVO and OAJ?
Data Model WG
Primary: Spectrum 1.2 RFE:- With exec approval
- Update abstract with statement of model update scope
- create/update issues for future enhancements ( vocabularies, VOUnit, invalid UCD sweep)
- correct typo: PARAMs with no datatype
- Find/update spec lib implementation
- implement python workflow scripts as jupyter notebooks
- proceed to RFC
- DM review of WD; organize resolution of any open questions.
- continue refinement of content and implementations
- detail stop of V1.0 of document
- goal: concrete WD for next interop
- revise model per discussion in Bologna interop
- define and execute appropriate implementations (in context of TimeSeries, Cube, Mango)
- goal: concrete WD for next interop
- Refine Note in support of HEIG creation
- continue informal discussions of the 'club'
- goal: formation of HE Interest group
- finalize Note, see if this can become a WD
- as 'view of Provenance' similar to ObsCore
- integrate toolkit usage into model repositories
- utilize to generate code package of vo-dml compliant models.
- work on Dataset, and High Energy should define additional workflow threads which exercise this model in other contexts.
- goal: implement any defined workflow cases
- VO-DML compliant version of Characterisation model
- goal: WD by Spring 2024
- formalizing model basis for Aladin FoV format used by STScI and others
- goal: no stated goals this semester
- continue discussions with CADC about how this model fits into the IVOA ecosystem
Grid and Web Services WG
- Execution Broker (Execution Planner)
- VOSpace
- SSO
- UWS
- Working Group new Name
Converging to "Distributed Services and Protocols" - proposed by Dave Morris
Registry WG
- Continue developing, maintaining and testing the new PyVO Registry search API
- Follow up discussion in Tucson’s Registry thematic session about “Python as a registry client for the end users”: https://wiki.ivoa.net/internal/IVOA/InterOpNov2023Registry/interop_202311_pyvo.pdf and https://wiki.ivoa.net/internal/IVOA/InterOpNov2023Registry/InterOpNov2023Registry_Notes-365days.pdf
- Investigate resource ranking in results
- Facilitate discovery of data collections by product type (eg “time series”)
- cf work on VODataService-1.3 (see standards section below)
- Curation and validation of the Registry
- Coordinate with Ops IG to continue ongoing validation and curation efforts among registry maintainers, resource curators, validators and standards authors
- Explore feasibility of a VOResource linter using XSD (vs schematron?) or/and checks with in Python (xpath & libxml2) to handle constraints (eg author names syntax)
- Continue WG + EDU IG collaboration to help/train data publishers register their services
- Follow-up of online brainstorming mtg of 2023-10-03, cf report http://mail.ivoa.net/pipermail/registry/2023-October/005545.html and notes http://mail.ivoa.net/pipermail/registry/attachments/20231014/414c4998/attachment-0003.pdf
- Continue involvement with DCP IG, esp. on the note “Data Origin and the VO” https://www.ivoa.net/documents/DataOrigin/
- Continue discussing whether/how does cloud information belong in the registry and how to register information about multiple locations for finding data e.g., on-prem or on S3...
- Follow-up on deprecation of the Registry Interface 1.0 old client search functionality on clients and servers
- discussed in Bologna cf notes https://wiki.ivoa.net/internal/IVOA/InterOpMay2023Registry/InterOpMay2023Registry-Notes.pdf
- Standards currently worked on (in order of priority):
- RegTAP-1.2 (editor: MD) https://github.com/ivoa-std/RegTAP
- storing coverage information (described in VOResource-1.2)
- WD published https://ivoa.net/documents/RegTAP/20220519/
- WD announced http://mail.ivoa.net/pipermail/registry/2022-May/005495.html
- Goal: PR then RFC in 2024
- VOResource-1.2 (editor: MD) https://github.com/ivoa-std/VOResource
- UAT-in-the-VO adoption
- licenses: use SPDX URIs for rightURIs
- generalizing altIdentifiers:
- doi is now a recognized identifier scheme (along with orcid and bibcode)
- altIdentifier is now an attribute of vr:ResourceName => data providers can use DOIs when declaring relationships.
- WD published https://ivoa.net/documents/VOResource/20230105/index.html
- VODataService-1.3 (editor: MD) https://github.com/ivoa-std/VODataService
- Introducing productTypeServed: cf http://mail.ivoa.net/pipermail/registry/2024-January/005550.html
- StandardsRegExt-1.1 (editor: RS) https://github.com/ivoa-std/StandardsRegExt
- Summary and issues corrected: https://wiki.ivoa.net/internal/IVOA/InterOpMay2023Registry/InterOpMay2023Reg_StandardsRegExt-1.1.pdf
- WD published https://ivoa.net/documents/StandardsRegExt/20230921/index.html
- WD announced http://mail.ivoa.net/pipermail/registry/2023-October/005547.html
- DocRegExt-1.0 (editor: MD) https://github.com/ivoa-std/DocRegExt
- RegTAP-1.2 (editor: MD) https://github.com/ivoa-std/RegTAP
Semantics WG
- ObsFacility: propose draft vocabulary + publish IVOA note(s) see 2022 presentation
- EPNcore: Prepare lists of terms in vocabularies (directly RDF/XML). See EPNTAP REC and EPNTAPV20RFC
- IHDEA: Coordination on helio reference frames and work on VEP.
- UCDList RFM 1.6 : simplify process + UCD as a IVOA vocabulary
- Linked-data exploration : onto-portal prototype (UAT, DCAT, RDF Graph on small subset of Registry/epncore/datalink resources)
- Product-type vocabulary: still some work to do before we converge…
Data Curation Preservation IG
- Good practices to provide Data in the VO -
- Provide a DOI reference note - (to improve Data citation)
- Promote Data Origin (see http://ivoa.net/documents/DataOrigin/) to working Group (DALI?) in order to have a basic Provenance in the VO
- Investigate standards use in Open Science and their potential usage in the VO (eg: DCAT)
Education IG
Knowledge Discovery IG
- ML-proofing existing and future science platforms:
- Are existing astronomy science platforms compatible with ML methods?
- Investigate whether science platforms can access tabular and non-tabular data through VO interfaces.
- Building libraries of well-established pre-trained models and integrating them in science platforms.
- Designing a questionnaire to collect user requirements for science platforms to support ML methods and see how it fits into the plan of science platform from the developers' and relevant service providers' side.
- Artificial Intelligence and Large Language Models in the VO
- Coordinating an IVOA, inter-group "tiger team" that will scope the current and future potential impact of AI on VO.
- Collecting use cases and requirements for the integration of commercial and/or ad hoc LLMs models in IVOA-relevant topics.
- Investigating best practices to integrate LLMs with VO tools for a fusion of domain knowledge and data.
Operations IG
Ongoing activity:- Monitor VO health
- Coordinate "weather reports" (service compliance)
- Address non-compliancy:
- Identify common/persistent errors
- Support/encourage service operators to improve service compliance
- Work with WGs on standards-related issues
- Validation:
- Ensure validators are available (IvoaValidatorsSummary)
- Encourage use of validators
- on deployed services
- during standard development
- Communication
- Provide forum for discussion of operational issues
- Host/invite "site reports"
- Work with DAL on standards issues:
- ConeSearch: UCD1+
- DALI: Sexagesimal markup standards
- SSA: A couple of Errata suggested
- Push for service metadata improvements:
- VOUnits, UCDs
- Upcoming themes:
- Operations in the cloud
- Authentication as an operational necessity
Radio IG
- Complete and implement (2 implementation) the radio Obscore Extension under the resonsability of DM WG
- Improve Pulsar Radio data Note in collaboration with TDIG
- Start version 2 of Implementation review note
Solar System IG
- Continue development on Observatory Facility Codes
- Vocabulary Development:
- Reference Frames
- Data Product Types
- Planetary Object Types
Theory IG
Time Domain IG
Priority in 2023B is to continue the discussions and review of the VOEvent standard, and align it with current practices for transient alert reporting, Expertise from ZTF, and broker alerts for Vera Rubin will be reviewed and hopefully incorporated into a new version of the standard.
-
VOEvent 2.1 working draft: new version coordinated by DAL, but the TD IG should follow and read the draft to provide feedback. The current, stable version of VOEvent is at https://www.ivoa.net/documents/VOEvent/20110711/
-
Pick up discussion on Time Series Cube Data Model (2019), to get status, establish if it leads to a recommendation, etc. Does the datacube only have a time dimension, or also a spectral dimension? Of relevance for upcoming spectral time domain surveys. We need to ensue that it also fits high energy data (i.e. event files) => https://www.ivoa.net/documents/Notes/CubeDM/20190318/index.html
- Time search postponed in the Cone Search standard to the next major release
- Evaluate synergies with HAPI protocol for heliophysics time domain data (https://vires.services/hapi/)
- Form Galluzzi's talk at interop: radio time/frequency datasets. Similar to GW transients. Are there standards for this?
-
Status of (S)TMOC => Space and Time coverage => Does this require updates? Is it currently being used? https://www.ivoa.net/documents/MOC/20220727/index.html
-
Cone search service that allows for time constraints
-
Other, dormant documents: <a href="https://wiki.ivoa.net/twiki/bin/edit/IVOA/VOEventRegExt?topicparent=IVOA.2023ARoadmap;nowysiwyg=0" rel="nofollow" title="VOEventRegExt (this topic does not yet exist; you can create it)"> VOEventRegExt </a> => Extension of the VO registry to describe a service providing VOEvents (dormant still 2014 ); STC-S => Space Time coordinate linear definitions (dormant still 2013)
Standard and Processes
<!--
* Set ALLOWTOPICRENAME = TWikiAdminGroup
-->
Registry Working Group : May 2024, 16:00-17:30, C122
* back to main programme page]*Schedule
| Registry Session: Tuesday May 21 2024 @ 16:00-17:30 (Session #7) Room C122 | |||
|---|---|---|---|
| Speaker | Title | Materials | Time |
| Tess Jaffe | Brief Registry Orientation | ||
| Pat Dowler | update on a re-usable OAI publishing registry | 10 + 5 | |
| Gilles/Renaud | Update on linking capabilities with tablesets | Slides Minutes | 10 + 5 |
| Markus+TJ | productTypeServed in VODataService | lecture notes | 10+5 |
| TJ | Registry spring cleaning Following up on Markus' Confessions of a Registry Janitor, I propose a yearly spring cleaning effort. I'll describe what I have in mind and show however far I get by then. |
10 + 5 | |
| Discussion | |||
Relevant Links and Information
- Moderator: TessJaffe
- Note Taker: RenaudSavalle
- Collaborative Notes: pad
Other sessions of interest
- TBD
IVOA Roadmap for 2024A
This outlines the roadmap for development activities by the various IVOA working and interest groups between the May and November 2024 Interops.Applications WG
Data Access Layer WG
Primary-
TAP
-
1.2 - Experiment with Open API spec for these changes (context: P3T)
-
-
DALI
-
Polygon definition clarification
-
PR-1.2 by Malta
-
-
ADQL
-
WD-2.2 by Malta (PEG grammar only)
-
-
SODA
-
return extended metadata
-
New transformation types
-
WD-1.1 by Malta
-
-
SIA
-
Issue errata for minor issues already noted in GitHub
-
-
SSA
-
Issue errata for minor issues already noted in GitHub
-
-
DAP (Data Access Protocol)
-
SIAP-2.0 into DAP
-
Gregory D.-F. become the 2nd Editor with Pat D.
-
Proceed with renaming, expansion of supported data product types
-
Miscellaneous changes
-
WD-1.0 by Malta
-
-
User Defined Functions catalogue
-
Ready to be endorsed
-
-
Conesearch
-
Modernize required UCDs (UCD 1.0-> UCD 1+)
-
RESPONSEFORMAT,MAXREC -
Allow services to reject unknown parameters (work to be covered by P3T)
-
WD-1.1 by Malta (PR pending validators and implementations)
-
-
LineTAP
-
Work to continue
-
Status about client and server implementations
-
Raise PRec once implementations ready
-
-
ObjObsSAP
- Work to continue
-
WD-1.0 by Malta
-
TAP
-
1.3 - User managed persistent tables
-
-
ADQL
-
2.3 -
INTERVALsupport, MOC support, Array support, string functions,OFFSET
-
Data Model WG
Grid and Web Services WG
Registry WG
Semantics WG
Data Curation Preservation IG
Education IG
Knowledge Discovery IG
Operations IG
Radio IG
Solar System IG
Time Domain IG
Standard and Processes
<!--* Set ALLOWTOPICRENAME = TWikiAdminGroup
-->
The ADASS XXII Guide to Eating in the Champaign-Urbana Area
in progress Find short descriptions of some of the many eating choices available below.|
View Eating for ADASS: Overview in a larger map |
 -- Notable for quality or convenience  --
Region containing multiple restaurants  -- Conference hotels  -- Conference Center
|
Hey Vegetarians!
Many of the restaurants described below are vegetarian and vegan-friendly. Cross-reference it with this listing from VegGuide.org.- Restaurant codes used in descriptions below
- * -- Recommended for quality or convenience
L -- Lunch only
A -- Serves Beer, wine, and/or spirits.
$ -- Reasonable Cost, particularly for Lunch
$$ -- A little more expensive, more appropriate for dinner
Near the Hotels: Kirby Avenue and Neil Street
I know, it's mostly chain restaurants within walking distance of the conference, but there are a few gems in there.
- Merry Ann's Diner (* L $)
- This Champaign mainstay is the real-deal--everything you want from a true American diner. Open Late.
Find it: from the hotels, it's just across Neil Street on the far north corner with Kirby. - Espresso Royale (* $)
- Coffee drinkers: If you are not staying at the I-Hotel, you wil probably be stopping here for a proper cupa.
Find it: On the hotel side of Neil street next door to the Hilton Garden Inn. - Cactus Grill (* $$)
- Not the same ol' Mexican
Find it: Next door to Espresso Royale. - Jarling's Custard Cup (* $)
- Who cares that it's November--this is the best frozen custard in town!
Find it: Walk west on Kirby Ave and find it on the south side, past State St - Seaboat (* $$)
- The best fish-and-chips in town by folks who care about fresh fried fish. Throw in some soulful sides like fried okra or sweet potato pie. Worth the slightly longer walk.
Find it: Walk west on Kirby Ave and find it on the south side, just past Jarlings and Vinny's Pizza.
East Campus/Krannert Neighborhood
If you want to catch a show at either the Krannert Center for Performing Arts or the Canopy Club, know that there are some nice places to get food and/or coffee before or after the show. You can get to this area via the Gold bus.
- The Bread Company (* A $/$$)
- European-style bakery and cafe. Lunch features fresh sandwiches, soups, and salads; dinner features European-style plates, fondue, pizzas, and pasta with a decent beer and wine list.
Find it: on the east (left) side of Goodwin Ave. just past the 4-way stop at Oregon.
- Timpone's (* A $$)
- Northern Italian with a great wine list.
Find it: on the east (left) side of Goodwin Ave. past the The Bread Company.
- Manolo's Pizza and Empanadas (* L $)
- What's better than a slice after a fun night on the town? Empanadas, is what. This place is awesome and open late; but be warned, at 1am, there may be a line.
Find It: This is just 2 doors down from the coffee shop on the corner on Oregon.
- The Red Herring (* L $)
- Vegetarian/Vegan "hole-in-the-wall"; worth the walk.
Find it: At the 4-way stop at Oregon. turn right. It's located on your left in the basement of its building.
Coffee
- Espresso Royale (* $)
- A great place to get an espresso after dinner.
- Intermezzo Cafe ($)
- This cafe is located inside the Krannert Center. The best thing about this cafe are the desserts; however, the center itself is fun to check out and getting coffee is a good excuse to check it out.
ADASS XXI BoF : Scientific Workflows in Astronomy
Presentations
- Introduction and Workflows@IVOA, André Schaaff
- Wf4Ever, Juande Santander Vela
- IceCore, Sami Maisala
- UVES workflow, Pascal Ballester
- Different Workflow levels in a project, Christian Surace
- OWL, Francesco Pierfederici
Discussion points
- How to boost the creation / sharing of Wfs knowledge in our community ?
- People could be reluctant to fill with care the information required as part of the workflow preservation (data used, services, references, etc.) or to make the effort to ensure that it is reusable by others
- Wf4Ever is a European project with a limited time life
- How to preserve Wfs on the long term ? What kind of things (user datasets, methodology, tools for backward compatibility, Licenses??, etc.)
- What can we learnt / what have we learned from other communities ?
- Publishing Wfs : review methods (peer review, volunteer, etc.)
- What parameters would you use to rate a Wf ? Or which ones would you like a rating system to provide you, so that you can select based on their quality / rating ?
Questions and Comments
- Eric N. (Fermi National Accelerator Laboratory): R has facility for reproducible research
- Data linked with method in same document
- Alberto A. (CfA, ADS): big difference between high throughput, and group of people working on different smaller steps;
- In Wf4Ever we are eating our own dog food
- Petr S.: we are moving in spirals, mainframes, PCs, clouds
- Services providers then have to bore the load of providing those computing facilities
- Doug M. (SAO): Workflows as mean of structured communication between users and facilities
- Rating workflows:
- rate according to how can it hold delta changes in data
- Adriaan R. (ASTRON): there is no way astronomers trust someone other than themselves. *having ways to assess authorship might allay some concerns.
The ADASS-er's Guide to Nightlife in Champaign-Urbana
We're in the heart of the semester right now, so there's quite a bit going on about town. For a more complete run down of happenings you might try these local community calendars:- The News-Gazette Events Calendar
- the217.com
- the calendar for the Krannert Center for Performing Arts
Theater and Comedy
Sunday, Nov. 4
- Theater: A Dream Play at the Krannert Center for the Performing Arts. 3pm matinee, $16.
- Take a bus to Campustown (2:16pm), getting off at the Transit Center. Walk east through the campus to Goodwin Avenue, and you will find Krannert a head of you.
- Theatre: Hamlet at the Station Theatre, featuring a woman in the title role and a modern setting. 8pm, $10.
- Take a bus to downtown Urbana; a Yellow North bus at 7pm from the E-14 lot stop, transfering to the Green at 7:24 in Campustown should have there comfortably on time. Leave earlier to get a BBQ dinner and a beer at Black Dog ahead of time.
Monday, Nov. 5
- Improv: Abe Froman Project, a local improv troupe @ Mike'N'Molly's
Wednesday, Nov. 7
- Theatre: Hamlet at the Station Theatre, featuring a woman in the title role and a modern setting. 8pm, $10.
- Take a bus to downtown Urbana; the last Gold East bus at 6:45pm from the E-14 lot stop will give you just enough time to get a BBQ dinner and a beer at Black Dog ahead of time.
Thursday, Nov. 8
- Theatre: Hamlet at the Station Theatre, featuring a woman in the title role and a modern setting. 8pm, $10.
- Take a bus to downtown Urbana; the last Gold East bus at 6:45pm from the E-14 lot stop will give you just enough time to get a BBQ dinner and a beer at Black Dog ahead of time.
Music
Astronomical Data Query Language
The Astronomical Data Query Language (ADQL) is the language used by the IVOA to represent astronomy queries posted to VO services. ADQL is based on the Structured Query Language, especially SQL 92; special restrictions and extensions have been defined in order to support generic and astronomy specific operations.Current recommendation
The current IVOA Recommendation is ADQL-2.0 (30 October 2008). ADQL-2.0-Next page collects proposed features and notes while developing the next revision of the specification. ADQL-2_0-Errata keeps track of the Errata status for this specific version of the recommendation. ADQL-2.1-Next page collects proposed features and notes while developing the next revision of the specification.ADQL-2.1 Proposed Recommendation
The DAL group are working on developing the next version (2.1) of the ADQL specification. The latest version of the specification is available as HTML and PDF at PR-ADQL-2.1-20210528 (28 May 2021).Document source
The source code for the document can be found online in the ADQL GitHub repository. The following command can be used to checkout a local copy of the document source.
git clone https://github.com/ivoa-std/ADQL.git
BNF grammar validation effort
At the October 2016 interop in Trieste the DAL group decided to delay putting the ADQL 2.1 working draft forward for the next stage of review because there were questions about the validity of the Backus–Naur form (BNF) definition of the language grammar included in the specification. Prior to this, the BNF definition of the language grammar has been treated as a human readable resource that was interpreted manually by people developing ADQL parser implementations. The October 2016 decision added a new requirement that the BNF definition of the language grammar needed to be machine readable and had to be validated against a set of standard queries. In order to accomplish this, the Lyonetia GitHub project has been set up to develop the tools and test data needed to create a machine readable version of the grammar. To follow and contribute to this work please refer to the project plan.Previous versions
- DAL PR (2021-05-28): PR-ADQL-2.1-20210528 (git commit 0d5911f or git tag v2.1_20210528-pr)
- DAL Internal WD (2017-11-29): WD-ADQL-2.1-20171129 (svn revision 4594)
- DAL Internal WD (2017-10-24): WD-ADQL-2.1-20171024 (svn revision 4555)
- DAL Internal WD (2017-10-23): WD-ADQL-2.1-20171023 (svn revision 4544)
- DAL Internal WD (2016-05-02): WD-ADQL-2.1-20160502 (svn revision 3385)
- DAL Internal WD (2015-06-01): WD-ADQL-2.1-20150601 (svn revision 2983)
Associated projects
- UDF Catalogue Collections of IVOA User Defined Functions
- Cosmopterix Docker containers for common database platforms.
- Lyonetia project to validate language grammar.
- ADQL function name prefixes to avoid name conflicts.
ADQL-2.0 Erratum 1: White space clarification
Author: Dave Morris Date last changed: 2018-02-23 Date accepted: 2018-02-22Rationale
The BNF grammar for the ADQL-2.0 language is listed in Appendix A of the recommendation. In this grammar aseparator rule is present, with the following definition:
<separator> ::= { <comment> | <space> | <newline> }...
However this nonterminal token is only referenced in the rule for the character_string_literal, i.e.
<character_string_literal> ::=
<quote> [ <character_representation>... ] <quote>
[ { <separator>... <quote>
[ <character_representation>... ] <quote> }... ]
It is uncontroversial that the intent is to allow comments and white-space wherever SQL-92 allows them. The ADQL standard however says differently, and there should be a clarification. Between two alternatives, adding a clarifying subsection to the ADQL-2.0 standard or removing the separator.
Erratum Content
In section 2 of the ADQL 2.0 Recommendation, add a section 2.1.4 with the following content:1.1.4 Whitespace and Comments The rules on where whitespace is allowed and required are as in SQL92; essentially, any <token> may be followed by a <separator>.
Impact Assessment
The change introduced by this erratum has no other impact on the current ADQL grammar and language apart from what already described in the previous sections of this note. Current implementations and parsers of ADQL query strings shouldn't be affected at all by this change. Moreover, current BNF grammar in ADQL-2.0 needs changes to be machine readable and validated and one of the issues in BNF is exactly how to deal with optional or required white spaces.Note
This Erratum was previously part of the ADQL2Err1 Errata Note (PDF former version). Erratum approved on 2018-02-22 by TCG, but last changes made as action deriving from that TCG tconf: removing unstable SVN reference and added formal author. -- MarcoMolinaro - 2018-02-23ADQL-2.0 Erratum 2: Mathematical Functions' Table issues
Author: Dave Morris Date last changed: 2018-02-14 Date accepted: 2018-02-22Rationale
IVOA DAL mailing list thread on July 2016 highlighted a possible issue with the description of the MOD function. Discussion went on also on this other thread. Subsequently a different thread highlighted other mismatches between Table 1 (Mathematical functions) descriptions and the ADQL-2.0 grammar. Here follows what emerged as possible erratum content from those discussions. MOD function description Considering what current TAP-1.0 services using ADQL implement and the usual SQL definition of the modulo operator: M % N = R with- R having the same sign as M
- |R| is less than |N|
- M = K * N + R for a given integer K
unsigned_integer x to be an optional argument (the seed) to the random mathematical function. This is not made explicit in the description in Table 1.
We suggest to amend the description changing the final "where x is a seed" into "where x is an optional seed" value.
ROUND optional places number
ADQL BNF grammar (pages 29-30 of ADQL-2.0) reports the signed_integer n of ROUND(x, n) to be an optional argument. This signed integer number of places to round the value of x to is not made explicit to be optional in Table 1 description. Also, as per SQL standard, it should default to 0 when not explicitly present.
We suggest to amend the description of round(x, n) in Table 1 by adding a final sentence reading "The integer n is optional and its value should default to 0.".
TRUNCATE optional places number
ADQL BNF grammar (page 30 of ADQL-2.0) reports the signed_integer n of TRUNCATE(x, n) to be an optional argument. This signed integer number of places to truncate the value of x to is not made explicit to be optional in Table 1 description. Also, as per SQL standard, it should default to 0 when not explicitly present.
We suggest to amend the description of truncate(x, n) in Table 1 by adding a final sentence reading "The integer n is optional and its value should default to 0.".
Erratum Content
This erratum changes the description of the mod, rand, round and truncate function in Table 1 on page 8 of the ADQL-2.0 recommendation from
mod(x, y)
Returns the remainder of y/x.
rand(x)
Returns a random value between 0.0 and 1.0, where x is a seed value.
round(x, n)
Rounds double value x to n number of decimal places, with the default
being to round to the nearest integer. To round to the left of the decimal
point, a negative number should be provided.
truncate(x, n)
Returns the result of truncating the argument x to n decimal places.
to
mod(x, y)
Returns the remainder of x/y.
rand(x)
Returns a random value between 0.0 and 1.0. The optional argument,
originally intended to provide a random seed, has undefined semantics.
Query writers are advised to use the form without the argument.
round(x, n)
Rounds double value x to n number of decimal places, with the default
being to round to the nearest integer. To round to the left of the decimal
point, a negative number should be provided. The integer n is optional
and its value should default to 0.
truncate(x, n)
Returns the result of truncating the argument x to n decimal places.
The integer n is optional and its value should default to 0.
Impact Assessment
The changes proposed by this erratum should impact no existing IVOA resource. Checks against the major TAP services show they already act following the behaviour here above described for the modulo function, while for the remaining 3 issues, all of them simply require some explicit wording in the description to follow the grammar constraints. Checks against a choice of RDBMS-es (PostgreSQL, MySQL, MariaDB, Apache Derby, HyperSQL and Oracle) shows the same behaviour too. See the mail threads referenced in the introduction of this page for details.ADQL-2.0 Erratum 3: Coordinate system argument as a string literal
Author: Grégory Mantelet Date last changed: 2020-05-13 Date accepted: 2020-06-11Rationale
It appears that the grammar of ADQL-2.0 is more permissive than intended for the first argument of thePOINT, BOX, CIRCLE and POLYGON functions. This alert has been raised while adapting the grammar of the functions above for an alternative function signature introduced in ADQL-2.1 (see the GitHub Issue #29). This argument represents the coordinate system of the geometry coordinates. The grammar of ADQL-2.0 admits any string expression ( <string_value_expression>) here, which includes string concatenation, column references, User Defined Functions (UDFs), any other function returning a string, but also numerics. Considering that a coordinate system cannot be represented by a numeric expression, this case should have already been excluded. In particular, allowing a non-constant expression as coordinate system makes its interpretation very difficult by an ADQL-SQL translation engine (indeed, parsing and transform functions are required on the database side). Also, users have no explicit way to know what kind of coordinate transformation is applied.
This Erratum restricts the coordinate system argument of POINT, BOX, CIRCLE and POLYGON to a string literal ( <character_string_literal>) to mitigate these problems.
Erratum Content
This Erratum changes the BNF grammar of ADQL-2.0, in the Appendix A, p.25, from:<coord_sys> ::= <string_value_expression>into:
<coord_sys> ::= <character_string_literal>The description of the first argument of the functions is also changed for:
-
BOX, section 2.4.3 (p.13), -
CIRCLE, section 2.4.5 (p.14), -
POINT, section 2.4.12 (p.18), and -
POLYGON, section 2.4.13 (p.18),
the coordinate system is a string value expression as defined in Section 2.4.1.into:
the coordinate system is a string literal as defined in Section 2.4.1.
Impact Assessment
Currently, very few ADQL services are able to interpret the coordinate system argument and to perform appropriate coordinate transformations. Currently, as far as we know, such services are able to do so only if the coordinate system is given as a string literal. This Erratum will not impact them. For ADQL services that can interpret coordinate systems and apply coordinates transformations system from column references (e.g., via an uploaded table) – no such service is known at the present time –, it is perfectly allowed to offer more features in an implementation as long as the required ADQL language is supported. From ADQL-2.1 on, the coordinate system argument is deprecated and becomes optional. Hence, no impact expected in future versions of ADQL where this argument will likely be removed.ADQL-2.0 Errata
Page status report for Errata related to the ADQL-2.0 specification. (ADQL-2.0 next features page is available here).Accepted Errata
- ADQL-2.0 Erratum 1: white space clarification
- ADQL 2.0 Erratum-2: mathematical functions' table issues (MOD, RAND, ROUND, TRUNCATE)...
- ADQL-2.0 Erratum 3: Coordinate system argument as a string literal
- ADQL-2.0 Erratum 4: NULL as valid value expression
Rejected Errata
Proposed Errata
ADQL-2.0 Next
This topic collects proposals for modifications of the ADQL-2.0 specification in order to improve the next revision of the specification. Following the guidelines to report and accept Errata to IVOA standards here you'll only find listed the proposed features and discussion for subsequents revisions of the this specification. Errata tracking process for ADQL-2.0 can be found at ADQL-2_0-Errata page.Proposed Features
Suggestion for revision of ADQL-2.0, in terms of errata content, clarifications and proposed new features, where initially collected in the TAP Implementation Notes wiki topic, later serial as an IVOA Note (its source available on volute), and then discussed at various Interoperability Meetings. Before the Banff Interop (Northern Fall 2014) a page (vote on Madrid's splinter topics) was issued to try to reach some consensus. With reference to the ADQL-2.1 internal WD (see [[ADQL][ADQL] topic), a session and splinter discussion took place at Sesto Interop. The outcome of that discussion (thank to notes from DaveMorris) is listed here as a starting point for further discussion:- No-rewrite rule dropped. *Legacy restriction dropped to enable new functionality to go ahead.
- BOOLEAN data type will be added to ADQL-2.1
- Optional functions
- Proposed text for describing how to handle optional features accepted in ADQL-2.1
- Acceptance criteria
- The group will work together to define some acceptance criteria for optional and mandatory features. Accepted as task to do - first draft by Oct.
- Target platforms
- The group will work together to collect details about the database platforms that are used to implement ADQL Which features are supported by which database platforms may form part of the acceptance criteria. Accepted as task to do - first draft by Oct.
- Generic user defined functions
- Allow user defined functions returning geometry values
- Accepted as mandatory core in ADQL-2.1.
- LOWER, ILIKE
- Accepted as optional in ADQL-2.1
- Some issues with indexing and platform support preventing them being mandatory.
- Set operators - UNION, INTERSECT, EXCEPT
- Not as easy as originally thought.
- Accepted as optional for ADQL-2.1 ( will probably only be implemented by experimental services)
- Hierarchical queries - WITH
- Not as easy as originally thought
- Accepted as optional for ADQL-2.1 ( will probably only be implemented by experimental services)
- CAST
- Accepted as optional in ADQL-2.1
- Probably mandatory in ADQL-3.0
- UCDCOL(’pos.eq.ra;meta.main’) => the first column with that UCD
- Problems with implementation - proposal withdrawn.
- IN_UNIT
- Simple unit transforms: dec + IN_UNIT(pmde,′ deg/yr′) ∗ 24.4
- Proposed text description needs more detail.
- Requires two independent implementations to be accepted as mandatory.
- Accepted as optional in ADQL-2.1
- Probably mandatory in ADQL-3.0
- Bitwise operators
- BIT_AND, BIT_OR, BIT_XOR, BIT_NOT
- Accepted as optional for ADQL-2.1.
- CROSSMATCH
- Some questions to answer about what the function returns: “closest within match limit” or “all pairs up to sr”
- Not accepted for ADQL-2.1 (yet).
- If LOWER, why not UPPER?
- Accepted as optional in ADQL-2.1
- SUBSTRING, CONVERT, TRANSLATE, TRIM
- To be discussed on the mailing list.
- ADQL data in TAPRegExt
- Should the registration details for the ADQL optional features be moved from TAPRegExt to a 'std:ADQL' registry entry ?
- Duplcate columns
- Questions raised about how we should cope with queries that produce columns with duplicate names. To be discussed on the mailing list.
- Deprecate REGION
- To be discussed on the mailing list.
- Function overloading
- Interval and 2D geom versions of CONTAINS etc. To be discussed on the mailing list.
ADQL-2.1 Next
This topic collects proposals for modifications of the ADQL-2.1 specification in order to improve the next revision of the specification. Following the guidelines to report and accept Errata to IVOA standards here you'll only find listed the proposed features and discussion for subsequents revisions of the this specification. Errata tracking process for ADQL-2.1 can be found at TBD page.Proposed Features
- Move from BNF to PEG grammar
- Filtering on timestamp values
- Support interval values
- Support MOC
- Support array values
- Support bitwise operators
- Support hexadecimal values
- Better support of boolean values (i.e. boolean type ; no more
=1) - Add a substring function (already existing in SQL-92)
- Extend
DISTANCE()to other geometries (not onlyPOINT()) - Add a function returning the polygonal intersection of two geometries:
INTERSECTION() - Unit annotation of values
- Consider the OGC Standard (OpenGIS ® Implementation Standard for Geographic information) for ADQL geometries evolution
ADQL-2.0 Erratum 4: NULL as valid value expression
Author: Grégory Mantelet Date last changed: 2023-01-31 Date accepted: 2023-04-05Rationale
The in-development validator for ADQL (ivoa/lyonetia#16 (comment)) as described in the official standard document highlighted a miss since ADQL-2.0. NULL values are not allowed anywhere except in the constraint IS NULL. However, it is a valid value in any existing DBMS. This Erratum aims to fix this miss by adding the NULL in the value expression definition.Erratum Content
This Erratum changes the BNF grammar of ADQL-2.0, in the Appendix A, p.35, from:into:
<value_expression> ::= <numeric_value_expression> | <string_value_expression> | <geometry_value_expression>
<value_expression> ::= NULL | <numeric_value_expression> | <string_value_expression> | <geometry_value_expression>
Impact Assessment
All existing Database Management Systems (e.g. PostgreSQL, MySQL, SQLite, SQLServer, ...) allow NULL values in all places where a value can be provided. DACHS and CADC already support it, and so, it should not be a problem for all existing services based on their TAP framework. However, services based on VOLLT are impacted until a new version is released or if it has been forked and already fixed on this particular issue. Anyway, the amount of ADQL queries where a NULL has to be explicitly written (apart from the special constraint IS NULL) is extremely low. Hence, very few inconvenience is expected while waiting for a migration for a newer version resolving this issue. The next version of VOLLT will anyway embed ADQL-2.1, in which this problem is already fixed (see ivoa-std/ADQL#72).ADQL 2.1 Proposed Recommendation: Request for Comments
ADQL defines an SQL-like grammar adapted for astronomical purpose. It is especially used by the TAP (Table Access Protocol) standard. Latest version of ADQL can be found at: Here is a diff view of all the modifications done on the above PR document since the beginning of this RFC period:
Reference Interoperable Implementations
- DACHS
- VOLLT/ADQLLib (branch adql2.1)
- Data providers:
- CEFCA/OAJ: CEFCA Catalogues Portal TAP services.
Implementations Validators
Two ADQL parsing validators available in the GitHub repository named Lyonetia:- One based on DACHS: src/adql/gavo
- One based on VOLLT: src/adql-validator
Comments from the IVOA Community during RFC/TCG review period: 25-Apr-2023 - 6-Jun-2023
The comments from the TCG members during the RFC/TCG review should be included in the next section. In order to add a comment to the document, please edit this page and add your comment to the list below in the format used for the example (include your Wiki Name so that authors can contact you for further information). When the author(s) of the document have considered the comment, they will provide a response after the comment. Additional discussion about any of the comments or responses can be conducted on the WG mailing list. However, please be sure to enter your initial comments here for full consideration in any future revisions of this documentCommunity Review by Theresa Dower
TheresaDower (as an implementer of parts of the ADQL 2.1 standard client and server side on odd architecture), 2023-04-27 I am so happy to see this moving forward!- I see no show-stoppers in new features for the MAST/STScI frequently annoying special case MSSQL server back end.
- I have already, through the vollt adql library reference implementation and my tweaks to a MAST/STScI fork, added some of the features. This includes COALESCE and ILIKE and some updates to math functionality, mostly for the sake of canned queries PyVO expects for RegTAP 1.1. This has gone well so far. Thank you for the work on this reference implementation!
- I think deprecating the coordinate system arguments is ultimately good; it removes unforced errors from expectations of servers quietly converting things in the back end, which it seems no one is actually doing (we're ignoring it and defaulting to ICRS)
- ... that said, given the way a couple PyVO client expectations forcing server support have already caught us by surprise with RegTAP, I am a little nervous about the actual shuffle in operations when clients start dropping the deprecated pieces, and servers are expected to handle it. Hopefully we can improve coordination rolling out changes in the most popular clients and services that speak ADQL; we are already in a better place for global volunteer effort coordination than the past few years.
- This is all to say... should I be pulling the whole vollt ADQL 2.1 branch for MAST TAP services soon? Is there anything outstanding in it I should wait on? Happy to block some time to work on it.
- GM Answer:
- First of all, thank you for all your positive comments.
- About the optional features and how they are supported by clients and servers, both should know that supported optional features must be declared in the capabilities of the used DAL service (e.g. TAP). If an unsupported feature is used anyway, the server must raise an error in a way depending on the underlying query protocol. It is then up to the client to correctly report this error. It is also up to the client to eventually provide a way to show which features are supported, so that the use can adapt the query expression.
- [Speaking as the VOLLT developer] The ADQL-2.1 branch contains the full ADQL-library implementation. So, if you want to use only the query parsing and manipulation library, it is all OK. However, the TAP-library is not yet fully-compatible with this new version of ADQL (especially for the features declaration). That's why this branch is still a branch and is not yet merged with master. I am currently working on completing the integration of ADQL-2.1 into TAP-lib. So, in theory, the TAP service in this branch it should work, but things like the declaration of supported optional features is not yet integrated. The best is probably to wait for this branch to be merged with the master one.
- -- GregoryMantelet 2023-05-03
Community Review by Markus Demleitner
I have submitted a bunch of minor changes in https://github.com/ivoa-std/ADQL/pull/91, too.- GM Answer: Review started -- GregoryMantelet 2023-07-25
- GM Answer: Not being at the origin of these two sentences while not really understanding them, I agree to remove ambiguous or at least possibly untrue content. Consider it done in PR#95 ; and your PR#91 for "MAY"s removal. -- GregoryMantelet 2023-07-26
- GM Answer: Agreed. Done in PR#95 -- GregoryMantelet 2023-07-26
- GM Answer: Personally, no strong opinion here, as long as we don't loose any information. Let's do that in ADQL-2.2 then. I created the issue #93 to remember doing this. -- GregoryMantelet 2023-07-26
ADQL also provides support for STC-S based geometric regions. While a fully normative definition of the argument of REGION has not been written, implementations are advised to follow Section~6 in TAP 1.0 \citep{2010ivoa.spec.0327D}.That's a lot more helpful than a reference to the WD that should have been pulled from the doc repo a long time ago. This will then also be consistent with what we write in 3.5.4 (where I'd not argue against dropping this discussion on p. 20 entriely).
- GM Answer: Agreed to replace the reference to STC-S WD by the TAP appendix. As you say, it is more consistent with 3.5.4 and it is safer to avoid citing something as unstable as a WD. This is done in PR#95. About dropping the discussion about STC-S, I would not do it in ADQL-2.1 ; it would be too soon. Let's speak about that in a future version of ADQL, when DALI will be ready to take over completely. -- GregoryMantelet 2023-07-26
- GM Answer: Do you speak about section 4.2.5 or 4.2.7? If about 4.2.5, I would keep this section ; it gives important information about the policy change in ADQL 2.1 regarding coordinate system management. But we should actually rename it: "Coordinate system" instead of "Coordsys" (which is ambiguous ; it could be about the COORDSYS function which is not the case here). If you speak about 4.2.7, I agree that this entire section should actually be removed (and maybe some content moved in the the COORDSYS and DISTANCE sections) because it duplicates what is already written in the COORDSYS and DISTANCE sections. I wait for your answer before doing anything in PR#95. -- GregoryMantelet 2023-07-26
- Ah, I see how ambiguous my comment was. I was, as you suspected, talking about 4.2.7 -- MarkusDemleitner - 2023-09-21
- Done in #97 -- GregoryMantelet 2023-09-22
- Fine compromise, thanks. -- MarkusDemleitner - 2023-09-21
- Done in #97 -- GregoryMantelet 2023-09-22
- GM Answer: The intent of this sentence was indeed what you rephrased. Sorry for the clumsy wording. Mark Taylor, on behalf of the Operations IG, wrote a similar comment. I made a correction based on his comment in PR#92 : 1/ Both specifications and Endorsed Notes can define an ivo_ prefix, 2/ cite the UDF catalogue instead of RegTAP. This highlights the UDF catalogue. However, I don't think it is a good idea to limit the usage of the ivo_ prefix to just the UDF catalogue. I think it is still perfectly fair to also allow Recommandations to define the ivo_ UDF they need instead of waiting for the UDF catalogue to be updated. In this ADQL-2.1 section, we already say that the ivo_ prefix is reserved only for specifications and Endorsed Notes. I think it is already enough authority, though I agree that possibly spreading ivo_ UDF in multiple documents may lead to a bit of confusion for users. However, the UDF definition is never lost: it is defined in an IVOA spec. or EN, and it is declared as supported by a TAP service through TAPRegExt. So, what I suggest here, is to keep the correction suggestion from Mark Taylor and to remove the clumsy and confusing paragraph that you just mention. I just did this second correction in PR#95 which will be merged with the correction from Mark Taylor in PR#92. Is it ok like this? -- GregoryMantelet 2023-07-26
- GM Answer: I agree: repetition is annoying. Because UPPER does not reference the same algorithm as LOWER and ILIKE, I preferred to move the case folding explanations in a separate sub-section. I hope this proposition reads better. See PR#95. -- GregoryMantelet 2023-07-26
- GM Answer: Agreed. Mark Taylor has also sent the same comment. I fixed it while joining and moving all duplicated explanations inside the PR#92. -- GregoryMantelet 2023-07-26
- GM Answer: You've certainly looked at this BNF more than me and you highly contributed to improve it. Me too, I really wait for the PEG grammar in ADQL-2.2. -- GregoryMantelet 2023-07-26
Comments from TCG member during the RFC/TCG Review Period: 25-Apr-2023 - 20-Jun-2023
WG chairs or vice chairs must read the Document, provide comments if any (including on topics not directly linked to the Group matters) or indicate that they have no comment. IG chairs or vice chairs are also encouraged to do the same, althought their inputs are not compulsory.TCG Chair & Vice Chair
With positive review by the TCG with a comments & feedback period successfully completed, the TCG chair/Vice Chair approve as well.Applications Working Group
We took note of all the other changes requested already. The document is now well structured and clear. Well done.Data Access Layer Working Group
Overall this is a substantial update to the ADQL spec which is clear and easy to read.Substantive
- §2.1.9 whitespace - In erratum 1, the section was titled "Whitespace and comments" which didn't refer to comments. Comments are defined in the BNF but are not allowed anywhere. Should this be made explicit in this section e.g. adding "Comments are not currently supported in ADQL."
- GM Answer: You're right. There is nothing about adding comments in ADQL though the BNF (and parsers) allows it. I just added a simple section description the syntax to use about comments and what parsers should do with them (in brief: nothing). See PR#100. -- GregoryMantelet 2023-10-11
- BNF - Should <numeric_primary> refer to <value_expression_primary> given that <value_expression_primary> can then refer to a numeric, string or geometry value expression?
- GM Answer: This is indeed a possibility given that, by instance, a column reference or a CAST function can return something else than a numeric. This problem may also happen with <character_primary> and <geometry_value_expression>. Unfortunately, I do not see any grammatical solution here. In such case, even a DBMS will throw an error only at execution time (instead of raising it at parsing time). This will be exactly what will happen with an ADQL query: it will pass the parsing but will fail on execution time. I guess it is the limit of a grammar. In my parser, I try to be as smart as possible in order to detect such problem, but it does not work in all cases. -- GregoryMantelet 2023-10-11
- §4.2.3 - Coordinate limits - Galactic longitudes are often presented as [-180, 180] rather than [0, 360] - is it worth noting that this isn't supported in ADQL?
- GM Answer: Grid WG raised the same warning. Since we deprecated the coordinate system argument in geometrical expression, it is true that such coordinates limits make no more sense. We solved it in PR#99. -- GregoryMantelet 2023-10-11
- §4.2.4 - Datatype functions - The opening sentence "The following functions" confused me as I assumed the paras following were somehow defining functions. Instead, I gather this is referring to the functions defined in §4.2.8 onwards? If that is the case, then maybe reword it to something like "The functions in this section". If it is a subset it would be better to state it like is done in §4.2.6
- GM Answer: Done in PR#100. -- GregoryMantelet 2023-10-11
- §4.2.5 - Coordinate system - "For interoperability reasons, queries against 2.0 services" - I assume this means "2.0 or later services"? If so it should state that.
- GM Answer: Done in PR#100. -- GregoryMantelet 2023-10-11
- §4.3 User defined functions - The changelog states "Recommendation for data providers to look at the Catalogue of ADQL User Defined Functions before implementing a UDF" however while the catalogue is mentioned this recommendation isn't specifically stated
- GM Answer: You're right. It is more a suggestion than a recommendation. In his RFC review, Markus also wrote a comment about recommending this EN. Then, I fixed something and accidentely removed this explicit recommendation. However, as Markus also said, it should be stated clearly in this document. My previous wording was quite clumsy. I hope the paragraph I've just added in PR#100 is now better. -- GregoryMantelet 2023-10-18
- GM Answer: You're right. It is more a suggestion than a recommendation. In his RFC review, Markus also wrote a comment about recommending this EN. Then, I fixed something and accidentely removed this explicit recommendation. However, as Markus also said, it should be stated clearly in this document. My previous wording was quite clumsy. I hope the paragraph I've just added in PR#100 is now better. -- GregoryMantelet 2023-10-18
- §4.3.1 Overview (UDF) - Should "Catalogue of ADQL User Defined Functions" have a citation reference here?
- GM Answer: Yes, it should, but it is not dependent from me. Ivoa-Tex has this rule of adding the citation only on the first occurence of the referenced resource. This Catalogue of UDF is indeed cited in section 4.2.5 ; there, we see the expected citation. -- GregoryMantelet 2023-10-18
- §4.6.2 Except - I assume that rows are excluded using the rules for duplicated rows in §4.6.4. Should this be stated?
- GM Answer: This was indeed done in a previous version of the ADQL 2.1 document, but it lead to a lot of repetition because ths rule about duplicated rows also apply to UNION and especially INTERSECT. But I assume we can have just a short sentence to tell readers to look at the section 4.6.4 to discover the exclusion rule but I fear it repeat again the same sentence in UNION, INTERSECT and EXCEPT sections while the very next section actually speak about duplicated rows. So, unless you really think it is absolutely important for readers, I think it is best to keep it like this. -- GregoryMantelet 2023-10-18
- §4.2.9 BOX
- "has been deprecated in this version of the standard" -> "has been deprecated in version 2.1 of the standard" so that it isn't confusing in v2.2
- Similarly "as of this version" should be replaced with "as of version 2.1" in §4.2.9, 4.2.11, 4.2.15, 4.2.18, and 4.2.19
- §4.4.2 LOWER - Should this mention that the conversion is according to the rules of the database's locale
- §4.4.3 UPPER - Should this mention that the conversion is according to the rules of the database's locale
- §4.5 Common Table Expressions - “Support for” is repeated in the sentence, making it a bit hard to read. It might be better if the second instance was dropped to read "MAY include support for the following optional common table expressions"
- §4.7.1 CAST - The type table appears above the "Input Types" paragraph making the last sentence ending in "support the following types:" rather confusing. I recommend the table be numbered and referred to in the text by number and, if possible moved to be below that paragraph.
- §4.9.1 IN_UNIT
- This section is the only one which refers to "numeric expression". I assume it intends "numeric value expression" as defined in the BNF? Should it be stated in that way (like in §4.2.16)
- "Consequently, regarding the unitless aspect of such numeric expression" -> Consequently, regarding the unitless aspect of such a numeric expression" (adding a)
- GM Answer: OK. Merged with main branch. -- GregoryMantelet 2023-10-18
Data Model Working Group
The revised ADQL spec v2.1 is well structured, clear and detailed. We observe that the authors integrated and answered to the comments received on this page. From the Data Model point of view, there is little implications and we have no further comments. -- MathieuServillat - 2023-11-10Grid & Web Services Working Group
First, great standard and a very good advance. I have some comments, see below. Minor vs Major version My first comment is philosophical. If this is a minor version, should it be backwards compatible with version 2.0? In the practical side, should a service ADQL 2.1 compatible be able to provide unambiguous replies to ADQL 2.0 queries? I partially (almost completely) implemented a parser for ADQL 2.1 for the Gaia archive (not my current position), compatible with ADQL 2.0 and sometimes the parsing was not able to get problems. I wonder if this is needed. Removing the first geometrical parameter could cause ambiguous parsing, in particular when we use columns instead of values (where we can use the type to evaluate the syntax). For example, POLYGON(t.a, t.b, 3, 4, 5, 6, 7,...) where t.a could be a region or a first coordinate. "counting" parameters (even or odd) could help infer the data type expected but good parsing is not guaranteed) There are some clever restrictions in the spec (like requesting the same type of values for DISTANCE, either pair of coordinates or POINTS) that help a lot. So, my general question is about backwards compatibility or not. In case this is the case, has this backwards compatibility been assessed in the spec for the operators that have changed signatures?- GM Answer: I understand your point about backwards compatibility and I also sympathize with the parser update to 2.1. I experienced the same tricky ambiguous syntaxes with the geometrical functions parsing. As you did, I succeeded to clear up ambiguities thanks to the number and datatypes of parameters. I agree this is not obvious and quite tricky on this aspect. Hopefully, this geometries ambiguities are only temporary. They exist only because the coordinate system argument is deprecated and kept there optional exactly to ensure backward compatibility. So, unless I have missed something, ADQL 2.0 queries can be successfully interpreted and executed with an ADQL-2.1 processor. In this sense, backward compatibility is respected. Besides, I hope transitionning to a PEG grammar can highly improve parsing of ADQL queries, but we have to wait for the next version of ADQL to confirm that. -- GregoryMantelet 2023-10-11
- GM Answer: Agreed. This minor revision of ADQL did not improve this part of the spec. I think this should be probably solved in a next version using region syntax described in DALI. I keep that in my notes for a next version of ADQL. Anyway, I think it is also good to keep some flexibility here ; I am quite in favor of recommending one or two syntaxes while still allowing any custom one, though I am not sure it would be really possible and if yes, how. Let's discuss about that while preparing next version. -- GregoryMantelet 2023-10-11
- GM Answer: You're right, and you are not the only one to make this comment. In my parser, I don't check whether the coordinates are in these ranges, and I don't know if any other implementation does it or not. In this regards, I think it would be reasonnable to change this part of the spec, either by removing these limits or by relaxing the constraint (e.g. by saying equatorial coordinates should be between these ranges). Let's try in PR#99 -- GregoryMantelet 2023-10-11
- GM Answer: You're right, this is quite difficult to implement, and that's why it is an optional feature. My parser does not do that for the moment (but I'd like to implement it some time). However, the DACHS parser supports this function. I don't know about any other ADQL parser. -- GregoryMantelet 2023-10-11
- GM Answer: My parser checks for such column name consistency already without WITH queries. When some WITH queries are added, my parser merely sees them as tables. Considering that, it is not that difficult for my parser. I honestly don't know about other parsers, and anyway, I would never require them to do so, because as you said, it is not particularly trivial to do (I have spent some time myself to reach this level of check in my parser). -- GregoryMantelet 2023-10-11
Registry Working Group
Semantics Working Group
Section 4.9.1 describes IN_UNIT(), and the many cases where it is returning an error message. Maybe it would be good to give one example of possible success query ? The reason for this is that in the current form, it is very difficult for a reader not familiar with the TAP standard to figure out where the unit of the first argument is retrieved from (as it is not specified in the ADQL query itself). Maybe rephrase : The first argument MUST be a numeric value expression. with : The first argument MUST be a numeric value expression. If the first argument is a column name, the VOUnits for this column ought to be found in the "unit" column of the TAP_SCHEMA.columns table of the TAP service. maybe also add in the Examples : -- ra : column expressed in 'deg' (column_name="ra" and unit="deg" in TAP_SCHEMA.columns) IN_UNIT(ra, 'rad') -- OK SebastienDerriere 2023-11-10- GM Answer: FIne with these changes. It is true a clarification was needed here for ADQL users. I applied all of them with a little more genericity regarding the way to get the unit for a column. See PullRequest #102. -- GregoryMantelet 2023-11-10
Data Curation & Preservation Interest Group
Interesting new features - thank Greg and all! It looks good, more user-friendly, even if the migration needs important work for data-center (eg: WITH, CAST, geometrical functions, units) Details :- (4.2.7) "coord_value" is not defined.
- GM Answer: This part is an excerpt from the full BNF. So, obviously not all terms are not defined here, and should not be, otherwise it would too long inside the text. However, you're right, this paragraph is about a new function signature with new parameters, so it makes sense here to also includes the definition for <coord_value>. I just added it (see GitHub/ADQL#88). -- GregoryMantelet 2023-04-27
- (4.2.13) is it possible to use boolean instead of values {0,1} for CONTAINS ?
- GM Answer: Making boolean functions really returning a boolean value instead of an integer value (0 or 1) was a work initiated with this version of ADQL, but considering the important and complex grammar refactoring work it requires, it has been decided to postpone this for a next version (see GitHub/ADQL#32 for more details). I will speak about that at the May 2023 IVOA Interoperability Meeting. -- GregoryMantelet 2023-04-27
- (4.3) can you add link to UDF functions validated by IVOA (ivo_..) ?
- GM Answer: This link is already available in the document. It is referenced in sections 4.2.5 (COORSYS) and 4.3.1 (UDF Overview) (last words of the last paragraph ; but since it is not the first reference to this document the LaTeX template unfortunately hides the link); but I agree the PDF and HTML rendering does not make that kind of references really obvious in the text (except for the first reference occurrence) (or I am using the LaTeX template in the wrong way, which is really possible). It is also the 3rd reference in the References appendix. -- GregoryMantelet 2023-04-27
- (4.6.1) remove item "the columns in the operands SHOULD have the same metadata, e.g. units, UCD, etc." -
Tests should be limited to datatype - "same UCD" is clearly wrong (eg.: JOIN on primary-foregin key use meta.id;id.main and meta.id) , furthermore VOTable have not always UCD.- GM Answer: For exactly the reason you mention, this item is a SHOULD instead of a MUST. It is just a recommendation and not a requirement to encourage data providers to provide set operation result with as rich and consistent metadata as possible. In this item, the UCD is just an example of metadata on which a data provider can choose to add a strong constraint while performing a set operation. So, it is up to the implementation to decide whether or not this constraint should be set, and on which metadata. Although I agree this can be tricky for UCD, this item could be perfectly valid for units and possibly for any other metadata a data providers can offer. So, since this item is just a recommendation (not a requirement) and is really up to implementation, I suggest to keep this item. -- GregoryMantelet 2023-04-27
For instance, coordinate system used in VizieR tables depends of each catalogues.
The coord syst is important in case of crossmatch (from an other table or from an UPLOAD) If the parameter is removed, it requires;
- TO FIND A WAY TO REPORT THE NATIVE COORDINATE SYSTEM TO USER - in TAP_SCHEMA for instance
- GM Answer: You are perfectly right. This is however out of the scope of the ADQL standard. This information must be provided by the TAP standard. I know that a new version of TAP is in preparation and that a talk will happen on this topic at the May 2023 IVOA Interoperability Meeting. It seems to be a very good timing to suggest this addition to TAP. Personally, I propose coordinate system information on columns in the VOLLT/TAP library. This can be used as inspiration for next version of TAP. -- GregoryMantelet 2023-04-27
- to use UDF functions to make the conversion (is it possible to add with explicit example using UDF)
- GM Answer: It may be possible, though such functions should be defined in the Catalogue of UDF Endorsed Note. Currently, only one conversion function is defined in there: gavo_transform(from_sys, to_sys, geo) (see sec. A.1.5). Note the function prefix: gavo_. So, it is not yet an official IVOA User Defined Function ; it is just a function defined in a single service implementation. I know that other TAP implementations have implemented a such function. If at least 2 TAP services have the same function (except for the function prefix), I think we can make this function official and give it an ivoa_ prefix ; from what I know there are a GAVO and VizieR implementation for a such function. Thus, maybe it is the good moment to propose a new version of this Endorsed Note. Anyway, to come back on giving examples in the ADQL standard, I don't think it is wise to do so now ; it is too early. I prefer to wait until an ivoa_ function is officially endorsed in this note rather than suggesting to use an unstable and soon-deprecated function. Thank you Gilles for this suggestion, it will probably help updating the UDF catalogue, and I keep this idea also for the next version of ADQL. -- GregoryMantelet 2023-04-27
Education Interest Group
Knowledge Discovery Interest Group
Operations Interest Group
This is generally well-written. Special commendation for the careful change list in Appendix C.1. I am happy to endorse it, but have a few suggestions that should be considered by the authors.- I made a few uncontroversial fixes in PR#89 (merged on the 7th May 2023).
- Sec 2.1.3 SQL reserved keywords list: The entry END-EXEC doesn't really make sense here, since it is not syntactically capable of use as an Identifier. I suggest its removal.
- GM Answer: Although I completely agree with this, I prefer to keep it here as it is a reserved keyword in SQL-92 and that I can find it in some SQL dialects. It is probably being too much careful here, though. If another person wants to drop it, I won't fight more than that. -- GregoryMantelet 2023-07-25
- Sec 2.2.3 Search Condition could contain a reference to the IN construct alongside the others listed.
- GM Answer: That's right. Added in PR#92. -- GregoryMantelet 2023-07-25
- Sec 4.2.8 (written by me, blame attributable accordingly): this section promotes use of the DISTANCE()<r idiom for crossmatching. It does not explicitly promote its use in cone-search-like WHERE clauses as well, but I now think it should do. I suggest appending the following paragraph to the end of this section: "Similar remarks apply to cone-search-like conditions in WHERE clauses: clients are encouraged to use, and services to provide efficient implementations of, conditions of the form WHERE DISTANCE(...) < r_max_deg."
- GM Answer: Are you sure it is needed? It is already explicitly said in the second paragraph that the WHERE syntax is allowed. It is also said that both syntaxes have to be supported and the server must guarantee the same level of performance. Would you mean that an example with the WHERE syntax is missing? -- GregoryMantelet 2023-07-25
- Only that the language throughout this section, including the title, is that of crossmatching which makes it look like that's the only context in which it applies. Where I mentioned WHERE in the second paragraph I was thinking of JOIN ... WHERE rather than SELECT ... WHERE. If you agree I could draft a PR to shift this emphasis and clarify a bit? I don't think a new example is required, though it might help. -- MarkTaylor - 2023-08-08
- Please, do so. I let you draft a PR to fix this. -- GregoryMantelet 2023-08-08
- Sec 4.2.13, 4.2.17, 4.2.18: the sections on CONTAINS, DISTANCE and INTERSECTS all contain a paragraph at the end along the lines: "If the geometric arguments are expressed in different coordinate systems, the XXX function is responsible for converting one, or both, of the arguments into a different coordinate system. If the XXX function cannot perform the required conversion then it SHOULD throw an error. Details of the mechanism for reporting the error condition are implementation dependent." This consideration only makes sense when the coordinates are specified using their optional, and now deprecated, coordsys arguments, so they are a bit confusing for a reader not familiar with the previous version of the standard. I suggest that these paragraphs are either removed, or that a comment is added to the effect that they only apply where the deprecated coordsys argument has been specified.
- GM Answer: You are right. This is ambiguous. Markus D. more or less stated the same thing. I am going to merge your remark and the comments of Markus together ; a common solution proposition will come in a PullRequest (in progress). -- GregoryMantelet 2023-07-25
- Sec 4.2.20 POLYGON. This section does not explicltly say what counts as inside or outside the specified polygon. It does say that it "corresponds semantically to the STC polygon," which probably answers the question, but the relevant STC definition discusses some non-sky geometries and is quite hard to understand (at least by me). I'm not sure what's best to do here, but possibly a reference to DALI which distills the STC advice into a more readable form might be helpful, if that is the intended semantics here (STC has some additional constraints about the maximum length of a polygon side). In practice it might be best to find out what the de facto standard PgSphere implementation does and make sure that advice is provided here which matches that. Or maybe I'm overthinking this and it's fine as it is.
- Commenting on my own comment, following AlbertoMicol 's polygon presentation at the Bologna Interop - my remarks here may be misguided or wrong. It could be worth considering whether the existing text is optimal in the light of Alberto's comments, but I no longer claim to have a good idea about what the right thing to do is. -- MarkTaylor - 2023-05-10
- GM Answer: Like you, I do not know exactly what should be done here for the moment. Alberto's presentation is a good start to clarify this polygon situation more generally in DAL. In the meantime, I am tempted to not change anything ; the risk is to add more ambiguous content in IVOA standards regarding the polygon definition. -- GregoryMantelet 2023-07-25
- Sec 4.3.1: "The ivo prefix is reserved for functions that have been defined in an IVOA specification" - this isn't quite true (assuming that "specification" refers to a Recommendation-track document) since the Catalogue of User Defined Functions referenced in the next paragraph is an Endorsed Note rather than an IVOA Specification. Could be reworded as "... have been defined in an IVOA specification or Endorsed Note". It might also be a good idea to use examples from the Catalogue of UDFs rather than RegTAP in the examples here, since that will be the more typical case.
- GM Answer: Correct. Fixed in PR#92. -- GregoryMantelet 2023-07-25
- Sec 4.3.2: This section discusses declaration of UDFs as TAPRegExt Language Features. It quotes BNF for UDF signature syntax from TAPRegExt, including the terminal <type_name>. TAPRegExt 1.0 [or 1.1] sec 2.3 says "The type_name nonterminal is not defined by the ADQL grammar [in version 2.0]. For the purposes of TAPRegExt, it is sufficient to assume it expands to some sort of SQL type specifier." Since ADQL 2.1 now has introduced a type system, it would be appropriate to say something about type_name in this ADQL section beyond implicitly deferring to TAPRegExt (which refers to an earlier version of ADQL). I suggest something like "The <type_name> SHOULD be one of the terms defined in Section 3."
- GM Answer: Agreed. This is done in PR#92. -- GregoryMantelet 2023-07-25
- Sec 4.6.1, 4.6.2, 4.6.3: These sections are quite repetitious.
- They each contain a list of MUST/SHOULD criteria which are almost, but not quite, the same as each other (I think by oversight rather than design). But in fact I don't think the lists quite make sense in any of these cases. The first two items (number/type equivalence of columns) are indeed MUST criteria for the operations, but the other two are not. I suggest retaining the first two as MUST criteria as at present, and replace the metadata advice with "In addition the operands SHOULD have the same metadata e.g. units, UCD etc. The metadata for the results SHOULD (or MUST?) in any case be generated from the left-hand operand."
- They each end with a warning that no clever unit manipulation will be done, ending with the displayed formula "2m = 2km." In my opinion the whole paragraph is kind of obvious and could be omitted; but at least I suggest to remove the (obviously untrue!) sentence "2m = 2km", which doesn't really add anything.
- It might be worth pulling out both these items and saying them once near the start of sec 4.6 rather than repeating them in each of the subsections 4.6.1, 4.6.2 and 4.6.3.
- GM Answer: I agree too. Comment similar to the one from Markus D. I will work on a common solution in a new PullRequest. -- GregoryMantelet 2023-07-25
- GM Answer: Actually, I removed all the duplications and the "2m = 2km" inside the PR#92. -- GregoryMantelet 2023-07-26
- I find the presentation style for ADQL examples in which each parameter is written on a different line, leading in most cases to examples extending over many lines when they could be fit on one, a bit annoying. But that's just my personal preference, if the authors prefer it as it stands that's fine.
- GM Answer: I personally completely agree with you. I did not touch this presentation style to preserve the style of the former editors. I am 100% ok to change that. Consider it done in PR#92. -- GregoryMantelet 2023-07-25
Radio Astronomy Interest Group
Solar System Interest Group
Nothing to add. -- Anne Raugh - 2023-11-10Theory Interest Group
Time Domain Interest Group
Standards and Processes Committee
TCG Vote : 6-Jun-2023 - 20-Jun-2023
If you have minor comments (typos) on the last version of the document please indicate it in the Comments column of the table and post them in the TCG comments section above with the date.| Group | Yes | No | Abstain | Comments |
| TCG | * | |||
| Apps | * | |||
| DAL | * | |||
| DM | * | |||
| GWS | * | |||
| Registry | * | |||
| Semantics | * | |||
| DCP | * | |||
| Edu | ||||
| KDIG | * | |||
| Ops | * | |||
| Radio | ||||
| SSIG | * | |||
| Theory | ||||
| TD | * | |||
| <nop>StdProc |
<!-- Set ALLOWTOPICRENAME =<span class="WYSIWYG_PROTECTED"> TWikiAdminGroup </span> -->
ADQL V2.0 P"> Recommended Edits to ADQL V2.0 PR: Geometric Function Semantics
This document provides suggested edits that addresses Ray Plante's RFC Comment that called for more explicit enumeration of the types and meanings associated with geometrical functions. (VOQL chair Editing: VOQL Chair has added the late comments from Ray at the end of this page, rather than at the -already closed- RFC pages to avoid confusion): In general, the aim of these suggestions is provide a crisp definition of each function at the beginning of the section. In many of the function descriptions currently, the type and meaning are given by example, which is not a good practice and can lead to ambiguities. In contrast, unambiguous statements up front will make the document a better reference for implementers, particularly when "looking up" a specific function. Note that bold within the suggested text below indicate new or changed words.- ADQL V2.0 P"> Recommended Edits to ADQL V2.0 PR: Geometric Function Semantics
- Late comments by RayPlante - 21 Oct 2008
2.4.2. AREA
Change the first sentence to:This function computes the area, in square degrees, of the region given by the function's only argument.Prepend the 2nd paragraph with the line:
The argument can be represented with one of the region functions, BOX, CIRCLE, POLYGON, or REGION.
2.4.3 BOX
Change 2nd sentence toA box is a special case of Polygon, defined purely for convenience, and it corresponds in meaning to the STC-S "Box" subphrase [4].The second paragraph sufficiently describes the arguments.
2.4.4 CENTROID
Change the first paragraph to:This function computes the centroid of the region given by the function's only argument and returns a POINT (See 2.4.11).Prepend the 2nd paragraph with the line:
The argument can be represented with one of the region functions, BOX, CIRCLE, POLYGON, or REGION.
2.4.5. CIRCLE
Change the first paragraph's first sentence to:This function expresses a circular region on the sky (a cone in space) and corresponds in meaning to the "Circle" STC-S subphrase [4].The rest of paragraph sufficiently describes the arguments.
2.4.6 CONTAINS
The return type is given in the 2nd paragraph after the first example, and arguments are defined in the last paragraph after the examples. It would be better to put a crisper definition up front. Append to the first paragraph:The first argument is a point or region value representing the contained geometry, and the second argument is a region value representing the containing region. The function returns 1 (meaning true) if the contained geometry is entirely within the boundary of the containing region and 0 (meaning false) otherwise. When the first argument is a point, it is considered inside the containing region if it lies on the containing region's border.Using the following text, move the contents of the last paragraph to a new paragraph after the first one:
Either argument can be given by the appropriate functions (the region functions--BOX, CIRCLE, POLYGON, or REGION--for the second argument, and the region functions or POINT for the first argument) or by a single column name or alias. When a column name or alias is provided, the value in the column or alias must be interpreted the appropriate value type. Since the two argument geometries may be expressed in different coordinate systems, the function is responsible for converting one (or both). If either argument cannot be converted to the proper geometry in a usable coordinate system, the function should throw an error message (as defined by the service making use of ADQL).Drop the last paragraph.
2.4.7 COORD1
This function returns the first coordinate value, in degrees, of a position given by the first argument. The argument may be given using the POINT function (See 2.4.12) or a column reference.
2.4.8 COORD2
This function returns the second coordinate value, in degrees, of a position given by the first argument. The argument may be given using the POINT function (See 2.4.12) or a column reference.
2.4.9 COORDSYS
This function returns the coordinate system string value from *the geometry given by the first argument. The argument value may be given as a geometry data type function (POINT, BOX, CIRCLE, POLYGON, or REGION) or as a column reference that can be interpreted as a geometry.
2.4.10 DISTANCE
This is sufficiently described.2.4.11 INTERSECTS
see suggestions for CONTAINS.2.4.12 POINT
This is sufficiently explicit.2.4.13 POLYGON
Insert the following into the second paragraph as the second sentence:This function corresponds in meaning to the "Polygon" STC-S sub-phrase [4].The explanation of the arguments is sufficient.
2.4.14 REGION
This is sufficiently described. VOQL chair Editing: VOQL Chair has added the late comments from Ray at this page, rather than at the -already closed- RFC pages to avoid confusion:Late comments by RayPlante - 21 Oct 2008
(Moved from RFC page) I recognize that these comments come after the official RFC, so I don't expect them to be answered by the authors. I hope some benefit could be gotten from at least the simpler items.Section 1
- Grammar: 3rd paragraph, 3rd sentence: I think you want to say "Similar to SQL, ..."
- I'm disturbed by the statement, "...this specification defines syntactical correctness only." For one, this is not true, since much of the semantics of the functions are described (e.g. Table 1). Second, I think we want to be a little stronger about the semantics. Do we really want "SELECT" to mean completely different things in different applications? See comments below about how to tighten this up. For here, I would prefer a statement that says something like:
This document provides the general semantics for the language elements; where these semantics are ambiguous, the specification of the service or application using ADQL should clarify how the elements should be applied.
Section 2.1.2
- "We can extend the list..." is not really appropriate for a spec. I suggest "The list includes SQL92 reserved keywords, keywords useful for astronomical purposes, and a subset of vendor-specific languages (e.g. TOP):"
Section 2.2
- I really don't like seeing TOP included in the syntax for 2 reasons:
- it is not SQL92; therefore, it forces some parsing of the query by front-ends to otherwise SQL92-compliant databases that do not support TOP,
- it is ill-defined: there's nothing that defines what the "first n-rows" are. It guarantees no repeatability and no ability to page beyond the TOP rows.
- Following up on my comment for section 1, I feel we need to be a bit more definitive regarding the semantics. I recommend inserting the following line into the paragraph after the syntax summary (beginning "The SELECT statement defines a query..."):
The query should be interpreted much like an SQL92 query: where ADQL and SQL92 keywords are identical, the ADQL keywords and their operands should be interpreted in the same way as defined in SQL92.
Section 2.4
- All functions should explain the types and meanings of the arguments and the returned value, just as is done in Table 1. Some of the function definitions do this sufficiently well, but others seem to provide this via examples, which is not a good practice and can lead to ambiguities. For simple changes to rectify this, consult my recommendations listed here.
ADQL geometric functi"> SQLServer & ADQL geometric functions
Page for discussions about supporting the ADQL geometric functions in SQLServer.Current deployments
ROE
The main science archives at ROE are hosted on a set of SQLServer 2012 databases. They have HTM-based regional queries enabled using SQL Server's built-in geometric functions, but these are not connected to the TAP services. Our current TAP services do not support the ADQL geometric functions.MAST
Many MAST holdings, including the Registry and a growing set of multimission data in ObsCore, are hosted on a set of SQLServer databases. Nearly completed TAP services are not currently published. Release of ObsTAP is held up on adding geometric functions for parsing ADQL to native SQL Server. to SQL Server-flavoured queries. For both RegTAP and ObsTAP, some canned TOPCAT queries don't work due to lack of NATURAL JOIN support in SQL Server. Services will be shipped preliminarily without that support, with the intention of adding it later. Brian McLean has written a series of database procedures mapping more pgSphere-like regional queries to the SQL Server's built-in geometric functions. Testing suggests similar performance to the old HTM-based system, and will be used in the ADQL parsing.A proposal for vector math in ADQL
With tables containing massive amounts of vectors becoming common (e.g., the collections of low-resolution spectra within Gaia DR3 or the Digitised Byurakan Surveys), giving TAP users a toolset to do server-side work with arrays becomes highly desirable and will significantly enhance the power of ADQL to do server-side analyses. This is an attempt to provide a baseline feature set for that. TAP servers supporting this should declare that by defining a language feature. While no IVOA specification exists for array operations, use the VECTORMATH key from GAVO's ADQL extensions standards record, like this:<languageFeatures type="ivo://org.gavo.dc/std/exts#extra-adql-keywords">
<feature>
<form>VECTORMATH</form>
<description>You can compute with vectors here. See
https://wiki.ivoa.net/twiki/bin/view/IVOA/ADQLVectorMath
for an overview of the functions and operators available.
</description>
</feature>
</languageFeatures>
Element Access
To access an element of a vector, write[element-index], where element-index is an integer-valued expression. In keeping with common SQL practices (and regrettably working against most programming languages), indexes in ADQL are 1-based (rather than 0-based). That is, the first element of an array with N elements has the index 1 and the last element has the index N.
Again in keeping with common SQL practices, accessing elements outside of that range gives NULL.
It is also possible to access to multiple elements of a vector (as a sub-array). To access, write [lower-bound:upper-bound], where lower and upper bounds are integer-valued expressions and are both included. Lower-bound must be equal or greater to 1 due to indexes in ADQL are 1-based and if upper-bound is greater to the lenght of the array not error is going to be returned.
Basic Math
- vec1+vec2 is the component-wise sum of two vectors. Where vec1 and vec2 have unequal length, the result is padded with NaNs to the length of the longer vector.
- vec1-vec2 is the component-wise difference of two vectors. Where vec1 and vec2 have unequal length, the result is padded with NaNs to the length of the longer vector.
- vec1*vec2 is the component-wise product of two vectors. Where vec1 and vec2 have unequal length, the result is padded with NaNs to the length of the longer vector.
- vec1/vec2 is the component-wise quotient of two vectors. Where vec1 and vec2 have unequal length, the result is padded with NaNs to the length of the longer vector.
- scal*vec and vec*scal is the scalar multiplication of a vector.
- vec/scal is the equivalent of (1/scal)*vec for a scalar. This is always floating point division, never integer division.
Vector computations
- arr_dot(vec1,vec2) is the scalar product of two vectors. Where vec1 and vec2 have unequal length, the shorter vector is padded with NaNs to the length of the longer vector. That is, the scalar product of vectors of unequal length is NaN.
Array Aggregation
These are functions that work like SQL aggregate functions, just on the elements of arrays. These ought to return the types of the elements of the argument (real, double precision, integers).- arr_avg(arr) returns the arithmetic mean of arr's elements
- arr_max(arr) returns the largest element of arr
- arr_min(arr) returns the smallest element of arr
- arr_sum(arr) returns the sum of arr's elements
- arr_count(arr) returns the number of elements in the array (the “array length”, where NaN elements count). This is always an integer, regardless of the array type.
Aggregate Functions for Arrays
The following standard ADQL aggregate functions, applied to arrays, work component-wise:- AVG
- MIN
- MAX
- SUM
Array Map
arr_map(expr_over_x, arr) computes a new array by binding each element of arr to x in turn and then computing expr_over_x. expr_over_x is an ADQL numeric_value_expression that can use column references as usual, except that the name x is reserved for the evaluation. For instance, arr_map(power(10, x), mags) will return an array [power(10, mags[1]), power(10, mags[2]), power(10, mags[3])...]. Admittedly, the artificial "x" here is not pretty. The clean solution would be to define some sort of lambda calculus for ADQL ("first class functions"), but that's almost certainly overdoing it (although: does anyone do that in SQL?). Perhaps it is preferable to use the array name itself, as in arr_map(power(10, mags), mags)? That would at least not clobber other names that SQL might want to use somewhere else? In implementation, at least Markus had to massage these column references on the translator level anyway. On the other hand, one might be tempted then to leave out the second argument at all, and that would require a lot more thought, first, as regards finding arrays in the expression (do we want to require translators to be able to do that?), and then what should happen if there are multiple arrays.Other array functions
Other functions related with vectors are:- arr_in(element, arr) returns true if the element is in the array or otherwise false.
Implementation Status
The SQL part of an implementation of this in postgresql is in DaCHS //adql RD, the create_array_operator script. The functionality can be tried out at the TAP service at http://dc.g-vo.org/tap. Suitable tables (i.e., with vector-like data) include sdssdr16.main, gaia.dr2epochflux, onebigb.ssa, or dfbsspec.spectra. In this implementation, array multiple access (sub-arrays) and function arr_in are not implemented. Test cases for implementors can be derived from sqlarraytest.py. This vector proposal has been also implemented at CEFCA as a part of the CEFCA Catalogues Portal. The functionality can be tried at the TAP service of the different projects/data releases offered.ADQL RFC
This document will act as RFC centre for the Astronomical Data Query Language v2.0 Proposed Recommendation. This document is currently under TCG evaluation for approval since 15 Sep 2008. TCG members shall approve or otherwise in the reserved section at the bottom of these pages.- Comments by BobHanisch
- Comments by FrancoisOchsenbein (from 04-May-2008 to voql@ivoa.net)
- At the Beijing meeeting, it was agreed to implement a sphericalDistance function as sphericalDistance(ra1, dec1, ra2, dec2). This function is completely independent of STC, and it follows the standard definitions of SQL functions: any of the 4 arguments may be either a number or a column reference.
- The definitions of functions in the document should be completed -- the units of arguments and returned values must be specified in the final document. And for my own point of view, I would strongly recommend to use radians for the sphericalDistance function, and steradians for the returned value of the AREA function, in order to be coherent with the other standard math functions existing in SQL.
- About the introduction of the REGION or Point objects: these are not numbers neither strings and cannot be directly manipulated by SQL, hence the introduction of the new <geometry_value_expression>. It is not clear how these objects are stored in relational tables; one may assume they are stored as e.g. string or binary objects, or they are computed from the other attributes of the table. I feel that some explanations about such non-atomic data would help the reader to understand how it works. The concept of a function which returns an object like CENTROID would also benefit of some explanation -- it differs from any SQL standard function since it does not return a number nor a string. What do you get if you issue a select statement like
Select centroid(mytable.centre) from mytable
which, according to the BNF, looks correct ?
SELECT list: The service is free to return a standard String serialization of the geometry.
- the ELLIPSE region is especially important in astronomical applications, like cross-matching or studies of surface brightness of galaxies. Was it omitted because the ellipse is a too complex geometry ?
- About the query expression: I share Jeff's comments about readability
where CIRCLE('ICRS', 123, 45,12) contains Point('ICRS', mytable.ra, mytable.dec)(b1)
looks more understandable than
where CONTAINS(Point('ICRS',mytable.ra, mytable.dec), CIRCLE('ICRS', 123, 45,12)) = 1(b2)
The functionnalities are equivalent, and there is surely a way of mapping exactly (b1) into (b2) which is SQL-92 compliant.
Assuming that a table contains an attribute "coo" which represents a point, the 2 expressions would be
where CIRCLE('ICRS', 123, 45,12) contains mytable.coo
versus
where CONTAINS(mytable.coo, CIRCLE('ICRS', 123, 45,12)) = 1
- I don't see the necessity of separating math and trigo functions; the table in section 2.3.1 gathers both functions... Maybe just modify the caption of this table ?
- Terminology: rather than <system_defined_function> I would prefer the term <adql_defined_function>
- about the section numbering: it looks strange to require a 6-level numbering to present ADQL; is it to compete with the STC document ?
It might look rational for the writer, but it looks really not balanced for the reader ... and the table of contents is therfore lacking the important topics related to the region stuff.
- Comments by Roy Williams
- Comments by MarkTaylor - 13 May 2008
- I agree with Francois that the the Functions table in sec 2.3 needs some more attention:
- are trig function arguments and return values in degrees or radians?
- how does the seed work in the
rand(x)function? I guessxis the seed - is it an integer or a float? It looks like you need a seed every time you invoke it, but that wouldn't be very convenient.
VOQL-TEG Answer: The seed is supposed to be an integer. We will detail the data type for each function argument in the next version of the document. - how come the
round(x,n)("Round to nearest integer") function has two arguments?
VOQL-TEG Answer: The function description is not complete. It rounds to the given number of decimal places, with the default being to round to the nearest integer. You use a negative number to round to the left of the decimal point. The function with two args is supported by the main manufacturers (both parameters are required in SQL Server). The function description will be corrected in the next version of the document.
- Some other functions need to be documented more carefully:
-
LONGITUDEandLATITUDE - the units of the returned values need to be specified (presumably degrees). So do the ranges: I guess LATITUDE will return a value in the range [-90,+90]. LONGITUDE suggests (to me) a quantity in the range [-180,+180], but given that it's basically talking about Right Ascension people will probably be expecting [0,360]. Or can it be either, left up to the implementation? (I suggest not). Either way, document the intention. Permitted ranges should also be noted for supplying latitude and (especially) longitude values as arguments to the various geometry functions - will values outside the nominal range of lat/long be wrapped automatically or are they errors?
-
AREA - what are the units of the return value?
- VOQL-TEG Answer: The
LONGITUDEandLATITUDEfunctions are meant to extract the first and second coordinate from an attribute ofPOINTtype. No coordinate transformation is supposed to take place.
-
- Coordinate systems: Is there a vocabulary for these? The only example given is 'ICRS' and the BNF is specified only as
<string_value_expression>. Are coordinate system specifications intended to be lifted from STC (or elsewhere) or is it intended that the implementation may define any values it likes for known coordinate systems? In practice - if only for testing purposes - it is often convenient to use "the default coordinate system", i.e. whatever is in use in the database; is there some way this can be selected?
VOQL-TEG Answer: In Trieste we agreed to define a vocabulary for the possible coordinate systems.
- It seems a bit strange that you can extract the longitude and latitude, but not the coordinate system from a POINT. But maybe it's OK - I must admit I don't have a clear idea of how these constructs are expected to be used.
VOQL-TEG Answer: A function for extracting the coordinate system from aPOINTwill be added.
- Comments by ArnoldRots - 16 May 2008
yyyy[-mm[-dd[Thh[:mm[:ss[.s...]]]]]]:
I.e., the limited ISO 8601 format only, as defined for FITS and in STC. VOQL-TEG Answer: RDBMSs implicitly convert strings to internal (datetime or timestamp) form using a variety of techniques; ISO8601 format is an acceptable format already. As with other string representations, it is a service capability (possibly mandatory) to understand specific formats. The PR introduces a "geometry" data type, but it really is a hodgepodge of stings - essentially, a geometry data type is defined, if I understand it correctly, as a string that contains a function call, with a variety of functions and parameters that seem to be reinventing STC. If one wants to introduce a new data type, there is a much more elegant and ready-made option: an STC-S string. I would be more inclined to call it an STC data type, since the term geometry data type is misleadingly narrow, but that can be discussed. VOQL-TEG Answer: As described in the attached presentation (and above) STC can be used within the REGION construct; it is up to a service to specify that it understands STC and they can support as little or much as is feasible. The ADQL language itself remains independent of any specific string representation or version thereof. In the Interop discussions it was explicitly stated that we expect TAP services to use standards (for region and the coordinate system arguments). The text within the PR will be expanded to make this clear. Having defined such a data type, one can use the to-be-developed STC library to verify its validity and manipulate (interpret) the string, and define a set of STC core functions to provide the necessary operations, such as intersection, union, contained_in, etc. The STC-S string has the added advantage that it can also handle coordinates, so that problem would be resolved, too. As it is, the PR basically develops its own STC model, through an ad-hoc set of functions and parameters. This, I think, is undesirable. If the IVOA has an accepted standard for certain structures, one should use it, rather than roll one's own. The reason is a very practical one: it encapsulates the model issues in that particular standard, allowing its related library to handle the details in a transparent and uniform way across applications, allowing models to evolve as necessary without impacting the users. If ADQL and SIAP and SSAP and Registry all would start using their own definitions of, say, Circle we create chaos for the user. I have heard many complain that incorporating STC greatly complicates the systems into which it is to be integrated. That is nonsense. I have said it numerous times and will say it again: There is nothing wrong with applications recognizing only a limited subset of STC structures and coordinate systems - returning an error if they encounter something they cannot handle. That is the way STC was incorporated into VOEvent - and it works. It's perfectly acceptable to accept only simple strings and a limited set of shapes and coordinate systems. It is not hard to parse those strings and the enormous advantage is that (a) one is compliant with an IVOA standard and (b) one is ready to expand functionality to include more cases, shapes, and coordinate systems at any time, without any changes to the interface, just by adding code to the server software. Aside from the fact that it is not a good idea to develop a new STC model for ADQL, there are a number of specific problems with the way it has been done, as well. The only coordinate system flavor recognized in the PR is 2-D spherical. Coordinates are specifically assumed to consist of a longitude and latitude. This may seem sensible, but it means that one is painting oneself into a corner for future extensions. VOQL-TEG Answer: The defined geometry types are 2-D and the LONGITUDE/LATITUDE functions unnecessarily restrict it to spherical coordinates (see above for more about LONG/LAT). VOQL-TEG Answer: The PR was arrived at by balancing usefulness and complexity and the TEG feels we have found the best comprimise. We have considered future extensibility and found that it is plausible to make only modest extensions beyond the current PR. It is feasible to add 1-D regions (intervals) and 2-D ellipse in a way that is consistent with the PR. After that, one would have to break with SQL to add much value. Another issue is that, again, if I understand it correctly, the parameters in the function calls may be valued or consist of column references. The problem is that there is no way to handle the coordinate system if the coordinate elements consist of references. I cannot say "function ('ICRS', table.ra, table.dec)" because I do not necessarily know whether the table contains ICRS positions - nor should I have to. VOQL-TEG Answer: The first argument should be another column in the table, or the whole construct (a POINT) should be taken from a single column of the table. See attached presentation. If I ask whether the positions in a table lie within a certain region that I define in a certain coordinate system, the server (knowing what its own coordinate system is, one would hope) should perform the necessary transformations. The PR specifies that when a server cannot perform a necessary transformation, it should return a NULL. I question whether this is a sensible thing: it is indistinguishable from "I have no data". I would much rather see a standardized warning returned.
- Comments by JohnTaylor - 22 May 2008
Comments by TCG
Chairs should add their comments under their name.Tom McGlynn (Applications WG)
I have lots of comments. The most substantial are that I think we need to clarify the nature of geometry objects within the database better than is currently done and also think about whether we should entangle the astronomical concept of coordinate systems with the purely geometric concepts of regions on a sphere -- I think we should not. The first I think needs to be addressed before I'd be at all comfortable with this -- it doesn't seem like the behavior is currently well defined. The second reflects my view of how things should be done, so there I would be more reluctant for my viewpoint to impede progress.- 2.1.2 Many existing databases have case sensitive identifiers (including ours). Requiring case insentitivity in ADQL will be extremely costly for us to implement.
- General Elements of the ADQL syntax that are extensions of standard SQL syntax should be clearly identified (e.g., TOP)
- Use of TOP: The syntax for limiting queries to a given number of rows is extremely varied among the databases. Use of the LIMIT syntax (used by MySQL for example) might be preferred since it allows not just the specification of the first N rows, but also the n-m'th row. However this isn't a big deal.
- Section 2.2.1 is completely incomprehensible to me. It needs to have examples of what is included and excluded.
- 2.3.1 The arguments (and return values) of the mathematical functions need to be clarified with regard to radians versus degrees.
- 2.3.2 Can geometric types be used in subselects. What is the effect of null values in the creation of a geometry type? Do they become null? Can tables support persistent geometry types? How do geometry types act under standard SQL operations? E.g., if I specify the same point in two different coordinates systems are they equal? Can >< operations be done on these? E.g., is CIRCLE(ICRS, x,y,10)>circle(ICRS,x,y,5), .... There needs to be a lot more discussion of how geometry objects work.
- 2.3.2 I don't like the contains function returning a numeric value. It's a geometric equivelent of the SQL IN function and should be a boolean. Intersects is better as an number but only because it could give a value (between 0 and 1) that reflects the degree of intersection. If we want these things to be boolean, then that's what they should be. SQL does not use integers as substitutes for boolean values and we shouldn't import that C atrocity into ADQL.
- 2.3.2.2.5.1 The units of area are not given. They should be steradians.
- 2.3.2.2.5.2 What is the centroid of a geometry? Does it need to be inside the geometry? How it it to be computed? Is it required to be computed the same way on all platforms? E.g., what is the centroid of a belt around the equator? One of the poles? Which one? Lots of questions here.
- 2.3.2.2.5.3 I don't believe that any geometry other than perhaps point should refer to a specific coordinate system. So I'd recommend a syntax of CIRCLE(POINT('ICRS', x,y),radius). Also, there should be no reference to a cone in the sky in the normative section, though it could be noted that circles may be helpful in implementing cone searches. My view is that we want to isolate the explicit astronomical elements of ADQL into a very few areas. The extensions to SQL should be made as primarily new mathematical capabilities (such as regions) indenpendent of astronomical peculiarities. Also, I believe that all angular measurements should be in radians.
- 2.3.2.2.5 generally. This is weirdly structured. The functions that generate area datatypes should be specified in one section and those things that operation on them to produce standard types (or perhaps booleans) elsewhere. I.e., point, circle and polygon (and box if retained in one section. Area, distance, latitude, longitude in another and perhaps contains and intersects in a third.
- 2.3.2.2.5.5 Again I think that all functions that take angular measurements should be in radians. Otherwise it gets very confusing as you combine sines and distances and such.
- 2.3.2.2.5.7-9 I think there is a very important issue concealed here. In my conception, we allow users to specify a coordinate system in creating a point primarily so that the local system can make any necessary transformation to get the point into what coordinate representation is used locally. However since the ADQL query may be a multidatabase query it's possible that multiple native coordinate representations are involved. So either we are going to specify a standard way in which geometries are passed between databases in a distributed query, or we need to tag each of the geometries that are created with the coordinate system in which they were created. As I mentioned above, I don't think we want to pollute geometries with this very astronomical concept. They should be defined on an abstract unit sphere.
- 2.3.2.2.5.10 I'd have made it POLYGON(POINT('ICRS', 10,10),POINT('ICRS',10,20),POINT('ICRS', 15,15). All the code for handling coordinate systems is embedded in POINT (and maybe LATITUDE and LONGITUDE). Despite being longer this is easier for me to read and it's more powerful.
- 2.3.2.2.5.11 I do not believe that the rectangle should be supported. Note that the statement that transforming a rectangle from one coordinate system to another results in a polygon is incorrect. The parallels that define a rectangle in one coordinate system do not transform to great circles. Rectangles can be supported using standard SQL on coordinates where needed and need no special support within ADQL.
- 2.3.2.2.5.12 I see no reason for a special REGION syntax.
- 2.3.3. Are user defined functions supported? It is unclear from this section. If they are not, then this section should be deleted until they are. Who are the users that define these functions? How and when do they get defined? How are they invoked after being defined? How persistent are they? Can they be shared between users?....
Christophe Arviset (TCG)
I approve the document. As a minor editorial comment, it would be useful to add page numbers. VOQL Chair Answer Page numbers will eventually be added (end VOQL Chair Answer)Keith Noddle (Data Access Layer)
I approve the document. As has been stated elsewhere, this is an evolutionary step towards a full ADQL specification and as such represents a substantial effort on the part of the authors; they are to be congratulated on their work. I feel sure that the compromises included in the document (especially with respect to REGION etc) will allow practical implementations to help guide the way forward in future versions of ADQL.Matthew Graham (Grid & Web Services WG)
I approve this document: it is reassuring to see that there are bona fide implementations for what is clearly a standard on an evolutionary track.Mireille Louys (Data Models WG)
The document is now easier to read and consistent, has integrated many of the improvements asked in the previous phase.- I just noted a detail : in section 2.4.15 Geometry in the SELECT clause, the example
- dissemination of this language:
Francois Ochsenbein (VOTable WG)
I approve the document (see however comments below) Section 2.3 Mathematical and Trigonometrical Functions: in the power function, the second argument is a double (not an integer); and atan2 is not exactly the arc tangent (it gives the polar angle of a point having the cartesian coordinates (x,y)) VOQL Chair Answer Will eventually be corrected (end VOQL Chair Answer) Section 2.4 I feel the authors made a good compromise between the divergent points of view (on my side, I still would have preferred ADQL to stick to a single reference system, i.e. remove completely the requirement of a COORDSYS -- any of the described regions can be expressed in the ICRS, and coordinate system transformation is outside the scope of ADQL). Nevertheless having to refer to a definition of the time to define a simple point on a sphere does not look rational...Pedro Osuna (VOQL WG)
I approve.Ray Plante (Resource Registry WG)
All in all, this is a well written document, and nicely separates itself as a language from the service that utilizes it. I traced the BNF through a few of the concepts, and it appears in good shape. I did not review this document during the RFC period as I should have, so some of my comments are out-of-scope of the TCG comment period. Nevertheless, I summarize them briefly here, and, for the benefit of the record and the authors, detail them above in the RFC section:- TOP is not SQL92 and is highly problematic, so I don't think it should not be included.
- The defined semantics for the language elements are not spelled out sufficiently to test whether an implementation is correct. Is this really what we want?
- There are a few minor wording improvements that could be made
I would like to see a more restricted, unambiguous, and implementable definition than what we have here if it is to go forward as a REC.I believe that this can be easily rectified with the following:
- a blanket statement near the beginning indicating that where a term in ADQL is identical to that in SQL92, the semantics of that term (and its arguments) are the same as in SQL92.
- spelling out the meanings and types of all arguments for functions. (Suggested sentences are given here.)
- you mention that the region functions are based on STC; I recommend that an explicit statement indicating that the arguments are identical in type and meaning as the arguments defined for STC/s.
When a geometry data type within the context of the WHERE clause, the given coordinate system may not match the system in use by the underlying database that the query is applied to. In this case, it is expected that the service or application will perform the necessary coordinate conversions to evaluate the constraints. If this conversion is not possible, then an appropriate error should be thrown as defined by the service or applicaiton using ADQL.VOQL Chair Answer * TOP falls under the category of ADQL Reserved Words, as it was understood as an extension to the SQL92 reserved keywords. First attempts on TAP to sketch query ranking functionalities for large datasets showed the need to provide such features in the language rather than in the service by combining the use of ORDER_BY and TOP. At that time, no problem was found to include TOP despite the fact that it can be named and implemented in different ways in different database systems. That is why it is defined as an ADQL extension and not and SQL reserved word. The first time TOP appears is in the April 2007 version. Later, in Beijing (May 2007), some people expressed concerns that TOP might have different implementations in different DB systems, but there was no final requirement from any party to remove it. Nor was there such a requirement within any of the members of the VOQL-TEG. In case the TOP would create troubles when implementing services making use of ADQL language, it could be deprecated in next ADQL version. For the time being, it would give a good functionality for eventual TAP-like services, and therefore it should stay in. * I am not sure I understand the part on implementations. The BNF parser is a full system very kindly developed by Jeff Lusted (Astrogrid) and which source code, packaging information, documentation, etc. is fully available at the pages given in the referred URL (you only have to click on the corresponding menu at the left to navigate through the source code, etc.). As an example, please find attached this URL where part of the java code can be seen: http://www.astrogrid.org/viewcvs/astrogrid/adql2/ The language is defined via a BNF (as was agreed several times in the past). The above implementation is a full implementation of the parser. Actually, I believe this is one of the most thorough implementation there has ever been of an IVOA proposed standard. The fact that Protocols (and Services making use of them) will make use of the ADQL within their future implementations shall not be confused with an implementation of the BNF of the language itself, which -as said above- is attained by the referenced parser. * A language should not give instructions for services to do any type of transformation. It is up to the service to understand that if coordinates are given to it in equatorial RA,DEC and the service only has galactic L,B then either an error shall be returned or the calculation done silently. * In response to: [...] blanket statement near the beginning indicating that where a term in ADQL is identical to that in SQL92, the semantics of that term (and its arguments) are the same as in SQL92[...] I believe the document clearly states what is SQL92 standard and what is ADQL specific. * In response to: [...]you mention that the region functions are based on STC; I recommend that an explicit statement indicating that the arguments are identical in type and meaning as the arguments defined for STC/s.[...] read points 2.4.1 and 2.4.14 of the document (see excerpt here) [...]A special attention has to be paid to the REGION function. As can be seen more in detail in Section 2.4.14, this construct is a general purpose function and it takes a string value expression as argument. The format of the string is to be specified by a service that accepts ADQL by referring to a standard format. Currently STC/s (See [3] and [4]) is the only standardized string representation a service can declare.[...] (end VOQL Chair Answer) RayPlante replies:
- I'm fine with your response regarding TOP (thanks).
- Thanks for the extra link to the implementation; I couldn't seem to find my way to the source from those navigation links on cited page. (I've queried Jeff from some extra detail.) I have no problems with the BN--it looks great. My main issue is with the semantics.
- In response to issue of transformations: It's fine if you do not want to go as far as indicating that it is an error when the transformation can not be done. However, is it your intention that a service be allowed to interpret the POINT positions as in a coordinate system other than the one provided? If the answer is yes, then we're okay; although, there is a bit of inconsistancy with CONTAINS and INTERSECTS addressing transformations but other not. It would be helpful if a statement regarding this point be made either way. The important point is that these functions are intended to mean something regardless of where ADQL is used; that meaning needs to be respected and, more importantly, made clear.
- In response to [...]I believe the document clearly states what is SQL92 standard and what is ADQL specific[...] I apologize if I missed this, but could you point to the statement[s] that are intended to clarify this?
- In response to reference to 2.4.1 and 2.4.14: Perhaps I was unclear. Is there an intended relationship between, say, CIRCLE() in ADQL and the "Circle" sub-phrase in SQL/s? If so, then stating this would help clarify the semantics of this function. Please see my suggested edit for 2.4.5 CIRCLE
Sebastien Derriere (Semantics WG)
I share some of the comments already done, and am reassured by the VOQL TEG answer. There are only minor issues I hope can be clarified:-
- Properly define the argument and return units (degree/radians) for trigonometric functions (as Mark Taylor above)
- Why does round(x,n) take 2 arguments? Is round(0.5)=1 ?
- section 2.3.2.2.2 indicates in "later additions" the "overlaps" keyword. But note this is a SQL reserved keyword in section 2.1.2.
Rob Seaman (VOEvent WG)
I approve.Bob Hanisch (Data Curation & Preservation IG)
Here are some comments that Arnold sent to Pedro et al., based on discussions that he, I, and Alex had in the past couple of weeks. 1. section 2.1.3 correctly states that ISO 8601 is an acceptable format and that therefore it would be perfectly in order for services to indicate that they understand it. But that was not the point I was making. I suggested that recognizing ISO 8601 date-time be mandatory, for the simple reason that it would guarantee a single date-time format that will be recognized by all services. As it currently stands, there is no such thing and clients would have to query each service as to what it does and does not understand - or break up the datetime in small pieces and handle them separately. I feel that time is sufficiently essential in astronomy to warrant requiring at least one common format (other than JD or MJD) that clients can count on to be understood by all servers. 2. The geomerical functions in Section 2.4 are better defined in the present version, but I still feel that the ADQL-specials complicate life unnecessarily. I do not see that there is much to be gained (and, imho, much more to be lost) by allowing the same thing to be specified by: CIRCLE ('UTC-FK5-GEO', 25.0, -20.0, 1) and: REGION ('Circle FK5 GEOCENTER 15.0 -20.0 1.0') I am afraid that the duplication is going to lead to a situation where some clients will support one, but not the other, and services that support the other, but not the one. That is not interoperability. Alternatively, everyone has to write code to understand both - the information is the same, but the formatting subtly different. (Actually, strictly speaking, the former would need to look up the AstroCoordSystem in the library that UTC-FK5-GEO points to, so it's worse off than using STC-S directly.) This is a waste of resources and would like to advocate that the only geometric function we recognize be REGION with as argument an STC-S string. I think the STC-S syntax in REGION is no more complicated or obtuse than the arguments in CIRCLE. It has been said that SQL cannot validate the STC-S string, but the same is true for CIRCLE: SQL can count the arguments and their types, but cannot make a judgment as to whether their values make sense - that's left to the CIRCLE function. The same is true for REGION: SQL can check the number and type of the parameters (one string), but it is left to REGION to validate the string. There is the issue of passing references (like "t.ra" and "t.dec") on. I am sure we can come to an agreement on that, for instance by escaping the names, like: REGION ('Circle FK5 GEOCENTER @t.ra @t.dec 1.0') 3. The meaning of a coordinate system specification in the presence of referenced coordinate values. To some extent, this refers to Section 2.4.9 which is not very clear, but it is more fundamental than that. What does it mean when I say (as in the example above): REGION ('Circle FK5 GEOCENTER 15.0 -20.0 1.0') or: CIRCLE ('UTC-FK5-GEO', t,ra, t.dec, 1) Clearly, it cannot mean: "I will interpret (t.ra,t.dec) as FK5", because they may not be and besides, the server should know. The most intuitive meaning would probably be: "I want (t.ra,t.dec) in FK5, so, if they aren't, do the transformation." But that may not be the most practical solution under all circumstances. The point is, though, this needs to be clarified and specified very explicitly. (End of comments.) Subsequently there was an e-mail response from Jeff Lusted: "I think most members of the VOQL TEG group are sympathetic to the concerns you express, but feel we have gone well beyond v1.0 and need implementation feedback before we make further progress. However, I speak for myself here, and the opinions in the following points are my own" If we need implementation feedback then I do not see how we are quite at the point of accepting this as a REC. We made an exception to this guideline for STC, at my (and others) urging. Some still think this was a mistake. However, STC had been implemented and had been used in several software environments. Who has an ADQL parser that can accept queries in the variety of syntaxes that are described in the document? I would like to see a more restricted, unambiguous, and implementable definition than what we have here if it is to go forward as a REC. VOQL Chair Answer: Issue 1 concerns imposing a specific date-time format. This is to be addressed in either Protocols or Services, rather than language Issue 2 concerns the usage and semantics of the REGION constructs. The usage of those has been discussed at length and agreed at the last interop meeting in Trieste. The point made worries about possible use of two ways of formatting a query, but does not imply an inherent problem. Issue 3 might require a small clarification of the final version of the document. With respect to implementations, there are at least two implementations of the ADQL doc: - a full parser implementing all the BNF defining the language (c.f.: http://deployer.astrogrid.org/software/adqlparser-2008.2/index.html) - a service for Table Access from CVO making use of ADQL A definitive implementation of a BNF-based language should be a parser that understands the language constructs, and this is already available (see above). Other implementations will take place whenever protocols making use of the language are available so that clients can implement them. A parallel example is easily found with the SQL language itself. I believe therefore the document is ready for REC. (end of VOQL Chair Answer) New Comment, 28 October 2008,by BobHanisch Following a discussion held 27 October 2008, in conjunction with the IVOA Interop meeting, some clarifying text has been added to the document that connects the two ways of expressing regions to the fundamental definitions in STC. In this sense, the function forms are convenient representations for certain database functions, and the function forms are semantically equivalent to the more general REGION expressions. With these clarifications my concerns are alleviated, and I approve the document for advancement to REC.Herve Wozniak (Theory IG)
Approved.Masatoshi Ohishi (Astro-RGIG)
Approved.- Name: aaron price
- Email: aaronp@aavso.org
- Company Name: AAVSO
- Company URL: http://www.aavso.org
- Telephone: 617-354-0484
- Fax:
- Country: USA
- Comment:
- WelcomeGuest to learn TWiki
- Sandbox web to try out TWiki
- AaronPriceSandbox just for me
- Show tool-tip topic info on mouse-over of WikiWord links, on or off:
- Set LINKTOOLTIPINFO = off
- Horizontal size of text edit box:
- Set EDITBOXWIDTH = 70
- Vertical size of text edit box:
- Set EDITBOXHEIGHT = 22
- Style of text edit box.
width: 99%for full window width (default),width: autoto disable.- Set EDITBOXSTYLE = width: 99%
- Optionally write protect your home page: (set it to your WikiName)
- Set ALLOWTOPICCHANGE =
Work around abusive users
Beginning
This work has been proposed by Mark Holliman et we had a first discussion during the Heidelberg IVOA interop in May 2013 (GWS Session 2). The aim is to define a Best practice against abusive uses of VO services.Inputs from Mark Holliman
Application Considerations to prevent Abusive User Behaviour
This page is meant to be a sounding board and general discussion area for the issue of preventing users from knowingly or unknowingly abusing services through VO enabled applications. The topic is of interest both to application developers and data providers. The goal is to make VO enabled applications less likely to enable problematic behaviour by default and to develop possible solutions for data providers that can alleviate or prevent service disruptions. Common examples of "abusive behaviour" include Denial of Service attacks (where a user overloads a service with requests, effectively crashing it), and ...Incident Examples and Proposed Solutions
- Topcat Multicone DoS on WFAU services
- Description: In January and March 2013 there were incidents where users unknowingly crashed the cone search services for WFAU's archives. In both cases the users were running multiple instances of Topcat, performing multi-cone queries with the multi-threaded setting activated. In the worst case the user was effectively running 600 concurrent connections to the cone-search services continously for 72 hours, with each query requesting 1 degree square of tabular results. The web container running the services was overloaded and eventually crashed, bring down access for all users.
- Lessons Learned: A bug was identified in Topcat whereby the "Stop" button didn't actually stop subsequent queries from being submitted. The user thought they had stopped their requests multiple times, when in fact the queries continued in the background alongside all subsequent queries they were submitting. This bug is now fixed.
- Proposed Solutions:
- VO Applications should use low default settings for options like multithreading. When possible, they should also provide warning messages when a user selects a setting that could be considered unreasonable.
- VO Services could provide 'HTTP 307 Temporary Redirect' messages to clients that are overloading their services. The temporary redirect would point to a page with time delayed redirect, and the time on that would be set by the server to delay query response. As more requests come in the wait time would grow. The client applications must be capable of understanding the redirect message and act accordingly.
Inputs from CDS
The VizieR solution for abusive users is still to set a delay for HTTP queries identified as "abusive". The solution is manual: it adds the IP in a "black" list where queries are affected by a "sleep 5" before being executed. But as the capacity of VizieR to ingest more queries has been increased during the last year and this action is not often used.We should also consider the fact that an abuse acces to a service could be somewhere in a workflow (a simple pipeline or a more complex execution plan). In this case the user will probably not know that he is guilty. For him, the execution of his query could be considered as slow (if a delay is used) or not possible(if a redirection is used) .
Inputs from ThomasBoch
In the CDS cross-match service, we output a 503 code (server unavailable) if the service is too busy to process the request. We should also consider HTTP code 429 (too many requests, see http://tools.ietf.org/html/rfc6585#section-4 )Inputs from CADC
In the past at the CADC we had similar problems, where our processing users were making a very large number of requests to our VOSpace virtual storage system. Sometimes a user would, inadvertently, because perhaps of a bug in their code or because they have a resource intensive script, be making so many requests to VOSpace that it made the system very slow for other users. Sometimes, the overload would exhaust our low level resources and make the system unusable. Initially, we implemented a "throttling" scheme, where users were grouped and limited to a certain number of simultaneous requests. When users reached their limit, they were throttled: we would issue of 503 "Service Unavailable" with a value in the "Retry-After", suggesting a time in which the user should retry their request. Our clients were written in a way to recognize these HTTP responses and delay actions appropriately. However, we found it hard to set the correct limit numbers and identify the groups, causing throttling too early or too late. We then realized that it is the limits of our low level resources that we must understand and protect. These are resources such as database connections, threads, file descriptors, etc.. So, using pools to gain access to these resources, we then issued the 503 responses when these pools were exhausted. We found this was the true indicator of the amount of load we could handle and required no maintenance or updating as our user base grew. Only the pool sizes need be adjusted when our low level resource capabilities changed. The other advantage of using pools to control resources is that the users' connections will wait in line for the pool resource. You have the ability to set the amount of time users wait before we give up and issue the 503 response. So far this seems to be working well and are now able to scale with our growing user base.Inputs from ???
Feel free to completeApplication Considerations to prevent Abusive User Behaviour
This page is meant to be a sounding board and general discussion area for the issue of preventing users from knowingly or unknowingly abusing services through VO enabled applications. The topic is of interest both to application developers and data providers. The goal is to make VO enabled applications less likely to enable problematic behaviour by default and to develop possible solutions for data providers that can alleviate or prevent service disruptions. Common examples of "abusive behaviour" include Denial of Service attacks (where a user overloads a service with requests, effectively crashing it), and ...Incident Examples and Proposed Solutions
- Topcat Multicone DoS on WFAU services
- Description: In January and March 2013 there were incidents where users unknowingly crashed the cone search services for WFAU's archives. In both cases the users were running multiple instances of Topcat, performing multi-cone queries with the multi-threaded setting activated. In the worst case the user was effectively running 600 concurrent connections to the cone-search services continously for 72 hours, with each query requesting 1 degree square of tabular results. The web container running the services was overloaded and eventually crashed, bring down access for all users.
- Lessons Learned: A bug was identified in Topcat whereby the "Stop" button didn't actually stop subsequent queries from being submitted. The user thought they had stopped their requests multiple times, when in fact the queries continued in the background alongside all subsequent queries they were submitting. This bug is now fixed.
- Proposed Solutions:
- VO Applications should use low default settings for options like multithreading. When possible, they should also provide warning messages when a user selects a setting that could be considered unreasonable.
- VO Services could provide 'HTTP 307 Temporary Redirect' messages to clients that are overloading their services. The temporary redirect would point to a page with time delayed redirect, and the time on that would be set by the server to delay query response. As more requests come in the wait time would grow. The client applications must be capable of understanding the redirect message and act accordingly.
* Collaborative Page for AccessData This page is to capture details of discussions around the AccessData capability being deveveloped for the Multi-dimensional data access CSP project. Primary discussion occurs on the dal@ivoa.net mailing list. Summaries from meetings, intermediate documents, and important links/resources can be added here. It is important to note that the grand plan for AccessData is to produce incremental versions of the document starting with basic features and adding more complex things later -- once the basics are implemented. Features must be prototyped and proven before they should be added to the specification. * Documents The initial draft document is WD-AccessData-1.0-20140312 and is available on this page only (see below) * Issues
+*Access Format*
This field is still being defined (some discussion is given in the document text). A draft specification will appear in a future version of the document. The intention is to specify both the basic file format as well as the astronomy-specific format, and details such as whether compression is used. This is required, for example, to tell the user what software will be required to do anything useful with the data product. While we can specify how to compose this field, and supply some standard values, it will really be up to the broader community and data providers to define specific values to represent all their data products. -- DougTody 2 March 2011 If the standardisation of the access_format field is not ready for this version, I would propose to leave it out completely to avoid people (mis)using it now, and to avoid non-compliant services polluting the vosphere after the access_format is formalised. -- AlbertoMicol 4 March 2011
- Semantics should probably define those. How is the timescale for a good enough list of values?
- Call to the community for contribution to build up examples
This field is still being defined (some discussion is given in the document text). A draft specification will appear in a future version of the document. The intention is to specify both the basic file format as well as the astronomy-specific format, and details such as whether compression is used. This is required, for example, to tell the user what software will be required to do anything useful with the data product. While we can specify how to compose this field, and supply some standard values, it will really be up to the broader community and data providers to define specific values to represent all their data products. -- DougTody 2 March 2011 If the standardisation of the access_format field is not ready for this version, I would propose to leave it out completely to avoid people (mis)using it now, and to avoid non-compliant services polluting the vosphere after the access_format is formalised. -- AlbertoMicol 4 March 2011
access_format in the draft refers to MIME types, but no real mime types are provided. Real MIME types for FITS files are application/fits for non-imaging, or multi-extension FITS files, and image/fits for FITS files which can be directly interpreted as images. Following IETF RFC4047, image/fits can be generalized for data cubes, as long as the corresponding WCS information is present, and software such as fv can interpret it as a 3D data cube. Even if all of the enumerated values shown in 4.7 and B5.2 (which by the way, are the same) were valid as MIME types, there would be a need to register all of those MIME types, and I think we are served with both image/fits and application/fits.
So, for the whole access_format section, I would use an approach similar to UCD specialization, where we have a vocabulary composed of data formats, dimensionality, and compression.
- Data formats:
fits,class,ms,aedm(corresponding to FITS, CLASS, Measurement Set, and ALMA Export Data Model). - Compression:
z,gz,zip,none,embedded(meaning tile compression in FITS, for instance) - Dimensionality:
image,datacube(regular mesh data cube from radio interferometry, or regridded radio OTF, for instance),spatial.spectral(irregular spatial mesh, with possibly irregular spectral binning, such as those from non-regridded radio OTF, IFU, or MOS instruments).
My Links
- ATasteOfTWiki - view a short introductory presentation on TWiki for beginners
- WelcomeGuest - starting points on TWiki
- TWikiUsersGuide - complete TWiki documentation, Quick Start to Reference
- Sandbox - try out TWiki on your own
- AdaNebotSandbox - just for me
My Personal Preferences
- Show tool-tip topic info on mouse-over of WikiWord links, on or off:
- Set LINKTOOLTIPINFO = off
- Preference for the editor, default is the WYSIWYG editor. The options are raw, wysiwyg:
- Set EDITMETHOD = wysiwyg
- More preferences
TWiki has system wide preferences settings defined in TWikiPreferences. You can customize preferences settings to your needs: To overload a system setting, (1) do a "raw view" on TWikiPreferences, (2) copy a Set VARIABLE = valuebullet, (3) do a "raw edit" of your user profile page, (4) add the bullet to the bullet list above, and (5) customize the value as needed. Make sure the settings render as real bullets (in "raw edit", a bullet requires 3 or 6 spaces before the asterisk).
Related Topics
- ChangePassword for changing your password
- ChangeEmailAddress for changing your email address
- UserList has a list of other TWiki users
- UserDocumentationCategory is a list of TWiki user documentation
- UserToolsCategory lists all TWiki user tools
- Comment:
My Links
- ATasteOfTWiki - view a short introductory presentation on TWiki for beginners
- WelcomeGuest - starting points on TWiki
- TWikiUsersGuide - complete TWiki documentation, Quick Start to Reference
- Sandbox - try out TWiki on your own
- AdamBoltonSandbox - just for me
My Personal Preferences
- Show tool-tip topic info on mouse-over of WikiWord links, on or off:
- Set LINKTOOLTIPINFO = off
- Preference for the editor, default is the WYSIWYG editor. The options are raw, wysiwyg:
- Set EDITMETHOD = wysiwyg
- More preferences TWiki has system wide preferences settings defined in TWikiPreferences. You can customize preferences settings to your needs: To overload a system setting, (1) do a "raw view" on TWikiPreferences, (2) copy a
Set VARIABLE = valuebullet, (3) do a "raw edit" of your user profile page, (4) add the bullet to the bullet list above, and (5) customize the value as needed. Make sure the settings render as real bullets (in "raw edit", a bullet requires 3 or 6 spaces before the asterisk).
Related Topics
- ChangePassword for changing your password
- ChangeEmailAddress for changing your email address
- UserList has a list of other TWiki users
- UserDocumentationCategory is a list of TWiki user documentation
- UserToolsCategory lists all TWiki user tools
- Comment:
The Data Access Layer (DAL) Tutorial at ADASS XIII
Collection of Slides and Information for the DAL Tutorial Session.- DAL Overview (D.Tody) (.ppt)
- How to Implement the Cone Search and SIA Services (G.Greene) (.ppt)
- How to Navigate VO Datasets using VO Protocols (T.McGlynn, R.Plante)(.ppt)
- How to Adapt Existing Archives to SIAP
The XMM Newton and ISO Cases (P.Osuna) (.ppt)
Interfacing the CDS Aladin image service and client viewer (F.Bonarrel) (.ppt)
ADQL function names
The ADQL standard itself does not impose any restrictions on the names that may be used for user defined functions. To avoid name collisions when solving common problems, we recommend that TAP service providers adopt a policy of consistently prefixing the names of user-defined functions provided by their services with an acronym that represents the institute, department or archive where the function originates. If the Wide Field Astronomy Unit (WFAU) use the department acronym,wfau, to prefix their user defined function names:
wfau_align()
wfau_convert()
and the German Astrophysical Virtual Observatory (GAVO) adopt a similar naming convention:
gavo_align()
gavo_convert()
this would help to avoid confusion when users encounter functions with similar names on different TAP services:
wfau_align()
gavo_align()
wfau_convert()
gavo_convert()
Note that the prefix indicates the origin of the function implementation, not the identity of the TAP service providing the function. If another TAP service provides a function with exactly the same behaviour as one of the WFAU functions, the function name should use the wfau prefix to indicate this.
wfau_align()
By ensuring that different implementations use different names, this scheme avoids potential conflicts when a user takes a query written for one TAP service and sends it to another TAP service. With the name prefixes, the second TAP service will reject the query if it does not provide exactly the same function as the first.
Without this fail fast approach, the second query may run without raising any errors or warnings, but due to differences in the algorithm used, the second query may produce subtly different results which could go undetected.
On the other hand, if the second TAP service does implement the same function, with exactly the same behaviour, then selecting this function using the name prefix gives the user a level of confidence that a query would produce equivalent results when run on different TAP services.
IVO standard functions
Theivo prefix is reserved for defining functions that are specified in IVOA recommendations or endorsed notes.
If one of the GAVO functions is adopted as part of an IVOA standard, then we could use the ivo prefix to indicate that it is an IVOA standard function.
ivo_align()
For backwards compatibility, a TAP services could provide the same function using different names:
gavo_align() -- original GAVO version
ivo_align() -- IVOA standard version
The same TAP service could also provide the WFAU implementation of the same function:
gavo_align() -- original GAVO version
wfau_align() -- alternative WFAU version
ivo_align() -- IVOA standard version
It is the responsibility of the IVOA working groups to ensure that the total set of ivo endorsed functions remains sensible.
In order to be accepted as an ivo function, a function MUST be defined in a format suitable for TAPRegExt and ideally have reference implementations for one or more of the major RDMS platforms.
Note that in general a TAP service is not required to implement these functions. However, if they do implement functions with these names, then their implementation MUST be consistent with the behaviour defined in the specification.
An example for a Recommendation that defines ivo functions is RegTAP, which defines the following functions:
-
ivo_nocasematchcase insensitive match. -
ivo_haswordword based search. -
ivo_hashlist_hashash '#' delimited word search. -
ivo_string_aggdelimited string aggregate function.
Prefix registry
The following table acts as an informal registry of function prefix names.| prefix | name | link |
|---|---|---|
| ivo | IVOA standard functions | http://ivoa.net/ |
| wfau | Wide Field Astronomy Unit, Institute for Astronomy, University of Edinburgh | http://www.roe.ac.uk/ifa/wfau/ |
| gavo | German Astrophysical Virtual Observatory | http://www.g-vo.org/pmwiki/ |
| mast | Mikulski Archive for Space Telescopes | http://archive.stsci.edu/ |
| eso | European Southern Observatory | http://archive.eso.org/ |
| cefca | Centro de Estudios de Física del Cosmos de Aragón (CEFCA) | https://archive.cefca.es/ |
Next steps
If you are adding new user defined functions to your TAP services, consider adding a prefix to the function name to indicate where the function comes from. If you already have user defined functions in your TAP services, introduce alternative names for them with an appropriate prefix and gradually encourage your users to use the prefixed names.Original text by -- DaveMorris with additions and suggestions from -- MarkusDemleitner
My Links
- ATasteOfTWiki - view a short introductory presentation on TWiki for beginners
- WelcomeGuest - starting points on TWiki
- TWikiUsersGuide - complete TWiki documentation, Quick Start to Reference
- Sandbox - try out TWiki on your own
- AdrianDamianSandbox - just for me
My Personal Preferences
- Show tool-tip topic info on mouse-over of WikiWord links, on or off:
- Set LINKTOOLTIPINFO = off
- Preference for the editor, default is the WYSIWYG editor. The options are raw, wysiwyg:
- Set EDITMETHOD = wysiwyg
- More preferences TWiki has system wide preferences settings defined in TWikiPreferences. You can customize preferences settings to your needs: To overload a system setting, (1) do a "raw view" on TWikiPreferences, (2) copy a
Set VARIABLE = valuebullet, (3) do a "raw edit" of your user profile page, (4) add the bullet to the bullet list above, and (5) customize the value as needed. Make sure the settings render as real bullets (in "raw edit", a bullet requires 3 or 6 spaces before the asterisk).
Related Topics
- ChangePassword for changing your password
- ChangeEmailAddress for changing your email address
- UserList has a list of other TWiki users
- UserDocumentationCategory is a list of TWiki user documentation
- UserToolsCategory lists all TWiki user tools
- Comment:
If you are interested in sharing transportation from your arrival airport to Baltimore, please enter your name, e-mail address, and flight arrival information below.
If anyone is flying via/from Houston Intercontinental (IAH) on the afternoon 26/Oct, drop me a note: IgorChilingarian
Ivan Zolotukhin: I'll be picking Igor Chilingarian on a car at BWI on Oct 26 evening
| Name | Arrival (airport, date, time, flight number) | |
|---|---|---|
| Norman Gray | norman@astro.gla.ac.uk | BWI, 25 Oct 14:10, C2355 |
| Ivan Zolotukhin | iz_at_sai.msu.ru | JFK, 25 Oct 14:30, DL 31 -- see note |
| Mireille Louys | louys@astro.u-strasbg.fr | IAD, 25 Oct, 15:40, KLM 0651 |
| Sebastien Derriere | derriere@astro.u-strasbg.fr | IAD, 25 Oct, 15:40, KLM 0651 |
| Thomas Boch | boch@newb6.u-strasbg.fr | IAD, 25 Oct, 15:40, KLM 0651 |
| Andrea Preite-Martinez | andrea.preitemartinez@iasf-roma.inaf.it | IAD, 25 Oct, 15:40, KLM 0651 |
| Hervé Wozniak | herve.wozniak@obs.univ-lyon1.fr | BWI, 25 Oct 16:20 BA 0229 |
| Alasdair Allan | AlasdairAllan <aa@astro.ex.ac.uk> | BWI, 25 Oct 18:37, AC 7934 |
| RickWagner | rick@ucsd.edu | BWI, 26 Oct, 10:32, DL 982 |
| Andy Lawrence | al@roe.ac.uk | BWI, 26 Oct, 10:51am, UA1226 from ORD |
| Vivekananda Moosani | vivekananda_moosani@persistent.co.in | BWI, 26 Oct, 14:00, DL 8939 |
| Petr Skoda | skoda@sunstel.asu.cas.cz | BWI, 26 Oct, 17:20, BA229 |
| Fabien Chereau | fchereau@eso.org | BWI, 26 Oct, 17:54, DL 734 |
| Arnold Rots | arots@head.cfa.harvard.edu | BWI, 26 Oct, 18:49, DL6931 from BOS |
| Sherry Winkelman | wink@head.cfa.harvard.edu | BWI, 26 Oct, 18:49, DL6931 from BOS |
| Matthew Graham | mjg@caltech.edu | BWI, 26 Oct, 18:59, UA0306 from LAX |
| Igor Chilingarian | Igor.Chilingarian_at_obspm.fr | BWI, 26 Oct, 23:03, CO 426 -- see below |
If anyone is flying via/from Houston Intercontinental (IAH) on the afternoon 26/Oct, drop me a note: IgorChilingarian
Ivan Zolotukhin: I'll be picking Igor Chilingarian on a car at BWI on Oct 26 evening
My Links
- ATasteOfTWiki - view a short introductory presentation on TWiki for beginners
- WelcomeGuest - starting points on TWiki
- TWikiUsersGuide - complete TWiki documentation, Quick Start to Reference
- Sandbox - try out TWiki on your own
- AitorIbarraSandbox - just for me
My Personal Preferences
- Show tool-tip topic info on mouse-over of WikiWord links, on or off:
- Set LINKTOOLTIPINFO = off
- Preference for the editor, default is the WYSIWYG editor. The options are raw, wysiwyg:
- Set EDITMETHOD = wysiwyg
- More preferences TWiki has system wide preferences settings defined in TWikiPreferences. You can customize preferences settings to your needs: To overload a system setting, (1) do a "raw view" on TWikiPreferences, (2) copy a
Set VARIABLE = valuebullet, (3) do a "raw edit" of your user profile page, (4) add the bullet to the bullet list above, and (5) customize the value as needed. Make sure the settings render as real bullets (in "raw edit", a bullet requires 3 or 6 spaces before the asterisk).
Related Topics
- ChangePassword for changing your password
- ChangeEmailAddress for changing your email address
- UserList has a list of other TWiki users
- UserDocumentationCategory is a list of TWiki user documentation
- UserToolsCategory lists all TWiki user tools
- Comment:
- Name: AjitKembhavi
- Email: akk@iucaa.ernet.in
- Company Name: IUCAA
- Company URL: http://www.iucaa.ernet.in/
- Telephone: +91-20-25604101
- Fax:
- Country: India
- Comment:
- WelcomeGuest to learn TWiki
- Sandbox web to try out TWiki
- AjitKembhaviSandbox just for me
- Horizontal size of text edit box:
- Set EDITBOXWIDTH = 70
- Vertical size of text edit box:
- Set EDITBOXHEIGHT = 17
- Style of text edit box.
width: 99%for full window width (default),width: autoto disable.- Set EDITBOXSTYLE = width: 99%
- Optionally write protect your home page: (set it to your WikiName)
- Set ALLOWTOPICCHANGE =
- TWikiPreferences has site-level preferences of TWiki.
- WebPreferences has preferences of the TWiki.IVOA web.
- TWikiUsers has a list of other TWiki users.
- Name: Alain SARKISSIAN
- Email: alain.sarkissian@aerov.jussieu.fr
- Company Name: Service d'Aeronomie
- Company URL:
- Telephone: 33 1 64 47 43 02
- Fax: 33 1 69 20 29 99
- Country: France
- Comment:
- WelcomeGuest to learn TWiki
- Sandbox web to try out TWiki
- AlainSARKISSIANSandbox just for me
- Horizontal size of text edit box:
- Set EDITBOXWIDTH = 70
- Vertical size of text edit box:
- Set EDITBOXHEIGHT = 17
- Style of text edit box.
width: 99%for full window width (default),width: autoto disable.- Set EDITBOXSTYLE = width: 99%
- Optionally write protect your home page: (set it to your WikiName)
- Set ALLOWTOPICCHANGE =
- TWikiPreferences has site-level preferences of TWiki.
- WebPreferences has preferences of the TWiki.IVOA web.
- TWikiUsers has a list of other TWiki users.
Alasdair Allan eSTAR Project
School of Physics
University of Exeter
Stocker Road
Exeter
EX4 4QL
Tel. +44-1392-264160
Email aa@astro.ex.ac.uk
Web http://www.astro.ex.ac.uk/people/aa/
- Name: Alasdair Gray
- Email: agray@dcs.gla.ac.uk
- Company Name: University of Glasgow, Computing Science
- Company URL: http://www.dcs.gla.ac.uk
- Telephone:
- Fax:
- Country: United Kingdom
- Comment:
- WelcomeGuest to learn TWiki
- Sandbox web to try out TWiki
- AlasdairGraySandbox just for me
- Show tool-tip topic info on mouse-over of WikiWord links, on or off:
- Set LINKTOOLTIPINFO = off
- Horizontal size of text edit box:
- Set EDITBOXWIDTH = 70
- Vertical size of text edit box:
- Set EDITBOXHEIGHT = 22
- Style of text edit box.
width: 99%for full window width (default),width: autoto disable.- Set EDITBOXSTYLE = width: 99%
- Optionally write protect your home page: (set it to your WikiName)
- Set ALLOWTOPICCHANGE =
Alberto Accomazzi
|
- TWikiPreferences has site-level preferences of TWiki.
- WebPreferences has preferences of the TWiki.IVOA web.
- TWikiUsers has a list of other TWiki users.
My Links
- ATasteOfTWiki - view a short introductory presentation on TWiki for beginners
- WelcomeGuest - starting points on TWiki
- TWikiUsersGuide - complete TWiki documentation, Quick Start to Reference
- Sandbox - try out TWiki on your own
- AlbertoKroneMartinsSandbox - just for me
My Personal Preferences
- Show tool-tip topic info on mouse-over of WikiWord links, on or off:
- Set LINKTOOLTIPINFO = off
- Preference for the editor, default is the WYSIWYG editor. The options are raw, wysiwyg:
- Set EDITMETHOD = wysiwyg
- More preferences TWiki has system wide preferences settings defined in TWikiPreferences. You can customize preferences settings to your needs: To overload a system setting, (1) do a "raw view" on TWikiPreferences, (2) copy a
Set VARIABLE = valuebullet, (3) do a "raw edit" of your user profile page, (4) add the bullet to the bullet list above, and (5) customize the value as needed. Make sure the settings render as real bullets (in "raw edit", a bullet requires 3 or 6 spaces before the asterisk).
Related Topics
- ChangePassword for changing your password
- ChangeEmailAddress for changing your email address
- UserList has a list of other TWiki users
- UserDocumentationCategory is a list of TWiki user documentation
- UserToolsCategory lists all TWiki user tools
- Comment:
- Name: Alberto Micol
- Email: amicol at eso.org
- Company Name: European Southern Observatory
- Company URL: http://www.eso.org/
- Telephone: +49 89 32006261
- Fax:
- Country: Spain
- Comment:
html based prefs. probably overwritten by CSS prefs
- Horizontal size of text edit box:
- Set EDITBOXWIDTH = 70
- Vertical size of text edit box:
- Set EDITBOXHEIGHT = 17
- Optionally write protect your home page: (set it to your WikiName)
- Set ALLOWTOPICCHANGE =
- TWikiPreferences has site-level preferences of TWiki.
- WebPreferences has preferences of the TWiki.IVOA web.
- TWikiUsers has a list of other TWiki users.
My Links
- ATasteOfTWiki - view a short introductory presentation on TWiki for beginners
- WelcomeGuest - starting points on TWiki
- TWikiUsersGuide - complete TWiki documentation, Quick Start to Reference
- Sandbox - try out TWiki on your own
- AlessandraZanichelliSandbox - just for me
My Personal Preferences
- Show tool-tip topic info on mouse-over of WikiWord links, on or off:
- Set LINKTOOLTIPINFO = off
- Preference for the editor, default is the WYSIWYG editor. The options are raw, wysiwyg:
- Set EDITMETHOD = wysiwyg
- More preferences TWiki has system wide preferences settings defined in TWikiPreferences. You can customize preferences settings to your needs: To overload a system setting, (1) do a "raw view" on TWikiPreferences, (2) copy a
Set VARIABLE = valuebullet, (3) do a "raw edit" of your user profile page, (4) add the bullet to the bullet list above, and (5) customize the value as needed. Make sure the settings render as real bullets (in "raw edit", a bullet requires 3 or 6 spaces before the asterisk).
Related Topics
- ChangePassword for changing your password
- ChangeEmailAddress for changing your email address
- UserList has a list of other TWiki users
- UserDocumentationCategory is a list of TWiki user documentation
- UserToolsCategory lists all TWiki user tools
- Comment:
Number of topics: 64
Topic revision: r152 - 2024-02-12 - GiuliaIafrate
Log in or Register
IVOA.net
Wiki Home
WebChanges
WebTopicList
WebStatistics Twiki Meta & Help
IVOA
Know
Main
Sandbox
TWiki
TWiki intro
TWiki tutorial
User registration
Notify me Working Groups Interest Groups
www.ivoa.net
Documents
Events
Members
XML Schema
Wiki Home
WebChanges
WebTopicList
WebStatistics Twiki Meta & Help
IVOA
Know
Main
Sandbox
TWiki
TWiki intro
TWiki tutorial
User registration
Notify me Working Groups Interest Groups
- Data Curation
- Education
- Knowledge Discovery
- Operations
- Radio Astronomy
- Solar System
- Theory
- Time Domain
www.ivoa.net
Documents
Events
Members
XML Schema
 |
|
Copyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors. Ideas, requests, problems regarding TWiki? Send feedback