IVOA Roadmap for 2023A

This outlines the roadmap for development activities by the various IVOA working and interest groups between the May 2023 and Nov 2023 Interops.

Applications WG

Standards and Related

• Support VOTable 1.5 (possibly get to RFC)
• Evolution of MOC adding frequency axis to MOC
• Following integration of VO-Standards and comon applications into science platform
• Migrate HiPS and SAMP documents to Github
• Clean up applications WG wiki pages.
  • Remove stale links and make sure all information is current

Python

• Support open development in PyVO
  • Maintenance, new features? Organize meetings with developers, mainteners, users
• Maintain votable parsing in Astropy core

Data Access Layer WG

Primary:
ADQL

• 2.1 RFC - complete and get to REC
• 2.2 - PEG, INTERVAL support, MOC support, Array support, string functions, OFFSET

Datalink 1.1

• RFC open - complete and get to REC

DALI 1.2

• PRec + RFC mid 2023

TAP 1.2

• User managed persistent tables
• Experiment with Open API spec for these changes

Conesearch 1.1

• RESPONSEFORMAT, MAXREC
• Allow services to reject unknown parameters
• Modernize required UCDs (UCD 1.0-> UCD 1+)
• PRec by next interop

SIAP-2.0 -> Data Access Protocol v1

• Proceed with renaming, expansion of supported data product types
• Miscellaneous changes
• Moving to WD status

SODA 1.1

• return extended metadata
• New transformation types
• Moving to WD status

Secondary
LineTAP

• Client and server implementations progressing
• Raise PRec once implementations ready

ObjVisSAP

• Examine renaming to avoid radio confusion
• Restart effort to complete the standard and get to PRec

SSA

• Issue errata for minor issues already noted in github

ADQL User defined functions

• Can functions be reconciled between GAVO and OAJ?

ProvTAP

• Continuing work towards mature WD status

Data Model WG

Primary:

Spectrum 1.2 RFE:

• confirm implementations are sufficient for RFC; possibly drop 'relorder' attribute
• proceed to RFC

Transform 1.0:

• implement python workflow scripts as jupyter notebooks
• proceed to RFC

ObsCore extension for Radio domain

• working implementations
• meeting in September
• goal: RFC ready for next interop

Secondary:

Mango 1.0:

• reducing content/scope; working implementation
• goal: concrete WD for next interop

Dataset Metadata 1.0:

• revise model per discussion in Bologna interop
• define and execute appropriate implementations (in context of TimeSeries, Cube, Mango)
• goal: concrete WD for next interop

Modeling High Energy Datasets:

• Meeting June 28
• evaluate and report on compatiblity of current models (eg: Cube) with High Energy data.
• evaluate and report on possible ObsCore extension, following Radio example.
• identify workflow threads to exercise models in this context

Provenance, one step:

• finalize Note, see if this can become a WD
  • as 'view of Provenance' similar to ObsCore

Other:

VODML Toolkit

• integrate toolkit usage into model repositories
• utilize to generate code package of vo-dml compliant models.

NDCube 1.0:

• work on Dataset, and High Energy should define additional workflow threads which exercise this model in other contexts.
• goal: implement any defined workflow cases

Characterisation 2.0:

• VO-DML compliant version of Characterisation model
• goal: WD by Spring 2024

Field of View:

• formalizing model basis for Aladin FoV format used by STScI and others
• goal: no stated goals this semester

CAOM:

• continue discussions with CADC about how this model fits into the IVOA ecosystem, if at all.

Grid and Web Services WG

Pending upgrades

Pending Update SSO 2.1, including issues at:
https://github.com/ivoa-std/SSO/issues

Also, reformat of GitHub repository.

Cross-science platforms execution project

The GWS WG is going to start a project to identify the main changes and new standards to allow remote execution on different astronomical science platforms.

1. discover computing services, answering the question "what computing services are available and suitable to run my task?"
2. access computing services, answering the question "how can I run my task on the selected computing service?"

https://wiki.ivoa.net/twiki/bin/view/IVOA/ComputingServicesAPI

The preliminary items to be analysed are:

Execution planner

Currently, a note has been prepared at
https://github.com/ivoa/ExecutionPlannerNote

This note should be either converted into a standard or, at least, upgraded making it a more powerful interface. To do that, it is necessary to standardise:
Characterisation of software (in terms not only of software dependencies but, also, dependencies that could affect execution like memory, requirements on computing architecture (e.g. CPUs and GPUs), data location, etc. Software should be exposed in a portable format
Metadata characterisation of the science platform node is also a requirement

Access to this metadata should be also characterised as per the definition of a software discovery service and a science platform resources discovery service. The execution planner could use metadata of both types to decide the best place to execute a certain workflow.

For the software characterisation and discovery, other CI/CD solutions could be expanded and already have the connections between code and containers. For the science platform node service, we need to have a dynamic service to capture the real-time status of the resources.

The execution planner and the science platform discovery service could be integrated into the same service and the software discovery looks to be a different one.

Endorsement of a standard for workflow serialisation

Identify one standard for workflow serialisation (e.g. CWL) and endorse it for the IVOA. This format should have software to parse and decompose a workflow in smaller executable elements in different languages if possible.

Execution interface

Identify an execution interface that could abstract the internal execution interfaces of the different science platforms. Probably something based on IVOA UWS could be used to control execution

Credentials Delegation

Review of related standards for authentication/authorisation combined with the execution interface. This could be a problem for a general approach as some computing centres have their local access policies

For this exercise, SKA Regional Centre Network members are available to collaborate, lead and implement although it would be also needed to have members of other projects that are implementing (or already have) running science platforms to create another independent implementation (candidates Vera Rubin, ESA, CADC, Gaia UK,...)

Registry WG

• Continue developing and maintaining the new pyvo Registry search API (released in 2022 for pyvo 1.4)
• Announce and perform deprecation of the Registry Interface 1.0 old client search functionality on clients and servers
• Standards being worked on:
  • VOResource-1.2 (editor: MD)
    • https://github.com/ivoa-std/VOResource
    • Status: ready to go for PR
  • StandardsRegExt -1.1 (editor: RS)
    • https://github.com/ivoa-std/StandardsRegExt
    • Status: ready to go for PR
  • DocRegExt -1.0 (editor: MD)
    • https://github.com/ivoa-std/DocRegExt
    • Goal: registering educational resources: tutorials, use cases, courses
    • Status: currently WD
• Facilitate discovery of data collections by product type (eg “time series”)
  • Bring ObsCore dataproduct_types into the VODataService Registry records via update of VODataService
  • Explore the possibility to add a mandatory tableset for CatalogService
• Curation and validation of the Registry
  • Coordinate with Ops IG to continue ongoing validation and curation efforts among registry maintainers, resource curators, and validator and standards authors
  • Implement actions from last Registry curation task force meeting and organize follow-up meetings
  • Explore feasibility of a VOResource linter using schematron or/and checks with in Python (xpath & libxml2) to handle constraints (eg author names syntax)
• Create Registry WG + EDU IG collaboration to help/teach data publishers register their services
  • Organize brainstorming meeting with Rubin Obs
• Continue discussing whether/how does cloud information belong in the registry and how to register information about multiple locations for finding data e.g., on-prem or on S3...

Semantics WG

• VOUnits 1.1 is PR since May 2023. Given that the update addresses the difficult question of empty and missing units, that RFC could be more involved.
• The UCD list 1.6 should be available for TCG review right after the Interop. We hope that it will be endorsed at the next TCG meeting, or perhaps the one after that.
• Once that is done, the cycle for the UCD list 1.7 will begin in earnest. This will, in particular, comprise UCDs useful to annotate columns containing metadata on ambient conditions for observations.
• Beyond the Semantics WG's documents, we intend to drive along a new release of VOResource with the main goal of clarifying how the UAT is intended to be used within VOResource. Unless the Registry WG has additional items in this update, we expect this will be another easy update.
• Outside of the standards track, we still work on the product-type vocabulary that needs to be ready for Datalink 1.1. We also work on the facility nomenclature and hope to produce a note discussion the intended practices between the IVOA and wikidata before the next Interop. Linking with semantic ecosystem of other communities, and making our vocabularies more interoperable with the rest of the science community.
• Finally, we will continue to process VEPs in the context of maintaining the vocabulary repository. By this activity's nature, we cannot predict at this point how much effort will be required for that.

Data Curation Preservation IG

• Good practices to provide Data in the VO -
• Provide a DOI reference note - (to improve Data citation)
• Promote Data Origin (see http://ivoa.net/documents/DataOrigin/) to working Group (DALI?) in order to have a basic Provenance in the VO

Education IG

2023

• Improving the newcomer's session with regard of feedback from the Bologna Interop meeting. Suggestion is to have a rather technical Session online a week before the Interop meetings, and a rather organisational/social Session at the in person meetings covering topics like * How to join WGs (turn up to the Session, subscribe to mail list...) * How to join a mail list * Explain how the working groups work together at the example of a specific topic of the running interop. * One question raised was that the established IVOA people seem to be friends, which makes it harder for newcomers to fit in. This might be influenced by Bologna being the first in person meeting after 3.5 years. Yet, maybe it's possible that we have a handful of volunteers who would attend the newcomers session and introduce themselves

• Moving forward with the Edu note in Registry to make tutorial findable in the registry
• Updating the IVOA webpage in regard of VO publishing topics
• Finding a solution for putting votut-latex currently hosted on SVN at GAVO to ivoa github (Not as trivial as it sounds)

2024/2025

• Presenting VO-publishing at AAS meeting (when in person meetings are possible)
• Organizing one week workshop/hackathon on VO publishing with the goal that participants will have a usable VO service running at the end of the workshop (when in person meetings are possible also see RegistryWG)

Knowledge Discovery IG

• ML-proofing existing and future science platforms:
  
  • Are existing astronomy science platforms compatible with ML methods?
  • Investigate whether science platforms can access tabular and non-tabular data through VO interfaces.
  • Building libraries of well-established pre-trained models and integrating them in science platforms.
  • Designing a questionnaire to collect user requirements for science platforms to support ML methods and see how it fits into the plan of science platform from the developers' and relevant service providers' side.
• Artificial Intelligence and Large Language Models in the VO
  • Coordinating an IVOA, inter-group "tiger team" that will scope the current and future potential impact of AI on VO.
  • Collecting use cases and requirements for the integration of commercial and/or ad hoc LLMs models in IVOA-relevant topics.
  • Investigating best practices to integrate LLMs with VO tools for a fusion of domain knowledge and data.

Operations IG

Ongoing activity:

• Monitor VO health
  • Coordinate "weather reports" (service compliance)
• Address non-compliancy:
  • Identify common/persistent errors
  • Support/encourage service operators to improve service compliance
  • Work with WGs on standards-related issues
• Validation:
  • Ensure validators are available (IvoaValidatorsSummary)
  • Encourage use of validators
    • on deployed services
    • during standard development
• Communication
  • Provide forum for discussion of operational issues
  • Host/invite "site reports"

Specific focus:

• Work with DAL on standards issues:
  • ConeSearch: UCD1+
  • DALI: Sexagesimal markup standards
  • SSA: A couple of Errata suggested
• Push for service metadata improvements:
  • VOUnits, UCDs
• Upcoming themes:
  • Operations in the cloud
  • Authentication as an operational necessity

Radio IG

• Pushing ObsCore Radio data extension (interferometry/visibility and single dish) forwards by solving issues and pushing implementations
• Update implementation review note with recent progress and additional implementations (projects such as NRAO, GBT, SKA, IRAM, SpanishVO, etc...)
• Upgrade the Pulsar and FRB Radio Data Discovery and Access note
• Continue work on integrating radio services and software in the VO and specifically in platforms (notebooks, clusters, DataLakes) for finding, downloading and processing radio data through the VO.

Solar System IG

• Continue development on Observation Facility Codes
• Get some prototype planetary reference frames incorporated into the Reference Frames vocabulary
  • This tutorial explains how the Navigation and Ancillary Information Facility (NAIF) software defines (and uses) reference frames and coordinate systems: NAIF Frames and Coordinate Systems Tutorial
  • The list of IAU defined body-fixed frames is included in the NAIF software documentation on frames, as an appendix: direct link to "frames.html" second appendix
• Ensure convergence of the EPN-TAP dataproduct_type with the Data Product Type vocabulary.
• Support PDS/SBN efforts to exercise EPN-TAP and planetary additions to related protocols by implementing IVOA services on small data sets from several different planetary disciplines.

Theory IG

Time Domain IG

Priority in this first semester is to review status of recommendation level and working draft level documents of relevance for time domain, and decide on priorities based on current status of the field of time domain astrophysics. In particular, upcoming surveys (LSST, LIGO) and the increase number of transient alerts across wavelengths (including high energy transients, of relevance in multi-messenger science), as well as the emergence of spectral time-domain surveys, such SDSS-V, set the following priorities:

• VOEvent 2.1 working draft: new version coordinated by DAL, but the TD IG should follow and read the draft to provide feedback. The current, stable version of VOEvent is at https://www.ivoa.net/documents/VOEvent/20110711/

• Evaluate if new version of VOEvent requires updates to the VOEvent Transport Protocol => Protocol for publishing VOEvent => https://www.ivoa.net/documents/VOEventTransport/20170320/index.html

• Pick up discussion on Time Series Cube Data Model (2019), to get status, establish if it leads to a recommendation, etc. Does the datacube only have a time dimension, or also a spectral dimension? Of relevance for upcoming spectral time domain surveys. We need to ensue that it also fits high energy data (i.e. event files) => https://www.ivoa.net/documents/Notes/CubeDM/20190318/index.html

• Status of (S)TMOC => Space and Time coverage => Does this require updates? Is it currently being used? https://www.ivoa.net/documents/MOC/20220727/index.html

• Cone search service that allows for time constraints

• Other, dormant documents: VOEventRegExt => Extension of the VO registry to describe a service providing VOEvents (dormant still 2014 ); STC-S => Space Time coordinate linear definitions (dormant still 2013)

Standard and Processes

<!--
* Set ALLOWTOPICRENAME = TWikiAdminGroup
-->