Difference: PDL1RFC (6 vs. 7)

This document will act as RFC centre for the PDL 1.0 Proposed Recommendation http://www.ivoa.net/documents/PDL/

• A standalone dynamic client. It embeds the automatic generation of the verification layer. This development has shown that a PDL description instance can be used for generating the checking algorithms and for generating a client with a dynamic-intelligent behavior helping the user in interacting with a service. This client could be used for interacting with services exposed using different job systems. The repository is https://code.google.com/p/vo-param/ which hosts also the PDL XSD schema.
• The Taverna Plugin. It is strongly oriented towards general physical and scientific interoperability. It uses the PDL description (including semantics aspects) for validating the ‘chaining’ of jobs composing a workflow. The repository for this implementation is at https://github.com/wf4ever/astrotaverna.

• A PDL server exposing existing code as a web service. It embeds the verification layer. This development was done to show that a PDL description instance can be used to generate the ad hoc server, exposing the described service. Its repository is: https://github.com/cmzwolf.
• PDL editor, a tool helping people to write PDL descriptions. Repository at: https://code.google.com/p/pdl-editor/

IVOA Community Comments Period: 24 December 2013 - 21 January 2014

Comments from the IVOA Community during the RFC period:

PDL as it stands is a serialization of an underlying data model for service parameters (where one could add several types that have been identified as necessary in DAL discussions, e.g., intervals, but that's another matter). I suspect it would be fairly straightforward to formulate the DM itself in VO-DML. The disadvantage would be that the schema generated by VO-DML itself would be different from the hand-written one, so implementations would need an update. On the other hand, the DM is specified explicitely rather than being somewhat implied by the combination of XML Schema and prose, it's much clearer what to do on extension, and you get a VOTable serialization for free, which is really nice when you want to embed PDL in, say, responses of DAL services. -- MarkusDemleitner - 2014-01-14

Answer : Yes PDL can be seen as a serialization of an underlying data model. By the way intervals are actually handled in PDL in a very powerful way (using the ValueInRange object, see paragraph 9.6.6 of the Proposed Recommendation document): one could express not only that a parameter is into a ‘simple’ numerical range, but defining complex intervals, something where INF and SUP limits are defined as functions and/or expressions involving numerical constants and/or the other parameters. From a practical point of view PDL could be expressed with VO-DML but this would imply to totally rewrite the PR document and to change all the existing implementations (and redeploy the service, with related clients, based on the actual implementations). In any case VO-DML would have an impact on other standards and PDL would be just one case. Several implementations are based on the PDL grammar (including some production services). These implementations took 2/3 years of work and in some cases are really complex (e.g. the automatic generator of checking algorithm and the generator of the dynamic interface are more or less compilers of PDL, written in Java). The cost for adapting these implementations would be prohibitive today. -- AndreSchaaff and Carlo Maria Zwolf

TCG Review Period

Comments from the TCG members during the TCG Review period: 5 February to 5 March 2014

Application Working Group ( _Mark Taylor, Pierre Fernique)

Some of my concerns from the earlier RFC period have been addressed.

Following the comments from Franck et al., I am prepared to believe that PDL is a good fit to the kind of constraints that are required for theoretical services for which this standard is targetted.

I am however a bit sceptical of sentences like:

• "PDL could be plugged as an additional layer to every existing IVOA service and is suitable for solving issues not covered by other IVOA standards and is particularly indicated for workflows." (sec 3.4.1)

• "The key point to retain is that PDL is simple for simple services is flexible and powerful enough for meeting description requirements coming with the most complex scientific codes (of course the associated description won't be simple)." (sec 13.1)

As long as the scope of the current version of this standard is mainly recognised as applying to theoretical type services, Applications WG recommends approval.

A few new errors have been introduced since the previous version:

• Sec 5: Still references rational and complex types, though these have been removed.

• Sec 10.6.2: isRational condition: I don't think this could ever evaluate to something non-true?

• Sec 13: examples contain validation errors, for instance SkossConcept (the spelling of this element has been changed to SkosConcept). Also <expression> open tag is closed by </Expression> close tag. This kind of thing is not helped by the fact that there does not appear to be(?) a consistent rule about whether element names start with lower or upper case letters.

Data Access Layer Working Group ( Patrick Dowler, Francois Bonnarel)

Data Model Working Group ( _Jesus Salgado, Omar Laurino)

Grid & Web Services Working Group (Andre Schaaff, Andreas Wicenec_)

Registry Working Group ( _Gretchen Greene, Pierre Le Sidaner)

Semantics Working Group ( _Norman Gray, Mireille Louys)

Data Curation & Preservation Interest Group (Alberto Accomazzi, Francoise Genova)

Education Interest Group ( _Massimo Ramella, Sudhanshu Barway )

Knowledge Discovery in Databases Interest Group ( George Djorgovski )

Theory Interest Group ( _Franck Le Petit, Rick Wagner)

As I already said, to my point of view, this standard fills a missing piece in the IVOA standards for interoperable services in astrophysics.

When I started to work on Theory at the IVOA (end of 2005), I asked to scientists what they would expect from VO-Theory. For a lot of scientists, that was not to have access to simulations but to be able to give online access to their codes. The interest to do it in a VO context was the hope to be able to pipe scientific codes together. As, for example, to post-process outputs of large MHD simulations (published in a database - so relying on what is today SimDM / SimDAL) with online codes to compute, for example line intensities that they could compare to observations.

At this time, the CEA by Astrogrid partly answered the need. It allowed to easily put online a scientific code, but had not detail descriptions for interoperability between applications in a scientific meaning (units, type of data, semantics tags, ...). The CEA has never been standardized, maybe because the funding difficulties of Astrogrid.

Today, PDL answers the need. With it, it is now possible to describe a scientific application in a standardized way.

It may seems complicate to use. But, in fact, for simple applications, the description is also simple. The standard is just comprehensive enough to allow description of complex applications. I also noticed that in France, several data centers plan to use PDL to describe their applications and give access to them in a VO context. In Spain, people working in European projects about workflows, not directly related to VO, find it useful and integrated it in Taverna for example.

I think that most comments done during the Hawaii InterOp have been taken into account in the revised version of the document.

For me, we can approve it.

-- FranckLePetit - 2014-03-06

Time Domain Interest Group ( _John Swinbank, Mike Fitzpatrick)

Archive : Previous Community period for comments

Please, don't modify the below part of this page.

N.B. : comments from the Community review and remarks from the Hawaii interop GWS session have been integrated in the last version

Comments from the IVOA Community during the RFC period:

The word "unique" is used a lot in the document, but the scope is not clear (and I think is not the same in all cases). Sometimes it seems to refer to the entirety of the VO (e.g., ServiceID) and at other times (mostly, I think) it refers to within a particular PDL instance. Yes?

the next update of the document will be clearer for theses differents points -- AndreSchaaff - 2013-09-22

I think we need to get some more people to read through carefully, especially some XML experts.

a talk and a discussion is planned during the Hawaii interop sessions -- AndreSchaaff - 2013-09-22

PierreLeSidenar GretchenGreene (9/11/2013) There is two line in the text with confusion:

It beginning of document it explains that PDL is made for describing parameters, physical quantities (via UCD), concept (via SKOS) and
constrain (via PDL grammar). It specifies already exist Jobs Description Language (WSDL or WADL) that describe how service work and
parameters basically. However, PDL describes completely the parameters.
then on page 9:
PDL is a VO standard for describing how to access data or driving online codes, exposed through the VO. PDL is not such a standard, UWS is the standard for REST service and as a service it can be described using Jobs Description Language. It is quite important that PDL is presented only as a grammar to describe parameter, not as all UWS

and page 34....the PDL server, embedding the veri...

It seems quite strange to have a PDL server, PDL can be used on server and client side, but we should have a PDL server as PDL is a grammar

A small point, but we should not confuse a grammar with a job system

the use of UWS will be refurbished in the next update of the document -- AndreSchaaff - 2013-09-22

Franck Le Petit (19/10/2013): As part of a (minor) author of this standard, I cannot say much.

To my point of view, this standard fills a missing piece in the IVOA standards for interoperable services in astrophysics. Up to now, we have nothing to get a "real" interoperability (in a computer - computer meaning) between services. Indeed, PDL goes further than previous standards because it allows to describe properly input and ouputs of services associating to each quantities semantic meaning, units and, if required, relationship between parameters. So it has everything to put a bit of "intelligence" in the communication between services.

This should be a useful standard. For Theory I.G., I see immediatly some several use cases. Any online application (online simulation code for example) can be described in details thanks to PDL. One of the difficulty often mentionned by the community to provide online codes is that, those ones are often complex to use and users may do mistake with online simulation codes. They could use them outside of their validity domain. The description of parameters with PDL allows to constrain those ones in validity domains, and so PDL answers this fear of the theorist community.

Describing precisly inputs and outputs of services with relationships between parameters, should also be a major step forward for workflows. Workflows engines have been a subject of dicussions in the community for a long time but very few of them are really used. Proper description of services with PDL should help to have more efficient interoperable services in workflows engines.

TCG Review Period was postponed after the Hawaii interop and the Community comments integration

Comments from TCG members during the TCG Review period: 15 November 2013 - 13 December 2013

General Comments

It is not obvious to me that PDL in its current form solves a pressing need.

A standard providing some machine-readable characterisation of the inputs and outputs of services does seem like a good idea.

However, the motivation of attempting to encode detailed constraints on the presented values of groups of variables is less obvious to me. It seems unrealistic to expect a generic standard to do a good job of performing a complete pre-submission test of whether a set of query parameters provides valid inputs to a given service - realistically only the service itself is going to be able to do that.

Section 13.1 says "[PDL is] ... flexible and powerful enough for meeting description requirements coming with the most complex scientific codes." but it's not clear to me that this is the case. The constraints part of PDL, which does indeed have considerable complexity, has the feel to me of something put together to handle a small number of specific use cases (like eq. 16 and 17) but that will not be able to handle other ones. For instance, as far as I can see it is not able to answer the following, I would have thought quite common, requirements:

• allow an input parameter to be a vector of unspecified length --- this could be performed quite easily by introducing an Any Expression object, standing for any Integer number.
• constrain two input parameters to be be vectors of the same length, without specifying what that length is --- it exists by defining that the size of the scalar product between two vectors is equal to one. Since the scalar product could be computed only between vectors of the same size, this conditions fit the need.
• constrain two input parameters to have the same units, without specifying what those units are ---The answer is not straightforward. One could introduce a specific condition for this need, but this is not in the PDL logic: The good approach would consist in implementing a Unit algebra (thus this should be in the implementation, not in the grammar). Naturally, an expression must be homogeneous: let v1 and v2 be to speeds. The expression v1+v2 has sense only if v1 and v2 has same units. With a Unit Algebra, the condition ‘two parameter have to be same unit’ would be a part implied by ‘the expression is homogeneous’. We could also verify that two Expression has same unit. Again this is on the implementation side and not in the grammar part. (Working with VO-Unit group for this part should be of great interest).
• write a constraint using a scalar function of multiple arguments ---It can be done by extending the schema with new functions, of kind f(Expression1, Expression2, …, ExpressionN).
• constrain coordinate values in a non-trivial way (e.g. maximum separation between two sky points or two dates, inclusion in a given sky coverage region) --- with a decomposition of the coordinates with maximum details, you could express (e.g. asc1, dec1 and asc2, dec2) any condition between these four parameters.
• cope with 2- or 3-dimensional input/output parameters (e.g. images) --- the adopted strategy is that this two/three dimensional structures or images are contained into files. It is possible to use a String for the path (or the URI) of the file and to use the Skos concept for specifying that the parameter is the URI of a specific file (in case of images the Skos could specify if it is pointing to a jpg or a fits image).

--- the need for numerical constraints comes from theoretical services, where often users need help for filling the set of parameters

The above comments constitute my personal misgivings about the usefulness of this standard. However, the current draft seems to have support from some other IVOA members who are better placed than me to assess its usefulness (I don't have experience with workflows). So if the consensus within the GWS WG or other IVOA members is that this does provide a useful part of the standards jigsaw, Applications WG will not block its acceptance.

However, I do have some more detailed comments which I would like the authors to address.

Detailed comments

• Sec 5: One of the children of the SingleParameter type is named SkossConcept . Is this spelt correctly? I was expecting the spelling SkosConcept .

• Sec 5: The Dimension child (meaning vector length) of SingleParameter is required, so as far as I can see, every scalar parameter definition has to contain the boilerplate:

      <Dimension xsi:type="AtomicConstantExpression" ConstantType="integer">
         <Constant>1</Constant>
      </Dimension>

which seems unnecessarily verbose. It would simplify PDL documents if this element was optional, and dimension assumed equal to 1 when absent. (Or perhaps, that it was not part of a SingleParameter but was added in a subclass). ---in a primitive version of PDL this was not so ‘boilerplate’. Then it appeared that the dimension of a vector parameter must be an expression. Consider for example a service computing polynomial interpolations. A first parameter could be the degree of the polynomial and the second parameter a vector containing the set of points. For this simple case one has to express the constraint that the degree of the polynomial must be equal to the size of the vector. This kind of argument explains why the dimension is an expression. Of course in a simple case this could be complex, just for saying that you have a scalar parameter.

• Sec 7: A list of permitted types for parameters and expressions is presented: boolean, string, rational, complex, integer, real, date . However, no details are given for how values of these types are to be serialized. --- this is out of the PDL scope, in SOAP services, this will serialized in the SOAP enveloppe, in REST-UWS services, it will be in POST Perhaps this is out of scope for the parameters described/constrained by a PDL document, since their values will be exchanged by some mechanism specific to the workflow engine (or whatever) within which they run. However, expression values do appear within the PDL document so serialization should be well-defined at least for that application. In turn:
  • integer and real : representations are fairly standard, though I'd argue it would be good practice to be explicit about the required format (perhaps defer to the XSD rules). --- it will be defined in the text as a default format (ex for real decimal separator is)
  • boolean : in the schema, though not the text, boolean is annotated as "A representation of a boolean - e.g. true/false on/off" which is not really specific enough (case senstitivity? are other representations permitted?). --- it will be defined as permitted values in the document (TRUE, FALSE, non case sensitive)
  • date : representation could be ISO-8601? if so that should be spelled out. Can dates be combined using the arithmetic operations described elsewhere in the document? --- it will be defined as a requested representation
  • complex , rational: there is really no clue how you'd write/parse expressions of these types. One place where a rational constant would make sense in the examples in the document (line 9 of the example on page 42) has <Constant>0.16666666</Constant> in any case - is the need for a rational well-motivated? (and if it is - how do you go about evaluating the isRational condition defined in 10.6.2?) --- the introduction of this kind of parameters comebacks to the genesis of PDL. We were looking for completeness. Then we realized that rational has few utility (real can replace it almost everywhere) and dealing with the algebra of complex was mathematically very heavy (moreover one should introduce somewhere an imaginary unit in the schema for complex operation). We abandoned it and we didn’t clean up the schema from this old remanent.

• Sec 9.1, 9.2: The AtomicParameter and AtomicConstant expressions seem, well, not very atomic. These are the simplest items described which do what they do, but both have also optional power (raise to a power) and operation (combine with other expression) children. My expectation would be to have a simple value-only instance, and more complex operations to apply on top of them where the additional complexity is required. But I admit this is just a matter of taste.

• Sec 9.1, 9.2: The symbol d_{oo} is defined in the text, but not used. --- it will be removed

• Sec 10: This section, I think, is defining a language for making assertions about the values of presented parameters. However, I have to admit to being confused by the details. This is partly down to the use of terms like "must", "valid", and "TRUE" without really explaining what's meant, for instance "The Criterion contained within a Always object must always be valid" . Does valid mean it must evaluate to true? What happens if it's not valid? --- that means 'it is good’ only when the evaluation of the criterion contained into an 'Always object' evaluates to 'TRUE'. What if it is not good, well it is wrong. « wrong » evaluation is catched for notifying errors to users.

• Sec 10: I don't understand under what circumstances one would use IfThenConditionalStatement , WhenConditionalStatement and And - as far as I can see they all assert the conjunction of two subordinate assertions, i.e. they all basically mean "and". Also, I can't see how ConstraintOnGroup and the Active label are supposed to work or what they do. These are probably my misunderstandings, but a more careful introduction to this section, or perhaps some examples, might make it easier to make sense of. --- the When statement was introduced pretty much for the sole purpose of dealing with the case of activating a ParameterGroup. 'When' and 'IfThen' are not completely equivalent. 'When' is a shorthand for and 'IfThen' clause where the Then part can only evaluate to true or "valid", so that the ParameterGroup can be either active or not. 'When' was introduced to have a restricted form of 'ConditionalStatement' that could have no "side effects" in the 'Then' part, and at the time seemed to the the most natural extension to the way that the reference implementation worked with the PDL.-

• PDL Schema (sec 13.4): The ParameterDependency type definition seems to be the victim of a cut'n'paste error - as well as the enumeration values "required" and "optional" it contains all the enumeration values from ParameterType as well ("rational", "complex", "integer", ...). --- will be fixed in the schema %/

• PDL Schema (sec 13.4): The ParameterType definition, as well as containing the types mentioned in sec 7 ( boolean, string, rational, complex, integer, real, date), also contains the types binary, Table, Image and Spectrum. While types along these lines would be very useful, there is no hint in the body of the document that they exist or how to handle them. --- will be cleared in the schema, it was declared in the initial version of PDL (all the file cases were declared but it appeared that for this usage (binary, Table, images, Spectrum) one can use a file, put the file Path or URI into a String parameter and specify in the Skos of this same parameter what is the nature of the file.

• PDL Schema (sec 13.4): There seem to be some items ( MinMaxArgument, cardinality, others?) in the schema which are not discussed in the text. --- in previous versions, all the parameter were only scalar, without vector aspect in the first versions: the min and max were introduced for defining the max (reap. the min) over a set of values. But with the vector aspect this lose sense. Must be cleared

• Implementation status: All credit to the authors for providing four implementations of different kinds on the RFC page. I haven't tried them all out, but I started up the PDL editor and tried to create a ConstantExpression with type COMPLEX and got the message "Type Error: the type COMPLEX does not correspond to any supported type" . The DocStd document requires "two interoperable implementations of each feature" . Can the authors confirm that all the features in the document are covered by two or more of the listed implementations? --- will be done.

Grid & Web Services Working Group (Andre Schaaff, Andreas Wicenec_)

Data Curation & Preservation Interest Group (Alberto Accomazzi, Francoise Genova)

PDL should be useful in short term to provide a frame to publish online applications. One of the benefit to have such a frame is that applications will be able to communicate (computer - computer communication). Precise description of parameters of an application taking into account relationships between parameters is something that does not exist up to now, and that is very useful to scientists wishing to publish their applications.

PDL description may appear complex but that is just because it is quite complete. For simple applications, their XML description is also simple. So the complexity of the standard should not be a blocking point. Maybe an implementation note with a few examples could help also.

Concerning Theory, we have an interest in this standard to allow scientists to give access to their numerical models. A few years ago, Astrogrid developed the CEA (Common Execution Architecture) that fullfiled this need. Unfortunatly, CEA should have been standardized but that has never been done, and, all these efforts have been lost. PDL goes further than the CEA.

So concerning Theory I.G. I would say we approve PDL as an IVOA standard (taking into account all Mark's remarks who read the document very carefully

-- FranckLePetit - 2013-12-15