GWSTelecon20230920 < IVOA

---+ Minutes: GWS Telecon 2023-09-20

Time: 20-09-2023; 13:00 UTC

VideoConf link: [[https://skatelescope.zoom.us/j/4371396863?pwd=bVl4ZThNeko4S3RSQnJxd0plVUkxZz09]]

Online notes: https://yopad.eu/p/IVOA_GWS_Science_Platforms_Federated_Execution
---++++ Background information

Computing Services: [[ComputingServicesAPI][https://wiki.ivoa.net/twiki/bin/view/IVOA/ComputingServicesAPI]]

Execution Planner: https://github.com/Zarquan/ExecutionPlanner
---++++ Main Topic. Science Platforms Federated Execution: Agenda
---++++ Attendees: 22

<div id="magicdomid6">
   * Jesus Salgado - JS
</div> <div id="magicdomid7">
   * Sara Bertocco - SB
</div> <div id="magicdomid8">
   * Dave Morris - DM
</div> <div id="magicdomid9">
   * Brian Major
</div> <div id="magicdomid10">
   * Claudio Gheller
</div> <div id="magicdomid11">
   * David Aikema
</div> <div id="magicdomid12">
   * Gerard Lemson
</div> <div id="magicdomid13">
   * Rob Simmonds
</div> <div id="magicdomid14">
   * Chenzhou Cui
</div> <div id="magicdomid15">
   * Paul Harrison
</div> <div id="magicdomid16">
   * Susana Sanchez
</div> <div id="magicdomid17">
   * Vicente Navarro
</div> <div id="magicdomid18">
   * Zhijun Xu
</div> <div id="magicdomid19">
   * Tess Jaffe
</div> <div id="magicdomid21">
   * Severin Gaudet
</div> <div id="magicdomid22">
   * Renaud Savalle
</div> <div id="magicdomid23">
   * David Aikema
</div> <div id="magicdomid24">
   * James Collison
</div> <div id="magicdomid25">
   * Sara Nieto
</div> <div id="magicdomid26">
   * Rajesh Tamhane
</div> <div id="magicdomid27">
   * Andreas Wicenec
</div> <div id="magicdomid28">
   * Rohini Joshi
<div id="magicdomid27"> </div> <div id="magicdomid28">
   * Iulia Cimpan
</div>
</div>
---++++ Agenda

   * Introduction and presentation of concept - Jesus Salgado 15’
   * Execution Planner status and missing points - Dave Morris 15’
   * Open Discussion - All 15’
   * Identification of collaborators and Future plans 15’
---++++ Introduction and presentation of concept - Jesus Salgado

<a href="https://docs.google.com/presentation/d/1zow9L1hAAn3oLo3BRb_EDHmXQkLQ8zxO5DWXDQ_d9JA/" title="GWS - Federated Computing">https://docs.google.com/presentation/d/1zow9L1hAAn3oLo3BRb_EDHmXQkLQ8zxO5DWXDQ_d9JA/</a>

<div id="magicdomid38">Connect sci plat on different centres.</div> <div id="magicdomid39">IVOA focus</div> <div id="magicdomid40">1 Discover data from different missions with different interfaces.</div> <div id="magicdomid41">2 connect services</div> <div id="magicdomid42">3 allow users to connect to services with interfaces</div> <div id="magicdomid43">We need to execute code as close as possible to the data.</div> <div id="magicdomid44">SRC net already provides some capabilities.</div> <div id="magicdomid45">The idea is to connect different platforms with different missions.</div> <div id="magicdomid46">A general picture of sci-plat</div> <div id="magicdomid47">Federated authentication, federated data lake,</div> <div id="magicdomid48">how to execute remotely (today's focus)</div> <div id="magicdomid49">standardize service to access data (not the main thing for today)</div> <div id="magicdomid50">How to use the software repository</div> <div id="magicdomid51">Some possible data mesh services, a sort of SODA approach. Define a set of operations on the data and expand soda to this set of operations.</div> <div id="magicdomid52">ExecutionPlanner description: run software on different platforms. Are you able to run this?</div> <div id="magicdomid53">A more complex approach uses an information system. The IVOA approach is more pragmatic.</div> <div id="magicdomid54">ExecPlanner: standardise the workflow description, and extend the UWS.</div> <div id="magicdomid55">Summary: federate auth protocol, maybe improve data access, simplify federated execution, and characterize workflow and hardware resources.</div> <div id="magicdomid56">Today's objectives: agree roadmap and collect interest and collaborators.</div> <div id="magicdomid57">Q(Simmonds): are there teams doing similar things?</div> <div id="magicdomid58">A(Jesus): It is some split what can be standard in IVOA and what can be standardized locally</div> <div id="magicdomid59">Q(Simmonds):: reimplement the grid?</div> <div id="magicdomid60">A(Jesus): we need a common solution in the current more heterogeneous environment (cloud, grid, ...)</div>

---++++ Execution Planner status and missing points - Dave Morris

<div id="magicdomid64">Dave will describe the common web interface to have in common over whatever implementation. A federated system of different technologies.</div> <div id="magicdomid65">The document is based on a couple of years of work (presentations and collecting feedback) and is still a work in progress.</div> <div id="magicdomid66">Scope: Collect a group working on a prototype and developing a common understanding.</div> <div id="magicdomid67">Concept: "executable thing" can be a Python program (requires Python env) or a docker container (requires a container execution environment). The executable thing is described (see the document). Discoverable URL.</div> <div id="magicdomid68">Service interaction. For data in IVOA, we have services to discover cutouts of data. Execution Planner does not care about service discovery.</div> <div id="magicdomid69">Services interaction: Dave describes the exchange between client and service. The client makes a request with the "executable thing description", and the service answers with an offer.</div> <div id="magicdomid70">...</div> <div id="magicdomid71">Sara: Is the user or the execPlanner that calculates how long the job is?</div> <div id="magicdomid72">Dave: The user.</div>
---++++ Open Discussion - All

Gheller: need to think about the data model description

Dave: Dave is thinking about it. It is a type, which is a URI and then the spec, the specification with type-specific details. The structure comes from Kubernetes. Will try to have a vocabulary as simple as possible.
<div id="magicdomid78">Jesus: should it be inside the execution planner standard or separate?</div> <div id="magicdomid79">Dave: by now a simple data model is proposed in ExecPlanner. More complex definitions need to go out of the ExecPlanner (see the GPU example in the document).</div> <div id="magicdomid80">Simmonds:</div> <div id="magicdomid81">Dave:</div> <div id="magicdomid82">Jesus: The data location is missing. Do you need a standard to do that?</div> <div id="magicdomid83">Simmonds: a lot of stuff has been done before (for data reconstruction).</div> <div id="magicdomid84">It seems to introduce complexity</div> <div id="magicdomid85">Dave: it seems because there are a lot of details. The implementation is really simple, it depends on the level of detail.</div> <div id="magicdomid86">Rohini:</div> <div id="magicdomid87">Dave: we have the distinction between batch and interactive. Everything is asynchronous. also when data has to be staged, the time needed is kept into account.</div> <div id="magicdomid88">Vincente Navarro: Developing a distributed pipeline implementation is difficult also in the same organization if you add multiple nodes. Provocative: Have you considered the possibility of using what already exists and putting it on top of an orchestration layer? We use CWL, airflow is becoming a standard. Why do not use an approach using these existing technologies?</div> <div id="magicdomid89">Dave: in a large federated environment someone has new tech, while others have pretty old systems, i.e. docker run, docker-compose, Kubernetes cluster... our interface does not care about details, only answer yes, I can run your container.</div> <div id="magicdomid90">Jesus: could we have a software stack to simplify this? The answer is no. Asked multiple partners.</div> <div id="magicdomid91">Vincente: S3 is the technology, not the standard. Is an industrial de facto standard and can help us to data access speeding up the process.</div> <div id="magicdomid92">Dave: totally agree that standardize....</div> <div id="magicdomid93">Vincente: there is plenty of work if you need, for example, to replicate the data volume.</div> <div id="magicdomid94">Severin: what do we need more on the specification to start with the prototyping?</div> <div id="magicdomid95">Dave: now we can start. what we publish now is an initial draft of the standard. The implementation of a really simple prototype could provide precious feedback to improve the standard.</div> <div id="magicdomid96">Jesus: will send an e-mail to understand if we are able to endorse the standard and find people to collaborate in prototyping. At IVOA Interop we will have a session to collaborate</div>
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
pdf	GWS_-_Federated_Computing.pdf	r1	manage	2551.0 K	2023-09-20 - 10:35	JesusSalgado
This topic: IVOA > WebHome > IvoaGridAndWebServices > GWSTelecon20230920
Topic revision: r6 - 2023-09-25 - JesusSalgado