{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Time series in the VO -- how to be not seen #\n",
    "\n",
    "## The use case ## \n",
    "\n",
    "Around March 2023 a collegue and VO expert approach me with an actually simple VO use case: how to find TESS lightcurves in the VO with VO tools. What at the first glance seemed like an easy and simple task, very soon turned into something very different. \n",
    "Here I will present the steps we did to solve this question and hope to demonstrate that this certainly is not a good way to for non VO experts to follow.\n",
    "\n",
    "## The first approach ##\n",
    "The first approach to find data in the VO of course is to send a query to the VO registry. So for TESS lightcurves the first step would be to search for TESS data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Table length=341>\n",
      "                  ivoid                    ...\n",
      "                                           ...\n",
      "                  object                   ...\n",
      "------------------------------------------ ...\n",
      "           ivo://archive.stsci.edu/caomtap ...\n",
      "ivo://archive.stsci.edu/catalogs/tessinput ...\n",
      "            ivo://archive.stsci.edu/cs/tic ...\n",
      "                   ivo://cadc.nrc.ca/argus ...\n",
      "                   ivo://cds.vizier/ii/366 ...\n",
      "                    ivo://cds.vizier/iv/38 ...\n",
      "                    ivo://cds.vizier/iv/39 ...\n",
      "            ivo://cds.vizier/j/a+a/569/a21 ...\n",
      "            ivo://cds.vizier/j/a+a/600/a30 ...\n",
      "                                       ... ...\n",
      "         ivo://cds.vizier/j/mnras/513/6042 ...\n",
      "         ivo://cds.vizier/j/mnras/515/3212 ...\n",
      "         ivo://cds.vizier/j/mnras/519/4699 ...\n",
      "      ivo://cds.vizier/j/other/coska/51.78 ...\n",
      "  ivo://cds.vizier/j/other/newa/101.102022 ...\n",
      "     ivo://cds.vizier/j/other/raa/23.c5016 ...\n",
      "         ivo://cds.vizier/j/pasp/131/c4401 ...\n",
      "                   ivo://cds.vizier/vi/154 ...\n",
      "                      ivo://cds/p/tess/2yr ...\n",
      "           ivo://nasa.heasarc/skyview/tess ...\n"
     ]
    }
   ],
   "source": [
    "import pyvo\n",
    "\n",
    "results=pyvo.registry.search(keywords='TESS')\n",
    "\n",
    "print (results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, 341 results are a bit overwhelming which on the one hand is good news, because it seems there is a lot of data available, and we have to narrow down our search. So an intuitive way would be to add something to the keywords -- though of course we (as: VO guys) are very well aware that this is not the proper way, it's the *obvious* way for VO users. So we went down the path:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Table length=6>\n",
      "             ivoid              ...\n",
      "                                ...\n",
      "             object             ...\n",
      "------------------------------- ...\n",
      " ivo://cds.vizier/j/a+a/635/a13 ...\n",
      " ivo://cds.vizier/j/a+a/655/a66 ...\n",
      "ivo://cds.vizier/j/a+a/664/a162 ...\n",
      "  ivo://cds.vizier/j/aj/162/127 ...\n",
      " ivo://cds.vizier/j/apjs/245/29 ...\n",
      " ivo://cds.vizier/j/apjs/259/33 ...\n"
     ]
    }
   ],
   "source": [
    "results=pyvo.registry.search(keywords='TESS lightcurves')\n",
    "print (results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unfortunately none of the results in Vizier actually provide the TESS lightcurves, so our query was too narrow, but the next best term 'time series' returns a similar result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ivo://cds.vizier/j/a+a/619/l10\n",
      "ivo://cds.vizier/j/a+a/625/l13\n",
      "ivo://cds.vizier/j/a+a/630/a106\n",
      "ivo://cds.vizier/j/a+a/638/a108\n",
      "ivo://cds.vizier/j/a+a/639/a35\n",
      "ivo://cds.vizier/j/a+a/643/a110\n",
      "ivo://cds.vizier/j/a+a/660/a86\n",
      "ivo://cds.vizier/j/a+a/664/a162\n",
      "ivo://cds.vizier/j/a+a/666/a155\n",
      "ivo://cds.vizier/j/a+a/667/a63\n",
      "ivo://cds.vizier/j/a+a/668/a158\n",
      "ivo://cds.vizier/j/a+a/671/l8\n",
      "ivo://cds.vizier/j/aca/70/1\n",
      "ivo://cds.vizier/j/aca/72/31\n",
      "ivo://cds.vizier/j/aj/158/243\n",
      "ivo://cds.vizier/j/aj/161/170\n",
      "ivo://cds.vizier/j/aj/161/171\n",
      "ivo://cds.vizier/j/aj/161/194\n",
      "ivo://cds.vizier/j/aj/161/46\n",
      "ivo://cds.vizier/j/aj/162/150\n",
      "ivo://cds.vizier/j/aj/163/19\n",
      "ivo://cds.vizier/j/aj/164/216\n",
      "ivo://cds.vizier/j/aj/164/70\n",
      "ivo://cds.vizier/j/apjs/245/13\n",
      "ivo://cds.vizier/j/apjs/259/50\n",
      "ivo://cds.vizier/j/mnras/473/2004\n",
      "ivo://cds.vizier/j/mnras/478/2676\n",
      "ivo://cds.vizier/j/mnras/496/5423\n"
     ]
    }
   ],
   "source": [
    "results=pyvo.registry.search(keywords='TESS time series')\n",
    "\n",
    "for i in results:\n",
    "    print (i['ivoid'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As time series are observational data we easily found the right way to figure it out by using obscore:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Table length=2>\n",
      "             ivoid                   res_type     ...  intf_types  intf_roles\n",
      "                                                  ...                        \n",
      "             object                   object      ...    object      object  \n",
      "------------------------------- ----------------- ... ------------ ----------\n",
      "ivo://archive.stsci.edu/caomtap vs:catalogservice ... vs:paramhttp        std\n",
      "        ivo://cadc.nrc.ca/argus vs:catalogservice ... vs:paramhttp        std\n"
     ]
    }
   ],
   "source": [
    "results=pyvo.registry.search(keywords='TESS', datamodel='obscore')\n",
    "print (results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "STScI CAOMTAP \n",
      " <Table length=12>\n",
      "dataproduct_type calib_level obs_collection ... facility_name instrument_name\n",
      "                                            ...                              \n",
      "     object         int32        object     ...     object         object    \n",
      "---------------- ----------- -------------- ... ------------- ---------------\n",
      "      timeseries           3           TESS ...     NASA/Ames      Photometer\n",
      "      timeseries           3           TESS ...     NASA/Ames      Photometer\n",
      "      timeseries           3           TESS ...     NASA/Ames      Photometer\n",
      "      timeseries           3           TESS ...     NASA/Ames      Photometer\n",
      "      timeseries           3           TESS ...     NASA/Ames      Photometer\n",
      "      timeseries           3           TESS ...     NASA/Ames      Photometer\n",
      "      timeseries           3           TESS ...     NASA/Ames      Photometer\n",
      "      timeseries           3           TESS ...     NASA/Ames      Photometer\n",
      "      timeseries           3           TESS ...     NASA/Ames      Photometer\n",
      "      timeseries           3           TESS ...     NASA/Ames      Photometer\n",
      "      timeseries           3           TESS ...     NASA/Ames      Photometer\n",
      "      timeseries           3           TESS ...     NASA/Ames      Photometer \n",
      "\n",
      "CADC TAP \n",
      " <Table length=2>\n",
      "calib_level     s_ra    ...       lastModified     \n",
      "                deg     ...                        \n",
      "   int32      float64   ...          str23         \n",
      "----------- ----------- ... -----------------------\n",
      "          2 73.38130182 ... 2021-07-06T18:05:41.157\n",
      "          3 73.38130182 ... 2021-07-06T18:05:41.157 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "for svc in results:\n",
    "    obsc_service = pyvo.dal.TAPService(svc.access_url)\n",
    "    obsc_result = obsc_service.search(\n",
    "        \"\"\"SELECT * FROM ivoa.Obscore\n",
    "        WHERE dataproduct_type='timeseries'\n",
    "        AND CONTAINS(POINT('ICRS', 073.3813019796800, -06.6957980314600), s_region)=1\n",
    "        AND obs_collection='TESS'\"\"\")\n",
    "#    obsc_result.broadcast_samp('topcat')\n",
    "    print (svc.short_name,\"\\n\", obsc_result, \"\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sending the whole thing to topcat we easily see that the results from CADC even contain datalinks to the data products -- which lead to the time series on the services of STSCI. \n",
    "This might explain though, why using google to search for \"TESS light curves\" will lead to the TESS documention of STSCI, but in now way ever points to CADC. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## How stupid am I ?##\n",
    "Apparently I fail at finding TESS light curves with VO tools only, unless relyin on obscore -- which is a cool standard, but for really advanced VO users. Yet, it puzzled me and I really felt like I was stupid and missing something (though in good company of Ada, and the thought, that my stupidity was on the same level as Adas was very soothing.) Finding TESS light curves and accessing them was so hard compared to the ones of Gaia. \n",
    "\n",
    "Is it, thought ? Actually, I never searched for Gaia light curves. I knew where to find them, because I use to hang out with the right crowd of cool kids (the IVOA folks), so I never needed to search for Gaia light curves. Well, let's test that:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Table length=1307>\n",
      "                   ivoid                    ...\n",
      "                                            ...\n",
      "                   object                   ...\n",
      "------------------------------------------- ...\n",
      "     ivo://archive.stsci.edu/catalogs/atlas ...\n",
      "   ivo://archive.stsci.edu/catalogs/gaiadr1 ...\n",
      "   ivo://archive.stsci.edu/catalogs/gaiadr2 ...\n",
      "     ivo://archive.stsci.edu/catalogs/ucac5 ...\n",
      "  ivo://astronet.ru/cas/gaiadr2-gaia_source ...\n",
      "                  ivo://cds.vizier/b/pollux ...\n",
      "                     ivo://cds.vizier/i/324 ...\n",
      "                     ivo://cds.vizier/i/325 ...\n",
      "                     ivo://cds.vizier/i/331 ...\n",
      "                                        ... ...\n",
      "                     ivo://svo.cab/cat/wdw2 ...\n",
      "                     ivo://svo.cab/cat/wdw3 ...\n",
      "    ivo://uni-heidelberg.de/gaia/conesearch ...\n",
      "           ivo://uni-heidelberg.de/gaia/tap ...\n",
      " ivo://uni-heidelberg.de/gaiadr1/conesearch ...\n",
      " ivo://uni-heidelberg.de/gaiadr2/conesearch ...\n",
      "ivo://uni-heidelberg.de/gaiaedr3/conesearch ...\n",
      "    ivo://uni-heidelberg.de/tgas/conesearch ...\n",
      "                ivo://vopdc.obspm/gepi/gaia ...\n",
      "            ivo://vopdc.obspm/imcce/miriade ...\n"
     ]
    }
   ],
   "source": [
    "results=pyvo.registry.search(keywords='Gaia')\n",
    "\n",
    "print (results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<Table length=5>\n",
      "             ivoid              ...\n",
      "                                ...\n",
      "             object             ...\n",
      "------------------------------- ...\n",
      " ivo://cds.vizier/j/a+a/620/a91 ...\n",
      " ivo://cds.vizier/j/a+a/649/a98 ...\n",
      " ivo://cds.vizier/j/a+a/661/a44 ...\n",
      "ivo://cds.vizier/j/a+a/664/a162 ...\n",
      "  ivo://org.gavo.dc/gaia/q2/ssa ...\n"
     ]
    }
   ],
   "source": [
    "results=pyvo.registry.search(keywords='Gaia lightcurves')\n",
    "print (results)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ivo://cds.vizier/j/a+a/587/a112\n",
      "ivo://cds.vizier/j/a+a/619/a51\n",
      "ivo://cds.vizier/j/a+a/619/l10\n",
      "ivo://cds.vizier/j/a+a/622/a60\n",
      "ivo://cds.vizier/j/a+a/623/a110\n",
      "ivo://cds.vizier/j/a+a/623/a83\n",
      "ivo://cds.vizier/j/a+a/627/a135\n",
      "ivo://cds.vizier/j/a+a/641/a51\n",
      "ivo://cds.vizier/j/a+a/649/a7\n",
      "ivo://cds.vizier/j/a+a/656/a66\n",
      "ivo://cds.vizier/j/a+a/657/a131\n",
      "ivo://cds.vizier/j/a+a/657/a31\n",
      "ivo://cds.vizier/j/a+a/664/a162\n",
      "ivo://cds.vizier/j/a+a/667/a83\n",
      "ivo://cds.vizier/j/a+a/668/a59\n",
      "ivo://cds.vizier/j/a+a/671/a50\n",
      "ivo://cds.vizier/j/a+a/672/a31\n",
      "ivo://cds.vizier/j/a+a/672/a69\n",
      "ivo://cds.vizier/j/aj/153/154\n",
      "ivo://cds.vizier/j/aj/153/204\n",
      "ivo://cds.vizier/j/aj/158/77\n",
      "ivo://cds.vizier/j/aj/159/114\n",
      "ivo://cds.vizier/j/aj/160/220\n",
      "ivo://cds.vizier/j/aj/161/170\n",
      "ivo://cds.vizier/j/aj/162/100\n",
      "ivo://cds.vizier/j/apj/835/173\n",
      "ivo://cds.vizier/j/apj/844/66\n",
      "ivo://cds.vizier/j/apj/884/20\n",
      "ivo://cds.vizier/j/apj/888/73\n",
      "ivo://cds.vizier/j/apj/898/173\n",
      "ivo://cds.vizier/j/apj/909/115\n",
      "ivo://cds.vizier/j/apj/911/51\n",
      "ivo://cds.vizier/j/mnras/462/3616\n",
      "ivo://cds.vizier/j/mnras/481/1195\n",
      "ivo://cds.vizier/j/mnras/486/4574\n",
      "ivo://cds.vizier/j/mnras/493/1996\n",
      "ivo://cds.vizier/j/mnras/499/4026\n",
      "ivo://cds.vizier/j/mnras/513/420\n",
      "ivo://cds.vizier/j/other/newa/88.j1607\n",
      "ivo://cds.vizier/j/other/rmxaa/55.337\n",
      "ivo://org.gavo.dc/gaia/q2/dr2epochflux\n",
      "ivo://org.gavo.dc/tap\n"
     ]
    }
   ],
   "source": [
    "results=pyvo.registry.search(keywords='Gaia time series')\n",
    "\n",
    "for i in results:\n",
    "    print (i['ivoid'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ivo://cds.vizier/i/350\n",
      "ivo://cds.vizier/i/353\n",
      "ivo://cds.vizier/i/354\n",
      "ivo://cds.vizier/i/355\n",
      "ivo://cds.vizier/i/356\n",
      "ivo://cds.vizier/i/357\n",
      "ivo://cds.vizier/i/358\n",
      "ivo://cds.vizier/i/359\n",
      "ivo://cds.vizier/i/360\n",
      "ivo://cds.vizier/ii/374\n",
      "ivo://cds.vizier/j/a+a/617/a135\n",
      "ivo://cds.vizier/j/a+a/624/a13\n",
      "ivo://cds.vizier/j/a+a/638/a76\n",
      "ivo://cds.vizier/j/a+a/653/a160\n",
      "ivo://cds.vizier/j/a+a/656/a110\n",
      "ivo://cds.vizier/j/a+a/659/a167\n",
      "ivo://cds.vizier/j/a+a/660/a34\n",
      "ivo://cds.vizier/j/a+a/661/a103\n",
      "ivo://cds.vizier/j/a+a/662/a40\n",
      "ivo://cds.vizier/j/a+a/664/a109\n",
      "ivo://cds.vizier/j/a+a/666/a128\n",
      "ivo://cds.vizier/j/a+a/666/l4\n",
      "ivo://cds.vizier/j/a+a/667/a74\n",
      "ivo://cds.vizier/j/a+a/668/a99\n",
      "ivo://cds.vizier/j/a+a/669/a139\n",
      "ivo://cds.vizier/j/a+a/669/a91\n",
      "ivo://cds.vizier/j/a+a/670/a102\n",
      "ivo://cds.vizier/j/a+a/670/a75\n",
      "ivo://cds.vizier/j/a+a/670/l2\n",
      "ivo://cds.vizier/j/a+a/671/a149\n",
      "ivo://cds.vizier/j/a+a/672/a166\n",
      "ivo://cds.vizier/j/a+a/672/a183\n",
      "ivo://cds.vizier/j/a+a/672/a31\n",
      "ivo://cds.vizier/j/a+a/672/a65\n",
      "ivo://cds.vizier/j/a+a/672/a69\n",
      "ivo://cds.vizier/j/a+a/672/a81\n",
      "ivo://cds.vizier/j/a+a/672/a85\n",
      "ivo://cds.vizier/j/apj/858/l7\n",
      "ivo://cds.vizier/j/apj/875/89\n",
      "ivo://cds.vizier/j/apj/891/39\n",
      "ivo://cds.vizier/j/apj/912/125\n",
      "ivo://cds.vizier/j/apj/917/21\n",
      "ivo://cds.vizier/j/apj/935/l9\n",
      "ivo://cds.vizier/j/apjs/238/16\n",
      "ivo://cds.vizier/j/apjs/263/31\n",
      "ivo://cds.vizier/j/apjs/265/12\n",
      "ivo://cds.vizier/j/mnras/503/3660\n",
      "ivo://cds.vizier/j/mnras/506/150\n",
      "ivo://cds.vizier/j/mnras/510/4308\n",
      "ivo://cds.vizier/j/mnras/516/l61\n",
      "ivo://cds.vizier/j/mnras/520/1390\n",
      "ivo://cds.vizier/j/mnras/521/2087\n",
      "ivo://cds.vizier/v/155\n",
      "ivo://cds/p/dm/flux-bp/i/355/gaiadr3\n",
      "ivo://cds/p/dm/flux-color-rp-g-bp/i/355/gaiadr3\n",
      "ivo://cds/p/dm/flux-g/i/355/gaiadr3\n",
      "ivo://cds/p/dm/i/355/gaiadr3\n",
      "ivo://cds/p/dm/radial-velocity/i/355/gaiadr3\n",
      "ivo://esavo/gaia/csdr3\n",
      "ivo://nasa.heasarc/milliquas\n",
      "ivo://org.gavo.dc/amlensing/q3/q\n",
      "ivo://org.gavo.dc/apfs/res/apfs_new/gedr3query\n",
      "ivo://org.gavo.dc/gaia/q3/cone\n",
      "ivo://org.gavo.dc/gaia/q3/dr3lite\n",
      "ivo://org.gavo.dc/gaia/s3/pub\n",
      "ivo://org.gavo.dc/gaia/s3/ssa\n",
      "ivo://org.gavo.dc/gedr3dist/q/cone\n",
      "ivo://org.gavo.dc/gedr3mock/q/pub\n",
      "ivo://org.gavo.dc/tap\n",
      "ivo://uni-heidelberg.de/gaia/conesearch\n",
      "ivo://uni-heidelberg.de/gaia/tap\n"
     ]
    }
   ],
   "source": [
    "results=pyvo.registry.search(keywords='Gaia dr3')\n",
    "for i in results:\n",
    "    print (i['ivoid'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}