DataLink Service

Authors: Héctor Cánovas and Jos de Bruijne

In each Gaia data release, the key parameters of all sources are stored in the gaia_source table that contains the (mean) astrometric, photometric, and radial-velocity data as well as astrophysical parameters. This table, along with the complementary tables for, for instance, variable stars or solar system objects, is accessible by means of the IVOA compliant TAP+ table access protocol that allows to explore astronomical datasets stored in relational data bases using the ADQL query language. In addition to the gaia_source table, Gaia DR3 includes vast amounts of non-tabular data such as mean spectra, epoch photometry, and Monte Carlo Markov Chain samples for millions of sources (while Gaia DR4 will include epoch astrometry and epoch photometry for the whole sample plus billions of mean and epoch spectra). Storing these non-tabular datasets as plain tables in a monolithic, relational data base is impractical. Instead, these products are hosted by a dedicated service designed to handle massive data requests that is accessible via the DataLink protocol. DataLink is a data access protocol compliant with the IVOA architecture that provides a linking mechanism between datasets offered by different services. In practice, it can be seen and used as a web service providing the list of additional data products available for each object outside the main catalogue(s).

Since the Archive upgrade to version 2.14 the VOTables generated by the Archive contain a new resource that facilitates the access to the DataLink products to IVOA compliant clients (like TOPCAT):

<RESOURCE type="meta" utype="adhoc:service" name="ancillary">
    <DESCRIPTION>Retrieve DataLink file containing ancillary data for source</DESCRIPTION>
    <PARAM name="standardID" datatype="char" arraysize="*" value="ivo://ivoa.net/std/DataLink#links-1.0"/>
    <PARAM name="accessURL" datatype="char" arraysize="*" value="https://gea.esac.esa.int/data-server/datalink/links"/>
    <PARAM name="contentType" datatype="char" arraysize="*" value="application/x-votable+xml;content=datalink"/>
    <GROUP name="inputParams">
         <PARAM name="ID"  datatype="char" arraysize="*" value="" ref="DESIGNATION"/>
    </GROUP>
</RESOURCE>

The entry point to the DataLink server is indicated by the "accessURL" parameter. To invoke the service and find out the resources associated to a given source it is necessary to combine the entry point with the target ID as:  

https://gea.esac.esa.int/data-server/datalink/links?ID=Gaia+DR3+30343944744320

The output is a xml file not intended for humans but for IVOA-compliant clients. Opening this file with TOPCAT reveals the following content:

Figure 1: Content of the xml file generated by the DataLink service when invoked as explained above.

This file contains the URLs that give access to the DataLink products associated to the target source. For information about how to access to these products from the Gaia Archive web interface and programmatically, please see the DataLink: Access from the Archive GUI and the Command line access: DataLink tutorials, respectively. The structure and content of the DataLink products are described in the Datamodel Chapter in the Gaia DR3 documentation and DataLink products serialisation tutorial.

DataLink: Access from the Archive web interface

Authors: Héctor Cánovas, Jos de Bruijne, and Alcione Mora

Last update: 2024-12-10

The Gaia DR3 data release includes vast amounts of non-tabular data such as high- and low-resolution (mean) spectra, epoch photometry, and Monte Carlo Markov Chain samples for millions of sources. These products are made available through a dedicated service designed to handle massive data requests using the DataLink protocol. This intermediate-level tutorial introduces the concepts needed to retrieve these non-tabular products using the Gaia ESA Archive web interface via its Advanced (ADQL) form. The complementary DataLink: command line access and DataLink: Python access tutorials describe the programmatic access to these products using the Unix curl command-line utility and the Python package Astroquery.Gaia, respectively, while the DataLink products serialisation tutorial describes the structure of these products. In case of difficulties following this tutorial, please consult the DataLink service and Advanced (ADQL) tab tutorials.

Tutorial content:

1. How it works
2. Basic use case
3. FPR tables
4. User tables
   1. List of sources
   2. List of coordinates

1. How it works

As explained in the DataLink Service tutorial above, the DataLink protocol allows users to search for and retrieve DataLink products associated with a list of Gaia designations, or alternatively, with a combination of Gaia source IDs and Gaia data releases. Direct access to this service through the web interface is not possible. Instead, users must first execute an ADQL query that generates an output table containing the source_id and/or designation fields. Since the release of GACSv3.6, the output table metadata includes a Gaia release tag (e.g., "Gaia DR3" or "Gaia DR2"), which is automatically fetched by the Archive web interface when preparing the request to the DataLink server.

2. Basic use case

One of the simplest use cases is likely: "I want to search for the DataLink products associated with the output of this Gaia DR3 query." The ADQL query in the example below performs a cone search with a radius of 0.25 degrees (see the Query examples section for more complex queries):

SELECT DISTANCE(266.41683, -29.00781, ra, dec) AS separation, *
FROM gaiadr3.gaia_source WHERE DISTANCE(266.41683, -29.00781, ra, dec) < 0.25
ORDER BY separation ASC

The first step to retrieve the DataLink products associated with this sample is to click on the double chain ("paperclip") icon in the job list area of the Advanced (ADQL) form. In this case, an error message will appear, explaining that the query result contains 19,758 sources, exceeding the threshold of 5,000 sources imposed to prevent overloading the DataLink server. To resolve this, you can either reduce the cone search radius or, preferably, use the has_<datalink_product> fields available in the gaiadr3.gaia_source table as follows:

SELECT DISTANCE(266.41683, -29.00781,ra, dec) AS separation, *
FROM gaiadr3.gaia_source WHERE DISTANCE(266.41683, -29.00781, ra, dec) <0.25 AND
-- Retrieve only sources with associated DataLink products 
has_epoch_photometry ='True' AND has_xp_sampled = 'True'
ORDER BY separation ASC

The output of this query contains just 26 sources, all of which have epoch photometry and both XP sampled and XP continuous spectra. Note: advanced users can use the job_upload mechanism to apply these filters to their previous queries. Clicking the double chain icon in the job list area again will launch the DataLink wizard, as shown in Fig. 1. This window lists all the available products associated with the sample generated by the previous query. One can retrieve only selected products (e.g., just RVS mean spectra) or download all products at once by clicking the "Save All Data" button. Note that, depending on the volume of data, preparing the dataset for download may take several minutes. In the DataLink wizard, one can also select different combinations of data structures and download formats (see the DataLink: products serialisation tutorial for details).

Figure 1: Screenshot of the Gaia ESA Archive web interface DataLink wizard that appears when clicking on the DataLink icon (double chain link encompassed by a red circle above) in the job lists area. The horizontal and vertical arrows point to the drop-down menus that allow to select the data structure (output serialisation) and file format of the files (see the DataLink: Products serialisation tutorial), respectively. 

The phot_variable_flag field in the main Gaia DR2 catalogue (gaia_dr2.gaia_source) allows to extract only the sources that include DR2 epoch photometry. The Gaia DR2 equivalent of the previous Gaia DR3 ADQL query is therefore:

SELECT DISTANCE(266.41683, -29.00781, ra, dec) AS separation, *
FROM gaiadr2.gaia_source
WHERE DISTANCE(266.41683, -29.00781, ra, dec) < 0.25 AND
-- Retrieve only sources with associated DataLink products 
phot_variable_flag = 'VARIABLE'
ORDER BY separation ASC

3. FPR tables

The Gaia Focused Product Release (FPR) contains 12 catalogues organised into five categories: "Crowded Fields," "Extra-galactic," "Solar System," "Spectroscopy," and "Variability." Among these, only the Extra-galactic (gaiafpr.lens_catalogue_name, gaiafpr.lens_observation, and gaiafpr.lens_outlier) and Variability (gaiafpr.vari_epoch_radial_velocity, gaiafpr.vari_long_period_variable, and gaiafpr.vari_rad_vel_statistics) tables include sources with associated DataLink products that were already published in Gaia Data Release 3. To access these products, it is necessary to JOIN these tables with any of the tables from Gaia Data Release 3 either using the JOB UPLOAD or the table upload mechanism in a two-step way, as shown in the example below:

SELECT TOP 10 * FROM gaiafpr.vari_epoch_radial_velocity

SELECT gaia.*
FROM gaiadr3.gaia_source AS gaia
JOIN job_upload."jobf127e639-2c33-11f1-9054-bc97e148b76b-O" AS USING (source_id)

Where the job id quoted in the last query ("f127e639-2c33-11f1-9054-bc97e148b76b-O") corresponds to the job id associated to the first query in this example.

Once the output result has been generated, please proceed as explained in the Basic use case section above.

4. User tables

Another popular use case is: "I want to retrieve the DataLink products associated with my list of sources (either valid Gaia source IDs/designations or coordinates)". In this case, the first step is to upload a table with the list of sources to one's user space, as detailed in the Upload a user table tutorial. The second step is to combine that table with any of the catalogues included in Gaia Data Release 3 (or Gaia DR2 if your list of sources belong to this release). Depending on the uploaded data (either source IDs/designations or coordinates), the ADQL query used to combine the two tables will involve a JOIN or a cross-match operation, as detailed below.

4.1 List of sources

If the uploaded list of sources contains valid Gaia source IDs and/or designations, a JOIN operation (as also explained in the Joining Catalogues section of the Catalogue combination tutorial) can be used to combine the tables as follows:

SELECT my_table.*, gaia.source_id
FROM <user_name>.<uploaded_table_name> AS my_table
JOIN gaiadr3.gaia_source AS gaia
  ON my_table.selected_source_ids = gaia.source_id

In this example, the field containing valid Gaia DR3 source IDs in the uploaded table is named as selected_source_ids. Once the output of this query has been generated, please proceed as explained in the Basic use case section above.

4.2 List of coordinates

In this alternative case, the uploaded table contains a list of coordinates; therefore, a cross-match is needed to identify the counterparts in the Gaia catalogue. The Cross-matching two catalogues (basic) section included in the Catalogue combination tutorial explains in detail how to do this using different approaches. For simplicity and to avoid duplication, we provide an example below that applies a basic cross-match - without extra filters or epoch propagation - to a list of equatorial coordinates uploaded to the Archive user space.

SELECT my_table.*, gaia.source_id
FROM <user_name>.<uploaded_table_name> AS my_table
JOIN gaiadr3.gaia_source AS gaia
  ON DISTANCE(my_table.ra, my_table.dec, gaia.ra, gaia.dec) < 5. / 3600.

In this example, the cross-match radius is set to 5 arcseconds, and the fields containing the right ascension and declination coordinates (in the table uploaded by the user) are named ra and dec, respectively. Once the output of this query has been generated, please proceed as explained in the Basic use case section above.

datalink products serialisation

Authors: Héctor Cánovas, Jos de Bruijne, María Henar, Elena Racero, and Alcione Mora

Last update: 2024-12-10

The DataLink IVOA protocol implemented in the Gaia ESA Archive gives access to six different products (epoch photometry, medium- and low-resolution spectra, and probability density distributions for the different astrophysical parameters) available for a significant fraction of the sources included in the main Gaia DR3 table (gaia_dr3.gaia_source). These products are serialised according to different data models, and all of them can be retrieved in multiple file formats as well as multiple data structures. This document describes the contents (both data and metadata) of the DataLink products in the various serialisations generated by the Archive. Further information about the implementation of the DataLink protocol in the Archive is briefly described in the DataLink Service, while the Datalink: Command line access and the DataLink: Access from the web interface describe how to retrieve the DataLink products through the Archive web interface and programmatically, respectively. This other tutorial shows how to use the Astroquery.Gaia Python package to download these products.

The DataLink products served by the Archive (and the data models applied to serialise them) are listed below:

(*) The DR2 serialisation follows the IVOA Times Series Cube serialisation.

The serialisation of each product is detailed in the following sections.

Tutorial content:

1. Retrieval parameters
   1. Data structure
   2. Download file format
   3. Output file naming
2. Data models
   1. Epoch Photometry
   2. MCMCs: GSP-Phot & MSC Samples
   3. XP Continuous Spectra
   4. XP Sampled mean Spectra and RVS mean Spectra

1. retrieval parameters

1.1 Data Structure

This parameter defines the structure of the file that is being prepared for download. There are two possible options:

1. INDIVIDUAL (default): one single file per product per selected source(s), with the data serialised in tabular format (one element per table cell).
2. RAW: one single file per product, with the data for multiple sources serialised in a tabular format (one or more elements per table cell).

The latter format is the one used internally by the DPAC consortium, and it is documented in the Gaia Data Release 3 documentation (see the Datamodel description chapter).

Since GACS 3.6, the COMBINED data structure has been deprecated. In practise, this structure can be described as a collection of vertically stacked data serialised in the INDIVIDUAL structure.

1.2 Download FILE Format

Available file download formats are: 

1. VOTable (both binary and plain-text formats, .xml extension)
2. FITS
3. CSV
4. *ECSV (Enhanced Character Separated Values)

The VOTable, FITS, and ECSV file formats provide the table fields and metadata with column descriptions, UCDs, UTYPEs, and units when applicable. The CSV file format only includes the column names.

1.3. output file naming

The data structure and download format define the names of the retrieved files as follows:

By default, the ouput data is downloaded as a compressed .gzip file. However, some internet browsers like, for instance, Safari, automatically expand these files (without asking the user). Note that the file name is agnostic to the data structure selected when retrieving ancillary data associated with a single source.

2. Data Models

The data model for the products serialised in the INDIVIDUAL data structure is described in the following subsections. FOr the sake of simplicity, only the content that differs from the DPAC (RAW) datamodel is detailed below.

2.1 EPOCH PHOTOMETRY

Since GACS 3.6, the INDIVIDUAL serialisation of the epoch photometry follows the IVOA Time Series: Annotation of light curves in VOTable (Case #1). For reference, the previous serialisation is detailed in the Archive Help historical content. The "source_id" and "solution_id" information is provided in the file metadata (as PARAM elements in the VOTable format), and all the table fields are described in the DPAC (RAW) data model except for the "rejected_by_photometry" and "<band>_other_flags" fields detailed below:

Example Python code to extract the Boolean flags encoded in the "g_other_flags" bitmask:

from astropy.table import Table
epoch   = Table.read('EPOCH_PHOTOMETRY-Gaia DR3 5616512450772689536.xml')
bitmask = epoch['g_other_flags'].tolist()
# Decode photometry_flag_sm_reject:
bit_pos      = 0
ind_flag_dec = [bool((inp >> bit_pos) & 1) for inp in bitmask]
# Decode photometry_flag_af2_unavailable:
bit_pos      = 15
ind_flag_dec = [bool((inp >> bit_pos) & 1) for inp in bitmask]

2.2 MCMC's: GSP-PHOT & MSC samples

Both the serialisation of the MCMC GSP-Phot samples and the MCMC MSC samples follows the DPAC data model, with the only exceptions being the "nsamples" field that is not included and the array serialisation (the array columns are flattened to one value per entry). No metadata is added into the file header, and all the information is repeated through the output table so each row is self-contained. 

2.3 XP CONTINUOUS spectra

The XP Continuous mean spectra are serialised without deviations from the original DPAC data model for any data structure. The table metadata contains neither units nor UTYPEs. 

2.4 XP SAMPLED mean spectra and rvs mean spectra

The INDIVIDUAL serialisation of the XP sampled mean spectra and the RVS mean spectra follows the IVOA Spectra Data Model. In the table below, the rows in white and green background indicate the fields included in the table metadata and data, respectively. Note that, due to the 8-characters length limit imposed by the FITS format to the keyword names, the (added) table metadata parameters are re-named when serialising this product in FITS format. None of the metadata fields is included in the files generated in .csv format, which follows a particular serialisation (similar to the DPAC RAW serialisation but including a wavelength column).

(*) The combined_transits, combined_ccds, and deblended_ccd PARAM elements are only present in the RVS mean spectra.

(**) XP sampled mean spectra flux and flux_error units are [W.m**-2.nm**-1], while the RVS mean spectra is normalised and hence its flux and flux_error have no units.

(***) The spatial location contains the ra and dec fields included in the XP sampled mean spectra and RVS mean spectra.

Figure 1: excerpt from the XP Sampled Spectrum table and metadata as shown by TOPCAT.

Datalink: python access

Authors: Héctor Cánovas and Jos de Bruijne

Last update: 2026-05-07

Tutorial - Programmatic download of large datasets through DataLink

Authors: Héctor Cánovas and Jos de Bruijne

Last update: 2026-05-07

Back to Gaia Help