How to extract Gaia data programmatically - Gaia Users

Python Access: Astroquery

ESA Gaia Archive provides a Python Astroquery (Astropy) package for easy access to the ESA Gaia Archive: astroquery.gaia. The latest Astroquery version can be instaled following the instructions given in the "How can I install the latest version of Astroquery.Gaia" FAQ.

Also, a generic package is provided for accessing any TAP and TAP+ compliant service: astroquery.utils.tap.

See a Python tutorial for Jupyter Notebook example here.

Command line access: TAP/TAP+

The entry point is a TAP (Table Access Protocol) server.

You may use HTTP protocol to execute TAP requests at https://gea.esac.esa.int/tap-server/tap. TAP provides two operation modes: Synchronous and Asynchronous:

• Synchronous: the response to the request will be generated as soon as the request received by the server.
• Asynchronous: the server will start a job that will execute the request. The first response to the request is the required information (a link) to obtain the job status. Once the job is finished, the results can be retrieved.

Our TAP server provides two access mode: public and authenticated:

• Public: this is the standard TAP access. A user can execute ADQL queries and upload tables to be used in a query 'on-the-fly' (these tables will be removed once the query is executed). The results are available to any other user and they will remain in the server for a limited space of time.
• Authenticated: some functionalities are restricted to authenticated users only The results are saved in a private user space and they will remain in the server for ever (they can be removed by the user).
  • ADQL queries and results are saved in a user private area.
  • Cross-match operations: a catalogue cross-match operation can be executed. Cross-match operations results are saved in a user private area.
  • Persintance of uploaded tables: a user can upload a table in a private space. These tables can be used in queries as well as in cross-matches operations.

Here you can find some examples about how to interact with a TAP server (we are using the curl tool):

1. Non authenticated access
2. Authenticated access
3. Interface

1. Non authenticated access

Content:

1.1. Getting all public tables​​​​

curl "https://gea.esac.esa.int/tap-server/tap/tables"	

1.2. Synchronous query

curl "https://gea.esac.esa.int/tap-server/tap/sync?REQUEST=doQuery&LANG=ADQL&FORMAT=votable&QUERY=SELECT+TOP+5+source_id,ra,dec+FROM+gaiadr1.gaia_source"

The retrieved results is a VO table by default (see '3.2. Synchronous Queries' section parameters to specify a different output format). The results can be saved in a file and inspected using any analysis tool like TOPCAT, for instance.

1.3. Synchronous query on an 'on-the-fly' uploaded table

curl --form UPLOAD="table_c,param:table1" --form table1=@test_ra_dec.vot --form LANG=ADQL --form REQUEST=doQuery --form QUERY="select top 5 * from tap_upload.table_c" https://gea.esac.esa.int/tap-server/tap/sync			

Where 'test_ra_dec.vot' is a file that contains the VOTable to be uploaded (in order to be used by the query)

The retrieved results is a VO table by default (see '3.2. Synchronous Queries' section parameters to specify a different output format). The results can be saved in a file and inspected using any analysis tool like TOPCAT, for instance.

1.4. Asynchronous query

curl -i -X POST --data "PHASE=run&LANG=ADQL&LANG=ADQL&REQUEST=doQuery&QUERY=select+top+5+*+from+gaiadr1.gaia_source" "https://gea.esac.esa.int/tap-server/tap/async"

curl -i -X POST --data "PHASE=run&LANG=ADQL&JOBNAME=optionalJobName&JOBDESCRIPTION=optionalDescription&LANG=ADQL&REQUEST=doQuery&QUERY=select+top+5+*+from+gaiadr1.gaia_source" "https://gea.esac.esa.int/tap-server/tap/async"			

The response will contain the URL of the job running at server side (see Location header):

HTTP/1.1 303 See Other
Date: Mon, 30 Jun 2014 14:44:39 GMT
Server: Apache-Coyote/1.1
Location: https://gea.esac.esa.int/tap-server/tap/async/1404139480755A
Content-Type: application/x-www-form-urlencoded
Connection: close
Transfer-Encoding: chunked			

To obtain the status of the running job:

curl "https://gea.esac.esa.int/tap-server/tap/async/1404139480755A"

The status response is something like:

<?xml version="1.0" encoding="UTF-8"?>
<uws:job xmlns:uws="http://www.ivoa.net/xml/UWS/v1.0" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
        <uws:jobId><![CDATA[1404139480755A]]></uws:jobId>
        <uws:runId xsi:nil="true" />
        <uws:ownerId><![CDATA[anonymous]]></uws:ownerId>
        <uws:phase>COMPLETED</uws:phase>
        <uws:quote xsi:nil="true" />
        <uws:startTime>2014-06-30T16:44:40.766+0200</uws:startTime>
        <uws:endTime>2014-06-30T16:44:40.830+0200</uws:endTime>
        <uws:executionDuration>0</uws:executionDuration>
        <uws:destruction>2014-07-07T16:44:40.754+0200</uws:destruction>
        <uws:parameters>
                <uws:parameter id="maxRec"><![CDATA[100000]]></uws:parameter>
                <uws:parameter id="query"><![CDATA[select top 5 * from gaiadr1.gaia_source]]></uws:parameter>
                <uws:parameter id="request"><![CDATA[doQuery]]></uws:parameter>
                <uws:parameter id="format"><![CDATA[votable]]></uws:parameter>
                <uws:parameter id="keepAuthenticatedUserJobs"><![CDATA[true]]></uws:parameter>
                <uws:parameter id="lang"><![CDATA[ADQL]]></uws:parameter>
                <uws:parameter id="version"><![CDATA[1.0]]></uws:parameter>
        </uws:parameters>
        <uws:results>
                <uws:result id="result" xlink:type="simple" xlink:href="http%3A%2F%2Fgea.esac.esa.int%2Ftap-server%2Ftap%2Fasync%2F1404139480755A%2Fresults%2Fresult" mime="application/x-votable+xml" size="8863" rows="6" />
        </uws:results>
        <uws:errorSummary xsi:nil="true" />

To obtain the results of the job (once the job is finished):

curl "https://gea.esac.esa.int/tap-server/tap/async/1404139480755A/results/result"

The retrieved results is a VO table by default (see '3.3 Asynchronous Queries' section parameters to specify a different output format).

The results can be saved in a file and inspected using any analysis tool like TOPCAT, for instance.

1.5. Python low-level script to execute an asynchronous query and wait until the results are available.

It is also possible to access the Archive using a low-level Python script that does not require external libraries like Astroquery.

#ASYNCHRONOUS REQUEST

#Python 2
#import httplib
#import urllib

#Python 3
import http.client as httplib
import urllib.parse as urllib
import time
from xml.dom.minidom import parseString

host = "gea.esac.esa.int"
port = 443
pathinfo = "/tap-server/tap/async"


#-------------------------------------
#Create job

params = urllib.urlencode({\
	"REQUEST": "doQuery", \
	"LANG":    "ADQL", \
	"FORMAT":  "votable_plain", \
	"PHASE":  "RUN", \
	"JOBNAME":  "Any name (optional)", \
	"JOBDESCRIPTION":  "Any description (optional)", \
	"QUERY":   "SELECT DISTANCE(POINT('ICRS',ra,dec), POINT('ICRS',266.41683,-29.00781)) AS dist, * FROM gaiadr1.gaia_source  WHERE 1=CONTAINS(POINT('ICRS',ra,dec),CIRCLE('ICRS',266.41683,-29.00781, 0.08333333)) ORDER BY dist ASC"
	})

headers = {\
	"Content-type": "application/x-www-form-urlencoded", \
	"Accept":       "text/plain" \
	}

connection = httplib.HTTPSConnection(host, port)
connection.request("POST",pathinfo,params,headers)

#Status
response = connection.getresponse()
print ("Status: " +str(response.status), "Reason: " + str(response.reason))

#Server job location (URL)
location = response.getheader("location")
print ("Location: " + location)

#Jobid
jobid = location[location.rfind('/')+1:]
print ("Job id: " + jobid)

connection.close()

#-------------------------------------
#Check job status, wait until finished

while True:
	connection = httplib.HTTPSConnection(host, port)
	connection.request("GET",pathinfo+"/"+jobid)
	response = connection.getresponse()
	data = response.read()
	#XML response: parse it to obtain the current status
	#(you may use pathinfo/jobid/phase entry point to avoid XML parsing)
	dom = parseString(data)
	phaseElement = dom.getElementsByTagName('uws:phase')[0]
	phaseValueElement = phaseElement.firstChild
	phase = phaseValueElement.toxml()
	print ("Status: " + phase)
	#Check finished
	if phase == 'COMPLETED': break
	#wait and repeat
	time.sleep(0.2)


connection.close()

#-------------------------------------
#Get results
connection = httplib.HTTPSConnection(host, port)
connection.request("GET",pathinfo+"/"+jobid+"/results/result")
response = connection.getresponse()
data = response.read().decode('iso-8859-1')
#print(type(data))
#print(data)
outputFileName = "example_votable_output.vot"
outputFile = open(outputFileName, "w")
outputFile.write(data)
outputFile.close()
connection.close()
print ("Data saved in: " + outputFileName)
				

The saved file is a VO table (by default, see '3.3 Asynchronous Queries' section parameters to specify a different output format). The file can be inspected using any analysis tool like TOPCAT, for instance.

2. Authenticated access

Content:

2.1. Login

curl -k -c cookies.txt -X POST -d username=USERNAME -d password=PASSWORD -L "https://gea.esac.esa.int/tap-server/login"

2.2. Logout

curl -k -b cookies.txt -X POST -d -L "https://gea.esac.esa.int/tap-server/logout"

2.3. Getting public and user tables

curl -k -b cookies.txt -X POST -L "https://gea.esac.esa.int/tap-server/tap/tables"

2.4. Upload user table

curl -k -b cookies.txt -X POST -F FILE=@file.name -F TABLE_NAME=table_name "https://gea.esac.esa.int/tap-server/Upload"

Where 'file.name' is a file that contains the VOTable to be uploaded.
TABLE_DESC parameter can be used to specify a table description.
FORMAT parameter can be used to specify the table data format. Available formats: 'VOTable', 'CSV' and 'ASCII': see section 3.7 for more details.

2.5. Delete user table

curl -k -b cookies.txt -X POST -F TABLE_NAME=table_name -F DELETE=TRUE "https://gea.esac.esa.int/tap-server/Upload"		

Where 'table.name' is the argument used when the table was uploaded (argument TABLE_NAME=table_name).

To force a table removal, use FORCE_REMOVAL=TRUE flag:

curl -k -b cookies.txt -X POST -F TABLE_NAME=table_name -F DELETE=TRUE -F FORCE_REMOVAL=TRUE "https://gea.esac.esa.int/tap-server/Upload"

2.6. Asynchronous query

curl -k -b cookies.txt -i -X POST --data "PHASE=run&LANG=ADQL&REQUEST=doQuery&QUERY=select+top+5+*+from+gaiadr1.gaia_source" "https://gea.esac.esa.int/tap-server/tap/async"

Note that there is the possibility to use the optional parameters "JOBNAME" to assign a name to the job and "JOBDESCRIPTION" to add a description:

curl -k -b cookies.txt -i -X POST --data "PHASE=run&LANG=ADQL&JOBNAME=optionalJobName&JOBDESCRIPTION=optionalDescription&REQUEST=doQuery&QUERY=select+top+5+*+from+gaiadr1.gaia_source" "https://gea.esac.esa.int/tap-server/tap/async"				

The response will contain the URL of the job running at server side:

HTTP/1.1 303 See Other
Date: Mon, 30 Jun 2014 15:02:00 GMT
Server: Apache-Coyote/1.1
Location: https://gea.esac.esa.int/tap-server/tap/async/1404140520859A
Content-Type: application/x-www-form-urlencoded
Connection: close
Transfer-Encoding: chunked

To obtain the status of the running job:

curl -k -b cookies.txt "https://gea.esac.esa.int/tap-server/tap/async/1404140520859A"

The status response is something like:

<?xml version="1.0" encoding="UTF-8"?>
<uws:job xmlns:uws="http://www.ivoa.net/xml/UWS/v1.0" xmlns:xlink="http://www.w3.org/1999/xlink" xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
        <uws:jobId><![CDATA[1404141177261A]]></uws:jobId>
        <uws:runId xsi:nil="true" />
        <uws:ownerId><![CDATA[USERNAME]]></uws:ownerId>
        <uws:phase>COMPLETED</uws:phase>
        <uws:quote xsi:nil="true" />
        <uws:startTime>2014-06-30T17:12:57.273+0200</uws:startTime>
        <uws:endTime>2014-06-30T17:12:57.864+0200</uws:endTime>
        <uws:executionDuration>0</uws:executionDuration>
        <uws:destruction>2014-07-07T17:23:31.362+0200</uws:destruction>
        <uws:parameters>
                <uws:parameter id="jobdescription"><![CDATA[]]></uws:parameter>
                <uws:parameter id="jobname"><![CDATA[]]></uws:parameter>
                <uws:parameter id="session"><![CDATA[1404141103551]]></uws:parameter>
                <uws:parameter id="maxRec"><![CDATA[100000]]></uws:parameter>
                <uws:parameter id="query"><![CDATA[SELECT DISTANCE(POINT('ICRS',ra,dec), POINT('ICRS',266.41683,-29.00781)) AS dist, *
FROM gaiadr1.gaia_source
WHERE 1=CONTAINS(POINT('ICRS',ra,dec),CIRCLE('ICRS',266.41683,-29.00781, 0.08333333)) ORDER BY dist ASC]]></uws:parameter>
                <uws:parameter id="request"><![CDATA[doQuery]]></uws:parameter>
                <uws:parameter id="keepAuthenticatedUserJobs"><![CDATA[true]]></uws:parameter>
                <uws:parameter id="format"><![CDATA[votable]]></uws:parameter>
                <uws:parameter id="lang"><![CDATA[ADQL]]></uws:parameter>
                <uws:parameter id="version"><![CDATA[1.0]]></uws:parameter>
        </uws:parameters>
        <uws:results>
                <uws:result id="result" xlink:type="simple" xlink:href="http%3A%2F%2Fgea.esac.esa.int%2Ftap-server%2Ftap%2Fasync%2F1404141177261A%2Fresults%2Fresult" mime="application/x-votable+xml" size="2468741" rows="4734" />
        </uws:results>
        <uws:errorSummary xsi:nil="true" />

To obtain the results of the job (once the job is finished):

curl -k -b cookies.txt "https://gea.esac.esa.int/tap-server/tap/async/1404140520859A/results/result"

The retrieved results is a VO table by default (see '3.3 Asynchronous Queries' section parameters to specify a different output format). The results can be saved in a file and inspected using any analysis tool like TOPCAT, for instance.

2.7. Shared tables

Shared tables are accessible once the user is authenticated. You can use the tables in the same way you use your own tables. For instance, if you want to get the first 5 rows of the table 'shared_table' from user 'user_joe' (i.e. 'user_joe._shared_table') in a synchronous mode, you can type:

curl -k -b cookies.txt "https://gea.esac.esa.int/tap-server/tap/sync?REQUEST=doQuery&LANG=ADQL&FORMAT=votable&QUERY=SELECT+TOP+5+*+FROM+user_joe.shared_table"

in asynchronous mode:

curl -k -b cookies.txt -i -X POST --data "PHASE=run&LANG=ADQL&REQUEST=doQuery&QUERY=select+top+5+*+from+user_joe.shared_table" "https://gea.esac.esa.int/tap-server/tap/async"

If you want to obtain a list of available tables and columns, you can type:

curl -k -b cookies.txt "https://gea.esac.esa.int/tap-server/tap/tables?share_info=true&share_accessible=true"					

If you want to obtain a list of available tables, you can type:

curl -k -b cookies.txt "https://gea.esac.esa.int/tap-server/tap/tables?share_info=true&share_accessible=true&only_tables=true"

2.8. Share capability

You have to create a shared group in order to share a table.
The following example creates a group named 'new_group' and adds two users ('user_id_1' and 'user_id_2') to it:

curl -k -b cookies.txt -X POST --data "action=CreateOrUpdateGroup&title=my_new_group&description=descripion&users_list=user_id1,user_id2" "https://gea.esac.esa.int/tap-server/tap/share"

The response will contain the group identifier.

OK
Group '<user_id>_1425480932331JC' successfully created.

Then, you can update a group (to add/remove users to a group, see 'Adding/Removing a user' below):

curl -k -b cookies.txt -X POST --data "action=CreateOrUpdateGroup&group_id=<group_identifier>&title=my_new_group&description=descripion&users_list=<new_users_list>" "https://gea.esac.esa.int/tap-server/tap/share"

To obtain a list of groups associated to the user (either the user is the owner or the user belongs to):

curl -k -b cookies.txt "https://gea.esac.esa.int/tap-server/tap/share?action=GetGroups"

The response:

<?xml version="1.0" encoding="UTF-8"?>
<sharedGroups>
<sharedGroup id="user_id_1425482038968JC" owner="user_id">
	<title><![CDATA[my_new_group]]></title>
	<description><![CDATA[descripion]]></description>
</sharedGroup>
<sharedGroup id="other_user_1417615009193I" owner="other_user">
	<title><![CDATA[gacs]]></title>
	<description><![CDATA[group of gacs developers]]></description>
</sharedGroup>
</sharedGroups>

In this case, the user 'user_id' is the owner of the group 'user_id_1425482038968JC', while the user 'other_user' owns 'other_user_1417615009193I' group.

To remove a group:

curl -k -b cookies.txt -X POST --data "action=RemoveGroup&group_id=<group_identifier>" "https://gea.esac.esa.int/tap-server/tap/share"

Adding a user:

curl -k -b cookies.txt -X POST --data "action=CreateUserGroup&group_id=<group_identifier>&user_id=<user_id>" "https://gea.esac.esa.int/tap-server/tap/share"

Removing a user:

curl -k -b cookies.txt -X POST --data "action=RemoveUserGroup&group_id=<group_identifier>&user_id=<user_id>" "https://gea.esac.esa.int/tap-server/tap/share"
						

To share a table named 'user_schema.table1' (full qualified table name) you need to create a shared item. In this example, a new item is created and shared to 'my_new_group' group:

curl -k -b cookies.txt -X POST --data "action=CreateOrUpdateItem&resource_type=0&title=user_schema.table1&description=description&items_list=group_id|Group|Read" "https://gea.esac.esa.int/tap-server/tap/share"

The response will contain the shared resource identifier.

OK
Resource '<user_id>_1425542442398JC' (type '0') successfully created.

Then, you can update a shared table:

curl -k -b cookies.txt -X POST --data "action=CreateOrUpdateItem&resource_id=<resource_identifier>&resource_type=0&title=user_schema.table1&description=description&items_list=group_id|Group|Read" "https://gea.esac.esa.int/tap-server/tap/share"	

To obtain a list of shared tables:

curl -k -b cookies.txt "https://gea.esac.esa.int/tap-server/tap/share?action=GetSharedItems"						

The response

<?xml version="1.0" encoding="UTF-8"?>
<sharedItems owner="user_id">
<sharedItem id="user_id_1425485873983JC" type="0">
	<title><![CDATA[user_schema.table]]></title>
	<description><![CDATA[description]]></description>
	<sharedToItems>
<sharedToItem shareTo="group_id_1425482038968JC" shareType="Group" shareMode="Read"/>
	</sharedToItems>
</sharedItem>
</sharedItems>

To remove a sharing (it does not remove the actual table):

curl -k -b cookies.txt -X POST --data "action=RemoveItem&resource_id=<resource_identifier>&resource_type=0" "https://gea.esac.esa.int/tap-server/tap/share"

2.9. Events capability

Each action (table creation/removal, sharing creation/removal, etc.) generates an event. Events are associated to a session and they are kept while the session (connecton) is open.
Events are grouped by types and each time has a 'latest modification time' (latest time that this type of event was generated).
The following example retrieves the events for a user:

curl -k -b cookies.txt "https://gea.esac.esa.int/tap-server/tap/event"

The response will contain the events and times associated.

event_type1=timestamp1
event_type2=timestamp2
...	

2.10. Notifications capability

Notifications provides information about some activities. Notifications are available for authenticated users only and are kept for a month.
The following example retrieves the notifications for a user:

curl -k -b cookies.txt "https://gea.esac.esa.int/tap-server/tap/notification?action=GetNotifications"

A response example:

notification_id1[type: notification_type1,notification_subtype1]=explanation1
notification_id2[type: notification_type2,notification_subtype2]=explanation2
...	

2.11. Table flags editing capability

Tables flags can be modified.
The following example explains how to set the flags of a table:

curl -k -b cookies.txt -X POST --data "ACTION=radec&TABLE_NAME=schema.table_name&RA=ra_column_name&DEC=dec_column_name" "https://gea.esac.esa.int/tap-server/TableTool"

Where schema.table_name is a full qualified table name (e.g. user_userid.table_name). ra_column specifies the ra column name. dec_column specifies the dec column name.

2.12. Listing jobs

Jobs can be listed using the following request:

curl -k -b cookies.txt "https://gea.esac.esa.int/tap-server/tap/jobs/list?offset=index&limit=jobs_number&order=order"

Where list specifies the jobs list (e.g. 'sync' or 'async'). It is mandatory.
index specifies the number of jobs to skip before beginning to return the first job. By default, it is '0'.
limit specifies the number of jobs to be returned. No limit by default is set.
order specifies the order of the results.

Example

curl -k -b cookies.txt "https://gea.esac.esa.int/tap-server/tap/jobs/async?offset=0&limit=20&order=CREATION_TIME:DESC"

2.13. Deleting jobs

Jobs can be deleted using their identifiers:

curl -k -b cookies.txt -X POST --data "JOB_IDS=job_id1,job_id2..." "https://gea.esac.esa.int/tap-server/tap/deletejobs"		

2.14. Queries on jobs

A job can be used as a table in a query.

Launch an asynchronous job:

curl -i -X POST --data "PHASE=run&LANG=ADQL&LANG=ADQL&REQUEST=doQuery&QUERY=select+top+5+*+from+gaiadr1.gaia_source" "https://gea.esac.esa.int/tap-server/tap/async"

Response:

HTTP/1.1 303 See Other
Date: Mon, 30 Jun 2014 14:44:39 GMT
Server: Apache-Coyote/1.1
Location: https://gea.esac.esa.int/tap-server/tap/async/1404139480755A
Content-Type: application/x-www-form-urlencoded
Connection: close
Transfer-Encoding: chunked

Use the job identifier (1404139480755A in this case) as a table:

curl "https://gea.esac.esa.int/tap-server/tap/sync?REQUEST=doQuery&LANG=ADQL&FORMAT=votable&QUERY=SELECT+TOP+5+source_id,ra,dec+FROM+JOB_UPLOAD.JOB1404139480755A"

NOTE always add 'JOB' to your job identifier, e.g. JOB1404139480755A

2.15. External TAP queries

A query can be executed in an external TAP server. When an external query is executed, an asynchronous job is launched to the remote TAP server. (See '3.15. TAP+ external TAP access capability' section parameters)

In synchronous mode, the results are served on the fly (in this example, we are accessing Herschel TAP service).

curl "https://gea.esac.esa.int/tap-server/tap/sync?REQUEST=external_tap&EXT_TAP_URL=http://archives.esac.esa.int/hsa/whsa-tap-server/tap&LANG=ADQL&FORMAT=votable_plain&QUERY=SELECT+TOP+5+observation_id,ra,dec+FROM+hsa.v_active_observation"

In asynchronous mode, the results are saved locally (in this example, we access to Vizier TAP service)

curl -i -X POST --data "REQUEST=external_tap&EXT_TAP_URL=http://tapvizier.u-strasbg.fr/TAPVizieR/tap&PHASE=run&LANG=ADQL&LANG=ADQL&QUERY=select+top+5+recno,lambda,flux+from+%22J/A%2BA/605/L8/quasar_a%22" "https://gea.esac.esa.int/tap-server/tap/async"

Response:

HTTP/1.1 303 See Other
Date: Mon, 30 Jun 2014 14:44:39 GMT
Server: Apache-Coyote/1.1
Location: https://gea.esac.esa.int/tap-server/tap/async/1404139480755A
Content-Type: application/x-www-form-urlencoded
Connection: close
Transfer-Encoding: chunked

Now, you have a local job (1404139480755A in this case) with the results of your external TAP query.

2.16. Global Tap Schema (GloTS) search

A global tap schema search service is available.

Search for services that contains a table name or description.

curl "https://gea.esac.esa.int/tap-server/tap/glots?ACTION=search_tables&keyword=quasars+new&MODE=all&FORMAT=csv"

In this case, we are searching for the keywords 'quasars new'

Response:

accessurl,ivoid,table_name,table_desc,utype
http://tapvizier.u-strasbg.fr/TAPVizieR/tap,ivo://cds.vizier/tap,J/A+AS/134/483/table3,"New QSOs from the Hamburg Quasar Survey Object ( Hagen H.-J., Engels D., Reimers D.)",
http://tapvizier.u-strasbg.fr/TAPVizieR/tap,ivo://cds.vizier/tap,J/AJ/113/2000/table1,"New quasar radio detections ( Bischof O.B., Becker R.H.)",
http://tapvizier.u-strasbg.fr/TAPVizieR/tap,ivo://cds.vizier/tap,J/AJ/122/518/table1,"New LBQS quasars ( Hewett P.C., Foltz C.B., Chaffee F.H.)",
http://tapvizier.u-strasbg.fr/TAPVizieR/tap,ivo://cds.vizier/tap,J/AJ/142/78/table1,"Parameters of 14 new 2.2<z<3.0 quasars in S82 ( Wu X.-B., Wang R., Schmidt K.B., Bian F., Jiang L., Fan X.)",
http://tapvizier.u-strasbg.fr/TAPVizieR/tap,ivo://cds.vizier/tap,J/AJ/145/159/table1,"Catalog of new quasars in the vicinity of M31 and M33 discovered by LAMOST in fields observed in 2010 and 2011 ( Huo Z.-Y., Liu X.-W., Xiang M.-S., et al.)",
http://tapvizier.u-strasbg.fr/TAPVizieR/tap,ivo://cds.vizier/tap,J/AJ/151/24/table1,"Parameters of the 1180 New Quasars Discovered in LAMOST DR1 ( Ai Y.L., Wu X.-B., Yang J., Yang Q., Wang F., Guo R., Zuo W., Dong X., Zhang Y.-X., Yuan H.-L., Song Y.-H., Wang J., Dong X., Yang M., Wu H., Shen S.-Y., Shi J.-R., He B.-L., Lei Y.-J., Li Y.-B., Luo A.-L., Zhao Y.-H., Zhang H.-T.)",
http://tapvizier.u-strasbg.fr/TAPVizieR/tap,ivo://cds.vizier/tap,J/ApJS/194/22/table1,"29 new, 12 plausible, and 3 previously known quasars behind the SMC ( Kozlowski S., Kochanek C. S., Udalski A.)",
http://tapvizier.u-strasbg.fr/TAPVizieR/tap,ivo://cds.vizier/tap,J/ApJS/227/11/table5,"Spectroscopic observations of the new PS1 quasars (77 unique sources) ( Banados E., Venemans B.P., Decarli R., et al.)",
http://tapvizier.u-strasbg.fr/TAPVizieR/tap,ivo://cds.vizier/tap,J/other/RAA/15.1438/table1,"Catalog of new quasars in fields in the vicinity of M31 and M33 discovered by LAMOST in the 2013 observational season ( Huo Z.-Y., Liu X.-W., Xiang M.-S., Shi J.-R., Yuan H.-B., Huang Y., Zhang Y., Hou Y.-H., Wang Y.-F., Yang M.)",
http://tapvizier.u-strasbg.fr/TAPVizieR/tap,ivo://cds.vizier/tap,VII/182/table6,"new UV excess quasars (paper II) ( Warren S.J., Hewett P.C., Irwin M.J., Osmer P.S.)", 						

In order to retrieve all columns of a table, we have to provide the 'ivoid' service identifier and the table name:

curl "https://gea.esac.esa.int/tap-server/tap/glots?ACTION=get_columns&IVOID=ivo%3A%2F%2Fcds.vizier%2Ftap&TABLE_NAME=J%2FA%2BAS%2F134%2F483%2Ftable3&FORMAT=csv"

Response:

ivoid,table_name,column_name,description,unit,ucd,utype,datatype,ndhnpmiighpa,principal,indexed,std
ivo://cds.vizier/tap,J/A+AS/134/483/table3,Bmag,B magnitude,mag,phot.mag;em.opt.B, ,DOUBLE,mt.size,0,0,1
ivo://cds.vizier/tap,J/A+AS/134/483/table3,DE2000,Declination (2000.0),deg,pos.eq.dec;meta.main, ,DOUBLE,mt.size,1,0,1
ivo://cds.vizier/tap,J/A+AS/134/483/table3,Name,Object name, ,meta.id;meta.main, ,VARCHAR,mt.size,0,0,1
ivo://cds.vizier/tap,J/A+AS/134/483/table3,RA2000,Right ascension (2000.0),deg,pos.eq.ra;meta.main, ,DOUBLE,mt.size,1,0,1
ivo://cds.vizier/tap,J/A+AS/134/483/table3,Run,Run number, ,meta.id;obs, ,SMALLINT,mt.size,0,0,1
ivo://cds.vizier/tap,J/A+AS/134/483/table3,l_Bmag,[~] Approximate flag on Bmag, ,meta.code.error, ,CHAR(1),mt.size,0,0,1
ivo://cds.vizier/tap,J/A+AS/134/483/table3,recno,Record number assigned by the VizieR team. Should Not be used for identification., ,meta.record, ,INTEGER,mt.size,0,0,1
ivo://cds.vizier/tap,J/A+AS/134/483/table3,z,Redshift, ,src.redshift, ,DOUBLE,mt.size,0,0,1

3. Interface

See the following specifications:

• TAP (Table Access Protocol) v1.0
• UWS (Universal Worker Service) v1.0
• ADQL (Astronomical Data Query Language) v2.0

Content:

3.1. TAP resources

https://gea.esac.esa.int/tap-server/tap/

3.2. Synchronous Queries

3.3. Asynchronous Queries

The response header will contain the location of the job.

curl --form UPLOAD="table_c,param:table1" --form table1=@test_ra_dec.vot --form LANG=ADQL --form REQUEST=doQuery --form QUERY="select top 5 * from tap_upload.table_c" https://gea.esac.esa.int/tap-server/tap/sync

Command line access: Datalink

The Datalink dedicated server provides a service listing all available resources per source linked at the [server]/datalink/links? entry point. To invoke the service the source ID value in the DR2 Gaia Source Catalogue should be passed as follows:

curl "https://geadata.esac.esa.int/data-server/datalink/links?ID=5350554572442654336&TABLE=dr2.gaia_source"

The response is something like:

<?xml version="1.0" encoding="UTF-8"?>
<VOTABLE version="1.3" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.ivoa.net/xml/VOTable/v1.2" xmlns="http://www.ivoa.net/xml/VOTable/v1.2">
	<RESOURCE type="results">
		<INFO name="QUERY_STATUS" value="OK" />
		<INFO name="PROVIDER" value=""></INFO>
		<INFO name="QUERY"><![CDATA[select datalink_url, epoch_photometry_url from user_dr2int5.gaia_source where source_id='5350554572442654336']]></INFO>
		<INFO name="PAGE"><![CDATA[null]]></INFO>
		<INFO name="PAGE_SIZE"><![CDATA[null]]></INFO>
		<INFO name="JOBID"><![CDATA[1518173886217ODL]]></INFO>
		<INFO name="JOBNAME"><![CDATA[Datalink job for user anonymous]]></INFO>
		<TABLE>
		<FIELD name="ID" ucd="meta.id;meta.main" datatype="char" arraysize="*"/>
		<FIELD name="datalink_url" ucd="meta.ref.url" datatype="char" arraysize="*"/>
		<FIELD name="semantics" ucd="meta.code" datatype="char" arraysize="*"/>
		<FIELD name="description" ucd="meta.note" datatype="char" arraysize="*"/>
		<FIELD name="content_type" ucd="phys.size;meta.file" datatype="char" arraysize="*"/>
		<FIELD name="content_length" ucd="meta.code.mime" datatype="long"/>
		<DATA>
		<TABLEDATA>
		<TR>
		<TD>5350554572442654336</TD>
		<TD>https://geadata.esac.esa.int/data-server/data?RETRIEVAL_TYPE=epoch_photometry&ID=5350554572442654336&TABLE=dr2.epoch_photometry&KEY_COLUMN_NAME=source_id</TD>
		<TD>#this</TD>
		<TD></TD>
		<TD></TD>
		<TD></TD>
		</TR>
		</TABLEDATA>
		</DATA>
		</TABLE>
	</RESOURCE>
</VOTABLE>

This service is public and can also be accessed through the main gaia_source catalogue table, where a dedicated column datalink_url contains the direct access to the content available through Datalink services associated to the individual resource (ID). For instance the following query will return the available links per source:

SELECT top 5 source_id, datalink_url FROM gaiadr2.gaia_source

Epoch Photometry

The Epoch Photometry service provides time series serialization of the lightcurves available in the Gaia archive and is linked via the Datalink service when available.

The light curves output data models are described here. Individual and combined structures follow light_curve, while raw is defined by epoch_photometry.

The entry point to the service is at https://geadata.esac.esa.int/data-server/data?RETRIEVAL_TYPE=epoch_photometry. By default, this service is also linked through the main DR2 Gaia Source table under the epoch_photometry_url column.

The service accepts a number of parameters depending on the requested resources:

The only mandatory request parameter is the ID of a Gaia DR2 source. If no additional parameters are included in the request, the output will be filtered to only include valid data and all photometry bands will be included when available. The output format will be all individual source ID files in binary VO Table format compressed in a zip file:

curl -k -b cookies.txt "https://geadata.esac.esa.int/data-server/data?RETRIEVAL_TYPE=epoch_photometry&ID=5350554572442654336"

curl -k -b cookies.txt "https://geadata.esac.esa.int/data-server/data?RETRIEVAL_TYPE=epoch_photometry&ID=5350554572442654336&DATA_STRUCTURE=COMBINED"						

Any combination of the optional request parameters from the above table is allowed. The following example shows how to retrieve all data (i.e. not filtered) associated to a source for the BP band with FITS as output format:

curl -k -b cookies.txt "https://geadata.esac.esa.int/data-server/data?RETRIEVAL_TYPE=epoch_photometry&ID=5350554572442654336&VALID_DATA=false&BAND=BP&FORMAT=FITS"

This service also allows to request several lightcurves from different sources in a single request. The list of multiple IDs can be passed directly in the request as a comma-separated values (e.g. 16061459700373248), ranges (e.g. 720000000000000-1800000000000000), or a combination of both. For example:

curl -k -b cookies.txt "https://geadata.esac.esa.int/data-server/data?RETRIEVAL_TYPE=epoch_photometry&ID=1460000000000000-2345017784412032,1514169246023424,16061459700373248,720000000000000-1800000000000000&FORMAT=FITS"

The service also provides a specific interface to support both user defined tables or jobs where the list of source ids is passed as a specific column of the job or the user table (this interface requires to first login as an authenticated user - see Sect. 2 of the Command line access tutorial).

curl -k -b cookies.txt "https://geadata.esac.esa.int/data-server/data?RETRIEVAL_TYPE=epoch_photometry&ID=job:1530537474992O.source_id"
curl -k -b cookies.txt "https://geadata.esac.esa.int/data-server/data?RETRIEVAL_TYPE=epoch_photometry&ID=user_schema.user_table.source_id"

The table below summarizes the different expected formats for a request of multiple IDs:

ID(s) Input Formats