Implementation Guides

A) Object Visibility Service

A basic web service (REST) complaint with ObjVisSAP can be downloaded as python source code from github under:
https://github.com/emiliosalazardonate/visibility-service/blob/master/README.md
It listening for requests in port 8000. It needs python 3.6 and uses two libraries django and astropy:

Django==3.0.3 astropy==3.0.4

To run the server:
/path/to/python3/bin/python manage.py runserver

Starting development server at http://127.0.0.1:8000/

B) Observation Locator Service

This page summarises how to implement an ObsLocTAP service. The full description of this protocol can be found under the IVOA (International Virtual Observatory Alliance) domain at:

http://www.ivoa.net/documents/ObsLocTAP/index.html

The Observation Locator Table Access Protocol (ObsLocTAP henceforth) specifies in a standard format the services to retrieve information about planned, scheduled and performed observations. By sharing this information, the scientific community can monitor the planning activities of the observatories, prepare coordinated observations or to allow the preparation of better proposals from the scientific point of view.

Following the approach of the newer IVOA protocols, ObsLocTAP is a Tabular Access Protocol (TAP) service. This service exposes a database table using a query language close to SQL but with astronomical operators that allow better exploration of astronomical resources. This language called Astronomical Data Query Language (ADQL)

http://www.ivoa.net/documents/REC/ADQL/ADQL-20081030.pdf

is a standard for the virtual observatory. 

The implementation of a full compatible TAP service from scratch is cumbersome. However, there are a set of toolkits that allow the creation of this services without major knowledge of the underlying standards and very fast. We will describe how to implement a TAP service using a very well toolkit called taptuto but there are others. Implementors could take a look into the different alternatives and give them a try.

Other very well known toolkits are:

Points 3 and 5 are applicable for all the toolkits to create a table ObsLocTAP compliant.

1. ObsLocTAP components

Three basic components are needed to implement an ObsLocTAP service

  • Database installation: In order to facilitate the geometrical queries needed for a TAP service, we recommend the use of a PostgreSQL DB instance
  • Creation of ObsLocTAP complaint table (database)
  • TAP service instance: Several frameworks can be used to create a TAP service. We will use, as example, the one created by Gregory Mantelet, currently working at CDS, due to its level of documentation. This package is open source
  • Publication of tables through TAP server (TAP_SCHEMA)

2. Database installation

There are several pages where we can find how to install a PosgreSQL server. In most of the cases, just following the instructions of https://www.postgresql.org/download/ should be enough to have a PosgreSQL server up and running. In fact, many of the different server images have already a PosgreSQL instance that can be used. Although PosgreSQL v10 contains some interesting features, v9.6 can guarantee at this stage operational performance and compatibility with extensions. There are multiple tutorials to install PosgreSQL depending on the operating system on the web.

It is recommendable to implement periodic dumps and backups in order to restore an operational database in case of failure (see https://www.postgresql.org/docs/9.6/backup.html). Also, it is recommendable to use a special account (e.g. postgres) for the database instance.

Once you have your PosgreSQL instance up and running, there is one spatial extension that would be needed for the TAP service instance. There are other spatial indexing extensions but the TAP service instance used in point 2 is making use of pgsphere.

Installation of the extension following the instructions at http://pgsphere.github.io/download.html is more or less transparent.

3. Creation of ObsLocTAP complaint table (database)

Create a database and a table with the following structure:

CREATE SCHEMA ivoa;
CREATE TABLE ivoa.obsplan (
    t_planning        double precision NOT NULL,
    target_name       character varying,
    obs_id            character varying NOT NULL,
    obs_collection    character varying NOT NULL,
    s_ra              double precision,
    s_dec             double precision,
    s_fov             double precision,
    s_region          scircle,
    s_resolution      double precision,
    t_min             double precision NOT NULL,
    t_max             double precision NOT NULL,
    t_exptime         double precision,
    t_resolution      double precision,
    em_min            double precision,
    em_max            double precision,
    em_res_power      double precision,
    o_ucd             character varying NOT NULL,
    pol_states        character varying,
    pol_xel           integer,
    facility_name     character varying NOT NULL,
    instrument_name   character varying NOT NULL,
    obs_release_date  timestamp,
    t_plan_exptime    double precision NOT NULL,
    category          character varying,
    priority          integer,
    execution_status  character varying NOT NULL
);
CREATE INDEX i_obsplan_fov
 ON ivoa.obsplan
 USING gist
 (s_region);

Please notice that the field of view (s_region column) is defined as a pgsphere object. Other types different than scircle could be used (see https://pgsphere.github.io/doc/types.html). For example, if the planned observation can be very well defined (including, e.g. the position angle), a typical type used for footprints would be polygon. Both, circle and polygon, can be used in the same way for ADQL searches. Unfortunately, two different types of pgsphere objects cannot be combined in the same database column (that means, you cannot have observations defined as circles and other defined as polygons into the same s_region column of the same table)

The last create index sentence defines how to create a gist index on the pgsphere object to increase the performance.

In this case, we have defined s_region as a scircle pgsphere object. Typical value would be a spoly. However, there is not an easy way to mix pgsphere circles and polygons in the same column as they do not have a generic data type. One possible solution is to store everything as spoly and convert the circles in approximated polygons.

The output of a select on s_region is typically a string representation of the Field of View. The conversion from a pgsphere output to a STC-S output when this column is used on a SELECT condition is typically done by the TAP server.

For example, polygon is defined as:

Polygon J2000 158.408362 57.781849 158.501991 57.812066 158.558573 57.762136 158.465026 57.73196

where the points of the polygon is defined after J2000 in pairs of ra dec values.

Whenever there is some uncertainty on the final pointing, a good option is to define the stc_s representation just as a circle, with the central position and the radius defined as:

Circle J200 <s_ra> <s_dec> <s_fov>

where the values of s_ra, s_dec and s_fov are the other columns in the obsplan table. Whenever the Field of View is better defined, an upgrade from the circle to something more detailed could be done.

See more info at http://ivoa.net/documents/STC-S/

It would be a good idea to populate this table with some dummy rows for testing during next steps.

If you are familiar with docker, we have created a docker instance with posgreSQL, pgsphere and the ivoa.obsplan table with some dummy fields for testing.

https://hub.docker.com/r/jsalgadodocker/postgres9.5-pgsphere-obsplan

4. TAP Service instance

There are some frameworks to be used to create a TAP instance. We recommend to use the one created by Gregory Mantelet at CDS as it is quite well documented and it has been updated to the latests ADQL (Astronomical Data Query Language) changes.

Installation instructions can be found here: http://cdsportal.u-strasbg.fr/taptuto/

Some notes:

  • ObsLocTAP database could/should be named ivoa  instead of the one mentioned into the example  MyStarCatalogues to clarify the schema
  • Table to be published will be called obsplan

The instructions provided by these pages are long but very descriptive so, in principle, you should be able to have a service running at the end of this process.

5. Publication of tables through TAP server

TAP_SCHEMA should contain the description of the published table(s) through the TAP service. Using the SQL script provided in the previous framework (or by other method), the characterization of the table in the TAP_SCHEMA should contain the following information:

ivoa.obsplan t_planning adql:DOUBLE d
ivoa.obsplan target_name adql:VARCHAR  
ivoa.obsplan obs_id adql:VARCHAR  
ivoa.obsplan obs_collection adql:VARCHAR  
ivoa.obsplan s_ra adql:DOUBLE deg
ivoa.obsplan s_dec adql:DOUBLE deg
ivoa.obsplan s_fov adql:DOUBLE deg
ivoa.obsplan s_resolution adql:DOUBLE arcsec
ivoa.obsplan t_min adql:DOUBLE d
ivoa.obsplan t_max adql:DOUBLE d
ivoa.obsplan t_exptime adql:DOUBLE s
ivoa.obsplan t_resolution adql:DOUBLE s
ivoa.obsplan em_min adql:DOUBLE m
ivoa.obsplan em_max adql:DOUBLE m
ivoa.obsplan em_res_power adql:DOUBLE  
ivoa.obsplan o_ucd adql:VARCHAR  
ivoa.obsplan pol_states adql:VARCHAR  
ivoa.obsplan pol_xel adql:BIGINT  
ivoa.obsplan facility_name adql:VARCHAR  
ivoa.obsplan instrument_name adql:VARCHAR  
ivoa.obsplan obs_release_date adql:TIMESTAMP date
ivoa.obsplan t_plan_exptime adql:DOUBLE s
ivoa.obsplan category adql:VARCHAR  
ivoa.obsplan priority adql:INTEGER  
ivoa.obsplan fov adql:VARCHAR  
ivoa.obsplan stc_s adql:VARCHAR  

 

The metadata that should be added to the basic/plain TAP_SCHEMA created by TAPTUTO to the TAP_SCHEMA would be the following:

INSERT INTO "TAP_SCHEMA".tables (schema_name,table_name,table_type,description,utype,dbname) VALUES (
'TAP_SCHEMA','TAP_SCHEMA.schemas','table','List of schemas published in this TAP service.',NULL,NULL);

INSERT INTO "TAP_SCHEMA".tables (schema_name,table_name,table_type,description,utype,dbname) VALUES (
'TAP_SCHEMA','TAP_SCHEMA.tables','table','List of tables published in this TAP service.',NULL,NULL);

INSERT INTO "TAP_SCHEMA".tables (schema_name,table_name,table_type,description,utype,dbname) VALUES (
'TAP_SCHEMA','TAP_SCHEMA.coosys','table','List of coordinate systems of coordinate columns published in this TAP service.',NULL,NULL);
INSERT INTO "TAP_SCHEMA".tables (schema_name,table_name,table_type,description,utype,dbname) VALUES (
'TAP_SCHEMA','TAP_SCHEMA.columns','table','List of columns of all tables listed in TAP_SCHEMA.TABLES and published in this TAP service.',NULL,NULL);
INSERT INTO "TAP_SCHEMA".tables (schema_name,table_name,table_type,description,utype,dbname) VALUES (
'TAP_SCHEMA','TAP_SCHEMA.keys','table','List all foreign keys but provides just the tables linked by the foreign key. To know which columns of these tables are linked, see in TAP_SCHEMA.key_columns using the key_id.',NULL,NULL);
INSERT INTO "TAP_SCHEMA".tables (schema_name,table_name,table_type,description,utype,dbname) VALUES (
'TAP_SCHEMA','TAP_SCHEMA.key_columns','table','List all foreign keys but provides just the columns linked by the foreign key. To know the table of these columns, see in TAP_SCHEMA.keys using the key_id.',NULL,NULL);
INSERT INTO "TAP_SCHEMA".tables (schema_name,table_name,table_type,description,utype,dbname) VALUES (
'ivoa','obsplan','table','ObsLocTAP compatible table','','');
INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES (
'ivoa.obsplan','t_planning','','d','','','adql:DOUBLE',0,0,0,0,'t_planning','');
INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES (
'ivoa.obsplan','target_name','','','','','adql:VARCHAR',32,0,0,0,'target_name','');
INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES (
'ivoa.obsplan','obs_id','','','','','adql:VARCHAR',32,0,0,0,'obs_id','');
INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES (
'ivoa.obsplan','obs_collection','','','','','adql:VARCHAR',32,0,0,0,'obs_collection','');
INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES (
'ivoa.obsplan','s_ra','','deg','','','adql:DOUBLE',0,0,0,0,'s_ra','');
INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES (
'ivoa.obsplan','s_dec','','deg','','','adql:DOUBLE',0,0,0,0,'s_dec','');
INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES (
'ivoa.obsplan','s_fov','','deg','','','adql:DOUBLE',0,0,0,0,'s_fov','');
INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES (
'ivoa.obsplan','s_resolution','','arcsec','','','adql:DOUBLE',0,0,0,0,'s_resolution','');
INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES (
'ivoa.obsplan','t_min','','d','','','adql:DOUBLE',0,0,0,0,'t_min','');
INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES (
'ivoa.obsplan','t_max','','d','','','adql:DOUBLE',0,0,0,0,'t_max','');
INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES (
'ivoa.obsplan','t_exptime','','s','','','adql:DOUBLE',0,0,0,0,'t_exptime','');
INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES (
'ivoa.obsplan','t_resolution','','s','','','adql:DOUBLE',0,0,0,0,'t_resolution','');
INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES (
'ivoa.obsplan','em_min','','m','','','adql:DOUBLE',0,0,0,0,'em_min','');
INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES (
'ivoa.obsplan','em_max','','m','','','adql:DOUBLE',0,0,0,0,'em_max','');
INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES (
'ivoa.obsplan','em_res_power','','','','','adql:DOUBLE',0,0,0,0,'em_res_power','');
INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES (
'ivoa.obsplan','o_ucd','','','','','adql:VARCHAR',32,0,0,0,'o_ucd','');
INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES (
'ivoa.obsplan','pol_states','','','','','adql:VARCHAR',32,0,0,0,'pol_states','');
INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES (
'ivoa.obsplan','pol_xel','','','','','adql:BIGINT',0,0,0,0,'pol_xel','');
INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES (
'ivoa.obsplan','facility_name','','','','','adql::VARCHAR',32,0,0,0,'facility_name','');
INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES (
'ivoa.obsplan','instrument_name','','','','','adql::VARCHAR',32,0,0,0,'instrument_name','');
INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES (
'ivoa.obsplan','obs_release_date','date','','','','adql::TIMESTAMP',0,0,0,0,'obs_release_date','');
INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES (
'ivoa.obsplan','t_plan_exptime','s','','','','adql::DOUBLE',0,0,0,0,'t_plan_exptime','');
INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES (
'ivoa.obsplan','category','','','','','adql::VARCHAR',32,0,0,0,'category','');
INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES (
'ivoa.obsplan','priority','','','','','adql::INTEGER',0,0,0,0,'priority','');
INSERT INTO "TAP_SCHEMA".columns (table_name,column_name,description,unit,ucd,utype,datatype,"size",principal,indexed,std,dbname,coosys_id) VALUES (
'ivoa.obsplan','s_region','','','','','adql::VARCHAR',32,0,0,0,'s_region','');

 

6. Testing the system

Once the service is up and running, some queries can be used to test the system. Some examples are included in the specification but, as an example, we present some queries used for the ObsLocTAP Integral Observatory service. The service URL for this service is:

https://iladev.esac.esa.int/tap-dev/tap/sync?

For other services, this service URL will be different although the rest of parameters (REQUEST, PHASE, LANG, QUERY) are the typical TAP service one so these will be common for the rest of services.

6.1. Search of observations on a certain time range

ADQL Query:

SELECT o.s_ra, o.s_dec, o.target_name, o.t_min, o.t_max
FROM ivoa.obsplan AS o
WHERE o.t_max > 56783 ORDER BY o.t_min

Service invocation:

https://iladev.esac.esa.int/tap-dev/tap/sync?REQUEST=doQuery&PHASE=RUN&LANG=ADQL&QUERY=select+o.s_ra%2C+o.s_dec%2C+o.target_name%2C++o.t_min%2C+o.t_max+from+ivoa.obsplan+as+o+where+o.t_max%3E56738+order+by+o.t_min

6.2. Search of observations on a certain astronomical area

ADQL Query:

SELECT * FROM ivoa.obsplan WHERE
1=INTERSECTS(s_region, CIRCLE('ICRS', 114.8251, 1.6179, 0.016666))

Service invocation:


https://iladev.esac.esa.int/tap-dev/tap/sync?REQUEST=doQuery&PHASE=RUN&LANG=ADQL&QUERY=SELECT%20%2A%20FROM%20ivoa.obsplan%20WHERE%201%3DINTERSECTS%28s_region%2C%20CIRCLE%28%27ICRS%27%2C%20114.8251%2C%201.6179%2C%200.016666%29%29

6.3. Combination of time and geometrical conditions 

ADQL Query:

SELECT * FROM ivoa.obsplan WHERE
t_max < 58502 AND
1=INTERSECTS(s_region, CIRCLE('ICRS', 114.8251, 1.6179, 0.016666))

Service invocation:

https://iladev.esac.esa.int/tap-dev/tap/sync?REQUEST=doQuery&PHASE=RUN&LANG=ADQL&QUERY=SELECT%20%2A%20FROM%20ivoa.obsplan%20WHERE%20t_max%20%3C%2058502%20AND%201%3DINTERSECTS%28s_region%2C%20CIRCLE%28%27ICRS%27%2C%20114.8251%2C%201.6179%2C%200.016666%29%29

7. Playing with dockers

Although it is not recommended to use docker containers for an operational archive, we have defined two dockers containers for the DB and the TAP server that you can use to have a ObsLocTAP server up and running in minutes in you have the docker command line client installed.

You just need to:

docker pull jsalgadodocker/pgsphere-obsplan:latest
docker pull jsalgadodocker/tapserver:latest
docker network create --driver=bridge db-network
docker image ls
docker run -p 8080:8080 --net=db-network --name tap jsalgadodocker/tapserver:latest
docker run -p 5432:5432 --net=db-network --name db  jsalgadodocker/pgsphere-obsplan:latest

You will have a TAP server with an obsplan table with some Integral records as test data running on your localhost. It could be tested by

http://localhost:8080/tap/tap/

Also, you can connect to port 5432 of your localhost with a dabase client (e.g. pgadmin3) to see the records or to ingest your own ones.

In order to make public your service, you can e.g. create a proxy pass redirect from an apache web server, use port 80 and open it to internet, etc.

(please send comments on this page to Jesus.Salgado@sciops.esa.int)