Guide to ISO Data Products - ISO

Guide to ISO Data Products

The ISO Data Archive contains various sets of data products that the user can download for independent scientific analysis. These are the pipeline products, that are kept on disk at the ISO Data Center in Villafranca, Spain and accessible through the ISO Data Archive interface. The pipeline products are subject to a scientific validation by the Instrument Teams. In order to facilitate browsing through the archive and identifying data to download, the so-called browse products are generated by an additional software.

In addition, the ISO Data Archive contains the so-called 'Highly Processed Data Products' (HPDP). These products include DATA (images, spectra etc.), which have been processed beyond the pipeline and/or using new, refined algorithms and therefore have been improved to any degree compared to the pipeline products, as well as any resulting CATALOGUES and ATLASES.

Pipeline Products

Every observation is run through an automatic data-analysis pipeline called Off-Line Processing, or OLP, to produce three sets of data products, namely:

ERD (Edited Raw Data)
SPD (Standard Processed Data)
AAR (Auto-Analysis Results)

The various types of analysis performed are described, as appropriate, in the ISO Handbook.

Detailed technical descriptions of the contents and formats of data product files are given in the corresponding sections of the ISO Handbook.

The type and sophistication of analysis performed for each product set is instrument dependent. In the Archive, products are further classified according to the following scheme:

- Raw Data Products: These are essentially unpacked telemetry in which no data reduction has taken place and thus form the starting point for data analysis performed by OLP or the user's own system. PHT ERD is an example.
- Basic Science Data Products: These data have been processed further to an intermediate level (with the use of calibration files where necessary), often containing physical units. They are therefore ready for assembly into structures of astronomical significance such as images or spectra by OLP or independently by observers themselves before scientific conclusions can be drawn. For example, the final result of the LWS pipeline is the LSAN file that contains a tabulation of flux in physical units measured during the observation.
- Fully Auto-processed Science Data Products: These data include a set of coherent, instrument-independent measurements of images or spectra designed to get as close as possible by automatic means to what could be produced by an astronomer using an interactive analysis system. Where possible, products follow FITS standard conventions to ease further analysis by standard high-level tools such as IDL or MIDAS. They allow the user to carry out initial survey programmes or to make an assessment of the relevance of a given observation to their scientific work, although the limitations must be emphasised of such an automatic procedure which takes no account of the individual circumstances of an observation and involves no scientific judgement. Significantly better results are likely to be possible using an interactive system. Nevertheless, the CAM pipeline for example produces individual FITS image in the CMAP file, that are combined where appropriate into mosaics in the CMOS file; gives a tabulation of sources detected in the images in the CPSL file; and gives individual source spectra as appropriate in the CSSP file.

For the spectrometers, LWS and SWS, the AAR products contain basic science data, for CAM and PHT these are the SPD products. An overview of the files per product level for each instrument can be found in list of files per product level. Each observation has a Data Quality Report including the quality information available.

The pipeline products undergo a scientific validation to establish confidence in the processing and calibration of the products. In the ISO Handbook an overview of the calibration status and the accuracies of the pipeline products are given. The scientific validation per AOT summarizes the current overall status.

Browse Products from the Pipeline Products: Icons, Postcards and Survey Products

>From the standard pipeline products a number of products are generated by an independent software to provide users with:

Survey Products
These are fully reduced standard data sets, either FITS images or ASCII FITS tables, for survey-type work. Although these products contain fully reduced data, it must be emphasised that the processing is done in a standard and automatic way which does not involve any scientific judgement. These products may serve for statistical or survey-type analysis of large samples.

Icons and Postcards
These are static representations in GIF-format of the survey products. Icons are small images intended to give an impression of the data: photometry, an image or a spectrum. Postcards are essentially enlarged icons with annotations added to give users an impression of the flux levels and wavelengths covered. Icons and postcards facilitate a quick-look to scan the data for their particular purposes and identify which data need to be retrieved. These should never be used for scientific work. The postcards are also used to give a quick look impression of the ISO data for a given observation when browsing through ISO information from other archives/databases within the so-called Postcard Server. Through calling a URL / Java Server Page (JSP) containing the ISO observation identifier (so called TDT number), it returns the ISO postcard (GIF image) of this observation and ancillary information embedded into an HTML page. The information embedded in the Postcard Server is dynamically generated via direct queries to the IDA database. This currently includes information related to the observation technical quality and the status of validation of the observation mode used. Links are also provided to specific sections of the ISO documentation (observation mode description and associated pipeline products accuracies and caveats). When relevant, specific caveats text is dynamically assembled depending on the observation parameters.

A short description follows of the main, instrument-dependent, characteristics of the browse products.

CAM

- Survey Products: These products are the equivalent of the AAR, giving the image from the CMOS file when it exists and CMAP otherwise.
- Icons and Postcards: The postcard is the image of the survey product projected in RA,DEC coordinates. It contains a grey-scale coded wedge to indicate flux levels. The icon is a small version of the CMAP/CMOS image shown in detector coordinates. For the CAM spectral observations (CAM04) the mean image of all wavelengths is displayed; the spectrum shown is that of the innermost 10x10 pixels. Up to four multi-filter or multi-PFOV measurements (for a given observation) are shown inside the corresponding icon.

LWS

remove all data points with bad status
clip outliers
remove individual scans which are inconsistent with the majority of the scans
for each detector the remaining scans are averaged

- Survey Products: The following processing steps are applied to the standard AAR LSAN file:; It should be noted that the resulting spectrum is not stiched together nor is it averaged across detectors. Also, no defringing is applied.
- Icons and Postcards: Postcard and icon are spectra. In the case of a raster map only the spectrum of the central point is presented.

PHT

spectrum or flux per filter against wavelength plots (single or multi-filter staring observations with PHT-P, spectrophotometry with PHT-S)
flux against aperture size (multi-aperture photometry)
image (single filter raster maps with PHT-P and PHT-C)
mosaic (raster map with PHT-S, multi-filter image for PHT-P and PHT-C)

- Survey Products: The survey product is the equivalent of AAR. No survey product is derived for polarization observations.
- Icons and Postcards: Depending on the observing mode, postcards and icons are:

SWS

remove all data points with bad or unreliable status
flatfield the data such that all detectors are scaled to the same level
clip outliers using sigma clipping
rebin the data to a grid with the nominal resolution

- Survey Products: The following processing steps are applied to the standard AAR file:
- Icons and Postcards: Postcard and icon are spectra which are presented in a mosaic form when the observation contains more than one wavelength range.

Highly Processed Data Products (HPDP)

The Off-Line Processing Pipeline copes well with a number of instrumental artifacts in an automatic fashion. The final products can however be improved by processing them further, in particular by means of the Interactive Analysis software packages, as documented in the ISO Handbook.

Many ISO papers are based on a systematic reduction of archive data, producing 'Highly Processed Data Products' (HPDP). These products include DATA (images, spectra etc.), which have been processed beyond the pipeline and/or using new, refined algorithms and therefore have been improved to any degree compared to the OLP 10 products, as well as any resulting CATALOGUES and ATLASES.

To query and download these data, access the ISO Data Archive and look at the specific help sections, at the top-right help menu or directly clicking on the "Highly Processed Data Products" panel.

Projects have been undertaken by the ISO Data Centre, in collaboration with the national instrument data centres, for systematic data reduction of specific instrument modes, that produce homogeneous sets of HPDP.

Browse Products from the HPDPs: Icons, Postcards and Survey Products

Browse Products have also been generated from the HPDPs which are default dataset.

ISO VO Compatible Products

The Survey Products are in most cases the ISO VO compatible products. They are generated from the standard pipeline AAR file or from the Highly Processed Data Products.

Only in the case of CAM, specific VO compatible products have been generated which are different from the standard Survey Products. The CAM products that are accessible from VO have been also produced based on the pipeline products. They will evolve in time according to the compliance with the image access protocol. More information is given in the CAM VO document.

VO products can be displayed from the ISO Data Archive using direct access to VO Tools as VOSpec (for spectra) and Aladin (for images).

iso

Guide to ISO Data Products - ISO