Analysis chain for point-like sources (xmmextractor)

Introduction

The purpose of xmmextractor is to produce a final set of XMM-Newton science products from all instruments for a point source with given sky coordinates, starting from the ODF.

The long-term goal is to provide an interactive tool, but at this stage xmmextractor can only reliably be used for an initial default processing of a given ODF.

Expected Outcome

  • Events files, Images, Spectra, Response Matrices, and Light Curves from MOS[1,2], PN, RGS[1,2], and OM.
  • A default configuration file in xml format (if none was provided with the call).
  • Log files with run-time information, errors, warnings, and science results.

SAS Tasks to be Used

Prerequisites

Useful Links

Caveats

Last Reviewed: 25 MAY 2023, for SAS v21.0

Last Updated: 30 OCTOBER 2020

 


Description

Starting from the uncalibrated files stored in the Observation Data File (ODF, see Short Guide to XMM-Newton Science Data Products), calibrated events files are produced (using most up-to-date calibration files) from which in turn spectra and light curves can be extracted using source-specific filter criteria. This is a multi-step procedure that needs to be carried out for each instrument separately.

The purpose of xmmextractor is to combine all these steps with a single command. All user input is provided via a configuration file. A default configuration file can be generated, based on the individual situation (e.g., pointing coordinates, exposure times, exposure numbers). A separate tool, odfParamCreator, is available for the creation of a default configuration file that can then be edited with a text editor.

Preparation

The ODF can be downloaded from the XMM-Newton Science Archive, XSA.

xmmextractor can be run from any working directory. All output will be stored in the current working directory. Various subdirectories will be created in which the products will be stored.

The following preparatory steps are required before running xmmextractor:

  • Download ODF data from XSA
  • Initialize HEADAS/HEASOFT
  • Initialize SAS, following the SAS Startup Thread.
    The tasks cifbuild and odfingest do not need to be run unless a customized session is to be generated (see below).
  • As part of the SAS setup, the environment variable SAS_ODF needs to point to the ODF directory (full path).
  • Create and change to the directory where the output should be stored. This directory will be called analysis directory.

The current implementation of xmmextractor only works reliably without any parameters. A call without parameters produces a full extraction of all data from all instruments in all modes. For a description of a default run, the following steps can be skipped, and proceed to Usage.

Preparation of a configuration file

A customized xmmextractor session is foreseen but currently not reliably possible. The concept of customizing an xmmextractor run is still explained here.

The infrastructure for customizing an xmmextractor run allows the flexiblility to analyse data from only a single instrument, or producing only spectra or light curves. A customized configuration file can be created and later provided when calling xmmextractor. The configuration file has to be formatted in the Extensible Markup Language (XML) which follows a hierarchical structure. A default file can be generated for later editing using the SAS task,

odfParamCreator outputFileName=rundefault.xml

This requires the prior setup of the SAS including the running the tasks cifbuild and odfingest.

The resulting xml file (in this case rundefault.xml), can be edited with a text editor. The structure of the configuration file is described in the Package description of odfParamCreator. The default configuration file directs xmmextractor to produce events files, spectra, and light curves for all instruments. Non-default extractions such as extractions only for one instrument, can be communicated to xmmextractor by editing the configuration file accordingly. An example is given below,

 

The first block, <OBSERVATION>, contains performance setup in the format: <PARAM id="analysisoption" default="0:all"/>, where the parameter name is given in quotes behind id= and the parameter value in quotes behind default=. The first parameter, analysisoption, determines which level of products are to be extracted. The default, 0:all, indicates that the full analysis is performed. Other options allow extraction of only events files, only spectra or light curves, etc. All options are explained in the description of xmmextractor. The remaining parameters are self-explanatory. The data reduction can be turned off for each instrument by changing default="yes" to default="no" behind the corresponding instrument for which no data analysis is desired.

The remaining blocks contain configuration information for each available instrument and are named <INSTRUMENT value="...."/>, where the name of the instrument is given in quotes. Within each instrument block, a separate block is present for each exposure, where the mode, exposure id, exposure time, and a flag whether or not this exposure is to be processed are given. Replacing process="yes" to process="no" means that the exposure is not processed. For each exposure, the products that the tool can produce are listed in blocks within which the task setup can be modified. Again, each yes can be replaced by no if not wanted. Parameter names and values can be modified. Not all details can be given here but should be found in the respective documentation for each task.

Usage

The most simple use and only fully functional mode is to type on the command line,

  xmmextractor >& screenoutput.log

where all screen output will be piped into the log file screenoutput.log if the command and the name of the log file are separated by >&. A default configuration file will be generated and can later be used to re run the tool with exactly the same setup. A copy can be created that can be modified and used to run the tool again with the modified setup.

After the configuration file has been adjusted to the individual needs, the tool can be run, giving the name of the configuration file:

  xmmextractor paramfile=rundefault.xml >& screenoutput.log

Note that without events files, no other products can be produced. Thus, events files must either already exist in the right directories, or the tool has to be run with analysisoption=1:events before using other options. If EPIC events files are to be recycled they need to be stored in the respective subdirectories pn or mos under the working directory and must be named according to the SAS naming convention

  • pn
    REVN_OBSID_EPN_SEEE_ImagingEvts.ds
    REVN_OBSID_EPN_SEEE_TimingEvts.ds
  • mos
    REVN_OBSID_EMOS[1,2]_SEEE_ImagingEvts.ds
    REVN_OBSID_EMOS[1,2]_SEEE_TimingEvts.ds

with REVN the 4-digit revolution number, OBSID the 10-digit observation ID, and EEE the exposure number.

If the tool is run again, existing files will not be overwritten. In particular, log files will be appended, and if events files have already been produced (which can take a long time), they will be used for the extraction of new light curves and spectra, for example for other sources in the field. Pipeline products can be used, but have to be renamed to SAS naming convention.

If events files have to be created, the full task can run for some 20-30 minutes or more. While running, brief pop ups of ds9 windows will occur for the only purpose of creating image plots as screen shots of ds9. They can be ignored or closed by hand.

Products

All products will be stored in the analysis directory from where the tool was launched. Performance output can be inspected from the three files,

  • xmmextractor.err: Containing errors
  • xmmextractor.warn: Containing warnings
  • xmmextractor.info: Containing runtime information

It is strongly recommended to follow up all errors and warnings before using the science products. If messages are not clear, the XMM-Newton HelpDesk can be contacted. The science products are stored in the subdirectories that have been generated in the analysis directory. In the following, the contents of each directory that was created is given:
 

mos Science files created by emproc:
  • REV#_OBSID_AttHk.ds: the reconstructed attitude file
  • REV#_OBSID_EMOS[1,2]_S***_**_Badpixels.ds: one table per reduced CCD containing the bad pixels
  • REV#_OBSID_EMOS[1,2]_S***_**_ImagingEvts.ds: calibrated and concatenated event list; see description of MOS imaging files
  • PLUS, if timing mode
  • REV#_OBSID_EMOS[1,2]_S***_**_TimingEvts.ds: calibrated and concatenated event list; see description of MOS timing files
pn Science files created by epproc:
  • REV#_OBSID_AttHk.ds: the reconstructed attitude file
  • REV#_OBSID_EPN_S***_**_Badpixels.ds: one table per reduced CCD containing the bad pixels
  • REV#_OBSID_EPN_S***_**_ImagingEvts.ds: calibrated and concatenated event list; see description of PN imaging files
  • OR, if timing mode
  • REV#_OBSID_EPN_S***_**_TimingEvts.ds: calibrated and concatenated event list; see description of PN timing files
rgs Science files created by rgsproc:
  • PxxxxxxyyyyR[1,2]eeeeEVENLI0000.FIT: event list; see description of RGS event list
  • PxxxxxxyyyyR[1,2]eeeeSRSPEC[1,2]001.FIT: source+background [first, second] order spectrum
  • PxxxxxxyyyyR[1,2]eeeeBGSPEC[1,2]001.FIT: background [first, second] order spectrum
  • PxxxxxxyyyyR[1,2]eeeeSRCLI_0000.FIT: source list
  • PxxxxxxyyyyR[1,2]eeeeRSPMAT[1,2]000.FIT: [first, second] order response matrix
  • PxxxxxxyyyyOBX000fluxed[1,2]000.FIT: combined fluxed [first, second] order spectrum
  • spatial_rgs[1,2].fit: fits image of dispersion angle versus cross-dispersion pixel
  • pi_rgs[1,2].fit: fits image of dispersion angle versus pi energy (also known as 'banana plot')
  • image_rgs[1,2].ps: RGS Spectroscopy diagnostic plots produced by rgsimplot
A number of RGS housekeeping files are stored, see list of output files of rgsproc
om Science files created by omichain, omfchain, and omgchain.
Example output can be seen for omichain
spectra Spectral products from EPIC cameras. Naming convention is: Name[m1,m2,pn]_SEEE_[ImagingEvts,TimingEvts]_type
with Name being the provided source name (default is the ObsID), EEE the exposure number, and type a type indicating the following:
  • type=[source,background]_spectrum.fits: source and background spectra. For spectral analysis with XSPEC, loading the source spectrum will automatically load background, effective areas and response file as well.
  • type=src.[arf,rmf]: Effective areas and response matrix.
  • type=[region,bkgregion,combinedregion].reg: ASCII Region files in ds9 format for source, background and both combined, respectively.
  • type=spectrum.ps: PS files showing image with extraction region marked
  • type=eregionanalyse.txt: script output from eregionanalyse for MOS1 and MOS2
lcurve Source and background light curves for MOS1, MOS2, PN, RGS1, and RGS2
Naming convention is: Name[m1,m2,pn]_SEEE_[ImagingEvts,TimingEvts]_type
with Name being the provided source name (default is the ObsID), EEE the exposure number, and type a type indicating the following:
  • type=[source,bkg,sourcebkgsubtracted].lc: Light curves for source and background and background-subtracted
  • type=[region,bkgregion,combinedregion].reg: ASCII Region files in ds9 format for source, background and both combined, respectively.
  • type=eregionanalyse.txt: script output from eregionanalyse for MOS1 and MOS2
In addition, filtered events files are stored by names [m1,m2,pn]_SEEE_[ImagingEvts,TimingEvts]_filtered.fit
images Images of events and exposure maps for all sources. Separate fits images are produced for each energy range defined by the parameters [m1,m2,pn]_ene_low and [m1,m2,pn]_ene_high (default is 4 ranges, b0, b1, b2, b3)
gti Good Time Intervals used for filtering
  • [m1,m2,pn]_SEEE_[ImagingEvts,TimingEvts]_back_lightc.fits: particle background light curve used to apply filter
  • [m1,m2,pn]_gti_SEEE_[ImagingEvts,TimingEvts].fits: Applied Good Time intervals
  • [m1,m2,pn]_SEEE_[ImagingEvts,TimingEvts]_img_gtifiltered.[fit.img]: filtered events files
epatplot Analysis plots for pile up, created by epatplot (see How to evaluate the pile-up fraction in an EPIC source)
results log files containing extraction parameters and science results:
  • science_target_name.log: Science information, e.g., count rate, background, regions
  • event_target_name.log: Information from EPIC Event Files
  • spectra_target_name.log: Extraction Details of EPIC spectra
  • lcurve_target_name.log: Extraction Details of EPIC light curves
  • rgs_target_name.log: Information from RGS Event File
  • rgs_lcurve_target_name.log: Extraction details of RGS light curves
  • gtievent_target_name.log: Details for good time interval filtering
  • edetect_chain_target_name.log: Count rate information from edetect_chain
  • om_chain_target_name.log: Detailed information from om_chain

 

Caveats

xmmextractor is a task that is still being developed. While basic usage such as first time data reduction with default parameters works, some of the higher-level options may produce problems with the present version. For reporting problems and asking questions, please don't hesitate to use the XMM-Newton HelpDesk.