Image of the Week

Run 1000 of the Initial Data Treatment system

   
 

Figure 1:IDT web-based front-end tool showing the 1000th IDT run. Click here to see how the monitoring system behaves during a few hours of the mission.

 

Yesterday, almost two years after the start of Gaia's nominal operations, the Initial Data Treatment system (IDT) finished its 1000th run - a remarkable milestone for this critical system in charge of the daily data processing in the Gaia Data Processing and Analysis Consortium (DPAC).

The IDT software is executed on some 28 computing nodes at the Gaia Science Operations Centre (SOC, at ESAC, near Madrid), receiving raw data from the spacecraft through the Mission Operations Centre (MOC, in Darmstadt), the MIT (MOC-SOC Interface Task), and the DCS (Decompression and Calibration Service). It processes spacecraft attitude data, astrometric, photometric, and spectroscopic instrument data, Basic Angle Monitor data, Wave Front Sensor data, and a handful of ancillary data packet types. Additionally, it runs complex algorithms to determine "intermediate data": refined attitude, basic angle variations, sky background, image parameters (fluxes, positions, quality, etc.) and a first cross-matching (identification of measurements in the reference on-ground catalogue). These algorithms require several instrumental calibrations determined either by IDT itself or by other DPAC systems.

All this must be done in real-time, on a daily basis throughout the whole mission, to feed the so-called First Look diagnostics and calibration system, as well as all the downstream Data Processing Centres (DPCs). Thus, IDT feeds the whole Gaia consortium.

The Initial Data Treatment works in "runs", which is a way to handle and identify a bunch of data measured and processed during a given time period. Typically, IDT does one run about every 23 hours of observations (there is no need to have exactly one IDT run per day). This allows enough data to be accumulated to get self-consistent outputs and perform representative diagnostics. During the first months of the mission, the IDT runs covered shorter time ranges to speed up some of these diagnostics and assess the health of the satellite and on-ground data processing systems.

IDT has its own diagnostics, generating over 9000 plots and statistics every day to determine the data quality and trends, allowing to detect potential inconsistencies or issues before they may become a problem (either on-board or on-ground). All this is accessible through a web-based front-end, but also PDF reports are automatically generated at the end of every run.

Figure 1 shows the main page of the tool, which shows the moment when Run 1000 was closed. One way to identify Gaia measurements is their acquisition time in 'mission revolutions' (spins of Gaia around its rotation axis). The four main plots of Figure 1 illustrate the mission time (vertical axes) of the several data types being received or processed at a given wallclock time (horizontal axes). Here IDT was recovering from a downtime to perform some maintenance, quickly reaching again the 'real-time line' (the dashed diagonal line on each plot). Vertical dashed lines indicate the Runs done. The progress indicators at the top show the numbers of 'Elementaries' (intermediate data outputs, one per measurement), the system status, the fraction of Runs done, the pending processing tasks, or the number of computing threads being used.

   
 

Figure 2: Sky-coverage of a typical IDT run. Click on the image for a high-res version.

 

To illustrate the daily workload and accuracy of the IDT system and Gaia, Figure 2 draws the equatorial sky coordinates (right ascension and declination), as well as the onboard-estimated brightnesses, of all objects detected, measured, downlinked and processed during a typical day, having "only" 43 million objects in this case (some days over a hundred million objects can be reached). The sky coordinates are those determined by IDT in real-time, which are already very accurate (around 100 milliarcseconds, or 1/36000th of a degree).

The original, artificial image has 603 megapixels which not all image viewers support, which is why a downsized version is provided here. However, even the original image size would not be enough to illustrate the resolution in high sky densities, that is, when observing the Galactic Plane. When zooming in, some features can be seen, including the "strips" of several scans of Gaia on the sky (one scanning revolution every 6 hours).

   
  Figure 3: Zoom of Figure 2. Click on the image for a high-res version.  

Figure 3 shows a zoom into the southern region, where some sky features and structures can be seen, as well as some further "strips" corresponding, in this case, to the seven CCD rows (or "cameras") of the Gaia instruments.

   
  Figure 4: Zoom of Figure 3. Click on the image for a high-res version.  

Finally, Figure 4 shows another zoom (with increased dot sizes for better visibility) on a couple of globular clusters, NGC 1831 and NGC 1866, the individual stars of which are detected and measured by Gaia without any problem. Only at this zoom level, individual objects as detected by Gaia can be seen.

Full-resolution images of Figures 2 and 3 can be downloaded here.

Credits: ESA/Gaia/DPAC/UB/IEEC

[Published: 10/06/2016]

 

Image of the Week Archive

2017
24/11: Gaia-GOSA service
27/10: German Gaia stamp in the making
19/10: Hertzsprung-russell diagram using Gaia DR1
05/10: Updated prediction to the Triton occultation campaign
31/08: Close stellar encounters from the first Gaia data release
16/08: Preliminary view of the Gaia sky in colour
07/07: Chariklo stellar occultation follow-up
24/04: Gaia reveals the composition of asteroids
20/04: Extra-galactic observations with Gaia
10/04: How faint are the faintest Gaia stars?
24/03: Pulsating stars to study Galactic structures
09/02: Known exoplanetary transits in Gaia data
31/01: Successful second DPAC Consortium Meeting
2016
23/12: Interactive and statistical visualisation of Gaia DR1 with vaex
16/12: Standard uncertainties for the photometric data (in GDR1)
25/11: Signature of the rotation of the galactic bar uncovered
15/11: Successful first DR1 Workshop
27/10: Microlensing Follow-Up
21/10: Asteroid Occultation
16/09: First DR1 results
14/09: Pluto Stellar Occultation
15/06: Happy Birthday, DPAC!
10/06: 1000th run of the Initial Data Treatment system
04/05: Complementing Gaia observations of the densest sky regions
22/04: A window to Gaia - the focal plane
05/04: Hipparcos interactive data access tool
24/03: Gaia spots a sunspot
29/02: Gaia sees exploding stars next door
11/02: A new heart for the Gaia Object Generator
04/02: Searching for solar siblings with Gaia
28/01: Globular cluster colour-magnitude diagrams
21/01: Gaia resolving power estimated with Pluto and Charon
12/01: 100th First-Look Weekly Report
06/01: Gaia intersects a Perseid meteoroid
2015
18/12: Tales of two clusters retold by Gaia
11/11: Lunar transit temperature plots
06/11: Gaia's sensors scan a lunar transit
03/11: Celebrity comet spotted among Gaia's stars
09/10: The SB2 stars as seen by Gaia's RVS
02/10: The colour of Gaia's eyes
24/09: Estimating distances from parallaxes
18/09: Gaia orbit reconstruction
31/07: Asteroids all around
17/07: Gaia satellite and amateur astronomers spot one in a billion star
03/07: Counting stars with Gaia
01/07: Avionics Model test bench arrives at ESOC
28/05: Short period/faint magnitude Cepheids in the Large Magellanic Cloud
19/05: Visualising Gaia Photometric Science Alerts
09/04: Gaia honours Einstein by observing his cross
02/04: 1 April - First Look Scientists play practical joke
05/03: RR Lyrae stars in the Large Magellanic Cloud as seen by Gaia
26/02: First Gaia BP/RP deblended spectra
19/02: 13 months of GBOT Gaia observations
12/02: Added Value Interface Portal for Gaia
04/02: Gaia's potential for the discovery of circumbinary planets
26/01: DIBs in three hot stars as seen by Gaia's RVS
15/01: The Tycho-Gaia Astrometric Solution
06/01: Close encounters of the stellar kind
2014
12/12: Gaia detects microlensing event
05/12: Cat's Eye Nebula as seen by Gaia
01/12: BFOSC observation of Gaia at L2
24/11: Gaia spectra of six stars
13/11: Omega Centauri as seen by Gaia
02/10: RVS Data Processing
12/09: Gaia discovers first supernova
04/08: Gaia flag arrives at ESAC
29/07: Gaia handover
15/07: Eclipsing binaries
03/07: Asteroids at the "photo finish"
19/06: Calibration image III - Messier 51
05/06: First Gaia BP/RP and RVS spectra
02/06: Sky coverage of Gaia during commissioning
03/04: Gaia source detection
21/02: Sky-background false detections in the sky mapper
14/02: Gaia calibration images II
06/02: Gaia calibration image I
28/01: Gaia telescope light path
17/01: First star shines for Gaia
14/01: Radiation Campaign #4
06/01: Asteroid detection by Gaia
2013
17/12: Gaia in the gantry
12/12: The sky in G magnitude
05/12: Pre-launch release of spectrophotometric standard stars
28/11: From one to one billion pixels
21/11: The Hipparcos all-sky map
15/10: Gaia Sunshield Deployment Test
08/10: Initial Gaia Source List
17/09: CU1 Operations Workshop
11/09: Apsis
26/08: Gaia arrival in French Guiana
20/08: Gaia cartoons
11/07: Model Soyuz Fregat video
01/07: Acoustic Testing
21/06: SOVT
03/06: CU4 meeting #15
04/04: DPCC (CNES) 
26/03: Gaia artist impression 
11/02: Gaia payload testing  
04/01: Space flyby with Gaia-like data
2012
10/12: DPAC OR#2. Testing with Planck
05/11: Galaxy detection with Gaia
09/10: Plot of part of the GUMS-10 catalogue
23/07: "Gaia" meets at Gaia
29/06: The Sky as seen by Gaia
31/05: Panorama of BAM clean room
29/03: GREAT school results
12/03: Scanning-law movie
21/02: Astrometric microlensing and Gaia
03/02: BAM with PMTS
12/01: FPA with all the CCDs and WFSs
2011
14/12: Deployable sunshield
10/11: Earth Trojan search
21/10: First Soyuz liftoff from the French Guiana
20/09: Fast 2D image reconstruction algorithm
05/09: RVS OMA
10/08: 3D distribution of the Gaia catalogue
13/07: Dynamical Attitude Model
22/06: Gaia's view of open clusters
27/05: Accuracy of the stellar transverse velocity
13/05: Vibration test of BAM mirrors
18/04: L. Lindegren, Dr. Honoris Causa of the Observatory of Paris
19/01: Detectability of stars close to Jupiter
05/01: Delivery of the WFS flight models
2010
21/12: The 100th member of CU3
17/11: Nano-JASMINE and AGIS
27/10: Eclipsing binary light curves fitted with DPAC code
13/10: Gaia broad band photometry
28/09: Measuring stellar parameters and interstellar extinction
14/09: M1 mirror
 
Please note: Entries from the period 2003-2010 are available in this PDF document.