Table of Contents Previous Chapter Next Chapter

9.0 ACIS Data Analysis and Database

To:ACIS Team
From:Jonathan W. Woo
Subject:ACIS Data Analysis and Database
Date:4 May 1995

This memo describes the following subjects related to the MIT ACIS lab calibrations: data formats, standard preliminary analysis procedures, utility software, and the database structure of events collected from flight devices.

9.1 Data Format

At present, there are four different data formats used in the initial data reduction processes. (1) Raw data read from ACIS CCDs are first recorded in the FITS files of full frame images. (2,3) From the raw image FITS files, events are then collected and stored in one of two event list formats--Ftool science file (FSF) format and ACIS RV (ARV) format, or in both. (4) Some event lists are also stored in IDL table files.

9.2 Raw Image Format

The raw data read out from ACIS CCDs by current test electronics are initially recorded in 2D FITS image files. Each value of readout pixels is registered as a 16 bits short integer record. In the beginning of each raw FITS file, extended FITS Header keywords can be found.

The following first group of keywords are there to conform to the FITS standard:

SIMPLE =  T  / file does conform to FITS standard
BITPIX =  16  / number of bits per data pixel
NAXIS  =   2  / number of data axes
NAXIS1 =  1120  / length of data axis 1
NAXIS2 =  256  / length of data axis 2 
EXTEND =  T  / FITS dataset may contain extensions

These keywords specify that this FITS file contains a 2D short integer (16 bits) image array (1120x256).

The next six keywords are used to map a subframe array captured/saved from the full frame array (when the entire CCD is read out). The full frame array includes all active image pixels and "expected" overclock pixels (OCs) that should be consistent with OCs in the subframe array.

RASTER1 =  1120  / Original column dimension of entire CCD
RASTER2 =  1030  / Original row dimension of entire CCD
CRPIX1  =  1  / starting column number of sub image array
CRPIX2  =  1  / starting row number of sub image array

CRVAL1  =  1  / starting column number of remapping array
CRVAL2  =  200  / starting row number of remapping array

Thus RASTER1 and RASTER2 define the "expected" dimension of the full frame array. These keywords are useful when a subframe array of a given device is compared with a different (in locationwise) subframe array of the same device. The values of CRPIX1 and CRPX2 is usually 1. CRVAL1 and CRVAL2 should be the coordinate values of the first pixel from the subframe array relative to the full frame array (see Figure 2).

FIGURE 2. Relative coordinate system
of a subframe array with respect to the full frame array.

The following keywords are there for record keeping purposes:

MISSION = 'AXAF    '    / Mission name
TELESCOP= 'MIT_CCD '    / Telescope (facility) name
INSTRUME= 'ACIS    '    / Instrument name
OBS_MODE= 'GROUND  '    / Observation mode
OBS_ID  = 'PTB Whitelight' / Observation purpose
DATAMODE= 'SPECIAL '    / data mode
CREATOR = '/usr/acis/s56dsp/hostvid'  / creator of data
                        / acquisition and control software
DETNAM  = 'w34c3   '    / CCD ID
AEID    = 'acis3   '    / Electronics ID
ORIGIN  = 'MIT     '    / SAO/MIT
SRC_ID  = 'BESSY   '    / Name of X-ray source device
OBJECT  = 'PTB     '    / Name of X-ray source anode
OBSERVER= 'sek'    / User ID of calibration operator
HOSTID  = '0xeffff284'    / host machine CPU ID
HOSTNAME= 'otto'    / Host machine name
DATE    = '27/04/95'    / FITS file creation date (dd/mm/yy)

These keywords are self-explanatory.

The next set of keywords specify the locations of the active readout image pixels in the readout array:

IAMINCOL=  5  / Minimum column number of Output Node A
IAMAXCOL=  260  / Maximum column number of Output Node A
IAMINROW=  1  / Minimum row number of Output Node A
IAMAXROW=  256  / Maximum row number of Output Node A
IBMINCOL=  285  / Minimum column number of Output Node B
IBMAXCOL=  540  / Maximum column number of Output Node B
IBMINROW=  1  / Minimum row number of Output Node B
IBMAXROW=  256  / Maximum row number of Output Node B
ICMINCOL=  565  / Minimum column number of Output Node C
ICMAXCOL=  820  / Maximum column number of Output Node C
ICMINROW=  1  / Minimum row number of Output Node C
ICMAXROW=  256  / Maximum row number of Output Node C
IDMINCOL=  845  / Minimum column number of Output Node D
IDMAXCOL=  1100  / Maximum column number of Output Node D
IDMINROW=  1  / Minimum row number of Output Node D
IDMAXROW=  256  / Maximum row number of Output Node D

These are the row and column boundaries of the four read out nodes which contains active image pixels which exclude overclock pixels, underclock pixels, and undefined parallel transfer clock pixels from the readout array.

The frame acquisition time (actually for the previous frame) and a rough exposure time (difference between two UNIX time calls at the two consecutive data synchronization marks) are recorded in the following keywords:

DATE-OBS=  '27/04/95'  / Date(dd/mm/yy) of exposure start (EDT)
TIME-OBS=  '05:08:39'  / Time(hh:mm:ss) of exposure start (EDT)
ONTIME  =  1.528E+00  / Exposure time (sec between two sync)

The following keywords are initially set by data acquisition programs:

FILENAME= 'i24ey001.1170.fits'
FILEPATH= '/otto/d3/w34c3/27Apr0357/'
OVERFILL=  24704  / over fill
VERBOSE =  1  / verbose (0=off;1=on)
TRIGGER =  F0F0  /
TRCOUNT =  16  /

Voltage, current, and temperature readings of various clocks and corresponding ADU values are recorded in the following keywords:

V_IA_H  =  +4.490  / Voltage of CCD Image Clock High (V)
A_IA_H  =  44262  / ADU of CCD Image Clock High (ADU)
V_IA_L  =  -5.033  / Voltage of CCD Image Clock Low (V)
A_IA_L  =  19884  / ADU of CCD Image Clock Low (ADU)
V_FS_H  =  +6.956  / Voltage of CCD Frame Storage Clock High (V)
A_FS_H  =  50576  / ADU of CCD Frame Storage Clock High (ADU)
V_FS_L  =  -1.888  / Voltage of CCD Frame Storage Clock Low (V)
A_FS_L  =  27935  / ADU of CCD Frame Storage Clock Low (ADU)
V_OR_H  =  0.0  / Voltage of CCD Output Register Clock High (V)

A_OR_H  =  0  / ADU of CCD Output Register Clock High (ADU)
V_OR_L  =  0.0  / Voltage of CCD Output Register Clock Low (V)
A_OR_L  =  0  / ADU of CCD Output Register Clock Low (ADU)
V_RG_H  =  +7.522  / Voltage of CCD Reset Clock High (V)
A_RG_H  =  52025  / ADU of CCD Reset Clock High (ADU)
V_RG_LP =  +4.018  / Voltage of CCD Reset Clock Low+ (V)
A_RG_LP =  43054  / ADU of CCD Reset Clock Low+ (ADU)
V_SCP   =  +7.027  / Voltage of CCD Scupper (V)
A_SCP   =  50758  / ADU of CCD Scupper (ADU)
V_OG_P  =  +1.020  / Voltage of CCD Output Gate+ (V)
A_OG_P  =  35378  / ADU of CCD Output Gate+ (ADU)
V_RDA   =  +7.021  / Voltage of CCD Reset Drain (V)
A_RDA   =  50742  / ADU of CCD Reset Drain (ADU)
V_HTR_C =  +0.014  / Amp of Heater Current
A_HTR_C =  32803  / ADU of Heater Current
V_HTR_V =  +0.014  / Heater Voltage (V)
A_HTR_V =  32804  / ADU of Heater Voltage
V_RG_LM =  0.0  / Voltage of CCD Reset Clock Low - (V)
A_RG_LM =  0  / ADU of CCD Reset Clock Low - (ADU)
V_VDD_A =  +19.170  / Voltage of CCD Vdd_A (V)
A_VDD_A =  49126  / ADU of CCD Vdd_A (ADU)
V_VDD_B =  +19.173  / Voltage of CCD Vdd_B (V)
A_VDD_B =  49129  / ADU of CCD Vdd_B (ADU)
V_VDD_C =  +19.105  / Voltage of CCD Vdd_C (V)
A_VDD_C =  49071  / ADU of CCD Vdd_C (ADU)
V_VDD_D =  +19.218  / Voltage of CCD Vdd_D (V)
A_VDD_D =  49167  / ADU of CCD Vdd_D (ADU)
V_BJ    =  0.0  / Voltage of CCD Back Junction (V)
A_BJ    =  0  / ADU of CCD Back Junction (ADU)
V_EB_A  =  +5800.000  / Voltage of node A Electronics Bias (V)
A_EB_A  =  5800  / ADU of node A Electronics Bias (ADU)
V_EB_B  =  +5500.000  / Voltage of node B Electronics Bias (V)
A_EB_B  =  5500  / ADU of node B Electronics Bias (ADU)
V_EB_C  =  +5700.000  / Voltage of node C Electronics Bias (V)
A_EB_C  =  5700  / ADU of node C Electronics Bias (ADU)
V_EB_D  =  +5900.000  / Voltage of node D Electronics Bias (V)
A_EB_D  =  5900  / ADU of node D Electronics Bias (ADU)
CAM_TEMP=  +0.013  / Camera Temperature (C)
CCD_TEMP=  -242.00  / CCD Temperature (C)

The DE binary bit settings for the "gse" data acquisition program are recorded in the following keywords:

DE_MODE0=  0  / CHKCR: MODE0 bit (0=off;1=on)
DE_MODE1=  0  / CHKCR: MODE1 bit (0=off;1=on)
DE_HSEQ =  1  / CHKCR: HSEQ bit(0=off;1=on) the hard sequencer
DE_TSAP =  0  / CHKCR/TESTAP asyn loopback test (global hk pipe)
DE_BOARD=  1  / CHKCR: BOARD bit (0=off;1=on)

DE_RASTR=  1  / CHKCR: RASTER bit (0=off;1=on)
DE_RIGHT=  0  / VHKCR: RIGHT bit (0=off;1=on)
DE_SHFT0=  0  / VHKCR: SHIFT0 bit (0=off;1=on)
DE_SHFT1=  0  / VHKCR: SHIFT1 bit (0=off;1=on)
DE_SHFT2=  0  / VHKCR: SHIFT2 bit (0=off;1=on)
DE_EN12 =  0  / VCRX: EN12 bit (0=off;1=on)
DE_TSDVP=  0  / VCRX: TESTDVP bit (0=off;1=on)
DE_HEATR=  0  / DHKCR: Turns the heater (0=off;1=on)
DE_SWAP =  0  / DHKCR: Swaps output shift reg phs 2 with 3

9.2.1 FSF Format

In the standard ACIS analysis procedure, an event (3x3 pixel island whose center pixel value is the local maximum above event threshold) detected by the event finding algorithm (adopted from rv_findevents) is stored in the FSF event files. The FSF event files are in a binary FITS table extension format similar to the ASCA science event file format. Each FSF file has at least the following three components: a dummy primary image extension with zero dimension, a binary table extension containing events, and a binary table extension of good time intervals (GTIs). Since most of the FITS Header keywords in FSF are adopted from ASCA convention, a detailed description of the keywords is not presented in this memo.

In the binary table extension, each event is defined by 10 fields:


The TIME field is the absolute time (seconds after 1994.0 in UNIX time) of the frame exposure when the event was detected. The center pixel position of a detected event (above a specified event threshold) is recorded in readout, physical CCD, and lab reference coordinate systems, respectively as READX & READY, CHIPX & CHIPY, and LABX & LABY columns. The FRAME field contains the integer numbers of the frames where the detected events extracted. The frame numbers are counted from the TSTART time when the first data frame is exposed.

Therefore, the frame number has only a relative meaning for a single FSF data set. In a mathematical expression, nframe = (tTSTART)/exposure. The PHAS field contains 9 short integer pulse height values of a 3x3 pixel island (see Figure 3). Finally, the CCDNODE field specifies the read-out node.

FIGURE 3. Sequence of pixel values from
3x3 pixel island stored in the event record of an FSF

9.2.2 ARV Format

In order to use the available software developed for the previous laboratory X-ray CCD calibration efforts, all events recorded in FSFs are converted to a binary record format called "ARV" (ACIS RV) which is a modified version of "RV" format. The ARV format is a binary format of individual events identified as a 3x3 pixel island distribution. Each data structure is a 32 Byte binary record consisting of frame number, exposure time (in msec), acistime (sec from 1994.0), nodenum, x & y position of the center pixel, and the 9 pixel values of the 3x3 island in the order shown in Figure 4. While acistime is 4 Byte long integer, the rest of the variables are all in 2 Byte short integers.

struct ds_data_str
    short nframe;
    short exposure;
    long acistime;
    short nodenum,x,y,data[9];

FIGURE 4. Sequence of pixel values
from 3x3 pixel island stored in ARV format.

9.2.3 IDL Format

Some events are recorded in IDL binary (32 Byte record) table files similar to ARV format (ask sjones about details).

9.3 Analysis Procedure

For a typical run of data acquisition, a couple hundred consecutive raw image frames and several (~10) bias frames are taken. ACISANAL1 (a tcsh script) is first run to collect events after correcting for the bias level for each pixel. An improved data acquisition time of each event is recalculated using the refined exposure time which is a constant value throughout a given data set. Once an event list is extracted by ACISANAL1, ACISANAL2 (a tcsh script) is run to generate a number density image of events, pulse-height-distributions (PHDs) of two grade combinations (g0 and g0234), light curves, pulse-height histograms of each grade, and pulse-height histograms of neighboring pixels of the center pixels in the 3x3 event islands.


This script first produces a mean clip bias frame from all the available bias frames in a data set, by executing the meanbiasclip program:

ls ${bias_heading}.????.fits | meanbiasclip ${bias}

The mean bias frame prepared in this step will be used in the acis_mit_findevents routine to estimate the bias level for each pixel.

After a FITS file containing the mean bias frame is created, the script then prepares to run the main routine, acis_mit_findevents to extract events. As a first step for the preparation, the script executes the following line to collect all data file names into a temporary file:

ls ${data_heading}.????.fits > tmp_data.list

And then, the script calculates the exposure time of the data set (which is a constant value for a single data set). It uses the first 20 files to estimate a rough exposure time by executing the get_exp program which makes a simple average of values stored in the FITS header keyword ONTIME from the first 20 raw image frames. The script calls get_exp again to make a fine adjustment of the exposure time by averaging all ONTIME values which are close to the initial rough exposure time within 50% level. With these two iterations, the calculated exposure time should be good to 1/1000 s.

The following two lines in the ACISANAL1 script are executed to calculate the exposure time:

set init_exp = `get_exp -1.0 20 < tmp_data.list | lines 1-1`
set exp = `get_exp ${init_exp} 0 < tmp_data.list | lines 1-1`

Once an accurate exposure time and an average bias frame are obtained, the ACISANAL1 script runs the acis_mit_findevents program to extract events. acis_mit_findevents first reads in the mean bias frame and subtracts it off from each data image frame before extracting events. For each quadrant, it adds back the mean value of the bias frame subtracted image over-clock region to each image pixel for a fine temporal bias level adjustment. Then, the program goes through each pixel of the bias subtracted image array to find events where each event's central pixel value is above the low event threshold and is also the local maximum in the 3x3 island. Events are then recorded in a FSF event binary table FITS file. Each event record is identified with the UNIX time (absolute time is good to within a sec and relative time among events in a single data set is good to within ~1 ms), the position (readout and device coordinates) of the center pixel, quadrant id, and the 9 pixel values of the 3x3 island. This step is executed by:

acis_mit_findevents -el ${low_ev_th} -i stdin -bs ${bias} -o
${acis_evts_fits} -h ${acis_hk_fits} -exp ${exp} < tmp_data.list

To do further analysis using available RV tools, the FSF event list is converted into ARV format by running the acis_fits_to_rv program.

acis_fits_to_rv ${acis_evts_fits} stdout > ${acis_evts}

Since it is convenient to calculate readout noise at this stage, the acis_readoutnoise tcsh script is called to calculate readout noise values for 4 image and overclock regions using the first two bias files.

acis_readoutnoise ${bias_heading}.0001.fits ${bias_heading}.0002.fits


ACISANAL2 is a tcsh script which creates secondary products from an ARV event file: pulse-height distributions (histograms and XSPEC input file--PHA), event number density map, light curves, and event corner pixel pulse-height distributions. A number density map is created as a 2D FITS image file, PHA files are in the XSPEC binary format, and the rest of the products are in QDP file formats.

The ACISANAL2 script first creates a number density map of detected events whose pixel grade distributions are either singles or splits (i.e., grades 0, 2, 3, or 4).

rv_gflt ${ev_th} ${sp_th} -p 0 4095 -g 0 2 3 4 < ${acis_evts} | rv_ef
${x_img} ${y_img} > ${heading}_sp${sp_th}_g0234_img.fits

For each of the four read-out nodes, two pulse-height histograms (g0 & g0234) are created. In the script, the following two lines in the foreach loop are executed:

rv_streamfilt -c ${ro_node} < ${acis_evts} | rv_gflt ${ev_th}
${sp_th} -p 0 4095 -g 0 | grad_ph ${sp_th} | awk '{print $4}' | histo
1 0.5 4095.5 4096 > ${heading}_c${ro_node}_sp${sp_th}_g0.qdp
rv_streamfilt -c ${ro_node} < ${acis_evts} | rv_gflt ${ev_th}
${sp_th} -p 0 4095 -g 0 2 3 4 | grad_ph ${sp_th} | awk '{print $4}' |
histo 1 0.5 4095.5 4096 > ${heading}_c${ro_node}_sp${sp_th}_g0234.qdp

Next, the script calculates the variation in count rate (counts/frame) for each read-out node as a function of frame number counted from the first raw image frame of a given set (~200 frames).

rv_streamfilt -c ${ro_node} < ${acis_evts} | rv_gflt ${ev_th}
${sp_th} -p 0 4095 -g 0 2 3 4 | grad_ph ${sp_th} | awk '{print $7}' |
lightcv > ${heading}_c${ro_node}_sp${sp_th}_g0234_lightcv.qdp

And, pulse-height histograms of each grade (0 to 7) are produced in QDP by the following tcsh command:

rv_streamfilt -c ${ro_node} < ${acis_evts} | rv_gflt ${ev_th}
${sp_th} -p 0 4095 | rv_ev2pcf ${ev_th} ${sp_th} /dev/null >

Neighbor histograms (corner pixel distributions and top, bottom, left, and right pixel distributions of detected events) are produced by the following command:

rv_streamfilt -c ${ro_node} < ${acis_evts} | neighborhist >

In QDP files, as a product of this step, "readout fraction" (total pixel values divided by the total number of pixels for each bin) is plotted against ADU.

To use XSPEC for the spectral analysis (with convolution of response matrix), two PHA files are produced (g0 & g0234):

rv_streamfilt -c ${ro_node} < ${acis_evts} | rv_gflt ${ev_th}
${sp_th} -p 0 4095 -g 0 2 3 4 | rv_ev2pha ${ev_th} ${sp_th} 1 >
rv_streamfilt -c ${ro_node} < ${acis_evts} | rv_gflt ${ev_th}
${sp_th} -p 0 4095 -g 0 | rv_ev2pha ${ev_th} ${sp_th} 1 >

9.3.3 Data Products

A summary table and sample plots of products generated by ACISANAL2 are presented in this section. A sample plot of light curve is shown in Figure 5, a sample plot of PH-histogram is in Figure 6, a sample plot of primary calibration file is in Figure 7, and a sample plot of read-out noise distribution is in Figure 8.

TABLE 25. Summary of products from ACISANAL2
Type Sample_filename Format Purpose
Evts Num Density test_sp20_g0234_img.fits FITS Event Flux
PH histogram test_c0_sp20_g0.qdp QDP Spectral Resolution
PH histogram test_c0_sp20_g0234.qdp QDP Spectral Resolution
PH spectrum test_c0_sp20_g0.pha XSPEC Spectral response/QE meas.
PH spectrum test_c0_sp20_g0234.pha XSPEC Spectral response/QE meas.
Primary Cal File test_c0_pcf.qdp QDP Grade branching ratio meas.
Light Curve test_c0_sp20_g0234_lightcv.qdp QDP Check source/detector stability
Noise Summary File readoutnoise.out ASCII Read-out Noise σ (ADU)
Noise Model readoutnoise00im.model ASCII 1 Gaussian fit
Noise spectrum readoutnoise00im.qdp QDP Read-out Noise meas.

FIGURE 5. A sample light curve

FIGURE 6. A sample PH histogram/spectrum.

FIGURE 7. A sample primary calibration file

FIGURE 8. A sample readout noise file

9.4 Utility Software

There are a few utility modules in the "/usr/acis/bin" directory. ana2gau calculates the FWHM value of 2-Gaussian model determined by a fit to the PH histogram. The ARV event list can be displayed in ascii format with acisrvtoascii.

9.5 Database

The ACIS flight device calibration database will be constructed as subdirectory trees. On the top level, there will be one directory for each device and the directory name will be named after the device id (e.g., c17-103-1). For a given device directory, there will be subdirectories to identify lab experiment setups (e.g., IFM, HIREFS, HE-A, HE-B, SES, BESSY). As a next lower level subtree, the date of a particular data acquisition will be used (e.g., 01may95). A FSF event FITS file, a house keeping FITS file, an ARV file, two bias FITS files, and the averaged bias FITS file will be stored in the next lower level which specifies the source energy range (e.g., carbon_k, oxygen_k). Any ASCII note regarding to the particular data set will be placed at the same directory.

As an example,

p4 jww@ohno> pard
p4 jww@ohno> ls

The secondary products from ACISANAL2 will be also stored in a subdirectory call "products":

p4 jww@ohno> pwd
p4 jww@ohno> ls
i05ey495_c0_pcf.qdp  i05ey495_neighborhist_c0.qdp
i05ey495_c0_sp20_g0.qdp  i05ey495_neighborhist_c1.qdp
i05ey495_c0_sp20_g0234.qdp  i05ey495_neighborhist_c2.qdp
i05ey495_c0_sp20_g0234_lightcv.qdp  i05ey495_neighborhist_c3.qdp
i05ey495_c0_sp20_g0.pha  i05ey495_sp20_g0234_img.fits
i05ey495_c0_sp20_g0234.pha  readoutnoise.mesg
i05ey495_c1_pcf.qdp  readoutnoise.out
i05ey495_c1_sp20_g0.qdp  readoutnoise00im.model
i05ey495_c1_sp20_g0234.qdp  readoutnoise00im.qdp

i05ey495_c1_sp20_g0234_lightcv.qdp  readoutnoise00oc.model
i05ey495_c1_sp20_g0.pha  readoutnoise00oc.qdp
i05ey495_c1_sp20_g0234.pha  readoutnoise01im.model
i05ey495_c2_pcf.qdp  readoutnoise01im.qdp
i05ey495_c2_sp20_g0.qdp  readoutnoise01oc.model
i05ey495_c2_sp20_g0234.qdp  readoutnoise01oc.qdp
i05ey495_c2_sp20_g0234_lightcv.qdp  readoutnoise02im.model
i05ey495_c2_sp20_g0.pha  readoutnoise02im.qdp
i05ey495_c2_sp20_g0234.pha  readoutnoise02oc.model
i05ey495_c3_pcf.qdp  readoutnoise02oc.qdp
i05ey495_c3_sp20_g0.qdp  readoutnoise03im.model
i05ey495_c3_sp20_g0234.qdp  readoutnoise03im.qdp
i05ey495_c3_sp20_g0234_lightcv.qdp  readoutnoise03oc.model
i05ey495_c3_sp20_g0.pha  readoutnoise03oc.qdp

A log summary file of these directory trees, called "data.log" can be found in the /usr/acis/database/ directory. This file can be used as a guide to locate events and products. The content of this file should be:

CDID          EXP    DATE(HH:MM)     SRCID      DIR    USER  MOD-DATE
c17-103-1    BESSY  09mar95(19:57)  i05ey495     *     jww   05/02/95

* = /ohno/d1/c17-103-1/bessy/09mar95/t1957_i05ey495

Table of Contents Previous Chapter Next Chapter
Peter G. Ford
Last modified: Mon Nov 25 15:29:51 EST