FIGURE 2. Relative coordinate system
of a subframe array with respect to the full frame array.
Table of Contents
Previous Chapter
Next Chapter
| To: | ACIS Team |
| From: | Jonathan W. Woo |
| Subject: | ACIS Data Analysis and Database |
| Date: | 4 May 1995 |
This memo describes the following subjects related to the MIT ACIS lab calibrations: data formats, standard preliminary analysis procedures, utility software, and the database structure of events collected from flight devices.
At present, there are four different data formats used in the initial data reduction processes. (1) Raw data read from ACIS CCDs are first recorded in the FITS files of full frame images. (2,3) From the raw image FITS files, events are then collected and stored in one of two event list formats--Ftool science file (FSF) format and ACIS RV (ARV) format, or in both. (4) Some event lists are also stored in IDL table files.
The raw data read out from ACIS CCDs by current test electronics are initially recorded in 2D FITS image files. Each value of readout pixels is registered as a 16 bits short integer record. In the beginning of each raw FITS file, extended FITS Header keywords can be found.
The following first group of keywords are there to conform to the FITS standard:
SIMPLE = T / file does conform to FITS standard BITPIX = 16 / number of bits per data pixel NAXIS = 2 / number of data axes NAXIS1 = 1120 / length of data axis 1 NAXIS2 = 256 / length of data axis 2 EXTEND = T / FITS dataset may contain extensions
These keywords specify that this FITS file contains a 2D short integer (16 bits) image array (1120x256).
The next six keywords are used to map a subframe array captured/saved from the full frame array (when the entire CCD is read out). The full frame array includes all active image pixels and "expected" overclock pixels (OCs) that should be consistent with OCs in the subframe array.
RASTER1 = 1120 / Original column dimension of entire CCD RASTER2 = 1030 / Original row dimension of entire CCD CRPIX1 = 1 / starting column number of sub image array CRPIX2 = 1 / starting row number of sub image array CRVAL1 = 1 / starting column number of remapping array CRVAL2 = 200 / starting row number of remapping array
Thus RASTER1 and RASTER2 define the "expected" dimension of the full frame array. These keywords are useful when a subframe array of a given device is compared with a different (in locationwise) subframe array of the same device. The values of CRPIX1 and CRPX2 is usually 1. CRVAL1 and CRVAL2 should be the coordinate values of the first pixel from the subframe array relative to the full frame array (see Figure 2).
FIGURE 2. Relative coordinate system
of a subframe array with respect to the full frame array.
The following keywords are there for record keeping purposes:
MISSION = 'AXAF ' / Mission name
TELESCOP= 'MIT_CCD ' / Telescope (facility) name
INSTRUME= 'ACIS ' / Instrument name
OBS_MODE= 'GROUND ' / Observation mode
OBS_ID = 'PTB Whitelight' / Observation purpose
DATAMODE= 'SPECIAL ' / data mode
CREATOR = '/usr/acis/s56dsp/hostvid' / creator of data
/ acquisition and control software
DETNAM = 'w34c3 ' / CCD ID
AEID = 'acis3 ' / Electronics ID
ORIGIN = 'MIT ' / SAO/MIT
SRC_ID = 'BESSY ' / Name of X-ray source device
OBJECT = 'PTB ' / Name of X-ray source anode
OBSERVER= 'sek' / User ID of calibration operator
HOSTID = '0xeffff284' / host machine CPU ID
HOSTNAME= 'otto' / Host machine name
DATE = '27/04/95' / FITS file creation date (dd/mm/yy)
These keywords are self-explanatory.
The next set of keywords specify the locations of the active readout image pixels in the readout array:
IAMINCOL= 5 / Minimum column number of Output Node A IAMAXCOL= 260 / Maximum column number of Output Node A IAMINROW= 1 / Minimum row number of Output Node A IAMAXROW= 256 / Maximum row number of Output Node A IBMINCOL= 285 / Minimum column number of Output Node B IBMAXCOL= 540 / Maximum column number of Output Node B IBMINROW= 1 / Minimum row number of Output Node B IBMAXROW= 256 / Maximum row number of Output Node B ICMINCOL= 565 / Minimum column number of Output Node C ICMAXCOL= 820 / Maximum column number of Output Node C ICMINROW= 1 / Minimum row number of Output Node C ICMAXROW= 256 / Maximum row number of Output Node C IDMINCOL= 845 / Minimum column number of Output Node D IDMAXCOL= 1100 / Maximum column number of Output Node D IDMINROW= 1 / Minimum row number of Output Node D IDMAXROW= 256 / Maximum row number of Output Node D
These are the row and column boundaries of the four read out nodes which contains active image pixels which exclude overclock pixels, underclock pixels, and undefined parallel transfer clock pixels from the readout array.
The frame acquisition time (actually for the previous frame) and a rough exposure time (difference between two UNIX time calls at the two consecutive data synchronization marks) are recorded in the following keywords:
DATE-OBS= '27/04/95' / Date(dd/mm/yy) of exposure start (EDT) TIME-OBS= '05:08:39' / Time(hh:mm:ss) of exposure start (EDT) ONTIME = 1.528E+00 / Exposure time (sec between two sync)
The following keywords are initially set by data acquisition programs:
FILENAME= 'i24ey001.1170.fits' FILEPATH= '/otto/d3/w34c3/27Apr0357/' OVERFILL= 24704 / over fill VERBOSE = 1 / verbose (0=off;1=on) TRIGGER = F0F0 / TRCOUNT = 16 /
Voltage, current, and temperature readings of various clocks and corresponding ADU values are recorded in the following keywords:
V_IA_H = +4.490 / Voltage of CCD Image Clock High (V) A_IA_H = 44262 / ADU of CCD Image Clock High (ADU) V_IA_L = -5.033 / Voltage of CCD Image Clock Low (V) A_IA_L = 19884 / ADU of CCD Image Clock Low (ADU) V_FS_H = +6.956 / Voltage of CCD Frame Storage Clock High (V) A_FS_H = 50576 / ADU of CCD Frame Storage Clock High (ADU) V_FS_L = -1.888 / Voltage of CCD Frame Storage Clock Low (V) A_FS_L = 27935 / ADU of CCD Frame Storage Clock Low (ADU) V_OR_H = 0.0 / Voltage of CCD Output Register Clock High (V) A_OR_H = 0 / ADU of CCD Output Register Clock High (ADU) V_OR_L = 0.0 / Voltage of CCD Output Register Clock Low (V) A_OR_L = 0 / ADU of CCD Output Register Clock Low (ADU) V_RG_H = +7.522 / Voltage of CCD Reset Clock High (V) A_RG_H = 52025 / ADU of CCD Reset Clock High (ADU) V_RG_LP = +4.018 / Voltage of CCD Reset Clock Low+ (V) A_RG_LP = 43054 / ADU of CCD Reset Clock Low+ (ADU) V_SCP = +7.027 / Voltage of CCD Scupper (V) A_SCP = 50758 / ADU of CCD Scupper (ADU) V_OG_P = +1.020 / Voltage of CCD Output Gate+ (V) A_OG_P = 35378 / ADU of CCD Output Gate+ (ADU) V_RDA = +7.021 / Voltage of CCD Reset Drain (V) A_RDA = 50742 / ADU of CCD Reset Drain (ADU) V_HTR_C = +0.014 / Amp of Heater Current A_HTR_C = 32803 / ADU of Heater Current V_HTR_V = +0.014 / Heater Voltage (V) A_HTR_V = 32804 / ADU of Heater Voltage V_RG_LM = 0.0 / Voltage of CCD Reset Clock Low - (V) A_RG_LM = 0 / ADU of CCD Reset Clock Low - (ADU) V_VDD_A = +19.170 / Voltage of CCD Vdd_A (V) A_VDD_A = 49126 / ADU of CCD Vdd_A (ADU) V_VDD_B = +19.173 / Voltage of CCD Vdd_B (V) A_VDD_B = 49129 / ADU of CCD Vdd_B (ADU) V_VDD_C = +19.105 / Voltage of CCD Vdd_C (V) A_VDD_C = 49071 / ADU of CCD Vdd_C (ADU) V_VDD_D = +19.218 / Voltage of CCD Vdd_D (V) A_VDD_D = 49167 / ADU of CCD Vdd_D (ADU) V_BJ = 0.0 / Voltage of CCD Back Junction (V) A_BJ = 0 / ADU of CCD Back Junction (ADU) V_EB_A = +5800.000 / Voltage of node A Electronics Bias (V) A_EB_A = 5800 / ADU of node A Electronics Bias (ADU) V_EB_B = +5500.000 / Voltage of node B Electronics Bias (V) A_EB_B = 5500 / ADU of node B Electronics Bias (ADU) V_EB_C = +5700.000 / Voltage of node C Electronics Bias (V) A_EB_C = 5700 / ADU of node C Electronics Bias (ADU) V_EB_D = +5900.000 / Voltage of node D Electronics Bias (V) A_EB_D = 5900 / ADU of node D Electronics Bias (ADU) CAM_TEMP= +0.013 / Camera Temperature (C) CCD_TEMP= -242.00 / CCD Temperature (C)
The DE binary bit settings for the "gse" data acquisition program are recorded in the following keywords:
DE_MODE0= 0 / CHKCR: MODE0 bit (0=off;1=on) DE_MODE1= 0 / CHKCR: MODE1 bit (0=off;1=on) DE_HSEQ = 1 / CHKCR: HSEQ bit(0=off;1=on) the hard sequencer DE_TSAP = 0 / CHKCR/TESTAP asyn loopback test (global hk pipe) DE_BOARD= 1 / CHKCR: BOARD bit (0=off;1=on) DE_RASTR= 1 / CHKCR: RASTER bit (0=off;1=on) DE_RIGHT= 0 / VHKCR: RIGHT bit (0=off;1=on) DE_SHFT0= 0 / VHKCR: SHIFT0 bit (0=off;1=on) DE_SHFT1= 0 / VHKCR: SHIFT1 bit (0=off;1=on) DE_SHFT2= 0 / VHKCR: SHIFT2 bit (0=off;1=on) DE_EN12 = 0 / VCRX: EN12 bit (0=off;1=on) DE_TSDVP= 0 / VCRX: TESTDVP bit (0=off;1=on) DE_HEATR= 0 / DHKCR: Turns the heater (0=off;1=on) DE_SWAP = 0 / DHKCR: Swaps output shift reg phs 2 with 3
In the standard ACIS analysis procedure, an event (3x3 pixel island whose center pixel value is the local maximum above event threshold) detected by the event finding algorithm (adopted from rv_findevents) is stored in the FSF event files. The FSF event files are in a binary FITS table extension format similar to the ASCA science event file format. Each FSF file has at least the following three components: a dummy primary image extension with zero dimension, a binary table extension containing events, and a binary table extension of good time intervals (GTIs). Since most of the FITS Header keywords in FSF are adopted from ASCA convention, a detailed description of the keywords is not presented in this memo.
In the binary table extension, each event is defined by 10 fields:
TIME, READX, READY, CHIPX, CHIPY, LABX, LABY, FRAME, PHAS, CCDNODE.
The TIME field is the absolute time (seconds after 1994.0 in UNIX time) of the frame exposure when the event was detected. The center pixel position of a detected event (above a specified event threshold) is recorded in readout, physical CCD, and lab reference coordinate systems, respectively as READX & READY, CHIPX & CHIPY, and LABX & LABY columns. The FRAME field contains the integer numbers of the frames where the detected events extracted. The frame numbers are counted from the TSTART time when the first data frame is exposed.
Therefore, the frame number has only a relative meaning for a single
FSF data set. In a mathematical expression,
nframe = (tTSTART)/exposure.
The PHAS field contains 9 short integer pulse
height values of a 3x3 pixel island
(see Figure 3).
Finally, the CCDNODE field specifies the read-out node.
FIGURE 3. Sequence of pixel values from 3x3 pixel island stored in the event record of an FSF
In order to use the available software developed for the previous laboratory X-ray CCD calibration efforts, all events recorded in FSFs are converted to a binary record format called "ARV" (ACIS RV) which is a modified version of "RV" format. The ARV format is a binary format of individual events identified as a 3x3 pixel island distribution. Each data structure is a 32 Byte binary record consisting of frame number, exposure time (in msec), acistime (sec from 1994.0), nodenum, x & y position of the center pixel, and the 9 pixel values of the 3x3 island in the order shown in Figure 4. While acistime is 4 Byte long integer, the rest of the variables are all in 2 Byte short integers.
struct ds_data_str
{
short nframe;
short exposure;
long acistime;
short nodenum,x,y,data[9];
};
FIGURE 4. Sequence of pixel values
from 3x3 pixel island stored in ARV format.
Some events are recorded in IDL binary (32 Byte record) table files similar to ARV format (ask sjones about details).
For a typical run of data acquisition, a couple hundred consecutive raw image frames and several (~10) bias frames are taken. ACISANAL1 (a tcsh script) is first run to collect events after correcting for the bias level for each pixel. An improved data acquisition time of each event is recalculated using the refined exposure time which is a constant value throughout a given data set. Once an event list is extracted by ACISANAL1, ACISANAL2 (a tcsh script) is run to generate a number density image of events, pulse-height-distributions (PHDs) of two grade combinations (g0 and g0234), light curves, pulse-height histograms of each grade, and pulse-height histograms of neighboring pixels of the center pixels in the 3x3 event islands.
This script first produces a mean clip bias frame from all the available bias frames in a data set, by executing the meanbiasclip program:
ls ${bias_heading}.????.fits | meanbiasclip ${bias}
The mean bias frame prepared in this step will be used in the acis_mit_findevents routine to estimate the bias level for each pixel.
After a FITS file containing the mean bias frame is created, the script then prepares to run the main routine, acis_mit_findevents to extract events. As a first step for the preparation, the script executes the following line to collect all data file names into a temporary file:
ls ${data_heading}.????.fits > tmp_data.list
And then, the script calculates the exposure time of the data set (which is a constant value for a single data set). It uses the first 20 files to estimate a rough exposure time by executing the get_exp program which makes a simple average of values stored in the FITS header keyword ONTIME from the first 20 raw image frames. The script calls get_exp again to make a fine adjustment of the exposure time by averaging all ONTIME values which are close to the initial rough exposure time within 50% level. With these two iterations, the calculated exposure time should be good to 1/1000 s.
The following two lines in the ACISANAL1 script are executed to calculate the exposure time:
set init_exp = `get_exp -1.0 20 < tmp_data.list | lines 1-1`
set exp = `get_exp ${init_exp} 0 < tmp_data.list | lines 1-1`
Once an accurate exposure time and an average bias frame are obtained, the ACISANAL1 script runs the acis_mit_findevents program to extract events. acis_mit_findevents first reads in the mean bias frame and subtracts it off from each data image frame before extracting events. For each quadrant, it adds back the mean value of the bias frame subtracted image over-clock region to each image pixel for a fine temporal bias level adjustment. Then, the program goes through each pixel of the bias subtracted image array to find events where each event's central pixel value is above the low event threshold and is also the local maximum in the 3x3 island. Events are then recorded in a FSF event binary table FITS file. Each event record is identified with the UNIX time (absolute time is good to within a sec and relative time among events in a single data set is good to within ~1 ms), the position (readout and device coordinates) of the center pixel, quadrant id, and the 9 pixel values of the 3x3 island. This step is executed by:
acis_mit_findevents -el ${low_ev_th} -i stdin -bs ${bias} -o
${acis_evts_fits} -h ${acis_hk_fits} -exp ${exp} < tmp_data.list
To do further analysis using available RV tools, the FSF event list is converted into ARV format by running the acis_fits_to_rv program.
acis_fits_to_rv ${acis_evts_fits} stdout > ${acis_evts}
Since it is convenient to calculate readout noise at this stage, the acis_readoutnoise tcsh script is called to calculate readout noise values for 4 image and overclock regions using the first two bias files.
acis_readoutnoise ${bias_heading}.0001.fits ${bias_heading}.0002.fits
ACISANAL2 is a tcsh script which creates secondary products from an ARV event file: pulse-height distributions (histograms and XSPEC input file--PHA), event number density map, light curves, and event corner pixel pulse-height distributions. A number density map is created as a 2D FITS image file, PHA files are in the XSPEC binary format, and the rest of the products are in QDP file formats.
The ACISANAL2 script first creates a number density map of detected events whose pixel grade distributions are either singles or splits (i.e., grades 0, 2, 3, or 4).
rv_gflt ${ev_th} ${sp_th} -p 0 4095 -g 0 2 3 4 < ${acis_evts} | rv_ef
${x_img} ${y_img} > ${heading}_sp${sp_th}_g0234_img.fits
For each of the four read-out nodes, two pulse-height histograms (g0 & g0234) are created. In the script, the following two lines in the foreach loop are executed:
rv_streamfilt -c ${ro_node} < ${acis_evts} | rv_gflt ${ev_th}
${sp_th} -p 0 4095 -g 0 | grad_ph ${sp_th} | awk '{print $4}' | histo
1 0.5 4095.5 4096 > ${heading}_c${ro_node}_sp${sp_th}_g0.qdp
rv_streamfilt -c ${ro_node} < ${acis_evts} | rv_gflt ${ev_th}
${sp_th} -p 0 4095 -g 0 2 3 4 | grad_ph ${sp_th} | awk '{print $4}' |
histo 1 0.5 4095.5 4096 > ${heading}_c${ro_node}_sp${sp_th}_g0234.qdp
Next, the script calculates the variation in count rate (counts/frame) for each read-out node as a function of frame number counted from the first raw image frame of a given set (~200 frames).
rv_streamfilt -c ${ro_node} < ${acis_evts} | rv_gflt ${ev_th}
${sp_th} -p 0 4095 -g 0 2 3 4 | grad_ph ${sp_th} | awk '{print $7}' |
lightcv > ${heading}_c${ro_node}_sp${sp_th}_g0234_lightcv.qdp
And, pulse-height histograms of each grade (0 to 7) are produced in QDP by the following tcsh command:
rv_streamfilt -c ${ro_node} < ${acis_evts} | rv_gflt ${ev_th}
${sp_th} -p 0 4095 | rv_ev2pcf ${ev_th} ${sp_th} /dev/null >
${heading}_c${ro_node}_pcf.qdp
Neighbor histograms (corner pixel distributions and top, bottom, left, and right pixel distributions of detected events) are produced by the following command:
rv_streamfilt -c ${ro_node} < ${acis_evts} | neighborhist >
${heading}_neighborhist_c${ro_node}.qdp
In QDP files, as a product of this step, "readout fraction" (total pixel values divided by the total number of pixels for each bin) is plotted against ADU.
To use XSPEC for the spectral analysis (with convolution of response matrix), two PHA files are produced (g0 & g0234):
rv_streamfilt -c ${ro_node} < ${acis_evts} | rv_gflt ${ev_th}
${sp_th} -p 0 4095 -g 0 2 3 4 | rv_ev2pha ${ev_th} ${sp_th} 1 >
${heading}_c${ro_node}g0234.pha
rv_streamfilt -c ${ro_node} < ${acis_evts} | rv_gflt ${ev_th}
${sp_th} -p 0 4095 -g 0 | rv_ev2pha ${ev_th} ${sp_th} 1 >
${heading}_c${ro_node}g0.pha
A summary table and sample plots of products generated by ACISANAL2 are presented in this section. A sample plot of light curve is shown in Figure 5, a sample plot of PH-histogram is in Figure 6, a sample plot of primary calibration file is in Figure 7, and a sample plot of read-out noise distribution is in Figure 8.
| Type | Sample_filename | Format | Purpose |
|---|---|---|---|
| Evts Num Density | test_sp20_g0234_img.fits | FITS | Event Flux |
| PH histogram | test_c0_sp20_g0.qdp | QDP | Spectral Resolution |
| PH histogram | test_c0_sp20_g0234.qdp | QDP | Spectral Resolution |
| PH spectrum | test_c0_sp20_g0.pha | XSPEC | Spectral response/QE meas. |
| PH spectrum | test_c0_sp20_g0234.pha | XSPEC | Spectral response/QE meas. |
| Primary Cal File | test_c0_pcf.qdp | QDP | Grade branching ratio meas. |
| Light Curve | test_c0_sp20_g0234_lightcv.qdp | QDP | Check source/detector stability |
| Noise Summary File | readoutnoise.out | ASCII | Read-out Noise σ (ADU) |
| Noise Model | readoutnoise00im.model | ASCII | 1 Gaussian fit |
| Noise spectrum | readoutnoise00im.qdp | QDP | Read-out Noise meas. |
