Category:Post-Processing
Special Considerations for EVLA Data Calibration and Imaging in AIPS
Overview
The old VLA with its once state of the art, but now dated, correlator and electronic has been turned off. The new electronics and correlator of the EVLA has been turned on and made available to users. For the time being, this availability will be “Shared Risk Observing” of two forms: “Open” (OSRO) with limited capabilities and “Resident” (RSRO) with potentially unlimited capabilities. OSRO data are expected to be similar to those of the VLA initially although with more spectral channels and bandwidth. OSRO data for continuum science will have, in each of 2 tunable spectral windows, 64 spectral channels in each of 4 polarization products covering ≤ 128 MHz. For line work, OSRO data will have one spectral window with 256 spectral channels in both RR and LL polarizations covering ≤ 128 MHz total bandwidth. These capabilities will increase gradually with time. RSRO data may have many thousands of spectral channels and, in time, up to 8 GHz of bandwidth per polarization. AIPS software will be important to both programs, although RSRO data are expected, by management, to be processed primarily in CASA.
At this writing, the EVLA has already produced amazing scientific results, but with considerable difficulties which are expected to be corrected over time. Delays are currently not set accurately, causing the data analysis to begin with the VLBI task FRING needed to correct large slopes in phase across the bandpass. The flagging information known to the on-line system (telescope off source and the like) is now transferred to CASA and so may reach AIPS in the form of already flagged data. That on-line flagging information is not transferred to AIPS in the form of a flag table and neither software package yet receives useful flagging data, such as the antenna has no receiver, from the correlator. Substantial flagging effort is therefore still required. Also, system temperatures and gains are not transferred as yet. This means that the data weights are not meaningful and should not be adjusted by the amplitude calibration. That calibration will not be as good as it will be since corrections for differences in gains and sensitivity in the directions of the primary and secondary calibrators and the target sources cannot be made. The weather table is now available with the data so that reasonable opacities may be determined. However, the “over-the-top”, table which is used in determining antenna positions, the frequency offset table, used in managing Doppler tracking, and the CQ table, used to correct amplitudes for spectral averaging in the presence of non-zero delays, are not available.
Although these issues should be corrected, quite possibly early in 2010, the following guide will not assume that they have been completed. Steps that can be omitted or simplified when they are will be described. This appendix is written with the assumption that the reader is moderately familiar with AIPS as described in the preceding chapters. It is also written with the assumption that you are using the 31DEC10 or later releases of the software.
Getting Your Data into AIPS
Your EVLA data are stored as an “ALMA Science Data Model” (ASDM) format file in “SDMBDF” (Science Data Model Binary Data Format) in the NRAO archive. They may be read out of the archive in that format, a CASA measurement set format, or in an AIPS-friendly uvfits format. This last is produced by the CASA uvfits writing software. Go the the web page:
and select the Advanced Query Tool. Fill out enough of the form to describe your data and submit the query. If the data are not yet public, you will need the Locked Project Access Key which may be obtained from the NRAO data analysts. To avoid the need for this key, you may log in to my.nrao.edu after which it will know if you are entitled to access to locked projects. The query will return a list of the data sets which meet your specifications. On this form, enter your e-mail address, choose AIPS Friendly names (almost certainly does not work), AIPS FITS under the EVLA-WIDAR section and choose the desired spectral and time averaging. If the delays are not accurately known, spectral averaging can be damaging to the data amplitudes. However, the data are recorded at one-second intervals which is rather short, making the data voluminous. Judicious averaging can help with data set size and processing times without compromising the science. Choose the data set(s) you wish to receive and submit the request. You will be told an estimate of the output data set sizes and the amount of time you will need to wait for the format translation to occur. A 19 Gbyte SDMBDF file run as a test with no averaging was estimated to produce a UVFITS file of 30.26 Gbytes and to take 103 minutes to prepare for download. That time assumes that your download job is the only one being performed. If your download fails, you will probably be told erroneously by e-mail that it worked. The output file will however be missing or incomplete. Try again before contacting NRAO for help. Unlocked files will be downloaded to the NRAO public ftp site:
ftp://ftp.aoc.nrao.edu/e2earchive/
and you may then use ftp to copy the file to your computer. Locked files will go to a protected ftp site and you must use ftp to download those, even within NRAO. The instructions for downloading will be e-mailed to you. Be sure to specify binary for the copy. If you are located in the AOC in Socorro, you may set an environment variable to the archive location, e.g.,
export E2E=/home/acorn2/ftp/pub/e2earchive CR for bash shells
setenv E2E /home/acorn2/ftp/pub/e2earchive CR for C shells such as tcsh
and simply unlocked data files directly from the public download area. Note that the file will be deleted automatically after 48 hours in both public and protected data areas. The data file may be read from disk into AIPS using UVLOD or FITLD, using:
> DEFAULT ’UVLOD’ ; INP CR to initialize and review the inputs needed.
> DATAIN ’E2E:filename’ CR where filename is the disk file name in logical area E2E; (see § 3.10.3).
> DOUVCOMP FALSE CR to write visibilities in uncompressed format. There are no
weights at present, so there is no loss of information in
compressed format, but the conversion from compressed format
costs more than reading the larger data files.
> OUTNA ’myname’ CR to set the AIPS name.
> OUTCL ’ ’ CR to take default (UVDATA) class.
> OUTSEQ 0 CR to take next higher sequence #.
> OUTDI 3 CR to write the data to disk 3 (one with enough space).
> INP CR to review the inputs.
> GO CR to run the program when you’re satisfied with inputs.
Watch the messages from UVLOD to see where your data set goes and whether the task ran properly. When it is finished, check the output header:
> INDI n; GETN m CR to select the data set on disk n and catalog number m.
> IMHEAD CR to examine the header.
Note that the header does not show the usual complement of AIPS extension files. CASA translates the on-line data into its internal format and then writes the uvfits file read by AIPS. Since CASA does not have files comparable to AIPS index and CL tables, it does not provide them. To build index and calibration tables, use;
> TASK ’INDXR’ ; INP CR to select the task and review its inputs.
> INFILE ’ ’;PRTLEV=0 CR to be sure not to use an input text file and to avoid excess messages.
> CPARM=0,0,1/2 CR to make a CL table 1 with a 30-second interval.
> BPARM τ , 0 CR to take default VLA gains and a zenith opacity of τ . Set τ = −1
for no opacity correction. You may set τ = 0, which is now
recommended, to get new default opacities. These are based on
a detailed model predicting the opacity at any frequency from
that at 22 GHz. The combination of weather and seasonal
model long used by FILLM and INDXR is now used solely to
estimate the 22 GHz opacity.
> GO CR to run the task after checking the inputs.
It is a good idea to list the structure of your data set and your antenna locations on the printer and to keep those listings next to your work station for reference:
> DEFAULT LISTR ; INP CR to initialize the LISTR inputs and review them.
> INDI n; GETN m CR to select the data set on disk n and catalog number m.
> OPTYPE ’SCAN’ ; DOCRT -1 CR to choose a scan listing on the printer.
> GO ; GO PRTAN CR to print the scan listing and the antenna file contents.
Read these with care. There have sometimes been problems with antenna identifications, with the order of the IF frequencies, and even with identification of sources by scan. Task SUFIX may be used to correct the last problem and, if desired, FLOPM may be used to reverse the frequency order. You may have to use SETJY to change the CALCODE of some sources if your calibration sources have a blank calibrator code or your target sources have a non-blank calibrator code.
Initial Calibration -- FRING
We have had difficulty setting all of the delays in the EVLA to values which are sufficiently accurate. If the delay is not set correctly, the interferometer phase will vary linearly with frequency, potentially wrapping through several turns of phase within a single spectral window (“IF band”). We hope that bad delays will not arise in future, allowing you to skip this section. But this is a problem familiar to VLBI users and AIPS has a well-tested method to correct the problem. Using your LISTR output, select a time range of about one minute toward the end of a scan on a strong point-source calibrator, usually your bandpass calibrator. Then
> DEFAULT FRING ; INP CR to initialize the FRING inputs and review them.
> INDI n; GETN m CR to select the data set on disk n and catalog number m.
> TIMERANG db,hb,mb,sb,de,he,me,se CR to specify the beginning day, hour, minute, and second and
ending day, hour, minute, and second (wrt REFDATE) of the
data to be included.
> SOLINT 1.05 CR to set the averaging interval in minutes slightly longer than the
data interval selected.
> DPARM(9) = 1 CR to fit only delay, not rate.
> INP CR to check the voluminous inputs.
> GO to run the task, writing SN table 1 with delays for each antenna,
IF, and polarization.
to specify the beginning day, hour, minute, and second and ending day, hour, minute, and second (wrt REFDATE) of the data to be included. Too much data will cause trouble. to set the averaging interval in minutes slightly longer than the data interval selected. to fit only delay, not rate. to check the voluminous inputs. to run the task, writing SN table 1 with delays for each antenna, IF, and polarization. The different IFs in current EVLA data sets may come from different basebands and therefore have different residual delays. The option APARM(5)=1 to force all IFs to have the same delay solution is therefore no longer appropriate. This SN table will need to be applied to the main CL table created by INDXR.
> TASK ’CLCAL’ ; INP CR to look at the necessary inputs.
> TIMERANG 0 CR to reset the time range.
> GAINUSE 0 ; GAINVER 0 CR to select the highest CL table as input and write one higher as
output (version 1 and 2, resp. in this case).
> SNVER 1 ; INVER 1 CR to use only the SN table just created.
> INP CR to review the inputs.
> GO CR to make an update calibration table.
Be sure to apply this (or higher) CL table with DOCALIB 1 in all later steps.
Initial Editing
There will be data validity information prepared both by the on-line control software and by the WIDAR correlator and this information will in time be available as an initial flag table. The tasks above will have applied this table for you by default since FLAGVER 0. On-line flags may already have caused data to be flagged within your data set (but not deleted) by CASA. Unfortunately, at this writing, no flag table is present and, even when it does appear, it may not be fully reliable initially. Thus, we need to look at the data to flag out whatever remains of the time off source not flagged in CASA using on-line flagging information. There have also been drop outs in which the visibility is pure zero, typically for all channels and IFs and a single integration. The drop outs should now be handled by UVLOD and FITLD. Note, however, that CASA and FITLD pass along all data samples, including those that are fully flagged. This makes the data set rather larger than one might wish. Use UVCOP to remove all fully flagged data samples. Before doing this, use TVFLG to look for any more data samples that might need to be flagged fully. Check especially samples at the beginnings and ends of scans. Try
> DEFAULT TVFLG ; INP to reset all adverbs and choose the task.
> INDI n; GETN m CR to select the data set on disk n and catalog number m.
> DOCAL 1 ; DOBAND -1 CR to apply the delay calibration. If a bandpass has been
determined, use DOBAND 3 or 1 to apply it.
> BCHAN c1 ; ECHAN c2 CR to average across a range of channels — not as flexible as
ICHANSEL but probably okay here.
> BIF j ; EIF BIF CR to edit one IF only, which will suffice for problems that are not
IF dependent, such as drop outs, antenna not on source, etc.
Choose an IF that is reasonably free of RFI.
> NCHAV ECHAN-BCHAN+1 CR to average all the channels into one number.
> CALCODE ’*’ CR to do just calibrators for the moment.
> DPARM(6) ∆ t CR to do no time averaging in the work file set ∆ t to the data interval in seconds.
> GO CR to start the task.
The default smoothing time shown in the display will probably be some multiple of ∆ t. Select sub-windows and change the smoothing time to one times the basic interval in order to edit in detail. Remember to change the initial setup so that the flags apply to all channels and all IFs. See § 4.4.3 for more information.
We note here that some users feel that the data need to be inspected more carefully than with just an average of most of the channels. POSSM (below) may be of use to find RFI. Avoiding the worst of that, you may still wish to run TVFLG to look at the average of a few channels at a time. Use NCHAV and CHINC appropriately. Task SPFLG (§ 10.2.2) is the ultimate weapon when looking for channel-dependent difficulties, but is onerous when there are many baselines.
Basic Calibration
Target Source Data -- Edit and SPLIT
Spectral-line Imaging Hints
Continuum Imaging Hints
The first problem that continuum observers will notice with their EVLA data is that the spectral and time resolution of the data, by default anyway, will be rather more than their science requires. It will be possible to instruct the software which extracts data from the archive to do some averaging in both frequency and time. However, detailed editing for RFI and other issues may require excellent resolution in both these domains. After the data have been edited, you can average data in both domains so long as you are careful not to average so much that you produce radial (bandwidth) and/or transverse (time) smearing within the image area. Note that the increased sensitivity of the EVLA will increase the area over which non-negligible astronomical objects may be found while the wide bandwidth will mean that lowest frequency part of your band will be sensitive, because of its larger primary beam, to a much larger area on the sky than the highest frequency part. The spectral averaging can be done with SPLIT; use APARM(1)=1 and set NCHAV, CHINC, and perhaps SMOOTH appropriately. Similarly, AVSPC can be used with AVOPTION=’SUBS’, setting CHANNEL and SMOOTH suitably. You will almost certainly wish to retain some spectral separation, so do not use the “channel 0” option.
Time averaging should be done with UVAVG:
> DEFAULT UVAVG ; INP to reset all adverbs and choose the task.
> INDI Sn; GETN Sm CR to select the calibrated target data set on disk Sn and catalog
number Sm.
> YINC ∆t CR to average to ∆t seconds.
> GO CR to produce the averaged data set.
UBAVG will do a more aggressive averaging, using baseline-dependent time intervals appropriate for the desired field of view. Do not use UBAVG if you are planning to use self-calibration since it destroys the time regularity in the data on which CALIB depends. IMAGR may now do this extra averaging for you on the fly to reduce the size of the work file it uses. Set IM2PARM(11) and (12).
Imaging of the continuum is discussed at great length in Chapter 5 and those details will not be repeated here. Bandwidth-synthesis imaging, which will be the only form of continuum imaging with the EVLA, will make certain adverbs more important. Set BCHAN and ECHAN to avoid the noisier edge channels. Set NCHAV = ECHAN - BCHAN + 1 and CHINC = NCHAV. This will then image all of your IFs and spectral channels into a single image, positioning each channel correctly in the uv plane. With the EVLA, you will be imaging a wider field of view than you did with the VLA. Use SETFC with IMSIZE 0 ; CELLSIZE 0 to see if you should image with a single facet or with multiple facets. If using multiple facets and trying for significant dynamic range, start imaging with OVERLAP 2 ; ONEBEAM -1, but consider OVRSWTCH = -0.05 or so to switch into faster methods of Cleaning when the dynamic range in the residual is small enough.
31DEC09 and later versions of IMAGR allow you to request automatic finding of the Clean boxes (IM2PARM of 1 through 6). In cases with low sidelobes, this works rather well, but you should probably keep an eye on what it does with DOTV 1 in any case. IM2PARM(12) controls the baseline-dependent time averaging while specifying the maximum field of view you expect. This allows you to reduce the size of the work file considerably which will at least reduce the time required for many of the steps in the imaging proportionally. It may be rather better than that if the work file is very large otherwise, requiring actual reading of the disk every time the data are accessed. Note, however, that the uniform weighting of your data will be affected. This averaging reduces the number of samples at short spacings disproportionally and so appears to reduce their weight in the imaging. Some UVTAPER could be reduce to compensate for this.
By default, bandwidth synthesis imaging assumes that the primary beam and all continuum sources are the same at every frequency. In fact, the primary beam size varies linearly with frequency (to first order anyway) and sources have spectral index. IMAGR will allow you to compensate for the average spectral index at almost no cost with IMAGRPRM(2). A far more accurate and expensive correction for spectral index may be made if you do the following. First image each spectral channel (or group of closely-spaced channels) separately. Combine them into a cube with FQUBE, transpose the cube with TRANS, and solve for spectral index images with SPIXR. To use these images, set IMAGRPRM(17) to a radius (> 0) in pixels of a smoothing area and put the image name parameters in the 3rd and 4th input image names. Note that this algorithm is expensive, but that it can be sped up with judicious use of the FQTOL parameter.
The change of primary beam with frequency may be corrected by setting IMAGRPRM(1) = 25 for the diameter of the EVLA dishes. Note that this algorithm is expensive, but that it can be sped up with judicious use of the FQTOL parameter. These two corrections work together, so that doing both costs very little more than doing just one of them.
If you are observing a strong source and trying for very high dynamic range, you will probably have to correct for errors that are baseline- rather than antenna-dependent. One source of these errors is the antenna polarization leakage which affects the parallel-hand visibilities in a non-closing fashion. Task BLCAL can be used after you have as good an image as you can get without it. This task will divide the data by the model and average over a user-specified time to find baseline-dependent corrections which may then be applied to the data by setting adverb BLVER. We recommend that you average the divided data over all of the times in your data to get a single correction for each baseline (and IF and polarization). If you use shorter intervals, you run the risk of forcing your data to look too much like your model. Since the polarization leakage is probably a function of frequency, an experimental version of BLCAL called BLCHN has been released. It determines the same correction but does not average over channels. The correction is saved in a table which POSSM and BPLOT are able to display. However, the calibration routines do not know how to apply this table, so BLCHN write out the corrected data as well as the table.
Concluding Remarks, References, Pictures
AIPS itself, and particularly this appendix, do not begin to cover all of the issues that will arise with EVLA data. The increased bandwidth will probably cause the polarization calibration to change from one complex number per antenna per IF to a complete complex spectrum much like a bandpass. This will force major revisions to the AIPS code which deals with this area. See § 4.6 for information about current AIPS methods of polarization calibration, which average all spectral channels within an IF except for those which are flagged.
The increased sensitivity of the EVLA will means that imaging will no longer be able to ignore effects that are difficult to correct such as pointing errors, beam squint, variable antenna polarization across the field, leakage of polarized signal into the parallel-hand visibilities, etc., etc. These are research topics which may have solutions in other software packages such as OBIT and CASA.
