Special Considerations for EVLA Data Calibration and Imaging in AIPS

Overview

The old VLA with its once state of the art, but now dated, correlator and electronic has been turned off. The new electronics and correlator of the EVLA has been turned on and made available to users. For the time being, this availability will be “Shared Risk Observing” of two forms: “Open” (OSRO) with limited capabilities and “Resident” (RSRO) with potentially unlimited capabilities. OSRO data are expected to be similar to those of the VLA initially although with more spectral channels and bandwidth. OSRO data for continuum science will have, in each of 2 tunable spectral windows, 64 spectral channels in each of 4 polarization products covering ≤ 128 MHz. For line work, OSRO data will have one spectral window with 256 spectral channels in both RR and LL polarizations covering ≤ 128 MHz total bandwidth. These capabilities will increase gradually with time. RSRO data may have many thousands of spectral channels and, in time, up to 8 GHz of bandwidth per polarization. AIPS software will be important to both programs, although RSRO data are expected, by management, to be processed primarily in CASA.

At this writing, the EVLA has already produced amazing scientific results, but with considerable difficulties which are expected to be corrected over time. Delays are currently not set accurately, causing the data analysis to begin with the VLBI task FRING needed to correct large slopes in phase across the bandpass. The flagging information known to the on-line system (telescope off source and the like) is now transferred to CASA and so may reach AIPS in the form of already flagged data. That on-line flagging information is not transferred to AIPS in the form of a flag table and neither software package yet receives useful flagging data, such as the antenna has no receiver, from the correlator. Substantial flagging effort is therefore still required. Also, system temperatures and gains are not transferred as yet. This means that the data weights are not meaningful and should not be adjusted by the amplitude calibration. That calibration will not be as good as it will be since corrections for differences in gains and sensitivity in the directions of the primary and secondary calibrators and the target sources cannot be made. The weather table is now available with the data so that reasonable opacities may be determined. However, the “over-the-top”, table which is used in determining antenna positions, the frequency offset table, used in managing Doppler tracking, and the CQ table, used to correct amplitudes for spectral averaging in the presence of non-zero delays, are not available.

Although these issues should be corrected, quite possibly early in 2010, the following guide will not assume that they have been completed. Steps that can be omitted or simplified when they are will be described. This appendix is written with the assumption that the reader is moderately familiar with AIPS as described in the preceding chapters. It is also written with the assumption that you are using the 31DEC10 or later releases of the software.

Getting Your Data into AIPS

Your EVLA data are stored as an “ALMA Science Data Model” (ASDM) format file in “SDMBDF” (Science Data Model Binary Data Format) in the NRAO archive. They may be read out of the archive in that format, a CASA measurement set format, or in an AIPS-friendly uvfits format. This last is produced by the CASA uvfits writing software. Go the the web page:

http://archive.cv.nrao.edu/

and select the Advanced Query Tool. Fill out enough of the form to describe your data and submit the query. If the data are not yet public, you will need the Locked Project Access Key which may be obtained from the NRAO data analysts. To avoid the need for this key, you may log in to my.nrao.edu after which it will know if you are entitled to access to locked projects. The query will return a list of the data sets which meet your specifications. On this form, enter your e-mail address, choose AIPS Friendly names (almost certainly does not work), AIPS FITS under the EVLA-WIDAR section and choose the desired spectral and time averaging. If the delays are not accurately known, spectral averaging can be damaging to the data amplitudes. However, the data are recorded at one-second intervals which is rather short, making the data voluminous. Judicious averaging can help with data set size and processing times without compromising the science. Choose the data set(s) you wish to receive and submit the request. You will be told an estimate of the output data set sizes and the amount of time you will need to wait for the format translation to occur. A 19 Gbyte SDMBDF file run as a test with no averaging was estimated to produce a UVFITS file of 30.26 Gbytes and to take 103 minutes to prepare for download. That time assumes that your download job is the only one being performed. If your download fails, you will probably be told erroneously by e-mail that it worked. The output file will however be missing or incomplete. Try again before contacting NRAO for help. Unlocked files will be downloaded to the NRAO public ftp site:

ftp://ftp.aoc.nrao.edu/e2earchive/

and you may then use ftp to copy the file to your computer. Locked files will go to a protected ftp site and you must use ftp to download those, even within NRAO. The instructions for downloading will be e-mailed to you. Be sure to specify binary for the copy. If you are located in the AOC in Socorro, you may set an environment variable to the archive location, e.g.,

export E2E=/home/acorn2/ftp/pub/e2earchive CR	for bash shells 
setenv E2E /home/acorn2/ftp/pub/e2earchive CR	for C shells such as tcsh

and simply unlocked data files directly from the public download area. Note that the file will be deleted automatically after 48 hours in both public and protected data areas. The data file may be read from disk into AIPS using UVLOD or FITLD, using:

> DEFAULT ’UVLOD’ ; INP CR       to initialize and review the inputs needed.
> DATAIN ’E2E:filename’ CR       where filename is the disk file name in logical area E2E; (see § 3.10.3).
> DOUVCOMP FALSE CR              to write visibilities in uncompressed format. There are no 
                                 weights at present, so there is no loss of information in 
                                 compressed format, but the conversion from compressed format 
                                 costs more than reading the larger data files.
> OUTNA ’myname’ CR              to set the AIPS name.
> OUTCL ’ ’ CR                   to take default (UVDATA) class.
> OUTSEQ 0 CR                    to take next higher sequence #.
> OUTDI 3 CR                     to write the data to disk 3 (one with enough space).
> INP CR                         to review the inputs.
> GO CR                          to run the program when you’re satisfied with inputs.

Watch the messages from UVLOD to see where your data set goes and whether the task ran properly. When it is finished, check the output header:

> INDI n; GETN m CR	         to select the data set on disk n and catalog number m.
> IMHEAD CR	                 to examine the header.

Note that the header does not show the usual complement of AIPS extension files. CASA translates the on-line data into its internal format and then writes the uvfits file read by AIPS. Since CASA does not have files comparable to AIPS index and CL tables, it does not provide them. To build index and calibration tables, use;

> TASK ’INDXR’ ; INP CR          to select the task and review its inputs.
> INFILE ’ ’;PRTLEV=0 CR         to be sure not to use an input text file and to avoid excess messages.
> CPARM=0,0,1/2 CR               to make a CL table 1 with a 30-second interval. 
> BPARM τ , 0 CR                 to take default VLA gains and a zenith opacity of τ . Set τ = −1
                                 for no opacity correction. You may set τ = 0, which is now
                                 recommended, to get new default opacities. These are based on
                                 a detailed model predicting the opacity at any frequency from
                                 that at 22 GHz. The combination of weather and seasonal
                                 model long used by FILLM and INDXR is now used solely to
                                 estimate the 22 GHz opacity.
> GO CR                          to run the task after checking the inputs.

It is a good idea to list the structure of your data set and your antenna locations on the printer and to keep those listings next to your work station for reference:

> DEFAULT LISTR ; INP CR         to initialize the LISTR inputs and review them.
> INDI n; GETN m CR              to select the data set on disk n and catalog number m.
> OPTYPE ’SCAN’ ; DOCRT -1 CR    to choose a scan listing on the printer.
> GO ; GO PRTAN CR               to print the scan listing and the antenna file contents.

Read these with care. There have sometimes been problems with antenna identifications, with the order of the IF frequencies, and even with identification of sources by scan. Task SUFIX may be used to correct the last problem and, if desired, FLOPM may be used to reverse the frequency order. You may have to use SETJY to change the CALCODE of some sources if your calibration sources have a blank calibrator code or your target sources have a non-blank calibrator code.

Initial Calibration -- FRING

We have had difficulty setting all of the delays in the EVLA to values which are sufficiently accurate. If the delay is not set correctly, the interferometer phase will vary linearly with frequency, potentially wrapping through several turns of phase within a single spectral window (“IF band”). We hope that bad delays will not arise in future, allowing you to skip this section. But this is a problem familiar to VLBI users and AIPS has a well-tested method to correct the problem. Using your LISTR output, select a time range of about one minute toward the end of a scan on a strong point-source calibrator, usually your bandpass calibrator. Then

 
> DEFAULT FRING ; INP CR	        to initialize the FRING inputs and review them.
> INDI n; GETN m CR	                to select the data set on disk n and catalog number m. 
> TIMERANG db,hb,mb,sb,de,he,me,se CR   to specify the beginning day, hour, minute, and second and 
                                        ending day, hour, minute, and second (wrt REFDATE) of the 
                                        data to be included. 
> SOLINT 1.05 CR                        to set the averaging interval in minutes slightly longer than the 
                                        data interval selected.
> DPARM(9) = 1 CR                       to fit only delay, not rate.
> INP CR                                to check the voluminous inputs.
> GO                                    to run the task, writing SN table 1 with delays for each antenna, 
                                        IF, and polarization.

to specify the beginning day, hour, minute, and second and ending day, hour, minute, and second (wrt REFDATE) of the data to be included. Too much data will cause trouble. to set the averaging interval in minutes slightly longer than the data interval selected. to fit only delay, not rate. to check the voluminous inputs. to run the task, writing SN table 1 with delays for each antenna, IF, and polarization. The different IFs in current EVLA data sets may come from different basebands and therefore have different residual delays. The option APARM(5)=1 to force all IFs to have the same delay solution is therefore no longer appropriate. This SN table will need to be applied to the main CL table created by INDXR.

> TASK ’CLCAL’ ; INP CR         to look at the necessary inputs. 
> TIMERANG 0 CR                 to reset the time range.
> GAINUSE 0 ; GAINVER 0 CR      to select the highest CL table as input and write one higher as 
                                output (version 1 and 2, resp. in this case).
> SNVER 1 ; INVER 1 CR          to use only the SN table just created. 
> INP CR                        to review the inputs.
> GO CR                         to make an update calibration table.

Be sure to apply this (or higher) CL table with DOCALIB 1 in all later steps.

Initial Editing

There will be data validity information prepared both by the on-line control software and by the WIDAR correlator and this information will in time be available as an initial flag table. The tasks above will have applied this table for you by default since FLAGVER 0. On-line flags may already have caused data to be flagged within your data set (but not deleted) by CASA. Unfortunately, at this writing, no flag table is present and, even when it does appear, it may not be fully reliable initially. Thus, we need to look at the data to flag out whatever remains of the time off source not flagged in CASA using on-line flagging information. There have also been drop outs in which the visibility is pure zero, typically for all channels and IFs and a single integration. The drop outs should now be handled by UVLOD and FITLD. Note, however, that CASA and FITLD pass along all data samples, including those that are fully flagged. This makes the data set rather larger than one might wish. Use UVCOP to remove all fully flagged data samples. Before doing this, use TVFLG to look for any more data samples that might need to be flagged fully. Check especially samples at the beginnings and ends of scans. Try

> DEFAULT TVFLG ; INP            to reset all adverbs and choose the task.
> INDI n; GETN m CR              to select the data set on disk n and catalog number m.
> DOCAL 1 ; DOBAND -1 CR         to apply the delay calibration. If a bandpass has been 
                                 determined, use DOBAND 3 or 1 to apply it.
> BCHAN c1 ; ECHAN c2 CR         to average across a range of channels — not as flexible as 
                                 ICHANSEL but probably okay here.
> BIF j ; EIF BIF CR             to edit one IF only, which will suffice for problems that are not 
                                 IF dependent, such as drop outs, antenna not on source, etc.
                                 Choose an IF that is reasonably free of RFI.
> NCHAV ECHAN-BCHAN+1 CR         to average all the channels into one number.
> CALCODE ’*’ CR                 to do just calibrators for the moment.
> DPARM(6) ∆ t CR                to do no time averaging in the work file set ∆ t to the data interval in seconds.
> GO CR                          to start the task.

The default smoothing time shown in the display will probably be some multiple of ∆ t. Select sub-windows and change the smoothing time to one times the basic interval in order to edit in detail. Remember to change the initial setup so that the flags apply to all channels and all IFs. See § 4.4.3 for more information.

We note here that some users feel that the data need to be inspected more carefully than with just an average of most of the channels. POSSM (below) may be of use to find RFI. Avoiding the worst of that, you may still wish to run TVFLG to look at the average of a few channels at a time. Use NCHAV and CHINC appropriately. Task SPFLG (§ 10.2.2) is the ultimate weapon when looking for channel-dependent difficulties, but is onerous when there are many baselines.

Basic Calibration

For both continuum and line observations, we must begin by determining which spectral channels are reliable and which are affected by the inevitable loss of signal-to-noise at band edges or are degraded by radio- frequency interference (RFI). Use POSSM to display spectra from the shorter baselines on the TV:

> DEFAULT POSSM ; INP           to set the task name and clear the adverbs.
> INDI n; GETN m CR             to select the data set on disk n and catalog number m.
> SOURCE ’bandpass cal’ CR      to select the strong bandpass calibrator.
> DOTV 1;NPLOTS 1 CR            to plot only on the TV, one baseline at a time.
> ANTEN n1,n2,n3,n4 CR          to select the antennas nearest the center of the array or the maintenance areas.
> BASELINE ANTEN CR             and only them.
> DOCAL 1;APARM 1,0 CR          to apply the FRING solutions and display vector averaged
                                spectra. Scalar averaged spectra will turn up at the edges 
                                reflecting the decreased signal to noise in the outer channels.
> BIF j ; EIF BIF CR            to plot one IF at a time.
> GO CR                         to run the task. Make notes of the desirable channels IF by IF.

If there is no RFI, then you may be able to use the same channel range for all IFs. If the RFI is particularly pernicious, you may have to edit it out of your data before continuing. Task FLGIT (§ 8.1) attempts to flag RFI that is both channel- and time-dependent in a non-interactive fashion. SPFLG (§ 10.2.2) is labor and time intensive but would be the most reliable method to deal with the problem. The basic EVLA calibration is much like that described in detail in Chapter 4 except that bandpass calibration is now required rather than merely recommended. Having chosen those channels which may be reliably used to normalize the bandpass functions,

> DEFAULT BPASS ; INP           to reset all adverbs and choose the task.
> INDI n; GETN m CR             to select the data set on disk n and catalog number m.
> DOCAL 1 CR                    to apply the delay calibration — very important.
> SOLINT 0 CR                   to compute a bandpass solution for each scan on the bandpass 
                                calibrator.
> CALSOUR ’bandpass cal’ CR     to select the strong bandpass calibrator.
> ICHANSEL c11,c12,1,if1,c21,c22,1,if2,c31,c32,1,if3,... CR 
                                to select the range(s) of channels which are reliable for
                                averaging in each IF. Remember these values — you will use 
                                them again.
> BPASSPRM(5) 1 ; BPASSPRM(10) 3 CR    
                                to normalize the results only after the solution is found using 
                                the channels selected by ICHANSEL.
> GO CR                         to make a bandpass (BP) table.

Do not use spectral smoothing at this point unless you want to use the same smoothing forever after. Apply the flag table. Consider correcting the bandpass function for the spectral index of bandpass cal if it is known — the EVLA bandwidths are large enough that this may matter. A model for the calibrator may be used; see § 4.3.3.1.

You now need to run SETJY with OPTYPE ’CALC’ and SOURCES set to point at your primary flux calibration sources. You should load the models for these sources that apply to your data with CALRD; see §4.3.3.1. Then run CALIB with the model once for each primary flux calibrator:

> DEFAULT CALIB ; INP          to reset all adverbs and choose the task.
> INDI n; GETN m CR            to select the data set on disk n and catalog number m.
> IN2DI n2; GET2N m2 CR        to select the model image on disk n2 and catalog number m2.
> DOCAL1;DOBAND3 CR            to apply the delay and bandpass calibration — very important.
> SOLINT 0 ; NMAPS 1 CR        to compute a solution for each calibration scan and use the 
                               source model.
> CALSOUR ’flux cal’ CR        to select the primary flux calibrator by whatever form of its 
                               name appears in your LISTR output.
> ICHANSEL c11,c12,1,if1,c21,c22,1,if2,c31,c32,1,if3,... CR
                               to select the range(s) of channels which are reliable for 
                               averaging in each IF. These must be the same values that 
                               you used in BPASS.
> SNVER 2 CR	               to put all CALIB solutions in solution table 2.
> GO CR	                       to find the complex gains for the flux calibrator.

Read the output closely. If solutions fail, examine your data closely for bad things. The primary flux calibrator should work without failure. After you have done each primary flux calibrator for which you have models, run CALIB on the remaining calibration sources:

> CLR2NAME ; NMAPS 0 CR        to do no models.
> CALSOUR ’other cal1’, ’other cal2’ CR      
                               to select the secondary calibrators by whatever names appear
                               in your LISTR output.
> GO CR                        to find the remaining complex gains.

Again, examine the output messages closely. There may be a few failures but there should not be many in a good data set. The RUN file procedure VLACALIB (see § 4.3.3.1) may be used but it does not offer the ICHANSEL option which may be required by your data. It also does a scalar averaging for the amplitudes. In 31DEC10, this averaging was changed to be a vector average of the spectral channels followed by a scalar average over time. Scalar averaging suffers from Ricean bias in the amplitudes and so should be used only when the calibration source is very strong or when the atmospheric phases are very unstable. At this point it is necessary to calibrate the fluxes of the secondary calibration sources using your SN table:

> TASK ’GETJY’ ; INP CR        to set the task name without changing other adverbs.
> SOURCE CALSOUR CR            to select the secondary sources by the list of name you just used.
> CALSOUR ’flux cal’ CR        to select the primary flux calibrator by whatever form of its name appears in your LISTR output.
> INP CR                       to check the inputs closely; remember to do all times, IFs, etc. with SNVER 2.
> GO CR                        to adjust the gains in the SN table and the fluxes in the SU (source) table.

Look at the messages with care — the fluxes in the various IFs should be consistent and the error bars should be reasonably small (< 10% at high frequencies, smaller at low frequencies). If not, look at your SN table with SNPLT to see if there are bad solutions. If there are, delete SN table 2, do more flagging with TVFLG or SPFLG, and repeat the process.

Finally, apply the gain solutions to your calibration table:

> DEFAULT CLCAL ; INP CR       to clear the adverbs.
> INDI n; GETN m CR            to select the data set on disk n and catalog number m.
> CALCODE ’*’ CR               to select all calibration sources.
> SNVER 2; INVERS SNVER CR     to select your solution table from CALIB. Do not include the 
                               SN table from FRING a second time!
> GO CR                        to apply SN table 2 to CL table 2, creating CL table 3.

Check the result using POSSM and/or VPLOT.

Target Source Data -- Edit and SPLIT

Spectral-line Imaging Hints

Continuum Imaging Hints

The first problem that continuum observers will notice with their EVLA data is that the spectral and time resolution of the data, by default anyway, will be rather more than their science requires. It will be possible to instruct the software which extracts data from the archive to do some averaging in both frequency and time. However, detailed editing for RFI and other issues may require excellent resolution in both these domains. After the data have been edited, you can average data in both domains so long as you are careful not to average so much that you produce radial (bandwidth) and/or transverse (time) smearing within the image area. Note that the increased sensitivity of the EVLA will increase the area over which non-negligible astronomical objects may be found while the wide bandwidth will mean that lowest frequency part of your band will be sensitive, because of its larger primary beam, to a much larger area on the sky than the highest frequency part. The spectral averaging can be done with SPLIT; use APARM(1)=1 and set NCHAV, CHINC, and perhaps SMOOTH appropriately. Similarly, AVSPC can be used with AVOPTION=’SUBS’, setting CHANNEL and SMOOTH suitably. You will almost certainly wish to retain some spectral separation, so do not use the “channel 0” option.

Time averaging should be done with UVAVG:

> DEFAULT UVAVG ; INP        to reset all adverbs and choose the task.
> INDI Sn; GETN Sm CR        to select the calibrated target data set on disk Sn and catalog
                             number Sm.
> YINC ∆t CR                 to average to ∆t seconds.
> GO CR                      to produce the averaged data set.

UBAVG will do a more aggressive averaging, using baseline-dependent time intervals appropriate for the desired field of view. Do not use UBAVG if you are planning to use self-calibration since it destroys the time regularity in the data on which CALIB depends. IMAGR may now do this extra averaging for you on the fly to reduce the size of the work file it uses. Set IM2PARM(11) and (12).

Imaging of the continuum is discussed at great length in Chapter 5 and those details will not be repeated here. Bandwidth-synthesis imaging, which will be the only form of continuum imaging with the EVLA, will make certain adverbs more important. Set BCHAN and ECHAN to avoid the noisier edge channels. Set NCHAV = ECHAN - BCHAN + 1 and CHINC = NCHAV. This will then image all of your IFs and spectral channels into a single image, positioning each channel correctly in the uv plane. With the EVLA, you will be imaging a wider field of view than you did with the VLA. Use SETFC with IMSIZE 0 ; CELLSIZE 0 to see if you should image with a single facet or with multiple facets. If using multiple facets and trying for significant dynamic range, start imaging with OVERLAP 2 ; ONEBEAM -1, but consider OVRSWTCH = -0.05 or so to switch into faster methods of Cleaning when the dynamic range in the residual is small enough.

31DEC09 and later versions of IMAGR allow you to request automatic finding of the Clean boxes (IM2PARM of 1 through 6). In cases with low sidelobes, this works rather well, but you should probably keep an eye on what it does with DOTV 1 in any case. IM2PARM(12) controls the baseline-dependent time averaging while specifying the maximum field of view you expect. This allows you to reduce the size of the work file considerably which will at least reduce the time required for many of the steps in the imaging proportionally. It may be rather better than that if the work file is very large otherwise, requiring actual reading of the disk every time the data are accessed. Note, however, that the uniform weighting of your data will be affected. This averaging reduces the number of samples at short spacings disproportionally and so appears to reduce their weight in the imaging. Some UVTAPER could be reduce to compensate for this.

By default, bandwidth synthesis imaging assumes that the primary beam and all continuum sources are the same at every frequency. In fact, the primary beam size varies linearly with frequency (to first order anyway) and sources have spectral index. IMAGR will allow you to compensate for the average spectral index at almost no cost with IMAGRPRM(2). A far more accurate and expensive correction for spectral index may be made if you do the following. First image each spectral channel (or group of closely-spaced channels) separately. Combine them into a cube with FQUBE, transpose the cube with TRANS, and solve for spectral index images with SPIXR. To use these images, set IMAGRPRM(17) to a radius (> 0) in pixels of a smoothing area and put the image name parameters in the 3rd and 4th input image names. Note that this algorithm is expensive, but that it can be sped up with judicious use of the FQTOL parameter.

The change of primary beam with frequency may be corrected by setting IMAGRPRM(1) = 25 for the diameter of the EVLA dishes. Note that this algorithm is expensive, but that it can be sped up with judicious use of the FQTOL parameter. These two corrections work together, so that doing both costs very little more than doing just one of them.

If you are observing a strong source and trying for very high dynamic range, you will probably have to correct for errors that are baseline- rather than antenna-dependent. One source of these errors is the antenna polarization leakage which affects the parallel-hand visibilities in a non-closing fashion. Task BLCAL can be used after you have as good an image as you can get without it. This task will divide the data by the model and average over a user-specified time to find baseline-dependent corrections which may then be applied to the data by setting adverb BLVER. We recommend that you average the divided data over all of the times in your data to get a single correction for each baseline (and IF and polarization). If you use shorter intervals, you run the risk of forcing your data to look too much like your model. Since the polarization leakage is probably a function of frequency, an experimental version of BLCAL called BLCHN has been released. It determines the same correction but does not average over channels. The correction is saved in a table which POSSM and BPLOT are able to display. However, the calibration routines do not know how to apply this table, so BLCHN write out the corrected data as well as the table.

Concluding Remarks, References, Pictures

AIPS itself, and particularly this appendix, do not begin to cover all of the issues that will arise with EVLA data. The increased bandwidth will probably cause the polarization calibration to change from one complex number per antenna per IF to a complete complex spectrum much like a bandpass. This will force major revisions to the AIPS code which deals with this area. See § 4.6 for information about current AIPS methods of polarization calibration, which average all spectral channels within an IF except for those which are flagged.

The increased sensitivity of the EVLA will means that imaging will no longer be able to ignore effects that are difficult to correct such as pointing errors, beam squint, variable antenna polarization across the field, leakage of polarized signal into the parallel-hand visibilities, etc., etc. These are research topics which may have solutions in other software packages such as OBIT and CASA.