Parseltongue scripts for loading, housekeeping and processing of e-MERLIN data ------------------------------------------------------------------------------ As well as the e-MERLIN pipeline, the scripts in this directory are intended to automate many of the initial tedious tasks which must be run on e-MERLIN data before the user can do anything useful with them. Aside from flagging (which the user must still do, sorry), all of these processes are standard and require no user interaction once they are running. Apart from eMERLIN_loadlocal.py (which loads data directly from the e-MERLIN archive at JBCA), these scripts can be run on any machine with working installations of AIPS and ParselTongue (and sufficient disk space to accommodate your data!). To process your data, follow the simple recipe: * load your data (with either eMERLIN_load.py or eMERLIN_loadlocal.py, see below) * flag your data using your preferred methods * average and combine your datasets (with eMERLIN_dbcon.py) * run the pipeline (eMERLIN_pipeline.py) You can, of course, do some or all of this by hand, and you will need to re-image your data in any case, but these scripts should speed things up. ----------------------------------------------------------------------------------------------- eMERLIN_loadlocal.py *** Intended for use on the Jodrell system only! *** *** If you are not at Jodrell, use eMERLIN_load.py instead. *** Script to load data directly from the e-MERLIN archive located at JBCA. The script will ask you for your AIPS number and which AIPS disk to load to. You can specify dates to load either by giving a text file containing a list of dates as an argument when invoking the script (e.g. "parseltongue eMERLIN_loadlocal.py dates.txt", where "dates.txt" contains a list of dates in yyyymmdd format, one per line) or, if no file is specified, the script will ask you for dates interactively (in the same format). The script searches the e-MERLIN archive on the Jodrell system for the specified dates and loads anything with the extension ".fits" to the AIPS number and disk specified by the user. For each file found, the script will run FITLD, UVFIX, MSORT and INDXR, tidying up the AIPS disk when loading is complete. Due to AIPS issues, the script will not be able to load files with names longer than 46 characters. It will log all messages to eMERLIN_loadlocal.log (clobbering any messages from previous runs). Following this script, you should flag your data before proceeding. eMERLIN_load.py Script to load data from disk. The script will ask you for your AIPS number and which AIPS disk to load to. Your data should be in one or more directories, named as they are on the archive (e.g. "/scratch/jaglan_1/M51/20110613_1/"). Put the script in the base directory where your data are located (e.g. "/scratch/jaglan_1/M51/") and invoke as "parseltongue eMERLIN_load.py". For each file found, the script will run FITLD, UVFIX, MSORT and INDXR, tidying up the disk when loading is complete. Due to AIPS issues, the script will not be able to load files with names longer than 46 characters. It will log all messages to eMERLIN_load.log (clobbering any messages from previous runs). Following this script, you should flag your data before proceeding. eMERLIN_dbcon.py Script to average and combine e-MERLIN datasets. Before running this script, your data should have been UVFIXd, MSORTd, INDXRd (by using the load scripts above) and (e.g.) SPFLG'd. The script will ask you for your AIPS number, the AIPS disk where your data are located, the desired number of spectral channels in your final database and how much time averaging you want. If you want no time averaging, enter "-1" at the appropriate prompt. If you want no spectral averaging, enter "-1" at the appropriate prompt. The script queries the specified AIPS catalog and exits if any stale files from a previous run are found, so delete any pre-existing files of class "SUB SP" or "DBCON" before you run it. It will average in time with UVAVG (if requested), average spectraly with AVSPC (if requested), then combine in a logical way with DBCON, finishing with a final run of MSORT and INDXR. It will log all messages to eMERLIN_dbcon.log (clobbering any messages from previous runs). Following this script, you data should be ready for calibration. eMERLIN_pipeline.py Data calibration pipeline for e-MERLIN data. The script can either load a file from disk, or process a file already loaded (in which case it assumes that your AIPS catalogue contains only one file, the flagged output from the dbcon script, and will complain if it finds other files). After checking the inputs, the pipeline runs an initial fringe fit before calibrating using standard procedures. Inputs are set in a text file which should be specified when invoking the script as e.g. "parseltongue eMERLIN_pipeline.py calib.inp". Requires the file "eMERLIN_tasks.py" to be present. The pipeline first deletes all SN, PL, BP and CL(>1) tables, so if you have already done some processing you are advised to backup your tables outside of AIPS before running it. The script carries out the following procedures: (optionally) load data from disk, (optionally) msort & indxr, output some pre-calibration diagnostic plots, fringe fit the data, calibrate the gain and phase, do a bandpass calibration, (optionally) iterate selfcal on the phase calibrator, map the phase cal and source. It also makes various diagnostic plots along the way which it will plot to disk. The pipeline requires a number of bits of information which it takes from a text file specified on the command line. An example input file ("calib.inputs") with explanations is located with the scripts. Most parameters are optional (although recommended) and will be set to sensible defaults if they are missing from the inputs file. You *must* specify your AIPS number, disk, sources and pointcal fluxes. Note that, unlike for the old MERLIN pipeline, there is no longer an option to specify continuum/wide-field/spectral imaging. Averaging should have been carried out by the eMERLIN_DBCON.py scipt (or by hand if you prefer), the pipeline carries out a bandpass calibration on all datasets and no further averaging is done here. calib.inputs Example set of inputs for the pipeline (eMERLIN_pipeline.py). eMERLIN_tasks.py File containing the task definitions for eMERLIN_pipeline.py. mkargo 2011