This document describes how to read Chilbolton radar data directly into matlab, using the mex-files loadchil and dirchil. Sorry about the monolithic document.
Under solaris the mex files are called loadchil.mexsol and dirchil.mexsol. On the Reading Meteorology Department UNIX system they can be found in /home/swrhgnrj/matlab. Make sure they are in your matlab path (consult the matlab path command).
dirchil is used to list information about Chilbolton data, although currently its functionality is rather primitive. The data can be read into matlab with loadchil, which on the other hand is quite sophisticated and should be able to cope with all the changes to the format that have taken place over recent years (time stamping, optional range correction, more than three parameters etc.). In addition it is able to do a number of processing tasks that previously had to be done within matlab:
loadchil |
loadchil file.dat raster [-scan scan] [options] |
In its simplest form data from a single raster `raster' is read from the file `file.dat' into the matlab workspace. If `scan' is not specified then the first scan of the raster is loaded. This form is suitable for scanning data (as opposed to `cloud' data). The available options are described in the options section. For example:
loadchil 6001new.dat 25 -scan 2In its functional form one could equivalently use
loadchil('6001new.dat','25','-scan','2')or even
loadchil('6001new.dat',25,'-scan',2)
loadchil file.dat raster1-rasterN [-scan scan] [options] |
This form and the next are suitable for cloud radar data in which a number of rasters are required from one or more files. In this form rasters in the range raster1 to rasterN are read from file1.dat, then for any number of file/raster range pairs. Rasters are concatenated. For example:
loadchil 9001c94z.dat 24 9002c94z.dat 1-10
loadchil -date date filepattern1 [filepattern2 ...] [options] |
With cloud radar data the raster concept is inconvenient and one often wishes to look at data in daily chunks, for which this form of the command is used. Here `date' is the date in the form YYYYMMDD or YYMMDD. The list of file patterns are expanded into a real file list (by expanding *'s and ?'s), and scanned for data taken on the required date (in fact it is the corresponding summary files that are scanned. For example:
loadchil -date 19990101 /cdrom/*c94z.dat
The following options can be specified after the file or file pattern list.
Display file and raster information as files and rasters are opened.
Only report errors.
Don't write those scalars to the matlab workspace that are only applicable to single scans and are therefore not appropriate for cloud radar data (day, month, year, file, raster, start_hour, start_min, start_sec). The variables min_gate and max_gate are not necessary if range correction is performed accurately so these are suppressed too.
Average every n rays; the parameters Zh and Ldr are converted into linear units before taking the average. If the radar is measuring v in cloud mode and averaging is performed then sigma_v, the standard deviation of the velocity, is calculated as well.
Average every n gates. The comments for -averays apply here.
Perform range correction as with the -correctrange option, but use a range offset of f metres. Usually f is negative.
Calibrate the Zcal field by adding f dB to it.
Summary files are generally required for loadchil to work. First it looks for them in the same directory as the data file, then it looks in the directory s. You can find the compiled-in default by running loadchil with no arguments.
Subtract the noise contribution to the measured Zh signal using gates at the far end of the ray; for cloud data these gates should be in the cloud-free stratosphere. The number of gates used can be specified using -noisegates. Pixels with no signal are set to NaN in every parameter field. This option can only be used if Zh is one of the parameters. If the noise floor varies due to interference by an oscillator then try -oscillator instead.
As -process, except the gates at the end of the ray are used to characterise an interference in the noise floor. Hence the number of gates used (specified by -noisegates) must be chosen to match the frequency of the oscillator that is causing the interference.
Use n gates at the end of each ray to characterise the noise - works in conjunction with -process or -oscillator. Use loadchil with no arguments to find the default value. Note that it is the number of gates after averaging. For example, if you use 50 noisegates with no vertical averaging then you ought to use 25 noisegates with 2-gate vertical averaging.
The number of noise-standard-deviations that a signal must be above to not be rejected when noise subtraction is performed. If the noise is normally distributed (as it should be) then the default value of 3.0 results in 99.87% of pure noise being rejected. Sometimes interference means that a higher value is necessary to get rid of all anomalous echos, although at the expense of effective sensitivity. If you are rejecting anomalous points of cloud some other way then a lower value may be appropriate.
A further way to get rid of anomalous echos. Any isolated pixels in a ray (after noise subtraction) are removed - this can often get rid of those ugly `lines' in the clear air caused by oscillator interference.
Attempt to suppress ground clutter. Echos with a sigma_v lower than MIN_V_STD and with a v between -MIN_V_STD and +MIN_V_STD are removed. Currently the compiled-in value of MIN_V_STD is 5 cm/s. These pixels are nearly always ground clutter, although sometimes good pixels will be removed.
To get the range accurately it is necessary to start digitising before the pulse has gone out so that the position of the meteorological echos can be measured with respect to the position of the transmit pulse. This means that the true range to the first pulse can be negative. Earlier data had range correction performed using the wrong range and the user was expected to do his/her own correction. Later data does not do range correction at all, but the range offset to use has been encoded into the header of each raster. This option uses all available information to correct the range and the reflectivity values accordingly. Since the earlier data does not have the range offset figure you must specify it with the -rangeoffset option if you intend to correct the range. This option may result in some negative ranges; the Zh values here will all be NaN.
The number of pulses used by the data acquisition system to calculate Zh is usually orders of magnitude more than the number used to calculate v. Hence v tends to get noticeably noisier at low signal to noise ratios. With this option, the equivalent noise floor for v is raised to f dB above that for Zh, so that unreliable values are rejected.
As `-v_floor' but for the parameter sigma_v. This is even more important however, since at low signal-to-noise ratios sigma_v can get very large and be misleading. I've found that f=2.5 dB works OK for the Galileo in the spring/summer '99 configuration.
Load scan n from each raster. The default is for the first scan to be read. Most rasters only contain one scan.
Output extra coordinate matrices x, y, z and r. These enable the data to be plotted easily, e.g. for an PPI:
pcolor(x,y,Zh); shading flat
Or an RHI:
pcolor(r,z,Zh); shading flat
Remove all data (probably clutter) with Ldr greater than f. A good value is -10 (dB).
Subtract noise from Zh, Zdr and Ldr using noise-equivalent reflectivities at 1 km in the H, V and cross-polar channels of fZ, fZV and fZHV (dBZ) respectively. These values vary occasionally when the system is changed. You can find appropriate values below.
Set the noise-elimination threshold to f. The default value is 1.19, which means that the any signal more than around 19% above the noise value is deemed to be a valid meteorological echo. If ever the PRF or dwell time of the 3 GHz radar ever changes then this value should be changed accordingly.
The radar parameters that can be read in are
Radar reflectivity (in dBZ) using horizontal polarisation.
Differential reflectivity (in dB).
Linear depolarisation ratio (in dB).
Differential phase shift (degrees).
Radial velocity (in m/s) measured at horizontal polarisation, where positive indicates towards the radar.
CAMRa measures two velocities: v1 is the velocity measured using both horizontally and vertically polarised pulses, and v2 is that measured using horizontal polarisation alone (so is the same as v above). Both are in m/s. The folding velocity of v1 is therefore twice that of v2.
Doppler spectral width (in m/s). As yet this parameter is not measured by any system.
Standard deviation of the raw (usually 64-pulse) mean velocities (in m/s). This parameter is calculated from v in cloud data when some averaging is applied.
Distance of each pixel in scanning data east of Chilbolton, in km. Only set with the -coords option.
Distance of each pixel in scanning data north of Chilbolton, in km. Only set with the -coords option.
Distance of each pixel in scanning data above the height of Chilbolton, in km. This takes no account of the curvature of the earth. Only set with the -coords option.
Distance of each pixel in scanning data from Chilbolton along the ground. Only set with the -coords option.
The range to the centre of each gate in kilometres.
The elevation angle of each ray in degrees.
The azimuth angle of each ray in degrees clockwise from due west.
The time of each ray in decimal hours UTC.
The date of each ray in the form YYYYMMDD.
The noise-equivalent reflectivity at 1 km (in dBZ) for each ray. This parameter is only calculated if the -process or -oscillator options are selected. If the data is calibrated using the -Zcal option then this parameter is calibrated also.
Note that the writing of many of these can be suppressed by employing the -basicsonly option.
The file and raster number of the last raster.
The system ID (0=unknown, 1=CAMRa, 2=Rabelais, 3=Galileo, 4=Singapore) and frequency (GHz) of the radar that took the data.
The second, minute, hour, day, month and year of the start of the last raster that was loaded. These are present only because they were produced by the previous program used to load scanning data.
The sweep mode in `universal radar format' (0=calibration scan, 1=PPI, 2=coplane, 3=RHI, 4=vertical, 5=dwell, 6=manual, 7=idle).
Range gate spacing in kilometres.
The minimum and maximum gates in clock units (only useful if you want to perform your own range correction).
The range offset that has been applied to the data, in kilometres.
OK, there's the detail, now for some examples for cloud data. I can't guarantee these options are necessarily correct because hardware and data formats often change without anyone being told. In particular the 1999 calibration figures need looking at.
The are an abundance of artifacts in 35 GHz data that my program can't remove - in particular look out for that big band between 2 and 3 km. The system also has a poor recovery time so above strong signals you can get a slightly elevated noise when from the 94 GHz radar we see there is no cloud. I've selected an increased noisefactor of 4.5 to get rid of some of this, but you might want to go higher.
loadchil -date date /cdrom/cdrom0/*35z.dat -averays 120 -process -noisegates 25 -noisefactor 4.5 -clean -Zcal -15.5 -rangeoffset -480 -basicsonly
On the evening of 28 October an attenuator setting was changed by 9.5 dB - calibration changed accordingly.
loadchil -date date /cdrom/cdrom0/*35z.dat -averays 120 -process -noisegates 25 -noisefactor 4.5 -clean -Zcal -25 -rangeoffset -480 -basicsonly
At around the time the instrument started recording mean Doppler velocity the range offset was encoded into the information record so doesn't have to be specified manually. Note that early on in these data I and Q were not centred every ray before calculating velocity so some of this data may be very dubious.
loadchil -date date /cdrom/cdrom0/*35d.dat -averays 120 -process -noisegates 25 -noisefactor 4.5 -clean -Zcal -25 -correctrange -basicsonly
The -oscillator option should get rid of most of those annoying lines, but data was only recorded up to 12 km and you need 25 gates at the top of the ray to characterise the oscillator interference because of it's frequency - Hence when cloud extends well above 10 km this affects noise subtraction further down the ray.
loadchil -date date /cdrom/cdrom0/*94z.dat -averays 120 -oscillator -noisegates 25 -noisefactor 2.0 -clean -Zcal +16 -rangeoffset -190 -basicsonly
The radar was put on the side of the main CAMRa dish, so the calibration figure will have changed - from the noise level I estimate this to be around 7 dB more, and this gives much better visual agreement with the 35 GHz from the same period. There's more gates to get the noise from (if you're not averaging any gates you can use -noisegates 50 if you want. The range offset doesn't need to be specified.
loadchil -date date /cdrom/cdrom0/*94d.dat -averays 120 -oscillator -noisegates 25 -noisefactor 2.0 -clean -Zcal +23 -correctrange -basicsonly
If you want the Zh, Zdr and Ldr to be corrected for noise, then the -polar option must be used. The current values are shown in this example:
loadchil FILEnew.dat raster -scan scan -Zcal +5 -polar -41.7 -43.1 -43.1 -maxLDR -10 -coords -basicsonly
dirchil is a simple utility for summarising the contents of Chilbolton radar data files, but it's not yet fully featured.
dirchil |
dirchil file_pattern1 [file_pattern2 ...] |
Expands the file_patterns into a real file list (using the *'s and the ?'s) and displays the available days contained within the files as a list in the form YYMMDD.
dirchil -date date file_pattern1 [file_pattern2 ...] |
Displays the files and raster ranges that contain the date `date' (in the form YYYYMMDD or YYMMDD) from the file pattern list.
The source can be found at http://www.met.rdg.ac.uk/radar/software.html. Successful compilation has been achieved with gcc under sparc/solaris and HP-UX, but no luck with i686/linux yet. It should be a simple matter to modify loadchil.c to output other information or whatever.
I'm sure you can think of a few more items but here's a few improvements I could make:
Please contact Robin Hogan with comments or queries.