Dependency perl code

Arguments for dependency perl code

The only way to use the DPC to generate new files is to run the update_files.pl script (the web interface is only a visual aid). The arguments for update_files.pl are:

• --checks=(all|ichecks), where all means that all the parent files will be checked, and ichecks that only the newest ichecks parent files are checked. This is only important for very large file systems, where comparing the age of parent files with all its children could take a long time.
• --depend=dependency file, where dependency file is the name of dependency file found in the config directory. If this isn't specfied the default is usually dependencies.dat.
• --level=ilev, where ilev is the level which will be updated. It is unlikely to be obvious which files can be found at which level, unless you refer to the web interface, which groups files by level. ilev will usually be a number, but can be set to compress if only the compression stage is required.
• --passive, no commands are carried out, but the command that would have been carried out are printed to screen instead. After making any changes it's recommended that this is used the first time that update_files.pl is run.
• --tmax=max-time-in-hours, where if the code exceeds the max-time-in-hours after completing a level it'll stop. If this is not set, the default is usually 12 hours. This is a safety feature. Sometimes the dependency perl program can run for a very long time, especially if it needs to run a lot of Matlab code, which tends to be slow. If the dependency perl program is run a second time while it's still completing a previous job, both jobs can try to create the same file which will uneccessarily increase the load on the processing machine and can cause further problems (see section on Cron jobs below).
• --update=(new|all), where new will only replace files that didn't already exist and all will replace any files the perl script believes need updating. If no option is provided, the default is all.

Cron jobs

Typically the dependency perl program is run through a cron job. Typically it is run once of twice a day following the expected arrival of input files, such as raw data from a remote site. It's worth making sure the max-time-in-hours is significantly less that the gap between cron job to ensure that the code is never running twice at the same time (see the --tmax argument above for why).