Atmospheric Observatory

Data Extractor

The data extracter is written in perl, like my other webpages, and can found in /home/sws09a/public_html/cgi-bin/extract/.

set_env_extract.pl

The filename I use for the file containing almost all the environment variables that are likely to be changed is normally set_environment.pl. However because the data extractor uses a lot of files from the manual observation suite, the manual observations code directory is included in the perl library directory through the line

use lib '/home/sws09a/public_html/cgi-bin/obs';

found in view.cgi, and the filename set_environment.pl is already taken in the manual observations code directory. So for this code I've used the filename set_env_extract.pl, which sets

• the directories which
  • find the code for this suite
  • find the configuration file for the METFiDAS data (variables.config).
  • store the CSV files for the data extracted
• The maximum number of CSV files to store, $maxFiles. If this is exceeded then the old files begin to be deleted.
• Location of style sheet.
• METFiDAS variables of this level and below can be extracted (the level option hasn't really be used for the manual observation data as yet).
• Various menu information
• Any manual observations that were only available before $dateInfo{'start'}{'year'} are not shown, and $dateInfo{'start'}{'year'} is set here.
• The possible time formats are listed
• The maximum number of Excel rows is set

The routines set_env_obs, which is found the manual observation suite and includes the directory containing the manual observation configuration file, is also called and described on the manual observation webpage.

Gary's software to process 1second raw data

Gary's software, /home/swslab/code/fido.pl, running on labserver as user swslab is still responsible for processing the 1 second raw data. I believe it makes sure that the all the data for a particularly day on is in the same file. I think his software adds in the columns Record and Clock, but I'm not sure what these are. My code just uses the Time column which is in the original files. It'd be worth asking Gary for more information.