This script runs a global search through Madrigal data from a given URL, and downloads all matching files.

This script is a stand-alone application, and can be run from anywhere with a connection to the internet. It runs on either unix or windows. It requires only the MadrigalWeb python module to be installed.

Usage:

globalDownload.py --url=<Madrigal url> --outputDir=<output directory> \ --user_fullname=<user fullname> --user_email=<user email> \ --user_affiliation=<user affiliation> --format=<ascii,hdf5> [options]

where:

--url=<Madrigal url> - url to homepage of site to be searched (ie, http://madrigal.haystack.mit.edu/madrigal/) This is required.

--outputDir=<output directory> - the output directory to store all files in. Default is to store all files in the same directory, and a number is added to the filename if a file might be overwritten. Set --tree flag to store all files in the same directory structure they appear in Madrigal. This allows all files to keep their original names.

--user_fullname=<user fullname> - the full user name (probably in quotes unless your name is Sting or Madonna)

--user_email=<user email>

--user_affiliation=<user affiliation> - user affiliation. Use quotes if it contains spaces.

--format=<ascii or hdf5>

and options are:

--startDate=<MM/DD/YYYY> - start date to filter experiments before. Defaults to allow all experiments.

--endDate=<MM/DD/YYYY> - end date to filter experiments after. Defaults to allow all experiments.

--inst=<instrument list> - comma separated list of instrument codes or names. See Madrigal documentation for this list. Defaults to allow all instruments. If names are given, the argument must be enclosed in double quotes. An asterick will perform matching as in glob. For example:

           --inst=10,30

           --inst="Jicamarca IS Radar,Arecibo*"

--expName - filter experiments by the experiment name. Give all or part of the experiment name. Matching is case insensitive. Default is no filtering by experiment name.

--kindat=<kind of data list> - comma separated list of kind of data codes. See Madrigal documentation for this list. Defaults to allow all kinds of data. If names are given, the argument must be enclosed in double quotes. An asterick will perform matching as in glob. For example:

            --kindat=3001,13201

            --kindat="INSCAL Basic Derived Parameters,*efwind*,2001"

--seasonalStartDate=<MM/DD> - seasonal start date to filter experiments before. Use this to select only part of the year to collect data. Defaults to Jan 1. Example:

--seasonalStartDate=07/01 would only allow experiments after July 1st from each year.

--seasonalEndDate=<MM/DD> - seasonal end date to filter experiments after. Use this to select only part of the year to collect data. Defaults to Dec 31. Example:

--seasonalEndDate=10/31 would only allow experiments before Oct 31 of each year.

--tree - add if you want to store the downloaded files in the same hierarchy as in Madrigal: <YYYY/<instCode>/<experimentDir>. Without --tree, stores all downloaded files in one directory.

--includeNonDefault - if given, include realtime files when there are no default. Default is to search only default files.

--verbose - if given, print each file processed info to stdout. Default is to run silently.

Example:

globalDownload.py --url=http://madrigal.haystack.mit.edu/madrigal --outputDir=/tmp --user_fullname="Bill Rideout" --user_email=brideout@haystack.mit.edu --user_affiliation=MIT --startDate=01/01/1998 --endDate=-01/30/1998 --inst=30

$Id: globalDownload.py 4111 2012-11-19 15:28:18Z brideout $

filterExperimentFilesUsingFileDesc ( expFileList,  fileDesc )

filterExperimentFilesUsingKindat ( expFileList,  kindat )

filterExperimentFilesUsingStatus ( expFileList )

filterExperimentsUsingExpName ( expList,  expName )

filterExperimentsUsingSeason (
        expList,
        seasonalStartDate,
        seasonalEndDate,
        )

getExperimentFileList (
        server,
        expList,
        includeNonDefault,
        )

        server - the active MadrigalData object to get information from

        expList - the list of desired MadrigalExperiment objects

        includeNonDefault - 1 if should include non-default files, 0 otherwise

getInstrumentList ( inst,  server )

getSubdirectoryFromFullFile ( fullFilename )

getTimesOfExperiment ( expList,  expId )