|
|
|
Remote access using IDL | Doc home | Madrigal home |
The following are the methods for accessing Madrigal data remotely:
A good way to learn how to use this IDL API is to run this example.
***********************************************************
;+
; NAME:
; madgetallinstruments(madurl)
;
; PURPOSE:
; Get information on all instruments with data at the Madrigal site specified
; by madUrl
;
; INPUTS:
; madurl - scalar string giving a fully qualified url to the Madrigal site
;
; OUTPUT: an array of structures with the following fields:
; 1. name - (string) Example: 'Millstone Hill Incoherent Scatter Radar'
; 2. code - (int) Example: 30
; 3. mnemonic - (3 char string) Example: 'mlh'
; 4. latitude - (double) Example: 45.0
; 5. longitude (double) Example: 110.0
; 6. altitude (double) Example: 0.015 (km)
; 7. category (string) Example: 'Incoherent scatter radars'
; EXAMPLE:
; result = madGetAllInstruments('http://madrigal.haystack.mit.edu/madrigal')
;
; $Id: madidl.html 4425 2014-07-30 18:47:27Z brideout $
;
FUNCTION madgetallinstruments, madurl
***********************************************************
;+
; NAME:
; madgetexperiments(madurl, instcode, startyear, startmonth, startday, starthour, startmin, startsec,
; endyear, endmonth, endday, endhour, endmin, endsec, local)
;
; PURPOSE:
; Get information on experiments for a given instrument and date range
;
; INPUTS:
; madurl - scalar string giving a fully qualified url to the Madrigal site
; instcode = int or array of ints representing instrument code(s). Special value of 0 selects all instruments.
; startyear, startmonth, startday, starthour, startmin, startsec - 6 ints representing start time
; endyear, endmonth, endday, endhour, endmin, endsec - 6 ints representing end time
; local - 0 if all sites desired, 1 (default) if only local experiments desired
;
; OUTPUT: an array of structures representing experiments between the dates with the following fields:
; 1. strid (string) Example: "10000111". Uniquely identifies experiment. Can be converted to a long long.
; Not stored as a long long because that just doesn't seem to work, so user needs to convert.
; 2. url (string) Example: 'http://www.haystack.mit.edu/cgi-bin/madtoc/1997/mlh/03dec97'
; Deprecated - use realUrl field instead (described below)
; 3. name (string) Example: 'Wide Latitude Substorm Study'
; 4. siteid (int) Example: 1
; 5. sitename (string) Example: 'Millstone Hill Observatory'
; 6. instcode (int) Code of instrument. Example: 30
; 7. instname (string) Instrument name. Example: 'Millstone Hill Incoherent Scatter Radar'
; 8. startyear, startmonth, startday, starthour, startmin, startsec - 6 ints representing start time of experiments
; 9. endyear, endmonth, endday, endhour, endmin, endsec - 6 ints representing end time of experiments
; 10. isLocal - 1 if a local experiment, 0 if not
; 11. madrigalUrl - url of Madrigal site. Used if not a local experiment.
; 12. pi - name of experiment PI
; 13. piemail - email of experiment PI
; 14. realUrl - real url to experiment web page
; If no experiements found, returns a Null Pointer (IDL does not allow empty arrays)
;
; Note that if the returned experiment is not local, the experiment id will be -1. This means that you
; will need to rerun madgetexperiments with the url of the
; non-local experiment (experiment.madrigalUrl).
; This is because
; while Madrigal sites share metadata about experiments, the real experiment ids are only
; known by the individual Madrigal sites. See example_madidl.pro for an example of this
; for an example of this.
; EXAMPLE:
; result = madGetExperiments('http://madrigal.haystack.mit.edu/madrigal', 30, 1998,1,1,0,0,0, 1998,1,31,23,59,59,1)
;
; $Id: madidl.html 4425 2014-07-30 18:47:27Z brideout $
;
FUNCTION madGetExperiments, madurl, instcode, startyear, startmonth, startday, starthour, startmin, startsec, $
endyear, endmonth, endday, endhour, endmin, endsec, local
***********************************************************
;+
; NAME:
; madgetexperimentfiles(madurl, expid, getnondefault)
;
; PURPOSE:
; returns a list of all Madrigal Experiment Files for a given experiment id
;
; INPUTS:
; madurl - scalar string giving a fully qualified url to the Madrigal site
; expid - experiment id as returned by madgetexperiments
; getnondefault - an optional argument. If 1, get all files, including non-default. If 0 (the default)
; get only default files. In general, non-default files should be used with caution, and most users
; should set this argument to 0.
;
; OUTPUT: an array of structures representing experiment files for that experiment with the following fields:
; 1. name (string) Example '/opt/madrigal/blah/mlh980120g.001'
; 2. kindat (int) Kindat code. Example: 3001
; 3. kindatdesc (string) Kindat description: Example 'Basic Derived Parameters'
; 4. category (int) (1=default, 2=variant, 3=history, 4=real-time)
; 5. status (string)('preliminary', 'final', or any other description)
; 6. permission (int) 0 for public, 1 for private
; 7. expId - experiment id (int) of the experiment this MadrigalExperimentFile belongs in
; If no experiement files found, returns a Null Pointer (IDL does not allow empty arrays)
; EXAMPLE:
; result = madGetExperimentFiles('http://madrigal.haystack.mit.edu/madrigal', 300041, 0)
;
; $Id: madidl.html 4425 2014-07-30 18:47:27Z brideout $
;
FUNCTION madGetExperimentFiles, madurl, expId, getNonDefault
***********************************************************
;+
; NAME:
; madgetexperimentfileparameters(madurl, fullFilename)
;
; PURPOSE:
; returns a list of all measured and derivable parameters in file
;
; INPUTS:
; madurl - scalar string giving a fully qualified url to the Madrigal site
; fullFilename - full path to experiment file as returned by madGetExperimentFiles.
;
; OUTPUT: an array of structures representing parameters for that experiment file with the following fields:
; 1. mnemonic (string) Example 'dti'
; 2. description (string) Example: "F10.7 Multiday average observed (Ott)"
; 3. isError (int) 1 if error parameter, 0 if not
; 4. units (string) Example "W/m2/Hz"
; 5. isMeasured (int) 1 if measured, 0 if derivable
; 6. category (string) Example: "Time Related Parameter"
; 7. isSure (int) 1 if parameter can be found for every record, 0 if can only be found for some
; 8. isAddIncrement - 1 if additional increment, 0 if normal, -1 if unknown (this capability
; only added with Madrigal 2.5)
; EXAMPLE:
; result = madGetExperimentFileParameters('http://madrigal.haystack.mit.edu/madrigal', '/opt/madrigal/blah/mlh980120g.001')
;
; $Id: madidl.html 4425 2014-07-30 18:47:27Z brideout $
;
FUNCTION madGetExperimentFileParameters, madurl, fullFilename
***********************************************************
;+
; NAME:
; madsimpleprint(madurl, fullFilename, user_fullname, user_email, user_affiliation)
;
; PURPOSE:
; madSimplePrint prints the data in the given file is a simple ascii format.
;
; madSimplePrint prints only the parameters in the file, without filters or derived
; parameters. To choose which parameters to print, to print derived parameters, or
; to filter the data, use isprint instead.
;
; INPUTS:
; madurl - scalar string giving a fully qualified url to the Madrigal site
; fullFilename - full path to experiment file as returned by madGetExperimentFiles.
; user_fullname - full name of user making request
; user_email - email address of user making request
; user_affiliation - affiliation of user making request
;
; OUTPUT: a structure with following fields:
; 1. parameters - an array of strings giving the mnemonics of the parameters. Equal to the number of columns in data
; 2. data - a two dimensional array of double. Number of columns = number of parameters above. Number of rows
; is the number of measurments. Note that Madrigal often contains a number a measurments at the same time
; (such as when a radar makes different range measurements at the same time), so two or more rows may have the same time.
; EXAMPLE:
; result = madSimplePrint('http://madrigal.haystack.mit.edu/madrigal', '/opt/madrigal/blah/mlh980120g.001', $
; 'Bill Rideout', 'brideout@haystack.mit.edu', 'MIT')
;
; $Id: madidl.html 4425 2014-07-30 18:47:27Z brideout $
;
FUNCTION madSimplePrint, madurl, fullFilename, user_fullname, user_email, user_affiliation
***********************************************************
;+
; NAME:
; madprint(madurl, fullFilename, parms, filters, user_fullname, user_email, user_affiliation, ignore_no_data, verbose)
;
; PURPOSE:
; madPrint returns a two-dimensional array of doubles based on user-specified parmeters and filters.
;
; See madSimplePrint to print a file with all data and just those parameters in the file itself.
;
; madPrint allows you to choose which parameters to print, to print derived parameters, or
; to filter the data.
;
; INPUTS:
; madurl - scalar string giving a fully qualified url to the Madrigal site
; fullFilename - full path to experiment file as returned by madGetExperimentFiles.
; parms - Comma delimited string listing requested parameters (no spaces allowed).
; filters - Space delimited string listing filters desired, as in isprint command (see
; http://madrigal.haystack.mit.edu/madrigal/ug_commandLine.html#isprint for details)
; user_fullname - full name of user making request
; user_email - email address of user making request
; user_affiliation - affiliation of user making request
; ignore_no_data - if 0 (the default), raises error if no data, If 1, ignores no data
; verbose - if 0 (the default) no printing progress information. If 1, print progress information.
;
; OUTPUT: a two dimensional array of doubles. Number of columns = number of parameters requested. Number of rows
; is the number of measurments. Note that Madrigal often contains a number a measurments at the same time
; (such as when a radar makes different range measurements at the same time), so two or more rows may have the same time.
; EXAMPLE:
; result = madPrint('http://madrigal.haystack.mit.edu/madrigal', '/opt/madrigal/blah/mlh980120g.001', $
; 'year,month,day,hour,min,sec,gdalt,ti', 'filter=gdalt,500,600 filter=ti,1900,2000' $
; 'Bill Rideout', 'brideout@haystack.mit.edu', 'MIT')
;
; $Id: madidl.html 4425 2014-07-30 18:47:27Z brideout $
;
FUNCTION madPrint, madurl, fullFilename, parms, filters, user_fullname, user_email, user_affiliation, ignore_no_data, verbose
*********************************************************** ;+ ; NAME: ; maddownloadfile, madurl, fullFilename, outputFile, user_fullname, user_email, user_affiliation, format ; ; PURPOSE: ; downloads a Madrigal file in simple ascii format to local computer ; ; INPUTS: ; madurl - scalar string giving a fully qualified url to the Madrigal site ; fullFilename - full path to experiment file as returned by madGetExperimentFiles. ; outputFile - path to save file locally ; user_fullname - full name of user making request ; user_email - email address of user making request ; user_affiliation - affiliation of user making request ; format - 'ascii'or 'hdf5'. Default is ascii. ; ; EXAMPLE: ; maddownloadfile, 'http://madrigal.haystack.mit.edu/madrigal', '/opt/madrigal/experiments/1998/mlh/20jan98/mlh980120g.002', '/tmp/junk.hdf5', & ; 'Bill Rideout', 'brideout@haystack.mit.edu', 'MIT', 'hdf5' ; ; $Id: madidl.html 4425 2014-07-30 18:47:27Z brideout $ ; PRO maddownloadfile, madurl, fullFilename, outputFile, user_fullname, user_email, user_affiliation, format
***********************************************************
;+
; NAME:
; madglobalprint, madurl, parms, output, user_fullname, user_email, user_affiliation, startDate, endDate,
; inst, filters, kindats, expName, fileDesc
;
; PURPOSE
; madglobalprint is a procedure to search through the entire Madrigal database
; for appropriate data to print in ascii to a file
;
; INPUTS:
;
; madurl - url to homepage of site to be searched (Example:
; 'http://www.haystack.mit.edu/madrigal/'
;
; parms - a comma delimited string listing the desired Madrigal
; parameters in mnemonic form.
; (Example: 'year,month,day,hour,min,sec,gdalt,dte,te').
; Ascii space-separated data will be returned in the same
; order as given in this string. See
; http://madrigal.haystack.mit.edu/cgi-bin/madrigal/getMetadata
; "Parameter code table" for all possible parameters.
;
; output - the local file name to store the resulting ascii data.
; (Example: '/tmp/isprint.txt')
;
; user_fullname - the full user name (Example: 'Bill Rideout')
;
; user email - Example: 'brideout@haystack.mit.edu'
;
; user_affiliation - Example: 'MIT'
;
; startDate - a time in IDL Julian Date format at which to begin the search
;
; endDate - a time in IDL Julian Date format at which to end the search
;
; inst - instrument code (integer). See
; http://madrigal.haystack.mit.edu/cgi-bin/madrigal/getMetadata
; "Instrument Table" for this list. Examples: 30 for Millstone
; Hill Incoherent Scatter Radar, 80 for Sondrestrom Incoherent
; Scatter Radar
;
; Optional inputs
;
; filters - is the optional filters requested in exactly the form given in isprint
; command line (example = 'filter=gdalt,,500 filter=ti,500,1000')
; See: http://www.haystack.mit.edu/madrigal/ug_commandLine.html for details
;
; kindats - is an optional array of kindat (kinds of data) codes to accept.
; The default is a null pointer, which will accept all kindats.
;
; expName - a case insensitive string as used by strmatch that matches the experiment
; name. Default is zero-length string, which matches all experiment names.
;
; fileDesc - a case insensitive string as used by strmatch that matches the file description.
; Default is zero-length string, which matches all file descriptions.
;
; OUTPUT: Nothing.
;
; Affects: Writes results to output file
;
;
; EXAMPLE:
; madglobalprint, 'http://www.haystack.mit.edu/madrigal/', $
; 'year,month,day,hour,min,sec,gdalt,dte,te', $
; '/tmp/isprint.txt', $
; 'Bill Rideout', 'brideout@haystack.mit.edu', 'MIT', $
; julday(1,19,1998,0,0,0), julday(1,21,1998,23,59,59), 30
;
; $Id: madidl.html 4425 2014-07-30 18:47:27Z brideout $
;
pro madglobalprint, madurl, parms, output, user_fullname, user_email, user_affiliation, startDate, endDate, $
inst, filters, kindats, expName, fileDesc
***********************************************************
;+
; NAME:
; madcalculator(madurl, year, month, day, hour, min, sec, startLat, endLat, stepLat,
; startLong, endLong, stepLong, startAlt, endAlt, stepAlt, parms)
;
; PURPOSE:
; madcalculator calculates requested derived parameter for a given time and grid of spatial points
;
; INPUTS:
; madurl - scalar string giving a fully qualified url to the Madrigal site
; year, month, day, hour, min, sec - time at which to run calculation
; startLat - Starting geodetic latitude, -90 to 90 (float)
; endLat - Ending geodetic latitude, -90 to 90 (float)
; stepLat - Latitude step (0.1 to 90) (float)
; startLong - Starting geodetic longitude, -180 to 180 (float)
; endLong - Ending geodetic longitude, -180 to 180 (float)
; stepLong - Longitude step (0.1 to 180) (float)
; startAlt - Starting geodetic altitude, >= 0 (float)
; endAlt - Ending geodetic altitude, > 0 (float)
; stepAlt - Altitude step (>= 0.1) (float)
; parms - comma delimited string of Madrigal parameters desired
;
; OUTPUT: a two dimensional array of doubles. Number of columns = 3 + number of parameters requested, because the
; first three parameters are always geodetic latitude, longitude, and altitude. Number of rows
; is the number of latitudes * number of longitudes * number of altitudes.
; EXAMPLE:
; result = madcalculator(1999,2,15,12,30,0,45,55,5,-170,-150,10,200,200,0,'bmag,bn')
;
; $Id: madidl.html 4425 2014-07-30 18:47:27Z brideout $
;
FUNCTION madcalculator, madurl, year, month, day, hour, min, sec, startLat, endLat, stepLat, $
startLong, endLong, stepLong, startAlt, endAlt, stepAlt, parms
; Simple example of all methods in the remote IDL interface to Madrigal (madidl)
;
; $Id: madidl.html 4425 2014-07-30 18:47:27Z brideout $
PRO example_madidl
; these are the instrument codes for the Millstone ISR. Run the function madGetAllInstruments
; to see a list of all instruments available in Madrigal
instList = [30,31,32]
; run this test for two different sites (Haystack and SRI), using the test files in each site
siteList = ['http://madrigal.haystack.mit.edu/madrigal', $
'http://isr.sri.com/madrigal/' ]
for i=0, n_elements(siteList)-1 do begin
site = siteList[i]
print, 'Testing Madrigal url ', site
; the highest level of Madrigal data is the instrument - here we get a list of all available instruments
; See madgetallinstruments.pro for a definition of the instrument structure returned in an array
result = madGetAllInstruments(site)
print, 'madGetAllInstruments returned', result
print, ""
; the next level of Madrigal is the experiment. An experiment has only one instrument associated with
; it. In this case, we ask for all experiments in Jan 1998 for any Millstone ISR radar. sonce this
; is a standard test file, its found at both Millstone and SRI
; See madgetexperiments.pro for a definition of the experiment structure returned in an array
result = madGetExperiments(site, instList, 1998,1,19,0,0,0, 1998,1,20,23,59,59,1)
print, 'madGetExperiments returned ', result
print, ""
; verify at least one returned - note a null pointer is returned since IDL does not support empty arrays
resultSize = size(result)
if (resultSize[1] eq 10) then continue ; no experiments found
; next we get an array of files in that experiment. See madgetexperimentfiles.pro for a definition
; of the experimentfile structure returned in an array
expId = long64(result[0].strid)
result = madGetExperimentFiles(site, expid, 1)
print, 'madGetExperimentFiles returned ', result
print, ""
; verify at least one returned - note a null pointer is returned since IDL does not support empty arrays
resultSize = size(result)
if (resultSize[1] eq 10) then continue ; no experiment files found
; next we want to get a list of all parameters. Note Madrigal allows access to both parameters in the file
; (called measured parameters) and parameters derivable form the measured ones (called derived parameters).
; See madgetexperimentfileparameters.pro for a definition of the parmeter structure returned in an array
fullFilename = result[0].name
result = madGetExperimentFileParameters(site, fullFilename)
print, 'madGetExperimentFileParameters returned ', result
print, ""
; the next method effectively downloads the file into an IDL 2-D array. It only downloads measured parameters
; in the file. See more advanced method madPrint to choose measured or derived parameters and/or to filter data.
; See madsimpleprint.pro for details of how the data is returned
result = madSimplePrint(site, fullFilename, 'Bill Rideout', 'brideout@haystack.mit.edu', 'MIT')
print, 'size of data returned by madSimplePrint is ', size(result.data)
print, 'madSimplePrint parameters are ', result.parameters
print, ""
; the next method is similar to madsimpleprint, except the user chooses which measured or derived parameters
; to download, and can filter data using the filter string as discussed in
; http://madrigal.haystack.mit.edu/madrigal/ug_commandLine.html#isprint
; See madprint.pro for details of how the data is returned
result = madPrint(site, fullFilename, 'year,month,day,hour,min,sec,gdalt,ti', 'filter=recno,5,6 filter=ti,1000,', $
'Bill Rideout', 'brideout@haystack.mit.edu', 'MIT')
print, 'size of data returned by madPrint is ', size(result)
print, ""
; the next method is similar to madSimplePrint, except that it downloads the data from
; a file into a simple column delimited formated file on your local computer
maddownloadfile, site, fullFilename, '/tmp/junk5.txt', 'Bill Rideout', 'brideout@haystack.mit.edu', 'MIT'
print, 'file has been downloaded to /tmp/junk5.txt'
; The next method is used to run the Madrigal derivation engine for any random time and array of geodetic points in space
; In other words, things such as Magnetic field and geophysicals indices can be calculated without a Madrigal file.
; See madCalculator.pro for details of how the data is returned
result = madCalculator(site, 1999,2,15,12,30,0,45,55,5,-170,-150,10,200,200,0,'bmag,bn')
print, 'madCalculator returned ', result
print, ""
; The next procedure madglobalPrint allows you to get data from multiple files in a particular Madrigal datbase
; at once. Since it can return very large data sets, it writes to a file rather than an IDL array
madglobalPrint, site, 'year,month,day,hour,min,sec,gdalt,dte,te', $
'/tmp/isprint.txt', 'Bill Rideout', 'brideout@haystack.mit.edu', 'MIT', $
julday(1,19,1998,0,0,0), julday(1,21,1998,23,59,59), 30, $
'', [3408, 3409, 3410], '*world*'
; The next procedure madglobaldownload allows you to download multiple files in a particular Madrigal datbase
; at once
print, 'Using madglobaldownload to get files in ascii format'
madglobaldownload, 'http://www.haystack.mit.edu/madrigal/', $
'/tmp', $
'Bill Rideout', 'brideout@haystack.mit.edu', 'MIT', $
julday(1,19,1998,0,0,0), julday(1,21,1998,23,59,59), 30, $
[3408, 3409, 3410]
; Next madglobaldownload is called to download Hdf5 files
print, 'Using madglobaldownload to get files in hdf5 format'
madglobaldownload, 'http://www.haystack.mit.edu/madrigal/', $
'/tmp', $
'Bill Rideout', 'brideout@haystack.mit.edu', 'MIT', $
julday(1,19,1998,0,0,0), julday(1,21,1998,23,59,59), 30, $
[3408, 3409, 3410], 'hdf5'
endfor ; next Madrigal site
; an example of getting data from a site from a different Madrigal site - search for Poker Flat data from Millstone site
site = 'http://madrigal.haystack.mit.edu/madrigal'
instList = [61] ; Poker Flat ISR
result = madGetExperiments(site, instList, 2008,4,1,0,0,0, 2008,4,30,23,59,59,0)
print, 'madGetExperiments returned ', result
; handle a non-local experiment
if (result[0].isLocal eq 0) then begin
site = result[0].madrigalUrl
; call madGetExperiments again with local url
result = madGetExperiments(site, instList, 2008,4,1,0,0,0, 2008,4,30,23,59,59,0)
endif
expId = long64(result[0].strid)
result = madGetExperimentFiles(site, expid, 1)
print, 'madGetExperimentFiles returned ', result
print, ""
END
|
|
|
Remote access using python | Doc home | Madrigal home |