|
|
|
Remote access using Matlab | Doc home | Madrigal home |
This page describes the remote Matlab API, and gives two examples of using this API. The first example uses all the basic methods, and outputs text. The second example creates a simple pcolor plot.
The remote Matlab API is organized in the same way as the Madrigal data model, from Instrument at the highest level, down to the level of data values. Readers who are not familiar with the Madrigal data model should read the material in that section before proceeding with this tutorial.
This API and example have been tested on both Windows and Linux, and require only access to the internet and Matlab 5 or greater to run. It is available for download here.
A good way to learn how to use this Matlab API is to run this example.
*********************************************************** function cgiUrl = getMadrigalCgiUrl(url) getMadrigalCgiUrl parse the main madrigal page to get the cgi url input: url to Madrigal output: cgi url for that Madrigal Site Note: parses the homepage for the accessData link
***********************************************************
function instArray = getInstrumentsWeb(cgiurl)
getInstrumentsWeb returns an array of instrument structs of instruments found on remote Madrigal server.
inputs: cgiurl (string) to Madrigal site cgi directory
(Example: 'http://madrigal.haystack.mit.edu/cgi-bin/madrigal/')
Note that method getMadrigalCgiUrl converts homepage url into cgiurl.
output:
instArray - array of instrument structs found
instrument struct has the fields:
instrument.name (string) Example: 'Millstone Hill Incoherent Scatter Radar'
instrument.code (int) Example: 30
instrument.mnemonic (3 char string) Example: 'mlh'
instrument.latitude (double) Example: 45.0
instrument.longitude (double) Example: 110.0
instrument.altitude (double) Example: 0.015 (km)
instrument.category (string) Instrument category description. May be
if Madrigal 2.6 or earlier.
Raises error if unable to return instrument array
Example:
getInstrumentsWeb('http://madrigal.haystack.mit.edu/cgi-bin/madrigal/')
***********************************************************
function expArray = getExperimentsWeb(cgiurl, instCodeArray, starttime, endtime, localFlag)
getExperimentsWeb returns an array of experiment structs given input filter arguments from a remote Madrigal server.
Inputs:
1. cgiurl (string) to Madrigal site cgi directory
(Example: 'http://madrigal.haystack.mit.edu/cgi-bin/madrigal/')
Note that method getMadrigalCgiUrl converts homepage url into cgiurl.
2. instCodeArray - a 1 X N array of ints containing selected instrument codes. Special value of 0 selects all instruments.
3. starttime - Matlab datenum double (must be UTC)
4. endtime - Matlab datenum double (must be UTC)
5. localFlag - 1 if local experiments only, 0 if all experiments
Returns a startime sorted array of Experiment struct (May be empty):
experiment.id (int) Example: 10000111
experiment.url (string) Example: 'http://madrigal.haystack.mit.edu/cgi-bin/madtoc/1997/mlh/03dec97'
Deprecated url used only in metadata. To see real url, use realUrl
field described below
experiment.name (string) Example: 'Wide Latitude Substorm Study'
experiment.siteid (int) Example: 1
experiment.sitename (string) Example: 'Millstone Hill Observatory'
experiment.instcode (int) Code of instrument. Example: 30
experiment.instname (string) Instrument name. Example: 'Millstone Hill Incoherent Scatter Radar'
experiment.starttime (double) Matlab datenum of experiment start
experiment.endtime (double) Matlab datenum of experiment end
experiment.isLocal (int) 1 if local, 0 if not
experiment.madrigalUrl (string) - home url of Madrigal site with this
experiment. Example 'http://madrigal.haystack.mit.edu/madrigal'
experiment.PI - experiment principal investigator. May be unknown for
Madrigal 2.5 and earlier sites.
experiment.PIEmail - PI email. May be unknown for Madrigal 2.6 or
earlier.
realUrl - real url to experiment valid for web browser
Raises error if unable to return experiment array
Example: expArray = getExperimentsWeb('http://madrigal.haystack.mit.edu/cgi-bin/madrigal/', ...
30, datenum('01/01/1998'), datenum('12/31/1998'), 1);
Note that if the returned
experiment is not local, the experiment.id will be -1. This means that you
will need to call getExperimentsWeb a second time with the cgiurl of the
non-local experiment (getCgiurlForExperiment(experiment.madrigalUrl)). This is because
while Madrigal sites share metadata about experiments, the real experiment ids are only
known by the individual Madrigal sites. See testMadmatlab.m
for an example of this.
***********************************************************
function cgiurl = getCgiurlForExperiment(experiment)
getCgiurlForExperiment returns cgiurl of experiment.url as returned by getExperimentsWeb.
inputs: experiment struct as returned by getExperimentsWeb or getExperiments.
output:
cgiurl of experiment
Simply truncates experiment.url to remove /madtoc///
Example: If expArray is the value returned in the getExperimentsWeb example, and
expArray(1).url = 'http://madrigal.haystack.mit.edu/cgi-bin/madrigal/madtoc/1998/mlh/07jan98', then
getCgiurlForExperiment(expArray(1))
returns:
'http://madrigal.haystack.mit.edu/cgi-bin/madrigal/'
***********************************************************
function expFileArray = getExperimentFilesWeb(cgiurl, experimentId)
getExperimentFilesWeb returns an array of experiment file structs given experiment id from a remote Madrigal server.
Note that it is assumed that experiment is local to cgiurl. If not,
empty list will be returned. Use getCgiurlForExperiment to get the correct
cgiurl for any given experiment struct.
Inputs:
1. cgiurl (string) to Madrigal site cgi directory that has that
experiment.
(Example: 'http://madrigal.haystack.mit.edu/cgi-bin/madrigal/')
Note that method getMadrigalCgiUrl converts homepage url into cgiurl.
2. experiment id (int) as returned by getExperiments or
getExperimentsWeb
Return array of Experiment File struct (May be empty):
file.name (string) Example '/opt/mdarigal/blah/mlh980120g.001'
file.kindat (int) Kindat code. Example: 3001
file.kindatdesc (string) Kindat description: Example 'Basic Derived Parameters'
file.category (int) (1=default, 2=variant, 3=history, 4=real-time)
file.status (string)('preliminary', 'final', or any other description)
file.permission (int) 0 for public, 1 for private
Raises error if unable to return experiment file array
Example: expFileArray =
getExperimentFilesWeb('http://madrigal.haystack.mit.edu/cgi-bin/madrigal/', 10001686);
***********************************************************
function parmArray = getParametersWeb(cgiurl, filename)
getParametesWeb returns an array of parameter structs given filename from a remote Madrigal server.
Note that it is assumed that filename is local to cgiurl. If not,
empty list will be returned.
Inputs:
1. cgiurl (string) to Madrigal site cgi directory that has that
filename.
(Example: 'http://madrigal.haystack.mit.edu/cgi-bin/madrigal/')
Note that method getMadrigalCgiUrl converts homepage url into cgiurl.
2. filename (string) as returned by getExperimentFiles or
getExperimentFilesWeb
Return array of Parameter struct:
parameter.mnemonic (string) Example 'dti'
parameter.description (string) Example:
"F10.7 Multiday average observed (Ott) - Units: W/m2/Hz"
parameter.isError (int) 1 if error parameter, 0 if not
parameter.units (string) Example "W/m2/Hz"
parameter.isMeasured (int) 1 if measured, 0 if derivable
parameter.category (string) Example: "Time Related Parameter"
parameter.isSure (int) 1 if can be found for all records, 0 if only
for some records (implies not all records have same measured
parameters)
Raises error if unable to return parameter array
Example: parmArray = getParametersWeb('http://madrigal.haystack.mit.edu/cgi-bin/madrigal/', ...
'/opt/madrigal/experiments/1998/mlh/07jan98/mil980107g.001')
***********************************************************
function records = isprintWeb(cgiUrl, file, parms, user_fullname, user_email, user_affiliation, filters, missing, assumed, knownbad)
isprintWeb Create an isprint-like 3D array of doubles via a command similar to the isprint command-line application, but access data via the web
The calling syntax is:
[records] = isprintWeb(cgiurl, file, parms, user_fullname, user_email, user_affiliation, [filters, [missing, [assumed, [knownbad] ] ] ])
where
cgiurl (string) to Madrigal site cgi directory that has that
filename.
(Example: 'http://madrigal.haystack.mit.edu/cgi-bin/madrigal/')
Note that method getMadrigalCgiUrl converts homepage url into cgiurl.
file is path to file
(example = '/home/brideout/data/mlh980120g.001')
parms is the desired parameters in the form of a comma-delimited
string of Madrigal mnemonics (example = 'gdlat,ti,dti')
user_fullname - is user name (string)
user_email - is user email address (string)
user_affiliation - is user affiliation (string)
filters is the optional filters requested in exactly the form given in isprint
command line (example = 'time1=15:00:00 date1=01/20/1998
time2=15:30:00 date2=01/20/1998 filter=ti,500,1000')
See: http://madrigal.haystack.mit.edu/madrigal/ug_commandLine.html for details
missing is an optional double to represent missing values. Defaults to NaN
assumed is an optional double to represent assumed values. Defaults to NaN
knownbad is an optional double to represent knownbad values. Defaults to NaN
The returned records is a three dimensional array of double with the dimensions:
[Number of rows, number of parameters requested, number of records]
If error or no data returned, will return error explanation string instead.
Example: data = isprintWeb('http://madrigal.haystack.mit.edu/cgi-bin/madrigal/', ...
'/opt/madrigal/experiments/1998/mlh/07jan98/mil980107g.001', ...
'gdlat,ti,dti', ...
'Bill Rideout', 'wrideout@haystack.mit.edu', 'MIT');
For now avoids limits with urlread by only asking for maxRecs records at once. Repeatedly
calls isprintUnguarded, which is a method without any additional filtering.
$Id: rt_matlab.html 4434 2014-07-30 19:55:26Z brideout $
***********************************************************
function records = isprintUnguarded(cgiUrl, file, parms, user_fullname, user_email, user_affiliation, filters, missing, assumed, knownbad)
isprintUnguarded - Create an isprint-like 3D array of doubles via a command similar to the isprint command-line application, but access data via the web
This method differs from isprintWeb in that no protection against
downloading too much data is provided. If too much data is requested,
urlread will fail.
The calling syntax for this method is:
[records] = isprintUnguarded(cgiurl, file, parms, user_fullname, user_email, user_affiliation, [filters, [missing, [assumed, [knownbad] ] ] ])
where
cgiurl (string) to Madrigal site cgi directory that has that
filename.
(Example: 'http://madrigal.haystack.mit.edu/cgi-bin/madrigal/')
Note that method getMadrigalCgiUrl converts homepage url into cgiurl.
file is path to file
(example = '/home/brideout/data/mlh980120g.001')
parms is the desired parameters in the form of a comma-delimited
string of Madrigal mnemonics (example = 'gdlat,ti,dti')
user_fullname - is user name (string)
user_email - is user email address (string)
user_affiliation - is user affiliation (string)
filters is the optional filters requested in exactly the form given in isprint
command line (example = 'time1=15:00:00 date1=01/20/1998
time2=15:30:00 date2=01/20/1998 filter=ti,500,1000')
See: http://madrigal.haystack.mit.edu/madrigal/ug_commandLine.html for details
missing is an optional double to represent missing values. Defaults to NaN
assumed is an optional double to represent assumed values. Defaults to NaN
knownbad is an optional double to represent knownbad values. Defaults to NaN
The returned records is a three dimensional array of double with the dimensions:
[Number of rows, number of parameters requested, number of records]
If error or no data returned, will return error explanation string instead.
Example: data = isprintUnguarded('http://madrigal.haystack.mit.edu/cgi-bin/madrigal/', ...
'/opt/madrigal/experiments/1998/mlh/07jan98/mil980107g.001', ...
'gdlat,ti,dti', ...
'Bill Rideout', 'wrideout@haystack.mit.edu', 'MIT');
Unlike isprintWeb, no effort is made to protect the user from
requesting too more data than the Matlab method urlread can handle
$Id: rt_matlab.html 4434 2014-07-30 19:55:26Z brideout $
***********************************************************
function result = madDownloadFile(cgiurl, fullFilename, outputFile, user_fullname, user_email, user_affiliation, format )
madDownloadFile downloads a Madrigal file to local computer in various
formats
The calling syntax is:
result = madDownloadFile(cgiurl, fullFilename, outputFile, [ format ] )
Inputs:
1. cgiurl (string) to Madrigal site cgi directory
(Example: 'http://madrigal.haystack.mit.edu/cgi-bin/madrigal/')
Note that method getMadrigalCgiUrl converts homepage url into
cgiurl.
2. fullFilename - file to download as returned by getExperimentFilesWeb.m
3. outputFile - name to save new file to
4. user_fullname - is user name (string)
5. user_email - is user email address (string)
6. user_affiliation - is user affiliation (string)
7. format - one of the following strings:
'simple', 'hdf5', 'cedar_ascii'
'simple' is the default if not specified. It
is recommended that users choose either 'simple' or 'hdf5'
'simple' is an ascii space delimited column
format. See Cedar
standard for description of complex cedar_ascii format
(not recommended)
'hdf5 option will only work with Madrigal 2.6 or greater'
$Id: rt_matlab.html 4434 2014-07-30 19:55:26Z brideout $
result = -1;
***********************************************************
function record = madCalculatorWeb(cgiUrl, ...
time, ...
startLat, ...
endLat, ...
stepLat, ...
startLong, ...
endLong, ...
stepLong, ...
startAlt, ...
endAlt, ...
stepAlt, ...
parms)
madCalculatorWeb Create a matrix of doubles via a the Madrigal derivation engine for a time and range of lat, long, and alt
The calling syntax is:
[record] = madCalculatorWeb(cgiurl, time, startLat, endLat, stepLat, startLong, endLong, stepLong,
startAlt, endAlt, stepAlt, parms)
where
cgiurl (string) to Madrigal site cgi directory that has that
filename.
(Example: 'http://madrigal.haystack.mit.edu/cgi-bin/madrigal/')
Note that method getMadrigalCgiUrl converts homepage url into cgiurl.
time - Matlab datenum double (must be UTC) 7. startLat - Starting geodetic latitude, -90 to 90 (required)
endLat - Ending geodetic latitude, -90 to 90 (required)
stepLat - Latitude step (0.1 to 90) (required)
startLong - Starting geodetic longitude, -180 to 180 (required)
endLong - Ending geodetic longitude, -180 to 180 (required)
stepLong - Longitude step (0.1 to 180) (required)
startAlt - Starting geodetic altitude, >= 0 (required)
endAlt - Ending geodetic altitude, > 0 (required)
stepAlt - Altitude step (>= 0.1) (required)
parms is the desired parameters in the form of a comma-delimited
string of Madrigal mnemonics (example = 'gdlat,ti,dti')
The returned record is a matrix of doubles with the dimensions:
[(num lat steps * num long steps * num alt steps), 3 + num of parms]
The first three columns will always be lat, long, and alt, so there are three
additional columns to the number of parameters requested via the parms argument.
If error or no data returned, will return error explanation string instead.
Example: result = madCalculatorWeb('http:/grail.haystack.mit.edu/cgi-bin/madrigal', ...
now,45,55,5,45,55,5,200,300,50,'bmag,bn');
***********************************************************
function [] = globalIsprint(url, ...
parms, ...
output, ...
user_fullname, ...
user_email, ...
user_affiliation, ...
startTime, ...
endTime, ...
inst, ...
filters, ...
kindats, ...
expName, ...
fileDesc)
globalIsprint is a script to search through the entire Madrigal database
for appropriate data to print in ascii to a file
Inputs:
url - url to homepage of site to be searched (Example:
'http://madrigal.haystack.mit.edu/madrigal/'
parms - a comma delimited string listing the desired Madrigal
parameters in mnemonic form.
(Example: 'year,month,day,hour,min,sec,gdalt,dte,te').
Ascii space-separated data will be returned in the same
order as given in this string. See
http://madrigal.haystack.mit.edu/cgi-bin/madrigal/getMetadata
"Parameter code table" for all possible parameters.
output - the local file name to store the resulting ascii data.
(Example: '/tmp/isprint.txt')
user_fullname - the full user name (Example: 'Bill Rideout')
user email - Example: 'brideout@haystack.mit.edu'
user_affiliation - Example: 'MIT'
startTime - a Matlab time to begin search at. Example:
datenum('20-Jan-1998 00:00:00') Time in UT
endTime - a Matlab time to end search at. Example:
datenum('21-Jan-1998 23:59:59') Time in UT
inst - instrument code (integer). See
http://madrigal.haystack.mit.edu/cgi-bin/madrigal/getMetadata
"Instrument Table" for this list. Examples: 30 for Millstone
Hill Incoherent Scatter Radar, 80 for Sondrestrom Incoherent
Scatter Radar
Optional inputs
filters - is the optional filters requested in exactly the form given in isprint
command line (example = 'filter=gdalt,,500 filter=ti,500,1000')
See: http://madrigal.haystack.mit.edu/madrigal/ug_commandLine.html for details
kindats - is an optional array of kindat (kinds of data) codes to accept.
The default is an empty array, which will accept all kindats.
expName - a case insensitive regular expression that matches the experiment
name. Default is zero-length string, which matches all experiment names.
fileDesc - a case insensitive regular expression that matches the file description.
Default is zero-length string, which matches all file descriptions.
Returns: Nothing.
Affects: Writes results to output file
Example: globalIsprint('http://madrigal.haystack.mit.edu/madrigal/', ...
'year,month,day,hour,min,sec,gdalt,dte,te', ...
'/tmp/isprint.txt', ...
'Bill Rideout', ...
'brideout@haystack.mit.edu', ...
'MIT', ...
datenum('20-Jan-1998 00:00:00'), ...
datenum('21-Jan-1998 23:59:59'), ...
30);
$Id: rt_matlab.html 4434 2014-07-30 19:55:26Z brideout $
***********************************************************
function [] = globalDownload(url, ...
outputDir, ...
user_fullname, ...
user_email, ...
user_affiliation, ...
format, ...
startTime, ...
endTime, ...
inst, ...
kindats, ...
expName, ...
fileDesc)
globalDownload is a script to search through the entire Madrigal database
for appropriate files in ascii or hdf5 format to store locally
Inputs:
url - url to homepage of site to be searched (Example:
'http://madrigal.haystack.mit.edu/madrigal/'
outputDir - the local directory to store the downloaded files.
(Example: '/tmp/isprint.txt')
user_fullname - the full user name (Example: 'Bill Rideout')
user email - Example: 'brideout@haystack.mit.edu'
user_affiliation - Example: 'MIT'
format - either "ascii" or "hdf5" Ascii is simple column
delimited acsii
startTime - a Matlab time to begin search at. Example:
datenum('20-Jan-1998 00:00:00') Time in UT
endTime - a Matlab time to end search at. Example:
datenum('21-Jan-1998 23:59:59') Time in UT
inst - instrument code (integer). See
http://madrigal.haystack.mit.edu/cgi-bin/madrigal/getMetadata
"Instrument Table" for this list. Examples: 30 for Millstone
Hill Incoherent Scatter Radar, 80 for Sondrestrom Incoherent
Scatter Radar
Optional inputs
kindats - is an optional array of kindat (kinds of data) codes to accept.
The default is an empty array, which will accept all kindats.
expName - a case insensitive regular expression that matches the experiment
name. Default is zero-length string, which matches all experiment names.
fileDesc - a case insensitive regular expression that matches the file description.
Default is zero-length string, which matches all file descriptions.
Returns: Nothing.
Affects: Writes downloaded files to outputDir
Example: globalDownload('http://madrigal.haystack.mit.edu/madrigal/', ...
'/tmp/download', ...
'Bill Rideout', ...
'brideout@haystack.mit.edu', ...
'MIT', ...
'ascii', ...
datenum('20-Jan-1998 00:00:00'), ...
datenum('21-Jan-1998 23:59:59'), ...
30);
$Id: rt_matlab.html 4434 2014-07-30 19:55:26Z brideout $
% demo program of madmatlab running on a pc or linux
madurl = 'http://madrigal.haystack.mit.edu/madrigal';
cgiurl = getMadrigalCgiUrl(madurl)
'List all instruments, and their latitudes and longitudes:'
instArray = getInstrumentsWeb(cgiurl);
for i = 1:length(instArray)
[s,errmsg] = sprintf('Instrument: %s, at lat %f and long %f', ...
instArray(i).name, ...
instArray(i).latitude, ...
instArray(i).longitude);
s
end
% now list all experiments from local Madrigal site with mlh (code 30) in
% 1998 - should be data if default files installed...
startdate = datenum('01/01/1998');
enddate = datenum('12/31/1998');
expArray = getExperimentsWeb(cgiurl, 30, startdate, enddate, 1);
for i = 1:length(expArray)
[s,errmsg] = sprintf('Experiment name: %s, at url %s and id %i', ...
expArray(i).name, ...
expArray(i).url, ...
expArray(i).id);
s
end
% now list all files in the first experiment
expFileArray = getExperimentFilesWeb(cgiurl, expArray(1).id);
for i = 1:length(expFileArray)
[s,errmsg] = sprintf('File name: %s, with kindat %i', ...
expFileArray(i).name, ...
expFileArray(i).kindat);
s
end
% now first 2 parameters in the last file
parmArray = getParametersWeb(cgiurl, expFileArray(end).name)
for i = 1:10
[s,errmsg] = sprintf('Parameter mnemonic: %s, description "%s" -- isMeasured = %i', ...
parmArray(i).mnemonic, ...
parmArray(i).description, ...
parmArray(i).isMeasured);
s
end
% run isprintWeb for that file for first two parameters
parmStr = sprintf('%s,%s', parmArray(1).mnemonic, parmArray(2).mnemonic);
data = isprintWeb(cgiurl, expFileArray(1).name, parmStr, 'Bill Rideout', 'wrideout@haystack.mit.edu', 'MIT');
% print first 10 records
data(:,:,1:10)
% download that data file in simple format
result = madDownloadFile(cgiurl, expFileArray(1).name, '/tmp/junk.txt', 'Bill Rideout', 'brideout@haystack.mit.edu', 'MIT');
'downloaded file to /tmp/junk.txt'
% run globalIsprint, which can gather data from multiple files at once
globalIsprint('http://madrigal.haystack.mit.edu/madrigal/', ...
'year,month,day,hour,min,sec,gdalt,dte,te', ...
'/tmp/isprint.txt', ...
'Bill Rideout', 'brideout@haystack.mit.edu', 'MIT', ...
datenum('20-Jan-1998 00:00:00'), datenum('21-Jan-1998 23:59:59'), 30);
'globalIsprint output saved to /tmp/isprint.txt'
% run globalDownload, which can download multiple files at once
globalDownload('http://madrigal.haystack.mit.edu/madrigal/', ...
'/tmp', ...
'Bill Rideout', ...
'brideout@haystack.mit.edu', ...
'MIT', ...
'ascii', ...
datenum('20-Jan-1998 00:00:00'), ...
datenum('21-Jan-1998 23:59:59'), ...
30);
'globalDownload saved files to /tmp'
% madCalculatorWeb runs the Madrigal derivation engine for any point
data = madCalculatorWeb(cgiurl, datenum(1999,2,15,12,30,0), 45,55,5,-170,-150,10,200,200,0,'sdwht,kp');
'madCalculator output'
% print data
data
'The following is an example of searching for non-local experiments'
% 61 is the instrument id of Poker Flat ISR - so this will return an
% experiment not from the Millstone Madrigal site
startdate = datenum('04/01/2008');
enddate = datenum('04/30/2008');
expArray = getExperimentsWeb(cgiurl, 61, startdate, enddate, 0);
% calling this now would fail: expFileArray = getExperimentFilesWeb(cgiurl, expArray(1).id);
% Instead, get the cgiurl of the non-local experiment
if (expArray(1).isLocal == 0)
cgiurl = getMadrigalCgiUrl(expArray(1).madrigalUrl)
expArray = getExperimentsWeb(cgiurl, 61, startdate, enddate, 0);
end
% now you can get the files
expFileArray = getExperimentFilesWeb(cgiurl, expArray(1).id);
for i = 1:length(expFileArray)
[s,errmsg] = sprintf('File name: %s, with kindat %i', ...
expFileArray(i).name, ...
expFileArray(i).kindat);
s
end
|
|
|
Remote access using Matlab | Doc home | Madrigal home |