Remote access using python

Doc home

Madrigal home

Remote access using python

This page describes the remote Python API, and gives some examples of using this API. These examples have been tested on both Windows and Linux, and require only access to the internet and python 2.3 to run. It is available for download here.

The remote Python API is organized in the same way as the Madrigal data model, from Instrument at the highest level, down to the level of data values. Readers who are not familiar with the Madrigal data model should read the material in that section before proceeding with this tutorial.

The basic object in the remote Python API is the MadrigalData, found in the madrigalWeb module. To initialize MadrigalData requires only the url of the home page on any Madrigal 2.3 (or above) site as an argument. Calling the methods of this object will return all possible information from one Madrigal site. The other objects in madrigalWeb are simply there to hold returned information - for example, the MadrigalExperiment object holds information about one experiment.

MadrigalData has the following methods:

• getAllInstruments - returns a list of all MadrigalInstrument objects at all Madrigal sites
• getExperiments - returns a list of all MadrigalExperiment objects that meet the criteria you set at any Madrigal site
• getExperimentFiles - returns a list of all default MadrigalExperimentFile objects for a given experiment id
• getExperimentFileParameters - returns a list of all measured and derivable parameters in file
• downloadFile - downloadFile will download a Cedar file in the specified format.
• simplePrint - prints the data in the given file is a simple ascii format.
• isprint - returns as a string the isprint output given file, parms, filters without headers or summary. This is the main method that accesses the raw data in a Madrigal data file.
• madCalculator - returns derived parameters for a range of latitudes, longitudes, and altitudes at a given time. Note that this method does not returned measured values from a file - use isprint for that. Instead, it is useful for accessing parameters available via the Madrigal derivation engine, such as magnetic field or MSIS parameters.
• madCalculator2 - madCalculator2 is similar to madCalculator, except that a random collection of points in space can be specified, rather than a grid of points.
• madCalculator3 - madCalculator3 is similar to madCalculator, except that multiple times can be specified, where each time can have its own unique spatial positions and 1D and 2D parms.
• madTimeCalculator - is similar to madCalculator, except that it returns data from a range of times, but only returns parameters such as Kp that are independent of position.
• radarToGeodetic - converts arrays of az, el, and ranges to geodetic locations.
• geodeticToRadar - converts arrays of points in space to az, el, and range.

See the Madrigal Python API reference guide for complete documentation.

Two applications written with the remote Python API follow. The first is a simple regression test that is run to test web services when Madrigal is installed. The second is a script that downloads realtime data from any desired Madrigal site.

Simple regression test

This simple script calls the following MadrigalData methods:

• getAllInstruments
• getExperiments
• getExperimentFiles
• downloadFile
• simplePrint
• isprint
• madCalculator

This example also shows how to get data from a different Madrigal site than the one you start with.

To use this regression test, cd to the examples directory in the installation directory, and type:

python exampleMadrigalWebServices.py

"""exampleMadrigalWebServices.py runs an example of the Madrigal Web Services interface
   for a given Madrigal server.

   usage:

   python exampleMadrigalWebServices.py 



"""

# $Id: rt_python.html 4115 2012-12-03 19:03:31Z brideout $

import madrigalWeb.madrigalWeb

# constants
user_fullname = 'Bill Rideout - automated test'
user_email = 'brideout@haystack.mit.edu'
user_affiliation = 'MIT Haystack'

madrigalUrl = 'http://madrigal.haystack.mit.edu/madrigal'


testData = madrigalWeb.madrigalWeb.MadrigalData(madrigalUrl)



print 'Example of call to getAllInstruments'
instList = testData.getAllInstruments()
# print out Millstone
for inst in instList:
    if inst.code == 30:
        print (str(inst) + '\n')
        

print 'Example of call to getExperiments'
expList = testData.getExperiments(30, 1998,1,19,0,0,0,1998,1,22,0,0,0)
for exp in expList:
    # should be only one
    print (str(exp) + '\n')


print 'Example of call to getExperimentFiles'
fileList = testData.getExperimentFiles(expList[0].id)
for thisFile in fileList:
    if thisFile.category == 1:
        print (str(thisFile.name) + '\n')
        thisFilename = thisFile.name
        break
    
print 'Example of downloadFile - simple and hdf5 formats:'
result = testData.downloadFile(thisFilename, "/tmp/test.txt", user_fullname, user_email, user_affiliation, "simple")
result = testData.downloadFile(thisFilename, "/tmp/test.hdf5", user_fullname, user_email, user_affiliation, "simple")

print 'Example of simplePrint - only first 1000 characters printed'
result = testData.simplePrint(thisFilename, user_fullname, user_email, user_affiliation)
print result[:1000]
print

print 'Example of call to getExperimentFileParameters - only first 10 printed'
fileParms = testData.getExperimentFileParameters(thisFilename)
for i in range(10):
    print fileParms[i]
print


print 'Example of call to isprint (prints data)'
print(testData.isprint(thisFilename,
                       'gdalt,ti',
                       'filter=gdalt,500,600 filter=ti,1900,2000',
                       user_fullname, user_email, user_affiliation))


print 'Example of call to madCalculator (gets derived data at any time)'
result = testData.madCalculator(1999,2,15,12,30,0,45,55,5,-170,-150,10,200,200,0,'sdwht,kp')
for line in result:
    for value in line:
        print ('%8.2e ' % (value))
    print('\n')

print 'Example of searching all Madrigal sites for an experiment - here we search for PFISR data'
expList = testData.getExperiments(61,2008,4,1,0,0,0,2008,4,30,0,0,0,local=0)
print expList[0]

print 'Since this experiment is not local (note the experiment id = -1), we need to create a new MadrigalData object to get it'
testData2 = madrigalWeb.madrigalWeb.MadrigalData(expList[0].madrigalUrl)

print 'Now repeat the same calls as above to get PFISR data from the SRI site'
expList2 = testData2.getExperiments(61,2008,4,1,0,0,0,2008,4,30,0,0,0,local=1)
print 'This is a PFISR experiment'
print expList2[0]

Script to download realtime data from Madrigal

The following is a demonstration script that shows how real-time data can be imported from any Madrigal site that is updated on a real-time basis.

In this example, data is imported from http://www.haystack.mit.edu/madrigal from "Millstone Hill IS Radar". The following Madrigal parameters are retrieved:

year,month,day,hour,min,sec,gdlat,glon,gdalt,az,el,vo,dvo

for all records from the past 15 minutes.

Although the particular Madrigal site (http://www.haystack.mit.edu/madrigal), the instrument ("Millstone Hill IS Radar"), the parameters, and the times are hard-coded in this example, they could be easily be modified to be arguments.

To avoid missing data, we choose one parameter to be the filter parameter: vo. By filtering on this parameter, any "missing" values are filtered out.

To run this script requires the python Madrigal API be installed, which can be downloaded from http://www.haystack.edu/madrigal/madDownload.html.

import os,sys,os.path
import string
import time


import madrigalWeb.madrigalWeb


#constants
madrigalUrl = 'http://www.haystack.mit.edu/madrigal'
instrument = 'Millstone Hill IS Radar'

user_fullname = 'Put your name here!!!'
user_email = 'your@email.here'
user_affiliation = 'Put your affiliation here!!!'


# each line of data contains the following parameters
params = 'year,month,day,hour,min,sec,gdlat,glon,gdalt,azm,elm,vo,dvo'
filterParm = 'vo'
timeDelay = 15

# create the main object to get all needed info from Madrigal
madrigalObj = madrigalWeb.madrigalWeb.MadrigalData(madrigalUrl)


# these next few lines convert instrument name to code
code = None
instList = madrigalObj.getAllInstruments()
for inst in instList:
    if inst.name.lower() == instrument.lower():
        code = inst.code
        break

if code == None:
    raise ValueError, 'Unknown instrument %s' % (instrument)


# next, get a list of real time experiments in the last timeDelay minutes
startTime = time.gmtime(time.time() - timeDelay*60.0)
endTime = time.gmtime(time.time())


try:
    expList = madrigalObj.getExperiments(code, startTime[0],
                                     startTime[1],
                                     startTime[2],
                                     startTime[3],
                                     startTime[4],
                                     startTime[5],
                                     endTime[0],
                                     endTime[1],
                                     endTime[2],
                                     endTime[3],
                                     endTime[4],
                                     endTime[5])

except:
    raise ValueError, 'No realtime experiments found'


# assume there's only one realtime experiment, and get the file names
fileList = madrigalObj.getExperimentFiles(expList[0].id)

if len(fileList) == 0:
    raise ValueError, 'No realtime experiment files found'


# get data from each of the files
startDateStr = time.strftime('%m/%d/%Y', startTime)
startDateStr = ' date1=' + startDateStr
startTimeStr = time.strftime('%H:%M:%S', startTime)
startTimeStr = ' time1=' + startTimeStr
endDateStr = time.strftime('%m/%d/%Y', endTime)
endDateStr = ' date2=' + endDateStr
endTimeStr = time.strftime('%H:%M:%S', endTime)
endTimeStr = ' time2=' + endTimeStr

filterString = 'filter=%s,-1E30,1E30' % (filterParm) + startDateStr + startTimeStr + endDateStr + endTimeStr
for dataFile in fileList:
    result = madrigalObj.isprint(dataFile.name, params, filterString,
                                 user_fullname, user_email, user_affiliation)
    # make sure it succeeded
    if result.find('No records were selected') != -1:
        continue
    if result.find('****') != -1:
        continue
    print result

Remote access using python

Doc home

Madrigal home