Creating and editing Madrigal data files

Doc home

Madrigal home

Creating and editing Madrigal data files

A key element in administering the Madrigal database is the ability to create and edit Madrigal data files. An ambitious Madrigal administrator could theoretically read the Cedar database format and write their own code from scratch. However, Madrigal provides API's and examples in three languages, Python, C, and Tcl, to make this chore easier. This section describes how to create Madrigal files using each of those three languages.

• Python
• Using Python to create catalog and header records
• C
• Tcl
• Scripts to modify existing Cedar files
  • mergeCedarFiles
  • mergeCedarFilesInTime
  • removeCedarRecords

Python

This section gives an introduction to using the madrigal python API to create new Cedar files, and to edit existing Cedar files. Examples are given of of both creating new files and editing existing files. Complete documentation is available as part of the Madrigal Python API documentation.

The python cedar module was written to simplify the task of creating Cedar-formatted files as much as possible. The user of this module only needs to know the following about the Cedar format:

• Cedar files are made up of a series of metadata and data records. Each data record consists of a measurement made with a single instrument over a single interval of time.
• A Cedar data record has three parts: a prolog, one-dimensional data, and two-dimensional data. The prolog defines the integration period start and end time, along with the instrument and a kind-of data code that indicates a particular data analysis algorithm. One-dimensional data describes measurements of parameters that do not vary over any spatial dimension. Two-dimensional data describes measurements that do vary over some number of spatial dimensions. Two-dimensional data must have at least one parameter that indicates a spatial dimension.
• A list of Cedar parameters and their units can be found here. A complete description of each parameter can also be seen by clicking on any parameter name in Madrigal. Parameters can be referred to in the python API either by case-insensitive mnemonics (e.g., "Gdalt") or by integer id. Parameters are set either to float values, or to the special values 'missing', 'assumed', or 'knownbad'. The values 'assumed' and 'knownbad' can only be applied to error parameters.
• In addition to data records, the Cedar format also allows two types of records that contain human-readable descriptions of the data: a catalog record which summarizes the entire file, and a header record that summarizes a subset of the file. The text in these records must be padded to a multiple of 80 characters, and should not contain line-feeds. See the Cedar format for the suggested layout of these catalog and header records.

The python cedar module hides the following details from the user:

• The details of the Cedar layout.
• The various 'flavors' of the Cedar format. The use can specify saving a file in one particular 'flavor', if desired
• The limited dynamic range of data fields, given that data is stored as 16 bit integers. An exception is raised if the user tries to set a parameter outside its dynamic range.
• The use of 'additional increment' parameters to increase of precision of a given parameter.

MadrigalCedarFile

The high level object in the cedar module is MadrigalCedarFile. This class emulates a python list, and so users may treat it just like a python list. The restriction enforced is that all items in the list must be either MadrigalCatalogRecords, MadrigalHeaderRecords, or MadrigalDataRecords. Each of these three classes supports the method getType(), which returns 'catalog', 'header', and 'data', respectively.

Using Python to add catalog and header records.

Cedar catalog and header records can be difficult to create. With the cedar module CatalogHeaderCreator, creating these records should now be much easier. Catalog and header records contain a lot of information that can be deduced from the data - this includes which parameters are present, minimum and maximums of certain parameters, and start and stop times. The only parts of the catalog or header record that can't be determined automatically are some optional descriptive text fields. With this new module, you simply pass in strings of any length if you want to fill in one of those optional descriptive text fields - the module will handle all formating for you.

Here's a list of the optional descriptive text fields for a catalog record:

• principleInvestigator: names of responsible Principal Investigator(s) or others knowledgeable about the experiment
• expPurpose: brief description of the experiment purpose
• expMode: further elaboration of meaning of experiment mode; e.g. antenna patterns and pulse sequences
• cycleTime: minutes for one full measurement cycle (this needs to be a number)
• correlativeExp: any correlative experiments (experiments with related data)
• sciRemarks: scientific remarks
• instRemarks: instrument remarks

Here's a list of the optional descriptive text fields for a header record:

• kindatDesc: description of how this data was analyzed (the kind of data)
• analyst: name of person(s) who analyzed this data
• comments: additional comments about data (describe any instrument-specific parameters)
• history: a description of the history of the processing of this file

Creating a new file example

"""createSample.py shows am example of creating an entirely new Madrigal
file using the Python cedar module.  In particular, it creates a file with
a catalog record, a header record, and two data records.  The data records
contain two 1D parameters (System temperature - SYSTMP and Transmitter
Frequency TFREQ) and five 2D parameters (GDALT, GDLAT, GLON, and TR, and DTR).
"""

import os, os.path
import types
import datetime

import madrigal.metadata
import madrigal.cedar

################# sample data #################

kinst = 30 # instrument identifier of Millstone Hill ISR
modexp = 230 # id of mode of experiment
kindat = 3408 # id of kind of data processing
nrow = 5 # all data records have 5 2D rows

SYSTMP = (120.0, 122.0)
TFREQ = (4.4E8, 4.4E8)

GDALT = ((70.0, 100.0, 200.0, 300.0, 400.0),
         (70.0, 100.0, 200.0, 300.0, 400.0))

GDLAT = ((42.0, 42.0, 42.0, 42.0, 42.0),
         (42.0, 42.0, 42.0, 42.0, 42.0))

GLON  = ((270.0, 270.0, 270.0, 270.0, 270.0),
         (270.0, 270.0, 270.0, 270.0, 270.0))

TR    = (('missing', 1.0, 1.0, 2.3, 3.0),
         ('missing', 1.0, 1.7, 2.4, 3.1))


DTR   = (('missing', 'assumed', 'assumed', 0.3, 0.7),
         ('missing', 'assumed', 0.7, 0.4, 0.5))

################# end sample data #################

newFile = '/tmp/testCedar.dat'

# create a new Madrigal file 
cedarObj = madrigal.cedar.MadrigalCedarFile(newFile, True)

# create all data records -  each record lasts one minute
startTime = datetime.datetime(2005, 3, 19, 12, 30, 0, 0)
recTime = datetime.timedelta(0,60)
for recno in range(2):
    endTime = startTime + recTime
    dataRec = madrigal.cedar.MadrigalDataRecord(kinst,
                                                kindat,
                                                startTime.year,
                                                startTime.month,
                                                startTime.day,
                                                startTime.hour,
                                                startTime.minute,
                                                startTime.second,
                                                startTime.microsecond/10000,
                                                endTime.year,
                                                endTime.month,
                                                endTime.day,
                                                endTime.hour,
                                                endTime.minute,
                                                endTime.second,
                                                endTime.microsecond/10000,
                                                ('systmp', 'tfreq'),
                                                ('gdalt', 'gdlat', 'glon', 'tr', 'dtr'),
                                                nrow)

    # set 1d values
    dataRec.set1D('systmp', SYSTMP[recno])
    dataRec.set1D('tfreq', TFREQ[recno])

    # set 2d values
    for n in range(nrow):
        dataRec.set2D('gdalt', n, GDALT[recno][n])
        dataRec.set2D('gdlat', n, GDLAT[recno][n])
        dataRec.set2D('glon',  n, GLON[recno][n])
        dataRec.set2D('tr',    n, TR[recno][n])
        dataRec.set2D('dtr',   n, DTR[recno][n])

    # append new data record
    cedarObj.append(dataRec)

    startTime += recTime                                 

# write new file
cedarObj.write()

# next, use the cedar.CatalogHeaderCreator class to add catalog and header 
catHeadObj = madrigal.cedar.CatalogHeaderCreator(newFile)
catHeadObj.createCatalog(principleInvestigator="John Holt", sciRemarks="Test data only - do not use")
catHeadObj.createHeader(analyst="Bill Rideout", comments="Do not use this data")
catHeadObj.write()

        
        

Editing an existing file example

"""editSample.py shows am example of editing existing data in a Madrigal
file using the Python cedar module.  In particular, it edits the sample file
mil980120g.001 to increase all Ti values by a factor of 1.2
"""

import os, os.path
import types

import madrigal.metadata
import madrigal.cedar

metaObj = madrigal.metadata.MadrigalDB()

orgFile = os.path.join(metaObj.getMadroot(), 'experiments/1998/mlh/20jan98/mil980120g.003')
newFile = '/tmp/mil980120g.003'

# read the Madrigal file into memory
cedarObj = madrigal.cedar.MadrigalCedarFile(orgFile)

# loop through each record, increasing all Ti values by a factor of 1.2
for record in cedarObj:
    # skip header and catalog records
    if record.getType() == 'data':
        # loop through each 2D roow
        for row in range(record.getNrow()):
            presentTi = record.get2D('Ti', row)
            # make sure its not a special string value, eg 'missing'
            if type(presentTi) != types.StringType:
                record.set2D('Ti', row, presentTi*1.2)

# write edited file
cedarObj.write('Madrigal', newFile)

        
        

C

The C API to create Madrigal data files is more complex than the Python API. The best way to learn how to use it is to follow the example below. All the C methods are documented in the Madrigal C API documention in the Madrigal developer's section.

/* 
*    Usage: testCreateRecord
*    This program creates a Madrigal file with a catalog, header, and data record
*        
*/

#include <stdlib.h>
#include <string.h>
#include <stdio.h>
#include <madrec.h>
#include <cedar.h>

int
main (argc, argv)
     int     argc;
     char    *argv[];
{
    Madrec *madrecp;    /* Output File - type 0-4 */
    int iotype=0, stat=0, record=0, i=0;
    int lprol=0, jpar=0, mpar=0, nrow=0, krec=0, kinst=0, kindat=0, ibyr=0, 
        ibmo=0, ibdy=0, ibh=0, ibm=0, ibs=0, ibcs=0, ieyr=0, iemo=0, iedy=0, 
        ieh=0, iem=0, ies=0, iecs=0;
    double kp[8] = {2.0, 3.3, 4.0, 3.0, 8.0, 7.7, 6.3, 5.0};
    double ap[8] = {10.0, 15.0, 16.0, 15.0, 24.0, 21.0, 18.0, 15.0};
    char text[161] = "";
  

    /* Create a madrec object for the output file */
    if ((madrecp = madrecCreate()) == (Madrec *) NULL) {
        fprintf(stderr, "create madrecw: %s\n", madrecGetError(madrecp));
        return(1);
    }

    /* Connect the output madrec object to a madrigal file */
    iotype = 20;
    /* iotype will set which of the Cedar format variations we will create */
    /* See cedarFromat.pdf for details */
    /* 20 - Madrigal file          */
    /* 21 - Blocked Binary file    */ 
    /* 22 - Cbf file               */
    /* 23 - Unblocked Binary file  */
    /* 24 - Ascii file             */

    stat = madrecOpen(madrecp, iotype, "madout");
    fprintf(stderr, "open madrecw: %s\n", madrecGetError(madrecp));
    
    /* first, add a catalog record - lenght must be 80*n */
    /* create some text */
    text[160] = '\0';
    for (i=0; i<160; i++)
        text[i] = ' ';
    strcpy(text, "This is a catalog record line 0");
    text[strlen("This is a catalog record line 0")] = ' ';
    strcpy(text + 80, "This is a catalog record line 1");
    text[80 + strlen("This is a catalog record line 1")] = ' ';
    madrecp->recordp = cedarCreateCatalogRecord(31, 30007,
                                2001, 8, 20,
                                0, 0, 0, 0,
                                2001, 8, 21,
                                23, 59, 59, 99,
				text);
				
    /* append some more text to the catalog record */
    for (i=0; i<160; i++)
        text[i] = ' ';
    strcpy(text, "This is a catalog record line 2");
    text[strlen("This is a catalog record line 2")] = ' ';
    strcpy(text + 80, "This is a catalog record line 3");
    text[80 + strlen("This is a catalog record line 3")] = ' ';
    stat = cedarAppendCatalogRecord(&(madrecp->recordp), text);
				
    stat = madrecPutNextRec(madrecp);
    fprintf(stderr, "putNextRec madrecw: %s\n", madrecGetError(madrecp));
    free(madrecp->recordp);
    
    /* next, add a header record - lenght must be 80*n */
    /* create some text */
    for (i=0; i<160; i++)
        text[i] = ' ';
    strcpy(text, "This is a header record line 0");
    text[strlen("This is a header record line 0")] = ' ';
    strcpy(text + 80, "This is a header record line 1");
    text[80 + strlen("This is a header record line 1")] = ' ';
    madrecp->recordp = cedarCreateHeaderRecord(31, 30007,
                                2001, 8, 20,
                                0, 0, 0, 0,
                                2001, 8, 21,
                                23, 59, 59, 99,
				2,2,
				text);
				
    /* append some more text to the header record */
    for (i=0; i<160; i++)
        text[i] = ' ';
    strcpy(text, "This is a header record line 2");
    text[strlen("This is a header record line 2")] = ' ';
    strcpy(text + 80, "This is a header record line 3");
    text[80 + strlen("This is a header record line 3")] = ' ';
    stat = cedarAppendHeaderRecord(&(madrecp->recordp), text);
				
    stat = madrecPutNextRec(madrecp);
    fprintf(stderr, "putNextRec madrecw: %s\n", madrecGetError(madrecp));
	
    lprol = 16;
    jpar = 3;
    mpar = 2;
    nrow = 8;
    krec = 1002;
    kinst = 210;
    kindat = 30007;
    ibyr = 2001;
    ibmo = 8;
    ibdy = 20;
    ibh = 0;
    ibm = 0;
    ibs = 0;
    ibcs = 0;
    ieyr = 2001;
    iemo = 8;
    iedy = 20;
    ieh = 23;
    iem = 59;
    ies = 59;
    iecs = 59;
    for (record=0; record<5; record++) {

        ibdy++;
        iedy++;

	/* Create a Cedar record in the madrec object */
        if (madrecp->recordp != (Int16 *)NULL) {
            free(madrecp->recordp);
        }
	madrecp->recordp = cedarCreateRecord(lprol, jpar, mpar, nrow, krec,
					     kinst, kindat, ibyr, ibmo, ibdy, 
					     ibh, ibm, ibs, ibcs, ieyr,
					     iemo, iedy, ieh, iem, ies,
					     iecs);
    
	/* Set 1d parameters */
	stat = cedarSet1dParm(madrecp->recordp, 340, 12.0, 0);
	stat = cedarSet1dParm(madrecp->recordp, 354, 1.5e-20, 1);
	stat = cedarSet1dParm(madrecp->recordp, 356, 1.2e-20, 2);
    
	/* Set 2d parms */
	stat = cedarSet2dParm(madrecp->recordp, 310, kp, 0);
	stat = cedarSet2dParm(madrecp->recordp, 335, ap, 1);
    
	/* cedarPrintRecord(madrecp->recordp); */
    
	stat = madrecPutNextRec(madrecp);
	fprintf(stderr, "putNextRec madrecw: %s\n", madrecGetError(madrecp));

    }

    stat = madrecClose(madrecp);      
    fprintf(stderr, "close madrecw: %s\n", madrecGetError(madrecp));

    madrecDestroy(madrecp);

    return(0);
}

Tcl

The Tcl API is not documented the way the Python and C API's are; users who wish to use it are referred to the source code available from the CVS repository at the OpenMadrigal site.The following example shows the creation of a Madrigal file with tcl:

# testWrite

# usage: testWrite

set madfile "madout"

# Create a madrec object
set status [catch mad mad1]
if {$status == 0} {
    puts "mad: [$mad1 get error]"
} else {
    puts "mad Error: [$mad1 get error]"
}

set status [$mad1 open 20 $madfile]
puts "open: [$mad1 get error]"

# Get parameter codes
cedarCode cedarCode

# Create a record
set lprol 16
set jpar 3
set mpar 2
set nrow 8
set krec 1002
set kinst 210
set kindat 30007
set ibyr 2001
set ibmo 8
set ibdy 31
set ibh 0
set ibm 0
set ibs 0
set ibcs 0
set ieyr 2001
set iemo 8
set iedy 31
set ieh 23
set iem 59
set ies 59
set iecs 59

set status [catch {set jdayno [jday 1 11 2001]} result]
puts "status = $status"
puts "result = $result"
puts "jdayno = $jdayno"
puts "date = 1 11 2001"
puts "jdayno = $jdayno"
puts "date = [jdater $jdayno]"

for {set i 0} {$i<5} {incr i} {
    
    set status [catch {$mad1 createRecord $lprol $jpar $mpar $nrow $krec \
			     $kinst $kindat \
			     $ibyr $ibmo $ibdy $ibh $ibm $ibs $ibcs \
			     $ieyr $iemo $iedy $ieh $iem $ies $iecs} result]
    if {$status != 0} {
        puts "createRecord Error: $result"
        exit
    }

    
    set status [catch {$mad1 set krec 1002}]
    if {$status == 0} {
	puts "set krec: [$mad1 get error]"
    } else {
	puts "set krec Error: [$mad1 get error]"
    }
    
    set status [catch {$mad1 set kinst 2222}]
    if {$status == 0} {
	puts "set kinst: [$mad1 get error]"
    } else {
	puts "set kinst Error: [$mad1 get error]"
    }
    
    set status [catch {$mad1 set kindat 3333}]
    if {$status == 0} {
	puts "set kindat: [$mad1 get error]"
    } else {
	puts "set kindat Error: [$mad1 get error]"
    }
    
    set status [catch {$mad1 set startTime 2002 9 32 1 1 1 1}]
    if {$status == 0} {
	puts "set startTime: [$mad1 get error]"
    } else {
	puts "set startTime Error: [$mad1 get error]"
    }
    
    set status [catch {$mad1 set endTime 2003 10 33 2 2 2 2}]
    if {$status == 0} {
	puts "set endTime: [$mad1 get error]"
    } else {
	puts "set endTime Error: [$mad1 get error]"
    }
    
    set status [catch {$mad1 set 1dParm 340 12.0 0}]
    if {$status == 0} {
	puts "set 1dParm: [$mad1 get error]"
    } else {
	puts "set 1dParm Error: [$mad1 get error]"
    }
	
    set status [catch {$mad1 set 1dParm 354 1.5e-20 1}]
    if {$status == 0} {
	puts "set 1dParm: [$mad1 get error]"
    } else {
	puts "set 1dParm Error: [$mad1 get error]"
    }
    
    set status [catch {$mad1 set 1dParm 356 1.2e-20 2}]
    if {$status == 0} {
	puts "set 1dParm: [$mad1 get error]"
    } else {
	puts "set 1dParm Error: [$mad1 get error]"
    }
    
    set parmvals [list 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0]
    set status [catch {$mad1 set 2dParm 310 $parmvals 0}]
    if {$status == 0} {
	puts "set 2dParm: [$mad1 get error]"
    } else {
	puts "set 2dParm Error: [$mad1 get error]"
    }
    
    set parmvals [list 10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0]
    set status [catch {$mad1 set 2dParm 335 $parmvals 1}]
    if {$status == 0} {
	puts "set 2dParm: [$mad1 get error]"
    } else {
	puts "set 2dParm Error: [$mad1 get error]"
    }
    
    # $mad1 printProlog


    $mad1 putNextRecord
    
    #$mad1 printRecord

}

$mad1 close

$mad1 destroy

exit

Scripts to modify existing Cedar files

The following scripts all modify existing Cedar files. All these scripts are installed in madroot/bin.

mergeCedarFiles

Usage: mergeCedarFiles [options] Options:

• -i file rec1 rec2 - Add records rec1 to rec2 of file to outFile
• -o outFile
• -t filetype - set type of outFile to fileType
  1. Madrigal
  2. Blocked Binary
  3. Cbf
  4. Unblocked binary
  5. Ascii

mergeCedarFilesInTime

Usage: mergeCedarFilesInTime [options] Options:

• -i file rec1 rec2 - Add records rec1 to rec2 of file to outFile. Two input files must be specified.
• -o outFile
• -t filetype - set type of outFile to fileType
  1. Madrigal
  2. Blocked Binary
  3. Cbf
  4. Unblocked binary
  5. Ascii

removeCedarRecords

Usage: removeCedarRecords [options] Options:

• -i infile - Only one input file must be specified.
• -o outFile - Only one output file must be specified.
• -r rec1 rec2 - A range of records to remove (inclusive). More than one range can be given.
• -t filetype - set type of outFile to fileType
  1. Madrigal
  2. Blocked Binary
  3. Cbf
  4. Unblocked binary
  5. Ascii

removeCedarRecords -i mil991102g.001 -o mil991102g.002 -r 10 12 -r 100 100

Creating and editing Madrigal data files

Doc home

Madrigal home