Previous: Doc home   Up: Doc home   Next: What's new in Madrigal 2.6?

Brief history of Madrigal

Madrigal is a database of ground-based measurements and models of the Earth's upper atmosphere and ionosphere. It is closely related to the Coupling, Energetics and Dynamics of Atmospheric Regions (CEDAR) program, which is devoted to the characterization and understanding of the atmosphere above about 60 km, with emphasis on the various processes that determine the basic structure and composition of the atmosphere, and on the mechanisms that couple different atmospheric regions. Instruments developed or upgraded under CEDAR include interferometers, spectrometers, imagers, lidars and medium, high-frequency and incoherent scatter radars. Models developed with CEDAR support or incorporating CEDAR data include first-principle global circulation models, and empirical models of temperature, density and winds. The success of CEDAR has been due, in large measure, to its ability to encourage collaborative efforts coalescing observations, theory and modeling. The CEDAR community includes about 800 scientists and students from around the world.

From the inception of the CEDAR program in 1988, there has been a great concern among the members of the CEDAR community to make the data collected by the CEDAR instruments easily accessible for joint studies. Consequently, a high priority was placed on establishing a repository for CEDAR data and model results. An incoherent scatter radar database had been established at the National Center for Atmospheric Research (NCAR) in 1985, and this evolved into the CEDAR Database in 1989. By the end of 1997, it had grown to include data from 44 instruments and 16 models. Over 200 users have requested information from the Database.

A central element of the CEDAR Database is a standard data format. This format evolved from the format used by the earlier incoherent scatter database, which in turn evolved from an earlier version of Madrigal developed at the MIT Haystack Observatory in 1980. Haystack continues to maintain and develop Madrigal as an open-source project with community contributions. The basic Madrigal data format is the same as at NCAR, and data files are easily exchanged between the two sites, but Madrigal has a significantly different emphasis. The function of NCAR in the CEDAR Data Base is first of all to archive a comprehensive collection of CEDAR data. Madrigal, on the other hand, has evolved into a robust, World Wide Web based system capable of managing and serving archival and real-time data from a wide variety of CEDAR instruments. The Madrigal database is focused on data that is being actively maintained by the groups that produced the data. With the Madrigal database, the site owner stores only their own data, which they can add to or update at any time. However, because the Madrigal database shares its metadata with all other Madrigal sites, users browsing any Madrigal site can search for data at any other Madrigal site.

CEDAR is now poised to realize the full scientific potential anticipated at its inception. CEDAR's scientific directions and future operational requirements were detailed in 1997 in the CEDAR Phase III Report. The report states that "CEDAR must continue to capitalize on the latest advancements in order to accomplish the best science most effectively. Where practical and advantageous, remote continuous autonomous operation of CEDAR instruments ought to be encouraged. Computer-aided collaborations, CEDAR Database use, and CEDAR program information should be available to the CEDAR community over the World Wide Web. Computer-aided collaborations and real-time access to data and predictive models should be encouraged." Madrigal is a key component to meeting these goals.

Madrigal data are arranged into "experiments", which may contain data files, images, documentation, links, etc. Almost all Millstone Hill Radar experiments since 1976, over 900 experiments in all, as well as many measurements by other instruments, are now available on-line. Each experiment has a unique URL, which serves to map the database contents into the World Wide Web, and is the fundamental mechanism through which the database can be distributed among multiple servers. An key feature of Madrigal is its seamless integration of archival and real-time data. A realtime file on Madrigal is accessed in exactly the same way as any archival file.

Madrigal has been installed at several locations in addition to Millstone Hill, including EISCAT, SRI International, and Jicamarca. The inventories of experiments available at each installation are available to the other installations through a cgi script. Typically each installation retrieves inventories from the other sites and updates a combined experiment list once a day. Users can view the combined inventory and seamlessly retrieve data regardless of where the data is actually stored.

Madrigal is an open source project with a central developer's web site and a central repository. A complete CVS archive of all Madrigal software, including software which is not included with the standard distribution, is available at the developer's web site. There is also an Open Madrigal mailing list and developer's forum. Any group wishing to install Madrigal to distribute their instrument's data is welcome to - see the www.OpenMadrigal.org web site for details. The central repository link is a place where all data from all Madrigal sites is archived. It can also be used to access data, although accessing the local sites is preferred.

Previous: Doc Home   Up: Doc home   Next: What's new in Madrigal 2.6?