Version 1.2.0 Level 2007.08.05
Contacts
Jeremy Fee, USGS Golden
David Oppenheimer, USGS Menlo
Park
Alan Jones, SUNY
Binghamton
The QDM (Quake Data Merge) Real Time Merged Catalog software is a system of programs that receives earthquake summary information and additional text from seismographic networks and produces a single merged catalog. The first implementation of the system was written by Andy Michael in a combination of C, Unix scripts, and Unix utilities such as sort, awk, grep, etc. This implementation is in Java to provide a platform-independent system.
The merged catalog follows the rules set out by the now-defunct Council of the National Seismic System. One additional rule has been instituted: a network can not place an earthquake outside of its authoritative region until 10 minutes after origin time. This rule was put in place to give the authoritative network a chance to put out the first location and prevent excessive changes in information soon after an earthquake.
The most likely method for receiving the event and text information is through the QDDS system. For more information, please contact David Oppenheimer.
The most likely use for the output of the QDM system is Bob Simpson's recenteqs programs that build the web pages. For more information on these programs, please contact Bob Simpson.
QDM and QDDS can be used in conjunction with Alan Jones' Windows Seismic/Eruption program (www.geol.binghamton.edu/faculty/jones).
QDDS/QDM are also used to distribute Eqnews messages. For more information, please contact Lisa Wald.
To install, you must first install the Java Software Development Kit (J2SE SDK) or the Java Runtime Environment (J2SE JRE) which is provided by Sun Microcomputers. Unless you intend to modify the source code, you only need JRE. Select J2SE JRE:
Pick the download for your environment, either Windows, Linux, Solaris, etc. Download and install.
Previous versions of QDM required that you have the xerces jar files on your system. This is no longer a requirement since the needed files are included in the QDM.jar.
Following are the steps to install QDM.
Before starting QDM you should know where the QDDS system (or another data source) will be placing the input data and where the output should be directed for the recenteqs system. The account the programs run under must have write permission on those directories. See the introduction for information on QDDS and recenteqs. You also need to pick an hour when the system will do its daily cleanup work.
Edit the file: QDM.config. It will look something like the example below. The comments after the ";" character are for explanations only. They should not appear in the actual file
POLL WAIT TIME: 20 ; How often, in seconds, to check polldir
POLL DIRECTORY: polldir/ ; Poll directory which contains input
OUTPUTSPOOL DIRECTORY: outputspool/ ; Output directory to store messages
OUTPUTSPOOL DIRECTORY: outputspool1/ ; Optional directory - Receives
copies of files in outspool
OUTPUTSPOOL DIRECTORY: outputspool2/ ; Optional directory - Receives
2nd copy of files in outspool
OUTPUTSPOOL DIRECTORY: outputspoolN/ ; Optional directory - Receives
Nth copy of files in outspool
CATALOG DIRECTORY: catalog/ ; Catalog containing records
for each network plus merged files
LOG DIRECTORY: logdir/ ; Store logs
OLDINPUT DIRECTORY: oldoutput/ ; Files from polldir are moved here
MAINTENANCE HOUR: 13.5 ; Hour of day to do maintenance
DAYS TO RETAIN OUTPUTSPOOL FILES: 14 ; Number of days to retain output files
DAYS TO RETAIN OLDINPUT FILES: 14 ; Number of days to retain old
input files
DAYS TO RETAIN LOG FILES: 5 ; Number of days to reatain log files
DAYS TO RETAIN EVENTS: 8 ; Throw event records out of catalog
; when they are older than
this many days
INITIAL TIME: 199912010000 ; If even is older than this, don't use
LOG LEVEL: 5 ; Level of logging desired. See below.
The purpose of the INITIAL TIME setting is for start-up purposes. That is, as the merge file is created, there may be a few events several days older than the current time making the merge file not complete. When the merge.nts file is used as an input to another system such as Seismic/Eruption, the merge.nts file is used to replace all events starting with the oldest event in merge.nts. By setting INITIAL TIME to the current time, the merge.nts file will be slowly created and should be complete.
The LOG LEVELs run from 1 to 5. A level of 1 just logs unexpected events, called Exceptions in Java. As the level moves from 2 to 5 more details are logged. If you are not having troubles and don't care about examining the logs later, a level of 1 or even 0 (no logging) is recommended.
Notice that the MAINTENANCE HOUR can be fractional so that, say, 13.5 will call for a daily maintenace at 1:30pm.
To run the program, execute:
java -jar QDM.jar
When the program starts, it checks to see if the necessary sub-directories are present. If they are not, it creates them. The regions.xml file must be there or the program will terminate. The QDM system looks for input in the polldir directory. The QDM system places the catalog information in the catalog directory, the text messages in the eqcomments directory, and the addition information links (addons) in the eqaddons directory. Copies of the text message and addon files and flag files when the merged catalog changes are also placed in the outputspool directory for use by the recenteqs system. This can be changed to some other directory by editing the QDM.config file.
If you have a memory problem, you may have to specify the Java heap space. For example, if you want to specify an initial Java heap size of 50 MB and a maximum Java heap size of 90 MB, you would start QDM as:
java -Xms50000000 -Xmx90000000 -jar QDM.jar
Once per day at an hour designated in QDM.config it does file maintenance. It removes old files and old events.
You can make a change to QDM.config while the system is running. If you do, the program detects that the file has a new date, reads the file, and changes its internal setting accordingly.
What if an event, comment, or addon is wrong on just this computer? Put the correct version of it in the polldir directory. To do this create the file somewhere else and then move or copy it to the polldir directory.
Note: The CUBE files contain a check byte which will not be correct if you edit one of the files by hand. QDDS does not look inside the files so it will not care but QDM does. It will discard an event with a bad check sum.
What if an event, comment, or addon is wrong on multiple computers? Resubmit it through the QDDS system.
What if you are sure you are providing correct input files and the QDM system is not processing them properly? Resubmit them to make sure. Make sure you know the QDM merging rules completely. For instance, a network is completely authoritative in its area and if it submits two identical events with different id numbers they will both wind up in merge.sum. Remember the 10 minute rule. Then, as a last resort, stop the QDM.jar program hand edit the files so that they look like what you think they should look like, and restart QDM.jar. Hand editting should only be done as a last resort and you should save copies of the bad files, document what happened, and let Alan Jones know via email
The QDM system creates the following directores and files in it's home directory:
For each network the following files exist (where nc is the network code):
The merged catalog information is stored in the files and is time order sorted:
QDM handles XML files. XML is a standardized format for storing and distributing information. A brief description of the format and philosophy can be found at XML. Modern languages (Java, Perl) have utilities for reading and writing XML files. Upon encountering an XML file containing a <file name="xxxxx"/> element, QDM will copy the file to its outputspool directories with the specified name. QDM will ignore files that begin with the string, "<?xml", but do not contain the file/name element, or are not well-formed xml files. This xml capability is currently used for distributing updated region polygon files (see next item), and, for communicating eq-in-the-news messages.
The QDM distribution contains two versions of the package:
The regions are stored in an XML file (regions.xml). The best way to describe the structure of the file is to show a portion of the file:
<?xml version="1.0"?>
<!--
<!DOCTYPE regions PUBLIC "-//gov.usgs.ehz//QDM Index 0.1//EN"
"regions.dtd">
-->
<regions>
<file name="regions.xml"/>
<update date="2002-01-04 13:33:14"/>
<format version="0.1"/>
<!--
# Region file for the QDM catalog
# DEFAULT indicates no polygon file. I.e. it matches any event.
# A network may have more than 1 region list, but each region may
# be associated with only ONE network.
# There may only be 1 DEFAULT region.
-->
<net code="AK" name="alaska">
<region code="AK">
<coordinate latitude="71.0000" longitude="-140.0000"/>
<coordinate latitude="57.0000" longitude="-140.0000"/>
<coordinate latitude="55.0000" longitude="-156.0000"/>
<coordinate latitude="59.0000" longitude="-164.0000"/>
<coordinate latitude="63.0000" longitude="-169.0000"/>
<coordinate latitude="71.0000" longitude="-169.0000"/>
</region>
</net>
<net code="AT" name="wcatwc">
<region code="AT">
<coordinate latitude="41.0000" longitude="-160.0000"/>
<coordinate latitude="41.1000" longitude="-160.0000"/>
<coordinate latitude="41.1000" longitude="-159.9000"/>
<coordinate latitude="41.0000" longitude="-159.9000"/>
<coordinate latitude="41.0000" longitude="-160.0000"/>
</region>
</net>
...
[etc.]
...
<net code="WY" name="yellowstone">
<region code="WY">
<coordinate latitude="44.0000" longitude="-109.7500"/>
<coordinate latitude="44.0000" longitude="-111.3333"/>
<coordinate latitude="45.1667" longitude="-111.3333"/>
<coordinate latitude="45.1667" longitude="-109.7500"/>
<coordinate latitude="44.0000" longitude="-109.7500"/>
</region>
</net>
<net code="US" name="DEFAULT">
<region code="US"/>
</net>
</regions>
The regions.xml file is read each time the program starts. The regions are used in the program to decide if an event is authoritive, that is, if it occurred in the region defined for that network.
When the day reaches the hour specified in QDM.config the following maintenance activities take place:
The QDM system uses the CUBE format.