The DataFetcher

The DataFetcher class is by detex to serve seismic data to other functions and classes. It is designed to use data from local directories as well as remote clients (like the obspy FDSN client). In the future I hope to add functionality to the DataFetcher to allow it to check data availability and quality. We will start by looking at the DataFetcher class docs, using the DataFetcher on local data directories, and then setting up a DataFetcher to use a remote client.

DataFetcher docs

Let’s print the current version of detex and the docstring associated with the DataFetcher class in order to get an idea of what it does and what options are available.

import detex
print('Current detex version is %s' % (detex.__version__))
print ('-------------------------------')
print (detex.getdata.DataFetcher.__doc__)
Current detex version is 1.0.5
-------------------------------

    

    Class to handle data acquisition
    
    Parameters 
    ----------
    method : str or int
        One of the approved methods for getting data as supported by detex
        Options are:
            "dir" : A data directory as created by makeDataDirectories
            "client" : an obspy client can be passed to get data
            useful if using an in-network database 
            "iris" : an iris client is initiated, also uses IRIS for inventory
            "uuss" : A client attached to the university of utah 
            seismograph stations is initiated using CWB for waveforms
            and IRIS is used for station inventories
    client : An obspy client object
        Client object used to get data, from obspy.fdsn, obspy.neic etc.
    removeResponse : bool
        If True remove response before returning stream.
    inventoryArg : None, obspy client object, or obspy Inventory object
        A seperate client for station inventories, only used if 
        removeResponse == True, also supports keyword "iris" for iris client
    directoryName : str
        A path to the continuous waveforms directory or event waveforms
        directory. If None is passed default names are used 
        (ContinuousWaveForms and EventWaveForms)
    opType : str
        Output type after removing instrument response. Choices are:
        "DISP" (m), "VEL" (m/s), or "ACC" (m/s**2)
    prefilt : list of real numbers
        Pre-filter parameters for removing instrument response. 
    conDatDuration : int or float
        Duration for continuous data in seconds
    conBuff : int or float
        The amount of data, in seconds, to download at the end of the 
        conDatDuration. Ideally should be equal to template length, important 
        in order to avoid missing potential events at the end of a stream
    timeBeforeOrigin : int or float
        Seconds before origin of each event to fetch (used in getTemData)
    timeAfterOrigin : int or float
        Seconds after origin of each event to fetch (used in getTemData)
    checkData : bool
        If True apply some data checks before returning streams, can be useful
        for older data sets. 
    fillZeros : bool
        If True fill data that are not available with 0s (provided some data are
        available)

Some of the more important parameters to pay attention to are the ones controlling the duration of files and the response removal.

  • Parameters that control data duration, number of files, and file type:
    1. timeBeforeOrigin
    2. timeAfterOrigin
    3. conDatDuration
    4. secBuf
  • Parameters that control response removal (more on obspy response removal here)
    1. removeResponse (True or False)
    2. opType (“DISP” (m), “VEL” (m/s), or “ACC” (m/s^2))
    3. prefilt

Also, for less than perfect data, the fillZeros parameter can be very important to avoid discarding data with small gaps. More on this in the clustering section.

DataFetcher with local directories

Often it can be faster to download, preform some processing, and save data once rather than using clients each time detex needs seismic data. This is not always the case, however, if a database on the same network as your workstation is the remote client you wish to use. As an example, let’s create a local data directory and then set up a DataFetcher instance to pull data from it.

Create local directories

In order to create the data directories we first need to let Detex know which stations and events it should look for. To do this we use the template key and station key files (more on that in the required files section.

For this example lets use a subset of the template key and station key used in the intro tutorial.

import detex
stakey = detex.util.readKey('StationKey.csv', key_type='station')
stakey
NETWORK STATION STARTTIME ENDTIME LAT LON ELEVATION CHANNELS
0 TA M17A 2009-04-01T00:00:00 2009-04-01T06:00:00 41.4729 -110.6664 2101 BHE-BHN-BHZ
1 TA M18A 2009-04-01T00:00:00 2009-04-01T06:00:00 41.4272 -110.0674 2103 BHE-BHN-BHZ
temkey = detex.util.readKey('TemplateKey.csv', key_type='template')
temkey
CONTRIBUTOR NAME TIME LAT LON DEPTH MTYPE MAG STATIONKEY
0 ANF 2008-03-24T17-54-19 2008-03-24T17-54-19 41.7268 -110.6584 7.2619 ML 2.01 StationKey.csv
1 ANF 2008-03-25T16-35-03 2008-03-25T16-35-03 41.7993 -110.5818 7.5726 ML 1.75 StationKey.csv
2 ANF 2008-03-28T20-41-02 2008-03-28T20-41-02 41.6896 -110.6455 7.2846 ML 1.88 StationKey.csv
3 ANF 2008-04-04T18-08-51 2008-04-04T18-08-51 41.7077 -110.6730 0.0898 ML 2.62 StationKey.csv
4 ANF 2008-04-28T16-55-58 2008-04-28T16-55-58 41.7997 -110.6390 8.5077 ML 2.39 StationKey.csv
5 ANF 2008-05-02T15-04-29 2008-05-02T15-04-29 41.6946 -110.6193 9.0421 ML 1.74 StationKey.csv
6 ANF 2008-06-06T19-39-39 2008-06-06T19-39-39 41.7436 -110.6124 7.2227 ML 1.88 StationKey.csv
7 ANF 2008-06-19T19-14-39 2008-06-19T19-14-39 41.7503 -110.6070 6.8870 ML 1.88 StationKey.csv

Next we need to call makeDataDirectories (or getAllData which was kept for backward compatibility).

%time detex.getdata.makeDataDirectories() # make directories and time how long it takes (the %time magic only works in ipython)
Getting template waveforms
EventWaveForms is not indexed, indexing now
Getting continuous data
ContinuousWaveForms is not indexed, indexing now
finished makeDataDirectories call
CPU times: user 34.9 s, sys: 1min 24s, total: 1min 58s
Wall time: 1min 13s

/home/derrick/anaconda/lib/python2.7/site-packages/obspy/mseed/core.py:610: UserWarning: The encoding specified in trace.stats.mseed.encoding does not match the dtype of the data.
A suitable encoding will be chosen.
  warnings.warn(msg, UserWarning)

Before we look at the downloaded data lets discuss some of the parameters that you should pay attention to when calling the makeDataDirectories function. You should notice that many of the makeDataDirectories function’s input arguments are similar to DataFetchers arguments. This is because under the hood the makeDataDirectories function is simply using a DataFetcher attached to a client (IRIS by default). If you wanted to use something besides IRIS you would just need to pass a DataFetcher instance attached to another client as the fetch argument.

One unique argument that makeDataDirectories needs is the formatOut, which is the format to use when saving the data to disk. Any format obspy can read/write should be acceptable. Options are: ‘mseed’, ‘sac’, ‘GSE2’, ‘sacxy’, ‘q’, ‘sh_asc’, ‘slist’, ‘tspair’, ‘segy’, ‘su’, ‘pickle’, ‘h5’ (if obspyh5 is installed). Default is mseed, although the makeDataDirectories call by default will remove instrument response thus necessitating that the data are in a float format and therefore devaluing the mseed compression advantage.

I recommend you look at the entire doc string of the function, but I wont print it here. You should think about what parameters will work best for your data set before just using the defaults.

Now let’s take a look at the newly created data directories. This is most easily accomplished by reading the SQLite database that was created to index the directories.

import os
index_file = os.path.join('ContinuousWaveForms', '.index.db')
ind = detex.util.loadSQLite(index_file, 'ind')
ind
Path FileName Starttime Endtime Gaps Nc Nt Duration Station
0 [0, 0, 0, 0] TA.M17A.2009-091T06-00-00.msd 1238565600 1238569320 0 3 3 3720.000000 TA.M17A
1 [0, 0, 0, 0] TA.M17A.2009-091T00-00-00.msd 1238544000 1238547720 0 3 3 3720.000000 TA.M17A
2 [0, 0, 0, 0] TA.M17A.2009-091T02-00-00.msd 1238551200 1238554920 0 3 3 3720.000000 TA.M17A
3 [0, 0, 0, 0] TA.M17A.2009-091T03-00-00.msd 1238554800 1238558520 0 3 3 3720.000001 TA.M17A
4 [0, 0, 0, 0] TA.M17A.2009-091T01-00-00.msd 1238547600 1238551320 0 3 3 3720.000000 TA.M17A
5 [0, 0, 0, 0] TA.M17A.2009-091T05-00-00.msd 1238562000 1238565720 0 3 3 3720.000000 TA.M17A
6 [0, 0, 0, 0] TA.M17A.2009-091T04-00-00.msd 1238558400 1238562120 0 3 3 3720.000000 TA.M17A
7 [0, 1, 0, 0] TA.M18A.2009-091T00-00-00.msd 1238544000 1238547720 0 3 3 3720.000000 TA.M18A
8 [0, 1, 0, 0] TA.M18A.2009-091T02-00-00.msd 1238551200 1238554920 0 3 3 3720.000001 TA.M18A
9 [0, 1, 0, 0] TA.M18A.2009-091T05-00-00.msd 1238562000 1238565720 0 3 3 3720.000000 TA.M18A
10 [0, 1, 0, 0] TA.M18A.2009-091T04-00-00.msd 1238558400 1238562120 0 3 3 3720.000000 TA.M18A
11 [0, 1, 0, 0] TA.M18A.2009-091T06-00-00.msd 1238565600 1238569320 0 3 3 3720.000000 TA.M18A
12 [0, 1, 0, 0] TA.M18A.2009-091T03-00-00.msd 1238554800 1238558520 0 3 3 3720.000000 TA.M18A
13 [0, 1, 0, 0] TA.M18A.2009-091T01-00-00.msd 1238547600 1238551320 0 3 3 3720.000000 TA.M18A

The fields in the database table “ind” are as follows:

Field Description
Path A list of indicies to reference values in the indkey table for building absolute paths
FileName The name of the particular file represented by the current row
Starttime time stamp of the start time in the file
Endtime time stamp of the end time in the file
Gaps The total number of gaps in the file
Nc The number of unique channels
Nt The number of traces (without gaps Nc = Nt)
Duration Duration of seismic data in seconds
Station network.station

When the DataFetcher loads files from a directory it first reads the index to find the paths to load. Because of this, the directory structure not important. For example, if you already have a directory that contains some files in an obspy readable format you can index it with the detex.util.indexDirectory function. Once indexed the directory can be used by the DataFetcher class.

It can be useful to use the index for data quality checks. For example, let’s look for files that are shorter than expected, that are missing channels, or that have gaps (even though we can see these TA data don’t have any such issues).

# look for gaps
ind_gaps = ind[ind.Gaps > 0]
print("There are %d files with gaps" % len(ind_gaps))

# Look for durations at least 2 minutes less than the expected duration
expected_duration = 3720
ind_short = ind[3720 - ind.Duration > 120]
print("There are %d files with shorter than expected durations" % len(ind_short))

# look for missing channels
expected_channels = 3
ind_missing = ind[ind.Nc < expected_channels]
print("There are %d files with less than %d channels" % (len(ind_missing), expected_channels))
There are 0 files with gaps
There are 0 files with shorter than expected durations
There are 0 files with less than 3 channels

Initiate DataFetcher

Now we are ready to create a DataFetcher instance and point it at the newly created directory. We will also explore some of the DataFetcher methods.

# Create two fetchers, one pointed at the continuous data and the other at the event data
con_fetcher = detex.getdata.DataFetcher('dir', directoryName='ContinuousWaveForms', removeResponse=False)
eve_fetcher = detex.getdata.DataFetcher('dir', directoryName='EventWaveForms', removeResponse=False)

As of version 1.0.4 the DataFetcher has 3 public methods:

  1. getStream - fetches an a stream from an arbitrary network, station, channel, location (which the user must define). If no data are fetchable then None is returned.
  2. getConData - creates a generator for fetching all data avaliable for the stations, channels, and date ranges found in a station key.
  3. getTemData - fetches data related to those described by the template key, but also needs a station key to know which stations to look for. Let’s look at an example use of each method:
# getStream example
import obspy

## set variables
utc1 = obspy.UTCDateTime('2009-091T04-13-00') - 5
utc2 = utc1 + 60
net = 'TA'
sta = 'M17A'
chan = 'BH?'

## fetch
st = con_fetcher.getStream(utc1, utc2, net, sta, chan)

## plot the boring data
%pylab inline 
st.plot()
Populating the interactive namespace from numpy and matplotlib

../../../_images/output_15_1.pngpng

# getConData example

## Read station key and use only TA M17A

stakey = detex.util.readKey('StationKey.csv', key_type='station')
stakey = stakey[stakey.STATION=='M17A']

## Create a generator for fetching continuous data
congen = con_fetcher.getConData(stakey) # note if we dont pass a duration the default is used

## loop over generator and calculate sta/lta values to see if we can find an event
from obspy.signal.trigger import classicSTALTA # for simplicity let's use the basic sta/lta
from obspy.signal.trigger import plotTrigger

sta = 0.5 # short term average in seconds
lta = 2 # long term average in seconds
ratio_max = 0 # int variables to keep track of max and time it occurs
time_max = 0
trace_max = None
cft_max = None

for st in congen: # iterate through the generator until it is exhausted
    trace = st.select(component = 'z')[0] # select vertical component
    trace.filter('bandpass', freqmin=1, freqmax=10, zerophase=True, corners=2) #filter
    sr = trace.stats.sampling_rate # get sampling rate
    starttime = trace.stats.starttime
    cft = classicSTALTA(trace.data, int(sta * sr), int(lta * sr)) # run sta/lta
    cft_max = max(cft) # get max value
    if cft_max > ratio_max: # if the max is greater than old max
        ratio_max = cft_max # set new max
        time_max = starttime + cft.argmax()/float(sr) # set time max
        trace_max = trace.copy()
        cft_max = cft

print("The max sta/lta was %.2f occured at %s" % (ratio_max, time_max))
plotTrigger(trace, cft, ratio_max*.92, ratio_max/1.5)

## Let's get a closer look

st = con_fetcher.getStream(time_max-10, time_max+35, 'TA', 'M17A', 'BHZ')
st.filter('bandpass', freqmin=1, freqmax=5, zerophase=True, corners=2)
st.plot()
        
The max sta/lta was 3.73 occured at 2009-04-01T01:44:09.950001Z

/home/derrick/anaconda/lib/python2.7/site-packages/obspy/signal/trigger.py:348: VisibleDeprecationWarning: boolean index did not match indexed array along dimension 0; dimension is 435 but corresponding boolean dimension is 434
  of.extend(ind2[np.diff(ind2) > 1].tolist())

../../../_images/output_16_2.pngpng

../../../_images/output_16_3.pngpng

# getConData example

## Create stream generator
evegen = eve_fetcher.getTemData("TemplateKey.csv", "StationKey.csv")
# note: the temkey and stakey parameters can either be paths to csv files or DataFrames

## iterate through each of the known events plot a spectrogram of the one with highest amplitude
amp_max = 0
tr_max = None
for st, evename in evegen:
    trace = st.select(component = 'z')[0]
    trace.detrend('linear')
    trace.filter('bandpass', freqmin=1, freqmax=10, zerophase=True, corners=2)
    z_max = max(trace.data)
    if z_max > amp_max:
        amp_max = z_max
        tr_max = trace.copy()
tr_max.plot()
tr_max.spectrogram()

../../../_images/output_17_0.pngpng

../../../_images/output_17_1.pngpng

DataFetcher with clients

Detex should be able to handle a wide variety of obspy client types, including FDSN, NEIC, EARTHWORM, etc. However, as of version 1.0.4 I have only tested IRIS extensively so using other clients may take a bit of debugging. More tests and bug fixes will follow in future versions.

IRIS FDSN client

In order to use the DataFetcher we first need to set up a client object. We will create an FDSN client then initiate an instance of the DataFetcher class and use the getStream function to fetch an obspy stream.

import detex
import obspy
from obspy.fdsn import Client

#setup client
client = Client("IRIS")

# setup fetcher
fetcher = detex.getdata.DataFetcher(method='client', client=client)

# set info
utc1 = obspy.UTCDateTime('2009-03-19T19-06-07') - 5
utc2 = utc1 + 60
net = 'TA'
sta = 'M17A'
chan = 'BH?'

# fetch a stream
st = fetcher.getStream(utc1, utc2, net, sta, chan)

# plot waveforms
%pylab inline 
st.filter('bandpass', freqmin=1, freqmax=10, corners=2, zerophase=True)
st.plot()
Populating the interactive namespace from numpy and matplotlib

WARNING: pylab import has clobbered these variables: ['trace']
`%matplotlib` prevents importing * from pylab and numpy

../../../_images/output_19_2.pngpng

If the waveforms look strange it is because this event is actually a blast at a surface coal mine.

The other methods demonstrated in previous sections also work with the DataFetcher attached to IRIS, so I wont illustrate them again here.

It should be noted that by default the instrument responses have been removed. This can be controlled with the removeResponse input argument which is either set to True or False.

This should give you all the information you need on how detex gets its data and how to set up a custom DataFetcher to be used by other detex classes.

Next Section

The next section covers how to perform waveform similarity analysis in preparation for subspace detection.