Clustering¶
This segment of the tutorial will teach you how to preform waveform similarity clustering in detex. The function used to preform clustering is createCluster of the construct module. The results are then stored in an instance of the ClusterStream class. Let’s start by looking at the createCluster function and some important parameters that go into it, then we will look at the ClusterStream class its methods.
CreateCluster¶
import detex
version = detex.__version__
print ("Detex version is %s\n" % version)
print (detex.construct.createCluster.__doc__)
Detex version is 1.0.5
Function to create an instance of the ClusterStream class
Parameters
-------
CCreq : float, between 0 and 1
The minimum correlation coefficient for grouping waveforms.
0.0 results in all waveforms grouping together and 1.0 will not
form any groups.
fetch_arg : str or detex.getdata.DataFetcher instance
Fetch_arg of detex.getdata.quickFetch, see docs for details.
filt : list
A list of the required input parameters for the obspy bandpass
filter [freqmin, freqmax, corners, zerophase].
stationKey : str or pd.DataFrame
Path to the station key or DataFrame of station key.
templateKey : str or pd.DataFrame
Path to the template key or loaded template key in DataFrame.
trim : list
A list with seconds to trim from events with respect to the origin
time reported in the template key. The default value of [10, 120]
means each event will be trimmed to only contain 10 seconds before
its origin time and 120 seconds after. The larger the values of this
argument the longer the computation time and chance of misalignment,
but values that are too small may trim out desired phases of the
waveform.
saveClust : bool
If true save the cluster object in the current working
directory. The name is controlled by the fileName parameter.
fileName : str
Path (or name) to save the clustering instance, only used
if saveClust is True.
decimate : int or None
A decimation factor to apply to all data in order to decrease run
time. Can be very useful if the the data are oversampled. For
example, if the data are sampled at 200 Hz but a 1 to 10 Hz
bandpass filter is applied it may be appropriate to apply a
decimation factor of 5 to bring the sampling rate down to 40 hz.
dytpe : str
An option to recast data type of the seismic data. Options are:
double- numpy float 64
single- numpy float 32, much faster and amenable to cuda GPU
processing, sacrifices precision.
eventsOnAllStations : bool
If True only use the events that occur on all stations, if
false let each station have an independent event list.
enforceOrigin : bool
If True make sure each trace starts at the reported origin time in
the template key. If not trim or merge with zeros. Required for
lag times to be meaningful for hypoDD input.
fillZeros : bool
If True fill zeros from trim[0] to trim[1]. Suggested for older
data or if only triggered data are available.
Returns
---------
An instance of the detex SSClustering class
As you can see there are a lot of input arguments, and a lot to think about when creating a cluster object. Let me elaborate on some of the arguments you should pay special attention to.
- fet_arg - make sure to look at the detex.getdata.quickFetch docs for this one. Basically, if you want to use a custom DataFetcher be sure to pass it to the createCluster call here or else detex will try to use a local directory with the default name of EventWaveForms.
- filt - parameters to apply a bandpass filter to the waveform similarity clustering and ALL all detex downstream operations. Make sure to think about this carefully before simply using the default, as the default values are not appropriate for all data sets.
- fillZeros - a parameter for handling data with gaps. If data are not avaliable for the entire range (defined by template key and trim parameter) detex will simply fill zeros so that each trace will have the length defined by the trim parameter. The created cluster instance can then be used later on by detex, although you should be careful going forward to no include a bunch of the zero data in your detector, more on that later.
- trim - a two element list that defines the length of each waveform. The first element is the time before the origin (as reported in the station key) and the second element is the number of seconds after the reported origin time.
Dealing with gaps¶
In order to see how some of these parameters affect the clustering process we will look at an early UUSS dataset that has some issues with gaps. Here are the stations and templates:
stakey = detex.util.readKey('StationKey.csv', key_type='station')
stakey
| NETWORK | STATION | STARTTIME | ENDTIME | LAT | LON | ELEVATION | CHANNELS | |
|---|---|---|---|---|---|---|---|---|
| 0 | UU | IMU | 2003-01-03T00:00:00 | 2004-04-04T00:00:00 | 38.6332 | -113.158 | 1833 | EHZ |
| 1 | UU | MSU | 2003-01-01T00:00:00 | 2004-04-04T00:00:00 | 38.5123 | -112.177 | 2105 | EHZ |
temkey = detex.util.readKey('TemplateKey.csv', key_type='template')
temkey
| TIME | NAME | LAT | LON | MAG | DEPTH | |
|---|---|---|---|---|---|---|
| 0 | 2002-02-01T04-37-29.94 | 2002-02-01T04-37-29.94 | 38.616833 | -112.463833 | 1.40 | 2.82 |
| 1 | 2002-02-02T03-38-36.21 | 2002-02-02T03-38-36.21 | 38.575667 | -112.714500 | 1.19 | 1.14 |
| 2 | 2002-02-22T22-59-28.69 | 2002-02-22T22-59-28.69 | 38.558667 | -112.460500 | 2.17 | 1.72 |
| 3 | 2002-02-22T23-10-36.58 | 2002-02-22T23-10-36.58 | 38.538667 | -112.455500 | 0.99 | 5.82 |
| 4 | 2002-02-26T11-45-48.87 | 2002-02-26T11-45-48.87 | 38.527333 | -112.467667 | 1.98 | 4.01 |
| 5 | 2002-02-26T12-09-47.92 | 2002-02-26T12-09-47.92 | 38.558500 | -112.456500 | 2.39 | -0.28 |
| 6 | 2002-03-04T02-53-36.92 | 2002-03-04T02-53-36.92 | 38.708000 | -112.535500 | 1.43 | -0.14 |
| 7 | 2002-03-04T17-13-19.61 | 2002-03-04T17-13-19.61 | 38.542333 | -112.455333 | 1.33 | 1.84 |
| 8 | 2002-03-05T00-32-02.35 | 2002-03-05T00-32-02.35 | 38.544667 | -112.457667 | 1.34 | 2.80 |
| 9 | 2002-03-10T16-03-15.42 | 2002-03-10T16-03-15.42 | 38.722000 | -112.545000 | 2.50 | -2.97 |
| 10 | 2002-03-21T20-16-14.74 | 2002-03-21T20-16-14.74 | 38.711333 | -112.541167 | 2.04 | 1.81 |
| 11 | 2002-03-23T16-39-08.76 | 2002-03-23T16-39-08.76 | 38.701667 | -112.542333 | 1.32 | 1.23 |
| 12 | 2002-05-12T18-32-19.43 | 2002-05-12T18-32-19.43 | 38.724333 | -112.539500 | 0.71 | 1.42 |
| 13 | 2002-05-15T15-35-00.17 | 2002-05-15T15-35-00.17 | 38.588667 | -112.586833 | 0.73 | 3.00 |
| 14 | 2002-07-06T22-05-16.73 | 2002-07-06T22-05-16.73 | 38.512000 | -112.460833 | 0.35 | 5.78 |
| 15 | 2002-08-01T20-19-21.00 | 2002-08-01T20-19-21.00 | 38.525833 | -112.455500 | 1.03 | 1.84 |
| 16 | 2002-08-19T19-35-07.44 | 2002-08-19T19-35-07.44 | 38.689000 | -112.543500 | 0.72 | 5.96 |
| 17 | 2002-08-19T21-09-35.83 | 2002-08-19T21-09-35.83 | 38.685833 | -112.541833 | 1.17 | 4.54 |
| 18 | 2002-09-07T19-55-27.85 | 2002-09-07T19-55-27.85 | 38.713000 | -112.647500 | 1.02 | 2.79 |
| 19 | 2002-09-17T14-50-53.10 | 2002-09-17T14-50-53.10 | 38.663167 | -112.630000 | 0.71 | 0.31 |
| 20 | 2002-10-06T11-57-35.19 | 2002-10-06T11-57-35.19 | 38.645667 | -112.461667 | 1.31 | 3.64 |
| 21 | 2002-11-05T23-12-44.72 | 2002-11-05T23-12-44.72 | 38.532333 | -112.466167 | 0.95 | 6.02 |
| 22 | 2002-11-05T23-13-27.56 | 2002-11-05T23-13-27.56 | 38.555167 | -112.468167 | 1.37 | 2.94 |
| 23 | 2002-11-06T00-44-00.38 | 2002-11-06T00-44-00.38 | 38.527833 | -112.482167 | 2.05 | 4.09 |
| 24 | 2002-11-06T01-48-04.90 | 2002-11-06T01-48-04.90 | 38.532167 | -112.463833 | 1.10 | 3.15 |
| 25 | 2002-11-06T02-24-03.05 | 2002-11-06T02-24-03.05 | 38.515833 | -112.463833 | 0.61 | 6.02 |
| 26 | 2002-11-08T18-27-28.27 | 2002-11-08T18-27-28.27 | 38.534667 | -112.468500 | 1.29 | 2.43 |
| 27 | 2002-11-08T21-48-47.69 | 2002-11-08T21-48-47.69 | 38.530833 | -112.470500 | 2.26 | 2.88 |
| 28 | 2002-11-08T22-00-02.41 | 2002-11-08T22-00-02.41 | 38.519500 | -112.468000 | 1.12 | 2.07 |
| 29 | 2002-11-09T21-38-33.05 | 2002-11-09T21-38-33.05 | 38.548167 | -112.468333 | 1.22 | 7.83 |
| ... | ... | ... | ... | ... | ... | ... |
| 192 | 2013-10-10T16-10-37.49 | 2013-10-10T16-10-37.49 | 38.597167 | -112.598167 | 1.96 | 5.21 |
| 193 | 2013-10-10T23-12-49.39 | 2013-10-10T23-12-49.39 | 38.606500 | -112.612833 | 2.48 | 3.32 |
| 194 | 2013-10-10T23-21-10.54 | 2013-10-10T23-21-10.54 | 38.601833 | -112.606333 | 2.36 | 3.91 |
| 195 | 2013-10-11T00-26-32.78 | 2013-10-11T00-26-32.78 | 38.603833 | -112.606833 | 1.34 | -0.16 |
| 196 | 2013-10-11T10-36-01.05 | 2013-10-11T10-36-01.05 | 38.613667 | -112.607667 | 0.86 | -0.01 |
| 197 | 2013-10-11T15-19-47.68 | 2013-10-11T15-19-47.68 | 38.598167 | -112.608333 | 1.96 | 4.14 |
| 198 | 2013-10-24T09-57-46.49 | 2013-10-24T09-57-46.49 | 38.589167 | -112.606333 | 1.54 | 1.48 |
| 199 | 2013-11-01T02-56-52.97 | 2013-11-01T02-56-52.97 | 38.734500 | -112.567167 | 1.31 | 0.69 |
| 200 | 2013-11-10T15-36-35.26 | 2013-11-10T15-36-35.26 | 38.714667 | -112.526333 | 0.70 | 9.28 |
| 201 | 2013-12-25T05-32-32.15 | 2013-12-25T05-32-32.15 | 38.633833 | -112.539500 | 0.78 | -0.10 |
| 202 | 2014-01-09T00-10-03.16 | 2014-01-09T00-10-03.16 | 38.554833 | -112.728833 | 1.93 | -0.24 |
| 203 | 2014-01-17T12-52-52.03 | 2014-01-17T12-52-52.03 | 38.637500 | -112.612000 | -9.99 | 1.49 |
| 204 | 2014-01-17T12-52-55.73 | 2014-01-17T12-52-55.73 | 38.609667 | -112.613000 | 1.84 | 0.45 |
| 205 | 2014-01-17T16-16-23.71 | 2014-01-17T16-16-23.71 | 38.607667 | -112.603667 | 0.96 | 1.93 |
| 206 | 2014-02-08T07-14-59.47 | 2014-02-08T07-14-59.47 | 38.664500 | -112.535667 | 0.86 | 4.32 |
| 207 | 2014-02-10T00-32-14.92 | 2014-02-10T00-32-14.92 | 38.734333 | -112.521500 | 1.24 | 2.86 |
| 208 | 2014-03-08T20-37-13.43 | 2014-03-08T20-37-13.43 | 38.722000 | -112.552667 | 0.94 | 4.96 |
| 209 | 2014-03-23T01-26-16.64 | 2014-03-23T01-26-16.64 | 38.674167 | -112.585833 | 1.92 | -3.09 |
| 210 | 2014-04-06T08-19-55.40 | 2014-04-06T08-19-55.40 | 38.563000 | -112.690167 | 1.92 | 1.88 |
| 211 | 2014-05-27T19-04-32.17 | 2014-05-27T19-04-32.17 | 38.594833 | -112.593833 | 1.20 | 0.38 |
| 212 | 2014-06-05T03-14-06.76 | 2014-06-05T03-14-06.76 | 38.683667 | -112.533167 | 2.01 | 3.51 |
| 213 | 2014-06-09T11-00-38.18 | 2014-06-09T11-00-38.18 | 38.727000 | -112.503167 | 0.57 | 1.75 |
| 214 | 2014-06-12T06-03-10.42 | 2014-06-12T06-03-10.42 | 38.678167 | -112.539500 | 1.70 | 0.99 |
| 215 | 2014-07-11T03-47-35.36 | 2014-07-11T03-47-35.36 | 38.689333 | -112.579500 | 0.78 | 3.96 |
| 216 | 2014-10-29T10-58-38.16 | 2014-10-29T10-58-38.16 | 38.574833 | -112.587333 | 1.64 | 3.12 |
| 217 | 2014-10-29T11-01-00.32 | 2014-10-29T11-01-00.32 | 38.592167 | -112.594500 | 0.94 | 0.46 |
| 218 | 2014-11-15T16-47-50.40 | 2014-11-15T16-47-50.40 | 38.586167 | -112.600333 | 0.79 | 2.93 |
| 219 | 2014-11-29T14-18-04.87 | 2014-11-29T14-18-04.87 | 38.698167 | -112.544500 | 0.89 | 0.16 |
| 220 | 2014-12-17T08-52-09.85 | 2014-12-17T08-52-09.85 | 38.579500 | -112.589833 | 1.47 | 0.23 |
| 221 | 2014-12-21T17-01-29.91 | 2014-12-21T17-01-29.91 | 38.586500 | -112.587167 | 0.56 | 0.69 |
222 rows × 6 columns
Because there are so many events it may take some time to get the data. We will skip getting the continuous data because it is not needed for this section of the tutorial. We should probably also start the logger in case we need more info than what is printed to the screen. We will also delete an old logger if there is one.
import os
if os.path.exists("detex_log.log"):
os.remove("detex_log.log")
detex.setLogger()
detex.getdata.makeDataDirectories(getContinuous=False)
Getting template waveforms
EventWaveForms is not indexed, indexing now
finished makeDataDirectories call
/home/derrick/anaconda/lib/python2.7/site-packages/obspy/mseed/core.py:610: UserWarning: The encoding specified in trace.stats.mseed.encoding does not match the dtype of the data.
A suitable encoding will be chosen.
warnings.warn(msg, UserWarning)
Now we will cluster these events while varying the input arguments. Let’s start by using the defaults.
%time cl = detex.createCluster() # notice we can call createCluster from the detex level
Cannot remove response without a valid inventoryArg, setting removeResponse to False
Starting IO operations and data checks
2004-07-29T23-09-58.55 on UU.IMU is out of length tolerance, removing
2005-01-02T06-35-22.46 on UU.IMU is out of length tolerance, removing
2005-09-30T13-47-54.79 on UU.IMU is out of length tolerance, removing
2005-11-28T16-18-36.82 on UU.IMU is out of length tolerance, removing
2005-12-21T04-48-05.24 on UU.IMU is out of length tolerance, removing
2004-07-29T23-09-58.55 is fractured or missing data, removing
2004-05-19T20-44-17.41 on UU.MSU is out of length tolerance, removing
2005-01-02T06-35-22.46 on UU.MSU is out of length tolerance, removing
2005-01-14T01-30-48.97 on UU.MSU is out of length tolerance, removing
2005-08-09T06-35-29.44 on UU.MSU is out of length tolerance, removing
2005-09-30T13-47-54.79 on UU.MSU is out of length tolerance, removing
2005-12-21T04-48-05.24 on UU.MSU is out of length tolerance, removing
2006-02-05T23-33-43.00 on UU.MSU is out of length tolerance, removing
performing cluster analysis on UU.IMU
performing cluster analysis on UU.MSU
ccReq for station UU.IMU updated to ccReq=0.500
ccReq for station UU.MSU updated to ccReq=0.500
writing ClusterStream instance as clust.pkl
CPU times: user 2min 36s, sys: 5min 50s, total: 8min 27s
Wall time: 2min 36s
No handlers could be found for logger "detex.getdata.__init__"
We see the wall time for the createCluster call was around 2 minutes (on my computer). Let’s make a function to see how many of the original 220 events were actually used
def check_cluster(cl):
for c in cl:
sta = c.station
num_events = len(c.key)
print '%s had %d events used in the analysis' % (sta, num_events)
print '\n'
def get_unused_events(cl, temkey):
for c in cl:
sta = c.station
unused = list(set(temkey.NAME) - set(c.key))
print 'Unused events on %s are:\n %s\n' % (sta, unused)
def get_info(cl, temkey_in='TemplateKey.csv'):
temkey = detex.util.readKey(temkey_in, 'template')
print 'There are %d events in the template key' % len(temkey)
check_cluster(cl)
get_unused_events(cl, temkey)
get_info(cl)
There are 222 events in the template key
UU.IMU had 213 events used in the analysis
UU.MSU had 214 events used in the analysis
Unused events on UU.IMU are:
['2005-07-28T16-14-45.72', '2004-07-29T23-09-58.55', '2005-01-02T06-35-22.46', '2010-03-04T00-02-53.67', '2005-07-30T08-58-00.45', '2005-12-21T04-48-05.24', '2005-09-30T13-47-54.79', '2005-11-28T16-18-36.82', '2009-10-07T18-44-39.86']
Unused events on UU.MSU are:
['2005-09-30T13-47-54.79', '2004-05-19T20-44-17.41', '2004-07-29T23-09-58.55', '2005-01-02T06-35-22.46', '2005-12-21T04-48-05.24', '2005-01-14T01-30-48.97', '2005-08-09T06-35-29.44', '2006-02-05T23-33-43.00']
Now let’s try using fillZeros as True rather than the default of False. This will force each event waveform to be exactly the length defined by the trim parameter by filling with zeros where necessary.
%time cl2 = detex.createCluster(fillZeros=True)
Cannot remove response without a valid inventoryArg, setting removeResponse to False
Starting IO operations and data checks
performing cluster analysis on UU.IMU
performing cluster analysis on UU.MSU
ccReq for station UU.IMU updated to ccReq=0.500
ccReq for station UU.MSU updated to ccReq=0.500
writing ClusterStream instance as clust.pkl
CPU times: user 2min 43s, sys: 5min 36s, total: 8min 20s
Wall time: 2min 24s
get_info(cl2)
There are 222 events in the template key
UU.IMU had 218 events used in the analysis
UU.MSU had 222 events used in the analysis
Unused events on UU.IMU are:
['2005-07-28T16-14-45.72', '2009-10-07T18-44-39.86', '2010-03-04T00-02-53.67', '2005-07-30T08-58-00.45']
Unused events on UU.MSU are:
[]
So setting fill_zeros to True caused detex to use all the events on MSU and all but four on IMU. The four IMU events that went unused were probably due to missing waveforms. We can verify this by looking in the log for indications the that the data were not available to download.
log = detex.util.readLog()
log
| Time | Mod | Level | Msg | |
|---|---|---|---|---|
| 0 | 2015-12-14 22:26:20,753 | detex | INFO | Starting logging, path to log file: /home/derr... |
| 1 | 2015-12-14 22:26:21,050 | detex.getdata.makeDataDirectories | INFO | Getting template waveforms |
| 2 | 2015-12-14 22:26:22,396 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 3 | 2015-12-14 22:26:22,402 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 4 | 2015-12-14 22:26:22,407 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 5 | 2015-12-14 22:26:22,413 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 6 | 2015-12-14 22:26:22,419 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 7 | 2015-12-14 22:26:22,424 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 8 | 2015-12-14 22:26:22,430 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 9 | 2015-12-14 22:26:23,903 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 10 | 2015-12-14 22:26:23,909 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 11 | 2015-12-14 22:26:23,915 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 12 | 2015-12-14 22:26:23,921 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 13 | 2015-12-14 22:26:23,926 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 14 | 2015-12-14 22:26:25,190 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 15 | 2015-12-14 22:26:25,196 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 16 | 2015-12-14 22:26:25,202 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 17 | 2015-12-14 22:26:25,208 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 18 | 2015-12-14 22:26:25,213 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 19 | 2015-12-14 22:26:25,219 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 20 | 2015-12-14 22:26:25,224 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 21 | 2015-12-14 22:26:25,230 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 22 | 2015-12-14 22:26:25,235 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 23 | 2015-12-14 22:26:26,237 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 24 | 2015-12-14 22:26:26,244 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 25 | 2015-12-14 22:26:26,250 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 26 | 2015-12-14 22:26:26,255 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 27 | 2015-12-14 22:26:26,261 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 28 | 2015-12-14 22:26:26,267 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 29 | 2015-12-14 22:26:26,272 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| ... | ... | ... | ... | ... |
| 174 | 2015-12-14 22:30:21,939 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 175 | 2015-12-14 22:30:21,945 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 176 | 2015-12-14 22:30:21,951 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 177 | 2015-12-14 22:30:21,956 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 178 | 2015-12-14 22:30:21,962 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 179 | 2015-12-14 22:30:21,968 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 180 | 2015-12-14 22:30:21,973 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 181 | 2015-12-14 22:30:21,979 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 182 | 2015-12-14 22:30:21,984 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 183 | 2015-12-14 22:30:21,990 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 184 | 2015-12-14 22:30:23,109 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 185 | 2015-12-14 22:30:24,314 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 186 | 2015-12-14 22:30:24,319 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 187 | 2015-12-14 22:30:24,325 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 188 | 2015-12-14 22:30:24,331 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 189 | 2015-12-14 22:30:24,336 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 190 | 2015-12-14 22:30:24,342 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 191 | 2015-12-14 22:30:24,347 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 192 | 2015-12-14 22:30:25,736 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 193 | 2015-12-14 22:30:25,742 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 194 | 2015-12-14 22:30:25,748 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 195 | 2015-12-14 22:30:25,754 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 196 | 2015-12-14 22:30:25,760 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 197 | 2015-12-14 22:30:26,965 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 198 | 2015-12-14 22:30:26,972 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 199 | 2015-12-14 22:30:26,979 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 200 | 2015-12-14 22:30:26,986 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 201 | 2015-12-14 22:30:26,993 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 202 | 2015-12-14 22:33:46,231 | detex.getdata.indexDirectory | INFO | EventWaveForms is not indexed, indexing now |
| 203 | 2015-12-14 22:33:48,172 | detex.getdata.makeDataDirectories | INFO | finished makeDataDirectories call |
204 rows × 4 columns
Time Trials¶
If you are trying to perform waveform clustering on a large data set it may be worth your time to understand how varying certain parameters can affect runtimes. Let’s isolate a few variables and compare run times from the default values. If you are running this on your computer at home it may take some time, skip ahead if you aren’t interested.
# Setup code for time trials
import time
def timeit(func): # decorator for timing function calls
def wraper(*args, **kwargs):
t = time.time()
out = func(*args, **kwargs)
return (time.time() - t, out)
return wraper
@timeit
def time_cluster(*args, **kwargs):
detex.createCluster(*args, **kwargs)
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
detex.verbose = False # silence detex
cols = ['waveform_duration', 'run_time']
df = pd.DataFrame(columns=cols)
trims = [(10, 120), (5, 60), (2, 30), (1, 15)]
for trim in trims:
rt = time_cluster(trim=trim)[0]
ser = pd.Series([sum(trim), rt], index=cols)
df.loc[len(df)] = ser
plt.plot(df.waveform_duration, df.run_time)
plt.title("Waveform Length vs Run Times")
plt.ylabel("run times (seconds)")
plt.xlabel("waveform lengths (seconds)")
<matplotlib.text.Text at 0x7f0ea9cc1290>
png
cols = ['num_events', 'run_time']
df = pd.DataFrame(columns=cols)
temkey = detex.util.readKey("TemplateKey.csv", "template")
temkey_lengths = [10, 20, 50, 100, 150, 200]
for tkl in temkey_lengths:
temkey2 = temkey.copy()
rt = time_cluster(templateKey=temkey2[:tkl+1])[0]
ser = pd.Series([tkl, rt], index=cols)
df.loc[len(df)] = ser
plt.plot(df.num_events, df.run_time)
plt.title("Number of Events vs Runtimes")
plt.xlabel("Number of Events")
plt.ylabel("Runtimes (seconds)")
<matplotlib.text.Text at 0x7f0e2c93a5d0>
png
Although a bit more complicated than this, we could qualitatively estimate that changing the waveform length scales the runtime by approximately N (linearly with time) whereas the number of events scales the runtime by approximately N2 (quadratic with time). Let’s see how decimating the data changes the runtimes.
# Test various decimation factors
rt_base = time_cluster()[0]
rt_decimate = time_cluster(decimate=2)[0]
print("Base run time: %.02f, Decimated run time: %.02f" % (rt_base, rt_decimate))
Base run time: 136.88, Decimated run time: 101.74
Interestingly, this didn’t seem to make much of a difference. The original data were sampled at 100 Hz so using a decimation factor of 2 would have reduced the sampling rate to 50 Hz. Since we left the default bandpass filter (1.0 to 10.0 Hz) it might make sense to use a decimation factor of 4 in order to bring the sampling rate down to 25 Hz.
ClusterStream and Cluster Classes¶
The ClusterStream and Cluster classes are used to control and visualize waveform similarity clustering. These classes are required to define the subspaces used in the detection process.
The ClusterStream is a container for one or more Cluster instances. There is a cluster instance for each station, although most attributes are accessible from the ClusterStream level. Let’s take create a ClusterStream instance and take a closer look.
import detex # reimport so we can start here
detex.verbose = False
cl = detex.createCluster()
The bulk of the information for the ClusterStream is stored in the trdf attribute, which, of course, is a pandas DataFrame.
cl.trdf
| Station | Link | CCs | Lags | Subsamp | Events | Stats | |
|---|---|---|---|---|---|---|---|
| 0 | UU.IMU | [[194.0, 195.0, 0.00121253182149, 2.0], [105.0... | 1 2 3 4 ... | 1 2 3 4 5 6 7 8... | 1 2 3 4 ... | [2002-02-01T04-37-29.94, 2002-02-02T03-38-36.2... | {u'2002-11-09T21-38-33.05': {u'Nc': 1, u'sampl... |
| 1 | UU.MSU | [[195.0, 196.0, 0.0035685379758, 2.0], [158.0,... | 1 2 3 4 ... | 1 2 3 4 5 6 7 ... | 1 2 3 4 ... | [2002-02-01T04-37-29.94, 2002-02-02T03-38-36.2... | {u'2002-11-09T21-38-33.05': {u'Nc': 1, u'sampl... |
In this DataFrame there is a row for each station. The columns are:
| Column | Description |
|---|---|
| CCs | A matrix of max correlation coef for each station pair |
| Lags | A matrix of lag samples corresponding to the highest correlation coef |
| Subsamp | The decimal fraction determined by subsample extrapolation |
| Events | The name of the events used |
| Stats | Selected stats of the events |
The CCs and Lags are DataFrames that have indices and rows that correspond to an element in the Events list. This is probably best illustrated by an example. Let’s say we want to find the max correlation ceof. between two events and the corresponding number of samples that would be required to shift the first event to line up with the second. First, we need to find where the events we want to find occur in the events list, then we can index them in the lags and ccs.
# Here are two events in the list
ev1 = '2010-07-10T08-57-51.25'
ev2 = '2014-11-29T14-18-04.87'
events = list(cl.trdf.loc[0, 'Events']) # cast from np array to list
# Find the index where each event occurs in the list
ev1_ind = events.index(ev1)
ev2_ind = events.index(ev2)
print ("%s index is %d, %s index is %d" % (ev1, ev1_ind, ev2, ev2_ind))
2010-07-10T08-57-51.25 index is 125, 2014-11-29T14-18-04.87 index is 210
cc = cl.trdf.loc[0, 'CCs']
lags = cl.trdf.loc[0, 'Lags']
coef = cc.loc[ev1_ind, ev2_ind]
lag = lags.loc[ev1_ind, ev2_ind]
print (coef, lag)
# events
(0.12697600464301972, 263)
Visualization Methods¶
The ClusterStream has several methods for visualizing. We can create a simple similarity matrix.
cl.simMatrix()
png
png
By default the events (x and y axis) are ordered based on origin time. We can also plot them based on the groups the events best fit in.
cl.simMatrix(groupClusts=True)
png
png
We can visualize and change the clustering structure for each station with the dendro and updateReqCC methods, just as in the intro tutorial.
cl.dendro()
png
png
cl[0].updateReqCC(.6)
cl[0].dendro()
png
We can plot the spatial relations of the events with the plotEvents method. This is used to get a quick and dirty idea of event locations and depths; it still needs a lot of work before it will produce presentable plots. The following is not the best example of a meaningful plot because there are so many colors and different groups but plotEvents can be useful, especially on smaller datasets.
cl[0].plotEvents()
png
png
png
Next section¶
The next section covers subspace detection.