# Clustering This segment of the tutorial will teach you how to preform waveform similarity clustering in detex. The function used to preform clustering is createCluster of the construct module. The results are then stored in an instance of the ClusterStream class. Let's start by looking at the createCluster function and some important parameters that go into it, then we will look at the ClusterStream class its methods. ## CreateCluster ```python import detex version = detex.__version__ print ("Detex version is %s\n" % version) print (detex.construct.createCluster.__doc__) ``` Detex version is 1.0.5 Function to create an instance of the ClusterStream class Parameters ------- CCreq : float, between 0 and 1 The minimum correlation coefficient for grouping waveforms. 0.0 results in all waveforms grouping together and 1.0 will not form any groups. fetch_arg : str or detex.getdata.DataFetcher instance Fetch_arg of detex.getdata.quickFetch, see docs for details. filt : list A list of the required input parameters for the obspy bandpass filter [freqmin, freqmax, corners, zerophase]. stationKey : str or pd.DataFrame Path to the station key or DataFrame of station key. templateKey : str or pd.DataFrame Path to the template key or loaded template key in DataFrame. trim : list A list with seconds to trim from events with respect to the origin time reported in the template key. The default value of [10, 120] means each event will be trimmed to only contain 10 seconds before its origin time and 120 seconds after. The larger the values of this argument the longer the computation time and chance of misalignment, but values that are too small may trim out desired phases of the waveform. saveClust : bool If true save the cluster object in the current working directory. The name is controlled by the fileName parameter. fileName : str Path (or name) to save the clustering instance, only used if saveClust is True. decimate : int or None A decimation factor to apply to all data in order to decrease run time. Can be very useful if the the data are oversampled. For example, if the data are sampled at 200 Hz but a 1 to 10 Hz bandpass filter is applied it may be appropriate to apply a decimation factor of 5 to bring the sampling rate down to 40 hz. dytpe : str An option to recast data type of the seismic data. Options are: double- numpy float 64 single- numpy float 32, much faster and amenable to cuda GPU processing, sacrifices precision. eventsOnAllStations : bool If True only use the events that occur on all stations, if false let each station have an independent event list. enforceOrigin : bool If True make sure each trace starts at the reported origin time in the template key. If not trim or merge with zeros. Required for lag times to be meaningful for hypoDD input. fillZeros : bool If True fill zeros from trim[0] to trim[1]. Suggested for older data or if only triggered data are available. Returns --------- An instance of the detex SSClustering class As you can see there are a lot of input arguments, and a lot to think about when creating a cluster object. Let me elaborate on some of the arguments you should pay special attention to. * fet_arg - make sure to look at the detex.getdata.quickFetch docs for this one. Basically, if you want to use a custom DataFetcher be sure to pass it to the createCluster call here or else detex will try to use a local directory with the default name of EventWaveForms. * filt - parameters to apply a bandpass filter to the waveform similarity clustering and ALL all detex downstream operations. Make sure to think about this carefully before simply using the default, as the default values are not appropriate for all data sets. * fillZeros - a parameter for handling data with gaps. If data are not avaliable for the entire range (defined by template key and trim parameter) detex will simply fill zeros so that each trace will have the length defined by the trim parameter. The created cluster instance can then be used later on by detex, although you should be careful going forward to no include a bunch of the zero data in your detector, more on that later. * trim - a two element list that defines the length of each waveform. The first element is the time before the origin (as reported in the station key) and the second element is the number of seconds after the reported origin time. ### Dealing with gaps In order to see how some of these parameters affect the clustering process we will look at an early UUSS dataset that has some issues with gaps. Here are the stations and templates: ```python stakey = detex.util.readKey('StationKey.csv', key_type='station') stakey ```
NETWORK STATION STARTTIME ENDTIME LAT LON ELEVATION CHANNELS
0 UU IMU 2003-01-03T00:00:00 2004-04-04T00:00:00 38.6332 -113.158 1833 EHZ
1 UU MSU 2003-01-01T00:00:00 2004-04-04T00:00:00 38.5123 -112.177 2105 EHZ
```python temkey = detex.util.readKey('TemplateKey.csv', key_type='template') temkey ```
TIME NAME LAT LON MAG DEPTH
0 2002-02-01T04-37-29.94 2002-02-01T04-37-29.94 38.616833 -112.463833 1.40 2.82
1 2002-02-02T03-38-36.21 2002-02-02T03-38-36.21 38.575667 -112.714500 1.19 1.14
2 2002-02-22T22-59-28.69 2002-02-22T22-59-28.69 38.558667 -112.460500 2.17 1.72
3 2002-02-22T23-10-36.58 2002-02-22T23-10-36.58 38.538667 -112.455500 0.99 5.82
4 2002-02-26T11-45-48.87 2002-02-26T11-45-48.87 38.527333 -112.467667 1.98 4.01
5 2002-02-26T12-09-47.92 2002-02-26T12-09-47.92 38.558500 -112.456500 2.39 -0.28
6 2002-03-04T02-53-36.92 2002-03-04T02-53-36.92 38.708000 -112.535500 1.43 -0.14
7 2002-03-04T17-13-19.61 2002-03-04T17-13-19.61 38.542333 -112.455333 1.33 1.84
8 2002-03-05T00-32-02.35 2002-03-05T00-32-02.35 38.544667 -112.457667 1.34 2.80
9 2002-03-10T16-03-15.42 2002-03-10T16-03-15.42 38.722000 -112.545000 2.50 -2.97
10 2002-03-21T20-16-14.74 2002-03-21T20-16-14.74 38.711333 -112.541167 2.04 1.81
11 2002-03-23T16-39-08.76 2002-03-23T16-39-08.76 38.701667 -112.542333 1.32 1.23
12 2002-05-12T18-32-19.43 2002-05-12T18-32-19.43 38.724333 -112.539500 0.71 1.42
13 2002-05-15T15-35-00.17 2002-05-15T15-35-00.17 38.588667 -112.586833 0.73 3.00
14 2002-07-06T22-05-16.73 2002-07-06T22-05-16.73 38.512000 -112.460833 0.35 5.78
15 2002-08-01T20-19-21.00 2002-08-01T20-19-21.00 38.525833 -112.455500 1.03 1.84
16 2002-08-19T19-35-07.44 2002-08-19T19-35-07.44 38.689000 -112.543500 0.72 5.96
17 2002-08-19T21-09-35.83 2002-08-19T21-09-35.83 38.685833 -112.541833 1.17 4.54
18 2002-09-07T19-55-27.85 2002-09-07T19-55-27.85 38.713000 -112.647500 1.02 2.79
19 2002-09-17T14-50-53.10 2002-09-17T14-50-53.10 38.663167 -112.630000 0.71 0.31
20 2002-10-06T11-57-35.19 2002-10-06T11-57-35.19 38.645667 -112.461667 1.31 3.64
21 2002-11-05T23-12-44.72 2002-11-05T23-12-44.72 38.532333 -112.466167 0.95 6.02
22 2002-11-05T23-13-27.56 2002-11-05T23-13-27.56 38.555167 -112.468167 1.37 2.94
23 2002-11-06T00-44-00.38 2002-11-06T00-44-00.38 38.527833 -112.482167 2.05 4.09
24 2002-11-06T01-48-04.90 2002-11-06T01-48-04.90 38.532167 -112.463833 1.10 3.15
25 2002-11-06T02-24-03.05 2002-11-06T02-24-03.05 38.515833 -112.463833 0.61 6.02
26 2002-11-08T18-27-28.27 2002-11-08T18-27-28.27 38.534667 -112.468500 1.29 2.43
27 2002-11-08T21-48-47.69 2002-11-08T21-48-47.69 38.530833 -112.470500 2.26 2.88
28 2002-11-08T22-00-02.41 2002-11-08T22-00-02.41 38.519500 -112.468000 1.12 2.07
29 2002-11-09T21-38-33.05 2002-11-09T21-38-33.05 38.548167 -112.468333 1.22 7.83
... ... ... ... ... ... ...
192 2013-10-10T16-10-37.49 2013-10-10T16-10-37.49 38.597167 -112.598167 1.96 5.21
193 2013-10-10T23-12-49.39 2013-10-10T23-12-49.39 38.606500 -112.612833 2.48 3.32
194 2013-10-10T23-21-10.54 2013-10-10T23-21-10.54 38.601833 -112.606333 2.36 3.91
195 2013-10-11T00-26-32.78 2013-10-11T00-26-32.78 38.603833 -112.606833 1.34 -0.16
196 2013-10-11T10-36-01.05 2013-10-11T10-36-01.05 38.613667 -112.607667 0.86 -0.01
197 2013-10-11T15-19-47.68 2013-10-11T15-19-47.68 38.598167 -112.608333 1.96 4.14
198 2013-10-24T09-57-46.49 2013-10-24T09-57-46.49 38.589167 -112.606333 1.54 1.48
199 2013-11-01T02-56-52.97 2013-11-01T02-56-52.97 38.734500 -112.567167 1.31 0.69
200 2013-11-10T15-36-35.26 2013-11-10T15-36-35.26 38.714667 -112.526333 0.70 9.28
201 2013-12-25T05-32-32.15 2013-12-25T05-32-32.15 38.633833 -112.539500 0.78 -0.10
202 2014-01-09T00-10-03.16 2014-01-09T00-10-03.16 38.554833 -112.728833 1.93 -0.24
203 2014-01-17T12-52-52.03 2014-01-17T12-52-52.03 38.637500 -112.612000 -9.99 1.49
204 2014-01-17T12-52-55.73 2014-01-17T12-52-55.73 38.609667 -112.613000 1.84 0.45
205 2014-01-17T16-16-23.71 2014-01-17T16-16-23.71 38.607667 -112.603667 0.96 1.93
206 2014-02-08T07-14-59.47 2014-02-08T07-14-59.47 38.664500 -112.535667 0.86 4.32
207 2014-02-10T00-32-14.92 2014-02-10T00-32-14.92 38.734333 -112.521500 1.24 2.86
208 2014-03-08T20-37-13.43 2014-03-08T20-37-13.43 38.722000 -112.552667 0.94 4.96
209 2014-03-23T01-26-16.64 2014-03-23T01-26-16.64 38.674167 -112.585833 1.92 -3.09
210 2014-04-06T08-19-55.40 2014-04-06T08-19-55.40 38.563000 -112.690167 1.92 1.88
211 2014-05-27T19-04-32.17 2014-05-27T19-04-32.17 38.594833 -112.593833 1.20 0.38
212 2014-06-05T03-14-06.76 2014-06-05T03-14-06.76 38.683667 -112.533167 2.01 3.51
213 2014-06-09T11-00-38.18 2014-06-09T11-00-38.18 38.727000 -112.503167 0.57 1.75
214 2014-06-12T06-03-10.42 2014-06-12T06-03-10.42 38.678167 -112.539500 1.70 0.99
215 2014-07-11T03-47-35.36 2014-07-11T03-47-35.36 38.689333 -112.579500 0.78 3.96
216 2014-10-29T10-58-38.16 2014-10-29T10-58-38.16 38.574833 -112.587333 1.64 3.12
217 2014-10-29T11-01-00.32 2014-10-29T11-01-00.32 38.592167 -112.594500 0.94 0.46
218 2014-11-15T16-47-50.40 2014-11-15T16-47-50.40 38.586167 -112.600333 0.79 2.93
219 2014-11-29T14-18-04.87 2014-11-29T14-18-04.87 38.698167 -112.544500 0.89 0.16
220 2014-12-17T08-52-09.85 2014-12-17T08-52-09.85 38.579500 -112.589833 1.47 0.23
221 2014-12-21T17-01-29.91 2014-12-21T17-01-29.91 38.586500 -112.587167 0.56 0.69

222 rows × 6 columns

Because there are so many events it may take some time to get the data. We will skip getting the continuous data because it is not needed for this section of the tutorial. We should probably also start the logger in case we need more info than what is printed to the screen. We will also delete an old logger if there is one. ```python import os if os.path.exists("detex_log.log"): os.remove("detex_log.log") detex.setLogger() detex.getdata.makeDataDirectories(getContinuous=False) ``` Getting template waveforms EventWaveForms is not indexed, indexing now finished makeDataDirectories call /home/derrick/anaconda/lib/python2.7/site-packages/obspy/mseed/core.py:610: UserWarning: The encoding specified in trace.stats.mseed.encoding does not match the dtype of the data. A suitable encoding will be chosen. warnings.warn(msg, UserWarning) Now we will cluster these events while varying the input arguments. Let's start by using the defaults. ```python %time cl = detex.createCluster() # notice we can call createCluster from the detex level ``` Cannot remove response without a valid inventoryArg, setting removeResponse to False Starting IO operations and data checks 2004-07-29T23-09-58.55 on UU.IMU is out of length tolerance, removing 2005-01-02T06-35-22.46 on UU.IMU is out of length tolerance, removing 2005-09-30T13-47-54.79 on UU.IMU is out of length tolerance, removing 2005-11-28T16-18-36.82 on UU.IMU is out of length tolerance, removing 2005-12-21T04-48-05.24 on UU.IMU is out of length tolerance, removing 2004-07-29T23-09-58.55 is fractured or missing data, removing 2004-05-19T20-44-17.41 on UU.MSU is out of length tolerance, removing 2005-01-02T06-35-22.46 on UU.MSU is out of length tolerance, removing 2005-01-14T01-30-48.97 on UU.MSU is out of length tolerance, removing 2005-08-09T06-35-29.44 on UU.MSU is out of length tolerance, removing 2005-09-30T13-47-54.79 on UU.MSU is out of length tolerance, removing 2005-12-21T04-48-05.24 on UU.MSU is out of length tolerance, removing 2006-02-05T23-33-43.00 on UU.MSU is out of length tolerance, removing performing cluster analysis on UU.IMU performing cluster analysis on UU.MSU ccReq for station UU.IMU updated to ccReq=0.500 ccReq for station UU.MSU updated to ccReq=0.500 writing ClusterStream instance as clust.pkl CPU times: user 2min 36s, sys: 5min 50s, total: 8min 27s Wall time: 2min 36s No handlers could be found for logger "detex.getdata.__init__" We see the wall time for the createCluster call was around 2 minutes (on my computer). Let's make a function to see how many of the original 220 events were actually used ```python def check_cluster(cl): for c in cl: sta = c.station num_events = len(c.key) print '%s had %d events used in the analysis' % (sta, num_events) print '\n' def get_unused_events(cl, temkey): for c in cl: sta = c.station unused = list(set(temkey.NAME) - set(c.key)) print 'Unused events on %s are:\n %s\n' % (sta, unused) def get_info(cl, temkey_in='TemplateKey.csv'): temkey = detex.util.readKey(temkey_in, 'template') print 'There are %d events in the template key' % len(temkey) check_cluster(cl) get_unused_events(cl, temkey) get_info(cl) ``` There are 222 events in the template key UU.IMU had 213 events used in the analysis UU.MSU had 214 events used in the analysis Unused events on UU.IMU are: ['2005-07-28T16-14-45.72', '2004-07-29T23-09-58.55', '2005-01-02T06-35-22.46', '2010-03-04T00-02-53.67', '2005-07-30T08-58-00.45', '2005-12-21T04-48-05.24', '2005-09-30T13-47-54.79', '2005-11-28T16-18-36.82', '2009-10-07T18-44-39.86'] Unused events on UU.MSU are: ['2005-09-30T13-47-54.79', '2004-05-19T20-44-17.41', '2004-07-29T23-09-58.55', '2005-01-02T06-35-22.46', '2005-12-21T04-48-05.24', '2005-01-14T01-30-48.97', '2005-08-09T06-35-29.44', '2006-02-05T23-33-43.00'] Now let's try using fillZeros as True rather than the default of False. This will force each event waveform to be exactly the length defined by the trim parameter by filling with zeros where necessary. ```python %time cl2 = detex.createCluster(fillZeros=True) ``` Cannot remove response without a valid inventoryArg, setting removeResponse to False Starting IO operations and data checks performing cluster analysis on UU.IMU performing cluster analysis on UU.MSU ccReq for station UU.IMU updated to ccReq=0.500 ccReq for station UU.MSU updated to ccReq=0.500 writing ClusterStream instance as clust.pkl CPU times: user 2min 43s, sys: 5min 36s, total: 8min 20s Wall time: 2min 24s ```python get_info(cl2) ``` There are 222 events in the template key UU.IMU had 218 events used in the analysis UU.MSU had 222 events used in the analysis Unused events on UU.IMU are: ['2005-07-28T16-14-45.72', '2009-10-07T18-44-39.86', '2010-03-04T00-02-53.67', '2005-07-30T08-58-00.45'] Unused events on UU.MSU are: [] So setting fill_zeros to True caused detex to use all the events on MSU and all but four on IMU. The four IMU events that went unused were probably due to missing waveforms. We can verify this by looking in the log for indications the that the data were not available to download. ```python log = detex.util.readLog() log ```
Time Mod Level Msg
0 2015-12-14 22:26:20,753 detex INFO Starting logging, path to log file: /home/derr...
1 2015-12-14 22:26:21,050 detex.getdata.makeDataDirectories INFO Getting template waveforms
2 2015-12-14 22:26:22,396 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
3 2015-12-14 22:26:22,402 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
4 2015-12-14 22:26:22,407 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
5 2015-12-14 22:26:22,413 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
6 2015-12-14 22:26:22,419 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
7 2015-12-14 22:26:22,424 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
8 2015-12-14 22:26:22,430 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
9 2015-12-14 22:26:23,903 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
10 2015-12-14 22:26:23,909 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
11 2015-12-14 22:26:23,915 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
12 2015-12-14 22:26:23,921 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
13 2015-12-14 22:26:23,926 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
14 2015-12-14 22:26:25,190 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
15 2015-12-14 22:26:25,196 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
16 2015-12-14 22:26:25,202 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
17 2015-12-14 22:26:25,208 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
18 2015-12-14 22:26:25,213 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
19 2015-12-14 22:26:25,219 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
20 2015-12-14 22:26:25,224 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
21 2015-12-14 22:26:25,230 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
22 2015-12-14 22:26:25,235 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
23 2015-12-14 22:26:26,237 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
24 2015-12-14 22:26:26,244 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
25 2015-12-14 22:26:26,250 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
26 2015-12-14 22:26:26,255 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
27 2015-12-14 22:26:26,261 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
28 2015-12-14 22:26:26,267 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
29 2015-12-14 22:26:26,272 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
... ... ... ... ...
174 2015-12-14 22:30:21,939 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
175 2015-12-14 22:30:21,945 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
176 2015-12-14 22:30:21,951 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
177 2015-12-14 22:30:21,956 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
178 2015-12-14 22:30:21,962 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
179 2015-12-14 22:30:21,968 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
180 2015-12-14 22:30:21,973 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
181 2015-12-14 22:30:21,979 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
182 2015-12-14 22:30:21,984 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
183 2015-12-14 22:30:21,990 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
184 2015-12-14 22:30:23,109 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
185 2015-12-14 22:30:24,314 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
186 2015-12-14 22:30:24,319 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
187 2015-12-14 22:30:24,325 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
188 2015-12-14 22:30:24,331 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
189 2015-12-14 22:30:24,336 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
190 2015-12-14 22:30:24,342 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
191 2015-12-14 22:30:24,347 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
192 2015-12-14 22:30:25,736 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
193 2015-12-14 22:30:25,742 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
194 2015-12-14 22:30:25,748 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
195 2015-12-14 22:30:25,754 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
196 2015-12-14 22:30:25,760 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
197 2015-12-14 22:30:26,965 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
198 2015-12-14 22:30:26,972 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
199 2015-12-14 22:30:26,979 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
200 2015-12-14 22:30:26,986 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
201 2015-12-14 22:30:26,993 detex.getdata._dataCheck WARNING Found non-int sampling_rates, rounded to neare...
202 2015-12-14 22:33:46,231 detex.getdata.indexDirectory INFO EventWaveForms is not indexed, indexing now
203 2015-12-14 22:33:48,172 detex.getdata.makeDataDirectories INFO finished makeDataDirectories call

204 rows × 4 columns

### Time Trials If you are trying to perform waveform clustering on a large data set it may be worth your time to understand how varying certain parameters can affect runtimes. Let's isolate a few variables and compare run times from the default values. If you are running this on your computer at home it may take some time, skip ahead if you aren't interested. ```python # Setup code for time trials import time def timeit(func): # decorator for timing function calls def wraper(*args, **kwargs): t = time.time() out = func(*args, **kwargs) return (time.time() - t, out) return wraper @timeit def time_cluster(*args, **kwargs): detex.createCluster(*args, **kwargs) ``` ```python import pandas as pd import matplotlib.pyplot as plt %matplotlib inline detex.verbose = False # silence detex cols = ['waveform_duration', 'run_time'] df = pd.DataFrame(columns=cols) trims = [(10, 120), (5, 60), (2, 30), (1, 15)] for trim in trims: rt = time_cluster(trim=trim)[0] ser = pd.Series([sum(trim), rt], index=cols) df.loc[len(df)] = ser plt.plot(df.waveform_duration, df.run_time) plt.title("Waveform Length vs Run Times") plt.ylabel("run times (seconds)") plt.xlabel("waveform lengths (seconds)") ``` ![png](output_18_1.png) ```python cols = ['num_events', 'run_time'] df = pd.DataFrame(columns=cols) temkey = detex.util.readKey("TemplateKey.csv", "template") temkey_lengths = [10, 20, 50, 100, 150, 200] for tkl in temkey_lengths: temkey2 = temkey.copy() rt = time_cluster(templateKey=temkey2[:tkl+1])[0] ser = pd.Series([tkl, rt], index=cols) df.loc[len(df)] = ser plt.plot(df.num_events, df.run_time) plt.title("Number of Events vs Runtimes") plt.xlabel("Number of Events") plt.ylabel("Runtimes (seconds)") ``` ![png](output_19_1.png) Although a bit more complicated than this, we could qualitatively estimate that changing the waveform length scales the runtime by approximately N (linearly with time) whereas the number of events scales the runtime by approximately N2 (quadratic with time). Let's see how decimating the data changes the runtimes. ```python # Test various decimation factors rt_base = time_cluster()[0] rt_decimate = time_cluster(decimate=2)[0] print("Base run time: %.02f, Decimated run time: %.02f" % (rt_base, rt_decimate)) ``` Base run time: 136.88, Decimated run time: 101.74 Interestingly, this didn't seem to make much of a difference. The original data were sampled at 100 Hz so using a decimation factor of 2 would have reduced the sampling rate to 50 Hz. Since we left the default bandpass filter (1.0 to 10.0 Hz) it might make sense to use a decimation factor of 4 in order to bring the sampling rate down to 25 Hz. ## ClusterStream and Cluster Classes The ClusterStream and Cluster classes are used to control and visualize waveform similarity clustering. These classes are required to define the subspaces used in the detection process. The ClusterStream is a container for one or more Cluster instances. There is a cluster instance for each station, although most attributes are accessible from the ClusterStream level. Let's take create a ClusterStream instance and take a closer look. ```python import detex # reimport so we can start here detex.verbose = False cl = detex.createCluster() ``` The bulk of the information for the ClusterStream is stored in the trdf attribute, which, of course, is a pandas DataFrame. ```python cl.trdf ```
Station Link CCs Lags Subsamp Events Stats
0 UU.IMU [[194.0, 195.0, 0.00121253182149, 2.0], [105.0... 1 2 3 4 ... 1 2 3 4 5 6 7 8... 1 2 3 4 ... [2002-02-01T04-37-29.94, 2002-02-02T03-38-36.2... {u'2002-11-09T21-38-33.05': {u'Nc': 1, u'sampl...
1 UU.MSU [[195.0, 196.0, 0.0035685379758, 2.0], [158.0,... 1 2 3 4 ... 1 2 3 4 5 6 7 ... 1 2 3 4 ... [2002-02-01T04-37-29.94, 2002-02-02T03-38-36.2... {u'2002-11-09T21-38-33.05': {u'Nc': 1, u'sampl...
In this DataFrame there is a row for each station. The columns are: | Column | Description | |:-----:| :---------: | | CCs | A matrix of max correlation coef for each station pair | | Lags | A matrix of lag samples corresponding to the highest correlation coef | | Subsamp | The decimal fraction determined by subsample extrapolation | | Events | The name of the events used | | Stats | Selected stats of the events | The CCs and Lags are DataFrames that have indices and rows that correspond to an element in the Events list. This is probably best illustrated by an example. Let's say we want to find the max correlation ceof. between two events and the corresponding number of samples that would be required to shift the first event to line up with the second. First, we need to find where the events we want to find occur in the events list, then we can index them in the lags and ccs. ```python # Here are two events in the list ev1 = '2010-07-10T08-57-51.25' ev2 = '2014-11-29T14-18-04.87' events = list(cl.trdf.loc[0, 'Events']) # cast from np array to list # Find the index where each event occurs in the list ev1_ind = events.index(ev1) ev2_ind = events.index(ev2) print ("%s index is %d, %s index is %d" % (ev1, ev1_ind, ev2, ev2_ind)) ``` 2010-07-10T08-57-51.25 index is 125, 2014-11-29T14-18-04.87 index is 210 ```python cc = cl.trdf.loc[0, 'CCs'] lags = cl.trdf.loc[0, 'Lags'] coef = cc.loc[ev1_ind, ev2_ind] lag = lags.loc[ev1_ind, ev2_ind] print (coef, lag) # events ``` (0.12697600464301972, 263) ### Visualization Methods The ClusterStream has several methods for visualizing. We can create a simple similarity matrix. ```python cl.simMatrix() ``` ![png](output_31_0.png) ![png](output_31_1.png) By default the events (x and y axis) are ordered based on origin time. We can also plot them based on the groups the events best fit in. ```python cl.simMatrix(groupClusts=True) ``` ![png](output_33_0.png) ![png](output_33_1.png) We can visualize and change the clustering structure for each station with the dendro and updateReqCC methods, just as in the intro tutorial. ```python cl.dendro() ``` ![png](output_35_0.png) ![png](output_35_1.png) ```python cl[0].updateReqCC(.6) cl[0].dendro() ``` ![png](output_36_0.png) We can plot the spatial relations of the events with the plotEvents method. This is used to get a quick and dirty idea of event locations and depths; it still needs a lot of work before it will produce presentable plots. The following is not the best example of a meaningful plot because there are so many colors and different groups but plotEvents can be useful, especially on smaller datasets. ```python cl[0].plotEvents() ``` ![png](output_38_0.png) ![png](output_38_1.png) ![png](output_38_2.png) # Next section The [next section](../SubspaceDetection/subspace_detection1.md) covers subspace detection.