# Clustering This segment of the tutorial will teach you how to preform waveform similarity clustering in detex. The function used to preform clustering is createCluster of the construct module. The results are then stored in an instance of the ClusterStream class. Let's start by looking at the createCluster function and some important parameters that go into it, then we will look at the ClusterStream class its methods. ## CreateCluster ```python import detex version = detex.__version__ print ("Detex version is %s\n" % version) print (detex.construct.createCluster.__doc__) ``` Detex version is 1.0.5 Function to create an instance of the ClusterStream class Parameters ------- CCreq : float, between 0 and 1 The minimum correlation coefficient for grouping waveforms. 0.0 results in all waveforms grouping together and 1.0 will not form any groups. fetch_arg : str or detex.getdata.DataFetcher instance Fetch_arg of detex.getdata.quickFetch, see docs for details. filt : list A list of the required input parameters for the obspy bandpass filter [freqmin, freqmax, corners, zerophase]. stationKey : str or pd.DataFrame Path to the station key or DataFrame of station key. templateKey : str or pd.DataFrame Path to the template key or loaded template key in DataFrame. trim : list A list with seconds to trim from events with respect to the origin time reported in the template key. The default value of [10, 120] means each event will be trimmed to only contain 10 seconds before its origin time and 120 seconds after. The larger the values of this argument the longer the computation time and chance of misalignment, but values that are too small may trim out desired phases of the waveform. saveClust : bool If true save the cluster object in the current working directory. The name is controlled by the fileName parameter. fileName : str Path (or name) to save the clustering instance, only used if saveClust is True. decimate : int or None A decimation factor to apply to all data in order to decrease run time. Can be very useful if the the data are oversampled. For example, if the data are sampled at 200 Hz but a 1 to 10 Hz bandpass filter is applied it may be appropriate to apply a decimation factor of 5 to bring the sampling rate down to 40 hz. dytpe : str An option to recast data type of the seismic data. Options are: double- numpy float 64 single- numpy float 32, much faster and amenable to cuda GPU processing, sacrifices precision. eventsOnAllStations : bool If True only use the events that occur on all stations, if false let each station have an independent event list. enforceOrigin : bool If True make sure each trace starts at the reported origin time in the template key. If not trim or merge with zeros. Required for lag times to be meaningful for hypoDD input. fillZeros : bool If True fill zeros from trim[0] to trim[1]. Suggested for older data or if only triggered data are available. Returns --------- An instance of the detex SSClustering class As you can see there are a lot of input arguments, and a lot to think about when creating a cluster object. Let me elaborate on some of the arguments you should pay special attention to. * fet_arg - make sure to look at the detex.getdata.quickFetch docs for this one. Basically, if you want to use a custom DataFetcher be sure to pass it to the createCluster call here or else detex will try to use a local directory with the default name of EventWaveForms. * filt - parameters to apply a bandpass filter to the waveform similarity clustering and ALL all detex downstream operations. Make sure to think about this carefully before simply using the default, as the default values are not appropriate for all data sets. * fillZeros - a parameter for handling data with gaps. If data are not avaliable for the entire range (defined by template key and trim parameter) detex will simply fill zeros so that each trace will have the length defined by the trim parameter. The created cluster instance can then be used later on by detex, although you should be careful going forward to no include a bunch of the zero data in your detector, more on that later. * trim - a two element list that defines the length of each waveform. The first element is the time before the origin (as reported in the station key) and the second element is the number of seconds after the reported origin time. ### Dealing with gaps In order to see how some of these parameters affect the clustering process we will look at an early UUSS dataset that has some issues with gaps. Here are the stations and templates: ```python stakey = detex.util.readKey('StationKey.csv', key_type='station') stakey ```
| NETWORK | STATION | STARTTIME | ENDTIME | LAT | LON | ELEVATION | CHANNELS | |
|---|---|---|---|---|---|---|---|---|
| 0 | UU | IMU | 2003-01-03T00:00:00 | 2004-04-04T00:00:00 | 38.6332 | -113.158 | 1833 | EHZ |
| 1 | UU | MSU | 2003-01-01T00:00:00 | 2004-04-04T00:00:00 | 38.5123 | -112.177 | 2105 | EHZ |
| TIME | NAME | LAT | LON | MAG | DEPTH | |
|---|---|---|---|---|---|---|
| 0 | 2002-02-01T04-37-29.94 | 2002-02-01T04-37-29.94 | 38.616833 | -112.463833 | 1.40 | 2.82 |
| 1 | 2002-02-02T03-38-36.21 | 2002-02-02T03-38-36.21 | 38.575667 | -112.714500 | 1.19 | 1.14 |
| 2 | 2002-02-22T22-59-28.69 | 2002-02-22T22-59-28.69 | 38.558667 | -112.460500 | 2.17 | 1.72 |
| 3 | 2002-02-22T23-10-36.58 | 2002-02-22T23-10-36.58 | 38.538667 | -112.455500 | 0.99 | 5.82 |
| 4 | 2002-02-26T11-45-48.87 | 2002-02-26T11-45-48.87 | 38.527333 | -112.467667 | 1.98 | 4.01 |
| 5 | 2002-02-26T12-09-47.92 | 2002-02-26T12-09-47.92 | 38.558500 | -112.456500 | 2.39 | -0.28 |
| 6 | 2002-03-04T02-53-36.92 | 2002-03-04T02-53-36.92 | 38.708000 | -112.535500 | 1.43 | -0.14 |
| 7 | 2002-03-04T17-13-19.61 | 2002-03-04T17-13-19.61 | 38.542333 | -112.455333 | 1.33 | 1.84 |
| 8 | 2002-03-05T00-32-02.35 | 2002-03-05T00-32-02.35 | 38.544667 | -112.457667 | 1.34 | 2.80 |
| 9 | 2002-03-10T16-03-15.42 | 2002-03-10T16-03-15.42 | 38.722000 | -112.545000 | 2.50 | -2.97 |
| 10 | 2002-03-21T20-16-14.74 | 2002-03-21T20-16-14.74 | 38.711333 | -112.541167 | 2.04 | 1.81 |
| 11 | 2002-03-23T16-39-08.76 | 2002-03-23T16-39-08.76 | 38.701667 | -112.542333 | 1.32 | 1.23 |
| 12 | 2002-05-12T18-32-19.43 | 2002-05-12T18-32-19.43 | 38.724333 | -112.539500 | 0.71 | 1.42 |
| 13 | 2002-05-15T15-35-00.17 | 2002-05-15T15-35-00.17 | 38.588667 | -112.586833 | 0.73 | 3.00 |
| 14 | 2002-07-06T22-05-16.73 | 2002-07-06T22-05-16.73 | 38.512000 | -112.460833 | 0.35 | 5.78 |
| 15 | 2002-08-01T20-19-21.00 | 2002-08-01T20-19-21.00 | 38.525833 | -112.455500 | 1.03 | 1.84 |
| 16 | 2002-08-19T19-35-07.44 | 2002-08-19T19-35-07.44 | 38.689000 | -112.543500 | 0.72 | 5.96 |
| 17 | 2002-08-19T21-09-35.83 | 2002-08-19T21-09-35.83 | 38.685833 | -112.541833 | 1.17 | 4.54 |
| 18 | 2002-09-07T19-55-27.85 | 2002-09-07T19-55-27.85 | 38.713000 | -112.647500 | 1.02 | 2.79 |
| 19 | 2002-09-17T14-50-53.10 | 2002-09-17T14-50-53.10 | 38.663167 | -112.630000 | 0.71 | 0.31 |
| 20 | 2002-10-06T11-57-35.19 | 2002-10-06T11-57-35.19 | 38.645667 | -112.461667 | 1.31 | 3.64 |
| 21 | 2002-11-05T23-12-44.72 | 2002-11-05T23-12-44.72 | 38.532333 | -112.466167 | 0.95 | 6.02 |
| 22 | 2002-11-05T23-13-27.56 | 2002-11-05T23-13-27.56 | 38.555167 | -112.468167 | 1.37 | 2.94 |
| 23 | 2002-11-06T00-44-00.38 | 2002-11-06T00-44-00.38 | 38.527833 | -112.482167 | 2.05 | 4.09 |
| 24 | 2002-11-06T01-48-04.90 | 2002-11-06T01-48-04.90 | 38.532167 | -112.463833 | 1.10 | 3.15 |
| 25 | 2002-11-06T02-24-03.05 | 2002-11-06T02-24-03.05 | 38.515833 | -112.463833 | 0.61 | 6.02 |
| 26 | 2002-11-08T18-27-28.27 | 2002-11-08T18-27-28.27 | 38.534667 | -112.468500 | 1.29 | 2.43 |
| 27 | 2002-11-08T21-48-47.69 | 2002-11-08T21-48-47.69 | 38.530833 | -112.470500 | 2.26 | 2.88 |
| 28 | 2002-11-08T22-00-02.41 | 2002-11-08T22-00-02.41 | 38.519500 | -112.468000 | 1.12 | 2.07 |
| 29 | 2002-11-09T21-38-33.05 | 2002-11-09T21-38-33.05 | 38.548167 | -112.468333 | 1.22 | 7.83 |
| ... | ... | ... | ... | ... | ... | ... |
| 192 | 2013-10-10T16-10-37.49 | 2013-10-10T16-10-37.49 | 38.597167 | -112.598167 | 1.96 | 5.21 |
| 193 | 2013-10-10T23-12-49.39 | 2013-10-10T23-12-49.39 | 38.606500 | -112.612833 | 2.48 | 3.32 |
| 194 | 2013-10-10T23-21-10.54 | 2013-10-10T23-21-10.54 | 38.601833 | -112.606333 | 2.36 | 3.91 |
| 195 | 2013-10-11T00-26-32.78 | 2013-10-11T00-26-32.78 | 38.603833 | -112.606833 | 1.34 | -0.16 |
| 196 | 2013-10-11T10-36-01.05 | 2013-10-11T10-36-01.05 | 38.613667 | -112.607667 | 0.86 | -0.01 |
| 197 | 2013-10-11T15-19-47.68 | 2013-10-11T15-19-47.68 | 38.598167 | -112.608333 | 1.96 | 4.14 |
| 198 | 2013-10-24T09-57-46.49 | 2013-10-24T09-57-46.49 | 38.589167 | -112.606333 | 1.54 | 1.48 |
| 199 | 2013-11-01T02-56-52.97 | 2013-11-01T02-56-52.97 | 38.734500 | -112.567167 | 1.31 | 0.69 |
| 200 | 2013-11-10T15-36-35.26 | 2013-11-10T15-36-35.26 | 38.714667 | -112.526333 | 0.70 | 9.28 |
| 201 | 2013-12-25T05-32-32.15 | 2013-12-25T05-32-32.15 | 38.633833 | -112.539500 | 0.78 | -0.10 |
| 202 | 2014-01-09T00-10-03.16 | 2014-01-09T00-10-03.16 | 38.554833 | -112.728833 | 1.93 | -0.24 |
| 203 | 2014-01-17T12-52-52.03 | 2014-01-17T12-52-52.03 | 38.637500 | -112.612000 | -9.99 | 1.49 |
| 204 | 2014-01-17T12-52-55.73 | 2014-01-17T12-52-55.73 | 38.609667 | -112.613000 | 1.84 | 0.45 |
| 205 | 2014-01-17T16-16-23.71 | 2014-01-17T16-16-23.71 | 38.607667 | -112.603667 | 0.96 | 1.93 |
| 206 | 2014-02-08T07-14-59.47 | 2014-02-08T07-14-59.47 | 38.664500 | -112.535667 | 0.86 | 4.32 |
| 207 | 2014-02-10T00-32-14.92 | 2014-02-10T00-32-14.92 | 38.734333 | -112.521500 | 1.24 | 2.86 |
| 208 | 2014-03-08T20-37-13.43 | 2014-03-08T20-37-13.43 | 38.722000 | -112.552667 | 0.94 | 4.96 |
| 209 | 2014-03-23T01-26-16.64 | 2014-03-23T01-26-16.64 | 38.674167 | -112.585833 | 1.92 | -3.09 |
| 210 | 2014-04-06T08-19-55.40 | 2014-04-06T08-19-55.40 | 38.563000 | -112.690167 | 1.92 | 1.88 |
| 211 | 2014-05-27T19-04-32.17 | 2014-05-27T19-04-32.17 | 38.594833 | -112.593833 | 1.20 | 0.38 |
| 212 | 2014-06-05T03-14-06.76 | 2014-06-05T03-14-06.76 | 38.683667 | -112.533167 | 2.01 | 3.51 |
| 213 | 2014-06-09T11-00-38.18 | 2014-06-09T11-00-38.18 | 38.727000 | -112.503167 | 0.57 | 1.75 |
| 214 | 2014-06-12T06-03-10.42 | 2014-06-12T06-03-10.42 | 38.678167 | -112.539500 | 1.70 | 0.99 |
| 215 | 2014-07-11T03-47-35.36 | 2014-07-11T03-47-35.36 | 38.689333 | -112.579500 | 0.78 | 3.96 |
| 216 | 2014-10-29T10-58-38.16 | 2014-10-29T10-58-38.16 | 38.574833 | -112.587333 | 1.64 | 3.12 |
| 217 | 2014-10-29T11-01-00.32 | 2014-10-29T11-01-00.32 | 38.592167 | -112.594500 | 0.94 | 0.46 |
| 218 | 2014-11-15T16-47-50.40 | 2014-11-15T16-47-50.40 | 38.586167 | -112.600333 | 0.79 | 2.93 |
| 219 | 2014-11-29T14-18-04.87 | 2014-11-29T14-18-04.87 | 38.698167 | -112.544500 | 0.89 | 0.16 |
| 220 | 2014-12-17T08-52-09.85 | 2014-12-17T08-52-09.85 | 38.579500 | -112.589833 | 1.47 | 0.23 |
| 221 | 2014-12-21T17-01-29.91 | 2014-12-21T17-01-29.91 | 38.586500 | -112.587167 | 0.56 | 0.69 |
222 rows × 6 columns
| Time | Mod | Level | Msg | |
|---|---|---|---|---|
| 0 | 2015-12-14 22:26:20,753 | detex | INFO | Starting logging, path to log file: /home/derr... |
| 1 | 2015-12-14 22:26:21,050 | detex.getdata.makeDataDirectories | INFO | Getting template waveforms |
| 2 | 2015-12-14 22:26:22,396 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 3 | 2015-12-14 22:26:22,402 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 4 | 2015-12-14 22:26:22,407 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 5 | 2015-12-14 22:26:22,413 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 6 | 2015-12-14 22:26:22,419 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 7 | 2015-12-14 22:26:22,424 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 8 | 2015-12-14 22:26:22,430 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 9 | 2015-12-14 22:26:23,903 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 10 | 2015-12-14 22:26:23,909 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 11 | 2015-12-14 22:26:23,915 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 12 | 2015-12-14 22:26:23,921 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 13 | 2015-12-14 22:26:23,926 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 14 | 2015-12-14 22:26:25,190 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 15 | 2015-12-14 22:26:25,196 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 16 | 2015-12-14 22:26:25,202 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 17 | 2015-12-14 22:26:25,208 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 18 | 2015-12-14 22:26:25,213 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 19 | 2015-12-14 22:26:25,219 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 20 | 2015-12-14 22:26:25,224 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 21 | 2015-12-14 22:26:25,230 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 22 | 2015-12-14 22:26:25,235 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 23 | 2015-12-14 22:26:26,237 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 24 | 2015-12-14 22:26:26,244 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 25 | 2015-12-14 22:26:26,250 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 26 | 2015-12-14 22:26:26,255 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 27 | 2015-12-14 22:26:26,261 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 28 | 2015-12-14 22:26:26,267 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 29 | 2015-12-14 22:26:26,272 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| ... | ... | ... | ... | ... |
| 174 | 2015-12-14 22:30:21,939 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 175 | 2015-12-14 22:30:21,945 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 176 | 2015-12-14 22:30:21,951 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 177 | 2015-12-14 22:30:21,956 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 178 | 2015-12-14 22:30:21,962 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 179 | 2015-12-14 22:30:21,968 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 180 | 2015-12-14 22:30:21,973 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 181 | 2015-12-14 22:30:21,979 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 182 | 2015-12-14 22:30:21,984 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 183 | 2015-12-14 22:30:21,990 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 184 | 2015-12-14 22:30:23,109 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 185 | 2015-12-14 22:30:24,314 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 186 | 2015-12-14 22:30:24,319 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 187 | 2015-12-14 22:30:24,325 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 188 | 2015-12-14 22:30:24,331 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 189 | 2015-12-14 22:30:24,336 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 190 | 2015-12-14 22:30:24,342 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 191 | 2015-12-14 22:30:24,347 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 192 | 2015-12-14 22:30:25,736 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 193 | 2015-12-14 22:30:25,742 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 194 | 2015-12-14 22:30:25,748 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 195 | 2015-12-14 22:30:25,754 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 196 | 2015-12-14 22:30:25,760 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 197 | 2015-12-14 22:30:26,965 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 198 | 2015-12-14 22:30:26,972 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 199 | 2015-12-14 22:30:26,979 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 200 | 2015-12-14 22:30:26,986 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 201 | 2015-12-14 22:30:26,993 | detex.getdata._dataCheck | WARNING | Found non-int sampling_rates, rounded to neare... |
| 202 | 2015-12-14 22:33:46,231 | detex.getdata.indexDirectory | INFO | EventWaveForms is not indexed, indexing now |
| 203 | 2015-12-14 22:33:48,172 | detex.getdata.makeDataDirectories | INFO | finished makeDataDirectories call |
204 rows × 4 columns
| Station | Link | CCs | Lags | Subsamp | Events | Stats | |
|---|---|---|---|---|---|---|---|
| 0 | UU.IMU | [[194.0, 195.0, 0.00121253182149, 2.0], [105.0... | 1 2 3 4 ... | 1 2 3 4 5 6 7 8... | 1 2 3 4 ... | [2002-02-01T04-37-29.94, 2002-02-02T03-38-36.2... | {u'2002-11-09T21-38-33.05': {u'Nc': 1, u'sampl... |
| 1 | UU.MSU | [[195.0, 196.0, 0.0035685379758, 2.0], [158.0,... | 1 2 3 4 ... | 1 2 3 4 5 6 7 ... | 1 2 3 4 ... | [2002-02-01T04-37-29.94, 2002-02-02T03-38-36.2... | {u'2002-11-09T21-38-33.05': {u'Nc': 1, u'sampl... |