Since this was my first time using a NN, GPU/TPU and Tensorflow I had a steep learning curve. I tried a lot of things and thought I'd share what I learnt.
\nIn summary I used Keras to build a model that
\nIs is very similar to what many others have done. I currently have a AUC score of 0.872, which puts me in the middle of the leaderboard.
\nWhat I learnt from experimenting with this was.
\nThis gave me an experimental test bed for further experimentation.
\nAs I was running a lot of experiments I found that I needed a better way to record the results and started using Neptune.ai, which has been very useful.
\nMy best result so far is B7 with a 512x512 scalogram
", "messages": [ { "id": 1449690, "postDate": "2021-08-05T00:02:07.277Z", "content": "Since this was my first time using a NN, GPU/TPU and Tensorflow I had a steep learning curve. I tried a lot of things and thought I'd share what I learnt.
\nIn summary I used Keras to build a model that
\nIs is very similar to what many others have done. I currently have a AUC score of 0.872, which puts me in the middle of the leaderboard.
\nWhat I learnt from experimenting with this was.
\nThis gave me an experimental test bed for further experimentation.
\nAs I was running a lot of experiments I found that I needed a better way to record the results and started using Neptune.ai, which has been very useful.
\nMy best result so far is B7 with a 512x512 scalogram
", "rawMarkdown": "Since this was my first time using a NN, GPU/TPU and Tensorflow I had a steep learning curve. I tried a lot of things and thought I'd share what I learnt. \n\nIn summary I used Keras to build a model that \n* applies a bandpass filter, \n* rescales the waveform to [-1, 1]\n* Applies a time-frequency transform (e.g., STFT, CQT, CWT) to create a scalogram\n* Classifies using Efficient Net.\n\nIs is very similar to what many others have done. I currently have a AUC score of 0.872, which puts me in the middle of the leaderboard.\n\nWhat I learnt from experimenting with this was. \n* The bandpass filter (20hz-500Hz) has quite an impact. I think this is because the noise below 20 Hz is high and signal is weaker at low frequencies, i.e., very low SNR. I chose 500 Hz because the highest signal frequency for a black hole event is 350 Hz. I could not find articles on the frequency range for other types of events so I picked 500 as there are some strong noise sources just above that. It also looks like the best SNR is around 350-500Hz\n* Rescaling was essential. I'd be interested to hear from an Kaggel Expert if they found this and if so why? \n* I started with nnAudio Constant Quality Transform (CQT) and found this a lot faster than librosa. The entire processing, i.e., load the data, bandpass filter, creating the a 64x64 scalogram in nnAUdio and classifying in EFN B4 took about 8hrs on a CPU and **2hrs on a GPU**. \n* I discovered a TF implementation of Continuous Wavelet Transform and updated it to TF 2.0, optimised the algorithm, and made it a Keras Layer. This enables me to have a single Kera model for the scalogram -> EFN that runs on a TPU. The entire processing time dropped to about **30 min**.\n* I pre-processed the numpy input files into TFRecords and first epoch takes **3 min** and subsequent took 30s.\n\nThis gave me an experimental test bed for further experimentation. \n* Larger scalograms, e.g., 256x256 and 512x512 gives a small boost.\n* Different CWT wavelet_width, 8 seems optimal, give a small boost. I found making this paramater trainable also gave a small boost.\n* Transfer Learning (i.e., freeze the EFN layer) followed by Fine Tuning (unfreeze). Marginal improvement\n* DIfferent EFN models, e.g., B0, B4, B7, with B7 having a slight boost\n\nAs I was running a lot of experiments I found that I needed a better way to record the results and started using Neptune.ai, which has been very useful.\n\nMy best result so far is B7 with a 512x512 scalogram ", "votes": 158 }, { "id": 1454700, "postDate": "2021-08-06T09:49:59.203Z", "content": "Good work.
\nYou're not just tweaking public notebooks. This is how to make a difference here. Keep pushing (but not too hard, I'd like to stay ahead of you β¦ )
", "rawMarkdown": "Good work.\n\nYou're not just tweaking public notebooks. This is how to make a difference here. Keep pushing (but not too hard, I'd like to stay ahead of you ... )", "votes": 21, "replies": [ { "id": 1458571, "postDate": "2021-08-08T00:28:43.560Z", "content": "and I'd like to catch up ;-)
", "rawMarkdown": "and I'd like to catch up ;-)", "votes": 11 } ] }, { "id": 1487386, "postDate": "2021-08-23T15:39:24.517Z", "content": "I tried to add your work into my pipeline. Hower the AUC is always around 0.5π¦β¦
", "rawMarkdown": "I tried to add your work into my pipeline. Hower the AUC is always around 0.5π¦...", "votes": 5, "replies": [ { "id": 1488625, "postDate": "2021-08-24T12:11:50.530Z", "content": "I set CQT trainable=True, AUC is 0.5.
", "rawMarkdown": "I set CQT trainable=True, AUC is 0.5.", "votes": 2 }, { "id": 1488703, "postDate": "2021-08-24T13:19:01.493Z", "content": "I also noticed that trainable CQT gives quite bad results.
", "rawMarkdown": "I also noticed that trainable CQT gives quite bad results.", "votes": 2 }, { "id": 1492122, "postDate": "2021-08-26T22:52:27.133Z", "content": "I experienced the same behavior. The loss doesn't decrease when the trainable=True
", "rawMarkdown": "I experienced the same behavior. The loss doesn't decrease when the trainable=True" }, { "id": 1516587, "postDate": "2021-09-18T13:38:01.623Z", "content": "Is this still the same or has anyone figure how to make trainable=True work for them? I understand if no one wants to share now, we will see after the end. π
Tried with trainable:
\nEpoch 1 - Score: 0.8622\nEpoch 1 - Save Best Score: 0.8622 Model\nEpoch 1 - Save Best Loss: 0.4206 Model\nEpoch 2 - Score: 0.8670\nEpoch 2 - Save Best Score: 0.8670 Model\nEpoch 2 - Save Best Loss: 0.4167 Model\nEpoch 3 - Score: 0.8690\nEpoch 3 - Save Best Score: 0.8690 Model\nEpoch 3 - Save Best Loss: 0.4150 Model\nLooks like its converging just fine(?), though it takes 2698s / epoch.
\nUPDATE
\nLet it ran 3 more epochs and it overfit at #4.
Regarding the rescaling. You always want your data to have zero mean and unity standard deviation. This allows deep neural networks to train well, see the famous paper on this: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf
\nHope I got youe question correclty.
", "rawMarkdown": "Regarding the rescaling. You always want your data to have zero mean and unity standard deviation. This allows deep neural networks to train well, see the famous paper on this: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf\n\nHope I got youe question correclty.", "votes": 3 }, { "id": 1466846, "postDate": "2021-08-11T16:47:49.447Z", "content": "Could you share the TF implementation of CWT you have been using?
\nI found this version and tried implementing it in PyTorch, however, the performance is significantly worse than Q-Transform.
\nMy PyTorch implementation is here
", "rawMarkdown": "Could you share the TF implementation of CWT you have been using?\n\nI found [this](https://github.com/nickgeoca/cwt-tensorflow) version and tried implementing it in PyTorch, however, the performance is significantly worse than Q-Transform.\n\nMy PyTorch implementation is [here](https://www.kaggle.com/anjum48/continuous-wavelet-transform-cwt-in-pytorch)", "votes": 3, "replies": [ { "id": 1467424, "postDate": "2021-08-12T01:24:26.667Z", "content": "I looked at your pytorch implementation and its great to see original work! You can find my TF implementation here, its based on an unoptimised TF 1.0 implementation
\n!pip install git+https://github.com//Kevin-McIsaac/cmorlet-tensorflow@Performance --no-deps > /dev/null\nThe repo is a mess as I've not merged all the changes yet. You can find more information on my work in the comments below.
\nWith regards to your CWT implementation the following might help:
Thank you so much for sharing! The 2D conv is a great suggestion, I'll give it a try and let you know how I get on :)
", "rawMarkdown": "Thank you so much for sharing! The 2D conv is a great suggestion, I'll give it a try and let you know how I get on :)", "votes": 3 }, { "id": 1468180, "postDate": "2021-08-12T09:40:57.087Z", "content": "Thank you both for sharing so much! So far I have just been using pywt which is quite slow..
", "rawMarkdown": "Thank you both for sharing so much! So far I have just been using pywt which is quite slow.." }, { "id": 1468219, "postDate": "2021-08-12T09:59:56.943Z", "content": "Seems sensible to me. I started out with PYCBC to get an understanding and quickly prototype an approach, then looked for faster tools.
", "rawMarkdown": "Seems sensible to me. I started out with PYCBC to get an understanding and quickly prototype an approach, then looked for faster tools.", "votes": 2 }, { "id": 1498875, "postDate": "2021-09-01T09:16:38.380Z", "content": "@kevinmcisaac i am getting this error upon using CWT and your TF ds this peculiarly happens when 7th epoch is about to complete. before that all epoch would run fine.
\nOutOfRangeError: 9 root error(s) found.\n (0) Out of range: {{function_node __inference_train_function_599323}} End of sequence\n [[{{node IteratorGetNext_6}}]]\n [[cluster_train_function/_execute_2_0/_91]]\n (1) Out of range: {{function_node __inference_train_function_599323}} End of sequence\n [[{{node IteratorGetNext_6}}]]\n (2) Cancelled: {{function_node __inference_train_function_599323}} Function was cancelled before it was started\n (3) Cancelled: {{function_node __inference_train_function_599323}} Function was cancelled before it was started\n (4) Cancelled: {{function_node __inference_train_function_599323}} Function was cancelled before it was started\n (5) Cancelled: {{function_node __inference_train_function_599323}} Function was cancelled before it was started\n (6) Cancelled: {{function_node __inference_train_function_599323}} Function was cancelled before it was started\n (7) Cancelled: {{function_node __inference_train_function_599323}} Function was cancelled before it was started\n (8) Cancelled: {{function_node __inference_train_function_599323}} Function was cancelled before it was started\n0 successful operations.\nany help appreciatedβ¦
\nThanks in advance. I am not so good in Keras.. its this competition m trying to learn .
", "rawMarkdown": "@kevinmcisaac i am getting this error upon using CWT and your TF ds this peculiarly happens when 7th epoch is about to complete. before that all epoch would run fine. \n\n```\nOutOfRangeError: 9 root error(s) found.\n (0) Out of range: {{function_node __inference_train_function_599323}} End of sequence\n\t [[{{node IteratorGetNext_6}}]]\n\t [[cluster_train_function/_execute_2_0/_91]]\n (1) Out of range: {{function_node __inference_train_function_599323}} End of sequence\n\t [[{{node IteratorGetNext_6}}]]\n (2) Cancelled: {{function_node __inference_train_function_599323}} Function was cancelled before it was started\n (3) Cancelled: {{function_node __inference_train_function_599323}} Function was cancelled before it was started\n (4) Cancelled: {{function_node __inference_train_function_599323}} Function was cancelled before it was started\n (5) Cancelled: {{function_node __inference_train_function_599323}} Function was cancelled before it was started\n (6) Cancelled: {{function_node __inference_train_function_599323}} Function was cancelled before it was started\n (7) Cancelled: {{function_node __inference_train_function_599323}} Function was cancelled before it was started\n (8) Cancelled: {{function_node __inference_train_function_599323}} Function was cancelled before it was started\n0 successful operations.\n```\nany help appreciated...\n\nThanks in advance. I am not so good in Keras.. its this competition m trying to learn ." }, { "id": 1498904, "postDate": "2021-09-01T09:53:27.870Z", "content": "I've not see this before. Can you post your code for \"fit\"
\nThe only thing that I've runing into that was similar is the need to use drop_remainder with the batches so that every batch is a full batch,
\ndataset = dataset.batch(batch_size, drop_remainder=drop_remainder)
i hope no issues with dataset path we use.I get two gcs paths that seem to include all files
\nwith strategy.scope():\n model = build_model(\n size=IMAGE_SIZE, \n efficientnet_size=EFFICIENTNET_SIZE,\n weights=WEIGHTS, \n count=train_image_count // BATCH_SIZE // REPLICAS // 4)\n\n model_ckpt = tf.keras.callbacks.ModelCheckpoint(\n str(SAVEDIR / f\"fold{fold}.h5\"), monitor=\"val_auc\", verbose=1, save_best_only=True,\n save_weights_only=True, mode=\"max\", save_freq=\"epoch\"\n )\nhistory = model.fit(\n get_dataset(files_train, labeled=True, batch_size=128, \n buffer_size=tf.data.experimental.AUTOTUNE, drop_remainder=True),\n# get_dataset(files_train, batch_size=BATCH_SIZE, shuffle=True, repeat=True, aug=True),\n epochs=EPOCHS,\n callbacks=[model_ckpt, get_lr_callback(BATCH_SIZE, REPLICAS)],\n steps_per_epoch=train_image_count // BATCH_SIZE // REPLICAS // 4,\n# validation_data=get_dataset(files_valid, batch_size=BATCH_SIZE * 4, repeat=True, shuffle=False, aug=True, val=True),\n validation_data=get_dataset(files_valid, labeled=True, batch_size=BATCH_SIZE * 4, shuffle=False, drop_remainder=False),\n verbose=1\n )\nBelow is error log
\n37 # validation_data=get_dataset(files_valid, batch_size=BATCH_SIZE * 4, repeat=True, shuffle=False, aug=True, val=True),\n 38 validation_data=get_dataset(files_valid, labeled=True, batch_size=BATCH_SIZE * 4, shuffle=False, drop_remainder=False),\n---> 39 verbose=1\n 40 )\n 41 \n/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)\n1143 epoch_logs.update(val_logs)\n1144 \n-> 1145 callbacks.on_epoch_end(epoch, epoch_logs)\n1146 training_logs = epoch_logs\n1147 if self.stop_training:\n/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py in on_epoch_end(self, epoch, logs)\n 426 for callback in self.callbacks:\n 427 if getattr(callback, '_supports_tf_logs', False):\n--> 428 callback.on_epoch_end(epoch, logs)\n 429 else:\n 430 if numpy_logs is None: # Only convert once.\n/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py in on_epoch_end(self, epoch, logs)\n1029 \n1030 def on_epoch_end(self, epoch, logs=None):\n-> 1031 self._finalize_progbar(logs, self._train_step)\n1032 \n1033 def on_test_end(self, logs=None):\n/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py in _finalize_progbar(self, logs, counter)\n1086 \n1087 def _finalize_progbar(self, logs, counter):\n-> 1088 logs = tf_utils.to_numpy_or_python_type(logs or {})\n1089 if self.target is None:\n1090 if counter is not None:\n/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/utils/tf_utils.py in to_numpy_or_python_type(tensors)\n 512 return t # Don't turn ragged or sparse tensors to NumPy.\n 513 \n--> 514 return nest.map_structure(_to_single_numpy_or_python_type, tensors)\n 515 \n 516 \n/opt/conda/lib/python3.7/site-packages/tensorflow/python/util/nest.py in map_structure(func, *structure, **kwargs)\n 657 \n 658 return pack_sequence_as(\n--> 659 structure[0], [func(*x) for x in entries],\n 660 expand_composites=expand_composites)\n 661 \n/opt/conda/lib/python3.7/site-packages/tensorflow/python/util/nest.py in <listcomp>(.0)\n 657 \n 658 return pack_sequence_as(\n--> 659 structure[0], [func(*x) for x in entries],\n 660 expand_composites=expand_composites)\n 661 \n/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/utils/tf_utils.py in _to_single_numpy_or_python_type(t)\n 508 def _to_single_numpy_or_python_type(t):\n 509 if isinstance(t, ops.Tensor):\n--> 510 x = t.numpy()\n 511 return x.item() if np.ndim(x) == 0 else x\n 512 return t # Don't turn ragged or sparse tensors to NumPy.\n/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/ops.py in numpy(self)\n1069 \"\"\"\n1070 # TODO(slebedev): Consider avoiding a copy for non-CPU or remote tensors.\n-> 1071 maybe_arr = self._numpy() # pylint: disable=protected-access\n1072 return maybe_arr.copy() if isinstance(maybe_arr, np.ndarray) else maybe_arr\n1073 \n/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/ops.py in _numpy(self)\n1037 return self._numpy_internal()\n1038 except core._NotOkStatusException as e: # pylint: disable=protected-access\n-> 1039 six.raise_from(core._status_to_exception(e.code, e.message), None) # pylint: disable=protected-access\n1040 \n1041 @property\ntOfRangeError: 9 root error(s) found.\n(0) Out of range: {{function_node __inference_train_function_1081877}} End of sequence\n[[{{node cond_11/else/_106/cond_11/IteratorGetNext}}]]\n[[Shape_19/_120]]\n(1) Out of range: {{function_node __inference_train_function_1081877}} End of sequence\n[[{{node cond_11/else/_106/cond_11/IteratorGetNext}}]]\n(2) Out of range: {{function_node __inference_train_function_1081877}} End of sequence\n[[{{node cond_11/else/_106/cond_11/IteratorGetNext}}]]\n[[GroupCrossDeviceControlEdges_0/Identity_7/_325]]\n(3) Out of range: {{function_node __inference_train_function_1081877}} End of sequence\n[[{{node cond_11/else/_106/cond_11/IteratorGetNext}}]]\n[[strided_slice_27/_232]]\n(4) Out of range: {{function_node __inference_train_function_1081877}} End of sequence\n[[{{node cond_11/else/_106/cond_11/IteratorGetNext}}]]\n[[tpu_compile_succeeded_assert/_10601166017813830618/_5/_257]]\n(5) Out of range: {{function_node __inference_train_function_1081877}} End of sequence\n[[{{node cond_11/else/_106/cond_11/IteratorGetNext}}]]\n[[Pad_5/paddings/_148]]\n(6) Out of range: {{function_node __inference_train_function_1081877}} End of sequence\n[[{{node cond_11/else/_106/cond_11/IteratorGetNext}}]]\n@kevinmcisaac when i do image counts for valid and train
\ni get
\n1)1680000 560000
print(train_image_count,valid_image_count)
\nhow can we use folds as per your tfrecords set. i get 80 files set..
\n2 ) How you performed it there are multiple ways mentioned
\nrescales the waveform to [-1, 1]
@kevinmcisaac just a slight reminder. I feel your datasets folds are not built properly if u see above. any suggestion ?
", "rawMarkdown": "@kevinmcisaac just a slight reminder. I feel your datasets folds are not built properly if u see above. any suggestion ?" } ] }, { "id": 1505096, "postDate": "2021-09-07T00:34:50.220Z", "content": "\"Rescaling was essential.\"
\nthe reason could be the use of stride=2 in the cnn network (especially the first one)
by using enlarged image, you could have a sub-pixel convolution?
", "rawMarkdown": "\"Rescaling was essential.\"\nthe reason could be the use of stride=2 in the cnn network (especially the first one)\n\nby using enlarged image, you could have a sub-pixel convolution?", "votes": 2 }, { "id": 1489163, "postDate": "2021-08-24T18:45:28.073Z", "content": "\n\nRescaling was essential. I'd be interested to hear from an Kaggel Expert if they found this and if so why?
\n
I guess it's due to original data has very small value(around 1e-20 range), that's why many people divide tha data with max() first to scale to [some value, 1].
", "rawMarkdown": "> Rescaling was essential. I'd be interested to hear from an Kaggel Expert if they found this and if so why?\n\nI guess it's due to original data has very small value(around 1e-20 range), that's why many people divide tha data with max() first to scale to [some value, 1]. ", "votes": 1 }, { "id": 1450145, "postDate": "2021-08-05T03:51:25.433Z", "content": "Any plans on sharing your keras layer of CWT ?
", "rawMarkdown": "Any plans on sharing your keras layer of CWT ?", "replies": [ { "id": 1458576, "postDate": "2021-08-08T00:31:18.317Z", "content": "I've not had time to clean up the repo, but you can use this
\n!pip install git+https://github.com//Kevin-McIsaac/cmorlet-tensorflow@Performance --no-deps > /dev/null
and there is basic documentation.
", "rawMarkdown": "I've not had time to clean up the repo, but you can use this\n!pip install git+https://github.com//Kevin-McIsaac/cmorlet-tensorflow@Performance --no-deps > /dev/null\n\nand there is basic documentation.", "votes": 5 }, { "id": 1459052, "postDate": "2021-08-08T07:37:53.680Z", "content": "So we have to just feed the npy files to the models by adding this layer ?
", "rawMarkdown": "So we have to just feed the npy files to the models by adding this layer ?" }, { "id": 1460514, "postDate": "2021-08-08T23:30:37.480Z", "content": "Yes (see earlier comments) though to get the TPU performance you must use TFRecords (see early comments for links to these datasets)
", "rawMarkdown": "Yes (see earlier comments) though to get the TPU performance you must use TFRecords (see early comments for links to these datasets)", "votes": 3 }, { "id": 1470768, "postDate": "2021-08-13T17:48:44.130Z", "content": "You can find a Keras CWT layer in this notebook. No bandpass filtering is required with this one, as it only computes the scaleogram for a frequency range of interest.
", "rawMarkdown": "You can find a Keras CWT layer in [this notebook](https://www.kaggle.com/mistag/wavelet1d-custom-keras-wavelet-transform-layer?scriptVersionId=71528111). No bandpass filtering is required with this one, as it only computes the scaleogram for a frequency range of interest." }, { "id": 1470965, "postDate": "2021-08-13T20:23:30.790Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ] }, { "id": 1559722, "postDate": "2021-10-27T07:09:24.783Z", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris
@kevinmcisaac nice work! may I ask what does the parameter wavelet_width: default = 1 in CWT ? what range of values should we consider ?
\nI saw the demo in your git repo where it uses 1, but not sure if that is the case for our data here
Thanks for sharing @kevinmcisaac , may I ask what do you mean by \"rescales the waveform to [-1, 1]\"? It's like standardize as -mean and /std, or min/max to [0, 1] then scale and shift to [-1, 1]? Thanks.
", "rawMarkdown": "Thanks for sharing @kevinmcisaac , may I ask what do you mean by \"rescales the waveform to [-1, 1]\"? It's like standardize as -mean and /std, or min/max to [0, 1] then scale and shift to [-1, 1]? Thanks.", "replies": [ { "id": 1499842, "postDate": "2021-09-02T01:22:00.070Z", "content": "Its Min/Max
", "rawMarkdown": "Its Min/Max", "votes": 1 } ] }, { "id": 1495884, "postDate": "2021-08-29T21:25:53.740Z", "content": "@kevinmcisaac Can you guide me what is bandpass filter and CWT
", "rawMarkdown": " @kevinmcisaac Can you guide me what is bandpass filter and CWT", "replies": [ { "id": 1495948, "postDate": "2021-08-30T00:15:50.607Z", "content": "Your best option is to do some reading on the internet.
\nhttps://en.wikipedia.org/wiki/Band-pass_filter
\nhttps://www.kaggle.com/asauve/a-gentle-introduction-to-wavelet-for-data-analysis
In short a filter removes some frequencies from a wave and a band pass only allows frequencies in the band to pass.
\nA CWT is an alternative approach to Short Term Fourier Transform or Constant Quality Transform to converting a wave from time/amplitude to time/Frequency
", "rawMarkdown": "Your best option is to do some reading on the internet.\nhttps://en.wikipedia.org/wiki/Band-pass_filter\nhttps://www.kaggle.com/asauve/a-gentle-introduction-to-wavelet-for-data-analysis\n\nIn short a filter removes some frequencies from a wave and a band pass only allows frequencies in the band to pass.\n\nA CWT is an alternative approach to Short Term Fourier Transform or Constant Quality Transform to converting a wave from time/amplitude to time/Frequency", "votes": 1 } ] }, { "id": 1490958, "postDate": "2021-08-26T03:31:55.107Z", "content": "I have learned a lot from your summary!
", "rawMarkdown": "I have learned a lot from your summary!" }, { "id": 1485990, "postDate": "2021-08-22T14:59:01.930Z", "content": "Hello, how is your single model's lb and cv?
", "rawMarkdown": "Hello, how is your single model's lb and cv?" }, { "id": 1480712, "postDate": "2021-08-19T06:31:35.727Z", "content": "Good post. Thanks for sharing your inputs. Well written indeed! @kevinmcisaac π
", "rawMarkdown": "Good post. Thanks for sharing your inputs. Well written indeed! @kevinmcisaac π" }, { "id": 1478898, "postDate": "2021-08-18T07:26:34.500Z", "content": "@kevinmcisaac does the order Bandpass filter makes any difference, Like apply CQT then BandPass
", "rawMarkdown": "@kevinmcisaac does the order Bandpass filter makes any difference, Like apply CQT then BandPass " }, { "id": 1478581, "postDate": "2021-08-18T04:16:57.947Z", "content": "I thought b7 on 512x512 would cause memory issue on many GPUs. Mind sharing GPUs you are using?
", "rawMarkdown": "I thought b7 on 512x512 would cause memory issue on many GPUs. Mind sharing GPUs you are using?", "replies": [ { "id": 1478728, "postDate": "2021-08-18T05:51:12.157Z", "content": "I've been running on a TPU and find I get better performance, say 10-20x
", "rawMarkdown": "I've been running on a TPU and find I get better performance, say 10-20x", "votes": 1 } ] }, { "id": 1464254, "postDate": "2021-08-10T14:01:40.647Z", "content": "Than you for sharing
", "rawMarkdown": "Than you for sharing " }, { "id": 1461695, "postDate": "2021-08-09T14:19:55.917Z", "content": "Good work. @kevinmcisaac
", "rawMarkdown": "Good work. @kevinmcisaac " }, { "id": 1459244, "postDate": "2021-08-08T09:37:33.613Z", "content": "@kevinmcisaac Hey Thanks for the detailed approach overview.
\nI just had a doubt. At what step are you resizing the scalograms?
.npy files -> CQT (keras layer) -> Resizing (?) -> EFN Model (keras model)
Won't this be a bottleneck here?
", "rawMarkdown": "@kevinmcisaac Hey Thanks for the detailed approach overview.\nI just had a doubt. At what step are you resizing the scalograms?\n\n``` .npy files -> CQT (keras layer) -> Resizing (?) -> EFN Model (keras model) ```\n\nWon't this be a bottleneck here?", "replies": [ { "id": 1460496, "postDate": "2021-08-08T23:09:08.840Z", "content": "@yerramvarun I don't resize the CWT for the EFN. My model looks like
\n\n model = tf.keras.Sequential([L.InputLayer(input_shape=(3, 4096),\n ComplexMorletCWT(n_scales = 64, stride=64, \n output='magnitude', data_format='channels_first'), \n L.Permute((2, 3, 1)), # transpose to channel last [:, time, n_scales, channels]\n efn.EfficientnetB4(include_top=False,weights='imagenet'),\n L.GlobalAveragePooling2D(),\n L.Dense(32,activation='relu'),\n L.Dense(1, activation='sigmoid')\n ])\n
\nI guess you are asking at the Keras documentation for EFN states each model uses a specific size. I found this confusing and discovered they worked Ok with other sizes. I'd be interested to hear from someone with more experience about this?
Note: I'm using Continuious Wavelett Transfroms (CWT) not CQT using the keras layer in this
\n!pip install git+https://github.com//Kevin-McIsaac/cmorlet-tensorflow@Performance --no-deps > /dev/null
That makes sense. Thanks for clearing that up for me
", "rawMarkdown": "That makes sense. Thanks for clearing that up for me", "votes": 2 }, { "id": 1494194, "postDate": "2021-08-28T13:36:27.907Z", "content": "where do you resize your image to different square sizes @kevinmcisaac
", "rawMarkdown": "where do you resize your image to different square sizes @kevinmcisaac " }, { "id": 1495919, "postDate": "2021-08-29T22:40:10.310Z", "content": "I use the n_scales and strides parameters to determine the size of the image, e.g., n_scales = 128, stride=32 creates a spectrogram that is 128x128. While efficient net was trained on specific image sizes, it can accept any size.
", "rawMarkdown": "I use the n_scales and strides parameters to determine the size of the image, e.g., n_scales = 128, stride=32 creates a spectrogram that is 128x128. While efficient net was trained on specific image sizes, it can accept any size." }, { "id": 1495924, "postDate": "2021-08-29T22:55:35.327Z", "content": "Let's say you want image of WIDTH x HEIGHT size. Then you should set n_scales=WIDTH, and strides = int(np.ceil(4096/HEIGHT)) (assuming your input has (3, 4096) shape). This works for me.
\n\nI started with nnAudio Constant Quality Transform (CQT) and found this a lot faster than librosa. The entire processing, i.e., load the data, bandpass filter, creating the a 64x64 scalogram in nnAUdio and classifying in EFN B4 took about 8hrs on a CPU and 2hrs on a GPU.
\n
can you tell me how many epochs you did in 2 hours you mentioned ?
", "rawMarkdown": "> I started with nnAudio Constant Quality Transform (CQT) and found this a lot faster than librosa. The entire processing, i.e., load the data, bandpass filter, creating the a 64x64 scalogram in nnAUdio and classifying in EFN B4 took about 8hrs on a CPU and 2hrs on a GPU. \n\ncan you tell me how many epochs you did in 2 hours you mentioned ?", "replies": [ { "id": 1458555, "postDate": "2021-08-08T00:08:40.940Z", "content": "My notes are not the best for the old nnAudio runs. For recent runs with a CWT that create an 64x64 image and B4 I get the following
\nTPU 32s per epoch, ran 30 epochs.
\nGPU 260s per epoch, 29 epochs
The total notebook time, including set and inferencing was 28 min and 2hrs 14 respectively.
", "rawMarkdown": "My notes are not the best for the old nnAudio runs. For recent runs with a CWT that create an 64x64 image and B4 I get the following\nTPU 32s per epoch, ran 30 epochs. \nGPU 260s per epoch, 29 epochs \n\nThe total notebook time, including set and inferencing was 28 min and 2hrs 14 respectively.\n\n", "votes": 2 }, { "id": 1458814, "postDate": "2021-08-08T04:51:42.973Z", "content": "Thanks! Can you tell me how you optimized the preprocessing / process of conversion to TFrecords ? When I was trying to do some custom preprocessing and trying to convert them to TFrecords, each notebook had only 5GB output size. So, it will take me 16 runs, each around 4GB. This seems to be a bottleneck.
", "rawMarkdown": "Thanks! Can you tell me how you optimized the preprocessing / process of conversion to TFrecords ? When I was trying to do some custom preprocessing and trying to convert them to TFrecords, each notebook had only 5GB output size. So, it will take me 16 runs, each around 4GB. This seems to be a bottleneck. " }, { "id": 1460513, "postDate": "2021-08-08T23:28:43.087Z", "content": "I found I could do this as three batches; 2 for train and 1 for submissions.
\nThese are public datasets named like this G2NET TFRecords with 20-500Hz bandpass Part I
\nYou can then look at the notebook that generated this which use this utility script
", "rawMarkdown": "I found I could do this as three batches; 2 for train and 1 for submissions. \n\nThese are public datasets named like this [G2NET TFRecords with 20-500Hz bandpass Part I](https://www.kaggle.com/kevinmcisaac/g2net-tfrecords-with-20500hz-bandpass)\n\nYou can then look at the notebook that generated this which use this [utility script](https://www.kaggle.com/kevinmcisaac/g2nettfrec)", "votes": 2 }, { "id": 1467960, "postDate": "2021-08-12T07:41:29.597Z", "content": "@kevinmcisaac can you tell me how many training samples are in each tfrecord of your dataset ? I can't seem to find any number from your utility script
", "rawMarkdown": "@kevinmcisaac can you tell me how many training samples are in each tfrecord of your dataset ? I can't seem to find any number from your utility script" }, { "id": 1468089, "postDate": "2021-08-12T09:00:06.677Z", "content": "Rather than pick a number of samples per TFR The approach I used was to create a specific number of TFrecord files so that each was large enough to get good read performance and there were enough file to enable overlapping parallel reads.
\nYou can see in the notebook I created 70 files so each trf file has 560000/70 = 8,000 samples
", "rawMarkdown": "Rather than pick a number of samples per TFR The approach I used was to create a specific number of TFrecord files so that each was large enough to get good read performance and there were enough file to enable overlapping parallel reads. \n\nYou can see in the notebook I created 70 files so each trf file has 560000/70 = 8,000 samples", "votes": 2 }, { "id": 1468528, "postDate": "2021-08-12T13:06:55.060Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1469484, "postDate": "2021-08-13T01:02:39.323Z", "content": "I converted the inputs to np.float32 as part of the preprocessing. Having read this I now understand why you are asking a however not familiar enough with TF 2.0 to be able to answer the question of using mixed precision.
\nIf you have the skills you could fork the repo and make the necessary changes. I'd be keen to hear how that goes.
", "rawMarkdown": "I converted the inputs to np.float32 as part of the preprocessing. Having read [this](https://cloud.google.com/blog/products/ai-machine-learning/bfloat16-the-secret-to-high-performance-on-cloud-tpus) I now understand why you are asking a however not familiar enough with TF 2.0 to be able to answer the question of using mixed precision.\n\nIf you have the skills you could fork the repo and make the necessary changes. I'd be keen to hear how that goes.", "votes": 1 }, { "id": 1471865, "postDate": "2021-08-14T14:16:33.637Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ] }, { "id": 1452494, "postDate": "2021-08-05T16:11:29.170Z", "content": "Can you tell which package you used for making a bandpass filter?
", "rawMarkdown": "Can you tell which package you used for making a bandpass filter?", "replies": [ { "id": 1452559, "postDate": "2021-08-05T16:28:27.047Z", "rawMarkdown": "", "votes": 3, "isDeleted": true }, { "id": 1458564, "postDate": "2021-08-08T00:16:35.877Z", "content": "I used scipy.signal. Here is the code I used.
\nfrom scipy import signal\n\nbHP, aHP = signal.butter(8, (20, 500), btype='bandpass', fs= 2048)\ndef filterSig(waves, a=aHP, b=bHP):\n '''Apply a 20Hz high pass filter to the three events'''\n return np.array([signal.filtfilt(b, a, wave) for wave in waves]) #lfilter introduces a larger spike around 20hz\nInitially issued gwpy but its a lot of baggage just to do a simple filter. I use this in a pre-processing step that re-writes the data s TFrecords to speed up data loading and keep the TPUs busy.
\nWhat I'd like to do is make a bandpass filter in Keras and include that in my NN
", "rawMarkdown": "I used scipy.signal. Here is the code I used.\n```\nfrom scipy import signal\n\nbHP, aHP = signal.butter(8, (20, 500), btype='bandpass', fs= 2048)\ndef filterSig(waves, a=aHP, b=bHP):\n '''Apply a 20Hz high pass filter to the three events'''\n return np.array([signal.filtfilt(b, a, wave) for wave in waves]) #lfilter introduces a larger spike around 20hz\n```\n\nInitially issued gwpy but its a lot of baggage just to do a simple filter. I use this in a pre-processing step that re-writes the data s TFrecords to speed up data loading and keep the TPUs busy.\n\nWhat I'd like to do is make a bandpass filter in Keras and include that in my NN", "votes": 10 }, { "id": 1459232, "postDate": "2021-08-08T09:30:52.680Z", "content": "I think for a 20-500Hz cuts you need to use something like:
\nnyq = 0.5 * fs # Nyquist frequency\nlow = 20 / nyq\nhigh = 500 / nyq\nb, a = butter(order, [low, high], btype='bandpass')\nFrom: https://scipy-cookbook.readthedocs.io/items/ButterworthBandpass.html
\nEdit: I just found out that if you pass the fs arg, you don't need to do this :)
@kevinmcisaac should this be applied before or after normalization ?
", "rawMarkdown": "@kevinmcisaac should this be applied before or after normalization ?", "votes": 1 }, { "id": 1463981, "postDate": "2021-08-10T12:07:08.570Z", "content": "I applied the scaling after the bandpass filter but I don't think the order matter.
", "rawMarkdown": "I applied the scaling after the bandpass filter but I don't think the order matter. ", "votes": 1 }, { "id": 1464039, "postDate": "2021-08-10T12:29:37.577Z", "content": "Just tried both after and before scaling. They both look the same
", "rawMarkdown": "Just tried both after and before scaling. They both look the same", "votes": 2 }, { "id": 1464081, "postDate": "2021-08-10T12:47:05.490Z", "content": "Good to know
", "rawMarkdown": "Good to know", "votes": 3 }, { "id": 1464124, "postDate": "2021-08-10T13:06:32.067Z", "rawMarkdown": "", "votes": 2, "isDeleted": true }, { "id": 1465337, "postDate": "2021-08-11T02:33:43.033Z", "content": "Yes I did. The results are reported here.
\nTL;DR i make the AUC 10% worse.
Thanks to all who replied. This info is quite useful.
", "rawMarkdown": "Thanks to all who replied. This info is quite useful.", "votes": 1 } ] }, { "id": 1497459, "postDate": "2021-08-31T08:03:53.073Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 1497469, "postDate": "2021-08-31T08:12:45.913Z", "content": "No, I change the size of n_scales and or stride to create the desired sized scalogram, e.g,
\n\n\nComplexMorletCWT(n_scales = 64, stride=64)
\n
creates a 64 x 64 image.
\nI simply divided by a fixed scale factor, i.e., the same number for each channel (detector) and all samples.
\nI started out by dividing each channel by the max of the absolute value of that channel, i.e., standardize into the range [-1,1] but then I thought this might lose some important information so I switch to a common number across all. Since there is a large dynamic range across the various samples, and across teh detectors, I just picked something that seemed reasonable for many samples (i.e., more art than science).
", "rawMarkdown": "I simply divided by a fixed scale factor, i.e., the same number for each channel (detector) and all samples. \n\nI started out by dividing each channel by the max of the absolute value of that channel, i.e., standardize into the range [-1,1] but then I thought this might lose some important information so I switch to a common number across all. Since there is a large dynamic range across the various samples, and across teh detectors, I just picked something that seemed reasonable for many samples (i.e., more art than science).", "votes": 4 } ] }, { "id": 1451854, "postDate": "2021-08-05T12:59:10.630Z", "rawMarkdown": "", "votes": 2, "isDeleted": true, "replies": [ { "id": 1452016, "postDate": "2021-08-05T13:55:38.030Z", "rawMarkdown": "", "votes": 2, "isDeleted": true }, { "id": 1458574, "postDate": "2021-08-08T00:30:34.733Z", "content": "I've not tried ENetV2 yet. How much improvement did you get, i.e., %?
", "rawMarkdown": "I've not tried ENetV2 yet. How much improvement did you get, i.e., %? ", "votes": 1 }, { "id": 1459249, "postDate": "2021-08-08T09:40:25.407Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 1478994, "postDate": "2021-08-18T08:33:41.167Z", "content": "How was efficientnetv2 XL comparable to efficientnet B7 ?
", "rawMarkdown": "How was efficientnetv2 XL comparable to efficientnet B7 ?", "votes": 1 } ] }, { "id": 1473217, "postDate": "2021-08-15T12:40:49.160Z", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!" }, { "id": 1464152, "postDate": "2021-08-10T13:15:57.927Z", "content": "Thanks for sharing ! ππ
", "rawMarkdown": "Thanks for sharing ! ππ" }, { "id": 1455557, "postDate": "2021-08-06T15:51:22.947Z", "content": "Thanks for sharing your insights
", "rawMarkdown": "Thanks for sharing your insights " }, { "id": 1454015, "postDate": "2021-08-06T04:32:03.510Z", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!\n" } ], "comments": [ { "id": 1454700, "author_name": "CPMP", "author_url": "", "post_date": "2021-08-06T09:49:59.203000", "content": "Good work.
\nYou're not just tweaking public notebooks. This is how to make a difference here. Keep pushing (but not too hard, I'd like to stay ahead of you β¦ )
", "votes": 21, "replies": [ { "id": 1458571, "author_name": "Fractal Feelings", "author_url": "", "post_date": "2021-08-08T00:28:43.560000", "content": "and I'd like to catch up ;-)
", "votes": 11, "replies": [] } ] }, { "id": 1487386, "author_name": "toxu", "author_url": "", "post_date": "2021-08-23T15:39:24.517000", "content": "I tried to add your work into my pipeline. Hower the AUC is always around 0.5π¦β¦
", "votes": 5, "replies": [ { "id": 1488625, "author_name": "Correlation", "author_url": "", "post_date": "2021-08-24T12:11:50.530000", "content": "I set CQT trainable=True, AUC is 0.5.
", "votes": 2, "replies": [] }, { "id": 1488703, "author_name": "Araik Tamazian", "author_url": "", "post_date": "2021-08-24T13:19:01.493000", "content": "I also noticed that trainable CQT gives quite bad results.
", "votes": 2, "replies": [] }, { "id": 1492122, "author_name": "Sinan Calisir", "author_url": "", "post_date": "2021-08-26T22:52:27.133000", "content": "I experienced the same behavior. The loss doesn't decrease when the trainable=True
", "votes": 0, "replies": [] }, { "id": 1516587, "author_name": "Yassine Alouini", "author_url": "", "post_date": "2021-09-18T13:38:01.623000", "content": "Is this still the same or has anyone figure how to make trainable=True work for them? I understand if no one wants to share now, we will see after the end. π
Tried with trainable:
\nEpoch 1 - Score: 0.8622\nEpoch 1 - Save Best Score: 0.8622 Model\nEpoch 1 - Save Best Loss: 0.4206 Model\nEpoch 2 - Score: 0.8670\nEpoch 2 - Save Best Score: 0.8670 Model\nEpoch 2 - Save Best Loss: 0.4167 Model\nEpoch 3 - Score: 0.8690\nEpoch 3 - Save Best Score: 0.8690 Model\nEpoch 3 - Save Best Loss: 0.4150 Model\nLooks like its converging just fine(?), though it takes 2698s / epoch.
\nUPDATE
\nLet it ran 3 more epochs and it overfit at #4.
Regarding the rescaling. You always want your data to have zero mean and unity standard deviation. This allows deep neural networks to train well, see the famous paper on this: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf
\nHope I got youe question correclty.
", "votes": 3, "replies": [] }, { "id": 1466846, "author_name": "datasaurus", "author_url": "", "post_date": "2021-08-11T16:47:49.447000", "content": "Could you share the TF implementation of CWT you have been using?
\nI found this version and tried implementing it in PyTorch, however, the performance is significantly worse than Q-Transform.
\nMy PyTorch implementation is here
", "votes": 3, "replies": [ { "id": 1467424, "author_name": "Fractal Feelings", "author_url": "", "post_date": "2021-08-12T01:24:26.667000", "content": "I looked at your pytorch implementation and its great to see original work! You can find my TF implementation here, its based on an unoptimised TF 1.0 implementation
\n!pip install git+https://github.com//Kevin-McIsaac/cmorlet-tensorflow@Performance --no-deps > /dev/null\nThe repo is a mess as I've not merged all the changes yet. You can find more information on my work in the comments below.
\nWith regards to your CWT implementation the following might help:
Thank you so much for sharing! The 2D conv is a great suggestion, I'll give it a try and let you know how I get on :)
", "votes": 3, "replies": [] }, { "id": 1468180, "author_name": "Hannes Γhler", "author_url": "", "post_date": "2021-08-12T09:40:57.087000", "content": "Thank you both for sharing so much! So far I have just been using pywt which is quite slow..
", "votes": 0, "replies": [] }, { "id": 1468219, "author_name": "Fractal Feelings", "author_url": "", "post_date": "2021-08-12T09:59:56.943000", "content": "Seems sensible to me. I started out with PYCBC to get an understanding and quickly prototype an approach, then looked for faster tools.
", "votes": 2, "replies": [] }, { "id": 1498875, "author_name": "Jaideep", "author_url": "", "post_date": "2021-09-01T09:16:38.380000", "content": "@kevinmcisaac i am getting this error upon using CWT and your TF ds this peculiarly happens when 7th epoch is about to complete. before that all epoch would run fine.
\nOutOfRangeError: 9 root error(s) found.\n (0) Out of range: {{function_node __inference_train_function_599323}} End of sequence\n [[{{node IteratorGetNext_6}}]]\n [[cluster_train_function/_execute_2_0/_91]]\n (1) Out of range: {{function_node __inference_train_function_599323}} End of sequence\n [[{{node IteratorGetNext_6}}]]\n (2) Cancelled: {{function_node __inference_train_function_599323}} Function was cancelled before it was started\n (3) Cancelled: {{function_node __inference_train_function_599323}} Function was cancelled before it was started\n (4) Cancelled: {{function_node __inference_train_function_599323}} Function was cancelled before it was started\n (5) Cancelled: {{function_node __inference_train_function_599323}} Function was cancelled before it was started\n (6) Cancelled: {{function_node __inference_train_function_599323}} Function was cancelled before it was started\n (7) Cancelled: {{function_node __inference_train_function_599323}} Function was cancelled before it was started\n (8) Cancelled: {{function_node __inference_train_function_599323}} Function was cancelled before it was started\n0 successful operations.\nany help appreciatedβ¦
\nThanks in advance. I am not so good in Keras.. its this competition m trying to learn .
", "votes": 0, "replies": [] }, { "id": 1498904, "author_name": "Fractal Feelings", "author_url": "", "post_date": "2021-09-01T09:53:27.870000", "content": "I've not see this before. Can you post your code for \"fit\"
\nThe only thing that I've runing into that was similar is the need to use drop_remainder with the batches so that every batch is a full batch,
\ndataset = dataset.batch(batch_size, drop_remainder=drop_remainder)
i hope no issues with dataset path we use.I get two gcs paths that seem to include all files
\nwith strategy.scope():\n model = build_model(\n size=IMAGE_SIZE, \n efficientnet_size=EFFICIENTNET_SIZE,\n weights=WEIGHTS, \n count=train_image_count // BATCH_SIZE // REPLICAS // 4)\n\n model_ckpt = tf.keras.callbacks.ModelCheckpoint(\n str(SAVEDIR / f\"fold{fold}.h5\"), monitor=\"val_auc\", verbose=1, save_best_only=True,\n save_weights_only=True, mode=\"max\", save_freq=\"epoch\"\n )\nhistory = model.fit(\n get_dataset(files_train, labeled=True, batch_size=128, \n buffer_size=tf.data.experimental.AUTOTUNE, drop_remainder=True),\n# get_dataset(files_train, batch_size=BATCH_SIZE, shuffle=True, repeat=True, aug=True),\n epochs=EPOCHS,\n callbacks=[model_ckpt, get_lr_callback(BATCH_SIZE, REPLICAS)],\n steps_per_epoch=train_image_count // BATCH_SIZE // REPLICAS // 4,\n# validation_data=get_dataset(files_valid, batch_size=BATCH_SIZE * 4, repeat=True, shuffle=False, aug=True, val=True),\n validation_data=get_dataset(files_valid, labeled=True, batch_size=BATCH_SIZE * 4, shuffle=False, drop_remainder=False),\n verbose=1\n )\nBelow is error log
\n37 # validation_data=get_dataset(files_valid, batch_size=BATCH_SIZE * 4, repeat=True, shuffle=False, aug=True, val=True),\n 38 validation_data=get_dataset(files_valid, labeled=True, batch_size=BATCH_SIZE * 4, shuffle=False, drop_remainder=False),\n---> 39 verbose=1\n 40 )\n 41 \n/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_batch_size, validation_freq, max_queue_size, workers, use_multiprocessing)\n1143 epoch_logs.update(val_logs)\n1144 \n-> 1145 callbacks.on_epoch_end(epoch, epoch_logs)\n1146 training_logs = epoch_logs\n1147 if self.stop_training:\n/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py in on_epoch_end(self, epoch, logs)\n 426 for callback in self.callbacks:\n 427 if getattr(callback, '_supports_tf_logs', False):\n--> 428 callback.on_epoch_end(epoch, logs)\n 429 else:\n 430 if numpy_logs is None: # Only convert once.\n/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py in on_epoch_end(self, epoch, logs)\n1029 \n1030 def on_epoch_end(self, epoch, logs=None):\n-> 1031 self._finalize_progbar(logs, self._train_step)\n1032 \n1033 def on_test_end(self, logs=None):\n/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/callbacks.py in _finalize_progbar(self, logs, counter)\n1086 \n1087 def _finalize_progbar(self, logs, counter):\n-> 1088 logs = tf_utils.to_numpy_or_python_type(logs or {})\n1089 if self.target is None:\n1090 if counter is not None:\n/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/utils/tf_utils.py in to_numpy_or_python_type(tensors)\n 512 return t # Don't turn ragged or sparse tensors to NumPy.\n 513 \n--> 514 return nest.map_structure(_to_single_numpy_or_python_type, tensors)\n 515 \n 516 \n/opt/conda/lib/python3.7/site-packages/tensorflow/python/util/nest.py in map_structure(func, *structure, **kwargs)\n 657 \n 658 return pack_sequence_as(\n--> 659 structure[0], [func(*x) for x in entries],\n 660 expand_composites=expand_composites)\n 661 \n/opt/conda/lib/python3.7/site-packages/tensorflow/python/util/nest.py in <listcomp>(.0)\n 657 \n 658 return pack_sequence_as(\n--> 659 structure[0], [func(*x) for x in entries],\n 660 expand_composites=expand_composites)\n 661 \n/opt/conda/lib/python3.7/site-packages/tensorflow/python/keras/utils/tf_utils.py in _to_single_numpy_or_python_type(t)\n 508 def _to_single_numpy_or_python_type(t):\n 509 if isinstance(t, ops.Tensor):\n--> 510 x = t.numpy()\n 511 return x.item() if np.ndim(x) == 0 else x\n 512 return t # Don't turn ragged or sparse tensors to NumPy.\n/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/ops.py in numpy(self)\n1069 \"\"\"\n1070 # TODO(slebedev): Consider avoiding a copy for non-CPU or remote tensors.\n-> 1071 maybe_arr = self._numpy() # pylint: disable=protected-access\n1072 return maybe_arr.copy() if isinstance(maybe_arr, np.ndarray) else maybe_arr\n1073 \n/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/ops.py in _numpy(self)\n1037 return self._numpy_internal()\n1038 except core._NotOkStatusException as e: # pylint: disable=protected-access\n-> 1039 six.raise_from(core._status_to_exception(e.code, e.message), None) # pylint: disable=protected-access\n1040 \n1041 @property\ntOfRangeError: 9 root error(s) found.\n(0) Out of range: {{function_node __inference_train_function_1081877}} End of sequence\n[[{{node cond_11/else/_106/cond_11/IteratorGetNext}}]]\n[[Shape_19/_120]]\n(1) Out of range: {{function_node __inference_train_function_1081877}} End of sequence\n[[{{node cond_11/else/_106/cond_11/IteratorGetNext}}]]\n(2) Out of range: {{function_node __inference_train_function_1081877}} End of sequence\n[[{{node cond_11/else/_106/cond_11/IteratorGetNext}}]]\n[[GroupCrossDeviceControlEdges_0/Identity_7/_325]]\n(3) Out of range: {{function_node __inference_train_function_1081877}} End of sequence\n[[{{node cond_11/else/_106/cond_11/IteratorGetNext}}]]\n[[strided_slice_27/_232]]\n(4) Out of range: {{function_node __inference_train_function_1081877}} End of sequence\n[[{{node cond_11/else/_106/cond_11/IteratorGetNext}}]]\n[[tpu_compile_succeeded_assert/_10601166017813830618/_5/_257]]\n(5) Out of range: {{function_node __inference_train_function_1081877}} End of sequence\n[[{{node cond_11/else/_106/cond_11/IteratorGetNext}}]]\n[[Pad_5/paddings/_148]]\n(6) Out of range: {{function_node __inference_train_function_1081877}} End of sequence\n[[{{node cond_11/else/_106/cond_11/IteratorGetNext}}]]\n@kevinmcisaac when i do image counts for valid and train
\ni get
\n1)1680000 560000
print(train_image_count,valid_image_count)
\nhow can we use folds as per your tfrecords set. i get 80 files set..
\n2 ) How you performed it there are multiple ways mentioned
\nrescales the waveform to [-1, 1]
@kevinmcisaac just a slight reminder. I feel your datasets folds are not built properly if u see above. any suggestion ?
", "votes": 0, "replies": [] } ] }, { "id": 1505096, "author_name": "hengck23", "author_url": "", "post_date": "2021-09-07T00:34:50.220000", "content": "\"Rescaling was essential.\"
\nthe reason could be the use of stride=2 in the cnn network (especially the first one)
by using enlarged image, you could have a sub-pixel convolution?
", "votes": 2, "replies": [] }, { "id": 1489163, "author_name": "Hao", "author_url": "", "post_date": "2021-08-24T18:45:28.073000", "content": "\n\nRescaling was essential. I'd be interested to hear from an Kaggel Expert if they found this and if so why?
\n
I guess it's due to original data has very small value(around 1e-20 range), that's why many people divide tha data with max() first to scale to [some value, 1].
", "votes": 1, "replies": [] }, { "id": 1450145, "author_name": "Mithil Salunkhe", "author_url": "", "post_date": "2021-08-05T03:51:25.433000", "content": "Any plans on sharing your keras layer of CWT ?
", "votes": 0, "replies": [ { "id": 1458576, "author_name": "Fractal Feelings", "author_url": "", "post_date": "2021-08-08T00:31:18.317000", "content": "I've not had time to clean up the repo, but you can use this
\n!pip install git+https://github.com//Kevin-McIsaac/cmorlet-tensorflow@Performance --no-deps > /dev/null
and there is basic documentation.
", "votes": 5, "replies": [] }, { "id": 1459052, "author_name": "Mithil Salunkhe", "author_url": "", "post_date": "2021-08-08T07:37:53.680000", "content": "So we have to just feed the npy files to the models by adding this layer ?
", "votes": 0, "replies": [] }, { "id": 1460514, "author_name": "Fractal Feelings", "author_url": "", "post_date": "2021-08-08T23:30:37.480000", "content": "Yes (see earlier comments) though to get the TPU performance you must use TFRecords (see early comments for links to these datasets)
", "votes": 3, "replies": [] }, { "id": 1470768, "author_name": "Geir Drange", "author_url": "", "post_date": "2021-08-13T17:48:44.130000", "content": "You can find a Keras CWT layer in this notebook. No bandpass filtering is required with this one, as it only computes the scaleogram for a frequency range of interest.
", "votes": 0, "replies": [] }, { "id": 1470965, "author_name": "", "author_url": "", "post_date": "2021-08-13T20:23:30.790000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 1559722, "author_name": "ChristopherZerafa", "author_url": "", "post_date": "2021-10-27T07:09:24.783000", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris
@kevinmcisaac nice work! may I ask what does the parameter wavelet_width: default = 1 in CWT ? what range of values should we consider ?
\nI saw the demo in your git repo where it uses 1, but not sure if that is the case for our data here
Thanks for sharing @kevinmcisaac , may I ask what do you mean by \"rescales the waveform to [-1, 1]\"? It's like standardize as -mean and /std, or min/max to [0, 1] then scale and shift to [-1, 1]? Thanks.
", "votes": 0, "replies": [ { "id": 1499842, "author_name": "Fractal Feelings", "author_url": "", "post_date": "2021-09-02T01:22:00.070000", "content": "Its Min/Max
", "votes": 1, "replies": [] } ] }, { "id": 1495884, "author_name": "Tanish Gupta", "author_url": "", "post_date": "2021-08-29T21:25:53.740000", "content": "@kevinmcisaac Can you guide me what is bandpass filter and CWT
", "votes": 0, "replies": [ { "id": 1495948, "author_name": "Fractal Feelings", "author_url": "", "post_date": "2021-08-30T00:15:50.607000", "content": "Your best option is to do some reading on the internet.
\nhttps://en.wikipedia.org/wiki/Band-pass_filter
\nhttps://www.kaggle.com/asauve/a-gentle-introduction-to-wavelet-for-data-analysis
In short a filter removes some frequencies from a wave and a band pass only allows frequencies in the band to pass.
\nA CWT is an alternative approach to Short Term Fourier Transform or Constant Quality Transform to converting a wave from time/amplitude to time/Frequency
", "votes": 1, "replies": [] } ] }, { "id": 1490958, "author_name": "Lowerce", "author_url": "", "post_date": "2021-08-26T03:31:55.107000", "content": "I have learned a lot from your summary!
", "votes": 0, "replies": [] }, { "id": 1485990, "author_name": "Zekun", "author_url": "", "post_date": "2021-08-22T14:59:01.930000", "content": "Hello, how is your single model's lb and cv?
", "votes": 0, "replies": [] }, { "id": 1480712, "author_name": "Kalilur Rahman", "author_url": "", "post_date": "2021-08-19T06:31:35.727000", "content": "Good post. Thanks for sharing your inputs. Well written indeed! @kevinmcisaac π
", "votes": 0, "replies": [] }, { "id": 1478898, "author_name": "Jaideep", "author_url": "", "post_date": "2021-08-18T07:26:34.500000", "content": "@kevinmcisaac does the order Bandpass filter makes any difference, Like apply CQT then BandPass
", "votes": 0, "replies": [] }, { "id": 1478581, "author_name": "dan", "author_url": "", "post_date": "2021-08-18T04:16:57.947000", "content": "I thought b7 on 512x512 would cause memory issue on many GPUs. Mind sharing GPUs you are using?
", "votes": 0, "replies": [ { "id": 1478728, "author_name": "Fractal Feelings", "author_url": "", "post_date": "2021-08-18T05:51:12.157000", "content": "I've been running on a TPU and find I get better performance, say 10-20x
", "votes": 1, "replies": [] } ] }, { "id": 1464254, "author_name": "Srushti Sawant", "author_url": "", "post_date": "2021-08-10T14:01:40.647000", "content": "Than you for sharing
", "votes": 0, "replies": [] }, { "id": 1461695, "author_name": "stallone", "author_url": "", "post_date": "2021-08-09T14:19:55.917000", "content": "Good work. @kevinmcisaac
", "votes": 0, "replies": [] }, { "id": 1459244, "author_name": "Yerram Varun", "author_url": "", "post_date": "2021-08-08T09:37:33.613000", "content": "@kevinmcisaac Hey Thanks for the detailed approach overview.
\nI just had a doubt. At what step are you resizing the scalograms?
.npy files -> CQT (keras layer) -> Resizing (?) -> EFN Model (keras model)
Won't this be a bottleneck here?
", "votes": 0, "replies": [ { "id": 1460496, "author_name": "Fractal Feelings", "author_url": "", "post_date": "2021-08-08T23:09:08.840000", "content": "@yerramvarun I don't resize the CWT for the EFN. My model looks like
\n\n model = tf.keras.Sequential([L.InputLayer(input_shape=(3, 4096),\n ComplexMorletCWT(n_scales = 64, stride=64, \n output='magnitude', data_format='channels_first'), \n L.Permute((2, 3, 1)), # transpose to channel last [:, time, n_scales, channels]\n efn.EfficientnetB4(include_top=False,weights='imagenet'),\n L.GlobalAveragePooling2D(),\n L.Dense(32,activation='relu'),\n L.Dense(1, activation='sigmoid')\n ])\n
\nI guess you are asking at the Keras documentation for EFN states each model uses a specific size. I found this confusing and discovered they worked Ok with other sizes. I'd be interested to hear from someone with more experience about this?
Note: I'm using Continuious Wavelett Transfroms (CWT) not CQT using the keras layer in this
\n!pip install git+https://github.com//Kevin-McIsaac/cmorlet-tensorflow@Performance --no-deps > /dev/null
That makes sense. Thanks for clearing that up for me
", "votes": 2, "replies": [] }, { "id": 1494194, "author_name": "Jaideep", "author_url": "", "post_date": "2021-08-28T13:36:27.907000", "content": "where do you resize your image to different square sizes @kevinmcisaac
", "votes": 0, "replies": [] }, { "id": 1495919, "author_name": "Fractal Feelings", "author_url": "", "post_date": "2021-08-29T22:40:10.310000", "content": "I use the n_scales and strides parameters to determine the size of the image, e.g., n_scales = 128, stride=32 creates a spectrogram that is 128x128. While efficient net was trained on specific image sizes, it can accept any size.
", "votes": 0, "replies": [] }, { "id": 1495924, "author_name": "Araik Tamazian", "author_url": "", "post_date": "2021-08-29T22:55:35.327000", "content": "Let's say you want image of WIDTH x HEIGHT size. Then you should set n_scales=WIDTH, and strides = int(np.ceil(4096/HEIGHT)) (assuming your input has (3, 4096) shape). This works for me.
\n\nI started with nnAudio Constant Quality Transform (CQT) and found this a lot faster than librosa. The entire processing, i.e., load the data, bandpass filter, creating the a 64x64 scalogram in nnAUdio and classifying in EFN B4 took about 8hrs on a CPU and 2hrs on a GPU.
\n
can you tell me how many epochs you did in 2 hours you mentioned ?
", "votes": 0, "replies": [ { "id": 1458555, "author_name": "Fractal Feelings", "author_url": "", "post_date": "2021-08-08T00:08:40.940000", "content": "My notes are not the best for the old nnAudio runs. For recent runs with a CWT that create an 64x64 image and B4 I get the following
\nTPU 32s per epoch, ran 30 epochs.
\nGPU 260s per epoch, 29 epochs
The total notebook time, including set and inferencing was 28 min and 2hrs 14 respectively.
", "votes": 2, "replies": [] }, { "id": 1458814, "author_name": "Md Zarif Ul Alam", "author_url": "", "post_date": "2021-08-08T04:51:42.973000", "content": "Thanks! Can you tell me how you optimized the preprocessing / process of conversion to TFrecords ? When I was trying to do some custom preprocessing and trying to convert them to TFrecords, each notebook had only 5GB output size. So, it will take me 16 runs, each around 4GB. This seems to be a bottleneck.
", "votes": 0, "replies": [] }, { "id": 1460513, "author_name": "Fractal Feelings", "author_url": "", "post_date": "2021-08-08T23:28:43.087000", "content": "I found I could do this as three batches; 2 for train and 1 for submissions.
\nThese are public datasets named like this G2NET TFRecords with 20-500Hz bandpass Part I
\nYou can then look at the notebook that generated this which use this utility script
", "votes": 2, "replies": [] }, { "id": 1467960, "author_name": "Md Zarif Ul Alam", "author_url": "", "post_date": "2021-08-12T07:41:29.597000", "content": "@kevinmcisaac can you tell me how many training samples are in each tfrecord of your dataset ? I can't seem to find any number from your utility script
", "votes": 0, "replies": [] }, { "id": 1468089, "author_name": "Fractal Feelings", "author_url": "", "post_date": "2021-08-12T09:00:06.677000", "content": "Rather than pick a number of samples per TFR The approach I used was to create a specific number of TFrecord files so that each was large enough to get good read performance and there were enough file to enable overlapping parallel reads.
\nYou can see in the notebook I created 70 files so each trf file has 560000/70 = 8,000 samples
", "votes": 2, "replies": [] }, { "id": 1468528, "author_name": "", "author_url": "", "post_date": "2021-08-12T13:06:55.060000", "content": "", "votes": 0, "replies": [] }, { "id": 1469484, "author_name": "Fractal Feelings", "author_url": "", "post_date": "2021-08-13T01:02:39.323000", "content": "I converted the inputs to np.float32 as part of the preprocessing. Having read this I now understand why you are asking a however not familiar enough with TF 2.0 to be able to answer the question of using mixed precision.
\nIf you have the skills you could fork the repo and make the necessary changes. I'd be keen to hear how that goes.
", "votes": 1, "replies": [] }, { "id": 1471865, "author_name": "", "author_url": "", "post_date": "2021-08-14T14:16:33.637000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 1452494, "author_name": "Araik Tamazian", "author_url": "", "post_date": "2021-08-05T16:11:29.170000", "content": "Can you tell which package you used for making a bandpass filter?
", "votes": 0, "replies": [ { "id": 1452559, "author_name": "", "author_url": "", "post_date": "2021-08-05T16:28:27.047000", "content": "", "votes": 3, "replies": [] }, { "id": 1458564, "author_name": "Fractal Feelings", "author_url": "", "post_date": "2021-08-08T00:16:35.877000", "content": "I used scipy.signal. Here is the code I used.
\nfrom scipy import signal\n\nbHP, aHP = signal.butter(8, (20, 500), btype='bandpass', fs= 2048)\ndef filterSig(waves, a=aHP, b=bHP):\n '''Apply a 20Hz high pass filter to the three events'''\n return np.array([signal.filtfilt(b, a, wave) for wave in waves]) #lfilter introduces a larger spike around 20hz\nInitially issued gwpy but its a lot of baggage just to do a simple filter. I use this in a pre-processing step that re-writes the data s TFrecords to speed up data loading and keep the TPUs busy.
\nWhat I'd like to do is make a bandpass filter in Keras and include that in my NN
", "votes": 10, "replies": [] }, { "id": 1459232, "author_name": "datasaurus", "author_url": "", "post_date": "2021-08-08T09:30:52.680000", "content": "I think for a 20-500Hz cuts you need to use something like:
\nnyq = 0.5 * fs # Nyquist frequency\nlow = 20 / nyq\nhigh = 500 / nyq\nb, a = butter(order, [low, high], btype='bandpass')\nFrom: https://scipy-cookbook.readthedocs.io/items/ButterworthBandpass.html
\nEdit: I just found out that if you pass the fs arg, you don't need to do this :)
@kevinmcisaac should this be applied before or after normalization ?
", "votes": 1, "replies": [] }, { "id": 1463981, "author_name": "Fractal Feelings", "author_url": "", "post_date": "2021-08-10T12:07:08.570000", "content": "I applied the scaling after the bandpass filter but I don't think the order matter.
", "votes": 1, "replies": [] }, { "id": 1464039, "author_name": "Mithil Salunkhe", "author_url": "", "post_date": "2021-08-10T12:29:37.577000", "content": "Just tried both after and before scaling. They both look the same
", "votes": 2, "replies": [] }, { "id": 1464081, "author_name": "Fractal Feelings", "author_url": "", "post_date": "2021-08-10T12:47:05.490000", "content": "Good to know
", "votes": 3, "replies": [] }, { "id": 1464124, "author_name": "", "author_url": "", "post_date": "2021-08-10T13:06:32.067000", "content": "", "votes": 2, "replies": [] }, { "id": 1465337, "author_name": "Fractal Feelings", "author_url": "", "post_date": "2021-08-11T02:33:43.033000", "content": "Yes I did. The results are reported here.
\nTL;DR i make the AUC 10% worse.
Thanks to all who replied. This info is quite useful.
", "votes": 1, "replies": [] } ] }, { "id": 1497459, "author_name": "", "author_url": "", "post_date": "2021-08-31T08:03:53.073000", "content": "", "votes": 0, "replies": [ { "id": 1497469, "author_name": "Fractal Feelings", "author_url": "", "post_date": "2021-08-31T08:12:45.913000", "content": "No, I change the size of n_scales and or stride to create the desired sized scalogram, e.g,
\n\n\nComplexMorletCWT(n_scales = 64, stride=64)
\n
creates a 64 x 64 image.
\nI simply divided by a fixed scale factor, i.e., the same number for each channel (detector) and all samples.
\nI started out by dividing each channel by the max of the absolute value of that channel, i.e., standardize into the range [-1,1] but then I thought this might lose some important information so I switch to a common number across all. Since there is a large dynamic range across the various samples, and across teh detectors, I just picked something that seemed reasonable for many samples (i.e., more art than science).
", "votes": 4, "replies": [] } ] }, { "id": 1451854, "author_name": "", "author_url": "", "post_date": "2021-08-05T12:59:10.630000", "content": "", "votes": 2, "replies": [ { "id": 1452016, "author_name": "", "author_url": "", "post_date": "2021-08-05T13:55:38.030000", "content": "", "votes": 2, "replies": [] }, { "id": 1458574, "author_name": "Fractal Feelings", "author_url": "", "post_date": "2021-08-08T00:30:34.733000", "content": "I've not tried ENetV2 yet. How much improvement did you get, i.e., %?
", "votes": 1, "replies": [] }, { "id": 1459249, "author_name": "", "author_url": "", "post_date": "2021-08-08T09:40:25.407000", "content": "", "votes": 1, "replies": [] }, { "id": 1478994, "author_name": "Mithil Salunkhe", "author_url": "", "post_date": "2021-08-18T08:33:41.167000", "content": "How was efficientnetv2 XL comparable to efficientnet B7 ?
", "votes": 1, "replies": [] } ] }, { "id": 1473217, "author_name": "Yuri Sun", "author_url": "", "post_date": "2021-08-15T12:40:49.160000", "content": "Thanks for sharing!
", "votes": 0, "replies": [] }, { "id": 1464152, "author_name": "Mohamed Iniesta", "author_url": "", "post_date": "2021-08-10T13:15:57.927000", "content": "Thanks for sharing ! ππ
", "votes": 0, "replies": [] }, { "id": 1455557, "author_name": "Saurav Maheshkar βοΈ", "author_url": "", "post_date": "2021-08-06T15:51:22.947000", "content": "Thanks for sharing your insights
", "votes": 0, "replies": [] }, { "id": 1454015, "author_name": "Sunnymoon Sultan", "author_url": "", "post_date": "2021-08-06T04:32:03.510000", "content": "Thanks for sharing!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1449690": "Since this was my first time using a NN, GPU/TPU and Tensorflow I had a steep learning curve. I tried a lot of things and thought I'd share what I learnt. \n\nIn summary I used Keras to build a model that \n* applies a bandpass filter, \n* rescales the waveform to [-1, 1]\n* Applies a time-frequency transform (e.g., STFT, CQT, CWT) to create a scalogram\n* Classifies using Efficient Net.\n\nIs is very similar to what many others have done. I currently have a AUC score of 0.872, which puts me in the middle of the leaderboard.\n\nWhat I learnt from experimenting with this was. \n* The bandpass filter (20hz-500Hz) has quite an impact. I think this is because the noise below 20 Hz is high and signal is weaker at low frequencies, i.e., very low SNR. I chose 500 Hz because the highest signal frequency for a black hole event is 350 Hz. I could not find articles on the frequency range for other types of events so I picked 500 as there are some strong noise sources just above that. It also looks like the best SNR is around 350-500Hz\n* Rescaling was essential. I'd be interested to hear from an Kaggel Expert if they found this and if so why? \n* I started with nnAudio Constant Quality Transform (CQT) and found this a lot faster than librosa. The entire processing, i.e., load the data, bandpass filter, creating the a 64x64 scalogram in nnAUdio and classifying in EFN B4 took about 8hrs on a CPU and **2hrs on a GPU**. \n* I discovered a TF implementation of Continuous Wavelet Transform and updated it to TF 2.0, optimised the algorithm, and made it a Keras Layer. This enables me to have a single Kera model for the scalogram -> EFN that runs on a TPU. The entire processing time dropped to about **30 min**.\n* I pre-processed the numpy input files into TFRecords and first epoch takes **3 min** and subsequent took 30s.\n\nThis gave me an experimental test bed for further experimentation. \n* Larger scalograms, e.g., 256x256 and 512x512 gives a small boost.\n* Different CWT wavelet_width, 8 seems optimal, give a small boost. I found making this paramater trainable also gave a small boost.\n* Transfer Learning (i.e., freeze the EFN layer) followed by Fine Tuning (unfreeze). Marginal improvement\n* DIfferent EFN models, e.g., B0, B4, B7, with B7 having a slight boost\n\nAs I was running a lot of experiments I found that I needed a better way to record the results and started using Neptune.ai, which has been very useful.\n\nMy best result so far is B7 with a 512x512 scalogram ", "1454700": "Good work.\n\nYou're not just tweaking public notebooks. This is how to make a difference here. Keep pushing (but not too hard, I'd like to stay ahead of you ... )", "1487386": "I tried to add your work into my pipeline. Hower the AUC is always around 0.5π¦...", "1475260": "Regarding the rescaling. You always want your data to have zero mean and unity standard deviation. This allows deep neural networks to train well, see the famous paper on this: http://proceedings.mlr.press/v9/glorot10a/glorot10a.pdf\n\nHope I got youe question correclty.", "1466846": "Could you share the TF implementation of CWT you have been using?\n\nI found [this](https://github.com/nickgeoca/cwt-tensorflow) version and tried implementing it in PyTorch, however, the performance is significantly worse than Q-Transform.\n\nMy PyTorch implementation is [here](https://www.kaggle.com/anjum48/continuous-wavelet-transform-cwt-in-pytorch)", "1505096": "\"Rescaling was essential.\"\nthe reason could be the use of stride=2 in the cnn network (especially the first one)\n\nby using enlarged image, you could have a sub-pixel convolution?", "1489163": "> Rescaling was essential. I'd be interested to hear from an Kaggel Expert if they found this and if so why?\n\nI guess it's due to original data has very small value(around 1e-20 range), that's why many people divide tha data with max() first to scale to [some value, 1]. ", "1450145": "Any plans on sharing your keras layer of CWT ?", "1559722": "Hey All,\n\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.\n\nThank you all,\n\nRegards,\nChris", "1504887": "@kevinmcisaac nice work! may I ask what does the parameter `wavelet_width: default = 1` in CWT ? what range of values should we consider ? \nI saw the demo in your git repo where it uses 1, but not sure if that is the case for our data here", "1499318": "Thanks for sharing @kevinmcisaac , may I ask what do you mean by \"rescales the waveform to [-1, 1]\"? It's like standardize as -mean and /std, or min/max to [0, 1] then scale and shift to [-1, 1]? Thanks.", "1495884": " @kevinmcisaac Can you guide me what is bandpass filter and CWT", "1490958": "I have learned a lot from your summary!", "1485990": "Hello, how is your single model's lb and cv?", "1480712": "Good post. Thanks for sharing your inputs. Well written indeed! @kevinmcisaac π", "1478898": "@kevinmcisaac does the order Bandpass filter makes any difference, Like apply CQT then BandPass ", "1478581": "I thought b7 on 512x512 would cause memory issue on many GPUs. Mind sharing GPUs you are using?", "1464254": "Than you for sharing ", "1461695": "Good work. @kevinmcisaac ", "1459244": "@kevinmcisaac Hey Thanks for the detailed approach overview.\nI just had a doubt. At what step are you resizing the scalograms?\n\n``` .npy files -> CQT (keras layer) -> Resizing (?) -> EFN Model (keras model) ```\n\nWon't this be a bottleneck here?", "1456944": "> I started with nnAudio Constant Quality Transform (CQT) and found this a lot faster than librosa. The entire processing, i.e., load the data, bandpass filter, creating the a 64x64 scalogram in nnAUdio and classifying in EFN B4 took about 8hrs on a CPU and 2hrs on a GPU. \n\ncan you tell me how many epochs you did in 2 hours you mentioned ?", "1452494": "Can you tell which package you used for making a bandpass filter?", "1497459": "", "1464122": "", "1460735": "", "1451854": "", "1473217": "Thanks for sharing!", "1464152": "Thanks for sharing ! ππ", "1455557": "Thanks for sharing your insights ", "1454015": "Thanks for sharing!\n" } }