Hi,
\nLike many other people I am using transfer learning with EfficientNetB7 from tensorflow to classify a signal. I use the image spectogram images kindly provided here by another Kaggle user https://www.kaggle.com/coldfir3/g2net-cqt-dataset-pt1-jpgrgb
\nI am using 140,000 images i batches of 32
\nJust to report what I am doing to already speed things up.
\n1) I import the images using image_dataset_from_directory in a (256 x 256) image_size
\n2) I use tf.data.experimental.AUTOTUNE to prefetch the images so that the gpu is always active
\n3) I use EfficientNetB7 from tensorflow. I remove the top layers and freeze the remaining weights. I then add layers of my own for classification. I also use preprocess_inputs. I only have 2561 trainable parameters
I am using a M1 Mac. Its still taking >2 hrs to go through the entire dataset. Any tips or any guesses on what I might be doing wrong?
\nThanks,
\nAkshay
Hi,
\nLike many other people I am using transfer learning with EfficientNetB7 from tensorflow to classify a signal. I use the image spectogram images kindly provided here by another Kaggle user https://www.kaggle.com/coldfir3/g2net-cqt-dataset-pt1-jpgrgb
\nI am using 140,000 images i batches of 32
\nJust to report what I am doing to already speed things up.
\n1) I import the images using image_dataset_from_directory in a (256 x 256) image_size
\n2) I use tf.data.experimental.AUTOTUNE to prefetch the images so that the gpu is always active
\n3) I use EfficientNetB7 from tensorflow. I remove the top layers and freeze the remaining weights. I then add layers of my own for classification. I also use preprocess_inputs. I only have 2561 trainable parameters
I am using a M1 Mac. Its still taking >2 hrs to go through the entire dataset. Any tips or any guesses on what I might be doing wrong?
\nThanks,
\nAkshay
\n\nAny tips or any guesses on what I might be doing wrong?
\n
Using B7 on local non-server PC. Even with 10-20x speedup in inference mode with frozen layers, it is still a big net that needs to pass your images through it, and it takes time.
", "rawMarkdown": "> Any tips or any guesses on what I might be doing wrong?\n\nUsing B7 on local non-server PC. Even with 10-20x speedup in inference mode with frozen layers, it is still a big net that needs to pass your images through it, and it takes time.", "votes": null }, { "id": "1494537", "postDate": "08/28/2021 18:45:00", "content": "@aghalsa M1 Mac with 16GB, I tried too. It is not good yet to train large models. Kaggle/Colab is much better. See this article from w&b - https://wandb.ai/vanpelt/m1-benchmark/reports/Can-Apple-s-M1-help-you-train-models-faster-cheaper-than-NVIDIA-s-V100---VmlldzozNTkyMzg. Maybe M1X for training :D
", "rawMarkdown": "aghalsa M1 Mac with 16GB, I tried too. It is not good yet to train large models. Kaggle/Colab is much better. See this article from w&b - https://wandb.ai/vanpelt/m1-benchmark/reports/Can-Apple-s-M1-help-you-train-models-faster-cheaper-than-NVIDIA-s-V100---VmlldzozNTkyMzg. Maybe M1X for training :D", "votes": null }, { "id": "1494547", "postDate": "08/28/2021 18:54:45", "content": "Thanks. Guess Il will start using collab. Is there a way to preprocess all the inputs once beforehand so I can play with meta parameters without having through wait this long again?
", "rawMarkdown": "Thanks. Guess Il will start using collab. Is there a way to preprocess all the inputs once beforehand so I can play with meta parameters without having through wait this long again?", "votes": null }, { "id": "1494554", "postDate": "08/28/2021 19:00:02", "content": "Thanks. Is there a way to guesstimate how long it should take. Also is there a way to preprocess all the inputs once beforehand so I can play with meta parameters without having through wait this long again?
", "rawMarkdown": "Thanks. Is there a way to guesstimate how long it should take. Also is there a way to preprocess all the inputs once beforehand so I can play with meta parameters without having through wait this long again?", "votes": null }, { "id": "1494562", "postDate": "08/28/2021 19:14:40", "content": "https://www.kaggle.com/hidehisaarai1213/g2net-tf-on-the-fly-cqt-tpu-training - 1 Epoch with B7 - 256, this notebook able to do in ~ 5mins @ Kaggle TPU.
", "rawMarkdown": "https://www.kaggle.com/hidehisaarai1213/g2net-tf-on-the-fly-cqt-tpu-training - 1 Epoch with B7 - 256, this notebook able to do in ~ 5mins @ Kaggle TPU.", "votes": null }, { "id": "1494575", "postDate": "08/28/2021 19:34:56", "content": "If you would not change earlier layers/input images, then you can pass all your images through those layers once, save results and use those results as \"images\" for upper layer training.
\nDoing it this way might be worse than just using a smaller net, though.
", "rawMarkdown": "If you would not change earlier layers/input images, then you can pass all your images through those layers once, save results and use those results as \"images\" for upper layer training.\n\nDoing it this way might be worse than just using a smaller net, though.", "votes": null }, { "id": "1498431", "postDate": "09/01/2021 01:37:27", "content": "@aghalsa to give you a comparison I'm using Keras/TF 2.0 with a TPU all run in Kaggle notebooks:
\nFor experimentation I set n_scales to 128 and stride to 32 to create a 128x128 image with three channels that is passed to B0. A full epoch takes under 3min.
\nIf I switch to 256x256 CWT and B7 it is about 15 min per epoch.
\nIn my experience the TPU is 8x faster than a GPU which is 100x faster than a CPU (using Kaggle resources). The bottom line is to get high performance you need to use a GPU or TPU with an optimised CWT and probably with TFRecords.
", "rawMarkdown": "aghalsa to give you a comparison I'm using Keras/TF 2.0 with a TPU all run in Kaggle notebooks:\n* I use TFRecords for read performance\n* An optimised CWT (https://github.com/Kevin-McIsaac/cmorlet-tensorflow) instead of nnAudio/CQT. The performance of these is similar but much better than librosa\n* EFN\n* Split my data 95%/5%.\n\nFor experimentation I set n_scales to 128 and stride to 32 to create a 128x128 image with three channels that is passed to B0. A full epoch takes under 3min. \n\nIf I switch to 256x256 CWT and B7 it is about 15 min per epoch.\n\nIn my experience the TPU is 8x faster than a GPU which is 100x faster than a CPU (using Kaggle resources). The bottom line is to get high performance you need to use a GPU or TPU with an optimised CWT and probably with TFRecords.", "votes": null }, { "id": "1561009", "postDate": "10/27/2021 09:09:17", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris
\n\nAny tips or any guesses on what I might be doing wrong?
\n
Using B7 on local non-server PC. Even with 10-20x speedup in inference mode with frozen layers, it is still a big net that needs to pass your images through it, and it takes time.
", "votes": null, "replies": [ { "id": 1494554, "author_name": "aghalsa", "author_url": "", "post_date": "08/28/2021 19:00:02", "content": "Thanks. Is there a way to guesstimate how long it should take. Also is there a way to preprocess all the inputs once beforehand so I can play with meta parameters without having through wait this long again?
", "votes": null, "replies": [] }, { "id": 1494575, "author_name": "fffrrt", "author_url": "", "post_date": "08/28/2021 19:34:56", "content": "If you would not change earlier layers/input images, then you can pass all your images through those layers once, save results and use those results as \"images\" for upper layer training.
\nDoing it this way might be worse than just using a smaller net, though.
", "votes": null, "replies": [] } ] }, { "id": 1494537, "author_name": "seshurajup", "author_url": "", "post_date": "08/28/2021 18:45:00", "content": "@aghalsa M1 Mac with 16GB, I tried too. It is not good yet to train large models. Kaggle/Colab is much better. See this article from w&b - https://wandb.ai/vanpelt/m1-benchmark/reports/Can-Apple-s-M1-help-you-train-models-faster-cheaper-than-NVIDIA-s-V100---VmlldzozNTkyMzg. Maybe M1X for training :D
", "votes": null, "replies": [ { "id": 1494547, "author_name": "aghalsa", "author_url": "", "post_date": "08/28/2021 18:54:45", "content": "Thanks. Guess Il will start using collab. Is there a way to preprocess all the inputs once beforehand so I can play with meta parameters without having through wait this long again?
", "votes": null, "replies": [] }, { "id": 1494562, "author_name": "seshurajup", "author_url": "", "post_date": "08/28/2021 19:14:40", "content": "https://www.kaggle.com/hidehisaarai1213/g2net-tf-on-the-fly-cqt-tpu-training - 1 Epoch with B7 - 256, this notebook able to do in ~ 5mins @ Kaggle TPU.
", "votes": null, "replies": [] } ] }, { "id": 1498431, "author_name": "kevinmcisaac", "author_url": "", "post_date": "09/01/2021 01:37:27", "content": "@aghalsa to give you a comparison I'm using Keras/TF 2.0 with a TPU all run in Kaggle notebooks:
\nFor experimentation I set n_scales to 128 and stride to 32 to create a 128x128 image with three channels that is passed to B0. A full epoch takes under 3min.
\nIf I switch to 256x256 CWT and B7 it is about 15 min per epoch.
\nIn my experience the TPU is 8x faster than a GPU which is 100x faster than a CPU (using Kaggle resources). The bottom line is to get high performance you need to use a GPU or TPU with an optimised CWT and probably with TFRecords.
", "votes": null, "replies": [] }, { "id": 1561009, "author_name": "zerafachris", "author_url": "", "post_date": "10/27/2021 09:09:17", "content": "Hey All,
\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.
\nThank you all,
\nRegards,
\nChris