I use 2023 competition data only, but 3 of my models have pre-train weights on 2022 data.
\nI use SED architecture, almost the same as this notebook with very minor modifications (e.g., emb_size, num_layers).
\nI tried 8 backbones in total, including 3 large and 5 small:
\nlarge backbones: efficientnetv2_m, eca_nfnet_l0, seresnext50
\nsmall backbones: efficientnet_v0, mnasnet_100, spnasnet_100, mobilenetv2_100, resnet34
As seresnext50 and resnet34 perform not well based on my cv, so I dropped them in the final ensemble.
\nI got the information of mnasnet_100, spnasnet_100, and mobilenetv2_100 by asking ChatGPT: What are small models offered by timm and ChatGPT replied with these models with the number of parameters, which is very helpful.
One of my inference notebooks can be found here
\nall models are trained on 5s clip.
soft label: in the one-hot gt, I give the primary label with \"1\" and the secondary label with \"0.3\".
data augmentation: add GaussianNoise, Background noise, PinkNoise, adjust Gain, use cut mix and mix up (very helpful).
Progressive pretraining: the hardest part of this competition is to find a good set of parameters when applying melspectrum, especially the n_fft, hop_length, and n_mels, which affect the input image (melspectrum) size. I tried many different params, and surprisingly, I found that if you have a model trained on params set A, then you load the model as a pre-trained model (except the first and last layer), and then train it on params set B, the model will perform much better than directly training on params set B, and this is the key idea of my solution. All of my models were trained on many different parameters and finally finetuned on n_mels=128, n_fft=1024, and hop_length=320 to make it the same size, so that can be ensemble efficiently.
loss function: BCEWithLogits + BCEFocal2WayLoss, same as previous years.
cv strategy: I use 5fold stratified cross-validation and select the best 1 or 2 fold according to cv.
ensemble strategy: each model has 1/2 fold for ensemble, and I have 6/7 models in total, with 6 different backbones as mentioned before. The reason I didn't use all folds is because of the computational limitation asked by the host.
cpu acceleration: I use ThreadPoolExecutor as I have mentioned here, and I also convert models to onnx which utilizes the model quantization technique and shows faster inference time.
I previously made a request regarding the submission time function, as mentioned here, when @ammarali32 and @jaafarmahmoud1 found 10 PM to 6 AM EST is a good time to make submissions. The reason behind this recommendation is that 10 PM EST is considered late for the American region, while it corresponds to 4 AM in Europe and 10 AM in Asia, which might be too early for many participants. Consequently, this particular timeframe becomes somewhat magical, as very few people tend to make submissions during this period. Therefore, if one chooses to make a submission during this time, they could potentially benefit from it.
\nbased on my experiments, I believe the model params have an impact on inference time when using CPU, i.e., with the same model architecture and the number of params, but different models, some may be fast some may not. @chrisqiu 's post verifies this situation. As the host is trying to build apps that can be deployed in the real world using the CPU, I think this issue is worth mentioning @stefankahl.
I found that both my CV and lb vary based on the different folds, which indicates a good data split may have a great impact on the result, and some luck is needed when setting the random seed for the data split.
I appreciate @stefankahl for hosting this amazing competition, @herbison for answering my questions regarding the submission timeout error, and thanks to the Kaggle community for sharing their insights, ideas, and thoughts. This is a wonderful learning experience and I hope everyone enjoys it!
", "messages": [ { "id": "2273030", "postDate": "05/25/2023 01:09:59", "content": "I use 2023 competition data only, but 3 of my models have pre-train weights on 2022 data.
\nI use SED architecture, almost the same as this notebook with very minor modifications (e.g., emb_size, num_layers).
\nI tried 8 backbones in total, including 3 large and 5 small:
\nlarge backbones: efficientnetv2_m, eca_nfnet_l0, seresnext50
\nsmall backbones: efficientnet_v0, mnasnet_100, spnasnet_100, mobilenetv2_100, resnet34
As seresnext50 and resnet34 perform not well based on my cv, so I dropped them in the final ensemble.
\nI got the information of mnasnet_100, spnasnet_100, and mobilenetv2_100 by asking ChatGPT: What are small models offered by timm and ChatGPT replied with these models with the number of parameters, which is very helpful.
One of my inference notebooks can be found here
\nall models are trained on 5s clip.
soft label: in the one-hot gt, I give the primary label with \"1\" and the secondary label with \"0.3\".
data augmentation: add GaussianNoise, Background noise, PinkNoise, adjust Gain, use cut mix and mix up (very helpful).
Progressive pretraining: the hardest part of this competition is to find a good set of parameters when applying melspectrum, especially the n_fft, hop_length, and n_mels, which affect the input image (melspectrum) size. I tried many different params, and surprisingly, I found that if you have a model trained on params set A, then you load the model as a pre-trained model (except the first and last layer), and then train it on params set B, the model will perform much better than directly training on params set B, and this is the key idea of my solution. All of my models were trained on many different parameters and finally finetuned on n_mels=128, n_fft=1024, and hop_length=320 to make it the same size, so that can be ensemble efficiently.
loss function: BCEWithLogits + BCEFocal2WayLoss, same as previous years.
cv strategy: I use 5fold stratified cross-validation and select the best 1 or 2 fold according to cv.
ensemble strategy: each model has 1/2 fold for ensemble, and I have 6/7 models in total, with 6 different backbones as mentioned before. The reason I didn't use all folds is because of the computational limitation asked by the host.
cpu acceleration: I use ThreadPoolExecutor as I have mentioned here, and I also convert models to onnx which utilizes the model quantization technique and shows faster inference time.
I previously made a request regarding the submission time function, as mentioned here, when @ammarali32 and @jaafarmahmoud1 found 10 PM to 6 AM EST is a good time to make submissions. The reason behind this recommendation is that 10 PM EST is considered late for the American region, while it corresponds to 4 AM in Europe and 10 AM in Asia, which might be too early for many participants. Consequently, this particular timeframe becomes somewhat magical, as very few people tend to make submissions during this period. Therefore, if one chooses to make a submission during this time, they could potentially benefit from it.
\nbased on my experiments, I believe the model params have an impact on inference time when using CPU, i.e., with the same model architecture and the number of params, but different models, some may be fast some may not. @chrisqiu 's post verifies this situation. As the host is trying to build apps that can be deployed in the real world using the CPU, I think this issue is worth mentioning @stefankahl.
I found that both my CV and lb vary based on the different folds, which indicates a good data split may have a great impact on the result, and some luck is needed when setting the random seed for the data split.
I appreciate @stefankahl for hosting this amazing competition, @herbison for answering my questions regarding the submission timeout error, and thanks to the Kaggle community for sharing their insights, ideas, and thoughts. This is a wonderful learning experience and I hope everyone enjoys it!
", "rawMarkdown": "#Training Data\n\nI use 2023 competition data only, but 3 of my models have pre-train weights on 2022 data.\n\n#Model Architecture\nI use SED architecture, almost the same as [this notebook](https://www.kaggle.com/code/kaerunantoka/birdclef2022-use-2nd-label-f0) with very minor modifications (e.g., emb_size, num_layers).\n\nI tried 8 backbones in total, including 3 large and 5 small: \nlarge backbones: efficientnetv2_m, eca_nfnet_l0, seresnext50\nsmall backbones: efficientnet_v0, mnasnet_100, spnasnet_100, mobilenetv2_100, resnet34\n\nAs seresnext50 and resnet34 perform not well based on my cv, so I dropped them in the final ensemble.\n\nI got the information of mnasnet_100, spnasnet_100, and mobilenetv2_100 by asking ChatGPT: `What are small models offered by timm` and ChatGPT replied with these models with the number of parameters, which is very helpful. \n\n#Code\nOne of my inference notebooks can be found [here](https://www.kaggle.com/code/leonshangguan/fork-of-fork-of-onnx-final-combine-bs4-b9daf6?scriptVersionId=130783179)\n\n#Training Strategy\n1. all models are trained on 5s clip.\n\n2. soft label: in the one-hot gt, I give the primary label with \"1\" and the secondary label with \"0.3\".\n\n3. data augmentation: add GaussianNoise, Background noise, PinkNoise, adjust Gain, use cut mix and mix up (very helpful).\n\n4. Progressive pretraining: the hardest part of this competition is to find a good set of parameters when applying melspectrum, especially the n_fft, hop_length, and n_mels, which affect the input image (melspectrum) size. I tried many different params, and surprisingly, I found that if you have a model trained on params set A, then you load the model as a pre-trained model (except the first and last layer), and then train it on params set B, the model will perform much better than directly training on params set B, and this is the key idea of my solution. All of my models were trained on many different parameters and finally finetuned on n_mels=128, n_fft=1024, and hop_length=320 to make it the same size, so that can be ensemble efficiently.\n\n5. loss function: BCEWithLogits + BCEFocal2WayLoss, same as previous years.\n\n6. cv strategy: I use 5fold stratified cross-validation and select the best 1 or 2 fold according to cv. \n\n7. ensemble strategy: each model has 1/2 fold for ensemble, and I have 6/7 models in total, with 6 different backbones as mentioned before. The reason I didn't use all folds is because of the computational limitation asked by the host.\n\n8. cpu acceleration: I use ThreadPoolExecutor as I have mentioned [here](https://www.kaggle.com/competitions/birdclef-2023/discussion/401587), and I also convert models to onnx which utilizes the model quantization technique and shows faster inference time.\n\n#Others \n1. more than 70 of my submissions encountered timeout error, and I therefore highly request the function that if more information can be released for the submission, for example, the running time @herbison . I suffered from it a lot, and many almost the same submissions sometimes pass but sometimes not, even with the identical code. \n\nI previously made a request regarding the submission time function, as mentioned [here](https://www.kaggle.com/competitions/uw-madison-gi-tract-image-segmentation/discussion/337343), when @ammarali32 and @jaafarmahmoud1 found 10 PM to 6 AM EST is a good time to make submissions. The reason behind this recommendation is that 10 PM EST is considered late for the American region, while it corresponds to 4 AM in Europe and 10 AM in Asia, which might be too early for many participants. Consequently, this particular timeframe becomes somewhat magical, as very few people tend to make submissions during this period. Therefore, if one chooses to make a submission during this time, they could potentially benefit from it.\n\n2. based on my experiments, I believe the model params have an impact on inference time when using CPU, i.e., with the same model architecture and the number of params, but different models, some may be fast some may not. @chrisqiu 's [post](https://www.kaggle.com/competitions/birdclef-2023/discussion/409331) verifies this situation. As the host is trying to build apps that can be deployed in the real world using the CPU, I think this issue is worth mentioning @stefankahl. \n\n3. I found that both my CV and lb vary based on the different folds, which indicates a good data split may have a great impact on the result, and some luck is needed when setting the random seed for the data split.\n\n\n#Acknowledgement\nI appreciate @stefankahl for hosting this amazing competition, @herbison for answering my questions regarding the submission timeout error, and thanks to the Kaggle community for sharing their insights, ideas, and thoughts. This is a wonderful learning experience and I hope everyone enjoys it!", "votes": null }, { "id": "2273058", "postDate": "05/25/2023 01:48:21", "content": "Congrats for your first solo gold! 💯
", "rawMarkdown": "Congrats for your first solo gold! 💯", "votes": null }, { "id": "2273061", "postDate": "05/25/2023 01:50:03", "content": "Thank you!!!
", "rawMarkdown": "Thank you!!!", "votes": null }, { "id": "2273257", "postDate": "05/25/2023 05:03:41", "content": "Wow Crazy ! Thanks for sharing and Congratulations🎉
", "rawMarkdown": "Wow Crazy ! Thanks for sharing and Congratulations🎉", "votes": null }, { "id": "2273263", "postDate": "05/25/2023 05:06:28", "content": "Congratulations and thanks for sharing (include inference code)!
", "rawMarkdown": "Congratulations and thanks for sharing (include inference code)!", "votes": null }, { "id": "2273278", "postDate": "05/25/2023 05:14:10", "content": "Thanks and congrats for winning the Prize!
", "rawMarkdown": "Thanks and congrats for winning the Prize!", "votes": null }, { "id": "2274211", "postDate": "05/25/2023 17:44:29", "content": "@leonshangguan thanks for sharing with us your 10th place solution 🙌🏼🙌🏼
", "rawMarkdown": "leonshangguan thanks for sharing with us your 10th place solution 🙌🏼🙌🏼", "votes": null }, { "id": "2274227", "postDate": "05/25/2023 17:58:16", "content": "I'm really glad you could reach a gold medal, your ensemble notebook help us a lot, was one of the reasons why we reach a medal. Thanks for sharing your solution, we also thought Onnx could improve the inference time but sadly we couldn't make it work, thanks for sharing the inference code also, surely can help a lot of kagglers (including us) in future competitions.
", "rawMarkdown": "I'm really glad you could reach a gold medal, your ensemble notebook help us a lot, was one of the reasons why we reach a medal. Thanks for sharing your solution, we also thought Onnx could improve the inference time but sadly we couldn't make it work, thanks for sharing the inference code also, surely can help a lot of kagglers (including us) in future competitions.", "votes": null }, { "id": "2274290", "postDate": "05/25/2023 19:14:31", "content": "Thanks for sharing your solution @leonshangguan. The discussion title is misleading in a way because it suggests you somehow had ChatGPT work for you, but you actually intelligently used it to an extent and still worked hard on this as can be seen by 70 submissions being timed out comment of yours. I'm getting to learn so much from so many people here, it's amazing! God bless!
", "rawMarkdown": "Thanks for sharing your solution @leonshangguan. The discussion title is misleading in a way because it suggests you somehow had ChatGPT work for you, but you actually intelligently used it to an extent and still worked hard on this as can be seen by 70 submissions being timed out comment of yours. I'm getting to learn so much from so many people here, it's amazing! God bless!", "votes": null } ], "comments": [ { "id": 2273058, "author_name": "jionie", "author_url": "", "post_date": "05/25/2023 01:48:21", "content": "Congrats for your first solo gold! 💯
", "votes": null, "replies": [ { "id": 2273061, "author_name": "leonshangguan", "author_url": "", "post_date": "05/25/2023 01:50:03", "content": "Thank you!!!
", "votes": null, "replies": [] } ] }, { "id": 2273257, "author_name": "scipygaurav", "author_url": "", "post_date": "05/25/2023 05:03:41", "content": "Wow Crazy ! Thanks for sharing and Congratulations🎉
", "votes": null, "replies": [] }, { "id": 2273263, "author_name": "atsunorifujita", "author_url": "", "post_date": "05/25/2023 05:06:28", "content": "Congratulations and thanks for sharing (include inference code)!
", "votes": null, "replies": [ { "id": 2273278, "author_name": "leonshangguan", "author_url": "", "post_date": "05/25/2023 05:14:10", "content": "Thanks and congrats for winning the Prize!
", "votes": null, "replies": [] } ] }, { "id": 2274211, "author_name": "bilalwaseer", "author_url": "", "post_date": "05/25/2023 17:44:29", "content": "@leonshangguan thanks for sharing with us your 10th place solution 🙌🏼🙌🏼
", "votes": null, "replies": [] }, { "id": 2274227, "author_name": "maxdiazbattan", "author_url": "", "post_date": "05/25/2023 17:58:16", "content": "I'm really glad you could reach a gold medal, your ensemble notebook help us a lot, was one of the reasons why we reach a medal. Thanks for sharing your solution, we also thought Onnx could improve the inference time but sadly we couldn't make it work, thanks for sharing the inference code also, surely can help a lot of kagglers (including us) in future competitions.
", "votes": null, "replies": [] }, { "id": 2274290, "author_name": "hnooruddin", "author_url": "", "post_date": "05/25/2023 19:14:31", "content": "Thanks for sharing your solution @leonshangguan. The discussion title is misleading in a way because it suggests you somehow had ChatGPT work for you, but you actually intelligently used it to an extent and still worked hard on this as can be seen by 70 submissions being timed out comment of yours. I'm getting to learn so much from so many people here, it's amazing! God bless!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2273030": "#Training Data\n\nI use 2023 competition data only, but 3 of my models have pre-train weights on 2022 data.\n\n#Model Architecture\nI use SED architecture, almost the same as [this notebook](https://www.kaggle.com/code/kaerunantoka/birdclef2022-use-2nd-label-f0) with very minor modifications (e.g., emb_size, num_layers).\n\nI tried 8 backbones in total, including 3 large and 5 small: \nlarge backbones: efficientnetv2_m, eca_nfnet_l0, seresnext50\nsmall backbones: efficientnet_v0, mnasnet_100, spnasnet_100, mobilenetv2_100, resnet34\n\nAs seresnext50 and resnet34 perform not well based on my cv, so I dropped them in the final ensemble.\n\nI got the information of mnasnet_100, spnasnet_100, and mobilenetv2_100 by asking ChatGPT: `What are small models offered by timm` and ChatGPT replied with these models with the number of parameters, which is very helpful. \n\n#Code\nOne of my inference notebooks can be found [here](https://www.kaggle.com/code/leonshangguan/fork-of-fork-of-onnx-final-combine-bs4-b9daf6?scriptVersionId=130783179)\n\n#Training Strategy\n1. all models are trained on 5s clip.\n\n2. soft label: in the one-hot gt, I give the primary label with \"1\" and the secondary label with \"0.3\".\n\n3. data augmentation: add GaussianNoise, Background noise, PinkNoise, adjust Gain, use cut mix and mix up (very helpful).\n\n4. Progressive pretraining: the hardest part of this competition is to find a good set of parameters when applying melspectrum, especially the n_fft, hop_length, and n_mels, which affect the input image (melspectrum) size. I tried many different params, and surprisingly, I found that if you have a model trained on params set A, then you load the model as a pre-trained model (except the first and last layer), and then train it on params set B, the model will perform much better than directly training on params set B, and this is the key idea of my solution. All of my models were trained on many different parameters and finally finetuned on n_mels=128, n_fft=1024, and hop_length=320 to make it the same size, so that can be ensemble efficiently.\n\n5. loss function: BCEWithLogits + BCEFocal2WayLoss, same as previous years.\n\n6. cv strategy: I use 5fold stratified cross-validation and select the best 1 or 2 fold according to cv. \n\n7. ensemble strategy: each model has 1/2 fold for ensemble, and I have 6/7 models in total, with 6 different backbones as mentioned before. The reason I didn't use all folds is because of the computational limitation asked by the host.\n\n8. cpu acceleration: I use ThreadPoolExecutor as I have mentioned [here](https://www.kaggle.com/competitions/birdclef-2023/discussion/401587), and I also convert models to onnx which utilizes the model quantization technique and shows faster inference time.\n\n#Others \n1. more than 70 of my submissions encountered timeout error, and I therefore highly request the function that if more information can be released for the submission, for example, the running time @herbison . I suffered from it a lot, and many almost the same submissions sometimes pass but sometimes not, even with the identical code. \n\nI previously made a request regarding the submission time function, as mentioned [here](https://www.kaggle.com/competitions/uw-madison-gi-tract-image-segmentation/discussion/337343), when @ammarali32 and @jaafarmahmoud1 found 10 PM to 6 AM EST is a good time to make submissions. The reason behind this recommendation is that 10 PM EST is considered late for the American region, while it corresponds to 4 AM in Europe and 10 AM in Asia, which might be too early for many participants. Consequently, this particular timeframe becomes somewhat magical, as very few people tend to make submissions during this period. Therefore, if one chooses to make a submission during this time, they could potentially benefit from it.\n\n2. based on my experiments, I believe the model params have an impact on inference time when using CPU, i.e., with the same model architecture and the number of params, but different models, some may be fast some may not. @chrisqiu 's [post](https://www.kaggle.com/competitions/birdclef-2023/discussion/409331) verifies this situation. As the host is trying to build apps that can be deployed in the real world using the CPU, I think this issue is worth mentioning @stefankahl. \n\n3. I found that both my CV and lb vary based on the different folds, which indicates a good data split may have a great impact on the result, and some luck is needed when setting the random seed for the data split.\n\n\n#Acknowledgement\nI appreciate @stefankahl for hosting this amazing competition, @herbison for answering my questions regarding the submission timeout error, and thanks to the Kaggle community for sharing their insights, ideas, and thoughts. This is a wonderful learning experience and I hope everyone enjoys it!", "2273058": "Congrats for your first solo gold! 💯", "2273061": "Thank you!!!", "2273257": "Wow Crazy ! Thanks for sharing and Congratulations🎉", "2273263": "Congratulations and thanks for sharing (include inference code)!", "2273278": "Thanks and congrats for winning the Prize!", "2274211": "leonshangguan thanks for sharing with us your 10th place solution 🙌🏼🙌🏼", "2274227": "I'm really glad you could reach a gold medal, your ensemble notebook help us a lot, was one of the reasons why we reach a medal. Thanks for sharing your solution, we also thought Onnx could improve the inference time but sadly we couldn't make it work, thanks for sharing the inference code also, surely can help a lot of kagglers (including us) in future competitions.", "2274290": "Thanks for sharing your solution @leonshangguan. The discussion title is misleading in a way because it suggests you somehow had ChatGPT work for you, but you actually intelligently used it to an extent and still worked hard on this as can be seen by 70 submissions being timed out comment of yours. I'm getting to learn so much from so many people here, it's amazing! God bless!" }, "source": "meta" }