Hi to all!
\nI have a rule, I do not take \"seriously\" to those competitions where the public test set is less than 40% of the total test set, because at the end of the competition the density of leading positions is such that a shakeup is inevitable. And cross-validation will not help you choose the best notebook.
\nWith this competition I broke my rules, we'll see what happens.
But I have already done a lot of work that gave me a lot of knowledge.
\nNow I want to share the results of my research.
\nFor a start, I used public notebooks as a basis, but I made my own changes to each one.
\nI have already published my best single model ( https://www.kaggle.com/aikhmelnytskyy/resnet-wavenet-my-best-single-model-ensemble ).
\nI mentioned this notebook for a reason because I tried instead of resnet34 all the basic models of tf.keras.applications. Google colab helped me a lot in this, I calmly ran several divisions on TPU in parallel, without spending a limit on kaggle, here is a sample code: https://www.kaggle.com/aikhmelnytskyy/resnet-tpu-on-colab-and-kaggle/comments.
What can be said here with the increasing complexity of the model, its accuracy decreases.
\nResnet34, Resnet50, InceptionResNetV2 and DenseNet121 showed the best results. As you can see, the resnet family has shown the best results with this architecture. With the efficientnet family, the results were much worse.
\nI can say that my biggest disappointment was using the DenseNet121 model. The fact is that with it I got a public accuracy of 0.830, but in an ensemble with other models, it significantly worsened my result (If anyone knows what's going on here, please write in the comment!!!)
If you want to use efficientnet, I advise you to take the SED model from the public notebook as a basis (I also recommend looking at this notebook, 6th place on Cornell Birdcall Identification https://www.kaggle.com/hidehisaarai1213/birdcall-resnestsed-effnet-b0-ema-all-th04 ). By the way, I tried to use resnet34 with the SED model, but the result was worse than with efficientnet.
\nNow about my main mistake / flaw of all my models - overfitting. My cross-validation shows a validation accuracy greater than 0.87 (training accuracy +0.95), but the best public accuracy is 0.851. Of course, for 79% of the test data, everything can change, but that's too much.
\nI train models that showed commensurate training and validation accuracy, but their accuracy on the public dataset was much lower.
\nIn summary, the best results in this competition are shown by the models of the ResNet and efficientnet families in combination with other methods. And the determining factor here is not so much the model as the method of processing input data.
\nGood luck to all!
", "messages": [ { "id": "1176825", "postDate": "01/29/2021 20:20:30", "content": "Hi to all!
\nI have a rule, I do not take \"seriously\" to those competitions where the public test set is less than 40% of the total test set, because at the end of the competition the density of leading positions is such that a shakeup is inevitable. And cross-validation will not help you choose the best notebook.
\nWith this competition I broke my rules, we'll see what happens.
But I have already done a lot of work that gave me a lot of knowledge.
\nNow I want to share the results of my research.
\nFor a start, I used public notebooks as a basis, but I made my own changes to each one.
\nI have already published my best single model ( https://www.kaggle.com/aikhmelnytskyy/resnet-wavenet-my-best-single-model-ensemble ).
\nI mentioned this notebook for a reason because I tried instead of resnet34 all the basic models of tf.keras.applications. Google colab helped me a lot in this, I calmly ran several divisions on TPU in parallel, without spending a limit on kaggle, here is a sample code: https://www.kaggle.com/aikhmelnytskyy/resnet-tpu-on-colab-and-kaggle/comments.
What can be said here with the increasing complexity of the model, its accuracy decreases.
\nResnet34, Resnet50, InceptionResNetV2 and DenseNet121 showed the best results. As you can see, the resnet family has shown the best results with this architecture. With the efficientnet family, the results were much worse.
\nI can say that my biggest disappointment was using the DenseNet121 model. The fact is that with it I got a public accuracy of 0.830, but in an ensemble with other models, it significantly worsened my result (If anyone knows what's going on here, please write in the comment!!!)
If you want to use efficientnet, I advise you to take the SED model from the public notebook as a basis (I also recommend looking at this notebook, 6th place on Cornell Birdcall Identification https://www.kaggle.com/hidehisaarai1213/birdcall-resnestsed-effnet-b0-ema-all-th04 ). By the way, I tried to use resnet34 with the SED model, but the result was worse than with efficientnet.
\nNow about my main mistake / flaw of all my models - overfitting. My cross-validation shows a validation accuracy greater than 0.87 (training accuracy +0.95), but the best public accuracy is 0.851. Of course, for 79% of the test data, everything can change, but that's too much.
\nI train models that showed commensurate training and validation accuracy, but their accuracy on the public dataset was much lower.
\nIn summary, the best results in this competition are shown by the models of the ResNet and efficientnet families in combination with other methods. And the determining factor here is not so much the model as the method of processing input data.
\nGood luck to all!
", "rawMarkdown": "Hi to all!\nI have a rule, I do not take \"seriously\" to those competitions where the public test set is less than 40% of the total test set, because at the end of the competition the density of leading positions is such that a shakeup is inevitable. And cross-validation will not help you choose the best notebook.\nWith this competition I broke my rules, we'll see what happens. \n\nBut I have already done a lot of work that gave me a lot of knowledge.\nNow I want to share the results of my research.\nFor a start, I used public notebooks as a basis, but I made my own changes to each one.\nI have already published my best single model ( https://www.kaggle.com/aikhmelnytskyy/resnet-wavenet-my-best-single-model-ensemble ).\nI mentioned this notebook for a reason because I tried instead of resnet34 all the basic models of tf.keras.applications. Google colab helped me a lot in this, I calmly ran several divisions on TPU in parallel, without spending a limit on kaggle, here is a sample code: https://www.kaggle.com/aikhmelnytskyy/resnet-tpu-on-colab-and-kaggle/comments. \n\nWhat can be said here with the increasing complexity of the model, its accuracy decreases.\n\nResnet34, Resnet50, InceptionResNetV2 and DenseNet121 showed the best results. As you can see, the resnet family has shown the best results with this architecture. With the efficientnet family, the results were much worse.\nI can say that my biggest disappointment was using the DenseNet121 model. The fact is that with it I got a public accuracy of 0.830, but in an ensemble with other models, it significantly worsened my result (If anyone knows what's going on here, please write in the comment!!!)\n\nIf you want to use efficientnet, I advise you to take the SED model from the public notebook as a basis (I also recommend looking at this notebook, 6th place on Cornell Birdcall Identification https://www.kaggle.com/hidehisaarai1213/birdcall-resnestsed-effnet-b0-ema-all-th04 ). By the way, I tried to use resnet34 with the SED model, but the result was worse than with efficientnet.\n\nNow about my main mistake / flaw of all my models - overfitting. My cross-validation shows a validation accuracy greater than 0.87 (training accuracy +0.95), but the best public accuracy is 0.851. Of course, for 79% of the test data, everything can change, but that's too much.\n\nI train models that showed commensurate training and validation accuracy, but their accuracy on the public dataset was much lower.\n\nIn summary, the best results in this competition are shown by the models of the ResNet and efficientnet families in combination with other methods. And the determining factor here is not so much the model as the method of processing input data. \n\nGood luck to all!", "votes": null }, { "id": "1176957", "postDate": "01/29/2021 23:21:04", "content": "\n\nbecause at the end of the competition the density of leading positions is such that a shakeup is inevitable.
\n
I tend to believe that @selimsef would be pretty safe if the competition stopped today. Imho, shaky LB should not be the reason to skip this particular competition.
", "rawMarkdown": "> because at the end of the competition the density of leading positions is such that a shakeup is inevitable. \n\nI tend to believe that @selimsef would be pretty safe if the competition stopped today. Imho, shaky LB should not be the reason to skip this particular competition.", "votes": null }, { "id": "1177016", "postDate": "01/30/2021 01:12:05", "content": "Regarding, the effect of larger model, I notice performance improvement when migrating from efficient-net b0 to b3, and performance further increases when migrating to b4. ( I am using SED )
", "rawMarkdown": "Regarding, the effect of larger model, I notice performance improvement when migrating from efficient-net b0 to b3, and performance further increases when migrating to b4. ( I am using SED )", "votes": null }, { "id": "1177132", "postDate": "01/30/2021 04:51:14", "content": "yeah, 0973 seems unreachible for me)
", "rawMarkdown": "yeah, 0973 seems unreachible for me)", "votes": null }, { "id": "1177296", "postDate": "01/30/2021 07:37:17", "content": "When I wrote that I do not take such competitions seriously, I meant that I do not spend a lot of time on them because the result in the bronze and silver zones still depends in many cases on luck. I didn't say I missed them. but this is only my personal approach. good luck
", "rawMarkdown": "When I wrote that I do not take such competitions seriously, I meant that I do not spend a lot of time on them because the result in the bronze and silver zones still depends in many cases on luck. I didn't say I missed them. but this is only my personal approach. good luck", "votes": null }, { "id": "1177312", "postDate": "01/30/2021 07:50:36", "content": "I spent much more time on my model where efnet showed results on a public test set at 0.800. so I recommended using SED for efnet. However, in my experiments with SED, I did not notice any improvement.
", "rawMarkdown": "I spent much more time on my model where efnet showed results on a public test set at 0.800. so I recommended using SED for efnet. However, in my experiments with SED, I did not notice any improvement.", "votes": null }, { "id": "1177331", "postDate": "01/30/2021 07:57:51", "content": "By the way, you reminded me of the model that took 6th place on the Cornell Birdcall Identification https://www.kaggle.com/hidehisaarai1213/birdcall-resnestsed-effnet-b0-ema-all-th04. There is used b0
", "rawMarkdown": "By the way, you reminded me of the model that took 6th place on the Cornell Birdcall Identification https://www.kaggle.com/hidehisaarai1213/birdcall-resnestsed-effnet-b0-ema-all-th04. There is used b0", "votes": null }, { "id": "1177334", "postDate": "01/30/2021 08:01:27", "content": "Kupchanski I wouldn't be so sure, I've already seen competitions in which the first places went down. But I mostly meant the bronze and silver zone. For me, the Golden Zone and the other zones are different competitions.
", "rawMarkdown": "Kupchanski I wouldn't be so sure, I've already seen competitions in which the first places went down. But I mostly meant the bronze and silver zone. For me, the Golden Zone and the other zones are different competitions.", "votes": null }, { "id": "1180254", "postDate": "02/01/2021 07:27:07", "content": "Thank you for sharing lots of attempts!
\nI have also used SED models and trained by changing CNN models(ResNet, DenseNet, EfficientNet, …).
\nBut I did not notice any improvements at all. Maybe I have to change other models not SED models.
Thank you for your observations Andrij
", "rawMarkdown": "Thank you for your observations Andrij", "votes": null }, { "id": "1191963", "postDate": "02/08/2021 20:08:00", "content": "Thank you Chris. As I wrote, my idea with the combination of ResNet + Wavenet was a pure experiment, although it showed some of the best results on public models, but I \"stuck to the ceiling\" with it at 0.850. I was only able to improve it a bit by changing the \"preprocessing settings\". If possible It would be interesting to hear your ideas on how to improve it. In my model, I use large values for dropout, but still can't avoid overfitting.
", "rawMarkdown": "Thank you Chris. As I wrote, my idea with the combination of ResNet + Wavenet was a pure experiment, although it showed some of the best results on public models, but I \"stuck to the ceiling\" with it at 0.850. I was only able to improve it a bit by changing the \"preprocessing settings\". If possible It would be interesting to hear your ideas on how to improve it. In my model, I use large values for dropout, but still can't avoid overfitting.", "votes": null } ], "comments": [ { "id": 1176957, "author_name": "ilu000", "author_url": "", "post_date": "01/29/2021 23:21:04", "content": "\n\nbecause at the end of the competition the density of leading positions is such that a shakeup is inevitable.
\n
I tend to believe that @selimsef would be pretty safe if the competition stopped today. Imho, shaky LB should not be the reason to skip this particular competition.
", "votes": null, "replies": [ { "id": 1177132, "author_name": "kupchanski", "author_url": "", "post_date": "01/30/2021 04:51:14", "content": "yeah, 0973 seems unreachible for me)
", "votes": null, "replies": [] }, { "id": 1177296, "author_name": "aikhmelnytskyy", "author_url": "", "post_date": "01/30/2021 07:37:17", "content": "When I wrote that I do not take such competitions seriously, I meant that I do not spend a lot of time on them because the result in the bronze and silver zones still depends in many cases on luck. I didn't say I missed them. but this is only my personal approach. good luck
", "votes": null, "replies": [] }, { "id": 1177334, "author_name": "aikhmelnytskyy", "author_url": "", "post_date": "01/30/2021 08:01:27", "content": "Kupchanski I wouldn't be so sure, I've already seen competitions in which the first places went down. But I mostly meant the bronze and silver zone. For me, the Golden Zone and the other zones are different competitions.
", "votes": null, "replies": [] } ] }, { "id": 1177016, "author_name": "garfieldchh", "author_url": "", "post_date": "01/30/2021 01:12:05", "content": "Regarding, the effect of larger model, I notice performance improvement when migrating from efficient-net b0 to b3, and performance further increases when migrating to b4. ( I am using SED )
", "votes": null, "replies": [ { "id": 1177312, "author_name": "aikhmelnytskyy", "author_url": "", "post_date": "01/30/2021 07:50:36", "content": "I spent much more time on my model where efnet showed results on a public test set at 0.800. so I recommended using SED for efnet. However, in my experiments with SED, I did not notice any improvement.
", "votes": null, "replies": [] }, { "id": 1177331, "author_name": "aikhmelnytskyy", "author_url": "", "post_date": "01/30/2021 07:57:51", "content": "By the way, you reminded me of the model that took 6th place on the Cornell Birdcall Identification https://www.kaggle.com/hidehisaarai1213/birdcall-resnestsed-effnet-b0-ema-all-th04. There is used b0
", "votes": null, "replies": [] } ] }, { "id": 1180254, "author_name": "kbh0287", "author_url": "", "post_date": "02/01/2021 07:27:07", "content": "Thank you for sharing lots of attempts!
\nI have also used SED models and trained by changing CNN models(ResNet, DenseNet, EfficientNet, …).
\nBut I did not notice any improvements at all. Maybe I have to change other models not SED models.
Thank you for your observations Andrij
", "votes": null, "replies": [ { "id": 1191963, "author_name": "aikhmelnytskyy", "author_url": "", "post_date": "02/08/2021 20:08:00", "content": "Thank you Chris. As I wrote, my idea with the combination of ResNet + Wavenet was a pure experiment, although it showed some of the best results on public models, but I \"stuck to the ceiling\" with it at 0.850. I was only able to improve it a bit by changing the \"preprocessing settings\". If possible It would be interesting to hear your ideas on how to improve it. In my model, I use large values for dropout, but still can't avoid overfitting.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1176825": "Hi to all!\nI have a rule, I do not take \"seriously\" to those competitions where the public test set is less than 40% of the total test set, because at the end of the competition the density of leading positions is such that a shakeup is inevitable. And cross-validation will not help you choose the best notebook.\nWith this competition I broke my rules, we'll see what happens. \n\nBut I have already done a lot of work that gave me a lot of knowledge.\nNow I want to share the results of my research.\nFor a start, I used public notebooks as a basis, but I made my own changes to each one.\nI have already published my best single model ( https://www.kaggle.com/aikhmelnytskyy/resnet-wavenet-my-best-single-model-ensemble ).\nI mentioned this notebook for a reason because I tried instead of resnet34 all the basic models of tf.keras.applications. Google colab helped me a lot in this, I calmly ran several divisions on TPU in parallel, without spending a limit on kaggle, here is a sample code: https://www.kaggle.com/aikhmelnytskyy/resnet-tpu-on-colab-and-kaggle/comments. \n\nWhat can be said here with the increasing complexity of the model, its accuracy decreases.\n\nResnet34, Resnet50, InceptionResNetV2 and DenseNet121 showed the best results. As you can see, the resnet family has shown the best results with this architecture. With the efficientnet family, the results were much worse.\nI can say that my biggest disappointment was using the DenseNet121 model. The fact is that with it I got a public accuracy of 0.830, but in an ensemble with other models, it significantly worsened my result (If anyone knows what's going on here, please write in the comment!!!)\n\nIf you want to use efficientnet, I advise you to take the SED model from the public notebook as a basis (I also recommend looking at this notebook, 6th place on Cornell Birdcall Identification https://www.kaggle.com/hidehisaarai1213/birdcall-resnestsed-effnet-b0-ema-all-th04 ). By the way, I tried to use resnet34 with the SED model, but the result was worse than with efficientnet.\n\nNow about my main mistake / flaw of all my models - overfitting. My cross-validation shows a validation accuracy greater than 0.87 (training accuracy +0.95), but the best public accuracy is 0.851. Of course, for 79% of the test data, everything can change, but that's too much.\n\nI train models that showed commensurate training and validation accuracy, but their accuracy on the public dataset was much lower.\n\nIn summary, the best results in this competition are shown by the models of the ResNet and efficientnet families in combination with other methods. And the determining factor here is not so much the model as the method of processing input data. \n\nGood luck to all!", "1176957": "> because at the end of the competition the density of leading positions is such that a shakeup is inevitable. \n\nI tend to believe that @selimsef would be pretty safe if the competition stopped today. Imho, shaky LB should not be the reason to skip this particular competition.", "1177016": "Regarding, the effect of larger model, I notice performance improvement when migrating from efficient-net b0 to b3, and performance further increases when migrating to b4. ( I am using SED )", "1177132": "yeah, 0973 seems unreachible for me)", "1177296": "When I wrote that I do not take such competitions seriously, I meant that I do not spend a lot of time on them because the result in the bronze and silver zones still depends in many cases on luck. I didn't say I missed them. but this is only my personal approach. good luck", "1177312": "I spent much more time on my model where efnet showed results on a public test set at 0.800. so I recommended using SED for efnet. However, in my experiments with SED, I did not notice any improvement.", "1177331": "By the way, you reminded me of the model that took 6th place on the Cornell Birdcall Identification https://www.kaggle.com/hidehisaarai1213/birdcall-resnestsed-effnet-b0-ema-all-th04. There is used b0", "1177334": "Kupchanski I wouldn't be so sure, I've already seen competitions in which the first places went down. But I mostly meant the bronze and silver zone. For me, the Golden Zone and the other zones are different competitions.", "1180254": "Thank you for sharing lots of attempts!\n\nI have also used SED models and trained by changing CNN models(ResNet, DenseNet, EfficientNet, ...).\nBut I did not notice any improvements at all. Maybe I have to change other models not SED models.", "1190920": "Thank you for your observations Andrij", "1191963": "Thank you Chris. As I wrote, my idea with the combination of ResNet + Wavenet was a pure experiment, although it showed some of the best results on public models, but I \"stuck to the ceiling\" with it at 0.850. I was only able to improve it a bit by changing the \"preprocessing settings\". If possible It would be interesting to hear your ideas on how to improve it. In my model, I use large values for dropout, but still can't avoid overfitting." }, "source": "meta" }