![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F15622327%2F880ab319638fe63c824473487dfcaad4%2Fmeme.png?generation=1718067598972288&alt=media\")
I mentioned earlier that this competition was disappointing for me. Now, we can see how disappointing it has been not only for me but for many others as well. Many participants have dropped from the silver and bronze ranks, while many others have risen to these ranks by simply copying and pasting a notebook. Please don't tell me about overfitting to the public leaderboard. This shows that β¦ . Congrats BirdCLEF!
\nUpdate:
\nWhenever I open a discussion, I never seek upvotes (my focus is on competitions not discussions). But here's the interesting part: every time I offer criticism, I seem to attract downvotes.
My advice to those disappointed by the results, that many others have shared before :
\nI have seen dozens of worse competitions than this one, and I'm not saying that because I'm 3rd.
\nThe only issue with this competition is that it was extremely hard to make progress, partly because we had no reliable validation set.
Thanks for your comment! I'm fully on board with your perspective, but I'd like to expand on two points for clarity:
\nMany individuals who engage in competitions are not solely focused on learning; they're also motivated by the opportunity to enhance their resumes. While they aim to improve their skills, they're also mindful of how their performance in the competition could bolster their chances of securing a job in the future.
A competition can significantly benefit from a well-designed validation set. Think of designing a good validation set like equipping oneself with a light before journeying into a dark jungle. Without it, navigating through the dense foliage becomes a chaotic and exhausting endeavor. However, with a well-designed validation set, individuals have a guiding light that illuminates their path, helping them efficiently reach their destination or goal.
I was so sure we would get at least silver, but now we didn't even get bronze. π
\n
same with COOLCAT team
", "rawMarkdown": "same with COOLCAT team", "votes": 2, "replies": [ { "id": 2865783, "postDate": "2024-06-11T01:32:51.007Z", "content": "and ours :-)
", "rawMarkdown": "and ours :-)", "votes": 1 }, { "id": 2865786, "postDate": "2024-06-11T01:35:41.267Z", "content": "Yeah you got hit the hardest tanxxx :( feels bad
", "rawMarkdown": "Yeah you got hit the hardest tanxxx :( feels bad", "votes": 1 }, { "id": 2865790, "postDate": "2024-06-11T01:44:45.630Z", "content": "It's fine, i've alreadly gotten used to it :)
", "rawMarkdown": "It's fine, i've alreadly gotten used to it :)", "votes": 2 } ] } ] }, { "id": 2865740, "postDate": "2024-06-11T00:57:00Z", "content": "I mentioned earlier that this competition was disappointing for me. Now, we can see how disappointing it has been not only for me but for many others as well. Many participants have dropped from the silver and bronze ranks, while many others have risen to these ranks by simply copying and pasting a notebook. Please don't tell me about overfitting to the public leaderboard. This shows that β¦ . Congrats BirdCLEF!
\nUpdate:
\nWhenever I open a discussion, I never seek upvotes (my focus is on competitions not discussions). But here's the interesting part: every time I offer criticism, I seem to attract downvotes.
It was for sure a very challenging competition, since we did not know what to optimize?! We were unable to set up any reliable local CV, I would be very interested if anyone was able to: ![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8913321%2Fdaa82880cd1660bebcfbc98e347c0918%2Ftestee.png?generation=1718069293546591&alt=media\")
It seems that everyone who forkmitted the public notebook did better than a lot of people with gold / silver public scores..
\nThen in the end, stability and randomness were also a great factor in this competition. Our best 0.7 public gave us a 0.63. Then that same model, by just duplicating the channels from 1 to 3 into our conv_next timm model gave us a public of 0.69, which we did not select but a 0.66 private that would led us to staying in silver.
", "rawMarkdown": "It was for sure a very challenging competition, since we did not know what to optimize?! We were unable to set up any reliable local CV, I would be very interested if anyone was able to: \n\nIt seems that everyone who forkmitted the public notebook did better than a lot of people with gold / silver public scores..\n\nThen in the end, stability and randomness were also a great factor in this competition. Our best 0.7 public gave us a 0.63. Then that same model, by just duplicating the channels from 1 to 3 into our conv_next timm model gave us a public of 0.69, which we did not select but a 0.66 private that would led us to staying in silver.\n\n", "votes": 6, "replies": [ { "id": 2865809, "postDate": "2024-06-11T02:21:20.083Z", "content": "What I can say instead of challenging is that they had a problem and wanted to see solutions and ideas. However, they ended up wasting the time and energy of many people for their own goals.
", "rawMarkdown": "What I can say instead of challenging is that they had a problem and wanted to see solutions and ideas. However, they ended up wasting the time and energy of many people for their own goals.", "votes": 1 } ] }, { "id": 2867684, "postDate": "2024-06-12T02:56:24.980Z", "content": "![\"As](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F10337191%2F33c1187e708e1e5c8bb213e6dbb218c0%2F1.png?generation=1718160961711110&alt=media\")
As I said, this shake is most likely caused by the different categories in the two test sets.π
", "rawMarkdown": "As I said, this shake is most likely caused by the different categories in the two test sets.π" }, { "id": 2867705, "postDate": "2024-06-12T03:29:05.813Z", "content": "Ok, so that's your theory. Are the private and public test data actually different like that? If so, the distribution of public and private is diff also, and that's not fair.
", "rawMarkdown": "Ok, so that's your theory. Are the private and public test data actually different like that? If so, the distribution of public and private is diff also, and that's not fair.", "replies": [ { "id": 2867739, "postDate": "2024-06-12T04:23:21.740Z", "content": "Indeed, from my experiments, the distributions of the public and private datasets are both similar to the unlabeled data. However, since I only performed domain adaptation on a subset of the classes in the unlabeled data (about 80 classes with confident/high probabilities from BirdNET), I was shaken from the golden position. In other words, I believe the purpose of this competition may be to classify the classes that are difficult for BirdNET to identify. This is my perspective, and I think it is very likely.
", "rawMarkdown": "Indeed, from my experiments, the distributions of the public and private datasets are both similar to the unlabeled data. However, since I only performed domain adaptation on a subset of the classes in the unlabeled data (about 80 classes with confident/high probabilities from BirdNET), I was shaken from the golden position. In other words, I believe the purpose of this competition may be to classify the classes that are difficult for BirdNET to identify. This is my perspective, and I think it is very likely.", "votes": 1 } ] } ] }, { "id": 2865782, "postDate": "2024-06-11T01:32:48.207Z", "content": "\n\nwhile many others have risen to these ranks by simply copying and pasting a notebook
\n
Which notebook was that?
", "rawMarkdown": "> while many others have risen to these ranks by simply copying and pasting a notebook\n\nWhich notebook was that?", "replies": [ { "id": 2865785, "postDate": "2024-06-11T01:34:27.760Z", "rawMarkdown": "", "votes": -1, "isDeleted": true }, { "id": 2865806, "postDate": "2024-06-11T02:17:01.197Z", "content": "I don't know as I never used others' notebook. We should ask 23rd place.
", "rawMarkdown": "I don't know as I never used others' notebook. We should ask 23rd place." }, { "id": 2865818, "postDate": "2024-06-11T02:41:38.117Z", "content": "I believe it was this one: https://www.kaggle.com/code/tc0000/birdclef-starter-notebook
\nPublic: 0.654307, Private: 0.649998
\nInterestingly, it was not the highest scoring public notebook on the public LB. That would be this one: https://www.kaggle.com/code/salmanahmedtamu/new-torch-jit-submission-nf
\nPublic: 0.658391, Private: 0.615320
So to do well in this competition, we should have submitted the 2nd best public notebook :P
\nIn all seriousness, I really enjoyed the challenge from the train/soundscape shift & I'm really looking forward to learning from these notebooks and also the top solutions.
", "rawMarkdown": "I believe it was this one: https://www.kaggle.com/code/tc0000/birdclef-starter-notebook\nPublic: 0.654307, Private: 0.649998\nInterestingly, it was not the highest scoring public notebook *on the public LB*. That would be this one: https://www.kaggle.com/code/salmanahmedtamu/new-torch-jit-submission-nf\nPublic: 0.658391, Private: 0.615320\n\nSo to do well in this competition, we should have submitted the 2nd best public notebook :P\n\nIn all seriousness, I really enjoyed the challenge from the train/soundscape shift & I'm really looking forward to learning from these notebooks and also the top solutions.", "votes": 5, "replies": [ { "id": 2865822, "postDate": "2024-06-11T02:49:42.293Z", "content": "Interesting! The person who published the first notebook placed 341st.
", "rawMarkdown": "Interesting! The person who published the first notebook placed 341st.", "votes": 1 }, { "id": 2865825, "postDate": "2024-06-11T02:53:08.437Z", "content": "I didn't look into the first notebook details, but I feel that even if there was another test dataset, we would still observe another shake-up. It's better to learn from others' results with caution and take away what makes sense.
", "rawMarkdown": "I didn't look into the first notebook details, but I feel that even if there was another test dataset, we would still observe another shake-up. It's better to learn from others' results with caution and take away what makes sense." }, { "id": 2866149, "postDate": "2024-06-11T06:54:41.073Z", "content": "I'm 342ndπ€.
", "rawMarkdown": "I'm 342ndπ€. ~~I should use his notebook~~" }, { "id": 2866161, "postDate": "2024-06-11T07:03:59.673Z", "content": "In the event of an exact score tie, the tiebreaker is the team which submitted earlier.
you can have a high rank if u submit that notebook earliest. :DD
", "rawMarkdown": "`In the event of an exact score tie, the tiebreaker is the team which submitted earlier. `\n\nyou can have a high rank if u submit that notebook earliest. :DD", "votes": 2 } ] } ] }, { "id": 2867682, "postDate": "2024-06-12T02:54:49.793Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2866225, "author_name": "Theo Viel", "author_url": "", "post_date": "2024-06-11T08:00:08.300000", "content": "My advice to those disappointed by the results, that many others have shared before :
\nI have seen dozens of worse competitions than this one, and I'm not saying that because I'm 3rd.
\nThe only issue with this competition is that it was extremely hard to make progress, partly because we had no reliable validation set.
Thanks for your comment! I'm fully on board with your perspective, but I'd like to expand on two points for clarity:
\nMany individuals who engage in competitions are not solely focused on learning; they're also motivated by the opportunity to enhance their resumes. While they aim to improve their skills, they're also mindful of how their performance in the competition could bolster their chances of securing a job in the future.
A competition can significantly benefit from a well-designed validation set. Think of designing a good validation set like equipping oneself with a light before journeying into a dark jungle. Without it, navigating through the dense foliage becomes a chaotic and exhausting endeavor. However, with a well-designed validation set, individuals have a guiding light that illuminates their path, helping them efficiently reach their destination or goal.
I was so sure we would get at least silver, but now we didn't even get bronze. π
\n
same with COOLCAT team
", "votes": 2, "replies": [ { "id": 2865783, "author_name": "Zhongkai Shangguan", "author_url": "", "post_date": "2024-06-11T01:32:51.007000", "content": "and ours :-)
", "votes": 1, "replies": [] }, { "id": 2865786, "author_name": "Hugo de Heer", "author_url": "", "post_date": "2024-06-11T01:35:41.267000", "content": "Yeah you got hit the hardest tanxxx :( feels bad
", "votes": 1, "replies": [] }, { "id": 2865790, "author_name": "tanxxx", "author_url": "", "post_date": "2024-06-11T01:44:45.630000", "content": "It's fine, i've alreadly gotten used to it :)
", "votes": 2, "replies": [] } ] } ] }, { "id": 2865784, "author_name": "Hugo de Heer", "author_url": "", "post_date": "2024-06-11T01:32:52.120000", "content": "It was for sure a very challenging competition, since we did not know what to optimize?! We were unable to set up any reliable local CV, I would be very interested if anyone was able to:
It seems that everyone who forkmitted the public notebook did better than a lot of people with gold / silver public scores..
\nThen in the end, stability and randomness were also a great factor in this competition. Our best 0.7 public gave us a 0.63. Then that same model, by just duplicating the channels from 1 to 3 into our conv_next timm model gave us a public of 0.69, which we did not select but a 0.66 private that would led us to staying in silver.
", "votes": 6, "replies": [ { "id": 2865809, "author_name": "Warrior", "author_url": "", "post_date": "2024-06-11T02:21:20.083000", "content": "What I can say instead of challenging is that they had a problem and wanted to see solutions and ideas. However, they ended up wasting the time and energy of many people for their own goals.
", "votes": 1, "replies": [] } ] }, { "id": 2867684, "author_name": "tanxxx", "author_url": "", "post_date": "2024-06-12T02:56:24.980000", "content": "
As I said, this shake is most likely caused by the different categories in the two test sets.π
", "votes": 0, "replies": [] }, { "id": 2867705, "author_name": "Levi", "author_url": "", "post_date": "2024-06-12T03:29:05.813000", "content": "Ok, so that's your theory. Are the private and public test data actually different like that? If so, the distribution of public and private is diff also, and that's not fair.
", "votes": 0, "replies": [ { "id": 2867739, "author_name": "tanxxx", "author_url": "", "post_date": "2024-06-12T04:23:21.740000", "content": "Indeed, from my experiments, the distributions of the public and private datasets are both similar to the unlabeled data. However, since I only performed domain adaptation on a subset of the classes in the unlabeled data (about 80 classes with confident/high probabilities from BirdNET), I was shaken from the golden position. In other words, I believe the purpose of this competition may be to classify the classes that are difficult for BirdNET to identify. This is my perspective, and I think it is very likely.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2865782, "author_name": "thacrobatheskis", "author_url": "", "post_date": "2024-06-11T01:32:48.207000", "content": "\n\nwhile many others have risen to these ranks by simply copying and pasting a notebook
\n
Which notebook was that?
", "votes": 0, "replies": [ { "id": 2865785, "author_name": "", "author_url": "", "post_date": "2024-06-11T01:34:27.760000", "content": "", "votes": -1, "replies": [] }, { "id": 2865806, "author_name": "Warrior", "author_url": "", "post_date": "2024-06-11T02:17:01.197000", "content": "I don't know as I never used others' notebook. We should ask 23rd place.
", "votes": 0, "replies": [] }, { "id": 2865818, "author_name": "thacrobatheskis", "author_url": "", "post_date": "2024-06-11T02:41:38.117000", "content": "I believe it was this one: https://www.kaggle.com/code/tc0000/birdclef-starter-notebook
\nPublic: 0.654307, Private: 0.649998
\nInterestingly, it was not the highest scoring public notebook on the public LB. That would be this one: https://www.kaggle.com/code/salmanahmedtamu/new-torch-jit-submission-nf
\nPublic: 0.658391, Private: 0.615320
So to do well in this competition, we should have submitted the 2nd best public notebook :P
\nIn all seriousness, I really enjoyed the challenge from the train/soundscape shift & I'm really looking forward to learning from these notebooks and also the top solutions.
", "votes": 5, "replies": [ { "id": 2865822, "author_name": "Warrior", "author_url": "", "post_date": "2024-06-11T02:49:42.293000", "content": "Interesting! The person who published the first notebook placed 341st.
", "votes": 1, "replies": [] }, { "id": 2865825, "author_name": "Warrior", "author_url": "", "post_date": "2024-06-11T02:53:08.437000", "content": "I didn't look into the first notebook details, but I feel that even if there was another test dataset, we would still observe another shake-up. It's better to learn from others' results with caution and take away what makes sense.
", "votes": 0, "replies": [] }, { "id": 2866149, "author_name": "SakuraYuyuko", "author_url": "", "post_date": "2024-06-11T06:54:41.073000", "content": "I'm 342ndπ€.
", "votes": 0, "replies": [] }, { "id": 2866161, "author_name": "Levi", "author_url": "", "post_date": "2024-06-11T07:03:59.673000", "content": "In the event of an exact score tie, the tiebreaker is the team which submitted earlier.
you can have a high rank if u submit that notebook earliest. :DD
", "votes": 2, "replies": [] } ] } ] }, { "id": 2867682, "author_name": "", "author_url": "", "post_date": "2024-06-12T02:54:49.793000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2866225": "My advice to those disappointed by the results, that many others have shared before :\n\n- Focus on the learnings. If you invested time on the competition then you gained experience coding and kaggling\n- Learn from others solutions, and understand the shortcomings of your solution. What did you miss to get a silver / bronze ? What is the one thing about the data that a lot of other participants leveraged that you did not think of ?\n- Luck (almost) always plays a part in Kaggle. Next competition you might move up 600 ranks, so keep pushing\n\nI have seen dozens of worse competitions than this one, and I'm not saying that because I'm 3rd. \nThe only issue with this competition is that it was extremely hard to make progress, partly because we had no reliable validation set. ", "2865746": "I was so sure we would get at least silver, but now we didn't even get bronze. π\n\n", "2865740": "I mentioned [earlier](https://www.kaggle.com/competitions/birdclef-2024/discussion/510838) that this competition was disappointing for me. Now, we can see how disappointing it has been not only for me but for many others as well. Many participants have dropped from the silver and bronze ranks, while many others have risen to these ranks by simply copying and pasting a notebook. Please don't tell me about overfitting to the public leaderboard. This shows that ... . Congrats BirdCLEF!\n\nUpdate:\nWhenever I open a discussion, I never seek upvotes (my focus is on competitions not discussions). But here's the interesting part: every time I offer criticism, I seem to attract downvotes. ", "2865784": "It was for sure a very challenging competition, since we did not know what to optimize?! We were unable to set up any reliable local CV, I would be very interested if anyone was able to: \n\nIt seems that everyone who forkmitted the public notebook did better than a lot of people with gold / silver public scores..\n\nThen in the end, stability and randomness were also a great factor in this competition. Our best 0.7 public gave us a 0.63. Then that same model, by just duplicating the channels from 1 to 3 into our conv_next timm model gave us a public of 0.69, which we did not select but a 0.66 private that would led us to staying in silver.\n\n", "2867684": "", "2865782": "> while many others have risen to these ranks by simply copying and pasting a notebook\n\nWhich notebook was that?", "2867682": "" } }