{ "id": 511498, "title": "big shaked by my notebook", "url": "/competitions/birdclef-2024/discussion/511498", "author_name": "", "post_date": "2024-06-11T00:24:37.172330600Z", "votes": 19, "comment_count": 20, "views": 0, "content": "

My public NOTEBOOK that I put out shook up quite a bit and was at the top of the list. I have a bit of mixed feelings.

", "messages": [ { "id": "2865724", "postDate": "06/11/2024 00:24:37", "content": "

My public NOTEBOOK that I put out shook up quite a bit and was at the top of the list. I have a bit of mixed feelings.

", "rawMarkdown": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8638407%2F3cd00634d5d6d168681e8f7b2759107a%2F2024-06-11%2010.37.12.png?generation=1718069853038774&alt=media)My public NOTEBOOK that I put out shook up quite a bit and was at the top of the list. I have a bit of mixed feelings.", "votes": null }, { "id": "2865734", "postDate": "06/11/2024 00:49:21", "content": "

yes

", "rawMarkdown": "yes", "votes": null }, { "id": "2865743", "postDate": "06/11/2024 00:58:08", "content": "

Yeah that is extremely unfortunate, I had considered being conservative and selecting your public notebook as well because I knew many would pick it. I opted not to because I thought I was able to beat it fairly reliably with an ensemble of smaller models I built but it seems I was incorrect, a fatal flaw haha

", "rawMarkdown": "Yeah that is extremely unfortunate, I had considered being conservative and selecting your public notebook as well because I knew many would pick it. I opted not to because I thought I was able to beat it fairly reliably with an ensemble of smaller models I built but it seems I was incorrect, a fatal flaw haha", "votes": null }, { "id": "2865749", "postDate": "06/11/2024 01:05:34", "content": "

Yeah, me too. And the score went down in the NOTEBOOK with richer AUGMENTATION. In the end I don't know what was the key in this competition haha

", "rawMarkdown": "Yeah, me too. And the score went down in the NOTEBOOK with richer AUGMENTATION. In the end I don't know what was the key in this competition haha", "votes": null }, { "id": "2865764", "postDate": "06/11/2024 01:15:02", "content": "

I think at the top it is great skill, in the middle and onward it is great luck hahaha unfortunate but we all learn!

", "rawMarkdown": "I think at the top it is great skill, in the middle and onward it is great luck hahaha unfortunate but we all learn!", "votes": null }, { "id": "2865787", "postDate": "06/11/2024 01:40:13", "content": "

We had a feeling that the more we tried, the better it gets on CV, but the worse on LB… heavy augmentations / pertaining / pseudo labeling (google model) / knowledge distillation (google model)/ species weighting all did not work.. Also, model stability seemed a real thing, training the same model again with different seed would matter a lot for some Timm encoders

", "rawMarkdown": "We had a feeling that the more we tried, the better it gets on CV, but the worse on LB... heavy augmentations / pertaining / pseudo labeling (google model) / knowledge distillation (google model)/ species weighting all did not work.. Also, model stability seemed a real thing, training the same model again with different seed would matter a lot for some Timm encoders", "votes": null }, { "id": "2865866", "postDate": "06/11/2024 03:35:30", "content": "

It's quite disappointing that such high scores could be achieved just by using the public notebooks.
\nThe leaderboard shows a score of \"0.649998\" ranging from 32nd to 245th place. Can someone please clarify up to which rank is awarded a bronze medal and which rank starts to receive a silver medal? Could someone clarify the criteria for awarding medals in this scenario?

", "rawMarkdown": "It's quite disappointing that such high scores could be achieved just by using the public notebooks.\nThe leaderboard shows a score of \"0.649998\" ranging from 32nd to 245th place. Can someone please clarify up to which rank is awarded a bronze medal and which rank starts to receive a silver medal? Could someone clarify the criteria for awarding medals in this scenario?", "votes": null }, { "id": "2865877", "postDate": "06/11/2024 03:45:21", "content": "

That is my concern as well. However, it might be better to write about it in the host's discussion than here!

", "rawMarkdown": "That is my concern as well. However, it might be better to write about it in the host's discussion than here!", "votes": null }, { "id": "2866025", "postDate": "06/11/2024 05:45:44", "content": "

I feel little bit sad because I tried lots of different argumentation and ran hundreds of experiment, but in the end the result seems a bit \"random\" (from my understanding, may be it is limited). Some argumentation seem to work but the private LB is lower than the model using less argumentation (the one with gassian noise, h-flip, xy_mask work for me the best). It is very unfortunate and sad, but at least we all learned something. 😉

", "rawMarkdown": "I feel little bit sad because I tried lots of different argumentation and ran hundreds of experiment, but in the end the result seems a bit \"random\" (from my understanding, may be it is limited). Some argumentation seem to work but the private LB is lower than the model using less argumentation (the one with gassian noise, h-flip, xy_mask work for me the best). It is very unfortunate and sad, but at least we all learned something. 😉", "votes": null }, { "id": "2866134", "postDate": "06/11/2024 06:47:15", "content": "

omg, there are 213 teams that use the author's notebook and create this shake. Never seen it before

", "rawMarkdown": "omg, there are 213 teams that use the author's notebook and create this shake. Never seen it before", "votes": null }, { "id": "2866627", "postDate": "06/11/2024 12:20:15", "content": "

@tc0000 I think Kaggle staff should make some rules; there should be a constrant to make some changes if you fork a notebok othervise cannot submit.

", "rawMarkdown": "tc0000 I think Kaggle staff should make some rules; there should be a constrant to make some changes if you fork a notebok othervise cannot submit.", "votes": null }, { "id": "2866789", "postDate": "06/11/2024 14:11:10", "content": "

It is safe to say that there won't be a write up posts for ranks 32 - 245th on private LB then.

", "rawMarkdown": "It is safe to say that there won't be a write up posts for ranks 32 - 245th on private LB then.", "votes": null }, { "id": "2867225", "postDate": "06/11/2024 17:59:14", "content": "

In every competition setting up a validation that represents the train / test difference is key. Here you train on xeno-canto data but infer on PAM recordings from a specific region. Have you thought about how to replicate this shift for validation?

", "rawMarkdown": "In every competition setting up a validation that represents the train / test difference is key. Here you train on xeno-canto data but infer on PAM recordings from a specific region. Have you thought about how to replicate this shift for validation?", "votes": null }, { "id": "2867566", "postDate": "06/11/2024 23:48:11", "content": "

Yes, there could be special rules, such as submissions can be made but no medals will be awarded.

", "rawMarkdown": "Yes, there could be special rules, such as submissions can be made but no medals will be awarded.", "votes": null }, { "id": "2867570", "postDate": "06/11/2024 23:55:17", "content": "

Without complex modeling, I did not consider PAM because the LBvsPBs of past competitions seemed to correlate rather well. (It should be done, that's for sure.)
\nIn fact, in my experiments, the data augmentation complexity was SHAKE to minimize the impact of domain shifts, such as regional or device, and the simple architecture was robust.

", "rawMarkdown": "Without complex modeling, I did not consider PAM because the LBvsPBs of past competitions seemed to correlate rather well. (It should be done, that's for sure.)\nIn fact, in my experiments, the data augmentation complexity was SHAKE to minimize the impact of domain shifts, such as regional or device, and the simple architecture was robust.", "votes": null }, { "id": "2867573", "postDate": "06/12/2024 00:01:46", "content": "

I have had difficulty with the use of augmentation in my experiments. In this experiment, I think augur surface-tation will be used for domain shifts, etc. At least in the CV and LB I used in my experiment, I felt that if I complicate the augur surface-tation, the result would not be very good.
\nI also felt that further elaboration would not be cosmetic, so I had to withdraw about a month ago.

", "rawMarkdown": "I have had difficulty with the use of augmentation in my experiments. In this experiment, I think augur surface-tation will be used for domain shifts, etc. At least in the CV and LB I used in my experiment, I felt that if I complicate the augur surface-tation, the result would not be very good.\nI also felt that further elaboration would not be cosmetic, so I had to withdraw about a month ago.", "votes": null }, { "id": "2868091", "postDate": "06/12/2024 07:23:23", "content": "

There are many entries tied for 32nd place. In such cases, are medals awarded based on the order displayed on the leaderboard (perhaps based on who submitted first)? I'm curious about how this has been handled in past competitions when there were many ties.

", "rawMarkdown": "There are many entries tied for 32nd place. In such cases, are medals awarded based on the order displayed on the leaderboard (perhaps based on who submitted first)? I'm curious about how this has been handled in past competitions when there were many ties.", "votes": null }, { "id": "2868137", "postDate": "06/12/2024 07:53:12", "content": "

Basically, it does not take into account which NOTEBOOK you scored in. These massive SHAKES happen more often than not, so…

\n

It seems that the rankings have already been finalized.

", "rawMarkdown": "Basically, it does not take into account which NOTEBOOK you scored in. These massive SHAKES happen more often than not, so...\n\nIt seems that the rankings have already been finalized.", "votes": null }, { "id": "2868488", "postDate": "06/12/2024 12:10:05", "content": "

Thank you, it seems that the rankings have already been finalized.
\nI was purely curious about how medals are awarded and rankings are determined when scores are tied. I suppose rankings might be decided based on the order of submissions or something, but now that the rankings are already set, I'm a bit unclear about how they are determined.

", "rawMarkdown": "Thank you, it seems that the rankings have already been finalized.\nI was purely curious about how medals are awarded and rankings are determined when scores are tied. I suppose rankings might be decided based on the order of submissions or something, but now that the rankings are already set, I'm a bit unclear about how they are determined.", "votes": null }, { "id": "2870295", "postDate": "06/13/2024 13:53:55", "content": "

It can be better

", "rawMarkdown": "It can be better", "votes": null }, { "id": "2871415", "postDate": "06/14/2024 07:31:17", "content": "

What’s worse? I have seen social media posts about users bragging their silver/bronze medal they gained from this competition, with the 0.649998 score, when it is obvious that the score was achieved by forking and submitting YOUR notebook with ZERO changes 🤦‍♂️🤦‍♂️

", "rawMarkdown": "What’s worse? I have seen social media posts about users bragging their silver/bronze medal they gained from this competition, with the 0.649998 score, when it is obvious that the score was achieved by forking and submitting YOUR notebook with ZERO changes 🤦‍♂️🤦‍♂️", "votes": null } ], "comments": [ { "id": 2865734, "author_name": "kristofasandor", "author_url": "", "post_date": "06/11/2024 00:49:21", "content": "

yes

", "votes": null, "replies": [] }, { "id": 2865743, "author_name": "cody11null", "author_url": "", "post_date": "06/11/2024 00:58:08", "content": "

Yeah that is extremely unfortunate, I had considered being conservative and selecting your public notebook as well because I knew many would pick it. I opted not to because I thought I was able to beat it fairly reliably with an ensemble of smaller models I built but it seems I was incorrect, a fatal flaw haha

", "votes": null, "replies": [ { "id": 2865749, "author_name": "tc0000", "author_url": "", "post_date": "06/11/2024 01:05:34", "content": "

Yeah, me too. And the score went down in the NOTEBOOK with richer AUGMENTATION. In the end I don't know what was the key in this competition haha

", "votes": null, "replies": [ { "id": 2865764, "author_name": "cody11null", "author_url": "", "post_date": "06/11/2024 01:15:02", "content": "

I think at the top it is great skill, in the middle and onward it is great luck hahaha unfortunate but we all learn!

", "votes": null, "replies": [] }, { "id": 2865787, "author_name": "hugodeheer", "author_url": "", "post_date": "06/11/2024 01:40:13", "content": "

We had a feeling that the more we tried, the better it gets on CV, but the worse on LB… heavy augmentations / pertaining / pseudo labeling (google model) / knowledge distillation (google model)/ species weighting all did not work.. Also, model stability seemed a real thing, training the same model again with different seed would matter a lot for some Timm encoders

", "votes": null, "replies": [ { "id": 2867225, "author_name": "christofhenkel", "author_url": "", "post_date": "06/11/2024 17:59:14", "content": "

In every competition setting up a validation that represents the train / test difference is key. Here you train on xeno-canto data but infer on PAM recordings from a specific region. Have you thought about how to replicate this shift for validation?

", "votes": null, "replies": [ { "id": 2867570, "author_name": "tc0000", "author_url": "", "post_date": "06/11/2024 23:55:17", "content": "

Without complex modeling, I did not consider PAM because the LBvsPBs of past competitions seemed to correlate rather well. (It should be done, that's for sure.)
\nIn fact, in my experiments, the data augmentation complexity was SHAKE to minimize the impact of domain shifts, such as regional or device, and the simple architecture was robust.

", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 2865866, "author_name": "kmatsu01", "author_url": "", "post_date": "06/11/2024 03:35:30", "content": "

It's quite disappointing that such high scores could be achieved just by using the public notebooks.
\nThe leaderboard shows a score of \"0.649998\" ranging from 32nd to 245th place. Can someone please clarify up to which rank is awarded a bronze medal and which rank starts to receive a silver medal? Could someone clarify the criteria for awarding medals in this scenario?

", "votes": null, "replies": [ { "id": 2865877, "author_name": "tc0000", "author_url": "", "post_date": "06/11/2024 03:45:21", "content": "

That is my concern as well. However, it might be better to write about it in the host's discussion than here!

", "votes": null, "replies": [] }, { "id": 2866134, "author_name": "levinguyen02", "author_url": "", "post_date": "06/11/2024 06:47:15", "content": "

omg, there are 213 teams that use the author's notebook and create this shake. Never seen it before

", "votes": null, "replies": [ { "id": 2866789, "author_name": "desalegngeb", "author_url": "", "post_date": "06/11/2024 14:11:10", "content": "

It is safe to say that there won't be a write up posts for ranks 32 - 245th on private LB then.

", "votes": null, "replies": [ { "id": 2868091, "author_name": "kmatsu01", "author_url": "", "post_date": "06/12/2024 07:23:23", "content": "

There are many entries tied for 32nd place. In such cases, are medals awarded based on the order displayed on the leaderboard (perhaps based on who submitted first)? I'm curious about how this has been handled in past competitions when there were many ties.

", "votes": null, "replies": [ { "id": 2868137, "author_name": "tc0000", "author_url": "", "post_date": "06/12/2024 07:53:12", "content": "

Basically, it does not take into account which NOTEBOOK you scored in. These massive SHAKES happen more often than not, so…

\n

It seems that the rankings have already been finalized.

", "votes": null, "replies": [ { "id": 2868488, "author_name": "kmatsu01", "author_url": "", "post_date": "06/12/2024 12:10:05", "content": "

Thank you, it seems that the rankings have already been finalized.
\nI was purely curious about how medals are awarded and rankings are determined when scores are tied. I suppose rankings might be decided based on the order of submissions or something, but now that the rankings are already set, I'm a bit unclear about how they are determined.

", "votes": null, "replies": [] } ] } ] } ] } ] } ] }, { "id": 2866025, "author_name": "sakurayuyuko", "author_url": "", "post_date": "06/11/2024 05:45:44", "content": "

I feel little bit sad because I tried lots of different argumentation and ran hundreds of experiment, but in the end the result seems a bit \"random\" (from my understanding, may be it is limited). Some argumentation seem to work but the private LB is lower than the model using less argumentation (the one with gassian noise, h-flip, xy_mask work for me the best). It is very unfortunate and sad, but at least we all learned something. 😉

", "votes": null, "replies": [ { "id": 2867573, "author_name": "tc0000", "author_url": "", "post_date": "06/12/2024 00:01:46", "content": "

I have had difficulty with the use of augmentation in my experiments. In this experiment, I think augur surface-tation will be used for domain shifts, etc. At least in the CV and LB I used in my experiment, I felt that if I complicate the augur surface-tation, the result would not be very good.
\nI also felt that further elaboration would not be cosmetic, so I had to withdraw about a month ago.

", "votes": null, "replies": [] } ] }, { "id": 2866627, "author_name": "vipin20", "author_url": "", "post_date": "06/11/2024 12:20:15", "content": "

@tc0000 I think Kaggle staff should make some rules; there should be a constrant to make some changes if you fork a notebok othervise cannot submit.

", "votes": null, "replies": [ { "id": 2867566, "author_name": "tc0000", "author_url": "", "post_date": "06/11/2024 23:48:11", "content": "

Yes, there could be special rules, such as submissions can be made but no medals will be awarded.

", "votes": null, "replies": [] } ] }, { "id": 2870295, "author_name": "tejaswinikapasiya", "author_url": "", "post_date": "06/13/2024 13:53:55", "content": "

It can be better

", "votes": null, "replies": [] }, { "id": 2871415, "author_name": "yeoyunsianggeremie", "author_url": "", "post_date": "06/14/2024 07:31:17", "content": "

What’s worse? I have seen social media posts about users bragging their silver/bronze medal they gained from this competition, with the 0.649998 score, when it is obvious that the score was achieved by forking and submitting YOUR notebook with ZERO changes 🤦‍♂️🤦‍♂️

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2865724": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8638407%2F3cd00634d5d6d168681e8f7b2759107a%2F2024-06-11%2010.37.12.png?generation=1718069853038774&alt=media)My public NOTEBOOK that I put out shook up quite a bit and was at the top of the list. I have a bit of mixed feelings.", "2865734": "yes", "2865743": "Yeah that is extremely unfortunate, I had considered being conservative and selecting your public notebook as well because I knew many would pick it. I opted not to because I thought I was able to beat it fairly reliably with an ensemble of smaller models I built but it seems I was incorrect, a fatal flaw haha", "2865749": "Yeah, me too. And the score went down in the NOTEBOOK with richer AUGMENTATION. In the end I don't know what was the key in this competition haha", "2865764": "I think at the top it is great skill, in the middle and onward it is great luck hahaha unfortunate but we all learn!", "2865787": "We had a feeling that the more we tried, the better it gets on CV, but the worse on LB... heavy augmentations / pertaining / pseudo labeling (google model) / knowledge distillation (google model)/ species weighting all did not work.. Also, model stability seemed a real thing, training the same model again with different seed would matter a lot for some Timm encoders", "2865866": "It's quite disappointing that such high scores could be achieved just by using the public notebooks.\nThe leaderboard shows a score of \"0.649998\" ranging from 32nd to 245th place. Can someone please clarify up to which rank is awarded a bronze medal and which rank starts to receive a silver medal? Could someone clarify the criteria for awarding medals in this scenario?", "2865877": "That is my concern as well. However, it might be better to write about it in the host's discussion than here!", "2866025": "I feel little bit sad because I tried lots of different argumentation and ran hundreds of experiment, but in the end the result seems a bit \"random\" (from my understanding, may be it is limited). Some argumentation seem to work but the private LB is lower than the model using less argumentation (the one with gassian noise, h-flip, xy_mask work for me the best). It is very unfortunate and sad, but at least we all learned something. 😉", "2866134": "omg, there are 213 teams that use the author's notebook and create this shake. Never seen it before", "2866627": "tc0000 I think Kaggle staff should make some rules; there should be a constrant to make some changes if you fork a notebok othervise cannot submit.", "2866789": "It is safe to say that there won't be a write up posts for ranks 32 - 245th on private LB then.", "2867225": "In every competition setting up a validation that represents the train / test difference is key. Here you train on xeno-canto data but infer on PAM recordings from a specific region. Have you thought about how to replicate this shift for validation?", "2867566": "Yes, there could be special rules, such as submissions can be made but no medals will be awarded.", "2867570": "Without complex modeling, I did not consider PAM because the LBvsPBs of past competitions seemed to correlate rather well. (It should be done, that's for sure.)\nIn fact, in my experiments, the data augmentation complexity was SHAKE to minimize the impact of domain shifts, such as regional or device, and the simple architecture was robust.", "2867573": "I have had difficulty with the use of augmentation in my experiments. In this experiment, I think augur surface-tation will be used for domain shifts, etc. At least in the CV and LB I used in my experiment, I felt that if I complicate the augur surface-tation, the result would not be very good.\nI also felt that further elaboration would not be cosmetic, so I had to withdraw about a month ago.", "2868091": "There are many entries tied for 32nd place. In such cases, are medals awarded based on the order displayed on the leaderboard (perhaps based on who submitted first)? I'm curious about how this has been handled in past competitions when there were many ties.", "2868137": "Basically, it does not take into account which NOTEBOOK you scored in. These massive SHAKES happen more often than not, so...\n\nIt seems that the rankings have already been finalized.", "2868488": "Thank you, it seems that the rankings have already been finalized.\nI was purely curious about how medals are awarded and rankings are determined when scores are tied. I suppose rankings might be decided based on the order of submissions or something, but now that the rankings are already set, I'm a bit unclear about how they are determined.", "2870295": "It can be better", "2871415": "What’s worse? I have seen social media posts about users bragging their silver/bronze medal they gained from this competition, with the 0.649998 score, when it is obvious that the score was achieved by forking and submitting YOUR notebook with ZERO changes 🤦‍♂️🤦‍♂️" }, "source": "meta" }