Do we think we'll see a high proportion of shake-up surprises between public and private ranking in this competition? Anyone know some math that might suggest one way or another, dataset size, etc.?
\nAnd for individuals/teams, are those with a high number of submissions more prone to an unfavorable shakeup from the public to private ranking? Is the opposite true for a low number of submissions? Strategies on picking your best and final?
\nCome on Shake UP!!!!! fingers crossed!
\nBest of luck everyone!
", "messages": [ { "id": 2270766, "postDate": "2023-05-23T11:51:15.587Z", "content": "This is my first Birdclef competition, and for this reason, I don't know how much you must rely on your CV, but I saw many Kagglers sharing their best models with a very huge gap between CV and LB, like CV 0.91 - LB 0.80 or even worse.
", "rawMarkdown": "This is my first Birdclef competition, and for this reason, I don't know how much you must rely on your CV, but I saw many Kagglers sharing their best models with a very huge gap between CV and LB, like CV 0.91 - LB 0.80 or even worse. ", "votes": 2, "replies": [ { "id": 2270883, "postDate": "2023-05-23T13:26:58.067Z", "content": "My first Birdclef too. The CV seemed dependent on ratio of padded to total validation rows. You can adjust with the pad argument. My validation set changed size with different approaches, so I sampled a constant size for a hopefully stable benchmark. It seemed well correlated with the LB for me, but yeah is CV was .87-.88, and LB is .80.
", "rawMarkdown": "My first Birdclef too. The CV seemed dependent on ratio of padded to total validation rows. You can adjust with the pad argument. My validation set changed size with different approaches, so I sampled a constant size for a hopefully stable benchmark. It seemed well correlated with the LB for me, but yeah is CV was .87-.88, and LB is .80.", "votes": 2, "replies": [ { "id": 2270987, "postDate": "2023-05-23T15:01:16.823Z", "content": "In our case, CV is 0.836, and LB 0.81.
", "rawMarkdown": "In our case, CV is 0.836, and LB 0.81." } ] } ] }, { "id": 2270087, "postDate": "2023-05-23T01:49:21.607Z", "content": "Do we think we'll see a high proportion of shake-up surprises between public and private ranking in this competition? Anyone know some math that might suggest one way or another, dataset size, etc.?
\nAnd for individuals/teams, are those with a high number of submissions more prone to an unfavorable shakeup from the public to private ranking? Is the opposite true for a low number of submissions? Strategies on picking your best and final?
\nCome on Shake UP!!!!! fingers crossed!
\nBest of luck everyone!
", "rawMarkdown": "Do we think we'll see a high proportion of shake-up surprises between public and private ranking in this competition? Anyone know some math that might suggest one way or another, dataset size, etc.?\n\nAnd for individuals/teams, are those with a high number of submissions more prone to an unfavorable shakeup from the public to private ranking? Is the opposite true for a low number of submissions? Strategies on picking your best and final?\n\nCome on Shake **UP**!!!!! fingers crossed!\n\nBest of luck everyone!", "votes": 2 }, { "id": 2270597, "postDate": "2023-05-23T09:02:53.783Z", "content": "I believe that in this comp, public LB is our CV, I expect that the shakeup won't be very violent and that solely trusting your CV is a pretty risky idea.
", "rawMarkdown": "I believe that in this comp, public LB is our CV, I expect that the shakeup won't be very violent and that solely trusting your CV is a pretty risky idea. \n" }, { "id": 2270322, "postDate": "2023-05-23T05:53:09.650Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2270766, "author_name": "Maximiliano Diaz Battan", "author_url": "", "post_date": "2023-05-23T11:51:15.587000", "content": "This is my first Birdclef competition, and for this reason, I don't know how much you must rely on your CV, but I saw many Kagglers sharing their best models with a very huge gap between CV and LB, like CV 0.91 - LB 0.80 or even worse.
", "votes": 2, "replies": [ { "id": 2270883, "author_name": "Andy Atkinson", "author_url": "", "post_date": "2023-05-23T13:26:58.067000", "content": "My first Birdclef too. The CV seemed dependent on ratio of padded to total validation rows. You can adjust with the pad argument. My validation set changed size with different approaches, so I sampled a constant size for a hopefully stable benchmark. It seemed well correlated with the LB for me, but yeah is CV was .87-.88, and LB is .80.
", "votes": 2, "replies": [ { "id": 2270987, "author_name": "Maximiliano Diaz Battan", "author_url": "", "post_date": "2023-05-23T15:01:16.823000", "content": "In our case, CV is 0.836, and LB 0.81.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2270597, "author_name": "JEANMPIA", "author_url": "", "post_date": "2023-05-23T09:02:53.783000", "content": "I believe that in this comp, public LB is our CV, I expect that the shakeup won't be very violent and that solely trusting your CV is a pretty risky idea.
", "votes": 0, "replies": [] }, { "id": 2270322, "author_name": "", "author_url": "", "post_date": "2023-05-23T05:53:09.650000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2270766": "This is my first Birdclef competition, and for this reason, I don't know how much you must rely on your CV, but I saw many Kagglers sharing their best models with a very huge gap between CV and LB, like CV 0.91 - LB 0.80 or even worse. ", "2270087": "Do we think we'll see a high proportion of shake-up surprises between public and private ranking in this competition? Anyone know some math that might suggest one way or another, dataset size, etc.?\n\nAnd for individuals/teams, are those with a high number of submissions more prone to an unfavorable shakeup from the public to private ranking? Is the opposite true for a low number of submissions? Strategies on picking your best and final?\n\nCome on Shake **UP**!!!!! fingers crossed!\n\nBest of luck everyone!", "2270597": "I believe that in this comp, public LB is our CV, I expect that the shakeup won't be very violent and that solely trusting your CV is a pretty risky idea. \n", "2270322": "" } }