Gold zone score is above 0.87.
\nI don't think it can be achived by participant split, at leaset for training set.
Does private test set have many same participants with training set?
\nOr, is there any other reason?
Gold zone score is above 0.87.
\nI don't think it can be achived by participant split, at leaset for training set.
Does private test set have many same participants with training set?
\nOr, is there any other reason?
I have the same doubt. It seems for every model, the accuracy has increased for the private lb. Even the models that didn't perform well at all on public LB.
", "rawMarkdown": "I have the same doubt. It seems for every model, the accuracy has increased for the private lb. Even the models that didn't perform well at all on public LB.", "votes": null }, { "id": "2242228", "postDate": "05/02/2023 06:11:10", "content": "I agree it is very weird. The public LB is similar to local CV score but the private LB is not.
", "rawMarkdown": "I agree it is very weird. The public LB is similar to local CV score but the private LB is not.", "votes": null }, { "id": "2242333", "postDate": "05/02/2023 07:35:57", "content": "They were very similar to my private CV scores, 0.01-0.02 higher than my local scores. I was surprised my public LB scores were so low.
", "rawMarkdown": "They were very similar to my private CV scores, 0.01-0.02 higher than my local scores. I was surprised my public LB scores were so low.", "votes": null }, { "id": "2242430", "postDate": "05/02/2023 09:16:42", "content": "@sohier @ashleychow
\nCan you follow up on that ?
On my local validation setup, I have only one patient with CV > 0.87 (id=26734)
\nThe private LB scores do not look right.
\nEither there is train / test overlap or the private test is surprisingly not noisy.
For reference, using a single model:
\nThanks for sharing Theo. Also congrats for your result!
\nMay I Ask how you split training data? I guess it's split by participant, but just making sure:)
", "rawMarkdown": "Thanks for sharing Theo. Also congrats for your result!\n\nMay I Ask how you split training data? I guess it's split by participant, but just making sure:)", "votes": null }, { "id": "2242481", "postDate": "05/02/2023 10:12:38", "content": "I think they used non-overlapping participant on the public, and the overlapping on the private.
\nMy single model CV with participant split:~0.80, public LB:~0.80
\nrandom split CV:~0.88(not verified), private LB:~0.88
random split CV is not verified but very likely(I did not check the random split CV after I crossed 0.87)
", "rawMarkdown": "I think they used non-overlapping participant on the public, and the overlapping on the private.\n\nMy single model CV with participant split:~0.80, public LB:~0.80\nrandom split CV:~0.88(not verified), private LB:~0.88\n\nrandom split CV is not verified but very likely(I did not check the random split CV after I crossed 0.87)", "votes": null }, { "id": "2242527", "postDate": "05/02/2023 10:40:22", "content": "Thanks and congratz to you as well !
\nI split by participant yes (groupkfold with k=4), as recommended by the hosts.
", "rawMarkdown": "Thanks and congratz to you as well !\n\nI split by participant yes (groupkfold with k=4), as recommended by the hosts.", "votes": null }, { "id": "2242536", "postDate": "05/02/2023 10:46:58", "content": "That is also my guess. My CV scores perfectly align in both cases, too.
", "rawMarkdown": "That is also my guess. My CV scores perfectly align in both cases, too.", "votes": null }, { "id": "2242637", "postDate": "05/02/2023 12:16:59", "content": "@hoyso48 Thanks for sharing, congrats for the solo winning! Looking forward your solution by the way;)
", "rawMarkdown": "hoyso48 Thanks for sharing, congrats for the solo winning! Looking forward your solution by the way;)", "votes": null }, { "id": "2242671", "postDate": "05/02/2023 12:43:58", "content": "private set is cleaned data?
", "rawMarkdown": "private set is cleaned data?", "votes": null }, { "id": "2243140", "postDate": "05/02/2023 17:47:56", "content": "I tried overfitting a model on the training data, it performed worse.
\nSo I don't think there's duplicates between train and test. There can still be signer overlap though.
haha, i'm trying the same experiment now too.
\nIf the private test data has signer (i.e. participant) overlap (but not duplicate videos), I wonder how we could design our models differently to take advantage of this?
", "rawMarkdown": "haha, i'm trying the same experiment now too.\n\nIf the private test data has signer (i.e. participant) overlap (but not duplicate videos), I wonder how we could design our models differently to take advantage of this?", "votes": null }, { "id": "2243176", "postDate": "05/02/2023 18:04:22", "content": "Does each participant only sign each word once? If so, perhaps we can classify the participant then remove any words they already signed in train data.
", "rawMarkdown": "Does each participant only sign each word once? If so, perhaps we can classify the participant then remove any words they already signed in train data.", "votes": null }, { "id": "2243351", "postDate": "05/02/2023 20:55:08", "content": "Now that the competition is over I can disclose that we split the dataset by signer with no overlap between the train, public, and private sets. There were only a handful of signers in the public set so we weren't too surprised to see that the public scores are somewhat idiosyncratic.
\nSimilarly, we expected to see a meaningful degree of variation in the difficulty of parsing videos from each signer. You can see some of that variation in the train set landmarks. Looking at the raw video reveals additional sources of difference, like lighting quality or distance from the camera.
\nOverall I am not concerned.
", "rawMarkdown": "Now that the competition is over I can disclose that we split the dataset by signer with no overlap between the train, public, and private sets. There were only a handful of signers in the public set so we weren't too surprised to see that the public scores are somewhat idiosyncratic. \n\nSimilarly, we expected to see a meaningful degree of variation in the difficulty of parsing videos from each signer. You can see some of that variation in the train set landmarks. Looking at the raw video reveals additional sources of difference, like lighting quality or distance from the camera.\n\nOverall I am not concerned.", "votes": null }, { "id": "2243393", "postDate": "05/02/2023 21:50:40", "content": "Good to know, thanks Sohier !
", "rawMarkdown": "Good to know, thanks Sohier !", "votes": null }, { "id": "2243422", "postDate": "05/02/2023 22:30:30", "content": "Thanks @sohier for the clarification, good to know what really happened!
", "rawMarkdown": "Thanks @sohier for the clarification, good to know what really happened!", "votes": null }, { "id": "2243430", "postDate": "05/02/2023 22:35:17", "content": "I also trained my submission for 300 epochs instead of 120 and submitted. It's LB performed worse.
", "rawMarkdown": "I also trained my submission for 300 epochs instead of 120 and submitted. It's LB performed worse.", "votes": null } ], "comments": [ { "id": 2242189, "author_name": "ahsuna123", "author_url": "", "post_date": "05/02/2023 05:16:40", "content": "I have the same doubt. It seems for every model, the accuracy has increased for the private lb. Even the models that didn't perform well at all on public LB.
", "votes": null, "replies": [] }, { "id": 2242228, "author_name": "cdeotte", "author_url": "", "post_date": "05/02/2023 06:11:10", "content": "I agree it is very weird. The public LB is similar to local CV score but the private LB is not.
", "votes": null, "replies": [] }, { "id": 2242333, "author_name": "joejeo1", "author_url": "", "post_date": "05/02/2023 07:35:57", "content": "They were very similar to my private CV scores, 0.01-0.02 higher than my local scores. I was surprised my public LB scores were so low.
", "votes": null, "replies": [] }, { "id": 2242430, "author_name": "theoviel", "author_url": "", "post_date": "05/02/2023 09:16:42", "content": "@sohier @ashleychow
\nCan you follow up on that ?
On my local validation setup, I have only one patient with CV > 0.87 (id=26734)
\nThe private LB scores do not look right.
\nEither there is train / test overlap or the private test is surprisingly not noisy.
For reference, using a single model:
\nThanks for sharing Theo. Also congrats for your result!
\nMay I Ask how you split training data? I guess it's split by participant, but just making sure:)
", "votes": null, "replies": [ { "id": 2242527, "author_name": "theoviel", "author_url": "", "post_date": "05/02/2023 10:40:22", "content": "Thanks and congratz to you as well !
\nI split by participant yes (groupkfold with k=4), as recommended by the hosts.
", "votes": null, "replies": [] } ] } ] }, { "id": 2242481, "author_name": "hoyso48", "author_url": "", "post_date": "05/02/2023 10:12:38", "content": "I think they used non-overlapping participant on the public, and the overlapping on the private.
\nMy single model CV with participant split:~0.80, public LB:~0.80
\nrandom split CV:~0.88(not verified), private LB:~0.88
random split CV is not verified but very likely(I did not check the random split CV after I crossed 0.87)
", "votes": null, "replies": [ { "id": 2242536, "author_name": "gregorlied", "author_url": "", "post_date": "05/02/2023 10:46:58", "content": "That is also my guess. My CV scores perfectly align in both cases, too.
", "votes": null, "replies": [] }, { "id": 2242637, "author_name": "bamps53", "author_url": "", "post_date": "05/02/2023 12:16:59", "content": "@hoyso48 Thanks for sharing, congrats for the solo winning! Looking forward your solution by the way;)
", "votes": null, "replies": [] } ] } ] }, { "id": 2242671, "author_name": "hengck23", "author_url": "", "post_date": "05/02/2023 12:43:58", "content": "private set is cleaned data?
", "votes": null, "replies": [] }, { "id": 2243140, "author_name": "theoviel", "author_url": "", "post_date": "05/02/2023 17:47:56", "content": "I tried overfitting a model on the training data, it performed worse.
\nSo I don't think there's duplicates between train and test. There can still be signer overlap though.
haha, i'm trying the same experiment now too.
\nIf the private test data has signer (i.e. participant) overlap (but not duplicate videos), I wonder how we could design our models differently to take advantage of this?
", "votes": null, "replies": [ { "id": 2243176, "author_name": "cdeotte", "author_url": "", "post_date": "05/02/2023 18:04:22", "content": "Does each participant only sign each word once? If so, perhaps we can classify the participant then remove any words they already signed in train data.
", "votes": null, "replies": [] } ] }, { "id": 2243430, "author_name": "cdeotte", "author_url": "", "post_date": "05/02/2023 22:35:17", "content": "I also trained my submission for 300 epochs instead of 120 and submitted. It's LB performed worse.
", "votes": null, "replies": [] } ] }, { "id": 2243351, "author_name": "sohier", "author_url": "", "post_date": "05/02/2023 20:55:08", "content": "Now that the competition is over I can disclose that we split the dataset by signer with no overlap between the train, public, and private sets. There were only a handful of signers in the public set so we weren't too surprised to see that the public scores are somewhat idiosyncratic.
\nSimilarly, we expected to see a meaningful degree of variation in the difficulty of parsing videos from each signer. You can see some of that variation in the train set landmarks. Looking at the raw video reveals additional sources of difference, like lighting quality or distance from the camera.
\nOverall I am not concerned.
", "votes": null, "replies": [ { "id": 2243393, "author_name": "theoviel", "author_url": "", "post_date": "05/02/2023 21:50:40", "content": "Good to know, thanks Sohier !
", "votes": null, "replies": [] }, { "id": 2243422, "author_name": "bamps53", "author_url": "", "post_date": "05/02/2023 22:30:30", "content": "Thanks @sohier for the clarification, good to know what really happened!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2242164": "Gold zone score is above 0.87.\nI don't think it can be achived by participant split, at leaset for training set.\n\nDoes private test set have many same participants with training set?\nOr, is there any other reason?", "2242189": "I have the same doubt. It seems for every model, the accuracy has increased for the private lb. Even the models that didn't perform well at all on public LB.", "2242228": "I agree it is very weird. The public LB is similar to local CV score but the private LB is not.", "2242333": "They were very similar to my private CV scores, 0.01-0.02 higher than my local scores. I was surprised my public LB scores were so low.", "2242430": "sohier @ashleychow \nCan you follow up on that ? \n\nOn my local validation setup, I have only one patient with CV > 0.87 (id=26734)\n\nThe private LB scores do not look right.\nEither there is train / test overlap or the private test is surprisingly not noisy.", "2242444": "For reference, using a single model: \n- Validation CV : 0.74\n- Training CV : 0.94\n- Public LB : 0.78, private LB 0.87 (not verified)", "2242468": "Thanks for sharing Theo. Also congrats for your result!\n\nMay I Ask how you split training data? I guess it's split by participant, but just making sure:)", "2242481": "I think they used non-overlapping participant on the public, and the overlapping on the private.\n\nMy single model CV with participant split:~0.80, public LB:~0.80\nrandom split CV:~0.88(not verified), private LB:~0.88\n\nrandom split CV is not verified but very likely(I did not check the random split CV after I crossed 0.87)", "2242527": "Thanks and congratz to you as well !\n\nI split by participant yes (groupkfold with k=4), as recommended by the hosts.", "2242536": "That is also my guess. My CV scores perfectly align in both cases, too.", "2242637": "hoyso48 Thanks for sharing, congrats for the solo winning! Looking forward your solution by the way;)", "2242671": "private set is cleaned data?", "2243140": "I tried overfitting a model on the training data, it performed worse.\nSo I don't think there's duplicates between train and test. There can still be signer overlap though.", "2243173": "haha, i'm trying the same experiment now too.\n\nIf the private test data has signer (i.e. participant) overlap (but not duplicate videos), I wonder how we could design our models differently to take advantage of this?", "2243176": "Does each participant only sign each word once? If so, perhaps we can classify the participant then remove any words they already signed in train data.", "2243351": "Now that the competition is over I can disclose that we split the dataset by signer with no overlap between the train, public, and private sets. There were only a handful of signers in the public set so we weren't too surprised to see that the public scores are somewhat idiosyncratic. \n\nSimilarly, we expected to see a meaningful degree of variation in the difficulty of parsing videos from each signer. You can see some of that variation in the train set landmarks. Looking at the raw video reveals additional sources of difference, like lighting quality or distance from the camera.\n\nOverall I am not concerned.", "2243393": "Good to know, thanks Sohier !", "2243422": "Thanks @sohier for the clarification, good to know what really happened!", "2243430": "I also trained my submission for 300 epochs instead of 120 and submitted. It's LB performed worse." }, "source": "meta" }