Hi everyone.
\nI hope you are enjoying this challenge.
I'm here for an important question. Is everyone facing a gap between CV and LB accuracy score?
\nWe may discuss here techniques to reduce this gap.
\nMy current best is 0.78 train accuracy, 0.67 validation accuracy and 0.48 LB accuracy.
\nI tried using quantization aware training but found no benefit by trying it.
", "messages": [ { "id": 2163552, "postDate": "2023-02-28T23:03:33.557Z", "content": "Because Popsign has to be user independent, I like to do leave-one-signer-out cross validation for all 21-folds and then get an average and standard deviation. That way I can do a t-test to determine if one method is going to be better than another.
", "rawMarkdown": "Because Popsign has to be user independent, I like to do leave-one-signer-out cross validation for all 21-folds and then get an average and standard deviation. That way I can do a t-test to determine if one method is going to be better than another. ", "votes": 16, "replies": [ { "id": 2164058, "postDate": "2023-03-01T09:23:10.037Z", "content": "Does this mean that there are new participants in hidden test sets?
", "rawMarkdown": "Does this mean that there are new participants in hidden test sets?" } ] }, { "id": 2163249, "postDate": "2023-02-28T17:26:13.377Z", "content": "Hi everyone.
\nI hope you are enjoying this challenge.
I'm here for an important question. Is everyone facing a gap between CV and LB accuracy score?
\nWe may discuss here techniques to reduce this gap.
\nMy current best is 0.78 train accuracy, 0.67 validation accuracy and 0.48 LB accuracy.
\nI tried using quantization aware training but found no benefit by trying it.
", "rawMarkdown": "Hi everyone.\nI hope you are enjoying this challenge.\n\nI'm here for an important question. Is everyone facing a gap between CV and LB accuracy score?\n\nWe may discuss here techniques to reduce this gap.\n\nMy current best is 0.78 train accuracy, 0.67 validation accuracy and 0.48 LB accuracy.\n\nI tried using quantization aware training but found no benefit by trying it.", "votes": 13 }, { "id": 2163584, "postDate": "2023-02-28T23:38:22.087Z", "content": "StratifiedgroupKFold: Stratified by sign grouped by participants
\nCV 0.56
\nLB 0.60
\nbut much longer running time than expected
Nicely done! Was your LB score an ensemble? That is, a model that internally ensembled all the KFold models?
", "rawMarkdown": "Nicely done! Was your LB score an ensemble? That is, a model that internally ensembled all the KFold models?", "replies": [ { "id": 2163606, "postDate": "2023-03-01T00:19:28.223Z", "content": "not yet, it is only one fold model
", "rawMarkdown": "not yet, it is only one fold model", "votes": 2 } ] } ] }, { "id": 2169300, "postDate": "2023-03-05T02:08:56.530Z", "content": "My new public notebook gets 0.74 optimistic CV, and 0.63 LB
\nhttps://www.kaggle.com/code/roberthatch/gislr-lb-0-63-on-the-shoulders
", "rawMarkdown": "My new public notebook gets 0.74 optimistic CV, and 0.63 LB\n\nhttps://www.kaggle.com/code/roberthatch/gislr-lb-0-63-on-the-shoulders", "votes": 5 }, { "id": 2165407, "postDate": "2023-03-02T07:29:45.477Z", "content": "same fold from this notebook -> https://www.kaggle.com/code/mayukh18/end-to-end-pytorch-training-submission
\ntrain_test_split(datax, datay, test_size=0.15, random_state=42)
CV 0.52 LB 0.45
\nCV 0.55 LB 0.48
\nCV 0.57 LB 0.49
\nCV 0.59 LB 0.50
CV and LB correlate very well!
", "rawMarkdown": "same fold from this notebook -> https://www.kaggle.com/code/mayukh18/end-to-end-pytorch-training-submission\n`train_test_split(datax, datay, test_size=0.15, random_state=42)`\n\nCV 0.52 LB 0.45\nCV 0.55 LB 0.48\nCV 0.57 LB 0.49\nCV 0.59 LB 0.50\n\nCV and LB correlate very well!", "votes": 3, "replies": [ { "id": 2166813, "postDate": "2023-03-03T03:24:15.440Z", "content": "I improved the model from here, but the gap between LB and CV increased tremendously.
\nProbably because of the leak. I need to groupKFold it with participant_id like Carno.
Using StratifiedGroupKFold, I get the following results from some models using one fold with the following setup: https://www.kaggle.com/code/clemchris/asl-sign-detection-pytorch-lightning
\nFold 0, Val Acc: 0.4304, LB: 0.48
\nFold 2, Val Acc: 0.4700, LB: 0.52
\nFold 4, Val Acc: 0.4572, LB: 0.52
thanks for reporting your results. are the results reported based on the same hyper-parameters in the notebook or after hyper-parameters optimization?
\ni tried to used the same hyper-parameters but could not get the same results.
\nby the way, you are using mean of the frames as input feature. if that works, it will also single frame (or the frame and neighbors that is close to mean) as input might work to some extent.
\nthe next step would probably see how you can pool better over the whole video (e.g. multiple instance, attention pool/average), rather than just average
Interesting that your LB is higher than the CV, I only saw people reporting lower LB than CV. Do you do something special that the others might not?
", "rawMarkdown": "Interesting that your LB is higher than the CV, I only saw people reporting lower LB than CV. Do you do something special that the others might not?", "votes": 1, "replies": [ { "id": 2169794, "postDate": "2023-03-05T12:47:29.333Z", "content": "So I guess most people don't split train and valid on a participant_id- but on a video level.
", "rawMarkdown": "So I guess most people don't split train and valid on a participant_id- but on a video level.", "votes": 1 }, { "id": 2170736, "postDate": "2023-03-06T08:51:07.023Z", "content": "Yes, I think this might be the reason. I am still seeing it when using less features (lips instead of all face, see the latest version of the notebook)
", "rawMarkdown": "Yes, I think this might be the reason. I am still seeing it when using less features (lips instead of all face, see the latest version of the notebook)", "votes": 1 } ] } ] }, { "id": 2163506, "postDate": "2023-02-28T21:48:07.633Z", "content": "it is better to do a few splitting and record results of:
\nwe need to visualise the data (tsne, etc) to see where the variation lies
", "rawMarkdown": "it is better to do a few splitting and record results of:\n- CV within user\n- CV between user\n- CV for each words for each of words ...\n\nwe need to visualise the data (tsne, etc) to see where the variation lies", "votes": 4, "replies": [ { "id": 2163532, "postDate": "2023-02-28T22:28:20.763Z", "content": "You are absolutely correct.
\nI think also I should go back some steps and analyze difference between signs between sequences of the same user and between sequences of different users.
\nFinding those differences may help finding ways to \"normalize\" the users just from a sequence or also it may help in finding useful ways to augment the dataset without adding external data.
", "rawMarkdown": "You are absolutely correct.\n\nI think also I should go back some steps and analyze difference between signs between sequences of the same user and between sequences of different users.\n\nFinding those differences may help finding ways to \"normalize\" the users just from a sequence or also it may help in finding useful ways to augment the dataset without adding external data." } ] }, { "id": 2163320, "postDate": "2023-02-28T18:05:43Z", "content": "Yes, I'm seeing similar gap between CV and LB scores. I wonder whether it's related to overfitting to the individual signers in the training set. Have you tried holding out all the samples from some of the signers and using those for validation? (I haven't yet.)
", "rawMarkdown": "Yes, I'm seeing similar gap between CV and LB scores. I wonder whether it's related to overfitting to the individual signers in the training set. Have you tried holding out all the samples from some of the signers and using those for validation? (I haven't yet.)", "votes": 1, "replies": [ { "id": 2163364, "postDate": "2023-02-28T18:53:26.230Z", "content": "Oh, yes, that's probably why. To get a more accurate CV, we will need to do like 7-fold GroupKFold, holding out 3 signers each fold. Note that will improve the correlation between CV and LB, but probably won't help LB any (even if you figure out how to have a single model that ensembles the base models?) It's mainly for getting more realistic CV.
", "rawMarkdown": "Oh, yes, that's probably why. To get a more accurate CV, we will need to do like 7-fold GroupKFold, holding out 3 signers each fold. Note that will improve the correlation between CV and LB, but probably won't help LB any (even if you figure out how to have a single model that ensembles the base models?) It's mainly for getting more realistic CV.", "votes": 1 }, { "id": 2163501, "postDate": "2023-02-28T21:35:58.163Z", "content": "I tried right now splitting signers, I noticed that my CV dropped from 0.67 to around 0.35.
\nThe model seemed to be heavily overfitting to the individual signers.
\nI'll try to find some way to get better performance with this split and see if with this kind of split I see a better correlation.
So, as I feared, it looks like the variability between signers is greater than the variability between a single signer doing a sign multiple times. Not exactly surprising, but given we've only got 21 signers in the test set, that's going to make it a challenge.
", "rawMarkdown": "So, as I feared, it looks like the variability between signers is greater than the variability between a single signer doing a sign multiple times. Not exactly surprising, but given we've only got 21 signers in the test set, that's going to make it a challenge." }, { "id": 2163554, "postDate": "2023-02-28T23:04:16.383Z", "content": "And of the 21, how many sign left-handed vs right-handed? Most likely it makes this issue even worse
", "rawMarkdown": "And of the 21, how many sign left-handed vs right-handed? Most likely it makes this issue even worse" }, { "id": 2163562, "postDate": "2023-02-28T23:12:31.693Z", "content": "One trick my students and I did to make some test models is just to sum the deltas of both hands. That is a crude way to handle handedness. Another potential option is to mirror all examples so you get the same number of left and right handed signs. Offhand, I can not think of any signs in this vocabulary for which that will not work, but no guarantees. I'll have to try it myself.
", "rawMarkdown": "One trick my students and I did to make some test models is just to sum the deltas of both hands. That is a crude way to handle handedness. Another potential option is to mirror all examples so you get the same number of left and right handed signs. Offhand, I can not think of any signs in this vocabulary for which that will not work, but no guarantees. I'll have to try it myself.", "votes": 4 }, { "id": 2163569, "postDate": "2023-02-28T23:18:32.607Z", "rawMarkdown": "", "isDeleted": true } ] } ] }, { "id": 2166918, "postDate": "2023-03-03T05:48:45.893Z", "content": "In my experiments, CV (single fold validation) & LB correlate well too (usually there's 0.07 gap between the scores).
\nCV 0.6310 LB 0.56
\nCV 0.6677 LB 0.60
That's the same gap as what I got on my first working LB: CV 0.64, LB 0.57.
", "rawMarkdown": "That's the same gap as what I got on my first working LB: CV 0.64, LB 0.57.", "votes": 1 } ] }, { "id": 2164085, "postDate": "2023-03-01T09:39:46.430Z", "content": "In addition to the problem I mentioned below (with very few signers in the training set and apparently large variability between signers), I also wonder how much loss of accuracy there is when the models are converted to TFLite. Presumably this would be easy enough to measure - just run your validation data through the final TFLite model - but my hunch is that the loss there will be very small.
", "rawMarkdown": "In addition to the problem I mentioned below (with very few signers in the training set and apparently large variability between signers), I also wonder how much loss of accuracy there is when the models are converted to TFLite. Presumably this would be easy enough to measure - just run your validation data through the final TFLite model - but my hunch is that the loss there will be very small." }, { "id": 2169885, "postDate": "2023-03-05T14:24:37.350Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2163552, "author_name": "Thad Starner", "author_url": "", "post_date": "2023-02-28T23:03:33.557000", "content": "Because Popsign has to be user independent, I like to do leave-one-signer-out cross validation for all 21-folds and then get an average and standard deviation. That way I can do a t-test to determine if one method is going to be better than another.
", "votes": 16, "replies": [ { "id": 2164058, "author_name": "Ryota", "author_url": "", "post_date": "2023-03-01T09:23:10.037000", "content": "Does this mean that there are new participants in hidden test sets?
", "votes": 0, "replies": [] } ] }, { "id": 2163584, "author_name": "Carno Zhao", "author_url": "", "post_date": "2023-02-28T23:38:22.087000", "content": "StratifiedgroupKFold: Stratified by sign grouped by participants
\nCV 0.56
\nLB 0.60
\nbut much longer running time than expected
Nicely done! Was your LB score an ensemble? That is, a model that internally ensembled all the KFold models?
", "votes": 0, "replies": [ { "id": 2163606, "author_name": "Carno Zhao", "author_url": "", "post_date": "2023-03-01T00:19:28.223000", "content": "not yet, it is only one fold model
", "votes": 2, "replies": [] } ] } ] }, { "id": 2169300, "author_name": "Robert Hatch", "author_url": "", "post_date": "2023-03-05T02:08:56.530000", "content": "My new public notebook gets 0.74 optimistic CV, and 0.63 LB
\nhttps://www.kaggle.com/code/roberthatch/gislr-lb-0-63-on-the-shoulders
", "votes": 5, "replies": [] }, { "id": 2165407, "author_name": "YYama", "author_url": "", "post_date": "2023-03-02T07:29:45.477000", "content": "same fold from this notebook -> https://www.kaggle.com/code/mayukh18/end-to-end-pytorch-training-submission
\ntrain_test_split(datax, datay, test_size=0.15, random_state=42)
CV 0.52 LB 0.45
\nCV 0.55 LB 0.48
\nCV 0.57 LB 0.49
\nCV 0.59 LB 0.50
CV and LB correlate very well!
", "votes": 3, "replies": [ { "id": 2166813, "author_name": "YYama", "author_url": "", "post_date": "2023-03-03T03:24:15.440000", "content": "I improved the model from here, but the gap between LB and CV increased tremendously.
\nProbably because of the leak. I need to groupKFold it with participant_id like Carno.
Using StratifiedGroupKFold, I get the following results from some models using one fold with the following setup: https://www.kaggle.com/code/clemchris/asl-sign-detection-pytorch-lightning
\nFold 0, Val Acc: 0.4304, LB: 0.48
\nFold 2, Val Acc: 0.4700, LB: 0.52
\nFold 4, Val Acc: 0.4572, LB: 0.52
thanks for reporting your results. are the results reported based on the same hyper-parameters in the notebook or after hyper-parameters optimization?
\ni tried to used the same hyper-parameters but could not get the same results.
\nby the way, you are using mean of the frames as input feature. if that works, it will also single frame (or the frame and neighbors that is close to mean) as input might work to some extent.
\nthe next step would probably see how you can pool better over the whole video (e.g. multiple instance, attention pool/average), rather than just average
Interesting that your LB is higher than the CV, I only saw people reporting lower LB than CV. Do you do something special that the others might not?
", "votes": 1, "replies": [ { "id": 2169794, "author_name": "nofreewill42", "author_url": "", "post_date": "2023-03-05T12:47:29.333000", "content": "So I guess most people don't split train and valid on a participant_id- but on a video level.
", "votes": 1, "replies": [] }, { "id": 2170736, "author_name": "clem-chris", "author_url": "", "post_date": "2023-03-06T08:51:07.023000", "content": "Yes, I think this might be the reason. I am still seeing it when using less features (lips instead of all face, see the latest version of the notebook)
", "votes": 1, "replies": [] } ] } ] }, { "id": 2163506, "author_name": "hengck23", "author_url": "", "post_date": "2023-02-28T21:48:07.633000", "content": "it is better to do a few splitting and record results of:
\nwe need to visualise the data (tsne, etc) to see where the variation lies
", "votes": 4, "replies": [ { "id": 2163532, "author_name": "Pietro Maldini", "author_url": "", "post_date": "2023-02-28T22:28:20.763000", "content": "You are absolutely correct.
\nI think also I should go back some steps and analyze difference between signs between sequences of the same user and between sequences of different users.
\nFinding those differences may help finding ways to \"normalize\" the users just from a sequence or also it may help in finding useful ways to augment the dataset without adding external data.
", "votes": 0, "replies": [] } ] }, { "id": 2163320, "author_name": "Andrew", "author_url": "", "post_date": "2023-02-28T18:05:43", "content": "Yes, I'm seeing similar gap between CV and LB scores. I wonder whether it's related to overfitting to the individual signers in the training set. Have you tried holding out all the samples from some of the signers and using those for validation? (I haven't yet.)
", "votes": 1, "replies": [ { "id": 2163364, "author_name": "Robert Hatch", "author_url": "", "post_date": "2023-02-28T18:53:26.230000", "content": "Oh, yes, that's probably why. To get a more accurate CV, we will need to do like 7-fold GroupKFold, holding out 3 signers each fold. Note that will improve the correlation between CV and LB, but probably won't help LB any (even if you figure out how to have a single model that ensembles the base models?) It's mainly for getting more realistic CV.
", "votes": 1, "replies": [] }, { "id": 2163501, "author_name": "Pietro Maldini", "author_url": "", "post_date": "2023-02-28T21:35:58.163000", "content": "I tried right now splitting signers, I noticed that my CV dropped from 0.67 to around 0.35.
\nThe model seemed to be heavily overfitting to the individual signers.
\nI'll try to find some way to get better performance with this split and see if with this kind of split I see a better correlation.
So, as I feared, it looks like the variability between signers is greater than the variability between a single signer doing a sign multiple times. Not exactly surprising, but given we've only got 21 signers in the test set, that's going to make it a challenge.
", "votes": 0, "replies": [] }, { "id": 2163554, "author_name": "Robert Hatch", "author_url": "", "post_date": "2023-02-28T23:04:16.383000", "content": "And of the 21, how many sign left-handed vs right-handed? Most likely it makes this issue even worse
", "votes": 0, "replies": [] }, { "id": 2163562, "author_name": "Thad Starner", "author_url": "", "post_date": "2023-02-28T23:12:31.693000", "content": "One trick my students and I did to make some test models is just to sum the deltas of both hands. That is a crude way to handle handedness. Another potential option is to mirror all examples so you get the same number of left and right handed signs. Offhand, I can not think of any signs in this vocabulary for which that will not work, but no guarantees. I'll have to try it myself.
", "votes": 4, "replies": [] }, { "id": 2163569, "author_name": "", "author_url": "", "post_date": "2023-02-28T23:18:32.607000", "content": "", "votes": 0, "replies": [] } ] } ] }, { "id": 2166918, "author_name": "HyeongChan Kim", "author_url": "", "post_date": "2023-03-03T05:48:45.893000", "content": "In my experiments, CV (single fold validation) & LB correlate well too (usually there's 0.07 gap between the scores).
\nCV 0.6310 LB 0.56
\nCV 0.6677 LB 0.60
That's the same gap as what I got on my first working LB: CV 0.64, LB 0.57.
", "votes": 1, "replies": [] } ] }, { "id": 2164085, "author_name": "Andrew", "author_url": "", "post_date": "2023-03-01T09:39:46.430000", "content": "In addition to the problem I mentioned below (with very few signers in the training set and apparently large variability between signers), I also wonder how much loss of accuracy there is when the models are converted to TFLite. Presumably this would be easy enough to measure - just run your validation data through the final TFLite model - but my hunch is that the loss there will be very small.
", "votes": 0, "replies": [] }, { "id": 2169885, "author_name": "", "author_url": "", "post_date": "2023-03-05T14:24:37.350000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2163552": "Because Popsign has to be user independent, I like to do leave-one-signer-out cross validation for all 21-folds and then get an average and standard deviation. That way I can do a t-test to determine if one method is going to be better than another. ", "2163249": "Hi everyone.\nI hope you are enjoying this challenge.\n\nI'm here for an important question. Is everyone facing a gap between CV and LB accuracy score?\n\nWe may discuss here techniques to reduce this gap.\n\nMy current best is 0.78 train accuracy, 0.67 validation accuracy and 0.48 LB accuracy.\n\nI tried using quantization aware training but found no benefit by trying it.", "2163584": "StratifiedgroupKFold: Stratified by sign grouped by participants\n\nCV 0.56\nLB 0.60\nbut much longer running time than expected ", "2169300": "My new public notebook gets 0.74 optimistic CV, and 0.63 LB\n\nhttps://www.kaggle.com/code/roberthatch/gislr-lb-0-63-on-the-shoulders", "2165407": "same fold from this notebook -> https://www.kaggle.com/code/mayukh18/end-to-end-pytorch-training-submission\n`train_test_split(datax, datay, test_size=0.15, random_state=42)`\n\nCV 0.52 LB 0.45\nCV 0.55 LB 0.48\nCV 0.57 LB 0.49\nCV 0.59 LB 0.50\n\nCV and LB correlate very well!", "2167450": "Using StratifiedGroupKFold, I get the following results from some models using one fold with the following setup: https://www.kaggle.com/code/clemchris/asl-sign-detection-pytorch-lightning\n\nFold 0, Val Acc: 0.4304, LB: 0.48\nFold 2, Val Acc: 0.4700, LB: 0.52\nFold 4, Val Acc: 0.4572, LB: 0.52", "2163506": "it is better to do a few splitting and record results of:\n- CV within user\n- CV between user\n- CV for each words for each of words ...\n\nwe need to visualise the data (tsne, etc) to see where the variation lies", "2163320": "Yes, I'm seeing similar gap between CV and LB scores. I wonder whether it's related to overfitting to the individual signers in the training set. Have you tried holding out all the samples from some of the signers and using those for validation? (I haven't yet.)", "2166918": "In my experiments, CV (single fold validation) & LB correlate well too (usually there's 0.07 gap between the scores).\n\nCV 0.6310 LB 0.56\nCV 0.6677 LB 0.60", "2164085": "In addition to the problem I mentioned below (with very few signers in the training set and apparently large variability between signers), I also wonder how much loss of accuracy there is when the models are converted to TFLite. Presumably this would be easy enough to measure - just run your validation data through the final TFLite model - but my hunch is that the loss there will be very small.", "2169885": "" } }