The data tab says that the curated train set and the test set came from the same dataset and both were labeled manually. However, I see that most public kernels and also my own pipeline (which differs from public kernels) have very high CV-LB difference that seems to be around 0.2.
\n\nAre you getting the same results?
", "messages": [ { "id": 515655, "postDate": "2019-04-12T22:45:21.723Z", "content": "The data tab says that the curated train set and the test set came from the same dataset and both were labeled manually. However, I see that most public kernels and also my own pipeline (which differs from public kernels) have very high CV-LB difference that seems to be around 0.2.
\n\nAre you getting the same results?
", "rawMarkdown": "The data tab says that the curated train set and the test set came from the same dataset and both were labeled manually. However, I see that most public kernels and also my own pipeline (which differs from public kernels) have very high CV-LB difference that seems to be around 0.2. \n\nAre you getting the same results?", "votes": 17 }, { "id": 521710, "postDate": "2019-04-23T10:08:47.610Z", "content": "5folds, CV:0.67, LB:0.635
\nUpdate: 5folds, CV:0.685, LB:0.665
Do you use noisy data when training?
", "rawMarkdown": "Do you use noisy data when training?" }, { "id": 521723, "postDate": "2019-04-23T10:39:06.370Z", "content": "Yes.
", "rawMarkdown": "Yes.", "votes": 1 }, { "id": 521747, "postDate": "2019-04-23T11:08:33.077Z", "content": "May I ask if you use both curated and noisy data for validation?
", "rawMarkdown": "May I ask if you use both curated and noisy data for validation?" }, { "id": 521753, "postDate": "2019-04-23T11:17:45.977Z", "content": "Yes. Since I started this competition today, I had no time to make any effort regarding validation.
\nI have focused on modeling and loss function.
Yeah I noticed if you validate with noisy data together then the CV more closely matches LB (you must check the lwlrap with noisy data included). Once you remove noisy data from validation after training together then you can see the lwlrap score is once again around 0.2 higher.
", "rawMarkdown": "Yeah I noticed if you validate with noisy data together then the CV more closely matches LB (you must check the lwlrap with noisy data included). Once you remove noisy data from validation after training together then you can see the lwlrap score is once again around 0.2 higher.", "votes": 2 } ] }, { "id": 518738, "postDate": "2019-04-17T17:52:39.173Z", "content": "\n\nHello,
\n\nJust a little clarification. Can we expect that the stage2 test data will have the same distribution as the stage1 test data? Thanks.
", "rawMarkdown": "@eduardofonseca @plakal\n\nHello, \n\nJust a little clarification. Can we expect that the stage2 test data will have the same distribution as the stage1 test data? Thanks.", "votes": 1, "replies": [ { "id": 518834, "postDate": "2019-04-17T22:34:44.837Z", "content": "The first and second stages were produced by a 1:3 random split of the original test dataset.
", "rawMarkdown": "The first and second stages were produced by a 1:3 random split of the original test dataset.", "votes": 7 } ] }, { "id": 517823, "postDate": "2019-04-16T15:24:24Z", "content": "same for me. I do the predictions over the whole validation set (20% holdout) and afterwards i calculate the lwlrap. CV is about 0.73, i don't know exactly because i didn't do k-fold, just the final run over all the data with similar settings, the LB was then 0.505
", "rawMarkdown": "same for me. I do the predictions over the whole validation set (20% holdout) and afterwards i calculate the lwlrap. CV is about 0.73, i don't know exactly because i didn't do k-fold, just the final run over all the data with similar settings, the LB was then 0.505", "votes": 1 }, { "id": 516317, "postDate": "2019-04-14T01:43:47.673Z", "content": "I have just made a few submissions and already seeing the similar relationships between cv and lb.
\n\nI only use train_curated and 5-folds like op.
", "rawMarkdown": "I have just made a few submissions and already seeing the similar relationships between cv and lb.\n\n- cv 0.683, lb 0.436\n- cv 0.739, lb 0.505\n\nI only use train_curated and 5-folds like op.", "votes": 1 }, { "id": 515871, "postDate": "2019-04-13T08:35:38.903Z", "content": "Yes, we also observe huge difference around 0.2 between CV and LB, but luckily its correlated, so when CV goes up LB goes up too.
", "rawMarkdown": "Yes, we also observe huge difference around 0.2 between CV and LB, but luckily its correlated, so when CV goes up LB goes up too.", "votes": 1, "replies": [ { "id": 515891, "postDate": "2019-04-13T09:16:47.103Z", "content": "Yeah, I also observe the correlation. However, such a huge gap is a bit disturbing, since it's stated that the train and test data have to be very similar.
", "rawMarkdown": "Yeah, I also observe the correlation. However, such a huge gap is a bit disturbing, since it's stated that the train and test data have to be very similar.", "votes": 1 } ] }, { "id": 515759, "postDate": "2019-04-13T04:58:59.637Z", "content": "Yes I am seeing the same. On my local CV I get on average 0.76 lwlrap score which is close to 0.2 difference from LB. I used the lwlrap implementation that was provided and computed on the entire validation set.
", "rawMarkdown": "Yes I am seeing the same. On my local CV I get on average 0.76 lwlrap score which is close to 0.2 difference from LB. I used the lwlrap implementation that was provided and computed on the entire validation set.", "votes": 1, "replies": [ { "id": 515766, "postDate": "2019-04-13T05:19:29.890Z", "content": "@eduardofonseca if he has any thoughts.
\n\nAnd just to be clear, your cross-validation set is a randomized subset of the curated train set, right (there are patterns in the order of rows in the CSV)? And you have all classes equally represented in your train set and your validation set? Are you sure you are not overfitting to your (now smaller) train set because you might be using a large model?
", "rawMarkdown": "@eduardofonseca if he has any thoughts.\n\nAnd just to be clear, your cross-validation set is a randomized subset of the curated train set, right (there are patterns in the order of rows in the CSV)? And you have all classes equally represented in your train set and your validation set? Are you sure you are not overfitting to your (now smaller) train set because you might be using a large model?\n" } ] }, { "id": 515674, "postDate": "2019-04-12T23:45:55.683Z", "content": "By CV, do you mean lwlrap computed on a held-out cross-validation set taken from the curated train set? Are you using the lwlrap implementation that we have provided, and are you computing it over the entire validation set? There was another thread about lwlrap where someone was computing lwlrap per batch and taking the average, which is not the way to combine lwlraps.
", "rawMarkdown": "By CV, do you mean lwlrap computed on a held-out cross-validation set taken from the curated train set? Are you using the lwlrap implementation that we have provided, and are you computing it over the entire validation set? There was another thread about lwlrap where someone was computing lwlrap per batch and taking the average, which is not the way to combine lwlraps.", "votes": 1, "replies": [ { "id": 515821, "postDate": "2019-04-13T07:14:07.900Z", "content": "Sure, I'm doing CV on the curated train set. I'm using the provided lwlrap implementation. The metric is computed on the entire validation set in a single pass.
", "rawMarkdown": "Sure, I'm doing CV on the curated train set. I'm using the provided lwlrap implementation. The metric is computed on the entire validation set in a single pass." }, { "id": 515893, "postDate": "2019-04-13T09:19:10.337Z", "content": "I've just tested my approach on a holdout set. The results are very close to my CV results and the gap is still around 0.2.
", "rawMarkdown": "I've just tested my approach on a holdout set. The results are very close to my CV results and the gap is still around 0.2.", "votes": 2 } ] }, { "id": 518604, "postDate": "2019-04-17T14:10:37.730Z", "content": "5 folds cv 0.803 lb 0.62
", "rawMarkdown": "5 folds cv 0.803 lb 0.62", "votes": 2 }, { "id": 515709, "postDate": "2019-04-13T01:59:54.457Z", "content": "@ddanevskyi Yes, it's same with me. As far as I see the statistics, train and test set are similar. So, I wonder how they are annotated, e.g. by different person and splitted?
", "rawMarkdown": "@ddanevskyi Yes, it's same with me. As far as I see the statistics, train and test set are similar. So, I wonder how they are annotated, e.g. by different person and splitted?", "votes": 2 }, { "id": 515851, "postDate": "2019-04-13T08:14:36.920Z", "content": "When your CV improves, does your LB score improve too?
", "rawMarkdown": "When your CV improves, does your LB score improve too?", "votes": -1, "replies": [ { "id": 515868, "postDate": "2019-04-13T08:29:00.057Z", "content": "Yes.
", "rawMarkdown": "Yes. " } ] }, { "id": 523696, "postDate": "2019-04-26T18:59:15.313Z", "content": "Well, my last observations are that LB-CV gap is model-specific and also that as local score improves, the gap seems to become smaller. Note, however, that these facts may be specific to my approach and may not work for you.
", "rawMarkdown": "Well, my last observations are that LB-CV gap is model-specific and also that as local score improves, the gap seems to become smaller. Note, however, that these facts may be specific to my approach and may not work for you." }, { "id": 521393, "postDate": "2019-04-22T21:18:42.897Z", "content": "Has anyone managed to find a way to close the gap? :) It seems that training on the noisy part of the train set helps a bit, but the difference is still around 0.15...
", "rawMarkdown": "Has anyone managed to find a way to close the gap? :) It seems that training on the noisy part of the train set helps a bit, but the difference is still around 0.15...", "replies": [ { "id": 521683, "postDate": "2019-04-23T09:06:18Z", "content": "Same here, using both noisy and curated data, 5Fold CV\n0.747 CV <-> 0.617 LB\na 0.13 difference
", "rawMarkdown": "Same here, using both noisy and curated data, 5Fold CV\n0.747 CV <-> 0.617 LB\na 0.13 difference" } ] }, { "id": 516143, "postDate": "2019-04-13T17:17:23.353Z", "content": "Hi,
\n\njust to confirm or deny some facts:
\n\n@ddanevskyi, you mention:
\n\n\n\n\nit's stated that the train and test data have to be very similar.
\n
really? I'm afraid I don’t remember that being stated anywhere... Can you point us where it is stated?
\n\nI am not sure I see the problem here. The fact that CV scores are higher than test scores can be just a feature, and it is not uncommon in recognition tasks.
\n\nthanks!
", "rawMarkdown": "Hi,\n\njust to confirm or deny some facts:\n\n- the curated train set and the test set come from the currently under development FSD, yes. \n- both are labeled manually, yes. Several people were involved in the annotation of both sets.\n\n@ddanevskyi, you mention:\n> it's stated that the train and test data have to be very similar.\n\nreally? I'm afraid I don’t remember that being stated anywhere... Can you point us where it is stated?\n\nI am not sure I see the problem here. The fact that CV scores are higher than test scores can be just a feature, and it is not uncommon in recognition tasks.\n\nthanks!", "replies": [ { "id": 516153, "postDate": "2019-04-13T17:33:31.233Z", "content": "Oh, maybe I got it wrong, but for some reason, I assumed that since train/test come from the same dataset and both were manually labeled, their distributions should be the same. If it's not the case, there is indeed no problem. Thank you for clarification.
", "rawMarkdown": "Oh, maybe I got it wrong, but for some reason, I assumed that since train/test come from the same dataset and both were manually labeled, their distributions should be the same. If it's not the case, there is indeed no problem. Thank you for clarification." } ] }, { "id": 515790, "postDate": "2019-04-13T06:16:00.507Z", "content": "As an alternative data point, I tried evaluating one of our baseline models that had been trained purely on the noisy data and got comparable lwlraps when evaluated on the test set and an equal-sized random subset of the curated train set (curated train was lower by ~0.004). I would double check that all classes are equally represented in your train and validation sets, and that your model has not gotten large enough that it overfits to your train and validation sets.
", "rawMarkdown": "As an alternative data point, I tried evaluating one of our baseline models that had been trained purely on the noisy data and got comparable lwlraps when evaluated on the test set and an equal-sized random subset of the curated train set (curated train was lower by ~0.004). I would double check that all classes are equally represented in your train and validation sets, and that your model has not gotten large enough that it overfits to your train and validation sets.", "replies": [ { "id": 515823, "postDate": "2019-04-13T07:21:04.370Z", "content": "It seems that almost everyone has this issue. My local scores are close to what @jamesrequa has reported and my LB score is also very close to his score. I really doubt that we have the same bug/leak since our pipelines are probably very different.
\n\nWhat is the CV score for your baseline model? Currently, I'm interested in only the curated set performance.
\n\nThank you in advance!
", "rawMarkdown": "It seems that almost everyone has this issue. My local scores are close to what @jamesrequa has reported and my LB score is also very close to his score. I really doubt that we have the same bug/leak since our pipelines are probably very different.\n\nWhat is the CV score for your baseline model? Currently, I'm interested in only the curated set performance.\n\nThank you in advance!", "votes": 1 }, { "id": 516117, "postDate": "2019-04-13T16:25:35.317Z", "content": "Our submitted baseline was trained using an internal copy of the train and test sets and was validated using the entire test set. I can re-run the best model using only the released data from Kaggle. Can you share the train and validation splits you used?
", "rawMarkdown": "Our submitted baseline was trained using an internal copy of the train and test sets and was validated using the entire test set. I can re-run the best model using only the released data from Kaggle. Can you share the train and validation splits you used?" }, { "id": 516157, "postDate": "2019-04-13T17:41:37.463Z", "content": "Sure, I used 5 fold cv with default KFold from sklearn with random_state=42. Currently, I'm using only curated data.
To close out this sub-thread, as a rough point of comparison, I tried training the baseline with the same hyperparameters and number of epochs as the original submission, but with a 80%-20% train-validation split chosen by your choice of fold selector: sklearn's KFold with shuffle=True and random_state=42 (note that I tried just one of the 5 splits). The validation lwlrap was ~0.705 compared with public test lwlrap of ~0.537 and full (public + private) test lwlrap of ~0.544.
\n\nI wouldn't get too alarmed by the difference in the lwlraps. As Eduardo said, train and test are taken from the same source of data and have been annotated in similar ways, but they are not identical in all respects. Perhaps you can figure out how to make use of the extra noisy data to your model's advantage. Also note that lwlrap is designed to be a weighted average of per-class lwlraps, so it might be time to look into how your models are doing for individual classes in order to raise your overall rank.
\n\nBTW, you didn't mention it and the KFold seems to use a default shuffle of False, but I hope you are shuffling when making your folds? The lines in the training CSV are not in completely random order.
\n\nAlso worth checking out: splitting your data blindly may increase bias given that this is a multi-label problem. Perhaps take a look at http://scikit.ml/stratification.html (although I don't have much experience with it myself)
", "rawMarkdown": "To close out this sub-thread, as a rough point of comparison, I tried training the baseline with the same hyperparameters and number of epochs as the original submission, but with a 80%-20% train-validation split chosen by your choice of fold selector: sklearn's KFold with shuffle=True and random_state=42 (note that I tried just one of the 5 splits). The validation lwlrap was ~0.705 compared with public test lwlrap of ~0.537 and full (public + private) test lwlrap of ~0.544.\n\nI wouldn't get too alarmed by the difference in the lwlraps. As Eduardo said, train and test are taken from the same source of data and have been annotated in similar ways, but they are not identical in all respects. Perhaps you can figure out how to make use of the extra noisy data to your model's advantage. Also note that lwlrap is designed to be a weighted average of per-class lwlraps, so it might be time to look into how your models are doing for individual classes in order to raise your overall rank.\n\nBTW, you didn't mention it and the KFold seems to use a default shuffle of False, but I hope you are shuffling when making your folds? The lines in the training CSV are not in completely random order.\n\nAlso worth checking out: splitting your data blindly may increase bias given that this is a multi-label problem. Perhaps take a look at http://scikit.ml/stratification.html (although I don't have much experience with it myself)", "votes": 5 } ] }, { "id": 522310, "postDate": "2019-04-24T07:54:02.883Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 521710, "author_name": "phalanx", "author_url": "", "post_date": "2019-04-23T10:08:47.610000", "content": "5folds, CV:0.67, LB:0.635
\nUpdate: 5folds, CV:0.685, LB:0.665
Do you use noisy data when training?
", "votes": 0, "replies": [] }, { "id": 521723, "author_name": "phalanx", "author_url": "", "post_date": "2019-04-23T10:39:06.370000", "content": "Yes.
", "votes": 1, "replies": [] }, { "id": 521747, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2019-04-23T11:08:33.077000", "content": "May I ask if you use both curated and noisy data for validation?
", "votes": 0, "replies": [] }, { "id": 521753, "author_name": "phalanx", "author_url": "", "post_date": "2019-04-23T11:17:45.977000", "content": "Yes. Since I started this competition today, I had no time to make any effort regarding validation.
\nI have focused on modeling and loss function.
Yeah I noticed if you validate with noisy data together then the CV more closely matches LB (you must check the lwlrap with noisy data included). Once you remove noisy data from validation after training together then you can see the lwlrap score is once again around 0.2 higher.
", "votes": 2, "replies": [] } ] }, { "id": 518738, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2019-04-17T17:52:39.173000", "content": "\n\nHello,
\n\nJust a little clarification. Can we expect that the stage2 test data will have the same distribution as the stage1 test data? Thanks.
", "votes": 1, "replies": [ { "id": 518834, "author_name": "Manoj Plakal", "author_url": "", "post_date": "2019-04-17T22:34:44.837000", "content": "The first and second stages were produced by a 1:3 random split of the original test dataset.
", "votes": 7, "replies": [] } ] }, { "id": 517823, "author_name": "Marek Wyborski", "author_url": "", "post_date": "2019-04-16T15:24:24", "content": "same for me. I do the predictions over the whole validation set (20% holdout) and afterwards i calculate the lwlrap. CV is about 0.73, i don't know exactly because i didn't do k-fold, just the final run over all the data with similar settings, the LB was then 0.505
", "votes": 1, "replies": [] }, { "id": 516317, "author_name": "Appian", "author_url": "", "post_date": "2019-04-14T01:43:47.673000", "content": "I have just made a few submissions and already seeing the similar relationships between cv and lb.
\n\nI only use train_curated and 5-folds like op.
", "votes": 1, "replies": [] }, { "id": 515871, "author_name": "Hidehisa Arai", "author_url": "", "post_date": "2019-04-13T08:35:38.903000", "content": "Yes, we also observe huge difference around 0.2 between CV and LB, but luckily its correlated, so when CV goes up LB goes up too.
", "votes": 1, "replies": [ { "id": 515891, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2019-04-13T09:16:47.103000", "content": "Yeah, I also observe the correlation. However, such a huge gap is a bit disturbing, since it's stated that the train and test data have to be very similar.
", "votes": 1, "replies": [] } ] }, { "id": 515759, "author_name": "James Requa", "author_url": "", "post_date": "2019-04-13T04:58:59.637000", "content": "Yes I am seeing the same. On my local CV I get on average 0.76 lwlrap score which is close to 0.2 difference from LB. I used the lwlrap implementation that was provided and computed on the entire validation set.
", "votes": 1, "replies": [ { "id": 515766, "author_name": "Manoj Plakal", "author_url": "", "post_date": "2019-04-13T05:19:29.890000", "content": "@eduardofonseca if he has any thoughts.
\n\nAnd just to be clear, your cross-validation set is a randomized subset of the curated train set, right (there are patterns in the order of rows in the CSV)? And you have all classes equally represented in your train set and your validation set? Are you sure you are not overfitting to your (now smaller) train set because you might be using a large model?
", "votes": 0, "replies": [] } ] }, { "id": 515674, "author_name": "Manoj Plakal", "author_url": "", "post_date": "2019-04-12T23:45:55.683000", "content": "By CV, do you mean lwlrap computed on a held-out cross-validation set taken from the curated train set? Are you using the lwlrap implementation that we have provided, and are you computing it over the entire validation set? There was another thread about lwlrap where someone was computing lwlrap per batch and taking the average, which is not the way to combine lwlraps.
", "votes": 1, "replies": [ { "id": 515821, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2019-04-13T07:14:07.900000", "content": "Sure, I'm doing CV on the curated train set. I'm using the provided lwlrap implementation. The metric is computed on the entire validation set in a single pass.
", "votes": 0, "replies": [] }, { "id": 515893, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2019-04-13T09:19:10.337000", "content": "I've just tested my approach on a holdout set. The results are very close to my CV results and the gap is still around 0.2.
", "votes": 2, "replies": [] } ] }, { "id": 518604, "author_name": "qrfaction", "author_url": "", "post_date": "2019-04-17T14:10:37.730000", "content": "5 folds cv 0.803 lb 0.62
", "votes": 2, "replies": [] }, { "id": 515709, "author_name": "AkiraSosa", "author_url": "", "post_date": "2019-04-13T01:59:54.457000", "content": "@ddanevskyi Yes, it's same with me. As far as I see the statistics, train and test set are similar. So, I wonder how they are annotated, e.g. by different person and splitted?
", "votes": 2, "replies": [] }, { "id": 515851, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2019-04-13T08:14:36.920000", "content": "When your CV improves, does your LB score improve too?
", "votes": -1, "replies": [ { "id": 515868, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2019-04-13T08:29:00.057000", "content": "Yes.
", "votes": 0, "replies": [] } ] }, { "id": 523696, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2019-04-26T18:59:15.313000", "content": "Well, my last observations are that LB-CV gap is model-specific and also that as local score improves, the gap seems to become smaller. Note, however, that these facts may be specific to my approach and may not work for you.
", "votes": 0, "replies": [] }, { "id": 521393, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2019-04-22T21:18:42.897000", "content": "Has anyone managed to find a way to close the gap? :) It seems that training on the noisy part of the train set helps a bit, but the difference is still around 0.15...
", "votes": 0, "replies": [ { "id": 521683, "author_name": "YLChan", "author_url": "", "post_date": "2019-04-23T09:06:18", "content": "Same here, using both noisy and curated data, 5Fold CV\n0.747 CV <-> 0.617 LB\na 0.13 difference
", "votes": 0, "replies": [] } ] }, { "id": 516143, "author_name": "Eduardo Fonseca", "author_url": "", "post_date": "2019-04-13T17:17:23.353000", "content": "Hi,
\n\njust to confirm or deny some facts:
\n\n@ddanevskyi, you mention:
\n\n\n\n\nit's stated that the train and test data have to be very similar.
\n
really? I'm afraid I don’t remember that being stated anywhere... Can you point us where it is stated?
\n\nI am not sure I see the problem here. The fact that CV scores are higher than test scores can be just a feature, and it is not uncommon in recognition tasks.
\n\nthanks!
", "votes": 0, "replies": [ { "id": 516153, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2019-04-13T17:33:31.233000", "content": "Oh, maybe I got it wrong, but for some reason, I assumed that since train/test come from the same dataset and both were manually labeled, their distributions should be the same. If it's not the case, there is indeed no problem. Thank you for clarification.
", "votes": 0, "replies": [] } ] }, { "id": 515790, "author_name": "Manoj Plakal", "author_url": "", "post_date": "2019-04-13T06:16:00.507000", "content": "As an alternative data point, I tried evaluating one of our baseline models that had been trained purely on the noisy data and got comparable lwlraps when evaluated on the test set and an equal-sized random subset of the curated train set (curated train was lower by ~0.004). I would double check that all classes are equally represented in your train and validation sets, and that your model has not gotten large enough that it overfits to your train and validation sets.
", "votes": 0, "replies": [ { "id": 515823, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2019-04-13T07:21:04.370000", "content": "It seems that almost everyone has this issue. My local scores are close to what @jamesrequa has reported and my LB score is also very close to his score. I really doubt that we have the same bug/leak since our pipelines are probably very different.
\n\nWhat is the CV score for your baseline model? Currently, I'm interested in only the curated set performance.
\n\nThank you in advance!
", "votes": 1, "replies": [] }, { "id": 516117, "author_name": "Manoj Plakal", "author_url": "", "post_date": "2019-04-13T16:25:35.317000", "content": "Our submitted baseline was trained using an internal copy of the train and test sets and was validated using the entire test set. I can re-run the best model using only the released data from Kaggle. Can you share the train and validation splits you used?
", "votes": 0, "replies": [] }, { "id": 516157, "author_name": "Dmytro Danevskyi", "author_url": "", "post_date": "2019-04-13T17:41:37.463000", "content": "Sure, I used 5 fold cv with default KFold from sklearn with random_state=42. Currently, I'm using only curated data.
To close out this sub-thread, as a rough point of comparison, I tried training the baseline with the same hyperparameters and number of epochs as the original submission, but with a 80%-20% train-validation split chosen by your choice of fold selector: sklearn's KFold with shuffle=True and random_state=42 (note that I tried just one of the 5 splits). The validation lwlrap was ~0.705 compared with public test lwlrap of ~0.537 and full (public + private) test lwlrap of ~0.544.
\n\nI wouldn't get too alarmed by the difference in the lwlraps. As Eduardo said, train and test are taken from the same source of data and have been annotated in similar ways, but they are not identical in all respects. Perhaps you can figure out how to make use of the extra noisy data to your model's advantage. Also note that lwlrap is designed to be a weighted average of per-class lwlraps, so it might be time to look into how your models are doing for individual classes in order to raise your overall rank.
\n\nBTW, you didn't mention it and the KFold seems to use a default shuffle of False, but I hope you are shuffling when making your folds? The lines in the training CSV are not in completely random order.
\n\nAlso worth checking out: splitting your data blindly may increase bias given that this is a multi-label problem. Perhaps take a look at http://scikit.ml/stratification.html (although I don't have much experience with it myself)
", "votes": 5, "replies": [] } ] }, { "id": 522310, "author_name": "", "author_url": "", "post_date": "2019-04-24T07:54:02.883000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "515655": "The data tab says that the curated train set and the test set came from the same dataset and both were labeled manually. However, I see that most public kernels and also my own pipeline (which differs from public kernels) have very high CV-LB difference that seems to be around 0.2. \n\nAre you getting the same results?", "521710": "5folds, CV:0.67, LB:0.635