Hi, I am new to kaggle. I am using resnet200d with all layers unfrozen. I ran the training for about 100 epochs and was able to get an F1 score of 90+ on train set and 80+ on validation but even then I get a public score of only 0.177. I have tried training on efficientnet-b3 as well but I am still getting the same score. I am probably doing something wrong maybe in the inference or while preparing the submission.csv. This is the notebook. I shall be very grateful if someone can help me figure this out. I have been seeing the number 0.177 all week 😐.
", "messages": [ { "id": "1298344", "postDate": "05/08/2021 18:46:51", "content": "Hi, I am new to kaggle. I am using resnet200d with all layers unfrozen. I ran the training for about 100 epochs and was able to get an F1 score of 90+ on train set and 80+ on validation but even then I get a public score of only 0.177. I have tried training on efficientnet-b3 as well but I am still getting the same score. I am probably doing something wrong maybe in the inference or while preparing the submission.csv. This is the notebook. I shall be very grateful if someone can help me figure this out. I have been seeing the number 0.177 all week 😐.
", "rawMarkdown": "Hi, I am new to kaggle. I am using resnet200d with all layers unfrozen. I ran the training for about 100 epochs and was able to get an F1 score of 90+ on train set and 80+ on validation but even then I get a public score of only 0.177. I have tried training on efficientnet-b3 as well but I am still getting the same score. I am probably doing something wrong maybe in the inference or while preparing the submission.csv. This is the [notebook](https://www.kaggle.com/rustyelectron/plant-path-inference-resnet200d). I shall be very grateful if someone can help me figure this out. I have been seeing the number 0.177 all week 😐.", "votes": null }, { "id": "1298786", "postDate": "05/09/2021 07:49:48", "content": "Can you share the training notebook as well? It's hard to tell what's wrong without it.
", "rawMarkdown": "Can you share the training notebook as well? It's hard to tell what's wrong without it.", "votes": null }, { "id": "1298793", "postDate": "05/09/2021 08:01:52", "content": "ya sure, here it is.
", "rawMarkdown": "ya sure, [here](https://www.kaggle.com/rustyelectron/plant-path-train-resnet200d-288x288) it is.", "votes": null }, { "id": "1298799", "postDate": "05/09/2021 08:13:50", "content": "First thing I noticed is that you don't do shuffle in your training dataloader, which is not good. (read more here https://datascience.stackexchange.com/questions/24511/why-should-the-data-be-shuffled-for-machine-learning-tasks )
", "rawMarkdown": "First thing I noticed is that you don't do shuffle in your training dataloader, which is not good. (read more here https://datascience.stackexchange.com/questions/24511/why-should-the-data-be-shuffled-for-machine-learning-tasks )", "votes": null }, { "id": "1298806", "postDate": "05/09/2021 08:20:22", "content": "Second, your inference is quite strange. Why do you the same thing (inference) several times? If you want to do TTA, it's not how it's done. You need to augment your test data, and then average your predictions.
", "rawMarkdown": "Second, your inference is quite strange. Why do you the same thing (inference) several times? If you want to do TTA, it's not how it's done. You need to augment your test data, and then average your predictions.", "votes": null }, { "id": "1298825", "postDate": "05/09/2021 08:42:34", "content": "Ok, I will add shuffling, btw is it okay if I shuffle the pandas dataframe using df.sample(frac=1) instead of shuffling at the DataLoader as it consumes extra GPU memory or is there a better way?
Thanks for the suggestion about TTA, I didn't understand it before. I get it now and i'll fix it.
\nIs there anything else in my notebook that might be responsible for the bad score?
", "rawMarkdown": "Ok, I will add shuffling, btw is it okay if I shuffle the pandas dataframe using `df.sample(frac=1)` instead of shuffling at the DataLoader as it consumes extra GPU memory or is there a better way? \n\nThanks for the suggestion about TTA, I didn't understand it before. I get it now and i'll fix it.\n\nIs there anything else in my notebook that might be responsible for the bad score?", "votes": null }, { "id": "1298834", "postDate": "05/09/2021 08:53:16", "content": "No, that's not good. You need to shuffle your data after each epoch because you will always have the risk to create batches that are not representative of the overall dataset, and therefore, your estimate of the gradient will be off. (from the link I gave earlier). You can't just shuffle DataFrame one time before a training.
", "rawMarkdown": "No, that's not good. You need to shuffle your data after each epoch because you will always have the risk to create batches that are not representative of the overall dataset, and therefore, your estimate of the gradient will be off. (from the link I gave earlier). You can't just shuffle DataFrame one time before a training.", "votes": null }, { "id": "1298928", "postDate": "05/09/2021 10:28:53", "content": "Thanks, I will make changes to the code based on your feedback.
", "rawMarkdown": "Thanks, I will make changes to the code based on your feedback.", "votes": null } ], "comments": [ { "id": 1298786, "author_name": "atamazian", "author_url": "", "post_date": "05/09/2021 07:49:48", "content": "Can you share the training notebook as well? It's hard to tell what's wrong without it.
", "votes": null, "replies": [ { "id": 1298793, "author_name": "rustyelectron", "author_url": "", "post_date": "05/09/2021 08:01:52", "content": "ya sure, here it is.
", "votes": null, "replies": [] }, { "id": 1298799, "author_name": "atamazian", "author_url": "", "post_date": "05/09/2021 08:13:50", "content": "First thing I noticed is that you don't do shuffle in your training dataloader, which is not good. (read more here https://datascience.stackexchange.com/questions/24511/why-should-the-data-be-shuffled-for-machine-learning-tasks )
", "votes": null, "replies": [] }, { "id": 1298806, "author_name": "atamazian", "author_url": "", "post_date": "05/09/2021 08:20:22", "content": "Second, your inference is quite strange. Why do you the same thing (inference) several times? If you want to do TTA, it's not how it's done. You need to augment your test data, and then average your predictions.
", "votes": null, "replies": [] }, { "id": 1298825, "author_name": "rustyelectron", "author_url": "", "post_date": "05/09/2021 08:42:34", "content": "Ok, I will add shuffling, btw is it okay if I shuffle the pandas dataframe using df.sample(frac=1) instead of shuffling at the DataLoader as it consumes extra GPU memory or is there a better way?
Thanks for the suggestion about TTA, I didn't understand it before. I get it now and i'll fix it.
\nIs there anything else in my notebook that might be responsible for the bad score?
", "votes": null, "replies": [] }, { "id": 1298834, "author_name": "atamazian", "author_url": "", "post_date": "05/09/2021 08:53:16", "content": "No, that's not good. You need to shuffle your data after each epoch because you will always have the risk to create batches that are not representative of the overall dataset, and therefore, your estimate of the gradient will be off. (from the link I gave earlier). You can't just shuffle DataFrame one time before a training.
", "votes": null, "replies": [] }, { "id": 1298928, "author_name": "rustyelectron", "author_url": "", "post_date": "05/09/2021 10:28:53", "content": "Thanks, I will make changes to the code based on your feedback.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1298344": "Hi, I am new to kaggle. I am using resnet200d with all layers unfrozen. I ran the training for about 100 epochs and was able to get an F1 score of 90+ on train set and 80+ on validation but even then I get a public score of only 0.177. I have tried training on efficientnet-b3 as well but I am still getting the same score. I am probably doing something wrong maybe in the inference or while preparing the submission.csv. This is the [notebook](https://www.kaggle.com/rustyelectron/plant-path-inference-resnet200d). I shall be very grateful if someone can help me figure this out. I have been seeing the number 0.177 all week 😐.", "1298786": "Can you share the training notebook as well? It's hard to tell what's wrong without it.", "1298793": "ya sure, [here](https://www.kaggle.com/rustyelectron/plant-path-train-resnet200d-288x288) it is.", "1298799": "First thing I noticed is that you don't do shuffle in your training dataloader, which is not good. (read more here https://datascience.stackexchange.com/questions/24511/why-should-the-data-be-shuffled-for-machine-learning-tasks )", "1298806": "Second, your inference is quite strange. Why do you the same thing (inference) several times? If you want to do TTA, it's not how it's done. You need to augment your test data, and then average your predictions.", "1298825": "Ok, I will add shuffling, btw is it okay if I shuffle the pandas dataframe using `df.sample(frac=1)` instead of shuffling at the DataLoader as it consumes extra GPU memory or is there a better way? \n\nThanks for the suggestion about TTA, I didn't understand it before. I get it now and i'll fix it.\n\nIs there anything else in my notebook that might be responsible for the bad score?", "1298834": "No, that's not good. You need to shuffle your data after each epoch because you will always have the risk to create batches that are not representative of the overall dataset, and therefore, your estimate of the gradient will be off. (from the link I gave earlier). You can't just shuffle DataFrame one time before a training.", "1298928": "Thanks, I will make changes to the code based on your feedback." }, "source": "meta" }