Hello,
\nIn all the kaggle competitions so far, I've heavily relied on public kernels. Now, I'm trying to write my own code. I've started out with fastai, and I'm using this notebook by @thedrcat as a reference. When I run the notebook by thedrcat, I get a score above 0.35. But, my notebook, which is almost the same, gives a very low score ~0.1. The only major changes are that I'm using resnet50 from the timm library and pip installing iterative-stratification directly. I don't understand why there is such a big difference in submission scores. I've been trying to fix this for the past few days without any success. Any help would be appreciated.
My training notebook: https://www.kaggle.com/novice03/fastai-training-with-timm
\nMy inference notebook: https://www.kaggle.com/novice03/fastai-inference
Edit: notebooks privated as issue fixed (thanks @imakarov)
", "messages": [ { "id": 1279364, "postDate": "2021-04-20T20:03:04.163Z", "content": "It could be that fastai treats the classes in the get_y functions of your DataBlock as strings and sorts them as strings accordingly. Which is why the order of the predicted confidences in not 0 through 18 but 0, 1, 10, 11… instead. To try to solve this issue you can tell fastai how to properly sort your labels: you can either provide a list of classes in MultiCategoryBlock(vocab=[str(i) for i in range(19)])or by specifying a different get_y function in the DataBlock like get_y=lambda x: [int(i) for i in x['image_labels']]
Ohh, this must be the problem. I completely overlooked this. Thank you so much for your help!
", "rawMarkdown": "Ohh, this must be the problem. I completely overlooked this. Thank you so much for your help!", "votes": 1 } ] }, { "id": 1278950, "postDate": "2021-04-20T12:52:43.273Z", "content": "Hi. What is sample_stats in your training notebook?
\nIf your resnet50 is pretrained by ImageNet, I guess you have to use these mean & std while training and inference;
\nmean = [0.485, 0.456, 0.406]
\nstd = [0.229, 0.224, 0.225]
\nActually I'm not familiar with fastai. Sorry if it's an incorrect suggestion.
Hello, sample_stats is the same as in this notebook (https://www.kaggle.com/thedrcat/fastai-cell-tile-prototyping-training). It is the mean and std calculated as an average across all cell images in the training set that is used in the notebook. Could using different normalization make such a big difference in scores?
", "rawMarkdown": "Hello, sample_stats is the same as in this notebook (https://www.kaggle.com/thedrcat/fastai-cell-tile-prototyping-training). It is the mean and std calculated as an average across all cell images in the training set that is used in the notebook. Could using different normalization make such a big difference in scores?", "votes": 1 }, { "id": 1278956, "postDate": "2021-04-20T13:01:18.993Z", "content": "I will run it again with imagenet stats and see if that makes a difference.
", "rawMarkdown": "I will run it again with imagenet stats and see if that makes a difference.", "votes": 1 }, { "id": 1278958, "postDate": "2021-04-20T13:02:16.190Z", "content": "I see.
\nIn my understanding, we have to use the same stats between training and inference while normalization.
\nIf you use the same ones between them, it's not the problem.
Normally that's true, because you want to preserve whatever the network learned about the color channels. But in this case because of the domain shift this should not matter, so it should be ok to normalize with the stats from the current dataset.
\nI don't know where the difference in scores is coming from.
", "rawMarkdown": "Normally that's true, because you want to preserve whatever the network learned about the color channels. But in this case because of the domain shift this should not matter, so it should be ok to normalize with the stats from the current dataset. \n\nI don't know where the difference in scores is coming from. ", "votes": 3 }, { "id": 1279180, "postDate": "2021-04-20T17:07:50.917Z", "content": "Yea, I'm going crazy trying to figure out what's wrong.
", "rawMarkdown": "Yea, I'm going crazy trying to figure out what's wrong.", "votes": 1 }, { "id": 1279195, "postDate": "2021-04-20T17:21:20.897Z", "content": "Discovering what is wrong when you expect something to be working leads either to obvious bugs or big insights - wishing you the latter! :)
", "rawMarkdown": "Discovering what is wrong when you expect something to be working leads either to obvious bugs or big insights - wishing you the latter! :) ", "votes": 2 } ] }, { "id": 1278635, "postDate": "2021-04-20T06:15:27.453Z", "content": "Hello,
\nIn all the kaggle competitions so far, I've heavily relied on public kernels. Now, I'm trying to write my own code. I've started out with fastai, and I'm using this notebook by @thedrcat as a reference. When I run the notebook by thedrcat, I get a score above 0.35. But, my notebook, which is almost the same, gives a very low score ~0.1. The only major changes are that I'm using resnet50 from the timm library and pip installing iterative-stratification directly. I don't understand why there is such a big difference in submission scores. I've been trying to fix this for the past few days without any success. Any help would be appreciated.
My training notebook: https://www.kaggle.com/novice03/fastai-training-with-timm
\nMy inference notebook: https://www.kaggle.com/novice03/fastai-inference
Edit: notebooks privated as issue fixed (thanks @imakarov)
", "rawMarkdown": "Hello, \nIn all the kaggle competitions so far, I've heavily relied on public kernels. Now, I'm trying to write my own code. I've started out with fastai, and I'm using [this](https://www.kaggle.com/thedrcat/fastai-cell-tile-prototyping-training) notebook by @thedrcat as a reference. When I run the notebook by thedrcat, I get a score above 0.35. But, my notebook, which is almost the same, gives a very low score ~0.1. The only major changes are that I'm using resnet50 from the timm library and pip installing iterative-stratification directly. I don't understand why there is such a big difference in submission scores. I've been trying to fix this for the past few days without any success. Any help would be appreciated.\n\nMy training notebook: https://www.kaggle.com/novice03/fastai-training-with-timm\nMy inference notebook: https://www.kaggle.com/novice03/fastai-inference\n\nEdit: notebooks privated as issue fixed (thanks @imakarov) ", "votes": 2 } ], "comments": [ { "id": 1279364, "author_name": "Ilya Makarov", "author_url": "", "post_date": "2021-04-20T20:03:04.163000", "content": "It could be that fastai treats the classes in the get_y functions of your DataBlock as strings and sorts them as strings accordingly. Which is why the order of the predicted confidences in not 0 through 18 but 0, 1, 10, 11… instead. To try to solve this issue you can tell fastai how to properly sort your labels: you can either provide a list of classes in MultiCategoryBlock(vocab=[str(i) for i in range(19)])or by specifying a different get_y function in the DataBlock like get_y=lambda x: [int(i) for i in x['image_labels']]
Ohh, this must be the problem. I completely overlooked this. Thank you so much for your help!
", "votes": 1, "replies": [] } ] }, { "id": 1278950, "author_name": "cool_rabbit", "author_url": "", "post_date": "2021-04-20T12:52:43.273000", "content": "Hi. What is sample_stats in your training notebook?
\nIf your resnet50 is pretrained by ImageNet, I guess you have to use these mean & std while training and inference;
\nmean = [0.485, 0.456, 0.406]
\nstd = [0.229, 0.224, 0.225]
\nActually I'm not familiar with fastai. Sorry if it's an incorrect suggestion.
Hello, sample_stats is the same as in this notebook (https://www.kaggle.com/thedrcat/fastai-cell-tile-prototyping-training). It is the mean and std calculated as an average across all cell images in the training set that is used in the notebook. Could using different normalization make such a big difference in scores?
", "votes": 1, "replies": [] }, { "id": 1278956, "author_name": "novice03", "author_url": "", "post_date": "2021-04-20T13:01:18.993000", "content": "I will run it again with imagenet stats and see if that makes a difference.
", "votes": 1, "replies": [] }, { "id": 1278958, "author_name": "cool_rabbit", "author_url": "", "post_date": "2021-04-20T13:02:16.190000", "content": "I see.
\nIn my understanding, we have to use the same stats between training and inference while normalization.
\nIf you use the same ones between them, it's not the problem.
Normally that's true, because you want to preserve whatever the network learned about the color channels. But in this case because of the domain shift this should not matter, so it should be ok to normalize with the stats from the current dataset.
\nI don't know where the difference in scores is coming from.
", "votes": 3, "replies": [] }, { "id": 1279180, "author_name": "novice03", "author_url": "", "post_date": "2021-04-20T17:07:50.917000", "content": "Yea, I'm going crazy trying to figure out what's wrong.
", "votes": 1, "replies": [] }, { "id": 1279195, "author_name": "Darek Kłeczek", "author_url": "", "post_date": "2021-04-20T17:21:20.897000", "content": "Discovering what is wrong when you expect something to be working leads either to obvious bugs or big insights - wishing you the latter! :)
", "votes": 2, "replies": [] } ] } ], "raw_markdown_by_id": { "1279364": "It could be that fastai treats the classes in the `get_y` functions of your `DataBlock` as strings and sorts them as strings accordingly. Which is why the order of the predicted confidences in not 0 through 18 but 0, 1, 10, 11... instead. To try to solve this issue you can tell fastai how to properly sort your labels: you can either provide a list of classes in `MultiCategoryBlock(vocab=[str(i) for i in range(19)])`or by specifying a different get_y function in the DataBlock like `get_y=lambda x: [int(i) for i in x['image_labels']]`", "1278950": "Hi. What is sample_stats in your training notebook?\nIf your resnet50 is pretrained by ImageNet, I guess you have to use these mean & std while training and inference;\nmean = [0.485, 0.456, 0.406]\nstd = [0.229, 0.224, 0.225]\nActually I'm not familiar with fastai. Sorry if it's an incorrect suggestion.", "1278635": "Hello, \nIn all the kaggle competitions so far, I've heavily relied on public kernels. Now, I'm trying to write my own code. I've started out with fastai, and I'm using [this](https://www.kaggle.com/thedrcat/fastai-cell-tile-prototyping-training) notebook by @thedrcat as a reference. When I run the notebook by thedrcat, I get a score above 0.35. But, my notebook, which is almost the same, gives a very low score ~0.1. The only major changes are that I'm using resnet50 from the timm library and pip installing iterative-stratification directly. I don't understand why there is such a big difference in submission scores. I've been trying to fix this for the past few days without any success. Any help would be appreciated.\n\nMy training notebook: https://www.kaggle.com/novice03/fastai-training-with-timm\nMy inference notebook: https://www.kaggle.com/novice03/fastai-inference\n\nEdit: notebooks privated as issue fixed (thanks @imakarov) " } }