In several notebooks, (here, here, here), the validation dataset is constructed using the following dataframe:\nid_mask_count = train.loc[train['EncodedPixels'].isnull() == False, 'Image_Label'].apply(lambda x: x.split('_')[0]).value_counts().\\\nreset_index().rename(columns={'index': 'img_id', 'Image_Label': 'count'})
I used a very similar code snippet to set up k-fold cross validation and had major issues with it yesterday.\nIt turns out that value_counts doesn't handle ties in a deterministic way (see e.g. here), meaning that the validation set differs from run to run.
In several notebooks, (here, here, here), the validation dataset is constructed using the following dataframe:\nid_mask_count = train.loc[train['EncodedPixels'].isnull() == False, 'Image_Label'].apply(lambda x: x.split('_')[0]).value_counts().\\\nreset_index().rename(columns={'index': 'img_id', 'Image_Label': 'count'})
I used a very similar code snippet to set up k-fold cross validation and had major issues with it yesterday.\nIt turns out that value_counts doesn't handle ties in a deterministic way (see e.g. here), meaning that the validation set differs from run to run.
Then you could just add a sort_value to sort by count and ìmg_id in the second place to fix this:
id_mask_count = train.loc[train['EncodedPixels'].isnull() == False, 'Image_Label'].apply(lambda x: x.split('_')[0]).value_counts().\\ reset_index().rename(columns={'index': 'img_id', 'Image_Label': 'count'}).sort_values(['count', 'img_id'])
Thanks so much for this notice Welinder. This helps me.
", "rawMarkdown": "Thanks so much for this notice Welinder. This helps me.", "votes": null }, { "id": "645351", "postDate": "10/10/2019 03:07:49", "content": "Someone find a good validation strategy? I can't make my val score stay close to LB. Now I'm getting 0.581/0.626(LB)
", "rawMarkdown": "Someone find a good validation strategy? I can't make my val score stay close to LB. Now I'm getting 0.581/0.626(LB)", "votes": null }, { "id": "645372", "postDate": "10/10/2019 03:54:08", "content": "Your validation score may be calculated on smoothed dice (2*(A intersect B)+ eps)/(|A| + |B| + eps) when training, not competition dice. You need to add the original dice in your validation strategy, because for both empty ground truth and empty prediction the dice score is undefined. Then LB score will stay close to validation score.
", "rawMarkdown": "Your validation score may be calculated on smoothed dice (2*(A intersect B)+ eps)/(|A| + |B| + eps) when training, not competition dice. You need to add the original dice in your validation strategy, because for both empty ground truth and empty prediction the dice score is undefined. Then LB score will stay close to validation score.", "votes": null } ], "comments": [ { "id": 644705, "author_name": "jottbe", "author_url": "", "post_date": "10/09/2019 07:56:42", "content": "Then you could just add a sort_value to sort by count and ìmg_id in the second place to fix this:
id_mask_count = train.loc[train['EncodedPixels'].isnull() == False, 'Image_Label'].apply(lambda x: x.split('_')[0]).value_counts().\\ reset_index().rename(columns={'index': 'img_id', 'Image_Label': 'count'}).sort_values(['count', 'img_id'])
Thanks so much for this notice Welinder. This helps me.
", "votes": null, "replies": [] }, { "id": 645351, "author_name": "igormunizims", "author_url": "", "post_date": "10/10/2019 03:07:49", "content": "Someone find a good validation strategy? I can't make my val score stay close to LB. Now I'm getting 0.581/0.626(LB)
", "votes": null, "replies": [ { "id": 645372, "author_name": "khahuras", "author_url": "", "post_date": "10/10/2019 03:54:08", "content": "Your validation score may be calculated on smoothed dice (2*(A intersect B)+ eps)/(|A| + |B| + eps) when training, not competition dice. You need to add the original dice in your validation strategy, because for both empty ground truth and empty prediction the dice score is undefined. Then LB score will stay close to validation score.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "644191": "In several notebooks, ([here](https://www.kaggle.com/artgor/segmentation-in-pytorch-using-convenient-tools), [here](https://www.kaggle.com/ryches/turbo-charging-andrew-s-pytorch), [here](https://www.kaggle.com/hung96ad/fine-turning-model-with-adabound)), the validation dataset is constructed using the following dataframe:\n`id_mask_count = train.loc[train['EncodedPixels'].isnull() == False, 'Image_Label'].apply(lambda x: x.split('_')[0]).value_counts().\\\nreset_index().rename(columns={'index': 'img_id', 'Image_Label': 'count'})`\n\nI used a very similar code snippet to set up k-fold cross validation and had major issues with it yesterday.\nIt turns out that `value_counts` doesn't handle ties in a deterministic way (see e.g. [here](https://github.com/pandas-dev/pandas/issues/15833)), meaning that the validation set differs from run to run.", "644705": "Then you could just add a `sort_value` to sort by `count` and `ìmg_id` in the second place to fix this:\n\n`id_mask_count = train.loc[train['EncodedPixels'].isnull() == False, 'Image_Label'].apply(lambda x: x.split('_')[0]).value_counts().\\ reset_index().rename(columns={'index': 'img_id', 'Image_Label': 'count'}).sort_values(['count', 'img_id'])`", "645248": "Thanks so much for this notice Welinder. This helps me.", "645351": "Someone find a good validation strategy? I can't make my val score stay close to LB. Now I'm getting 0.581/0.626(LB)", "645372": "Your validation score may be calculated on smoothed dice (2*(A intersect B)+ eps)/(|A| + |B| + eps) when training, not competition dice. You need to add the original dice in your validation strategy, because for both empty ground truth and empty prediction the dice score is undefined. Then LB score will stay close to validation score." }, "source": "meta" }