{ "id": 116570, "title": "Questions regarding qubvel/segmentation_models", "url": "/competitions/understanding_cloud_organization/discussion/116570", "author_name": "", "post_date": "2019-11-10T02:49:08.802534900Z", "votes": 1, "comment_count": 5, "views": 0, "content": "

I am trying to understand a strange behaviour during training. In this scenario, I am using the exact same training and validation data in this model (so would expect loss to be very similar as they are essentially the same data):

\n\n

\nsm.FPN(\n backbone=\"resnet18\",\n classes=classes,\n input_shape=input_shape,\n activation=\"sigmoid\",\n encoder_freeze=true,\n )\n\nloss function and metrics are:\n\nfrom segmentation_models.losses import bce_jaccard_loss\nfrom segmentation_models.metrics import f_score\n

\n\n

However the training and validation scores are very different:\n\"\"

\n\n

I have tested it with a simple model (same loss, metric, train and validation data)\n\nfrom keras.models import Model\nfrom keras.layers import Input, Dense, Conv2D\na = Input(shape=(320, 480, 3))\nb = Conv2D(1, (1, 1))(a)\nmodel = Model(inputs=a, outputs=b)\n\nIn this case the train and validation data behave as expected

\n\n

\"\"

\n\n

Please can you help me understand why the train and validation metrics can be so different in the sm models?

\n\n

Thanks!

", "messages": [ { "id": "669414", "postDate": "11/10/2019 02:49:08", "content": "

I am trying to understand a strange behaviour during training. In this scenario, I am using the exact same training and validation data in this model (so would expect loss to be very similar as they are essentially the same data):

\n\n

\nsm.FPN(\n backbone=\"resnet18\",\n classes=classes,\n input_shape=input_shape,\n activation=\"sigmoid\",\n encoder_freeze=true,\n )\n\nloss function and metrics are:\n\nfrom segmentation_models.losses import bce_jaccard_loss\nfrom segmentation_models.metrics import f_score\n

\n\n

However the training and validation scores are very different:\n\"\"

\n\n

I have tested it with a simple model (same loss, metric, train and validation data)\n\nfrom keras.models import Model\nfrom keras.layers import Input, Dense, Conv2D\na = Input(shape=(320, 480, 3))\nb = Conv2D(1, (1, 1))(a)\nmodel = Model(inputs=a, outputs=b)\n\nIn this case the train and validation data behave as expected

\n\n

\"\"

\n\n

Please can you help me understand why the train and validation metrics can be so different in the sm models?

\n\n

Thanks!

", "rawMarkdown": "I am trying to understand a strange behaviour during training. In this scenario, I am using the exact same training and validation data in this model (so would expect loss to be very similar as they are essentially the same data):\n\n```\nsm.FPN(\n backbone=\"resnet18\",\n classes=classes,\n input_shape=input_shape,\n activation=\"sigmoid\",\n encoder_freeze=true,\n )\n```\nloss function and metrics are:\n```\nfrom segmentation_models.losses import bce_jaccard_loss\nfrom segmentation_models.metrics import f_score\n```\n\nHowever the training and validation scores are very different:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2106773%2F7ccb4a7d30acca4daf9fec724ea30cc9%2F1.png?generation=1573353961247701&alt=media)\n\n\nI have tested it with a simple model (same loss, metric, train and validation data)\n```\nfrom keras.models import Model\nfrom keras.layers import Input, Dense, Conv2D\na = Input(shape=(320, 480, 3))\nb = Conv2D(1, (1, 1))(a)\nmodel = Model(inputs=a, outputs=b)\n```\nIn this case the train and validation data behave as expected\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2106773%2Fa74950c8e67e8a00e0b3c7253aa9c653%2F2.png?generation=1573353980773341&alt=media)\n\n\nPlease can you help me understand why the train and validation metrics can be so different in the sm models?\n\nThanks!", "votes": null }, { "id": "669423", "postDate": "11/10/2019 03:20:29", "content": "

Can you post a screenshot here about the train epoch between 1 and 10 ?\nAugmentations also .

", "rawMarkdown": "Can you post a screenshot here about the train epoch between ` 1 and 10 ` ?\nAugmentations also .", "votes": null }, { "id": "669720", "postDate": "11/10/2019 11:11:09", "content": "

Thanks, see below the epochs 1-10 under segmentation model. I am not using any augmentations:

\n\n

\"\"

", "rawMarkdown": "Thanks, see below the epochs 1-10 under segmentation model. I am not using any augmentations:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2106773%2F7a8a7dc748008b2126ff6578e91a2e54%2F3.png?generation=1573384261375666&alt=media)", "votes": null }, { "id": "669783", "postDate": "11/10/2019 13:26:29", "content": "

I am not familiar with qubvel/segmentation_models,

\n\n
\n

a = Input(shape=(320, 480, 3))\n b = Conv2D(1, (1, 1))(a)\n model = Model(inputs=a, outputs=b)\n In this case the train and validation data behave as expected

\n
\n\n

similar to above \"train_data = val_data = single sample\" case, maybe can use single sample to train/validate the SM model, should get zero loss for both train and validation if the training pipeline is correct.

", "rawMarkdown": "I am not familiar with qubvel/segmentation_models, \n\n> a = Input(shape=(320, 480, 3))\nb = Conv2D(1, (1, 1))(a)\nmodel = Model(inputs=a, outputs=b)\nIn this case the train and validation data behave as expected\n\nsimilar to above \"train_data = val_data = single sample\" case, maybe can use single sample to train/validate the SM model, should get zero loss for both train and validation if the training pipeline is correct.", "votes": null }, { "id": "669784", "postDate": "11/10/2019 13:29:11", "content": "

indeed, it is strange, in both cases i use exact same pipeline. As you can see the simple model produces loss that rae the same for both train and validation. However for the SM model it doesnt.\nI thought it might be due to some dropout layers that are only active during train and not in validation, but it doesnt seem to be the reasons (ie, issue persists even if dropout is set to off)

", "rawMarkdown": "indeed, it is strange, in both cases i use exact same pipeline. As you can see the simple model produces loss that rae the same for both train and validation. However for the SM model it doesnt.\nI thought it might be due to some dropout layers that are only active during train and not in validation, but it doesnt seem to be the reasons (ie, issue persists even if dropout is set to off)", "votes": null }, { "id": "670020", "postDate": "11/10/2019 22:45:26", "content": "

Have you tested by setting encoder_freeze = False?

", "rawMarkdown": "Have you tested by setting encoder_freeze = False?", "votes": null } ], "comments": [ { "id": 669423, "author_name": "xujingzhao", "author_url": "", "post_date": "11/10/2019 03:20:29", "content": "

Can you post a screenshot here about the train epoch between 1 and 10 ?\nAugmentations also .

", "votes": null, "replies": [] }, { "id": 669720, "author_name": "heisenger", "author_url": "", "post_date": "11/10/2019 11:11:09", "content": "

Thanks, see below the epochs 1-10 under segmentation model. I am not using any augmentations:

\n\n

\"\"

", "votes": null, "replies": [] }, { "id": 669783, "author_name": "wschong", "author_url": "", "post_date": "11/10/2019 13:26:29", "content": "

I am not familiar with qubvel/segmentation_models,

\n\n
\n

a = Input(shape=(320, 480, 3))\n b = Conv2D(1, (1, 1))(a)\n model = Model(inputs=a, outputs=b)\n In this case the train and validation data behave as expected

\n
\n\n

similar to above \"train_data = val_data = single sample\" case, maybe can use single sample to train/validate the SM model, should get zero loss for both train and validation if the training pipeline is correct.

", "votes": null, "replies": [] }, { "id": 669784, "author_name": "heisenger", "author_url": "", "post_date": "11/10/2019 13:29:11", "content": "

indeed, it is strange, in both cases i use exact same pipeline. As you can see the simple model produces loss that rae the same for both train and validation. However for the SM model it doesnt.\nI thought it might be due to some dropout layers that are only active during train and not in validation, but it doesnt seem to be the reasons (ie, issue persists even if dropout is set to off)

", "votes": null, "replies": [] }, { "id": 670020, "author_name": "prachi1211", "author_url": "", "post_date": "11/10/2019 22:45:26", "content": "

Have you tested by setting encoder_freeze = False?

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "669414": "I am trying to understand a strange behaviour during training. In this scenario, I am using the exact same training and validation data in this model (so would expect loss to be very similar as they are essentially the same data):\n\n```\nsm.FPN(\n backbone=\"resnet18\",\n classes=classes,\n input_shape=input_shape,\n activation=\"sigmoid\",\n encoder_freeze=true,\n )\n```\nloss function and metrics are:\n```\nfrom segmentation_models.losses import bce_jaccard_loss\nfrom segmentation_models.metrics import f_score\n```\n\nHowever the training and validation scores are very different:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2106773%2F7ccb4a7d30acca4daf9fec724ea30cc9%2F1.png?generation=1573353961247701&alt=media)\n\n\nI have tested it with a simple model (same loss, metric, train and validation data)\n```\nfrom keras.models import Model\nfrom keras.layers import Input, Dense, Conv2D\na = Input(shape=(320, 480, 3))\nb = Conv2D(1, (1, 1))(a)\nmodel = Model(inputs=a, outputs=b)\n```\nIn this case the train and validation data behave as expected\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2106773%2Fa74950c8e67e8a00e0b3c7253aa9c653%2F2.png?generation=1573353980773341&alt=media)\n\n\nPlease can you help me understand why the train and validation metrics can be so different in the sm models?\n\nThanks!", "669423": "Can you post a screenshot here about the train epoch between ` 1 and 10 ` ?\nAugmentations also .", "669720": "Thanks, see below the epochs 1-10 under segmentation model. I am not using any augmentations:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2106773%2F7a8a7dc748008b2126ff6578e91a2e54%2F3.png?generation=1573384261375666&alt=media)", "669783": "I am not familiar with qubvel/segmentation_models, \n\n> a = Input(shape=(320, 480, 3))\nb = Conv2D(1, (1, 1))(a)\nmodel = Model(inputs=a, outputs=b)\nIn this case the train and validation data behave as expected\n\nsimilar to above \"train_data = val_data = single sample\" case, maybe can use single sample to train/validate the SM model, should get zero loss for both train and validation if the training pipeline is correct.", "669784": "indeed, it is strange, in both cases i use exact same pipeline. As you can see the simple model produces loss that rae the same for both train and validation. However for the SM model it doesnt.\nI thought it might be due to some dropout layers that are only active during train and not in validation, but it doesnt seem to be the reasons (ie, issue persists even if dropout is set to off)", "670020": "Have you tested by setting encoder_freeze = False?" }, "source": "meta" }