Dear all,
\n\nI am looking for a way to merge the predictions of my different models.
\n\nThe models are: \n- U-Net with ResNet-34 backbone\n- LinkNet with EfficientNetB3 backbone.
\n\nBoth those architectures were trained on 5 folds, resulting in 10 models.
\n\nFor now, I have tried to intersect the ten predicted masks, a mask having the shape (batch_size, image_width, image_height, n_chanels = 4). This results in a tensor of zeros.
\n\nFollowing is the code:
\n\n\npreds = np.ones((BATCH_SIZE, *IMG_SHAPE, N_CLASSES))
\nfor modelResUnet, modelEffLinknet in zip(modelsResUnet, modelsEffLinknet):\n preds *= make_preds(modelResUnet, test_imgs, batch_idx)\n preds *= make_preds(modelEffLinknet, test_imgs, batch_idx)\npreds = np.around(preds)\n
My second attempt being to average the predictions of each model's folds and finally computing the intersection of each model's average prediction.
\n\n```\npredsResUnet = []\npredsEffLinknet = []
\n\nfor modelResUnet, modelEffLinknet in zip(modelsResUnet, modelsEffLinknet):\n predsResUnet.append(make_preds(modelResUnet, test_imgs, batch_idx))\n predsEffLinknet.append(make_preds(modelEffLinknet, test_imgs, batch_idx))
\n\npredsResUnet = np.stack(predsResUnet)\npredsResUnet = np.mean(predsResUnet, axis = 0)
\n\npredsEffLinknet = np.stack(predsEffLinknet)\npredsEffLinknet = np.mean(predsEffLinknet, axis = 0)
\n\npreds = predsResUnet * predsEffLinknet\npreds = np.around(preds)`\n```
\n\nThe LB score of this second attempt is 0.626.
\n\nI also wanted to add a classifier or a regressor on top of this 'stage', however I am not sure what would be a good way of doing it. The idea is to feed as input the predicted masks and as labels the true mask (rle decoded data).
\n\nHas someone any thoughts about this matter or would be interested in teaming up to try something similar?
\n\nP.-S.: I am aware that some things might be unclear so do not hesitate to ask for clarification.
\n\nBy the way, if someone is intersted, using only the folds from the UNet model gave a LB score of 0.616. And for the LinkNet folds a LB score of 0.602, using a code similar to second code snippet
", "messages": [ { "id": "626708", "postDate": "09/14/2019 17:56:03", "content": "Dear all,
\n\nI am looking for a way to merge the predictions of my different models.
\n\nThe models are: \n- U-Net with ResNet-34 backbone\n- LinkNet with EfficientNetB3 backbone.
\n\nBoth those architectures were trained on 5 folds, resulting in 10 models.
\n\nFor now, I have tried to intersect the ten predicted masks, a mask having the shape (batch_size, image_width, image_height, n_chanels = 4). This results in a tensor of zeros.
\n\nFollowing is the code:
\n\n\npreds = np.ones((BATCH_SIZE, *IMG_SHAPE, N_CLASSES))
\nfor modelResUnet, modelEffLinknet in zip(modelsResUnet, modelsEffLinknet):\n preds *= make_preds(modelResUnet, test_imgs, batch_idx)\n preds *= make_preds(modelEffLinknet, test_imgs, batch_idx)\npreds = np.around(preds)\n
My second attempt being to average the predictions of each model's folds and finally computing the intersection of each model's average prediction.
\n\n```\npredsResUnet = []\npredsEffLinknet = []
\n\nfor modelResUnet, modelEffLinknet in zip(modelsResUnet, modelsEffLinknet):\n predsResUnet.append(make_preds(modelResUnet, test_imgs, batch_idx))\n predsEffLinknet.append(make_preds(modelEffLinknet, test_imgs, batch_idx))
\n\npredsResUnet = np.stack(predsResUnet)\npredsResUnet = np.mean(predsResUnet, axis = 0)
\n\npredsEffLinknet = np.stack(predsEffLinknet)\npredsEffLinknet = np.mean(predsEffLinknet, axis = 0)
\n\npreds = predsResUnet * predsEffLinknet\npreds = np.around(preds)`\n```
\n\nThe LB score of this second attempt is 0.626.
\n\nI also wanted to add a classifier or a regressor on top of this 'stage', however I am not sure what would be a good way of doing it. The idea is to feed as input the predicted masks and as labels the true mask (rle decoded data).
\n\nHas someone any thoughts about this matter or would be interested in teaming up to try something similar?
\n\nP.-S.: I am aware that some things might be unclear so do not hesitate to ask for clarification.
\n\nBy the way, if someone is intersted, using only the folds from the UNet model gave a LB score of 0.616. And for the LinkNet folds a LB score of 0.602, using a code similar to second code snippet
", "rawMarkdown": "Dear all,\n\nI am looking for a way to merge the predictions of my different models.\n\nThe models are: \n- U-Net with ResNet-34 backbone\n- LinkNet with EfficientNetB3 backbone.\n\nBoth those architectures were trained on 5 folds, resulting in 10 models.\n\nFor now, I have tried to intersect the ten predicted masks, a mask having the shape (batch_size, image_width, image_height, n_chanels = 4). This results in a tensor of zeros.\n\nFollowing is the code: \n\n```\npreds = np.ones((BATCH_SIZE, *IMG_SHAPE, N_CLASSES)) \nfor modelResUnet, modelEffLinknet in zip(modelsResUnet, modelsEffLinknet):\n preds *= make_preds(modelResUnet, test_imgs, batch_idx)\n preds *= make_preds(modelEffLinknet, test_imgs, batch_idx)\npreds = np.around(preds)\n```\n\nMy second attempt being to average the predictions of each model's folds and finally computing the intersection of each model's average prediction.\n\n```\npredsResUnet = []\npredsEffLinknet = []\n \nfor modelResUnet, modelEffLinknet in zip(modelsResUnet, modelsEffLinknet):\n predsResUnet.append(make_preds(modelResUnet, test_imgs, batch_idx))\n predsEffLinknet.append(make_preds(modelEffLinknet, test_imgs, batch_idx))\n\npredsResUnet = np.stack(predsResUnet)\npredsResUnet = np.mean(predsResUnet, axis = 0)\n \npredsEffLinknet = np.stack(predsEffLinknet)\npredsEffLinknet = np.mean(predsEffLinknet, axis = 0)\n \npreds = predsResUnet * predsEffLinknet\npreds = np.around(preds)`\n```\n\nThe LB score of this second attempt is 0.626.\n\nI also wanted to add a classifier or a regressor on top of this 'stage', however I am not sure what would be a good way of doing it. The idea is to feed as input the predicted masks and as labels the true mask (rle decoded data).\n\nHas someone any thoughts about this matter or would be interested in teaming up to try something similar?\n\nP.-S.: I am aware that some things might be unclear so do not hesitate to ask for clarification.\n\nBy the way, if someone is intersted, using only the folds from the UNet model gave a LB score of 0.616. And for the LinkNet folds a LB score of 0.602, using a code similar to second code snippet", "votes": null }, { "id": "626758", "postDate": "09/14/2019 19:52:41", "content": "Oh cool! I too am interested in this topic. All the ensembles I tried are giving me less scores than the individual models. However I didn't do kfold but trained on a 80-20 split, so I have 2 models only.
", "rawMarkdown": "Oh cool! I too am interested in this topic. All the ensembles I tried are giving me less scores than the individual models. However I didn't do kfold but trained on a 80-20 split, so I have 2 models only.", "votes": null }, { "id": "626983", "postDate": "09/15/2019 08:02:19", "content": "Hi @mightyrains, interesting, maybe your models are not diverse enough, which would be a reason why it does not improve your score. I remember that with using only one UNet model with the resnet34 backbone gave me a score of 0.602 and as written above with 5 folds 0.616, so maybe if you try using k-folds you will hopefully see an improvement
", "rawMarkdown": "Hi @mightyrains, interesting, maybe your models are not diverse enough, which would be a reason why it does not improve your score. I remember that with using only one UNet model with the resnet34 backbone gave me a score of 0.602 and as written above with 5 folds 0.616, so maybe if you try using k-folds you will hopefully see an improvement", "votes": null } ], "comments": [ { "id": 626758, "author_name": "mightyrains", "author_url": "", "post_date": "09/14/2019 19:52:41", "content": "Oh cool! I too am interested in this topic. All the ensembles I tried are giving me less scores than the individual models. However I didn't do kfold but trained on a 80-20 split, so I have 2 models only.
", "votes": null, "replies": [ { "id": 626983, "author_name": "jonasfreibs", "author_url": "", "post_date": "09/15/2019 08:02:19", "content": "Hi @mightyrains, interesting, maybe your models are not diverse enough, which would be a reason why it does not improve your score. I remember that with using only one UNet model with the resnet34 backbone gave me a score of 0.602 and as written above with 5 folds 0.616, so maybe if you try using k-folds you will hopefully see an improvement
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "626708": "Dear all,\n\nI am looking for a way to merge the predictions of my different models.\n\nThe models are: \n- U-Net with ResNet-34 backbone\n- LinkNet with EfficientNetB3 backbone.\n\nBoth those architectures were trained on 5 folds, resulting in 10 models.\n\nFor now, I have tried to intersect the ten predicted masks, a mask having the shape (batch_size, image_width, image_height, n_chanels = 4). This results in a tensor of zeros.\n\nFollowing is the code: \n\n```\npreds = np.ones((BATCH_SIZE, *IMG_SHAPE, N_CLASSES)) \nfor modelResUnet, modelEffLinknet in zip(modelsResUnet, modelsEffLinknet):\n preds *= make_preds(modelResUnet, test_imgs, batch_idx)\n preds *= make_preds(modelEffLinknet, test_imgs, batch_idx)\npreds = np.around(preds)\n```\n\nMy second attempt being to average the predictions of each model's folds and finally computing the intersection of each model's average prediction.\n\n```\npredsResUnet = []\npredsEffLinknet = []\n \nfor modelResUnet, modelEffLinknet in zip(modelsResUnet, modelsEffLinknet):\n predsResUnet.append(make_preds(modelResUnet, test_imgs, batch_idx))\n predsEffLinknet.append(make_preds(modelEffLinknet, test_imgs, batch_idx))\n\npredsResUnet = np.stack(predsResUnet)\npredsResUnet = np.mean(predsResUnet, axis = 0)\n \npredsEffLinknet = np.stack(predsEffLinknet)\npredsEffLinknet = np.mean(predsEffLinknet, axis = 0)\n \npreds = predsResUnet * predsEffLinknet\npreds = np.around(preds)`\n```\n\nThe LB score of this second attempt is 0.626.\n\nI also wanted to add a classifier or a regressor on top of this 'stage', however I am not sure what would be a good way of doing it. The idea is to feed as input the predicted masks and as labels the true mask (rle decoded data).\n\nHas someone any thoughts about this matter or would be interested in teaming up to try something similar?\n\nP.-S.: I am aware that some things might be unclear so do not hesitate to ask for clarification.\n\nBy the way, if someone is intersted, using only the folds from the UNet model gave a LB score of 0.616. And for the LinkNet folds a LB score of 0.602, using a code similar to second code snippet", "626758": "Oh cool! I too am interested in this topic. All the ensembles I tried are giving me less scores than the individual models. However I didn't do kfold but trained on a 80-20 split, so I have 2 models only.", "626983": "Hi @mightyrains, interesting, maybe your models are not diverse enough, which would be a reason why it does not improve your score. I remember that with using only one UNet model with the resnet34 backbone gave me a score of 0.602 and as written above with 5 folds 0.616, so maybe if you try using k-folds you will hopefully see an improvement" }, "source": "meta" }