{
  "id": 296221,
  "title": "*Notebook threw exception*,help ༼ つ ◕_◕ ༽つ",
  "url": "/competitions/sartorius-cell-instance-segmentation/discussion/296221",
  "author_name": "",
  "post_date": "2021-12-20T13:35:51.315283700Z",
  "votes": 2,
  "comment_count": 5,
  "views": 0,
  "content": "<h2>Defining the variables</h2>\n<pre><code>TEST_IMGS_PATH = \"../input/sartorius-cell-instance-segmentation/test/\"\n\nRESNET_MEAN = (0.485, 0.456, 0.406)\nRESNET_STD = (0.229, 0.224, 0.225)\n\nDEVICE = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nprint(\"Using : \",DEVICE)\n</code></pre>\n<h2>Loading the model</h2>\n<pre><code>model = smp.DeepLabV3Plus(\"resnet34\")\n\nif torch.cuda.is_available():\n    model.load_state_dict(torch.load(\"../input/deeplabv3plus-resnet34-sartorius/best_model.pth\"))\nelse:\n    model.load_state_dict(torch.load(\"../input/deeplabv3plus-resnet34-sartorius/best_model.pth\",map_location=torch.device('cpu')))\nmodel.to(DEVICE)\n</code></pre>\n<h2>Helper Functions</h2>\n<pre><code>def read_img(path,resize_shape=(512,512)):\n    img=cv2.cvtColor( cv2.imread(path),cv2.COLOR_BGR2RGB)\n    img=cv2.resize(img,resize_shape)\n    return img.astype(np.double)\n\ndef remove_overlapping_pixels(mask, other_masks):\n    for other_mask in other_masks:\n        if np.sum(np.logical_and(mask, other_mask)) &gt; 0:\n            mask[np.logical_and(mask, other_mask)] = 0\n    return mask\n\ndef preds_postprocess(mask,threshold=0.5,min_size=300):\n    mask=cv2.threshold(mask,threshold,1,cv2.THRESH_BINARY)[1]\n    n_component, component = cv2.connectedComponents(mask.astype(np.uint8))\n    predictions=[]\n    for c in range(1,n_component):\n        p = (component == c)\n        if p.sum() &gt; min_size:\n            a_prediction = np.zeros((520, 704), np.float32)\n            a_prediction[p] = 1\n            predictions.append(a_prediction)\n    return predictions\n\ntest_transforms = Compose([Resize(512,512),Normalize(mean=RESNET_MEAN,std=RESNET_STD), ToTensorV2()])\n</code></pre>\n<h2><code>Predicting Loop</code></h2>\n<pre><code>pred_list=[]\nmodel.eval()\nfor idx,img_name in enumerate (os.listdir(TEST_IMGS_PATH)):\n    id=img_name.split(\".png\")[0]\n    image=read_img(TEST_IMGS_PATH +img_name)\n    image=test_transforms(image=image)[\"image\"].to(DEVICE)\n    single_mask_pred=model(image.unsqueeze(0)).cpu().detach().squeeze(0).numpy().reshape(512,512)\n\n\n    single_mask_pred=cv2.resize(single_mask_pred,(704,520),interpolation = cv2.INTER_AREA)\n\n    img_preds=preds_postprocess(single_mask_pred)\n\n    masks=[]\n    gc.collect()\n    for img_pred in img_preds:\n        fixed_mask=remove_overlapping_pixels(img_pred,masks)\n        masks.append(fixed_mask)\n        pred_list.append((str(id),rle_encode(fixed_mask)))\n        gc.collect()\n\n\nsub_df = pd.DataFrame(pred_list,columns=['id','predicted']) \n</code></pre>\n<h4>When i run it on the public test set everything works perfectly,so pprobably there's cases of input that im not handling</h4>\n<p><img src=\"https://i.imgur.com/cOO0w3b.png\" alt=\"OUTPUT\"></p>\n<h2>Thanks to everyone in advance ^^</h2>",
  "messages": [
    {
      "id": "1624026",
      "postDate": "12/20/2021 13:35:51",
      "content": "<h2>Defining the variables</h2>\n<pre><code>TEST_IMGS_PATH = \"../input/sartorius-cell-instance-segmentation/test/\"\n\nRESNET_MEAN = (0.485, 0.456, 0.406)\nRESNET_STD = (0.229, 0.224, 0.225)\n\nDEVICE = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nprint(\"Using : \",DEVICE)\n</code></pre>\n<h2>Loading the model</h2>\n<pre><code>model = smp.DeepLabV3Plus(\"resnet34\")\n\nif torch.cuda.is_available():\n    model.load_state_dict(torch.load(\"../input/deeplabv3plus-resnet34-sartorius/best_model.pth\"))\nelse:\n    model.load_state_dict(torch.load(\"../input/deeplabv3plus-resnet34-sartorius/best_model.pth\",map_location=torch.device('cpu')))\nmodel.to(DEVICE)\n</code></pre>\n<h2>Helper Functions</h2>\n<pre><code>def read_img(path,resize_shape=(512,512)):\n    img=cv2.cvtColor( cv2.imread(path),cv2.COLOR_BGR2RGB)\n    img=cv2.resize(img,resize_shape)\n    return img.astype(np.double)\n\ndef remove_overlapping_pixels(mask, other_masks):\n    for other_mask in other_masks:\n        if np.sum(np.logical_and(mask, other_mask)) &gt; 0:\n            mask[np.logical_and(mask, other_mask)] = 0\n    return mask\n\ndef preds_postprocess(mask,threshold=0.5,min_size=300):\n    mask=cv2.threshold(mask,threshold,1,cv2.THRESH_BINARY)[1]\n    n_component, component = cv2.connectedComponents(mask.astype(np.uint8))\n    predictions=[]\n    for c in range(1,n_component):\n        p = (component == c)\n        if p.sum() &gt; min_size:\n            a_prediction = np.zeros((520, 704), np.float32)\n            a_prediction[p] = 1\n            predictions.append(a_prediction)\n    return predictions\n\ntest_transforms = Compose([Resize(512,512),Normalize(mean=RESNET_MEAN,std=RESNET_STD), ToTensorV2()])\n</code></pre>\n<h2><code>Predicting Loop</code></h2>\n<pre><code>pred_list=[]\nmodel.eval()\nfor idx,img_name in enumerate (os.listdir(TEST_IMGS_PATH)):\n    id=img_name.split(\".png\")[0]\n    image=read_img(TEST_IMGS_PATH +img_name)\n    image=test_transforms(image=image)[\"image\"].to(DEVICE)\n    single_mask_pred=model(image.unsqueeze(0)).cpu().detach().squeeze(0).numpy().reshape(512,512)\n\n\n    single_mask_pred=cv2.resize(single_mask_pred,(704,520),interpolation = cv2.INTER_AREA)\n\n    img_preds=preds_postprocess(single_mask_pred)\n\n    masks=[]\n    gc.collect()\n    for img_pred in img_preds:\n        fixed_mask=remove_overlapping_pixels(img_pred,masks)\n        masks.append(fixed_mask)\n        pred_list.append((str(id),rle_encode(fixed_mask)))\n        gc.collect()\n\n\nsub_df = pd.DataFrame(pred_list,columns=['id','predicted']) \n</code></pre>\n<h4>When i run it on the public test set everything works perfectly,so pprobably there's cases of input that im not handling</h4>\n<p><img src=\"https://i.imgur.com/cOO0w3b.png\" alt=\"OUTPUT\"></p>\n<h2>Thanks to everyone in advance ^^</h2>",
      "rawMarkdown": "## Defining the variables\n```\nTEST_IMGS_PATH = \"../input/sartorius-cell-instance-segmentation/test/\"\n\nRESNET_MEAN = (0.485, 0.456, 0.406)\nRESNET_STD = (0.229, 0.224, 0.225)\n\nDEVICE = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nprint(\"Using : \",DEVICE)\n```\n\n## Loading the model\n```\nmodel = smp.DeepLabV3Plus(\"resnet34\")\n\nif torch.cuda.is_available():\n    model.load_state_dict(torch.load(\"../input/deeplabv3plus-resnet34-sartorius/best_model.pth\"))\nelse:\n    model.load_state_dict(torch.load(\"../input/deeplabv3plus-resnet34-sartorius/best_model.pth\",map_location=torch.device('cpu')))\nmodel.to(DEVICE)\n```\n## Helper Functions\n```\ndef read_img(path,resize_shape=(512,512)):\n    img=cv2.cvtColor( cv2.imread(path),cv2.COLOR_BGR2RGB)\n    img=cv2.resize(img,resize_shape)\n    return img.astype(np.double)\n\ndef remove_overlapping_pixels(mask, other_masks):\n    for other_mask in other_masks:\n        if np.sum(np.logical_and(mask, other_mask)) > 0:\n            mask[np.logical_and(mask, other_mask)] = 0\n    return mask\n\ndef preds_postprocess(mask,threshold=0.5,min_size=300):\n    mask=cv2.threshold(mask,threshold,1,cv2.THRESH_BINARY)[1]\n    n_component, component = cv2.connectedComponents(mask.astype(np.uint8))\n    predictions=[]\n    for c in range(1,n_component):\n        p = (component == c)\n        if p.sum() > min_size:\n            a_prediction = np.zeros((520, 704), np.float32)\n            a_prediction[p] = 1\n            predictions.append(a_prediction)\n    return predictions\n\ntest_transforms = Compose([Resize(512,512),Normalize(mean=RESNET_MEAN,std=RESNET_STD), ToTensorV2()])\n```\n## `Predicting Loop`\n\n```\npred_list=[]\nmodel.eval()\nfor idx,img_name in enumerate (os.listdir(TEST_IMGS_PATH)):\n    id=img_name.split(\".png\")[0]\n    image=read_img(TEST_IMGS_PATH +img_name)\n    image=test_transforms(image=image)[\"image\"].to(DEVICE)\n    single_mask_pred=model(image.unsqueeze(0)).cpu().detach().squeeze(0).numpy().reshape(512,512)\n    \n    \n    single_mask_pred=cv2.resize(single_mask_pred,(704,520),interpolation = cv2.INTER_AREA)\n    \n    img_preds=preds_postprocess(single_mask_pred)\n    \n    masks=[]\n    gc.collect()\n    for img_pred in img_preds:\n        fixed_mask=remove_overlapping_pixels(img_pred,masks)\n        masks.append(fixed_mask)\n        pred_list.append((str(id),rle_encode(fixed_mask)))\n        gc.collect()\n        \n    \nsub_df = pd.DataFrame(pred_list,columns=['id','predicted']) \n```\n\n#### When i run it on the public test set everything works perfectly,so pprobably there's cases of input that im not handling\n![OUTPUT](https://i.imgur.com/cOO0w3b.png)\n## Thanks to everyone in advance ^^",
      "votes": null
    },
    {
      "id": "1630063",
      "postDate": "12/26/2021 23:37:53",
      "content": "<p>I recommend you use a much larger dataset as a fake test dataset to test your program, which serves as a more reresentative way.</p>",
      "rawMarkdown": "I recommend you use a much larger dataset as a fake test dataset to test your program, which serves as a more reresentative way.",
      "votes": null
    },
    {
      "id": "1630128",
      "postDate": "12/27/2021 01:39:02",
      "content": "<p>Thanks for the reply ^^ . then i'll test it by continuosly testing on the train data i guess. I'll try :)</p>",
      "rawMarkdown": "Thanks for the reply ^^ . then i'll test it by continuosly testing on the train data i guess. I'll try :)",
      "votes": null
    },
    {
      "id": "1630971",
      "postDate": "12/27/2021 22:13:32",
      "content": "<p>What kind of error did you get, exactly? </p>",
      "rawMarkdown": "What kind of error did you get, exactly?",
      "votes": null
    },
    {
      "id": "1630981",
      "postDate": "12/27/2021 22:46:07",
      "content": "<p>Kaggle won't tell you, it's just \"Notebook threw exception\"</p>",
      "rawMarkdown": "Kaggle won't tell you, it's just \"Notebook threw exception\"",
      "votes": null
    },
    {
      "id": "1632792",
      "postDate": "12/30/2021 04:33:06",
      "content": "<p>Maybe there are other formats than png.</p>",
      "rawMarkdown": "Maybe there are other formats than png.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1630063,
      "author_name": "rkxfrom2021",
      "author_url": "",
      "post_date": "12/26/2021 23:37:53",
      "content": "<p>I recommend you use a much larger dataset as a fake test dataset to test your program, which serves as a more reresentative way.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1630128,
          "author_name": "albertozorzetto",
          "author_url": "",
          "post_date": "12/27/2021 01:39:02",
          "content": "<p>Thanks for the reply ^^ . then i'll test it by continuosly testing on the train data i guess. I'll try :)</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1630971,
      "author_name": "ivanpan",
      "author_url": "",
      "post_date": "12/27/2021 22:13:32",
      "content": "<p>What kind of error did you get, exactly? </p>",
      "votes": null,
      "replies": [
        {
          "id": 1630981,
          "author_name": "albertozorzetto",
          "author_url": "",
          "post_date": "12/27/2021 22:46:07",
          "content": "<p>Kaggle won't tell you, it's just \"Notebook threw exception\"</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1632792,
      "author_name": "junxhuang",
      "author_url": "",
      "post_date": "12/30/2021 04:33:06",
      "content": "<p>Maybe there are other formats than png.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1624026": "## Defining the variables\n```\nTEST_IMGS_PATH = \"../input/sartorius-cell-instance-segmentation/test/\"\n\nRESNET_MEAN = (0.485, 0.456, 0.406)\nRESNET_STD = (0.229, 0.224, 0.225)\n\nDEVICE = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nprint(\"Using : \",DEVICE)\n```\n\n## Loading the model\n```\nmodel = smp.DeepLabV3Plus(\"resnet34\")\n\nif torch.cuda.is_available():\n    model.load_state_dict(torch.load(\"../input/deeplabv3plus-resnet34-sartorius/best_model.pth\"))\nelse:\n    model.load_state_dict(torch.load(\"../input/deeplabv3plus-resnet34-sartorius/best_model.pth\",map_location=torch.device('cpu')))\nmodel.to(DEVICE)\n```\n## Helper Functions\n```\ndef read_img(path,resize_shape=(512,512)):\n    img=cv2.cvtColor( cv2.imread(path),cv2.COLOR_BGR2RGB)\n    img=cv2.resize(img,resize_shape)\n    return img.astype(np.double)\n\ndef remove_overlapping_pixels(mask, other_masks):\n    for other_mask in other_masks:\n        if np.sum(np.logical_and(mask, other_mask)) > 0:\n            mask[np.logical_and(mask, other_mask)] = 0\n    return mask\n\ndef preds_postprocess(mask,threshold=0.5,min_size=300):\n    mask=cv2.threshold(mask,threshold,1,cv2.THRESH_BINARY)[1]\n    n_component, component = cv2.connectedComponents(mask.astype(np.uint8))\n    predictions=[]\n    for c in range(1,n_component):\n        p = (component == c)\n        if p.sum() > min_size:\n            a_prediction = np.zeros((520, 704), np.float32)\n            a_prediction[p] = 1\n            predictions.append(a_prediction)\n    return predictions\n\ntest_transforms = Compose([Resize(512,512),Normalize(mean=RESNET_MEAN,std=RESNET_STD), ToTensorV2()])\n```\n## `Predicting Loop`\n\n```\npred_list=[]\nmodel.eval()\nfor idx,img_name in enumerate (os.listdir(TEST_IMGS_PATH)):\n    id=img_name.split(\".png\")[0]\n    image=read_img(TEST_IMGS_PATH +img_name)\n    image=test_transforms(image=image)[\"image\"].to(DEVICE)\n    single_mask_pred=model(image.unsqueeze(0)).cpu().detach().squeeze(0).numpy().reshape(512,512)\n    \n    \n    single_mask_pred=cv2.resize(single_mask_pred,(704,520),interpolation = cv2.INTER_AREA)\n    \n    img_preds=preds_postprocess(single_mask_pred)\n    \n    masks=[]\n    gc.collect()\n    for img_pred in img_preds:\n        fixed_mask=remove_overlapping_pixels(img_pred,masks)\n        masks.append(fixed_mask)\n        pred_list.append((str(id),rle_encode(fixed_mask)))\n        gc.collect()\n        \n    \nsub_df = pd.DataFrame(pred_list,columns=['id','predicted']) \n```\n\n#### When i run it on the public test set everything works perfectly,so pprobably there's cases of input that im not handling\n![OUTPUT](https://i.imgur.com/cOO0w3b.png)\n## Thanks to everyone in advance ^^",
    "1630063": "I recommend you use a much larger dataset as a fake test dataset to test your program, which serves as a more reresentative way.",
    "1630128": "Thanks for the reply ^^ . then i'll test it by continuosly testing on the train data i guess. I'll try :)",
    "1630971": "What kind of error did you get, exactly?",
    "1630981": "Kaggle won't tell you, it's just \"Notebook threw exception\"",
    "1632792": "Maybe there are other formats than png."
  },
  "source": "meta"
}