{
  "id": 285927,
  "title": "Reproducing Sartorius LIVECell paper results",
  "url": "/competitions/sartorius-cell-instance-segmentation/discussion/285927",
  "author_name": "",
  "post_date": "2021-11-07T02:38:04.443047500Z",
  "votes": 9,
  "comment_count": 3,
  "views": 0,
  "content": "<p>I have tried to use a CenterMask2 model, pretrained on the LIVECell 2021 data, with and without training, with very poor results despite the zero-shot learning performance stated in the <a href=\"https://www.nature.com/articles/s41592-021-01249-6\" target=\"_blank\">Sartorius LIVECell paper</a>.</p>\n<p>So, I set out to reproduce the results of the paper, i.e. taking the pretrained model and applying it to the LIVECell test set. I found that the average precision on the test set is only 0.2%.</p>\n<p>My notebook is <a href=\"https://www.kaggle.com/kontoudi/reproducing-sartorius-results-on-centermask2\" target=\"_blank\">here</a>.</p>\n<p>I was wondering if anyone else has done the same thing and what were the results. Also, I'll be more than happy to have some feedback on the notebook as the result is really puzzling to me.</p>",
  "messages": [
    {
      "id": "1573882",
      "postDate": "11/07/2021 02:38:04",
      "content": "<p>I have tried to use a CenterMask2 model, pretrained on the LIVECell 2021 data, with and without training, with very poor results despite the zero-shot learning performance stated in the <a href=\"https://www.nature.com/articles/s41592-021-01249-6\" target=\"_blank\">Sartorius LIVECell paper</a>.</p>\n<p>So, I set out to reproduce the results of the paper, i.e. taking the pretrained model and applying it to the LIVECell test set. I found that the average precision on the test set is only 0.2%.</p>\n<p>My notebook is <a href=\"https://www.kaggle.com/kontoudi/reproducing-sartorius-results-on-centermask2\" target=\"_blank\">here</a>.</p>\n<p>I was wondering if anyone else has done the same thing and what were the results. Also, I'll be more than happy to have some feedback on the notebook as the result is really puzzling to me.</p>",
      "rawMarkdown": "I have tried to use a CenterMask2 model, pretrained on the LIVECell 2021 data, with and without training, with very poor results despite the zero-shot learning performance stated in the [Sartorius LIVECell paper](https://www.nature.com/articles/s41592-021-01249-6).\n\nSo, I set out to reproduce the results of the paper, i.e. taking the pretrained model and applying it to the LIVECell test set. I found that the average precision on the test set is only 0.2%.\n\nMy notebook is [here](https://www.kaggle.com/kontoudi/reproducing-sartorius-results-on-centermask2).\n\nI was wondering if anyone else has done the same thing and what were the results. Also, I'll be more than happy to have some feedback on the notebook as the result is really puzzling to me.",
      "votes": null
    },
    {
      "id": "1576174",
      "postDate": "11/09/2021 00:50:51",
      "content": "<p>I am writing again just to let you know that I found out what was the issue. Now <a href=\"https://www.kaggle.com/kontoudi/reproducing-sartorius-results-on-centermask2\" target=\"_blank\">my notebook</a> correctly reproduces the performance of the paper. <br>\nHope it helps!</p>",
      "rawMarkdown": "I am writing again just to let you know that I found out what was the issue. Now [my notebook](https://www.kaggle.com/kontoudi/reproducing-sartorius-results-on-centermask2) correctly reproduces the performance of the paper. \nHope it helps!",
      "votes": null
    },
    {
      "id": "1576287",
      "postDate": "11/09/2021 03:54:58",
      "content": "<p>since you already setup livecell+detectron, i suggest you can try this:<br>\n<a href=\"https://github.com/easton-cau/SOTR\" target=\"_blank\">https://github.com/easton-cau/SOTR</a></p>\n<p>state of art instance segmentation with transformer (map 0.42 on coco)<br>\nit is also based on detectron</p>",
      "rawMarkdown": "since you already setup livecell+detectron, i suggest you can try this:\nhttps://github.com/easton-cau/SOTR\n\nstate of art instance segmentation with transformer (map 0.42 on coco)\nit is also based on detectron",
      "votes": null
    },
    {
      "id": "1576761",
      "postDate": "11/09/2021 12:50:52",
      "content": "<p>Thanks for the suggestion :) I didn't have this one on my list. <br>\nI don't see any LICENSE in their repo though. I think we should use only open source code, right?</p>",
      "rawMarkdown": "Thanks for the suggestion :) I didn't have this one on my list. \nI don't see any LICENSE in their repo though. I think we should use only open source code, right?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1576174,
      "author_name": "kontoudi",
      "author_url": "",
      "post_date": "11/09/2021 00:50:51",
      "content": "<p>I am writing again just to let you know that I found out what was the issue. Now <a href=\"https://www.kaggle.com/kontoudi/reproducing-sartorius-results-on-centermask2\" target=\"_blank\">my notebook</a> correctly reproduces the performance of the paper. <br>\nHope it helps!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1576287,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "11/09/2021 03:54:58",
      "content": "<p>since you already setup livecell+detectron, i suggest you can try this:<br>\n<a href=\"https://github.com/easton-cau/SOTR\" target=\"_blank\">https://github.com/easton-cau/SOTR</a></p>\n<p>state of art instance segmentation with transformer (map 0.42 on coco)<br>\nit is also based on detectron</p>",
      "votes": null,
      "replies": [
        {
          "id": 1576761,
          "author_name": "kontoudi",
          "author_url": "",
          "post_date": "11/09/2021 12:50:52",
          "content": "<p>Thanks for the suggestion :) I didn't have this one on my list. <br>\nI don't see any LICENSE in their repo though. I think we should use only open source code, right?</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1573882": "I have tried to use a CenterMask2 model, pretrained on the LIVECell 2021 data, with and without training, with very poor results despite the zero-shot learning performance stated in the [Sartorius LIVECell paper](https://www.nature.com/articles/s41592-021-01249-6).\n\nSo, I set out to reproduce the results of the paper, i.e. taking the pretrained model and applying it to the LIVECell test set. I found that the average precision on the test set is only 0.2%.\n\nMy notebook is [here](https://www.kaggle.com/kontoudi/reproducing-sartorius-results-on-centermask2).\n\nI was wondering if anyone else has done the same thing and what were the results. Also, I'll be more than happy to have some feedback on the notebook as the result is really puzzling to me.",
    "1576174": "I am writing again just to let you know that I found out what was the issue. Now [my notebook](https://www.kaggle.com/kontoudi/reproducing-sartorius-results-on-centermask2) correctly reproduces the performance of the paper. \nHope it helps!",
    "1576287": "since you already setup livecell+detectron, i suggest you can try this:\nhttps://github.com/easton-cau/SOTR\n\nstate of art instance segmentation with transformer (map 0.42 on coco)\nit is also based on detectron",
    "1576761": "Thanks for the suggestion :) I didn't have this one on my list. \nI don't see any LICENSE in their repo though. I think we should use only open source code, right?"
  },
  "source": "meta"
}