{
  "id": 296361,
  "title": "Warnings! Data leakage in the provided LIVECell_dataset_2021.",
  "url": "/competitions/sartorius-cell-instance-segmentation/discussion/296361",
  "author_name": "",
  "post_date": "2021-12-21T07:44:56.053956700Z",
  "votes": 19,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Hi, </p>\n<p>I found data leakage in the provided <strong>LIVECell_dataset_2021</strong>.</p>\n<p>In the providied annotations, there ara <strong>30 images</strong> appeared in both the train annotations and the val annotations. </p>\n<h2>Specific annotations files:</h2>\n<p>-&gt; /LIVECell_dataset_2021/annotations/LIVECell_single_cells/huh7/livecell_huh7_train.json<br>\n-&gt; /LIVECell_dataset_2021/annotations/LIVECell_single_cells/huh7/livecell_huh7_val.json<br>\n-&gt; /LIVECell_dataset_2021/annotations/LIVECell_single_cells/skov3/livecell_skov3_train.json<br>\n-&gt; /LIVECell_dataset_2021/annotations/LIVECell_single_cells/skov3/livecell_skov3_val.json</p>\n<h2>leakage image list, 30 in total.</h2>\n<p>Huh7_Phase_A11_2_02d04h00m_3.tif<br>\nHuh7_Phase_A11_2_02d00h00m_4.tif<br>\nHuh7_Phase_A11_2_02d04h00m_2.tif<br>\nHuh7_Phase_A11_2_01d20h00m_3.tif<br>\nHuh7_Phase_A11_2_02d00h00m_2.tif<br>\nHuh7_Phase_A10_2_00d00h00m_3.tif<br>\nHuh7_Phase_A10_2_00d12h00m_1.tif<br>\nHuh7_Phase_A11_2_01d16h00m_2.tif<br>\nHuh7_Phase_A11_2_01d08h00m_4.tif<br>\nHuh7_Phase_A11_2_01d20h00m_2.tif<br>\nHuh7_Phase_A10_2_00d20h00m_1.tif<br>\nHuh7_Phase_A10_2_01d00h00m_1.tif<br>\nHuh7_Phase_A11_2_01d16h00m_1.tif<br>\nHuh7_Phase_A11_2_02d04h00m_4.tif<br>\nHuh7_Phase_A10_2_01d00h00m_2.tif<br>\nHuh7_Phase_A10_2_00d16h00m_3.tif<br>\nHuh7_Phase_A10_2_01d00h00m_4.tif<br>\nSKOV3_Phase_H4_1_00d04h00m_1.tif<br>\nSKOV3_Phase_G4_1_00d20h00m_4.tif<br>\nSKOV3_Phase_G4_1_01d08h00m_1.tif<br>\nSKOV3_Phase_G4_1_01d00h00m_3.tif<br>\nSKOV3_Phase_H4_1_00d04h00m_2.tif<br>\nSKOV3_Phase_G4_1_01d04h00m_3.tif<br>\nSKOV3_Phase_H4_1_00d08h00m_4.tif<br>\nSKOV3_Phase_G4_1_00d20h00m_3.tif<br>\nSKOV3_Phase_H4_1_00d16h00m_1.tif<br>\nSKOV3_Phase_H4_1_00d08h00m_2.tif<br>\nSKOV3_Phase_H4_1_00d12h00m_1.tif<br>\nSKOV3_Phase_G4_1_01d12h00m_4.tif<br>\nSKOV3_Phase_H4_1_00d00h00m_2.tif</p>",
  "messages": [
    {
      "id": "1624757",
      "postDate": "12/21/2021 07:44:56",
      "content": "<p>Hi, </p>\n<p>I found data leakage in the provided <strong>LIVECell_dataset_2021</strong>.</p>\n<p>In the providied annotations, there ara <strong>30 images</strong> appeared in both the train annotations and the val annotations. </p>\n<h2>Specific annotations files:</h2>\n<p>-&gt; /LIVECell_dataset_2021/annotations/LIVECell_single_cells/huh7/livecell_huh7_train.json<br>\n-&gt; /LIVECell_dataset_2021/annotations/LIVECell_single_cells/huh7/livecell_huh7_val.json<br>\n-&gt; /LIVECell_dataset_2021/annotations/LIVECell_single_cells/skov3/livecell_skov3_train.json<br>\n-&gt; /LIVECell_dataset_2021/annotations/LIVECell_single_cells/skov3/livecell_skov3_val.json</p>\n<h2>leakage image list, 30 in total.</h2>\n<p>Huh7_Phase_A11_2_02d04h00m_3.tif<br>\nHuh7_Phase_A11_2_02d00h00m_4.tif<br>\nHuh7_Phase_A11_2_02d04h00m_2.tif<br>\nHuh7_Phase_A11_2_01d20h00m_3.tif<br>\nHuh7_Phase_A11_2_02d00h00m_2.tif<br>\nHuh7_Phase_A10_2_00d00h00m_3.tif<br>\nHuh7_Phase_A10_2_00d12h00m_1.tif<br>\nHuh7_Phase_A11_2_01d16h00m_2.tif<br>\nHuh7_Phase_A11_2_01d08h00m_4.tif<br>\nHuh7_Phase_A11_2_01d20h00m_2.tif<br>\nHuh7_Phase_A10_2_00d20h00m_1.tif<br>\nHuh7_Phase_A10_2_01d00h00m_1.tif<br>\nHuh7_Phase_A11_2_01d16h00m_1.tif<br>\nHuh7_Phase_A11_2_02d04h00m_4.tif<br>\nHuh7_Phase_A10_2_01d00h00m_2.tif<br>\nHuh7_Phase_A10_2_00d16h00m_3.tif<br>\nHuh7_Phase_A10_2_01d00h00m_4.tif<br>\nSKOV3_Phase_H4_1_00d04h00m_1.tif<br>\nSKOV3_Phase_G4_1_00d20h00m_4.tif<br>\nSKOV3_Phase_G4_1_01d08h00m_1.tif<br>\nSKOV3_Phase_G4_1_01d00h00m_3.tif<br>\nSKOV3_Phase_H4_1_00d04h00m_2.tif<br>\nSKOV3_Phase_G4_1_01d04h00m_3.tif<br>\nSKOV3_Phase_H4_1_00d08h00m_4.tif<br>\nSKOV3_Phase_G4_1_00d20h00m_3.tif<br>\nSKOV3_Phase_H4_1_00d16h00m_1.tif<br>\nSKOV3_Phase_H4_1_00d08h00m_2.tif<br>\nSKOV3_Phase_H4_1_00d12h00m_1.tif<br>\nSKOV3_Phase_G4_1_01d12h00m_4.tif<br>\nSKOV3_Phase_H4_1_00d00h00m_2.tif</p>",
      "rawMarkdown": "Hi, \n\nI found data leakage in the provided **LIVECell_dataset_2021**.\n\nIn the providied annotations, there ara **30 images** appeared in both the train annotations and the val annotations. \n\n## Specific annotations files:\n-> /LIVECell_dataset_2021/annotations/LIVECell_single_cells/huh7/livecell_huh7_train.json\n-> /LIVECell_dataset_2021/annotations/LIVECell_single_cells/huh7/livecell_huh7_val.json\n-> /LIVECell_dataset_2021/annotations/LIVECell_single_cells/skov3/livecell_skov3_train.json\n-> /LIVECell_dataset_2021/annotations/LIVECell_single_cells/skov3/livecell_skov3_val.json\n\n## leakage image list, 30 in total.\nHuh7_Phase_A11_2_02d04h00m_3.tif\nHuh7_Phase_A11_2_02d00h00m_4.tif\nHuh7_Phase_A11_2_02d04h00m_2.tif\nHuh7_Phase_A11_2_01d20h00m_3.tif\nHuh7_Phase_A11_2_02d00h00m_2.tif\nHuh7_Phase_A10_2_00d00h00m_3.tif\nHuh7_Phase_A10_2_00d12h00m_1.tif\nHuh7_Phase_A11_2_01d16h00m_2.tif\nHuh7_Phase_A11_2_01d08h00m_4.tif\nHuh7_Phase_A11_2_01d20h00m_2.tif\nHuh7_Phase_A10_2_00d20h00m_1.tif\nHuh7_Phase_A10_2_01d00h00m_1.tif\nHuh7_Phase_A11_2_01d16h00m_1.tif\nHuh7_Phase_A11_2_02d04h00m_4.tif\nHuh7_Phase_A10_2_01d00h00m_2.tif\nHuh7_Phase_A10_2_00d16h00m_3.tif\nHuh7_Phase_A10_2_01d00h00m_4.tif\nSKOV3_Phase_H4_1_00d04h00m_1.tif\nSKOV3_Phase_G4_1_00d20h00m_4.tif\nSKOV3_Phase_G4_1_01d08h00m_1.tif\nSKOV3_Phase_G4_1_01d00h00m_3.tif\nSKOV3_Phase_H4_1_00d04h00m_2.tif\nSKOV3_Phase_G4_1_01d04h00m_3.tif\nSKOV3_Phase_H4_1_00d08h00m_4.tif\nSKOV3_Phase_G4_1_00d20h00m_3.tif\nSKOV3_Phase_H4_1_00d16h00m_1.tif\nSKOV3_Phase_H4_1_00d08h00m_2.tif\nSKOV3_Phase_H4_1_00d12h00m_1.tif\nSKOV3_Phase_G4_1_01d12h00m_4.tif\nSKOV3_Phase_H4_1_00d00h00m_2.tif",
      "votes": null
    },
    {
      "id": "1624770",
      "postDate": "12/21/2021 08:02:02",
      "content": "<p>It's better to combine the train, val and test datasets into a single dataframe, drop duplicates and then create your own train-val split to use all available data. Atleast that's how I am pre-training my models.</p>",
      "rawMarkdown": "It's better to combine the train, val and test datasets into a single dataframe, drop duplicates and then create your own train-val split to use all available data. Atleast that's how I am pre-training my models.",
      "votes": null
    },
    {
      "id": "1624777",
      "postDate": "12/21/2021 08:14:02",
      "content": "<p>Maybe you misunderstood what I said. I mean that there is a data leakage in the provided annotation files, that is, the same image appears in the train annotations and val annotations at the same time.</p>",
      "rawMarkdown": "Maybe you misunderstood what I said. I mean that there is a data leakage in the provided annotation files, that is, the same image appears in the train annotations and val annotations at the same time.",
      "votes": null
    },
    {
      "id": "1627196",
      "postDate": "12/23/2021 16:01:52",
      "content": "<p>This similar pictures are the same cells in different time? they may be Photographed in different time。</p>",
      "rawMarkdown": "This similar pictures are the same cells in different time? they may be Photographed in different time。",
      "votes": null
    },
    {
      "id": "1627231",
      "postDate": "12/23/2021 16:43:32",
      "content": "<p>The leakage images are used for training and also for validation.</p>",
      "rawMarkdown": "The leakage images are used for training and also for validation.",
      "votes": null
    },
    {
      "id": "1632205",
      "postDate": "12/29/2021 10:49:14",
      "content": "<p>Hi, the phenomena of the same image showing up in both train and val is noted in the publication.</p>\n<p>For more information about duplicate images and images that got annotated twice please see<a href=\"https://www.nature.com/articles/s41592-021-01249-6#Sec27\" target=\"_blank\"> this link</a> to our supplementary. There should be no data leakage in the train and test data for this competition thought. </p>",
      "rawMarkdown": "Hi, the phenomena of the same image showing up in both train and val is noted in the publication.\n\nFor more information about duplicate images and images that got annotated twice please see[ this link](https://www.nature.com/articles/s41592-021-01249-6#Sec27 ) to our supplementary. There should be no data leakage in the train and test data for this competition thought.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1624770,
      "author_name": "samfc10",
      "author_url": "",
      "post_date": "12/21/2021 08:02:02",
      "content": "<p>It's better to combine the train, val and test datasets into a single dataframe, drop duplicates and then create your own train-val split to use all available data. Atleast that's how I am pre-training my models.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1624777,
          "author_name": "blueboy97",
          "author_url": "",
          "post_date": "12/21/2021 08:14:02",
          "content": "<p>Maybe you misunderstood what I said. I mean that there is a data leakage in the provided annotation files, that is, the same image appears in the train annotations and val annotations at the same time.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1627196,
      "author_name": "macxiao",
      "author_url": "",
      "post_date": "12/23/2021 16:01:52",
      "content": "<p>This similar pictures are the same cells in different time? they may be Photographed in different time。</p>",
      "votes": null,
      "replies": [
        {
          "id": 1627231,
          "author_name": "blueboy97",
          "author_url": "",
          "post_date": "12/23/2021 16:43:32",
          "content": "<p>The leakage images are used for training and also for validation.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1632205,
      "author_name": "christoffersartorius",
      "author_url": "",
      "post_date": "12/29/2021 10:49:14",
      "content": "<p>Hi, the phenomena of the same image showing up in both train and val is noted in the publication.</p>\n<p>For more information about duplicate images and images that got annotated twice please see<a href=\"https://www.nature.com/articles/s41592-021-01249-6#Sec27\" target=\"_blank\"> this link</a> to our supplementary. There should be no data leakage in the train and test data for this competition thought. </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1624757": "Hi, \n\nI found data leakage in the provided **LIVECell_dataset_2021**.\n\nIn the providied annotations, there ara **30 images** appeared in both the train annotations and the val annotations. \n\n## Specific annotations files:\n-> /LIVECell_dataset_2021/annotations/LIVECell_single_cells/huh7/livecell_huh7_train.json\n-> /LIVECell_dataset_2021/annotations/LIVECell_single_cells/huh7/livecell_huh7_val.json\n-> /LIVECell_dataset_2021/annotations/LIVECell_single_cells/skov3/livecell_skov3_train.json\n-> /LIVECell_dataset_2021/annotations/LIVECell_single_cells/skov3/livecell_skov3_val.json\n\n## leakage image list, 30 in total.\nHuh7_Phase_A11_2_02d04h00m_3.tif\nHuh7_Phase_A11_2_02d00h00m_4.tif\nHuh7_Phase_A11_2_02d04h00m_2.tif\nHuh7_Phase_A11_2_01d20h00m_3.tif\nHuh7_Phase_A11_2_02d00h00m_2.tif\nHuh7_Phase_A10_2_00d00h00m_3.tif\nHuh7_Phase_A10_2_00d12h00m_1.tif\nHuh7_Phase_A11_2_01d16h00m_2.tif\nHuh7_Phase_A11_2_01d08h00m_4.tif\nHuh7_Phase_A11_2_01d20h00m_2.tif\nHuh7_Phase_A10_2_00d20h00m_1.tif\nHuh7_Phase_A10_2_01d00h00m_1.tif\nHuh7_Phase_A11_2_01d16h00m_1.tif\nHuh7_Phase_A11_2_02d04h00m_4.tif\nHuh7_Phase_A10_2_01d00h00m_2.tif\nHuh7_Phase_A10_2_00d16h00m_3.tif\nHuh7_Phase_A10_2_01d00h00m_4.tif\nSKOV3_Phase_H4_1_00d04h00m_1.tif\nSKOV3_Phase_G4_1_00d20h00m_4.tif\nSKOV3_Phase_G4_1_01d08h00m_1.tif\nSKOV3_Phase_G4_1_01d00h00m_3.tif\nSKOV3_Phase_H4_1_00d04h00m_2.tif\nSKOV3_Phase_G4_1_01d04h00m_3.tif\nSKOV3_Phase_H4_1_00d08h00m_4.tif\nSKOV3_Phase_G4_1_00d20h00m_3.tif\nSKOV3_Phase_H4_1_00d16h00m_1.tif\nSKOV3_Phase_H4_1_00d08h00m_2.tif\nSKOV3_Phase_H4_1_00d12h00m_1.tif\nSKOV3_Phase_G4_1_01d12h00m_4.tif\nSKOV3_Phase_H4_1_00d00h00m_2.tif",
    "1624770": "It's better to combine the train, val and test datasets into a single dataframe, drop duplicates and then create your own train-val split to use all available data. Atleast that's how I am pre-training my models.",
    "1624777": "Maybe you misunderstood what I said. I mean that there is a data leakage in the provided annotation files, that is, the same image appears in the train annotations and val annotations at the same time.",
    "1627196": "This similar pictures are the same cells in different time? they may be Photographed in different time。",
    "1627231": "The leakage images are used for training and also for validation.",
    "1632205": "Hi, the phenomena of the same image showing up in both train and val is noted in the publication.\n\nFor more information about duplicate images and images that got annotated twice please see[ this link](https://www.nature.com/articles/s41592-021-01249-6#Sec27 ) to our supplementary. There should be no data leakage in the train and test data for this competition thought."
  },
  "source": "meta"
}