It seems that images of the same cells are taken several times, sometimes with different quality and artifacts. Itself it's not a problem (I hope there is no leak between train and test data though). What's the real issue -- inconsistent annotations. One of the common cases is here.
\n@ChristofferSartorius I have a few questions:
\nAre you aware of these duplicates and annotation inconsistency?
\nIs there overlapping of test data and training data provided?
\nData quality is the real problem to get good performance while overfitting on the leak data from test can mess up things.
It seems that images of the same cells are taken several times, sometimes with different quality and artifacts. Itself it's not a problem (I hope there is no leak between train and test data though). What's the real issue -- inconsistent annotations. One of the common cases is here.
\n@ChristofferSartorius I have a few questions:
\nAre you aware of these duplicates and annotation inconsistency?
\nIs there overlapping of test data and training data provided?
\nData quality is the real problem to get good performance while overfitting on the leak data from test can mess up things.
Sorry for asking a silly question, but what's data leak?😬
", "rawMarkdown": "Sorry for asking a silly question, but what's data leak?😬", "votes": null }, { "id": "1594665", "postDate": "11/25/2021 04:13:03", "content": "Great discovery! What method did you use to find the duplicate images?
", "rawMarkdown": "Great discovery! What method did you use to find the duplicate images?", "votes": null }, { "id": "1594796", "postDate": "11/25/2021 06:08:20", "content": "I found these duplicates when looked at low score train data. It appears that such duplicates are only happens to corto images.
", "rawMarkdown": "I found these duplicates when looked at low score train data. It appears that such duplicates are only happens to corto images.", "votes": null }, { "id": "1594798", "postDate": "11/25/2021 06:10:05", "content": "", "rawMarkdown": "[Leakage](https://en.wikipedia.org/wiki/Leakage_(machine_learning))", "votes": null }, { "id": "1594902", "postDate": "11/25/2021 08:20:16", "content": "Btw, the images are from the same sample, i.e. they have the same sample_id. I believe that a sample has multiple images taken over time, see also the sample_date and plate_time fields. However, as you say, only for the cort images, different images of the same sample are similar to each other and, apparently, the masks seem to be quite inconsistent.
This is what LiveCELL paper says about acquiring images:
\n\n\nSeveral wells for each cell type were seeded in 96-well plates (Corning) and imaged over the course of 3–5 d, every 4 h
\n
Maybe here it was done the same way and you are seeing the same cells 4 hours apart?
", "rawMarkdown": "This is what LiveCELL paper says about acquiring images:\n>Several wells for each cell type were seeded in 96-well plates (Corning) and imaged over the course of 3–5 d, every 4 h\n\nMaybe here it was done the same way and you are seeing the same cells 4 hours apart?", "votes": null }, { "id": "1595021", "postDate": "11/25/2021 10:32:06", "content": "I also think that's the case since those images share the same sample_id but have different sample_date and plate_time fields which are within a couple of hours/days.
Yeah, for me the root cause of these duplicates is not that important as handling it properly:
\nThis may add more complexity for us to prepare folds properly, some of the duplicates are have quite different parts of images.
", "rawMarkdown": "Yeah, for me the root cause of these duplicates is not that important as handling it properly:\n- having consistent annotations\n- prevent data leaking\n\nThis may add more complexity for us to prepare folds properly, some of the duplicates are have quite different parts of images.", "votes": null }, { "id": "1595095", "postDate": "11/25/2021 12:09:15", "content": "I tested a bit, it appears test data has some of such duplicates with train data, but score difference was <0.001 when I replaced those duplicated prediction with dummy value.
", "rawMarkdown": "I tested a bit, it appears test data has some of such duplicates with train data, but score difference was <0.001 when I replaced those duplicated prediction with dummy value.", "votes": null } ], "comments": [ { "id": 1594586, "author_name": "manyuli", "author_url": "", "post_date": "11/25/2021 01:49:57", "content": "Sorry for asking a silly question, but what's data leak?😬
", "votes": null, "replies": [ { "id": 1594798, "author_name": "rednikotin", "author_url": "", "post_date": "11/25/2021 06:10:05", "content": "", "votes": null, "replies": [] } ] }, { "id": 1594665, "author_name": "apparition", "author_url": "", "post_date": "11/25/2021 04:13:03", "content": "Great discovery! What method did you use to find the duplicate images?
", "votes": null, "replies": [ { "id": 1594796, "author_name": "rednikotin", "author_url": "", "post_date": "11/25/2021 06:08:20", "content": "I found these duplicates when looked at low score train data. It appears that such duplicates are only happens to corto images.
", "votes": null, "replies": [] }, { "id": 1594902, "author_name": "omallo", "author_url": "", "post_date": "11/25/2021 08:20:16", "content": "Btw, the images are from the same sample, i.e. they have the same sample_id. I believe that a sample has multiple images taken over time, see also the sample_date and plate_time fields. However, as you say, only for the cort images, different images of the same sample are similar to each other and, apparently, the masks seem to be quite inconsistent.
This is what LiveCELL paper says about acquiring images:
\n\n\nSeveral wells for each cell type were seeded in 96-well plates (Corning) and imaged over the course of 3–5 d, every 4 h
\n
Maybe here it was done the same way and you are seeing the same cells 4 hours apart?
", "votes": null, "replies": [ { "id": 1595021, "author_name": "omallo", "author_url": "", "post_date": "11/25/2021 10:32:06", "content": "I also think that's the case since those images share the same sample_id but have different sample_date and plate_time fields which are within a couple of hours/days.
Yeah, for me the root cause of these duplicates is not that important as handling it properly:
\nThis may add more complexity for us to prepare folds properly, some of the duplicates are have quite different parts of images.
", "votes": null, "replies": [] } ] }, { "id": 1595095, "author_name": "rednikotin", "author_url": "", "post_date": "11/25/2021 12:09:15", "content": "I tested a bit, it appears test data has some of such duplicates with train data, but score difference was <0.001 when I replaced those duplicated prediction with dummy value.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1594293": "It seems that images of the same cells are taken several times, sometimes with different quality and artifacts. Itself it's not a problem (I hope there is no leak between train and test data though). What's the real issue -- inconsistent annotations. One of the common cases is [here](https://www.kaggle.com/rednikotin/duplicates-sartorius).\n\n@ChristofferSartorius I have a few questions:\nAre you aware of these duplicates and annotation inconsistency? \nIs there overlapping of test data and training data provided?\nData quality is the real problem to get good performance while overfitting on the leak data from test can mess up things.", "1594586": "Sorry for asking a silly question, but what's data leak?😬", "1594665": "Great discovery! What method did you use to find the duplicate images?", "1594796": "I found these duplicates when looked at low score train data. It appears that such duplicates are only happens to corto images.", "1594798": "[Leakage](https://en.wikipedia.org/wiki/Leakage_(machine_learning))", "1594902": "Btw, the images are from the same sample, i.e. they have the same `sample_id`. I believe that a sample has multiple images taken over time, see also the `sample_date` and `plate_time` fields. However, as you say, only for the cort images, different images of the same sample are similar to each other and, apparently, the masks seem to be quite inconsistent.", "1594982": "This is what LiveCELL paper says about acquiring images:\n>Several wells for each cell type were seeded in 96-well plates (Corning) and imaged over the course of 3–5 d, every 4 h\n\nMaybe here it was done the same way and you are seeing the same cells 4 hours apart?", "1595021": "I also think that's the case since those images share the same `sample_id` but have different `sample_date` and `plate_time` fields which are within a couple of hours/days.", "1595049": "Yeah, for me the root cause of these duplicates is not that important as handling it properly:\n- having consistent annotations\n- prevent data leaking\n\nThis may add more complexity for us to prepare folds properly, some of the duplicates are have quite different parts of images.", "1595095": "I tested a bit, it appears test data has some of such duplicates with train data, but score difference was <0.001 when I replaced those duplicated prediction with dummy value." }, "source": "meta" }