![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2F8d2166da33e933acf89e5acc9c865b74%2FScreenshot%202025-04-25%20at%2022.04.25.png?generation=1745586325626721&alt=media\")
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2F1c29e40e7d196d4c741b0118d2031752%2FScreenshot%202025-04-25%20at%2022.04.47.png?generation=1745586341084858&alt=media\")
I released the corrected dataset of pixel anomalies found on the @brendanartley's external dataset.
\nlink: https://www.kaggle.com/datasets/tatamikenn/byu-cryoet-dataset-with-pixel-anomalies-corrected/data
\nSome of the original data (from the CryoET Data Portal) appear to have been incorrectly quantized, resulting in pixel values ranging from -128 to 127 — as if they were stored as int8, even though the expected data range should be from 0 to 255 like uint8.
\nWhich causes significant corruption of source image (see the pictures in visualization section).
The key correction applied is as follows:
\nx = x.astype().astype()\nFor more detail, you can refer to the following notebooks:
\nBefore correction:
\n
\n
After correction:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2Fbc8601a58e8092323221848400cbfc03%2FScreenshot%202025-04-25%20at%2022.04.36.png?generation=1745586366776285&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2Fb7da5f6b1117cc4de4b7510b9e379a29%2FScreenshot%202025-04-25%20at%2022.04.57.png?generation=1745586379171915&alt=media\")
I released the corrected dataset of pixel anomalies found on the @brendanartley's external dataset.
\nlink: https://www.kaggle.com/datasets/tatamikenn/byu-cryoet-dataset-with-pixel-anomalies-corrected/data
\nSome of the original data (from the CryoET Data Portal) appear to have been incorrectly quantized, resulting in pixel values ranging from -128 to 127 — as if they were stored as int8, even though the expected data range should be from 0 to 255 like uint8.
\nWhich causes significant corruption of source image (see the pictures in visualization section).
The key correction applied is as follows:
\nx = x.astype().astype()\nFor more detail, you can refer to the following notebooks:
\nBefore correction:
\n
\n
After correction:
\n
\n
Thanks @tatamikenn!
", "rawMarkdown": "Thanks @tatamikenn!", "votes": null }, { "id": "3187056", "postDate": "04/25/2025 13:55:58", "content": "Thanks man- there seem to be a few blackened images like this in the original training set as well? Will the same correction work there?
", "rawMarkdown": "Thanks man- there seem to be a few blackened images like this in the original training set as well? Will the same correction work there?", "votes": null }, { "id": "3187090", "postDate": "04/25/2025 14:50:37", "content": "Thank you for the helpful information.
", "rawMarkdown": "Thank you for the helpful information.", "votes": null }, { "id": "3187178", "postDate": "04/25/2025 16:41:40", "content": "thanks man god job
", "rawMarkdown": "thanks man god job", "votes": null }, { "id": "3187364", "postDate": "04/25/2025 22:06:19", "content": "Thank you and great work!
\nIn your dataset in incorrect labels, there are tomogram names and z_slice values for those tomograms, are only THOSE slices incorrectly quantized?
\nI would generally expect not just one , but nearby slices to be incorrectly quantized too….
Don't care much of this column. These are slices which exist annotations. I added it to check the image around motors.
", "rawMarkdown": "Don't care much of this column. These are slices which exist annotations. I added it to check the image around motors.", "votes": null }, { "id": "3188959", "postDate": "04/28/2025 14:37:14", "content": "Thanks for sharings it helps alot
", "rawMarkdown": "Thanks for sharings it helps alot", "votes": null }, { "id": "3198930", "postDate": "05/10/2025 07:53:09", "content": "Hi Thanks for the insights!
\nBut I have a quesiton. Let's see we will implement a data normalization for preprocessing.
\nI.e. Normalize slice data using the 2nd and 98th percentiles and Normalized image in the range [0, 255]. In such case, I think this irregularity will not be an issue any more. Is it?
Best regards
\nLeo
I think fixing the anomalies of the test was key to get stable score in private. We didn't do that in our final subs, and unfortunately shaked down significantly. It is sad how this was not really represented in public lb tomos leading to all our work get punished severely :(
", "rawMarkdown": "I think fixing the anomalies of the test was key to get stable score in private. We didn't do that in our final subs, and unfortunately shaked down significantly. It is sad how this was not really represented in public lb tomos leading to all our work get punished severely :(", "votes": null }, { "id": "3217405", "postDate": "06/05/2025 01:53:26", "content": "I'm realized that I forget to add these logics to correct the wrong quantiled tomogram in submission code too, right before clicking \"Submit\" button for the last submission, but decided not gonna fix and still continue to submit caused of 12h deadline limitation. My reasons for this:
\nBut, now I'm making late submissions to verify these as well.
", "rawMarkdown": "I'm realized that I forget to add these logics to correct the wrong quantiled tomogram in submission code too, right before clicking \"Submit\" button for the last submission, but decided not gonna fix and still continue to submit caused of 12h deadline limitation. My reasons for this:\n- I've submit that logic and observe no change in public LB, like yours\n- Tomograms from public/private test seem to be produced/belong privately to a lab, with good and unified data collecting/preprocessing techniques, so such simple errors seem not very likely\n- Host had train models, visualize, label, review the tomogram again. If I were the host, I will fix this before release the competition dataset instead of forcing Kaggler to \"probe/guess the dataset\"\n\nBut, now I'm making late submissions to verify these as well.", "votes": null }, { "id": "3217406", "postDate": "06/05/2025 01:56:57", "content": "\n\nI've submit that logic and observe no change in public LB, like yours
\n
I observed similar result in the public LB. So this correction was not applied in my final sub. However, not sure in private LB.
", "rawMarkdown": "> I've submit that logic and observe no change in public LB, like yours\n\nI observed similar result in the public LB. So this correction was not applied in my final sub. However, not sure in private LB.", "votes": null }, { "id": "3217410", "postDate": "06/05/2025 02:02:16", "content": "I believe correction by quantization is not robust way of fixing this anomaly because it depends on the choice of the quantiles. Note that the absolute error value is at most 127.
", "rawMarkdown": "I believe correction by quantization is not robust way of fixing this anomaly because it depends on the choice of the quantiles. Note that the absolute error value is at most 127.", "votes": null }, { "id": "3217413", "postDate": "06/05/2025 02:05:12", "content": "Our private lb scores honestly looks very random. I don't see clear correlation, or why we shaked down.
\nSome random ensembles produced much better results than others without clear reason. But the only sub that has clearly a different pattern was the one with the anomalies fix added. That's why i believe it was the reason.
The difference is huge. I am talking about smth like this:
\nPublic: 0.841, private 0.785 (w/o)
\nPublic: 0.841, private 0.818 (with fix)
@mohammad2012191 thank you for sharing the result. It is noticeable gain.
\nMaybe test tomogram also contains this anomaly, but I think we need more submissions, considering high randomness in this competition.
Thanks @tatamikenn!
", "votes": null, "replies": [] }, { "id": 3187056, "author_name": "yahalom", "author_url": "", "post_date": "04/25/2025 13:55:58", "content": "Thanks man- there seem to be a few blackened images like this in the original training set as well? Will the same correction work there?
", "votes": null, "replies": [] }, { "id": 3187090, "author_name": "sohilaelmassry", "author_url": "", "post_date": "04/25/2025 14:50:37", "content": "Thank you for the helpful information.
", "votes": null, "replies": [] }, { "id": 3187178, "author_name": "analyticaobscura", "author_url": "", "post_date": "04/25/2025 16:41:40", "content": "thanks man god job
", "votes": null, "replies": [] }, { "id": 3187364, "author_name": "iamudit", "author_url": "", "post_date": "04/25/2025 22:06:19", "content": "Thank you and great work!
\nIn your dataset in incorrect labels, there are tomogram names and z_slice values for those tomograms, are only THOSE slices incorrectly quantized?
\nI would generally expect not just one , but nearby slices to be incorrectly quantized too….
Don't care much of this column. These are slices which exist annotations. I added it to check the image around motors.
", "votes": null, "replies": [] } ] }, { "id": 3188959, "author_name": "saraharshadbcs", "author_url": "", "post_date": "04/28/2025 14:37:14", "content": "Thanks for sharings it helps alot
", "votes": null, "replies": [] }, { "id": 3198930, "author_name": "yxyyxy", "author_url": "", "post_date": "05/10/2025 07:53:09", "content": "Hi Thanks for the insights!
\nBut I have a quesiton. Let's see we will implement a data normalization for preprocessing.
\nI.e. Normalize slice data using the 2nd and 98th percentiles and Normalized image in the range [0, 255]. In such case, I think this irregularity will not be an issue any more. Is it?
Best regards
\nLeo
I believe correction by quantization is not robust way of fixing this anomaly because it depends on the choice of the quantiles. Note that the absolute error value is at most 127.
", "votes": null, "replies": [] } ] }, { "id": 3217394, "author_name": "mohammad2012191", "author_url": "", "post_date": "06/05/2025 01:28:14", "content": "I think fixing the anomalies of the test was key to get stable score in private. We didn't do that in our final subs, and unfortunately shaked down significantly. It is sad how this was not really represented in public lb tomos leading to all our work get punished severely :(
", "votes": null, "replies": [ { "id": 3217405, "author_name": "dangnh0611", "author_url": "", "post_date": "06/05/2025 01:53:26", "content": "I'm realized that I forget to add these logics to correct the wrong quantiled tomogram in submission code too, right before clicking \"Submit\" button for the last submission, but decided not gonna fix and still continue to submit caused of 12h deadline limitation. My reasons for this:
\nBut, now I'm making late submissions to verify these as well.
", "votes": null, "replies": [ { "id": 3217406, "author_name": "tatamikenn", "author_url": "", "post_date": "06/05/2025 01:56:57", "content": "\n\nI've submit that logic and observe no change in public LB, like yours
\n
I observed similar result in the public LB. So this correction was not applied in my final sub. However, not sure in private LB.
", "votes": null, "replies": [ { "id": 3217413, "author_name": "mohammad2012191", "author_url": "", "post_date": "06/05/2025 02:05:12", "content": "Our private lb scores honestly looks very random. I don't see clear correlation, or why we shaked down.
\nSome random ensembles produced much better results than others without clear reason. But the only sub that has clearly a different pattern was the one with the anomalies fix added. That's why i believe it was the reason.
The difference is huge. I am talking about smth like this:
\nPublic: 0.841, private 0.785 (w/o)
\nPublic: 0.841, private 0.818 (with fix)
@mohammad2012191 thank you for sharing the result. It is noticeable gain.
\nMaybe test tomogram also contains this anomaly, but I think we need more submissions, considering high randomness in this competition.