![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2F38c849b28e87ecc571400e6a1bb145ea%2Fwrong_label_0.jpeg?generation=1747561711220736&alt=media\")
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2F98154d245e5e7cb4248e78e27c7fbf05%2Fwrong_label_1.jpeg?generation=1747561731213769&alt=media\")
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2Fbe197bfac516a4f74da996a248c3d37c%2Fwrong_label_2.jpeg?generation=1747561742452971&alt=media\")
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2Fe45806a4129b3185c4eb491f723344ee%2Fwrong_label_3.jpeg?generation=1747561754244417&alt=media\")
As many contestants may have already noticed, I found dozens of tomograms in the training dataset labeled as \"negative\" that actually appear to contain flagellar motor-like structures (see picture below).
\nWe can correct these labels in the training set, but I'm concerned that some test tomograms might also include similar mislabeled structures.
\nHas the negative test dataset been more thoroughly validated to ensure it doesn't include motors?
@braxtonowens @jacksonpond
\nI would greatly appreciate it if you could kindly clarify these concerns.
Sample Tomograms:
\nđ I'm not a domain expert, so any advice on how to interpret the \"negative\" or \"positive\" labels in these sample tomograms would be greatly appreciated!
\n
\n
\n
\n
As many contestants may have already noticed, I found dozens of tomograms in the training dataset labeled as \"negative\" that actually appear to contain flagellar motor-like structures (see picture below).
\nWe can correct these labels in the training set, but I'm concerned that some test tomograms might also include similar mislabeled structures.
\nHas the negative test dataset been more thoroughly validated to ensure it doesn't include motors?
@braxtonowens @jacksonpond
\nI would greatly appreciate it if you could kindly clarify these concerns.
Sample Tomograms:
\nđ I'm not a domain expert, so any advice on how to interpret the \"negative\" or \"positive\" labels in these sample tomograms would be greatly appreciated!
\n
\n
\n
\n
Some days ago they had to rescore due some FN fix. So I think yes, test should be more thoroughly validated.
\nEDIT: here
", "rawMarkdown": "Some days ago they had to rescore due some FN fix. So I think yes, test should be more thoroughly validated.\n\nEDIT: [here](https://www.kaggle.com/competitions/byu-locating-bacterial-flagellar-motors-2025/discussion/574729)", "votes": null }, { "id": "3204585", "postDate": "05/18/2025 14:05:02", "content": "Hey, this is a very valid concern. And as Ăngel brought up, it is the purpose of the leaderboard rescore. Since both our datasets were human annotated, we had a small amount of errors. However, upon analysis of a bunch of the top models, we were able to identify a significant number of mislabeled tomograms, mostly false negatives, that we corrected.
\nThis isnât me confirming perfection in the test set, rather our intentions to make the competition as competitive as possible. I can confirm that there are instances in which test tomograms are consistently mislabeled by almost all the submissions. So the non-perfect LB scores are still caused by non-perfect models.
", "rawMarkdown": "Hey, this is a very valid concern. And as Ăngel brought up, it is the purpose of the leaderboard rescore. Since both our datasets were human annotated, we had a small amount of errors. However, upon analysis of a bunch of the top models, we were able to identify a significant number of mislabeled tomograms, mostly false negatives, that we corrected.\n\nThis isnât me confirming perfection in the test set, rather our intentions to make the competition as competitive as possible. I can confirm that there are instances in which test tomograms are consistently mislabeled by almost all the submissions. So the non-perfect LB scores are still caused by non-perfect models.", "votes": null }, { "id": "3204668", "postDate": "05/18/2025 16:08:11", "content": "@andrewjdarley Great andrew. Thank you provide such a challenge competition with some dirty labeled data (it's very hard to accurately model your visual preference from the given label), absolutely increasing my skillset as a data scientist. I'm currently developing another way to handle this in unsupervised anomaly detection manner (one-class approach). Hope I can finish it before close.
", "rawMarkdown": "andrewjdarley Great andrew. Thank you provide such a challenge competition with some dirty labeled data (it's very hard to accurately model your visual preference from the given label), absolutely increasing my skillset as a data scientist. I'm currently developing another way to handle this in unsupervised anomaly detection manner (one-class approach). Hope I can finish it before close.", "votes": null }, { "id": "3205059", "postDate": "05/19/2025 08:35:21", "content": "\nI looked through the negative tomograms in the training dataset and found a general tendency:
\nConsidering that the winning solution will be used for subtomogram averaging (STA), I think low-quality subtomograms should be removed, even if they contain motor-like structures. So, cases 1 and 2 make sense.
\nHowever, what about case 3? Are those marked as negative simply because theyâre located in regions where humans are more likely to overlook them?
\nOr is there another valid reason for excluding them?
Example picture of case 3:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2Ff6c5ec000aa0e93fffd9ed2f0b172760%2FScreenshot%202025-05-19%20at%2017.19.08.png?generation=1747642938120760&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2F477fa1f9d891c85487552c37f96541d0%2FScreenshot%202025-05-19%20at%2017.19.39.png?generation=1747642951414867&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2Ffda0a9ca63470df100968b9e9e25d9d6%2FScreenshot%202025-05-19%20at%2017.20.41.png?generation=1747642964307156&alt=media\")
Nice Work đ
", "rawMarkdown": "Nice Work đ", "votes": null }, { "id": "3205531", "postDate": "05/20/2025 04:23:48", "content": "Thanks for sharing. I also noticed a sample, tomo_401341, where only one motor is labeled at slice 166, but there appears to be another motor-like structure around slice 197.
| row_id | \ntomo_id | \nMotor axis 0 | \nMotor axis 1 | \nMotor axis 2 | \nArray shape (axis 0) | \nArray shape (axis 1) | \nArray shape (axis 2) | \nVoxel spacing | \nNumber of motors | \n
|---|---|---|---|---|---|---|---|---|---|
| 191 | \ntomo_401341 | \n166.0 | \n612.0 | \n722.0 | \n300 | \n960 | \n928 | \n13.1 | \n1 | \n
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3964695%2F02a52711ad0d52f67b92afeafa134bef%2FScreenshot%202025-05-20%20000824.png?generation=1747714511875735&alt=media\")
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3964695%2F826d3999f9749ca4762d68b7061760d8%2FScreenshot%202025-05-20%20001005.png?generation=1747714522500658&alt=media\")
\n### slice 197:\n",
"votes": null
},
{
"id": "3205542",
"postDate": "05/20/2025 04:40:01",
"content": "@maxchen303 This case shows you don't naively use single point as your ground truth.
", "rawMarkdown": "maxchen303 This case shows you don't naively use single point as your ground truth.", "votes": null }, { "id": "3205549", "postDate": "05/20/2025 05:05:00", "content": "I found many of those as well, and relabeled the most obvious cases, but I think that this competition is about \"predict the labeler\" so I'm not sure whether labling everything will make our LB scores better if test cases were labeled with the same set of heuristic rules as the train.
", "rawMarkdown": "I found many of those as well, and relabeled the most obvious cases, but I think that this competition is about \"predict the labeler\" so I'm not sure whether labling everything will make our LB scores better if test cases were labeled with the same set of heuristic rules as the train.", "votes": null }, { "id": "3205703", "postDate": "05/20/2025 09:02:44", "content": "So may be would be good idea pseudolabel most confident train predictions right?
", "rawMarkdown": "So may be would be good idea pseudolabel most confident train predictions right?", "votes": null }, { "id": "3205713", "postDate": "05/20/2025 09:11:27", "content": "Hi.
\n\n\nâŚupon analysis of a bunch of the top models, we were able to identify a significant number of mislabeled tomograms, mostly false negatives, that we corrected.
\n
They're using most confident submissions to carefully check test annotations. So I think will be more \"predict label\" than \"labeler\".
", "rawMarkdown": "Hi.\n\n>...upon analysis of a bunch of the top models, we were able to identify a significant number of mislabeled tomograms, mostly false negatives, that we corrected.\n\nThey're using most confident submissions to carefully check test annotations. So I think will be more \"predict label\" than \"labeler\".", "votes": null }, { "id": "3205760", "postDate": "05/20/2025 10:21:00", "content": "Good Work đ
", "rawMarkdown": "Good Work đ", "votes": null }, { "id": "3205926", "postDate": "05/20/2025 14:47:00", "content": "I consulted some of the samplers regarding case 3, and they provided the following motivation. First, to some degree it is a coincidence that these frequently occur near the foil, probably due to train domain sampling randomness. Itâs caused a lot of correlations in the train data that arenât present in the test set.
\nSecond, there are instances in which a motor can be present without a visible flagellum. However these frequently represent sick or starving bacteria in unusual circumstances. The postdocs that work on this skip annotation in this case for our purposes of subtomogram averaging since you canât expect the structure of the bacteria to be helpful when theyâre so unhealthy.
\nThis means that in this case, it is a bit of a âpredict the labelerâ kind of competition which is tough. Thatâs just the pattern we used and it might seem inconsistent, especially to our models. It does represent the underlying purpose of this competition though. Let me know if you have any further questions.
", "rawMarkdown": "I consulted some of the samplers regarding case 3, and they provided the following motivation. First, to some degree it is a coincidence that these frequently occur near the foil, probably due to train domain sampling randomness. Itâs caused a lot of correlations in the train data that arenât present in the test set. \n\nSecond, there are instances in which a motor can be present without a visible flagellum. However these frequently represent sick or starving bacteria in unusual circumstances. The postdocs that work on this skip annotation in this case for our purposes of subtomogram averaging since you canât expect the structure of the bacteria to be helpful when theyâre so unhealthy. \n\nThis means that in this case, it is a bit of a âpredict the labelerâ kind of competition which is tough. Thatâs just the pattern we used and it might seem inconsistent, especially to our models. It does represent the underlying purpose of this competition though. Let me know if you have any further questions.", "votes": null }, { "id": "3206255", "postDate": "05/21/2025 04:59:23", "content": "\nThank you for the detailed clarification â that makes the intent and background much clearer.
\nIt seems that separating the presence of a motor from its structural quality (e.g., blurred, artifact-affected, missing flagella, etc.) might help both in terms of machine learning usability and interpretability for future datasets.
\nOf course, I understand that for this competition the current labeling policy reflects the intended challenge. Just wanted to share this thought for possible consideration in similar tasks going forward.
", "rawMarkdown": "andrewjdarley \n\nThank you for the detailed clarification â that makes the intent and background much clearer.\n\nIt seems that separating the presence of a motor from its structural quality (e.g., blurred, artifact-affected, missing flagella, etc.) might help both in terms of machine learning usability and interpretability for future datasets. \n\nOf course, I understand that for this competition the current labeling policy reflects the intended challenge. Just wanted to share this thought for possible consideration in similar tasks going forward.", "votes": null }, { "id": "3206285", "postDate": "05/21/2025 05:57:13", "content": "\nI found another confusing case.
\nThis tomogram slice contain clear image of a motor with a flagellum, but the entire motor region is overlap with foil region. I think the quality can be different from other subtomograms of a motor not covered with foil region, and possibly harms STA quality.
\nNote: This case is also labeled as positive.
\nlabeled as negative:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2F417cf061d88e340b5ce1786fa5431e47%2FScreenshot%202025-05-21%20at%2014.35.27.png?generation=1747806805268907&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2Fcd21984f3b884b1ac2c5ed383b1cf52d%2FScreenshot%202025-05-21%20at%2015.26.19.png?generation=1747808806486813&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2Fd7e1f55c10334398439d9cfd2bc2c89c%2FScreenshot%202025-05-21%20at%2015.30.54.png?generation=1747809071165915&alt=media\")
labeled as positive:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4910466%2Fd1e4032703fb22f1aea393388c60bef6%2FScreenshot%202025-05-21%20at%2018.33.48.png?generation=1747820101329639&alt=media\")
Thanks for bringing this up, it is a case I had not thought of. In this competition we donât have a locational restriction on motors even though that could quite possibly make sense. Expect this pattern to be present in the test set as they are clean motors, although distorted. Perhaps for future use this is something weâd throw out.
", "rawMarkdown": "Thanks for bringing this up, it is a case I had not thought of. In this competition we donât have a locational restriction on motors even though that could quite possibly make sense. Expect this pattern to be present in the test set as they are clean motors, although distorted. Perhaps for future use this is something weâd throw out.", "votes": null }, { "id": "3206632", "postDate": "05/21/2025 15:07:13", "content": "@andrewjdarley That's why every time I add strong cv model, the maximum ensemble function shows that this model predicts new false positives.
", "rawMarkdown": "andrewjdarley That's why every time I add strong cv model, the maximum ensemble function shows that this model predicts new false positives.", "votes": null }, { "id": "3207680", "postDate": "05/23/2025 05:55:03", "content": "\n\nThis means that in this case, it is a bit of a âpredict the labelerâ kind of competition which is tough. Thatâs just the pattern we used and it might seem inconsistent, especially to our models. It does represent the underlying purpose of this competition though. Let me know if you have any further questions.
\n
@andrewjdarley
\nJust to clarify, does this mean that the ambiguous labeling of motors without visible flagella also applies to the test dataset? Or is the test set curated differently to avoid such inconsistencies?
@tatamikenn This kind of label noise, if present in the test set, could significantly impact model performance and lead to frustrating local validation/leaderboard discrepancies.If \"negative\" can contain motor-like structures, it fundamentally changes the problem definition and evaluation.
", "rawMarkdown": "tatamikenn This kind of label noise, if present in the test set, could significantly impact model performance and lead to frustrating local validation/leaderboard discrepancies.If \"negative\" can contain motor-like structures, it fundamentally changes the problem definition and evaluation.", "votes": null }, { "id": "3214782", "postDate": "06/01/2025 04:26:56", "content": "@andrewjdarley does this negative-labeled tomo exist motor? My point generator produces a lot of points in this place.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4310004%2F7a426c9694b4ce973e3a27955d298fce%2F12.png?generation=1748751924202698&alt=media\")
Which tomoid is this? I can show it to an expert.
", "rawMarkdown": "Which tomoid is this? I can show it to an expert.", "votes": null }, { "id": "3216270", "postDate": "06/03/2025 10:15:07", "content": "@andrewjdarley tomo_098751
", "rawMarkdown": "andrewjdarley tomo_098751", "votes": null }, { "id": "3217068", "postDate": "06/04/2025 13:45:29", "content": "I canât give an entirely clear answer but Iâm confident that in more than 80% of the tomograms containing no flagellum the motor is not marked. However I did not label all the test data so I couldnât tell you with perfect certainty.
", "rawMarkdown": "I canât give an entirely clear answer but Iâm confident that in more than 80% of the tomograms containing no flagellum the motor is not marked. However I did not label all the test data so I couldnât tell you with perfect certainty.", "votes": null } ], "comments": [ { "id": 3204463, "author_name": "sacuscreed", "author_url": "", "post_date": "05/18/2025 10:50:26", "content": "Some days ago they had to rescore due some FN fix. So I think yes, test should be more thoroughly validated.
\nEDIT: here
", "votes": null, "replies": [] }, { "id": 3204585, "author_name": "andrewjdarley", "author_url": "", "post_date": "05/18/2025 14:05:02", "content": "Hey, this is a very valid concern. And as Ăngel brought up, it is the purpose of the leaderboard rescore. Since both our datasets were human annotated, we had a small amount of errors. However, upon analysis of a bunch of the top models, we were able to identify a significant number of mislabeled tomograms, mostly false negatives, that we corrected.
\nThis isnât me confirming perfection in the test set, rather our intentions to make the competition as competitive as possible. I can confirm that there are instances in which test tomograms are consistently mislabeled by almost all the submissions. So the non-perfect LB scores are still caused by non-perfect models.
", "votes": null, "replies": [ { "id": 3204668, "author_name": "tom99763", "author_url": "", "post_date": "05/18/2025 16:08:11", "content": "@andrewjdarley Great andrew. Thank you provide such a challenge competition with some dirty labeled data (it's very hard to accurately model your visual preference from the given label), absolutely increasing my skillset as a data scientist. I'm currently developing another way to handle this in unsupervised anomaly detection manner (one-class approach). Hope I can finish it before close.
", "votes": null, "replies": [] }, { "id": 3205059, "author_name": "tatamikenn", "author_url": "", "post_date": "05/19/2025 08:35:21", "content": "\nI looked through the negative tomograms in the training dataset and found a general tendency:
\nConsidering that the winning solution will be used for subtomogram averaging (STA), I think low-quality subtomograms should be removed, even if they contain motor-like structures. So, cases 1 and 2 make sense.
\nHowever, what about case 3? Are those marked as negative simply because theyâre located in regions where humans are more likely to overlook them?
\nOr is there another valid reason for excluding them?
Example picture of case 3:
\n
\n
\n
I consulted some of the samplers regarding case 3, and they provided the following motivation. First, to some degree it is a coincidence that these frequently occur near the foil, probably due to train domain sampling randomness. Itâs caused a lot of correlations in the train data that arenât present in the test set.
\nSecond, there are instances in which a motor can be present without a visible flagellum. However these frequently represent sick or starving bacteria in unusual circumstances. The postdocs that work on this skip annotation in this case for our purposes of subtomogram averaging since you canât expect the structure of the bacteria to be helpful when theyâre so unhealthy.
\nThis means that in this case, it is a bit of a âpredict the labelerâ kind of competition which is tough. Thatâs just the pattern we used and it might seem inconsistent, especially to our models. It does represent the underlying purpose of this competition though. Let me know if you have any further questions.
", "votes": null, "replies": [ { "id": 3206255, "author_name": "tatamikenn", "author_url": "", "post_date": "05/21/2025 04:59:23", "content": "\nThank you for the detailed clarification â that makes the intent and background much clearer.
\nIt seems that separating the presence of a motor from its structural quality (e.g., blurred, artifact-affected, missing flagella, etc.) might help both in terms of machine learning usability and interpretability for future datasets.
\nOf course, I understand that for this competition the current labeling policy reflects the intended challenge. Just wanted to share this thought for possible consideration in similar tasks going forward.
", "votes": null, "replies": [ { "id": 3206285, "author_name": "tatamikenn", "author_url": "", "post_date": "05/21/2025 05:57:13", "content": "\nI found another confusing case.
\nThis tomogram slice contain clear image of a motor with a flagellum, but the entire motor region is overlap with foil region. I think the quality can be different from other subtomograms of a motor not covered with foil region, and possibly harms STA quality.
\nNote: This case is also labeled as positive.
\nlabeled as negative:
\n
\n
\n
labeled as positive:
\n
Thanks for bringing this up, it is a case I had not thought of. In this competition we donât have a locational restriction on motors even though that could quite possibly make sense. Expect this pattern to be present in the test set as they are clean motors, although distorted. Perhaps for future use this is something weâd throw out.
", "votes": null, "replies": [ { "id": 3206632, "author_name": "tom99763", "author_url": "", "post_date": "05/21/2025 15:07:13", "content": "@andrewjdarley That's why every time I add strong cv model, the maximum ensemble function shows that this model predicts new false positives.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 3207680, "author_name": "tatamikenn", "author_url": "", "post_date": "05/23/2025 05:55:03", "content": "\n\nThis means that in this case, it is a bit of a âpredict the labelerâ kind of competition which is tough. Thatâs just the pattern we used and it might seem inconsistent, especially to our models. It does represent the underlying purpose of this competition though. Let me know if you have any further questions.
\n
@andrewjdarley
\nJust to clarify, does this mean that the ambiguous labeling of motors without visible flagella also applies to the test dataset? Or is the test set curated differently to avoid such inconsistencies?
I canât give an entirely clear answer but Iâm confident that in more than 80% of the tomograms containing no flagellum the motor is not marked. However I did not label all the test data so I couldnât tell you with perfect certainty.
", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 3205094, "author_name": "zainafzal1906", "author_url": "", "post_date": "05/19/2025 10:17:26", "content": "Nice Work đ
", "votes": null, "replies": [] }, { "id": 3205531, "author_name": "maxchen303", "author_url": "", "post_date": "05/20/2025 04:23:48", "content": "Thanks for sharing. I also noticed a sample, tomo_401341, where only one motor is labeled at slice 166, but there appears to be another motor-like structure around slice 197.
| row_id | \ntomo_id | \nMotor axis 0 | \nMotor axis 1 | \nMotor axis 2 | \nArray shape (axis 0) | \nArray shape (axis 1) | \nArray shape (axis 2) | \nVoxel spacing | \nNumber of motors | \n
|---|---|---|---|---|---|---|---|---|---|
| 191 | \ntomo_401341 | \n166.0 | \n612.0 | \n722.0 | \n300 | \n960 | \n928 | \n13.1 | \n1 | \n
@maxchen303 This case shows you don't naively use single point as your ground truth.
", "votes": null, "replies": [] }, { "id": 3205703, "author_name": "sacuscreed", "author_url": "", "post_date": "05/20/2025 09:02:44", "content": "So may be would be good idea pseudolabel most confident train predictions right?
", "votes": null, "replies": [] } ] }, { "id": 3205549, "author_name": "sakvaua", "author_url": "", "post_date": "05/20/2025 05:05:00", "content": "I found many of those as well, and relabeled the most obvious cases, but I think that this competition is about \"predict the labeler\" so I'm not sure whether labling everything will make our LB scores better if test cases were labeled with the same set of heuristic rules as the train.
", "votes": null, "replies": [ { "id": 3205713, "author_name": "sacuscreed", "author_url": "", "post_date": "05/20/2025 09:11:27", "content": "Hi.
\n\n\nâŚupon analysis of a bunch of the top models, we were able to identify a significant number of mislabeled tomograms, mostly false negatives, that we corrected.
\n
They're using most confident submissions to carefully check test annotations. So I think will be more \"predict label\" than \"labeler\".
", "votes": null, "replies": [] } ] }, { "id": 3205760, "author_name": "aliraza1709", "author_url": "", "post_date": "05/20/2025 10:21:00", "content": "Good Work đ
", "votes": null, "replies": [] }, { "id": 3207862, "author_name": "bruhjjwal", "author_url": "", "post_date": "05/23/2025 10:40:42", "content": "@tatamikenn This kind of label noise, if present in the test set, could significantly impact model performance and lead to frustrating local validation/leaderboard discrepancies.If \"negative\" can contain motor-like structures, it fundamentally changes the problem definition and evaluation.
", "votes": null, "replies": [] }, { "id": 3214782, "author_name": "tom99763", "author_url": "", "post_date": "06/01/2025 04:26:56", "content": "@andrewjdarley does this negative-labeled tomo exist motor? My point generator produces a lot of points in this place.
\n
Which tomoid is this? I can show it to an expert.
", "votes": null, "replies": [ { "id": 3216270, "author_name": "tom99763", "author_url": "", "post_date": "06/03/2025 10:15:07", "content": "@andrewjdarley tomo_098751
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3204442": "As many contestants may have already noticed, I found dozens of tomograms in the training dataset labeled as \"negative\" that actually appear to contain flagellar motor-like structures (see picture below).\nWe can correct these labels in the training set, but I'm concerned that **some test tomograms might also include similar mislabeled structures**.\nHas the negative test dataset been more thoroughly validated to ensure it doesn't include motors?\n\n@braxtonowens @jacksonpond\nI would greatly appreciate it if you could kindly clarify these concerns.\n\n**Sample Tomograms**:\n\nđ I'm not a domain expert, so any advice on how to interpret the \"negative\" or \"positive\" labels in these sample tomograms would be greatly appreciated!\n\n\n\n\n", "3204463": "Some days ago they had to rescore due some FN fix. So I think yes, test should be more thoroughly validated.\n\nEDIT: [here](https://www.kaggle.com/competitions/byu-locating-bacterial-flagellar-motors-2025/discussion/574729)", "3204585": "Hey, this is a very valid concern. And as Ăngel brought up, it is the purpose of the leaderboard rescore. Since both our datasets were human annotated, we had a small amount of errors. However, upon analysis of a bunch of the top models, we were able to identify a significant number of mislabeled tomograms, mostly false negatives, that we corrected.\n\nThis isnât me confirming perfection in the test set, rather our intentions to make the competition as competitive as possible. I can confirm that there are instances in which test tomograms are consistently mislabeled by almost all the submissions. So the non-perfect LB scores are still caused by non-perfect models.", "3204668": "andrewjdarley Great andrew. Thank you provide such a challenge competition with some dirty labeled data (it's very hard to accurately model your visual preference from the given label), absolutely increasing my skillset as a data scientist. I'm currently developing another way to handle this in unsupervised anomaly detection manner (one-class approach). Hope I can finish it before close.", "3205059": "andrewjdarley\n\nI looked through the negative tomograms in the training dataset and found a general tendency:\n\n1. Blurred images tend to be marked as negative.\n2. Motors with artifacts tend to be marked as negative.\n3. Motors near the foil edge tend to be marked as negative (possibly due to missing flagella).\n\nConsidering that the winning solution will be used for subtomogram averaging (STA), I think low-quality subtomograms should be removed, even if they contain motor-like structures. So, cases 1 and 2 make sense.\n\nHowever, what about case 3? Are those marked as negative simply because theyâre located in regions where humans are more likely to overlook them?\nOr is there another valid reason for excluding them?\n\n**Example picture of case 3**:\n\n\n\n", "3205094": "Nice Work đ", "3205531": "Thanks for sharing. I also noticed a sample, `tomo_401341`, where only one motor is labeled at slice 166, but there appears to be another motor-like structure around slice 197.\n\n| row_id | tomo_id | Motor axis 0 | Motor axis 1 | Motor axis 2 | Array shape (axis 0) | Array shape (axis 1) | Array shape (axis 2) | Voxel spacing | Number of motors |\n|--------|-------------|---------------|---------------|---------------|------------------------|------------------------|------------------------|----------------|-------------------|\n| 191 | tomo_401341 | 166.0 | 612.0 | 722.0 | 300 | 960 | 928 | 13.1 | 1 |\n\n\n### slice 166:\n\n### slice 197:\n",
"3205542": "maxchen303 This case shows you don't naively use single point as your ground truth.",
"3205549": "I found many of those as well, and relabeled the most obvious cases, but I think that this competition is about \"predict the labeler\" so I'm not sure whether labling everything will make our LB scores better if test cases were labeled with the same set of heuristic rules as the train.",
"3205703": "So may be would be good idea pseudolabel most confident train predictions right?",
"3205713": "Hi.\n\n>...upon analysis of a bunch of the top models, we were able to identify a significant number of mislabeled tomograms, mostly false negatives, that we corrected.\n\nThey're using most confident submissions to carefully check test annotations. So I think will be more \"predict label\" than \"labeler\".",
"3205760": "Good Work đ",
"3205926": "I consulted some of the samplers regarding case 3, and they provided the following motivation. First, to some degree it is a coincidence that these frequently occur near the foil, probably due to train domain sampling randomness. Itâs caused a lot of correlations in the train data that arenât present in the test set. \n\nSecond, there are instances in which a motor can be present without a visible flagellum. However these frequently represent sick or starving bacteria in unusual circumstances. The postdocs that work on this skip annotation in this case for our purposes of subtomogram averaging since you canât expect the structure of the bacteria to be helpful when theyâre so unhealthy. \n\nThis means that in this case, it is a bit of a âpredict the labelerâ kind of competition which is tough. Thatâs just the pattern we used and it might seem inconsistent, especially to our models. It does represent the underlying purpose of this competition though. Let me know if you have any further questions.",
"3206255": "andrewjdarley \n\nThank you for the detailed clarification â that makes the intent and background much clearer.\n\nIt seems that separating the presence of a motor from its structural quality (e.g., blurred, artifact-affected, missing flagella, etc.) might help both in terms of machine learning usability and interpretability for future datasets. \n\nOf course, I understand that for this competition the current labeling policy reflects the intended challenge. Just wanted to share this thought for possible consideration in similar tasks going forward.",
"3206285": "andrewjdarley \n\nI found another confusing case.\n\nThis tomogram slice contain clear image of a motor with a flagellum, but the entire motor region is overlap with foil region. I think the quality can be different from other subtomograms of a motor not covered with foil region, and possibly harms STA quality.\n\nNote: This case is also labeled as **positive**.\n\n**labeled as negative**:\n\n\n\n\n\n**labeled as positive**:\n\n",
"3206626": "Thanks for bringing this up, it is a case I had not thought of. In this competition we donât have a locational restriction on motors even though that could quite possibly make sense. Expect this pattern to be present in the test set as they are clean motors, although distorted. Perhaps for future use this is something weâd throw out.",
"3206632": "andrewjdarley That's why every time I add strong cv model, the maximum ensemble function shows that this model predicts new false positives.",
"3207680": "> This means that in this case, it is a bit of a âpredict the labelerâ kind of competition which is tough. Thatâs just the pattern we used and it might seem inconsistent, especially to our models. It does represent the underlying purpose of this competition though. Let me know if you have any further questions.\n\n@andrewjdarley\nJust to clarify, does this mean that the ambiguous labeling of motors without visible flagella also applies to the test dataset? Or is the test set curated differently to avoid such inconsistencies?",
"3207862": "tatamikenn This kind of label noise, if present in the test set, could significantly impact model performance and lead to frustrating local validation/leaderboard discrepancies.If \"negative\" can contain motor-like structures, it fundamentally changes the problem definition and evaluation.",
"3214782": "andrewjdarley does this negative-labeled tomo exist motor? My point generator produces a lot of points in this place.\n\n",
"3215943": "Which tomoid is this? I can show it to an expert.",
"3216270": "andrewjdarley tomo_098751",
"3217068": "I canât give an entirely clear answer but Iâm confident that in more than 80% of the tomograms containing no flagellum the motor is not marked. However I did not label all the test data so I couldnât tell you with perfect certainty."
},
"source": "meta"
}