![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F7e8bb892dd7ab2ea73bc6ee4f1307192%2FSelection_692.png?generation=1731800092073125&alt=media\")
The hidden test set is being updated to address a data issue. Once that is complete, all previous submission notebook will be re-run against the updated dataset. This process will take some time, during which, the leaderboard will be in a state of weirdness. This thread will be updated once the process is complete.
", "messages": [ { "id": "3045508", "postDate": "11/14/2024 14:24:11", "content": "The hidden test set is being updated to address a data issue. Once that is complete, all previous submission notebook will be re-run against the updated dataset. This process will take some time, during which, the leaderboard will be in a state of weirdness. This thread will be updated once the process is complete.
", "rawMarkdown": "The hidden test set is being updated to address a data issue. Once that is complete, all previous submission notebook will be re-run against the updated dataset. This process will take some time, during which, the leaderboard will be in a state of weirdness. This thread will be updated once the process is complete.", "votes": null }, { "id": "3045538", "postDate": "11/14/2024 15:03:22", "content": "Thank you for updating the test set. Can you also confirm that this script is used for scoring? (With beta=4 and distance_multiplier=0.5)
\nhttps://www.kaggle.com/code/metric/czi-cryoet-84969
", "rawMarkdown": "Thank you for updating the test set. Can you also confirm that this script is used for scoring? (With beta=4 and distance_multiplier=0.5)\n\nhttps://www.kaggle.com/code/metric/czi-cryoet-84969", "votes": null }, { "id": "3045869", "postDate": "11/14/2024 21:38:53", "content": "Yes, correct.
", "rawMarkdown": "Yes, correct.", "votes": null }, { "id": "3047356", "postDate": "11/16/2024 15:33:28", "content": "thanks for the update!
", "rawMarkdown": "thanks for the update!", "votes": null }, { "id": "3047520", "postDate": "11/16/2024 19:15:21", "content": "Even if I am in the top 2 at this moment, I must admit that there is a problem with evaluation algorithm š. The results are too high !
\nDieter said the code is this one :
\nhttps://www.kaggle.com/code/metric/czi-cryoet-84969
I went and gave a look to it. I think there is a mistake in this line :
\nraw_matches = ref_tree.query_ball_tree(candidate_tree, r=reference_radius)
It should be the inverse to have true positives in ref_tree instead of candidate tree :
\nraw_matches = candidate_tree.query_ball_tree(ref_tree, r=reference_radius)
The result is that submitting multiple candidate points in the radius give false 'true positives'.
\nPlease let me know if I am wrong.
\nI hope this helps !
it would be good if you could illustrate the problem with dummy data.
\ni am suggesting using linear_sum_assignment for cross checking the eval code
\nhttps://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.linear_sum_assignment.html
No problem, I made a quick notebook. You can see that before correction we have too much \"true positives\", and even negative (-1) \"false negatives\".
\nThe score is also 1,36 > 1, which should not be realistic š
\nhttps://www.kaggle.com/code/mjz1977/czii-metric-is-it-a-mistake
After correction it is OK
", "rawMarkdown": "No problem, I made a quick notebook. You can see that before correction we have too much \"true positives\", and even negative (-1) \"false negatives\".\nThe score is also 1,36 > 1, which should not be realistic š \nhttps://www.kaggle.com/code/mjz1977/czii-metric-is-it-a-mistake\n\nAfter correction it is OK", "votes": null }, { "id": "3047601", "postDate": "11/16/2024 23:35:46", "content": "@inversion @kharrington yet another evaluation bug discovered by @mjz1977
\nš š š
@mjz1977 good š. Thank you for the notebook. We've been checking it over and will followup/update on Monday.
", "rawMarkdown": "mjz1977 good š. Thank you for the notebook. We've been checking it over and will followup/update on Monday.", "votes": null }, { "id": "3047844", "postDate": "11/17/2024 08:57:21", "content": "Probably has an easy and fast fix. But since the score counts hits as coordinates inside a ground truth radius why not to give han explicit spherical mask and do evalute with CELoss directly? The statistics of ML are still my Achilles heel. Sorry if is a terrible suggestion.
", "rawMarkdown": "Probably has an easy and fast fix. But since the score counts hits as coordinates inside a ground truth radius why not to give han explicit spherical mask and do evalute with CELoss directly? The statistics of ML are still my Achilles heel. Sorry if is a terrible suggestion.", "votes": null }, { "id": "3048815", "postDate": "11/18/2024 11:32:51", "content": "I think that evaluation should be OK after the correction of wrong line. If staff go to another method, they will have to validate it before, and perhaps some other issues will appear š
", "rawMarkdown": "I think that evaluation should be OK after the correction of wrong line. If staff go to another method, they will have to validate it before, and perhaps some other issues will appear š", "votes": null }, { "id": "3049179", "postDate": "11/18/2024 19:50:36", "content": "@mjz1977 Would you mind reaching out to me via email or Kaggle contact form?
", "rawMarkdown": "mjz1977 Would you mind reaching out to me via email or Kaggle contact form?", "votes": null } ], "comments": [ { "id": 3045538, "author_name": "christofhenkel", "author_url": "", "post_date": "11/14/2024 15:03:22", "content": "Thank you for updating the test set. Can you also confirm that this script is used for scoring? (With beta=4 and distance_multiplier=0.5)
\nhttps://www.kaggle.com/code/metric/czi-cryoet-84969
", "votes": null, "replies": [ { "id": 3045869, "author_name": "inversion", "author_url": "", "post_date": "11/14/2024 21:38:53", "content": "Yes, correct.
", "votes": null, "replies": [] } ] }, { "id": 3047356, "author_name": "hengck23", "author_url": "", "post_date": "11/16/2024 15:33:28", "content": "thanks for the update!
", "votes": null, "replies": [] }, { "id": 3047520, "author_name": "mjz1977", "author_url": "", "post_date": "11/16/2024 19:15:21", "content": "Even if I am in the top 2 at this moment, I must admit that there is a problem with evaluation algorithm š. The results are too high !
\nDieter said the code is this one :
\nhttps://www.kaggle.com/code/metric/czi-cryoet-84969
I went and gave a look to it. I think there is a mistake in this line :
\nraw_matches = ref_tree.query_ball_tree(candidate_tree, r=reference_radius)
It should be the inverse to have true positives in ref_tree instead of candidate tree :
\nraw_matches = candidate_tree.query_ball_tree(ref_tree, r=reference_radius)
The result is that submitting multiple candidate points in the radius give false 'true positives'.
\nPlease let me know if I am wrong.
\nI hope this helps !
it would be good if you could illustrate the problem with dummy data.
\ni am suggesting using linear_sum_assignment for cross checking the eval code
\nhttps://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.linear_sum_assignment.html
No problem, I made a quick notebook. You can see that before correction we have too much \"true positives\", and even negative (-1) \"false negatives\".
\nThe score is also 1,36 > 1, which should not be realistic š
\nhttps://www.kaggle.com/code/mjz1977/czii-metric-is-it-a-mistake
After correction it is OK
", "votes": null, "replies": [ { "id": 3047621, "author_name": "kharrington", "author_url": "", "post_date": "11/17/2024 00:46:12", "content": "@mjz1977 good š. Thank you for the notebook. We've been checking it over and will followup/update on Monday.
", "votes": null, "replies": [] } ] } ] }, { "id": 3047601, "author_name": "hengck23", "author_url": "", "post_date": "11/16/2024 23:35:46", "content": "@inversion @kharrington yet another evaluation bug discovered by @mjz1977
\nš š š
Probably has an easy and fast fix. But since the score counts hits as coordinates inside a ground truth radius why not to give han explicit spherical mask and do evalute with CELoss directly? The statistics of ML are still my Achilles heel. Sorry if is a terrible suggestion.
", "votes": null, "replies": [ { "id": 3048815, "author_name": "mjz1977", "author_url": "", "post_date": "11/18/2024 11:32:51", "content": "I think that evaluation should be OK after the correction of wrong line. If staff go to another method, they will have to validate it before, and perhaps some other issues will appear š
", "votes": null, "replies": [] } ] } ] }, { "id": 3049179, "author_name": "inversion", "author_url": "", "post_date": "11/18/2024 19:50:36", "content": "@mjz1977 Would you mind reaching out to me via email or Kaggle contact form?
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "3045508": "The hidden test set is being updated to address a data issue. Once that is complete, all previous submission notebook will be re-run against the updated dataset. This process will take some time, during which, the leaderboard will be in a state of weirdness. This thread will be updated once the process is complete.", "3045538": "Thank you for updating the test set. Can you also confirm that this script is used for scoring? (With beta=4 and distance_multiplier=0.5)\n\nhttps://www.kaggle.com/code/metric/czi-cryoet-84969", "3045869": "Yes, correct.", "3047356": "thanks for the update!", "3047520": "Even if I am in the top 2 at this moment, I must admit that there is a problem with evaluation algorithm š. The results are too high !\n\nDieter said the code is this one :\nhttps://www.kaggle.com/code/metric/czi-cryoet-84969\n\nI went and gave a look to it. I think there is a mistake in this line :\n`raw_matches = ref_tree.query_ball_tree(candidate_tree, r=reference_radius)`\n\nIt should be the inverse to have true positives in ref_tree instead of candidate tree :\n`raw_matches = candidate_tree.query_ball_tree(ref_tree, r=reference_radius)`\n\nThe result is that submitting multiple candidate points in the radius give false 'true positives'.\n\nPlease let me know if I am wrong.\nI hope this helps !", "3047535": "it would be good if you could illustrate the problem with dummy data.\n\ni am suggesting using linear_sum_assignment for cross checking the eval code\nhttps://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.linear_sum_assignment.html", "3047545": "No problem, I made a quick notebook. You can see that before correction we have too much \"true positives\", and even negative (-1) \"false negatives\".\nThe score is also 1,36 > 1, which should not be realistic š \nhttps://www.kaggle.com/code/mjz1977/czii-metric-is-it-a-mistake\n\nAfter correction it is OK", "3047601": "inversion @kharrington yet another evaluation bug discovered by @mjz1977 \nš š š\n\n", "3047621": "mjz1977 good š. Thank you for the notebook. We've been checking it over and will followup/update on Monday.", "3047844": "Probably has an easy and fast fix. But since the score counts hits as coordinates inside a ground truth radius why not to give han explicit spherical mask and do evalute with CELoss directly? The statistics of ML are still my Achilles heel. Sorry if is a terrible suggestion.", "3048815": "I think that evaluation should be OK after the correction of wrong line. If staff go to another method, they will have to validate it before, and perhaps some other issues will appear š", "3049179": "mjz1977 Would you mind reaching out to me via email or Kaggle contact form?" }, "source": "meta" }