{ "id": 575280, "title": "What is the score in the leaderboard and why the score I got locally is so different from the score I submitted?", "url": "/competitions/byu-locating-bacterial-flagellar-motors-2025/discussion/575280", "author_name": "Yuzhi Xu", "post_date": "2025-04-27T12:46:15.804000", "votes": 4, "comment_count": 34, "views": 0, "content": "

Is the final score the same as result calculated from the metric notebook?

\n

The score that I submit is about 0.6, but the score I get locally is over 0.9.

\n

The way I run locally is as follows. I use the notebook to divide the tomograms in to training set and validation set. And I use the tomograms in validation set which only have one motor to calculate the metric. And I also add some tomograms which have no motor, and the proportion is the same as the (tomograms num without motor)/(total tomograms num).

", "messages": [ { "id": 3192378, "postDate": "2025-05-02T19:01:32.947Z", "content": "

You are in violation. You work as a team, which means you exchange info. Yet you have not teamed on Kaggle.

\n

Private sharing outside kaggle teams is prohibited by the rules.

\n

Issue with the way you do it is that you can submit more in a day than if you were in a single Kaggle team. That's why it is unfair and that's why it is prohibited by Kaggle rules.

", "rawMarkdown": "You are in violation. You work as a team, which means you exchange info. Yet you have not teamed on Kaggle.\n\nPrivate sharing outside kaggle teams is prohibited by the rules. \n\nIssue with the way you do it is that you can submit more in a day than if you were in a single Kaggle team. That's why it is unfair and that's why it is prohibited by Kaggle rules.", "votes": 8, "replies": [ { "id": 3192550, "postDate": "2025-05-03T03:27:26.540Z", "content": "

We have been using nearly all our submissions for the last 2 weeks just fixing data because the false labels in the test set really messed stuff up for us. A theoretical 20 subs a day since that point is a pretty large advantage…

", "rawMarkdown": "We have been using nearly all our submissions for the last 2 weeks just fixing data because the false labels in the test set really messed stuff up for us. A theoretical 20 subs a day since that point is a pretty large advantage...", "votes": 4 }, { "id": 3192707, "postDate": "2025-05-03T09:29:35.783Z", "content": "

I concur. I am not active here, but in arc prize I am wasting my single sub a day trying to debug my submission code. I would be done in a day with 20 subs.

", "rawMarkdown": "I concur. I am not active here, but in arc prize I am wasting my single sub a day trying to debug my submission code. I would be done in a day with 20 subs.", "votes": 2 } ] }, { "id": 3188337, "postDate": "2025-04-27T12:46:15.803Z", "content": "

Is the final score the same as result calculated from the metric notebook?

\n

The score that I submit is about 0.6, but the score I get locally is over 0.9.

\n

The way I run locally is as follows. I use the notebook to divide the tomograms in to training set and validation set. And I use the tomograms in validation set which only have one motor to calculate the metric. And I also add some tomograms which have no motor, and the proportion is the same as the (tomograms num without motor)/(total tomograms num).

", "rawMarkdown": "Is the final score the same as result calculated from the [metric notebook](https://www.kaggle.com/code/metric/byu-biophysics-91249)?\n\nThe score that I submit is about 0.6, but the score I get locally is over 0.9.\n\nThe way I run locally is as follows. I use the [notebook](https://www.kaggle.com/code/andrewjdarley/parse-data/notebook) to divide the tomograms in to training set and validation set. And I use the tomograms in validation set which only have one motor to calculate the metric. And I also add some tomograms which have no motor, and the proportion is the same as the (tomograms num without motor)/(total tomograms num).", "votes": 4 }, { "id": 3188580, "postDate": "2025-04-27T21:44:46.617Z", "content": "

Yes, the leaderboard score uses that metric. But it's calculated on a hidden set of tomograms, so the result can be different from your validation score from a number of reasons (overfitting your validation set, different sizes, voxel spacings, etc.).

", "rawMarkdown": "Yes, the leaderboard score uses that metric. But it's calculated on a hidden set of tomograms, so the result can be different from your validation score from a number of reasons (overfitting your validation set, different sizes, voxel spacings, etc.).", "votes": 1 }, { "id": 3201108, "postDate": "2025-05-13T12:28:49.760Z", "content": "
\n

Thanks for the heads up. We're not in violation, all 4 of us on the team will be merged subsequently and there are no additional accounts submitted

\n
\n

The fourth member of the team, will he/she finally combine with you or will he/she remain in solo mode?

", "rawMarkdown": ">Thanks for the heads up. We're not in violation, all 4 of us on the team will be merged subsequently and there are no additional accounts submitted\n\nThe fourth member of the team, will he/she finally combine with you or will he/she remain in solo mode?", "replies": [ { "id": 3201113, "postDate": "2025-05-13T12:34:49.010Z", "content": "

merge

", "rawMarkdown": "merge ", "votes": 1 }, { "id": 3201117, "postDate": "2025-05-13T12:41:47.690Z", "content": "

Okay, thanks!

", "rawMarkdown": "Okay, thanks!" }, { "id": 3202495, "postDate": "2025-05-15T13:24:58.003Z", "rawMarkdown": "", "votes": 2, "isDeleted": true }, { "id": 3202504, "postDate": "2025-05-15T13:43:45.590Z", "content": "

Thank you for your attention. Although we are in the same school, I don’t know him and have no idea who he is. Our teammate is someone else, who tried the 3D segmentation approach alone. He didn’t submit many attempts, and none of them achieved good results.

", "rawMarkdown": "Thank you for your attention. Although we are in the same school, I don’t know him and have no idea who he is. Our teammate is someone else, who tried the 3D segmentation approach alone. He didn’t submit many attempts, and none of them achieved good results.", "votes": -1 }, { "id": 3202511, "postDate": "2025-05-15T13:57:11.310Z", "content": "

Right now, he's in the Top 18, It's a shame that all together you exceed the number of submissions.

", "rawMarkdown": "Right now, he's in the Top 18, It's a shame that all together you exceed the number of submissions." }, { "id": 3202516, "postDate": "2025-05-15T14:03:19.417Z", "rawMarkdown": "", "votes": 3, "isDeleted": true }, { "id": 3202518, "postDate": "2025-05-15T14:08:47.767Z", "content": "

Wow is that another drama?

", "rawMarkdown": "Wow is that another drama?", "votes": 4 }, { "id": 3202519, "postDate": "2025-05-15T14:08:53.667Z", "content": "

This is an interesting story. At first, we discovered that we were at the same school, so my teammate privately messaged him to ask if he wanted to team up, but he never replied, so nothing came of it.😂

", "rawMarkdown": "This is an interesting story. At first, we discovered that we were at the same school, so my teammate privately messaged him to ask if he wanted to team up, but he never replied, so nothing came of it.😂", "votes": -5 }, { "id": 3202525, "postDate": "2025-05-15T14:15:56.110Z", "content": "

Respectfully, I do believe what you have done already counts as cheating according to Kaggle's ruleset as you have had a massive advantage by getting more daily subs than the rest of the community… I think this issue would stand regardless of if your 4th teammate merged or not. I would suggest reaching out to Kaggle admins.

", "rawMarkdown": "Respectfully, I do believe what you have done already counts as cheating according to Kaggle's ruleset as you have had a massive advantage by getting more daily subs than the rest of the community... I think this issue would stand regardless of if your 4th teammate merged or not. I would suggest reaching out to Kaggle admins. ", "votes": 5 }, { "id": 3202530, "postDate": "2025-05-15T14:22:49.417Z", "content": "

Thank you for your suggestion. We will explain the situation. May I ask how to get in touch with the administrator?

", "rawMarkdown": "Thank you for your suggestion. We will explain the situation. May I ask how to get in touch with the administrator?", "votes": 1 }, { "id": 3202553, "postDate": "2025-05-15T14:47:56.287Z", "content": "

Honestly, not too sure! I know you can @ admins as well as discord but I am not sure outside of that. :/

", "rawMarkdown": "Honestly, not too sure! I know you can @ admins as well as discord but I am not sure outside of that. :/" }, { "id": 3202557, "postDate": "2025-05-15T14:54:02.720Z", "content": "

@yyyy0201 You can contact kaggle staffs by clicking contact button from their profile.

", "rawMarkdown": "@yyyy0201 You can contact kaggle staffs by clicking contact button from their profile." }, { "id": 3202576, "postDate": "2025-05-15T15:33:47.590Z", "content": "

As pointed out you are in violation of rules, and you still are. Honestly, I think there is a good chance you are just removed silently from the competition when private LB is finalizing. So dont complain then.

", "rawMarkdown": "As pointed out you are in violation of rules, and you still are. Honestly, I think there is a good chance you are just removed silently from the competition when private LB is finalizing. So dont complain then.", "votes": 2 }, { "id": 3202790, "postDate": "2025-05-15T23:16:49.653Z", "content": "

No, I'm telling you very clearly that I'm not.

", "rawMarkdown": "No, I'm telling you very clearly that I'm not." }, { "id": 3202817, "postDate": "2025-05-16T00:40:05.400Z", "content": "

@majiaqi111 Make sure you never use part of their results or exchange privately. I believe you are not cheater cuz I saw you join many comps before.
\nActually I have asked the kaggle staffs, they say they'll investigate @yyyy0201's situation when this competition closes.

", "rawMarkdown": "@majiaqi111 Make sure you never use part of their results or exchange privately. I believe you are not cheater cuz I saw you join many comps before. \nActually I have asked the kaggle staffs, they say they'll investigate @yyyy0201's situation when this competition closes.", "votes": 1 }, { "id": 3202824, "postDate": "2025-05-16T01:03:26.850Z", "content": "

Yes, I can ensure that.

", "rawMarkdown": "Yes, I can ensure that." }, { "id": 3202832, "postDate": "2025-05-16T01:56:11.717Z", "content": "

I’m not really sure what there is to investigate. As we know for the last several weeks they have been allowed more than 5 subs per day. Due to the inconsistent lb of this competition that is a massive advantage, that’s clearly against the rules, and they have had this advantage for weeks if not months… if this is permitted now will other kagglers be able to private share before teaming to advance ourselves on the leaderboard?

", "rawMarkdown": "I’m not really sure what there is to investigate. As we know for the last several weeks they have been allowed more than 5 subs per day. Due to the inconsistent lb of this competition that is a massive advantage, that’s clearly against the rules, and they have had this advantage for weeks if not months… if this is permitted now will other kagglers be able to private share before teaming to advance ourselves on the leaderboard? ", "votes": 3 }, { "id": 3202926, "postDate": "2025-05-16T05:53:00.137Z", "content": "

\"Our teammate is someone else, who tried the 3D segmentation approach alone\". It turns out that everyone can contact one another, ask each other to try different methods, and verify what works, using up someone else’s submission quota in the process then team up at the end. That’s the biggest solution in the competition.

", "rawMarkdown": "\"Our teammate is someone else, who tried the 3D segmentation approach **alone**\". It turns out that everyone can contact one another, ask each other to try different methods, and verify what works, using up someone else’s submission quota in the process then team up at the end. That’s the biggest solution in the competition." }, { "id": 3202950, "postDate": "2025-05-16T06:39:30.783Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3207704, "postDate": "2025-05-23T06:41:25.043Z", "content": "

OK, thanks! I'll try later.

", "rawMarkdown": "OK, thanks! I'll try later.\n\n", "votes": 1 } ] }, { "id": 3189400, "postDate": "2025-04-29T06:21:32.273Z", "content": "

Is there anything anyone knows about the test set? We get very stable performance across all five folds of a cross-validation on the training set and we do very minimal to no tuning of hyperparameters, so we would not expect too much overfitting. Yet, the best CV f2 is 0.949 and the best lb score is 0.811. There must be a shift in distribution: either there is much fewer/more motors in the lb than the train set or there is some other domain shift that we were not able to account for.

", "rawMarkdown": "Is there anything anyone knows about the test set? We get very stable performance across all five folds of a cross-validation on the training set and we do very minimal to no tuning of hyperparameters, so we would not expect too much overfitting. Yet, the best CV f2 is 0.949 and the best lb score is 0.811. There must be a shift in distribution: either there is much fewer/more motors in the lb than the train set or there is some other domain shift that we were not able to account for. ", "replies": [ { "id": 3189428, "postDate": "2025-04-29T07:29:09.817Z", "content": "

Are you also using Yolo based models or 3D unet or something else??

", "rawMarkdown": "Are you also using Yolo based models or 3D unet or something else??\n", "votes": 1, "isDeleted": true }, { "id": 3189434, "postDate": "2025-04-29T07:48:10.207Z", "content": "

Whether to use additional datasets?
\n
\nOur best single model also scores around LB 84

", "rawMarkdown": "Whether to use additional datasets?\n~~Our best single model also scores around LB 81~~\nOur best single model also scores around LB 84\n", "replies": [ { "id": 3189636, "postDate": "2025-04-29T14:08:43.043Z", "content": "

just the official dataset so far

", "rawMarkdown": "just the official dataset so far", "votes": 1 }, { "id": 3191805, "postDate": "2025-05-02T06:47:33.490Z", "content": "

Who is \"our\"? I see you are alone on leaderboard?

", "rawMarkdown": "Who is \"our\"? I see you are alone on leaderboard?", "votes": 7 }, { "id": 3192201, "postDate": "2025-05-02T15:13:27.290Z", "content": "

We haven't merged teams yet~

", "rawMarkdown": "We haven't merged teams yet~", "votes": -8 }, { "id": 3192284, "postDate": "2025-05-02T17:00:09.727Z", "content": "

Which means you are breaking the rules by private sharing. That is reason for disqualification. Especially in a competition where you need to rely on leaderboard you cheat by having effectively more submissions than other teams.

", "rawMarkdown": "Which means you are breaking the rules by private sharing. That is reason for disqualification. Especially in a competition where you need to rely on leaderboard you cheat by having effectively more submissions than other teams. ", "votes": 6 }, { "id": 3193170, "postDate": "2025-05-04T01:58:54.927Z", "content": "

@yyyy0201 I see that you and your teammates are from the same institution. However, you exchange private notebooks when you and your partners haven't been merged yet. That’s a violation, as it allows you to easily make massive submissions in a day and share the results which is unfair.

", "rawMarkdown": "@yyyy0201 I see that you and your teammates are from the same institution. However, you exchange private notebooks when you and your partners haven't been merged yet. That’s a violation, as it allows you to easily make massive submissions in a day and share the results which is unfair.", "votes": 5 } ] } ] }, { "id": 3202968, "postDate": "2025-05-16T07:02:05.333Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 3192378, "author_name": "CPMP", "author_url": "", "post_date": "2025-05-02T19:01:32.947000", "content": "

You are in violation. You work as a team, which means you exchange info. Yet you have not teamed on Kaggle.

\n

Private sharing outside kaggle teams is prohibited by the rules.

\n

Issue with the way you do it is that you can submit more in a day than if you were in a single Kaggle team. That's why it is unfair and that's why it is prohibited by Kaggle rules.

", "votes": 8, "replies": [ { "id": 3192550, "author_name": "Cody_Null", "author_url": "", "post_date": "2025-05-03T03:27:26.540000", "content": "

We have been using nearly all our submissions for the last 2 weeks just fixing data because the false labels in the test set really messed stuff up for us. A theoretical 20 subs a day since that point is a pretty large advantage…

", "votes": 4, "replies": [] }, { "id": 3192707, "author_name": "CPMP", "author_url": "", "post_date": "2025-05-03T09:29:35.783000", "content": "

I concur. I am not active here, but in arc prize I am wasting my single sub a day trying to debug my submission code. I would be done in a day with 20 subs.

", "votes": 2, "replies": [] } ] }, { "id": 3188580, "author_name": "tennogh", "author_url": "", "post_date": "2025-04-27T21:44:46.617000", "content": "

Yes, the leaderboard score uses that metric. But it's calculated on a hidden set of tomograms, so the result can be different from your validation score from a number of reasons (overfitting your validation set, different sizes, voxel spacings, etc.).

", "votes": 1, "replies": [] }, { "id": 3201108, "author_name": "Manuel Campos", "author_url": "", "post_date": "2025-05-13T12:28:49.760000", "content": "
\n

Thanks for the heads up. We're not in violation, all 4 of us on the team will be merged subsequently and there are no additional accounts submitted

\n
\n

The fourth member of the team, will he/she finally combine with you or will he/she remain in solo mode?

", "votes": 0, "replies": [ { "id": 3201113, "author_name": "yyyy0201", "author_url": "", "post_date": "2025-05-13T12:34:49.010000", "content": "

merge

", "votes": 1, "replies": [] }, { "id": 3201117, "author_name": "Manuel Campos", "author_url": "", "post_date": "2025-05-13T12:41:47.690000", "content": "

Okay, thanks!

", "votes": 0, "replies": [] }, { "id": 3202495, "author_name": "", "author_url": "", "post_date": "2025-05-15T13:24:58.003000", "content": "", "votes": 2, "replies": [] }, { "id": 3202504, "author_name": "yyyy0201", "author_url": "", "post_date": "2025-05-15T13:43:45.590000", "content": "

Thank you for your attention. Although we are in the same school, I don’t know him and have no idea who he is. Our teammate is someone else, who tried the 3D segmentation approach alone. He didn’t submit many attempts, and none of them achieved good results.

", "votes": -1, "replies": [] }, { "id": 3202511, "author_name": "Manuel Campos", "author_url": "", "post_date": "2025-05-15T13:57:11.310000", "content": "

Right now, he's in the Top 18, It's a shame that all together you exceed the number of submissions.

", "votes": 0, "replies": [] }, { "id": 3202516, "author_name": "", "author_url": "", "post_date": "2025-05-15T14:03:19.417000", "content": "", "votes": 3, "replies": [] }, { "id": 3202518, "author_name": "Tom", "author_url": "", "post_date": "2025-05-15T14:08:47.767000", "content": "

Wow is that another drama?

", "votes": 4, "replies": [] }, { "id": 3202519, "author_name": "yyyy0201", "author_url": "", "post_date": "2025-05-15T14:08:53.667000", "content": "

This is an interesting story. At first, we discovered that we were at the same school, so my teammate privately messaged him to ask if he wanted to team up, but he never replied, so nothing came of it.😂

", "votes": -5, "replies": [] }, { "id": 3202525, "author_name": "Cody_Null", "author_url": "", "post_date": "2025-05-15T14:15:56.110000", "content": "

Respectfully, I do believe what you have done already counts as cheating according to Kaggle's ruleset as you have had a massive advantage by getting more daily subs than the rest of the community… I think this issue would stand regardless of if your 4th teammate merged or not. I would suggest reaching out to Kaggle admins.

", "votes": 5, "replies": [] }, { "id": 3202530, "author_name": "yyyy0201", "author_url": "", "post_date": "2025-05-15T14:22:49.417000", "content": "

Thank you for your suggestion. We will explain the situation. May I ask how to get in touch with the administrator?

", "votes": 1, "replies": [] }, { "id": 3202553, "author_name": "Cody_Null", "author_url": "", "post_date": "2025-05-15T14:47:56.287000", "content": "

Honestly, not too sure! I know you can @ admins as well as discord but I am not sure outside of that. :/

", "votes": 0, "replies": [] }, { "id": 3202557, "author_name": "Tom", "author_url": "", "post_date": "2025-05-15T14:54:02.720000", "content": "

@yyyy0201 You can contact kaggle staffs by clicking contact button from their profile.

", "votes": 0, "replies": [] }, { "id": 3202576, "author_name": "Dieter", "author_url": "", "post_date": "2025-05-15T15:33:47.590000", "content": "

As pointed out you are in violation of rules, and you still are. Honestly, I think there is a good chance you are just removed silently from the competition when private LB is finalizing. So dont complain then.

", "votes": 2, "replies": [] }, { "id": 3202790, "author_name": "Luck is what I need", "author_url": "", "post_date": "2025-05-15T23:16:49.653000", "content": "

No, I'm telling you very clearly that I'm not.

", "votes": 0, "replies": [] }, { "id": 3202817, "author_name": "Tom", "author_url": "", "post_date": "2025-05-16T00:40:05.400000", "content": "

@majiaqi111 Make sure you never use part of their results or exchange privately. I believe you are not cheater cuz I saw you join many comps before.
\nActually I have asked the kaggle staffs, they say they'll investigate @yyyy0201's situation when this competition closes.

", "votes": 1, "replies": [] }, { "id": 3202824, "author_name": "Luck is what I need", "author_url": "", "post_date": "2025-05-16T01:03:26.850000", "content": "

Yes, I can ensure that.

", "votes": 0, "replies": [] }, { "id": 3202832, "author_name": "Cody_Null", "author_url": "", "post_date": "2025-05-16T01:56:11.717000", "content": "

I’m not really sure what there is to investigate. As we know for the last several weeks they have been allowed more than 5 subs per day. Due to the inconsistent lb of this competition that is a massive advantage, that’s clearly against the rules, and they have had this advantage for weeks if not months… if this is permitted now will other kagglers be able to private share before teaming to advance ourselves on the leaderboard?

", "votes": 3, "replies": [] }, { "id": 3202926, "author_name": "Tom", "author_url": "", "post_date": "2025-05-16T05:53:00.137000", "content": "

\"Our teammate is someone else, who tried the 3D segmentation approach alone\". It turns out that everyone can contact one another, ask each other to try different methods, and verify what works, using up someone else’s submission quota in the process then team up at the end. That’s the biggest solution in the competition.

", "votes": 0, "replies": [] }, { "id": 3202950, "author_name": "", "author_url": "", "post_date": "2025-05-16T06:39:30.783000", "content": "", "votes": 0, "replies": [] }, { "id": 3207704, "author_name": "", "author_url": "", "post_date": "2025-05-23T06:41:25.043000", "content": "

OK, thanks! I'll try later.

", "votes": 1, "replies": [] } ] }, { "id": 3189400, "author_name": "FabianIsensee", "author_url": "", "post_date": "2025-04-29T06:21:32.273000", "content": "

Is there anything anyone knows about the test set? We get very stable performance across all five folds of a cross-validation on the training set and we do very minimal to no tuning of hyperparameters, so we would not expect too much overfitting. Yet, the best CV f2 is 0.949 and the best lb score is 0.811. There must be a shift in distribution: either there is much fewer/more motors in the lb than the train set or there is some other domain shift that we were not able to account for.

", "votes": 0, "replies": [ { "id": 3189428, "author_name": "", "author_url": "", "post_date": "2025-04-29T07:29:09.817000", "content": "

Are you also using Yolo based models or 3D unet or something else??

", "votes": 1, "replies": [] }, { "id": 3189434, "author_name": "yyyy0201", "author_url": "", "post_date": "2025-04-29T07:48:10.207000", "content": "

Whether to use additional datasets?
\n
\nOur best single model also scores around LB 84

", "votes": 0, "replies": [ { "id": 3189636, "author_name": "FabianIsensee", "author_url": "", "post_date": "2025-04-29T14:08:43.043000", "content": "

just the official dataset so far

", "votes": 1, "replies": [] }, { "id": 3191805, "author_name": "Dieter", "author_url": "", "post_date": "2025-05-02T06:47:33.490000", "content": "

Who is \"our\"? I see you are alone on leaderboard?

", "votes": 7, "replies": [] }, { "id": 3192201, "author_name": "yyyy0201", "author_url": "", "post_date": "2025-05-02T15:13:27.290000", "content": "

We haven't merged teams yet~

", "votes": -8, "replies": [] }, { "id": 3192284, "author_name": "Dieter", "author_url": "", "post_date": "2025-05-02T17:00:09.727000", "content": "

Which means you are breaking the rules by private sharing. That is reason for disqualification. Especially in a competition where you need to rely on leaderboard you cheat by having effectively more submissions than other teams.

", "votes": 6, "replies": [] }, { "id": 3193170, "author_name": "Tom", "author_url": "", "post_date": "2025-05-04T01:58:54.927000", "content": "

@yyyy0201 I see that you and your teammates are from the same institution. However, you exchange private notebooks when you and your partners haven't been merged yet. That’s a violation, as it allows you to easily make massive submissions in a day and share the results which is unfair.

", "votes": 5, "replies": [] } ] } ] }, { "id": 3202968, "author_name": "", "author_url": "", "post_date": "2025-05-16T07:02:05.333000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3192378": "You are in violation. You work as a team, which means you exchange info. Yet you have not teamed on Kaggle.\n\nPrivate sharing outside kaggle teams is prohibited by the rules. \n\nIssue with the way you do it is that you can submit more in a day than if you were in a single Kaggle team. That's why it is unfair and that's why it is prohibited by Kaggle rules.", "3188337": "Is the final score the same as result calculated from the [metric notebook](https://www.kaggle.com/code/metric/byu-biophysics-91249)?\n\nThe score that I submit is about 0.6, but the score I get locally is over 0.9.\n\nThe way I run locally is as follows. I use the [notebook](https://www.kaggle.com/code/andrewjdarley/parse-data/notebook) to divide the tomograms in to training set and validation set. And I use the tomograms in validation set which only have one motor to calculate the metric. And I also add some tomograms which have no motor, and the proportion is the same as the (tomograms num without motor)/(total tomograms num).", "3188580": "Yes, the leaderboard score uses that metric. But it's calculated on a hidden set of tomograms, so the result can be different from your validation score from a number of reasons (overfitting your validation set, different sizes, voxel spacings, etc.).", "3201108": ">Thanks for the heads up. We're not in violation, all 4 of us on the team will be merged subsequently and there are no additional accounts submitted\n\nThe fourth member of the team, will he/she finally combine with you or will he/she remain in solo mode?", "3189400": "Is there anything anyone knows about the test set? We get very stable performance across all five folds of a cross-validation on the training set and we do very minimal to no tuning of hyperparameters, so we would not expect too much overfitting. Yet, the best CV f2 is 0.949 and the best lb score is 0.811. There must be a shift in distribution: either there is much fewer/more motors in the lb than the train set or there is some other domain shift that we were not able to account for. ", "3202968": "" } }