Should motor coordinates be integers? I made one submission, and it scored pretty low, so I was wondering if my submitting floats and -1.0 for absent motors are the culprits. The kaggle score code works fine with such a format.
\nThanks!
Should motor coordinates be integers? I made one submission, and it scored pretty low, so I was wondering if my submitting floats and -1.0 for absent motors are the culprits. The kaggle score code works fine with such a format.
\nThanks!
They should be floats because if you use integers then you increase the risk of moving yourself outside of the threshold distance. If you are scoring significantly lower on the leaderboard than you are on your cross validation score then I recommend checking these two things:
\nIf the competition metric provided by the host works with floats, I suppose that’s what is used for scoring as well and it should be fine?
\nIn the host provided notebook, they are using round to return integers, which wouldn’t be needed if it worked with floats?
\nI haven’t attempted using floats to be honest and I am unable to test right now, but I’m very curious nonetheless!
", "rawMarkdown": "If the competition metric provided by the host works with floats, I suppose that’s what is used for scoring as well and it should be fine?\n\nIn the host provided notebook, they are using round to return integers, which wouldn’t be needed if it worked with floats?\n\nI haven’t attempted using floats to be honest and I am unable to test right now, but I’m very curious nonetheless!", "votes": null }, { "id": "3156104", "postDate": "03/21/2025 18:24:22", "content": "You should be good to use floats for your submissions.
\nThe rounding in my notebook is an artifact of our preparation for the competition before we finalized the evaluation code. It isn't necessary though.
", "rawMarkdown": "You should be good to use floats for your submissions. \n\nThe rounding in my notebook is an artifact of our preparation for the competition before we finalized the evaluation code. It isn't necessary though.", "votes": null }, { "id": "3156855", "postDate": "03/22/2025 16:46:16", "content": "I have the same problem (but I don't use floats in prediction): competition metric on validation set (I double checked that there is no leak) is about 0.7, but LB score is 0.08 🙈. I checked my submission notebook with predicting validation images from train set instead of 3 test examples and got the same 0.7 metric. The predicted coordinates for tests examples look OK, almost the same like for public notebooks with 0.6 score. Probably there is domain shift in test images, I don't see any other reason
", "rawMarkdown": "I have the same problem (but I don't use floats in prediction): competition metric on validation set (I double checked that there is no leak) is about 0.7, but LB score is 0.08 🙈. I checked my submission notebook with predicting validation images from train set instead of 3 test examples and got the same 0.7 metric. The predicted coordinates for tests examples look OK, almost the same like for public notebooks with 0.6 score. Probably there is domain shift in test images, I don't see any other reason", "votes": null }, { "id": "3158090", "postDate": "03/24/2025 08:25:59", "content": "just say, even i set all result -1, the score compute more than 10 hours, don't know if this is a bug in score script.
", "rawMarkdown": "just say, even i set all result -1, the score compute more than 10 hours, don't know if this is a bug in score script.", "votes": null } ], "comments": [ { "id": 3156043, "author_name": "connorjd", "author_url": "", "post_date": "03/21/2025 16:37:11", "content": "They should be floats because if you use integers then you increase the risk of moving yourself outside of the threshold distance. If you are scoring significantly lower on the leaderboard than you are on your cross validation score then I recommend checking these two things:
\nIf the competition metric provided by the host works with floats, I suppose that’s what is used for scoring as well and it should be fine?
\nIn the host provided notebook, they are using round to return integers, which wouldn’t be needed if it worked with floats?
\nI haven’t attempted using floats to be honest and I am unable to test right now, but I’m very curious nonetheless!
", "votes": null, "replies": [] }, { "id": 3156104, "author_name": "andrewjdarley", "author_url": "", "post_date": "03/21/2025 18:24:22", "content": "You should be good to use floats for your submissions.
\nThe rounding in my notebook is an artifact of our preparation for the competition before we finalized the evaluation code. It isn't necessary though.
", "votes": null, "replies": [] }, { "id": 3156855, "author_name": "ivashnyov", "author_url": "", "post_date": "03/22/2025 16:46:16", "content": "I have the same problem (but I don't use floats in prediction): competition metric on validation set (I double checked that there is no leak) is about 0.7, but LB score is 0.08 🙈. I checked my submission notebook with predicting validation images from train set instead of 3 test examples and got the same 0.7 metric. The predicted coordinates for tests examples look OK, almost the same like for public notebooks with 0.6 score. Probably there is domain shift in test images, I don't see any other reason
", "votes": null, "replies": [] }, { "id": 3158090, "author_name": "lilaiyuan", "author_url": "", "post_date": "03/24/2025 08:25:59", "content": "just say, even i set all result -1, the score compute more than 10 hours, don't know if this is a bug in score script.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3155895": "Should motor coordinates be integers? I made one submission, and it scored pretty low, so I was wondering if my submitting floats and -1.0 for absent motors are the culprits. The kaggle score code works fine with such a format.\nThanks!", "3156043": "They should be floats because if you use integers then you increase the risk of moving yourself outside of the threshold distance. If you are scoring significantly lower on the leaderboard than you are on your cross validation score then I recommend checking these two things:\n1. You don't have a data leak between your train and validation data that is causing it to overfit\n2. You are not mixing up your X, Y, Z coordinates when you write the axis to the csv. \n - For this I recommend you run your data against the ground truth train data and check that you are not mixing up axis", "3156046": "If the competition metric provided by the host works with floats, I suppose that’s what is used for scoring as well and it should be fine?\n\nIn the host provided notebook, they are using round to return integers, which wouldn’t be needed if it worked with floats?\n\nI haven’t attempted using floats to be honest and I am unable to test right now, but I’m very curious nonetheless!", "3156104": "You should be good to use floats for your submissions. \n\nThe rounding in my notebook is an artifact of our preparation for the competition before we finalized the evaluation code. It isn't necessary though.", "3156855": "I have the same problem (but I don't use floats in prediction): competition metric on validation set (I double checked that there is no leak) is about 0.7, but LB score is 0.08 🙈. I checked my submission notebook with predicting validation images from train set instead of 3 test examples and got the same 0.7 metric. The predicted coordinates for tests examples look OK, almost the same like for public notebooks with 0.6 score. Probably there is domain shift in test images, I don't see any other reason", "3158090": "just say, even i set all result -1, the score compute more than 10 hours, don't know if this is a bug in score script." }, "source": "meta" }