{ "id": 334654, "title": "Some ground truths are wrong", "url": "/competitions/smartphone-decimeter-2022/discussion/334654", "author_name": "", "post_date": "2022-07-02T13:53:38.417186600Z", "votes": 39, "comment_count": 11, "views": 0, "content": "

I noticed that some of the ground truths are wrong.
\nAlthough I only checked train data after 2021-03-10, at least in the following data, the coordinates of ground truth are offset by about 3 meters to the left and right in the direction of the car's travel.

\n
2021-04-21-US-MTV-2\n2021-04-29-US-MTV-1\n2021-04-29-US-MTV-2\n2021-03-16-US-MTV-2\n
\n

This is a code to verify the ground truth.
\nThe following is a plot of the difference between the baseline position and the ground truth in the horizontal direction.
\n\"gsdc1\"

\n

Normally, the distribution of error should follow a Gaussian distribution, but two peaks can clearly be seen in the right figure. In my experience, this happens when there is a mistake in the coordinate conversion, such as a lever arm between antennas, when the reference coordinates are obtained by GNSS/INS and then converted to the phone position. In this drive, the same road is traveled several times in opposite directions, thus the distribution is divided into two.

\n

The following is a plot of these tracks on a Google Earth satellite photo. The satellite photo locations are fairly accurate, and the data, which ground truth believes to be incorrect, clearly deviates significantly from the left-turn lane.
\n\"gsdc2\"

\n

@mohammedkhider
\nIncorrect ground truth is fatal to the competition; if the ground truth of the test data is incorrect, the public and private scores may change. Could you please check and correct the ground truth in the train and test data as soon as possible?

", "messages": [ { "id": "1840702", "postDate": "07/02/2022 13:53:38", "content": "

I noticed that some of the ground truths are wrong.
\nAlthough I only checked train data after 2021-03-10, at least in the following data, the coordinates of ground truth are offset by about 3 meters to the left and right in the direction of the car's travel.

\n
2021-04-21-US-MTV-2\n2021-04-29-US-MTV-1\n2021-04-29-US-MTV-2\n2021-03-16-US-MTV-2\n
\n

This is a code to verify the ground truth.
\nThe following is a plot of the difference between the baseline position and the ground truth in the horizontal direction.
\n\"gsdc1\"

\n

Normally, the distribution of error should follow a Gaussian distribution, but two peaks can clearly be seen in the right figure. In my experience, this happens when there is a mistake in the coordinate conversion, such as a lever arm between antennas, when the reference coordinates are obtained by GNSS/INS and then converted to the phone position. In this drive, the same road is traveled several times in opposite directions, thus the distribution is divided into two.

\n

The following is a plot of these tracks on a Google Earth satellite photo. The satellite photo locations are fairly accurate, and the data, which ground truth believes to be incorrect, clearly deviates significantly from the left-turn lane.
\n\"gsdc2\"

\n

@mohammedkhider
\nIncorrect ground truth is fatal to the competition; if the ground truth of the test data is incorrect, the public and private scores may change. Could you please check and correct the ground truth in the train and test data as soon as possible?

", "rawMarkdown": "I noticed that some of the ground truths are wrong.\nAlthough I only checked train data after 2021-03-10, at least in the following data, the coordinates of ground truth are offset by about 3 meters to the left and right in the direction of the car's travel. \n\n```\n2021-04-21-US-MTV-2\n2021-04-29-US-MTV-1\n2021-04-29-US-MTV-2\n2021-03-16-US-MTV-2\n```\n\nThis is a [code ](https://www.kaggle.com/code/taroz1461/some-ground-truths-are-wrong) to verify the ground truth. \nThe following is a plot of the difference between the baseline position and the ground truth in the horizontal direction. \n![gsdc1](https://user-images.githubusercontent.com/7933764/177003340-0e4ff56b-8f51-4236-a8ea-adc0f086c44d.jpg)\n\nNormally, the distribution of error should follow a Gaussian distribution, but two peaks can clearly be seen in the right figure. In my experience, this happens when there is a mistake in the coordinate conversion, such as a lever arm between antennas, when the reference coordinates are obtained by GNSS/INS and then converted to the phone position. In this drive, the same road is traveled several times in opposite directions, thus the distribution is divided into two.\n\nThe following is a plot of these tracks on a Google Earth satellite photo. The satellite photo locations are fairly accurate, and the data, which ground truth believes to be incorrect, clearly deviates significantly from the left-turn lane.\n![gsdc2](https://user-images.githubusercontent.com/7933764/177003361-ecb63cee-640f-4c1b-beac-00b9c6086309.jpg)\n\n@mohammedkhider\nIncorrect ground truth is fatal to the competition; if the ground truth of the test data is incorrect, the public and private scores may change. Could you please check and correct the ground truth in the train and test data as soon as possible?", "votes": null }, { "id": "1840757", "postDate": "07/02/2022 14:35:16", "content": "

Good spot. As you say, possibly related to the correction for the \"lever arm\" coordinate difference between phone and GT antenna.

", "rawMarkdown": "Good spot. As you say, possibly related to the correction for the \"lever arm\" coordinate difference between phone and GT antenna.", "votes": null }, { "id": "1843120", "postDate": "07/04/2022 15:01:16", "content": "

I have updated the code and checked all the train data.
\nSo far, in my opinion, here is the drive where the grand truth is wrong.

\n

Definitely wrong

\n
2020-08-06-US-MTV-1\n2020-08-06-US-MTV-2\n2020-11-23-US-MTV-1\n2021-03-16-US-MTV-2\n2021-04-21-US-MTV-2\n2021-04-29-US-MTV-1\n2021-04-29-US-MTV-2\n2021-12-15-US-MTV-1\n2021-12-28-US-MTV-1\n
\n

Doubtful

\n
2021-04-26-US-SVL-2\n2021-12-08-US-LAX-3\n
\n

If the train data contains such strange ground truth, it is natural to assume that the test data also contains it…

", "rawMarkdown": "I have updated the [code](https://www.kaggle.com/code/taroz1461/some-ground-truths-are-wrong) and checked all the train data. \nSo far, in my opinion, here is the drive where the grand truth is wrong.\n\n**Definitely wrong**\n```\n2020-08-06-US-MTV-1\n2020-08-06-US-MTV-2\n2020-11-23-US-MTV-1\n2021-03-16-US-MTV-2\n2021-04-21-US-MTV-2\n2021-04-29-US-MTV-1\n2021-04-29-US-MTV-2\n2021-12-15-US-MTV-1\n2021-12-28-US-MTV-1\n```\n\n**Doubtful**\n```\n2021-04-26-US-SVL-2\n2021-12-08-US-LAX-3\n```\n\nIf the train data contains such strange ground truth, it is natural to assume that the test data also contains it...", "votes": null }, { "id": "1843462", "postDate": "07/04/2022 22:04:46", "content": "

Fu, Khider & van Diggelen (2020) says (page 1933):

\n

\"In practice, we have compensated these lever arms. However, several centimeters of errors are inevitable because the lever arms are coarsely measured with regular rulers.\"

", "rawMarkdown": "Fu, Khider & van Diggelen (2020) says (page 1933):\n\n\"In practice, we have compensated these lever arms. However, several centimeters of errors are inevitable because the lever arms are coarsely measured with regular rulers.\"", "votes": null }, { "id": "1843492", "postDate": "07/04/2022 23:36:01", "content": "

Sure, an error of a few centimeters is acceptable, but I think it is problematic that there are several meters of error in some drives even though it is a decimeter challenge.

", "rawMarkdown": "Sure, an error of a few centimeters is acceptable, but I think it is problematic that there are several meters of error in some drives even though it is a decimeter challenge.", "votes": null }, { "id": "1844386", "postDate": "07/05/2022 14:12:16", "content": "

I absolutely agree that it is a concern. Your data suggest an error of something like 3.5 to 5.0 metres for that journey. There are also some weird glitches in the data like that discussed here. Nonetheless, the data seem to be good enough for the top teams to be down to ~1.5 to 2.0 metres on the competition metric. Changing the data now would be very disruptive to the competition (as it was with SETI last year).

", "rawMarkdown": "I absolutely agree that it is a concern. Your data suggest an error of something like 3.5 to 5.0 metres for that journey. There are also some weird glitches in the data like that discussed [here](https://www.kaggle.com/competitions/smartphone-decimeter-2022/discussion/326120). Nonetheless, the data seem to be good enough for the top teams to be down to ~1.5 to 2.0 metres on the competition metric. Changing the data now would be very disruptive to the competition (as it was with SETI last year).", "votes": null }, { "id": "1844434", "postDate": "07/05/2022 14:52:26", "content": "

I think it would be a real shame if the score goes down even though the phone’s position was estimated correctly.

\n

I am sure that there is not much time left until the end of the competition. However, in my opinion, the host of the competition should (1) at least confirm that the ground truth in the test data is correct and replace it if it is wrong (the current LB ranking may change a little), and (2) replace the wrong ground truths in the train data if possible.

", "rawMarkdown": "I think it would be a real shame if the score goes down even though the phone’s position was estimated correctly. \n\nI am sure that there is not much time left until the end of the competition. However, in my opinion, the host of the competition should (1) at least confirm that the ground truth in the test data is correct and replace it if it is wrong (the current LB ranking may change a little), and (2) replace the wrong ground truths in the train data if possible.", "votes": null }, { "id": "1848616", "postDate": "07/08/2022 18:56:32", "content": "

Thanks for examining the ground truth, Taro. Catching errors in the ground truth without a better \"ground truth\" is a challenge.

\n

For the training traces you pointed out, we're examining them internally one by one. Fortunately, they are only a small portion of all the training datasets. Since the time is too tight now, we're not going to remove or correct the ground truth of these traces. However, we will give an update here next week about the affected training traces.

\n

For all the 36 test traces, the WLS errors all appear as one cluster (peak) with reasonable offset to the origin. In addition, we tested these traces with an advanced positioning engine and the positioning metrics are reasonably within the normal bound. Accordingly, we are confident about the correctness of the ground truth in the public and private datasets (and leaderboards).

", "rawMarkdown": "Thanks for examining the ground truth, Taro. Catching errors in the ground truth without a better \"ground truth\" is a challenge. \n\nFor the training traces you pointed out, we're examining them internally one by one. Fortunately, they are only a small portion of all the training datasets. Since the time is too tight now, we're not going to remove or correct the ground truth of these traces. However, we will give an update here next week about the affected training traces.\n \nFor all the 36 test traces, the WLS errors all appear as one cluster (peak) with reasonable offset to the origin. In addition, we tested these traces with an advanced positioning engine and the positioning metrics are reasonably within the normal bound. Accordingly, we are confident about the correctness of the ground truth in the public and private datasets (and leaderboards).", "votes": null }, { "id": "1850146", "postDate": "07/10/2022 06:32:03", "content": "

The good news is that the ground truth of the test data is not affected by the error. Then the fairness of LB will be ensured. Thank you for the confirmation.

", "rawMarkdown": "The good news is that the ground truth of the test data is not affected by the error. Then the fairness of LB will be ensured. Thank you for the confirmation.", "votes": null }, { "id": "1860765", "postDate": "07/18/2022 14:42:48", "content": "

We made this post to discuss the training traces discussed here.

", "rawMarkdown": "We made [this post](https://www.kaggle.com/competitions/smartphone-decimeter-2022/discussion/337416) to discuss the training traces discussed here.", "votes": null }, { "id": "1863083", "postDate": "07/20/2022 06:32:56", "content": "

Hi @taroz1461 , Are you certain about your definetly wrong list as of today?

", "rawMarkdown": "Hi @taroz1461 , Are you certain about your definetly wrong list as of today?", "votes": null }, { "id": "1865027", "postDate": "07/21/2022 13:05:19", "content": "

@rodrigobrugue
\nUnchanged from above.
\nExcept for the above training data, the rest of the data seems to be reliable.

", "rawMarkdown": "rodrigobrugue \nUnchanged from above.\nExcept for the above training data, the rest of the data seems to be reliable.", "votes": null } ], "comments": [ { "id": 1840757, "author_name": "jbomitchell", "author_url": "", "post_date": "07/02/2022 14:35:16", "content": "

Good spot. As you say, possibly related to the correction for the \"lever arm\" coordinate difference between phone and GT antenna.

", "votes": null, "replies": [] }, { "id": 1843120, "author_name": "taroz1461", "author_url": "", "post_date": "07/04/2022 15:01:16", "content": "

I have updated the code and checked all the train data.
\nSo far, in my opinion, here is the drive where the grand truth is wrong.

\n

Definitely wrong

\n
2020-08-06-US-MTV-1\n2020-08-06-US-MTV-2\n2020-11-23-US-MTV-1\n2021-03-16-US-MTV-2\n2021-04-21-US-MTV-2\n2021-04-29-US-MTV-1\n2021-04-29-US-MTV-2\n2021-12-15-US-MTV-1\n2021-12-28-US-MTV-1\n
\n

Doubtful

\n
2021-04-26-US-SVL-2\n2021-12-08-US-LAX-3\n
\n

If the train data contains such strange ground truth, it is natural to assume that the test data also contains it…

", "votes": null, "replies": [ { "id": 1863083, "author_name": "rodrigobrugue", "author_url": "", "post_date": "07/20/2022 06:32:56", "content": "

Hi @taroz1461 , Are you certain about your definetly wrong list as of today?

", "votes": null, "replies": [] }, { "id": 1865027, "author_name": "taroz1461", "author_url": "", "post_date": "07/21/2022 13:05:19", "content": "

@rodrigobrugue
\nUnchanged from above.
\nExcept for the above training data, the rest of the data seems to be reliable.

", "votes": null, "replies": [] } ] }, { "id": 1843462, "author_name": "jbomitchell", "author_url": "", "post_date": "07/04/2022 22:04:46", "content": "

Fu, Khider & van Diggelen (2020) says (page 1933):

\n

\"In practice, we have compensated these lever arms. However, several centimeters of errors are inevitable because the lever arms are coarsely measured with regular rulers.\"

", "votes": null, "replies": [ { "id": 1843492, "author_name": "taroz1461", "author_url": "", "post_date": "07/04/2022 23:36:01", "content": "

Sure, an error of a few centimeters is acceptable, but I think it is problematic that there are several meters of error in some drives even though it is a decimeter challenge.

", "votes": null, "replies": [] }, { "id": 1844386, "author_name": "jbomitchell", "author_url": "", "post_date": "07/05/2022 14:12:16", "content": "

I absolutely agree that it is a concern. Your data suggest an error of something like 3.5 to 5.0 metres for that journey. There are also some weird glitches in the data like that discussed here. Nonetheless, the data seem to be good enough for the top teams to be down to ~1.5 to 2.0 metres on the competition metric. Changing the data now would be very disruptive to the competition (as it was with SETI last year).

", "votes": null, "replies": [] }, { "id": 1844434, "author_name": "taroz1461", "author_url": "", "post_date": "07/05/2022 14:52:26", "content": "

I think it would be a real shame if the score goes down even though the phone’s position was estimated correctly.

\n

I am sure that there is not much time left until the end of the competition. However, in my opinion, the host of the competition should (1) at least confirm that the ground truth in the test data is correct and replace it if it is wrong (the current LB ranking may change a little), and (2) replace the wrong ground truths in the train data if possible.

", "votes": null, "replies": [] } ] }, { "id": 1848616, "author_name": "gymf123", "author_url": "", "post_date": "07/08/2022 18:56:32", "content": "

Thanks for examining the ground truth, Taro. Catching errors in the ground truth without a better \"ground truth\" is a challenge.

\n

For the training traces you pointed out, we're examining them internally one by one. Fortunately, they are only a small portion of all the training datasets. Since the time is too tight now, we're not going to remove or correct the ground truth of these traces. However, we will give an update here next week about the affected training traces.

\n

For all the 36 test traces, the WLS errors all appear as one cluster (peak) with reasonable offset to the origin. In addition, we tested these traces with an advanced positioning engine and the positioning metrics are reasonably within the normal bound. Accordingly, we are confident about the correctness of the ground truth in the public and private datasets (and leaderboards).

", "votes": null, "replies": [ { "id": 1850146, "author_name": "taroz1461", "author_url": "", "post_date": "07/10/2022 06:32:03", "content": "

The good news is that the ground truth of the test data is not affected by the error. Then the fairness of LB will be ensured. Thank you for the confirmation.

", "votes": null, "replies": [] }, { "id": 1860765, "author_name": "gymf123", "author_url": "", "post_date": "07/18/2022 14:42:48", "content": "

We made this post to discuss the training traces discussed here.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1840702": "I noticed that some of the ground truths are wrong.\nAlthough I only checked train data after 2021-03-10, at least in the following data, the coordinates of ground truth are offset by about 3 meters to the left and right in the direction of the car's travel. \n\n```\n2021-04-21-US-MTV-2\n2021-04-29-US-MTV-1\n2021-04-29-US-MTV-2\n2021-03-16-US-MTV-2\n```\n\nThis is a [code ](https://www.kaggle.com/code/taroz1461/some-ground-truths-are-wrong) to verify the ground truth. \nThe following is a plot of the difference between the baseline position and the ground truth in the horizontal direction. \n![gsdc1](https://user-images.githubusercontent.com/7933764/177003340-0e4ff56b-8f51-4236-a8ea-adc0f086c44d.jpg)\n\nNormally, the distribution of error should follow a Gaussian distribution, but two peaks can clearly be seen in the right figure. In my experience, this happens when there is a mistake in the coordinate conversion, such as a lever arm between antennas, when the reference coordinates are obtained by GNSS/INS and then converted to the phone position. In this drive, the same road is traveled several times in opposite directions, thus the distribution is divided into two.\n\nThe following is a plot of these tracks on a Google Earth satellite photo. The satellite photo locations are fairly accurate, and the data, which ground truth believes to be incorrect, clearly deviates significantly from the left-turn lane.\n![gsdc2](https://user-images.githubusercontent.com/7933764/177003361-ecb63cee-640f-4c1b-beac-00b9c6086309.jpg)\n\n@mohammedkhider\nIncorrect ground truth is fatal to the competition; if the ground truth of the test data is incorrect, the public and private scores may change. Could you please check and correct the ground truth in the train and test data as soon as possible?", "1840757": "Good spot. As you say, possibly related to the correction for the \"lever arm\" coordinate difference between phone and GT antenna.", "1843120": "I have updated the [code](https://www.kaggle.com/code/taroz1461/some-ground-truths-are-wrong) and checked all the train data. \nSo far, in my opinion, here is the drive where the grand truth is wrong.\n\n**Definitely wrong**\n```\n2020-08-06-US-MTV-1\n2020-08-06-US-MTV-2\n2020-11-23-US-MTV-1\n2021-03-16-US-MTV-2\n2021-04-21-US-MTV-2\n2021-04-29-US-MTV-1\n2021-04-29-US-MTV-2\n2021-12-15-US-MTV-1\n2021-12-28-US-MTV-1\n```\n\n**Doubtful**\n```\n2021-04-26-US-SVL-2\n2021-12-08-US-LAX-3\n```\n\nIf the train data contains such strange ground truth, it is natural to assume that the test data also contains it...", "1843462": "Fu, Khider & van Diggelen (2020) says (page 1933):\n\n\"In practice, we have compensated these lever arms. However, several centimeters of errors are inevitable because the lever arms are coarsely measured with regular rulers.\"", "1843492": "Sure, an error of a few centimeters is acceptable, but I think it is problematic that there are several meters of error in some drives even though it is a decimeter challenge.", "1844386": "I absolutely agree that it is a concern. Your data suggest an error of something like 3.5 to 5.0 metres for that journey. There are also some weird glitches in the data like that discussed [here](https://www.kaggle.com/competitions/smartphone-decimeter-2022/discussion/326120). Nonetheless, the data seem to be good enough for the top teams to be down to ~1.5 to 2.0 metres on the competition metric. Changing the data now would be very disruptive to the competition (as it was with SETI last year).", "1844434": "I think it would be a real shame if the score goes down even though the phone’s position was estimated correctly. \n\nI am sure that there is not much time left until the end of the competition. However, in my opinion, the host of the competition should (1) at least confirm that the ground truth in the test data is correct and replace it if it is wrong (the current LB ranking may change a little), and (2) replace the wrong ground truths in the train data if possible.", "1848616": "Thanks for examining the ground truth, Taro. Catching errors in the ground truth without a better \"ground truth\" is a challenge. \n\nFor the training traces you pointed out, we're examining them internally one by one. Fortunately, they are only a small portion of all the training datasets. Since the time is too tight now, we're not going to remove or correct the ground truth of these traces. However, we will give an update here next week about the affected training traces.\n \nFor all the 36 test traces, the WLS errors all appear as one cluster (peak) with reasonable offset to the origin. In addition, we tested these traces with an advanced positioning engine and the positioning metrics are reasonably within the normal bound. Accordingly, we are confident about the correctness of the ground truth in the public and private datasets (and leaderboards).", "1850146": "The good news is that the ground truth of the test data is not affected by the error. Then the fairness of LB will be ensured. Thank you for the confirmation.", "1860765": "We made [this post](https://www.kaggle.com/competitions/smartphone-decimeter-2022/discussion/337416) to discuss the training traces discussed here.", "1863083": "Hi @taroz1461 , Are you certain about your definetly wrong list as of today?", "1865027": "rodrigobrugue \nUnchanged from above.\nExcept for the above training data, the rest of the data seems to be reliable." }, "source": "meta" }