![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1829450%2F3e57ddeb2288dd2af9a7664355416291%2Fdefogsample.png?generation=1682244053699840&alt=media\")
I see a common pattern in the defog dataset that events are present in a series, but both Task and Valid flag of those parts are False. That means these events won't take any effect at all during scoring. It is understood that where Task and Valid are both False, those parts are not annotated.
\nSo if they're not annotated, why are FOG events being labeled there? And what is the meaning of these labels?
\nAnother interesting pattern is that ALL StartHesitation events in defog dataset have Task and Valid as False (that means there is no StartHesitation in defog and should we just predict all 0?)
\n
EDIT:
\nIf you look at the green clusters on the plot, you'll see they should align (overlap) with the 2 clusters where Valid=Task=True just before them, instead of not-overlap. So is there any data mistake with this defog dataset?
Code to generate above plot:
\nx = pd.read_csv('data/train/defog/e069a57511.csv')
\nplt.plot(x.AccV, label='AccV')
\nplt.plot(x.StartHesitation, label='StartHe')
\nplt.plot(x.Turn2, label='Turn')\nplt.plot(x.Walking3, label='Walking')
\nplt.plot(x.Task-3, label='Task')\nplt.plot(x.Task-1.5, label='Valid')
\nplt.title('defog sample e069a57511')
\nplt.legend();
I see a common pattern in the defog dataset that events are present in a series, but both Task and Valid flag of those parts are False. That means these events won't take any effect at all during scoring. It is understood that where Task and Valid are both False, those parts are not annotated.
\nSo if they're not annotated, why are FOG events being labeled there? And what is the meaning of these labels?
\nAnother interesting pattern is that ALL StartHesitation events in defog dataset have Task and Valid as False (that means there is no StartHesitation in defog and should we just predict all 0?)
\n
EDIT:
\nIf you look at the green clusters on the plot, you'll see they should align (overlap) with the 2 clusters where Valid=Task=True just before them, instead of not-overlap. So is there any data mistake with this defog dataset?
Code to generate above plot:
\nx = pd.read_csv('data/train/defog/e069a57511.csv')
\nplt.plot(x.AccV, label='AccV')
\nplt.plot(x.StartHesitation, label='StartHe')
\nplt.plot(x.Turn2, label='Turn')\nplt.plot(x.Walking3, label='Walking')
\nplt.plot(x.Task-3, label='Task')\nplt.plot(x.Task-1.5, label='Valid')
\nplt.title('defog sample e069a57511')
\nplt.legend();
Hi @khahuras,
\nThank you for the heads up. We are looking into it now.
", "rawMarkdown": "Hi @khahuras,\n\nThank you for the heads up. We are looking into it now.", "votes": 12, "replies": [ { "id": 2235653, "postDate": "2023-04-26T08:25:45.913Z", "content": "I think this means that there is a time lag between the sensor data and the correct label.
\nIt is likely that the start time of the sensor and the start time of the recorded data that the annotator uses for labeling are different.
\nOther defog data also has a time lag.
", "rawMarkdown": "I think this means that there is a time lag between the sensor data and the correct label.\n\nIt is likely that the start time of the sensor and the start time of the recorded data that the annotator uses for labeling are different.\n\nOther defog data also has a time lag.", "votes": 4 } ] }, { "id": 2244400, "postDate": "2023-05-03T15:48:41.090Z", "content": "Data Update and Rescore by Kaggle Team
\nhttps://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/406700
Thanks for creating this topic @khahuras !
", "rawMarkdown": "Thanks for creating this topic @khahuras !", "votes": 2 } ] }, { "id": 2239041, "postDate": "2023-04-29T05:43:03.477Z", "content": "I created a notebook to correct for the misalignment.
\nWhen I trained the model with that corrected data, loss decreased, but LB worsened.
\nAssuming a correlation between LB and CV, it may be possible to assume that the test data is also misaligned.
Hi @hiroakifukuse, I think there is a way to try your method wisely. Let's train on your corrected data. Then predict on test data, and shift the predictions to misalign deliberately. Maybe you will have a very good score on LB!
", "rawMarkdown": "Hi @hiroakifukuse, I think there is a way to try your method wisely. Let's train on your corrected data. Then predict on test data, and shift the predictions to misalign deliberately. Maybe you will have a very good score on LB!", "votes": 3 }, { "id": 2244099, "postDate": "2023-05-03T12:48:53.363Z", "content": "Considering the test set is just a superset of what we have, it is a likely prediction that it is misaligned as well (imagining that they had an unaccounted offset between video annotation and the recordings).
", "rawMarkdown": "Considering the test set is just a superset of what we have, it is a likely prediction that it is misaligned as well (imagining that they had an unaccounted offset between video annotation and the recordings)." } ] }, { "id": 2238583, "postDate": "2023-04-28T16:40:18.033Z", "content": "@ryanholbrook Can we get an update on this as soon as possible, or at least some information on whether this is confirmed or not? I know that validating something like that and communicating with the host might take a bit of time but I (and likely many others) find it hard to keep working on this competition not knowing whether the data is corrupt or not.
", "rawMarkdown": "@ryanholbrook Can we get an update on this as soon as possible, or at least some information on whether this is confirmed or not? I know that validating something like that and communicating with the host might take a bit of time but I (and likely many others) find it hard to keep working on this competition not knowing whether the data is corrupt or not.", "votes": 8, "replies": [ { "id": 2238613, "postDate": "2023-04-28T17:07:05.723Z", "content": "Hi @fritzcremer, I believe the problem has been diagnosed. We should have a fix up sometime next week.
", "rawMarkdown": "Hi @fritzcremer, I believe the problem has been diagnosed. We should have a fix up sometime next week. ", "votes": 17 } ] }, { "id": 2234420, "postDate": "2023-04-25T07:30:47.020Z", "content": "I changed this topic's name to get more attention, as it is affecting all modelling steps, as well as scoring (average precision will be affected if this problem isn't answered or resolved)
\n", "rawMarkdown": "I changed this topic's name to get more attention, as it is affecting all modelling steps, as well as scoring (average precision will be affected if this problem isn't answered or resolved)\n\n@jeffhausdorff @ryanholbrook ", "votes": 3 }, { "id": 2234019, "postDate": "2023-04-24T19:23:03.783Z", "content": "Is this alignment problem still there if one uses events.csv to get the labels?
", "rawMarkdown": "Is this alignment problem still there if one uses events.csv to get the labels?", "votes": 3 }, { "id": 2232704, "postDate": "2023-04-24T14:48:53.893Z", "content": "@jeffhausdorff @ryanholbrook
\nIf you look at the green clusters on the plot, you'll see they should align (overlap) with the 2 clusters where Valid=Task=True just before them, instead of not-overlap. So is there any data mistake with this defog dataset?
your assertion that: \"ALL StartHesitation events in defog dataset have Task and Valid as False\" is false.
\nTime series 0d7ab3a9f9 in the defog train set do have a StartHesitation event with 88 timestamps where Valid an Task are both True (from Time 18150 -> 18237)
", "rawMarkdown": "your assertion that: \"*ALL StartHesitation events in defog dataset have Task and Valid as False*\" is **false**. \n\nTime series 0d7ab3a9f9 in the defog train set do have a StartHesitation event with 88 timestamps where Valid an Task are both True (from Time 18150 -> 18237)", "votes": 1 }, { "id": 2238691, "postDate": "2023-04-28T18:30:45.380Z", "content": "I do not see any mistake, your start hesitation color is blurring the pattern but still both Task and Valid are aligned with Turn green color.
", "rawMarkdown": "I do not see any mistake, your start hesitation color is blurring the pattern but still both Task and Valid are aligned with Turn green color.", "votes": 1 }, { "id": 2242078, "postDate": "2023-05-02T03:11:42.237Z", "content": "Thank you for sharing! I also have the same question. After I set the label labeled as unavailable data to 0, LB significantly decreased.
\nOn the contrary, the code that ignores the Valid and Task tags has a higher LB.
", "rawMarkdown": "Thank you for sharing! I also have the same question. After I set the label labeled as unavailable data to 0, LB significantly decreased.\n\nOn the contrary, the code that ignores the Valid and Task tags has a higher LB.", "votes": 2 }, { "id": 2238519, "postDate": "2023-04-28T15:33:44.370Z", "content": "Just started looking at this competition and noticed the same thing. Eg., looked at series 6041cad8ec and events vs annotations line up much better if I shift the events by approximately 20000 samples which looks similar to your plot:
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F94532%2Fa9aef1a60229c38ea13e59296fa3ed29%2FUnshifted.png?generation=1682695860449698&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F94532%2F6d92efa11f168cd28d07777aecea0503%2FShifted.png?generation=1682695882283507&alt=media\")
(Event is any event, and Annotated is Valid or Task.)
Interesting! Do you think that's the same for all samples or does it just happen to be 20k for this example?
\nHope this issue gets resolved quickly.
is there a typo?
\nplt.plot(x.Task-1.5, label='Valid') x.Task --> x.Valid
Thanks, for pointing that issue btw
", "rawMarkdown": "is there a typo? \n`plt.plot(x.Task-1.5, label='Valid')` x.Task --> x.Valid\n\nThanks, for pointing that issue btw", "votes": 2 }, { "id": 2294044, "postDate": "2023-06-09T17:18:44.913Z", "content": "Thanks for pointing out this alignment problem @khahuras!💪
", "rawMarkdown": "Thanks for pointing out this alignment problem @khahuras!💪" }, { "id": 2254485, "postDate": "2023-05-11T03:33:21.783Z", "content": "thx for sharing😁
", "rawMarkdown": "thx for sharing😁" }, { "id": 2237714, "postDate": "2023-04-27T22:23:46.600Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 2234757, "author_name": "Ryan Holbrook", "author_url": "", "post_date": "2023-04-25T13:22:07.187000", "content": "Hi @khahuras,
\nThank you for the heads up. We are looking into it now.
", "votes": 12, "replies": [ { "id": 2235653, "author_name": "hisashi_h", "author_url": "", "post_date": "2023-04-26T08:25:45.913000", "content": "I think this means that there is a time lag between the sensor data and the correct label.
\nIt is likely that the start time of the sensor and the start time of the recorded data that the annotator uses for labeling are different.
\nOther defog data also has a time lag.
", "votes": 4, "replies": [] } ] }, { "id": 2244400, "author_name": "Kha Vo", "author_url": "", "post_date": "2023-05-03T15:48:41.090000", "content": "Data Update and Rescore by Kaggle Team
\nhttps://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/406700
Thanks for creating this topic @khahuras !
", "votes": 2, "replies": [] } ] }, { "id": 2239041, "author_name": "Hi F", "author_url": "", "post_date": "2023-04-29T05:43:03.477000", "content": "I created a notebook to correct for the misalignment.
\nWhen I trained the model with that corrected data, loss decreased, but LB worsened.
\nAssuming a correlation between LB and CV, it may be possible to assume that the test data is also misaligned.
Hi @hiroakifukuse, I think there is a way to try your method wisely. Let's train on your corrected data. Then predict on test data, and shift the predictions to misalign deliberately. Maybe you will have a very good score on LB!
", "votes": 3, "replies": [] }, { "id": 2244099, "author_name": "Kate", "author_url": "", "post_date": "2023-05-03T12:48:53.363000", "content": "Considering the test set is just a superset of what we have, it is a likely prediction that it is misaligned as well (imagining that they had an unaccounted offset between video annotation and the recordings).
", "votes": 0, "replies": [] } ] }, { "id": 2238583, "author_name": "Fritz Cremer", "author_url": "", "post_date": "2023-04-28T16:40:18.033000", "content": "@ryanholbrook Can we get an update on this as soon as possible, or at least some information on whether this is confirmed or not? I know that validating something like that and communicating with the host might take a bit of time but I (and likely many others) find it hard to keep working on this competition not knowing whether the data is corrupt or not.
", "votes": 8, "replies": [ { "id": 2238613, "author_name": "Ryan Holbrook", "author_url": "", "post_date": "2023-04-28T17:07:05.723000", "content": "Hi @fritzcremer, I believe the problem has been diagnosed. We should have a fix up sometime next week.
", "votes": 17, "replies": [] } ] }, { "id": 2234420, "author_name": "Kha Vo", "author_url": "", "post_date": "2023-04-25T07:30:47.020000", "content": "I changed this topic's name to get more attention, as it is affecting all modelling steps, as well as scoring (average precision will be affected if this problem isn't answered or resolved)
\n", "votes": 3, "replies": [] }, { "id": 2234019, "author_name": "Ahmet Erdem", "author_url": "", "post_date": "2023-04-24T19:23:03.783000", "content": "Is this alignment problem still there if one uses events.csv to get the labels?
", "votes": 3, "replies": [] }, { "id": 2232704, "author_name": "Kha Vo", "author_url": "", "post_date": "2023-04-24T14:48:53.893000", "content": "@jeffhausdorff @ryanholbrook
\nIf you look at the green clusters on the plot, you'll see they should align (overlap) with the 2 clusters where Valid=Task=True just before them, instead of not-overlap. So is there any data mistake with this defog dataset?
your assertion that: \"ALL StartHesitation events in defog dataset have Task and Valid as False\" is false.
\nTime series 0d7ab3a9f9 in the defog train set do have a StartHesitation event with 88 timestamps where Valid an Task are both True (from Time 18150 -> 18237)
", "votes": 1, "replies": [] }, { "id": 2238691, "author_name": "Omar Nurhusien", "author_url": "", "post_date": "2023-04-28T18:30:45.380000", "content": "I do not see any mistake, your start hesitation color is blurring the pattern but still both Task and Valid are aligned with Turn green color.
", "votes": 1, "replies": [] }, { "id": 2242078, "author_name": "Lingduo Wang", "author_url": "", "post_date": "2023-05-02T03:11:42.237000", "content": "Thank you for sharing! I also have the same question. After I set the label labeled as unavailable data to 0, LB significantly decreased.
\nOn the contrary, the code that ignores the Valid and Task tags has a higher LB.
", "votes": 2, "replies": [] }, { "id": 2238519, "author_name": "Christoffer Karlsson", "author_url": "", "post_date": "2023-04-28T15:33:44.370000", "content": "Just started looking at this competition and noticed the same thing. Eg., looked at series 6041cad8ec and events vs annotations line up much better if I shift the events by approximately 20000 samples which looks similar to your plot:
(Event is any event, and Annotated is Valid or Task.)
Interesting! Do you think that's the same for all samples or does it just happen to be 20k for this example?
\nHope this issue gets resolved quickly.
is there a typo?
\nplt.plot(x.Task-1.5, label='Valid') x.Task --> x.Valid
Thanks, for pointing that issue btw
", "votes": 2, "replies": [] }, { "id": 2294044, "author_name": "Vladimir Simões da Luz Junior", "author_url": "", "post_date": "2023-06-09T17:18:44.913000", "content": "Thanks for pointing out this alignment problem @khahuras!💪
", "votes": 0, "replies": [] }, { "id": 2254485, "author_name": "马甲线来来来", "author_url": "", "post_date": "2023-05-11T03:33:21.783000", "content": "thx for sharing😁
", "votes": 0, "replies": [] }, { "id": 2237714, "author_name": "", "author_url": "", "post_date": "2023-04-27T22:23:46.600000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "2231440": "I see a common pattern in the defog dataset that events are present in a series, but both Task and Valid flag of those parts are False. That means these events won't take any effect at all during scoring. It is understood that where Task and Valid are both False, those parts are not annotated. \n\nSo if they're not annotated, why are FOG events being labeled there? And what is the meaning of these labels?\n\nAnother interesting pattern is that ALL StartHesitation events in defog dataset have Task and Valid as False (that means there is no StartHesitation in defog and should we just predict all 0?)\n\n\n\nEDIT:\nIf you look at the green clusters on the plot, you'll see they should align (overlap) with the 2 clusters where Valid=Task=True just before them, instead of not-overlap. So is there any data mistake with this defog dataset?\n\nCode to generate above plot:\n\nx = pd.read_csv('data/train/defog/e069a57511.csv')\nplt.plot(x.AccV, label='AccV')\nplt.plot(x.StartHesitation, label='StartHe')\nplt.plot(x.Turn*2, label='Turn')\nplt.plot(x.Walking*3, label='Walking')\nplt.plot(x.Task*-3, label='Task')\nplt.plot(x.Task*-1.5, label='Valid')\nplt.title('defog sample e069a57511')\nplt.legend();\n\n@jeffhausdorff @ryanholbrook ", "2234757": "Hi @khahuras,\n\nThank you for the heads up. We are looking into it now.", "2244400": "Data Update and Rescore by Kaggle Team\nhttps://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/406700", "2239041": "I created [a notebook](https://www.kaggle.com/code/hiroakifukuse/pfogp-correct-defog-data-discrepancies) to correct for the misalignment.\nWhen I trained the model with that corrected data, loss decreased, but LB worsened.\nAssuming a correlation between LB and CV, it may be possible to assume that the test data is also misaligned.", "2238583": "@ryanholbrook Can we get an update on this as soon as possible, or at least some information on whether this is confirmed or not? I know that validating something like that and communicating with the host might take a bit of time but I (and likely many others) find it hard to keep working on this competition not knowing whether the data is corrupt or not.", "2234420": "I changed this topic's name to get more attention, as it is affecting all modelling steps, as well as scoring (average precision will be affected if this problem isn't answered or resolved)\n\n@jeffhausdorff @ryanholbrook ", "2234019": "Is this alignment problem still there if one uses events.csv to get the labels?", "2232704": "@jeffhausdorff @ryanholbrook \nIf you look at the green clusters on the plot, you'll see they should align (overlap) with the 2 clusters where Valid=Task=True just before them, instead of not-overlap. So is there any data mistake with this defog dataset?", "2241353": "your assertion that: \"*ALL StartHesitation events in defog dataset have Task and Valid as False*\" is **false**. \n\nTime series 0d7ab3a9f9 in the defog train set do have a StartHesitation event with 88 timestamps where Valid an Task are both True (from Time 18150 -> 18237)", "2238691": "I do not see any mistake, your start hesitation color is blurring the pattern but still both Task and Valid are aligned with Turn green color.", "2242078": "Thank you for sharing! I also have the same question. After I set the label labeled as unavailable data to 0, LB significantly decreased.\n\nOn the contrary, the code that ignores the Valid and Task tags has a higher LB.", "2238519": "Just started looking at this competition and noticed the same thing. Eg., looked at series `6041cad8ec` and events vs annotations line up much better if I shift the events by approximately 20000 samples which looks similar to your plot:\n\n\n\n\n\n(`Event` is any event, and `Annotated` is `Valid` or `Task`.)", "2237688": "is there a typo? \n`plt.plot(x.Task-1.5, label='Valid')` x.Task --> x.Valid\n\nThanks, for pointing that issue btw", "2294044": "Thanks for pointing out this alignment problem @khahuras!💪", "2254485": "thx for sharing😁", "2237714": "" } }