Very motivating competition!
\nThis is from the Evaluation page describing the Id for events in the submission file:
\n\n\nThe Id values in the submission file should have the form {SeriesId}_{Time} where SeriesId is the identifier of the data series and Time is the time-step in that series of the predictions
\n
In the time series data time is discretised into time steps. This is clear. But what is considered the exact time-step of prediction for an event? With sampling rates of 100 Hz (or 128 Hz), the sampling interval is 10 (or 7.8) ms. I guess its very hard to exactly detect the beginning of an event with such a high precision. Probably the human annotators had a relatively high variance with respect to sample counts around something like a true initialisation. Actually, I think clinically it won't matter too much whether an event is detected, say 100 ms earlier or later (like +/- 10 samples). But it would completely change the Id of that event! For example Ids for predictions {SeriesId}_110 and {SeriesId}_112 were just 20 ms apart. Maybe the annotator marked the event at time step 113. How is the grader going to deal with that? Is there a tolerance for Init Time detection?
\nOr do I get that wrong? From the exemplary submission file it looks like events are just sequentially counted. But if so, how then is Init Time encoded?
\nThnx for any clarification.
", "messages": [ { "id": "2175997", "postDate": "03/10/2023 10:01:48", "content": "Very motivating competition!
\nThis is from the Evaluation page describing the Id for events in the submission file:
\n\n\nThe Id values in the submission file should have the form {SeriesId}_{Time} where SeriesId is the identifier of the data series and Time is the time-step in that series of the predictions
\n
In the time series data time is discretised into time steps. This is clear. But what is considered the exact time-step of prediction for an event? With sampling rates of 100 Hz (or 128 Hz), the sampling interval is 10 (or 7.8) ms. I guess its very hard to exactly detect the beginning of an event with such a high precision. Probably the human annotators had a relatively high variance with respect to sample counts around something like a true initialisation. Actually, I think clinically it won't matter too much whether an event is detected, say 100 ms earlier or later (like +/- 10 samples). But it would completely change the Id of that event! For example Ids for predictions {SeriesId}_110 and {SeriesId}_112 were just 20 ms apart. Maybe the annotator marked the event at time step 113. How is the grader going to deal with that? Is there a tolerance for Init Time detection?
\nOr do I get that wrong? From the exemplary submission file it looks like events are just sequentially counted. But if so, how then is Init Time encoded?
\nThnx for any clarification.
", "rawMarkdown": "Very motivating competition! \n\nThis is from the [**Evaluation** page](https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/overview/evaluation) describing the *Id* for events in the *submission file*:\n>The Id values in the submission file should have the form {SeriesId}_{Time} where SeriesId is the identifier of the data series and Time is the time-step in that series of the predictions\n\nIn the time series data *time* is discretised into time steps. This is clear. But what is considered the exact time-step of prediction for an event? With sampling rates of 100 Hz (or 128 Hz), the sampling interval is 10 (or 7.8) ms. I guess its very hard to exactly detect the beginning of an event with such a high precision. Probably the human annotators had a relatively high variance with respect to sample counts around something like a *true* initialisation. Actually, I think clinically it won't matter too much whether an event is detected, say 100 ms earlier or later (like +/- 10 samples). But it would completely change the *Id* of that event! For example *Id*s for predictions {SeriesId}_110 and {SeriesId}_112 were just 20 ms apart. Maybe the annotator marked the event at *time step* 113. How is the grader going to deal with that? Is there a tolerance for *Init Time* detection?\n\nOr do I get that wrong? From the exemplary *submission file* it looks like events are just sequentially counted. But if so, how then is *Init Time* encoded?\n\nThnx for any clarification.", "votes": null }, { "id": "2176020", "postDate": "03/10/2023 10:30:32", "content": "Hey @michaelfauler,
\nNote that you'll want to predict every timestep for which a FOG event was present, not just the initial and ending times. So even if the annotated times are a bit noisy, their variance will still likely be relatively small compared to the total duration of the event, which is what you're being scored on.
\nJust to make it concrete, if you wanted to predict a StartHesitation event from step 110 to 300, your submission file might look like (leaving off the other two event types):
Id,StartHesitation\n...\n{SeriesId}_108,0.0\n{SeriesId}_109,0.2\n{SeriesId}_110,0.8\n{SeriesId}_111,0.9\n{SeriesId}_112,1.0\n{SeriesId}_113,1.0\n...\n{SeriesId}_298,1.0\n{SeriesId}_299,0.9\n{SeriesId}_300,0.8\n{SeriesId}_301,0.3\n{SeriesId}_302,0.0\nAnd note that the metric accepts confidence scores or probabilities instead of just binary indicators, so you can include uncertainty in your predictions, if you like.
\nHope this answers your concerns!
", "rawMarkdown": "Hey @michaelfauler,\n\nNote that you'll want to predict every timestep for which a FOG event was present, not just the initial and ending times. So even if the annotated times are a bit noisy, their variance will still likely be relatively small compared to the total duration of the event, which is what you're being scored on.\n\nJust to make it concrete, if you wanted to predict a `StartHesitation` event from step 110 to 300, your submission file might look like (leaving off the other two event types):\n\n```\nId,StartHesitation\n...\n{SeriesId}_108,0.0\n{SeriesId}_109,0.2\n{SeriesId}_110,0.8\n{SeriesId}_111,0.9\n{SeriesId}_112,1.0\n{SeriesId}_113,1.0\n...\n{SeriesId}_298,1.0\n{SeriesId}_299,0.9\n{SeriesId}_300,0.8\n{SeriesId}_301,0.3\n{SeriesId}_302,0.0\n```\n\nAnd note that the metric accepts confidence scores or probabilities instead of just binary indicators, so you can include uncertainty in your predictions, if you like.\n\nHope this answers your concerns!", "votes": null }, { "id": "2176487", "postDate": "03/10/2023 17:19:28", "content": "Yes, thank you.
\nSo, that means that longer lasting events will get an higher weight in the Mean Average Precision metric compared to the very short ones.
Hey @michaelfauler,
\nNote that you'll want to predict every timestep for which a FOG event was present, not just the initial and ending times. So even if the annotated times are a bit noisy, their variance will still likely be relatively small compared to the total duration of the event, which is what you're being scored on.
\nJust to make it concrete, if you wanted to predict a StartHesitation event from step 110 to 300, your submission file might look like (leaving off the other two event types):
Id,StartHesitation\n...\n{SeriesId}_108,0.0\n{SeriesId}_109,0.2\n{SeriesId}_110,0.8\n{SeriesId}_111,0.9\n{SeriesId}_112,1.0\n{SeriesId}_113,1.0\n...\n{SeriesId}_298,1.0\n{SeriesId}_299,0.9\n{SeriesId}_300,0.8\n{SeriesId}_301,0.3\n{SeriesId}_302,0.0\nAnd note that the metric accepts confidence scores or probabilities instead of just binary indicators, so you can include uncertainty in your predictions, if you like.
\nHope this answers your concerns!
", "votes": null, "replies": [ { "id": 2176487, "author_name": "michaelfauler", "author_url": "", "post_date": "03/10/2023 17:19:28", "content": "Yes, thank you.
\nSo, that means that longer lasting events will get an higher weight in the Mean Average Precision metric compared to the very short ones.