Great to see the results to date. Curious if any teams are using the 24/7 daily living data; using it might help to avoid over fitting and to enhance generalizability.
\nWhile labels for FOG events do not exist for the daily living data, there are some ways of evaluating if the model results applied to that data set \"make sense.\"
\nFor example, among non-freezers, the number of FOG events and the total FOG duration (eg over the week) should be 0 or close to that and much larger in the freezers.
it would also be interesting to see if total FOG duration (summed across the week, for example) differs between those with and without FOG and how big the effect size is
\ne.g., Hedges's g (https://rowannicholls.github.io/python/statistics/effect_size.html#hedgess-g-1)
\nThanks! Jeff
Great to see the results to date. Curious if any teams are using the 24/7 daily living data; using it might help to avoid over fitting and to enhance generalizability.
\nWhile labels for FOG events do not exist for the daily living data, there are some ways of evaluating if the model results applied to that data set \"make sense.\"
\nFor example, among non-freezers, the number of FOG events and the total FOG duration (eg over the week) should be 0 or close to that and much larger in the freezers.
it would also be interesting to see if total FOG duration (summed across the week, for example) differs between those with and without FOG and how big the effect size is
\ne.g., Hedges's g (https://rowannicholls.github.io/python/statistics/effect_size.html#hedgess-g-1)
\nThanks! Jeff
Interesting post @jeffhausdorff - is predicting on the daily living dataset one of the ways you'd be hoping to test/use successful models?
\nThere's been some discussion in this thread about the legitimacy of explicit time features. If we use these in modelling then performance on the defog/tdcsfog datasets will likely be better, but generalisation to less contrived situations (such as daily living) will likely suffer.
\nThis notebook by @lingduowang shows the feature importance of one of the better performing public models. Time_frac - an explicit time feature - dominates the feature importance.
Any thoughts on this?
", "rawMarkdown": "Interesting post @jeffhausdorff - is predicting on the daily living dataset one of the ways you'd be hoping to test/use successful models?\n\nThere's been some discussion [in this thread](https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/400311) about the legitimacy of explicit time features. If we use these in modelling then performance on the defog/tdcsfog datasets will likely be better, but generalisation to less contrived situations (such as daily living) will likely suffer.\n\n[This notebook](https://www.kaggle.com/code/lingduowang/feature-importance-visualize-in-lgbm) by @lingduowang shows the feature importance of one of the better performing public models. `Time_frac` - an explicit time feature - dominates the feature importance.\n\nAny thoughts on this?", "votes": null }, { "id": "2234098", "postDate": "04/24/2023 21:50:40", "content": "Good points. Yes, a long-term goal is to predict on daily living datasets. That would be very useful and powerful. The more generalizable and robust the models are based on tDCS and DeFOG, the more likely they will do well on other data, collected in other patients with FOG (and more likely to do well on the test data here). And more general approaches and models are more likely to be successfully applied to daily-living data. We included tDCS and DeFOG data - two similar, but different FOG-provoking protocols, to try to capture a broad range of freezing (among a relatively broad range of patients).
", "rawMarkdown": "Good points. Yes, a long-term goal is to predict on daily living datasets. That would be very useful and powerful. The more generalizable and robust the models are based on tDCS and DeFOG, the more likely they will do well on other data, collected in other patients with FOG (and more likely to do well on the test data here). And more general approaches and models are more likely to be successfully applied to daily-living data. We included tDCS and DeFOG data - two similar, but different FOG-provoking protocols, to try to capture a broad range of freezing (among a relatively broad range of patients).", "votes": null }, { "id": "2238153", "postDate": "04/28/2023 08:48:58", "content": "I don't quite see the usefulness of the variable \"Time_frac\" for a real model. From what I see in the code, it is the most representative variable of the model, but it doesn't make sense in the real life. The episodes cannot be a function of time as they can happen any time. In the episodes for model training, it is clear that they can be related since specific episodes are chosen, the data is cut, and the episode is left within this data (and it has a sort of time relation). But if we want to obtain a model for real life continuous use, \"Time_frac\" should not be in the equation since an episode will never depend on time. I understand that it may be important during model training and for scoring, but ideally, we should obtain models independent of time.
", "rawMarkdown": "I don't quite see the usefulness of the variable \"Time_frac\" for a real model. From what I see in the code, it is the most representative variable of the model, but it doesn't make sense in the real life. The episodes cannot be a function of time as they can happen any time. In the episodes for model training, it is clear that they can be related since specific episodes are chosen, the data is cut, and the episode is left within this data (and it has a sort of time relation). But if we want to obtain a model for real life continuous use, \"Time_frac\" should not be in the equation since an episode will never depend on time. I understand that it may be important during model training and for scoring, but ideally, we should obtain models independent of time.", "votes": null }, { "id": "2238335", "postDate": "04/28/2023 12:34:16", "content": "I fully agree. Using Time_frac or similar approaches likely will not lead to a robust, generalizable solution. Also, I may be wrong, but my guess is that even for this specific competition, Time_frac and other uses of time along those lines might help to get in the general area of a FOG event, but it won't be very accurate or precise.
", "rawMarkdown": "I fully agree. Using Time_frac or similar approaches likely will not lead to a robust, generalizable solution. Also, I may be wrong, but my guess is that even for this specific competition, Time_frac and other uses of time along those lines might help to get in the general area of a FOG event, but it won't be very accurate or precise.", "votes": null }, { "id": "2240503", "postDate": "04/30/2023 15:37:34", "content": "It seems to me that the only way to fight this would be if the time-series were completely random splits of data, also with already started or unfinished events: this way the time passed since the start of the time-series would give no information on the targets.
\nI'm surprised the host is not much worried about this problem.
", "rawMarkdown": "It seems to me that the only way to fight this would be if the time-series were completely random splits of data, also with already started or unfinished events: this way the time passed since the start of the time-series would give no information on the targets. \n\nI'm surprised the host is not much worried about this problem.", "votes": null }, { "id": "2242327", "postDate": "05/02/2023 07:31:34", "content": "This is a concern. However, we anticipate that the best models will be generalizable, will need to be trained on a relatively larger dataset, and will work similarly on all datasets. In that case, too much reliance on the structure of a specific dataset structure won't be so helpful. A model that trains on all datasets will see a richer and more fully representative sample of FOG episodes; freezing can be highly variable across and within subjects.
", "rawMarkdown": "This is a concern. However, we anticipate that the best models will be generalizable, will need to be trained on a relatively larger dataset, and will work similarly on all datasets. In that case, too much reliance on the structure of a specific dataset structure won't be so helpful. A model that trains on all datasets will see a richer and more fully representative sample of FOG episodes; freezing can be highly variable across and within subjects.", "votes": null }, { "id": "2242419", "postDate": "05/02/2023 09:05:34", "content": "Are the time-series where you are checking the generalizability built in the same way the ones for training are, i.e. the events are generally in the middle of the time-series?
\nIf you already accounted for that sorry for this further message.
", "rawMarkdown": "Are the time-series where you are checking the generalizability built in the same way the ones for training are, i.e. the events are generally in the middle of the time-series?\n\nIf you already accounted for that sorry for this further message.", "votes": null } ], "comments": [ { "id": 2231755, "author_name": "jagofc", "author_url": "", "post_date": "04/23/2023 16:13:32", "content": "Interesting post @jeffhausdorff - is predicting on the daily living dataset one of the ways you'd be hoping to test/use successful models?
\nThere's been some discussion in this thread about the legitimacy of explicit time features. If we use these in modelling then performance on the defog/tdcsfog datasets will likely be better, but generalisation to less contrived situations (such as daily living) will likely suffer.
\nThis notebook by @lingduowang shows the feature importance of one of the better performing public models. Time_frac - an explicit time feature - dominates the feature importance.
Any thoughts on this?
", "votes": null, "replies": [ { "id": 2234098, "author_name": "jeffhausdorff", "author_url": "", "post_date": "04/24/2023 21:50:40", "content": "Good points. Yes, a long-term goal is to predict on daily living datasets. That would be very useful and powerful. The more generalizable and robust the models are based on tDCS and DeFOG, the more likely they will do well on other data, collected in other patients with FOG (and more likely to do well on the test data here). And more general approaches and models are more likely to be successfully applied to daily-living data. We included tDCS and DeFOG data - two similar, but different FOG-provoking protocols, to try to capture a broad range of freezing (among a relatively broad range of patients).
", "votes": null, "replies": [] }, { "id": 2238153, "author_name": "quimquadrada", "author_url": "", "post_date": "04/28/2023 08:48:58", "content": "I don't quite see the usefulness of the variable \"Time_frac\" for a real model. From what I see in the code, it is the most representative variable of the model, but it doesn't make sense in the real life. The episodes cannot be a function of time as they can happen any time. In the episodes for model training, it is clear that they can be related since specific episodes are chosen, the data is cut, and the episode is left within this data (and it has a sort of time relation). But if we want to obtain a model for real life continuous use, \"Time_frac\" should not be in the equation since an episode will never depend on time. I understand that it may be important during model training and for scoring, but ideally, we should obtain models independent of time.
", "votes": null, "replies": [ { "id": 2238335, "author_name": "jeffhausdorff", "author_url": "", "post_date": "04/28/2023 12:34:16", "content": "I fully agree. Using Time_frac or similar approaches likely will not lead to a robust, generalizable solution. Also, I may be wrong, but my guess is that even for this specific competition, Time_frac and other uses of time along those lines might help to get in the general area of a FOG event, but it won't be very accurate or precise.
", "votes": null, "replies": [] }, { "id": 2240503, "author_name": "albertoannoni", "author_url": "", "post_date": "04/30/2023 15:37:34", "content": "It seems to me that the only way to fight this would be if the time-series were completely random splits of data, also with already started or unfinished events: this way the time passed since the start of the time-series would give no information on the targets.
\nI'm surprised the host is not much worried about this problem.
", "votes": null, "replies": [ { "id": 2242327, "author_name": "jeffhausdorff", "author_url": "", "post_date": "05/02/2023 07:31:34", "content": "This is a concern. However, we anticipate that the best models will be generalizable, will need to be trained on a relatively larger dataset, and will work similarly on all datasets. In that case, too much reliance on the structure of a specific dataset structure won't be so helpful. A model that trains on all datasets will see a richer and more fully representative sample of FOG episodes; freezing can be highly variable across and within subjects.
", "votes": null, "replies": [ { "id": 2242419, "author_name": "albertoannoni", "author_url": "", "post_date": "05/02/2023 09:05:34", "content": "Are the time-series where you are checking the generalizability built in the same way the ones for training are, i.e. the events are generally in the middle of the time-series?
\nIf you already accounted for that sorry for this further message.
", "votes": null, "replies": [] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "2228601": "Great to see the results to date. Curious if any teams are using the 24/7 daily living data; using it might help to avoid over fitting and to enhance generalizability. \n\nWhile labels for FOG events do not exist for the daily living data, there are some ways of evaluating if the model results applied to that data set \"make sense.\"\nFor example, among non-freezers, the number of FOG events and the total FOG duration (eg over the week) should be 0 or close to that and much larger in the freezers. \n\nit would also be interesting to see if total FOG duration (summed across the week, for example) differs between those with and without FOG and how big the effect size is\ne.g., Hedges's g (https://rowannicholls.github.io/python/statistics/effect_size.html#hedgess-g-1)\nThanks! Jeff", "2231755": "Interesting post @jeffhausdorff - is predicting on the daily living dataset one of the ways you'd be hoping to test/use successful models?\n\nThere's been some discussion [in this thread](https://www.kaggle.com/competitions/tlvmc-parkinsons-freezing-gait-prediction/discussion/400311) about the legitimacy of explicit time features. If we use these in modelling then performance on the defog/tdcsfog datasets will likely be better, but generalisation to less contrived situations (such as daily living) will likely suffer.\n\n[This notebook](https://www.kaggle.com/code/lingduowang/feature-importance-visualize-in-lgbm) by @lingduowang shows the feature importance of one of the better performing public models. `Time_frac` - an explicit time feature - dominates the feature importance.\n\nAny thoughts on this?", "2234098": "Good points. Yes, a long-term goal is to predict on daily living datasets. That would be very useful and powerful. The more generalizable and robust the models are based on tDCS and DeFOG, the more likely they will do well on other data, collected in other patients with FOG (and more likely to do well on the test data here). And more general approaches and models are more likely to be successfully applied to daily-living data. We included tDCS and DeFOG data - two similar, but different FOG-provoking protocols, to try to capture a broad range of freezing (among a relatively broad range of patients).", "2238153": "I don't quite see the usefulness of the variable \"Time_frac\" for a real model. From what I see in the code, it is the most representative variable of the model, but it doesn't make sense in the real life. The episodes cannot be a function of time as they can happen any time. In the episodes for model training, it is clear that they can be related since specific episodes are chosen, the data is cut, and the episode is left within this data (and it has a sort of time relation). But if we want to obtain a model for real life continuous use, \"Time_frac\" should not be in the equation since an episode will never depend on time. I understand that it may be important during model training and for scoring, but ideally, we should obtain models independent of time.", "2238335": "I fully agree. Using Time_frac or similar approaches likely will not lead to a robust, generalizable solution. Also, I may be wrong, but my guess is that even for this specific competition, Time_frac and other uses of time along those lines might help to get in the general area of a FOG event, but it won't be very accurate or precise.", "2240503": "It seems to me that the only way to fight this would be if the time-series were completely random splits of data, also with already started or unfinished events: this way the time passed since the start of the time-series would give no information on the targets. \n\nI'm surprised the host is not much worried about this problem.", "2242327": "This is a concern. However, we anticipate that the best models will be generalizable, will need to be trained on a relatively larger dataset, and will work similarly on all datasets. In that case, too much reliance on the structure of a specific dataset structure won't be so helpful. A model that trains on all datasets will see a richer and more fully representative sample of FOG episodes; freezing can be highly variable across and within subjects.", "2242419": "Are the time-series where you are checking the generalizability built in the same way the ones for training are, i.e. the events are generally in the middle of the time-series?\n\nIf you already accounted for that sorry for this further message." }, "source": "meta" }