I have computed period for training data using gastpy, but this does not improve my CV. Before embarking on the significant effort to get period for test data, I wonder if anyone saw a boost using period as a feature?
\n\nIt could well be that it helps on test even if it does not on train. I think I'll start computation anyway, but any feedback is more than welcome.
", "messages": [ { "id": 414247, "postDate": "2018-11-02T12:06:55.153Z", "content": "I have computed period for training data using gastpy, but this does not improve my CV. Before embarking on the significant effort to get period for test data, I wonder if anyone saw a boost using period as a feature?
\n\nIt could well be that it helps on test even if it does not on train. I think I'll start computation anyway, but any feedback is more than welcome.
", "rawMarkdown": "I have computed period for training data using gastpy, but this does not improve my CV. Before embarking on the significant effort to get period for test data, I wonder if anyone saw a boost using period as a feature? \n\nIt could well be that it helps on test even if it does not on train. I think I'll start computation anyway, but any feedback is more than welcome.", "votes": 10 }, { "id": 414259, "postDate": "2018-11-02T12:40:19.713Z", "content": "I have found, that some features need co-features to be active. Period is probably one of such features - it seems to be obvious that it should be an important one. Up to now, I have not found a co-feature which can activate it.
", "rawMarkdown": "I have found, that some features need co-features to be active. Period is probably one of such features - it seems to be obvious that it should be an important one. Up to now, I have not found a co-feature which can activate it. ", "votes": 3, "replies": [ { "id": 414309, "postDate": "2018-11-02T14:09:25.930Z", "content": "Thanks Grzegorz, I am in good company!
", "rawMarkdown": "Thanks Grzegorz, I am in good company!" } ] }, { "id": 415008, "postDate": "2018-11-04T05:23:58.590Z", "content": "I know it's a totally unreasonable request; but if anyone does end up brute forcing the period calculations for the test set, consider how much duplicated compute / wasted electricity would be saved by open sourcing it :-)
", "rawMarkdown": "I know it's a totally unreasonable request; but if anyone _does_ end up brute forcing the period calculations for the test set, consider how much duplicated compute / wasted electricity would be saved by open sourcing it :-)", "votes": 4, "replies": [ { "id": 415026, "postDate": "2018-11-04T07:34:00.467Z", "content": "I am ok to do this if I can find some usefulness for the period.
", "rawMarkdown": "I am ok to do this if I can find some usefulness for the period. ", "votes": 2 } ] }, { "id": 415783, "postDate": "2018-11-05T17:23:02.747Z", "content": "I haven't found, how to use it yet. Neither best period, neither its power does not help my CV. But I definitelly plan to continue on examining this path further - you also need period for making phase curves. Together with localizing M0 (basic minimum or maximum), you can calculate assymetry of light curve. Or you can feed it into RNN. It should be helpful, because profesional astronomer is able to detect a variability type from a phase curves in many cases. But it might as well be a dead end for sure.
", "rawMarkdown": "I haven't found, how to use it yet. Neither best period, neither its power does not help my CV. But I definitelly plan to continue on examining this path further - you also need period for making phase curves. Together with localizing M0 (basic minimum or maximum), you can calculate assymetry of light curve. Or you can feed it into RNN. It *should* be helpful, because profesional astronomer is able to detect a variability type from a phase curves in many cases. But it might as well be a dead end for sure.", "votes": 1 }, { "id": 414940, "postDate": "2018-11-03T23:38:06.577Z", "content": "Yea i tried adding period features but with little success/improvement to local cv. Best I could get was 0.005 improvement which isn't awful but also not worth investing in right now. Coming back to the competition after 4-5 days, I went from position ~30 to ~70 so i'm guessing there's been some interesting developments since I last submitted.
", "rawMarkdown": "Yea i tried adding period features but with little success/improvement to local cv. Best I could get was 0.005 improvement which isn't awful but also not worth investing in right now. Coming back to the competition after 4-5 days, I went from position ~30 to ~70 so i'm guessing there's been some interesting developments since I last submitted.", "votes": 1 }, { "id": 414347, "postDate": "2018-11-02T15:29:21.167Z", "content": "May I ask how much time it took for you to compute just for train?
", "rawMarkdown": "May I ask how much time it took for you to compute just for train?", "votes": 1, "replies": [ { "id": 414357, "postDate": "2018-11-02T15:47:01.130Z", "content": "I think it was about one hour.
", "rawMarkdown": "I think it was about one hour." } ] }, { "id": 414571, "postDate": "2018-11-03T03:39:29.727Z", "content": "I just kind of confirmed by hand that it is a very useful feature to distinguish between certain classes / class groups. It can also serve as a way to increase data density for periodic objects via phase space conversion. See my kernel around the end.
", "rawMarkdown": "I just kind of confirmed by hand that it is a very useful feature to distinguish between certain classes / class groups. It can also serve as a way to increase data density for periodic objects via phase space conversion. See my [kernel](https://www.kaggle.com/mithrillion/strategies-for-flux-time-series-preprocessing) around the end.", "votes": 2, "replies": [ { "id": 414599, "postDate": "2018-11-03T06:00:26.857Z", "content": "Is your local CV improved with it? Your kernel has no model training AFAIK.
", "rawMarkdown": "Is your local CV improved with it? Your kernel has no model training AFAIK." }, { "id": 414602, "postDate": "2018-11-03T06:06:32.083Z", "content": "I guess if it improves \"naked-eye CV\" it should be self-evident whether it is useful or not... But I'll try after I finish work some more explorations. Don't have the CPU hour for mass extraction yet.
", "rawMarkdown": "I guess if it improves \"naked-eye CV\" it should be self-evident whether it is useful or not... But I'll try after I finish work some more explorations. Don't have the CPU hour for mass extraction yet." }, { "id": 414609, "postDate": "2018-11-03T06:37:35.413Z", "content": "I never trust any feature engineering until it is validated by a better model.
", "rawMarkdown": "I never trust any feature engineering until it is validated by a better model." }, { "id": 415506, "postDate": "2018-11-05T08:40:49.410Z", "content": "To be clear: is period able to add further value when you have features that already distinguish the easy cases pretty well?
\n\nThe only way to know is to check if using period improves CV and LB scores. I see your kernels, but they lack model validation.
", "rawMarkdown": "To be clear: is period able to add further value when you have features that already distinguish the easy cases pretty well? \n\nThe only way to know is to check if using period improves CV and LB scores. I see your kernels, but they lack model validation." } ] }, { "id": 426791, "postDate": "2018-11-23T21:31:45.727Z", "content": "You're the first now. Can you tell if period is useful for you? :)
", "rawMarkdown": "You're the first now. Can you tell if period is useful for you? :) ", "replies": [ { "id": 426969, "postDate": "2018-11-24T09:09:56.550Z", "content": "Not useful for me, not used in my 0.801 lgb model.
\n\nIt does not mean it cannot be useful, I have not tried it recently.
", "rawMarkdown": "Not useful for me, not used in my 0.801 lgb model.\n\nIt does not mean it cannot be useful, I have not tried it recently.", "votes": 2 }, { "id": 426978, "postDate": "2018-11-24T09:32:14.837Z", "content": "Thanks for info!!!
", "rawMarkdown": "Thanks for info!!!" } ] }, { "id": 420315, "postDate": "2018-11-13T12:50:54.093Z", "content": "How do you calculate the period?
", "rawMarkdown": "How do you calculate the period?\n", "replies": [ { "id": 420389, "postDate": "2018-11-13T14:54:28.660Z", "content": "The introductory kernel shared by organizers has gatspy code for computing the period: https://www.kaggle.com/michaelapers/the-plasticc-astronomy-starter-kit . There are other packages to compute it like cesium, discussed in the forum or in public kernels, have a look.
", "rawMarkdown": "The introductory kernel shared by organizers has gatspy code for computing the period: https://www.kaggle.com/michaelapers/the-plasticc-astronomy-starter-kit . There are other packages to compute it like cesium, discussed in the forum or in public kernels, have a look." } ] }, { "id": 415773, "postDate": "2018-11-05T17:11:37.857Z", "content": "@CPMP could you define what exactly you mean by the \"period\"? There is different periodicity in time traces and so the \"period\" is just too broad. For instance, there is a \"period\" of large gaps (when the telescope is not watching) of about every 6 months (useless feature), there is an average period of the observations in the other periods (subject to the weather conditions, by the way, so also useless, I guess), and there are other options that can have a \"period\"
", "rawMarkdown": "@CPMP could you define what exactly you mean by the \"period\"? There is different periodicity in time traces and so the \"period\" is just too broad. For instance, there is a \"period\" of large gaps (when the telescope is not watching) of about every 6 months (useless feature), there is an average period of the observations in the other periods (subject to the weather conditions, by the way, so also useless, I guess), and there are other options that can have a \"period\" ", "replies": [ { "id": 415776, "postDate": "2018-11-05T17:15:04.280Z", "content": "I mean the period computed by Lomb Scargle algorithm. It is used i several popular public kernels, but I don't find it useful for me. So far at least.
", "rawMarkdown": "I mean the period computed by Lomb Scargle algorithm. It is used i several popular public kernels, but I don't find it useful for me. So far at least." } ] }, { "id": 414325, "postDate": "2018-11-02T14:50:20.460Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 414327, "postDate": "2018-11-02T14:58:49.523Z", "content": "Yes, everybody says it is useful, but it is not useful for me. This is why I ask.
", "rawMarkdown": "Yes, everybody says it is useful, but it is not useful for me. This is why I ask." } ] } ], "comments": [ { "id": 414259, "author_name": "Grzegorz Sionkowski", "author_url": "", "post_date": "2018-11-02T12:40:19.713000", "content": "I have found, that some features need co-features to be active. Period is probably one of such features - it seems to be obvious that it should be an important one. Up to now, I have not found a co-feature which can activate it.
", "votes": 3, "replies": [ { "id": 414309, "author_name": "CPMP", "author_url": "", "post_date": "2018-11-02T14:09:25.930000", "content": "Thanks Grzegorz, I am in good company!
", "votes": 0, "replies": [] } ] }, { "id": 415008, "author_name": "عثمان", "author_url": "", "post_date": "2018-11-04T05:23:58.590000", "content": "I know it's a totally unreasonable request; but if anyone does end up brute forcing the period calculations for the test set, consider how much duplicated compute / wasted electricity would be saved by open sourcing it :-)
", "votes": 4, "replies": [ { "id": 415026, "author_name": "CPMP", "author_url": "", "post_date": "2018-11-04T07:34:00.467000", "content": "I am ok to do this if I can find some usefulness for the period.
", "votes": 2, "replies": [] } ] }, { "id": 415783, "author_name": "Michal Haltuf", "author_url": "", "post_date": "2018-11-05T17:23:02.747000", "content": "I haven't found, how to use it yet. Neither best period, neither its power does not help my CV. But I definitelly plan to continue on examining this path further - you also need period for making phase curves. Together with localizing M0 (basic minimum or maximum), you can calculate assymetry of light curve. Or you can feed it into RNN. It should be helpful, because profesional astronomer is able to detect a variability type from a phase curves in many cases. But it might as well be a dead end for sure.
", "votes": 1, "replies": [] }, { "id": 414940, "author_name": "yvan", "author_url": "", "post_date": "2018-11-03T23:38:06.577000", "content": "Yea i tried adding period features but with little success/improvement to local cv. Best I could get was 0.005 improvement which isn't awful but also not worth investing in right now. Coming back to the competition after 4-5 days, I went from position ~30 to ~70 so i'm guessing there's been some interesting developments since I last submitted.
", "votes": 1, "replies": [] }, { "id": 414347, "author_name": "Fatih Öztürk", "author_url": "", "post_date": "2018-11-02T15:29:21.167000", "content": "May I ask how much time it took for you to compute just for train?
", "votes": 1, "replies": [ { "id": 414357, "author_name": "CPMP", "author_url": "", "post_date": "2018-11-02T15:47:01.130000", "content": "I think it was about one hour.
", "votes": 0, "replies": [] } ] }, { "id": 414571, "author_name": "Mithrillion", "author_url": "", "post_date": "2018-11-03T03:39:29.727000", "content": "I just kind of confirmed by hand that it is a very useful feature to distinguish between certain classes / class groups. It can also serve as a way to increase data density for periodic objects via phase space conversion. See my kernel around the end.
", "votes": 2, "replies": [ { "id": 414599, "author_name": "CPMP", "author_url": "", "post_date": "2018-11-03T06:00:26.857000", "content": "Is your local CV improved with it? Your kernel has no model training AFAIK.
", "votes": 0, "replies": [] }, { "id": 414602, "author_name": "Mithrillion", "author_url": "", "post_date": "2018-11-03T06:06:32.083000", "content": "I guess if it improves \"naked-eye CV\" it should be self-evident whether it is useful or not... But I'll try after I finish work some more explorations. Don't have the CPU hour for mass extraction yet.
", "votes": 0, "replies": [] }, { "id": 414609, "author_name": "CPMP", "author_url": "", "post_date": "2018-11-03T06:37:35.413000", "content": "I never trust any feature engineering until it is validated by a better model.
", "votes": 0, "replies": [] }, { "id": 415506, "author_name": "CPMP", "author_url": "", "post_date": "2018-11-05T08:40:49.410000", "content": "To be clear: is period able to add further value when you have features that already distinguish the easy cases pretty well?
\n\nThe only way to know is to check if using period improves CV and LB scores. I see your kernels, but they lack model validation.
", "votes": 0, "replies": [] } ] }, { "id": 426791, "author_name": "Sergey Zlobin", "author_url": "", "post_date": "2018-11-23T21:31:45.727000", "content": "You're the first now. Can you tell if period is useful for you? :)
", "votes": 0, "replies": [ { "id": 426969, "author_name": "CPMP", "author_url": "", "post_date": "2018-11-24T09:09:56.550000", "content": "Not useful for me, not used in my 0.801 lgb model.
\n\nIt does not mean it cannot be useful, I have not tried it recently.
", "votes": 2, "replies": [] }, { "id": 426978, "author_name": "Sergey Zlobin", "author_url": "", "post_date": "2018-11-24T09:32:14.837000", "content": "Thanks for info!!!
", "votes": 0, "replies": [] } ] }, { "id": 420315, "author_name": "Mikhail Karchevskiy", "author_url": "", "post_date": "2018-11-13T12:50:54.093000", "content": "How do you calculate the period?
", "votes": 0, "replies": [ { "id": 420389, "author_name": "CPMP", "author_url": "", "post_date": "2018-11-13T14:54:28.660000", "content": "The introductory kernel shared by organizers has gatspy code for computing the period: https://www.kaggle.com/michaelapers/the-plasticc-astronomy-starter-kit . There are other packages to compute it like cesium, discussed in the forum or in public kernels, have a look.
", "votes": 0, "replies": [] } ] }, { "id": 415773, "author_name": "Blonde", "author_url": "", "post_date": "2018-11-05T17:11:37.857000", "content": "@CPMP could you define what exactly you mean by the \"period\"? There is different periodicity in time traces and so the \"period\" is just too broad. For instance, there is a \"period\" of large gaps (when the telescope is not watching) of about every 6 months (useless feature), there is an average period of the observations in the other periods (subject to the weather conditions, by the way, so also useless, I guess), and there are other options that can have a \"period\"
", "votes": 0, "replies": [ { "id": 415776, "author_name": "CPMP", "author_url": "", "post_date": "2018-11-05T17:15:04.280000", "content": "I mean the period computed by Lomb Scargle algorithm. It is used i several popular public kernels, but I don't find it useful for me. So far at least.
", "votes": 0, "replies": [] } ] }, { "id": 414325, "author_name": "", "author_url": "", "post_date": "2018-11-02T14:50:20.460000", "content": "", "votes": 0, "replies": [ { "id": 414327, "author_name": "CPMP", "author_url": "", "post_date": "2018-11-02T14:58:49.523000", "content": "Yes, everybody says it is useful, but it is not useful for me. This is why I ask.
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "414247": "I have computed period for training data using gastpy, but this does not improve my CV. Before embarking on the significant effort to get period for test data, I wonder if anyone saw a boost using period as a feature? \n\nIt could well be that it helps on test even if it does not on train. I think I'll start computation anyway, but any feedback is more than welcome.", "414259": "I have found, that some features need co-features to be active. Period is probably one of such features - it seems to be obvious that it should be an important one. Up to now, I have not found a co-feature which can activate it. ", "415008": "I know it's a totally unreasonable request; but if anyone _does_ end up brute forcing the period calculations for the test set, consider how much duplicated compute / wasted electricity would be saved by open sourcing it :-)", "415783": "I haven't found, how to use it yet. Neither best period, neither its power does not help my CV. But I definitelly plan to continue on examining this path further - you also need period for making phase curves. Together with localizing M0 (basic minimum or maximum), you can calculate assymetry of light curve. Or you can feed it into RNN. It *should* be helpful, because profesional astronomer is able to detect a variability type from a phase curves in many cases. But it might as well be a dead end for sure.", "414940": "Yea i tried adding period features but with little success/improvement to local cv. Best I could get was 0.005 improvement which isn't awful but also not worth investing in right now. Coming back to the competition after 4-5 days, I went from position ~30 to ~70 so i'm guessing there's been some interesting developments since I last submitted.", "414347": "May I ask how much time it took for you to compute just for train?", "414571": "I just kind of confirmed by hand that it is a very useful feature to distinguish between certain classes / class groups. It can also serve as a way to increase data density for periodic objects via phase space conversion. See my [kernel](https://www.kaggle.com/mithrillion/strategies-for-flux-time-series-preprocessing) around the end.", "426791": "You're the first now. Can you tell if period is useful for you? :) ", "420315": "How do you calculate the period?\n", "415773": "@CPMP could you define what exactly you mean by the \"period\"? There is different periodicity in time traces and so the \"period\" is just too broad. For instance, there is a \"period\" of large gaps (when the telescope is not watching) of about every 6 months (useless feature), there is an average period of the observations in the other periods (subject to the weather conditions, by the way, so also useless, I guess), and there are other options that can have a \"period\" ", "414325": "" } }