I have used many features extracted from flux data but my CV score is always between 0.6 and 0.7 I couldn't do better. But I saw there are teams who had less than 0.5 CV score. Are you using other features? I mean the metadata's features like 'ra', 'decl', 'gal_l', 'gal_b', 'ddf' and \"distmod\". Is there anyone who has found an idea to take advantage of these features ? \nI had better score when I removed 'ra', 'decl', 'gal_l', 'gal_b'. \nAny ideas?
", "messages": [ { "id": "430795", "postDate": "12/01/2018 00:56:59", "content": "I have used many features extracted from flux data but my CV score is always between 0.6 and 0.7 I couldn't do better. But I saw there are teams who had less than 0.5 CV score. Are you using other features? I mean the metadata's features like 'ra', 'decl', 'gal_l', 'gal_b', 'ddf' and \"distmod\". Is there anyone who has found an idea to take advantage of these features ? \nI had better score when I removed 'ra', 'decl', 'gal_l', 'gal_b'. \nAny ideas?
", "rawMarkdown": "I have used many features extracted from flux data but my CV score is always between 0.6 and 0.7 I couldn't do better. But I saw there are teams who had less than 0.5 CV score. Are you using other features? I mean the metadata's features like **'ra', 'decl', 'gal_l', 'gal_b', 'ddf' and \"distmod\"**. Is there anyone who has found an idea to take advantage of these features ? \nI had better score when I removed 'ra', 'decl', 'gal_l', 'gal_b'. \nAny ideas?", "votes": null }, { "id": "430838", "postDate": "12/01/2018 03:24:04", "content": "My best model does not use any of 'ra', 'decl', 'gall', 'galb', 'ddf' and \"distmod\". But it uses hostgal_photoz which is almost equivalent to distmod. Location isn't relevant IMHO.
", "rawMarkdown": "My best model does not use any of 'ra', 'decl', 'gall', 'galb', 'ddf' and \"distmod\". But it uses hostgal_photoz which is almost equivalent to distmod. Location isn't relevant IMHO.", "votes": null }, { "id": "430840", "postDate": "12/01/2018 03:31:57", "content": "Yeah that's what models are predicting ! But logically, it should exist a relation between classes and features related to the location... I tried to apply cos, sin,tan, every combination of multiplication and addition between these features but it seems that they don't have any impact.\nI even searched about the relations between flux,location and velocity. \nI thought maybe there are some extracted features that people with astronomical knowledge are using!
", "rawMarkdown": "Yeah that's what models are predicting ! But logically, it should exist a relation between classes and features related to the location... I tried to apply cos, sin,tan, every combination of multiplication and addition between these features but it seems that they don't have any impact.\nI even searched about the relations between flux,location and velocity. \nI thought maybe there are some extracted features that people with astronomical knowledge are using!", "votes": null }, { "id": "430881", "postDate": "12/01/2018 05:20:13", "content": "\n\n\nit should exist a relation between classes and features related to the location…
\n
Why? Commonly accepted models of the universe say it is the same in every direction.
", "rawMarkdown": "> it should exist a relation between classes and features related to the location… \n\nWhy? Commonly accepted models of the universe say it is the same in every direction.", "votes": null }, { "id": "430985", "postDate": "12/01/2018 11:22:10", "content": "If your sample weights and weighted log loss calculations are correct, then I can confirm that 0.50-52 CV is really achievable just by feature engineering on flux information. However, I also really tried so many feature engineering after this point, and it never gets better. I really don't have a tiny clue about what those guys did to achieve 0.4X CV.
\n\nIt would be great if some confirm that it's also achievable just by keeping feature engineering on flux info. There must be some other features like 'mjd_diff' to give boosts.
", "rawMarkdown": "If your sample weights and weighted log loss calculations are correct, then I can confirm that 0.50-52 CV is really achievable just by feature engineering on flux information. However, I also really tried so many feature engineering after this point, and it never gets better. I really don't have a tiny clue about what those guys did to achieve 0.4X CV.\n\nIt would be great if some confirm that it's also achievable just by keeping feature engineering on flux info. There must be some other features like 'mjd_diff' to give boosts.", "votes": null }, { "id": "431088", "postDate": "12/01/2018 15:57:14", "content": "same here. i have already tried different feat engineering on flux, both library based and handcrafted, but I'm still stuck.
", "rawMarkdown": "same here. i have already tried different feat engineering on flux, both library based and handcrafted, but I'm still stuck.", "votes": null }, { "id": "431127", "postDate": "12/01/2018 16:47:19", "content": "@baba. just think about it: should the type of supernova in a galaxy be different if the galaxy is on your right vs. if the galaxy is on your left? sure there might be some event or predistribution of elements/conditions in a galaxy that causes a patch of say class 99, but training on position just tells your model to look at a feature that is not a characteristic of the event, but a detail of our viewpoint of the universe. observed from a planet elsewhere in the universe your model would break. it's overfitting.
", "rawMarkdown": "baba. just think about it: should the type of supernova in a galaxy be different if the galaxy is on your right vs. if the galaxy is on your left? sure there might be some event or predistribution of elements/conditions in a galaxy that causes a patch of say class 99, but training on position just tells your model to look at a feature that is not a characteristic of the event, but a detail of our viewpoint of the universe. observed from a planet elsewhere in the universe your model would break. it's overfitting.", "votes": null }, { "id": "431152", "postDate": "12/01/2018 17:13:33", "content": "Yeah you're right \nI was thinking that there is maybe some clusters based on the location that belong to the same class.\nNow, I understand why it s better to get rid of these features\nThank you !
", "rawMarkdown": "Yeah you're right \nI was thinking that there is maybe some clusters based on the location that belong to the same class.\nNow, I understand why it s better to get rid of these features\nThank you !", "votes": null }, { "id": "431155", "postDate": "12/01/2018 17:16:40", "content": "I am using these weights found in some kernels\nw = train_metadata.target.value_counts()\nweights = {i : np.sum(w) / w[i] for i in w.index}\nis it okey?
", "rawMarkdown": "I am using these weights found in some kernels\nw = train_metadata.target.value_counts()\nweights = {i : np.sum(w) / w[i] for i in w.index}\nis it okey?", "votes": null }, { "id": "431174", "postDate": "12/01/2018 18:02:46", "content": "Have you found out the information hidden at https://www.kaggle.com/c/PLAsTiCC-2018/discussion/70725#416740 ? Flux interactived with mjd or meta info?
", "rawMarkdown": "Have you found out the information hidden at https://www.kaggle.com/c/PLAsTiCC-2018/discussion/70725#416740 ? Flux interactived with mjd or meta info?", "votes": null }, { "id": "431182", "postDate": "12/01/2018 18:58:41", "content": "I'm also not using any of the positional features that CPMP listed in my model. The simulation doesn't seem to include positional information other than simulating the Milky Way galaxy. Your model might think that those features are important because they do separate galactic objects from extragalactic ones. However, the photoz (or distmod) splits galactic from extragalactic perfectly, so you lose nothing from dropping the positional features.
\n\nIn reality, things will be quite different. By the time that LSST comes online, the Gaia satellite will have mapped out a huge fraction of the stars in our galaxy, so cross referencing the coordinates of a transient to the Gaia catalog will provide a lot of information. Additionally, we have lots more information about the galaxies that the transients went off in that could be incorporated into the models. Maybe we'll need a PLAsTiCC 2.0...
", "rawMarkdown": "I'm also not using any of the positional features that CPMP listed in my model. The simulation doesn't seem to include positional information other than simulating the Milky Way galaxy. Your model might think that those features are important because they do separate galactic objects from extragalactic ones. However, the photoz (or distmod) splits galactic from extragalactic perfectly, so you lose nothing from dropping the positional features.\n\nIn reality, things will be quite different. By the time that LSST comes online, the Gaia satellite will have mapped out a huge fraction of the stars in our galaxy, so cross referencing the coordinates of a transient to the Gaia catalog will provide a lot of information. Additionally, we have lots more information about the galaxies that the transients went off in that could be incorporated into the models. Maybe we'll need a PLAsTiCC 2.0...", "votes": null }, { "id": "431203", "postDate": "12/01/2018 20:30:45", "content": "\n\n\nMaybe we'll need a PLAsTiCC 2.0…
\n
That would be awesome.
", "rawMarkdown": "> Maybe we'll need a PLAsTiCC 2.0…\n\nThat would be awesome.", "votes": null }, { "id": "431252", "postDate": "12/02/2018 00:11:55", "content": "@kyleboone Although galaxies are fixed in the sky (sorry for my crude expression, I am not an astronomer), but we make measurements on moving platform with ever changing environment. So positional features may have links with time, like days, years or months. I assume flux or flux errors at least at dawn and dusk will be different from those at midnight. Full moon may hinder visibility of a star. Have you tried features from mjd with periodicity and its harmonic combinations of galactic coordinates?
", "rawMarkdown": "[@kyleboone](https://www.kaggle.com/kyleboone ) Although galaxies are fixed in the sky (sorry for my crude expression, I am not an astronomer), but we make measurements on moving platform with ever changing environment. So positional features may have links with time, like days, years or months. I assume flux or flux errors at least at dawn and dusk will be different from those at midnight. Full moon may hinder visibility of a star. Have you tried features from mjd with periodicity and its harmonic combinations of galactic coordinates?", "votes": null }, { "id": "431260", "postDate": "12/02/2018 00:40:44", "content": "I also want PLAsTICC 2.0 !
", "rawMarkdown": "I also want PLAsTICC 2.0 !", "votes": null }, { "id": "431342", "postDate": "12/02/2018 04:42:46", "content": "The presence of the Moon does increase the sky background a lot, but if you have a good pipeline to reduce the images that just means that you'll have larger error bars, not that there will be a bias or anything like that. We are provided the error bars, and they should take these effects into account.
\n\nAlso, we typically try to observe away from the Moon, so there might actually be gaps in the lightcurve when objects are close to the Moon. I haven't looked for that.
\n\nRegardless, all of these effects should be fully captured by the lightcurve fluxes and flux errors. You might find correlations with things like the Moon location, but I don't think that they will help you with classification.
", "rawMarkdown": "The presence of the Moon does increase the sky background a lot, but if you have a good pipeline to reduce the images that just means that you'll have larger error bars, not that there will be a bias or anything like that. We are provided the error bars, and they should take these effects into account.\n\nAlso, we typically try to observe away from the Moon, so there might actually be gaps in the lightcurve when objects are close to the Moon. I haven't looked for that.\n\nRegardless, all of these effects should be fully captured by the lightcurve fluxes and flux errors. You might find correlations with things like the Moon location, but I don't think that they will help you with classification.", "votes": null }, { "id": "431470", "postDate": "12/02/2018 10:49:13", "content": "@KALE I couldn't. Wonder if there is someone else understood the thread and got that 0.15 gain as well.
", "rawMarkdown": "KALE I couldn't. Wonder if there is someone else understood the thread and got that 0.15 gain as well.", "votes": null }, { "id": "431545", "postDate": "12/02/2018 13:22:37", "content": "\n\n\nhttps://www.kaggle.com/c/PLAsTiCC-2018/discussion/70725#416740
\n
I may have started something in that thread, but be reassured. I didn't found another feature that yielded 0.15 improvement. And both @kyleboone and I gave quite precise hints elsewhere in the forum about the missing feature.
", "rawMarkdown": "> https://www.kaggle.com/c/PLAsTiCC-2018/discussion/70725#416740 \n\nI may have started something in that thread, but be reassured. I didn't found another feature that yielded 0.15 improvement. And both @kyleboone and I gave quite precise hints elsewhere in the forum about the missing feature.", "votes": null } ], "comments": [ { "id": 430838, "author_name": "cpmpml", "author_url": "", "post_date": "12/01/2018 03:24:04", "content": "My best model does not use any of 'ra', 'decl', 'gall', 'galb', 'ddf' and \"distmod\". But it uses hostgal_photoz which is almost equivalent to distmod. Location isn't relevant IMHO.
", "votes": null, "replies": [ { "id": 430840, "author_name": "rinnqd", "author_url": "", "post_date": "12/01/2018 03:31:57", "content": "Yeah that's what models are predicting ! But logically, it should exist a relation between classes and features related to the location... I tried to apply cos, sin,tan, every combination of multiplication and addition between these features but it seems that they don't have any impact.\nI even searched about the relations between flux,location and velocity. \nI thought maybe there are some extracted features that people with astronomical knowledge are using!
", "votes": null, "replies": [] }, { "id": 430881, "author_name": "cpmpml", "author_url": "", "post_date": "12/01/2018 05:20:13", "content": "\n\n\nit should exist a relation between classes and features related to the location…
\n
Why? Commonly accepted models of the universe say it is the same in every direction.
", "votes": null, "replies": [] }, { "id": 431127, "author_name": "yvanscher", "author_url": "", "post_date": "12/01/2018 16:47:19", "content": "@baba. just think about it: should the type of supernova in a galaxy be different if the galaxy is on your right vs. if the galaxy is on your left? sure there might be some event or predistribution of elements/conditions in a galaxy that causes a patch of say class 99, but training on position just tells your model to look at a feature that is not a characteristic of the event, but a detail of our viewpoint of the universe. observed from a planet elsewhere in the universe your model would break. it's overfitting.
", "votes": null, "replies": [] }, { "id": 431152, "author_name": "rinnqd", "author_url": "", "post_date": "12/01/2018 17:13:33", "content": "Yeah you're right \nI was thinking that there is maybe some clusters based on the location that belong to the same class.\nNow, I understand why it s better to get rid of these features\nThank you !
", "votes": null, "replies": [] }, { "id": 431182, "author_name": "kyleboone", "author_url": "", "post_date": "12/01/2018 18:58:41", "content": "I'm also not using any of the positional features that CPMP listed in my model. The simulation doesn't seem to include positional information other than simulating the Milky Way galaxy. Your model might think that those features are important because they do separate galactic objects from extragalactic ones. However, the photoz (or distmod) splits galactic from extragalactic perfectly, so you lose nothing from dropping the positional features.
\n\nIn reality, things will be quite different. By the time that LSST comes online, the Gaia satellite will have mapped out a huge fraction of the stars in our galaxy, so cross referencing the coordinates of a transient to the Gaia catalog will provide a lot of information. Additionally, we have lots more information about the galaxies that the transients went off in that could be incorporated into the models. Maybe we'll need a PLAsTiCC 2.0...
", "votes": null, "replies": [] }, { "id": 431203, "author_name": "cpmpml", "author_url": "", "post_date": "12/01/2018 20:30:45", "content": "\n\n\nMaybe we'll need a PLAsTiCC 2.0…
\n
That would be awesome.
", "votes": null, "replies": [] }, { "id": 431252, "author_name": "zeemeen", "author_url": "", "post_date": "12/02/2018 00:11:55", "content": "@kyleboone Although galaxies are fixed in the sky (sorry for my crude expression, I am not an astronomer), but we make measurements on moving platform with ever changing environment. So positional features may have links with time, like days, years or months. I assume flux or flux errors at least at dawn and dusk will be different from those at midnight. Full moon may hinder visibility of a star. Have you tried features from mjd with periodicity and its harmonic combinations of galactic coordinates?
", "votes": null, "replies": [] }, { "id": 431260, "author_name": "blondinka", "author_url": "", "post_date": "12/02/2018 00:40:44", "content": "I also want PLAsTICC 2.0 !
", "votes": null, "replies": [] }, { "id": 431342, "author_name": "kyleboone", "author_url": "", "post_date": "12/02/2018 04:42:46", "content": "The presence of the Moon does increase the sky background a lot, but if you have a good pipeline to reduce the images that just means that you'll have larger error bars, not that there will be a bias or anything like that. We are provided the error bars, and they should take these effects into account.
\n\nAlso, we typically try to observe away from the Moon, so there might actually be gaps in the lightcurve when objects are close to the Moon. I haven't looked for that.
\n\nRegardless, all of these effects should be fully captured by the lightcurve fluxes and flux errors. You might find correlations with things like the Moon location, but I don't think that they will help you with classification.
", "votes": null, "replies": [] } ] }, { "id": 430985, "author_name": "fatihozturk", "author_url": "", "post_date": "12/01/2018 11:22:10", "content": "If your sample weights and weighted log loss calculations are correct, then I can confirm that 0.50-52 CV is really achievable just by feature engineering on flux information. However, I also really tried so many feature engineering after this point, and it never gets better. I really don't have a tiny clue about what those guys did to achieve 0.4X CV.
\n\nIt would be great if some confirm that it's also achievable just by keeping feature engineering on flux info. There must be some other features like 'mjd_diff' to give boosts.
", "votes": null, "replies": [ { "id": 431088, "author_name": "niclasdoce", "author_url": "", "post_date": "12/01/2018 15:57:14", "content": "same here. i have already tried different feat engineering on flux, both library based and handcrafted, but I'm still stuck.
", "votes": null, "replies": [] }, { "id": 431155, "author_name": "rinnqd", "author_url": "", "post_date": "12/01/2018 17:16:40", "content": "I am using these weights found in some kernels\nw = train_metadata.target.value_counts()\nweights = {i : np.sum(w) / w[i] for i in w.index}\nis it okey?
", "votes": null, "replies": [] }, { "id": 431174, "author_name": "cczaixian", "author_url": "", "post_date": "12/01/2018 18:02:46", "content": "Have you found out the information hidden at https://www.kaggle.com/c/PLAsTiCC-2018/discussion/70725#416740 ? Flux interactived with mjd or meta info?
", "votes": null, "replies": [] }, { "id": 431470, "author_name": "fatihozturk", "author_url": "", "post_date": "12/02/2018 10:49:13", "content": "@KALE I couldn't. Wonder if there is someone else understood the thread and got that 0.15 gain as well.
", "votes": null, "replies": [] }, { "id": 431545, "author_name": "cpmpml", "author_url": "", "post_date": "12/02/2018 13:22:37", "content": "\n\n\nhttps://www.kaggle.com/c/PLAsTiCC-2018/discussion/70725#416740
\n
I may have started something in that thread, but be reassured. I didn't found another feature that yielded 0.15 improvement. And both @kyleboone and I gave quite precise hints elsewhere in the forum about the missing feature.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "430795": "I have used many features extracted from flux data but my CV score is always between 0.6 and 0.7 I couldn't do better. But I saw there are teams who had less than 0.5 CV score. Are you using other features? I mean the metadata's features like **'ra', 'decl', 'gal_l', 'gal_b', 'ddf' and \"distmod\"**. Is there anyone who has found an idea to take advantage of these features ? \nI had better score when I removed 'ra', 'decl', 'gal_l', 'gal_b'. \nAny ideas?", "430838": "My best model does not use any of 'ra', 'decl', 'gall', 'galb', 'ddf' and \"distmod\". But it uses hostgal_photoz which is almost equivalent to distmod. Location isn't relevant IMHO.", "430840": "Yeah that's what models are predicting ! But logically, it should exist a relation between classes and features related to the location... I tried to apply cos, sin,tan, every combination of multiplication and addition between these features but it seems that they don't have any impact.\nI even searched about the relations between flux,location and velocity. \nI thought maybe there are some extracted features that people with astronomical knowledge are using!", "430881": "> it should exist a relation between classes and features related to the location… \n\nWhy? Commonly accepted models of the universe say it is the same in every direction.", "430985": "If your sample weights and weighted log loss calculations are correct, then I can confirm that 0.50-52 CV is really achievable just by feature engineering on flux information. However, I also really tried so many feature engineering after this point, and it never gets better. I really don't have a tiny clue about what those guys did to achieve 0.4X CV.\n\nIt would be great if some confirm that it's also achievable just by keeping feature engineering on flux info. There must be some other features like 'mjd_diff' to give boosts.", "431088": "same here. i have already tried different feat engineering on flux, both library based and handcrafted, but I'm still stuck.", "431127": "baba. just think about it: should the type of supernova in a galaxy be different if the galaxy is on your right vs. if the galaxy is on your left? sure there might be some event or predistribution of elements/conditions in a galaxy that causes a patch of say class 99, but training on position just tells your model to look at a feature that is not a characteristic of the event, but a detail of our viewpoint of the universe. observed from a planet elsewhere in the universe your model would break. it's overfitting.", "431152": "Yeah you're right \nI was thinking that there is maybe some clusters based on the location that belong to the same class.\nNow, I understand why it s better to get rid of these features\nThank you !", "431155": "I am using these weights found in some kernels\nw = train_metadata.target.value_counts()\nweights = {i : np.sum(w) / w[i] for i in w.index}\nis it okey?", "431174": "Have you found out the information hidden at https://www.kaggle.com/c/PLAsTiCC-2018/discussion/70725#416740 ? Flux interactived with mjd or meta info?", "431182": "I'm also not using any of the positional features that CPMP listed in my model. The simulation doesn't seem to include positional information other than simulating the Milky Way galaxy. Your model might think that those features are important because they do separate galactic objects from extragalactic ones. However, the photoz (or distmod) splits galactic from extragalactic perfectly, so you lose nothing from dropping the positional features.\n\nIn reality, things will be quite different. By the time that LSST comes online, the Gaia satellite will have mapped out a huge fraction of the stars in our galaxy, so cross referencing the coordinates of a transient to the Gaia catalog will provide a lot of information. Additionally, we have lots more information about the galaxies that the transients went off in that could be incorporated into the models. Maybe we'll need a PLAsTiCC 2.0...", "431203": "> Maybe we'll need a PLAsTiCC 2.0…\n\nThat would be awesome.", "431252": "[@kyleboone](https://www.kaggle.com/kyleboone ) Although galaxies are fixed in the sky (sorry for my crude expression, I am not an astronomer), but we make measurements on moving platform with ever changing environment. So positional features may have links with time, like days, years or months. I assume flux or flux errors at least at dawn and dusk will be different from those at midnight. Full moon may hinder visibility of a star. Have you tried features from mjd with periodicity and its harmonic combinations of galactic coordinates?", "431260": "I also want PLAsTICC 2.0 !", "431342": "The presence of the Moon does increase the sky background a lot, but if you have a good pipeline to reduce the images that just means that you'll have larger error bars, not that there will be a bias or anything like that. We are provided the error bars, and they should take these effects into account.\n\nAlso, we typically try to observe away from the Moon, so there might actually be gaps in the lightcurve when objects are close to the Moon. I haven't looked for that.\n\nRegardless, all of these effects should be fully captured by the lightcurve fluxes and flux errors. You might find correlations with things like the Moon location, but I don't think that they will help you with classification.", "431470": "KALE I couldn't. Wonder if there is someone else understood the thread and got that 0.15 gain as well.", "431545": "> https://www.kaggle.com/c/PLAsTiCC-2018/discussion/70725#416740 \n\nI may have started something in that thread, but be reassured. I didn't found another feature that yielded 0.15 improvement. And both @kyleboone and I gave quite precise hints elsewhere in the forum about the missing feature." }, "source": "meta" }