![\"confusion](\"https://storage.googleapis.com/kaggle-forum-message-attachments/416235/10623/confmatrix.png\")
My best model has hard time separating classes 42, 52, 62, 67, and 90. Separating these classes is the key to winning the competition, more than class 99 IMHO. See the confusion matrix:
\n\n
My best model has hard time separating classes 42, 52, 62, 67, and 90. Separating these classes is the key to winning the competition, more than class 99 IMHO. See the confusion matrix:
\n\n
I also made the same observations. The classes 42, 52, 62, 67, and 90 seem to be different types of supernovae. (This seems to be an argument for not focusing on the calculation of periods.) Most publications I have seen at actually handle those.
", "rawMarkdown": "I also made the same observations. The classes 42, 52, 62, 67, and 90 seem to be different types of supernovae. (This seems to be an argument for not focusing on the calculation of periods.) Most publications I have seen at actually handle those.", "votes": 5, "replies": [ { "id": 416493, "postDate": "2018-11-06T18:35:27.600Z", "content": "\n\n\nThis seems to be an argument for not focusing on the calculation of periods.
\n
Good point!
", "rawMarkdown": "> This seems to be an argument for not focusing on the calculation of periods.\n\nGood point!", "votes": 1 }, { "id": 416516, "postDate": "2018-11-06T19:43:38.323Z", "content": "I believe (haven't studied in deep yet), that supernovae can be in principle differentiated using lightcurve analysis as well (well, not period, of course, but speed of brightness decay, \"half-width\" and similar parameters). It could be helpful, if we managed to link these classes with specific supernovae types (i.e. 42 = Ia, just example).
", "rawMarkdown": "I believe (haven't studied in deep yet), that supernovae can be in principle differentiated using lightcurve analysis as well (well, not period, of course, but speed of brightness decay, \"half-width\" and similar parameters). It could be helpful, if we managed to link these classes with specific supernovae types (i.e. 42 = Ia, just example).", "votes": 2 }, { "id": 416817, "postDate": "2018-11-07T10:13:38.437Z", "content": "Period features (via LSSA) is basically curve fitting -> use fitted parameters as features. The same idea can definitely be used for other types of objects if you know the typical shape of the light curves and have a nice family of functions to approximate them. Actually if we know more about how supernovae differ, we might not need to know which label is which, as long as a fitted curve can describe such differences.
", "rawMarkdown": "Period features (via LSSA) is basically curve fitting -> use fitted parameters as features. The same idea can definitely be used for other types of objects if you know the typical shape of the light curves and have a nice family of functions to approximate them. Actually if we know more about how supernovae differ, we might not need to know which label is which, as long as a fitted curve can describe such differences." }, { "id": 416823, "postDate": "2018-11-07T10:23:54.230Z", "content": "Template fitting is indeed a technique used by astronomers AFAIK. I have not given it a try, and I am not even sure where to start on this.
", "rawMarkdown": "Template fitting is indeed a technique used by astronomers AFAIK. I have not given it a try, and I am not even sure where to start on this.", "votes": 1 }, { "id": 417141, "postDate": "2018-11-07T20:23:23.407Z", "content": "I am experiencing the same issue where class 90 is swallowing the other classifications of the aperiodic classes (supernova/nova?) since we have so many more training examples of that class as compared to the other \"spiky\" ones. As far as I can tell, several models can use high-level observation statistics (avg flux, max flux, std dev) to differentiate between the periodic classes and the episodic ones and discriminate well within the periodic classes. However when it comes to the episodic ones, my models give up and just say \"dump it in class 90.\"
\n\nThe last public light curve competition (SNPhotCC) from the LSST group focused exclusively on supernova classification. The paper below lists some of the key findings/techniques from that competition. As Michal and Mithrillion pointed out, previous contestants heavily relied on curve fitting and using the model parameters as features in the classifiers. On page 8, they list a specific curve that the was used to generate the samples in the prior competition.
\n\n\n\nFurthermore, these parameters were sensitive to redshift so closer SN will will have different parameters than ones of the same type that are far away.
\n\nJust like many people are using branching models on the Milky Way/Extragalatic feature, I've been trying to test for periodicity in the light curves and then have separate models for the periodic vs cataclysmic classes. One problem I have been facing is that there are some periodic training examples that also have transient spikes/weakening the strength of periodicity tests.
", "rawMarkdown": "I am experiencing the same issue where class 90 is swallowing the other classifications of the aperiodic classes (supernova/nova?) since we have so many more training examples of that class as compared to the other \"spiky\" ones. As far as I can tell, several models can use high-level observation statistics (avg flux, max flux, std dev) to differentiate between the periodic classes and the episodic ones and discriminate well within the periodic classes. However when it comes to the episodic ones, my models give up and just say \"dump it in class 90.\"\n\nThe last public light curve competition (SNPhotCC) from the LSST group focused exclusively on supernova classification. The paper below lists some of the key findings/techniques from that competition. As Michal and Mithrillion pointed out, previous contestants heavily relied on curve fitting and using the model parameters as features in the classifiers. On page 8, they list a specific curve that the was used to generate the samples in the prior competition.\n\nhttp://astro.uchicago.edu/~frieman/Courses/A411-old/References/supernova-photometric-classification-1008.1024.pdf\n\nFurthermore, these parameters were sensitive to redshift so closer SN will will have different parameters than ones of the same type that are far away. \n\nJust like many people are using branching models on the Milky Way/Extragalatic feature, I've been trying to test for periodicity in the light curves and then have separate models for the periodic vs cataclysmic classes. One problem I have been facing is that there are some periodic training examples that also have transient spikes/weakening the strength of periodicity tests.", "votes": 11 }, { "id": 417174, "postDate": "2018-11-07T22:14:16.433Z", "content": "Thanks a lot for sharing this.
", "rawMarkdown": "Thanks a lot for sharing this." }, { "id": 417353, "postDate": "2018-11-08T06:22:09.700Z", "content": "\n\n\nI am experiencing the same issue where class 90 is swallowing the other classifications of the aperiodic classes (supernova/nova?) since we have so many more training examples of that class as compared to the other \"spiky\" ones.
\n
I used the neural network model from Kernels and it seems to be right for that model. The result is more than 2 millions (out of 3.5 millions) of 'class 90' samples. I suspect this is wrong.\n@CPMP How many samples of 'class 90' do you have in your result? I used the maximum value in a row to determine the class for a sample.
", "rawMarkdown": "> I am experiencing the same issue where class 90 is swallowing the other classifications of the aperiodic classes (supernova/nova?) since we have so many more training examples of that class as compared to the other \"spiky\" ones.\n\nI used the neural network model from Kernels and it seems to be right for that model. The result is more than 2 millions (out of 3.5 millions) of 'class 90' samples. I suspect this is wrong.\n@CPMP How many samples of 'class 90' do you have in your result? I used the maximum value in a row to determine the class for a sample." }, { "id": 417561, "postDate": "2018-11-08T13:02:59.463Z", "content": "\n\n\nHow many samples of 'class 90' do you have in your result?
\n
The confusion matrix is computed on out of fold predictions for all the training data. I therefore have as many examples in class 90 as there are in training data.
", "rawMarkdown": "> How many samples of 'class 90' do you have in your result?\n\nThe confusion matrix is computed on out of fold predictions for all the training data. I therefore have as many examples in class 90 as there are in training data.", "votes": -2 }, { "id": 417576, "postDate": "2018-11-08T13:32:23.523Z", "content": "I mean the number of 'class 90' samples in the submission file. I suspect there shouldn't be so many samples with high 'class 90' probability.
", "rawMarkdown": "I mean the number of 'class 90' samples in the submission file. I suspect there shouldn't be so many samples with high 'class 90' probability. " }, { "id": 417586, "postDate": "2018-11-08T13:46:40.737Z", "content": "Here are the mean probabilities for the submission file corresponding to that confusion matrix. I did not compute classes as you suggest, as this is not how submissions are evaluated ;)
\n\nclass_6 1.776892e-03\nclass_15 1.072527e-01\nclass_16 2.913544e-02\nclass_42 1.477831e-01\nclass_52 9.749433e-02\nclass_53 5.449143e-04\nclass_62 7.669120e-02\nclass_64 1.632178e-02\nclass_65 2.614934e-02\nclass_67 6.618325e-02\nclass_88 3.137809e-02\nclass_90 3.096742e-01\nclass_92 5.297100e-02\nclass_95 2.522340e-02\nclass_99 1.800000e-01\nThanks!
\n\nHmm.. So I checked it the same way as you and the mean values in the Kernel is ok. At least for 'class 90': 0.275285. And in the train set it's about 30% samples with it.
\n\nMaybe the fractions of the classes in the train set and in the test set represent the real frequency in our Universe.
", "rawMarkdown": "Thanks!\n\nHmm.. So I checked it the same way as you and the mean values in the Kernel is ok. At least for 'class 90': 0.275285. And in the train set it's about 30% samples with it.\n\nMaybe the fractions of the classes in the train set and in the test set represent the real frequency in our Universe.\n", "votes": 1 }, { "id": 432209, "postDate": "2018-12-03T14:57:37.460Z", "content": "Thanks! I will spend too much time in thinking of class 99 if I did not see this. Is that true that you just set all class 99 to be 0.18 in your case?
", "rawMarkdown": "Thanks! I will spend too much time in thinking of class 99 if I did not see this. Is that true that you just set all class 99 to be 0.18 in your case?" }, { "id": 432619, "postDate": "2018-12-04T05:26:43.383Z", "content": "\n\n\nIs that true that you just set all class 99 to be 0.18 in your case?
\n
No. I started from Olivier's method, see his public kernel.
", "rawMarkdown": "> Is that true that you just set all class 99 to be 0.18 in your case?\n\nNo. I started from Olivier's method, see his public kernel." } ] }, { "id": 416411, "postDate": "2018-11-06T16:12:55.993Z", "content": "I fully agree. My model is not that accurate yet but I face similar issues distinguishing classes 42, 52, 62, 67, and 90. My confusion matrix looks quite similar. \n![\"confusion](\"https://storage.googleapis.com/kaggle-forum-message-attachments/416411/10624/confusion_matrix.png\")
Very similar indeed. Although mine is better in general, yours is better for some classes. There is room for improvement for both of us!
", "rawMarkdown": "Very similar indeed. Although mine is better in general, yours is better for some classes. There is room for improvement for both of us!" } ] }, { "id": 416357, "postDate": "2018-11-06T15:34:19.153Z", "content": "I saw something similar (with a much simpler model; just a straightforward RandomForest); are you using any kind of oversampling process, like SMOTE? Although SMOTE and the like help, I suspect these oversampling techniques may end misleading the model if over abused by adding noise to the training data set, thus degrading the \"signal\" of other classes.
\n\nMaybe a kind of hard negative mining may be key to improve these particular classes. Also, stacking generalization may also help.
", "rawMarkdown": "I saw something similar (with a much simpler model; just a straightforward RandomForest); are you using any kind of oversampling process, like SMOTE? Although SMOTE and the like help, I suspect these oversampling techniques may end misleading the model if over abused by adding noise to the training data set, thus degrading the \"signal\" of other classes.\n\nMaybe a kind of hard negative mining may be key to improve these particular classes. Also, stacking generalization may also help.", "votes": 2, "replies": [ { "id": 416362, "postDate": "2018-11-06T15:41:10.197Z", "content": "I am not using SMOTE. I never was able to use SMOTE successfully actually.
", "rawMarkdown": "I am not using SMOTE. I never was able to use SMOTE successfully actually." } ] }, { "id": 418761, "postDate": "2018-11-10T15:10:13.767Z", "content": "most of the object of these hard classes are being mistakenly classified as class90
", "rawMarkdown": "most of the object of these hard classes are being mistakenly classified as class90 ", "replies": [ { "id": 418767, "postDate": "2018-11-10T15:19:26.350Z", "content": "Some, not most ;)
", "rawMarkdown": "Some, not most ;)" }, { "id": 419023, "postDate": "2018-11-11T05:20:14.700Z", "content": "i mean on my single lgbm model. most of my mistakes land on class 90.
", "rawMarkdown": "i mean on my single lgbm model. most of my mistakes land on class 90." }, { "id": 419051, "postDate": "2018-11-11T06:26:39.377Z", "content": "I agree most mistakes land on class 90. It is true for my model as well.
", "rawMarkdown": "I agree most mistakes land on class 90. It is true for my model as well." }, { "id": 419533, "postDate": "2018-11-12T06:14:43.773Z", "content": "do you know if 52, 67, and 90 are types of supernovae?
", "rawMarkdown": "do you know if 52, 67, and 90 are types of supernovae?" }, { "id": 419598, "postDate": "2018-11-12T09:26:39.530Z", "content": "I am not an astronomer, but they all have some kind of brightness burst, therefore I think some of these classes are supernova. But there are other sources that can have a brightness burst AFAIK.
", "rawMarkdown": "I am not an astronomer, but they all have some kind of brightness burst, therefore I think some of these classes are supernova. But there are other sources that can have a brightness burst AFAIK." }, { "id": 419613, "postDate": "2018-11-12T09:44:00.563Z", "content": "these classes are very difficult for me. i tried manual calculation of time to decay or how much decay after 15 days(http://astronomy.swin.edu.au/cosmos/T/Type+Ia+Supernova+Light+Curves) but nothing improved my prediction. im stuck for 2 weeks now haha
", "rawMarkdown": "these classes are very difficult for me. i tried manual calculation of time to decay or how much decay after 15 days(http://astronomy.swin.edu.au/cosmos/T/Type+Ia+Supernova+Light+Curves) but nothing improved my prediction. im stuck for 2 weeks now haha" } ] }, { "id": 420584, "postDate": "2018-11-13T21:10:15.107Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 416477, "author_name": "Helgi", "author_url": "", "post_date": "2018-11-06T17:39:36.820000", "content": "I also made the same observations. The classes 42, 52, 62, 67, and 90 seem to be different types of supernovae. (This seems to be an argument for not focusing on the calculation of periods.) Most publications I have seen at actually handle those.
", "votes": 5, "replies": [ { "id": 416493, "author_name": "CPMP", "author_url": "", "post_date": "2018-11-06T18:35:27.600000", "content": "\n\n\nThis seems to be an argument for not focusing on the calculation of periods.
\n
Good point!
", "votes": 1, "replies": [] }, { "id": 416516, "author_name": "Michal Haltuf", "author_url": "", "post_date": "2018-11-06T19:43:38.323000", "content": "I believe (haven't studied in deep yet), that supernovae can be in principle differentiated using lightcurve analysis as well (well, not period, of course, but speed of brightness decay, \"half-width\" and similar parameters). It could be helpful, if we managed to link these classes with specific supernovae types (i.e. 42 = Ia, just example).
", "votes": 2, "replies": [] }, { "id": 416817, "author_name": "Mithrillion", "author_url": "", "post_date": "2018-11-07T10:13:38.437000", "content": "Period features (via LSSA) is basically curve fitting -> use fitted parameters as features. The same idea can definitely be used for other types of objects if you know the typical shape of the light curves and have a nice family of functions to approximate them. Actually if we know more about how supernovae differ, we might not need to know which label is which, as long as a fitted curve can describe such differences.
", "votes": 0, "replies": [] }, { "id": 416823, "author_name": "CPMP", "author_url": "", "post_date": "2018-11-07T10:23:54.230000", "content": "Template fitting is indeed a technique used by astronomers AFAIK. I have not given it a try, and I am not even sure where to start on this.
", "votes": 1, "replies": [] }, { "id": 417141, "author_name": "Mike Holcomb", "author_url": "", "post_date": "2018-11-07T20:23:23.407000", "content": "I am experiencing the same issue where class 90 is swallowing the other classifications of the aperiodic classes (supernova/nova?) since we have so many more training examples of that class as compared to the other \"spiky\" ones. As far as I can tell, several models can use high-level observation statistics (avg flux, max flux, std dev) to differentiate between the periodic classes and the episodic ones and discriminate well within the periodic classes. However when it comes to the episodic ones, my models give up and just say \"dump it in class 90.\"
\n\nThe last public light curve competition (SNPhotCC) from the LSST group focused exclusively on supernova classification. The paper below lists some of the key findings/techniques from that competition. As Michal and Mithrillion pointed out, previous contestants heavily relied on curve fitting and using the model parameters as features in the classifiers. On page 8, they list a specific curve that the was used to generate the samples in the prior competition.
\n\n\n\nFurthermore, these parameters were sensitive to redshift so closer SN will will have different parameters than ones of the same type that are far away.
\n\nJust like many people are using branching models on the Milky Way/Extragalatic feature, I've been trying to test for periodicity in the light curves and then have separate models for the periodic vs cataclysmic classes. One problem I have been facing is that there are some periodic training examples that also have transient spikes/weakening the strength of periodicity tests.
", "votes": 11, "replies": [] }, { "id": 417174, "author_name": "CPMP", "author_url": "", "post_date": "2018-11-07T22:14:16.433000", "content": "Thanks a lot for sharing this.
", "votes": 0, "replies": [] }, { "id": 417353, "author_name": "Sergey Zlobin", "author_url": "", "post_date": "2018-11-08T06:22:09.700000", "content": "\n\n\nI am experiencing the same issue where class 90 is swallowing the other classifications of the aperiodic classes (supernova/nova?) since we have so many more training examples of that class as compared to the other \"spiky\" ones.
\n
I used the neural network model from Kernels and it seems to be right for that model. The result is more than 2 millions (out of 3.5 millions) of 'class 90' samples. I suspect this is wrong.\n@CPMP How many samples of 'class 90' do you have in your result? I used the maximum value in a row to determine the class for a sample.
", "votes": 0, "replies": [] }, { "id": 417561, "author_name": "CPMP", "author_url": "", "post_date": "2018-11-08T13:02:59.463000", "content": "\n\n\nHow many samples of 'class 90' do you have in your result?
\n
The confusion matrix is computed on out of fold predictions for all the training data. I therefore have as many examples in class 90 as there are in training data.
", "votes": -2, "replies": [] }, { "id": 417576, "author_name": "Sergey Zlobin", "author_url": "", "post_date": "2018-11-08T13:32:23.523000", "content": "I mean the number of 'class 90' samples in the submission file. I suspect there shouldn't be so many samples with high 'class 90' probability.
", "votes": 0, "replies": [] }, { "id": 417586, "author_name": "CPMP", "author_url": "", "post_date": "2018-11-08T13:46:40.737000", "content": "Here are the mean probabilities for the submission file corresponding to that confusion matrix. I did not compute classes as you suggest, as this is not how submissions are evaluated ;)
\n\nclass_6 1.776892e-03\nclass_15 1.072527e-01\nclass_16 2.913544e-02\nclass_42 1.477831e-01\nclass_52 9.749433e-02\nclass_53 5.449143e-04\nclass_62 7.669120e-02\nclass_64 1.632178e-02\nclass_65 2.614934e-02\nclass_67 6.618325e-02\nclass_88 3.137809e-02\nclass_90 3.096742e-01\nclass_92 5.297100e-02\nclass_95 2.522340e-02\nclass_99 1.800000e-01\nThanks!
\n\nHmm.. So I checked it the same way as you and the mean values in the Kernel is ok. At least for 'class 90': 0.275285. And in the train set it's about 30% samples with it.
\n\nMaybe the fractions of the classes in the train set and in the test set represent the real frequency in our Universe.
", "votes": 1, "replies": [] }, { "id": 432209, "author_name": "Ningxiao Zhang", "author_url": "", "post_date": "2018-12-03T14:57:37.460000", "content": "Thanks! I will spend too much time in thinking of class 99 if I did not see this. Is that true that you just set all class 99 to be 0.18 in your case?
", "votes": 0, "replies": [] }, { "id": 432619, "author_name": "CPMP", "author_url": "", "post_date": "2018-12-04T05:26:43.383000", "content": "\n\n\nIs that true that you just set all class 99 to be 0.18 in your case?
\n
No. I started from Olivier's method, see his public kernel.
", "votes": 0, "replies": [] } ] }, { "id": 416411, "author_name": "Nikita Kozodoi", "author_url": "", "post_date": "2018-11-06T16:12:55.993000", "content": "I fully agree. My model is not that accurate yet but I face similar issues distinguishing classes 42, 52, 62, 67, and 90. My confusion matrix looks quite similar. \n
Very similar indeed. Although mine is better in general, yours is better for some classes. There is room for improvement for both of us!
", "votes": 0, "replies": [] } ] }, { "id": 416357, "author_name": "Andreu Sancho-Asensio", "author_url": "", "post_date": "2018-11-06T15:34:19.153000", "content": "I saw something similar (with a much simpler model; just a straightforward RandomForest); are you using any kind of oversampling process, like SMOTE? Although SMOTE and the like help, I suspect these oversampling techniques may end misleading the model if over abused by adding noise to the training data set, thus degrading the \"signal\" of other classes.
\n\nMaybe a kind of hard negative mining may be key to improve these particular classes. Also, stacking generalization may also help.
", "votes": 2, "replies": [ { "id": 416362, "author_name": "CPMP", "author_url": "", "post_date": "2018-11-06T15:41:10.197000", "content": "I am not using SMOTE. I never was able to use SMOTE successfully actually.
", "votes": 0, "replies": [] } ] }, { "id": 418761, "author_name": "dylonLL", "author_url": "", "post_date": "2018-11-10T15:10:13.767000", "content": "most of the object of these hard classes are being mistakenly classified as class90
", "votes": 0, "replies": [ { "id": 418767, "author_name": "CPMP", "author_url": "", "post_date": "2018-11-10T15:19:26.350000", "content": "Some, not most ;)
", "votes": 0, "replies": [] }, { "id": 419023, "author_name": "dylonLL", "author_url": "", "post_date": "2018-11-11T05:20:14.700000", "content": "i mean on my single lgbm model. most of my mistakes land on class 90.
", "votes": 0, "replies": [] }, { "id": 419051, "author_name": "CPMP", "author_url": "", "post_date": "2018-11-11T06:26:39.377000", "content": "I agree most mistakes land on class 90. It is true for my model as well.
", "votes": 0, "replies": [] }, { "id": 419533, "author_name": "dylonLL", "author_url": "", "post_date": "2018-11-12T06:14:43.773000", "content": "do you know if 52, 67, and 90 are types of supernovae?
", "votes": 0, "replies": [] }, { "id": 419598, "author_name": "CPMP", "author_url": "", "post_date": "2018-11-12T09:26:39.530000", "content": "I am not an astronomer, but they all have some kind of brightness burst, therefore I think some of these classes are supernova. But there are other sources that can have a brightness burst AFAIK.
", "votes": 0, "replies": [] }, { "id": 419613, "author_name": "dylonLL", "author_url": "", "post_date": "2018-11-12T09:44:00.563000", "content": "these classes are very difficult for me. i tried manual calculation of time to decay or how much decay after 15 days(http://astronomy.swin.edu.au/cosmos/T/Type+Ia+Supernova+Light+Curves) but nothing improved my prediction. im stuck for 2 weeks now haha
", "votes": 0, "replies": [] } ] }, { "id": 420584, "author_name": "", "author_url": "", "post_date": "2018-11-13T21:10:15.107000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "416235": "My best model has hard time separating classes 42, 52, 62, 67, and 90. Separating these classes is the key to winning the competition, more than class 99 IMHO. See the confusion matrix:\n\n![confusion matrix][1]\n\n\n [1]: https://storage.googleapis.com/kaggle-forum-message-attachments/416235/10623/confmatrix.png", "416477": "I also made the same observations. The classes 42, 52, 62, 67, and 90 seem to be different types of supernovae. (This seems to be an argument for not focusing on the calculation of periods.) Most publications I have seen at actually handle those.", "416411": "I fully agree. My model is not that accurate yet but I face similar issues distinguishing classes 42, 52, 62, 67, and 90. My confusion matrix looks quite similar. \n![confusion matrix][1]\n [1]: https://storage.googleapis.com/kaggle-forum-message-attachments/416411/10624/confusion_matrix.png", "416357": "I saw something similar (with a much simpler model; just a straightforward RandomForest); are you using any kind of oversampling process, like SMOTE? Although SMOTE and the like help, I suspect these oversampling techniques may end misleading the model if over abused by adding noise to the training data set, thus degrading the \"signal\" of other classes.\n\nMaybe a kind of hard negative mining may be key to improve these particular classes. Also, stacking generalization may also help.", "418761": "most of the object of these hard classes are being mistakenly classified as class90 ", "420584": "" } }