{ "id": 72613, "title": "Improved Confusion Matrix", "url": "/competitions/PLAsTiCC-2018/discussion/72613", "author_name": "", "post_date": "2018-11-25T11:51:14.739657200Z", "votes": 54, "comment_count": 43, "views": 0, "content": "

Onodera asked for our confusion matrix. Here is the one for our current best model, a lgb model that scores 0.801 on the LB. As we can see, hard classes are still hard to distinguish, especially 52 vs 90.

\n\n

\"confusion

", "messages": [ { "id": "427377", "postDate": "11/25/2018 11:51:14", "content": "

Onodera asked for our confusion matrix. Here is the one for our current best model, a lgb model that scores 0.801 on the LB. As we can see, hard classes are still hard to distinguish, especially 52 vs 90.

\n\n

\"confusion

", "rawMarkdown": "Onodera [asked][1] for our confusion matrix. Here is the one for our current best model, a lgb model that scores 0.801 on the LB. As we can see, hard classes are still hard to distinguish, especially 52 vs 90.\n\n\n![confusion matrix][2]\n\n\n [1]: https://www.kaggle.com/c/PLAsTiCC-2018/discussion/69538#427281\n [2]: https://storage.googleapis.com/kaggle-forum-message-attachments/427377/10715/cm.png", "votes": null }, { "id": "427380", "postDate": "11/25/2018 11:59:55", "content": "

Thank you for sharing! Actually doesn't look very different from the one you had at ~0.9.

", "rawMarkdown": "Thank you for sharing! Actually doesn't look very different from the one you had at ~0.9.", "votes": null }, { "id": "427384", "postDate": "11/25/2018 12:11:56", "content": "

I disagree. For instance class 52 self prediction (diagonal cell) moved from 0.36 to 0.45, class 42 moved from 0.50 to 0.58, and class 67 moved from 0.60 to 0.67 . If you don't think it is significant, the LB score thinks it is ;)

\n\n

But you are right in a way, the areas where confusion is largest are the same, mostly between classes 42, 52, 62, 67, and 90.

", "rawMarkdown": "I disagree. For instance class 52 self prediction (diagonal cell) moved from 0.36 to 0.45, class 42 moved from 0.50 to 0.58, and class 67 moved from 0.60 to 0.67 . If you don't think it is significant, the LB score thinks it is ;) \n\nBut you are right in a way, the areas where confusion is largest are the same, mostly between classes 42, 52, 62, 67, and 90.", "votes": null }, { "id": "427391", "postDate": "11/25/2018 12:29:11", "content": "

Thanks for the hint. So the improvement didn't come from the perfect predictions of easy ones but better predictions of difficult ones.

", "rawMarkdown": "Thanks for the hint. So the improvement didn't come from the perfect predictions of easy ones but better predictions of difficult ones.", "votes": null }, { "id": "427394", "postDate": "11/25/2018 12:37:18", "content": "

Seems so. I think it is the key, as I wrote previously. But progress is hard.

", "rawMarkdown": "Seems so. I think it is the key, as I wrote previously. But progress is hard.", "votes": null }, { "id": "427418", "postDate": "11/25/2018 13:29:19", "content": "

I mean it is not that you solved completely one of the hard classes, which would lead me to investigate more the specific class. You improved by about the same amount in every one of them, which is also very informative about how we could catch up. Of course I didn't mean to say it is not significant

", "rawMarkdown": "I mean it is not that you solved completely one of the hard classes, which would lead me to investigate more the specific class. You improved by about the same amount in every one of them, which is also very informative about how we could catch up. Of course I didn't mean to say it is not significant", "votes": null }, { "id": "427439", "postDate": "11/25/2018 14:06:36", "content": "

No pb, I reacted to your 'doesn't look very different'.

", "rawMarkdown": "No pb, I reacted to your 'doesn't look very different'.", "votes": null }, { "id": "427440", "postDate": "11/25/2018 14:07:20", "content": "

Wow splendid result! Especially class52! Thank you!

", "rawMarkdown": "Wow splendid result! Especially class52! Thank you!", "votes": null }, { "id": "427452", "postDate": "11/25/2018 14:42:17", "content": "

We also have problems with 52 vs 90. Do we know what are they, 90 could be nova type Ia, and what is 52, type Ib?

", "rawMarkdown": "We also have problems with 52 vs 90. Do we know what are they, 90 could be nova type Ia, and what is 52, type Ib?", "votes": null }, { "id": "427453", "postDate": "11/25/2018 14:42:42", "content": "

I have absolutely no clue.

", "rawMarkdown": "I have absolutely no clue.", "votes": null }, { "id": "427464", "postDate": "11/25/2018 15:14:34", "content": "

let's ask @kyleboone ...

\n\n

Kyle, could you tell us what do you think classes 52 and 90 are ?

", "rawMarkdown": "let's ask @kyleboone ... \n\nKyle, could you tell us what do you think classes 52 and 90 are ?", "votes": null }, { "id": "427477", "postDate": "11/25/2018 15:41:32", "content": "

How would you use that information if you had it?

", "rawMarkdown": "How would you use that information if you had it?", "votes": null }, { "id": "427519", "postDate": "11/25/2018 17:24:24", "content": "

Class 90 is Type Ia supernovae. Class 52 is probably some kind of supernova, but I'm not sure which one. Type Ib seems like it could fit. I honestly haven't really tried to identify which class is which. Supernova light curves all look pretty similar which is why this challenge exists in the first place. The features that you use to distinguish them are what you would expect: the peak brightness, the lightcurve width, etc. I'm not sure what you will learn by identifying which one is which.

", "rawMarkdown": "Class 90 is Type Ia supernovae. Class 52 is probably some kind of supernova, but I'm not sure which one. Type Ib seems like it could fit. I honestly haven't really tried to identify which class is which. Supernova light curves all look pretty similar which is why this challenge exists in the first place. The features that you use to distinguish them are what you would expect: the peak brightness, the lightcurve width, etc. I'm not sure what you will learn by identifying which one is which.", "votes": null }, { "id": "427526", "postDate": "11/25/2018 17:48:18", "content": "

@kyleboone, does it mean that surpernovae types are distinguished using more info by astronomers, like spectrogram? And we are asked to see if how far we can go without that info?

", "rawMarkdown": "kyleboone, does it mean that surpernovae types are distinguished using more info by astronomers, like spectrogram? And we are asked to see if how far we can go without that info?", "votes": null }, { "id": "427574", "postDate": "11/25/2018 18:55:56", "content": "

BTW anyone tried 'soft' confusion matrix here? I'm using the following for a probability-based confusion matrix:\n```\nsoft_pred = pd.DataFrame(oof_preds, columns=classes)\nsoft_pred['target'] = y\nsoft_conf_mat = soft_pred.groupby('target').sum()

\n\n

divide my rowsum/colsum as needed

\n\n

```\nI feel that it is a lot more informative than hard confusion matrix given that you can see how dispersed probability might be affecting your performance.

", "rawMarkdown": "BTW anyone tried 'soft' confusion matrix here? I'm using the following for a probability-based confusion matrix:\n```\nsoft_pred = pd.DataFrame(oof_preds, columns=classes)\nsoft_pred['target'] = y\nsoft_conf_mat = soft_pred.groupby('target').sum()\n# divide my rowsum/colsum as needed\n```\nI feel that it is a lot more informative than hard confusion matrix given that you can see how dispersed probability might be affecting your performance.", "votes": null }, { "id": "427586", "postDate": "11/25/2018 19:19:20", "content": "

Cool

", "rawMarkdown": "Cool", "votes": null }, { "id": "427592", "postDate": "11/25/2018 19:54:38", "content": "

The different kinds of supernovae are defined based on different features in their spectra. Traditionally, they have been classified using their spectra. However, getting spectra of every object that we find takes a lot of telescope time, and it won't be possible for future experiments like LSST. The point of this challenge is to find new ways to identify objects without spectra. The training sample represents the small fraction of LSST objects that we will have spectra for to determine their types. The test set is the rest of the objects that will be discovered that we need to classify.

", "rawMarkdown": "The different kinds of supernovae are defined based on different features in their spectra. Traditionally, they have been classified using their spectra. However, getting spectra of every object that we find takes a lot of telescope time, and it won't be possible for future experiments like LSST. The point of this challenge is to find new ways to identify objects without spectra. The training sample represents the small fraction of LSST objects that we will have spectra for to determine their types. The test set is the rest of the objects that will be discovered that we need to classify.", "votes": null }, { "id": "427637", "postDate": "11/25/2018 22:26:57", "content": "

Thank you

", "rawMarkdown": "Thank you", "votes": null }, { "id": "427700", "postDate": "11/26/2018 01:07:14", "content": "

Great!

", "rawMarkdown": "Great!", "votes": null }, { "id": "427783", "postDate": "11/26/2018 06:11:18", "content": "

Very impressive! Can I know approximately how many features there are in your lgb model?

", "rawMarkdown": "Very impressive! Can I know approximately how many features there are in your lgb model?", "votes": null }, { "id": "427850", "postDate": "11/26/2018 09:13:49", "content": "

Between 100 and 150 features ;)

", "rawMarkdown": "Between 100 and 150 features ;)", "votes": null }, { "id": "427891", "postDate": "11/26/2018 10:39:27", "content": "

Very nice, thank you!

", "rawMarkdown": "Very nice, thank you!", "votes": null }, { "id": "428090", "postDate": "11/26/2018 18:28:10", "content": "

Thanks ! nice result.

", "rawMarkdown": "Thanks ! nice result.", "votes": null }, { "id": "428193", "postDate": "11/26/2018 22:30:58", "content": "

Thanks , Master. I am learning a lot with the kernels and discussions in this competition.Is anyone using stacking techniques , training models with the predictions of base learners? My results are better with a single model (Xbg , Lgb , or NN ) than stacking them . I get CV scores of 0.63-0.64 using lgbm, xgb 0.64-0.65 and NN around 0.78. Stacking results (0.93)are worse than averaging(0.63) . why? Base learners (lgb and xgb with different parameters) correlated? Any suggestion? I am stuck at 1.08-1.1 in LB score. .Thanks in advance

", "rawMarkdown": "Thanks , Master. I am learning a lot with the kernels and discussions in this competition.Is anyone using stacking techniques , training models with the predictions of base learners? My results are better with a single model (Xbg , Lgb , or NN ) than stacking them . I get CV scores of 0.63-0.64 using lgbm, xgb 0.64-0.65 and NN around 0.78. Stacking results (0.93)are worse than averaging(0.63) . why? Base learners (lgb and xgb with different parameters) correlated? Any suggestion? I am stuck at 1.08-1.1 in LB score. .Thanks in advance", "votes": null }, { "id": "428421", "postDate": "11/27/2018 08:51:56", "content": "

We are not using stacking, yet.

", "rawMarkdown": "We are not using stacking, yet.", "votes": null }, { "id": "428451", "postDate": "11/27/2018 09:50:36", "content": "

good kernel

", "rawMarkdown": "good kernel", "votes": null }, { "id": "428672", "postDate": "11/27/2018 17:42:51", "content": "

Some improvement moves the LB score to 0.798, breaking the 0.8 bar :) The confusion matrix shows some improvement, especially for class_52.

\n\n

\"confusion

", "rawMarkdown": "Some improvement moves the LB score to 0.798, breaking the 0.8 bar :) The confusion matrix shows some improvement, especially for class_52.\n\n![confusion matrix][1]\n\n\n [1]: https://storage.googleapis.com/kaggle-forum-message-attachments/428672/10739/lb_0.798.png", "votes": null }, { "id": "428857", "postDate": "11/28/2018 01:27:39", "content": "

Thanks a lot

", "rawMarkdown": "Thanks a lot", "votes": null }, { "id": "429436", "postDate": "11/28/2018 21:05:31", "content": "

May I ask do you consider of the redshift when you stack the data?

", "rawMarkdown": "May I ask do you consider of the redshift when you stack the data?", "votes": null }, { "id": "429450", "postDate": "11/28/2018 21:41:03", "content": "

CV: 0.52 LB: 0.931\n\"enter

", "rawMarkdown": "CV: 0.52 LB: 0.931\n![enter image description here][1]\n\n\n [1]: https://storage.googleapis.com/kaggle-forum-message-attachments/429450/10745/cm1.png", "votes": null }, { "id": "429729", "postDate": "11/29/2018 09:00:20", "content": "

I concatenate the predictions obtained from each base learner and use them as features for the stacking model. I consider hostgal_photoz but not hostgal_spec in the base learners. I think the key here is not in the models but in the feature engineering.

", "rawMarkdown": "I concatenate the predictions obtained from each base learner and use them as features for the stacking model. I consider hostgal_photoz but not hostgal_spec in the base learners. I think the key here is not in the models but in the feature engineering.", "votes": null }, { "id": "429743", "postDate": "11/29/2018 09:40:24", "content": "
\n

the key here is not in the models but in the feature engineering.

\n
\n\n

That's my belief as well. Proved to be quite true until now, but rapid progress by the two other leaders may indicate that they use some model we don't use.

", "rawMarkdown": "> the key here is not in the models but in the feature engineering.\n\nThat's my belief as well. Proved to be quite true until now, but rapid progress by the two other leaders may indicate that they use some model we don't use.", "votes": null }, { "id": "429895", "postDate": "11/29/2018 14:17:46", "content": "

CV: 0.525, LB: 0.957, 60 features

\n\n

\"cm\"

", "rawMarkdown": "CV: 0.525, LB: 0.957, 60 features\n\n![cm][1]\n\n\n [1]: https://storage.googleapis.com/kaggle-forum-message-attachments/429895/10750/confusion_matrix.png", "votes": null }, { "id": "431144", "postDate": "12/01/2018 17:08:18", "content": "

My best Genetic Programs

\n\n

\"No

", "rawMarkdown": "My best Genetic Programs\n\n![No 4][1]\n\n\n\n [1]: https://storage.googleapis.com/kaggle-forum-message-attachments/431144/10765/GPIV.png", "votes": null }, { "id": "431619", "postDate": "12/02/2018 16:11:53", "content": "

How are you not at the top of this competition with this matrix? 97% for class 52! Do you have a leak somewhere? Or am I missing something?

", "rawMarkdown": "How are you not at the top of this competition with this matrix? 97% for class 52! Do you have a leak somewhere? Or am I missing something?", "votes": null }, { "id": "431621", "postDate": "12/02/2018 16:13:19", "content": "
\n

How are you not at the top of this competition with this matrix?

\n
\n\n

I was thinking the same

", "rawMarkdown": "> How are you not at the top of this competition with this matrix? \n\nI was thinking the same", "votes": null }, { "id": "431628", "postDate": "12/02/2018 16:25:55", "content": "

I think he isn't in the top because the GP can map the objects in the train set however the objects in the test have a lot of distincts values that are no present in the train set.

", "rawMarkdown": "I think he isn't in the top because the GP can map the objects in the train set however the objects in the test have a lot of distincts values that are no present in the train set.", "votes": null }, { "id": "431630", "postDate": "12/02/2018 16:33:13", "content": "

Overfitting then?

", "rawMarkdown": "Overfitting then?", "votes": null }, { "id": "431655", "postDate": "12/02/2018 17:31:05", "content": "
\n

however the objects in the test have a lot of distincts values that are no present in the train set.

\n
\n\n

This is true whatever the model used. This confusion matrix should yield to a lead of the competition if it was computed on out of fold data. I guess it is not the case.

\n\n

Edit: I see it is 'rebased and class weighted'. Not sure about what this means, and we may not be comparing apple to apple here.

", "rawMarkdown": "> however the objects in the test have a lot of distincts values that are no present in the train set.\n\nThis is true whatever the model used. This confusion matrix should yield to a lead of the competition if it was computed on out of fold data. I guess it is not the case.\n\nEdit: I see it is 'rebased and class weighted'. Not sure about what this means, and we may not be comparing apple to apple here.", "votes": null }, { "id": "432099", "postDate": "12/03/2018 12:08:35", "content": "

Thanks for sharing.

", "rawMarkdown": "Thanks for sharing.", "votes": null }, { "id": "432811", "postDate": "12/04/2018 10:42:31", "content": "

Class 42, 62 and 90 are difficult for my current model.\"Confusion

", "rawMarkdown": "Class 42, 62 and 90 are difficult for my current model.![Confusion Matrix][1]\n\n\n [1]: https://www.kaggleusercontent.com/kf/8061007/eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..36BosCL-vOysc0lq3XvDZw._t-tLyBSVE402Nn9ZDs8o8YfuwyrfR2EwUnlq11yH0hVEaRqVQI9JdeBr8MstNlrrCaezpXyb-3zFV3MLXgiCqZ4CVeMqPLgNSqCHzlu0qHvybNks5erbVapDUrMYYEksKMXdtJqFmydSkBpZR2XIOsDMoaXnHod0j4znYKgYuvXkRrKMWuWY3Exf_EtNCC6.f5MkEm0Z9wN0EBrCTWE4Iw/__results___files/__results___17_17.png", "votes": null }, { "id": "434281", "postDate": "12/06/2018 06:35:40", "content": "

@Scirpus Can you share what was the CV and LB score for the model that had the GPIV confusion matrix output.

", "rawMarkdown": "Scirpus Can you share what was the CV and LB score for the model that had the GPIV confusion matrix output.", "votes": null }, { "id": "434558", "postDate": "12/06/2018 15:34:17", "content": "

I'll give you one fold https://www.kaggle.com/scirpus/genetic-programming-confusion-matrix/code

\n\n

Kaggle Kernels isn't really up for the other 4 as it hits a limit sorry

\n\n

Just download the code rather than reading it online - I don't think it was only Ralph who wrecked the internet ;)

", "rawMarkdown": "I'll give you one fold https://www.kaggle.com/scirpus/genetic-programming-confusion-matrix/code\n\nKaggle Kernels isn't really up for the other 4 as it hits a limit sorry\n\nJust download the code rather than reading it online - I don't think it was only Ralph who wrecked the internet ;)", "votes": null }, { "id": "435544", "postDate": "12/08/2018 07:38:48", "content": "

CV : 0.440, LB : 0.905, 100 features, single lgb model. It's so hard to distinguish class 52.

", "rawMarkdown": "CV : 0.440, LB : 0.905, 100 features, single lgb model. It's so hard to distinguish class 52.", "votes": null } ], "comments": [ { "id": 427380, "author_name": "iprapas", "author_url": "", "post_date": "11/25/2018 11:59:55", "content": "

Thank you for sharing! Actually doesn't look very different from the one you had at ~0.9.

", "votes": null, "replies": [ { "id": 427384, "author_name": "cpmpml", "author_url": "", "post_date": "11/25/2018 12:11:56", "content": "

I disagree. For instance class 52 self prediction (diagonal cell) moved from 0.36 to 0.45, class 42 moved from 0.50 to 0.58, and class 67 moved from 0.60 to 0.67 . If you don't think it is significant, the LB score thinks it is ;)

\n\n

But you are right in a way, the areas where confusion is largest are the same, mostly between classes 42, 52, 62, 67, and 90.

", "votes": null, "replies": [] }, { "id": 427418, "author_name": "iprapas", "author_url": "", "post_date": "11/25/2018 13:29:19", "content": "

I mean it is not that you solved completely one of the hard classes, which would lead me to investigate more the specific class. You improved by about the same amount in every one of them, which is also very informative about how we could catch up. Of course I didn't mean to say it is not significant

", "votes": null, "replies": [] }, { "id": 427439, "author_name": "cpmpml", "author_url": "", "post_date": "11/25/2018 14:06:36", "content": "

No pb, I reacted to your 'doesn't look very different'.

", "votes": null, "replies": [] } ] }, { "id": 427391, "author_name": "aerdem4", "author_url": "", "post_date": "11/25/2018 12:29:11", "content": "

Thanks for the hint. So the improvement didn't come from the perfect predictions of easy ones but better predictions of difficult ones.

", "votes": null, "replies": [ { "id": 427394, "author_name": "cpmpml", "author_url": "", "post_date": "11/25/2018 12:37:18", "content": "

Seems so. I think it is the key, as I wrote previously. But progress is hard.

", "votes": null, "replies": [] } ] }, { "id": 427440, "author_name": "onodera", "author_url": "", "post_date": "11/25/2018 14:07:20", "content": "

Wow splendid result! Especially class52! Thank you!

", "votes": null, "replies": [] }, { "id": 427452, "author_name": "blondinka", "author_url": "", "post_date": "11/25/2018 14:42:17", "content": "

We also have problems with 52 vs 90. Do we know what are they, 90 could be nova type Ia, and what is 52, type Ib?

", "votes": null, "replies": [ { "id": 427453, "author_name": "cpmpml", "author_url": "", "post_date": "11/25/2018 14:42:42", "content": "

I have absolutely no clue.

", "votes": null, "replies": [] }, { "id": 427464, "author_name": "blondinka", "author_url": "", "post_date": "11/25/2018 15:14:34", "content": "

let's ask @kyleboone ...

\n\n

Kyle, could you tell us what do you think classes 52 and 90 are ?

", "votes": null, "replies": [] }, { "id": 427477, "author_name": "cpmpml", "author_url": "", "post_date": "11/25/2018 15:41:32", "content": "

How would you use that information if you had it?

", "votes": null, "replies": [] }, { "id": 427519, "author_name": "kyleboone", "author_url": "", "post_date": "11/25/2018 17:24:24", "content": "

Class 90 is Type Ia supernovae. Class 52 is probably some kind of supernova, but I'm not sure which one. Type Ib seems like it could fit. I honestly haven't really tried to identify which class is which. Supernova light curves all look pretty similar which is why this challenge exists in the first place. The features that you use to distinguish them are what you would expect: the peak brightness, the lightcurve width, etc. I'm not sure what you will learn by identifying which one is which.

", "votes": null, "replies": [] }, { "id": 427526, "author_name": "cpmpml", "author_url": "", "post_date": "11/25/2018 17:48:18", "content": "

@kyleboone, does it mean that surpernovae types are distinguished using more info by astronomers, like spectrogram? And we are asked to see if how far we can go without that info?

", "votes": null, "replies": [] }, { "id": 427592, "author_name": "kyleboone", "author_url": "", "post_date": "11/25/2018 19:54:38", "content": "

The different kinds of supernovae are defined based on different features in their spectra. Traditionally, they have been classified using their spectra. However, getting spectra of every object that we find takes a lot of telescope time, and it won't be possible for future experiments like LSST. The point of this challenge is to find new ways to identify objects without spectra. The training sample represents the small fraction of LSST objects that we will have spectra for to determine their types. The test set is the rest of the objects that will be discovered that we need to classify.

", "votes": null, "replies": [] }, { "id": 427637, "author_name": "blondinka", "author_url": "", "post_date": "11/25/2018 22:26:57", "content": "

Thank you

", "votes": null, "replies": [] } ] }, { "id": 427574, "author_name": "mithrillion", "author_url": "", "post_date": "11/25/2018 18:55:56", "content": "

BTW anyone tried 'soft' confusion matrix here? I'm using the following for a probability-based confusion matrix:\n```\nsoft_pred = pd.DataFrame(oof_preds, columns=classes)\nsoft_pred['target'] = y\nsoft_conf_mat = soft_pred.groupby('target').sum()

\n\n

divide my rowsum/colsum as needed

\n\n

```\nI feel that it is a lot more informative than hard confusion matrix given that you can see how dispersed probability might be affecting your performance.

", "votes": null, "replies": [] }, { "id": 427586, "author_name": "hl3101", "author_url": "", "post_date": "11/25/2018 19:19:20", "content": "

Cool

", "votes": null, "replies": [] }, { "id": 427700, "author_name": "bioinformatics", "author_url": "", "post_date": "11/26/2018 01:07:14", "content": "

Great!

", "votes": null, "replies": [] }, { "id": 427783, "author_name": "lfcpeng17", "author_url": "", "post_date": "11/26/2018 06:11:18", "content": "

Very impressive! Can I know approximately how many features there are in your lgb model?

", "votes": null, "replies": [ { "id": 427850, "author_name": "cpmpml", "author_url": "", "post_date": "11/26/2018 09:13:49", "content": "

Between 100 and 150 features ;)

", "votes": null, "replies": [] } ] }, { "id": 427891, "author_name": "xboard", "author_url": "", "post_date": "11/26/2018 10:39:27", "content": "

Very nice, thank you!

", "votes": null, "replies": [] }, { "id": 428090, "author_name": "luciusfox", "author_url": "", "post_date": "11/26/2018 18:28:10", "content": "

Thanks ! nice result.

", "votes": null, "replies": [] }, { "id": 428193, "author_name": "joxemi", "author_url": "", "post_date": "11/26/2018 22:30:58", "content": "

Thanks , Master. I am learning a lot with the kernels and discussions in this competition.Is anyone using stacking techniques , training models with the predictions of base learners? My results are better with a single model (Xbg , Lgb , or NN ) than stacking them . I get CV scores of 0.63-0.64 using lgbm, xgb 0.64-0.65 and NN around 0.78. Stacking results (0.93)are worse than averaging(0.63) . why? Base learners (lgb and xgb with different parameters) correlated? Any suggestion? I am stuck at 1.08-1.1 in LB score. .Thanks in advance

", "votes": null, "replies": [ { "id": 428421, "author_name": "cpmpml", "author_url": "", "post_date": "11/27/2018 08:51:56", "content": "

We are not using stacking, yet.

", "votes": null, "replies": [] }, { "id": 429436, "author_name": "vinzzz", "author_url": "", "post_date": "11/28/2018 21:05:31", "content": "

May I ask do you consider of the redshift when you stack the data?

", "votes": null, "replies": [] }, { "id": 429729, "author_name": "joxemi", "author_url": "", "post_date": "11/29/2018 09:00:20", "content": "

I concatenate the predictions obtained from each base learner and use them as features for the stacking model. I consider hostgal_photoz but not hostgal_spec in the base learners. I think the key here is not in the models but in the feature engineering.

", "votes": null, "replies": [] }, { "id": 429743, "author_name": "cpmpml", "author_url": "", "post_date": "11/29/2018 09:40:24", "content": "
\n

the key here is not in the models but in the feature engineering.

\n
\n\n

That's my belief as well. Proved to be quite true until now, but rapid progress by the two other leaders may indicate that they use some model we don't use.

", "votes": null, "replies": [] } ] }, { "id": 428451, "author_name": "bhavan123", "author_url": "", "post_date": "11/27/2018 09:50:36", "content": "

good kernel

", "votes": null, "replies": [] }, { "id": 428672, "author_name": "cpmpml", "author_url": "", "post_date": "11/27/2018 17:42:51", "content": "

Some improvement moves the LB score to 0.798, breaking the 0.8 bar :) The confusion matrix shows some improvement, especially for class_52.

\n\n

\"confusion

", "votes": null, "replies": [] }, { "id": 428857, "author_name": "aaronlau", "author_url": "", "post_date": "11/28/2018 01:27:39", "content": "

Thanks a lot

", "votes": null, "replies": [] }, { "id": 429450, "author_name": "titericz", "author_url": "", "post_date": "11/28/2018 21:41:03", "content": "

CV: 0.52 LB: 0.931\n\"enter

", "votes": null, "replies": [] }, { "id": 429895, "author_name": "iprapas", "author_url": "", "post_date": "11/29/2018 14:17:46", "content": "

CV: 0.525, LB: 0.957, 60 features

\n\n

\"cm\"

", "votes": null, "replies": [] }, { "id": 431144, "author_name": "scirpus", "author_url": "", "post_date": "12/01/2018 17:08:18", "content": "

My best Genetic Programs

\n\n

\"No

", "votes": null, "replies": [ { "id": 431619, "author_name": "sdoria", "author_url": "", "post_date": "12/02/2018 16:11:53", "content": "

How are you not at the top of this competition with this matrix? 97% for class 52! Do you have a leak somewhere? Or am I missing something?

", "votes": null, "replies": [] }, { "id": 431621, "author_name": "cpmpml", "author_url": "", "post_date": "12/02/2018 16:13:19", "content": "
\n

How are you not at the top of this competition with this matrix?

\n
\n\n

I was thinking the same

", "votes": null, "replies": [] }, { "id": 431628, "author_name": "joaopmpeinado", "author_url": "", "post_date": "12/02/2018 16:25:55", "content": "

I think he isn't in the top because the GP can map the objects in the train set however the objects in the test have a lot of distincts values that are no present in the train set.

", "votes": null, "replies": [] }, { "id": 431630, "author_name": "sdoria", "author_url": "", "post_date": "12/02/2018 16:33:13", "content": "

Overfitting then?

", "votes": null, "replies": [] }, { "id": 431655, "author_name": "cpmpml", "author_url": "", "post_date": "12/02/2018 17:31:05", "content": "
\n

however the objects in the test have a lot of distincts values that are no present in the train set.

\n
\n\n

This is true whatever the model used. This confusion matrix should yield to a lead of the competition if it was computed on out of fold data. I guess it is not the case.

\n\n

Edit: I see it is 'rebased and class weighted'. Not sure about what this means, and we may not be comparing apple to apple here.

", "votes": null, "replies": [] }, { "id": 434281, "author_name": "azacharia", "author_url": "", "post_date": "12/06/2018 06:35:40", "content": "

@Scirpus Can you share what was the CV and LB score for the model that had the GPIV confusion matrix output.

", "votes": null, "replies": [] }, { "id": 434558, "author_name": "scirpus", "author_url": "", "post_date": "12/06/2018 15:34:17", "content": "

I'll give you one fold https://www.kaggle.com/scirpus/genetic-programming-confusion-matrix/code

\n\n

Kaggle Kernels isn't really up for the other 4 as it hits a limit sorry

\n\n

Just download the code rather than reading it online - I don't think it was only Ralph who wrecked the internet ;)

", "votes": null, "replies": [] } ] }, { "id": 432099, "author_name": "raghupalem", "author_url": "", "post_date": "12/03/2018 12:08:35", "content": "

Thanks for sharing.

", "votes": null, "replies": [] }, { "id": 432811, "author_name": "subrahmanyamv", "author_url": "", "post_date": "12/04/2018 10:42:31", "content": "

Class 42, 62 and 90 are difficult for my current model.\"Confusion

", "votes": null, "replies": [] }, { "id": 435544, "author_name": "hmdhmd", "author_url": "", "post_date": "12/08/2018 07:38:48", "content": "

CV : 0.440, LB : 0.905, 100 features, single lgb model. It's so hard to distinguish class 52.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "427377": "Onodera [asked][1] for our confusion matrix. Here is the one for our current best model, a lgb model that scores 0.801 on the LB. As we can see, hard classes are still hard to distinguish, especially 52 vs 90.\n\n\n![confusion matrix][2]\n\n\n [1]: https://www.kaggle.com/c/PLAsTiCC-2018/discussion/69538#427281\n [2]: https://storage.googleapis.com/kaggle-forum-message-attachments/427377/10715/cm.png", "427380": "Thank you for sharing! Actually doesn't look very different from the one you had at ~0.9.", "427384": "I disagree. For instance class 52 self prediction (diagonal cell) moved from 0.36 to 0.45, class 42 moved from 0.50 to 0.58, and class 67 moved from 0.60 to 0.67 . If you don't think it is significant, the LB score thinks it is ;) \n\nBut you are right in a way, the areas where confusion is largest are the same, mostly between classes 42, 52, 62, 67, and 90.", "427391": "Thanks for the hint. So the improvement didn't come from the perfect predictions of easy ones but better predictions of difficult ones.", "427394": "Seems so. I think it is the key, as I wrote previously. But progress is hard.", "427418": "I mean it is not that you solved completely one of the hard classes, which would lead me to investigate more the specific class. You improved by about the same amount in every one of them, which is also very informative about how we could catch up. Of course I didn't mean to say it is not significant", "427439": "No pb, I reacted to your 'doesn't look very different'.", "427440": "Wow splendid result! Especially class52! Thank you!", "427452": "We also have problems with 52 vs 90. Do we know what are they, 90 could be nova type Ia, and what is 52, type Ib?", "427453": "I have absolutely no clue.", "427464": "let's ask @kyleboone ... \n\nKyle, could you tell us what do you think classes 52 and 90 are ?", "427477": "How would you use that information if you had it?", "427519": "Class 90 is Type Ia supernovae. Class 52 is probably some kind of supernova, but I'm not sure which one. Type Ib seems like it could fit. I honestly haven't really tried to identify which class is which. Supernova light curves all look pretty similar which is why this challenge exists in the first place. The features that you use to distinguish them are what you would expect: the peak brightness, the lightcurve width, etc. I'm not sure what you will learn by identifying which one is which.", "427526": "kyleboone, does it mean that surpernovae types are distinguished using more info by astronomers, like spectrogram? And we are asked to see if how far we can go without that info?", "427574": "BTW anyone tried 'soft' confusion matrix here? I'm using the following for a probability-based confusion matrix:\n```\nsoft_pred = pd.DataFrame(oof_preds, columns=classes)\nsoft_pred['target'] = y\nsoft_conf_mat = soft_pred.groupby('target').sum()\n# divide my rowsum/colsum as needed\n```\nI feel that it is a lot more informative than hard confusion matrix given that you can see how dispersed probability might be affecting your performance.", "427586": "Cool", "427592": "The different kinds of supernovae are defined based on different features in their spectra. Traditionally, they have been classified using their spectra. However, getting spectra of every object that we find takes a lot of telescope time, and it won't be possible for future experiments like LSST. The point of this challenge is to find new ways to identify objects without spectra. The training sample represents the small fraction of LSST objects that we will have spectra for to determine their types. The test set is the rest of the objects that will be discovered that we need to classify.", "427637": "Thank you", "427700": "Great!", "427783": "Very impressive! Can I know approximately how many features there are in your lgb model?", "427850": "Between 100 and 150 features ;)", "427891": "Very nice, thank you!", "428090": "Thanks ! nice result.", "428193": "Thanks , Master. I am learning a lot with the kernels and discussions in this competition.Is anyone using stacking techniques , training models with the predictions of base learners? My results are better with a single model (Xbg , Lgb , or NN ) than stacking them . I get CV scores of 0.63-0.64 using lgbm, xgb 0.64-0.65 and NN around 0.78. Stacking results (0.93)are worse than averaging(0.63) . why? Base learners (lgb and xgb with different parameters) correlated? Any suggestion? I am stuck at 1.08-1.1 in LB score. .Thanks in advance", "428421": "We are not using stacking, yet.", "428451": "good kernel", "428672": "Some improvement moves the LB score to 0.798, breaking the 0.8 bar :) The confusion matrix shows some improvement, especially for class_52.\n\n![confusion matrix][1]\n\n\n [1]: https://storage.googleapis.com/kaggle-forum-message-attachments/428672/10739/lb_0.798.png", "428857": "Thanks a lot", "429436": "May I ask do you consider of the redshift when you stack the data?", "429450": "CV: 0.52 LB: 0.931\n![enter image description here][1]\n\n\n [1]: https://storage.googleapis.com/kaggle-forum-message-attachments/429450/10745/cm1.png", "429729": "I concatenate the predictions obtained from each base learner and use them as features for the stacking model. I consider hostgal_photoz but not hostgal_spec in the base learners. I think the key here is not in the models but in the feature engineering.", "429743": "> the key here is not in the models but in the feature engineering.\n\nThat's my belief as well. Proved to be quite true until now, but rapid progress by the two other leaders may indicate that they use some model we don't use.", "429895": "CV: 0.525, LB: 0.957, 60 features\n\n![cm][1]\n\n\n [1]: https://storage.googleapis.com/kaggle-forum-message-attachments/429895/10750/confusion_matrix.png", "431144": "My best Genetic Programs\n\n![No 4][1]\n\n\n\n [1]: https://storage.googleapis.com/kaggle-forum-message-attachments/431144/10765/GPIV.png", "431619": "How are you not at the top of this competition with this matrix? 97% for class 52! Do you have a leak somewhere? Or am I missing something?", "431621": "> How are you not at the top of this competition with this matrix? \n\nI was thinking the same", "431628": "I think he isn't in the top because the GP can map the objects in the train set however the objects in the test have a lot of distincts values that are no present in the train set.", "431630": "Overfitting then?", "431655": "> however the objects in the test have a lot of distincts values that are no present in the train set.\n\nThis is true whatever the model used. This confusion matrix should yield to a lead of the competition if it was computed on out of fold data. I guess it is not the case.\n\nEdit: I see it is 'rebased and class weighted'. Not sure about what this means, and we may not be comparing apple to apple here.", "432099": "Thanks for sharing.", "432811": "Class 42, 62 and 90 are difficult for my current model.![Confusion Matrix][1]\n\n\n [1]: https://www.kaggleusercontent.com/kf/8061007/eyJhbGciOiJkaXIiLCJlbmMiOiJBMTI4Q0JDLUhTMjU2In0..36BosCL-vOysc0lq3XvDZw._t-tLyBSVE402Nn9ZDs8o8YfuwyrfR2EwUnlq11yH0hVEaRqVQI9JdeBr8MstNlrrCaezpXyb-3zFV3MLXgiCqZ4CVeMqPLgNSqCHzlu0qHvybNks5erbVapDUrMYYEksKMXdtJqFmydSkBpZR2XIOsDMoaXnHod0j4znYKgYuvXkRrKMWuWY3Exf_EtNCC6.f5MkEm0Z9wN0EBrCTWE4Iw/__results___files/__results___17_17.png", "434281": "Scirpus Can you share what was the CV and LB score for the model that had the GPIV confusion matrix output.", "434558": "I'll give you one fold https://www.kaggle.com/scirpus/genetic-programming-confusion-matrix/code\n\nKaggle Kernels isn't really up for the other 4 as it hits a limit sorry\n\nJust download the code rather than reading it online - I don't think it was only Ralph who wrecked the internet ;)", "435544": "CV : 0.440, LB : 0.905, 100 features, single lgb model. It's so hard to distinguish class 52." }, "source": "meta" }