{ "id": 74564, "title": "New Confusion matrix", "url": "/competitions/PLAsTiCC-2018/discussion/74564", "author_name": "", "post_date": "2018-12-13T13:44:55.328950900Z", "votes": 18, "comment_count": 36, "views": 0, "content": "

I shared the confusion matrix for a 0.801 single lgb model few weeks ago. We are now approaching the 0.75 bar with a single model (0.752 to be precise). Here is its confusion matrix. We can see that hard classes are a bit easier. For this model CV is 0.348 and gap is 0.404. I doubt it will be enough to catch up with leaders, but we'll see when we ensemble.

\n\n

\"confusion

", "messages": [ { "id": "438332", "postDate": "12/13/2018 13:44:55", "content": "

I shared the confusion matrix for a 0.801 single lgb model few weeks ago. We are now approaching the 0.75 bar with a single model (0.752 to be precise). Here is its confusion matrix. We can see that hard classes are a bit easier. For this model CV is 0.348 and gap is 0.404. I doubt it will be enough to catch up with leaders, but we'll see when we ensemble.

\n\n

\"confusion

", "rawMarkdown": "I shared [the confusion matrix for a 0.801 single lgb model][1] few weeks ago. We are now approaching the 0.75 bar with a single model (0.752 to be precise). Here is its confusion matrix. We can see that hard classes are a bit easier. For this model CV is 0.348 and gap is 0.404. I doubt it will be enough to catch up with leaders, but we'll see when we ensemble.\n\n![confusion matrix][2]\n\n\n [1]: https://www.kaggle.com/c/PLAsTiCC-2018/discussion/72613\n [2]: https://storage.googleapis.com/kaggle-forum-message-attachments/438332/10871/lgb_752.png", "votes": null }, { "id": "438338", "postDate": "12/13/2018 13:53:10", "content": "

Please don't improve anymore, I can't catch:)

", "rawMarkdown": "Please don't improve anymore, I can't catch:)", "votes": null }, { "id": "438341", "postDate": "12/13/2018 14:05:17", "content": "

Still suffered from the agony of big gap between LB and CV, can't figure out the magic behind it.

", "rawMarkdown": "Still suffered from the agony of big gap between LB and CV, can't figure out the magic behind it.", "votes": null }, { "id": "438342", "postDate": "12/13/2018 14:05:52", "content": "

Please don't improve anymore, or you catch us:)

", "rawMarkdown": "Please don't improve anymore, or you catch us:)", "votes": null }, { "id": "438352", "postDate": "12/13/2018 14:28:34", "content": "

Yes. We got CV 0.39X, but LB...QQ

", "rawMarkdown": "Yes. We got CV 0.39X, but LB...QQ", "votes": null }, { "id": "438356", "postDate": "12/13/2018 14:32:54", "content": "

Your solo performance is impressive, but we'll try to keep you at bay ;)

", "rawMarkdown": "Your solo performance is impressive, but we'll try to keep you at bay ;)", "votes": null }, { "id": "438358", "postDate": "12/13/2018 14:33:37", "content": "

I'm sorry but we'll try to disappoint you ;)

", "rawMarkdown": "I'm sorry but we'll try to disappoint you ;)", "votes": null }, { "id": "438363", "postDate": "12/13/2018 14:37:18", "content": "

It will be good to see other leaders confusion matrix. If they are not distributed the same way then further improvements will be possible after competition end.

", "rawMarkdown": "It will be good to see other leaders confusion matrix. If they are not distributed the same way then further improvements will be possible after competition end.", "votes": null }, { "id": "438377", "postDate": "12/13/2018 15:01:43", "content": "

Thanks, and good luck:)

", "rawMarkdown": "Thanks, and good luck:)", "votes": null }, { "id": "438383", "postDate": "12/13/2018 15:13:32", "content": "

No Thanks, but good luck:)

", "rawMarkdown": "No Thanks, but good luck:)", "votes": null }, { "id": "438401", "postDate": "12/13/2018 15:44:09", "content": "

Is it possible that CV is better but LB becomes worse? Thanks!

", "rawMarkdown": "Is it possible that CV is better but LB becomes worse? Thanks!", "votes": null }, { "id": "438411", "postDate": "12/13/2018 16:04:47", "content": "

Yes. This is called overfitting. And it happens easily here.

", "rawMarkdown": "Yes. This is called overfitting. And it happens easily here.", "votes": null }, { "id": "438415", "postDate": "12/13/2018 16:15:05", "content": "

My matrix is much worse (: Do you use lightGBM ? How many features do you use?

", "rawMarkdown": "My matrix is much worse (: Do you use lightGBM ? How many features do you use?", "votes": null }, { "id": "438419", "postDate": "12/13/2018 16:17:50", "content": "

Haha, you are totally right!

", "rawMarkdown": "Haha, you are totally right!", "votes": null }, { "id": "438420", "postDate": "12/13/2018 16:21:07", "content": "

So CV can be trusted?

", "rawMarkdown": "So CV can be trusted?", "votes": null }, { "id": "438421", "postDate": "12/13/2018 16:21:51", "content": "

That is really impressive score for a single model :) Don't you mind sharing the number of features that your team is using to achieve that result?

\n\n

Good luck guys!

", "rawMarkdown": "That is really impressive score for a single model :) Don't you mind sharing the number of features that your team is using to achieve that result?\n\nGood luck guys!", "votes": null }, { "id": "438428", "postDate": "12/13/2018 16:40:22", "content": "

Less than 200 features, but some are pretty heavy to compute.

", "rawMarkdown": "Less than 200 features, but some are pretty heavy to compute.", "votes": null }, { "id": "438431", "postDate": "12/13/2018 16:41:29", "content": "

The opposite: CV is not very reliable here. This is because train and test data distributions are very different.

", "rawMarkdown": "The opposite: CV is not very reliable here. This is because train and test data distributions are very different.", "votes": null }, { "id": "438432", "postDate": "12/13/2018 16:42:18", "content": "

This model is a lightgbm model, but we use other models too.

", "rawMarkdown": "This model is a lightgbm model, but we use other models too.", "votes": null }, { "id": "438437", "postDate": "12/13/2018 16:53:32", "content": "

Thanks! Let us have a talk after the competition ends :)

", "rawMarkdown": "Thanks! Let us have a talk after the competition ends :)", "votes": null }, { "id": "438464", "postDate": "12/13/2018 18:08:45", "content": "

Well, at least you showed me that class 52 does not have to be the sacrificial lamb for the greater good!

", "rawMarkdown": "Well, at least you showed me that class 52 does not have to be the sacrificial lamb for the greater good!", "votes": null }, { "id": "438576", "postDate": "12/13/2018 22:45:28", "content": "

Does it mean that you can’t judge whether a feature/features describe test set well and the score gets better until you submit a prediction and see the LB score?

", "rawMarkdown": "Does it mean that you can’t judge whether a feature/features describe test set well and the score gets better until you submit a prediction and see the LB score?", "votes": null }, { "id": "438587", "postDate": "12/13/2018 23:05:56", "content": "

I think we all face the same question.

", "rawMarkdown": "I think we all face the same question.", "votes": null }, { "id": "438639", "postDate": "12/14/2018 01:14:51", "content": "

Seeing your confusion matrix has been very helpful to me, so in the spirit of fairness I have attached my best one:

\n\n

\"Confusion

\n\n

Overall they are fairly similar, but there are some noticeable differences between the two! This single model has a CV of 0.405 and a LB of 0.730.

", "rawMarkdown": "Seeing your confusion matrix has been very helpful to me, so in the spirit of fairness I have attached my best one:\n\n![Confusion matrix][1]\n\nOverall they are fairly similar, but there are some noticeable differences between the two! This single model has a CV of 0.405 and a LB of 0.730.\n\n\n [1]: https://storage.googleapis.com/kaggle-forum-message-attachments/438639/10876/confusion_matrix.png", "votes": null }, { "id": "438679", "postDate": "12/14/2018 02:41:22", "content": "

@kyleboone, how many features do you have?

", "rawMarkdown": "kyleboone, how many features do you have?", "votes": null }, { "id": "438682", "postDate": "12/14/2018 02:52:32", "content": "

Around 200, but I'm doing things a bit differently than everyone else ;)

", "rawMarkdown": "Around 200, but I'm doing things a bit differently than everyone else ;)", "votes": null }, { "id": "438694", "postDate": "12/14/2018 03:17:51", "content": "

alright goodluck! im stuck for quite some time now and still trying to find useful info to improve my model up to the last minute

", "rawMarkdown": "alright goodluck! im stuck for quite some time now and still trying to find useful info to improve my model up to the last minute", "votes": null }, { "id": "438699", "postDate": "12/14/2018 03:30:53", "content": "

Hmm... Sounds like LB probing...\nAnyways, I believe you have been following your strategy to climbing up the LB. I'm looking forward to seeing your strategy after the competition ends.

", "rawMarkdown": "Hmm... Sounds like LB probing...\nAnyways, I believe you have been following your strategy to climbing up the LB. I'm looking forward to seeing your strategy after the competition ends.", "votes": null }, { "id": "438727", "postDate": "12/14/2018 04:48:49", "content": "

Kyle,

\n\n
\n

Seeing your confusion matrix has been very helpful to me,

\n
\n\n

I'm curious about why it was useful. I try to share as much as I can without helping competition too directly, and I did not think this would help anyone really...

\n\n
\n

in the spirit of fairness I have attached my best one

\n
\n\n

fair play is essential, thanks for sharing as well.

\n\n
\n

I'm doing things a bit differently than everyone else ;)

\n
\n\n

That's what I' was afraid of, you probably have a different angle than us. For one, you don't say what you use for modeling. ;)

\n\n

Well, only few days left before we know...

\n\n

Looking at your matrix, it seems our model categorical accuracy (sum of diagonal) is better than yours, but LB is lower. This is very interesting.

", "rawMarkdown": "Kyle,\n\n> Seeing your confusion matrix has been very helpful to me,\n\nI'm curious about why it was useful. I try to share as much as I can without helping competition too directly, and I did not think this would help anyone really...\n\n> in the spirit of fairness I have attached my best one\n\nfair play is essential, thanks for sharing as well.\n\n> I'm doing things a bit differently than everyone else ;)\n\nThat's what I' was afraid of, you probably have a different angle than us. For one, you don't say what you use for modeling. ;)\n\nWell, only few days left before we know...\n\nLooking at your matrix, it seems our model categorical accuracy (sum of diagonal) is better than yours, but LB is lower. This is very interesting.", "votes": null }, { "id": "438745", "postDate": "12/14/2018 05:50:50", "content": "

Well, it's probably not that different. I don't really know what you guys are doing either. You will probably find some interesting things if you look where our confusion matrices differ.

", "rawMarkdown": "Well, it's probably not that different. I don't really know what you guys are doing either. You will probably find some interesting things if you look where our confusion matrices differ.", "votes": null }, { "id": "439096", "postDate": "12/14/2018 18:22:59", "content": "

pretty heavy to compute - does that translate to having to step through each light curve? I had some really cool methods to extract some pretty useful features on the training set but I had to abandon them because I didn't see how I could pull it off on the test set. I need to re-read your posts on writing efficient code.

", "rawMarkdown": "pretty heavy to compute - does that translate to having to step through each light curve? I had some really cool methods to extract some pretty useful features on the training set but I had to abandon them because I didn't see how I could pull it off on the test set. I need to re-read your posts on writing efficient code.", "votes": null }, { "id": "439115", "postDate": "12/14/2018 18:59:09", "content": "

Sure, some computation require to process each lightcurve. For test data we have to run it in parallel. When my team mate said it takes 2 weeks overall I think he is close to the truth.

", "rawMarkdown": "Sure, some computation require to process each lightcurve. For test data we have to run it in parallel. When my team mate said it takes 2 weeks overall I think he is close to the truth.", "votes": null }, { "id": "439199", "postDate": "12/14/2018 22:10:06", "content": "

Very Impressive... the difference between CV and LB is only 0.325.

", "rawMarkdown": "Very Impressive... the difference between CV and LB is only 0.325.", "votes": null }, { "id": "439283", "postDate": "12/15/2018 04:53:12", "content": "

I believe this small gap has to do with the special treatment mentioned by Kyle. Now the question is if the LB is randomly split and any special processing can be generalized to the private.

", "rawMarkdown": "I believe this small gap has to do with the special treatment mentioned by Kyle. Now the question is if the LB is randomly split and any special processing can be generalized to the private.", "votes": null }, { "id": "439286", "postDate": "12/15/2018 04:56:22", "content": "

In hindsight I wish I’d had the patience. Using only kernels I would’ve had to run about 600 hours but I could’ve split that 20 ways and merged the results.

", "rawMarkdown": "In hindsight I wish I’d had the patience. Using only kernels I would’ve had to run about 600 hours but I could’ve split that 20 ways and merged the results.", "votes": null }, { "id": "439668", "postDate": "12/16/2018 03:06:34", "content": "

Are you required to use only one single model? is it in the rules?

", "rawMarkdown": "Are you required to use only one single model? is it in the rules?", "votes": null }, { "id": "439702", "postDate": "12/16/2018 05:44:13", "content": "

No, you can use whatever you see fit.

", "rawMarkdown": "No, you can use whatever you see fit.", "votes": null } ], "comments": [ { "id": 438338, "author_name": "aerdem4", "author_url": "", "post_date": "12/13/2018 13:53:10", "content": "

Please don't improve anymore, I can't catch:)

", "votes": null, "replies": [ { "id": 438356, "author_name": "cpmpml", "author_url": "", "post_date": "12/13/2018 14:32:54", "content": "

Your solo performance is impressive, but we'll try to keep you at bay ;)

", "votes": null, "replies": [] }, { "id": 438377, "author_name": "aerdem4", "author_url": "", "post_date": "12/13/2018 15:01:43", "content": "

Thanks, and good luck:)

", "votes": null, "replies": [] } ] }, { "id": 438341, "author_name": "andrew60909", "author_url": "", "post_date": "12/13/2018 14:05:17", "content": "

Still suffered from the agony of big gap between LB and CV, can't figure out the magic behind it.

", "votes": null, "replies": [ { "id": 438352, "author_name": "takuok", "author_url": "", "post_date": "12/13/2018 14:28:34", "content": "

Yes. We got CV 0.39X, but LB...QQ

", "votes": null, "replies": [] } ] }, { "id": 438342, "author_name": "mamasinkgs", "author_url": "", "post_date": "12/13/2018 14:05:52", "content": "

Please don't improve anymore, or you catch us:)

", "votes": null, "replies": [ { "id": 438358, "author_name": "cpmpml", "author_url": "", "post_date": "12/13/2018 14:33:37", "content": "

I'm sorry but we'll try to disappoint you ;)

", "votes": null, "replies": [] }, { "id": 438383, "author_name": "mamasinkgs", "author_url": "", "post_date": "12/13/2018 15:13:32", "content": "

No Thanks, but good luck:)

", "votes": null, "replies": [] } ] }, { "id": 438363, "author_name": "cpmpml", "author_url": "", "post_date": "12/13/2018 14:37:18", "content": "

It will be good to see other leaders confusion matrix. If they are not distributed the same way then further improvements will be possible after competition end.

", "votes": null, "replies": [ { "id": 438415, "author_name": "suvalex", "author_url": "", "post_date": "12/13/2018 16:15:05", "content": "

My matrix is much worse (: Do you use lightGBM ? How many features do you use?

", "votes": null, "replies": [] }, { "id": 438432, "author_name": "cpmpml", "author_url": "", "post_date": "12/13/2018 16:42:18", "content": "

This model is a lightgbm model, but we use other models too.

", "votes": null, "replies": [] } ] }, { "id": 438401, "author_name": "lucaskg", "author_url": "", "post_date": "12/13/2018 15:44:09", "content": "

Is it possible that CV is better but LB becomes worse? Thanks!

", "votes": null, "replies": [ { "id": 438411, "author_name": "cpmpml", "author_url": "", "post_date": "12/13/2018 16:04:47", "content": "

Yes. This is called overfitting. And it happens easily here.

", "votes": null, "replies": [] }, { "id": 438419, "author_name": "lucaskg", "author_url": "", "post_date": "12/13/2018 16:17:50", "content": "

Haha, you are totally right!

", "votes": null, "replies": [] }, { "id": 438420, "author_name": "lucaskg", "author_url": "", "post_date": "12/13/2018 16:21:07", "content": "

So CV can be trusted?

", "votes": null, "replies": [] }, { "id": 438431, "author_name": "cpmpml", "author_url": "", "post_date": "12/13/2018 16:41:29", "content": "

The opposite: CV is not very reliable here. This is because train and test data distributions are very different.

", "votes": null, "replies": [] }, { "id": 438576, "author_name": "yuooka", "author_url": "", "post_date": "12/13/2018 22:45:28", "content": "

Does it mean that you can’t judge whether a feature/features describe test set well and the score gets better until you submit a prediction and see the LB score?

", "votes": null, "replies": [] }, { "id": 438587, "author_name": "cpmpml", "author_url": "", "post_date": "12/13/2018 23:05:56", "content": "

I think we all face the same question.

", "votes": null, "replies": [] }, { "id": 438699, "author_name": "yuooka", "author_url": "", "post_date": "12/14/2018 03:30:53", "content": "

Hmm... Sounds like LB probing...\nAnyways, I believe you have been following your strategy to climbing up the LB. I'm looking forward to seeing your strategy after the competition ends.

", "votes": null, "replies": [] } ] }, { "id": 438421, "author_name": "meykds", "author_url": "", "post_date": "12/13/2018 16:21:51", "content": "

That is really impressive score for a single model :) Don't you mind sharing the number of features that your team is using to achieve that result?

\n\n

Good luck guys!

", "votes": null, "replies": [ { "id": 438428, "author_name": "cpmpml", "author_url": "", "post_date": "12/13/2018 16:40:22", "content": "

Less than 200 features, but some are pretty heavy to compute.

", "votes": null, "replies": [] }, { "id": 438437, "author_name": "meykds", "author_url": "", "post_date": "12/13/2018 16:53:32", "content": "

Thanks! Let us have a talk after the competition ends :)

", "votes": null, "replies": [] }, { "id": 439096, "author_name": "jimpsull", "author_url": "", "post_date": "12/14/2018 18:22:59", "content": "

pretty heavy to compute - does that translate to having to step through each light curve? I had some really cool methods to extract some pretty useful features on the training set but I had to abandon them because I didn't see how I could pull it off on the test set. I need to re-read your posts on writing efficient code.

", "votes": null, "replies": [] }, { "id": 439115, "author_name": "cpmpml", "author_url": "", "post_date": "12/14/2018 18:59:09", "content": "

Sure, some computation require to process each lightcurve. For test data we have to run it in parallel. When my team mate said it takes 2 weeks overall I think he is close to the truth.

", "votes": null, "replies": [] }, { "id": 439286, "author_name": "jimpsull", "author_url": "", "post_date": "12/15/2018 04:56:22", "content": "

In hindsight I wish I’d had the patience. Using only kernels I would’ve had to run about 600 hours but I could’ve split that 20 ways and merged the results.

", "votes": null, "replies": [] } ] }, { "id": 438464, "author_name": "mithrillion", "author_url": "", "post_date": "12/13/2018 18:08:45", "content": "

Well, at least you showed me that class 52 does not have to be the sacrificial lamb for the greater good!

", "votes": null, "replies": [] }, { "id": 438639, "author_name": "kyleboone", "author_url": "", "post_date": "12/14/2018 01:14:51", "content": "

Seeing your confusion matrix has been very helpful to me, so in the spirit of fairness I have attached my best one:

\n\n

\"Confusion

\n\n

Overall they are fairly similar, but there are some noticeable differences between the two! This single model has a CV of 0.405 and a LB of 0.730.

", "votes": null, "replies": [ { "id": 438679, "author_name": "niclasdoce", "author_url": "", "post_date": "12/14/2018 02:41:22", "content": "

@kyleboone, how many features do you have?

", "votes": null, "replies": [] }, { "id": 438682, "author_name": "kyleboone", "author_url": "", "post_date": "12/14/2018 02:52:32", "content": "

Around 200, but I'm doing things a bit differently than everyone else ;)

", "votes": null, "replies": [] }, { "id": 438694, "author_name": "niclasdoce", "author_url": "", "post_date": "12/14/2018 03:17:51", "content": "

alright goodluck! im stuck for quite some time now and still trying to find useful info to improve my model up to the last minute

", "votes": null, "replies": [] }, { "id": 438727, "author_name": "cpmpml", "author_url": "", "post_date": "12/14/2018 04:48:49", "content": "

Kyle,

\n\n
\n

Seeing your confusion matrix has been very helpful to me,

\n
\n\n

I'm curious about why it was useful. I try to share as much as I can without helping competition too directly, and I did not think this would help anyone really...

\n\n
\n

in the spirit of fairness I have attached my best one

\n
\n\n

fair play is essential, thanks for sharing as well.

\n\n
\n

I'm doing things a bit differently than everyone else ;)

\n
\n\n

That's what I' was afraid of, you probably have a different angle than us. For one, you don't say what you use for modeling. ;)

\n\n

Well, only few days left before we know...

\n\n

Looking at your matrix, it seems our model categorical accuracy (sum of diagonal) is better than yours, but LB is lower. This is very interesting.

", "votes": null, "replies": [] }, { "id": 438745, "author_name": "kyleboone", "author_url": "", "post_date": "12/14/2018 05:50:50", "content": "

Well, it's probably not that different. I don't really know what you guys are doing either. You will probably find some interesting things if you look where our confusion matrices differ.

", "votes": null, "replies": [] }, { "id": 439199, "author_name": "titericz", "author_url": "", "post_date": "12/14/2018 22:10:06", "content": "

Very Impressive... the difference between CV and LB is only 0.325.

", "votes": null, "replies": [] }, { "id": 439283, "author_name": "jiweiliu", "author_url": "", "post_date": "12/15/2018 04:53:12", "content": "

I believe this small gap has to do with the special treatment mentioned by Kyle. Now the question is if the LB is randomly split and any special processing can be generalized to the private.

", "votes": null, "replies": [] } ] }, { "id": 439668, "author_name": "greenwing1985", "author_url": "", "post_date": "12/16/2018 03:06:34", "content": "

Are you required to use only one single model? is it in the rules?

", "votes": null, "replies": [ { "id": 439702, "author_name": "cpmpml", "author_url": "", "post_date": "12/16/2018 05:44:13", "content": "

No, you can use whatever you see fit.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "438332": "I shared [the confusion matrix for a 0.801 single lgb model][1] few weeks ago. We are now approaching the 0.75 bar with a single model (0.752 to be precise). Here is its confusion matrix. We can see that hard classes are a bit easier. For this model CV is 0.348 and gap is 0.404. I doubt it will be enough to catch up with leaders, but we'll see when we ensemble.\n\n![confusion matrix][2]\n\n\n [1]: https://www.kaggle.com/c/PLAsTiCC-2018/discussion/72613\n [2]: https://storage.googleapis.com/kaggle-forum-message-attachments/438332/10871/lgb_752.png", "438338": "Please don't improve anymore, I can't catch:)", "438341": "Still suffered from the agony of big gap between LB and CV, can't figure out the magic behind it.", "438342": "Please don't improve anymore, or you catch us:)", "438352": "Yes. We got CV 0.39X, but LB...QQ", "438356": "Your solo performance is impressive, but we'll try to keep you at bay ;)", "438358": "I'm sorry but we'll try to disappoint you ;)", "438363": "It will be good to see other leaders confusion matrix. If they are not distributed the same way then further improvements will be possible after competition end.", "438377": "Thanks, and good luck:)", "438383": "No Thanks, but good luck:)", "438401": "Is it possible that CV is better but LB becomes worse? Thanks!", "438411": "Yes. This is called overfitting. And it happens easily here.", "438415": "My matrix is much worse (: Do you use lightGBM ? How many features do you use?", "438419": "Haha, you are totally right!", "438420": "So CV can be trusted?", "438421": "That is really impressive score for a single model :) Don't you mind sharing the number of features that your team is using to achieve that result?\n\nGood luck guys!", "438428": "Less than 200 features, but some are pretty heavy to compute.", "438431": "The opposite: CV is not very reliable here. This is because train and test data distributions are very different.", "438432": "This model is a lightgbm model, but we use other models too.", "438437": "Thanks! Let us have a talk after the competition ends :)", "438464": "Well, at least you showed me that class 52 does not have to be the sacrificial lamb for the greater good!", "438576": "Does it mean that you can’t judge whether a feature/features describe test set well and the score gets better until you submit a prediction and see the LB score?", "438587": "I think we all face the same question.", "438639": "Seeing your confusion matrix has been very helpful to me, so in the spirit of fairness I have attached my best one:\n\n![Confusion matrix][1]\n\nOverall they are fairly similar, but there are some noticeable differences between the two! This single model has a CV of 0.405 and a LB of 0.730.\n\n\n [1]: https://storage.googleapis.com/kaggle-forum-message-attachments/438639/10876/confusion_matrix.png", "438679": "kyleboone, how many features do you have?", "438682": "Around 200, but I'm doing things a bit differently than everyone else ;)", "438694": "alright goodluck! im stuck for quite some time now and still trying to find useful info to improve my model up to the last minute", "438699": "Hmm... Sounds like LB probing...\nAnyways, I believe you have been following your strategy to climbing up the LB. I'm looking forward to seeing your strategy after the competition ends.", "438727": "Kyle,\n\n> Seeing your confusion matrix has been very helpful to me,\n\nI'm curious about why it was useful. I try to share as much as I can without helping competition too directly, and I did not think this would help anyone really...\n\n> in the spirit of fairness I have attached my best one\n\nfair play is essential, thanks for sharing as well.\n\n> I'm doing things a bit differently than everyone else ;)\n\nThat's what I' was afraid of, you probably have a different angle than us. For one, you don't say what you use for modeling. ;)\n\nWell, only few days left before we know...\n\nLooking at your matrix, it seems our model categorical accuracy (sum of diagonal) is better than yours, but LB is lower. This is very interesting.", "438745": "Well, it's probably not that different. I don't really know what you guys are doing either. You will probably find some interesting things if you look where our confusion matrices differ.", "439096": "pretty heavy to compute - does that translate to having to step through each light curve? I had some really cool methods to extract some pretty useful features on the training set but I had to abandon them because I didn't see how I could pull it off on the test set. I need to re-read your posts on writing efficient code.", "439115": "Sure, some computation require to process each lightcurve. For test data we have to run it in parallel. When my team mate said it takes 2 weeks overall I think he is close to the truth.", "439199": "Very Impressive... the difference between CV and LB is only 0.325.", "439283": "I believe this small gap has to do with the special treatment mentioned by Kyle. Now the question is if the LB is randomly split and any special processing can be generalized to the private.", "439286": "In hindsight I wish I’d had the patience. Using only kernels I would’ve had to run about 600 hours but I could’ve split that 20 ways and merged the results.", "439668": "Are you required to use only one single model? is it in the rules?", "439702": "No, you can use whatever you see fit." }, "source": "meta" }