{ "id": 75179, "title": "Top teams' solutions are quite different (finally reached 0.5x !!!)", "url": "/competitions/PLAsTiCC-2018/discussion/75179", "author_name": "", "post_date": "2018-12-19T07:17:39.740482100Z", "votes": 31, "comment_count": 25, "views": 0, "content": "

I haven't read all of them very carefully, but seems we improved the score by different ways! It's very impressive to me. maybe we can get to 0.5x if we get average of gold medal teams model! (especially, top 5 teams)

\n\n

UPDATED:

\n\n

We finally reached 0.5x using 1st, 2nd, 3rd, 4th, 5th subs!\nReallly Great :)

\n\n

", "messages": [ { "id": "441864", "postDate": "12/19/2018 07:17:39", "content": "

I haven't read all of them very carefully, but seems we improved the score by different ways! It's very impressive to me. maybe we can get to 0.5x if we get average of gold medal teams model! (especially, top 5 teams)

\n\n

UPDATED:

\n\n

We finally reached 0.5x using 1st, 2nd, 3rd, 4th, 5th subs!\nReallly Great :)

\n\n

", "rawMarkdown": "I haven't read all of them very carefully, but seems we improved the score by different ways! It's very impressive to me. maybe we can get to 0.5x if we get average of gold medal teams model! (especially, top 5 teams)\n\nUPDATED:\n\nWe finally reached 0.5x using 1st, 2nd, 3rd, 4th, 5th subs!\nReallly Great :)\n\n", "votes": null }, { "id": "441887", "postDate": "12/19/2018 07:54:24", "content": "

It's our teams' best sub, which scored 0.680 on Public LB, and 0.700 on Private LB.

", "rawMarkdown": "It's our teams' best sub, which scored 0.680 on Public LB, and 0.700 on Private LB.", "votes": null }, { "id": "442310", "postDate": "12/19/2018 19:23:08", "content": "

Here is my best submission which got 0.670 public/0.685 private. I tried doing a 50/50 weighting of our two models, and got a public LB of 0.631!

", "rawMarkdown": "Here is my best submission which got 0.670 public/0.685 private. I tried doing a 50/50 weighting of our two models, and got a public LB of 0.631!", "votes": null }, { "id": "442319", "postDate": "12/19/2018 19:41:36", "content": "

WOW, seems it's really possible to get to 0.5x!!!

", "rawMarkdown": "WOW, seems it's really possible to get to 0.5x!!!", "votes": null }, { "id": "442373", "postDate": "12/19/2018 21:49:23", "content": "

Yes, it's really amazing how many different approaches were used in this competition!

\n\n

I'm curious what score could be achieved with combination of all approaches.. It looks organizers should be satisfied with competition results :)

", "rawMarkdown": "Yes, it's really amazing how many different approaches were used in this competition! \n\nI'm curious what score could be achieved with combination of all approaches.. It looks organizers should be satisfied with competition results :)", "votes": null }, { "id": "442374", "postDate": "12/19/2018 21:49:57", "content": "

Here's our best -- public 0.694/private 0.699. Just replacing class_99 with the mamas formula brings it down to public 0.650/private 0.656.

", "rawMarkdown": "Here's our best -- public 0.694/private 0.699. Just replacing class_99 with the mamas formula brings it down to public 0.650/private 0.656.", "votes": null }, { "id": "442386", "postDate": "12/19/2018 22:25:33", "content": "

Thanks ! I tried 0.33 : 0.33 : 0.33 weighted average of our 1st-2nd-3rd subs, it scored public 0.617/private 0.630!

", "rawMarkdown": "Thanks ! I tried 0.33 : 0.33 : 0.33 weighted average of our 1st-2nd-3rd subs, it scored public 0.617/private 0.630!", "votes": null }, { "id": "442850", "postDate": "12/20/2018 15:55:51", "content": "

Public 0.691 / Private 0.704. Upload button doesn't work for me, I don't know why. After Kyle's suggestion, it worked.

", "rawMarkdown": "Public 0.691 / Private 0.704. Upload button doesn't work for me, I don't know why. After Kyle's suggestion, it worked.", "votes": null }, { "id": "442851", "postDate": "12/20/2018 15:58:53", "content": "

@psilogram You have much better public-private gap than us. Do you have any idea why it could be? I guess it is because you have the best model (except class99) among us maybe? I also see that in my submission history, while the model gets better, the gap gets smaller.

", "rawMarkdown": "psilogram You have much better public-private gap than us. Do you have any idea why it could be? I guess it is because you have the best model (except class99) among us maybe? I also see that in my submission history, while the model gets better, the gap gets smaller.", "votes": null }, { "id": "442860", "postDate": "12/20/2018 16:02:16", "content": "

Here is our best, private 0.75229. We managed to select our best sub for private ;)

", "rawMarkdown": "Here is our best, private 0.75229. We managed to select our best sub for private ;)", "votes": null }, { "id": "442861", "postDate": "12/20/2018 16:04:22", "content": "

I don't know if upload works or not indeed.

", "rawMarkdown": "I don't know if upload works or not indeed.", "votes": null }, { "id": "442864", "postDate": "12/20/2018 16:07:42", "content": "

There is no progress bar when you upload, but if you wait for a minute (or however long it takes your computer to upload a large file) it shows up as an attachment.

", "rawMarkdown": "There is no progress bar when you upload, but if you wait for a minute (or however long it takes your computer to upload a large file) it shows up as an attachment.", "votes": null }, { "id": "442876", "postDate": "12/20/2018 16:21:15", "content": "

I guess because our LB probing and class_99 formula was so bad! The probing is a double-edged sword since part of the gain from adjusting predictions is due to overfitting the public LB.

", "rawMarkdown": "I guess because our LB probing and class_99 formula was so bad! The probing is a double-edged sword since part of the gain from adjusting predictions is due to overfitting the public LB.", "votes": null }, { "id": "442893", "postDate": "12/20/2018 16:48:26", "content": "

20 mins here, but it worked indeed.

", "rawMarkdown": "20 mins here, but it worked indeed.", "votes": null }, { "id": "442895", "postDate": "12/20/2018 16:51:20", "content": "

It may be worth applying the class_99 magic formula when blending all these subs.

", "rawMarkdown": "It may be worth applying the class_99 magic formula when blending all these subs.", "votes": null }, { "id": "442912", "postDate": "12/20/2018 17:21:30", "content": "

Don't worry. This is what @mamas always do.

", "rawMarkdown": "Don't worry. This is what @mamas always do.", "votes": null }, { "id": "442990", "postDate": "12/20/2018 20:41:29", "content": "

Thanks, \nFinally, reached 0.5x using weighed average and my team's class99 handling !!!\nsee https://www.kaggle.com/c/PLAsTiCC-2018/discussion/75179

", "rawMarkdown": "Thanks, \nFinally, reached 0.5x using weighed average and my team's class99 handling !!!\nsee https://www.kaggle.com/c/PLAsTiCC-2018/discussion/75179", "votes": null }, { "id": "443181", "postDate": "12/21/2018 06:53:34", "content": "

Very cool! Can you upload the final file with 0.59?

", "rawMarkdown": "Very cool! Can you upload the final file with 0.59?", "votes": null }, { "id": "443226", "postDate": "12/21/2018 08:44:05", "content": "

My first guess was also class99 probing but going over all our submission, before and after the probing and with different class99 handling, It looks like our gap is always quit high. Now I think it might be the proportion between different types of objects in the two sets.

", "rawMarkdown": "My first guess was also class99 probing but going over all our submission, before and after the probing and with different class99 handling, It looks like our gap is always quit high. Now I think it might be the proportion between different types of objects in the two sets.", "votes": null }, { "id": "443260", "postDate": "12/21/2018 10:05:24", "content": "

@yuval6967 the problem is not class99 probing. Gap is similar to the rank without probing. If we assume that all of us have your team's 99 formula, Silogram's team could be ranked the first, your team could be ranked the 4th. So when the models on known classes get better, the gap gets closer. This is my guess. If I Iook at my old submissions around 0.900 I have 0.040 public-private gap, while my best submission has 0.013 gap.

\n\n

Real Ranking:\n1-Kyle, 2-Silogram&Mike, 3-Major Tom, 4-Ahmet

\n\n

Gap Ranking:\n1-Silogram&Mike, 2-Ahmet, 3-Kyle, 4-Major Tom

\n\n

Benefit from class99 handling Ranking (from what I read from the solutions):\n1-Major Tom, 2-Kyle, 3-Ahmet, 4-Silogram&Mike

", "rawMarkdown": "yuval6967 the problem is not class99 probing. Gap is similar to the rank without probing. If we assume that all of us have your team's 99 formula, Silogram's team could be ranked the first, your team could be ranked the 4th. So when the models on known classes get better, the gap gets closer. This is my guess. If I Iook at my old submissions around 0.900 I have 0.040 public-private gap, while my best submission has 0.013 gap.\n\nReal Ranking:\n1-Kyle, 2-Silogram&Mike, 3-Major Tom, 4-Ahmet\n\nGap Ranking:\n1-Silogram&Mike, 2-Ahmet, 3-Kyle, 4-Major Tom\n\nBenefit from class99 handling Ranking (from what I read from the solutions):\n1-Major Tom, 2-Kyle, 3-Ahmet, 4-Silogram&Mike", "votes": null }, { "id": "443284", "postDate": "12/21/2018 10:59:18", "content": "

I attached 0.59 sub :)

", "rawMarkdown": "I attached 0.59 sub :)", "votes": null }, { "id": "443286", "postDate": "12/21/2018 11:01:55", "content": "

This is the result of 5 teams with my team's class99 handling.\nseems Ahmet's class99 handling maybe better than our's.\nprobably, if there is no class99, Silogram & Mike would have won this competition.\n

", "rawMarkdown": "This is the result of 5 teams with my team's class99 handling.\nseems Ahmet's class99 handling maybe better than our's.\nprobably, if there is no class99, Silogram & Mike would have won this competition.\n", "votes": null }, { "id": "443307", "postDate": "12/21/2018 11:55:27", "content": "

@mamasinkgs you get 0.667 for Silogram&Mike but they claim they get 0.656 with your method. there may be a difference between applying it first then blending vs blending then applying. I was usually applying my method on individual models and then blend, I assumed this could be less error prone.

", "rawMarkdown": "mamasinkgs you get 0.667 for Silogram&Mike but they claim they get 0.656 with your method. there may be a difference between applying it first then blending vs blending then applying. I was usually applying my method on individual models and then blend, I assumed this could be less error prone.", "votes": null }, { "id": "443330", "postDate": "12/21/2018 12:52:20", "content": "

I was usually applying my method on individual models and then blend\nhmm, you may be right, because I always blend model and then apply class99 method.\nI should have tried it.. :( :( :(

", "rawMarkdown": "` I was usually applying my method on individual models and then blend`\nhmm, you may be right, because I always blend model and then apply class99 method.\nI should have tried it.. :( :( :(", "votes": null }, { "id": "443337", "postDate": "12/21/2018 13:00:52", "content": "

@mamasinkgs btw can you try https://github.com/aerdem4/kaggle-plasticc/blob/master/ensemble.py for ensembling top 5 model? This should work better than normal averaging

", "rawMarkdown": "mamasinkgs btw can you try https://github.com/aerdem4/kaggle-plasticc/blob/master/ensemble.py for ensembling top 5 model? This should work better than normal averaging", "votes": null }, { "id": "443689", "postDate": "12/22/2018 06:39:27", "content": "

best moyen

", "rawMarkdown": "best moyen", "votes": null } ], "comments": [ { "id": 441887, "author_name": "mamasinkgs", "author_url": "", "post_date": "12/19/2018 07:54:24", "content": "

It's our teams' best sub, which scored 0.680 on Public LB, and 0.700 on Private LB.

", "votes": null, "replies": [] }, { "id": 442310, "author_name": "kyleboone", "author_url": "", "post_date": "12/19/2018 19:23:08", "content": "

Here is my best submission which got 0.670 public/0.685 private. I tried doing a 50/50 weighting of our two models, and got a public LB of 0.631!

", "votes": null, "replies": [ { "id": 442319, "author_name": "mamasinkgs", "author_url": "", "post_date": "12/19/2018 19:41:36", "content": "

WOW, seems it's really possible to get to 0.5x!!!

", "votes": null, "replies": [] } ] }, { "id": 442373, "author_name": "antklen", "author_url": "", "post_date": "12/19/2018 21:49:23", "content": "

Yes, it's really amazing how many different approaches were used in this competition!

\n\n

I'm curious what score could be achieved with combination of all approaches.. It looks organizers should be satisfied with competition results :)

", "votes": null, "replies": [] }, { "id": 442374, "author_name": "psilogram", "author_url": "", "post_date": "12/19/2018 21:49:57", "content": "

Here's our best -- public 0.694/private 0.699. Just replacing class_99 with the mamas formula brings it down to public 0.650/private 0.656.

", "votes": null, "replies": [ { "id": 442386, "author_name": "mamasinkgs", "author_url": "", "post_date": "12/19/2018 22:25:33", "content": "

Thanks ! I tried 0.33 : 0.33 : 0.33 weighted average of our 1st-2nd-3rd subs, it scored public 0.617/private 0.630!

", "votes": null, "replies": [] }, { "id": 442851, "author_name": "aerdem4", "author_url": "", "post_date": "12/20/2018 15:58:53", "content": "

@psilogram You have much better public-private gap than us. Do you have any idea why it could be? I guess it is because you have the best model (except class99) among us maybe? I also see that in my submission history, while the model gets better, the gap gets smaller.

", "votes": null, "replies": [] }, { "id": 442876, "author_name": "psilogram", "author_url": "", "post_date": "12/20/2018 16:21:15", "content": "

I guess because our LB probing and class_99 formula was so bad! The probing is a double-edged sword since part of the gain from adjusting predictions is due to overfitting the public LB.

", "votes": null, "replies": [] }, { "id": 443226, "author_name": "yuval6967", "author_url": "", "post_date": "12/21/2018 08:44:05", "content": "

My first guess was also class99 probing but going over all our submission, before and after the probing and with different class99 handling, It looks like our gap is always quit high. Now I think it might be the proportion between different types of objects in the two sets.

", "votes": null, "replies": [] }, { "id": 443260, "author_name": "aerdem4", "author_url": "", "post_date": "12/21/2018 10:05:24", "content": "

@yuval6967 the problem is not class99 probing. Gap is similar to the rank without probing. If we assume that all of us have your team's 99 formula, Silogram's team could be ranked the first, your team could be ranked the 4th. So when the models on known classes get better, the gap gets closer. This is my guess. If I Iook at my old submissions around 0.900 I have 0.040 public-private gap, while my best submission has 0.013 gap.

\n\n

Real Ranking:\n1-Kyle, 2-Silogram&Mike, 3-Major Tom, 4-Ahmet

\n\n

Gap Ranking:\n1-Silogram&Mike, 2-Ahmet, 3-Kyle, 4-Major Tom

\n\n

Benefit from class99 handling Ranking (from what I read from the solutions):\n1-Major Tom, 2-Kyle, 3-Ahmet, 4-Silogram&Mike

", "votes": null, "replies": [] }, { "id": 443286, "author_name": "mamasinkgs", "author_url": "", "post_date": "12/21/2018 11:01:55", "content": "

This is the result of 5 teams with my team's class99 handling.\nseems Ahmet's class99 handling maybe better than our's.\nprobably, if there is no class99, Silogram & Mike would have won this competition.\n

", "votes": null, "replies": [] }, { "id": 443307, "author_name": "aerdem4", "author_url": "", "post_date": "12/21/2018 11:55:27", "content": "

@mamasinkgs you get 0.667 for Silogram&Mike but they claim they get 0.656 with your method. there may be a difference between applying it first then blending vs blending then applying. I was usually applying my method on individual models and then blend, I assumed this could be less error prone.

", "votes": null, "replies": [] }, { "id": 443330, "author_name": "mamasinkgs", "author_url": "", "post_date": "12/21/2018 12:52:20", "content": "

I was usually applying my method on individual models and then blend\nhmm, you may be right, because I always blend model and then apply class99 method.\nI should have tried it.. :( :( :(

", "votes": null, "replies": [] }, { "id": 443337, "author_name": "aerdem4", "author_url": "", "post_date": "12/21/2018 13:00:52", "content": "

@mamasinkgs btw can you try https://github.com/aerdem4/kaggle-plasticc/blob/master/ensemble.py for ensembling top 5 model? This should work better than normal averaging

", "votes": null, "replies": [] } ] }, { "id": 442850, "author_name": "aerdem4", "author_url": "", "post_date": "12/20/2018 15:55:51", "content": "

Public 0.691 / Private 0.704. Upload button doesn't work for me, I don't know why. After Kyle's suggestion, it worked.

", "votes": null, "replies": [ { "id": 442861, "author_name": "cpmpml", "author_url": "", "post_date": "12/20/2018 16:04:22", "content": "

I don't know if upload works or not indeed.

", "votes": null, "replies": [] }, { "id": 442864, "author_name": "kyleboone", "author_url": "", "post_date": "12/20/2018 16:07:42", "content": "

There is no progress bar when you upload, but if you wait for a minute (or however long it takes your computer to upload a large file) it shows up as an attachment.

", "votes": null, "replies": [] }, { "id": 442893, "author_name": "cpmpml", "author_url": "", "post_date": "12/20/2018 16:48:26", "content": "

20 mins here, but it worked indeed.

", "votes": null, "replies": [] }, { "id": 442990, "author_name": "mamasinkgs", "author_url": "", "post_date": "12/20/2018 20:41:29", "content": "

Thanks, \nFinally, reached 0.5x using weighed average and my team's class99 handling !!!\nsee https://www.kaggle.com/c/PLAsTiCC-2018/discussion/75179

", "votes": null, "replies": [] } ] }, { "id": 442860, "author_name": "cpmpml", "author_url": "", "post_date": "12/20/2018 16:02:16", "content": "

Here is our best, private 0.75229. We managed to select our best sub for private ;)

", "votes": null, "replies": [ { "id": 442895, "author_name": "cpmpml", "author_url": "", "post_date": "12/20/2018 16:51:20", "content": "

It may be worth applying the class_99 magic formula when blending all these subs.

", "votes": null, "replies": [] }, { "id": 442912, "author_name": "yuval6967", "author_url": "", "post_date": "12/20/2018 17:21:30", "content": "

Don't worry. This is what @mamas always do.

", "votes": null, "replies": [] } ] }, { "id": 443181, "author_name": "sergeyzlobin", "author_url": "", "post_date": "12/21/2018 06:53:34", "content": "

Very cool! Can you upload the final file with 0.59?

", "votes": null, "replies": [ { "id": 443284, "author_name": "mamasinkgs", "author_url": "", "post_date": "12/21/2018 10:59:18", "content": "

I attached 0.59 sub :)

", "votes": null, "replies": [] } ] }, { "id": 443689, "author_name": "rooshroosh", "author_url": "", "post_date": "12/22/2018 06:39:27", "content": "

best moyen

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "441864": "I haven't read all of them very carefully, but seems we improved the score by different ways! It's very impressive to me. maybe we can get to 0.5x if we get average of gold medal teams model! (especially, top 5 teams)\n\nUPDATED:\n\nWe finally reached 0.5x using 1st, 2nd, 3rd, 4th, 5th subs!\nReallly Great :)\n\n", "441887": "It's our teams' best sub, which scored 0.680 on Public LB, and 0.700 on Private LB.", "442310": "Here is my best submission which got 0.670 public/0.685 private. I tried doing a 50/50 weighting of our two models, and got a public LB of 0.631!", "442319": "WOW, seems it's really possible to get to 0.5x!!!", "442373": "Yes, it's really amazing how many different approaches were used in this competition! \n\nI'm curious what score could be achieved with combination of all approaches.. It looks organizers should be satisfied with competition results :)", "442374": "Here's our best -- public 0.694/private 0.699. Just replacing class_99 with the mamas formula brings it down to public 0.650/private 0.656.", "442386": "Thanks ! I tried 0.33 : 0.33 : 0.33 weighted average of our 1st-2nd-3rd subs, it scored public 0.617/private 0.630!", "442850": "Public 0.691 / Private 0.704. Upload button doesn't work for me, I don't know why. After Kyle's suggestion, it worked.", "442851": "psilogram You have much better public-private gap than us. Do you have any idea why it could be? I guess it is because you have the best model (except class99) among us maybe? I also see that in my submission history, while the model gets better, the gap gets smaller.", "442860": "Here is our best, private 0.75229. We managed to select our best sub for private ;)", "442861": "I don't know if upload works or not indeed.", "442864": "There is no progress bar when you upload, but if you wait for a minute (or however long it takes your computer to upload a large file) it shows up as an attachment.", "442876": "I guess because our LB probing and class_99 formula was so bad! The probing is a double-edged sword since part of the gain from adjusting predictions is due to overfitting the public LB.", "442893": "20 mins here, but it worked indeed.", "442895": "It may be worth applying the class_99 magic formula when blending all these subs.", "442912": "Don't worry. This is what @mamas always do.", "442990": "Thanks, \nFinally, reached 0.5x using weighed average and my team's class99 handling !!!\nsee https://www.kaggle.com/c/PLAsTiCC-2018/discussion/75179", "443181": "Very cool! Can you upload the final file with 0.59?", "443226": "My first guess was also class99 probing but going over all our submission, before and after the probing and with different class99 handling, It looks like our gap is always quit high. Now I think it might be the proportion between different types of objects in the two sets.", "443260": "yuval6967 the problem is not class99 probing. Gap is similar to the rank without probing. If we assume that all of us have your team's 99 formula, Silogram's team could be ranked the first, your team could be ranked the 4th. So when the models on known classes get better, the gap gets closer. This is my guess. If I Iook at my old submissions around 0.900 I have 0.040 public-private gap, while my best submission has 0.013 gap.\n\nReal Ranking:\n1-Kyle, 2-Silogram&Mike, 3-Major Tom, 4-Ahmet\n\nGap Ranking:\n1-Silogram&Mike, 2-Ahmet, 3-Kyle, 4-Major Tom\n\nBenefit from class99 handling Ranking (from what I read from the solutions):\n1-Major Tom, 2-Kyle, 3-Ahmet, 4-Silogram&Mike", "443284": "I attached 0.59 sub :)", "443286": "This is the result of 5 teams with my team's class99 handling.\nseems Ahmet's class99 handling maybe better than our's.\nprobably, if there is no class99, Silogram & Mike would have won this competition.\n", "443307": "mamasinkgs you get 0.667 for Silogram&Mike but they claim they get 0.656 with your method. there may be a difference between applying it first then blending vs blending then applying. I was usually applying my method on individual models and then blend, I assumed this could be less error prone.", "443330": "` I was usually applying my method on individual models and then blend`\nhmm, you may be right, because I always blend model and then apply class99 method.\nI should have tried it.. :( :( :(", "443337": "mamasinkgs btw can you try https://github.com/aerdem4/kaggle-plasticc/blob/master/ensemble.py for ensembling top 5 model? This should work better than normal averaging", "443689": "best moyen" }, "source": "meta" }