{ "id": 174729, "title": "Does anyone else wish more competitions were single model only?", "url": "/competitions/siim-isic-melanoma-classification/discussion/174729", "author_name": "Signal", "post_date": "2020-08-15T03:19:01.053000", "votes": 53, "comment_count": 20, "views": 0, "content": "

I realize the benefits of ensembles. I also realize you can produce a better score in most cases by using them. But I enjoy the simplicity of single models. Remember the Netflix prize? It started off really interesting, but over the years it became almost a mess. I believe the final prize, was such a confusing ensemble of spaghetti and duct tape, that even though it beat Netflix's current model at the time, it was so messy and complicated that they didn't even use it. I would think the benefit to a lot of sponsors in the ability to actually deploy the models. I almost wish there was a single model prize and an ensemble prize.

", "messages": [ { "id": 970949, "postDate": "2020-08-15T03:19:01.053Z", "content": "

I realize the benefits of ensembles. I also realize you can produce a better score in most cases by using them. But I enjoy the simplicity of single models. Remember the Netflix prize? It started off really interesting, but over the years it became almost a mess. I believe the final prize, was such a confusing ensemble of spaghetti and duct tape, that even though it beat Netflix's current model at the time, it was so messy and complicated that they didn't even use it. I would think the benefit to a lot of sponsors in the ability to actually deploy the models. I almost wish there was a single model prize and an ensemble prize.

", "rawMarkdown": "I realize the benefits of ensembles. I also realize you can produce a better score in most cases by using them. But I enjoy the simplicity of single models. Remember the Netflix prize? It started off really interesting, but over the years it became almost a mess. I believe the final prize, was such a confusing ensemble of spaghetti and duct tape, that even though it beat Netflix's current model at the time, it was so messy and complicated that they didn't even use it. I would think the benefit to a lot of sponsors in the ability to actually deploy the models. I almost wish there was a single model prize and an ensemble prize.", "votes": 52 }, { "id": 971118, "postDate": "2020-08-15T07:36:13.837Z", "content": "

Easy solution: kernel competition with very tight runtime limits.

\n

This also removes all those ugly blending kernels that imho destroy some fun of this competition.

", "rawMarkdown": "Easy solution: kernel competition with very tight runtime limits.\n\nThis also removes all those ugly blending kernels that imho destroy some fun of this competition.", "votes": 33, "replies": [ { "id": 972187, "postDate": "2020-08-16T10:07:36.790Z", "content": "

I fully agree. Defining “single model” is quite difficult, but kernel competitions with runtime limits are the indirect way to impose constraints that motivate investing more time in refining a single model. People would still do blends, but only simple ones would be able to make it in the final solution.

", "rawMarkdown": "I fully agree. Defining “single model” is quite difficult, but kernel competitions with runtime limits are the indirect way to impose constraints that motivate investing more time in refining a single model. People would still do blends, but only simple ones would be able to make it in the final solution.", "votes": 1 } ] }, { "id": 971017, "postDate": "2020-08-15T05:11:12.590Z", "content": "

Because the thing is, \"only one model\" is a rule impossible to impose with a grey boundary. If I put multiple resnets inside the same pytorch class is that an ensemble or just a stupidly big model ? Are random forests one model or an ensemble ?

\n

Should we still allow unlimited TTA steps ? This is also something that has limits in industry. If a service is already saturating a GPU I'm not sure you'll justify needing 25 more for your 25 TTA steps. The code is more elegant than ensembles though yes…

\n

That said I agree with the sentiment.

\n

I think a lot of troubles like this are somewhat solved by requiring you to use a Kaggle kernel for inference. It also puts a clear limit on what you can do. If you can't produce predictions in say 6 hours (to be defined by organizers), then go away.

", "rawMarkdown": "Because the thing is, \"only one model\" is a rule impossible to impose with a grey boundary. If I put multiple resnets inside the same pytorch class is that an ensemble or just a stupidly big model ? Are random forests one model or an ensemble ?\n\nShould we still allow unlimited TTA steps ? This is also something that has limits in industry. If a service is already saturating a GPU I'm not sure you'll justify needing 25 more for your 25 TTA steps. The code is more elegant than ensembles though yes...\n\nThat said I agree with the sentiment.\n\nI think a lot of troubles like this are somewhat solved by requiring you to use a Kaggle kernel for inference. It also puts a clear limit on what you can do. If you can't produce predictions in say 6 hours (to be defined by organizers), then go away.", "votes": 18, "replies": [ { "id": 971163, "postDate": "2020-08-15T08:27:18.973Z", "content": "

+!

\n

Add to it using k model instances from k fold CV. Is that considered as a single model or not? And we can mention other blurry boundary like bagging, swa, etc.

", "rawMarkdown": "+!\n\nAdd to it using k model instances from k fold CV. Is that considered as a single model or not? And we can mention other blurry boundary like bagging, swa, etc.", "votes": 4 } ] }, { "id": 971162, "postDate": "2020-08-15T08:25:51.997Z", "content": "

This is a misconception. Netflix used parts of the solution.

\n

The benefit of top ensemble is not to be reused as is, it is to know what is achievable. Then if you get close to it with a simpler model then you're happy. If you don't have the top ensemble then you don't know how good your single model is.

\n

I guess here the host wants to know how far the limit can be pushed.

", "rawMarkdown": "This is a misconception. Netflix used parts of the solution.\n\nThe benefit of top ensemble is not to be reused as is, it is to know what is achievable. Then if you get close to it with a simpler model then you're happy. If you don't have the top ensemble then you don't know how good your single model is.\n\nI guess here the host wants to know how far the limit can be pushed.", "votes": 13 }, { "id": 970975, "postDate": "2020-08-15T03:58:17.297Z", "content": "

Great suggestion. i would enjoy a single model competition. Building 100 diverse models and ensembling is tedious. Optimizing a single model would be more fun.

", "rawMarkdown": "Great suggestion. i would enjoy a single model competition. Building 100 diverse models and ensembling is tedious. Optimizing a single model would be more fun.", "votes": 14, "replies": [ { "id": 972328, "postDate": "2020-08-16T13:20:06.803Z", "content": "

\"Optimizing a single model would be more fun.\"

\n

i think the outcome will be distilling 100 of models into one.

", "rawMarkdown": "\"Optimizing a single model would be more fun.\"\n\ni think the outcome will be distilling 100 of models into one.", "votes": 2 }, { "id": 972364, "postDate": "2020-08-16T13:44:11.037Z", "content": "

this is what i hope to see:
\n\"\"

\n

instead of being stuck in ensemble 100 of models, i would like to move forward to self supervised methods. like using massive data crawled from web …

", "rawMarkdown": "this is what i hope to see:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fbbdf79410cc18ee764bd01c67cb437c6%2FSelection_042.png?generation=1597585385065427&alt=media)\n\ninstead of being stuck in ensemble 100 of models, i would like to move forward to self supervised methods. like using massive data crawled from web ...", "votes": 4 } ] }, { "id": 973336, "postDate": "2020-08-17T08:52:34.333Z", "content": "

yes,i also wish more competitions were lottery free only 😪
\nsick of Unseen graphemes,Crazy M5, seeking for magics, Mystery Images etc

", "rawMarkdown": "yes,i also wish more competitions were **lottery free only** 😪\nsick of **Unseen graphemes**,**Crazy M5**, **seeking for magics**, **Mystery Images** etc\n\n", "votes": 5 }, { "id": 971132, "postDate": "2020-08-15T07:52:32.897Z", "content": "

Kernel competitions is clearly the way to go.

\n

These crazy blending are neither fun nor useful for the organizers. That just encourages spending time blending 100+ models instead of thinking about efficient arch.

\n

For instance, here we don't even know what the best public kernel score is blending

", "rawMarkdown": "Kernel competitions is clearly the way to go. \n\nThese crazy blending are neither fun nor useful for the organizers. That just encourages spending time blending 100+ models instead of thinking about efficient arch.\n\nFor instance, here we don't even know what the best public kernel score is blending", "votes": 4, "replies": [ { "id": 971153, "postDate": "2020-08-15T08:16:34.157Z", "content": "

@serigne agreed. And its bad for the sponsors too. I mean, its really nice to have techniques you can explain and understand, because those become the key takeaways.

", "rawMarkdown": "@serigne agreed. And its bad for the sponsors too. I mean, its really nice to have techniques you can explain and understand, because those become the key takeaways." } ] }, { "id": 971059, "postDate": "2020-08-15T06:22:14.333Z", "content": "

Agree, single models would be fun, but difficult to enforce.
\nI think Kaggle can start off by making all competitions, \"notebooks only\".

", "rawMarkdown": "Agree, single models would be fun, but difficult to enforce.\nI think Kaggle can start off by making all competitions, \"notebooks only\".", "votes": 1 }, { "id": 970983, "postDate": "2020-08-15T04:21:27.003Z", "content": "

I agree. I think we would learn a lot more if we were working and optimizing a single model. It is easy to lose yourself and the context of the problem when optimizing and ensembling lots of models.

", "rawMarkdown": "I agree. I think we would learn a lot more if we were working and optimizing a single model. It is easy to lose yourself and the context of the problem when optimizing and ensembling lots of models.", "votes": -1 }, { "id": 971451, "postDate": "2020-08-15T14:33:13.460Z", "content": "

Is it just me or is running TTA 15 times for each fold very expensive?

", "rawMarkdown": "Is it just me or is running TTA 15 times for each fold very expensive?", "votes": -2, "replies": [ { "id": 971707, "postDate": "2020-08-15T20:18:19.810Z", "content": "

Each TTA pass takes me about 8 seconds, I do 24 TTA and its about 3 1/2 minutes. Use a large batch size like say 512 - 1024, inference doesn't require as much memory.

", "rawMarkdown": "Each TTA pass takes me about 8 seconds, I do 24 TTA and its about 3 1/2 minutes. Use a large batch size like say 512 - 1024, inference doesn't require as much memory.", "votes": 1 } ] }, { "id": 974072, "postDate": "2020-08-17T17:53:59.287Z", "content": "

I don't think the ensembling/stacking itself is wrong. It is part of ML. (averaging models , or weighted them).
\nBut doing a blending base on \"random\" coefficient without any validation strategy is wrong and should not be accepted I think.

", "rawMarkdown": "I don't think the ensembling/stacking itself is wrong. It is part of ML. (averaging models , or weighted them).\nBut doing a blending base on \"random\" coefficient without any validation strategy is wrong and should not be accepted I think." }, { "id": 972451, "postDate": "2020-08-16T15:04:31.653Z", "content": "

As others said, there are no clear boundaries to define single models, but this is the closest thing right now.

\n

https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/

", "rawMarkdown": "As others said, there are no clear boundaries to define single models, but this is the closest thing right now.\n\nhttps://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/" }, { "id": 971473, "postDate": "2020-08-15T15:03:47.987Z", "content": "

yep! i really wish it was :)

", "rawMarkdown": "yep! i really wish it was :)\n" }, { "id": 971152, "postDate": "2020-08-15T08:16:30.077Z", "content": "

As much as I don't like working with kernels, I must admit kernel competition are better in every aspect.

", "rawMarkdown": "As much as I don't like working with kernels, I must admit kernel competition are better in every aspect. " }, { "id": 971471, "postDate": "2020-08-15T15:03:17.237Z", "rawMarkdown": "", "votes": -2, "isDeleted": true } ], "comments": [ { "id": 971118, "author_name": "Psi", "author_url": "", "post_date": "2020-08-15T07:36:13.837000", "content": "

Easy solution: kernel competition with very tight runtime limits.

\n

This also removes all those ugly blending kernels that imho destroy some fun of this competition.

", "votes": 33, "replies": [ { "id": 972187, "author_name": "Nikita Kozodoi", "author_url": "", "post_date": "2020-08-16T10:07:36.790000", "content": "

I fully agree. Defining “single model” is quite difficult, but kernel competitions with runtime limits are the indirect way to impose constraints that motivate investing more time in refining a single model. People would still do blends, but only simple ones would be able to make it in the final solution.

", "votes": 1, "replies": [] } ] }, { "id": 971017, "author_name": "Arnaud Roussel", "author_url": "", "post_date": "2020-08-15T05:11:12.590000", "content": "

Because the thing is, \"only one model\" is a rule impossible to impose with a grey boundary. If I put multiple resnets inside the same pytorch class is that an ensemble or just a stupidly big model ? Are random forests one model or an ensemble ?

\n

Should we still allow unlimited TTA steps ? This is also something that has limits in industry. If a service is already saturating a GPU I'm not sure you'll justify needing 25 more for your 25 TTA steps. The code is more elegant than ensembles though yes…

\n

That said I agree with the sentiment.

\n

I think a lot of troubles like this are somewhat solved by requiring you to use a Kaggle kernel for inference. It also puts a clear limit on what you can do. If you can't produce predictions in say 6 hours (to be defined by organizers), then go away.

", "votes": 18, "replies": [ { "id": 971163, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-15T08:27:18.973000", "content": "

+!

\n

Add to it using k model instances from k fold CV. Is that considered as a single model or not? And we can mention other blurry boundary like bagging, swa, etc.

", "votes": 4, "replies": [] } ] }, { "id": 971162, "author_name": "CPMP", "author_url": "", "post_date": "2020-08-15T08:25:51.997000", "content": "

This is a misconception. Netflix used parts of the solution.

\n

The benefit of top ensemble is not to be reused as is, it is to know what is achievable. Then if you get close to it with a simpler model then you're happy. If you don't have the top ensemble then you don't know how good your single model is.

\n

I guess here the host wants to know how far the limit can be pushed.

", "votes": 13, "replies": [] }, { "id": 970975, "author_name": "Chris Deotte", "author_url": "", "post_date": "2020-08-15T03:58:17.297000", "content": "

Great suggestion. i would enjoy a single model competition. Building 100 diverse models and ensembling is tedious. Optimizing a single model would be more fun.

", "votes": 14, "replies": [ { "id": 972328, "author_name": "hengck23", "author_url": "", "post_date": "2020-08-16T13:20:06.803000", "content": "

\"Optimizing a single model would be more fun.\"

\n

i think the outcome will be distilling 100 of models into one.

", "votes": 2, "replies": [] }, { "id": 972364, "author_name": "hengck23", "author_url": "", "post_date": "2020-08-16T13:44:11.037000", "content": "

this is what i hope to see:
\n\"\"

\n

instead of being stuck in ensemble 100 of models, i would like to move forward to self supervised methods. like using massive data crawled from web …

", "votes": 4, "replies": [] } ] }, { "id": 973336, "author_name": "Mobassir", "author_url": "", "post_date": "2020-08-17T08:52:34.333000", "content": "

yes,i also wish more competitions were lottery free only 😪
\nsick of Unseen graphemes,Crazy M5, seeking for magics, Mystery Images etc

", "votes": 5, "replies": [] }, { "id": 971132, "author_name": "Serigne ", "author_url": "", "post_date": "2020-08-15T07:52:32.897000", "content": "

Kernel competitions is clearly the way to go.

\n

These crazy blending are neither fun nor useful for the organizers. That just encourages spending time blending 100+ models instead of thinking about efficient arch.

\n

For instance, here we don't even know what the best public kernel score is blending

", "votes": 4, "replies": [ { "id": 971153, "author_name": "Signal", "author_url": "", "post_date": "2020-08-15T08:16:34.157000", "content": "

@serigne agreed. And its bad for the sponsors too. I mean, its really nice to have techniques you can explain and understand, because those become the key takeaways.

", "votes": 0, "replies": [] } ] }, { "id": 971059, "author_name": "Vee", "author_url": "", "post_date": "2020-08-15T06:22:14.333000", "content": "

Agree, single models would be fun, but difficult to enforce.
\nI think Kaggle can start off by making all competitions, \"notebooks only\".

", "votes": 1, "replies": [] }, { "id": 970983, "author_name": "Santiago Viquez", "author_url": "", "post_date": "2020-08-15T04:21:27.003000", "content": "

I agree. I think we would learn a lot more if we were working and optimizing a single model. It is easy to lose yourself and the context of the problem when optimizing and ensembling lots of models.

", "votes": -1, "replies": [] }, { "id": 971451, "author_name": "Aman Dalmia", "author_url": "", "post_date": "2020-08-15T14:33:13.460000", "content": "

Is it just me or is running TTA 15 times for each fold very expensive?

", "votes": -2, "replies": [ { "id": 971707, "author_name": "Signal", "author_url": "", "post_date": "2020-08-15T20:18:19.810000", "content": "

Each TTA pass takes me about 8 seconds, I do 24 TTA and its about 3 1/2 minutes. Use a large batch size like say 512 - 1024, inference doesn't require as much memory.

", "votes": 1, "replies": [] } ] }, { "id": 974072, "author_name": "Shiro", "author_url": "", "post_date": "2020-08-17T17:53:59.287000", "content": "

I don't think the ensembling/stacking itself is wrong. It is part of ML. (averaging models , or weighted them).
\nBut doing a blending base on \"random\" coefficient without any validation strategy is wrong and should not be accepted I think.

", "votes": 0, "replies": [] }, { "id": 972451, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2020-08-16T15:04:31.653000", "content": "

As others said, there are no clear boundaries to define single models, but this is the closest thing right now.

\n

https://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/

", "votes": 0, "replies": [] }, { "id": 971473, "author_name": "Harikrishnan TP", "author_url": "", "post_date": "2020-08-15T15:03:47.987000", "content": "

yep! i really wish it was :)

", "votes": 0, "replies": [] }, { "id": 971152, "author_name": "Eek The Cat", "author_url": "", "post_date": "2020-08-15T08:16:30.077000", "content": "

As much as I don't like working with kernels, I must admit kernel competition are better in every aspect.

", "votes": 0, "replies": [] }, { "id": 971471, "author_name": "", "author_url": "", "post_date": "2020-08-15T15:03:17.237000", "content": "", "votes": -2, "replies": [] } ], "raw_markdown_by_id": { "970949": "I realize the benefits of ensembles. I also realize you can produce a better score in most cases by using them. But I enjoy the simplicity of single models. Remember the Netflix prize? It started off really interesting, but over the years it became almost a mess. I believe the final prize, was such a confusing ensemble of spaghetti and duct tape, that even though it beat Netflix's current model at the time, it was so messy and complicated that they didn't even use it. I would think the benefit to a lot of sponsors in the ability to actually deploy the models. I almost wish there was a single model prize and an ensemble prize.", "971118": "Easy solution: kernel competition with very tight runtime limits.\n\nThis also removes all those ugly blending kernels that imho destroy some fun of this competition.", "971017": "Because the thing is, \"only one model\" is a rule impossible to impose with a grey boundary. If I put multiple resnets inside the same pytorch class is that an ensemble or just a stupidly big model ? Are random forests one model or an ensemble ?\n\nShould we still allow unlimited TTA steps ? This is also something that has limits in industry. If a service is already saturating a GPU I'm not sure you'll justify needing 25 more for your 25 TTA steps. The code is more elegant than ensembles though yes...\n\nThat said I agree with the sentiment.\n\nI think a lot of troubles like this are somewhat solved by requiring you to use a Kaggle kernel for inference. It also puts a clear limit on what you can do. If you can't produce predictions in say 6 hours (to be defined by organizers), then go away.", "971162": "This is a misconception. Netflix used parts of the solution.\n\nThe benefit of top ensemble is not to be reused as is, it is to know what is achievable. Then if you get close to it with a simpler model then you're happy. If you don't have the top ensemble then you don't know how good your single model is.\n\nI guess here the host wants to know how far the limit can be pushed.", "970975": "Great suggestion. i would enjoy a single model competition. Building 100 diverse models and ensembling is tedious. Optimizing a single model would be more fun.", "973336": "yes,i also wish more competitions were **lottery free only** 😪\nsick of **Unseen graphemes**,**Crazy M5**, **seeking for magics**, **Mystery Images** etc\n\n", "971132": "Kernel competitions is clearly the way to go. \n\nThese crazy blending are neither fun nor useful for the organizers. That just encourages spending time blending 100+ models instead of thinking about efficient arch.\n\nFor instance, here we don't even know what the best public kernel score is blending", "971059": "Agree, single models would be fun, but difficult to enforce.\nI think Kaggle can start off by making all competitions, \"notebooks only\".", "970983": "I agree. I think we would learn a lot more if we were working and optimizing a single model. It is easy to lose yourself and the context of the problem when optimizing and ensembling lots of models.", "971451": "Is it just me or is running TTA 15 times for each fold very expensive?", "974072": "I don't think the ensembling/stacking itself is wrong. It is part of ML. (averaging models , or weighted them).\nBut doing a blending base on \"random\" coefficient without any validation strategy is wrong and should not be accepted I think.", "972451": "As others said, there are no clear boundaries to define single models, but this is the closest thing right now.\n\nhttps://www.kaggle.com/c/osic-pulmonary-fibrosis-progression/", "971473": "yep! i really wish it was :)\n", "971152": "As much as I don't like working with kernels, I must admit kernel competition are better in every aspect. ", "971471": "" } }