Here I posted a public notebook about ensembling.
\n[7.739 on pubLB]Ensembling for better performance
\nThe ensembling of 7.745 and 8.333 improved to 7.739
\nI know it's so early to focus on how to ensemble, but I just wanted to share my experience & start discussing.
", "messages": [ { "id": 1221443, "postDate": "2021-03-01T01:51:08.343Z", "content": "Here I posted a public notebook about ensembling.
\n[7.739 on pubLB]Ensembling for better performance
\nThe ensembling of 7.745 and 8.333 improved to 7.739
\nI know it's so early to focus on how to ensemble, but I just wanted to share my experience & start discussing.
", "rawMarkdown": "Here I posted a public notebook about ensembling.\n\n[[7.739 on pubLB]Ensembling for better performance](https://www.kaggle.com/satokiogiso/7-739-on-publb-ensembling-for-better-performance)\n\nThe ensembling of 7.745 and 8.333 improved to 7.739\n\nI know it's so early to focus on how to ensemble, but I just wanted to share my experience & start discussing.", "votes": 15 }, { "id": 1221808, "postDate": "2021-03-01T09:55:08.890Z", "content": "The ensemble works very well for me.
\nBest single RNN:
\nCV: 7.0
\nLB: 6.1
\nAveraging 5 RNNs (different hyperparameters):
\nCV: 6.6
\nLB: 5.7
Averaging models with just different hyper parameters works?
\nI thought that doesn't introduce much diversity among models.
\nI've never tried that, that's interesting!
Do you usually try the method?
\nOr does it work only for some specific tasks?
Hi Kouki, I meant hyperparameters in the general sense (sorry can't say more at the moment). Yes, it is very specific to this dataset and it is not about learning rate, weight decay, etc.
", "rawMarkdown": "Hi Kouki, I meant hyperparameters in the general sense (sorry can't say more at the moment). Yes, it is very specific to this dataset and it is not about learning rate, weight decay, etc.", "votes": 7 }, { "id": 1222473, "postDate": "2021-03-01T20:22:47.390Z", "content": "Okay, I think I get it. Thanks!
", "rawMarkdown": "Okay, I think I get it. Thanks!", "votes": 1 }, { "id": 1249604, "postDate": "2021-03-23T12:01:35.450Z", "content": "Hyperparameters here refers to (hyper)parameters for entire ML pipeline. This might include various data pre-processing, different data sampling, various features extracted etc. Think about the pattern most frequently used in NLP models, where what we call a model include various text transformation steps, not only the ML algorithm.
", "rawMarkdown": "Hyperparameters here refers to (hyper)parameters for entire ML pipeline. This might include various data pre-processing, different data sampling, various features extracted etc. Think about the pattern most frequently used in NLP models, where what we call a model include various text transformation steps, not only the ML algorithm.", "votes": 3 } ] }, { "id": 1222158, "postDate": "2021-03-01T15:42:46.710Z", "content": "Thanks for sharing, it seems to make the model outputs more stable :)
", "rawMarkdown": "Thanks for sharing, it seems to make the model outputs more stable :)", "votes": 1 }, { "id": 1221929, "postDate": "2021-03-01T12:10:27.657Z", "content": "@satokiogiso - thanks for sharing that. Ensembling is an excellent technique, as Galton realised more than a century ago: Francis Galton, Vox Populi, Nature, 75, 450-451 (1907)
\nhttps://galton.org/cgi-bin/searchImages/search/essays/pages/galton-1907-vox-populi_2.htm
Yeah ensembling is a very common technique. I just posted this because nobody has shared notebook about it in this competition:) Didn’t know this kind of literature exists from more than a century ago…
", "rawMarkdown": "Yeah ensembling is a very common technique. I just posted this because nobody has shared notebook about it in this competition:) Didn’t know this kind of literature exists from more than a century ago...", "votes": 2 }, { "id": 1249629, "postDate": "2021-03-23T12:19:36.750Z", "rawMarkdown": "", "votes": 2, "isDeleted": true } ] }, { "id": 1244203, "postDate": "2021-03-18T19:06:29.480Z", "rawMarkdown": "", "votes": 2, "isDeleted": true } ], "comments": [ { "id": 1221808, "author_name": "Jiwei Liu", "author_url": "", "post_date": "2021-03-01T09:55:08.890000", "content": "The ensemble works very well for me.
\nBest single RNN:
\nCV: 7.0
\nLB: 6.1
\nAveraging 5 RNNs (different hyperparameters):
\nCV: 6.6
\nLB: 5.7
Averaging models with just different hyper parameters works?
\nI thought that doesn't introduce much diversity among models.
\nI've never tried that, that's interesting!
Do you usually try the method?
\nOr does it work only for some specific tasks?
Hi Kouki, I meant hyperparameters in the general sense (sorry can't say more at the moment). Yes, it is very specific to this dataset and it is not about learning rate, weight decay, etc.
", "votes": 7, "replies": [] }, { "id": 1222473, "author_name": "Kouki", "author_url": "", "post_date": "2021-03-01T20:22:47.390000", "content": "Okay, I think I get it. Thanks!
", "votes": 1, "replies": [] }, { "id": 1249604, "author_name": "Gabriel Preda", "author_url": "", "post_date": "2021-03-23T12:01:35.450000", "content": "Hyperparameters here refers to (hyper)parameters for entire ML pipeline. This might include various data pre-processing, different data sampling, various features extracted etc. Think about the pattern most frequently used in NLP models, where what we call a model include various text transformation steps, not only the ML algorithm.
", "votes": 3, "replies": [] } ] }, { "id": 1222158, "author_name": "Old Monk", "author_url": "", "post_date": "2021-03-01T15:42:46.710000", "content": "Thanks for sharing, it seems to make the model outputs more stable :)
", "votes": 1, "replies": [] }, { "id": 1221929, "author_name": "John Mitchell", "author_url": "", "post_date": "2021-03-01T12:10:27.657000", "content": "@satokiogiso - thanks for sharing that. Ensembling is an excellent technique, as Galton realised more than a century ago: Francis Galton, Vox Populi, Nature, 75, 450-451 (1907)
\nhttps://galton.org/cgi-bin/searchImages/search/essays/pages/galton-1907-vox-populi_2.htm
Yeah ensembling is a very common technique. I just posted this because nobody has shared notebook about it in this competition:) Didn’t know this kind of literature exists from more than a century ago…
", "votes": 2, "replies": [] }, { "id": 1249629, "author_name": "", "author_url": "", "post_date": "2021-03-23T12:19:36.750000", "content": "", "votes": 2, "replies": [] } ] }, { "id": 1244203, "author_name": "", "author_url": "", "post_date": "2021-03-18T19:06:29.480000", "content": "", "votes": 2, "replies": [] } ], "raw_markdown_by_id": { "1221443": "Here I posted a public notebook about ensembling.\n\n[[7.739 on pubLB]Ensembling for better performance](https://www.kaggle.com/satokiogiso/7-739-on-publb-ensembling-for-better-performance)\n\nThe ensembling of 7.745 and 8.333 improved to 7.739\n\nI know it's so early to focus on how to ensemble, but I just wanted to share my experience & start discussing.", "1221808": "The ensemble works very well for me. \nBest single RNN:\nCV: 7.0\nLB: 6.1\nAveraging 5 RNNs (different hyperparameters):\nCV: 6.6\nLB: 5.7 ", "1222158": "Thanks for sharing, it seems to make the model outputs more stable :)", "1221929": "@satokiogiso - thanks for sharing that. Ensembling is an excellent technique, as Galton realised more than a century ago: Francis Galton, Vox Populi, Nature, 75, 450-451 (1907)\n[https://galton.org/cgi-bin/searchImages/search/essays/pages/galton-1907-vox-populi_2.htm](url)", "1244203": "" } }