{ "id": 72627, "title": "Blending is all you need", "url": "/competitions/quora-insincere-questions-classification/discussion/72627", "author_name": "Shujian Liu", "post_date": "2018-11-25T15:24:11.400000", "votes": 87, "comment_count": 20, "views": 0, "content": "

There are several levels of blending:

\n\n
    \n

  1. Take the average of word embeddings, as I have discussed earlier: https://www.kaggle.com/c/quora-insincere-questions-classification/discussion/71778

    2. \n

  2. Combine LSTM and CRU horizontally, as in Bojan’s kernel: https://www.kaggle.com/tunguz/lstm-gru-why-not-both (or vertically: https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/52644)

    3. \n

  3. Take the average of predictions from many models, as in SRK’s kernel: https://www.kaggle.com/sudalairajkumar/a-look-at-different-embeddings

    4. \n

  4. There is actually another way of blending: snapshot ensemble. It takes the average of the model weights from several epochs from one single model. Chenglong used this approach in his 4th solution to Mercari: https://github.com/ChenglongChen/tensorflow-XNN#summary. I tried this method in this kernel: https://www.kaggle.com/shujian/single-rnn-with-5-folds-snapshot-ensemble (I don’t have any submissions left today so I am not sure about the LB score but the idea is cool). Here is a picture: \"snapshot

    5. \n
\n\n

That’s why blending is all you need :)

", "messages": [ { "id": 427467, "postDate": "2018-11-25T15:24:11.400Z", "content": "

There are several levels of blending:

\n\n
    \n

  1. Take the average of word embeddings, as I have discussed earlier: https://www.kaggle.com/c/quora-insincere-questions-classification/discussion/71778

    2. \n

  2. Combine LSTM and CRU horizontally, as in Bojan’s kernel: https://www.kaggle.com/tunguz/lstm-gru-why-not-both (or vertically: https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/52644)

    3. \n

  3. Take the average of predictions from many models, as in SRK’s kernel: https://www.kaggle.com/sudalairajkumar/a-look-at-different-embeddings

    4. \n

  4. There is actually another way of blending: snapshot ensemble. It takes the average of the model weights from several epochs from one single model. Chenglong used this approach in his 4th solution to Mercari: https://github.com/ChenglongChen/tensorflow-XNN#summary. I tried this method in this kernel: https://www.kaggle.com/shujian/single-rnn-with-5-folds-snapshot-ensemble (I don’t have any submissions left today so I am not sure about the LB score but the idea is cool). Here is a picture: \"snapshot

    5. \n
\n\n

That’s why blending is all you need :)

", "rawMarkdown": "There are several levels of blending:\n\n1. Take the average of word embeddings, as I have discussed earlier: https://www.kaggle.com/c/quora-insincere-questions-classification/discussion/71778\n\n2. Combine LSTM and CRU horizontally, as in Bojan’s kernel: https://www.kaggle.com/tunguz/lstm-gru-why-not-both (or vertically: https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/52644)\n\n3. Take the average of predictions from many models, as in SRK’s kernel: https://www.kaggle.com/sudalairajkumar/a-look-at-different-embeddings\n\n4. There is actually another way of blending: snapshot ensemble. It takes the average of the model weights from several epochs from one single model. Chenglong used this approach in his 4th solution to Mercari: https://github.com/ChenglongChen/tensorflow-XNN#summary. I tried this method in this kernel: https://www.kaggle.com/shujian/single-rnn-with-5-folds-snapshot-ensemble (I don’t have any submissions left today so I am not sure about the LB score but the idea is cool). Here is a picture: ![snapshot ensemble][1]\n\n\nThat’s why blending is all you need :)\n\n\n\n [1]: https://vitalab.github.io/deep-learning/images/snapshot/1.png", "votes": 87 }, { "id": 427490, "postDate": "2018-11-25T15:55:45.947Z", "content": "

Upvoted because of the title.

", "rawMarkdown": "Upvoted because of the title.", "votes": 5, "replies": [ { "id": 427494, "postDate": "2018-11-25T15:57:21.723Z", "content": "

:)

", "rawMarkdown": ":)", "votes": 3 }, { "id": 427498, "postDate": "2018-11-25T16:04:51.143Z", "content": "

Upvoted the comment because of upvote! :D

", "rawMarkdown": "Upvoted the comment because of upvote! :D", "votes": 4 } ] }, { "id": 427476, "postDate": "2018-11-25T15:41:02.790Z", "content": "

Snapshot worked for me in TGS competition....But here it is taking extra-time and requires more epoches to be really effective. So I dropped it for now.

", "rawMarkdown": "Snapshot worked for me in TGS competition....But here it is taking extra-time and requires more epoches to be really effective. So I dropped it for now. ", "votes": 4, "replies": [ { "id": 427478, "postDate": "2018-11-25T15:42:08.167Z", "content": "

Good point!

", "rawMarkdown": "Good point!" }, { "id": 427483, "postDate": "2018-11-25T15:48:41.433Z", "content": "

I burned some submissions to try it . I combined both k-fold and snapshot ( 2 cycles per fold and 7 epoches per cycle ) . But I had to use much shallower network and much less embedded words to fit the 2hrs limitation. It scored \"only\".. 0.692 on LB.

", "rawMarkdown": "I burned some submissions to try it . I combined both k-fold and snapshot ( 2 cycles per fold and 7 epoches per cycle ) . But I had to use much shallower network and much less embedded words to fit the 2hrs limitation. It scored \"only\".. 0.692 on LB. ", "votes": 5 }, { "id": 427486, "postDate": "2018-11-25T15:50:09.240Z", "content": "

WOW. Not bad.

", "rawMarkdown": "WOW. Not bad." } ] }, { "id": 427545, "postDate": "2018-11-25T18:22:36.637Z", "content": "

Yep, that's really helpful! Thanksgiving!😊✌️👍

", "rawMarkdown": "Yep, that's really helpful! Thanksgiving!😊✌️👍", "votes": 1, "replies": [ { "id": 427548, "postDate": "2018-11-25T18:26:39.883Z", "content": "

Happy thanksgiving!

", "rawMarkdown": "Happy thanksgiving!", "votes": 2 } ] }, { "id": 429806, "postDate": "2018-11-29T11:19:50.903Z", "content": "

Nice Kernel

", "rawMarkdown": "Nice Kernel", "votes": 1 }, { "id": 429128, "postDate": "2018-11-28T11:43:22.520Z", "content": "

Thank you for sharing this. You have saved lot of research time and made things more clear.

", "rawMarkdown": "Thank you for sharing this. You have saved lot of research time and made things more clear. ", "votes": 1 }, { "id": 428956, "postDate": "2018-11-28T06:23:06.127Z", "content": "

Great!

", "rawMarkdown": "Great!", "votes": 1 }, { "id": 428306, "postDate": "2018-11-27T03:53:43.093Z", "content": "

awesome

", "rawMarkdown": "awesome", "votes": 1 }, { "id": 428260, "postDate": "2018-11-27T01:47:46.120Z", "content": "

Thank you for sharing in this competition.

", "rawMarkdown": "Thank you for sharing in this competition.", "votes": 1 }, { "id": 427585, "postDate": "2018-11-25T19:17:41.510Z", "content": "

Thank you for all those references during competition. Really helps a lot! :D

", "rawMarkdown": "Thank you for all those references during competition. Really helps a lot! :D", "votes": 2, "replies": [ { "id": 427587, "postDate": "2018-11-25T19:19:45.007Z", "content": "

You are welcome.

", "rawMarkdown": "You are welcome.", "votes": 3 } ] }, { "id": 429597, "postDate": "2018-11-29T03:56:07.753Z", "rawMarkdown": "", "votes": 3, "isDeleted": true }, { "id": 427963, "postDate": "2018-11-26T13:34:50.813Z", "content": "

Thank you

", "rawMarkdown": "Thank you", "votes": 1 }, { "id": 427866, "postDate": "2018-11-26T10:05:18.510Z", "content": "

That`s helpful. Thanks

", "rawMarkdown": "That`s helpful. Thanks", "votes": 1 }, { "id": 442176, "postDate": "2018-12-19T15:21:32.937Z", "content": "

This is supremely helpful! Thanks!

", "rawMarkdown": "This is supremely helpful! Thanks! " } ], "comments": [ { "id": 427490, "author_name": "Miha Skalic", "author_url": "", "post_date": "2018-11-25T15:55:45.947000", "content": "

Upvoted because of the title.

", "votes": 5, "replies": [ { "id": 427494, "author_name": "Shujian Liu", "author_url": "", "post_date": "2018-11-25T15:57:21.723000", "content": "

:)

", "votes": 3, "replies": [] }, { "id": 427498, "author_name": "Miha Skalic", "author_url": "", "post_date": "2018-11-25T16:04:51.143000", "content": "

Upvoted the comment because of upvote! :D

", "votes": 4, "replies": [] } ] }, { "id": 427476, "author_name": "Serigne ", "author_url": "", "post_date": "2018-11-25T15:41:02.790000", "content": "

Snapshot worked for me in TGS competition....But here it is taking extra-time and requires more epoches to be really effective. So I dropped it for now.

", "votes": 4, "replies": [ { "id": 427478, "author_name": "Shujian Liu", "author_url": "", "post_date": "2018-11-25T15:42:08.167000", "content": "

Good point!

", "votes": 0, "replies": [] }, { "id": 427483, "author_name": "Serigne ", "author_url": "", "post_date": "2018-11-25T15:48:41.433000", "content": "

I burned some submissions to try it . I combined both k-fold and snapshot ( 2 cycles per fold and 7 epoches per cycle ) . But I had to use much shallower network and much less embedded words to fit the 2hrs limitation. It scored \"only\".. 0.692 on LB.

", "votes": 5, "replies": [] }, { "id": 427486, "author_name": "Shujian Liu", "author_url": "", "post_date": "2018-11-25T15:50:09.240000", "content": "

WOW. Not bad.

", "votes": 0, "replies": [] } ] }, { "id": 427545, "author_name": "Arunkumar Venkataramanan", "author_url": "", "post_date": "2018-11-25T18:22:36.637000", "content": "

Yep, that's really helpful! Thanksgiving!😊✌️👍

", "votes": 1, "replies": [ { "id": 427548, "author_name": "Shujian Liu", "author_url": "", "post_date": "2018-11-25T18:26:39.883000", "content": "

Happy thanksgiving!

", "votes": 2, "replies": [] } ] }, { "id": 429806, "author_name": "Bhavesh Gulecha", "author_url": "", "post_date": "2018-11-29T11:19:50.903000", "content": "

Nice Kernel

", "votes": 1, "replies": [] }, { "id": 429128, "author_name": "jhajhria", "author_url": "", "post_date": "2018-11-28T11:43:22.520000", "content": "

Thank you for sharing this. You have saved lot of research time and made things more clear.

", "votes": 1, "replies": [] }, { "id": 428956, "author_name": "godsome", "author_url": "", "post_date": "2018-11-28T06:23:06.127000", "content": "

Great!

", "votes": 1, "replies": [] }, { "id": 428306, "author_name": "AntanTyagi", "author_url": "", "post_date": "2018-11-27T03:53:43.093000", "content": "

awesome

", "votes": 1, "replies": [] }, { "id": 428260, "author_name": "Bai", "author_url": "", "post_date": "2018-11-27T01:47:46.120000", "content": "

Thank you for sharing in this competition.

", "votes": 1, "replies": [] }, { "id": 427585, "author_name": "Mihajlo T.", "author_url": "", "post_date": "2018-11-25T19:17:41.510000", "content": "

Thank you for all those references during competition. Really helps a lot! :D

", "votes": 2, "replies": [ { "id": 427587, "author_name": "Shujian Liu", "author_url": "", "post_date": "2018-11-25T19:19:45.007000", "content": "

You are welcome.

", "votes": 3, "replies": [] } ] }, { "id": 429597, "author_name": "", "author_url": "", "post_date": "2018-11-29T03:56:07.753000", "content": "", "votes": 3, "replies": [] }, { "id": 427963, "author_name": "Nahidul Islam", "author_url": "", "post_date": "2018-11-26T13:34:50.813000", "content": "

Thank you

", "votes": 1, "replies": [] }, { "id": 427866, "author_name": "Alexandr M", "author_url": "", "post_date": "2018-11-26T10:05:18.510000", "content": "

That`s helpful. Thanks

", "votes": 1, "replies": [] }, { "id": 442176, "author_name": "Panchajanya Banerjee (Pancham)", "author_url": "", "post_date": "2018-12-19T15:21:32.937000", "content": "

This is supremely helpful! Thanks!

", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "427467": "There are several levels of blending:\n\n1. Take the average of word embeddings, as I have discussed earlier: https://www.kaggle.com/c/quora-insincere-questions-classification/discussion/71778\n\n2. Combine LSTM and CRU horizontally, as in Bojan’s kernel: https://www.kaggle.com/tunguz/lstm-gru-why-not-both (or vertically: https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/52644)\n\n3. Take the average of predictions from many models, as in SRK’s kernel: https://www.kaggle.com/sudalairajkumar/a-look-at-different-embeddings\n\n4. There is actually another way of blending: snapshot ensemble. It takes the average of the model weights from several epochs from one single model. Chenglong used this approach in his 4th solution to Mercari: https://github.com/ChenglongChen/tensorflow-XNN#summary. I tried this method in this kernel: https://www.kaggle.com/shujian/single-rnn-with-5-folds-snapshot-ensemble (I don’t have any submissions left today so I am not sure about the LB score but the idea is cool). Here is a picture: ![snapshot ensemble][1]\n\n\nThat’s why blending is all you need :)\n\n\n\n [1]: https://vitalab.github.io/deep-learning/images/snapshot/1.png", "427490": "Upvoted because of the title.", "427476": "Snapshot worked for me in TGS competition....But here it is taking extra-time and requires more epoches to be really effective. So I dropped it for now. ", "427545": "Yep, that's really helpful! Thanksgiving!😊✌️👍", "429806": "Nice Kernel", "429128": "Thank you for sharing this. You have saved lot of research time and made things more clear. ", "428956": "Great!", "428306": "awesome", "428260": "Thank you for sharing in this competition.", "427585": "Thank you for all those references during competition. Really helps a lot! :D", "429597": "", "427963": "Thank you", "427866": "That`s helpful. Thanks", "442176": "This is supremely helpful! Thanks! " } }