{
  "id": 77879,
  "title": "which ensemble methods will you choose",
  "url": "/competitions/quora-insincere-questions-classification/discussion/77879",
  "author_name": "",
  "post_date": "2019-01-17T09:02:06.013272500Z",
  "votes": 14,
  "comment_count": 10,
  "views": 0,
  "content": "<p>There are two representative methods :</p>\n\n<ol>\n<li>Five folds CV. In this way, we will get a local cv score. It is very reliable to have a local cv score.</li>\n</ol>\n\n<p>like this kernel, for the cv score is low, 699-&gt;688 occured for a seed change.  : <a href=\"https://www.kaggle.com/chenshengabc/bilstm-attention-kfold-clr-extra-features-c-603dc6\">https://www.kaggle.com/chenshengabc/bilstm-attention-kfold-clr-extra-features-c-603dc6</a></p>\n\n<ol>\n<li>No valid set or only 0.001, use the magic number to ensemble different models/word embeddings. We can do a 5fold stakcing off kernel and copy the coefficient to run all data in kernel. It is a good way to make use of all data and get a better diversity. But there are no reliable cv score.</li>\n</ol>\n\n<p>like this kernel:\n<a href=\"https://www.kaggle.com/hung96ad/magic-numbers-is-all-you-need-0-696-lb\">https://www.kaggle.com/hung96ad/magic-numbers-is-all-you-need-0-696-lb</a></p>",
  "messages": [
    {
      "id": "457349",
      "postDate": "01/17/2019 09:02:06",
      "content": "<p>There are two representative methods :</p>\n\n<ol>\n<li>Five folds CV. In this way, we will get a local cv score. It is very reliable to have a local cv score.</li>\n</ol>\n\n<p>like this kernel, for the cv score is low, 699-&gt;688 occured for a seed change.  : <a href=\"https://www.kaggle.com/chenshengabc/bilstm-attention-kfold-clr-extra-features-c-603dc6\">https://www.kaggle.com/chenshengabc/bilstm-attention-kfold-clr-extra-features-c-603dc6</a></p>\n\n<ol>\n<li>No valid set or only 0.001, use the magic number to ensemble different models/word embeddings. We can do a 5fold stakcing off kernel and copy the coefficient to run all data in kernel. It is a good way to make use of all data and get a better diversity. But there are no reliable cv score.</li>\n</ol>\n\n<p>like this kernel:\n<a href=\"https://www.kaggle.com/hung96ad/magic-numbers-is-all-you-need-0-696-lb\">https://www.kaggle.com/hung96ad/magic-numbers-is-all-you-need-0-696-lb</a></p>",
      "rawMarkdown": "There are two representative methods :\n \n1. Five folds CV. In this way, we will get a local cv score. It is very reliable to have a local cv score.\n\nlike this kernel, for the cv score is low, 699-&gt;688 occured for a seed change.  : https://www.kaggle.com/chenshengabc/bilstm-attention-kfold-clr-extra-features-c-603dc6\n\n2. No valid set or only 0.001, use the magic number to ensemble different models/word embeddings. We can do a 5fold stakcing off kernel and copy the coefficient to run all data in kernel. It is a good way to make use of all data and get a better diversity. But there are no reliable cv score.\n\nlike this kernel:\nhttps://www.kaggle.com/hung96ad/magic-numbers-is-all-you-need-0-696-lb",
      "votes": null
    },
    {
      "id": "457411",
      "postDate": "01/17/2019 11:11:11",
      "content": "<p>Was about to ask this question. I am getting bad scores with ensembles. </p>",
      "rawMarkdown": "Was about to ask this question. I am getting bad scores with ensembles.",
      "votes": null
    },
    {
      "id": "457443",
      "postDate": "01/17/2019 12:43:35",
      "content": "<p>Hey Sarath, by \"bad\" you meant ensembles with <strong>different</strong> models, am I right? </p>",
      "rawMarkdown": "Hey Sarath, by \"bad\" you meant ensembles with **different** models, am I right?",
      "votes": null
    },
    {
      "id": "457480",
      "postDate": "01/17/2019 14:20:31",
      "content": "<p>The second way, I will build a model with different modules (Bi-LSTM, Capluse, CNN) and use softmax to weight output of modules. Using 5-fold, I calculate the weight of the module in each fold and get the average value as the final weight. Also, we can get the cv score.</p>",
      "rawMarkdown": "The second way, I will build a model with different modules (Bi-LSTM, Capluse, CNN) and use softmax to weight output of modules. Using 5-fold, I calculate the weight of the module in each fold and get the average value as the final weight. Also, we can get the cv score.",
      "votes": null
    },
    {
      "id": "457483",
      "postDate": "01/17/2019 14:29:46",
      "content": "<p>Yes, it's not sure to boost.</p>",
      "rawMarkdown": "Yes, it's not sure to boost.",
      "votes": null
    },
    {
      "id": "457516",
      "postDate": "01/17/2019 15:36:44",
      "content": "<p>So you run 5*3 = 15 models? </p>",
      "rawMarkdown": "So you run 5*3 = 15 models?",
      "votes": null
    },
    {
      "id": "457523",
      "postDate": "01/17/2019 16:04:16",
      "content": "<p>There is only one model, and the model contains three submodules. The output of the three modules is merged by the weight computed softmax function. The kernel only get the weight of three modules and cost too much to submit the result. The other kernel independently trains three modules and ensembles them through previously calculated weights.</p>",
      "rawMarkdown": "There is only one model, and the model contains three submodules. The output of the three modules is merged by the weight computed softmax function. The kernel only get the weight of three modules and cost too much to submit the result. The other kernel independently trains three modules and ensembles them through previously calculated weights.",
      "votes": null
    },
    {
      "id": "457739",
      "postDate": "01/18/2019 02:21:19",
      "content": "<p>@Neuron - yes. I tried my best model with 5-6 other models. Tried averaging, weighted averaging , stacking. I am not an expert in this, but nothing helped me. </p>",
      "rawMarkdown": "Neuron - yes. I tried my best model with 5-6 other models. Tried averaging, weighted averaging , stacking. I am not an expert in this, but nothing helped me.",
      "votes": null
    },
    {
      "id": "457790",
      "postDate": "01/18/2019 05:31:40",
      "content": "<p>Same as 1 model with 5 folds CV?</p>",
      "rawMarkdown": "Same as 1 model with 5 folds CV?",
      "votes": null
    },
    {
      "id": "457851",
      "postDate": "01/18/2019 07:55:55",
      "content": "<p>The first kernel with 5 folds, but the second kernel uses all data to train.</p>",
      "rawMarkdown": "The first kernel with 5 folds, but the second kernel uses all data to train.",
      "votes": null
    },
    {
      "id": "457867",
      "postDate": "01/18/2019 08:32:30",
      "content": "<p>How is your local cv score?</p>",
      "rawMarkdown": "How is your local cv score?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 457411,
      "author_name": "s4sarath",
      "author_url": "",
      "post_date": "01/17/2019 11:11:11",
      "content": "<p>Was about to ask this question. I am getting bad scores with ensembles. </p>",
      "votes": null,
      "replies": [
        {
          "id": 457443,
          "author_name": "ratthachat",
          "author_url": "",
          "post_date": "01/17/2019 12:43:35",
          "content": "<p>Hey Sarath, by \"bad\" you meant ensembles with <strong>different</strong> models, am I right? </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 457483,
          "author_name": "canming",
          "author_url": "",
          "post_date": "01/17/2019 14:29:46",
          "content": "<p>Yes, it's not sure to boost.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 457739,
          "author_name": "s4sarath",
          "author_url": "",
          "post_date": "01/18/2019 02:21:19",
          "content": "<p>@Neuron - yes. I tried my best model with 5-6 other models. Tried averaging, weighted averaging , stacking. I am not an expert in this, but nothing helped me. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 457480,
      "author_name": "salonsai",
      "author_url": "",
      "post_date": "01/17/2019 14:20:31",
      "content": "<p>The second way, I will build a model with different modules (Bi-LSTM, Capluse, CNN) and use softmax to weight output of modules. Using 5-fold, I calculate the weight of the module in each fold and get the average value as the final weight. Also, we can get the cv score.</p>",
      "votes": null,
      "replies": [
        {
          "id": 457516,
          "author_name": "baomengjiao",
          "author_url": "",
          "post_date": "01/17/2019 15:36:44",
          "content": "<p>So you run 5*3 = 15 models? </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 457523,
          "author_name": "salonsai",
          "author_url": "",
          "post_date": "01/17/2019 16:04:16",
          "content": "<p>There is only one model, and the model contains three submodules. The output of the three modules is merged by the weight computed softmax function. The kernel only get the weight of three modules and cost too much to submit the result. The other kernel independently trains three modules and ensembles them through previously calculated weights.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 457790,
          "author_name": "baomengjiao",
          "author_url": "",
          "post_date": "01/18/2019 05:31:40",
          "content": "<p>Same as 1 model with 5 folds CV?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 457851,
          "author_name": "salonsai",
          "author_url": "",
          "post_date": "01/18/2019 07:55:55",
          "content": "<p>The first kernel with 5 folds, but the second kernel uses all data to train.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 457867,
          "author_name": "baomengjiao",
          "author_url": "",
          "post_date": "01/18/2019 08:32:30",
          "content": "<p>How is your local cv score?</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "457349": "There are two representative methods :\n \n1. Five folds CV. In this way, we will get a local cv score. It is very reliable to have a local cv score.\n\nlike this kernel, for the cv score is low, 699-&gt;688 occured for a seed change.  : https://www.kaggle.com/chenshengabc/bilstm-attention-kfold-clr-extra-features-c-603dc6\n\n2. No valid set or only 0.001, use the magic number to ensemble different models/word embeddings. We can do a 5fold stakcing off kernel and copy the coefficient to run all data in kernel. It is a good way to make use of all data and get a better diversity. But there are no reliable cv score.\n\nlike this kernel:\nhttps://www.kaggle.com/hung96ad/magic-numbers-is-all-you-need-0-696-lb",
    "457411": "Was about to ask this question. I am getting bad scores with ensembles.",
    "457443": "Hey Sarath, by \"bad\" you meant ensembles with **different** models, am I right?",
    "457480": "The second way, I will build a model with different modules (Bi-LSTM, Capluse, CNN) and use softmax to weight output of modules. Using 5-fold, I calculate the weight of the module in each fold and get the average value as the final weight. Also, we can get the cv score.",
    "457483": "Yes, it's not sure to boost.",
    "457516": "So you run 5*3 = 15 models?",
    "457523": "There is only one model, and the model contains three submodules. The output of the three modules is merged by the weight computed softmax function. The kernel only get the weight of three modules and cost too much to submit the result. The other kernel independently trains three modules and ensembles them through previously calculated weights.",
    "457739": "Neuron - yes. I tried my best model with 5-6 other models. Tried averaging, weighted averaging , stacking. I am not an expert in this, but nothing helped me.",
    "457790": "Same as 1 model with 5 folds CV?",
    "457851": "The first kernel with 5 folds, but the second kernel uses all data to train.",
    "457867": "How is your local cv score?"
  },
  "source": "meta"
}