{
  "id": 90862,
  "title": "Finding threshold class-wise.",
  "url": "/competitions/imet-2019-fgvc6/discussion/90862",
  "author_name": "",
  "post_date": "2019-04-28T11:52:04.904713600Z",
  "votes": 7,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hello everyone.</p>\n\n<p>One of the problems in this competition is finding optimal threshold for class.\nThe simplest way is to use one value for all classes. However, in this particular competition it is not the best way to do because a lot of classes are very rare which leads to low probabilities and biased threshold.</p>\n\n<p>One way to overstep it is to find threshold for each particular class and use it later. I am not sure that it is the most optimal way to do that (it can lead to overfit and optimizing fbeta for each label ~ optimal fbeta for multilabel) but it increased my score for ~0.005</p>\n\n<p>Code snippet</p>\n\n<p>```\ndef f2_threshold_labelwise(y_true, probs, tvals=np.linspace(0.1, 0.9, 100), disable_tqdm=False, n_jobs=4):\n    def _process_single(class_idx):\n        class_probs = probs[:, class_idx]\n        labels_true = y_true[:, class_idx]</p>\n\n<pre><code>    f2_scores = [fbeta_score(labels_true, class_probs &amp;gt;= t, 2) for t in tvals]\n    return tvals[np.argmax(f2_scores)]\n\nreturn Parallel(n_jobs=n_jobs)(delayed(_process_single)(idx) for idx in tqdm_notebook(range(y_true.shape[1]), disable=disable_tqdm, leave=False))\n</code></pre>\n\n<p>```</p>",
  "messages": [
    {
      "id": "524278",
      "postDate": "04/28/2019 11:52:04",
      "content": "<p>Hello everyone.</p>\n\n<p>One of the problems in this competition is finding optimal threshold for class.\nThe simplest way is to use one value for all classes. However, in this particular competition it is not the best way to do because a lot of classes are very rare which leads to low probabilities and biased threshold.</p>\n\n<p>One way to overstep it is to find threshold for each particular class and use it later. I am not sure that it is the most optimal way to do that (it can lead to overfit and optimizing fbeta for each label ~ optimal fbeta for multilabel) but it increased my score for ~0.005</p>\n\n<p>Code snippet</p>\n\n<p>```\ndef f2_threshold_labelwise(y_true, probs, tvals=np.linspace(0.1, 0.9, 100), disable_tqdm=False, n_jobs=4):\n    def _process_single(class_idx):\n        class_probs = probs[:, class_idx]\n        labels_true = y_true[:, class_idx]</p>\n\n<pre><code>    f2_scores = [fbeta_score(labels_true, class_probs &amp;gt;= t, 2) for t in tvals]\n    return tvals[np.argmax(f2_scores)]\n\nreturn Parallel(n_jobs=n_jobs)(delayed(_process_single)(idx) for idx in tqdm_notebook(range(y_true.shape[1]), disable=disable_tqdm, leave=False))\n</code></pre>\n\n<p>```</p>",
      "rawMarkdown": "Hello everyone.\n\nOne of the problems in this competition is finding optimal threshold for class.\nThe simplest way is to use one value for all classes. However, in this particular competition it is not the best way to do because a lot of classes are very rare which leads to low probabilities and biased threshold.\n\nOne way to overstep it is to find threshold for each particular class and use it later. I am not sure that it is the most optimal way to do that (it can lead to overfit and optimizing fbeta for each label ~ optimal fbeta for multilabel) but it increased my score for ~0.005\n\nCode snippet\n\n```\ndef f2_threshold_labelwise(y_true, probs, tvals=np.linspace(0.1, 0.9, 100), disable_tqdm=False, n_jobs=4):\n    def _process_single(class_idx):\n        class_probs = probs[:, class_idx]\n        labels_true = y_true[:, class_idx]\n        \n        f2_scores = [fbeta_score(labels_true, class_probs &gt;= t, 2) for t in tvals]\n        return tvals[np.argmax(f2_scores)]\n    \n    return Parallel(n_jobs=n_jobs)(delayed(_process_single)(idx) for idx in tqdm_notebook(range(y_true.shape[1]), disable=disable_tqdm, leave=False))\n```",
      "votes": null
    },
    {
      "id": "524318",
      "postDate": "04/28/2019 13:45:06",
      "content": "<p>Thanks for sharing! I have tried this idea but only for top 5 classes that have most samples, which didn't boost my score. IMO, for classes that only have few samples, this method could lead to overfitting.  But it's reasonable that this could boost score, for those classes only have few samples they could be missed in training set even with k-fold cv then the threshold should be very different.</p>",
      "rawMarkdown": "Thanks for sharing! I have tried this idea but only for top 5 classes that have most samples, which didn't boost my score. IMO, for classes that only have few samples, this method could lead to overfitting.  But it's reasonable that this could boost score, for those classes only have few samples they could be missed in training set even with k-fold cv then the threshold should be very different.",
      "votes": null
    },
    {
      "id": "524539",
      "postDate": "04/29/2019 04:21:59",
      "content": "<p>Thanks for sharing! But it may increase the risk of overfitting.</p>",
      "rawMarkdown": "Thanks for sharing! But it may increase the risk of overfitting.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 524318,
      "author_name": "jionie",
      "author_url": "",
      "post_date": "04/28/2019 13:45:06",
      "content": "<p>Thanks for sharing! I have tried this idea but only for top 5 classes that have most samples, which didn't boost my score. IMO, for classes that only have few samples, this method could lead to overfitting.  But it's reasonable that this could boost score, for those classes only have few samples they could be missed in training set even with k-fold cv then the threshold should be very different.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 524539,
      "author_name": "seefun",
      "author_url": "",
      "post_date": "04/29/2019 04:21:59",
      "content": "<p>Thanks for sharing! But it may increase the risk of overfitting.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "524278": "Hello everyone.\n\nOne of the problems in this competition is finding optimal threshold for class.\nThe simplest way is to use one value for all classes. However, in this particular competition it is not the best way to do because a lot of classes are very rare which leads to low probabilities and biased threshold.\n\nOne way to overstep it is to find threshold for each particular class and use it later. I am not sure that it is the most optimal way to do that (it can lead to overfit and optimizing fbeta for each label ~ optimal fbeta for multilabel) but it increased my score for ~0.005\n\nCode snippet\n\n```\ndef f2_threshold_labelwise(y_true, probs, tvals=np.linspace(0.1, 0.9, 100), disable_tqdm=False, n_jobs=4):\n    def _process_single(class_idx):\n        class_probs = probs[:, class_idx]\n        labels_true = y_true[:, class_idx]\n        \n        f2_scores = [fbeta_score(labels_true, class_probs &gt;= t, 2) for t in tvals]\n        return tvals[np.argmax(f2_scores)]\n    \n    return Parallel(n_jobs=n_jobs)(delayed(_process_single)(idx) for idx in tqdm_notebook(range(y_true.shape[1]), disable=disable_tqdm, leave=False))\n```",
    "524318": "Thanks for sharing! I have tried this idea but only for top 5 classes that have most samples, which didn't boost my score. IMO, for classes that only have few samples, this method could lead to overfitting.  But it's reasonable that this could boost score, for those classes only have few samples they could be missed in training set even with k-fold cv then the threshold should be very different.",
    "524539": "Thanks for sharing! But it may increase the risk of overfitting."
  },
  "source": "meta"
}