{
  "id": 181298,
  "title": "Simple threshold optimizer",
  "url": "/competitions/birdsong-recognition/discussion/181298",
  "author_name": "",
  "post_date": "2020-09-08T11:05:33.727065Z",
  "votes": 2,
  "comment_count": 7,
  "views": 0,
  "content": "<p>I implemented a simple threshold optimizer to the F1 metric validation, to use with a validation set, didn't get any better, landed on the same values/score, but maybe will work better for someone else.</p>\n<pre><code>def row_wise_f1_score_micro_numpy(y_true, y_pred, threshold=0.5, count=5):\n\n    @author shonenkov \n\n    y_true - 2d npy vector with gt\n    y_pred - 2d npy vector with prediction\n    threshold - for round labels\n    count - number of preds (used sorting by confidence)\n    \"\"\"\n    def meth_agn_v2(x, threshold):\n        idx, = np.where(x &gt; threshold)\n        return idx[np.argsort(x[idx])[::-1]]\n\n    F1 = []\n    for preds, trues in zip(y_pred, y_true):\n        TP, FN, FP = 0, 0, 0\n        preds = meth_agn_v2(preds, threshold)[:count]\n        trues = meth_agn_v2(trues, threshold)\n        for true in trues:\n            if true in preds:\n                TP += 1\n            else:\n                FN += 1\n        for pred in preds:\n            if pred not in trues:\n                FP += 1\n        F1.append(2*TP / (2*TP + FN + FP))\n    return np.mean(F1)\n</code></pre>\n<pre><code>def threshopt_rw(pred,target, runs):\n    besthreshscore = 0\n    bestthresh = 0\n    for ii in range(runs):\n      thresh = random.random()\n      try:\n        threshscore = row_wise_f1_score_micro_numpy(pred.cpu().numpy(), target.cpu().numpy(),threshold=thresh)\n      except:\n        threshscore = 0\n      # print('zero')\n      if threshscore &gt; besthreshscore:        \n          besthreshscore = threshscore\n          bestthresh = thresh\n      ii   \n    #print('testno.',ii,'best thresh', thresh, \"Score\", threshscore)\n    return bestthresh, besthreshscore`\n</code></pre>",
  "messages": [
    {
      "id": "1002703",
      "postDate": "09/08/2020 11:05:33",
      "content": "<p>I implemented a simple threshold optimizer to the F1 metric validation, to use with a validation set, didn't get any better, landed on the same values/score, but maybe will work better for someone else.</p>\n<pre><code>def row_wise_f1_score_micro_numpy(y_true, y_pred, threshold=0.5, count=5):\n\n    @author shonenkov \n\n    y_true - 2d npy vector with gt\n    y_pred - 2d npy vector with prediction\n    threshold - for round labels\n    count - number of preds (used sorting by confidence)\n    \"\"\"\n    def meth_agn_v2(x, threshold):\n        idx, = np.where(x &gt; threshold)\n        return idx[np.argsort(x[idx])[::-1]]\n\n    F1 = []\n    for preds, trues in zip(y_pred, y_true):\n        TP, FN, FP = 0, 0, 0\n        preds = meth_agn_v2(preds, threshold)[:count]\n        trues = meth_agn_v2(trues, threshold)\n        for true in trues:\n            if true in preds:\n                TP += 1\n            else:\n                FN += 1\n        for pred in preds:\n            if pred not in trues:\n                FP += 1\n        F1.append(2*TP / (2*TP + FN + FP))\n    return np.mean(F1)\n</code></pre>\n<pre><code>def threshopt_rw(pred,target, runs):\n    besthreshscore = 0\n    bestthresh = 0\n    for ii in range(runs):\n      thresh = random.random()\n      try:\n        threshscore = row_wise_f1_score_micro_numpy(pred.cpu().numpy(), target.cpu().numpy(),threshold=thresh)\n      except:\n        threshscore = 0\n      # print('zero')\n      if threshscore &gt; besthreshscore:        \n          besthreshscore = threshscore\n          bestthresh = thresh\n      ii   \n    #print('testno.',ii,'best thresh', thresh, \"Score\", threshscore)\n    return bestthresh, besthreshscore`\n</code></pre>",
      "rawMarkdown": "I implemented a simple threshold optimizer to the F1 metric validation, to use with a validation set, didn't get any better, landed on the same values/score, but maybe will work better for someone else.\n\n```\ndef row_wise_f1_score_micro_numpy(y_true, y_pred, threshold=0.5, count=5):\n   \n    @author shonenkov \n    \n    y_true - 2d npy vector with gt\n    y_pred - 2d npy vector with prediction\n    threshold - for round labels\n    count - number of preds (used sorting by confidence)\n    \"\"\"\n    def meth_agn_v2(x, threshold):\n        idx, = np.where(x > threshold)\n        return idx[np.argsort(x[idx])[::-1]]\n\n    F1 = []\n    for preds, trues in zip(y_pred, y_true):\n        TP, FN, FP = 0, 0, 0\n        preds = meth_agn_v2(preds, threshold)[:count]\n        trues = meth_agn_v2(trues, threshold)\n        for true in trues:\n            if true in preds:\n                TP += 1\n            else:\n                FN += 1\n        for pred in preds:\n            if pred not in trues:\n                FP += 1\n        F1.append(2*TP / (2*TP + FN + FP))\n    return np.mean(F1)\n```\n\n\n```\ndef threshopt_rw(pred,target, runs):\n    besthreshscore = 0\n    bestthresh = 0\n    for ii in range(runs):\n      thresh = random.random()\n      try:\n        threshscore = row_wise_f1_score_micro_numpy(pred.cpu().numpy(), target.cpu().numpy(),threshold=thresh)\n      except:\n        threshscore = 0\n      # print('zero')\n      if threshscore > besthreshscore:        \n          besthreshscore = threshscore\n          bestthresh = thresh\n      ii   \n    #print('testno.',ii,'best thresh', thresh, \"Score\", threshscore)\n    return bestthresh, besthreshscore`\n```",
      "votes": null
    },
    {
      "id": "1002725",
      "postDate": "09/08/2020 11:32:30",
      "content": "<p>thanks for your sharing, my current validation metrics is a joke…..<br>\nWill try yours and see if any better</p>",
      "rawMarkdown": "thanks for your sharing, my current validation metrics is a joke.....\nWill try yours and see if any better",
      "votes": null
    },
    {
      "id": "1003087",
      "postDate": "09/08/2020 16:46:05",
      "content": "<p>Thank you for sharing, find the best range of threshold on this competition is a big deal. <br>\nAnyways, I hope you can get the best structure. 🤘</p>",
      "rawMarkdown": "Thank you for sharing, find the best range of threshold on this competition is a big deal. \nAnyways, I hope you can get the best structure. 🤘",
      "votes": null
    },
    {
      "id": "1003164",
      "postDate": "09/08/2020 17:42:50",
      "content": "<p>scikit learn f1_score with average='samples' does the job of <code>row_wise_f1_score_micro_numpy</code>.  </p>",
      "rawMarkdown": "scikit learn f1_score with average='samples' does the job of `row_wise_f1_score_micro_numpy`.",
      "votes": null
    },
    {
      "id": "1003233",
      "postDate": "09/08/2020 18:54:03",
      "content": "<p>yes they give the same score in training/validation, but I think this code can handle the float values the loss gives, tuning within the float value, and the threshold tuning better than scikit learn f1_score with average='samples'  which can't handle the same.</p>",
      "rawMarkdown": "yes they give the same score in training/validation, but I think this code can handle the float values the loss gives, tuning within the float value, and the threshold tuning better than scikit learn f1_score with average='samples'  which can't handle the same.",
      "votes": null
    },
    {
      "id": "1003309",
      "postDate": "09/08/2020 20:11:50",
      "content": "<p>I have tried.</p>",
      "rawMarkdown": "I have tried.",
      "votes": null
    },
    {
      "id": "1008157",
      "postDate": "09/12/2020 19:30:52",
      "content": "<p>Good one, thank you.</p>",
      "rawMarkdown": "Good one, thank you.",
      "votes": null
    },
    {
      "id": "1008158",
      "postDate": "09/12/2020 19:31:16",
      "content": "<p>What is your metric? F1-Score?</p>",
      "rawMarkdown": "What is your metric? F1-Score?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1002725,
      "author_name": "fiyeroleung",
      "author_url": "",
      "post_date": "09/08/2020 11:32:30",
      "content": "<p>thanks for your sharing, my current validation metrics is a joke…..<br>\nWill try yours and see if any better</p>",
      "votes": null,
      "replies": [
        {
          "id": 1008158,
          "author_name": "aliabdin1",
          "author_url": "",
          "post_date": "09/12/2020 19:31:16",
          "content": "<p>What is your metric? F1-Score?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1003087,
      "author_name": "",
      "author_url": "",
      "post_date": "09/08/2020 16:46:05",
      "content": "<p>Thank you for sharing, find the best range of threshold on this competition is a big deal. <br>\nAnyways, I hope you can get the best structure. 🤘</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1003164,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "09/08/2020 17:42:50",
      "content": "<p>scikit learn f1_score with average='samples' does the job of <code>row_wise_f1_score_micro_numpy</code>.  </p>",
      "votes": null,
      "replies": [
        {
          "id": 1003233,
          "author_name": "kirderf",
          "author_url": "",
          "post_date": "09/08/2020 18:54:03",
          "content": "<p>yes they give the same score in training/validation, but I think this code can handle the float values the loss gives, tuning within the float value, and the threshold tuning better than scikit learn f1_score with average='samples'  which can't handle the same.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1003309,
          "author_name": "cpmpml",
          "author_url": "",
          "post_date": "09/08/2020 20:11:50",
          "content": "<p>I have tried.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1008157,
      "author_name": "aliabdin1",
      "author_url": "",
      "post_date": "09/12/2020 19:30:52",
      "content": "<p>Good one, thank you.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1002703": "I implemented a simple threshold optimizer to the F1 metric validation, to use with a validation set, didn't get any better, landed on the same values/score, but maybe will work better for someone else.\n\n```\ndef row_wise_f1_score_micro_numpy(y_true, y_pred, threshold=0.5, count=5):\n   \n    @author shonenkov \n    \n    y_true - 2d npy vector with gt\n    y_pred - 2d npy vector with prediction\n    threshold - for round labels\n    count - number of preds (used sorting by confidence)\n    \"\"\"\n    def meth_agn_v2(x, threshold):\n        idx, = np.where(x > threshold)\n        return idx[np.argsort(x[idx])[::-1]]\n\n    F1 = []\n    for preds, trues in zip(y_pred, y_true):\n        TP, FN, FP = 0, 0, 0\n        preds = meth_agn_v2(preds, threshold)[:count]\n        trues = meth_agn_v2(trues, threshold)\n        for true in trues:\n            if true in preds:\n                TP += 1\n            else:\n                FN += 1\n        for pred in preds:\n            if pred not in trues:\n                FP += 1\n        F1.append(2*TP / (2*TP + FN + FP))\n    return np.mean(F1)\n```\n\n\n```\ndef threshopt_rw(pred,target, runs):\n    besthreshscore = 0\n    bestthresh = 0\n    for ii in range(runs):\n      thresh = random.random()\n      try:\n        threshscore = row_wise_f1_score_micro_numpy(pred.cpu().numpy(), target.cpu().numpy(),threshold=thresh)\n      except:\n        threshscore = 0\n      # print('zero')\n      if threshscore > besthreshscore:        \n          besthreshscore = threshscore\n          bestthresh = thresh\n      ii   \n    #print('testno.',ii,'best thresh', thresh, \"Score\", threshscore)\n    return bestthresh, besthreshscore`\n```",
    "1002725": "thanks for your sharing, my current validation metrics is a joke.....\nWill try yours and see if any better",
    "1003087": "Thank you for sharing, find the best range of threshold on this competition is a big deal. \nAnyways, I hope you can get the best structure. 🤘",
    "1003164": "scikit learn f1_score with average='samples' does the job of `row_wise_f1_score_micro_numpy`.",
    "1003233": "yes they give the same score in training/validation, but I think this code can handle the float values the loss gives, tuning within the float value, and the threshold tuning better than scikit learn f1_score with average='samples'  which can't handle the same.",
    "1003309": "I have tried.",
    "1008157": "Good one, thank you.",
    "1008158": "What is your metric? F1-Score?"
  },
  "source": "meta"
}