{
  "id": 120061,
  "title": "My implementation of competition metric",
  "url": "/competitions/tensorflow2-question-answering/discussion/120061",
  "author_name": "",
  "post_date": "2019-12-03T11:55:29.295605700Z",
  "votes": 36,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Since there is  a lot of uncertainty related to the competition metric. I shared my implementation here <a href=\"https://www.kaggle.com/c/tensorflow2-question-answering/discussion/120030#686602\">https://www.kaggle.com/c/tensorflow2-question-answering/discussion/120030#686602</a></p>\n\n<p>My implementation matches LB score and on my validation set, for various models ranging from 0.5 - 0.73. It also reflects effects of putting blanks for predictions with low confidence.</p>",
  "messages": [
    {
      "id": "686675",
      "postDate": "12/03/2019 11:55:29",
      "content": "<p>Since there is  a lot of uncertainty related to the competition metric. I shared my implementation here <a href=\"https://www.kaggle.com/c/tensorflow2-question-answering/discussion/120030#686602\">https://www.kaggle.com/c/tensorflow2-question-answering/discussion/120030#686602</a></p>\n\n<p>My implementation matches LB score and on my validation set, for various models ranging from 0.5 - 0.73. It also reflects effects of putting blanks for predictions with low confidence.</p>",
      "rawMarkdown": "Since there is  a lot of uncertainty related to the competition metric. I shared my implementation here https://www.kaggle.com/c/tensorflow2-question-answering/discussion/120030#686602\n\nMy implementation matches LB score and on my validation set, for various models ranging from 0.5 - 0.73. It also reflects effects of putting blanks for predictions with low confidence.",
      "votes": null
    },
    {
      "id": "686707",
      "postDate": "12/03/2019 12:29:21",
      "content": "<p>Thank you! So nice  to have a metric.</p>",
      "rawMarkdown": "Thank you! So nice  to have a metric.",
      "votes": null
    },
    {
      "id": "686711",
      "postDate": "12/03/2019 12:40:14",
      "content": "<p>Thank you for sharing!</p>",
      "rawMarkdown": "Thank you for sharing!",
      "votes": null
    },
    {
      "id": "691933",
      "postDate": "12/10/2019 17:02:36",
      "content": "<p>Now thanks to top kaggler <a href=\"/christofhenkel\">@christofhenkel</a> and <a href=\"/boliu0\">@boliu0</a> , the LB metric is updated.\nI want to check my understanding about new metric.</p>\n\n<p>From <a href=\"https://www.kaggle.com/c/tensorflow2-question-answering/discussion/120030#686602\">old metric</a>, we have to care False Positive case (= label has no gold answer but predict has answer).\nThen <a href=\"/christofhenkel\">@christofhenkel</a> 's great metric need to change like below:</p>\n\n<p>```\ndef in_shorts(row):\n    return row['short_answer'] in row['short_answers']</p>\n\n<p>from sklearn.metrics import f1_score</p>\n\n<p>def get_f1(answer_df,label_df):\n    short_label =  (label_df['short_answers'] != '').astype(int)\n    long_label =  (label_df['long_answer'] != '').astype(int)</p>\n\n<pre><code>long_predict = np.zeros(answer_df.shape[0])\nlong_predict[(answer_df['long_answer'] == label_df['long_answer']) &amp; (answer_df['long_answer'] != '')] = 1\nlong_predict[(label_df['long_answer'] == '') &amp; (answer_df['long_answer'] != '')] = 1  # false positive\n\nshort_predict = np.zeros(answer_df.shape[0])\nshort_predict[(label_df['short_answers'] == '') &amp; (answer_df['short_answer'] != '')] = 1  # false positive\na = pd.concat([answer_df[['short_answer']],label_df[['short_answers']]], axis = 1)\na['short_answers'] = a['short_answers'].apply(lambda x: x.split())\nshort_predict[a.apply(lambda x: in_shorts(x), axis = 1) &amp; (a['short_answer'] != '')] = 1\n\nlong_f1 = f1_score(long_label.values,long_predict)\nshort_f1 = f1_score(short_label.values,short_predict)\nmicro_f1 = f1_score(np.concatenate([long_label,short_label]),np.concatenate([long_predict,short_predict]))\nreturn micro_f1, long_f1, short_f1\n</code></pre>\n\n<p>```</p>\n\n<p>Please correct me if I'm wrong 🙇 </p>",
      "rawMarkdown": "Now thanks to top kaggler @christofhenkel and @boliu0 , the LB metric is updated.\nI want to check my understanding about new metric.\n\nFrom [old metric](https://www.kaggle.com/c/tensorflow2-question-answering/discussion/120030#686602), we have to care False Positive case (= label has no gold answer but predict has answer).\nThen @christofhenkel 's great metric need to change like below:\n\n```\ndef in_shorts(row):\n    return row['short_answer'] in row['short_answers']\n\n\nfrom sklearn.metrics import f1_score\n\n\ndef get_f1(answer_df,label_df):\n    short_label =  (label_df['short_answers'] != '').astype(int)\n    long_label =  (label_df['long_answer'] != '').astype(int)\n\n    long_predict = np.zeros(answer_df.shape[0])\n    long_predict[(answer_df['long_answer'] == label_df['long_answer']) &amp; (answer_df['long_answer'] != '')] = 1\n    long_predict[(label_df['long_answer'] == '') &amp; (answer_df['long_answer'] != '')] = 1  # false positive\n\n    short_predict = np.zeros(answer_df.shape[0])\n    short_predict[(label_df['short_answers'] == '') &amp; (answer_df['short_answer'] != '')] = 1  # false positive\n    a = pd.concat([answer_df[['short_answer']],label_df[['short_answers']]], axis = 1)\n    a['short_answers'] = a['short_answers'].apply(lambda x: x.split())\n    short_predict[a.apply(lambda x: in_shorts(x), axis = 1) &amp; (a['short_answer'] != '')] = 1\n\n    long_f1 = f1_score(long_label.values,long_predict)\n    short_f1 = f1_score(short_label.values,short_predict)\n    micro_f1 = f1_score(np.concatenate([long_label,short_label]),np.concatenate([long_predict,short_predict]))\n    return micro_f1, long_f1, short_f1\n```\n\nPlease correct me if I'm wrong 🙇",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 686707,
      "author_name": "yihdarshieh",
      "author_url": "",
      "post_date": "12/03/2019 12:29:21",
      "content": "<p>Thank you! So nice  to have a metric.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 686711,
      "author_name": "kentaronakanishi",
      "author_url": "",
      "post_date": "12/03/2019 12:40:14",
      "content": "<p>Thank you for sharing!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 691933,
      "author_name": "kentaronakanishi",
      "author_url": "",
      "post_date": "12/10/2019 17:02:36",
      "content": "<p>Now thanks to top kaggler <a href=\"/christofhenkel\">@christofhenkel</a> and <a href=\"/boliu0\">@boliu0</a> , the LB metric is updated.\nI want to check my understanding about new metric.</p>\n\n<p>From <a href=\"https://www.kaggle.com/c/tensorflow2-question-answering/discussion/120030#686602\">old metric</a>, we have to care False Positive case (= label has no gold answer but predict has answer).\nThen <a href=\"/christofhenkel\">@christofhenkel</a> 's great metric need to change like below:</p>\n\n<p>```\ndef in_shorts(row):\n    return row['short_answer'] in row['short_answers']</p>\n\n<p>from sklearn.metrics import f1_score</p>\n\n<p>def get_f1(answer_df,label_df):\n    short_label =  (label_df['short_answers'] != '').astype(int)\n    long_label =  (label_df['long_answer'] != '').astype(int)</p>\n\n<pre><code>long_predict = np.zeros(answer_df.shape[0])\nlong_predict[(answer_df['long_answer'] == label_df['long_answer']) &amp; (answer_df['long_answer'] != '')] = 1\nlong_predict[(label_df['long_answer'] == '') &amp; (answer_df['long_answer'] != '')] = 1  # false positive\n\nshort_predict = np.zeros(answer_df.shape[0])\nshort_predict[(label_df['short_answers'] == '') &amp; (answer_df['short_answer'] != '')] = 1  # false positive\na = pd.concat([answer_df[['short_answer']],label_df[['short_answers']]], axis = 1)\na['short_answers'] = a['short_answers'].apply(lambda x: x.split())\nshort_predict[a.apply(lambda x: in_shorts(x), axis = 1) &amp; (a['short_answer'] != '')] = 1\n\nlong_f1 = f1_score(long_label.values,long_predict)\nshort_f1 = f1_score(short_label.values,short_predict)\nmicro_f1 = f1_score(np.concatenate([long_label,short_label]),np.concatenate([long_predict,short_predict]))\nreturn micro_f1, long_f1, short_f1\n</code></pre>\n\n<p>```</p>\n\n<p>Please correct me if I'm wrong 🙇 </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "686675": "Since there is  a lot of uncertainty related to the competition metric. I shared my implementation here https://www.kaggle.com/c/tensorflow2-question-answering/discussion/120030#686602\n\nMy implementation matches LB score and on my validation set, for various models ranging from 0.5 - 0.73. It also reflects effects of putting blanks for predictions with low confidence.",
    "686707": "Thank you! So nice  to have a metric.",
    "686711": "Thank you for sharing!",
    "691933": "Now thanks to top kaggler @christofhenkel and @boliu0 , the LB metric is updated.\nI want to check my understanding about new metric.\n\nFrom [old metric](https://www.kaggle.com/c/tensorflow2-question-answering/discussion/120030#686602), we have to care False Positive case (= label has no gold answer but predict has answer).\nThen @christofhenkel 's great metric need to change like below:\n\n```\ndef in_shorts(row):\n    return row['short_answer'] in row['short_answers']\n\n\nfrom sklearn.metrics import f1_score\n\n\ndef get_f1(answer_df,label_df):\n    short_label =  (label_df['short_answers'] != '').astype(int)\n    long_label =  (label_df['long_answer'] != '').astype(int)\n\n    long_predict = np.zeros(answer_df.shape[0])\n    long_predict[(answer_df['long_answer'] == label_df['long_answer']) &amp; (answer_df['long_answer'] != '')] = 1\n    long_predict[(label_df['long_answer'] == '') &amp; (answer_df['long_answer'] != '')] = 1  # false positive\n\n    short_predict = np.zeros(answer_df.shape[0])\n    short_predict[(label_df['short_answers'] == '') &amp; (answer_df['short_answer'] != '')] = 1  # false positive\n    a = pd.concat([answer_df[['short_answer']],label_df[['short_answers']]], axis = 1)\n    a['short_answers'] = a['short_answers'].apply(lambda x: x.split())\n    short_predict[a.apply(lambda x: in_shorts(x), axis = 1) &amp; (a['short_answer'] != '')] = 1\n\n    long_f1 = f1_score(long_label.values,long_predict)\n    short_f1 = f1_score(short_label.values,short_predict)\n    micro_f1 = f1_score(np.concatenate([long_label,short_label]),np.concatenate([long_predict,short_predict]))\n    return micro_f1, long_f1, short_f1\n```\n\nPlease correct me if I'm wrong 🙇"
  },
  "source": "meta"
}