{
  "id": 122504,
  "title": "Another Issue with Score Metric",
  "url": "/competitions/bengaliai-cv19/discussion/122504",
  "author_name": "Bojan Tunguz",
  "post_date": "2019-12-20T15:12:06.953000",
  "votes": 18,
  "comment_count": 3,
  "views": 0,
  "content": "<p>The way it's set up now seems not to <em>weigh</em> different components, but just multiply one of them by a factor of two. There ought to be a factor of 3/4 to get a <em>weighted</em> average:</p>\n\n<p>```js\nimport numpy as np\nimport sklearn.metrics</p>\n\n<p>scores = dict()\nfor component in ['consonant_diacritic', 'grapheme_root', 'vowel_diacritic']:\n    y_true_subset = solution[solution[component] == component]['target'].values\n    y_pred_subset = submission[submission[component] == component]['target'].values\n    scores[component] = sklearn.metrics.recall_score(\n        y_true_subset, y_pred_subset, average='macro')\nscores['grapheme_root'] = scores['grapheme_root'] * 2\nfinal_score = 0.75 * np.mean(list(scores.values()))\n```</p>",
  "messages": [
    {
      "id": 699545,
      "postDate": "2019-12-20T15:12:06.953Z",
      "content": "<p>The way it's set up now seems not to <em>weigh</em> different components, but just multiply one of them by a factor of two. There ought to be a factor of 3/4 to get a <em>weighted</em> average:</p>\n\n<p>```js\nimport numpy as np\nimport sklearn.metrics</p>\n\n<p>scores = dict()\nfor component in ['consonant_diacritic', 'grapheme_root', 'vowel_diacritic']:\n    y_true_subset = solution[solution[component] == component]['target'].values\n    y_pred_subset = submission[submission[component] == component]['target'].values\n    scores[component] = sklearn.metrics.recall_score(\n        y_true_subset, y_pred_subset, average='macro')\nscores['grapheme_root'] = scores['grapheme_root'] * 2\nfinal_score = 0.75 * np.mean(list(scores.values()))\n```</p>",
      "rawMarkdown": "The way it's set up now seems not to *weigh* different components, but just multiply one of them by a factor of two. There ought to be a factor of 3/4 to get a *weighted* average:\n\n```js\nimport numpy as np\nimport sklearn.metrics\n\nscores = dict()\nfor component in ['consonant_diacritic', 'grapheme_root', 'vowel_diacritic']:\n    y_true_subset = solution[solution[component] == component]['target'].values\n    y_pred_subset = submission[submission[component] == component]['target'].values\n    scores[component] = sklearn.metrics.recall_score(\n        y_true_subset, y_pred_subset, average='macro')\nscores['grapheme_root'] = scores['grapheme_root'] * 2\nfinal_score = 0.75 * np.mean(list(scores.values()))\n```",
      "votes": 17
    },
    {
      "id": 699623,
      "postDate": "2019-12-20T16:53:00.167Z",
      "content": "<p>Sorry, I didn't test the final weights against the real C# metric implementation. I've updated the evaluation page; thanks for the catch!</p>",
      "rawMarkdown": "Sorry, I didn't test the final weights against the real C# metric implementation. I've updated the evaluation page; thanks for the catch!",
      "votes": 2,
      "replies": [
        {
          "id": 735567,
          "postDate": "2020-02-03T08:14:19.097Z",
          "rawMarkdown": "",
          "isDeleted": true
        }
      ]
    },
    {
      "id": 735566,
      "postDate": "2020-02-03T08:10:31.240Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 699623,
      "author_name": "Sohier Dane",
      "author_url": "",
      "post_date": "2019-12-20T16:53:00.167000",
      "content": "<p>Sorry, I didn't test the final weights against the real C# metric implementation. I've updated the evaluation page; thanks for the catch!</p>",
      "votes": 2,
      "replies": [
        {
          "id": 735567,
          "author_name": "",
          "author_url": "",
          "post_date": "2020-02-03T08:14:19.097000",
          "content": "",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 735566,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-02-03T08:10:31.240000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "699545": "The way it's set up now seems not to *weigh* different components, but just multiply one of them by a factor of two. There ought to be a factor of 3/4 to get a *weighted* average:\n\n```js\nimport numpy as np\nimport sklearn.metrics\n\nscores = dict()\nfor component in ['consonant_diacritic', 'grapheme_root', 'vowel_diacritic']:\n    y_true_subset = solution[solution[component] == component]['target'].values\n    y_pred_subset = submission[submission[component] == component]['target'].values\n    scores[component] = sklearn.metrics.recall_score(\n        y_true_subset, y_pred_subset, average='macro')\nscores['grapheme_root'] = scores['grapheme_root'] * 2\nfinal_score = 0.75 * np.mean(list(scores.values()))\n```",
    "699623": "Sorry, I didn't test the final weights against the real C# metric implementation. I've updated the evaluation page; thanks for the catch!",
    "735566": ""
  }
}