{
  "id": 318702,
  "title": "A problem when I train model with this data.",
  "url": "/competitions/snakeclef2022/discussion/318702",
  "author_name": "",
  "post_date": "2022-04-13T14:29:12.904072500Z",
  "votes": null,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I divide one-tenth of the dataset as a validation set and the rest as a training set, I use ResNext50 model with ImageNet pre-training weights for training, and I can get an accuracy of 75.1933 on the validation set, but the corresponding f1_score is only 55.73. Finally, the score after submission is only 46.034. <br>\nWhat is the reason for this?</p>",
  "messages": [
    {
      "id": "1754304",
      "postDate": "04/13/2022 14:29:12",
      "content": "<p>I divide one-tenth of the dataset as a validation set and the rest as a training set, I use ResNext50 model with ImageNet pre-training weights for training, and I can get an accuracy of 75.1933 on the validation set, but the corresponding f1_score is only 55.73. Finally, the score after submission is only 46.034. <br>\nWhat is the reason for this?</p>",
      "rawMarkdown": "I divide one-tenth of the dataset as a validation set and the rest as a training set, I use ResNext50 model with ImageNet pre-training weights for training, and I can get an accuracy of 75.1933 on the validation set, but the corresponding f1_score is only 55.73. Finally, the score after submission is only 46.034. \nWhat is the reason for this?",
      "votes": null
    },
    {
      "id": "1754318",
      "postDate": "04/13/2022 14:40:27",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/hyyyrwang\" target=\"_blank\">@hyyyrwang</a>,</p>\n<p>it is very hard to say based on your description. First of all, Accuracy is a different metric than F1-macro. To have an accuracy of 75% and an F1 of 56% is expected.</p>\n<p>Anyway, I would suspect one of the following:</p>\n<ul>\n<li>Your split does not respect Observations, thus, images of the same individual (understand visually similar pictures) might occur in both, the training and validation set.</li>\n<li>Your split does not respect species distribution.</li>\n<li>Check that your F1 is the macro averaged one.</li>\n</ul>\n<p>Lukas</p>",
      "rawMarkdown": "Hi @hyyyrwang,\n\nit is very hard to say based on your description. First of all, Accuracy is a different metric than F1-macro. To have an accuracy of 75% and an F1 of 56% is expected.\n\nAnyway, I would suspect one of the following:\n- Your split does not respect Observations, thus, images of the same individual (understand visually similar pictures) might occur in both, the training and validation set.\n- Your split does not respect species distribution.\n- Check that your F1 is the macro averaged one.\n\nLukas",
      "votes": null
    },
    {
      "id": "1759222",
      "postDate": "04/18/2022 12:51:42",
      "content": "<p>Hi, different countries are considered when calculating the macro-F1 in the test dataset, but the one you are using maybe just focuss on an average of species?😊</p>",
      "rawMarkdown": "Hi, different countries are considered when calculating the macro-F1 in the test dataset, but the one you are using maybe just focuss on an average of species?😊",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1754318,
      "author_name": "picekl",
      "author_url": "",
      "post_date": "04/13/2022 14:40:27",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/hyyyrwang\" target=\"_blank\">@hyyyrwang</a>,</p>\n<p>it is very hard to say based on your description. First of all, Accuracy is a different metric than F1-macro. To have an accuracy of 75% and an F1 of 56% is expected.</p>\n<p>Anyway, I would suspect one of the following:</p>\n<ul>\n<li>Your split does not respect Observations, thus, images of the same individual (understand visually similar pictures) might occur in both, the training and validation set.</li>\n<li>Your split does not respect species distribution.</li>\n<li>Check that your F1 is the macro averaged one.</li>\n</ul>\n<p>Lukas</p>",
      "votes": null,
      "replies": [
        {
          "id": 1759222,
          "author_name": "w3579628328",
          "author_url": "",
          "post_date": "04/18/2022 12:51:42",
          "content": "<p>Hi, different countries are considered when calculating the macro-F1 in the test dataset, but the one you are using maybe just focuss on an average of species?😊</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1754304": "I divide one-tenth of the dataset as a validation set and the rest as a training set, I use ResNext50 model with ImageNet pre-training weights for training, and I can get an accuracy of 75.1933 on the validation set, but the corresponding f1_score is only 55.73. Finally, the score after submission is only 46.034. \nWhat is the reason for this?",
    "1754318": "Hi @hyyyrwang,\n\nit is very hard to say based on your description. First of all, Accuracy is a different metric than F1-macro. To have an accuracy of 75% and an F1 of 56% is expected.\n\nAnyway, I would suspect one of the following:\n- Your split does not respect Observations, thus, images of the same individual (understand visually similar pictures) might occur in both, the training and validation set.\n- Your split does not respect species distribution.\n- Check that your F1 is the macro averaged one.\n\nLukas",
    "1759222": "Hi, different countries are considered when calculating the macro-F1 in the test dataset, but the one you are using maybe just focuss on an average of species?😊"
  },
  "source": "meta"
}