{
  "id": 317310,
  "title": "Overfitting - Am I alone?",
  "url": "/competitions/happy-whale-and-dolphin/discussion/317310",
  "author_name": "gergopool",
  "post_date": "2022-04-06T12:06:53.644000",
  "votes": 3,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Hi everyone,</p>\n<p>I am still trying to make my first submission, and ran into some issues I would like to share. My solution relies on a cropped dataset and on <a href=\"https://www.kaggle.com/code/clemchris/pytorch-backfin-convnext-arcface\" target=\"_blank\">this code</a>.</p>\n<ol>\n<li><p>Did anyone experience a massive overfit? Surprisingly, I can achieve 100% train accuracy in 20-30 epochs while the validation accuracy stays around 0%. In terms of loss it means a 0.01 vs 20 difference. I cannot even blame weak augmentations, because I take random crops. </p></li>\n<li><p>Why do self-supervised methods not help? I started with self-supervised pre-training and what I experienced that neither of my approaches actually helped to kick off the training. The resnet50-imagenet weights converge equally fast as a 50-epoch self-trained model, initialized with the same downloaded weights. This is just very surprising to me, did anyone make similar experiments?</p></li>\n</ol>\n<p>Thanks.</p>",
  "messages": [
    {
      "id": 1747135,
      "postDate": "2022-04-06T12:06:53.643Z",
      "content": "<p>Hi everyone,</p>\n<p>I am still trying to make my first submission, and ran into some issues I would like to share. My solution relies on a cropped dataset and on <a href=\"https://www.kaggle.com/code/clemchris/pytorch-backfin-convnext-arcface\" target=\"_blank\">this code</a>.</p>\n<ol>\n<li><p>Did anyone experience a massive overfit? Surprisingly, I can achieve 100% train accuracy in 20-30 epochs while the validation accuracy stays around 0%. In terms of loss it means a 0.01 vs 20 difference. I cannot even blame weak augmentations, because I take random crops. </p></li>\n<li><p>Why do self-supervised methods not help? I started with self-supervised pre-training and what I experienced that neither of my approaches actually helped to kick off the training. The resnet50-imagenet weights converge equally fast as a 50-epoch self-trained model, initialized with the same downloaded weights. This is just very surprising to me, did anyone make similar experiments?</p></li>\n</ol>\n<p>Thanks.</p>",
      "rawMarkdown": "Hi everyone,\n\nI am still trying to make my first submission, and ran into some issues I would like to share. My solution relies on a cropped dataset and on [this code](https://www.kaggle.com/code/clemchris/pytorch-backfin-convnext-arcface).\n\n1. Did anyone experience a massive overfit? Surprisingly, I can achieve 100% train accuracy in 20-30 epochs while the validation accuracy stays around 0%. In terms of loss it means a 0.01 vs 20 difference. I cannot even blame weak augmentations, because I take random crops. \n\n2. Why do self-supervised methods not help? I started with self-supervised pre-training and what I experienced that neither of my approaches actually helped to kick off the training. The resnet50-imagenet weights converge equally fast as a 50-epoch self-trained model, initialized with the same downloaded weights. This is just very surprising to me, did anyone make similar experiments?\n\nThanks.",
      "votes": 3
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "1747135": "Hi everyone,\n\nI am still trying to make my first submission, and ran into some issues I would like to share. My solution relies on a cropped dataset and on [this code](https://www.kaggle.com/code/clemchris/pytorch-backfin-convnext-arcface).\n\n1. Did anyone experience a massive overfit? Surprisingly, I can achieve 100% train accuracy in 20-30 epochs while the validation accuracy stays around 0%. In terms of loss it means a 0.01 vs 20 difference. I cannot even blame weak augmentations, because I take random crops. \n\n2. Why do self-supervised methods not help? I started with self-supervised pre-training and what I experienced that neither of my approaches actually helped to kick off the training. The resnet50-imagenet weights converge equally fast as a 50-epoch self-trained model, initialized with the same downloaded weights. This is just very surprising to me, did anyone make similar experiments?\n\nThanks."
  }
}