{
  "id": 506529,
  "title": "Unsupervised Contrastive Pre-Training Pipeline for Audio Representations",
  "url": "/competitions/birdclef-2024/discussion/506529",
  "author_name": "",
  "post_date": "2024-05-22T08:08:56.530587200Z",
  "votes": 3,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hey everyone!</p>\n<p>Pretraining with contrastive learning seems like a good idea in this competition. I have worked on implementing an unsupervised approach, which I would like to share. This is an implementation inspired by the paper Contrastive Learning of General-Purpose Audio Representations (<a href=\"https://arxiv.org/abs/2010.10915\" target=\"_blank\">COLA</a>) and adapted from this GitHub <a href=\"https://github.com/CVxTz/COLA_pytorch\" target=\"_blank\">repository</a>.</p>\n<p><a href=\"https://www.kaggle.com/code/arindamroy23/pretrain-eff0-unsup-contra\" target=\"_blank\">Notebook</a></p>\n<h1>Overview</h1>\n<p>The goal of this notebook is to enable training on any amount of mel spectrogram data, provided it's stored in .npy format. This flexible approach allows you to scale your training data without constraints.</p>\n<h1>Key Ideas</h1>\n<ul>\n<li>Unsupervised Learning: Leverages contrastive learning to pretrain models without labeled data.</li>\n<li>Flexible Data Input: Accepts mel spectrogram data in .npy format, making it easy to add and train on new datasets.</li>\n<li>Adaptability: Any encoder model can be replaced and trained in the pipeline. </li>\n</ul>\n<h2>Notes</h2>\n<p>This notebook is not trained on the full dataset by default. For comprehensive training, please remove limit_train_batches and limit_val_batches from the trainer instance.<br>\nFeel free to check out the notebook and give it a try! Your feedback and suggestions are highly appreciated.</p>\n<p>If you have any questions or need further assistance, please don't hesitate to ask.<br>\nCheers!</p>",
  "messages": [
    {
      "id": "2828699",
      "postDate": "05/22/2024 08:08:56",
      "content": "<p>Hey everyone!</p>\n<p>Pretraining with contrastive learning seems like a good idea in this competition. I have worked on implementing an unsupervised approach, which I would like to share. This is an implementation inspired by the paper Contrastive Learning of General-Purpose Audio Representations (<a href=\"https://arxiv.org/abs/2010.10915\" target=\"_blank\">COLA</a>) and adapted from this GitHub <a href=\"https://github.com/CVxTz/COLA_pytorch\" target=\"_blank\">repository</a>.</p>\n<p><a href=\"https://www.kaggle.com/code/arindamroy23/pretrain-eff0-unsup-contra\" target=\"_blank\">Notebook</a></p>\n<h1>Overview</h1>\n<p>The goal of this notebook is to enable training on any amount of mel spectrogram data, provided it's stored in .npy format. This flexible approach allows you to scale your training data without constraints.</p>\n<h1>Key Ideas</h1>\n<ul>\n<li>Unsupervised Learning: Leverages contrastive learning to pretrain models without labeled data.</li>\n<li>Flexible Data Input: Accepts mel spectrogram data in .npy format, making it easy to add and train on new datasets.</li>\n<li>Adaptability: Any encoder model can be replaced and trained in the pipeline. </li>\n</ul>\n<h2>Notes</h2>\n<p>This notebook is not trained on the full dataset by default. For comprehensive training, please remove limit_train_batches and limit_val_batches from the trainer instance.<br>\nFeel free to check out the notebook and give it a try! Your feedback and suggestions are highly appreciated.</p>\n<p>If you have any questions or need further assistance, please don't hesitate to ask.<br>\nCheers!</p>",
      "rawMarkdown": "Hey everyone!\n\nPretraining with contrastive learning seems like a good idea in this competition. I have worked on implementing an unsupervised approach, which I would like to share. This is an implementation inspired by the paper Contrastive Learning of General-Purpose Audio Representations ([COLA](https://arxiv.org/abs/2010.10915)) and adapted from this GitHub [repository](https://github.com/CVxTz/COLA_pytorch).\n\n\n[Notebook](https://www.kaggle.com/code/arindamroy23/pretrain-eff0-unsup-contra)\n\n\n# Overview\nThe goal of this notebook is to enable training on any amount of mel spectrogram data, provided it's stored in .npy format. This flexible approach allows you to scale your training data without constraints.\n\n\n# Key Ideas\n- Unsupervised Learning: Leverages contrastive learning to pretrain models without labeled data.\n- Flexible Data Input: Accepts mel spectrogram data in .npy format, making it easy to add and train on new datasets.\n- Adaptability: Any encoder model can be replaced and trained in the pipeline. \n\n\n## Notes\nThis notebook is not trained on the full dataset by default. For comprehensive training, please remove limit_train_batches and limit_val_batches from the trainer instance.\nFeel free to check out the notebook and give it a try! Your feedback and suggestions are highly appreciated.\n\n\nIf you have any questions or need further assistance, please don't hesitate to ask.\nCheers!",
      "votes": null
    },
    {
      "id": "2830529",
      "postDate": "05/23/2024 08:21:32",
      "content": "<p>How far did you get with this method ?  (is your current LB score trained with contrastive learning ? )</p>",
      "rawMarkdown": "How far did you get with this method ?  (is your current LB score trained with contrastive learning ? )",
      "votes": null
    },
    {
      "id": "2830574",
      "postDate": "05/23/2024 08:55:56",
      "content": "<p>I am still training various Effnet models post pretraining. Current LB does not contain these contrastive pretrained model. </p>\n<p>My training pipeline has errors, that I am solving. So may or may not work in the future. Dont know yet.</p>",
      "rawMarkdown": "I am still training various Effnet models post pretraining. Current LB does not contain these contrastive pretrained model. \n\nMy training pipeline has errors, that I am solving. So may or may not work in the future. Dont know yet.",
      "votes": null
    },
    {
      "id": "2832976",
      "postDate": "05/24/2024 02:14:10",
      "content": "<p>I have tried two experiments via contrastive learning to pretrain, followed by training with a sigmoid function, but it did not improve the leaderboard</p>",
      "rawMarkdown": "I have tried two experiments via contrastive learning to pretrain, followed by training with a sigmoid function, but it did not improve the leaderboard",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2830529,
      "author_name": "nyleve",
      "author_url": "",
      "post_date": "05/23/2024 08:21:32",
      "content": "<p>How far did you get with this method ?  (is your current LB score trained with contrastive learning ? )</p>",
      "votes": null,
      "replies": [
        {
          "id": 2830574,
          "author_name": "arindamroy23",
          "author_url": "",
          "post_date": "05/23/2024 08:55:56",
          "content": "<p>I am still training various Effnet models post pretraining. Current LB does not contain these contrastive pretrained model. </p>\n<p>My training pipeline has errors, that I am solving. So may or may not work in the future. Dont know yet.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2832976,
      "author_name": "sentrankim",
      "author_url": "",
      "post_date": "05/24/2024 02:14:10",
      "content": "<p>I have tried two experiments via contrastive learning to pretrain, followed by training with a sigmoid function, but it did not improve the leaderboard</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2828699": "Hey everyone!\n\nPretraining with contrastive learning seems like a good idea in this competition. I have worked on implementing an unsupervised approach, which I would like to share. This is an implementation inspired by the paper Contrastive Learning of General-Purpose Audio Representations ([COLA](https://arxiv.org/abs/2010.10915)) and adapted from this GitHub [repository](https://github.com/CVxTz/COLA_pytorch).\n\n\n[Notebook](https://www.kaggle.com/code/arindamroy23/pretrain-eff0-unsup-contra)\n\n\n# Overview\nThe goal of this notebook is to enable training on any amount of mel spectrogram data, provided it's stored in .npy format. This flexible approach allows you to scale your training data without constraints.\n\n\n# Key Ideas\n- Unsupervised Learning: Leverages contrastive learning to pretrain models without labeled data.\n- Flexible Data Input: Accepts mel spectrogram data in .npy format, making it easy to add and train on new datasets.\n- Adaptability: Any encoder model can be replaced and trained in the pipeline. \n\n\n## Notes\nThis notebook is not trained on the full dataset by default. For comprehensive training, please remove limit_train_batches and limit_val_batches from the trainer instance.\nFeel free to check out the notebook and give it a try! Your feedback and suggestions are highly appreciated.\n\n\nIf you have any questions or need further assistance, please don't hesitate to ask.\nCheers!",
    "2830529": "How far did you get with this method ?  (is your current LB score trained with contrastive learning ? )",
    "2830574": "I am still training various Effnet models post pretraining. Current LB does not contain these contrastive pretrained model. \n\nMy training pipeline has errors, that I am solving. So may or may not work in the future. Dont know yet.",
    "2832976": "I have tried two experiments via contrastive learning to pretrain, followed by training with a sigmoid function, but it did not improve the leaderboard"
  },
  "source": "meta"
}