{
  "id": 313161,
  "title": "upsampling with TFRecord and TF",
  "url": "/competitions/happy-whale-and-dolphin/discussion/313161",
  "author_name": "",
  "post_date": "2022-03-15T22:21:31.136000300Z",
  "votes": 6,
  "comment_count": 3,
  "views": 0,
  "content": "<p>I'm really beginner of TF and TFRecord. Therefore, this may be stupid question.</p>\n<p>How can I upsampling(Oversamping) same image?<br>\nIf I use Pytorch, I can do like</p>\n<pre><code>train = pd.read_csv(\"train.csv\")\nover_sample_df = train.sample(over_sample_size)\ntrain = pd.concat([train,over_sample_df ])\n</code></pre>\n<p>Now, I do upsampling when I make TFRecord by just duplicating same records.<br>\nBut this is maybe inefficient  because many same  images are in TFRecord.</p>\n<p>If you know better way to upsampling with TF and TFRecord, please teach me.</p>",
  "messages": [
    {
      "id": "1723942",
      "postDate": "03/15/2022 22:21:31",
      "content": "<p>I'm really beginner of TF and TFRecord. Therefore, this may be stupid question.</p>\n<p>How can I upsampling(Oversamping) same image?<br>\nIf I use Pytorch, I can do like</p>\n<pre><code>train = pd.read_csv(\"train.csv\")\nover_sample_df = train.sample(over_sample_size)\ntrain = pd.concat([train,over_sample_df ])\n</code></pre>\n<p>Now, I do upsampling when I make TFRecord by just duplicating same records.<br>\nBut this is maybe inefficient  because many same  images are in TFRecord.</p>\n<p>If you know better way to upsampling with TF and TFRecord, please teach me.</p>",
      "rawMarkdown": "I'm really beginner of TF and TFRecord. Therefore, this may be stupid question.\n\nHow can I upsampling(Oversamping) same image?\nIf I use Pytorch, I can do like\n```\ntrain = pd.read_csv(\"train.csv\")\nover_sample_df = train.sample(over_sample_size)\ntrain = pd.concat([train,over_sample_df ])\n```\n\nNow, I do upsampling when I make TFRecord by just duplicating same records.\nBut this is maybe inefficient  because many same  images are in TFRecord.\n\nIf you know better way to upsampling with TF and TFRecord, please teach me.",
      "votes": null
    },
    {
      "id": "1724036",
      "postDate": "03/16/2022 01:32:10",
      "content": "<p>Use flat_map, refer to <a href=\"https://stackoverflow.com/questions/47236465/oversampling-functionality-in-tensorflow-dataset-api/47236466#47236466\" target=\"_blank\">https://stackoverflow.com/questions/47236465/oversampling-functionality-in-tensorflow-dataset-api/47236466#47236466</a></p>",
      "rawMarkdown": "Use flat_map, refer to https://stackoverflow.com/questions/47236465/oversampling-functionality-in-tensorflow-dataset-api/47236466#47236466",
      "votes": null
    },
    {
      "id": "1725004",
      "postDate": "03/16/2022 18:04:00",
      "content": "<p>Thank you for your great sharing！ I'll check it</p>",
      "rawMarkdown": "Thank you for your great sharing！ I'll check it",
      "votes": null
    },
    {
      "id": "1725146",
      "postDate": "03/16/2022 21:06:48",
      "content": "<p>I guess this is the most efficient option in TF: <a href=\"https://www.tensorflow.org/api_docs/python/tf/data/Dataset#rejection_resample\" target=\"_blank\">https://www.tensorflow.org/api_docs/python/tf/data/Dataset#rejection_resample</a>. You will need to increase the number of epochs and adjust the schedule to keep training the same number of examples in total.</p>\n<p>Also, I wonder if balanced classes would work here. Single/few-example classes might be overfitting. On the hand, maybe Arcface can compensate this? Any thoughts someone?</p>",
      "rawMarkdown": "I guess this is the most efficient option in TF: https://www.tensorflow.org/api_docs/python/tf/data/Dataset#rejection_resample. You will need to increase the number of epochs and adjust the schedule to keep training the same number of examples in total.\n\nAlso, I wonder if balanced classes would work here. Single/few-example classes might be overfitting. On the hand, maybe Arcface can compensate this? Any thoughts someone?",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1724036,
      "author_name": "kejiewang",
      "author_url": "",
      "post_date": "03/16/2022 01:32:10",
      "content": "<p>Use flat_map, refer to <a href=\"https://stackoverflow.com/questions/47236465/oversampling-functionality-in-tensorflow-dataset-api/47236466#47236466\" target=\"_blank\">https://stackoverflow.com/questions/47236465/oversampling-functionality-in-tensorflow-dataset-api/47236466#47236466</a></p>",
      "votes": null,
      "replies": [
        {
          "id": 1725004,
          "author_name": "whitelily",
          "author_url": "",
          "post_date": "03/16/2022 18:04:00",
          "content": "<p>Thank you for your great sharing！ I'll check it</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1725146,
      "author_name": "greendolphin",
      "author_url": "",
      "post_date": "03/16/2022 21:06:48",
      "content": "<p>I guess this is the most efficient option in TF: <a href=\"https://www.tensorflow.org/api_docs/python/tf/data/Dataset#rejection_resample\" target=\"_blank\">https://www.tensorflow.org/api_docs/python/tf/data/Dataset#rejection_resample</a>. You will need to increase the number of epochs and adjust the schedule to keep training the same number of examples in total.</p>\n<p>Also, I wonder if balanced classes would work here. Single/few-example classes might be overfitting. On the hand, maybe Arcface can compensate this? Any thoughts someone?</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1723942": "I'm really beginner of TF and TFRecord. Therefore, this may be stupid question.\n\nHow can I upsampling(Oversamping) same image?\nIf I use Pytorch, I can do like\n```\ntrain = pd.read_csv(\"train.csv\")\nover_sample_df = train.sample(over_sample_size)\ntrain = pd.concat([train,over_sample_df ])\n```\n\nNow, I do upsampling when I make TFRecord by just duplicating same records.\nBut this is maybe inefficient  because many same  images are in TFRecord.\n\nIf you know better way to upsampling with TF and TFRecord, please teach me.",
    "1724036": "Use flat_map, refer to https://stackoverflow.com/questions/47236465/oversampling-functionality-in-tensorflow-dataset-api/47236466#47236466",
    "1725004": "Thank you for your great sharing！ I'll check it",
    "1725146": "I guess this is the most efficient option in TF: https://www.tensorflow.org/api_docs/python/tf/data/Dataset#rejection_resample. You will need to increase the number of epochs and adjust the schedule to keep training the same number of examples in total.\n\nAlso, I wonder if balanced classes would work here. Single/few-example classes might be overfitting. On the hand, maybe Arcface can compensate this? Any thoughts someone?"
  },
  "source": "meta"
}