{
  "id": 240049,
  "title": "Computation Power Bottleneck",
  "url": "/competitions/seti-breakthrough-listen/discussion/240049",
  "author_name": "Manav",
  "post_date": "2021-05-18T12:04:29.969000",
  "votes": 4,
  "comment_count": 7,
  "views": 0,
  "content": "<p>Hi Fellow Kagglers,</p>\n<p>Is anyone else facing any challenges with computational power with this competition dataset or is it just me? I have exhausted all my GPU and TPU hours within just 2 days in this week 😂😅</p>\n<p>Example:- B0 2 fold takes me around  5.5 hours on Pytorch (no aug)</p>\n<p>I have gone over the code and implementation probably 6-7 times now, fixed some things but with minimal gains. (And in turn lost my hours checking the fix 😪)</p>\n<p>Is anyone else facing this problem or have I written some super inefficient code?<br>\nWould really appreciate anyone sharing their findings…</p>\n<p>Thanks in advance</p>",
  "messages": [
    {
      "id": 1313097,
      "postDate": "2021-05-18T12:04:29.970Z",
      "content": "<p>Hi Fellow Kagglers,</p>\n<p>Is anyone else facing any challenges with computational power with this competition dataset or is it just me? I have exhausted all my GPU and TPU hours within just 2 days in this week 😂😅</p>\n<p>Example:- B0 2 fold takes me around  5.5 hours on Pytorch (no aug)</p>\n<p>I have gone over the code and implementation probably 6-7 times now, fixed some things but with minimal gains. (And in turn lost my hours checking the fix 😪)</p>\n<p>Is anyone else facing this problem or have I written some super inefficient code?<br>\nWould really appreciate anyone sharing their findings…</p>\n<p>Thanks in advance</p>",
      "rawMarkdown": "Hi Fellow Kagglers,\n\nIs anyone else facing any challenges with computational power with this competition dataset or is it just me? I have exhausted all my GPU and TPU hours within just 2 days in this week 😂😅\n\nExample:- B0 2 fold takes me around  5.5 hours on Pytorch (no aug)\n\nI have gone over the code and implementation probably 6-7 times now, fixed some things but with minimal gains. (And in turn lost my hours checking the fix 😪)\n\nIs anyone else facing this problem or have I written some super inefficient code?\nWould really appreciate anyone sharing their findings...\n\nThanks in advance",
      "votes": 4
    },
    {
      "id": 1313264,
      "postDate": "2021-05-18T13:27:45.847Z",
      "content": "<p>You can use colab. its not as fast a kaggle  but it is unlimited.<a href=\"https://colab.research.google.com/\" target=\"_blank\">https://colab.research.google.com/</a> . keep in mind that the disk size is limited so you would  have to use tfrecords.</p>",
      "rawMarkdown": "You can use colab. its not as fast a kaggle  but it is unlimited.https://colab.research.google.com/ . keep in mind that the disk size is limited so you would  have to use tfrecords.",
      "replies": [
        {
          "id": 1313275,
          "postDate": "2021-05-18T13:37:36.557Z",
          "content": "<p>Hi <a href=\"https://www.kaggle.com/mithilsalunkhe\" target=\"_blank\">@mithilsalunkhe</a> ,<br>\nThanks for your advise. I am certainly aware of colab, but working with TFRecords on Pytorch is certainly a challenge. Anyway, I guess something is better than nothing 😅<br>\nAnyway, are you also facing similar issues?</p>",
          "rawMarkdown": "Hi @mithilsalunkhe ,\nThanks for your advise. I am certainly aware of colab, but working with TFRecords on Pytorch is certainly a challenge. Anyway, I guess something is better than nothing 😅\nAnyway, are you also facing similar issues?",
          "votes": 1
        },
        {
          "id": 1313296,
          "postDate": "2021-05-18T13:52:52.983Z",
          "content": "<p><a href=\"https://www.kaggle.com/manabendrarout\" target=\"_blank\">@manabendrarout</a> My main problem is the file size. I use colab pro + local Gpu so Setting up on colab everyday is just pain. Btw For me i takes around 20 - 30 minutes per epoch for something Like EfnB3 with 512 image size. And you should Use float16 for your training. Really helps in training Time</p>",
          "rawMarkdown": "@manabendrarout My main problem is the file size. I use colab pro + local Gpu so Setting up on colab everyday is just pain. Btw For me i takes around 20 - 30 minutes per epoch for something Like EfnB3 with 512 image size. And you should Use float16 for your training. Really helps in training Time",
          "votes": 2
        },
        {
          "id": 1314833,
          "postDate": "2021-05-19T12:05:34.907Z",
          "content": "<p><a href=\"https://www.kaggle.com/manabendrarout\" target=\"_blank\">@manabendrarout</a> I think this can help you out <a href=\"https://github.com/vahidk/tfrecord\" target=\"_blank\">https://github.com/vahidk/tfrecord</a></p>",
          "rawMarkdown": "@manabendrarout I think this can help you out https://github.com/vahidk/tfrecord",
          "votes": 2
        },
        {
          "id": 1314937,
          "postDate": "2021-05-19T13:13:56.783Z",
          "content": "<p>Thanks a lot <a href=\"https://www.kaggle.com/mithilsalunkhe\" target=\"_blank\">@mithilsalunkhe</a> !<br>\nThis really helps… Greatly appreciated 👍</p>",
          "rawMarkdown": "Thanks a lot @mithilsalunkhe !\nThis really helps... Greatly appreciated 👍"
        },
        {
          "id": 1315186,
          "postDate": "2021-05-19T16:08:53.960Z",
          "content": "<p>When I have created a private Kaggle dataset, can I use this with a colab TPU directly? Or do I have to copy the data to some other \"gcs-account\"?</p>",
          "rawMarkdown": "When I have created a private Kaggle dataset, can I use this with a colab TPU directly? Or do I have to copy the data to some other \"gcs-account\"?"
        },
        {
          "id": 1315678,
          "postDate": "2021-05-20T04:13:03.997Z",
          "content": "<p>You have to use google cloud Buckets if you want to use Colab Tpu. Or else you should stick to kaggle Tpu</p>",
          "rawMarkdown": "You have to use google cloud Buckets if you want to use Colab Tpu. Or else you should stick to kaggle Tpu",
          "votes": -1
        }
      ]
    }
  ],
  "comments": [
    {
      "id": 1313264,
      "author_name": "Mithil Salunkhe",
      "author_url": "",
      "post_date": "2021-05-18T13:27:45.847000",
      "content": "<p>You can use colab. its not as fast a kaggle  but it is unlimited.<a href=\"https://colab.research.google.com/\" target=\"_blank\">https://colab.research.google.com/</a> . keep in mind that the disk size is limited so you would  have to use tfrecords.</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1313275,
          "author_name": "Manav",
          "author_url": "",
          "post_date": "2021-05-18T13:37:36.557000",
          "content": "<p>Hi <a href=\"https://www.kaggle.com/mithilsalunkhe\" target=\"_blank\">@mithilsalunkhe</a> ,<br>\nThanks for your advise. I am certainly aware of colab, but working with TFRecords on Pytorch is certainly a challenge. Anyway, I guess something is better than nothing 😅<br>\nAnyway, are you also facing similar issues?</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1313296,
          "author_name": "Mithil Salunkhe",
          "author_url": "",
          "post_date": "2021-05-18T13:52:52.983000",
          "content": "<p><a href=\"https://www.kaggle.com/manabendrarout\" target=\"_blank\">@manabendrarout</a> My main problem is the file size. I use colab pro + local Gpu so Setting up on colab everyday is just pain. Btw For me i takes around 20 - 30 minutes per epoch for something Like EfnB3 with 512 image size. And you should Use float16 for your training. Really helps in training Time</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 1314833,
          "author_name": "Mithil Salunkhe",
          "author_url": "",
          "post_date": "2021-05-19T12:05:34.907000",
          "content": "<p><a href=\"https://www.kaggle.com/manabendrarout\" target=\"_blank\">@manabendrarout</a> I think this can help you out <a href=\"https://github.com/vahidk/tfrecord\" target=\"_blank\">https://github.com/vahidk/tfrecord</a></p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 1314937,
          "author_name": "Manav",
          "author_url": "",
          "post_date": "2021-05-19T13:13:56.783000",
          "content": "<p>Thanks a lot <a href=\"https://www.kaggle.com/mithilsalunkhe\" target=\"_blank\">@mithilsalunkhe</a> !<br>\nThis really helps… Greatly appreciated 👍</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1315186,
          "author_name": "AgentAuers",
          "author_url": "",
          "post_date": "2021-05-19T16:08:53.960000",
          "content": "<p>When I have created a private Kaggle dataset, can I use this with a colab TPU directly? Or do I have to copy the data to some other \"gcs-account\"?</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1315678,
          "author_name": "Mithil Salunkhe",
          "author_url": "",
          "post_date": "2021-05-20T04:13:03.997000",
          "content": "<p>You have to use google cloud Buckets if you want to use Colab Tpu. Or else you should stick to kaggle Tpu</p>",
          "votes": -1,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1313097": "Hi Fellow Kagglers,\n\nIs anyone else facing any challenges with computational power with this competition dataset or is it just me? I have exhausted all my GPU and TPU hours within just 2 days in this week 😂😅\n\nExample:- B0 2 fold takes me around  5.5 hours on Pytorch (no aug)\n\nI have gone over the code and implementation probably 6-7 times now, fixed some things but with minimal gains. (And in turn lost my hours checking the fix 😪)\n\nIs anyone else facing this problem or have I written some super inefficient code?\nWould really appreciate anyone sharing their findings...\n\nThanks in advance",
    "1313264": "You can use colab. its not as fast a kaggle  but it is unlimited.https://colab.research.google.com/ . keep in mind that the disk size is limited so you would  have to use tfrecords."
  }
}