{
  "id": 251778,
  "title": "Has anyone tried running experiments in colab pro ?",
  "url": "/competitions/g2net-gravitational-wave-detection/discussion/251778",
  "author_name": "Atharva Ingle",
  "post_date": "2021-07-08T18:28:51.433000",
  "votes": 10,
  "comment_count": 7,
  "views": 0,
  "content": "<p>This competition data is huge. I have colab pro subscription still couldn't unzip this data successfully. Colab pro provides around 100 GB of usable disk space on a GPU runtime. Have anyone tried some hacks to make it work on colab or is there any data preprocessing we can use to reduce the data size </p>",
  "messages": [
    {
      "id": 1381226,
      "postDate": "2021-07-08T18:28:51.433Z",
      "content": "<p>This competition data is huge. I have colab pro subscription still couldn't unzip this data successfully. Colab pro provides around 100 GB of usable disk space on a GPU runtime. Have anyone tried some hacks to make it work on colab or is there any data preprocessing we can use to reduce the data size </p>",
      "rawMarkdown": "This competition data is huge. I have colab pro subscription still couldn't unzip this data successfully. Colab pro provides around 100 GB of usable disk space on a GPU runtime. Have anyone tried some hacks to make it work on colab or is there any data preprocessing we can use to reduce the data size ",
      "votes": 10
    },
    {
      "id": 1381448,
      "postDate": "2021-07-09T02:53:38.813Z",
      "content": "<p>At first, the data I produced was 100-200GB, but when I gradually understood the data, I found that I only needed to produce 40GB or so to achieve the score of single model CV86 +   LB86.5.</p>",
      "rawMarkdown": "At first, the data I produced was 100-200GB, but when I gradually understood the data, I found that I only needed to produce 40GB or so to achieve the score of single model CV86 +   LB86.5.",
      "votes": 3
    },
    {
      "id": 1481530,
      "postDate": "2021-08-19T15:04:22.667Z",
      "content": "<p>I applied bandpass filtering and converted the data to the TFRecords format. The total size was about 50GB, which is storable on the disk of Google Colab.</p>",
      "rawMarkdown": "I applied bandpass filtering and converted the data to the TFRecords format. The total size was about 50GB, which is storable on the disk of Google Colab.",
      "votes": 1
    },
    {
      "id": 1384939,
      "postDate": "2021-07-12T10:47:04.103Z",
      "content": "<p>Preprocess the data. This data is provided in float64 format. You can just make a kaggle notebook and save the data as float16</p>",
      "rawMarkdown": "Preprocess the data. This data is provided in float64 format. You can just make a kaggle notebook and save the data as float16\n",
      "votes": 1
    },
    {
      "id": 1381848,
      "postDate": "2021-07-09T09:21:25.803Z",
      "content": "<p>I think that in case of such big data it's the best to store it on Google Drive, and then use them in Colab by mounting your Google Drive.</p>",
      "rawMarkdown": "I think that in case of such big data it's the best to store it on Google Drive, and then use them in Colab by mounting your Google Drive.",
      "votes": 1
    },
    {
      "id": 1383411,
      "postDate": "2021-07-10T20:03:07Z",
      "content": "<p>Amazing resources! thank you for sharing… upvoted….</p>",
      "rawMarkdown": "Amazing resources! thank you for sharing... upvoted....",
      "votes": -9
    },
    {
      "id": 1561180,
      "postDate": "2021-10-27T12:15:06.773Z",
      "content": "<p>Hey All,</p>\n<p>Thank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey <a href=\"https://forms.gle/QP9L16niPexozyhu5\" target=\"_blank\">https://forms.gle/QP9L16niPexozyhu5</a>.</p>\n<p>Thank you all,</p>\n<p>Regards,<br>\nChris</p>",
      "rawMarkdown": "Hey All,\n\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.\n\nThank you all,\n\nRegards,\nChris"
    },
    {
      "id": 1484676,
      "postDate": "2021-08-21T13:53:30.423Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 1381448,
      "author_name": "zhangeng",
      "author_url": "",
      "post_date": "2021-07-09T02:53:38.813000",
      "content": "<p>At first, the data I produced was 100-200GB, but when I gradually understood the data, I found that I only needed to produce 40GB or so to achieve the score of single model CV86 +   LB86.5.</p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 1481530,
      "author_name": "The fearless",
      "author_url": "",
      "post_date": "2021-08-19T15:04:22.667000",
      "content": "<p>I applied bandpass filtering and converted the data to the TFRecords format. The total size was about 50GB, which is storable on the disk of Google Colab.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1384939,
      "author_name": "Mithil Salunkhe",
      "author_url": "",
      "post_date": "2021-07-12T10:47:04.103000",
      "content": "<p>Preprocess the data. This data is provided in float64 format. You can just make a kaggle notebook and save the data as float16</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1381848,
      "author_name": "Araik Tamazian",
      "author_url": "",
      "post_date": "2021-07-09T09:21:25.803000",
      "content": "<p>I think that in case of such big data it's the best to store it on Google Drive, and then use them in Colab by mounting your Google Drive.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1383411,
      "author_name": "Abhishek Kumar",
      "author_url": "",
      "post_date": "2021-07-10T20:03:07",
      "content": "<p>Amazing resources! thank you for sharing… upvoted….</p>",
      "votes": -9,
      "replies": []
    },
    {
      "id": 1561180,
      "author_name": "ChristopherZerafa",
      "author_url": "",
      "post_date": "2021-10-27T12:15:06.773000",
      "content": "<p>Hey All,</p>\n<p>Thank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey <a href=\"https://forms.gle/QP9L16niPexozyhu5\" target=\"_blank\">https://forms.gle/QP9L16niPexozyhu5</a>.</p>\n<p>Thank you all,</p>\n<p>Regards,<br>\nChris</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1484676,
      "author_name": "",
      "author_url": "",
      "post_date": "2021-08-21T13:53:30.423000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1381226": "This competition data is huge. I have colab pro subscription still couldn't unzip this data successfully. Colab pro provides around 100 GB of usable disk space on a GPU runtime. Have anyone tried some hacks to make it work on colab or is there any data preprocessing we can use to reduce the data size ",
    "1381448": "At first, the data I produced was 100-200GB, but when I gradually understood the data, I found that I only needed to produce 40GB or so to achieve the score of single model CV86 +   LB86.5.",
    "1481530": "I applied bandpass filtering and converted the data to the TFRecords format. The total size was about 50GB, which is storable on the disk of Google Colab.",
    "1384939": "Preprocess the data. This data is provided in float64 format. You can just make a kaggle notebook and save the data as float16\n",
    "1381848": "I think that in case of such big data it's the best to store it on Google Drive, and then use them in Colab by mounting your Google Drive.",
    "1383411": "Amazing resources! thank you for sharing... upvoted....",
    "1561180": "Hey All,\n\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.\n\nThank you all,\n\nRegards,\nChris",
    "1484676": ""
  }
}