{
  "id": 181300,
  "title": "A notebook to run for more than 9 hours",
  "url": "/competitions/landmark-recognition-2020/discussion/181300",
  "author_name": "",
  "post_date": "2020-09-08T11:24:01.663321Z",
  "votes": null,
  "comment_count": 6,
  "views": 0,
  "content": "<p>For some of my preprocessing work, I want to run a notebook for more than 9 hours. But the notebooks created here stops after 9 hours of execution?</p>\n<p>Is there any settings/solution to this?</p>\n<p>Please note that I don't have good enough internet bandwidth to download 98GB training data to my local system or upload to google colab.</p>",
  "messages": [
    {
      "id": "1002715",
      "postDate": "09/08/2020 11:24:01",
      "content": "<p>For some of my preprocessing work, I want to run a notebook for more than 9 hours. But the notebooks created here stops after 9 hours of execution?</p>\n<p>Is there any settings/solution to this?</p>\n<p>Please note that I don't have good enough internet bandwidth to download 98GB training data to my local system or upload to google colab.</p>",
      "rawMarkdown": "For some of my preprocessing work, I want to run a notebook for more than 9 hours. But the notebooks created here stops after 9 hours of execution?\n\nIs there any settings/solution to this?\n\nPlease note that I don't have good enough internet bandwidth to download 98GB training data to my local system or upload to google colab.",
      "votes": null
    },
    {
      "id": "1002746",
      "postDate": "09/08/2020 11:53:04",
      "content": "<p>You can download directly from kaggle on google colab. For kaggle: save weights after 9 hours, and repeat.</p>",
      "rawMarkdown": "You can download directly from kaggle on google colab. For kaggle: save weights after 9 hours, and repeat.",
      "votes": null
    },
    {
      "id": "1002782",
      "postDate": "09/08/2020 12:23:10",
      "content": "<p><a href=\"https://www.kaggle.com/philippsinger\" target=\"_blank\">@philippsinger</a> thanks for your reply</p>\n<p>But how to download directly from kaggle on google colab?<br>\nAnd i think on google colab also, there is some issue that the data is not persistant after session ends.</p>\n<p>So, that will be a big trouble i guess.</p>",
      "rawMarkdown": "philippsinger thanks for your reply\n\nBut how to download directly from kaggle on google colab?\nAnd i think on google colab also, there is some issue that the data is not persistant after session ends.\n\nSo, that will be a big trouble i guess.",
      "votes": null
    },
    {
      "id": "1002911",
      "postDate": "09/08/2020 14:18:27",
      "content": "<p>That is a general problem of colab, you can only run it interactively. You can use kaggle api for example, but there might be other options, you can check kaggle forums.</p>",
      "rawMarkdown": "That is a general problem of colab, you can only run it interactively. You can use kaggle api for example, but there might be other options, you can check kaggle forums.",
      "votes": null
    },
    {
      "id": "1004010",
      "postDate": "09/09/2020 11:57:34",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/philippsinger\" target=\"_blank\">@philippsinger</a> </p>\n<p>In Google Colab I'm facing memory issues. The data is downloaded as .zip and expanded for use. It flows out of memory during expansion.</p>\n<p>Google Colab Pro: GPU runtime with 180Gb is provided. But the data is somewhat around 100GB</p>",
      "rawMarkdown": "Hi @philippsinger \n\nIn Google Colab I'm facing memory issues. The data is downloaded as .zip and expanded for use. It flows out of memory during expansion.\n\nGoogle Colab Pro: GPU runtime with 180Gb is provided. But the data is somewhat around 100GB",
      "votes": null
    },
    {
      "id": "1004042",
      "postDate": "09/09/2020 12:22:37",
      "content": "<p><a href=\"https://www.kaggle.com/jagadish13\" target=\"_blank\">@jagadish13</a> Then load it to google drive and mount it on colab. You can connect gdrive on kaggle and use a kernel for that.</p>",
      "rawMarkdown": "jagadish13 Then load it to google drive and mount it on colab. You can connect gdrive on kaggle and use a kernel for that.",
      "votes": null
    },
    {
      "id": "1004152",
      "postDate": "09/09/2020 13:49:30",
      "content": "<p>Right, but google drive provides storage of only 14GB</p>",
      "rawMarkdown": "Right, but google drive provides storage of only 14GB",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1002746,
      "author_name": "philippsinger",
      "author_url": "",
      "post_date": "09/08/2020 11:53:04",
      "content": "<p>You can download directly from kaggle on google colab. For kaggle: save weights after 9 hours, and repeat.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1002782,
          "author_name": "rohitdeepu17",
          "author_url": "",
          "post_date": "09/08/2020 12:23:10",
          "content": "<p><a href=\"https://www.kaggle.com/philippsinger\" target=\"_blank\">@philippsinger</a> thanks for your reply</p>\n<p>But how to download directly from kaggle on google colab?<br>\nAnd i think on google colab also, there is some issue that the data is not persistant after session ends.</p>\n<p>So, that will be a big trouble i guess.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1002911,
          "author_name": "philippsinger",
          "author_url": "",
          "post_date": "09/08/2020 14:18:27",
          "content": "<p>That is a general problem of colab, you can only run it interactively. You can use kaggle api for example, but there might be other options, you can check kaggle forums.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1004010,
          "author_name": "jagadish13",
          "author_url": "",
          "post_date": "09/09/2020 11:57:34",
          "content": "<p>Hi <a href=\"https://www.kaggle.com/philippsinger\" target=\"_blank\">@philippsinger</a> </p>\n<p>In Google Colab I'm facing memory issues. The data is downloaded as .zip and expanded for use. It flows out of memory during expansion.</p>\n<p>Google Colab Pro: GPU runtime with 180Gb is provided. But the data is somewhat around 100GB</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1004042,
          "author_name": "philippsinger",
          "author_url": "",
          "post_date": "09/09/2020 12:22:37",
          "content": "<p><a href=\"https://www.kaggle.com/jagadish13\" target=\"_blank\">@jagadish13</a> Then load it to google drive and mount it on colab. You can connect gdrive on kaggle and use a kernel for that.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1004152,
          "author_name": "jagadish13",
          "author_url": "",
          "post_date": "09/09/2020 13:49:30",
          "content": "<p>Right, but google drive provides storage of only 14GB</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1002715": "For some of my preprocessing work, I want to run a notebook for more than 9 hours. But the notebooks created here stops after 9 hours of execution?\n\nIs there any settings/solution to this?\n\nPlease note that I don't have good enough internet bandwidth to download 98GB training data to my local system or upload to google colab.",
    "1002746": "You can download directly from kaggle on google colab. For kaggle: save weights after 9 hours, and repeat.",
    "1002782": "philippsinger thanks for your reply\n\nBut how to download directly from kaggle on google colab?\nAnd i think on google colab also, there is some issue that the data is not persistant after session ends.\n\nSo, that will be a big trouble i guess.",
    "1002911": "That is a general problem of colab, you can only run it interactively. You can use kaggle api for example, but there might be other options, you can check kaggle forums.",
    "1004010": "Hi @philippsinger \n\nIn Google Colab I'm facing memory issues. The data is downloaded as .zip and expanded for use. It flows out of memory during expansion.\n\nGoogle Colab Pro: GPU runtime with 180Gb is provided. But the data is somewhat around 100GB",
    "1004042": "jagadish13 Then load it to google drive and mount it on colab. You can connect gdrive on kaggle and use a kernel for that.",
    "1004152": "Right, but google drive provides storage of only 14GB"
  },
  "source": "meta"
}