{
  "id": 381587,
  "title": "Is there any ways to automate sequential execution of notebooks ?",
  "url": "/competitions/otto-recommender-system/discussion/381587",
  "author_name": "",
  "post_date": "2023-01-27T09:51:44.403215Z",
  "votes": 1,
  "comment_count": 6,
  "views": 0,
  "content": "<p>Hello community, given the number of notebooks I use due to lack of memory, my final notebook uses several datasets from these same notebooks ( one for candidates generation, multiple for feature extraction … ).</p>\n<p>I would like to know if there is a way to automate the execution and the saving of the outputs of my parent notebooks, to finally execute my final notebook using the latest version of each dataset.</p>\n<p>Thanks !!</p>",
  "messages": [
    {
      "id": "2117469",
      "postDate": "01/27/2023 09:51:44",
      "content": "<p>Hello community, given the number of notebooks I use due to lack of memory, my final notebook uses several datasets from these same notebooks ( one for candidates generation, multiple for feature extraction … ).</p>\n<p>I would like to know if there is a way to automate the execution and the saving of the outputs of my parent notebooks, to finally execute my final notebook using the latest version of each dataset.</p>\n<p>Thanks !!</p>",
      "rawMarkdown": "Hello community, given the number of notebooks I use due to lack of memory, my final notebook uses several datasets from these same notebooks ( one for candidates generation, multiple for feature extraction ... ).\n\n I would like to know if there is a way to automate the execution and the saving of the outputs of my parent notebooks, to finally execute my final notebook using the latest version of each dataset.\n\nThanks !!",
      "votes": null
    },
    {
      "id": "2117475",
      "postDate": "01/27/2023 09:59:08",
      "content": "<p>yes. just turn them into py files 😃</p>",
      "rawMarkdown": "yes. just turn them into py files 😃",
      "votes": null
    },
    {
      "id": "2117481",
      "postDate": "01/27/2023 10:03:26",
      "content": "<p>I'm limited by RAM, I can't use one single kernel to run all of them. How would you do ? 👀<br>\nMaybe I didn't speak about it but I'm only using kaggle kernels</p>",
      "rawMarkdown": "I'm limited by RAM, I can't use one single kernel to run all of them. How would you do ? 👀\n\nMaybe I didn't speak about it but I'm only using kaggle kernels",
      "votes": null
    },
    {
      "id": "2117535",
      "postDate": "01/27/2023 10:41:16",
      "content": "<p>You may use <strong>%%python</strong> header to isolate each step in one single kernel, then after each step the RAM will be released completely.<br>\nBut of course you will lose the parallel computation advantage by using multiple notebooks.</p>",
      "rawMarkdown": "You may use **%%python** header to isolate each step in one single kernel, then after each step the RAM will be released completely.\nBut of course you will lose the parallel computation advantage by using multiple notebooks.",
      "votes": null
    },
    {
      "id": "2117715",
      "postDate": "01/27/2023 13:38:13",
      "content": "<p>You may want to look at: <a href=\"https://nbconvert.readthedocs.io/en/latest/execute_api.html\" target=\"_blank\">https://nbconvert.readthedocs.io/en/latest/execute_api.html</a>, I am using this for otto, as all my code is scattered into various notebooks</p>",
      "rawMarkdown": "You may want to look at: https://nbconvert.readthedocs.io/en/latest/execute_api.html, I am using this for otto, as all my code is scattered into various notebooks",
      "votes": null
    },
    {
      "id": "2117725",
      "postDate": "01/27/2023 13:50:26",
      "content": "<p>Great ! I didn't know this lib, I'll test it. Thanks <a href=\"https://www.kaggle.com/nikhilmishradev\" target=\"_blank\">@nikhilmishradev</a> </p>",
      "rawMarkdown": "Great ! I didn't know this lib, I'll test it. Thanks @nikhilmishradev",
      "votes": null
    },
    {
      "id": "2117727",
      "postDate": "01/27/2023 13:51:32",
      "content": "<p>I hadn't thought of that. I tried using excessive del + gc, but didn't work, thanks <a href=\"https://www.kaggle.com/buumoo\" target=\"_blank\">@buumoo</a> </p>",
      "rawMarkdown": "I hadn't thought of that. I tried using excessive del + gc, but didn't work, thanks @buumoo",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2117475,
      "author_name": "kvlmll",
      "author_url": "",
      "post_date": "01/27/2023 09:59:08",
      "content": "<p>yes. just turn them into py files 😃</p>",
      "votes": null,
      "replies": [
        {
          "id": 2117481,
          "author_name": "rayanaay",
          "author_url": "",
          "post_date": "01/27/2023 10:03:26",
          "content": "<p>I'm limited by RAM, I can't use one single kernel to run all of them. How would you do ? 👀<br>\nMaybe I didn't speak about it but I'm only using kaggle kernels</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2117535,
      "author_name": "buumoo",
      "author_url": "",
      "post_date": "01/27/2023 10:41:16",
      "content": "<p>You may use <strong>%%python</strong> header to isolate each step in one single kernel, then after each step the RAM will be released completely.<br>\nBut of course you will lose the parallel computation advantage by using multiple notebooks.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2117727,
          "author_name": "rayanaay",
          "author_url": "",
          "post_date": "01/27/2023 13:51:32",
          "content": "<p>I hadn't thought of that. I tried using excessive del + gc, but didn't work, thanks <a href=\"https://www.kaggle.com/buumoo\" target=\"_blank\">@buumoo</a> </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2117715,
      "author_name": "nikhilmishradev",
      "author_url": "",
      "post_date": "01/27/2023 13:38:13",
      "content": "<p>You may want to look at: <a href=\"https://nbconvert.readthedocs.io/en/latest/execute_api.html\" target=\"_blank\">https://nbconvert.readthedocs.io/en/latest/execute_api.html</a>, I am using this for otto, as all my code is scattered into various notebooks</p>",
      "votes": null,
      "replies": [
        {
          "id": 2117725,
          "author_name": "rayanaay",
          "author_url": "",
          "post_date": "01/27/2023 13:50:26",
          "content": "<p>Great ! I didn't know this lib, I'll test it. Thanks <a href=\"https://www.kaggle.com/nikhilmishradev\" target=\"_blank\">@nikhilmishradev</a> </p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2117469": "Hello community, given the number of notebooks I use due to lack of memory, my final notebook uses several datasets from these same notebooks ( one for candidates generation, multiple for feature extraction ... ).\n\n I would like to know if there is a way to automate the execution and the saving of the outputs of my parent notebooks, to finally execute my final notebook using the latest version of each dataset.\n\nThanks !!",
    "2117475": "yes. just turn them into py files 😃",
    "2117481": "I'm limited by RAM, I can't use one single kernel to run all of them. How would you do ? 👀\n\nMaybe I didn't speak about it but I'm only using kaggle kernels",
    "2117535": "You may use **%%python** header to isolate each step in one single kernel, then after each step the RAM will be released completely.\nBut of course you will lose the parallel computation advantage by using multiple notebooks.",
    "2117715": "You may want to look at: https://nbconvert.readthedocs.io/en/latest/execute_api.html, I am using this for otto, as all my code is scattered into various notebooks",
    "2117725": "Great ! I didn't know this lib, I'll test it. Thanks @nikhilmishradev",
    "2117727": "I hadn't thought of that. I tried using excessive del + gc, but didn't work, thanks @buumoo"
  },
  "source": "meta"
}