Hello everyone!
\nThis is my first real competition. I've heard that in some cases you can \"hit the ceiling\" if you don't have enough power resources. So I am wondering if this is the case (I have 2020 m1 8gb macbook).
\nI am also interested in where people usually solve this kind of competitions: kaggle notebooks, google colab or any other IDEs?
\nThank you for your help and good luck!
Hello everyone!
\nThis is my first real competition. I've heard that in some cases you can \"hit the ceiling\" if you don't have enough power resources. So I am wondering if this is the case (I have 2020 m1 8gb macbook).
\nI am also interested in where people usually solve this kind of competitions: kaggle notebooks, google colab or any other IDEs?
\nThank you for your help and good luck!
Hello!
\nThe dataset for this competition is substantial, requiring significant resources to handle efficiently. Even with 32GB of RAM, I found myself needing to employ various strategies to manage the data effectively. Exploring techniques such as feature engineering to reduce RAM consumption presents a valuable learning opportunity. Alternatively, utilizing platforms like Google Colab or Kaggle Notebooks may be the most efficient path forward.
", "rawMarkdown": "Hello!\n\nThe dataset for this competition is substantial, requiring significant resources to handle efficiently. Even with 32GB of RAM, I found myself needing to employ various strategies to manage the data effectively. Exploring techniques such as feature engineering to reduce RAM consumption presents a valuable learning opportunity. Alternatively, utilizing platforms like Google Colab or Kaggle Notebooks may be the most efficient path forward.", "votes": null }, { "id": "2657174", "postDate": "02/18/2024 11:16:35", "content": "I'm thinking of downsampling the data for the first exploratory phase, that could be an option. Not sure if it makes sense with time-series data, though. I've just started the competition so can't say more.
", "rawMarkdown": "I'm thinking of downsampling the data for the first exploratory phase, that could be an option. Not sure if it makes sense with time-series data, though. I've just started the competition so can't say more.", "votes": null }, { "id": "2659991", "postDate": "02/20/2024 09:13:30", "content": "Here in kaggle we have 30gb of available RAM and it's challenging already. In free Colab there are 12gb and you have 8gb locally. So go for Kaggle if you don't have some other paid account with greater resources. And check for different techniques to reduce RAM use during data exploration. Tip - column dtypes could be kept at the minimum required (category instead of object makes a huge difference), rewrite variables instead of making copies, or delete the unused ones and so on.
", "rawMarkdown": "Here in kaggle we have 30gb of available RAM and it's challenging already. In free Colab there are 12gb and you have 8gb locally. So go for Kaggle if you don't have some other paid account with greater resources. And check for different techniques to reduce RAM use during data exploration. Tip - column dtypes could be kept at the minimum required (category instead of object makes a huge difference), rewrite variables instead of making copies, or delete the unused ones and so on.", "votes": null }, { "id": "2660676", "postDate": "02/20/2024 18:53:21", "content": "I think it would only make sense if you downsample evenly over each timestep.
", "rawMarkdown": "I think it would only make sense if you downsample evenly over each timestep.", "votes": null } ], "comments": [ { "id": 2656565, "author_name": "marcopg", "author_url": "", "post_date": "02/17/2024 19:59:34", "content": "Hello!
\nThe dataset for this competition is substantial, requiring significant resources to handle efficiently. Even with 32GB of RAM, I found myself needing to employ various strategies to manage the data effectively. Exploring techniques such as feature engineering to reduce RAM consumption presents a valuable learning opportunity. Alternatively, utilizing platforms like Google Colab or Kaggle Notebooks may be the most efficient path forward.
", "votes": null, "replies": [] }, { "id": 2657174, "author_name": "codenamev", "author_url": "", "post_date": "02/18/2024 11:16:35", "content": "I'm thinking of downsampling the data for the first exploratory phase, that could be an option. Not sure if it makes sense with time-series data, though. I've just started the competition so can't say more.
", "votes": null, "replies": [ { "id": 2660676, "author_name": "simonveitner", "author_url": "", "post_date": "02/20/2024 18:53:21", "content": "I think it would only make sense if you downsample evenly over each timestep.
", "votes": null, "replies": [] } ] }, { "id": 2659991, "author_name": "eu1234", "author_url": "", "post_date": "02/20/2024 09:13:30", "content": "Here in kaggle we have 30gb of available RAM and it's challenging already. In free Colab there are 12gb and you have 8gb locally. So go for Kaggle if you don't have some other paid account with greater resources. And check for different techniques to reduce RAM use during data exploration. Tip - column dtypes could be kept at the minimum required (category instead of object makes a huge difference), rewrite variables instead of making copies, or delete the unused ones and so on.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2656555": "Hello everyone! \nThis is my first real competition. I've heard that in some cases you can \"hit the ceiling\" if you don't have enough power resources. So I am wondering if this is the case (I have 2020 m1 8gb macbook).\nI am also interested in where people usually solve this kind of competitions: kaggle notebooks, google colab or any other IDEs?\nThank you for your help and good luck!", "2656565": "Hello!\n\nThe dataset for this competition is substantial, requiring significant resources to handle efficiently. Even with 32GB of RAM, I found myself needing to employ various strategies to manage the data effectively. Exploring techniques such as feature engineering to reduce RAM consumption presents a valuable learning opportunity. Alternatively, utilizing platforms like Google Colab or Kaggle Notebooks may be the most efficient path forward.", "2657174": "I'm thinking of downsampling the data for the first exploratory phase, that could be an option. Not sure if it makes sense with time-series data, though. I've just started the competition so can't say more.", "2659991": "Here in kaggle we have 30gb of available RAM and it's challenging already. In free Colab there are 12gb and you have 8gb locally. So go for Kaggle if you don't have some other paid account with greater resources. And check for different techniques to reduce RAM use during data exploration. Tip - column dtypes could be kept at the minimum required (category instead of object makes a huge difference), rewrite variables instead of making copies, or delete the unused ones and so on.", "2660676": "I think it would only make sense if you downsample evenly over each timestep." }, "source": "meta" }