{
  "id": 498230,
  "title": "Try and make best use of your resources",
  "url": "/competitions/home-credit-credit-risk-model-stability/discussion/498230",
  "author_name": "Ravi Ramakrishnan",
  "post_date": "2024-04-27T12:25:08.371000",
  "votes": 17,
  "comment_count": 0,
  "views": 0,
  "content": "<p>Hello all,</p>\n<p>Hope you all are doing well. The competition is approaching the last month now and I thought we could perhaps discuss how we may collectively manage our resources at our disposal to better effectuate a more organized submission pipeline to garner better effect on the CV and LB scores in totality. Hope the below points add value to your experiments-</p>\n<h1>Make best use of a half-day inference period</h1>\n<p>I must thank the competition host for being generous and providing us a half-day (12-hour) code submission period for this competition. We have a number of code competitions ongoing and I am happy to opine that this one is the only challenge in the list of assignments that offers this additional period. We need to make full usage of the extended inference period to perhaps make an attempt to climb the leaderboard and build and infer a stable model. </p>\n<ul>\n<li><strong>Do not train in the inference kernel</strong>. The inference kernel is meant to infer and submit only. Training may be done locally and models may be saved using <strong>joblib/ dill/ pickle modules</strong>. LightGBM also has a model saving option using <code>lightgbm.Booster.save_model()</code> and <code>lightgbm.Booster.load_model()</code> as well. Please use these options to good effect and reduce the inference time significantly!</li>\n<li><strong>Use scripts and pipeline architecture to good effect to fit() and transform()</strong>. We have a lot of excellent public materials that offer a great starter resource pool for data preparation. We may develop this and build a pipeline that <strong>fits</strong> the data to the train set and <strong>transforms</strong> the test set. Perhaps designing a script for this is valuable. This can be done in 2 ways-<br>\na. <strong>Utility script</strong> - A kaggle kernel may be dedicated to this pipeline and may be added to the main submission kernel as a utility script<br>\nb. <strong>Using a .py file in a dataset</strong> - I prefer this approach. We may design a script and then save it as a .py file and import and use in a dataset. We need to use <code>shutil.copyfile</code> and then use it akin to a module import. This is as follows- </li>\n</ul>\n<pre><code> shutil\nshutil.copyfile(src = , dst = )\n myscript  *\n</code></pre>\n<ul>\n<li><strong>Clean your memory</strong> off unwanted objects to perhaps reduce the chances of RAM issues. Explicitly deleting objects is a good way to remove them from the process. Use the code as below-<br>\n<code>del Xtest, selected_columns</code></li>\n</ul>\n<p>One may also use a function/ class method as below to clean the memory as below-</p>\n<pre><code> torch, ctypes, gc, os\n os  path, walk, getpid\n psutil  Process\n\n ():\n    \n\n    gc.collect();\n    libc = ctypes.CDLL()\n    libc.malloc_trim()\n    pid        = getpid()\n    py         = Process(pid)\n    memory_use = py.memory_info()[] /  ** \n\n    torch.cuda.empty_cache()\n\n    ()\n</code></pre>\n<h1>Set SMART goals to effectively use time</h1>\n<p>SMART goal setting is key for various endeavors and this is no different. SMART stands for the below-</p>\n<ul>\n<li>S - Specific</li>\n<li>M- Measurable</li>\n<li>A- Attainable</li>\n<li>R- Relevant </li>\n<li>T- Timebound<br>\nEffective goal setting and regular work can help you a lot in this challenge</li>\n</ul>\n<h1>Be creative in your training process</h1>\n<ul>\n<li>Try and save training datasets in 1 kernel. Say, you want to analyse the public dataset with 389 features. You don't need to make and delete it every time on GPU settings. Just save it once using parquet format and use it in all your experiments. This can be done with a simple CPU kernel. My dataset <a href=\"https://www.kaggle.com/datasets/ravi20076/homecreditquality2024xytrain389features/data\" target=\"_blank\">here</a> hosts the table with 389 columns for ready use. Please feel free to include it in your pipelines as and when needed</li>\n<li><strong>Bypassing this step can allow you to train +n LGBM model per week!</strong>. From my usual experiments, I fathom that one needs approximately 60-120 minutes for training a LGBM model depending on the number of folds and features. The process of preparing data takes approximately 20-25 minutes on CPU. Bypassing this step frees up 20 minutes per average of GPU cost and enables you to save multiple hours per week, thus allowing you to use the time saved to better and directed efforts! These experiments could be of great use to you later in the assignment!</li>\n<li><strong>Estimate and plan experiments to better use your 30-GPU hours</strong>. A LGBM model training takes 60-120 minutes on average and Catboost takes 4+ hours from public work. Divide the 30-hours well and plan your time ahead to experiment wisely. Resource planning is a key career growth skill as well. </li>\n<li><strong>Stick to your training plan</strong> and don't deviate in normal circumstances </li>\n<li><strong>Submit wisely, but submit too</strong>. Managing the 150+ submissions from today is of utmost importance</li>\n<li>Spend time in rechecking the code pipeline for bugs</li>\n</ul>\n<h1>Team up well</h1>\n<ul>\n<li>People facing difficulties with hardware may think of teaming up to perhaps augment the overall free compute available through the week. Team up with others interested in the challenge, else the experience is likely to degrade. Teaming up well is a cornerstone for success!</li>\n<li>Try and manage roles within the team and distribute work according to each member's strength and weaknesses</li>\n<li>Divide daily submissions within the team and adhere to the submission plan well</li>\n<li>Communicate effectively and set objectives well</li>\n</ul>\n<h1>Develop a success strategy</h1>\n<ul>\n<li>By now, you should know of how your 2 final submissions will eventually look like and should design a roadmap to achieve it by May22-23 keeping the last week for fine tuning and minor improvements</li>\n<li>Try and build your efforts to work on both submissions. Perhaps assigning resources in a team to work in both submission directions will be key. </li>\n</ul>\n<h1>Concluding thoughts</h1>\n<p>I may opine that we all wish to earn gold medals and prizes in our competitions, but we have a select number of slots for the same. Others may perhaps value the lessons learnt and use them in daily life outside of Kaggle. This is the real life gold medal one earns from Kaggle and it well and truly sustains! </p>\n<p>All the best and best wishes for a long and successful Kaggle journey!</p>",
  "messages": [
    {
      "id": 2778884,
      "postDate": "2024-04-27T12:25:08.370Z",
      "content": "<p>Hello all,</p>\n<p>Hope you all are doing well. The competition is approaching the last month now and I thought we could perhaps discuss how we may collectively manage our resources at our disposal to better effectuate a more organized submission pipeline to garner better effect on the CV and LB scores in totality. Hope the below points add value to your experiments-</p>\n<h1>Make best use of a half-day inference period</h1>\n<p>I must thank the competition host for being generous and providing us a half-day (12-hour) code submission period for this competition. We have a number of code competitions ongoing and I am happy to opine that this one is the only challenge in the list of assignments that offers this additional period. We need to make full usage of the extended inference period to perhaps make an attempt to climb the leaderboard and build and infer a stable model. </p>\n<ul>\n<li><strong>Do not train in the inference kernel</strong>. The inference kernel is meant to infer and submit only. Training may be done locally and models may be saved using <strong>joblib/ dill/ pickle modules</strong>. LightGBM also has a model saving option using <code>lightgbm.Booster.save_model()</code> and <code>lightgbm.Booster.load_model()</code> as well. Please use these options to good effect and reduce the inference time significantly!</li>\n<li><strong>Use scripts and pipeline architecture to good effect to fit() and transform()</strong>. We have a lot of excellent public materials that offer a great starter resource pool for data preparation. We may develop this and build a pipeline that <strong>fits</strong> the data to the train set and <strong>transforms</strong> the test set. Perhaps designing a script for this is valuable. This can be done in 2 ways-<br>\na. <strong>Utility script</strong> - A kaggle kernel may be dedicated to this pipeline and may be added to the main submission kernel as a utility script<br>\nb. <strong>Using a .py file in a dataset</strong> - I prefer this approach. We may design a script and then save it as a .py file and import and use in a dataset. We need to use <code>shutil.copyfile</code> and then use it akin to a module import. This is as follows- </li>\n</ul>\n<pre><code> shutil\nshutil.copyfile(src = , dst = )\n myscript  *\n</code></pre>\n<ul>\n<li><strong>Clean your memory</strong> off unwanted objects to perhaps reduce the chances of RAM issues. Explicitly deleting objects is a good way to remove them from the process. Use the code as below-<br>\n<code>del Xtest, selected_columns</code></li>\n</ul>\n<p>One may also use a function/ class method as below to clean the memory as below-</p>\n<pre><code> torch, ctypes, gc, os\n os  path, walk, getpid\n psutil  Process\n\n ():\n    \n\n    gc.collect();\n    libc = ctypes.CDLL()\n    libc.malloc_trim()\n    pid        = getpid()\n    py         = Process(pid)\n    memory_use = py.memory_info()[] /  ** \n\n    torch.cuda.empty_cache()\n\n    ()\n</code></pre>\n<h1>Set SMART goals to effectively use time</h1>\n<p>SMART goal setting is key for various endeavors and this is no different. SMART stands for the below-</p>\n<ul>\n<li>S - Specific</li>\n<li>M- Measurable</li>\n<li>A- Attainable</li>\n<li>R- Relevant </li>\n<li>T- Timebound<br>\nEffective goal setting and regular work can help you a lot in this challenge</li>\n</ul>\n<h1>Be creative in your training process</h1>\n<ul>\n<li>Try and save training datasets in 1 kernel. Say, you want to analyse the public dataset with 389 features. You don't need to make and delete it every time on GPU settings. Just save it once using parquet format and use it in all your experiments. This can be done with a simple CPU kernel. My dataset <a href=\"https://www.kaggle.com/datasets/ravi20076/homecreditquality2024xytrain389features/data\" target=\"_blank\">here</a> hosts the table with 389 columns for ready use. Please feel free to include it in your pipelines as and when needed</li>\n<li><strong>Bypassing this step can allow you to train +n LGBM model per week!</strong>. From my usual experiments, I fathom that one needs approximately 60-120 minutes for training a LGBM model depending on the number of folds and features. The process of preparing data takes approximately 20-25 minutes on CPU. Bypassing this step frees up 20 minutes per average of GPU cost and enables you to save multiple hours per week, thus allowing you to use the time saved to better and directed efforts! These experiments could be of great use to you later in the assignment!</li>\n<li><strong>Estimate and plan experiments to better use your 30-GPU hours</strong>. A LGBM model training takes 60-120 minutes on average and Catboost takes 4+ hours from public work. Divide the 30-hours well and plan your time ahead to experiment wisely. Resource planning is a key career growth skill as well. </li>\n<li><strong>Stick to your training plan</strong> and don't deviate in normal circumstances </li>\n<li><strong>Submit wisely, but submit too</strong>. Managing the 150+ submissions from today is of utmost importance</li>\n<li>Spend time in rechecking the code pipeline for bugs</li>\n</ul>\n<h1>Team up well</h1>\n<ul>\n<li>People facing difficulties with hardware may think of teaming up to perhaps augment the overall free compute available through the week. Team up with others interested in the challenge, else the experience is likely to degrade. Teaming up well is a cornerstone for success!</li>\n<li>Try and manage roles within the team and distribute work according to each member's strength and weaknesses</li>\n<li>Divide daily submissions within the team and adhere to the submission plan well</li>\n<li>Communicate effectively and set objectives well</li>\n</ul>\n<h1>Develop a success strategy</h1>\n<ul>\n<li>By now, you should know of how your 2 final submissions will eventually look like and should design a roadmap to achieve it by May22-23 keeping the last week for fine tuning and minor improvements</li>\n<li>Try and build your efforts to work on both submissions. Perhaps assigning resources in a team to work in both submission directions will be key. </li>\n</ul>\n<h1>Concluding thoughts</h1>\n<p>I may opine that we all wish to earn gold medals and prizes in our competitions, but we have a select number of slots for the same. Others may perhaps value the lessons learnt and use them in daily life outside of Kaggle. This is the real life gold medal one earns from Kaggle and it well and truly sustains! </p>\n<p>All the best and best wishes for a long and successful Kaggle journey!</p>",
      "rawMarkdown": "Hello all,\n\nHope you all are doing well. The competition is approaching the last month now and I thought we could perhaps discuss how we may collectively manage our resources at our disposal to better effectuate a more organized submission pipeline to garner better effect on the CV and LB scores in totality. Hope the below points add value to your experiments-\n\n# Make best use of a half-day inference period\nI must thank the competition host for being generous and providing us a half-day (12-hour) code submission period for this competition. We have a number of code competitions ongoing and I am happy to opine that this one is the only challenge in the list of assignments that offers this additional period. We need to make full usage of the extended inference period to perhaps make an attempt to climb the leaderboard and build and infer a stable model. \n- **Do not train in the inference kernel**. The inference kernel is meant to infer and submit only. Training may be done locally and models may be saved using **joblib/ dill/ pickle modules**. LightGBM also has a model saving option using `lightgbm.Booster.save_model()` and `lightgbm.Booster.load_model()` as well. Please use these options to good effect and reduce the inference time significantly!\n- **Use scripts and pipeline architecture to good effect to fit() and transform()**. We have a lot of excellent public materials that offer a great starter resource pool for data preparation. We may develop this and build a pipeline that **fits** the data to the train set and **transforms** the test set. Perhaps designing a script for this is valuable. This can be done in 2 ways-\na. **Utility script** - A kaggle kernel may be dedicated to this pipeline and may be added to the main submission kernel as a utility script\nb. **Using a .py file in a dataset** - I prefer this approach. We may design a script and then save it as a .py file and import and use in a dataset. We need to use `shutil.copyfile` and then use it akin to a module import. This is as follows- \n\n```python\nimport shutil\nshutil.copyfile(src = 'your dataset .py file path', dst = 'myscript.py')\nfrom myscript import *\n```\n- **Clean your memory** off unwanted objects to perhaps reduce the chances of RAM issues. Explicitly deleting objects is a good way to remove them from the process. Use the code as below-\n`del Xtest, selected_columns`\n\nOne may also use a function/ class method as below to clean the memory as below-\n\n```python\nimport torch, ctypes, gc, os\nfrom os import path, walk, getpid\nfrom psutil import Process\n\ndef CleanMemory():\n    \"This function cleans the memory off unused objects and displays the cleaned state RAM usage\"\n\n    gc.collect();\n    libc = ctypes.CDLL(\"libc.so.6\")\n    libc.malloc_trim(0)\n    pid        = getpid()\n    py         = Process(pid)\n    memory_use = py.memory_info()[0] / 2. ** 30\n\n    torch.cuda.empty_cache()\n\n    print(f\"\\nRAM usage = {memory_use :.4} GB\")\n```\n\n# Set SMART goals to effectively use time \nSMART goal setting is key for various endeavors and this is no different. SMART stands for the below-\n- S - Specific\n- M- Measurable\n- A- Attainable\n- R- Relevant \n- T- Timebound\nEffective goal setting and regular work can help you a lot in this challenge\n\n# Be creative in your training process\n- Try and save training datasets in 1 kernel. Say, you want to analyse the public dataset with 389 features. You don't need to make and delete it every time on GPU settings. Just save it once using parquet format and use it in all your experiments. This can be done with a simple CPU kernel. My dataset [here](https://www.kaggle.com/datasets/ravi20076/homecreditquality2024xytrain389features/data) hosts the table with 389 columns for ready use. Please feel free to include it in your pipelines as and when needed\n- **Bypassing this step can allow you to train +n LGBM model per week!**. From my usual experiments, I fathom that one needs approximately 60-120 minutes for training a LGBM model depending on the number of folds and features. The process of preparing data takes approximately 20-25 minutes on CPU. Bypassing this step frees up 20 minutes per average of GPU cost and enables you to save multiple hours per week, thus allowing you to use the time saved to better and directed efforts! These experiments could be of great use to you later in the assignment!\n- **Estimate and plan experiments to better use your 30-GPU hours**. A LGBM model training takes 60-120 minutes on average and Catboost takes 4+ hours from public work. Divide the 30-hours well and plan your time ahead to experiment wisely. Resource planning is a key career growth skill as well. \n- **Stick to your training plan** and don't deviate in normal circumstances \n- **Submit wisely, but submit too**. Managing the 150+ submissions from today is of utmost importance\n- Spend time in rechecking the code pipeline for bugs\n\n# Team up well\n- People facing difficulties with hardware may think of teaming up to perhaps augment the overall free compute available through the week. Team up with others interested in the challenge, else the experience is likely to degrade. Teaming up well is a cornerstone for success!\n- Try and manage roles within the team and distribute work according to each member's strength and weaknesses\n- Divide daily submissions within the team and adhere to the submission plan well\n- Communicate effectively and set objectives well\n\n# Develop a success strategy\n- By now, you should know of how your 2 final submissions will eventually look like and should design a roadmap to achieve it by May22-23 keeping the last week for fine tuning and minor improvements\n- Try and build your efforts to work on both submissions. Perhaps assigning resources in a team to work in both submission directions will be key. \n\n# Concluding thoughts\nI may opine that we all wish to earn gold medals and prizes in our competitions, but we have a select number of slots for the same. Others may perhaps value the lessons learnt and use them in daily life outside of Kaggle. This is the real life gold medal one earns from Kaggle and it well and truly sustains! \n\nAll the best and best wishes for a long and successful Kaggle journey!\n\n",
      "votes": 17
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "2778884": "Hello all,\n\nHope you all are doing well. The competition is approaching the last month now and I thought we could perhaps discuss how we may collectively manage our resources at our disposal to better effectuate a more organized submission pipeline to garner better effect on the CV and LB scores in totality. Hope the below points add value to your experiments-\n\n# Make best use of a half-day inference period\nI must thank the competition host for being generous and providing us a half-day (12-hour) code submission period for this competition. We have a number of code competitions ongoing and I am happy to opine that this one is the only challenge in the list of assignments that offers this additional period. We need to make full usage of the extended inference period to perhaps make an attempt to climb the leaderboard and build and infer a stable model. \n- **Do not train in the inference kernel**. The inference kernel is meant to infer and submit only. Training may be done locally and models may be saved using **joblib/ dill/ pickle modules**. LightGBM also has a model saving option using `lightgbm.Booster.save_model()` and `lightgbm.Booster.load_model()` as well. Please use these options to good effect and reduce the inference time significantly!\n- **Use scripts and pipeline architecture to good effect to fit() and transform()**. We have a lot of excellent public materials that offer a great starter resource pool for data preparation. We may develop this and build a pipeline that **fits** the data to the train set and **transforms** the test set. Perhaps designing a script for this is valuable. This can be done in 2 ways-\na. **Utility script** - A kaggle kernel may be dedicated to this pipeline and may be added to the main submission kernel as a utility script\nb. **Using a .py file in a dataset** - I prefer this approach. We may design a script and then save it as a .py file and import and use in a dataset. We need to use `shutil.copyfile` and then use it akin to a module import. This is as follows- \n\n```python\nimport shutil\nshutil.copyfile(src = 'your dataset .py file path', dst = 'myscript.py')\nfrom myscript import *\n```\n- **Clean your memory** off unwanted objects to perhaps reduce the chances of RAM issues. Explicitly deleting objects is a good way to remove them from the process. Use the code as below-\n`del Xtest, selected_columns`\n\nOne may also use a function/ class method as below to clean the memory as below-\n\n```python\nimport torch, ctypes, gc, os\nfrom os import path, walk, getpid\nfrom psutil import Process\n\ndef CleanMemory():\n    \"This function cleans the memory off unused objects and displays the cleaned state RAM usage\"\n\n    gc.collect();\n    libc = ctypes.CDLL(\"libc.so.6\")\n    libc.malloc_trim(0)\n    pid        = getpid()\n    py         = Process(pid)\n    memory_use = py.memory_info()[0] / 2. ** 30\n\n    torch.cuda.empty_cache()\n\n    print(f\"\\nRAM usage = {memory_use :.4} GB\")\n```\n\n# Set SMART goals to effectively use time \nSMART goal setting is key for various endeavors and this is no different. SMART stands for the below-\n- S - Specific\n- M- Measurable\n- A- Attainable\n- R- Relevant \n- T- Timebound\nEffective goal setting and regular work can help you a lot in this challenge\n\n# Be creative in your training process\n- Try and save training datasets in 1 kernel. Say, you want to analyse the public dataset with 389 features. You don't need to make and delete it every time on GPU settings. Just save it once using parquet format and use it in all your experiments. This can be done with a simple CPU kernel. My dataset [here](https://www.kaggle.com/datasets/ravi20076/homecreditquality2024xytrain389features/data) hosts the table with 389 columns for ready use. Please feel free to include it in your pipelines as and when needed\n- **Bypassing this step can allow you to train +n LGBM model per week!**. From my usual experiments, I fathom that one needs approximately 60-120 minutes for training a LGBM model depending on the number of folds and features. The process of preparing data takes approximately 20-25 minutes on CPU. Bypassing this step frees up 20 minutes per average of GPU cost and enables you to save multiple hours per week, thus allowing you to use the time saved to better and directed efforts! These experiments could be of great use to you later in the assignment!\n- **Estimate and plan experiments to better use your 30-GPU hours**. A LGBM model training takes 60-120 minutes on average and Catboost takes 4+ hours from public work. Divide the 30-hours well and plan your time ahead to experiment wisely. Resource planning is a key career growth skill as well. \n- **Stick to your training plan** and don't deviate in normal circumstances \n- **Submit wisely, but submit too**. Managing the 150+ submissions from today is of utmost importance\n- Spend time in rechecking the code pipeline for bugs\n\n# Team up well\n- People facing difficulties with hardware may think of teaming up to perhaps augment the overall free compute available through the week. Team up with others interested in the challenge, else the experience is likely to degrade. Teaming up well is a cornerstone for success!\n- Try and manage roles within the team and distribute work according to each member's strength and weaknesses\n- Divide daily submissions within the team and adhere to the submission plan well\n- Communicate effectively and set objectives well\n\n# Develop a success strategy\n- By now, you should know of how your 2 final submissions will eventually look like and should design a roadmap to achieve it by May22-23 keeping the last week for fine tuning and minor improvements\n- Try and build your efforts to work on both submissions. Perhaps assigning resources in a team to work in both submission directions will be key. \n\n# Concluding thoughts\nI may opine that we all wish to earn gold medals and prizes in our competitions, but we have a select number of slots for the same. Others may perhaps value the lessons learnt and use them in daily life outside of Kaggle. This is the real life gold medal one earns from Kaggle and it well and truly sustains! \n\nAll the best and best wishes for a long and successful Kaggle journey!\n\n"
  }
}