{ "id": 368000, "title": "πŸ’‘ how to move faster on an ML project and achieve more with less compute/time/energy (relevant to this competition)", "url": "/competitions/otto-recommender-system/discussion/368000", "author_name": "", "post_date": "2022-11-22T22:57:58.350075400Z", "votes": 31, "comment_count": 14, "views": 0, "content": "

Inspired by the very positive reception of my recent posts (How to thrive in this competition without going crazy ❀️‍πŸ”₯, let me share a couple of things that have been on my mind recently.

\n

A couple of techniques that will allow you to make faster progress on ML projects and that you can use in this competition!

\n

Get out of the waiting hell

\n

In any project, your time is the most valuable asset you have!

\n

So how does it normally play out?

\n

You tweak a small piece of code, run your notebook, and wait for 20 minutes only to find you made a typo and need to rerun the experiment again! 😱

\n

What is the solution? The debug flag!

\n

This is from the top of a notebook I am currently working on:

\n
VALIDATE = True\nDEBUG = True\n\nSEED = 0\n
\n

With DEBUG == True the notebook will execute on around 1% of data, if not less. It is blazingly fast.

\n

I am keeping this variable right at the top of my notebook to get in the habit of using it!

\n

Whenever I do, I save time. Whenever I don't, I end up rerunning the same code over and over again to fix small issues… This not only means I need to wait longer for the results, but I also need to keep spending my attention on babysitting the notebook! Troubleshooting it over and over again.

\n

With first running with DEBUG == True, I get to fix all the issues at one go and can only (in most cases) come back to evaluate the results.

\n

Separate writing code from running experiments

\n

This is a tough one though somewhat related to the point above.

\n

You need to make your life to stop being GIL locked and embrace paralellism as polars does natively πŸ˜„

\n

Essentially, you don't want your ability to write code (or do other meaningful things in life) to be blocked by waiting for the running of code to finish!

\n

Using the DEBUG flag is helpful, but it is only a start.

\n

The mindset is the key.

\n

I am trying to get in the mindset of separating writing code from running experiments (which is really hard to do when all you have at your disposal is your local machine, it is just so tempting to keep tweaking and running things live, but this changes you into a zombie, your brain is gone after an hour two of such an activity).

\n

So the general setup is:

\n
    \n
  1. Write code that trains a model and get it running.
    2. \n
  2. Write code that runs experiments using the code above. Could be looking for better hyperparams, etc.
    3. \n
  3. Once you have written the code, RUN IT, and go do something else!
    4. \n
\n

It sounds super simple, but this right here holds the mystery to achieving more on the things that matter.

\n

Prioritize relentlessly

\n

The search space of an ML solution is immense. For all intents and purposes, it is infinite.

\n

It is so easy to go down a rabbit hole of tweaking just this one small part of a solution… You can spend weeks developing a better model of a given arch for instance.

\n

But what you need to do is this. Ask yourself -- what is the highest point of leverage in this project I could be working on? Usually, it will stand out as a sore thumb!

\n

And you don't even have to get it perfectly right. Just roughly working on something that has a chance of helping your solution is 1000x better than being pulled into solving an interesting but low-impact problem (AKA time sink).

\n

Grow the solution organically

\n

There were competitions where I got crushed by the complexity of my code. I started working on a part of a solution, then another, but it all became so complex that a) I lost interest in the competition b) it stopped being fun c) the task of integrating the pieces felt overwhelming and not worth it.

\n

The solution to this is to keep growing your code organically!

\n

What do I mean by this?

\n

You probably don't need to go super deep on tweaking our ranking model IF you don't have the code for outputting a submission!

\n

You probably don't need to integrate this super elaborate score from a super elaborate DL model until you figure out how to ensemble simple solutions.

\n

You probably don't need to go super duper deep on co-visitation matrices IF you know you are building towards a solution that will involve a ranking model.

\n

Better put in place an end-to-end pipeline of roughly what you'd like to attempt AND only then keep growing each of the components. And keep integrating the pieces as you go so that it all runs!

\n

This also guarantees you receive feedback on your changes and know whether you are introducing bugs or moving the right direction.

\n

Summary

\n

Hoping what I wrote above can be of genuine help to you πŸ™ If you find it useful, please upvote, and tomorrow I will post how I am addressing dealing with RAM in this competition. You need so much of it!

\n

I nearly bought a fun piece of equipment to address this problem πŸ˜„ But decided otherwise as not sure where I would keep it plus have a bunch of other expenses so the family budget is not in a happy spot. But it is a good story I feel πŸ™‚ And am using a different solution that I would like to share and that so far works!

\n

Anyhow, let's see how it goes and whether I am on the right track with these posts πŸ™‚ Would hate to spam people with stuff they don't find interesting.

\n

So please upvote if you feel this is useful content and hopefully talk to you tomorrow! πŸ™‚

\n

Other resources you might find useful:

\n", "messages": [ { "id": "2040323", "postDate": "11/22/2022 22:57:58", "content": "

Inspired by the very positive reception of my recent posts (How to thrive in this competition without going crazy ❀️‍πŸ”₯, let me share a couple of things that have been on my mind recently.

\n

A couple of techniques that will allow you to make faster progress on ML projects and that you can use in this competition!

\n

Get out of the waiting hell

\n

In any project, your time is the most valuable asset you have!

\n

So how does it normally play out?

\n

You tweak a small piece of code, run your notebook, and wait for 20 minutes only to find you made a typo and need to rerun the experiment again! 😱

\n

What is the solution? The debug flag!

\n

This is from the top of a notebook I am currently working on:

\n
VALIDATE = True\nDEBUG = True\n\nSEED = 0\n
\n

With DEBUG == True the notebook will execute on around 1% of data, if not less. It is blazingly fast.

\n

I am keeping this variable right at the top of my notebook to get in the habit of using it!

\n

Whenever I do, I save time. Whenever I don't, I end up rerunning the same code over and over again to fix small issues… This not only means I need to wait longer for the results, but I also need to keep spending my attention on babysitting the notebook! Troubleshooting it over and over again.

\n

With first running with DEBUG == True, I get to fix all the issues at one go and can only (in most cases) come back to evaluate the results.

\n

Separate writing code from running experiments

\n

This is a tough one though somewhat related to the point above.

\n

You need to make your life to stop being GIL locked and embrace paralellism as polars does natively πŸ˜„

\n

Essentially, you don't want your ability to write code (or do other meaningful things in life) to be blocked by waiting for the running of code to finish!

\n

Using the DEBUG flag is helpful, but it is only a start.

\n

The mindset is the key.

\n

I am trying to get in the mindset of separating writing code from running experiments (which is really hard to do when all you have at your disposal is your local machine, it is just so tempting to keep tweaking and running things live, but this changes you into a zombie, your brain is gone after an hour two of such an activity).

\n

So the general setup is:

\n
    \n
  1. Write code that trains a model and get it running.
    2. \n
  2. Write code that runs experiments using the code above. Could be looking for better hyperparams, etc.
    3. \n
  3. Once you have written the code, RUN IT, and go do something else!
    4. \n
\n

It sounds super simple, but this right here holds the mystery to achieving more on the things that matter.

\n

Prioritize relentlessly

\n

The search space of an ML solution is immense. For all intents and purposes, it is infinite.

\n

It is so easy to go down a rabbit hole of tweaking just this one small part of a solution… You can spend weeks developing a better model of a given arch for instance.

\n

But what you need to do is this. Ask yourself -- what is the highest point of leverage in this project I could be working on? Usually, it will stand out as a sore thumb!

\n

And you don't even have to get it perfectly right. Just roughly working on something that has a chance of helping your solution is 1000x better than being pulled into solving an interesting but low-impact problem (AKA time sink).

\n

Grow the solution organically

\n

There were competitions where I got crushed by the complexity of my code. I started working on a part of a solution, then another, but it all became so complex that a) I lost interest in the competition b) it stopped being fun c) the task of integrating the pieces felt overwhelming and not worth it.

\n

The solution to this is to keep growing your code organically!

\n

What do I mean by this?

\n

You probably don't need to go super deep on tweaking our ranking model IF you don't have the code for outputting a submission!

\n

You probably don't need to integrate this super elaborate score from a super elaborate DL model until you figure out how to ensemble simple solutions.

\n

You probably don't need to go super duper deep on co-visitation matrices IF you know you are building towards a solution that will involve a ranking model.

\n

Better put in place an end-to-end pipeline of roughly what you'd like to attempt AND only then keep growing each of the components. And keep integrating the pieces as you go so that it all runs!

\n

This also guarantees you receive feedback on your changes and know whether you are introducing bugs or moving the right direction.

\n

Summary

\n

Hoping what I wrote above can be of genuine help to you πŸ™ If you find it useful, please upvote, and tomorrow I will post how I am addressing dealing with RAM in this competition. You need so much of it!

\n

I nearly bought a fun piece of equipment to address this problem πŸ˜„ But decided otherwise as not sure where I would keep it plus have a bunch of other expenses so the family budget is not in a happy spot. But it is a good story I feel πŸ™‚ And am using a different solution that I would like to share and that so far works!

\n

Anyhow, let's see how it goes and whether I am on the right track with these posts πŸ™‚ Would hate to spam people with stuff they don't find interesting.

\n

So please upvote if you feel this is useful content and hopefully talk to you tomorrow! πŸ™‚

\n

Other resources you might find useful:

\n", "rawMarkdown": "Inspired by the very positive reception of my recent posts ([How to thrive in this competition without going crazy ❀️‍πŸ”₯](https://www.kaggle.com/competitions/otto-recommender-system/discussion/367503), let me share a couple of things that have been on my mind recently.\n\nA couple of techniques that will allow you to make faster progress on ML projects and that you can use in this competition!\n\n### Get out of the waiting hell\n\nIn any project, your time is the most valuable asset you have!\n\nSo how does it normally play out?\n\nYou tweak a small piece of code, run your notebook, and wait for 20 minutes only to find you made a typo and need to rerun the experiment again! 😱\n\nWhat is the solution? The `debug` flag!\n\nThis is from the top of a notebook I am currently working on:\n```\nVALIDATE = True\nDEBUG = True\n\nSEED = 0\n```\n\nWith `DEBUG == True` the notebook will execute on around 1% of data, if not less. It is blazingly fast.\n\nI am keeping this variable right at the top of my notebook to get in the habit of using it!\n\nWhenever I do, I save time. Whenever I don't, I end up rerunning the same code over and over again to fix small issues... This not only means I need to wait longer for the results, but I also need to keep spending my attention on babysitting the notebook! Troubleshooting it over and over again.\n\nWith first running with `DEBUG == True`, I get to fix all the issues at one go and can only (in most cases) come back to evaluate the results.\n\n### Separate writing code from running experiments\n\nThis is a tough one though somewhat related to the point above.\n\nYou need to make your life to stop being GIL locked and embrace paralellism as polars does natively πŸ˜„\n\nEssentially, you don't want your ability to write code (or do other meaningful things in life) to be blocked by waiting for the running of code to finish!\n\nUsing the `DEBUG` flag is helpful, but it is only a start.\n\nThe mindset is the key.\n\nI am trying to get in the mindset of separating writing code from running experiments (which is really hard to do when all you have at your disposal is your local machine, it is just so tempting to keep tweaking and running things live, but this changes you into a zombie, your brain is gone after an hour two of such an activity).\n\nSo the general setup is:\n1. Write code that trains a model and get it running.\n2. Write code that runs experiments using the code above. Could be looking for better hyperparams, etc.\n3. Once you have written the code, RUN IT, and go do something else!\n\nIt sounds super simple, but this right here holds the mystery to achieving more on the things that matter.\n\n### Prioritize relentlessly\n\nThe search space of an ML solution is immense. For all intents and purposes, it is infinite.\n\nIt is so easy to go down a rabbit hole of tweaking just this one small part of a solution... You can spend weeks developing a better model of a given arch for instance.\n\nBut what you need to do is this. Ask yourself -- what is the highest point of leverage in this project I could be working on? Usually, it will stand out as a sore thumb!\n\nAnd you don't even have to get it perfectly right. Just roughly working on something that has a chance of helping your solution is 1000x better than being pulled into solving an interesting but low-impact problem (AKA time sink).\n\n### Grow the solution organically\n\nThere were competitions where I got crushed by the complexity of my code. I started working on a part of a solution, then another, but it all became so complex that a) I lost interest in the competition b) it stopped being fun c) the task of integrating the pieces felt overwhelming and not worth it.\n\nThe solution to this is to keep growing your code organically!\n\nWhat do I mean by this?\n\nYou probably don't need to go super deep on tweaking our ranking model IF you don't have the code for outputting a submission!\n\nYou probably don't need to integrate this super elaborate score from a super elaborate DL model until you figure out how to ensemble simple solutions.\n\nYou probably don't need to go super duper deep on co-visitation matrices IF you know you are building towards a solution that will involve a ranking model.\n\nBetter put in place an end-to-end pipeline of roughly what you'd like to attempt AND only then keep growing each of the components. And keep integrating the pieces as you go so that it all runs!\n\nThis also guarantees you receive feedback on your changes and know whether you are introducing bugs or moving the right direction.\n\n### Summary\n\nHoping what I wrote above can be of genuine help to you πŸ™ If you find it useful, please upvote, and tomorrow I will post how I am addressing dealing with RAM in this competition. You need so much of it!\n\nI nearly bought a fun piece of equipment to address this problem πŸ˜„ But decided otherwise as not sure where I would keep it plus have a bunch of other expenses so the family budget is not in a happy spot. But it is a good story I feel πŸ™‚ And am using a different solution that I would like to share and that so far works!\n\nAnyhow, let's see how it goes and whether I am on the right track with these posts πŸ™‚ Would hate to spam people with stuff they don't find interesting.\n\nSo please upvote if you feel this is useful content and hopefully talk to you tomorrow! πŸ™‚\n\n### Other resources you might find useful:\n\n* [πŸ’‘ [2 methods] How-to ensemble predictions πŸ…πŸ…πŸ…](https://www.kaggle.com/code/radek1/2-methods-how-to-ensemble-predictions)\n* [local validation tracks public LB perfecty -- here is the setup](https://www.kaggle.com/competitions/otto-recommender-system/discussion/364991)\n* [πŸ’‘ For my friends from Twitter and LinkedIn -- here is how to dive into this competition 🐳](https://www.kaggle.com/competitions/otto-recommender-system/discussion/368560)\n* [Full dataset processed to CSV/parquet files with optimized memory footprint](https://www.kaggle.com/competitions/otto-recommender-system/discussion/363843)\n* [co-visitation matrix - simplified, imprvd logic πŸ”₯](https://www.kaggle.com/code/radek1/co-visitation-matrix-simplified-imprvd-logic)\n* [πŸ’‘ Word2Vec How-to [training and submission]πŸš€πŸš€πŸš€](https://www.kaggle.com/code/radek1/word2vec-how-to-training-and-submission)", "votes": null }, { "id": "2040470", "postDate": "11/23/2022 04:16:38", "content": "

Great advice! Thank you Radek, looking forward to your new posts!

", "rawMarkdown": "Great advice! Thank you Radek, looking forward to your new posts!", "votes": null }, { "id": "2040475", "postDate": "11/23/2022 04:21:36", "content": "

Thank you very much, @zachary666! Really appreciate your kind comment and the voice of confidence! Thank you! πŸ™

", "rawMarkdown": "Thank you very much, @zachary666! Really appreciate your kind comment and the voice of confidence! Thank you! πŸ™", "votes": null }, { "id": "2044501", "postDate": "11/26/2022 15:15:27", "content": "

Really can't keep up with your amazing sharing! So much great stuff from you! Thank you very much!

", "rawMarkdown": "Really can't keep up with your amazing sharing! So much great stuff from you! Thank you very much!", "votes": null }, { "id": "2044924", "postDate": "11/26/2022 20:33:03", "content": "

Thank you for this. This is super useful.

", "rawMarkdown": "Thank you for this. This is super useful.", "votes": null }, { "id": "2044964", "postDate": "11/26/2022 21:52:33", "content": "

Great to hear, @lordxerxes! πŸ™‚ Glad to be of help

", "rawMarkdown": "Great to hear, @lordxerxes! πŸ™‚ Glad to be of help", "votes": null }, { "id": "2044965", "postDate": "11/26/2022 21:53:12", "content": "

Haha, thanks @danielliao! πŸ™‚ Really glad you are finding this useful!

", "rawMarkdown": "Haha, thanks @danielliao! πŸ™‚ Really glad you are finding this useful!", "votes": null }, { "id": "2045229", "postDate": "11/27/2022 07:29:58", "content": "

Hi @radek1, thank you so much for the advice on fast debugging with just 1% of the dataset.

\n

I have implemented the DEBUG option for @cdeotte Candidate ReRank CV notebook in order to debug faster, the full dataset notebook runs just under 40 mins, and the DEBUG version is just under 4 mins. This is the best I can do without changing much of the original notebook. Is 4 min fast enough, what do you think?

\n

Is the DEBUG version solely for speeding up to see whether all codes runs without error? Does it do anything else for us?

\n

Below is the code added or changed (very little) to the original notebook. Do you mind to have a look to see whether I have done it properly? πŸ™ Thanks!

\n

\"\"

\n

\"\"

\n

\"\"

\n

\"\"

", "rawMarkdown": "Hi @radek1, thank you so much for the advice on fast debugging with just 1% of the dataset. \n\nI have implemented the DEBUG option for @cdeotte Candidate ReRank CV notebook in order to debug faster, the full dataset notebook runs just under 40 mins, and the DEBUG version is just under 4 mins. This is the best I can do without changing much of the original notebook. Is 4 min fast enough, what do you think?\n\nIs the DEBUG version solely for speeding up to see whether all codes runs without error? Does it do anything else for us?\n\nBelow is the code added or changed (very little) to the original notebook. Do you mind to have a look to see whether I have done it properly? πŸ™ Thanks!\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F850197%2F490c42bcf2869356acc1b9d33c638550%2FScreen%20Shot%202022-11-27%20at%2015.06.37.png?generation=1669533977515751&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F850197%2Fb84e0c710cd9c313b33973021e98ea39%2FScreen%20Shot%202022-11-27%20at%2015.06.50.png?generation=1669533988729535&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F850197%2F36c62b2f43c28941e130889d1560ffdf%2FScreen%20Shot%202022-11-27%20at%2015.06.58.png?generation=1669534000990752&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F850197%2F9262c5a744e69262d26f316dfd87787e%2FScreen%20Shot%202022-11-27%20at%2015.07.07.png?generation=1669534011721440&alt=media)", "votes": null }, { "id": "2045245", "postDate": "11/27/2022 07:37:38", "content": "
\n

Is the DEBUG version solely for speeding up to see whether all codes runs without error?

\n
\n

Yes, that is the way I do it πŸ™‚

\n

The faster the code can run, the better πŸ™‚ 4 minutes is okay but in my experience, this would still interfere with the way I work on stuff. What are you going to do in those 4 minutes? I get distracted, go on Twitter, etc.

\n

Of course, life isn't perfect, I still often wait for stuff, but if I can just make sure the notebook runs in a couple of seconds before doing the full run, then I can take my attention and do something else, work on other stuff, without having to wait πŸ™‚ But 4 minutes vs 40 minutes and finding out at the 39th minute you have a typo is already an enormous step in the right direction!

", "rawMarkdown": "> Is the DEBUG version solely for speeding up to see whether all codes runs without error? \n\nYes, that is the way I do it πŸ™‚\n\nThe faster the code can run, the better πŸ™‚ 4 minutes is okay but in my experience, this would still interfere with the way I work on stuff. What are you going to do in those 4 minutes? I get distracted, go on Twitter, etc.\n\nOf course, life isn't perfect, I still often wait for stuff, but if I can just make sure the notebook runs in a couple of seconds before doing the full run, then I can take my attention and do something else, work on other stuff, without having to wait πŸ™‚ But 4 minutes vs 40 minutes and finding out at the 39th minute you have a typo is already an enormous step in the right direction!", "votes": null }, { "id": "2045328", "postDate": "11/27/2022 09:03:36", "content": "

Thank you so much Radek! You are absolutely right, even 4 minutes we can still easily get distracted. It would be great if it can just take a few seconds to run. I will dig a little more to see whether I can make it faster.

", "rawMarkdown": "Thank you so much Radek! You are absolutely right, even 4 minutes we can still easily get distracted. It would be great if it can just take a few seconds to run. I will dig a little more to see whether I can make it faster.", "votes": null }, { "id": "2045409", "postDate": "11/27/2022 10:18:03", "content": "

Hi Radek, I am happy to report that I have reduced the time to 90s from 230s, adding a few more lines of code.

\n

Now, I managed to make it run within 34s.

\n

Without your great feedback I would dream to reduce the time from 4 mins to 30s. Thanks @radek1 πŸ™

", "rawMarkdown": "Hi Radek, I am happy to report that I have reduced the time to 90s from 230s, adding a few more lines of code.\n\nNow, I managed to make it run within 34s. \n\nWithout your great feedback I would dream to reduce the time from 4 mins to 30s. Thanks @radek1 πŸ™", "votes": null }, { "id": "2120439", "postDate": "01/29/2023 15:44:20", "content": "

Great ! I didn't know about the first trick, thanks

", "rawMarkdown": "Great ! I didn't know about the first trick, thanks", "votes": null }, { "id": "2120583", "postDate": "01/29/2023 17:32:21", "content": "

@danielliao , how were you able to do it?

", "rawMarkdown": "danielliao , how were you able to do it?", "votes": null }, { "id": "2120585", "postDate": "01/29/2023 17:33:33", "content": "

Who later got to know how to run it in much less time?

", "rawMarkdown": "Who later got to know how to run it in much less time?", "votes": null }, { "id": "2122254", "postDate": "01/30/2023 19:53:21", "content": "

My trick of that sort is to always round cv metric. It is important to avoid paying attention to those constantly changing multiple digits and tell youself honestly \"those changes didn't improve the model\".

", "rawMarkdown": "My trick of that sort is to always round cv metric. It is important to avoid paying attention to those constantly changing multiple digits and tell youself honestly \"those changes didn't improve the model\".", "votes": null } ], "comments": [ { "id": 2040470, "author_name": "zachary666", "author_url": "", "post_date": "11/23/2022 04:16:38", "content": "

Great advice! Thank you Radek, looking forward to your new posts!

", "votes": null, "replies": [ { "id": 2040475, "author_name": "radek1", "author_url": "", "post_date": "11/23/2022 04:21:36", "content": "

Thank you very much, @zachary666! Really appreciate your kind comment and the voice of confidence! Thank you! πŸ™

", "votes": null, "replies": [] } ] }, { "id": 2044501, "author_name": "danielliao", "author_url": "", "post_date": "11/26/2022 15:15:27", "content": "

Really can't keep up with your amazing sharing! So much great stuff from you! Thank you very much!

", "votes": null, "replies": [ { "id": 2044965, "author_name": "radek1", "author_url": "", "post_date": "11/26/2022 21:53:12", "content": "

Haha, thanks @danielliao! πŸ™‚ Really glad you are finding this useful!

", "votes": null, "replies": [] } ] }, { "id": 2044924, "author_name": "lordxerxes", "author_url": "", "post_date": "11/26/2022 20:33:03", "content": "

Thank you for this. This is super useful.

", "votes": null, "replies": [ { "id": 2044964, "author_name": "radek1", "author_url": "", "post_date": "11/26/2022 21:52:33", "content": "

Great to hear, @lordxerxes! πŸ™‚ Glad to be of help

", "votes": null, "replies": [] } ] }, { "id": 2045229, "author_name": "danielliao", "author_url": "", "post_date": "11/27/2022 07:29:58", "content": "

Hi @radek1, thank you so much for the advice on fast debugging with just 1% of the dataset.

\n

I have implemented the DEBUG option for @cdeotte Candidate ReRank CV notebook in order to debug faster, the full dataset notebook runs just under 40 mins, and the DEBUG version is just under 4 mins. This is the best I can do without changing much of the original notebook. Is 4 min fast enough, what do you think?

\n

Is the DEBUG version solely for speeding up to see whether all codes runs without error? Does it do anything else for us?

\n

Below is the code added or changed (very little) to the original notebook. Do you mind to have a look to see whether I have done it properly? πŸ™ Thanks!

\n

\"\"

\n

\"\"

\n

\"\"

\n

\"\"

", "votes": null, "replies": [ { "id": 2045245, "author_name": "radek1", "author_url": "", "post_date": "11/27/2022 07:37:38", "content": "
\n

Is the DEBUG version solely for speeding up to see whether all codes runs without error?

\n
\n

Yes, that is the way I do it πŸ™‚

\n

The faster the code can run, the better πŸ™‚ 4 minutes is okay but in my experience, this would still interfere with the way I work on stuff. What are you going to do in those 4 minutes? I get distracted, go on Twitter, etc.

\n

Of course, life isn't perfect, I still often wait for stuff, but if I can just make sure the notebook runs in a couple of seconds before doing the full run, then I can take my attention and do something else, work on other stuff, without having to wait πŸ™‚ But 4 minutes vs 40 minutes and finding out at the 39th minute you have a typo is already an enormous step in the right direction!

", "votes": null, "replies": [] }, { "id": 2045328, "author_name": "danielliao", "author_url": "", "post_date": "11/27/2022 09:03:36", "content": "

Thank you so much Radek! You are absolutely right, even 4 minutes we can still easily get distracted. It would be great if it can just take a few seconds to run. I will dig a little more to see whether I can make it faster.

", "votes": null, "replies": [] }, { "id": 2045409, "author_name": "danielliao", "author_url": "", "post_date": "11/27/2022 10:18:03", "content": "

Hi Radek, I am happy to report that I have reduced the time to 90s from 230s, adding a few more lines of code.

\n

Now, I managed to make it run within 34s.

\n

Without your great feedback I would dream to reduce the time from 4 mins to 30s. Thanks @radek1 πŸ™

", "votes": null, "replies": [ { "id": 2120583, "author_name": "lordxerxes", "author_url": "", "post_date": "01/29/2023 17:32:21", "content": "

@danielliao , how were you able to do it?

", "votes": null, "replies": [] } ] } ] }, { "id": 2120439, "author_name": "rayanaay", "author_url": "", "post_date": "01/29/2023 15:44:20", "content": "

Great ! I didn't know about the first trick, thanks

", "votes": null, "replies": [] }, { "id": 2120585, "author_name": "lordxerxes", "author_url": "", "post_date": "01/29/2023 17:33:33", "content": "

Who later got to know how to run it in much less time?

", "votes": null, "replies": [] }, { "id": 2122254, "author_name": "artemfedorov", "author_url": "", "post_date": "01/30/2023 19:53:21", "content": "

My trick of that sort is to always round cv metric. It is important to avoid paying attention to those constantly changing multiple digits and tell youself honestly \"those changes didn't improve the model\".

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2040323": "Inspired by the very positive reception of my recent posts ([How to thrive in this competition without going crazy ❀️‍πŸ”₯](https://www.kaggle.com/competitions/otto-recommender-system/discussion/367503), let me share a couple of things that have been on my mind recently.\n\nA couple of techniques that will allow you to make faster progress on ML projects and that you can use in this competition!\n\n### Get out of the waiting hell\n\nIn any project, your time is the most valuable asset you have!\n\nSo how does it normally play out?\n\nYou tweak a small piece of code, run your notebook, and wait for 20 minutes only to find you made a typo and need to rerun the experiment again! 😱\n\nWhat is the solution? The `debug` flag!\n\nThis is from the top of a notebook I am currently working on:\n```\nVALIDATE = True\nDEBUG = True\n\nSEED = 0\n```\n\nWith `DEBUG == True` the notebook will execute on around 1% of data, if not less. It is blazingly fast.\n\nI am keeping this variable right at the top of my notebook to get in the habit of using it!\n\nWhenever I do, I save time. Whenever I don't, I end up rerunning the same code over and over again to fix small issues... This not only means I need to wait longer for the results, but I also need to keep spending my attention on babysitting the notebook! Troubleshooting it over and over again.\n\nWith first running with `DEBUG == True`, I get to fix all the issues at one go and can only (in most cases) come back to evaluate the results.\n\n### Separate writing code from running experiments\n\nThis is a tough one though somewhat related to the point above.\n\nYou need to make your life to stop being GIL locked and embrace paralellism as polars does natively πŸ˜„\n\nEssentially, you don't want your ability to write code (or do other meaningful things in life) to be blocked by waiting for the running of code to finish!\n\nUsing the `DEBUG` flag is helpful, but it is only a start.\n\nThe mindset is the key.\n\nI am trying to get in the mindset of separating writing code from running experiments (which is really hard to do when all you have at your disposal is your local machine, it is just so tempting to keep tweaking and running things live, but this changes you into a zombie, your brain is gone after an hour two of such an activity).\n\nSo the general setup is:\n1. Write code that trains a model and get it running.\n2. Write code that runs experiments using the code above. Could be looking for better hyperparams, etc.\n3. Once you have written the code, RUN IT, and go do something else!\n\nIt sounds super simple, but this right here holds the mystery to achieving more on the things that matter.\n\n### Prioritize relentlessly\n\nThe search space of an ML solution is immense. For all intents and purposes, it is infinite.\n\nIt is so easy to go down a rabbit hole of tweaking just this one small part of a solution... You can spend weeks developing a better model of a given arch for instance.\n\nBut what you need to do is this. Ask yourself -- what is the highest point of leverage in this project I could be working on? Usually, it will stand out as a sore thumb!\n\nAnd you don't even have to get it perfectly right. Just roughly working on something that has a chance of helping your solution is 1000x better than being pulled into solving an interesting but low-impact problem (AKA time sink).\n\n### Grow the solution organically\n\nThere were competitions where I got crushed by the complexity of my code. I started working on a part of a solution, then another, but it all became so complex that a) I lost interest in the competition b) it stopped being fun c) the task of integrating the pieces felt overwhelming and not worth it.\n\nThe solution to this is to keep growing your code organically!\n\nWhat do I mean by this?\n\nYou probably don't need to go super deep on tweaking our ranking model IF you don't have the code for outputting a submission!\n\nYou probably don't need to integrate this super elaborate score from a super elaborate DL model until you figure out how to ensemble simple solutions.\n\nYou probably don't need to go super duper deep on co-visitation matrices IF you know you are building towards a solution that will involve a ranking model.\n\nBetter put in place an end-to-end pipeline of roughly what you'd like to attempt AND only then keep growing each of the components. And keep integrating the pieces as you go so that it all runs!\n\nThis also guarantees you receive feedback on your changes and know whether you are introducing bugs or moving the right direction.\n\n### Summary\n\nHoping what I wrote above can be of genuine help to you πŸ™ If you find it useful, please upvote, and tomorrow I will post how I am addressing dealing with RAM in this competition. You need so much of it!\n\nI nearly bought a fun piece of equipment to address this problem πŸ˜„ But decided otherwise as not sure where I would keep it plus have a bunch of other expenses so the family budget is not in a happy spot. But it is a good story I feel πŸ™‚ And am using a different solution that I would like to share and that so far works!\n\nAnyhow, let's see how it goes and whether I am on the right track with these posts πŸ™‚ Would hate to spam people with stuff they don't find interesting.\n\nSo please upvote if you feel this is useful content and hopefully talk to you tomorrow! πŸ™‚\n\n### Other resources you might find useful:\n\n* [πŸ’‘ [2 methods] How-to ensemble predictions πŸ…πŸ…πŸ…](https://www.kaggle.com/code/radek1/2-methods-how-to-ensemble-predictions)\n* [local validation tracks public LB perfecty -- here is the setup](https://www.kaggle.com/competitions/otto-recommender-system/discussion/364991)\n* [πŸ’‘ For my friends from Twitter and LinkedIn -- here is how to dive into this competition 🐳](https://www.kaggle.com/competitions/otto-recommender-system/discussion/368560)\n* [Full dataset processed to CSV/parquet files with optimized memory footprint](https://www.kaggle.com/competitions/otto-recommender-system/discussion/363843)\n* [co-visitation matrix - simplified, imprvd logic πŸ”₯](https://www.kaggle.com/code/radek1/co-visitation-matrix-simplified-imprvd-logic)\n* [πŸ’‘ Word2Vec How-to [training and submission]πŸš€πŸš€πŸš€](https://www.kaggle.com/code/radek1/word2vec-how-to-training-and-submission)", "2040470": "Great advice! Thank you Radek, looking forward to your new posts!", "2040475": "Thank you very much, @zachary666! Really appreciate your kind comment and the voice of confidence! Thank you! πŸ™", "2044501": "Really can't keep up with your amazing sharing! So much great stuff from you! Thank you very much!", "2044924": "Thank you for this. This is super useful.", "2044964": "Great to hear, @lordxerxes! πŸ™‚ Glad to be of help", "2044965": "Haha, thanks @danielliao! πŸ™‚ Really glad you are finding this useful!", "2045229": "Hi @radek1, thank you so much for the advice on fast debugging with just 1% of the dataset. \n\nI have implemented the DEBUG option for @cdeotte Candidate ReRank CV notebook in order to debug faster, the full dataset notebook runs just under 40 mins, and the DEBUG version is just under 4 mins. This is the best I can do without changing much of the original notebook. Is 4 min fast enough, what do you think?\n\nIs the DEBUG version solely for speeding up to see whether all codes runs without error? Does it do anything else for us?\n\nBelow is the code added or changed (very little) to the original notebook. Do you mind to have a look to see whether I have done it properly? πŸ™ Thanks!\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F850197%2F490c42bcf2869356acc1b9d33c638550%2FScreen%20Shot%202022-11-27%20at%2015.06.37.png?generation=1669533977515751&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F850197%2Fb84e0c710cd9c313b33973021e98ea39%2FScreen%20Shot%202022-11-27%20at%2015.06.50.png?generation=1669533988729535&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F850197%2F36c62b2f43c28941e130889d1560ffdf%2FScreen%20Shot%202022-11-27%20at%2015.06.58.png?generation=1669534000990752&alt=media)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F850197%2F9262c5a744e69262d26f316dfd87787e%2FScreen%20Shot%202022-11-27%20at%2015.07.07.png?generation=1669534011721440&alt=media)", "2045245": "> Is the DEBUG version solely for speeding up to see whether all codes runs without error? \n\nYes, that is the way I do it πŸ™‚\n\nThe faster the code can run, the better πŸ™‚ 4 minutes is okay but in my experience, this would still interfere with the way I work on stuff. What are you going to do in those 4 minutes? I get distracted, go on Twitter, etc.\n\nOf course, life isn't perfect, I still often wait for stuff, but if I can just make sure the notebook runs in a couple of seconds before doing the full run, then I can take my attention and do something else, work on other stuff, without having to wait πŸ™‚ But 4 minutes vs 40 minutes and finding out at the 39th minute you have a typo is already an enormous step in the right direction!", "2045328": "Thank you so much Radek! You are absolutely right, even 4 minutes we can still easily get distracted. It would be great if it can just take a few seconds to run. I will dig a little more to see whether I can make it faster.", "2045409": "Hi Radek, I am happy to report that I have reduced the time to 90s from 230s, adding a few more lines of code.\n\nNow, I managed to make it run within 34s. \n\nWithout your great feedback I would dream to reduce the time from 4 mins to 30s. Thanks @radek1 πŸ™", "2120439": "Great ! I didn't know about the first trick, thanks", "2120583": "danielliao , how were you able to do it?", "2120585": "Who later got to know how to run it in much less time?", "2122254": "My trick of that sort is to always round cv metric. It is important to avoid paying attention to those constantly changing multiple digits and tell youself honestly \"those changes didn't improve the model\"." }, "source": "meta" }