As a beginner, I am eager to learn how pros optimize their parameters. Are there any specific strategies or is it a matter of trial and error? I would greatly appreciate it if you could share any valuable resources or reading materials. 😃
", "messages": [ { "id": "2684245", "postDate": "03/06/2024 14:00:16", "content": "As a beginner, I am eager to learn how pros optimize their parameters. Are there any specific strategies or is it a matter of trial and error? I would greatly appreciate it if you could share any valuable resources or reading materials. 😃
", "rawMarkdown": "As a beginner, I am eager to learn how pros optimize their parameters. Are there any specific strategies or is it a matter of trial and error? I would greatly appreciate it if you could share any valuable resources or reading materials. 😃", "votes": null }, { "id": "2684306", "postDate": "03/06/2024 14:44:37", "content": "I start with a fixed learning rate and try values like 1e-3, 5e-4, 1e-4, etc and then get familiar with the loss landscape of each fold i.e. seeing how tweaking changes losses and convergence. It's basically trial and error but with an intuition that you build over time.
", "rawMarkdown": "I start with a fixed learning rate and try values like 1e-3, 5e-4, 1e-4, etc and then get familiar with the loss landscape of each fold i.e. seeing how tweaking changes losses and convergence. It's basically trial and error but with an intuition that you build over time.", "votes": null }, { "id": "2685093", "postDate": "03/07/2024 01:23:28", "content": "Thank you for you reply！
", "rawMarkdown": "Thank you for you reply！", "votes": null }, { "id": "2685250", "postDate": "03/07/2024 05:28:37", "content": "I think that people who share the training notebook do a lot of trial and errors in their local machine.
\nKaggle notebook is not suitable for parameter tuning because of not enough gpu time.
\nTo run a lot of experiments efficiently, setting hyper parameters only by config file like yaml may be important
There is also an official guide written by google researchers.
\nhttps://github.com/google-research/tuning_playbook
I will read this， thank you！
", "rawMarkdown": "I will read this， thank you！", "votes": null }, { "id": "2685956", "postDate": "03/07/2024 14:40:22", "content": "In addition to to what others say, it is also important to make your experiment pipeline as fast as possible. This includes fast preprocess, fast dataloader, fast training, fast inference, fast metric computation. Additionally it helps if we can experiment successfully with subsets of entire data and/or small models and/or short train schedules.
\nUtilizing multiple GPU and removing all bottle necks is crucial. We want to minimize disk reading, minimize data dtypes, remove unnecessary features and processes, and avoid performing the exact same process procedure more than once. Everything needs to be optimized and accelerated.
\nAfter this is established we then begin the art of experimentation. Experimentation is certainly an art form. We are guided by experience, intuition, and the knowledge of how things work. It is important to reflect after each experiment to understand the models and data better. We constantly redefine plans for future experiments based on previous results and gained understanding.
\nFinally, remember to enjoy the journey. Experimentation is a fun way to learn data science. We learn how models work and behave and we learn about data. Experimentation is also exciting, waking up to see new results, and rewards of discoveries!
", "rawMarkdown": "In addition to to what others say, it is also important to make your experiment pipeline as fast as possible. This includes fast preprocess, fast dataloader, fast training, fast inference, fast metric computation. Additionally it helps if we can experiment successfully with subsets of entire data and/or small models and/or short train schedules.\n\nUtilizing multiple GPU and removing all bottle necks is crucial. We want to minimize disk reading, minimize data dtypes, remove unnecessary features and processes, and avoid performing the exact same process procedure more than once. Everything needs to be optimized and accelerated.\n\nAfter this is established we then begin the art of experimentation. Experimentation is certainly an art form. We are guided by experience, intuition, and the knowledge of how things work. It is important to reflect after each experiment to understand the models and data better. We constantly redefine plans for future experiments based on previous results and gained understanding. \n\nFinally, remember to enjoy the journey. Experimentation is a fun way to learn data science. We learn how models work and behave and we learn about data. Experimentation is also exciting, waking up to see new results, and rewards of discoveries!", "votes": null }, { "id": "2686649", "postDate": "03/08/2024 00:58:34", "content": "I'm very glad to see your reply! I have a question about how to accelerate these processes and remove unnecessary features in the code, how can I implement that? Every time I participate in a competition, I normalize the data and directly input it, but this seems like a significant issue.🤣
", "rawMarkdown": "I'm very glad to see your reply! I have a question about how to accelerate these processes and remove unnecessary features in the code, how can I implement that? Every time I participate in a competition, I normalize the data and directly input it, but this seems like a significant issue.🤣", "votes": null }, { "id": "2686657", "postDate": "03/08/2024 01:20:02", "content": "Here are a three specific tips:
\nRemove Features
\nWhen i say \"remove features\", i'm referring to input data. It is not always the case that using all the provided data is best. For example, train.csv has 100k rows and EEG parquets have 20 columns and Spectrogram parquets have 400 columns. (That's approx 40 million elements!) Maybe we can build a great model using only 10k rows and a subset of columns. If we do this, then we speed up experiments by 10x or more! After we tune our model, we can then add more features and more data (but experimentation is best using small stuff).
Accelerate
\nOne of the most overlooked speed components is preprocess and dataloader. Try running your dataloader without training a model. For example, time how long does the following take:
%%\n batch loader:\n pass\nThis should be lightning fast. If it is not, find the bottlenecks (disk reading, repeated operations, inefficient operations, etc) and remove them. Also utilize CPU/GPU multiprocessing in dataloader. When we train a model on GPU we want to feed large batch sizes the fastest we can because GPUs want lots of data quickly. If we run nvidia-smi we should see 100% GPU utilization. Anything less indicates a dataloader bottleneck and unnecessary slow down.
Training
\nUse mixed precision, multiple GPU, small models (at first), efficiently designed short train schedules (i.e. the largest possible learning rates without overfitting instead of many epochs of low learning rates), and other tricks to accelerate model training like freezing layers, PEFT, loading pretrained models, using diverse subsets of train data (instead of all train data)
This is just information from another world. You are truly a genius. Thank you for sharing. You are an inspiration.
", "rawMarkdown": "This is just information from another world. You are truly a genius. Thank you for sharing. You are an inspiration.", "votes": null }, { "id": "2689889", "postDate": "03/10/2024 07:24:23", "content": "Thank you for your instructions！
", "rawMarkdown": "Thank you for your instructions！", "votes": null }, { "id": "2704359", "postDate": "03/18/2024 18:04:57", "content": "Thank for sharing Chris!
", "rawMarkdown": "Thank for sharing Chris!", "votes": null } ], "comments": [ { "id": 2684306, "author_name": "gunesevitan", "author_url": "", "post_date": "03/06/2024 14:44:37", "content": "I start with a fixed learning rate and try values like 1e-3, 5e-4, 1e-4, etc and then get familiar with the loss landscape of each fold i.e. seeing how tweaking changes losses and convergence. It's basically trial and error but with an intuition that you build over time.
", "votes": null, "replies": [ { "id": 2685093, "author_name": "seeingtimes", "author_url": "", "post_date": "03/07/2024 01:23:28", "content": "Thank you for you reply！
", "votes": null, "replies": [] } ] }, { "id": 2685250, "author_name": "clearwaterkzk", "author_url": "", "post_date": "03/07/2024 05:28:37", "content": "I think that people who share the training notebook do a lot of trial and errors in their local machine.
\nKaggle notebook is not suitable for parameter tuning because of not enough gpu time.
\nTo run a lot of experiments efficiently, setting hyper parameters only by config file like yaml may be important
There is also an official guide written by google researchers.
\nhttps://github.com/google-research/tuning_playbook
I will read this， thank you！
", "votes": null, "replies": [] } ] }, { "id": 2685956, "author_name": "cdeotte", "author_url": "", "post_date": "03/07/2024 14:40:22", "content": "In addition to to what others say, it is also important to make your experiment pipeline as fast as possible. This includes fast preprocess, fast dataloader, fast training, fast inference, fast metric computation. Additionally it helps if we can experiment successfully with subsets of entire data and/or small models and/or short train schedules.
\nUtilizing multiple GPU and removing all bottle necks is crucial. We want to minimize disk reading, minimize data dtypes, remove unnecessary features and processes, and avoid performing the exact same process procedure more than once. Everything needs to be optimized and accelerated.
\nAfter this is established we then begin the art of experimentation. Experimentation is certainly an art form. We are guided by experience, intuition, and the knowledge of how things work. It is important to reflect after each experiment to understand the models and data better. We constantly redefine plans for future experiments based on previous results and gained understanding.
\nFinally, remember to enjoy the journey. Experimentation is a fun way to learn data science. We learn how models work and behave and we learn about data. Experimentation is also exciting, waking up to see new results, and rewards of discoveries!
", "votes": null, "replies": [ { "id": 2686649, "author_name": "seeingtimes", "author_url": "", "post_date": "03/08/2024 00:58:34", "content": "I'm very glad to see your reply! I have a question about how to accelerate these processes and remove unnecessary features in the code, how can I implement that? Every time I participate in a competition, I normalize the data and directly input it, but this seems like a significant issue.🤣
", "votes": null, "replies": [ { "id": 2686657, "author_name": "cdeotte", "author_url": "", "post_date": "03/08/2024 01:20:02", "content": "Here are a three specific tips:
\nRemove Features
\nWhen i say \"remove features\", i'm referring to input data. It is not always the case that using all the provided data is best. For example, train.csv has 100k rows and EEG parquets have 20 columns and Spectrogram parquets have 400 columns. (That's approx 40 million elements!) Maybe we can build a great model using only 10k rows and a subset of columns. If we do this, then we speed up experiments by 10x or more! After we tune our model, we can then add more features and more data (but experimentation is best using small stuff).
Accelerate
\nOne of the most overlooked speed components is preprocess and dataloader. Try running your dataloader without training a model. For example, time how long does the following take:
%%\n batch loader:\n pass\nThis should be lightning fast. If it is not, find the bottlenecks (disk reading, repeated operations, inefficient operations, etc) and remove them. Also utilize CPU/GPU multiprocessing in dataloader. When we train a model on GPU we want to feed large batch sizes the fastest we can because GPUs want lots of data quickly. If we run nvidia-smi we should see 100% GPU utilization. Anything less indicates a dataloader bottleneck and unnecessary slow down.
Training
\nUse mixed precision, multiple GPU, small models (at first), efficiently designed short train schedules (i.e. the largest possible learning rates without overfitting instead of many epochs of low learning rates), and other tricks to accelerate model training like freezing layers, PEFT, loading pretrained models, using diverse subsets of train data (instead of all train data)
This is just information from another world. You are truly a genius. Thank you for sharing. You are an inspiration.
", "votes": null, "replies": [] }, { "id": 2689889, "author_name": "seeingtimes", "author_url": "", "post_date": "03/10/2024 07:24:23", "content": "Thank you for your instructions！
", "votes": null, "replies": [] }, { "id": 2704359, "author_name": "arindamroy23", "author_url": "", "post_date": "03/18/2024 18:04:57", "content": "Thank for sharing Chris!
", "votes": null, "replies": [] } ] } ] } ] } ], "raw_markdown_by_id": { "2684245": "As a beginner, I am eager to learn how pros optimize their parameters. Are there any specific strategies or is it a matter of trial and error? I would greatly appreciate it if you could share any valuable resources or reading materials. 😃", "2684306": "I start with a fixed learning rate and try values like 1e-3, 5e-4, 1e-4, etc and then get familiar with the loss landscape of each fold i.e. seeing how tweaking changes losses and convergence. It's basically trial and error but with an intuition that you build over time.", "2685093": "Thank you for you reply！", "2685250": "I think that people who share the training notebook do a lot of trial and errors in their local machine.\nKaggle notebook is not suitable for parameter tuning because of not enough gpu time.\nTo run a lot of experiments efficiently, setting hyper parameters only by config file like yaml may be important\n\nThere is also an official guide written by google researchers.\n[https://github.com/google-research/tuning_playbook](https://github.com/google-research/tuning_playbook)", "2685636": "I will read this， thank you！", "2685956": "In addition to to what others say, it is also important to make your experiment pipeline as fast as possible. This includes fast preprocess, fast dataloader, fast training, fast inference, fast metric computation. Additionally it helps if we can experiment successfully with subsets of entire data and/or small models and/or short train schedules.\n\nUtilizing multiple GPU and removing all bottle necks is crucial. We want to minimize disk reading, minimize data dtypes, remove unnecessary features and processes, and avoid performing the exact same process procedure more than once. Everything needs to be optimized and accelerated.\n\nAfter this is established we then begin the art of experimentation. Experimentation is certainly an art form. We are guided by experience, intuition, and the knowledge of how things work. It is important to reflect after each experiment to understand the models and data better. We constantly redefine plans for future experiments based on previous results and gained understanding. \n\nFinally, remember to enjoy the journey. Experimentation is a fun way to learn data science. We learn how models work and behave and we learn about data. Experimentation is also exciting, waking up to see new results, and rewards of discoveries!", "2686649": "I'm very glad to see your reply! I have a question about how to accelerate these processes and remove unnecessary features in the code, how can I implement that? Every time I participate in a competition, I normalize the data and directly input it, but this seems like a significant issue.🤣", "2686657": "Here are a three specific tips:\n**Remove Features**\nWhen i say \"remove features\", i'm referring to input data. It is not always the case that using all the provided data is best. For example, `train.csv` has 100k rows and EEG parquets have 20 columns and Spectrogram parquets have 400 columns. (That's approx 40 million elements!) Maybe we can build a great model using only 10k rows and a subset of columns. If we do this, then we speed up experiments by 10x or more! After we tune our model, we can then add more features and more data (but experimentation is best using small stuff).\n\n**Accelerate**\nOne of the most overlooked speed components is preprocess and dataloader. Try running your dataloader without training a model. For example, time how long does the following take:\n\n %%time\n for batch in loader:\n pass\n\nThis should be lightning fast. If it is not, find the bottlenecks (disk reading, repeated operations, inefficient operations, etc) and remove them. Also utilize CPU/GPU multiprocessing in dataloader. When we train a model on GPU we want to feed large batch sizes the fastest we can because GPUs want lots of data quickly. If we run `nvidia-smi` we should see `100%` GPU utilization. Anything less indicates a dataloader bottleneck and unnecessary slow down.\n\n**Training**\nUse mixed precision, multiple GPU, small models (at first), efficiently designed short train schedules (i.e. the largest possible learning rates without overfitting instead of many epochs of low learning rates), and other tricks to accelerate model training like freezing layers, PEFT, loading pretrained models, using diverse subsets of train data (instead of all train data)", "2689791": "This is just information from another world. You are truly a genius. Thank you for sharing. You are an inspiration.", "2689889": "Thank you for your instructions！", "2704359": "Thank for sharing Chris!" }, "source": "meta" }