![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F337744%2Fe51b120191c980e6e21f5c40be7bf955%2FScreen%20Shot%202020-12-03%20at%206.25.04%20PM.png?generation=1607037924271686&alt=media\")
This is my first major Kaggle competition so I wanted to know how do most of the fellow Kagglers keep track of their experiments since it is a very long competition. Some tips will be really helpful!
", "messages": [ { "id": 1100085, "postDate": "2020-12-02T20:23:48.897Z", "content": "This is my first major Kaggle competition so I wanted to know how do most of the fellow Kagglers keep track of their experiments since it is a very long competition. Some tips will be really helpful!
", "rawMarkdown": "This is my first major Kaggle competition so I wanted to know how do most of the fellow Kagglers keep track of their experiments since it is a very long competition. Some tips will be really helpful!", "votes": 24 }, { "id": 1101464, "postDate": "2020-12-03T23:26:24.100Z", "content": "Spread sheet! This one is from the MoA competition. I keep record of the notebooks, CV and LB scores, and all hyperparameters. I start with no hyperparameters in the spreadsheet and start adding them as I change them.
\n
Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!", "votes": 1 } ] }, { "id": 1210655, "postDate": "2021-02-19T15:33:02.190Z", "content": "During our journey in this competition we realized that structuring and organizing experiments is extremely useful to optimize hypothesis testing, make sure the same things are not run twice and better understand where to go next. We started with a good-old Excel sheet but keeping it up to date was becoming more and more difficult.
\nIn the last few weeks, I decided to try Neptune and was pleased with its features. The free version has 100Gb storage, which is enough to store model weights and configurations for quite a lot of experiments. It makes so much easier to aggregate results from different environments (Kaggle, Colab) in one place and analyze the performance. Here is how the table with some of my models looks like:
\n![\"neptune\"](\"https://i.postimg.cc/5NzVqDqv/rsz-neptune.png\")
This is great!
\nThanks for sharing
\nAlso congratulations for the gold
That is so awesome, thank you for giving Neptune a go!
", "rawMarkdown": "That is so awesome, thank you for giving Neptune a go!" } ] }, { "id": 1101898, "postDate": "2020-12-04T11:25:54.697Z", "content": "I'm using MLFlow. I have set up an instance on a Cloud env and integration in the Notebook is simply inserting some python lines of code. If you use PyTorch Lightning there is also automated logging during the training
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1526569%2Fc00d67d722a9843bde415040f42cd101%2FScreenshot%202020-12-04%20at%2012.23.49.png?generation=1607081135790873&alt=media\")
I am using neptune.ml.
\nHere is a notebook that shows how to set it up and use it's features: https://www.kaggle.com/nroman/tracking-experiments-with-neptune
How do you manage kfold validation with neptune, do a new experiment for each fold?
", "rawMarkdown": "How do you manage kfold validation with neptune, do a new experiment for each fold?", "votes": 1 }, { "id": 1102859, "postDate": "2020-12-05T12:03:34.430Z", "content": "Yes, new experiment for each fold.
", "rawMarkdown": "Yes, new experiment for each fold.", "votes": 1 }, { "id": 1105478, "postDate": "2020-12-07T23:09:56.753Z", "content": "I also use neptune.ai and I love it. And yes one experiment per fold.
", "rawMarkdown": "I also use neptune.ai and I love it. And yes one experiment per fold." } ] }, { "id": 1102181, "postDate": "2020-12-04T17:00:07.850Z", "content": "Real men do it in Vim π
", "rawMarkdown": "Real men do it in Vim π", "votes": 3, "replies": [ { "id": 1102203, "postDate": "2020-12-04T17:23:05.157Z", "content": "Can't lose your records if you can't quit!
", "rawMarkdown": "Can't lose your records if you can't quit!", "votes": 5 }, { "id": 1102226, "postDate": "2020-12-04T17:55:16.080Z", "content": "This is why I have 2 monitors. One is always stuck in Vim :D
", "rawMarkdown": "This is why I have 2 monitors. One is always stuck in Vim :D ", "votes": 1 } ] }, { "id": 1100321, "postDate": "2020-12-03T02:01:57.393Z", "content": "I am using wandb. I think mlflow is also major one.
", "rawMarkdown": "I am using [wandb](https://www.wandb.com/). I think [mlflow](https://mlflow.org/) is also major one.", "votes": 3 }, { "id": 1106425, "postDate": "2020-12-08T20:31:25.117Z", "content": "Title: Haven-AI: Build Large-Scale Machine Learning Projects and Manage Thousands of Experiments
\nhttps://www.youtube.com/watch?v=Ngc7da0DUws
always generate logfile (parameters and loss curve should be saved). make sure the code is kept.
if you need to change your code, you should start a new file. you should always make sure that you can roll back to the previous version of your code to repeat your previous experiment results.
\n(you can do version control or just good file organization)
when you have the raw results(logfiles), you can use excel, etc to do summary, trend.
Thank you! Will surely follow this
", "rawMarkdown": "Thank you! Will surely follow this" } ] }, { "id": 1100791, "postDate": "2020-12-03T10:48:00.493Z", "content": "I use neptune.ai to keep track of all parameters and logs of each experiment.
", "rawMarkdown": "I use [neptune.ai](https://neptune.ai/) to keep track of all parameters and logs of each experiment.", "votes": 1 }, { "id": 1100789, "postDate": "2020-12-03T10:44:33.780Z", "content": "I use Google Sheet to keep track of all experiments with notebook versions, scores and high level approach like this:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1905996%2Fbb6c4325f6c1a4f52ea8f33b6908eba1%2FScreenshot%202020-12-03%20at%204.13.02%20PM.png?generation=1606992201354611&alt=media\")
I use wandb but that doesn't keep track of version wise training/inference notebooks.
Thank You! This is really helpful
", "rawMarkdown": "Thank You! This is really helpful" }, { "id": 1100939, "postDate": "2020-12-03T13:41:56.037Z", "content": "\n\nI use wandb but that doesn't keep track of version wise training/inference notebooks.
\n
For more advanced use see tutorials (in wandb website) on how you can use the artifacts to keep track of your pipeline, datasets, model versions etc
PS: In any case what you need to keep track is your training runs - inference is just the result of that
\nPS2: google-sheets are also great, I still use them
nice question! wandb and neptune-ai offering a lot of such possibilities + hyperparam optimization.
\nPS: I have a public nb that shows step-by-step how to start with wandb (it is applied on different dataset but you get the idea) - also there are many tutorials online - if you need any help let me know
Thanks! It will be really helpful if you could share your notebook
", "rawMarkdown": "Thanks! It will be really helpful if you could share your notebook" }, { "id": 1100936, "postDate": "2020-12-03T13:37:21.187Z", "content": "sorry I didn't want to be considered as spam/self promotion or wth - have a look a starting point here and hp optimisation here
\nthere are many helpful tutorials online too
Thank you! This will help a lot
", "rawMarkdown": "Thank you! This will help a lot", "votes": 1 } ] }, { "id": 1100747, "postDate": "2020-12-03T09:54:33.673Z", "content": "There are plenty of options:
\nAll of these are commonly used in the community, so you can find a lot of tutorials (including official) that help you set everything up very quickly for both metrics logging or experiment tracking (hyperparameters etc.)
", "rawMarkdown": "All of these are commonly used in the community, so you can find a lot of tutorials (including official) that help you set everything up very quickly for both metrics logging or experiment tracking (hyperparameters etc.)", "votes": 1 }, { "id": 1100915, "postDate": "2020-12-03T13:14:11.433Z", "content": "Thank you! Very helpful
", "rawMarkdown": "Thank you! Very helpful" } ] }, { "id": 1100344, "postDate": "2020-12-03T03:00:23.053Z", "content": "google sheets )))
", "rawMarkdown": "google sheets )))", "votes": 2 }, { "id": 1210703, "postDate": "2021-02-19T16:08:15.547Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1101464, "author_name": "Tolga", "author_url": "", "post_date": "2020-12-03T23:26:24.100000", "content": "Spread sheet! This one is from the MoA competition. I keep record of the notebooks, CV and LB scores, and all hyperparameters. I start with no hyperparameters in the spreadsheet and start adding them as I change them.
\n
Thanks for sharing!
", "votes": 1, "replies": [] } ] }, { "id": 1210655, "author_name": "Nikita Kozodoi", "author_url": "", "post_date": "2021-02-19T15:33:02.190000", "content": "During our journey in this competition we realized that structuring and organizing experiments is extremely useful to optimize hypothesis testing, make sure the same things are not run twice and better understand where to go next. We started with a good-old Excel sheet but keeping it up to date was becoming more and more difficult.
\nIn the last few weeks, I decided to try Neptune and was pleased with its features. The free version has 100Gb storage, which is enough to store model weights and configurations for quite a lot of experiments. It makes so much easier to aggregate results from different environments (Kaggle, Colab) in one place and analyze the performance. Here is how the table with some of my models looks like:
\n
This is great!
\nThanks for sharing
\nAlso congratulations for the gold
That is so awesome, thank you for giving Neptune a go!
", "votes": 0, "replies": [] } ] }, { "id": 1101898, "author_name": "Luigi Saetta", "author_url": "", "post_date": "2020-12-04T11:25:54.697000", "content": "I'm using MLFlow. I have set up an instance on a Cloud env and integration in the Notebook is simply inserting some python lines of code. If you use PyTorch Lightning there is also automated logging during the training
\n
I am using neptune.ml.
\nHere is a notebook that shows how to set it up and use it's features: https://www.kaggle.com/nroman/tracking-experiments-with-neptune
How do you manage kfold validation with neptune, do a new experiment for each fold?
", "votes": 1, "replies": [] }, { "id": 1102859, "author_name": "Roman", "author_url": "", "post_date": "2020-12-05T12:03:34.430000", "content": "Yes, new experiment for each fold.
", "votes": 1, "replies": [] }, { "id": 1105478, "author_name": "Peter Cnudde", "author_url": "", "post_date": "2020-12-07T23:09:56.753000", "content": "I also use neptune.ai and I love it. And yes one experiment per fold.
", "votes": 0, "replies": [] } ] }, { "id": 1102181, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-12-04T17:00:07.850000", "content": "Real men do it in Vim π
", "votes": 3, "replies": [ { "id": 1102203, "author_name": "Sohier Dane", "author_url": "", "post_date": "2020-12-04T17:23:05.157000", "content": "Can't lose your records if you can't quit!
", "votes": 5, "replies": [] }, { "id": 1102226, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-12-04T17:55:16.080000", "content": "This is why I have 2 monitors. One is always stuck in Vim :D
", "votes": 1, "replies": [] } ] }, { "id": 1100321, "author_name": "takuoko", "author_url": "", "post_date": "2020-12-03T02:01:57.393000", "content": "I am using wandb. I think mlflow is also major one.
", "votes": 3, "replies": [] }, { "id": 1106425, "author_name": "hengck23", "author_url": "", "post_date": "2020-12-08T20:31:25.117000", "content": "Title: Haven-AI: Build Large-Scale Machine Learning Projects and Manage Thousands of Experiments
\nhttps://www.youtube.com/watch?v=Ngc7da0DUws
always generate logfile (parameters and loss curve should be saved). make sure the code is kept.
if you need to change your code, you should start a new file. you should always make sure that you can roll back to the previous version of your code to repeat your previous experiment results.
\n(you can do version control or just good file organization)
when you have the raw results(logfiles), you can use excel, etc to do summary, trend.
Thank you! Will surely follow this
", "votes": 0, "replies": [] } ] }, { "id": 1100791, "author_name": "Yovin Yahathugoda", "author_url": "", "post_date": "2020-12-03T10:48:00.493000", "content": "I use neptune.ai to keep track of all parameters and logs of each experiment.
", "votes": 1, "replies": [] }, { "id": 1100789, "author_name": "Kaushal Shah", "author_url": "", "post_date": "2020-12-03T10:44:33.780000", "content": "I use Google Sheet to keep track of all experiments with notebook versions, scores and high level approach like this:
\n
I use wandb but that doesn't keep track of version wise training/inference notebooks.
Thank You! This is really helpful
", "votes": 0, "replies": [] }, { "id": 1100939, "author_name": "Ioannis M", "author_url": "", "post_date": "2020-12-03T13:41:56.037000", "content": "\n\nI use wandb but that doesn't keep track of version wise training/inference notebooks.
\n
For more advanced use see tutorials (in wandb website) on how you can use the artifacts to keep track of your pipeline, datasets, model versions etc
PS: In any case what you need to keep track is your training runs - inference is just the result of that
\nPS2: google-sheets are also great, I still use them
nice question! wandb and neptune-ai offering a lot of such possibilities + hyperparam optimization.
\nPS: I have a public nb that shows step-by-step how to start with wandb (it is applied on different dataset but you get the idea) - also there are many tutorials online - if you need any help let me know
Thanks! It will be really helpful if you could share your notebook
", "votes": 0, "replies": [] }, { "id": 1100936, "author_name": "Ioannis M", "author_url": "", "post_date": "2020-12-03T13:37:21.187000", "content": "sorry I didn't want to be considered as spam/self promotion or wth - have a look a starting point here and hp optimisation here
\nthere are many helpful tutorials online too
Thank you! This will help a lot
", "votes": 1, "replies": [] } ] }, { "id": 1100747, "author_name": "mtszkw", "author_url": "", "post_date": "2020-12-03T09:54:33.673000", "content": "There are plenty of options:
\nAll of these are commonly used in the community, so you can find a lot of tutorials (including official) that help you set everything up very quickly for both metrics logging or experiment tracking (hyperparameters etc.)
", "votes": 1, "replies": [] }, { "id": 1100915, "author_name": "Debarshi Chanda", "author_url": "", "post_date": "2020-12-03T13:14:11.433000", "content": "Thank you! Very helpful
", "votes": 0, "replies": [] } ] }, { "id": 1100344, "author_name": "Kupchanski", "author_url": "", "post_date": "2020-12-03T03:00:23.053000", "content": "google sheets )))
", "votes": 2, "replies": [] }, { "id": 1210703, "author_name": "", "author_url": "", "post_date": "2021-02-19T16:08:15.547000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1100085": "This is my first major Kaggle competition so I wanted to know how do most of the fellow Kagglers keep track of their experiments since it is a very long competition. Some tips will be really helpful!", "1101464": "Spread sheet! This one is from the MoA competition. I keep record of the notebooks, CV and LB scores, and all hyperparameters. I start with no hyperparameters in the spreadsheet and start adding them as I change them.\n\n", "1210655": "During our journey in this competition we realized that structuring and organizing experiments is extremely useful to optimize hypothesis testing, make sure the same things are not run twice and better understand where to go next. We started with a good-old Excel sheet but keeping it up to date was becoming more and more difficult.\n\nIn the last few weeks, I decided to try [Neptune](https://neptune.ai) and was pleased with its features. The free version has 100Gb storage, which is enough to store model weights and configurations for quite a lot of experiments. It makes so much easier to aggregate results from different environments (Kaggle, Colab) in one place and analyze the performance. Here is how the table with some of my models looks like:\n", "1101898": "I'm using MLFlow. I have set up an instance on a Cloud env and integration in the Notebook is simply inserting some python lines of code. If you use PyTorch Lightning there is also automated logging during the training\n\n", "1102651": "I am using neptune.ml.\nHere is a notebook that shows how to set it up and use it's features: https://www.kaggle.com/nroman/tracking-experiments-with-neptune", "1102181": "Real men do it in Vim π", "1100321": "I am using [wandb](https://www.wandb.com/). I think [mlflow](https://mlflow.org/) is also major one.", "1106425": "Title: Haven-AI: Build Large-Scale Machine Learning Projects and Manage Thousands of Experiments\nhttps://www.youtube.com/watch?v=Ngc7da0DUws", "1105539": "- always generate logfile (parameters and loss curve should be saved). make sure the code is kept.\n- if you need to change your code, you should start a new file. you should always make sure that you can roll back to the previous version of your code to repeat your previous experiment results.\n(you can do version control or just good file organization)\n\n- when you have the raw results(logfiles), you can use excel, etc to do summary, trend.", "1100791": "I use [neptune.ai](https://neptune.ai/) to keep track of all parameters and logs of each experiment.", "1100789": "I use Google Sheet to keep track of all experiments with notebook versions, scores and high level approach like this:\n\n\n\nI use `wandb` but that doesn't keep track of version wise training/inference notebooks.", "1100604": "nice question! `wandb` and `neptune-ai` offering a lot of such possibilities + hyperparam optimization. \nPS: I have a public nb that shows step-by-step how to start with wandb (it is applied on different dataset but you get the idea) - also there are many tutorials online - if you need any help let me know ", "1100747": "There are plenty of options:\n- Tensorboard which can be ran locally (on your computer) or even in Kaggle notebook\n- Alternatively if you need nice UI and more functionalities: Weights&Biases, Neptune.ai ([described here](https://towardsdatascience.com/track-and-organize-your-ml-projects-e44e6c7c3f9d)), MLFlow", "1100344": "google sheets )))", "1210703": "" } }