Now,I use the init lr is 1e-4,and I use cosin lr_scheduler with warmup.But I only in lb 0.898,and can't to improve the score!I also try other method: like label_smooth, bi_templete_loss and so on.But this can't give me some improve.I think the reason is I am not choose a good lr. Anyone can help me and give me some advice?Thanks!
", "messages": [ { "id": "1133900", "postDate": "12/31/2020 16:13:56", "content": "Now,I use the init lr is 1e-4,and I use cosin lr_scheduler with warmup.But I only in lb 0.898,and can't to improve the score!I also try other method: like label_smooth, bi_templete_loss and so on.But this can't give me some improve.I think the reason is I am not choose a good lr. Anyone can help me and give me some advice?Thanks!
", "rawMarkdown": "Now,I use the init lr is 1e-4,and I use cosin lr_scheduler with warmup.But I only in lb 0.898,and can't to improve the score!I also try other method: like label_smooth, bi_templete_loss and so on.But this can't give me some improve.I think the reason is I am not choose a good lr. Anyone can help me and give me some advice?Thanks!", "votes": null }, { "id": "1134182", "postDate": "01/01/2021 00:30:22", "content": "Did you already try tools like fastai's lr_find (or equivalent approaches for generic PyTorch, keras, PyTorch Lightning or whatever framework you use)? Schedules are of course harder to decide upon and cross-validating some candidates (like flat cosine, one-cycle etc.) is probably the only thing you can do.
", "rawMarkdown": "Did you already try tools like fastai's [lr_find](https://docs.fast.ai/callback.schedule.html#Learner.lr_find) (or equivalent approaches for generic [PyTorch](https://github.com/davidtvs/pytorch-lr-finder), keras, PyTorch Lightning or whatever framework you use)? Schedules are of course harder to decide upon and cross-validating some candidates (like flat cosine, one-cycle etc.) is probably the only thing you can do.", "votes": null }, { "id": "1134224", "postDate": "01/01/2021 02:47:14", "content": "Hi,I use pytorch.I am not use lr_finder.what do you mean ' cross-validating some candidates (like flat cosine, one-cycle etc.)'?You mean is I should use the difference scheduler in difference fold?Thanks!
", "rawMarkdown": "Hi,I use pytorch.I am not use lr_finder.what do you mean ' cross-validating some candidates (like flat cosine, one-cycle etc.)'?You mean is I should use the difference scheduler in difference fold?Thanks!", "votes": null }, { "id": "1134239", "postDate": "01/01/2021 03:44:12", "content": "By the way,how can I find lr use pyorch?Thanks!
", "rawMarkdown": "By the way,how can I find lr use pyorch?Thanks!", "votes": null }, { "id": "1134375", "postDate": "01/01/2021 08:08:52", "content": "The repo I linked has explanations and usage examples.
\nRegarding CV: I meant to look at which approach gives the best competition metric when you assess it by cross validation (=e.g. with 5-fold, for each approach fit to each of the 5 training folds, calculate the performance on the validation fold, then compare performance between approaches).
", "rawMarkdown": "The [repo I linked](https://github.com/davidtvs/pytorch-lr-finder) has explanations and usage examples.\n\nRegarding CV: I meant to look at which approach gives the best competition metric when you assess it by cross validation (=e.g. with 5-fold, for each approach fit to each of the 5 training folds, calculate the performance on the validation fold, then compare performance between approaches).", "votes": null }, { "id": "1135356", "postDate": "01/02/2021 07:22:54", "content": "Wouldn't that be computationally expensive for someone using kaggle kernels?
", "rawMarkdown": "Wouldn't that be computationally expensive for someone using kaggle kernels?", "votes": null }, { "id": "1135416", "postDate": "01/02/2021 08:29:33", "content": "It's not actually that bad, it usually takes less than an epoch takes and you just do it once.
", "rawMarkdown": "It's not actually that bad, it usually takes less than an epoch takes and you just do it once.", "votes": null }, { "id": "1140351", "postDate": "01/06/2021 00:24:00", "content": "Hi, @bcwang , you can also try Pytorch lightning. It's a wrapper on top of Pytorch and has some nice built-in functions like lr_find, support for TPUs, hyperparameters search functions etc
", "rawMarkdown": "Hi, @bcwang , you can also try [Pytorch lightning](https://github.com/PyTorchLightning/pytorch-lightning). It's a wrapper on top of Pytorch and has some nice built-in functions like lr_find, support for TPUs, hyperparameters search functions etc", "votes": null }, { "id": "1140352", "postDate": "01/06/2021 00:24:00", "content": "Hi, @bcwang , you can also try Pytorch lightning. It's a wrapper on top of Pytorch and has some nice built-in functions like lr_find, support for TPUs, hyperparameters search functions etc
", "rawMarkdown": "Hi, @bcwang , you can also try [Pytorch lightning](https://github.com/PyTorchLightning/pytorch-lightning). It's a wrapper on top of Pytorch and has some nice built-in functions like lr_find, support for TPUs, hyperparameters search functions etc", "votes": null }, { "id": "1140662", "postDate": "01/06/2021 07:12:24", "content": "How much score you can get using single fold, no TTA ?
", "rawMarkdown": "How much score you can get using single fold, no TTA ?", "votes": null } ], "comments": [ { "id": 1134182, "author_name": "bjoernholzhauer", "author_url": "", "post_date": "01/01/2021 00:30:22", "content": "Did you already try tools like fastai's lr_find (or equivalent approaches for generic PyTorch, keras, PyTorch Lightning or whatever framework you use)? Schedules are of course harder to decide upon and cross-validating some candidates (like flat cosine, one-cycle etc.) is probably the only thing you can do.
", "votes": null, "replies": [ { "id": 1134224, "author_name": "bcwang", "author_url": "", "post_date": "01/01/2021 02:47:14", "content": "Hi,I use pytorch.I am not use lr_finder.what do you mean ' cross-validating some candidates (like flat cosine, one-cycle etc.)'?You mean is I should use the difference scheduler in difference fold?Thanks!
", "votes": null, "replies": [] }, { "id": 1134239, "author_name": "bcwang", "author_url": "", "post_date": "01/01/2021 03:44:12", "content": "By the way,how can I find lr use pyorch?Thanks!
", "votes": null, "replies": [ { "id": 1140351, "author_name": "suryajrrafl", "author_url": "", "post_date": "01/06/2021 00:24:00", "content": "Hi, @bcwang , you can also try Pytorch lightning. It's a wrapper on top of Pytorch and has some nice built-in functions like lr_find, support for TPUs, hyperparameters search functions etc
", "votes": null, "replies": [] }, { "id": 1140352, "author_name": "suryajrrafl", "author_url": "", "post_date": "01/06/2021 00:24:00", "content": "Hi, @bcwang , you can also try Pytorch lightning. It's a wrapper on top of Pytorch and has some nice built-in functions like lr_find, support for TPUs, hyperparameters search functions etc
", "votes": null, "replies": [] } ] }, { "id": 1134375, "author_name": "bjoernholzhauer", "author_url": "", "post_date": "01/01/2021 08:08:52", "content": "The repo I linked has explanations and usage examples.
\nRegarding CV: I meant to look at which approach gives the best competition metric when you assess it by cross validation (=e.g. with 5-fold, for each approach fit to each of the 5 training folds, calculate the performance on the validation fold, then compare performance between approaches).
", "votes": null, "replies": [] }, { "id": 1135356, "author_name": "zowlex", "author_url": "", "post_date": "01/02/2021 07:22:54", "content": "Wouldn't that be computationally expensive for someone using kaggle kernels?
", "votes": null, "replies": [] }, { "id": 1135416, "author_name": "bjoernholzhauer", "author_url": "", "post_date": "01/02/2021 08:29:33", "content": "It's not actually that bad, it usually takes less than an epoch takes and you just do it once.
", "votes": null, "replies": [] } ] }, { "id": 1140662, "author_name": "clwwlc", "author_url": "", "post_date": "01/06/2021 07:12:24", "content": "How much score you can get using single fold, no TTA ?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1133900": "Now,I use the init lr is 1e-4,and I use cosin lr_scheduler with warmup.But I only in lb 0.898,and can't to improve the score!I also try other method: like label_smooth, bi_templete_loss and so on.But this can't give me some improve.I think the reason is I am not choose a good lr. Anyone can help me and give me some advice?Thanks!", "1134182": "Did you already try tools like fastai's [lr_find](https://docs.fast.ai/callback.schedule.html#Learner.lr_find) (or equivalent approaches for generic [PyTorch](https://github.com/davidtvs/pytorch-lr-finder), keras, PyTorch Lightning or whatever framework you use)? Schedules are of course harder to decide upon and cross-validating some candidates (like flat cosine, one-cycle etc.) is probably the only thing you can do.", "1134224": "Hi,I use pytorch.I am not use lr_finder.what do you mean ' cross-validating some candidates (like flat cosine, one-cycle etc.)'?You mean is I should use the difference scheduler in difference fold?Thanks!", "1134239": "By the way,how can I find lr use pyorch?Thanks!", "1134375": "The [repo I linked](https://github.com/davidtvs/pytorch-lr-finder) has explanations and usage examples.\n\nRegarding CV: I meant to look at which approach gives the best competition metric when you assess it by cross validation (=e.g. with 5-fold, for each approach fit to each of the 5 training folds, calculate the performance on the validation fold, then compare performance between approaches).", "1135356": "Wouldn't that be computationally expensive for someone using kaggle kernels?", "1135416": "It's not actually that bad, it usually takes less than an epoch takes and you just do it once.", "1140351": "Hi, @bcwang , you can also try [Pytorch lightning](https://github.com/PyTorchLightning/pytorch-lightning). It's a wrapper on top of Pytorch and has some nice built-in functions like lr_find, support for TPUs, hyperparameters search functions etc", "1140352": "Hi, @bcwang , you can also try [Pytorch lightning](https://github.com/PyTorchLightning/pytorch-lightning). It's a wrapper on top of Pytorch and has some nice built-in functions like lr_find, support for TPUs, hyperparameters search functions etc", "1140662": "How much score you can get using single fold, no TTA ?" }, "source": "meta" }