![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1517144%2F452881d2732d16858181ec249a81d19d%2FCapture.PNG?generation=1587716308839677&alt=media\")
I wanted to share a notebook Comparing Two Different Schedulers which compares two different learning rate schedulers. The starter notebook scheduler (I call it TPU scheduler) and One cycle scheduler implemented in the fastai module. Any feedback would be greatly appreciated.
\n\n
UPDATE: There was mistake in the notebook for the second experiment which was fixed and updated.
", "messages": [ { "id": "818938", "postDate": "04/24/2020 08:20:11", "content": "I wanted to share a notebook Comparing Two Different Schedulers which compares two different learning rate schedulers. The starter notebook scheduler (I call it TPU scheduler) and One cycle scheduler implemented in the fastai module. Any feedback would be greatly appreciated.
\n\n
UPDATE: There was mistake in the notebook for the second experiment which was fixed and updated.
", "rawMarkdown": "I wanted to share a notebook [Comparing Two Different Schedulers](https://www.kaggle.com/ibrahimsherify/comparing-two-different-schedulers?scriptVersionId=32625540) which compares two different learning rate schedulers. The starter [notebook](https://www.kaggle.com/mgornergoogle/five-flowers-with-keras-and-xception-on-tpu) scheduler (I call it TPU scheduler) and One cycle scheduler implemented in the fastai [module](https://docs.fast.ai/callbacks.one_cycle.html#Training-with-the-1cycle-policy). Any feedback would be greatly appreciated.\n\n\n\nUPDATE: There was mistake in the notebook for the second experiment which was fixed and updated.", "votes": null }, { "id": "818967", "postDate": "04/24/2020 08:40:59", "content": "Thanks man :)
", "rawMarkdown": "Thanks man :)", "votes": null }, { "id": "819025", "postDate": "04/24/2020 09:43:12", "content": "welcome my friend :D
", "rawMarkdown": "welcome my friend :D", "votes": null }, { "id": "819073", "postDate": "04/24/2020 10:38:51", "content": "Great
", "rawMarkdown": "Great", "votes": null }, { "id": "819169", "postDate": "04/24/2020 11:59:09", "content": "Looks interesting
", "rawMarkdown": "Looks interesting", "votes": null }, { "id": "819657", "postDate": "04/24/2020 18:51:14", "content": "The two schedulers don't look that different to me and you get to the same f1 score in the end.
\n\nLooking at your loss curves however, you have a big hit on the validation loss around epochs 3-4-5. This is usually a sign that you are breaking the pre-trained weights. They recover afterwards but it is still probably not optimal.
", "rawMarkdown": "The two schedulers don't look that different to me and you get to the same f1 score in the end.\n\nLooking at your loss curves however, you have a big hit on the validation loss around epochs 3-4-5. This is usually a sign that you are breaking the pre-trained weights. They recover afterwards but it is still probably not optimal.", "votes": null }, { "id": "819719", "postDate": "04/24/2020 19:46:11", "content": "I tested it multiple times and the f1 score gives better results sometimes for the one cycle and sometimes for the other so I couldn't concretely find which is better. I also think the problem is I didn't use an optimal learning rate for the one cycle scheduler, for example using learning rate finder like the fastai would help I think.
", "rawMarkdown": "I tested it multiple times and the f1 score gives better results sometimes for the one cycle and sometimes for the other so I couldn't concretely find which is better. I also think the problem is I didn't use an optimal learning rate for the one cycle scheduler, for example using learning rate finder like the fastai would help I think.", "votes": null }, { "id": "820099", "postDate": "04/25/2020 06:19:29", "content": "Thanks for sharing these insight
", "rawMarkdown": "Thanks for sharing these insight", "votes": null } ], "comments": [ { "id": 818967, "author_name": "albeffe", "author_url": "", "post_date": "04/24/2020 08:40:59", "content": "Thanks man :)
", "votes": null, "replies": [ { "id": 819025, "author_name": "ibrahimsherify", "author_url": "", "post_date": "04/24/2020 09:43:12", "content": "welcome my friend :D
", "votes": null, "replies": [] } ] }, { "id": 819073, "author_name": "hnakhavan", "author_url": "", "post_date": "04/24/2020 10:38:51", "content": "Great
", "votes": null, "replies": [] }, { "id": 819169, "author_name": "atamazian", "author_url": "", "post_date": "04/24/2020 11:59:09", "content": "Looks interesting
", "votes": null, "replies": [] }, { "id": 819657, "author_name": "mgorner", "author_url": "", "post_date": "04/24/2020 18:51:14", "content": "The two schedulers don't look that different to me and you get to the same f1 score in the end.
\n\nLooking at your loss curves however, you have a big hit on the validation loss around epochs 3-4-5. This is usually a sign that you are breaking the pre-trained weights. They recover afterwards but it is still probably not optimal.
", "votes": null, "replies": [ { "id": 819719, "author_name": "ibrahimsherify", "author_url": "", "post_date": "04/24/2020 19:46:11", "content": "I tested it multiple times and the f1 score gives better results sometimes for the one cycle and sometimes for the other so I couldn't concretely find which is better. I also think the problem is I didn't use an optimal learning rate for the one cycle scheduler, for example using learning rate finder like the fastai would help I think.
", "votes": null, "replies": [] }, { "id": 820099, "author_name": "kurianbenoy", "author_url": "", "post_date": "04/25/2020 06:19:29", "content": "Thanks for sharing these insight
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "818938": "I wanted to share a notebook [Comparing Two Different Schedulers](https://www.kaggle.com/ibrahimsherify/comparing-two-different-schedulers?scriptVersionId=32625540) which compares two different learning rate schedulers. The starter [notebook](https://www.kaggle.com/mgornergoogle/five-flowers-with-keras-and-xception-on-tpu) scheduler (I call it TPU scheduler) and One cycle scheduler implemented in the fastai [module](https://docs.fast.ai/callbacks.one_cycle.html#Training-with-the-1cycle-policy). Any feedback would be greatly appreciated.\n\n\n\nUPDATE: There was mistake in the notebook for the second experiment which was fixed and updated.", "818967": "Thanks man :)", "819025": "welcome my friend :D", "819073": "Great", "819169": "Looks interesting", "819657": "The two schedulers don't look that different to me and you get to the same f1 score in the end.\n\nLooking at your loss curves however, you have a big hit on the validation loss around epochs 3-4-5. This is usually a sign that you are breaking the pre-trained weights. They recover afterwards but it is still probably not optimal.", "819719": "I tested it multiple times and the f1 score gives better results sometimes for the one cycle and sometimes for the other so I couldn't concretely find which is better. I also think the problem is I didn't use an optimal learning rate for the one cycle scheduler, for example using learning rate finder like the fastai would help I think.", "820099": "Thanks for sharing these insight" }, "source": "meta" }