I am trying to use the LRFinder function to determine the lr_scheduler optimal range with this code:
from torch_lr_finder import LRFinder\n\ncriterion = pytorch_neg_multi_log_likelihood_batch\noptimizer = optim.AdamW(model.parameters())\nlr_finder = LRFinder(model, optimizer, forward, device)\nlr_finder.range_test(train_loader, end_lr=10, num_iter=1000)\nAnd I got this error:
\n---------------------------------------------------------------------------\nValueError Traceback (most recent call last)\n<ipython-input-22-d6535fcb2010> in <module>\n----> 1 lr_finder.range_test(train_loader_for_LRF, end_lr=10, num_iter=1000)\n\n~/anaconda3/envs/lyft_env/lib/python3.8/site-packages/torch_lr_finder/lr_finder.py in range_test(self, train_loader, val_loader, start_lr, end_lr, num_iter, step_mode, smooth_f, diverge_th, accumulation_steps, non_blocking_transfer)\n 315 for iteration in tqdm(range(num_iter)):\n 316 # Train on batch and retrieve loss\n--> 317 loss = self._train_batch(\n 318 train_iter,\n 319 accumulation_steps,\n\n~/anaconda3/envs/lyft_env/lib/python3.8/site-packages/torch_lr_finder/lr_finder.py in _train_batch(self, train_iter, accumulation_steps, non_blocking_transfer)\n 369 self.optimizer.zero_grad()\n 370 for i in range(accumulation_steps):\n--> 371 inputs, labels = next(train_iter)\n 372 inputs, labels = self._move_to_device(\n 373 inputs, labels, non_blocking=non_blocking_transfer\n\n~/anaconda3/envs/lyft_env/lib/python3.8/site-packages/torch_lr_finder/lr_finder.py in __next__(self)\n 57 try:\n 58 batch = next(self._iterator)\n---> 59 inputs, labels = self.inputs_labels_from_batch(batch)\n 60 except StopIteration:\n 61 if not self.auto_reset:\n\n~/anaconda3/envs/lyft_env/lib/python3.8/site-packages/torch_lr_finder/lr_finder.py in inputs_labels_from_batch(self, batch_data)\n 31 def inputs_labels_from_batch(self, batch_data):\n 32 if not isinstance(batch_data, list) and not isinstance(batch_data, tuple):\n---> 33 raise ValueError(\n 34 \"Your batch type is not supported: {}. Please inherit from \"\n 35 \"`TrainDataLoaderIter` or `ValDataLoaderIter` and override the \"\n\nValueError: Your batch type is not supported: <class 'dict'>. Please inherit from `TrainDataLoaderIter` or `ValDataLoaderIter` and override the `inputs_labels_from_batch` method.\nThe pytorch_neg_multi_log_likelihood_batch and the train_loader are taken from the Lyft: Complete train and prediction pipeline notebook
So in short Your batch type is not supported: <class 'dict'>. which make sense, but how can I convert the AgentDataset output, that I could use the LRFinder?
Thanks for help!
", "messages": [ { "id": "1073348", "postDate": "11/09/2020 12:56:57", "content": "I am trying to use the LRFinder function to determine the lr_scheduler optimal range with this code:
from torch_lr_finder import LRFinder\n\ncriterion = pytorch_neg_multi_log_likelihood_batch\noptimizer = optim.AdamW(model.parameters())\nlr_finder = LRFinder(model, optimizer, forward, device)\nlr_finder.range_test(train_loader, end_lr=10, num_iter=1000)\nAnd I got this error:
\n---------------------------------------------------------------------------\nValueError Traceback (most recent call last)\n<ipython-input-22-d6535fcb2010> in <module>\n----> 1 lr_finder.range_test(train_loader_for_LRF, end_lr=10, num_iter=1000)\n\n~/anaconda3/envs/lyft_env/lib/python3.8/site-packages/torch_lr_finder/lr_finder.py in range_test(self, train_loader, val_loader, start_lr, end_lr, num_iter, step_mode, smooth_f, diverge_th, accumulation_steps, non_blocking_transfer)\n 315 for iteration in tqdm(range(num_iter)):\n 316 # Train on batch and retrieve loss\n--> 317 loss = self._train_batch(\n 318 train_iter,\n 319 accumulation_steps,\n\n~/anaconda3/envs/lyft_env/lib/python3.8/site-packages/torch_lr_finder/lr_finder.py in _train_batch(self, train_iter, accumulation_steps, non_blocking_transfer)\n 369 self.optimizer.zero_grad()\n 370 for i in range(accumulation_steps):\n--> 371 inputs, labels = next(train_iter)\n 372 inputs, labels = self._move_to_device(\n 373 inputs, labels, non_blocking=non_blocking_transfer\n\n~/anaconda3/envs/lyft_env/lib/python3.8/site-packages/torch_lr_finder/lr_finder.py in __next__(self)\n 57 try:\n 58 batch = next(self._iterator)\n---> 59 inputs, labels = self.inputs_labels_from_batch(batch)\n 60 except StopIteration:\n 61 if not self.auto_reset:\n\n~/anaconda3/envs/lyft_env/lib/python3.8/site-packages/torch_lr_finder/lr_finder.py in inputs_labels_from_batch(self, batch_data)\n 31 def inputs_labels_from_batch(self, batch_data):\n 32 if not isinstance(batch_data, list) and not isinstance(batch_data, tuple):\n---> 33 raise ValueError(\n 34 \"Your batch type is not supported: {}. Please inherit from \"\n 35 \"`TrainDataLoaderIter` or `ValDataLoaderIter` and override the \"\n\nValueError: Your batch type is not supported: <class 'dict'>. Please inherit from `TrainDataLoaderIter` or `ValDataLoaderIter` and override the `inputs_labels_from_batch` method.\nThe pytorch_neg_multi_log_likelihood_batch and the train_loader are taken from the Lyft: Complete train and prediction pipeline notebook
So in short Your batch type is not supported: <class 'dict'>. which make sense, but how can I convert the AgentDataset output, that I could use the LRFinder?
Thanks for help!
", "rawMarkdown": "I am trying to use the `LRFinder` function to determine the `lr_scheduler` optimal range with this code:\n```\nfrom torch_lr_finder import LRFinder\n\ncriterion = pytorch_neg_multi_log_likelihood_batch\noptimizer = optim.AdamW(model.parameters())\nlr_finder = LRFinder(model, optimizer, forward, device)\nlr_finder.range_test(train_loader, end_lr=10, num_iter=1000)\n```\n\nAnd I got this error:\n```\n---------------------------------------------------------------------------\nValueError Traceback (most recent call last)\nI'd say the best way would be to inherit from it and replace the getitem such that instead of a dict it gives you something else (I guess a list is what this function wants). You will likely have to change the model also if you're using the one we provide.
", "rawMarkdown": "I'd say the best way would be to inherit from it and replace the getitem such that instead of a dict it gives you something else (I guess a list is what this function wants). You will likely have to change the model also if you're using the one we provide.", "votes": null }, { "id": "1074077", "postDate": "11/10/2020 08:03:24", "content": "There is TrainDataLoaderIter class in the lr_finder package, which can be used to modify the output of your dataset just for LRFinder.
\nIn my case (I have a bit different pipeline than one in the public notebook) it looks like that.
\nfrom torch_lr_finder import LRFinder, TrainDataLoaderIter\n[...]\ntrain_dataloader = [...] # initialize your dataloader\nclass TrainIter(TrainDataLoaderIter):\n def inputs_labels_from_batch(self, batch_data):\n agents = batch_data\n image = batch_data[\"image\"]\n agent_state = batch_get_agent_state(batch_data)\n\n targets = batch_data['target_positions']\n avails = batch_data['target_availabilities']\n return [(image, agent_state), (targets, avails)] # returns [input, targets]\n\ntrain_data_iter = TrainIter(train_dataloader)\n[...] # model initialization, optimizer, LRFinder etc.\nHowever, I found out that LRFinder overstates the best learning rate and it does yield a pretty plot only at the beginning of the training. In later stages, it's useless for me.
", "rawMarkdown": "There is TrainDataLoaderIter class in the lr_finder package, which can be used to modify the output of your dataset just for LRFinder. \n\nIn my case (I have a bit different pipeline than one in the public notebook) it looks like that. \n\n```\nfrom torch_lr_finder import LRFinder, TrainDataLoaderIter\n[...]\ntrain_dataloader = [...] # initialize your dataloader\nclass TrainIter(TrainDataLoaderIter):\n def inputs_labels_from_batch(self, batch_data):\n agents = batch_data\n image = batch_data[\"image\"]\n agent_state = batch_get_agent_state(batch_data)\n\n targets = batch_data['target_positions']\n avails = batch_data['target_availabilities']\n return [(image, agent_state), (targets, avails)] # returns [input, targets]\n\ntrain_data_iter = TrainIter(train_dataloader)\n[...] # model initialization, optimizer, LRFinder etc.\n```\n\nHowever, I found out that LRFinder overstates the best learning rate and it does yield a pretty plot only at the beginning of the training. In later stages, it's useless for me.", "votes": null }, { "id": "1074093", "postDate": "11/10/2020 08:35:26", "content": "Thank you @szacho, this is really helpful. What is the batch_get_agent_state() content?
That's just my custom function that takes some additional input for the model, for the public notebook you only need an image
", "rawMarkdown": "That's just my custom function that takes some additional input for the model, for the public notebook you only need an image", "votes": null } ], "comments": [ { "id": 1073519, "author_name": "lucabergamini", "author_url": "", "post_date": "11/09/2020 15:34:25", "content": "I'd say the best way would be to inherit from it and replace the getitem such that instead of a dict it gives you something else (I guess a list is what this function wants). You will likely have to change the model also if you're using the one we provide.
", "votes": null, "replies": [] }, { "id": 1074077, "author_name": "szacho", "author_url": "", "post_date": "11/10/2020 08:03:24", "content": "There is TrainDataLoaderIter class in the lr_finder package, which can be used to modify the output of your dataset just for LRFinder.
\nIn my case (I have a bit different pipeline than one in the public notebook) it looks like that.
\nfrom torch_lr_finder import LRFinder, TrainDataLoaderIter\n[...]\ntrain_dataloader = [...] # initialize your dataloader\nclass TrainIter(TrainDataLoaderIter):\n def inputs_labels_from_batch(self, batch_data):\n agents = batch_data\n image = batch_data[\"image\"]\n agent_state = batch_get_agent_state(batch_data)\n\n targets = batch_data['target_positions']\n avails = batch_data['target_availabilities']\n return [(image, agent_state), (targets, avails)] # returns [input, targets]\n\ntrain_data_iter = TrainIter(train_dataloader)\n[...] # model initialization, optimizer, LRFinder etc.\nHowever, I found out that LRFinder overstates the best learning rate and it does yield a pretty plot only at the beginning of the training. In later stages, it's useless for me.
", "votes": null, "replies": [ { "id": 1074093, "author_name": "bessenyeiszilrd", "author_url": "", "post_date": "11/10/2020 08:35:26", "content": "Thank you @szacho, this is really helpful. What is the batch_get_agent_state() content?
That's just my custom function that takes some additional input for the model, for the public notebook you only need an image
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1073348": "I am trying to use the `LRFinder` function to determine the `lr_scheduler` optimal range with this code:\n```\nfrom torch_lr_finder import LRFinder\n\ncriterion = pytorch_neg_multi_log_likelihood_batch\noptimizer = optim.AdamW(model.parameters())\nlr_finder = LRFinder(model, optimizer, forward, device)\nlr_finder.range_test(train_loader, end_lr=10, num_iter=1000)\n```\n\nAnd I got this error:\n```\n---------------------------------------------------------------------------\nValueError Traceback (most recent call last)\n