![\"Mean](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1035002%2F05a5209502afd0e81baacd9d2d0ef00e%2Faveragenll.png?generation=1600410251301531&alt=media\")
One of the battles many of us are fighting against this data is that the data loading is a serious bottleneck. Obviously it would be great to remove this bottleneck and improve training throughput but so far it doesn't seem like there has been a great way around the rasterization slow down. Rather than trying to load the data faster maybe it is prudent to load only the most useful data.
\nIn my preliminary experiments, I am seeing that the loss numbers per batch are significantly skewed.
\n
This histogram is showing the distribution of the mean NLL per batch across my last 200k training batches. There is a significant mass centered around 20 and then a long tail with a max reaching all the way to >900. My thought is that it likely isnt particularly worthwhile to continue optimizing for samples that are already well predicted by our model. This seems to be a common theme in self-driving cars, there is a lot of data, but it is difficult to hone in on the important data and not waste time on the mundane and predictable.
\n![\"Describe](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1035002%2Ff191ffc2d0e491f7ab37f7934c322315%2Fdistribution_describe.png?generation=1600410606005697&alt=media\")
I looked at OHEM(Online Hard Example Mining), but this doesn't seem ideal to me as the focus is put on the hard samples but all data is still loaded. One alternate idea is to do some sort of weighted sampler that increases the prevalence of scenes based on their loss on previous epochs. Then on future epochs the harder samples will happen more frequently and unuseful training data will not be loaded. The issue with this is that running through all of the data even once is fairly time-consuming. I have begun logging these numbers because I can't imagine I will have time to do a continuous run but maybe if I capture the data early it will be useful to survey the mean across my various different experiments per training sample.
\nIn theory with the correctly chosen points we could likely converge a model much faster with much less processing from the dataloader. Does anyone else have any ideas in this regard?
", "messages": [ { "id": 1015380, "postDate": "2020-09-18T06:38:13.030Z", "content": "One of the battles many of us are fighting against this data is that the data loading is a serious bottleneck. Obviously it would be great to remove this bottleneck and improve training throughput but so far it doesn't seem like there has been a great way around the rasterization slow down. Rather than trying to load the data faster maybe it is prudent to load only the most useful data.
\nIn my preliminary experiments, I am seeing that the loss numbers per batch are significantly skewed.
\n
This histogram is showing the distribution of the mean NLL per batch across my last 200k training batches. There is a significant mass centered around 20 and then a long tail with a max reaching all the way to >900. My thought is that it likely isnt particularly worthwhile to continue optimizing for samples that are already well predicted by our model. This seems to be a common theme in self-driving cars, there is a lot of data, but it is difficult to hone in on the important data and not waste time on the mundane and predictable.
\n
I looked at OHEM(Online Hard Example Mining), but this doesn't seem ideal to me as the focus is put on the hard samples but all data is still loaded. One alternate idea is to do some sort of weighted sampler that increases the prevalence of scenes based on their loss on previous epochs. Then on future epochs the harder samples will happen more frequently and unuseful training data will not be loaded. The issue with this is that running through all of the data even once is fairly time-consuming. I have begun logging these numbers because I can't imagine I will have time to do a continuous run but maybe if I capture the data early it will be useful to survey the mean across my various different experiments per training sample.
\nIn theory with the correctly chosen points we could likely converge a model much faster with much less processing from the dataloader. Does anyone else have any ideas in this regard?
", "rawMarkdown": "One of the battles many of us are fighting against this data is that the data loading is a serious bottleneck. Obviously it would be great to remove this bottleneck and improve training throughput but so far it doesn't seem like there has been a great way around the rasterization slow down. Rather than trying to load the data faster maybe it is prudent to load only the most useful data. \n\nIn my preliminary experiments, I am seeing that the loss numbers per batch are significantly skewed.\n\n\n\nThis histogram is showing the distribution of the mean NLL per batch across my last 200k training batches. There is a significant mass centered around 20 and then a long tail with a max reaching all the way to >900. My thought is that it likely isnt particularly worthwhile to continue optimizing for samples that are already well predicted by our model. This seems to be a common theme in self-driving cars, there is a lot of data, but it is difficult to hone in on the important data and not waste time on the mundane and predictable. \n\n\n\nI looked at OHEM(Online Hard Example Mining), but this doesn't seem ideal to me as the focus is put on the hard samples but all data is still loaded. One alternate idea is to do some sort of weighted sampler that increases the prevalence of scenes based on their loss on previous epochs. Then on future epochs the harder samples will happen more frequently and unuseful training data will not be loaded. The issue with this is that running through all of the data even once is fairly time-consuming. I have begun logging these numbers because I can't imagine I will have time to do a continuous run but maybe if I capture the data early it will be useful to survey the mean across my various different experiments per training sample. \n\nIn theory with the correctly chosen points we could likely converge a model much faster with much less processing from the dataloader. Does anyone else have any ideas in this regard?", "votes": 32 }, { "id": 1016386, "postDate": "2020-09-18T22:50:20.873Z", "content": "Can you try increasing the min_frame_history and min_frame_future and retrying the test?
\ni.e. >AgentDataset(cfg, sample_zarr, rasterizer, min_frame_history=20, min_frame_future=20)
The defaults are set to 10 and 1 respectively. Changing these to a bigger number means you actually get agents with more of a future and a past, as having an agent for which you have to only predict 1 frame in the future won't be nearly as useful as having to 20 target values.
\nI did a quick check on the sample dataset, and if you increase min_frame_history and min_frame_future to 20 the amount of agents go from 111634 to 52277. If you up it to 50 it drops down to 19560.
\nI'm willing to bet doing this will increase your mean loss significantly.
", "rawMarkdown": "Can you try increasing the min_frame_history and min_frame_future and retrying the test? \ni.e. >`AgentDataset(cfg, sample_zarr, rasterizer, min_frame_history=20, min_frame_future=20)`\n\nThe defaults are set to 10 and 1 respectively. Changing these to a bigger number means you actually get agents with more of a future and a past, as having an agent for which you have to only predict 1 frame in the future won't be nearly as useful as having to 20 target values. \n\nI did a quick check on the sample dataset, and if you increase min_frame_history and min_frame_future to 20 the amount of agents go from 111634 to 52277. If you up it to 50 it drops down to 19560. \n\nI'm willing to bet doing this will increase your mean loss significantly. ", "votes": 7, "replies": [ { "id": 1016400, "postDate": "2020-09-18T23:31:44.717Z", "content": "@fnands The number might go down. But its a very bad idea. You have to predict 50 frames into the future. You can reduce agent by making history to zero even though it compromises accuracy. But why would you want to decrease future frames? reducing future up to 30 makes little sense(50 should be out of the question). But I am sure test data is set in such a way to account for diverse futures. with up to 30 you will definitely wont meet optimal(its my thinking). This reduction also can only be used to test if test data is diverse enough but not to arrive at best solution.
", "rawMarkdown": "@fnands The number might go down. But its a very bad idea. You have to predict 50 frames into the future. You can reduce agent by making history to zero even though it compromises accuracy. But why would you want to decrease future frames? reducing future up to 30 makes little sense(50 should be out of the question). But I am sure test data is set in such a way to account for diverse futures. with up to 30 you will definitely wont meet optimal(its my thinking). This reduction also can only be used to test if test data is diverse enough but not to arrive at best solution. ", "votes": -3 }, { "id": 1016702, "postDate": "2020-09-19T07:11:05.693Z", "content": "Changing these parameters will increase the average number of future frames, and I believe therefore increase the mean loss.
\nI think you are misunderstanding how the AgentDataset works, and I did too, as it's not clearly explained.
\nThe AgentDataset doesn't iterate through each agent by track_id, it iterates through each agent by the agent_index, which can change per frame, even for the same \"track_id\", i.e. the same vehicle.
\nAt the moment the default minimum min_frame_future = 1, meaning if you don't set it, you will get many images in your dataset with only 1 real future frame. It doesn't matter if you set future_num_frames to 50, as you will just end up with 1 frame with non-zero 'target_positions' and 49 frames of padded on zeroes.
Do we know how the test set was generated? i.e. is there a min_frame_future implemented implicitly there?
", "rawMarkdown": "Do we know how the test set was generated? i.e. is there a min_frame_future implemented implicitly there?", "votes": 5 }, { "id": 1018425, "postDate": "2020-09-19T17:14:36.917Z", "content": "@fergusoci great question, I just checked the history for the first couple test cases.
\ntest_dataset[34]['history_availabilities'].sum() = 3.0
\ntest_dataset[138]['history_availabilities'].sum() = 4.0
so, there are test cases with only very few history frames. I would suspect the same for the future frames.
", "rawMarkdown": "@fergusoci great question, I just checked the history for the first couple test cases.\n`test_dataset[34]['history_availabilities'].sum() = 3.0`\n`test_dataset[138]['history_availabilities'].sum() = 4.0`\n\nso, there are test cases with only very few history frames. I would suspect the same for the future frames.", "votes": 6 }, { "id": 1018434, "postDate": "2020-09-19T17:19:47.780Z", "content": "I believe that is 10. I believe that is what the create_chopped_dataset function does, find all the samples that satisfy the constraints. Looking in the evaluation portion of this notebook the create_chopped_dataset takes the MIN_FUTURE_STEPS argument and it is set to 10 in this file
", "rawMarkdown": "I believe that is 10. I believe that is what the create_chopped_dataset function does, find all the samples that satisfy the constraints. Looking in the evaluation portion of [this notebook](https://github.com/lyft/l5kit/blob/master/examples/agent_motion_prediction/agent_motion_prediction.ipynb) the create_chopped_dataset takes the MIN_FUTURE_STEPS argument and it is set to 10 in [this file](https://github.com/lyft/l5kit/blob/master/l5kit/l5kit/evaluation/chop_dataset.py)", "votes": 5 }, { "id": 1018488, "postDate": "2020-09-19T17:51:13.080Z", "content": "@ilu000 , I did a quick test and it seems there are a few cases in the test set with very short histories (1 frame, even when using the test mask), although the vast majority give you plenty of history to work with. So I guess the cases where we have only 1 frame of history to go on can be pretty challenging.
\nTo answer @fergusoci's question I think @ryches is right, as we know they used create_chopped_dataset to make the test set, and I guess we can assume the used MIN_FUTURE_STEPS = 10, which could maybe mean that setting min_frame_future=10 will get you a better approximation of the test set.
@ryches good find for the MIN_FUTURE_STEPS
\nWe also know that num_frames_to_copy is set to 100, but that doesn't carry over to a minimum history length. So, I assume we see anything from 1 to 100 for history_availabilities in the test set with a peak at 100.
I might be wrong but I dont agree with few mentioned statement like min history length =1 in some cases. If some of ye all want to look at Ground truth and target availability with sample.zaar. I will also do the same for eval data so everyone can directly see the groundtrutt.csv , use it eval and you dont have to generate that zarr. You can visit my dataset. https://www.kaggle.com/deepakrajpurushothaman/sample-chopped-100 .
", "rawMarkdown": "I might be wrong but I dont agree with few mentioned statement like min history length =1 in some cases. If some of ye all want to look at Ground truth and target availability with sample.zaar. I will also do the same for eval data so everyone can directly see the groundtrutt.csv , use it eval and you dont have to generate that zarr. You can visit my dataset. https://www.kaggle.com/deepakrajpurushothaman/sample-chopped-100 . ", "votes": -1 }, { "id": 1019408, "postDate": "2020-09-20T12:10:32.017Z", "content": "@deepakrajpurushothaman i can't follow your thoughts. History is always 100 for the entire test set. But the availability (history_availabilities) is lower, sometimes as low as 1 as @fnands said above.
\nFor the future frames we can only speculate, but given the hardcoded value for MIN_FUTURE_STEPS = 10 in the repository this value is probably a good guess.
Does anybody know how validate.zarr was created? Was that done using create_chopped_dataset() too? Dug around, but can't see any mention of it anywhere…
The validate file was not created with that function. The create_chopped_dataset function must be run against it to make data that is representative of the test set. I've found good correlation between val and test by doing this. The validate is raw like the train files
", "rawMarkdown": "The validate file was not created with that function. The create_chopped_dataset function must be run against it to make data that is representative of the test set. I've found good correlation between val and test by doing this. The validate is raw like the train files", "votes": 6 }, { "id": 1021371, "postDate": "2020-09-21T19:37:56.680Z", "content": "The examples in the l5kit repo show this process
", "rawMarkdown": "The examples in the l5kit repo show this process", "votes": 1 }, { "id": 1021383, "postDate": "2020-09-21T19:44:49.900Z", "content": "Great, thanks.
", "rawMarkdown": "Great, thanks. ", "votes": 1 }, { "id": 1045772, "postDate": "2020-10-11T02:51:11.557Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1045773, "postDate": "2020-10-11T02:52:03.320Z", "content": "Should I follow the example in https://github.com/lyft/l5kit/blob/master/examples/agent_motion_prediction/agent_motion_prediction.ipynb to test the model on validation set?
", "rawMarkdown": "Should I follow the example in [https://github.com/lyft/l5kit/blob/master/examples/agent_motion_prediction/agent_motion_prediction.ipynb](url) to test the model on validation set?" } ] }, { "id": 1015520, "postDate": "2020-09-18T08:31:58.563Z", "content": "Good observation. Continuing your line of thought, at first we can load the data without rasterization, and analyze history_positions and target_positions. If the targets can be accurately predicted with a constant speed model, then skip or undersample these items.
Exactly what I though. You can easily predict the motion for many agents just by extrapolating their initial positions/velocities, so these probably don't teach you much.
\nAs a first experiment you can probably just completely drop these from the training set.
", "rawMarkdown": "Exactly what I though. You can easily predict the motion for many agents just by extrapolating their initial positions/velocities, so these probably don't teach you much. \n\nAs a first experiment you can probably just completely drop these from the training set. ", "votes": 2 }, { "id": 1069586, "postDate": "2020-11-04T16:24:45.537Z", "content": "Thanks for your insights @zaharch! What do you mean by constant speed model? Is it related to this notebook by you - https://www.kaggle.com/zaharch/kalman-filter-baseline?
", "rawMarkdown": "Thanks for your insights @zaharch! What do you mean by constant speed model? Is it related to this notebook by you - https://www.kaggle.com/zaharch/kalman-filter-baseline?" }, { "id": 1069598, "postDate": "2020-11-04T16:55:50.703Z", "content": "Not necessarily related, by constant speed I mean estimating speed by any simple method. For example just calculating it from the last two history position points.
", "rawMarkdown": "Not necessarily related, by constant speed I mean estimating speed by any simple method. For example just calculating it from the last two history position points.", "votes": 1 } ] }, { "id": 1046803, "postDate": "2020-10-12T02:48:43.560Z", "content": "great
", "rawMarkdown": "great" }, { "id": 1015804, "postDate": "2020-09-18T12:46:28.713Z", "content": "I idea of 'weighted sampler' gets my attention. How do you go about doing that?
", "rawMarkdown": "I idea of 'weighted sampler' gets my attention. How do you go about doing that?", "replies": [ { "id": 1015866, "postDate": "2020-09-18T13:45:28.877Z", "content": "In pytorch it isn't too difficult. There is a built in weighted sampler you can configure and then use with your dataloaders. https://pytorch.org/docs/1.1.0/_modules/torch/utils/data/sampler.html
\nI tried working in that direction but ran into an issue with the sampling not being able to work when the dataset size is more than 2^24 samples. https://github.com/pytorch/pytorch/issues/2576
", "rawMarkdown": "In pytorch it isn't too difficult. There is a built in weighted sampler you can configure and then use with your dataloaders. https://pytorch.org/docs/1.1.0/_modules/torch/utils/data/sampler.html\n\nI tried working in that direction but ran into an issue with the sampling not being able to work when the dataset size is more than 2^24 samples. https://github.com/pytorch/pytorch/issues/2576", "votes": 1 }, { "id": 1016304, "postDate": "2020-09-18T20:02:49.450Z", "content": "Thank! I will have a look at it.
", "rawMarkdown": "Thank! I will have a look at it." }, { "id": 1059838, "postDate": "2020-10-25T14:15:34.860Z", "content": "Can we use a 50% subset of the training set when we are using weighted sampler? But I am confused that how we can use weighted sampler according to the label or the loss. If we use sampler refers to losses, should we go through all the data in the training set?
", "rawMarkdown": "Can we use a 50% subset of the training set when we are using weighted sampler? But I am confused that how we can use weighted sampler according to the label or the loss. If we use sampler refers to losses, should we go through all the data in the training set?" } ] }, { "id": 1020035, "postDate": "2020-09-20T20:34:53.787Z", "content": "thank you!
", "rawMarkdown": "thank you! ", "votes": 2 } ], "comments": [ { "id": 1016386, "author_name": "fnands", "author_url": "", "post_date": "2020-09-18T22:50:20.873000", "content": "Can you try increasing the min_frame_history and min_frame_future and retrying the test?
\ni.e. >AgentDataset(cfg, sample_zarr, rasterizer, min_frame_history=20, min_frame_future=20)
The defaults are set to 10 and 1 respectively. Changing these to a bigger number means you actually get agents with more of a future and a past, as having an agent for which you have to only predict 1 frame in the future won't be nearly as useful as having to 20 target values.
\nI did a quick check on the sample dataset, and if you increase min_frame_history and min_frame_future to 20 the amount of agents go from 111634 to 52277. If you up it to 50 it drops down to 19560.
\nI'm willing to bet doing this will increase your mean loss significantly.
", "votes": 7, "replies": [ { "id": 1016400, "author_name": "The Brown Iceman", "author_url": "", "post_date": "2020-09-18T23:31:44.717000", "content": "@fnands The number might go down. But its a very bad idea. You have to predict 50 frames into the future. You can reduce agent by making history to zero even though it compromises accuracy. But why would you want to decrease future frames? reducing future up to 30 makes little sense(50 should be out of the question). But I am sure test data is set in such a way to account for diverse futures. with up to 30 you will definitely wont meet optimal(its my thinking). This reduction also can only be used to test if test data is diverse enough but not to arrive at best solution.
", "votes": -3, "replies": [] }, { "id": 1016702, "author_name": "fnands", "author_url": "", "post_date": "2020-09-19T07:11:05.693000", "content": "Changing these parameters will increase the average number of future frames, and I believe therefore increase the mean loss.
\nI think you are misunderstanding how the AgentDataset works, and I did too, as it's not clearly explained.
\nThe AgentDataset doesn't iterate through each agent by track_id, it iterates through each agent by the agent_index, which can change per frame, even for the same \"track_id\", i.e. the same vehicle.
\nAt the moment the default minimum min_frame_future = 1, meaning if you don't set it, you will get many images in your dataset with only 1 real future frame. It doesn't matter if you set future_num_frames to 50, as you will just end up with 1 frame with non-zero 'target_positions' and 49 frames of padded on zeroes.
Do we know how the test set was generated? i.e. is there a min_frame_future implemented implicitly there?
", "votes": 5, "replies": [] }, { "id": 1018425, "author_name": "Pascal Pfeiffer", "author_url": "", "post_date": "2020-09-19T17:14:36.917000", "content": "@fergusoci great question, I just checked the history for the first couple test cases.
\ntest_dataset[34]['history_availabilities'].sum() = 3.0
\ntest_dataset[138]['history_availabilities'].sum() = 4.0
so, there are test cases with only very few history frames. I would suspect the same for the future frames.
", "votes": 6, "replies": [] }, { "id": 1018434, "author_name": "ryches", "author_url": "", "post_date": "2020-09-19T17:19:47.780000", "content": "I believe that is 10. I believe that is what the create_chopped_dataset function does, find all the samples that satisfy the constraints. Looking in the evaluation portion of this notebook the create_chopped_dataset takes the MIN_FUTURE_STEPS argument and it is set to 10 in this file
", "votes": 5, "replies": [] }, { "id": 1018488, "author_name": "fnands", "author_url": "", "post_date": "2020-09-19T17:51:13.080000", "content": "@ilu000 , I did a quick test and it seems there are a few cases in the test set with very short histories (1 frame, even when using the test mask), although the vast majority give you plenty of history to work with. So I guess the cases where we have only 1 frame of history to go on can be pretty challenging.
\nTo answer @fergusoci's question I think @ryches is right, as we know they used create_chopped_dataset to make the test set, and I guess we can assume the used MIN_FUTURE_STEPS = 10, which could maybe mean that setting min_frame_future=10 will get you a better approximation of the test set.
@ryches good find for the MIN_FUTURE_STEPS
\nWe also know that num_frames_to_copy is set to 100, but that doesn't carry over to a minimum history length. So, I assume we see anything from 1 to 100 for history_availabilities in the test set with a peak at 100.
I might be wrong but I dont agree with few mentioned statement like min history length =1 in some cases. If some of ye all want to look at Ground truth and target availability with sample.zaar. I will also do the same for eval data so everyone can directly see the groundtrutt.csv , use it eval and you dont have to generate that zarr. You can visit my dataset. https://www.kaggle.com/deepakrajpurushothaman/sample-chopped-100 .
", "votes": -1, "replies": [] }, { "id": 1019408, "author_name": "Pascal Pfeiffer", "author_url": "", "post_date": "2020-09-20T12:10:32.017000", "content": "@deepakrajpurushothaman i can't follow your thoughts. History is always 100 for the entire test set. But the availability (history_availabilities) is lower, sometimes as low as 1 as @fnands said above.
\nFor the future frames we can only speculate, but given the hardcoded value for MIN_FUTURE_STEPS = 10 in the repository this value is probably a good guess.
Does anybody know how validate.zarr was created? Was that done using create_chopped_dataset() too? Dug around, but can't see any mention of it anywhere…
The validate file was not created with that function. The create_chopped_dataset function must be run against it to make data that is representative of the test set. I've found good correlation between val and test by doing this. The validate is raw like the train files
", "votes": 6, "replies": [] }, { "id": 1021371, "author_name": "ryches", "author_url": "", "post_date": "2020-09-21T19:37:56.680000", "content": "The examples in the l5kit repo show this process
", "votes": 1, "replies": [] }, { "id": 1021383, "author_name": "fergusoci", "author_url": "", "post_date": "2020-09-21T19:44:49.900000", "content": "Great, thanks.
", "votes": 1, "replies": [] }, { "id": 1045772, "author_name": "", "author_url": "", "post_date": "2020-10-11T02:51:11.557000", "content": "", "votes": 0, "replies": [] }, { "id": 1045773, "author_name": "Yannik", "author_url": "", "post_date": "2020-10-11T02:52:03.320000", "content": "Should I follow the example in https://github.com/lyft/l5kit/blob/master/examples/agent_motion_prediction/agent_motion_prediction.ipynb to test the model on validation set?
", "votes": 0, "replies": [] } ] }, { "id": 1015520, "author_name": "nosound", "author_url": "", "post_date": "2020-09-18T08:31:58.563000", "content": "Good observation. Continuing your line of thought, at first we can load the data without rasterization, and analyze history_positions and target_positions. If the targets can be accurately predicted with a constant speed model, then skip or undersample these items.
Exactly what I though. You can easily predict the motion for many agents just by extrapolating their initial positions/velocities, so these probably don't teach you much.
\nAs a first experiment you can probably just completely drop these from the training set.
", "votes": 2, "replies": [] }, { "id": 1069586, "author_name": "Arpit", "author_url": "", "post_date": "2020-11-04T16:24:45.537000", "content": "Thanks for your insights @zaharch! What do you mean by constant speed model? Is it related to this notebook by you - https://www.kaggle.com/zaharch/kalman-filter-baseline?
", "votes": 0, "replies": [] }, { "id": 1069598, "author_name": "nosound", "author_url": "", "post_date": "2020-11-04T16:55:50.703000", "content": "Not necessarily related, by constant speed I mean estimating speed by any simple method. For example just calculating it from the last two history position points.
", "votes": 1, "replies": [] } ] }, { "id": 1046803, "author_name": "Rohit Dube", "author_url": "", "post_date": "2020-10-12T02:48:43.560000", "content": "great
", "votes": 0, "replies": [] }, { "id": 1015804, "author_name": "The Brown Iceman", "author_url": "", "post_date": "2020-09-18T12:46:28.713000", "content": "I idea of 'weighted sampler' gets my attention. How do you go about doing that?
", "votes": 0, "replies": [ { "id": 1015866, "author_name": "ryches", "author_url": "", "post_date": "2020-09-18T13:45:28.877000", "content": "In pytorch it isn't too difficult. There is a built in weighted sampler you can configure and then use with your dataloaders. https://pytorch.org/docs/1.1.0/_modules/torch/utils/data/sampler.html
\nI tried working in that direction but ran into an issue with the sampling not being able to work when the dataset size is more than 2^24 samples. https://github.com/pytorch/pytorch/issues/2576
", "votes": 1, "replies": [] }, { "id": 1016304, "author_name": "The Brown Iceman", "author_url": "", "post_date": "2020-09-18T20:02:49.450000", "content": "Thank! I will have a look at it.
", "votes": 0, "replies": [] }, { "id": 1059838, "author_name": "Yannik", "author_url": "", "post_date": "2020-10-25T14:15:34.860000", "content": "Can we use a 50% subset of the training set when we are using weighted sampler? But I am confused that how we can use weighted sampler according to the label or the loss. If we use sampler refers to losses, should we go through all the data in the training set?
", "votes": 0, "replies": [] } ] }, { "id": 1020035, "author_name": "berkaycihan", "author_url": "", "post_date": "2020-09-20T20:34:53.787000", "content": "thank you!
", "votes": 2, "replies": [] } ], "raw_markdown_by_id": { "1015380": "One of the battles many of us are fighting against this data is that the data loading is a serious bottleneck. Obviously it would be great to remove this bottleneck and improve training throughput but so far it doesn't seem like there has been a great way around the rasterization slow down. Rather than trying to load the data faster maybe it is prudent to load only the most useful data. \n\nIn my preliminary experiments, I am seeing that the loss numbers per batch are significantly skewed.\n\n\n\nThis histogram is showing the distribution of the mean NLL per batch across my last 200k training batches. There is a significant mass centered around 20 and then a long tail with a max reaching all the way to >900. My thought is that it likely isnt particularly worthwhile to continue optimizing for samples that are already well predicted by our model. This seems to be a common theme in self-driving cars, there is a lot of data, but it is difficult to hone in on the important data and not waste time on the mundane and predictable. \n\n\n\nI looked at OHEM(Online Hard Example Mining), but this doesn't seem ideal to me as the focus is put on the hard samples but all data is still loaded. One alternate idea is to do some sort of weighted sampler that increases the prevalence of scenes based on their loss on previous epochs. Then on future epochs the harder samples will happen more frequently and unuseful training data will not be loaded. The issue with this is that running through all of the data even once is fairly time-consuming. I have begun logging these numbers because I can't imagine I will have time to do a continuous run but maybe if I capture the data early it will be useful to survey the mean across my various different experiments per training sample. \n\nIn theory with the correctly chosen points we could likely converge a model much faster with much less processing from the dataloader. Does anyone else have any ideas in this regard?", "1016386": "Can you try increasing the min_frame_history and min_frame_future and retrying the test? \ni.e. >`AgentDataset(cfg, sample_zarr, rasterizer, min_frame_history=20, min_frame_future=20)`\n\nThe defaults are set to 10 and 1 respectively. Changing these to a bigger number means you actually get agents with more of a future and a past, as having an agent for which you have to only predict 1 frame in the future won't be nearly as useful as having to 20 target values. \n\nI did a quick check on the sample dataset, and if you increase min_frame_history and min_frame_future to 20 the amount of agents go from 111634 to 52277. If you up it to 50 it drops down to 19560. \n\nI'm willing to bet doing this will increase your mean loss significantly. ", "1015520": "Good observation. Continuing your line of thought, at first we can load the data without rasterization, and analyze `history_positions` and `target_positions`. If the targets can be accurately predicted with a constant speed model, then skip or undersample these items.", "1046803": "great", "1015804": "I idea of 'weighted sampler' gets my attention. How do you go about doing that?", "1020035": "thank you! " } }