\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4706215%2F95718930bd30d9c7b54044db4449359c%2Fplot1.png?generation=1606362496690194&alt=media\")
After I read discussion of @ryches, Improving convergence and sample efficiency
\nhttps://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/183814
I tried to calculate cumulative distance of target_positions.
\ntrain.zarr with min_frame_future=10
\nlength of AgentDataset: 17,003,687
This plot is sample distribution about non-zero-end of target_positions. number of samples are 850,184 (1/20 of AgentDatast). Most sample has 50 target_positions data.
\n
And histogram of cumulative distance is as follows. (number of samples are 17,003,687)
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4706215%2F4618fd472d51e0cda5ab12a44fab159e%2Fplot2.png?generation=1606362515038305&alt=media\")
And this is cumulative graph.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4706215%2F422cd1396d43aee8a97963a6713fb017%2Fplot3.png?generation=1606362532269209&alt=media\")
About 60% of data have less than 3 of cumulative distance.
\n| distance | \npercentile | \n
|---|---|
| 0 ~ 1 | \n0.3108957002248925 | \n
| 1 ~ 2 | \n0.15160976917937763 | \n
| 2 ~ 3 | \n0.08709361738164917 | \n
| 3 ~ 4 | \n0.057570243617852124 | \n
| 4 ~ 5 | \n0.030621018508934505 | \n
| 5 ~ 6 | \n0.01739799855090196 | \n
| 6 ~ 7 | \n0.011909304338825422 | \n
| 7 ~ 8 | \n0.009271287156662589 | \n
| 8 ~ 9 | \n0.007785491140741341 | \n
| 9 ~ 10 | \n0.006836990580862512 | \n
So i undersampled and made mask of data which of cumulative distance is less than 10 randomly by the inverse of percentile.
\n| distance | \nratio | \n
|---|---|
| 0 ~ 1 | \n0.021991267733573804 | \n
| 1 ~ 2 | \n0.04509597645237031 | \n
| 2 ~ 3 | \n0.0785016260250442 | \n
| 3 ~ 4 | \n0.1187591045514077 | \n
| 4 ~ 5 | \n0.22327769988668522 | \n
| 5 ~ 6 | \n0.3929756955007945 | \n
| 6 ~ 7 | \n0.5740881571540075 | \n
| 7 ~ 8 | \n0.7374370424875981 | \n
| 8 ~ 9 | \n0.8781707482890396 | \n
| 9 ~ 10 | \n1.0 | \n
This is histogram of undersampled data.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4706215%2F77659f08f6f965d19c7edceddf6ed204%2Fplot4.png?generation=1606362629250668&alt=media\")
The total length of undersampled data is 6,415,723. It is 37.7% of original data.
\nAnd I trained with this 6M data on resnet18 for 4 epochs.
\ntrain loss is average of 2k iter(2k x 16 = 32k samples)
| epoch | \ntr_loss | \nval_loss | \nLB | \n
|---|---|---|---|
| 1 | \n25.582 | \n23.814 | \n22.676 | \n
| 2 | \n20.947 | \n21.242 | \n20.529 | \n
| 3 | \n18.935 | \n21.900 | \n20.585 | \n
| 4 | \n17.359 | \n21.878 | \n20.484 | \n
| parameters | \nvalues | \n
|---|---|
| raster_size | \n267, 267 | \n
| pixel_size | \n0.5, 0.5 | \n
| history_num_frames | \n1 | \n
| history_delta_time | \n1 | \n
| batch_size | \n16 | \n
| AdamW | \nlr=1.0e-4, weight_decay=0.01 | \n
| CosineAnnealingWarmRestarts | \nT_0=80k, T_mult=1, eta_min=1.0e-5 | \n
I think I may have trained in wrong way. I used quite large weight_decay. Anyway training one epoch took ~17 hours on GCE with vCPU:16, GPU: T4 and ~13 hours with vCPU:24, GPU: T4.
\nI guess random undersampling by cumulative distance is not good idea. The information losses may be quite significant. Samples of small cumulative distance may have various trajectories. I think reducing overlapped sample is more appropriate. I wanted to do more experiment but I couldn't because of lack of time.
\nAnd my final submission is just resnet18 with train.zarr, no undersampling. I also tried mobilenetV2 but resnet18 is better.
\n| epoch | \nprivate LB | \npublic LB | \n
|---|---|---|
| 1 | \n19.101 | \n19.886 | \n
| 2 | \n18.185 | \n18.675 | \n
| parameters | \nvalues | \n
|---|---|
| raster_size | \n267, 267 | \n
| pixel_size | \n0.5, 0.5 | \n
| history_num_frames | \n1 | \n
| history_delta_time | \n1 | \n
| batch_size | \n16 | \n
| AdamW | \nweight_decay=2.43e-5 | \n
| first epoch with OneCycleLR | \nmax_lr=1.0e-4, div_factor=10 | \n
| second epoch with CosineAnnealing | \nlr=1.0e-5, eta_min=1.0e-6 | \n
After I read discussion of @ryches, Improving convergence and sample efficiency
\nhttps://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/183814
I tried to calculate cumulative distance of target_positions.
\ntrain.zarr with min_frame_future=10
\nlength of AgentDataset: 17,003,687
This plot is sample distribution about non-zero-end of target_positions. number of samples are 850,184 (1/20 of AgentDatast). Most sample has 50 target_positions data.
\n
And histogram of cumulative distance is as follows. (number of samples are 17,003,687)
\n
And this is cumulative graph.
\n
About 60% of data have less than 3 of cumulative distance.
\n| distance | \npercentile | \n
|---|---|
| 0 ~ 1 | \n0.3108957002248925 | \n
| 1 ~ 2 | \n0.15160976917937763 | \n
| 2 ~ 3 | \n0.08709361738164917 | \n
| 3 ~ 4 | \n0.057570243617852124 | \n
| 4 ~ 5 | \n0.030621018508934505 | \n
| 5 ~ 6 | \n0.01739799855090196 | \n
| 6 ~ 7 | \n0.011909304338825422 | \n
| 7 ~ 8 | \n0.009271287156662589 | \n
| 8 ~ 9 | \n0.007785491140741341 | \n
| 9 ~ 10 | \n0.006836990580862512 | \n
So i undersampled and made mask of data which of cumulative distance is less than 10 randomly by the inverse of percentile.
\n| distance | \nratio | \n
|---|---|
| 0 ~ 1 | \n0.021991267733573804 | \n
| 1 ~ 2 | \n0.04509597645237031 | \n
| 2 ~ 3 | \n0.0785016260250442 | \n
| 3 ~ 4 | \n0.1187591045514077 | \n
| 4 ~ 5 | \n0.22327769988668522 | \n
| 5 ~ 6 | \n0.3929756955007945 | \n
| 6 ~ 7 | \n0.5740881571540075 | \n
| 7 ~ 8 | \n0.7374370424875981 | \n
| 8 ~ 9 | \n0.8781707482890396 | \n
| 9 ~ 10 | \n1.0 | \n
This is histogram of undersampled data.
\n
The total length of undersampled data is 6,415,723. It is 37.7% of original data.
\nAnd I trained with this 6M data on resnet18 for 4 epochs.
\ntrain loss is average of 2k iter(2k x 16 = 32k samples)
| epoch | \ntr_loss | \nval_loss | \nLB | \n
|---|---|---|---|
| 1 | \n25.582 | \n23.814 | \n22.676 | \n
| 2 | \n20.947 | \n21.242 | \n20.529 | \n
| 3 | \n18.935 | \n21.900 | \n20.585 | \n
| 4 | \n17.359 | \n21.878 | \n20.484 | \n
| parameters | \nvalues | \n
|---|---|
| raster_size | \n267, 267 | \n
| pixel_size | \n0.5, 0.5 | \n
| history_num_frames | \n1 | \n
| history_delta_time | \n1 | \n
| batch_size | \n16 | \n
| AdamW | \nlr=1.0e-4, weight_decay=0.01 | \n
| CosineAnnealingWarmRestarts | \nT_0=80k, T_mult=1, eta_min=1.0e-5 | \n
I think I may have trained in wrong way. I used quite large weight_decay. Anyway training one epoch took ~17 hours on GCE with vCPU:16, GPU: T4 and ~13 hours with vCPU:24, GPU: T4.
\nI guess random undersampling by cumulative distance is not good idea. The information losses may be quite significant. Samples of small cumulative distance may have various trajectories. I think reducing overlapped sample is more appropriate. I wanted to do more experiment but I couldn't because of lack of time.
\nAnd my final submission is just resnet18 with train.zarr, no undersampling. I also tried mobilenetV2 but resnet18 is better.
\n| epoch | \nprivate LB | \npublic LB | \n
|---|---|---|
| 1 | \n19.101 | \n19.886 | \n
| 2 | \n18.185 | \n18.675 | \n
| parameters | \nvalues | \n
|---|---|
| raster_size | \n267, 267 | \n
| pixel_size | \n0.5, 0.5 | \n
| history_num_frames | \n1 | \n
| history_delta_time | \n1 | \n
| batch_size | \n16 | \n
| AdamW | \nweight_decay=2.43e-5 | \n
| first epoch with OneCycleLR | \nmax_lr=1.0e-4, div_factor=10 | \n
| second epoch with CosineAnnealing | \nlr=1.0e-5, eta_min=1.0e-6 | \n
I had similar results. Any kind of resampling of the data caused worse performance.
", "rawMarkdown": "I had similar results. Any kind of resampling of the data caused worse performance.", "votes": null }, { "id": "1091501", "postDate": "11/26/2020 04:25:44", "content": "I agree. Undersampling doesn't guarantee better result.
", "rawMarkdown": "I agree. Undersampling doesn't guarantee better result.", "votes": null }, { "id": "1091597", "postDate": "11/26/2020 06:07:30", "content": "I have also try oversample which also don't work too well. I guess the reason could be that to learn those outlier, you will impact the learning on other examples too much. Or, I simply didn't train the entire data set. Perhaps, if you train the entire epoch, it will be OK.
", "rawMarkdown": "I have also try oversample which also don't work too well. I guess the reason could be that to learn those outlier, you will impact the learning on other examples too much. Or, I simply didn't train the entire data set. Perhaps, if you train the entire epoch, it will be OK.", "votes": null }, { "id": "1091599", "postDate": "11/26/2020 06:09:23", "content": "I think batch size also matter a lot. I was using batch_size = 16 and couldn't get pass the 23 threshold. Switching to 64 can bring you lower loss when train more data.
", "rawMarkdown": "I think batch size also matter a lot. I was using batch_size = 16 and couldn't get pass the 23 threshold. Switching to 64 can bring you lower loss when train more data.", "votes": null }, { "id": "1091624", "postDate": "11/26/2020 06:25:33", "content": "I inspected the high loss samples quite a bit and it didn't seem like there was a great deal of systemic problems. So training on one outlier didn't seem to be particularly likely to reduce loss on other outliers. I tried an augmentation where I shifted a single channel out of the full history or surrounding vehicles by a couple pixels in a random direction and often found that that could shift a prediction from 1k+ loss down to under 50
", "rawMarkdown": "I inspected the high loss samples quite a bit and it didn't seem like there was a great deal of systemic problems. So training on one outlier didn't seem to be particularly likely to reduce loss on other outliers. I tried an augmentation where I shifted a single channel out of the full history or surrounding vehicles by a couple pixels in a random direction and often found that that could shift a prediction from 1k+ loss down to under 50", "votes": null }, { "id": "1091626", "postDate": "11/26/2020 06:26:54", "content": "It was interesting to see the system so fragile to this augmentation, I presume the error of the perception system could cause these small single pixel shifts so making it robust to this would probably be important.
", "rawMarkdown": "It was interesting to see the system so fragile to this augmentation, I presume the error of the perception system could cause these small single pixel shifts so making it robust to this would probably be important.", "votes": null } ], "comments": [ { "id": 1091493, "author_name": "ryches", "author_url": "", "post_date": "11/26/2020 04:19:14", "content": "I had similar results. Any kind of resampling of the data caused worse performance.
", "votes": null, "replies": [ { "id": 1091501, "author_name": "sunghyunjun", "author_url": "", "post_date": "11/26/2020 04:25:44", "content": "I agree. Undersampling doesn't guarantee better result.
", "votes": null, "replies": [] }, { "id": 1091597, "author_name": "louis925", "author_url": "", "post_date": "11/26/2020 06:07:30", "content": "I have also try oversample which also don't work too well. I guess the reason could be that to learn those outlier, you will impact the learning on other examples too much. Or, I simply didn't train the entire data set. Perhaps, if you train the entire epoch, it will be OK.
", "votes": null, "replies": [] }, { "id": 1091624, "author_name": "ryches", "author_url": "", "post_date": "11/26/2020 06:25:33", "content": "I inspected the high loss samples quite a bit and it didn't seem like there was a great deal of systemic problems. So training on one outlier didn't seem to be particularly likely to reduce loss on other outliers. I tried an augmentation where I shifted a single channel out of the full history or surrounding vehicles by a couple pixels in a random direction and often found that that could shift a prediction from 1k+ loss down to under 50
", "votes": null, "replies": [] }, { "id": 1091626, "author_name": "ryches", "author_url": "", "post_date": "11/26/2020 06:26:54", "content": "It was interesting to see the system so fragile to this augmentation, I presume the error of the perception system could cause these small single pixel shifts so making it robust to this would probably be important.
", "votes": null, "replies": [] } ] }, { "id": 1091599, "author_name": "louis925", "author_url": "", "post_date": "11/26/2020 06:09:23", "content": "I think batch size also matter a lot. I was using batch_size = 16 and couldn't get pass the 23 threshold. Switching to 64 can bring you lower loss when train more data.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1091489": "After I read discussion of @ryches, Improving convergence and sample efficiency\nhttps://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/183814\n\nI tried to calculate cumulative distance of target_positions.\n\ntrain.zarr with min_frame_future=10\nlength of AgentDataset: 17,003,687\n\nThis plot is sample distribution about non-zero-end of target_positions. number of samples are 850,184 (1/20 of AgentDatast). Most sample has 50 target_positions data.\n\n\n\nAnd histogram of cumulative distance is as follows. (number of samples are 17,003,687)\n\n\nAnd this is cumulative graph.\n\n\nAbout 60% of data have less than 3 of cumulative distance.\n\n|distance|percentile|\n|---|---|\n|0 ~ 1|0.3108957002248925|\n|1 ~ 2|0.15160976917937763|\n|2 ~ 3|0.08709361738164917|\n|3 ~ 4|0.057570243617852124|\n|4 ~ 5|0.030621018508934505|\n|5 ~ 6|0.01739799855090196|\n|6 ~ 7|0.011909304338825422|\n|7 ~ 8|0.009271287156662589|\n|8 ~ 9|0.007785491140741341|\n|9 ~ 10|0.006836990580862512|\n\n\nSo i undersampled and made mask of data which of cumulative distance is less than 10 randomly by the inverse of percentile.\n\n|distance|ratio|\n|---|---|\n|0 ~ 1|0.021991267733573804|\n|1 ~ 2|0.04509597645237031|\n|2 ~ 3|0.0785016260250442|\n|3 ~ 4|0.1187591045514077|\n|4 ~ 5|0.22327769988668522|\n|5 ~ 6|0.3929756955007945|\n|6 ~ 7|0.5740881571540075|\n|7 ~ 8|0.7374370424875981|\n|8 ~ 9|0.8781707482890396|\n|9 ~ 10|1.0|\n\n\nThis is histogram of undersampled data.\n\n\nThe total length of undersampled data is 6,415,723. It is 37.7% of original data.\n\nAnd I trained with this 6M data on resnet18 for 4 epochs. \ntrain loss is average of 2k iter(2k x 16 = 32k samples)\n\n|epoch|tr_loss|val_loss|LB|\n| --- | --- | --- | --- |\n|1|25.582|23.814|22.676|\n|2|20.947|21.242|20.529|\n|3|18.935|21.900|20.585|\n|4|17.359|21.878|20.484|\n\n|parameters|values|\n|---|---|\n|raster_size|267, 267|\n|pixel_size|0.5, 0.5|\n|history_num_frames|1|\n|history_delta_time|1|\n|batch_size|16|\n|AdamW|lr=1.0e-4, weight_decay=0.01|\n|CosineAnnealingWarmRestarts|T_0=80k, T_mult=1, eta_min=1.0e-5|\n\n\nI think I may have trained in wrong way. I used quite large weight_decay. Anyway training one epoch took ~17 hours on GCE with vCPU:16, GPU: T4 and ~13 hours with vCPU:24, GPU: T4.\n\nI guess random undersampling by cumulative distance is not good idea. The information losses may be quite significant. Samples of small cumulative distance may have various trajectories. I think reducing overlapped sample is more appropriate. I wanted to do more experiment but I couldn't because of lack of time.\n\nAnd my final submission is just resnet18 with train.zarr, no undersampling. I also tried mobilenetV2 but resnet18 is better.\n\n|epoch|private LB|public LB\n|---|---|---|\n|1|19.101|19.886|\n|2|18.185|18.675|\n\n|parameters|values|\n|---|---|\n|raster_size|267, 267|\n|pixel_size|0.5, 0.5|\n|history_num_frames|1|\n|history_delta_time|1|\n|batch_size|16|\n|AdamW|weight_decay=2.43e-5|\n|first epoch with OneCycleLR|max_lr=1.0e-4, div_factor=10|\n|second epoch with CosineAnnealing|lr=1.0e-5, eta_min=1.0e-6|", "1091493": "I had similar results. Any kind of resampling of the data caused worse performance.", "1091501": "I agree. Undersampling doesn't guarantee better result.", "1091597": "I have also try oversample which also don't work too well. I guess the reason could be that to learn those outlier, you will impact the learning on other examples too much. Or, I simply didn't train the entire data set. Perhaps, if you train the entire epoch, it will be OK.", "1091599": "I think batch size also matter a lot. I was using batch_size = 16 and couldn't get pass the 23 threshold. Switching to 64 can bring you lower loss when train more data.", "1091624": "I inspected the high loss samples quite a bit and it didn't seem like there was a great deal of systemic problems. So training on one outlier didn't seem to be particularly likely to reduce loss on other outliers. I tried an augmentation where I shifted a single channel out of the full history or surrounding vehicles by a couple pixels in a random direction and often found that that could shift a prediction from 1k+ loss down to under 50", "1091626": "It was interesting to see the system so fragile to this augmentation, I presume the error of the perception system could cause these small single pixel shifts so making it robust to this would probably be important." }, "source": "meta" }