![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fb8df48f983e647e2e913a98f13f7ca6f%2FSelection_209.png?generation=1604820518323524&alt=media\")
first, why is there multi-mode?
\nso is three mode enough?
\nI see that the public notebook are setting K=3, but K can be greater than 3. since \"regression loss\" seek for least square you will end up in a solution that is equal distance to each of the non-matching K cluster. If you have a matching K cluster, your regression loss will be low.
\nrelated: top-K ranking loss, classifiation with subclass, trajectory clustering
\ne.g. : https://arxiv.org/abs/1909.05235 - subclass softmax
\n'Multiple Centers Now, we assume that each class has K centers. Then, the similarity between the example x ….'
first, why is there multi-mode?
\nso is three mode enough?
\nI see that the public notebook are setting K=3, but K can be greater than 3. since \"regression loss\" seek for least square you will end up in a solution that is equal distance to each of the non-matching K cluster. If you have a matching K cluster, your regression loss will be low.
\nrelated: top-K ranking loss, classifiation with subclass, trajectory clustering
\ne.g. : https://arxiv.org/abs/1909.05235 - subclass softmax
\n'Multiple Centers Now, we assume that each class has K centers. Then, the similarity between the example x ….'
this is what you get if you plot the target on world coords.
\nthere could be some experiment design flaw here. The test data should be of a completely different road scene from the training dataset to avoid bias.
\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fe05ba536a3afc41526f9167fadf877b9%2FSelection_208.png?generation=1604820537854691&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fd03cabdbc4e906b59cfc51e63c093365%2FSelection_207.png?generation=1604820558413844&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F8666565343532f56a65a79a324bf676c%2FSelection_218.png?generation=1604822508676863&alt=media\")
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F7d827e2b859272da2fef33dd56529976%2FSelection_219.png?generation=1604822525999439&alt=media\")
\ntypo error : there is no red …
Do you have a hint how to implement additional inputs? I guess you are talking about the model architecture itself. Do you have an example how to add non image data to a pretained resnet34?
", "rawMarkdown": "Do you have a hint how to implement additional inputs? I guess you are talking about the model architecture itself. Do you have an example how to add non image data to a pretained resnet34?" }, { "id": 1072888, "postDate": "2020-11-08T20:14:48.760Z", "content": "In the forward function, you can extract features from the backbone, then flatten it and concatenate with any additional features you want using torch.cat(). After that, you will have to adjust the number of in_features in your dense layer (just increase it by the length of your non-image input).
", "rawMarkdown": "In the forward function, you can extract features from the backbone, then flatten it and concatenate with any additional features you want using torch.cat(). After that, you will have to adjust the number of in_features in your dense layer (just increase it by the length of your non-image input). ", "votes": 2 }, { "id": 1072896, "postDate": "2020-11-08T20:43:51.733Z", "content": "Thanks for the quick reply @szacho
\nDo you have a code example? I thought about something like this:
\n1# forward
\n def forward(self, x, non-image-data):\n x = self.backbone.conv1(x)\n x = self.backbone.bn1(x)\n x = self.backbone.relu(x)\n x = self.backbone.maxpool(x)\n\n x = self.backbone.layer1(x)\n x = self.backbone.layer2(x)\n x = self.backbone.layer3(x)\n x = self.backbone.layer4(x)\n\n x = self.backbone.avgpool(x)\n x = torch.flatten(x, 1)\n y = non-image-data\n x = torch.cat((x, y), dim=1)\n\n x = self.head(x)\n x = self.logit(x)\n [ ...... ]\n2# init
\ndef __init__(self, cfg: Dict, num_modes=3):\n [ ...... ]\n # You can add more layers here.\n self.head = nn.Sequential(\n # nn.Dropout(0.2),\n nn.Linear(in_features=backbone_out_features+len(non-image-data), out_features=4096),\n )\n self.num_preds = num_targets * num_modes\n self.num_modes = num_modes\n\n self.logit = nn.Linear(4096, out_features=self.num_preds + num_modes)\nWhat do you think?
", "rawMarkdown": "Thanks for the quick reply @szacho \n\nDo you have a code example? I thought about something like this:\n\n1# forward\n\n```\n def forward(self, x, non-image-data):\n x = self.backbone.conv1(x)\n x = self.backbone.bn1(x)\n x = self.backbone.relu(x)\n x = self.backbone.maxpool(x)\n\n x = self.backbone.layer1(x)\n x = self.backbone.layer2(x)\n x = self.backbone.layer3(x)\n x = self.backbone.layer4(x)\n\n x = self.backbone.avgpool(x)\n x = torch.flatten(x, 1)\n y = non-image-data\n x = torch.cat((x, y), dim=1)\n\n x = self.head(x)\n x = self.logit(x)\n [ ...... ]\n```\n\n2# init\n```\ndef __init__(self, cfg: Dict, num_modes=3):\n [ ...... ]\n # You can add more layers here.\n self.head = nn.Sequential(\n # nn.Dropout(0.2),\n nn.Linear(in_features=backbone_out_features+len(non-image-data), out_features=4096),\n )\n self.num_preds = num_targets * num_modes\n self.num_modes = num_modes\n\n self.logit = nn.Linear(4096, out_features=self.num_preds + num_modes)\n```\n\nWhat do you think?", "votes": 1 }, { "id": 1073382, "postDate": "2020-11-09T13:25:13.987Z", "content": "@benbla
\nSeems fine, I implemented it in a very similar if not the same way. Good job!
here is the loss breakdown by label:
\nfor non-chopped data:\nlocal cv loss: 9.346676 (lb = 15.034)\nbreakdown: \nPERCEPTION_LABEL_CAR 9.259042 \nPERCEPTION_LABEL_CYCLIST 17.221859 \nPERCEPTION_LABEL_PEDESTRIAN 4.827463\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Ffc4872f9d3751a914d92103db8769fc0%2FSelection_063.png?generation=1604974926304836&alt=media\")
i wonder are these ground truth senor errors?
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fd29aed8a457cdcd270ca91cd9562457b%2FSelection_067.png?generation=1604975711807483&alt=media\")
Hi, @szacho. Did you feed \"history_positions\" and other data into the model? During training, is \"history_availabilities \" useful to imply the available agent in history frames?
", "rawMarkdown": "Hi, @szacho. Did you feed \"history_positions\" and other data into the model? During training, is \"history_availabilities \" useful to imply the available agent in history frames?" }, { "id": 1074375, "postDate": "2020-11-10T15:29:06.357Z", "content": "@spicychicken38
\nIntegrating \"history_positions\" leads to a huge overfit and a big derivation between loss and lb score to me.
\n\n@spicychicken38
\n
\n Integrating \"history_positions\" leads to a huge overfit and a big derivation between loss and lb score to me.
Hi, @benbla. I don't think this huge deviation is caused by overfitting. According to my experiment, there is a data mismatch that is caused by the differences between train.zarr and validation.zarr(or test.zarr), which can lead to this huge deviation. Because the \"history_availabilities\" in train.zarr are always 1, which means agents exist in every frame in the training sample that they belong to. However, this is totally different in the chopped validation set or the testing set. So I think this might be the main reason why we have such a huge deviation when we are adding \"history_positions\".
", "rawMarkdown": "> @spicychicken38 \n> Integrating \"history_positions\" leads to a huge overfit and a big derivation between loss and lb score to me.\n\nHi, @benbla. I don't think this huge deviation is caused by overfitting. According to my experiment, there is a data mismatch that is caused by the differences between train.zarr and validation.zarr(or test.zarr), which can lead to this huge deviation. Because the \"history_availabilities\" in train.zarr are always 1, which means agents exist in every frame in the training sample that they belong to. However, this is totally different in the chopped validation set or the testing set. So I think this might be the main reason why we have such a huge deviation when we are adding \"history_positions\".", "votes": 1 } ] }, { "id": 1074389, "postDate": "2020-11-10T15:44:36.343Z", "content": "@benbla In my case, there is also a huge deviation between validation score and training loss. But I can not certainly say that it is overfitting. I also think that it is also possible that the differences of \"history_availabilities\" between the training set and the chopped validation set to cause this huge deviation. But I can not verify it. As mention here, @frankpanxj also meet the same problem.
", "rawMarkdown": "@benbla In my case, there is also a huge deviation between validation score and training loss. But I can not certainly say that it is overfitting. I also think that it is also possible that the differences of \"history_availabilities\" between the training set and the chopped validation set to cause this huge deviation. But I can not verify it. As mention [here](https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/185762), @frankpanxj also meet the same problem." }, { "id": 1074393, "postDate": "2020-11-10T15:50:07.757Z", "content": "@spicychicken38 No, I feed the agent state vector as described in this article: https://arxiv.org/abs/1809.10732
", "rawMarkdown": "@spicychicken38 No, I feed the agent state vector as described in this article: https://arxiv.org/abs/1809.10732" }, { "id": 1074406, "postDate": "2020-11-10T16:06:30.183Z", "content": "@szacho But we don't have the velocity, acceleration mentioned in the article. How you implement this?
", "rawMarkdown": "@szacho But we don't have the velocity, acceleration mentioned in the article. How you implement this?" }, { "id": 1074459, "postDate": "2020-11-10T17:07:33.470Z", "content": "@spicychicken38 Velocity is available in zarr data, but you can get everything from data returned by AgentDataset - you have history_positions and history_yaws in time. Features like velocity, acceleration, and yaw change rate can be calculated using basic physics knowledge.
", "rawMarkdown": "@spicychicken38 Velocity is available in zarr data, but you can get everything from data returned by AgentDataset - you have history_positions and history_yaws in time. Features like velocity, acceleration, and yaw change rate can be calculated using basic physics knowledge.", "votes": 1 }, { "id": 1074777, "postDate": "2020-11-11T03:38:06.713Z", "content": "@szacho Thanks for your views. I will try to do some experiments on that later!
", "rawMarkdown": "@szacho Thanks for your views. I will try to do some experiments on that later!" }, { "id": 1075234, "postDate": "2020-11-11T14:09:26.740Z", "content": "@szacho As you mentioned we can get the velocity from AgentDataset. But what about the agent's label_probabilities? seems agent_samples.py won't return them. If possible, can suggest some method to get them along with the \"data\" object?
", "rawMarkdown": "@szacho As you mentioned we can get the velocity from AgentDataset. But what about the agent's **label_probabilities**? seems **agent_samples.py** won't return them. If possible, can suggest some method to get them along with the \"data\" object?\n" }, { "id": 1075254, "postDate": "2020-11-11T14:19:33.403Z", "content": "for the file:
\n/l5kit/sampling/agent_sampling.py
def generate_agent_sample(...):\n ...\n \"extent\": agent_extent_m,\n #\"agent\": selected_agent, # <add here>\n \"label_probabilities\": selected_agent[\"label_probabilities\"], # <add here>\nfor the file:
\n/l5kit/dataset/ego.py
class EgoDataset(Dataset):\n...\ndef get_frame(...):\n\n \"extent\": data[\"extent\"],\n #\"agent\": data[\"agent\"], #<add here>\n \"label_probabilities\": data[\"label_probabilities\"], #<add here>\n }\nHi @hengck23 , Thanks for your insights. How to add this custom l5kit library to one's kaggle notebook? Instead I thought of calculating velocities in rudimentary way like this (I guess we have to calculate yaw rate and acceleration ourselves anyway):
\n if data['history_availabilities'].to(device)shape[1] > 1: \n velocities = torch.sqrt(torch.sum((pose_data[:,1:,:] - pose_data[:,0:-1, :]) ** 2, 2)) / dt\n avg_velocity = torch.mean(velocities, dim=1).view(bs,1)\n #acceleration = torch.abs(velocities[:, 1:] - velocities[:, 0:-1])\n avg_acceleration = torch.mean(torch.abs(velocities[:, 1:] - velocities[:, 0:-1]) / dt, dim=1).view(bs,1)\n yaw_rate = torch.mean(torch.abs(yaw_data[:, 1:] - yaw_data[:, 0:-1]) / dt, dim=1).view(bs,1)\n\n avg_velocity = avg_velocity / MAX_VELOCITY # 20 m/s\n avg_acceleration = avg_acceleration / MAX_ACCELERATION # 2m/s^2\n yaw_rate = yaw_rate / MAX_YAW_RATE # 45 deg/s\n\n else:\n avg_velocity = 0.0; avg_acceleration = 0.0; yaw_rate = 0.0; \nIs there anything I'm missing here? Any any exception handling required?
", "rawMarkdown": "Hi @hengck23 , Thanks for your insights. How to add this custom l5kit library to one's kaggle notebook? Instead I thought of calculating velocities in rudimentary way like this (I guess we have to calculate yaw rate and acceleration ourselves anyway):\n\n```\n if data['history_availabilities'].to(device)shape[1] > 1: \n velocities = torch.sqrt(torch.sum((pose_data[:,1:,:] - pose_data[:,0:-1, :]) ** 2, 2)) / dt\n avg_velocity = torch.mean(velocities, dim=1).view(bs,1)\n #acceleration = torch.abs(velocities[:, 1:] - velocities[:, 0:-1])\n avg_acceleration = torch.mean(torch.abs(velocities[:, 1:] - velocities[:, 0:-1]) / dt, dim=1).view(bs,1)\n yaw_rate = torch.mean(torch.abs(yaw_data[:, 1:] - yaw_data[:, 0:-1]) / dt, dim=1).view(bs,1)\n \n avg_velocity = avg_velocity / MAX_VELOCITY # 20 m/s\n avg_acceleration = avg_acceleration / MAX_ACCELERATION # 2m/s^2\n yaw_rate = yaw_rate / MAX_YAW_RATE # 45 deg/s\n\n else:\n avg_velocity = 0.0; avg_acceleration = 0.0; yaw_rate = 0.0; \n```\n\nIs there anything I'm missing here? Any any exception handling required?", "votes": 1 }, { "id": 1082506, "postDate": "2020-11-18T00:59:28.477Z", "content": "Adding these state inputs leads to large overfitting for me, anyone else seeing similar results ?
", "rawMarkdown": "Adding these state inputs leads to large overfitting for me, anyone else seeing similar results ?" } ] }, { "id": 1075328, "postDate": "2020-11-11T15:35:08.927Z", "content": "@hengck23 Oh you are using l5kit/ as part of your code. I am using it as a library. Sure I will do require changes. Thanks for your quick response.
", "rawMarkdown": "@hengck23 Oh you are using l5kit/ as part of your code. I am using it as a library. Sure I will do require changes. Thanks for your quick response." }, { "id": 1078444, "postDate": "2020-11-14T19:50:00.310Z", "content": "Speaking of velocities, agent_sampling.py and ego.py have been updated. You can now get velocity (and speed) through the AgentDataset() class.
Where can I find the updates ?
", "rawMarkdown": "Where can I find the updates ?" }, { "id": 1078462, "postDate": "2020-11-14T20:17:03.597Z", "content": "l5kit github repository, here
", "rawMarkdown": "l5kit github repository, [here](https://github.com/lyft/l5kit)", "votes": 1 } ] }, { "id": 1076901, "postDate": "2020-11-13T02:52:24.260Z", "content": "surprise, surprise, surprise …
\nI put the conclusion first and then the observations.
conclusion:
\nwe probably want to detect, predict, post or pre-processing them separately. they are easy to identify via past histories.
\nobservation:
\nsee below simulation:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fcadc4dbd60a83b50138a2a7daaf45c3b%2FSelection_149.png?generation=1605235825678435&alt=media\")
hence I trained a mode-7 predictor from my current mode-3 predictor. i just freeze the backbone weights and change the head. it is very fast to train. on chopped validation data, kaggle metric reduces from 15+ to 11+ within hours.
\nbut when I inspect the results, I was surprised
", "rawMarkdown": "surprise, surprise, surprise ...\nI put the conclusion first and then the observations.\n\nconclusion:\n- there are two types of trajectories: \n1. parametric curves, that obey kinematics. you can get very accurate results via curve fitting (and maybe using lstm of past history points)\n2. non-parametric and irregular curves\n\nwe probably want to detect, predict, post or pre-processing them separately. they are easy to identify via past histories.\n\n--- \n\nobservation:\n- I study the metric and note that the effect of l2 loss is very much greater than the confidence values (because world coords is in meters?). i.e. in a prediction of mode M, as long as one of them has low l2 loss (even if the confidence is low) the kaggle metric will be low.\n\nsee below simulation:\n \n\n\nhence I trained a mode-7 predictor from my current mode-3 predictor. i just freeze the backbone weights and change the head. it is very fast to train. on chopped validation data, kaggle metric reduces from 15+ to 11+ within hours.\n\nbut when I inspect the results, I was surprised", "votes": 4, "replies": [ { "id": 1076902, "postDate": "2020-11-13T02:53:07.980Z", "content": "![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F28b6cea52808ab6ba5a725609a2e1da3%2FSelection_144.png?generation=1605235977467342&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F3057c4f9087c723954509a2d25e0a2ce%2FSelection_157.png?generation=1605261672198016&alt=media\")
\nNote: if there are shakeup, it can be due to just a few outliers
so, what are the cases, mode-7 performs better?
\ntruth: disconnected black dots
\n(history is not shown)
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F490491b0c3190dc69ef6a15d00b88b86%2FSelection_145.png?generation=1605236075073834&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F26cb2e998011236e99e4b3de568a11f2%2FSelection_146.png?generation=1605236092430499&alt=media\")
you can see that we can gain a lot if we can estimate the \"straight trajectory\" more accurate … which seems to be an easy task
", "rawMarkdown": "so, what are the cases, mode-7 performs better?\ntruth: disconnected black dots\n(history is not shown)\n\n\n\n\n\n\nyou can see that we can gain a lot if we can estimate the \"straight trajectory\" more accurate ... which seems to be an easy task ", "votes": 1 }, { "id": 1077784, "postDate": "2020-11-13T23:19:28Z", "content": "Nice catch but the competition metric is biased towards long trajectories. Sigma is fixed to 1 which is not real! It should increase as the predictions goes further in time.
", "rawMarkdown": "Nice catch but the competition metric is biased towards long trajectories. Sigma is fixed to 1 which is not real! It should increase as the predictions goes further in time.\n" }, { "id": 1078686, "postDate": "2020-11-15T07:12:14.137Z", "content": "as an example
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F004de876653e436c02f85469f7911ca4%2FSelection_185.png?generation=1605424264162784&alt=media\")
you can add other labels for specific post-processing, e.g. di-biasing (the kaggle multiplier trick in regression problem, which is essentially manually adjusting the std of error)
", "rawMarkdown": "as an example\n\n\nyou can add other labels for specific post-processing, e.g. di-biasing (the kaggle multiplier trick in regression problem, which is essentially manually adjusting the std of error)", "votes": 1 }, { "id": 1078739, "postDate": "2020-11-15T08:49:06.343Z", "content": "How do you set the is_parametric flag? It's not clear to me. Do you use a custom curve fitting function?
e.g. for each train sample, you have x =[history, future]
\nthen you devise a curve function, predict_future = f(history). you will have to design this by hand, making some assumptions. you can google for some kinematics model.
\nnow each train sample will have x =[history, f(history), future].
\ncompute error of f(history), e.g. l2 loss (f(history), future). now if this error is small, you can label is as True (i.e. you can recover future values from f(history))
", "rawMarkdown": "e.g. for each train sample, you have x =[history, future]\n\nthen you devise a curve function, predict\\_future = f(history). you will have to design this by hand, making some assumptions. you can google for some kinematics model.\n\nnow each train sample will have x =[history, f(history), future].\n\ncompute error of f(history), e.g. l2 loss (f(history), future). now if this error is small, you can label is as True (i.e. you can recover future values from f(history))", "votes": 1 }, { "id": 1078778, "postDate": "2020-11-15T09:44:17.033Z", "content": "Thanks @hengck23, this helps :)
", "rawMarkdown": "Thanks @hengck23, this helps :)" } ] }, { "id": 1072969, "postDate": "2020-11-09T00:44:23.703Z", "content": "there is much more information in the zarr file (instead of the AgentDataset)
\ne.g. you get velocity and label probability
\nAGENT_DTYPE = [\n (\"centroid\", np.float64, (2,)),\n (\"extent\", np.float32, (3,)),\n (\"yaw\", np.float32),\n (\"velocity\", np.float32, (2,)),\n (\"track_id\", np.uint64),\n (\"label_probabilities\", np.float32, (len(LABELS),)),\n]\nhere is an example of reading the raw zarr file. note that you can recover the whole track!
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fa3d90739319a1ec4bea51004002f61d3%2FSelection_230.png?generation=1604882661363275&alt=media\")
you can submit the top-1, top-2 and top-3 results to prob the correctness of your prediction. then it may be possible to recalibrate or adjust the temperate of the confidence.
\nsome of the predictions are clearly only single-mode, e.g. moving in a straight lane or not moving. in that case a single high confidence trajectory prediction may be better?
", "rawMarkdown": "you can submit the top-1, top-2 and top-3 results to prob the correctness of your prediction. then it may be possible to recalibrate or adjust the temperate of the confidence.\n\nsome of the predictions are clearly only single-mode, e.g. moving in a straight lane or not moving. in that case a single high confidence trajectory prediction may be better?", "votes": 1 }, { "id": 1067368, "postDate": "2020-11-02T13:52:56.493Z", "content": "self-supervised?
\nfor the test data, say we ask for number of history frame = 30.
\nwe use t-30 to t-20 as input and train an embedding to predict t-20 to t.
would it work?
\nhow about :
\nfuture = predict (history) and then history1 = predict(future) … like to and from lanuage translation …
\nwould that be a kind of augmentation (i.e. history -->history1)?
MANTRA: Memory Augmented Networks for Multiple Trajectory Prediction
\nhttps://openaccess.thecvf.com/content_CVPR_2020/papers/Marchetti_MANTRA_Memory_Augmented_Networks_for_Multiple_Trajectory_Prediction_CVPR_2020_paper.pdf
![\"\"](\"https://images.deepai.org/converted-papers/2006.03340/x3.png\")
In the original paper of the base model that lyft provided to us it seems to show that 3 is empirically the most performant number of modes https://arxiv.org/pdf/1809.10732.pdf
\n1, 2, and 4 modes do worse. This has been reproduced by a few other papers as well.
", "rawMarkdown": "In the original paper of the base model that lyft provided to us it seems to show that 3 is empirically the most performant number of modes https://arxiv.org/pdf/1809.10732.pdf\n\n1, 2, and 4 modes do worse. This has been reproduced by a few other papers as well. ", "votes": 1, "replies": [ { "id": 1064288, "postDate": "2020-10-29T23:49:07.947Z", "content": "thanks for the paper.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fc0e2508c364917765fd2d65e8436fcc8%2FSelection_030.png?generation=1604014930131979&alt=media\")
it seems that it is due to the nature road junction. in that case, we can just train for going straight, turn left and turn right. i need to check the distribution/frequency of turns of junctions. thanks!
", "rawMarkdown": "thanks for the paper.\n\n\n\nit seems that it is due to the nature road junction. in that case, we can just train for going straight, turn left and turn right. i need to check the distribution/frequency of turns of junctions. thanks!", "votes": 1 }, { "id": 1065049, "postDate": "2020-10-30T19:38:03.467Z", "content": "Intresting, thank you for the paper.
\nThe relations are greatly explained and visualized.
", "rawMarkdown": "Intresting, thank you for the paper.\n\nThe relations are greatly explained and visualized." }, { "id": 1065578, "postDate": "2020-10-31T13:13:12.893Z", "content": "Does anyone know if the mentioned MTP loss is somewhere available as public code?
", "rawMarkdown": "Does anyone know if the mentioned MTP loss is somewhere available as public code?" }, { "id": 1065608, "postDate": "2020-10-31T13:58:34.417Z", "content": "Here, https://github.com/nutonomy/nuscenes-devkit/blob/master/python-sdk/nuscenes/prediction/models/mtp.py
", "rawMarkdown": "Here, https://github.com/nutonomy/nuscenes-devkit/blob/master/python-sdk/nuscenes/prediction/models/mtp.py ", "votes": 1 }, { "id": 1065732, "postDate": "2020-10-31T16:56:38.637Z", "content": "Thank you :)
", "rawMarkdown": "Thank you :)" } ] }, { "id": 1076954, "postDate": "2020-11-13T04:28:13.600Z", "content": "to disable image rasterizer and just do prediction using history information:
\n dm = LocalDataManager()\n rasterizer = None #build_rasterizer(cfg, dm)\n\n zarr = ChunkedDataset( dm.require('scenes/train.zarr')).open()\n dataset = MyAgentDataset(cfg, zarr, rasterizer) \n\n\n\nin ego.py: (around line 99)\n\nclass EgoDataset(Dataset):\n\ndef get_frame(self, scene_index ...):\n\n # 0,1,C -> C,0,1\n # <hck>\n if data[\"image\"] is not None:\n image = data[\"image\"].transpose(2, 0, 1)\n else:\n image = 0 #dummy\ni am thinking of using:
\nso i need to create 2 dataset, one with rasterizer and one without (to speed up cpu data loading).
", "rawMarkdown": "to disable image rasterizer and just do prediction using history information:\n\n```\n dm = LocalDataManager()\n rasterizer = None #build_rasterizer(cfg, dm)\n\n zarr = ChunkedDataset( dm.require('scenes/train.zarr')).open()\n dataset = MyAgentDataset(cfg, zarr, rasterizer) \n\n\n\nin ego.py: (around line 99)\n\nclass EgoDataset(Dataset):\n\ndef get_frame(self, scene_index ...):\n\n # 0,1,C -> C,0,1\n #I reported this as a bug long time ago on github and I guess it is now fixed
", "rawMarkdown": "I reported this as a bug long time ago on github and I guess it is now fixed" }, { "id": 1077851, "postDate": "2020-11-14T02:13:08.017Z", "content": "You mean that by doing disabling the rasterizer, we can use history information for improvement regardless of the differences of history_availabilities between train.zarr and test.zarr?
", "rawMarkdown": "You mean that by doing disabling the rasterizer, we can use history information for improvement regardless of the differences of history_availabilities between train.zarr and test.zarr?" }, { "id": 1077896, "postDate": "2020-11-14T04:22:54.570Z", "content": "rendering long history frames is very slow. hence I want to experiment using history positions only without image. (or few history images)
\nthis is non- fixed length sequence learning, e.g
\ninput:\nx(t), x(t-1),x(t-2) ... <end>\nyou can treat it as LSTM seq with an \"end\" token. alternatively you can pad to equal length.
", "rawMarkdown": "rendering long history frames is very slow. hence I want to experiment using history positions only without image. (or few history images)\n\nthis is non- fixed length sequence learning, e.g\n\n```\n\ninput:\nx(t), x(t-1),x(t-2) ...You can use StubRasterizer to render only blank images
", "rawMarkdown": "You can use [StubRasterizer](https://github.com/lyft/l5kit/blob/master/l5kit/l5kit/rasterization/stub_rasterizer.py) to render only blank images" } ] }, { "id": 1065510, "postDate": "2020-10-31T11:17:03.593Z", "content": "i wonder did anyone plot and visualise the predicted trajectory?
\ni don't think it would be a perfect parametric curve. it might look a little \"wiggling\" and not smooth. So there are be some post-processing required, or better still, should we predict the parameters of the curve of trajectory instead?
\nhistory trajectory can also be represented as a parametric curve
", "rawMarkdown": "i wonder did anyone plot and visualise the predicted trajectory?\n\ni don't think it would be a perfect parametric curve. it might look a little \"wiggling\" and not smooth. So there are be some post-processing required, or better still, should we predict the parameters of the curve of trajectory instead?\n\nhistory trajectory can also be represented as a parametric curve", "votes": 2, "replies": [ { "id": 1065524, "postDate": "2020-10-31T11:33:06.847Z", "content": "There has been some discussion about maybe using a Kalman filter to smooth out predictions, although at the moment my predicted trajectories tend to be pretty smooth: ![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F314368%2Fe89a6a4e6fe1239bf3aa224d2fe83537%2Finbox_314368_2ae6a5e2632e97085e567e4951c51536_mask.png?generation=1604143906678498&alt=media\")
The targets often have some amount of noise in them, so being smoother than the targets might not give that much improvement
", "rawMarkdown": "There has been some discussion about maybe using a Kalman filter to smooth out predictions, although at the moment my predicted trajectories tend to be pretty smooth: \n\nThe targets often have some amount of noise in them, so being smoother than the targets might not give that much improvement", "votes": 5 }, { "id": 1066394, "postDate": "2020-11-01T17:54:15.257Z", "content": "seems that predicting the end-point correctly is very important.
", "rawMarkdown": "seems that predicting the end-point correctly is very important. " } ] }, { "id": 1077979, "postDate": "2020-11-14T07:13:11.497Z", "content": "![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F3093b436d5ee8a87c7594c04ed1fb923%2FSelection_164.png?generation=1605337987662608&alt=media\")
I believe, that picture is rather misleading with different counts in each bin. You may subsample the 50 avails part to match the count for the others, or plot e.g. violin graphs to get a better view on the loss distribution.
", "rawMarkdown": "I believe, that picture is rather misleading with different counts in each bin. You may subsample the 50 avails part to match the count for the others, or plot e.g. violin graphs to get a better view on the loss distribution." }, { "id": 1078093, "postDate": "2020-11-14T10:50:16.457Z", "content": "![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F84f497214505a4fcaaac8589277f494c%2FSelection_165.png?generation=1605350990328844&alt=media\")
count of samples vs samples' target availability
", "rawMarkdown": "\n\ncount of samples vs samples' target availability", "votes": 1 }, { "id": 1078110, "postDate": "2020-11-14T11:31:02.323Z", "content": "I know, that's why your first Image gives a misleading view on the loss distribution. It looks like the loss distribution for 50 avails is much broader than for example 49 avails, but in reality they are very similar.
", "rawMarkdown": "I know, that's why your first Image gives a misleading view on the loss distribution. It looks like the loss distribution for 50 avails is much broader than for example 49 avails, but in reality they are very similar." }, { "id": 1078120, "postDate": "2020-11-14T11:43:59.517Z", "content": "i think the loss for e.g. availability=10 would be small because only \"10 nearby points are counted\" in the evaluation metric.
\nin the case where availability=50, errors can get accumulated for many predicted points. hence the loss will be higher.
", "rawMarkdown": "i think the loss for e.g. availability=10 would be small because only \"10 nearby points are counted\" in the evaluation metric.\n\nin the case where availability=50, errors can get accumulated for many predicted points. hence the loss will be higher." }, { "id": 1078142, "postDate": "2020-11-14T12:17:25.673Z", "content": "of course that is the case, you can see it in the l5kit time_displace error, which is exactly evaluating this. But naturally, 49 and 50 are similar, which is not what your first plot suggests, that's all I am saying.
Yes, the plot is rather misleading because the frequency of availability=50 makes it difficult to evaluate the distribution of losses. Because you are viewing thousands of samples rather than just a few it makes the spread look much more drastic even though it could just be a very small subset that makes the graph look like the spread is actually much greater. If you had thousands of samples for any other availability near 50 you might get a visualization that looks similar to availability=50
", "rawMarkdown": "Yes, the plot is rather misleading because the frequency of availability=50 makes it difficult to evaluate the distribution of losses. Because you are viewing thousands of samples rather than just a few it makes the spread look much more drastic even though it could just be a very small subset that makes the graph look like the spread is actually much greater. If you had thousands of samples for any other availability near 50 you might get a visualization that looks similar to availability=50", "votes": 1 } ] }, { "id": 1077899, "postDate": "2020-11-14T04:29:24.903Z", "content": "\"ensembling\" that works. i implemented it and it has some improvement
\n1. train 2 models:\nmodel1(x) : x -->embed1(x)-->head1(x)\nmodel2(x) : x -->embed2(x)-->head2(x)\n\n2. measure loss on validation set.\n- we plot loss1 vs loss2 to visually inspect correlation and diversity\n- we compute upper bound for improvement: best loss = average { min(loss1(x),loss2(x)) } \n\n3. make ensemble model\nmode: x -->embed1(x),embed2(x) --> concat --> head(concat)\n\n- we freeze embed1,embed2 to prevent overfitting. you should see some improvement\nWhat I am curious about is how do mode7 make inference? Take the top 3 confidence?
", "rawMarkdown": "What I am curious about is how do mode7 make inference? Take the top 3 confidence?", "replies": [ { "id": 1076928, "postDate": "2020-11-13T03:37:25.307Z", "content": "this is local validation. i use all 7 mode
", "rawMarkdown": "this is local validation. i use all 7 mode" }, { "id": 1076957, "postDate": "2020-11-13T04:32:26.447Z", "content": "What about \"total confidences should sum to 1\"? Simply increase the top 1 confidence and make the total sum to 1?
", "rawMarkdown": "What about \"total confidences should sum to 1\"? Simply increase the top 1 confidence and make the total sum to 1?" }, { "id": 1076964, "postDate": "2020-11-13T04:44:47.037Z", "content": "i use all 7 mode, meaning that I have 7 confidence values and 7 set of x,y coords.
\nthis will not be able to make a submission, but it is for analyzing results and data.
the evaluation function is not affected as the function does not assume 3 mode.
\nbut it does show that if you can make more modes, e.g. via more prediction from one model or concate predictions from multiple models, you may be able to get very low metric score.
\nbut of course, you will have another problem of selecting the correct 3 for submission. --- that is the problem of ensmbling
", "rawMarkdown": "i use all 7 mode, meaning that I have 7 confidence values and 7 set of x,y coords.\nthis will not be able to make a submission, but it is for analyzing results and data.\n\nthe evaluation function is not affected as the function does not assume 3 mode.\n\n---\n\nbut it does show that if you can make more modes, e.g. via more prediction from one model or concate predictions from multiple models, you may be able to get very low metric score. \n\n\nbut of course, you will have another problem of selecting the correct 3 for submission. --- that is the problem of ensmbling\n" }, { "id": 1077620, "postDate": "2020-11-13T18:34:57.503Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 1072972, "postDate": "2020-11-09T00:47:29.280Z", "content": "you might wan to check:
\n\"Cycling rules in the US
\nThere are differences in the cycling laws across the states, so please click here to check the bike laws based on your destination.
There are, however, a few general rules applicable wherever within the USA:
\nIn the United States, everyone must drive on the right-hand side of the roadway. Never ride your bike against the traffic flow.\"
", "rawMarkdown": "you might wan to check:\n\n\"Cycling rules in the US\nThere are differences in the cycling laws across the states, so please click here to check the bike laws based on your destination.\n\nThere are, however, a few general rules applicable wherever within the USA:\n\nIn the United States, everyone must drive on the right-hand side of the roadway. Never ride your bike against the traffic flow.\"" }, { "id": 1066996, "postDate": "2020-11-02T09:32:20.790Z", "content": "i was reading the paper introduced by @ryches
\nMultimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks
\nhttps://arxiv.org/pdf/1809.10732.pdf
\"We collected 240 hours of data by manually driving SDV in
\nPittsburgh, PA and Phoenix, AZ in various traffic conditions
\n(e.g., varying times of day, days of the week). Traffic actors
\nwere tracked using Unscented Kalman filter (UKF) \"
Now I need to read the document of l5kit to see how the data is collected. i don't think anyone would annotate the location of the agent frame by frame. there must be some kind of processing data.
", "rawMarkdown": "i was reading the paper introduced by @ryches \n\nMultimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks\nhttps://arxiv.org/pdf/1809.10732.pdf\n\n\n\"We collected 240 hours of data by manually driving SDV in\nPittsburgh, PA and Phoenix, AZ in various traffic conditions\n(e.g., varying times of day, days of the week). Traffic actors\nwere tracked using Unscented Kalman filter (UKF) \"\n\nNow I need to read the document of l5kit to see how the data is collected. i don't think anyone would annotate the location of the agent frame by frame. there must be some kind of processing data." }, { "id": 1064337, "postDate": "2020-10-30T02:32:16.760Z", "content": "Using classification loss instead of regression for post trajectory selection makes sense. covernet uses the similar concept.
\nhttps://arxiv.org/pdf/1911.10298.pdf
for classification loss they first sample sets of trajectories from train data and then select smallest set of fix trajectories using NP hard set cover problem.
\nBut from my experiments, nnl loss performs better than covernets classification loss for this data. Two reasons that I could think of are first different types of agent like pedestrians and cyclist in this data which has very jittery trajectories and secondly covernet loss is usually better for higher number of modes.
The problem I had with covernet is the trajectories they provided are configured for 6 seconds at a .5s resolution and the ones they provided were not the hybrid or dynamic which performed significantly better. It would be much better if they had provided their methods for generating these
", "rawMarkdown": "The problem I had with covernet is the trajectories they provided are configured for 6 seconds at a .5s resolution and the ones they provided were not the hybrid or dynamic which performed significantly better. It would be much better if they had provided their methods for generating these", "votes": 1 }, { "id": 1064811, "postDate": "2020-10-30T14:51:50.403Z", "content": "Methods used to generate sets of trajectories are described in the article. I created hybrid and dynamic sets and set up the CoverNet pipeline. However, the main problem with this approach is that you have a lower boundary for NLL. I did a test to answer the question \"What would be the nll value if CoverNet predicted everything correctly (unimodal)?\" - and given a fixed trajectory set (~1400 trajectories, epsilon 2.0) it was about 9 to 12 nll score. Of course, you cannot assume 100% accuracy, so it's probably not worth further investigation.
\nAlthough, those trajectories sets are useful for debugging.
\nAnd this is how fixed trajectories set (epsilon 2) looks like in Lyft competition:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F3611454%2F128398b61a49bbb0c49dcaa4ddb4c084%2Fdownload.png?generation=1604069404715931&alt=media\")
l believe having the correct way to train the mixture expert is a key to get good results and win the competition. And since the mixture number is only 3 and small, there are many tricks to get better results. (what's more, the car only travels on road, or nearly on road)
\ni am quite busy at the moment and I haven't tried this out yet. but this is what one can do:
\n1) say we divide trajectectory into type = left, front and right .
\n2) we train 3 predictors for each type using :
individual predictor(type, image) --> function( cat [ embed (image), type ] )
\nthis is the upper bound of our performance if we have guess the mode type correctly. Note that embedding weights are shared.
\n3) then, we have
\nfinal_predictor (image) -> type = function( …), image --> individual predictor
this reminds me of the early of face detection and face point localization where estimation of face orientation first and then apply specific face detector to specific orientation.
\nno sure how it will work here or comparing to end-to-end methods. in theory end-to-end should out-performance. and 2 stage method maybe a good baseline.
", "rawMarkdown": "l believe having the correct way to train the mixture expert is a key to get good results and win the competition. And since the mixture number is only 3 and small, there are many tricks to get better results. (what's more, the car only travels on road, or nearly on road)\n\ni am quite busy at the moment and I haven't tried this out yet. but this is what one can do:\n1) say we divide trajectectory into type = left, front and right .\n2) we train 3 predictors for each type using :\n\nindividual predictor(type, image) --> function( cat [ embed (image), type ] )\n\nthis is the upper bound of our performance if we have guess the mode type correctly. Note that embedding weights are shared.\n\n3) then, we have\nfinal\\_predictor (image) -> type = function( ...), image --> individual predictor\n\nthis reminds me of the early of face detection and face point localization where estimation of face orientation first and then apply specific face detector to specific orientation.\n\nno sure how it will work here or comparing to end-to-end methods. in theory end-to-end should out-performance. and 2 stage method maybe a good baseline.", "votes": 4 }, { "id": 1065847, "postDate": "2020-10-31T22:27:18.307Z", "content": "if we predict turn in the first step before trajectories don't you think it can lead to mode collapse and since no one can estimate the direction of agent accurately all the time, for some of them it can be terribly wrong? . I think MTP (https://arxiv.org/pdf/1809.10732.pdf) follows similar type of approach.
\nFirst they find best possible mode using classification loss and then backprop only for that mode to estimate trajectories. I ran into model collapse problem with it though, and not so good results.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4573165%2F0a27d166e533fa8c2a7962ead42a7d4f%2Fdownload.png?generation=1604183925157810&alt=media\")
So in this case all three modes are going straight. Loss is 1674 for this sample.
", "rawMarkdown": "if we predict turn in the first step before trajectories don't you think it can lead to mode collapse and since no one can estimate the direction of agent accurately all the time, for some of them it can be terribly wrong? . I think MTP (https://arxiv.org/pdf/1809.10732.pdf) follows similar type of approach.\nFirst they find best possible mode using classification loss and then backprop only for that mode to estimate trajectories. I ran into model collapse problem with it though, and not so good results.\n\n\nSo in this case all three modes are going straight. Loss is 1674 for this sample." }, { "id": 1066387, "postDate": "2020-11-01T17:35:26.197Z", "content": "if you can find a way to label the type of trajectory, then you can do ensembling.
\nit is just like object detection framework:
classification --> trajectory selection
\nbox regression -->estimation of the trajectory
but of course, you still can do ensembling without classification framework. In this case, assume you train 3 model using resnet34, efficient-b0, resnet-50. the predictions are:
\nyou would have to cluster all the t's into groups. (think of this as non-max suppresion in object detection). you may end up with group1,2,3,4..5 (e.g. 5 groups)
\nfor each group, you need to think of a way to find the center trajectory
\nand also a way to compute the group score. And for grouping, you need to away to measure trajectory distance (just like iou in non-max suppression)
there are papers that use transformer or train a network for non-max suppression to replace heuristics non-max-suppresion. these methods can be applied here too.
\ndef decide_to_ensemble_net(trajectory1, trajectory2):\n ...\n it is like 2 layer 2 stacking net\nif you can do ensembling, you can out-performance one single model. this opens up for the further application of pesudo-label, semi-supervised, weak-supervised methods.
", "rawMarkdown": "if you can find a way to label the type of trajectory, then you can do ensembling.\nit is just like object detection framework:\n\nclassification --> trajectory selection\nbox regression -->estimation of the trajectory\n\n---\n\nbut of course, you still can do ensembling without classification framework. In this case, assume you train 3 model using resnet34, efficient-b0, resnet-50. the predictions are:\n- ca0, ca1, ca2, ta0, ta1, ta2 (c for confidence, t for predicted trajectory)\n- cb0, cb1, cb2, tb0, tb1, tb2 \n- cc0, cc1, cc2, tc0, tc1, tc2 \n\nyou would have to cluster all the t's into groups. (think of this as non-max suppresion in object detection). you may end up with group1,2,3,4..5 (e.g. 5 groups)\n\nfor each group, you need to think of a way to find the center trajectory\nand also a way to compute the group score. And for grouping, you need to away to measure trajectory distance (just like iou in non-max suppression)\n\n\n---\n\nthere are papers that use transformer or train a network for non-max suppression to replace heuristics non-max-suppresion. these methods can be applied here too.\n\n```\n\ndef decide_to_ensemble_net(trajectory1, trajectory2):\n ...\n it is like 2 layer 2 stacking net\n```\n\n---\n\nif you can do ensembling, you can out-performance one single model. this opens up for the further application of pesudo-label, semi-supervised, weak-supervised methods.\n\n", "votes": 4 }, { "id": 1067227, "postDate": "2020-11-02T12:04:14.710Z", "content": "coverNet: https://www.youtube.com/watch?v=fTM6ZLwmp10
\nclassifiy trajectory
TPNet: https://www.youtube.com/watch?v=Bfvy7qby1fg
\npropose trajectory
@hengck23 Great insights, wondering if you have any code reference for classification head and regression head that could be adapted to this ??
", "rawMarkdown": "@hengck23 Great insights, wondering if you have any code reference for classification head and regression head that could be adapted to this ??", "votes": -1 }, { "id": 1071328, "postDate": "2020-11-06T18:28:14.923Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 1064055, "postDate": "2020-10-29T16:31:51.690Z", "content": "Hey @engck23, cool to see that you are jumping in!
\nYeah it's something I thought of. I was playing around with a model which has a prediction head and classification head, so in principle, you could predict 6 trajectories, and then only take the top 3 based on the classification score from the other head.
\nNot sure if it will help, but it makes sense to me that there could be more then 3 viable routes, and the additional exploration might help.
\nDon't have any concrete results as to whether or not it actually helps.
this is what you get if you plot the target on world coords.
\nthere could be some experiment design flaw here. The test data should be of a completely different road scene from the training dataset to avoid bias.
\n
\n
\ntypo error : there is no red …
Do you have a hint how to implement additional inputs? I guess you are talking about the model architecture itself. Do you have an example how to add non image data to a pretained resnet34?
", "votes": 0, "replies": [] }, { "id": 1072888, "author_name": "Michał Szachniewicz", "author_url": "", "post_date": "2020-11-08T20:14:48.760000", "content": "In the forward function, you can extract features from the backbone, then flatten it and concatenate with any additional features you want using torch.cat(). After that, you will have to adjust the number of in_features in your dense layer (just increase it by the length of your non-image input).
", "votes": 2, "replies": [] }, { "id": 1072896, "author_name": "Benedikt Droste", "author_url": "", "post_date": "2020-11-08T20:43:51.733000", "content": "Thanks for the quick reply @szacho
\nDo you have a code example? I thought about something like this:
\n1# forward
\n def forward(self, x, non-image-data):\n x = self.backbone.conv1(x)\n x = self.backbone.bn1(x)\n x = self.backbone.relu(x)\n x = self.backbone.maxpool(x)\n\n x = self.backbone.layer1(x)\n x = self.backbone.layer2(x)\n x = self.backbone.layer3(x)\n x = self.backbone.layer4(x)\n\n x = self.backbone.avgpool(x)\n x = torch.flatten(x, 1)\n y = non-image-data\n x = torch.cat((x, y), dim=1)\n\n x = self.head(x)\n x = self.logit(x)\n [ ...... ]\n2# init
\ndef __init__(self, cfg: Dict, num_modes=3):\n [ ...... ]\n # You can add more layers here.\n self.head = nn.Sequential(\n # nn.Dropout(0.2),\n nn.Linear(in_features=backbone_out_features+len(non-image-data), out_features=4096),\n )\n self.num_preds = num_targets * num_modes\n self.num_modes = num_modes\n\n self.logit = nn.Linear(4096, out_features=self.num_preds + num_modes)\nWhat do you think?
", "votes": 1, "replies": [] }, { "id": 1073382, "author_name": "Michał Szachniewicz", "author_url": "", "post_date": "2020-11-09T13:25:13.987000", "content": "@benbla
\nSeems fine, I implemented it in a very similar if not the same way. Good job!
here is the loss breakdown by label:
\nfor non-chopped data:\nlocal cv loss: 9.346676 (lb = 15.034)\nbreakdown: \nPERCEPTION_LABEL_CAR 9.259042 \nPERCEPTION_LABEL_CYCLIST 17.221859 \nPERCEPTION_LABEL_PEDESTRIAN 4.827463\n
i wonder are these ground truth senor errors?
\n
Hi, @szacho. Did you feed \"history_positions\" and other data into the model? During training, is \"history_availabilities \" useful to imply the available agent in history frames?
", "votes": 0, "replies": [] }, { "id": 1074375, "author_name": "Benedikt Droste", "author_url": "", "post_date": "2020-11-10T15:29:06.357000", "content": "@spicychicken38
\nIntegrating \"history_positions\" leads to a huge overfit and a big derivation between loss and lb score to me.
\n\n@spicychicken38
\n
\n Integrating \"history_positions\" leads to a huge overfit and a big derivation between loss and lb score to me.
Hi, @benbla. I don't think this huge deviation is caused by overfitting. According to my experiment, there is a data mismatch that is caused by the differences between train.zarr and validation.zarr(or test.zarr), which can lead to this huge deviation. Because the \"history_availabilities\" in train.zarr are always 1, which means agents exist in every frame in the training sample that they belong to. However, this is totally different in the chopped validation set or the testing set. So I think this might be the main reason why we have such a huge deviation when we are adding \"history_positions\".
", "votes": 1, "replies": [] } ] }, { "id": 1074389, "author_name": "Yannik", "author_url": "", "post_date": "2020-11-10T15:44:36.343000", "content": "@benbla In my case, there is also a huge deviation between validation score and training loss. But I can not certainly say that it is overfitting. I also think that it is also possible that the differences of \"history_availabilities\" between the training set and the chopped validation set to cause this huge deviation. But I can not verify it. As mention here, @frankpanxj also meet the same problem.
", "votes": 0, "replies": [] }, { "id": 1074393, "author_name": "Michał Szachniewicz", "author_url": "", "post_date": "2020-11-10T15:50:07.757000", "content": "@spicychicken38 No, I feed the agent state vector as described in this article: https://arxiv.org/abs/1809.10732
", "votes": 0, "replies": [] }, { "id": 1074406, "author_name": "Yannik", "author_url": "", "post_date": "2020-11-10T16:06:30.183000", "content": "@szacho But we don't have the velocity, acceleration mentioned in the article. How you implement this?
", "votes": 0, "replies": [] }, { "id": 1074459, "author_name": "Michał Szachniewicz", "author_url": "", "post_date": "2020-11-10T17:07:33.470000", "content": "@spicychicken38 Velocity is available in zarr data, but you can get everything from data returned by AgentDataset - you have history_positions and history_yaws in time. Features like velocity, acceleration, and yaw change rate can be calculated using basic physics knowledge.
", "votes": 1, "replies": [] }, { "id": 1074777, "author_name": "Yannik", "author_url": "", "post_date": "2020-11-11T03:38:06.713000", "content": "@szacho Thanks for your views. I will try to do some experiments on that later!
", "votes": 0, "replies": [] }, { "id": 1075234, "author_name": "Pallavi Ramicetty", "author_url": "", "post_date": "2020-11-11T14:09:26.740000", "content": "@szacho As you mentioned we can get the velocity from AgentDataset. But what about the agent's label_probabilities? seems agent_samples.py won't return them. If possible, can suggest some method to get them along with the \"data\" object?
", "votes": 0, "replies": [] }, { "id": 1075254, "author_name": "hengck23", "author_url": "", "post_date": "2020-11-11T14:19:33.403000", "content": "for the file:
\n/l5kit/sampling/agent_sampling.py
def generate_agent_sample(...):\n ...\n \"extent\": agent_extent_m,\n #\"agent\": selected_agent, # <add here>\n \"label_probabilities\": selected_agent[\"label_probabilities\"], # <add here>\nfor the file:
\n/l5kit/dataset/ego.py
class EgoDataset(Dataset):\n...\ndef get_frame(...):\n\n \"extent\": data[\"extent\"],\n #\"agent\": data[\"agent\"], #<add here>\n \"label_probabilities\": data[\"label_probabilities\"], #<add here>\n }\nHi @hengck23 , Thanks for your insights. How to add this custom l5kit library to one's kaggle notebook? Instead I thought of calculating velocities in rudimentary way like this (I guess we have to calculate yaw rate and acceleration ourselves anyway):
\n if data['history_availabilities'].to(device)shape[1] > 1: \n velocities = torch.sqrt(torch.sum((pose_data[:,1:,:] - pose_data[:,0:-1, :]) ** 2, 2)) / dt\n avg_velocity = torch.mean(velocities, dim=1).view(bs,1)\n #acceleration = torch.abs(velocities[:, 1:] - velocities[:, 0:-1])\n avg_acceleration = torch.mean(torch.abs(velocities[:, 1:] - velocities[:, 0:-1]) / dt, dim=1).view(bs,1)\n yaw_rate = torch.mean(torch.abs(yaw_data[:, 1:] - yaw_data[:, 0:-1]) / dt, dim=1).view(bs,1)\n\n avg_velocity = avg_velocity / MAX_VELOCITY # 20 m/s\n avg_acceleration = avg_acceleration / MAX_ACCELERATION # 2m/s^2\n yaw_rate = yaw_rate / MAX_YAW_RATE # 45 deg/s\n\n else:\n avg_velocity = 0.0; avg_acceleration = 0.0; yaw_rate = 0.0; \nIs there anything I'm missing here? Any any exception handling required?
", "votes": 1, "replies": [] }, { "id": 1082506, "author_name": "Saurabh7", "author_url": "", "post_date": "2020-11-18T00:59:28.477000", "content": "Adding these state inputs leads to large overfitting for me, anyone else seeing similar results ?
", "votes": 0, "replies": [] } ] }, { "id": 1075328, "author_name": "Pallavi Ramicetty", "author_url": "", "post_date": "2020-11-11T15:35:08.927000", "content": "@hengck23 Oh you are using l5kit/ as part of your code. I am using it as a library. Sure I will do require changes. Thanks for your quick response.
", "votes": 0, "replies": [] }, { "id": 1078444, "author_name": "InDSweTrust", "author_url": "", "post_date": "2020-11-14T19:50:00.310000", "content": "Speaking of velocities, agent_sampling.py and ego.py have been updated. You can now get velocity (and speed) through the AgentDataset() class.
Where can I find the updates ?
", "votes": 0, "replies": [] }, { "id": 1078462, "author_name": "InDSweTrust", "author_url": "", "post_date": "2020-11-14T20:17:03.597000", "content": "l5kit github repository, here
", "votes": 1, "replies": [] } ] }, { "id": 1076901, "author_name": "hengck23", "author_url": "", "post_date": "2020-11-13T02:52:24.260000", "content": "surprise, surprise, surprise …
\nI put the conclusion first and then the observations.
conclusion:
\nwe probably want to detect, predict, post or pre-processing them separately. they are easy to identify via past histories.
\nobservation:
\nsee below simulation:
\n
hence I trained a mode-7 predictor from my current mode-3 predictor. i just freeze the backbone weights and change the head. it is very fast to train. on chopped validation data, kaggle metric reduces from 15+ to 11+ within hours.
\nbut when I inspect the results, I was surprised
", "votes": 4, "replies": [ { "id": 1076902, "author_name": "hengck23", "author_url": "", "post_date": "2020-11-13T02:53:07.980000", "content": "
\nNote: if there are shakeup, it can be due to just a few outliers
so, what are the cases, mode-7 performs better?
\ntruth: disconnected black dots
\n(history is not shown)
you can see that we can gain a lot if we can estimate the \"straight trajectory\" more accurate … which seems to be an easy task
", "votes": 1, "replies": [] }, { "id": 1077784, "author_name": "A_Elsheikh", "author_url": "", "post_date": "2020-11-13T23:19:28", "content": "Nice catch but the competition metric is biased towards long trajectories. Sigma is fixed to 1 which is not real! It should increase as the predictions goes further in time.
", "votes": 0, "replies": [] }, { "id": 1078686, "author_name": "hengck23", "author_url": "", "post_date": "2020-11-15T07:12:14.137000", "content": "as an example
\n
you can add other labels for specific post-processing, e.g. di-biasing (the kaggle multiplier trick in regression problem, which is essentially manually adjusting the std of error)
", "votes": 1, "replies": [] }, { "id": 1078739, "author_name": "Ram Ramrakhya", "author_url": "", "post_date": "2020-11-15T08:49:06.343000", "content": "How do you set the is_parametric flag? It's not clear to me. Do you use a custom curve fitting function?
e.g. for each train sample, you have x =[history, future]
\nthen you devise a curve function, predict_future = f(history). you will have to design this by hand, making some assumptions. you can google for some kinematics model.
\nnow each train sample will have x =[history, f(history), future].
\ncompute error of f(history), e.g. l2 loss (f(history), future). now if this error is small, you can label is as True (i.e. you can recover future values from f(history))
", "votes": 1, "replies": [] }, { "id": 1078778, "author_name": "Ram Ramrakhya", "author_url": "", "post_date": "2020-11-15T09:44:17.033000", "content": "Thanks @hengck23, this helps :)
", "votes": 0, "replies": [] } ] }, { "id": 1072969, "author_name": "hengck23", "author_url": "", "post_date": "2020-11-09T00:44:23.703000", "content": "there is much more information in the zarr file (instead of the AgentDataset)
\ne.g. you get velocity and label probability
\nAGENT_DTYPE = [\n (\"centroid\", np.float64, (2,)),\n (\"extent\", np.float32, (3,)),\n (\"yaw\", np.float32),\n (\"velocity\", np.float32, (2,)),\n (\"track_id\", np.uint64),\n (\"label_probabilities\", np.float32, (len(LABELS),)),\n]\nhere is an example of reading the raw zarr file. note that you can recover the whole track!
\n
you can submit the top-1, top-2 and top-3 results to prob the correctness of your prediction. then it may be possible to recalibrate or adjust the temperate of the confidence.
\nsome of the predictions are clearly only single-mode, e.g. moving in a straight lane or not moving. in that case a single high confidence trajectory prediction may be better?
", "votes": 1, "replies": [] }, { "id": 1067368, "author_name": "hengck23", "author_url": "", "post_date": "2020-11-02T13:52:56.493000", "content": "self-supervised?
\nfor the test data, say we ask for number of history frame = 30.
\nwe use t-30 to t-20 as input and train an embedding to predict t-20 to t.
would it work?
\nhow about :
\nfuture = predict (history) and then history1 = predict(future) … like to and from lanuage translation …
\nwould that be a kind of augmentation (i.e. history -->history1)?
MANTRA: Memory Augmented Networks for Multiple Trajectory Prediction
\nhttps://openaccess.thecvf.com/content_CVPR_2020/papers/Marchetti_MANTRA_Memory_Augmented_Networks_for_Multiple_Trajectory_Prediction_CVPR_2020_paper.pdf
In the original paper of the base model that lyft provided to us it seems to show that 3 is empirically the most performant number of modes https://arxiv.org/pdf/1809.10732.pdf
\n1, 2, and 4 modes do worse. This has been reproduced by a few other papers as well.
", "votes": 1, "replies": [ { "id": 1064288, "author_name": "hengck23", "author_url": "", "post_date": "2020-10-29T23:49:07.947000", "content": "thanks for the paper.
\n
it seems that it is due to the nature road junction. in that case, we can just train for going straight, turn left and turn right. i need to check the distribution/frequency of turns of junctions. thanks!
", "votes": 1, "replies": [] }, { "id": 1065049, "author_name": "Ali Abdin", "author_url": "", "post_date": "2020-10-30T19:38:03.467000", "content": "Intresting, thank you for the paper.
\nThe relations are greatly explained and visualized.
", "votes": 0, "replies": [] }, { "id": 1065578, "author_name": "Ali Abdin", "author_url": "", "post_date": "2020-10-31T13:13:12.893000", "content": "Does anyone know if the mentioned MTP loss is somewhere available as public code?
", "votes": 0, "replies": [] }, { "id": 1065608, "author_name": "Michał Szachniewicz", "author_url": "", "post_date": "2020-10-31T13:58:34.417000", "content": "Here, https://github.com/nutonomy/nuscenes-devkit/blob/master/python-sdk/nuscenes/prediction/models/mtp.py
", "votes": 1, "replies": [] }, { "id": 1065732, "author_name": "Ali Abdin", "author_url": "", "post_date": "2020-10-31T16:56:38.637000", "content": "Thank you :)
", "votes": 0, "replies": [] } ] }, { "id": 1076954, "author_name": "hengck23", "author_url": "", "post_date": "2020-11-13T04:28:13.600000", "content": "to disable image rasterizer and just do prediction using history information:
\n dm = LocalDataManager()\n rasterizer = None #build_rasterizer(cfg, dm)\n\n zarr = ChunkedDataset( dm.require('scenes/train.zarr')).open()\n dataset = MyAgentDataset(cfg, zarr, rasterizer) \n\n\n\nin ego.py: (around line 99)\n\nclass EgoDataset(Dataset):\n\ndef get_frame(self, scene_index ...):\n\n # 0,1,C -> C,0,1\n # <hck>\n if data[\"image\"] is not None:\n image = data[\"image\"].transpose(2, 0, 1)\n else:\n image = 0 #dummy\ni am thinking of using:
\nso i need to create 2 dataset, one with rasterizer and one without (to speed up cpu data loading).
", "votes": 2, "replies": [ { "id": 1077782, "author_name": "A_Elsheikh", "author_url": "", "post_date": "2020-11-13T23:15:21.343000", "content": "I reported this as a bug long time ago on github and I guess it is now fixed
", "votes": 0, "replies": [] }, { "id": 1077851, "author_name": "Yannik", "author_url": "", "post_date": "2020-11-14T02:13:08.017000", "content": "You mean that by doing disabling the rasterizer, we can use history information for improvement regardless of the differences of history_availabilities between train.zarr and test.zarr?
", "votes": 0, "replies": [] }, { "id": 1077896, "author_name": "hengck23", "author_url": "", "post_date": "2020-11-14T04:22:54.570000", "content": "rendering long history frames is very slow. hence I want to experiment using history positions only without image. (or few history images)
\nthis is non- fixed length sequence learning, e.g
\ninput:\nx(t), x(t-1),x(t-2) ... <end>\nyou can treat it as LSTM seq with an \"end\" token. alternatively you can pad to equal length.
", "votes": 0, "replies": [] }, { "id": 1078104, "author_name": "Yosshi999", "author_url": "", "post_date": "2020-11-14T11:21:04.800000", "content": "You can use StubRasterizer to render only blank images
", "votes": 0, "replies": [] } ] }, { "id": 1065510, "author_name": "hengck23", "author_url": "", "post_date": "2020-10-31T11:17:03.593000", "content": "i wonder did anyone plot and visualise the predicted trajectory?
\ni don't think it would be a perfect parametric curve. it might look a little \"wiggling\" and not smooth. So there are be some post-processing required, or better still, should we predict the parameters of the curve of trajectory instead?
\nhistory trajectory can also be represented as a parametric curve
", "votes": 2, "replies": [ { "id": 1065524, "author_name": "fnands", "author_url": "", "post_date": "2020-10-31T11:33:06.847000", "content": "There has been some discussion about maybe using a Kalman filter to smooth out predictions, although at the moment my predicted trajectories tend to be pretty smooth:
The targets often have some amount of noise in them, so being smoother than the targets might not give that much improvement
", "votes": 5, "replies": [] }, { "id": 1066394, "author_name": "hengck23", "author_url": "", "post_date": "2020-11-01T17:54:15.257000", "content": "seems that predicting the end-point correctly is very important.
", "votes": 0, "replies": [] } ] }, { "id": 1077979, "author_name": "hengck23", "author_url": "", "post_date": "2020-11-14T07:13:11.497000", "content": "
I believe, that picture is rather misleading with different counts in each bin. You may subsample the 50 avails part to match the count for the others, or plot e.g. violin graphs to get a better view on the loss distribution.
", "votes": 0, "replies": [] }, { "id": 1078093, "author_name": "hengck23", "author_url": "", "post_date": "2020-11-14T10:50:16.457000", "content": "
count of samples vs samples' target availability
", "votes": 1, "replies": [] }, { "id": 1078110, "author_name": "Pascal Pfeiffer", "author_url": "", "post_date": "2020-11-14T11:31:02.323000", "content": "I know, that's why your first Image gives a misleading view on the loss distribution. It looks like the loss distribution for 50 avails is much broader than for example 49 avails, but in reality they are very similar.
", "votes": 0, "replies": [] }, { "id": 1078120, "author_name": "hengck23", "author_url": "", "post_date": "2020-11-14T11:43:59.517000", "content": "i think the loss for e.g. availability=10 would be small because only \"10 nearby points are counted\" in the evaluation metric.
\nin the case where availability=50, errors can get accumulated for many predicted points. hence the loss will be higher.
", "votes": 0, "replies": [] }, { "id": 1078142, "author_name": "Pascal Pfeiffer", "author_url": "", "post_date": "2020-11-14T12:17:25.673000", "content": "of course that is the case, you can see it in the l5kit time_displace error, which is exactly evaluating this. But naturally, 49 and 50 are similar, which is not what your first plot suggests, that's all I am saying.
Yes, the plot is rather misleading because the frequency of availability=50 makes it difficult to evaluate the distribution of losses. Because you are viewing thousands of samples rather than just a few it makes the spread look much more drastic even though it could just be a very small subset that makes the graph look like the spread is actually much greater. If you had thousands of samples for any other availability near 50 you might get a visualization that looks similar to availability=50
", "votes": 1, "replies": [] } ] }, { "id": 1077899, "author_name": "hengck23", "author_url": "", "post_date": "2020-11-14T04:29:24.903000", "content": "\"ensembling\" that works. i implemented it and it has some improvement
\n1. train 2 models:\nmodel1(x) : x -->embed1(x)-->head1(x)\nmodel2(x) : x -->embed2(x)-->head2(x)\n\n2. measure loss on validation set.\n- we plot loss1 vs loss2 to visually inspect correlation and diversity\n- we compute upper bound for improvement: best loss = average { min(loss1(x),loss2(x)) } \n\n3. make ensemble model\nmode: x -->embed1(x),embed2(x) --> concat --> head(concat)\n\n- we freeze embed1,embed2 to prevent overfitting. you should see some improvement\nWhat I am curious about is how do mode7 make inference? Take the top 3 confidence?
", "votes": 0, "replies": [ { "id": 1076928, "author_name": "hengck23", "author_url": "", "post_date": "2020-11-13T03:37:25.307000", "content": "this is local validation. i use all 7 mode
", "votes": 0, "replies": [] }, { "id": 1076957, "author_name": "Yannik", "author_url": "", "post_date": "2020-11-13T04:32:26.447000", "content": "What about \"total confidences should sum to 1\"? Simply increase the top 1 confidence and make the total sum to 1?
", "votes": 0, "replies": [] }, { "id": 1076964, "author_name": "hengck23", "author_url": "", "post_date": "2020-11-13T04:44:47.037000", "content": "i use all 7 mode, meaning that I have 7 confidence values and 7 set of x,y coords.
\nthis will not be able to make a submission, but it is for analyzing results and data.
the evaluation function is not affected as the function does not assume 3 mode.
\nbut it does show that if you can make more modes, e.g. via more prediction from one model or concate predictions from multiple models, you may be able to get very low metric score.
\nbut of course, you will have another problem of selecting the correct 3 for submission. --- that is the problem of ensmbling
", "votes": 0, "replies": [] }, { "id": 1077620, "author_name": "", "author_url": "", "post_date": "2020-11-13T18:34:57.503000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1072972, "author_name": "hengck23", "author_url": "", "post_date": "2020-11-09T00:47:29.280000", "content": "you might wan to check:
\n\"Cycling rules in the US
\nThere are differences in the cycling laws across the states, so please click here to check the bike laws based on your destination.
There are, however, a few general rules applicable wherever within the USA:
\nIn the United States, everyone must drive on the right-hand side of the roadway. Never ride your bike against the traffic flow.\"
", "votes": 0, "replies": [] }, { "id": 1066996, "author_name": "hengck23", "author_url": "", "post_date": "2020-11-02T09:32:20.790000", "content": "i was reading the paper introduced by @ryches
\nMultimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks
\nhttps://arxiv.org/pdf/1809.10732.pdf
\"We collected 240 hours of data by manually driving SDV in
\nPittsburgh, PA and Phoenix, AZ in various traffic conditions
\n(e.g., varying times of day, days of the week). Traffic actors
\nwere tracked using Unscented Kalman filter (UKF) \"
Now I need to read the document of l5kit to see how the data is collected. i don't think anyone would annotate the location of the agent frame by frame. there must be some kind of processing data.
", "votes": 0, "replies": [] }, { "id": 1064337, "author_name": "SujaydKhandekar", "author_url": "", "post_date": "2020-10-30T02:32:16.760000", "content": "Using classification loss instead of regression for post trajectory selection makes sense. covernet uses the similar concept.
\nhttps://arxiv.org/pdf/1911.10298.pdf
for classification loss they first sample sets of trajectories from train data and then select smallest set of fix trajectories using NP hard set cover problem.
\nBut from my experiments, nnl loss performs better than covernets classification loss for this data. Two reasons that I could think of are first different types of agent like pedestrians and cyclist in this data which has very jittery trajectories and secondly covernet loss is usually better for higher number of modes.
The problem I had with covernet is the trajectories they provided are configured for 6 seconds at a .5s resolution and the ones they provided were not the hybrid or dynamic which performed significantly better. It would be much better if they had provided their methods for generating these
", "votes": 1, "replies": [] }, { "id": 1064811, "author_name": "Michał Szachniewicz", "author_url": "", "post_date": "2020-10-30T14:51:50.403000", "content": "Methods used to generate sets of trajectories are described in the article. I created hybrid and dynamic sets and set up the CoverNet pipeline. However, the main problem with this approach is that you have a lower boundary for NLL. I did a test to answer the question \"What would be the nll value if CoverNet predicted everything correctly (unimodal)?\" - and given a fixed trajectory set (~1400 trajectories, epsilon 2.0) it was about 9 to 12 nll score. Of course, you cannot assume 100% accuracy, so it's probably not worth further investigation.
\nAlthough, those trajectories sets are useful for debugging.
\nAnd this is how fixed trajectories set (epsilon 2) looks like in Lyft competition:
\n
l believe having the correct way to train the mixture expert is a key to get good results and win the competition. And since the mixture number is only 3 and small, there are many tricks to get better results. (what's more, the car only travels on road, or nearly on road)
\ni am quite busy at the moment and I haven't tried this out yet. but this is what one can do:
\n1) say we divide trajectectory into type = left, front and right .
\n2) we train 3 predictors for each type using :
individual predictor(type, image) --> function( cat [ embed (image), type ] )
\nthis is the upper bound of our performance if we have guess the mode type correctly. Note that embedding weights are shared.
\n3) then, we have
\nfinal_predictor (image) -> type = function( …), image --> individual predictor
this reminds me of the early of face detection and face point localization where estimation of face orientation first and then apply specific face detector to specific orientation.
\nno sure how it will work here or comparing to end-to-end methods. in theory end-to-end should out-performance. and 2 stage method maybe a good baseline.
", "votes": 4, "replies": [] }, { "id": 1065847, "author_name": "SujaydKhandekar", "author_url": "", "post_date": "2020-10-31T22:27:18.307000", "content": "if we predict turn in the first step before trajectories don't you think it can lead to mode collapse and since no one can estimate the direction of agent accurately all the time, for some of them it can be terribly wrong? . I think MTP (https://arxiv.org/pdf/1809.10732.pdf) follows similar type of approach.
\nFirst they find best possible mode using classification loss and then backprop only for that mode to estimate trajectories. I ran into model collapse problem with it though, and not so good results.
\n
So in this case all three modes are going straight. Loss is 1674 for this sample.
", "votes": 0, "replies": [] }, { "id": 1066387, "author_name": "hengck23", "author_url": "", "post_date": "2020-11-01T17:35:26.197000", "content": "if you can find a way to label the type of trajectory, then you can do ensembling.
\nit is just like object detection framework:
classification --> trajectory selection
\nbox regression -->estimation of the trajectory
but of course, you still can do ensembling without classification framework. In this case, assume you train 3 model using resnet34, efficient-b0, resnet-50. the predictions are:
\nyou would have to cluster all the t's into groups. (think of this as non-max suppresion in object detection). you may end up with group1,2,3,4..5 (e.g. 5 groups)
\nfor each group, you need to think of a way to find the center trajectory
\nand also a way to compute the group score. And for grouping, you need to away to measure trajectory distance (just like iou in non-max suppression)
there are papers that use transformer or train a network for non-max suppression to replace heuristics non-max-suppresion. these methods can be applied here too.
\ndef decide_to_ensemble_net(trajectory1, trajectory2):\n ...\n it is like 2 layer 2 stacking net\nif you can do ensembling, you can out-performance one single model. this opens up for the further application of pesudo-label, semi-supervised, weak-supervised methods.
", "votes": 4, "replies": [] }, { "id": 1067227, "author_name": "hengck23", "author_url": "", "post_date": "2020-11-02T12:04:14.710000", "content": "coverNet: https://www.youtube.com/watch?v=fTM6ZLwmp10
\nclassifiy trajectory
TPNet: https://www.youtube.com/watch?v=Bfvy7qby1fg
\npropose trajectory
@hengck23 Great insights, wondering if you have any code reference for classification head and regression head that could be adapted to this ??
", "votes": -1, "replies": [] }, { "id": 1071328, "author_name": "", "author_url": "", "post_date": "2020-11-06T18:28:14.923000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1064055, "author_name": "fnands", "author_url": "", "post_date": "2020-10-29T16:31:51.690000", "content": "Hey @engck23, cool to see that you are jumping in!
\nYeah it's something I thought of. I was playing around with a model which has a prediction head and classification head, so in principle, you could predict 6 trajectories, and then only take the top 3 based on the classification score from the other head.
\nNot sure if it will help, but it makes sense to me that there could be more then 3 viable routes, and the additional exploration might help.
\nDon't have any concrete results as to whether or not it actually helps.