{ "id": 199499, "title": "Best single model", "url": "/competitions/lyft-motion-prediction-autonomous-vehicles/discussion/199499", "author_name": "nosound", "post_date": "2020-11-26T00:53:01.233000", "votes": 21, "comment_count": 42, "views": 0, "content": "

What is your best single model?

\n

Mine is mixnet_l from timm, with standard 3 trajectories output, trained on 215 (epochs) x 64 (batch size) x 2500 (iterations)=34.4m samples.

\n\n

Slightly more fitted on the training data, which indicates that the training data is not infinite. I made sure that no sample point is attended twice, but the correlations between the samples are still significant which results in this lower training loss. But the validation and the LB were very similar along the way.

\n

Note that mixnet_l is the heavy one in the family. It gave me a big boost to move from mixnet_s to mixnet_m and then to mixnet_l, approximately 1 score point per change. This is in contradiction with several discussions on the forum that smaller models are better. Looking forward to learn what other models have worked.

", "messages": [ { "id": 1091365, "postDate": "2020-11-26T00:53:01.233Z", "content": "

What is your best single model?

\n

Mine is mixnet_l from timm, with standard 3 trajectories output, trained on 215 (epochs) x 64 (batch size) x 2500 (iterations)=34.4m samples.

\n\n

Slightly more fitted on the training data, which indicates that the training data is not infinite. I made sure that no sample point is attended twice, but the correlations between the samples are still significant which results in this lower training loss. But the validation and the LB were very similar along the way.

\n

Note that mixnet_l is the heavy one in the family. It gave me a big boost to move from mixnet_s to mixnet_m and then to mixnet_l, approximately 1 score point per change. This is in contradiction with several discussions on the forum that smaller models are better. Looking forward to learn what other models have worked.

", "rawMarkdown": "What is your best single model?\n\nMine is `mixnet_l` from [timm](https://rwightman.github.io/pytorch-image-models/results/), with standard 3 trajectories output, trained on `215 (epochs) x 64 (batch size) x 2500 (iterations)=34.4m` samples.\n\n- Train loss: 11.48\n- Validation: 12.02\n- LB public: 12.285\n- LB private: 11.484 (13th place)\n\nSlightly more fitted on the training data, which indicates that the training data is not infinite. I made sure that no sample point is attended twice, but the correlations between the samples are still significant which results in this lower training loss. But the validation and the LB were very similar along the way.\n\nNote that `mixnet_l` is the heavy one in the family. It gave me a big boost to move from `mixnet_s` to `mixnet_m` and then to `mixnet_l`, approximately `1` score point per change. This is in contradiction with several discussions on the forum that smaller models are better. Looking forward to learn what other models have worked.\n", "votes": 21 }, { "id": 1091404, "postDate": "2020-11-26T01:56:13.583Z", "content": "

we have two single models (b3 & b6), that each would be enough to win. Will talk more about them in our summary

", "rawMarkdown": "we have two single models (b3 & b6), that each would be enough to win. Will talk more about them in our summary", "votes": 9, "replies": [ { "id": 1091723, "postDate": "2020-11-26T08:26:22.370Z", "content": "

Really interested in seeing how you got efficientnets to work! I fell flat in that regard, and couldn't figure out why they were faring so poorly…

", "rawMarkdown": "Really interested in seeing how you got efficientnets to work! I fell flat in that regard, and couldn't figure out why they were faring so poorly...", "votes": 1 }, { "id": 1091738, "postDate": "2020-11-26T08:41:18.377Z", "content": "

@fergusoci in my experience, i found efficientnets to have tendency to converge slower than resnets. could that be the reason, that you didn't train long enough. Anyone with similar experience?

", "rawMarkdown": "@fergusoci in my experience, i found efficientnets to have tendency to converge slower than resnets. could that be the reason, that you didn't train long enough. Anyone with similar experience?" }, { "id": 1096593, "postDate": "2020-11-30T16:16:44.243Z", "content": "

@christofhenkel Do you mind if I ask you - how did you adapt your efficientnets to more than 3 input channels? I'm thinking that I might have missed something there. I was using something like:

\n

`
\nmodel = EfficientNet.from_name('eff-b0')

\n

model._conv_stem.in_channels = num_in_channels

\n

conv_stem_weight = torch.cat([model._conv_stem.weight.clone()] * int(math.ceil(num_in_channels/3)), axis=1)

\n

model._conv_stem.weight = torch.nn.Parameter(conv_stem_weight[:, :num_in_channels, :num_in_channels, :num_in_channels], requires_grad=True)
\n``
\n@zaharch Maybe you also know the answer to this, given that you opted for mixnet.

\n

Cheers!

", "rawMarkdown": "@christofhenkel Do you mind if I ask you - how did you adapt your efficientnets to more than 3 input channels? I'm thinking that I might have missed something there. I was using something like:\n\n`\nmodel = EfficientNet.from_name('eff-b0')\n\nmodel._conv_stem.in_channels = num_in_channels\n\nconv_stem_weight = torch.cat([model._conv_stem.weight.clone()] * int(math.ceil(num_in_channels/3)), axis=1)\n\nmodel._conv_stem.weight = torch.nn.Parameter(conv_stem_weight[:, :num_in_channels, :num_in_channels, :num_in_channels], requires_grad=True)\n``\n@zaharch Maybe you also know the answer to this, given that you opted for mixnet.\n\nCheers!" }, { "id": 1096608, "postDate": "2020-11-30T16:32:33.360Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1096717, "postDate": "2020-11-30T18:09:23.963Z", "content": "

Actually, I think the issue I may have had was the grad_fn for the conv_stem was not retained from the pretrained model. So effectively the graph was broken at that point. I wonder why it even trained moderately well in that case…

", "rawMarkdown": "Actually, I think the issue I may have had was the grad_fn for the conv_stem was not retained from the pretrained model. So effectively the graph was broken at that point. I wonder why it even trained moderately well in that case..." }, { "id": 1096732, "postDate": "2020-11-30T18:25:24.193Z", "content": "

we used something like this. We used the efficientnet from timm

\n
        elif \"efficientnet\" in cfg[\"model_params\"][\"model_architecture\"]:\n            self.backbone.conv_stem = Conv2dSame(\n                num_in_channels,\n                self.backbone.conv_stem.out_channels,\n                kernel_size=self.backbone.conv_stem.kernel_size,\n                stride=self.backbone.conv_stem.stride,\n                padding=self.backbone.conv_stem.padding,\n                bias=False,\n            )\n
", "rawMarkdown": "we used something like this. We used the efficientnet from timm\n\n```\n elif \"efficientnet\" in cfg[\"model_params\"][\"model_architecture\"]:\n self.backbone.conv_stem = Conv2dSame(\n num_in_channels,\n self.backbone.conv_stem.out_channels,\n kernel_size=self.backbone.conv_stem.kernel_size,\n stride=self.backbone.conv_stem.stride,\n padding=self.backbone.conv_stem.padding,\n bias=False,\n )\n```", "votes": 1 }, { "id": 1096741, "postDate": "2020-11-30T18:30:53.587Z", "content": "

Thanks. Yeah, I can see where I went wrong now. Appreciate the help. 👍

", "rawMarkdown": "Thanks. Yeah, I can see where I went wrong now. Appreciate the help. 👍" }, { "id": 1096742, "postDate": "2020-11-30T18:31:45.943Z", "content": "

I am expecting your final solution. what is your diamond solution

", "rawMarkdown": "I am expecting your final solution. what is your diamond solution" }, { "id": 1096757, "postDate": "2020-11-30T18:50:11.203Z", "content": "

we are still waiting on finalization of the LB…

", "rawMarkdown": "we are still waiting on finalization of the LB..." }, { "id": 1096790, "postDate": "2020-11-30T19:28:08.227Z", "content": "

@christofhenkel , I am curious, why do you wait for the finalization to post your summary?

", "rawMarkdown": "@christofhenkel , I am curious, why do you wait for the finalization to post your summary?" }, { "id": 1097140, "postDate": "2020-12-01T00:48:12.447Z", "content": "

https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/199226

\n

based on this it seems like it will be quite a while before that is finalized

", "rawMarkdown": "https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/199226\n\nbased on this it seems like it will be quite a while before that is finalized" } ] }, { "id": 1091376, "postDate": "2020-11-26T01:07:32.150Z", "content": "

The best single model for me was the xception41 from timm, trained on 224x224 resolution on the full dataset. I have not tried to submit it but on validation it reached around 10.37 after a week of training.

\n

The few days earlier checkpoint of the same model scored 11.1 on public LB, 10.2 on private and 11.1 on validation. The validation score was still dropping, but we switched to the full dataset only a week or so before deadline. Some heavier models seems to reach the lower score with the same number of epochs, but was slower to train.

", "rawMarkdown": "The best single model for me was the xception41 from timm, trained on 224x224 resolution on the full dataset. I have not tried to submit it but on validation it reached around 10.37 after a week of training.\n\nThe few days earlier checkpoint of the same model scored 11.1 on public LB, 10.2 on private and 11.1 on validation. The validation score was still dropping, but we switched to the full dataset only a week or so before deadline. Some heavier models seems to reach the lower score with the same number of epochs, but was slower to train.", "votes": 7, "replies": [ { "id": 1091385, "postDate": "2020-11-26T01:14:15.963Z", "content": "

Wow, awesome results with a single model. Waiting to read your more detailed summary, interesting the list of changes that you applied to that model.

", "rawMarkdown": "Wow, awesome results with a single model. Waiting to read your more detailed summary, interesting the list of changes that you applied to that model.", "votes": 1 }, { "id": 1091386, "postDate": "2020-11-26T01:15:09.113Z", "content": "

thanks!

\n

\"after a week of training.\" … that's is a long time. i trained my models only for 2 days at most.
\nmaybe i should have trained longer.

\n

are are your history? are you using 10? are you training on chopped data or the original data?

", "rawMarkdown": "thanks!\n\n\"after a week of training.\" ... that's is a long time. i trained my models only for 2 days at most.\nmaybe i should have trained longer.\n\nare are your history? are you using 10? are you training on chopped data or the original data?", "votes": 1, "replies": [ { "id": 1091531, "postDate": "2020-11-26T05:05:16.147Z", "content": "

We trained with min_history=0 and didn't chop the training dataset

", "rawMarkdown": "We trained with min_history=0 and didn't chop the training dataset", "votes": 1 } ] }, { "id": 1091645, "postDate": "2020-11-26T06:55:32.417Z", "content": "

We used the reduced range of history samples at 0, 1, 2, 4, 8 frame offsets.
\nWe trained on the original data but used min_history_frames set to 0 to make sure we are using harder frames for training, especially since low history frames are used in the chopped dataset.

\n

We found using the full dataset helps quite a bit to improve the validation score, but switched to it quite late in the competition.

", "rawMarkdown": "We used the reduced range of history samples at 0, 1, 2, 4, 8 frame offsets.\nWe trained on the original data but used min_history_frames set to 0 to make sure we are using harder frames for training, especially since low history frames are used in the chopped dataset.\n\nWe found using the full dataset helps quite a bit to improve the validation score, but switched to it quite late in the competition.", "votes": 2 } ] }, { "id": 1091787, "postDate": "2020-11-26T09:31:00.780Z", "content": "

Best single model was resnet18 224x224 trained on chopped version of train_full:

\n\n

Final submission was just ensemble of checkpoints from this model.

", "rawMarkdown": "Best single model was resnet18 224x224 trained on chopped version of train_full:\n\n- Valid loss: 12.321\n- LB Public: 12.807\n- LB Private: 11.626\n\nFinal submission was just ensemble of checkpoints from this model.", "votes": 5, "replies": [ { "id": 1091869, "postDate": "2020-11-26T10:52:18.357Z", "content": "

Impressive result for simple resnet18. Looks like simple models are very competitive after all.

", "rawMarkdown": "Impressive result for simple `resnet18`. Looks like simple models are very competitive after all.", "votes": 2 }, { "id": 1091875, "postDate": "2020-11-26T10:54:37.143Z", "content": "

I think the second winning model after EF is ResNet18 alone

", "rawMarkdown": "I think the second winning model after EF is ResNet18 alone" }, { "id": 1091954, "postDate": "2020-11-26T12:18:08.087Z", "content": "

Well done! How many times did you iterate through your chopped dataset? And what history_num_frames did you opt for?

", "rawMarkdown": "Well done! How many times did you iterate through your chopped dataset? And what history_num_frames did you opt for?" }, { "id": 1091974, "postDate": "2020-11-26T12:39:47.950Z", "content": "

The final model was trained using 10 chopped versions of train_full, with history set to 10. I pre generated the rasters and left it training for over a week on a single 2080ti, which was around 15 repeated passes of all chopped versions. I'll perhaps add a summary later today :)

", "rawMarkdown": "The final model was trained using 10 chopped versions of train_full, with history set to 10. I pre generated the rasters and left it training for over a week on a single 2080ti, which was around 15 repeated passes of all chopped versions. I'll perhaps add a summary later today :)", "votes": 1 }, { "id": 1091989, "postDate": "2020-11-26T12:56:56.423Z", "content": "

Interesting, I had found that the models stopped learning after about 12 views of the same image. Did you find any useful augmentation techniques?

", "rawMarkdown": "Interesting, I had found that the models stopped learning after about 12 views of the same image. Did you find any useful augmentation techniques?" } ] }, { "id": 1091396, "postDate": "2020-11-26T01:29:43.917Z", "content": "

Mine is the baseline model Resnet34, batch_size 64, train on 24M samples

\n", "rawMarkdown": "Mine is the baseline model Resnet34, batch_size 64, train on 24M samples\n- train loss: 16.961 (complete training avg), 9.201 (last 200 batches avg)\n- val loss: 18.310\n- LB public: 17.549\n- LB private: 16.981\nLooks like this model is having a hard time on just the validation set :)", "votes": 5, "replies": [ { "id": 1091878, "postDate": "2020-11-26T10:55:09.183Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 1091389, "postDate": "2020-11-26T01:16:09.393Z", "content": "

Interesting. when you say no sample was attended to twice what exactly do you mean by that? How did you implement that? Like a sampler without replacement? Was there anything else special you did to get such low scores? I stuck mostly with resnet18 and tried upping to resnet50, but saw virtually no difference. Maybe down to the sampling

", "rawMarkdown": "Interesting. when you say no sample was attended to twice what exactly do you mean by that? How did you implement that? Like a sampler without replacement? Was there anything else special you did to get such low scores? I stuck mostly with resnet18 and tried upping to resnet50, but saw virtually no difference. Maybe down to the sampling", "votes": 3, "replies": [ { "id": 1091905, "postDate": "2020-11-26T11:15:08.547Z", "content": "

Yes, it's like a sampler without replacement. I generated a permutation with a seed and picked indices from it sequentially. With a seed, - so I can stop and rerun it.

", "rawMarkdown": "Yes, it's like a sampler without replacement. I generated a permutation with a seed and picked indices from it sequentially. With a seed, - so I can stop and rerun it. " } ] }, { "id": 1091879, "postDate": "2020-11-26T10:55:53.067Z", "content": "

Have Anyone tried with GeM Pooling i got a reasonable score in it better than before i had

", "rawMarkdown": "Have Anyone tried with GeM Pooling i got a reasonable score in it better than before i had", "votes": 1, "replies": [ { "id": 1091907, "postDate": "2020-11-26T11:16:10.117Z", "content": "

Yes, our best single model uses GeM

", "rawMarkdown": "Yes, our best single model uses GeM", "votes": 7 }, { "id": 1098643, "postDate": "2020-12-01T18:49:46.220Z", "content": "

I wonder what is the p that you use for GeM pooling?

", "rawMarkdown": "I wonder what is the p that you use for GeM pooling?" }, { "id": 1098779, "postDate": "2020-12-01T21:01:44.470Z", "content": "

Automatically trained, never checked how it ended up

", "rawMarkdown": "Automatically trained, never checked how it ended up" } ] }, { "id": 1091537, "postDate": "2020-11-26T05:12:26.817Z", "content": "

I tried ResNeSt , ResNeXt, MixNet, B0,B3, MobileNetV3, ResNet34,18,50,101 , RegNet, ReXNet, SeResNext,and lot more and totally we had 24 models,
\nBut we didn't have any use of them since we cannot train any of these into a better one

", "rawMarkdown": "I tried ResNeSt , ResNeXt, MixNet, B0,B3, MobileNetV3, ResNet34,18,50,101 , RegNet, ReXNet, SeResNext,and lot more and totally we had 24 models, \nBut we didn't have any use of them since we cannot train any of these into a better one", "votes": 1 }, { "id": 1091453, "postDate": "2020-11-26T03:17:22.673Z", "content": "

Congrats! How did you select 3.44m sample? Is it coming from train zarr or full train zarr?

", "rawMarkdown": "Congrats! How did you select 3.44m sample? Is it coming from train zarr or full train zarr?", "votes": 1, "replies": [ { "id": 1091855, "postDate": "2020-11-26T10:43:59.373Z", "content": "

It is a full train, I will give more info in my writeup, working on it

", "rawMarkdown": "It is a full train, I will give more info in my writeup, working on it" } ] }, { "id": 1091372, "postDate": "2020-11-26T01:00:42.490Z", "content": "

good work!

\n

\"215x64x2500\", how do i interprete the dims?
\nwhat is the image size and history you used?

", "rawMarkdown": "good work!\n\n\"215x64x2500\", how do i interprete the dims?\nwhat is the image size and history you used?", "votes": 2, "replies": [ { "id": 1091374, "postDate": "2020-11-26T01:03:19.823Z", "content": "

I added the dims description to the original post. Regarding the rest of the details, I will write a summary tomorrow! Need to sleep.

", "rawMarkdown": "I added the dims description to the original post. Regarding the rest of the details, I will write a summary tomorrow! Need to sleep.", "votes": 1 }, { "id": 1091382, "postDate": "2020-11-26T01:13:10.377Z", "content": "

Hmm… I thought people usually refer one epoch to one pass of all the training examples.

", "rawMarkdown": "Hmm... I thought people usually refer one epoch to one pass of all the training examples.", "votes": 2 }, { "id": 1091387, "postDate": "2020-11-26T01:15:28.907Z", "content": "

I was always bad with definitions, please excuse. My epoch is 2500 iterations in this case.

", "rawMarkdown": "I was always bad with definitions, please excuse. My epoch is 2500 iterations in this case.", "votes": 1 }, { "id": 1091392, "postDate": "2020-11-26T01:20:01.160Z", "content": "

Nevertheless, it is a very interesting idea! Glad the Mixnet works!
\nNow I got to learn some new models other than old fashion Resnet!

", "rawMarkdown": "Nevertheless, it is a very interesting idea! Glad the Mixnet works! \nNow I got to learn some new models other than old fashion Resnet!", "votes": 2 } ] }, { "id": 1096982, "postDate": "2020-11-30T23:05:44.913Z", "content": "

Hi @zaharch excellent work. Well done

", "rawMarkdown": "Hi @zaharch excellent work. Well done" }, { "id": 1091910, "postDate": "2020-11-26T11:19:16.987Z", "content": "

The main solution for this competition i think need more CPU and High computing

", "rawMarkdown": "The main solution for this competition i think need more CPU and High computing" }, { "id": 1091535, "postDate": "2020-11-26T05:09:37.610Z", "content": "

Look like we get some better score if we subtract 0.009 from transformed points in inference part
\nlike

\n
# convert into world coordinates and compute offsets\n        for idx in range(len(preds)):\n            for mode in range(3):\n                preds[idx, mode, :, :] = transform_points(preds[idx, mode, :, :], world_from_agents[idx]) - centroids[idx][:2]-0.009\n
", "rawMarkdown": "Look like we get some better score if we subtract `0.009` from transformed points in inference part\nlike \n``` \n# convert into world coordinates and compute offsets\n for idx in range(len(preds)):\n for mode in range(3):\n preds[idx, mode, :, :] = transform_points(preds[idx, mode, :, :], world_from_agents[idx]) - centroids[idx][:2]-0.009\n```" } ], "comments": [ { "id": 1091404, "author_name": "Dieter", "author_url": "", "post_date": "2020-11-26T01:56:13.583000", "content": "

we have two single models (b3 & b6), that each would be enough to win. Will talk more about them in our summary

", "votes": 9, "replies": [ { "id": 1091723, "author_name": "fergusoci", "author_url": "", "post_date": "2020-11-26T08:26:22.370000", "content": "

Really interested in seeing how you got efficientnets to work! I fell flat in that regard, and couldn't figure out why they were faring so poorly…

", "votes": 1, "replies": [] }, { "id": 1091738, "author_name": "Miroslav Valan", "author_url": "", "post_date": "2020-11-26T08:41:18.377000", "content": "

@fergusoci in my experience, i found efficientnets to have tendency to converge slower than resnets. could that be the reason, that you didn't train long enough. Anyone with similar experience?

", "votes": 0, "replies": [] }, { "id": 1096593, "author_name": "fergusoci", "author_url": "", "post_date": "2020-11-30T16:16:44.243000", "content": "

@christofhenkel Do you mind if I ask you - how did you adapt your efficientnets to more than 3 input channels? I'm thinking that I might have missed something there. I was using something like:

\n

`
\nmodel = EfficientNet.from_name('eff-b0')

\n

model._conv_stem.in_channels = num_in_channels

\n

conv_stem_weight = torch.cat([model._conv_stem.weight.clone()] * int(math.ceil(num_in_channels/3)), axis=1)

\n

model._conv_stem.weight = torch.nn.Parameter(conv_stem_weight[:, :num_in_channels, :num_in_channels, :num_in_channels], requires_grad=True)
\n``
\n@zaharch Maybe you also know the answer to this, given that you opted for mixnet.

\n

Cheers!

", "votes": 0, "replies": [] }, { "id": 1096608, "author_name": "", "author_url": "", "post_date": "2020-11-30T16:32:33.360000", "content": "", "votes": 0, "replies": [] }, { "id": 1096717, "author_name": "fergusoci", "author_url": "", "post_date": "2020-11-30T18:09:23.963000", "content": "

Actually, I think the issue I may have had was the grad_fn for the conv_stem was not retained from the pretrained model. So effectively the graph was broken at that point. I wonder why it even trained moderately well in that case…

", "votes": 0, "replies": [] }, { "id": 1096732, "author_name": "Dieter", "author_url": "", "post_date": "2020-11-30T18:25:24.193000", "content": "

we used something like this. We used the efficientnet from timm

\n
        elif \"efficientnet\" in cfg[\"model_params\"][\"model_architecture\"]:\n            self.backbone.conv_stem = Conv2dSame(\n                num_in_channels,\n                self.backbone.conv_stem.out_channels,\n                kernel_size=self.backbone.conv_stem.kernel_size,\n                stride=self.backbone.conv_stem.stride,\n                padding=self.backbone.conv_stem.padding,\n                bias=False,\n            )\n
", "votes": 1, "replies": [] }, { "id": 1096741, "author_name": "fergusoci", "author_url": "", "post_date": "2020-11-30T18:30:53.587000", "content": "

Thanks. Yeah, I can see where I went wrong now. Appreciate the help. 👍

", "votes": 0, "replies": [] }, { "id": 1096742, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2020-11-30T18:31:45.943000", "content": "

I am expecting your final solution. what is your diamond solution

", "votes": 0, "replies": [] }, { "id": 1096757, "author_name": "Dieter", "author_url": "", "post_date": "2020-11-30T18:50:11.203000", "content": "

we are still waiting on finalization of the LB…

", "votes": 0, "replies": [] }, { "id": 1096790, "author_name": "nosound", "author_url": "", "post_date": "2020-11-30T19:28:08.227000", "content": "

@christofhenkel , I am curious, why do you wait for the finalization to post your summary?

", "votes": 0, "replies": [] }, { "id": 1097140, "author_name": "ryches", "author_url": "", "post_date": "2020-12-01T00:48:12.447000", "content": "

https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/199226

\n

based on this it seems like it will be quite a while before that is finalized

", "votes": 0, "replies": [] } ] }, { "id": 1091376, "author_name": "Dmytro Poplavskiy", "author_url": "", "post_date": "2020-11-26T01:07:32.150000", "content": "

The best single model for me was the xception41 from timm, trained on 224x224 resolution on the full dataset. I have not tried to submit it but on validation it reached around 10.37 after a week of training.

\n

The few days earlier checkpoint of the same model scored 11.1 on public LB, 10.2 on private and 11.1 on validation. The validation score was still dropping, but we switched to the full dataset only a week or so before deadline. Some heavier models seems to reach the lower score with the same number of epochs, but was slower to train.

", "votes": 7, "replies": [ { "id": 1091385, "author_name": "nosound", "author_url": "", "post_date": "2020-11-26T01:14:15.963000", "content": "

Wow, awesome results with a single model. Waiting to read your more detailed summary, interesting the list of changes that you applied to that model.

", "votes": 1, "replies": [] }, { "id": 1091386, "author_name": "hengck23", "author_url": "", "post_date": "2020-11-26T01:15:09.113000", "content": "

thanks!

\n

\"after a week of training.\" … that's is a long time. i trained my models only for 2 days at most.
\nmaybe i should have trained longer.

\n

are are your history? are you using 10? are you training on chopped data or the original data?

", "votes": 1, "replies": [ { "id": 1091531, "author_name": "Artem.Sanakoev", "author_url": "", "post_date": "2020-11-26T05:05:16.147000", "content": "

We trained with min_history=0 and didn't chop the training dataset

", "votes": 1, "replies": [] } ] }, { "id": 1091645, "author_name": "Dmytro Poplavskiy", "author_url": "", "post_date": "2020-11-26T06:55:32.417000", "content": "

We used the reduced range of history samples at 0, 1, 2, 4, 8 frame offsets.
\nWe trained on the original data but used min_history_frames set to 0 to make sure we are using harder frames for training, especially since low history frames are used in the chopped dataset.

\n

We found using the full dataset helps quite a bit to improve the validation score, but switched to it quite late in the competition.

", "votes": 2, "replies": [] } ] }, { "id": 1091787, "author_name": "Tom Aindow", "author_url": "", "post_date": "2020-11-26T09:31:00.780000", "content": "

Best single model was resnet18 224x224 trained on chopped version of train_full:

\n\n

Final submission was just ensemble of checkpoints from this model.

", "votes": 5, "replies": [ { "id": 1091869, "author_name": "nosound", "author_url": "", "post_date": "2020-11-26T10:52:18.357000", "content": "

Impressive result for simple resnet18. Looks like simple models are very competitive after all.

", "votes": 2, "replies": [] }, { "id": 1091875, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2020-11-26T10:54:37.143000", "content": "

I think the second winning model after EF is ResNet18 alone

", "votes": 0, "replies": [] }, { "id": 1091954, "author_name": "fergusoci", "author_url": "", "post_date": "2020-11-26T12:18:08.087000", "content": "

Well done! How many times did you iterate through your chopped dataset? And what history_num_frames did you opt for?

", "votes": 0, "replies": [] }, { "id": 1091974, "author_name": "Tom Aindow", "author_url": "", "post_date": "2020-11-26T12:39:47.950000", "content": "

The final model was trained using 10 chopped versions of train_full, with history set to 10. I pre generated the rasters and left it training for over a week on a single 2080ti, which was around 15 repeated passes of all chopped versions. I'll perhaps add a summary later today :)

", "votes": 1, "replies": [] }, { "id": 1091989, "author_name": "fergusoci", "author_url": "", "post_date": "2020-11-26T12:56:56.423000", "content": "

Interesting, I had found that the models stopped learning after about 12 views of the same image. Did you find any useful augmentation techniques?

", "votes": 0, "replies": [] } ] }, { "id": 1091396, "author_name": "Louis Yang", "author_url": "", "post_date": "2020-11-26T01:29:43.917000", "content": "

Mine is the baseline model Resnet34, batch_size 64, train on 24M samples

\n", "votes": 5, "replies": [ { "id": 1091878, "author_name": "", "author_url": "", "post_date": "2020-11-26T10:55:09.183000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1091389, "author_name": "ryches", "author_url": "", "post_date": "2020-11-26T01:16:09.393000", "content": "

Interesting. when you say no sample was attended to twice what exactly do you mean by that? How did you implement that? Like a sampler without replacement? Was there anything else special you did to get such low scores? I stuck mostly with resnet18 and tried upping to resnet50, but saw virtually no difference. Maybe down to the sampling

", "votes": 3, "replies": [ { "id": 1091905, "author_name": "nosound", "author_url": "", "post_date": "2020-11-26T11:15:08.547000", "content": "

Yes, it's like a sampler without replacement. I generated a permutation with a seed and picked indices from it sequentially. With a seed, - so I can stop and rerun it.

", "votes": 0, "replies": [] } ] }, { "id": 1091879, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2020-11-26T10:55:53.067000", "content": "

Have Anyone tried with GeM Pooling i got a reasonable score in it better than before i had

", "votes": 1, "replies": [ { "id": 1091907, "author_name": "Psi", "author_url": "", "post_date": "2020-11-26T11:16:10.117000", "content": "

Yes, our best single model uses GeM

", "votes": 7, "replies": [] }, { "id": 1098643, "author_name": "Louis Yang", "author_url": "", "post_date": "2020-12-01T18:49:46.220000", "content": "

I wonder what is the p that you use for GeM pooling?

", "votes": 0, "replies": [] }, { "id": 1098779, "author_name": "Psi", "author_url": "", "post_date": "2020-12-01T21:01:44.470000", "content": "

Automatically trained, never checked how it ended up

", "votes": 0, "replies": [] } ] }, { "id": 1091537, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2020-11-26T05:12:26.817000", "content": "

I tried ResNeSt , ResNeXt, MixNet, B0,B3, MobileNetV3, ResNet34,18,50,101 , RegNet, ReXNet, SeResNext,and lot more and totally we had 24 models,
\nBut we didn't have any use of them since we cannot train any of these into a better one

", "votes": 1, "replies": [] }, { "id": 1091453, "author_name": "Kerem Turgutlu", "author_url": "", "post_date": "2020-11-26T03:17:22.673000", "content": "

Congrats! How did you select 3.44m sample? Is it coming from train zarr or full train zarr?

", "votes": 1, "replies": [ { "id": 1091855, "author_name": "nosound", "author_url": "", "post_date": "2020-11-26T10:43:59.373000", "content": "

It is a full train, I will give more info in my writeup, working on it

", "votes": 0, "replies": [] } ] }, { "id": 1091372, "author_name": "hengck23", "author_url": "", "post_date": "2020-11-26T01:00:42.490000", "content": "

good work!

\n

\"215x64x2500\", how do i interprete the dims?
\nwhat is the image size and history you used?

", "votes": 2, "replies": [ { "id": 1091374, "author_name": "nosound", "author_url": "", "post_date": "2020-11-26T01:03:19.823000", "content": "

I added the dims description to the original post. Regarding the rest of the details, I will write a summary tomorrow! Need to sleep.

", "votes": 1, "replies": [] }, { "id": 1091382, "author_name": "Louis Yang", "author_url": "", "post_date": "2020-11-26T01:13:10.377000", "content": "

Hmm… I thought people usually refer one epoch to one pass of all the training examples.

", "votes": 2, "replies": [] }, { "id": 1091387, "author_name": "nosound", "author_url": "", "post_date": "2020-11-26T01:15:28.907000", "content": "

I was always bad with definitions, please excuse. My epoch is 2500 iterations in this case.

", "votes": 1, "replies": [] }, { "id": 1091392, "author_name": "Louis Yang", "author_url": "", "post_date": "2020-11-26T01:20:01.160000", "content": "

Nevertheless, it is a very interesting idea! Glad the Mixnet works!
\nNow I got to learn some new models other than old fashion Resnet!

", "votes": 2, "replies": [] } ] }, { "id": 1096982, "author_name": "Patrick Uzuwe", "author_url": "", "post_date": "2020-11-30T23:05:44.913000", "content": "

Hi @zaharch excellent work. Well done

", "votes": 0, "replies": [] }, { "id": 1091910, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2020-11-26T11:19:16.987000", "content": "

The main solution for this competition i think need more CPU and High computing

", "votes": 0, "replies": [] }, { "id": 1091535, "author_name": "Mohammed Rizin V K", "author_url": "", "post_date": "2020-11-26T05:09:37.610000", "content": "

Look like we get some better score if we subtract 0.009 from transformed points in inference part
\nlike

\n
# convert into world coordinates and compute offsets\n        for idx in range(len(preds)):\n            for mode in range(3):\n                preds[idx, mode, :, :] = transform_points(preds[idx, mode, :, :], world_from_agents[idx]) - centroids[idx][:2]-0.009\n
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1091365": "What is your best single model?\n\nMine is `mixnet_l` from [timm](https://rwightman.github.io/pytorch-image-models/results/), with standard 3 trajectories output, trained on `215 (epochs) x 64 (batch size) x 2500 (iterations)=34.4m` samples.\n\n- Train loss: 11.48\n- Validation: 12.02\n- LB public: 12.285\n- LB private: 11.484 (13th place)\n\nSlightly more fitted on the training data, which indicates that the training data is not infinite. I made sure that no sample point is attended twice, but the correlations between the samples are still significant which results in this lower training loss. But the validation and the LB were very similar along the way.\n\nNote that `mixnet_l` is the heavy one in the family. It gave me a big boost to move from `mixnet_s` to `mixnet_m` and then to `mixnet_l`, approximately `1` score point per change. This is in contradiction with several discussions on the forum that smaller models are better. Looking forward to learn what other models have worked.\n", "1091404": "we have two single models (b3 & b6), that each would be enough to win. Will talk more about them in our summary", "1091376": "The best single model for me was the xception41 from timm, trained on 224x224 resolution on the full dataset. I have not tried to submit it but on validation it reached around 10.37 after a week of training.\n\nThe few days earlier checkpoint of the same model scored 11.1 on public LB, 10.2 on private and 11.1 on validation. The validation score was still dropping, but we switched to the full dataset only a week or so before deadline. Some heavier models seems to reach the lower score with the same number of epochs, but was slower to train.", "1091787": "Best single model was resnet18 224x224 trained on chopped version of train_full:\n\n- Valid loss: 12.321\n- LB Public: 12.807\n- LB Private: 11.626\n\nFinal submission was just ensemble of checkpoints from this model.", "1091396": "Mine is the baseline model Resnet34, batch_size 64, train on 24M samples\n- train loss: 16.961 (complete training avg), 9.201 (last 200 batches avg)\n- val loss: 18.310\n- LB public: 17.549\n- LB private: 16.981\nLooks like this model is having a hard time on just the validation set :)", "1091389": "Interesting. when you say no sample was attended to twice what exactly do you mean by that? How did you implement that? Like a sampler without replacement? Was there anything else special you did to get such low scores? I stuck mostly with resnet18 and tried upping to resnet50, but saw virtually no difference. Maybe down to the sampling", "1091879": "Have Anyone tried with GeM Pooling i got a reasonable score in it better than before i had", "1091537": "I tried ResNeSt , ResNeXt, MixNet, B0,B3, MobileNetV3, ResNet34,18,50,101 , RegNet, ReXNet, SeResNext,and lot more and totally we had 24 models, \nBut we didn't have any use of them since we cannot train any of these into a better one", "1091453": "Congrats! How did you select 3.44m sample? Is it coming from train zarr or full train zarr?", "1091372": "good work!\n\n\"215x64x2500\", how do i interprete the dims?\nwhat is the image size and history you used?", "1096982": "Hi @zaharch excellent work. Well done", "1091910": "The main solution for this competition i think need more CPU and High computing", "1091535": "Look like we get some better score if we subtract `0.009` from transformed points in inference part\nlike \n``` \n# convert into world coordinates and compute offsets\n for idx in range(len(preds)):\n for mode in range(3):\n preds[idx, mode, :, :] = transform_points(preds[idx, mode, :, :], world_from_agents[idx]) - centroids[idx][:2]-0.009\n```" } }