Congratulations to all the winners and thank the host for this very interesting competition. I have to admit that I did not expect a medal at all at the beginning. At some point I almost gave up, as the training was painfully slow, and I just could not get a reasonable score. This result gives me great motivation to keep trying in the future.
\nForgive me if my terminology does not make sense. Any feedback would be greatly appreciated.
\nTo my understanding, this is not a problem of finding three most likely future routes, which is equivalent to finding the one most likely route, as the second most likely route will always be the most likely route shifted by one nanometre. Instead, this is a problem of finding three routes that could best represent the probability distribution. Ideally, we want to include the less likely, but nonetheless typical routes. The most damage to our score probably will be caused by those less likely but very different routes. Therefore, we want diversity in our predictions, and simple ensemble might not work.
\nMy work is based on the model shared by @corochann here. While this simple approach to achieve multi-mode prediction is surprisingly effective, there is one thing that does not make sense to me, and I have been fighting with this problem most of the time:
\nAt every training step, the coordinates of all three predictions are all pulled towards the ground truth. The confidence of the prediction that is most close to the ground truth is increased, while the confidences for the other two predictions are decreased. Therefore, for the other two predictions that are relatively further from the ground truth, we are decreasing their confidence values (meaning now we think they are less likely) but pushing their coordinates to the ground truth (meaning making them more likely). Although at the early training stage this should not be such a big problem and the model is able to converge, I just cannot believe it can converge to an optimal point.
\nOther models such as classification models or NLP models would not have this problem as the target possibilities are fixed and finite. Here we are basically assigning confidence values to moving targets.
I thought about several solutions:
\nSo, my final solution might seem surprisingly simple to most people. I just used the good old Resnet18, a very small image setting of 150x150 with only 5 history frames, which enabled me to fit in a batch of 512 samples into my 8G VRAM. The optimizer is again good old Adam, with a learning rate starting from 0.0001 and reduce by half every 50000 steps. I trained on the full dataset, not because I think I need so much data, but to mitigate the problem of overlapping samples as discussed here.
\nI trained 400k steps, which intermittently took me more than 15 days!!! After this, my computer and I were so exhausted, so we did not try other models or optimizers.
\n(I tried accumulating gradients to increase the effect batch size but again the training was too slow, and the early result was not fantastic, possibly because of the incompatibility with Batch Normalization.)
\nI guess, if we could make the batch size even bigger, and image size a little larger, train longer, or maybe use a more sophisticated model, there is potential to further improve the score significantly.
\nBy the way, I have never really solved the problem of deviation between my training loss and validation loss. After removing some problematic parts of my model and setting the min future and history frames in alignment with the validation dataset, the problem was only partly solved. My training loss has reached below 10 but validation loss was never below 12.10. This is not too bad, but I know some of you get much better alignment. How did you guys get the scores aligned?
", "messages": [ { "id": "1091588", "postDate": "11/26/2020 06:01:12", "content": "Congratulations to all the winners and thank the host for this very interesting competition. I have to admit that I did not expect a medal at all at the beginning. At some point I almost gave up, as the training was painfully slow, and I just could not get a reasonable score. This result gives me great motivation to keep trying in the future.
\nForgive me if my terminology does not make sense. Any feedback would be greatly appreciated.
\nTo my understanding, this is not a problem of finding three most likely future routes, which is equivalent to finding the one most likely route, as the second most likely route will always be the most likely route shifted by one nanometre. Instead, this is a problem of finding three routes that could best represent the probability distribution. Ideally, we want to include the less likely, but nonetheless typical routes. The most damage to our score probably will be caused by those less likely but very different routes. Therefore, we want diversity in our predictions, and simple ensemble might not work.
\nMy work is based on the model shared by @corochann here. While this simple approach to achieve multi-mode prediction is surprisingly effective, there is one thing that does not make sense to me, and I have been fighting with this problem most of the time:
\nAt every training step, the coordinates of all three predictions are all pulled towards the ground truth. The confidence of the prediction that is most close to the ground truth is increased, while the confidences for the other two predictions are decreased. Therefore, for the other two predictions that are relatively further from the ground truth, we are decreasing their confidence values (meaning now we think they are less likely) but pushing their coordinates to the ground truth (meaning making them more likely). Although at the early training stage this should not be such a big problem and the model is able to converge, I just cannot believe it can converge to an optimal point.
\nOther models such as classification models or NLP models would not have this problem as the target possibilities are fixed and finite. Here we are basically assigning confidence values to moving targets.
I thought about several solutions:
\nSo, my final solution might seem surprisingly simple to most people. I just used the good old Resnet18, a very small image setting of 150x150 with only 5 history frames, which enabled me to fit in a batch of 512 samples into my 8G VRAM. The optimizer is again good old Adam, with a learning rate starting from 0.0001 and reduce by half every 50000 steps. I trained on the full dataset, not because I think I need so much data, but to mitigate the problem of overlapping samples as discussed here.
\nI trained 400k steps, which intermittently took me more than 15 days!!! After this, my computer and I were so exhausted, so we did not try other models or optimizers.
\n(I tried accumulating gradients to increase the effect batch size but again the training was too slow, and the early result was not fantastic, possibly because of the incompatibility with Batch Normalization.)
\nI guess, if we could make the batch size even bigger, and image size a little larger, train longer, or maybe use a more sophisticated model, there is potential to further improve the score significantly.
\nBy the way, I have never really solved the problem of deviation between my training loss and validation loss. After removing some problematic parts of my model and setting the min future and history frames in alignment with the validation dataset, the problem was only partly solved. My training loss has reached below 10 but validation loss was never below 12.10. This is not too bad, but I know some of you get much better alignment. How did you guys get the scores aligned?
", "rawMarkdown": "Congratulations to all the winners and thank the host for this very interesting competition. I have to admit that I did not expect a medal at all at the beginning. At some point I almost gave up, as the training was painfully slow, and I just could not get a reasonable score. This result gives me great motivation to keep trying in the future.\n\nForgive me if my terminology does not make sense. Any feedback would be greatly appreciated. \n\nTo my understanding, this is not a problem of finding three most likely future routes, which is equivalent to finding the one most likely route, as the second most likely route will always be the most likely route shifted by one nanometre. Instead, this is a problem of finding three routes that could best represent the probability distribution. Ideally, we want to include the less likely, but nonetheless typical routes. The most damage to our score probably will be caused by those less likely but very different routes. Therefore, we want diversity in our predictions, and simple ensemble might not work. \n\nMy work is based on the model shared by @corochann [here](https://www.kaggle.com/corochann/lyft-training-with-multi-mode-confidence). While this simple approach to achieve multi-mode prediction is surprisingly effective, there is one thing that does not make sense to me, and I have been fighting with this problem most of the time: \n\nAt every training step, the coordinates of all three predictions are all pulled towards the ground truth. The confidence of the prediction that is most close to the ground truth is increased, while the confidences for the other two predictions are decreased. Therefore, for the other two predictions that are relatively further from the ground truth, we are decreasing their confidence values (meaning now we think they are less likely) but pushing their coordinates to the ground truth (meaning making them more likely). Although at the early training stage this should not be such a big problem and the model is able to converge, I just cannot believe it can converge to an optimal point. \nOther models such as classification models or NLP models would not have this problem as the target possibilities are fixed and finite. Here we are basically assigning confidence values to moving targets. \n\nI thought about several solutions: \n- I constructed a “diversity” factor in the loss function. It is basically the average distance between three predictions. By adding this to the loss I was hoping I could gently push three predictions away from each other, reduce the effect of three predictions being pulled together. However, my experiments were of no success. The model either totally ignored this factor or used this factor as the only way to gain lower loss. I did not try many times because every experiment took too long. \n- Make the coordinate space discrete and finite and assign a confidence value to every possible point in this space at each step. Then generate randomly many possible future routes. Finally do a k-means clustering to cluster the routes into three groups and take the centre of each group as the final prediction. I did not even finish the implementation of this idea as the computer power needed would be out of my reach. \n- Very large batch size. It was until very late into the competition I suddenly realized that maybe increasing the batch size could mitigate (certainly not resolve) this problem. By letting the model see as many future possibilities as possible at each step, the model might learn to maintain the diversity of its three predictio[](url)ns. It indeed worked, although to be honest I am not sure it was only because of the problem I mentioned above. \n\nSo, my final solution might seem surprisingly simple to most people. I just used the good old Resnet18, a very small image setting of 150x150 with only 5 history frames, which enabled me to fit in a batch of 512 samples into my 8G VRAM. The optimizer is again good old Adam, with a learning rate starting from 0.0001 and reduce by half every 50000 steps. I trained on the full dataset, not because I think I need so much data, but to mitigate the problem of overlapping samples as discussed [here](https://www.kaggle.com/c/lyft-motion-prediction-autonomous-vehicles/discussion/185762). \n\nI trained 400k steps, which intermittently took me more than 15 days!!! After this, my computer and I were so exhausted, so we did not try other models or optimizers. \n\n(I tried accumulating gradients to increase the effect batch size but again the training was too slow, and the early result was not fantastic, possibly because of the incompatibility with Batch Normalization.) \n\nI guess, if we could make the batch size even bigger, and image size a little larger, train longer, or maybe use a more sophisticated model, there is potential to further improve the score significantly. \n\nBy the way, I have never really solved the problem of deviation between my training loss and validation loss. After removing some problematic parts of my model and setting the min future and history frames in alignment with the validation dataset, the problem was only partly solved. My training loss has reached below 10 but validation loss was never below 12.10. This is not too bad, but I know some of you get much better alignment. How did you guys get the scores aligned?", "votes": null }, { "id": "1091593", "postDate": "11/26/2020 06:05:37", "content": "Interesting. So you think the main thing was a large batch size that contributed to better results?
", "rawMarkdown": "Interesting. So you think the main thing was a large batch size that contributed to better results?", "votes": null }, { "id": "1091598", "postDate": "11/26/2020 06:08:08", "content": "That was my theory, and it did help, but to be honest I am not sure if that is the only explanation.
", "rawMarkdown": "That was my theory, and it did help, but to be honest I am not sure if that is the only explanation.", "votes": null }, { "id": "1091601", "postDate": "11/26/2020 06:11:39", "content": "WOW! Batch size 512! That is huge!
", "rawMarkdown": "WOW! Batch size 512! That is huge!", "votes": null }, { "id": "1091603", "postDate": "11/26/2020 06:13:03", "content": "Yeah, I think larger batch size should give you better result in long run. Even though it train slower (since you have less backprop per samples)
", "rawMarkdown": "Yeah, I think larger batch size should give you better result in long run. Even though it train slower (since you have less backprop per samples)", "votes": null }, { "id": "1091618", "postDate": "11/26/2020 06:23:34", "content": "Thanks for sharing. It seems like large batch size is the key factor.
", "rawMarkdown": "Thanks for sharing. It seems like large batch size is the key factor.", "votes": null }, { "id": "1091628", "postDate": "11/26/2020 06:28:45", "content": "I noticed that going from 32 to 16 bs really hurt results. I was also splitting across 2 gpus so my effective batch size was even smaller than that. I guess I should've tried even larger.
", "rawMarkdown": "I noticed that going from 32 to 16 bs really hurt results. I was also splitting across 2 gpus so my effective batch size was even smaller than that. I guess I should've tried even larger.", "votes": null }, { "id": "1091668", "postDate": "11/26/2020 07:28:15", "content": "Same, I tried BS 16 which affected results in a negative way. Did not try anything above 32…honestly, skimming through some recent write ups, I see that experimenting with hyperparams and backbones would have been extremely beneficial :) Most of us probably rushed into implementing complex ideas without getting some \"basics\" right
\n@frankpanxj Congrats on your result and thank you for sharing!
", "rawMarkdown": "Same, I tried BS 16 which affected results in a negative way. Did not try anything above 32...honestly, skimming through some recent write ups, I see that experimenting with hyperparams and backbones would have been extremely beneficial :) Most of us probably rushed into implementing complex ideas without getting some \"basics\" right\n\n@frankpanxj Congrats on your result and thank you for sharing!", "votes": null }, { "id": "1091676", "postDate": "11/26/2020 07:34:18", "content": "I implemented so many papers to no avail. Finding out that batch size might have been my limiting factor is very disappointing. Lesson learned if that is the case.
", "rawMarkdown": "I implemented so many papers to no avail. Finding out that batch size might have been my limiting factor is very disappointing. Lesson learned if that is the case.", "votes": null }, { "id": "1091684", "postDate": "11/26/2020 07:45:35", "content": "We live and we learn….but I can say that regardless of the result, the amount we learned is priceless. Armed with new knowledge and experience, we wont make the same mistakes again in future competitions;)
", "rawMarkdown": "We live and we learn....but I can say that regardless of the result, the amount we learned is priceless. Armed with new knowledge and experience, we wont make the same mistakes again in future competitions;)", "votes": null }, { "id": "1091742", "postDate": "11/26/2020 08:44:42", "content": "Batch size did not matter too much for us I believe, but BS is always an interplay with LR and mostly we used BS >= 64
", "rawMarkdown": "Batch size did not matter too much for us I believe, but BS is always an interplay with LR and mostly we used BS >= 64", "votes": null }, { "id": "1091758", "postDate": "11/26/2020 08:58:38", "content": "@philippsinger what's your default with BS and LR (pick your optimizer).
\nI start with Adam and BS=32 and set LR between 5e-5 to 5e-4 depending on my assumption how valuable are imagenet features for the domain task. Changes in BS and LR lean toward directly proportional
", "rawMarkdown": "philippsinger what's your default with BS and LR (pick your optimizer). \n\nI start with Adam and BS=32 and set LR between 5e-5 to 5e-4 depending on my assumption how valuable are imagenet features for the domain task. Changes in BS and LR lean toward directly proportional", "votes": null }, { "id": "1091765", "postDate": "11/26/2020 09:09:44", "content": "Nevertheless congratulations is good learning. I did come late to the competition and no time to train. I implemented a solution inspired by one of the top contenders in the nuscenes challenge (https://arxiv.org/pdf/2005.02545.pdf). Still training. I got the impression that for this particular competition computational power is important. I don't think I can fit anything beyond 224x224 in my 8GB GPU.
", "rawMarkdown": "Nevertheless congratulations is good learning. I did come late to the competition and no time to train. I implemented a solution inspired by one of the top contenders in the nuscenes challenge (https://arxiv.org/pdf/2005.02545.pdf). Still training. I got the impression that for this particular competition computational power is important. I don't think I can fit anything beyond 224x224 in my 8GB GPU.", "votes": null }, { "id": "1091779", "postDate": "11/26/2020 09:23:49", "content": "Computational power is important and allows you to iterate through a large number of ideas, but in this case that turned out to be not 100% necessary.
\nAs we can see, @frankpanxj was able to achieve a top-tier solution without any impressive hardware, but by thinking \"outside the box\" and trying some seemingly simple things that others wouldn't even think of trying….and waiting patiently for 15 days;)
\nThis is what I really like about this solution.
@ture05 I also implemented the joint multi head attention. Looked reasonably promising but seemed to converge to a similar result to my more vanilla models. Seems I did not train for long enough though so maybe it does eventually converge to a better result.
", "rawMarkdown": "ture05 I also implemented the joint multi head attention. Looked reasonably promising but seemed to converge to a similar result to my more vanilla models. Seems I did not train for long enough though so maybe it does eventually converge to a better result.", "votes": null }, { "id": "1091794", "postDate": "11/26/2020 09:37:53", "content": "It really depends on the problem, but for us a good start here was Adam BS 64 and LR 1e-4. I always use linear or cosine decay with rare exceptions.
", "rawMarkdown": "It really depends on the problem, but for us a good start here was Adam BS 64 and LR 1e-4. I always use linear or cosine decay with rare exceptions.", "votes": null }, { "id": "1091804", "postDate": "11/26/2020 09:51:00", "content": "Kudos to @frankpanxj definitely computational power and patience is what's needed.
\n@ryches How far did you go? Currently 50k iter ( 64 bs) and it's outperforming the baseline (for the same iterations in training). I did add a few things, more car states and my encoding was slightly different.
", "rawMarkdown": "Kudos to @frankpanxj definitely computational power and patience is what's needed. \n\n@ryches How far did you go? Currently 50k iter ( 64 bs) and it's outperforming the baseline (for the same iterations in training). I did add a few things, more car states and my encoding was slightly different.", "votes": null }, { "id": "1091807", "postDate": "11/26/2020 09:54:28", "content": "I allowed it to run for about 24 hours, not sure exactly the number of samples it was exposed to, but at that point it had converged to 17.5 validation loss.
", "rawMarkdown": "I allowed it to run for about 24 hours, not sure exactly the number of samples it was exposed to, but at that point it had converged to 17.5 validation loss.", "votes": null }, { "id": "1091852", "postDate": "11/26/2020 10:39:49", "content": "Thanks for sharing. And, congrats.
\nCould you tell us how your loss was getting down? Even if you have log of loss, please show us.
\nOur team might give up training too early…
Thanks for your kind words @indswetrust 😂. The only reason I could hold out that long was just because I was observing steadly decreasing loss every day, so there is no reason to stop.
", "rawMarkdown": "Thanks for your kind words @indswetrust 😂. The only reason I could hold out that long was just because I was observing steadly decreasing loss every day, so there is no reason to stop.", "votes": null }, { "id": "1091933", "postDate": "11/26/2020 11:46:31", "content": "iter training_loss (avg) eval_loss
\n10000 23.5 27.526
\n20000 20.01 23.999
\n30000 16.69 20.397
\n40000 17.12 18.814
\n50000 15.25 17.781
\n60000 13.1 15.989
\n70000 13.93 15.538
\n80000 14.29 15.311
\n90000 12.74 15.17
\n100000 12.38 14.593
\n110000 12.54 14.021
\n120000 11.53 13.705
\n130000 12.41 13.868
\n140000 11.81 13.485
\n150000 10.79 13.321
\n160000 10.92 12.983
\n170000 10.81 12.987
\n180000 10.98 13.04
\n190000 11.83 12.893
\n200000 10.94 12.806
\n210000 10.7 12.63
\n220000 10.48 12.666
\n230000 10.6 12.527
\n240000 No data, the machine shut down for no reason 😂
\n250000 10.43 12.555
\n260000 10.46 12.409
\n270000 10.69 12.426
\n280000 10.35 12.376
\n290000 10.55 12.347
\n300000 9.92 12.342
\n310000 11.07 12.314
\n320000 10.55 12.3
\n330000 9.94 12.317
\nhere I started the second round just before the competition deadline, restarted from lr of 0.00001 (drop by half every 20000 steps)
\n340000 10.67 12.465
\n350000 11.33 12.385
\n360000 10.1 12.352
\n370000 10.12 12.295
\n380000 10.83 12.269
\n390000 10.29 12.232
\nduring the final steps I checkpointed every 3000 steps
\n393000 9.98 12.2
\n396000 10.38 12.21
\n399000 10.5 12.189
\n402000 10.66 12.156
\n405000 10.25 12.241
\n408000 10.72 12.168
Congrats and thanks for mention!
\nWe also discussed your 2nd thought of predicting probability distribution directly and assign 3 trajectory in later step, which may be an interesting approach for future!
Thanks!
\nGreat patience you have.
Interesting. So you think the main thing was a large batch size that contributed to better results?
", "votes": null, "replies": [ { "id": 1091598, "author_name": "frankpanxj", "author_url": "", "post_date": "11/26/2020 06:08:08", "content": "That was my theory, and it did help, but to be honest I am not sure if that is the only explanation.
", "votes": null, "replies": [] }, { "id": 1091603, "author_name": "louis925", "author_url": "", "post_date": "11/26/2020 06:13:03", "content": "Yeah, I think larger batch size should give you better result in long run. Even though it train slower (since you have less backprop per samples)
", "votes": null, "replies": [] }, { "id": 1091628, "author_name": "ryches", "author_url": "", "post_date": "11/26/2020 06:28:45", "content": "I noticed that going from 32 to 16 bs really hurt results. I was also splitting across 2 gpus so my effective batch size was even smaller than that. I guess I should've tried even larger.
", "votes": null, "replies": [] }, { "id": 1091668, "author_name": "indswetrust", "author_url": "", "post_date": "11/26/2020 07:28:15", "content": "Same, I tried BS 16 which affected results in a negative way. Did not try anything above 32…honestly, skimming through some recent write ups, I see that experimenting with hyperparams and backbones would have been extremely beneficial :) Most of us probably rushed into implementing complex ideas without getting some \"basics\" right
\n@frankpanxj Congrats on your result and thank you for sharing!
", "votes": null, "replies": [] }, { "id": 1091676, "author_name": "ryches", "author_url": "", "post_date": "11/26/2020 07:34:18", "content": "I implemented so many papers to no avail. Finding out that batch size might have been my limiting factor is very disappointing. Lesson learned if that is the case.
", "votes": null, "replies": [] }, { "id": 1091684, "author_name": "indswetrust", "author_url": "", "post_date": "11/26/2020 07:45:35", "content": "We live and we learn….but I can say that regardless of the result, the amount we learned is priceless. Armed with new knowledge and experience, we wont make the same mistakes again in future competitions;)
", "votes": null, "replies": [] }, { "id": 1091742, "author_name": "philippsinger", "author_url": "", "post_date": "11/26/2020 08:44:42", "content": "Batch size did not matter too much for us I believe, but BS is always an interplay with LR and mostly we used BS >= 64
", "votes": null, "replies": [] }, { "id": 1091758, "author_name": "valanm", "author_url": "", "post_date": "11/26/2020 08:58:38", "content": "@philippsinger what's your default with BS and LR (pick your optimizer).
\nI start with Adam and BS=32 and set LR between 5e-5 to 5e-4 depending on my assumption how valuable are imagenet features for the domain task. Changes in BS and LR lean toward directly proportional
", "votes": null, "replies": [] }, { "id": 1091794, "author_name": "philippsinger", "author_url": "", "post_date": "11/26/2020 09:37:53", "content": "It really depends on the problem, but for us a good start here was Adam BS 64 and LR 1e-4. I always use linear or cosine decay with rare exceptions.
", "votes": null, "replies": [] } ] }, { "id": 1091601, "author_name": "louis925", "author_url": "", "post_date": "11/26/2020 06:11:39", "content": "WOW! Batch size 512! That is huge!
", "votes": null, "replies": [] }, { "id": 1091618, "author_name": "sunghyunjun", "author_url": "", "post_date": "11/26/2020 06:23:34", "content": "Thanks for sharing. It seems like large batch size is the key factor.
", "votes": null, "replies": [] }, { "id": 1091765, "author_name": "ture05", "author_url": "", "post_date": "11/26/2020 09:09:44", "content": "Nevertheless congratulations is good learning. I did come late to the competition and no time to train. I implemented a solution inspired by one of the top contenders in the nuscenes challenge (https://arxiv.org/pdf/2005.02545.pdf). Still training. I got the impression that for this particular competition computational power is important. I don't think I can fit anything beyond 224x224 in my 8GB GPU.
", "votes": null, "replies": [ { "id": 1091779, "author_name": "indswetrust", "author_url": "", "post_date": "11/26/2020 09:23:49", "content": "Computational power is important and allows you to iterate through a large number of ideas, but in this case that turned out to be not 100% necessary.
\nAs we can see, @frankpanxj was able to achieve a top-tier solution without any impressive hardware, but by thinking \"outside the box\" and trying some seemingly simple things that others wouldn't even think of trying….and waiting patiently for 15 days;)
\nThis is what I really like about this solution.
@ture05 I also implemented the joint multi head attention. Looked reasonably promising but seemed to converge to a similar result to my more vanilla models. Seems I did not train for long enough though so maybe it does eventually converge to a better result.
", "votes": null, "replies": [] }, { "id": 1091929, "author_name": "frankpanxj", "author_url": "", "post_date": "11/26/2020 11:41:56", "content": "Thanks for your kind words @indswetrust 😂. The only reason I could hold out that long was just because I was observing steadly decreasing loss every day, so there is no reason to stop.
", "votes": null, "replies": [] } ] }, { "id": 1091804, "author_name": "ture05", "author_url": "", "post_date": "11/26/2020 09:51:00", "content": "Kudos to @frankpanxj definitely computational power and patience is what's needed.
\n@ryches How far did you go? Currently 50k iter ( 64 bs) and it's outperforming the baseline (for the same iterations in training). I did add a few things, more car states and my encoding was slightly different.
", "votes": null, "replies": [ { "id": 1091807, "author_name": "ryches", "author_url": "", "post_date": "11/26/2020 09:54:28", "content": "I allowed it to run for about 24 hours, not sure exactly the number of samples it was exposed to, but at that point it had converged to 17.5 validation loss.
", "votes": null, "replies": [] } ] }, { "id": 1091852, "author_name": "yasagure", "author_url": "", "post_date": "11/26/2020 10:39:49", "content": "Thanks for sharing. And, congrats.
\nCould you tell us how your loss was getting down? Even if you have log of loss, please show us.
\nOur team might give up training too early…
iter training_loss (avg) eval_loss
\n10000 23.5 27.526
\n20000 20.01 23.999
\n30000 16.69 20.397
\n40000 17.12 18.814
\n50000 15.25 17.781
\n60000 13.1 15.989
\n70000 13.93 15.538
\n80000 14.29 15.311
\n90000 12.74 15.17
\n100000 12.38 14.593
\n110000 12.54 14.021
\n120000 11.53 13.705
\n130000 12.41 13.868
\n140000 11.81 13.485
\n150000 10.79 13.321
\n160000 10.92 12.983
\n170000 10.81 12.987
\n180000 10.98 13.04
\n190000 11.83 12.893
\n200000 10.94 12.806
\n210000 10.7 12.63
\n220000 10.48 12.666
\n230000 10.6 12.527
\n240000 No data, the machine shut down for no reason 😂
\n250000 10.43 12.555
\n260000 10.46 12.409
\n270000 10.69 12.426
\n280000 10.35 12.376
\n290000 10.55 12.347
\n300000 9.92 12.342
\n310000 11.07 12.314
\n320000 10.55 12.3
\n330000 9.94 12.317
\nhere I started the second round just before the competition deadline, restarted from lr of 0.00001 (drop by half every 20000 steps)
\n340000 10.67 12.465
\n350000 11.33 12.385
\n360000 10.1 12.352
\n370000 10.12 12.295
\n380000 10.83 12.269
\n390000 10.29 12.232
\nduring the final steps I checkpointed every 3000 steps
\n393000 9.98 12.2
\n396000 10.38 12.21
\n399000 10.5 12.189
\n402000 10.66 12.156
\n405000 10.25 12.241
\n408000 10.72 12.168
Thanks!
\nGreat patience you have.
Congrats and thanks for mention!
\nWe also discussed your 2nd thought of predicting probability distribution directly and assign 3 trajectory in later step, which may be an interesting approach for future!