![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F12da144697967d8007803e8b1bec2e53%2FSelection_207.png?generation=1605919579946411&alt=media\")
As this competition is coming to the end, we start questioning how and what we could have done better.
\nDue to the nature of the data and the problem, the training (even some partial dateset enough to draw some conclusion about the model architecture) takes longer that we are use too,and we did not got the chance to test a lot of ideas.
\nOne of the things that we could still try and play around with now then are a few days left of the competition is use multiple models.
\nComparing with other competition where a vote system (classification) or a normal/weighted mean (regression) theoretically will help, in this case it is a little bit trickier. Averaging two trajectory one where the vehicle will go front and the other one left will result in a diagonal trajectory that can possibly not even be on a street.
\nAlso, if we try a different approach, to use 3 models, take the trajectory with the most degree of confidence from each of them, normalize the confidences to sum 1 and then create a new submission with the newly created a data we have a high risk of pointing the same, almost identical trajectory 3 times.
\nWhat I did not try yet is stacking or blending approaches (build multilevel layers). This is what I will focus in the last days of the competition.
\nDid you had any success with using multiple models or any idea that you did not test yet?
\nWhat are you focusing now in the last days?
Good luck
", "messages": [ { "id": 1085541, "postDate": "2020-11-21T00:46:57.180Z", "content": "
the distribution of LB scores is pretty interesting …
", "rawMarkdown": "\n\nthe distribution of LB scores is pretty interesting ...", "votes": 9, "replies": [ { "id": 1085763, "postDate": "2020-11-21T07:30:48.947Z", "content": "It is interesting to see if this pattern will still be on the private leaderboard.
", "rawMarkdown": "It is interesting to see if this pattern will still be on the private leaderboard. ", "votes": 1 }, { "id": 1086072, "postDate": "2020-11-21T10:42:47.710Z", "content": "looks a little bit like technical analysis in the stock market
\nSo, you found method2 instead of method1, @hengck23 ?
The LB just got even more crazy…
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F18102%2F7f3d99784455f18a14b750a3f4c9f3b2%2FScreenshot%202020-11-23%20at%2023.37.37.png?generation=1606171119312827&alt=media\")
In any other competition I would suspect a leak…
\nGiven the size of the data and all the possible different ways to tackle the problem I hope it is not the case.
Anyway just a few more nights till we find out!
", "rawMarkdown": "The LB just got even more crazy... \n\n\n\nIn any other competition I would suspect a leak...\nGiven the size of the data and all the possible different ways to tackle the problem I hope it is not the case.\n\nAnyway just a few more nights till we find out!", "votes": 7, "replies": [ { "id": 1088785, "postDate": "2020-11-24T00:14:38.657Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1088787, "postDate": "2020-11-24T00:16:43.803Z", "content": "Time to unleash your final model that has been training for a month :)
", "rawMarkdown": "Time to unleash your final model that has been training for a month :)", "votes": 1 }, { "id": 1088810, "postDate": "2020-11-24T00:42:54.203Z", "content": "i don't think it is a leak. if we have used more modes (e.g. 9), low lb score prediction is possible. this means that the data did indeed contain enough information for low score.
", "rawMarkdown": "i don't think it is a leak. if we have used more modes (e.g. 9), low lb score prediction is possible. this means that the data did indeed contain enough information for low score. ", "votes": 3 }, { "id": 1089211, "postDate": "2020-11-24T09:46:46.150Z", "content": "It would be really fun to know how much CO2 emission/electricity is needed to place high in this comp. Both for training the best solution but also for all the experimentation that was needed to obtain a certain score.
", "rawMarkdown": "It would be really fun to know how much CO2 emission/electricity is needed to place high in this comp. Both for training the best solution but also for all the experimentation that was needed to obtain a certain score.", "votes": 3 }, { "id": 1089228, "postDate": "2020-11-24T10:06:20.780Z", "content": "Madness! Can't wait to see the solution…
", "rawMarkdown": "Madness! Can't wait to see the solution..." }, { "id": 1089436, "postDate": "2020-11-24T13:47:51.847Z", "content": "It for sure is a stressful time in the comp, especially for those in the top 5-10…hang in there people, just a bit left!
", "rawMarkdown": "It for sure is a stressful time in the comp, especially for those in the top 5-10...hang in there people, just a bit left!", "votes": 2 } ] }, { "id": 1084861, "postDate": "2020-11-20T13:18:23.973Z", "content": "As this competition is coming to the end, we start questioning how and what we could have done better.
\nDue to the nature of the data and the problem, the training (even some partial dateset enough to draw some conclusion about the model architecture) takes longer that we are use too,and we did not got the chance to test a lot of ideas.
\nOne of the things that we could still try and play around with now then are a few days left of the competition is use multiple models.
\nComparing with other competition where a vote system (classification) or a normal/weighted mean (regression) theoretically will help, in this case it is a little bit trickier. Averaging two trajectory one where the vehicle will go front and the other one left will result in a diagonal trajectory that can possibly not even be on a street.
\nAlso, if we try a different approach, to use 3 models, take the trajectory with the most degree of confidence from each of them, normalize the confidences to sum 1 and then create a new submission with the newly created a data we have a high risk of pointing the same, almost identical trajectory 3 times.
\nWhat I did not try yet is stacking or blending approaches (build multilevel layers). This is what I will focus in the last days of the competition.
\nDid you had any success with using multiple models or any idea that you did not test yet?
\nWhat are you focusing now in the last days?
Good luck
", "rawMarkdown": "As this competition is coming to the end, we start questioning how and what we could have done better. \nDue to the nature of the data and the problem, the training (even some partial dateset enough to draw some conclusion about the model architecture) takes longer that we are use too,and we did not got the chance to test a lot of ideas.\nOne of the things that we could still try and play around with now then are a few days left of the competition is use multiple models.\nComparing with other competition where a vote system (classification) or a normal/weighted mean (regression) theoretically will help, in this case it is a little bit trickier. Averaging two trajectory one where the vehicle will go front and the other one left will result in a diagonal trajectory that can possibly not even be on a street. \nAlso, if we try a different approach, to use 3 models, take the trajectory with the most degree of confidence from each of them, normalize the confidences to sum 1 and then create a new submission with the newly created a data we have a high risk of pointing the same, almost identical trajectory 3 times.\nWhat I did not try yet is stacking or blending approaches (build multilevel layers). This is what I will focus in the last days of the competition.\nDid you had any success with using multiple models or any idea that you did not test yet?\nWhat are you focusing now in the last days?\n\nGood luck ", "votes": 8 }, { "id": 1085071, "postDate": "2020-11-20T16:44:11.357Z", "content": "Me neither. I'm tring summarizing 9 trajectories predicted by single model into 3 modes, but currently it produces much bigger losses than the one by the model originally having exactly 3 modes.
\nAs discussed in other threads, the loss function (log-sum-exp) is like taking min of the each trajectory's L2 loss. The mean minimum loss of 9 trajs are much better than the one of originally-3-modes model (8.0 vs 16.3 on chpped-dev), so I believe this is a promising way, but now I don't have any successful ideas.
Interesting approach @yufuin
", "rawMarkdown": "Interesting approach @yufuin " } ] }, { "id": 1089168, "postDate": "2020-11-24T08:58:29.027Z", "content": "I started this competition with my friend , but since this got a bit tougher ,I had to leave in the middle ;(
", "rawMarkdown": "I started this competition with my friend , but since this got a bit tougher ,I had to leave in the middle ;(" }, { "id": 1089163, "postDate": "2020-11-24T08:52:39.287Z", "content": "Another thing I'm doing is gathering some other information from zarr files.
\nFor example, we can use the label probabilities (CAR, CYCLIST or PEDESTRIAN) from zarr.agents[i][5] and the current velocity from zarr.agents[i][3].
\nHowever, for my models using such information doesn't help to improve dev/test scores…
maybe different agents need different encoders.
", "rawMarkdown": "maybe different agents need different encoders." } ] }, { "id": 1085772, "postDate": "2020-11-21T07:34:42.950Z", "content": "One more thing I had tried was to juggle a little bit with the probabilities. For example, every possible route with a confidence lower than 0.05 or 0.01 will now have the probabily 0 and the remaining difference will be added proportional to the other 2 probabilities. With this approach I had tried to improve the score by increasing the probabilities of the most likely 2 scenarios in the cases where a route has very small chances of being a correct one.
", "rawMarkdown": "One more thing I had tried was to juggle a little bit with the probabilities. For example, every possible route with a confidence lower than 0.05 or 0.01 will now have the probabily 0 and the remaining difference will be added proportional to the other 2 probabilities. With this approach I had tried to improve the score by increasing the probabilities of the most likely 2 scenarios in the cases where a route has very small chances of being a correct one. ", "replies": [ { "id": 1085999, "postDate": "2020-11-21T10:16:28.927Z", "content": "If anything, I would do the opposite. Adding a small number to already high confidences helps them nothing.
", "rawMarkdown": "If anything, I would do the opposite. Adding a small number to already high confidences helps them nothing." }, { "id": 1086006, "postDate": "2020-11-21T10:17:54.333Z", "content": "I also tried this way but got a bad result (2x -> 3x). did u get good result ?
", "rawMarkdown": "I also tried this way but got a bad result (2x -> 3x). did u get good result ? " }, { "id": 1086065, "postDate": "2020-11-21T10:38:15.737Z", "content": "@zaharch
\nWhat I wanted to see in that approach is that in cases where a trajectory is very unlikely, if removing that specific trajectory and increases the confidences of the other will help to improve even a little bit the score (the metric will be better if you estimate a good trajectory with 0.83 compared with an estimation of 0.81).
\nI also tried something similar with you thoughts, to equilibrate a little bit the confidences with a algorithm.
\n@journey Unfortunately, none of the above methodology worked for me
the distribution of LB scores is pretty interesting …
", "votes": 9, "replies": [ { "id": 1085763, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-11-21T07:30:48.947000", "content": "It is interesting to see if this pattern will still be on the private leaderboard.
", "votes": 1, "replies": [] }, { "id": 1086072, "author_name": "Pascal Pfeiffer", "author_url": "", "post_date": "2020-11-21T10:42:47.710000", "content": "looks a little bit like technical analysis in the stock market
\nSo, you found method2 instead of method1, @hengck23 ?
The LB just got even more crazy…
\n
In any other competition I would suspect a leak…
\nGiven the size of the data and all the possible different ways to tackle the problem I hope it is not the case.
Anyway just a few more nights till we find out!
", "votes": 7, "replies": [ { "id": 1088785, "author_name": "", "author_url": "", "post_date": "2020-11-24T00:14:38.657000", "content": "", "votes": 0, "replies": [] }, { "id": 1088787, "author_name": "Louis Yang", "author_url": "", "post_date": "2020-11-24T00:16:43.803000", "content": "Time to unleash your final model that has been training for a month :)
", "votes": 1, "replies": [] }, { "id": 1088810, "author_name": "hengck23", "author_url": "", "post_date": "2020-11-24T00:42:54.203000", "content": "i don't think it is a leak. if we have used more modes (e.g. 9), low lb score prediction is possible. this means that the data did indeed contain enough information for low score.
", "votes": 3, "replies": [] }, { "id": 1089211, "author_name": "Miroslav Valan", "author_url": "", "post_date": "2020-11-24T09:46:46.150000", "content": "It would be really fun to know how much CO2 emission/electricity is needed to place high in this comp. Both for training the best solution but also for all the experimentation that was needed to obtain a certain score.
", "votes": 3, "replies": [] }, { "id": 1089228, "author_name": "Tom Aindow", "author_url": "", "post_date": "2020-11-24T10:06:20.780000", "content": "Madness! Can't wait to see the solution…
", "votes": 0, "replies": [] }, { "id": 1089436, "author_name": "InDSweTrust", "author_url": "", "post_date": "2020-11-24T13:47:51.847000", "content": "It for sure is a stressful time in the comp, especially for those in the top 5-10…hang in there people, just a bit left!
", "votes": 2, "replies": [] } ] }, { "id": 1085071, "author_name": "yufuin", "author_url": "", "post_date": "2020-11-20T16:44:11.357000", "content": "Me neither. I'm tring summarizing 9 trajectories predicted by single model into 3 modes, but currently it produces much bigger losses than the one by the model originally having exactly 3 modes.
\nAs discussed in other threads, the loss function (log-sum-exp) is like taking min of the each trajectory's L2 loss. The mean minimum loss of 9 trajs are much better than the one of originally-3-modes model (8.0 vs 16.3 on chpped-dev), so I believe this is a promising way, but now I don't have any successful ideas.
Interesting approach @yufuin
", "votes": 0, "replies": [] } ] }, { "id": 1089168, "author_name": "Manish Sharma", "author_url": "", "post_date": "2020-11-24T08:58:29.027000", "content": "I started this competition with my friend , but since this got a bit tougher ,I had to leave in the middle ;(
", "votes": 0, "replies": [] }, { "id": 1089163, "author_name": "yufuin", "author_url": "", "post_date": "2020-11-24T08:52:39.287000", "content": "Another thing I'm doing is gathering some other information from zarr files.
\nFor example, we can use the label probabilities (CAR, CYCLIST or PEDESTRIAN) from zarr.agents[i][5] and the current velocity from zarr.agents[i][3].
\nHowever, for my models using such information doesn't help to improve dev/test scores…
maybe different agents need different encoders.
", "votes": 0, "replies": [] } ] }, { "id": 1085772, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-11-21T07:34:42.950000", "content": "One more thing I had tried was to juggle a little bit with the probabilities. For example, every possible route with a confidence lower than 0.05 or 0.01 will now have the probabily 0 and the remaining difference will be added proportional to the other 2 probabilities. With this approach I had tried to improve the score by increasing the probabilities of the most likely 2 scenarios in the cases where a route has very small chances of being a correct one.
", "votes": 0, "replies": [ { "id": 1085999, "author_name": "nosound", "author_url": "", "post_date": "2020-11-21T10:16:28.927000", "content": "If anything, I would do the opposite. Adding a small number to already high confidences helps them nothing.
", "votes": 0, "replies": [] }, { "id": 1086006, "author_name": "JIN", "author_url": "", "post_date": "2020-11-21T10:17:54.333000", "content": "I also tried this way but got a bad result (2x -> 3x). did u get good result ?
", "votes": 0, "replies": [] }, { "id": 1086065, "author_name": "Vlad Vaduva", "author_url": "", "post_date": "2020-11-21T10:38:15.737000", "content": "@zaharch
\nWhat I wanted to see in that approach is that in cases where a trajectory is very unlikely, if removing that specific trajectory and increases the confidences of the other will help to improve even a little bit the score (the metric will be better if you estimate a good trajectory with 0.83 compared with an estimation of 0.81).
\nI also tried something similar with you thoughts, to equilibrate a little bit the confidences with a algorithm.
\n@journey Unfortunately, none of the above methodology worked for me