Hi all,
\nNow we all know the importance of post processing. But I'm wondering how do you validate your post processing.
More concretely say you have post processing A and B, how can you tell which method would have better score and how much would it be better without submitting to LB?
", "messages": [ { "id": "1313155", "postDate": "05/18/2021 12:29:01", "content": "Hi all,
\nNow we all know the importance of post processing. But I'm wondering how do you validate your post processing.
More concretely say you have post processing A and B, how can you tell which method would have better score and how much would it be better without submitting to LB?
", "rawMarkdown": "Hi all,\nNow we all know the importance of post processing. But I'm wondering how do you validate your post processing.\n\nMore concretely say you have post processing A and B, how can you tell which method would have better score and how much would it be better **without** submitting to LB?", "votes": null }, { "id": "1313195", "postDate": "05/18/2021 12:48:06", "content": "I used three primary methods:
\nI knew my wifi predictions had about 4-5m error on average, so if the post processing took a path too far from that (say, 8m away), then I'd either average back with wifi, or just reset back to the wifi x,y and start over
If the post processing distorted the predicted path shape too much (delta x,y between waypoints), then my script would automatically average back to be closer to the predicted shape again
I would do 3-4 runs of post processing, and then try to do a \"smart ensemble\", example: if 3 of 4 runs had path centroids near each other and 1 run was far out of alignment, the ensemble would average only the 3 and throw out the (presumably) bad one. Or sometimes, I would sort the runs for each path by which one was closest to the predicted delta x,y, and pick that one, throwing out the others.
Despite all of that, I wish I had a better way to do cross validation on the post processing - I could never really be sure that those 3 methods were working without checking the public LB, and then I was worried about overfitting :)
\nI'll try better CV on post-processing next time!
", "rawMarkdown": "I used three primary methods:\n\n1. I knew my wifi predictions had about 4-5m error on average, so if the post processing took a path too far from that (say, 8m away), then I'd either average back with wifi, or just reset back to the wifi x,y and start over\n\n2. If the post processing distorted the predicted path shape too much (delta x,y between waypoints), then my script would automatically average back to be closer to the predicted shape again\n\n3. I would do 3-4 runs of post processing, and then try to do a \"smart ensemble\", example: if 3 of 4 runs had path centroids near each other and 1 run was far out of alignment, the ensemble would average only the 3 and throw out the (presumably) bad one. Or sometimes, I would sort the runs for each path by which one was closest to the predicted delta x,y, and pick that one, throwing out the others.\n\nDespite all of that, I wish I had a better way to do cross validation on the post processing - I could never really be sure that those 3 methods were working without checking the public LB, and then I was worried about overfitting :)\n\nI'll try better CV on post-processing next time!", "votes": null }, { "id": "1313305", "postDate": "05/18/2021 13:58:10", "content": "The txt size of the test set>=2M (look in linux), the time span>=60s, and the number of path points>=5. In the first half of the month,I also selected 626 paths as fake test using above restrictions . No matter whether post-processing is performed or not, my fake test scores are consistent with the lb scores.
\nlink
I think I did this the \"correct\" way, but it was a bit tedious and not sure how much value it added, essentially pretending that my holdout set was the submission, and the 80% in-fold set of paths was the training data, and stuck to this quite strictly at each step. Here was my process:
\nFirst, I used the provided github step function to get \"ground truth\" x and y values for every step, and then linear interpolation to get ground truth locations at any time stamp I wanted. Different models used different timestamps, so this was important. (In reading others' solutions, my biggest mistake was putting too much trust in the provided function, I should have modeled these deltas/shapes myself!)
\nNext, I did my base modeling, LSTM at granular time steps, and MLP at regular interval (not wifi block times, but conceptually similar). I then repeated for 80% of the paths, with 20% holdout. (I was planning on doing 5-fold, but it would have been very computationally intensive)
\nThen, at every step in post processing, I passed in two separate datasets, and spit out two datasets, one for sub, and one for val. Importantly, my val file also carried through those interpolated \"ground truth\" values from the first step. So I was able to easily calculate my mean distance and floor accuracy metrics at any point, even at stages where the granularity was different from the final waypoints. At the point where I interpolated my predictions in time to match the submission data, I similarly interpolated my val set to match actual waypoints, and switched over to using those known waypoints to calculate metrics.
\nOne tricky part was that several of these post-processing steps also required two separate inputs for sub vs val. For example, for \"snap to grid\", I had two sets of \"training\" waypoints, where the val version was generated from only the out of fold paths. Similarly, I calculated the leaky start/end points separately for the val set, only using the 80%.
\nWhile tedious, this worked pretty well for me, and always seemed pretty correlated to the public LB.
", "rawMarkdown": "I think I did this the \"correct\" way, but it was a bit tedious and not sure how much value it added, essentially pretending that my holdout set was the submission, and the 80% in-fold set of paths was the training data, and stuck to this quite strictly at each step. Here was my process:\n\nFirst, I used the provided github step function to get \"ground truth\" x and y values for every step, and then linear interpolation to get ground truth locations at any time stamp I wanted. Different models used different timestamps, so this was important. (In reading others' solutions, my biggest mistake was putting too much trust in the provided function, I should have modeled these deltas/shapes myself!)\n\nNext, I did my base modeling, LSTM at granular time steps, and MLP at regular interval (not wifi block times, but conceptually similar). I then repeated for 80% of the paths, with 20% holdout. (I was planning on doing 5-fold, but it would have been very computationally intensive)\n\nThen, at every step in post processing, I passed in two separate datasets, and spit out two datasets, one for sub, and one for val. Importantly, my val file also carried through those interpolated \"ground truth\" values from the first step. So I was able to easily calculate my mean distance and floor accuracy metrics at any point, even at stages where the granularity was different from the final waypoints. At the point where I interpolated my predictions in time to match the submission data, I similarly interpolated my val set to match actual waypoints, and switched over to using those known waypoints to calculate metrics.\n\nOne tricky part was that several of these post-processing steps also required two separate inputs for sub vs val. For example, for \"snap to grid\", I had two sets of \"training\" waypoints, where the val version was generated from only the out of fold paths. Similarly, I calculated the leaky start/end points separately for the val set, only using the 80%.\n\nWhile tedious, this worked pretty well for me, and always seemed pretty correlated to the public LB.", "votes": null }, { "id": "1313337", "postDate": "05/18/2021 14:12:41", "content": "Oh man, this was smart to look at the characteristics of the sub paths. I think it now makes sense why my (and others') cv scores were systematically higher, because we probably have a bunch of short paths that are much harder predict.
", "rawMarkdown": "Oh man, this was smart to look at the characteristics of the sub paths. I think it now makes sense why my (and others') cv scores were systematically higher, because we probably have a bunch of short paths that are much harder predict.", "votes": null }, { "id": "1313348", "postDate": "05/18/2021 14:15:10", "content": "make sense
", "rawMarkdown": "make sense", "votes": null }, { "id": "1314062", "postDate": "05/18/2021 23:27:34", "content": "Thanks for the detailed steps. I'm very impressed by this part \"\"ground truth\" x and y values for every step, and then linear interpolation to get ground truth locations at any time stamp\". I should have done that too.
\nI also agreed on the tricky part some post processing requires other inputs like waypoints which complicates the validation step.
\nVery well done. Thank you!
", "rawMarkdown": "Thanks for the detailed steps. I'm very impressed by this part \"\"ground truth\" x and y values for every step, and then linear interpolation to get ground truth locations at any time stamp\". I should have done that too.\n\nI also agreed on the tricky part some post processing requires other inputs like waypoints which complicates the validation step.\n\nVery well done. Thank you!", "votes": null }, { "id": "1314374", "postDate": "05/19/2021 06:18:00", "content": "Thanks for sharing. I should have done this in the early stage of this competition.
", "rawMarkdown": "Thanks for sharing. I should have done this in the early stage of this competition.", "votes": null }, { "id": "1314379", "postDate": "05/19/2021 06:20:28", "content": "Thank you chris for sharing. You have very smart trick so that post processing wouldn't make it worse!
", "rawMarkdown": "Thank you chris for sharing. You have very smart trick so that post processing wouldn't make it worse!", "votes": null }, { "id": "1314951", "postDate": "05/19/2021 13:18:29", "content": "I just applied postprocessing to my validation set and checked the score, and it is correlated with LB.
", "rawMarkdown": "I just applied postprocessing to my validation set and checked the score, and it is correlated with LB.", "votes": null }, { "id": "1315023", "postDate": "05/19/2021 13:56:37", "content": "thank you ， I think that combining strategy of both you and me will be the host's real train&test split method.
", "rawMarkdown": "thank you ， I think that combining strategy of both you and me will be the host's real train&test split method.", "votes": null }, { "id": "1315451", "postDate": "05/19/2021 19:45:17", "content": "Thanks, and congrats to you! My posts started getting way longer when earlier this week I suddenly and mysteriously found myself with way more free time…
", "rawMarkdown": "Thanks, and congrats to you! My posts started getting way longer when earlier this week I suddenly and mysteriously found myself with way more free time...", "votes": null }, { "id": "1315594", "postDate": "05/20/2021 01:12:35", "content": "This is very interesting question and I want to hear answers of other kagglers.
\nI think some post-processing techniques are based on explicit assumptions/hypotheses about test set like there would be a leakage or train and test waypoints would be overlapped. We can infer these hypotheses throughout EDA of train set or train/val split validation, but it would be difficult to validate it without submitting to LB. These techniques would not work if the host split train/test set in different ways. Of course, the elegant solution of your teammate is not the case.
I used three primary methods:
\nI knew my wifi predictions had about 4-5m error on average, so if the post processing took a path too far from that (say, 8m away), then I'd either average back with wifi, or just reset back to the wifi x,y and start over
If the post processing distorted the predicted path shape too much (delta x,y between waypoints), then my script would automatically average back to be closer to the predicted shape again
I would do 3-4 runs of post processing, and then try to do a \"smart ensemble\", example: if 3 of 4 runs had path centroids near each other and 1 run was far out of alignment, the ensemble would average only the 3 and throw out the (presumably) bad one. Or sometimes, I would sort the runs for each path by which one was closest to the predicted delta x,y, and pick that one, throwing out the others.
Despite all of that, I wish I had a better way to do cross validation on the post processing - I could never really be sure that those 3 methods were working without checking the public LB, and then I was worried about overfitting :)
\nI'll try better CV on post-processing next time!
", "votes": null, "replies": [ { "id": 1314379, "author_name": "higepon", "author_url": "", "post_date": "05/19/2021 06:20:28", "content": "Thank you chris for sharing. You have very smart trick so that post processing wouldn't make it worse!
", "votes": null, "replies": [] } ] }, { "id": 1313305, "author_name": "max2020", "author_url": "", "post_date": "05/18/2021 13:58:10", "content": "The txt size of the test set>=2M (look in linux), the time span>=60s, and the number of path points>=5. In the first half of the month,I also selected 626 paths as fake test using above restrictions . No matter whether post-processing is performed or not, my fake test scores are consistent with the lb scores.
\nlink
Oh man, this was smart to look at the characteristics of the sub paths. I think it now makes sense why my (and others') cv scores were systematically higher, because we probably have a bunch of short paths that are much harder predict.
", "votes": null, "replies": [] }, { "id": 1313348, "author_name": "max2020", "author_url": "", "post_date": "05/18/2021 14:15:10", "content": "make sense
", "votes": null, "replies": [] }, { "id": 1314374, "author_name": "higepon", "author_url": "", "post_date": "05/19/2021 06:18:00", "content": "Thanks for sharing. I should have done this in the early stage of this competition.
", "votes": null, "replies": [] } ] }, { "id": 1313323, "author_name": "paulfornia", "author_url": "", "post_date": "05/18/2021 14:08:03", "content": "I think I did this the \"correct\" way, but it was a bit tedious and not sure how much value it added, essentially pretending that my holdout set was the submission, and the 80% in-fold set of paths was the training data, and stuck to this quite strictly at each step. Here was my process:
\nFirst, I used the provided github step function to get \"ground truth\" x and y values for every step, and then linear interpolation to get ground truth locations at any time stamp I wanted. Different models used different timestamps, so this was important. (In reading others' solutions, my biggest mistake was putting too much trust in the provided function, I should have modeled these deltas/shapes myself!)
\nNext, I did my base modeling, LSTM at granular time steps, and MLP at regular interval (not wifi block times, but conceptually similar). I then repeated for 80% of the paths, with 20% holdout. (I was planning on doing 5-fold, but it would have been very computationally intensive)
\nThen, at every step in post processing, I passed in two separate datasets, and spit out two datasets, one for sub, and one for val. Importantly, my val file also carried through those interpolated \"ground truth\" values from the first step. So I was able to easily calculate my mean distance and floor accuracy metrics at any point, even at stages where the granularity was different from the final waypoints. At the point where I interpolated my predictions in time to match the submission data, I similarly interpolated my val set to match actual waypoints, and switched over to using those known waypoints to calculate metrics.
\nOne tricky part was that several of these post-processing steps also required two separate inputs for sub vs val. For example, for \"snap to grid\", I had two sets of \"training\" waypoints, where the val version was generated from only the out of fold paths. Similarly, I calculated the leaky start/end points separately for the val set, only using the 80%.
\nWhile tedious, this worked pretty well for me, and always seemed pretty correlated to the public LB.
", "votes": null, "replies": [ { "id": 1314062, "author_name": "higepon", "author_url": "", "post_date": "05/18/2021 23:27:34", "content": "Thanks for the detailed steps. I'm very impressed by this part \"\"ground truth\" x and y values for every step, and then linear interpolation to get ground truth locations at any time stamp\". I should have done that too.
\nI also agreed on the tricky part some post processing requires other inputs like waypoints which complicates the validation step.
\nVery well done. Thank you!
", "votes": null, "replies": [] }, { "id": 1315451, "author_name": "paulfornia", "author_url": "", "post_date": "05/19/2021 19:45:17", "content": "Thanks, and congrats to you! My posts started getting way longer when earlier this week I suddenly and mysteriously found myself with way more free time…
", "votes": null, "replies": [] } ] }, { "id": 1314951, "author_name": "mamasinkgs", "author_url": "", "post_date": "05/19/2021 13:18:29", "content": "I just applied postprocessing to my validation set and checked the score, and it is correlated with LB.
", "votes": null, "replies": [ { "id": 1315023, "author_name": "max2020", "author_url": "", "post_date": "05/19/2021 13:56:37", "content": "thank you ， I think that combining strategy of both you and me will be the host's real train&test split method.
", "votes": null, "replies": [] } ] }, { "id": 1315594, "author_name": "tomooinubushi", "author_url": "", "post_date": "05/20/2021 01:12:35", "content": "This is very interesting question and I want to hear answers of other kagglers.
\nI think some post-processing techniques are based on explicit assumptions/hypotheses about test set like there would be a leakage or train and test waypoints would be overlapped. We can infer these hypotheses throughout EDA of train set or train/val split validation, but it would be difficult to validate it without submitting to LB. These techniques would not work if the host split train/test set in different ways. Of course, the elegant solution of your teammate is not the case.