As it has been said already, at inference time we don't have the target values for the current task container for each user.
\nSo far, I neglected this detail and in my feature engineering I update the target after each question. My CV is higher than my LB, but there is very good correlation.
Inspired by what @mamasinkgs wrote in another post, I decided to remove this leakage and make sure I update the target after each task_container_id, rather than after each question_id. My CV is now spot on wrt LB, but LB is lower.
\nDo people have similar experience with that?
", "messages": [ { "id": "1136779", "postDate": "01/03/2021 12:23:28", "content": "As it has been said already, at inference time we don't have the target values for the current task container for each user.
\nSo far, I neglected this detail and in my feature engineering I update the target after each question. My CV is higher than my LB, but there is very good correlation.
Inspired by what @mamasinkgs wrote in another post, I decided to remove this leakage and make sure I update the target after each task_container_id, rather than after each question_id. My CV is now spot on wrt LB, but LB is lower.
\nDo people have similar experience with that?
", "rawMarkdown": "As it has been said already, at inference time we don't have the target values for the current task container for each user. \nSo far, I neglected this detail and in my feature engineering I update the target after each question. My CV is higher than my LB, but there is very good correlation.\n\nInspired by what @mamasinkgs wrote in another post, I decided to remove this leakage and make sure I update the target after each task_container_id, rather than after each question_id. My CV is now spot on wrt LB, but LB is lower.\n\nDo people have similar experience with that?", "votes": null }, { "id": "1136811", "postDate": "01/03/2021 13:02:05", "content": "I experienced quite the opposite, it gave me a small bump in my LB score.
", "rawMarkdown": "I experienced quite the opposite, it gave me a small bump in my LB score.", "votes": null }, { "id": "1136814", "postDate": "01/03/2021 13:03:42", "content": "cool, thx. I must be doing something wrong with the implementation.
", "rawMarkdown": "cool, thx. I must be doing something wrong with the implementation.", "votes": null }, { "id": "1136866", "postDate": "01/03/2021 13:55:18", "content": "During the time I attempted to make this change to my training pipeline, I ended up discovering a myriad of issues with my data preparation. The result of which after they were all patched was my cv was considerably less. But due to aforementioned issues, I can't definitively attribute it to the leakage or the other problems that were corrected.
", "rawMarkdown": "During the time I attempted to make this change to my training pipeline, I ended up discovering a myriad of issues with my data preparation. The result of which after they were all patched was my cv was considerably less. But due to aforementioned issues, I can't definitively attribute it to the leakage or the other problems that were corrected.", "votes": null }, { "id": "1136872", "postDate": "01/03/2021 13:59:58", "content": "What about your LB?
\nI expect my CV to go down \"by construction\" after fixing this leakage, but I was expecting my LB to go up instead…
I haven't made a LB submission with any of my own models.
", "rawMarkdown": "I haven't made a LB submission with any of my own models.", "votes": null }, { "id": "1140426", "postDate": "01/06/2021 02:18:05", "content": "Never!!!
\nIt is dangerous. The shake game is not fun at all.🤥🤥🤥
I experienced quite the opposite, it gave me a small bump in my LB score.
", "votes": null, "replies": [ { "id": 1136814, "author_name": "stecasasso", "author_url": "", "post_date": "01/03/2021 13:03:42", "content": "cool, thx. I must be doing something wrong with the implementation.
", "votes": null, "replies": [] } ] }, { "id": 1136866, "author_name": "authman", "author_url": "", "post_date": "01/03/2021 13:55:18", "content": "During the time I attempted to make this change to my training pipeline, I ended up discovering a myriad of issues with my data preparation. The result of which after they were all patched was my cv was considerably less. But due to aforementioned issues, I can't definitively attribute it to the leakage or the other problems that were corrected.
", "votes": null, "replies": [ { "id": 1136872, "author_name": "stecasasso", "author_url": "", "post_date": "01/03/2021 13:59:58", "content": "What about your LB?
\nI expect my CV to go down \"by construction\" after fixing this leakage, but I was expecting my LB to go up instead…
I haven't made a LB submission with any of my own models.
", "votes": null, "replies": [] } ] }, { "id": 1140426, "author_name": "xiaowangiiiii", "author_url": "", "post_date": "01/06/2021 02:18:05", "content": "Never!!!
\nIt is dangerous. The shake game is not fun at all.🤥🤥🤥