![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1161354%2F837bee55a510aed5293ba08e8be31d51%2FSans%20titre.png?generation=1609089826464525&alt=media\")
Hey all !
\nSo I see a lot of people struggeling with the loop feature engineering, I would like to share with you my personnal pipeline that help me avoiding leakages and bugs for submission.
\nMy pipeline is based on the train/test df as well as two cache files :
\nMy pipeline below:
\n
In the general sequence, I loop through the row of the train/test set and add the relevant informations from my two caches to build the feature matrix used to train my model. Two things to note:
\nGood luck all for the last few days!
", "messages": [ { "id": 1128741, "postDate": "2020-12-27T17:27:54.863Z", "content": "Hey all !
\nSo I see a lot of people struggeling with the loop feature engineering, I would like to share with you my personnal pipeline that help me avoiding leakages and bugs for submission.
\nMy pipeline is based on the train/test df as well as two cache files :
\nMy pipeline below:
\n
In the general sequence, I loop through the row of the train/test set and add the relevant informations from my two caches to build the feature matrix used to train my model. Two things to note:
\nGood luck all for the last few days!
", "rawMarkdown": "Hey all !\n\nSo I see a lot of people struggeling with the loop feature engineering, I would like to share with you my personnal pipeline that help me avoiding leakages and bugs for submission.\n\nMy pipeline is based on the train/test df as well as two cache files :\n- question_cache that is precomputed using the full history and gather information about questions (difficulty, etc...)\n- user_cache that is computed iterativaly to avoid any time leakage.\n\nMy pipeline below:\n\n\n\nIn the general sequence, I loop through the row of the train/test set and add the relevant informations from my two caches to build the feature matrix used to train my model. Two things to note:\n- It is important to create the feature matrix BEFORE updating the user_cache to avoid leakage of information not yet seen (answered_correctly for example)\n- It is very important to consider the questions by batches, as the data is served like this through the API. To make sure there is no batch leaks, I include in my pipeline a temporary file where I store the rows from a unique batch. Only when the batch change, I iterate over this template to first create all the row in the feature matrix and then only update the user_cache.\n- Another very important point: Before starting the predictions, you shall precompute the user_cache using the full train df. A lot of information is available from users from the test set in the train set. I was initially not doing it and was stuck in the 0.77X area. This allowed me to jump in the 0.78X one.\n\nGood luck all for the last few days!", "votes": 10 } ], "comments": [], "raw_markdown_by_id": { "1128741": "Hey all !\n\nSo I see a lot of people struggeling with the loop feature engineering, I would like to share with you my personnal pipeline that help me avoiding leakages and bugs for submission.\n\nMy pipeline is based on the train/test df as well as two cache files :\n- question_cache that is precomputed using the full history and gather information about questions (difficulty, etc...)\n- user_cache that is computed iterativaly to avoid any time leakage.\n\nMy pipeline below:\n\n\n\nIn the general sequence, I loop through the row of the train/test set and add the relevant informations from my two caches to build the feature matrix used to train my model. Two things to note:\n- It is important to create the feature matrix BEFORE updating the user_cache to avoid leakage of information not yet seen (answered_correctly for example)\n- It is very important to consider the questions by batches, as the data is served like this through the API. To make sure there is no batch leaks, I include in my pipeline a temporary file where I store the rows from a unique batch. Only when the batch change, I iterate over this template to first create all the row in the feature matrix and then only update the user_cache.\n- Another very important point: Before starting the predictions, you shall precompute the user_cache using the full train df. A lot of information is available from users from the test set in the train set. I was initially not doing it and was stuck in the 0.77X area. This allowed me to jump in the 0.78X one.\n\nGood luck all for the last few days!" } }