First and foremost I would like to thank the coordinators of RIIID for conducting this awesome competition and all the Kaggle Staffs who kept their constant vigilance for bugs and issues. Despite having data leaks and many such issues, the kaggle team handled the issue promptly and efficiently. The competition API was one of the most frustrating and yet necessary aspect of a competition such as this one.
\nThe list of people I would like to thank is numerous and large, a few notable mentions but not limited to are:
\nSo what did I do? My model idea is quite simple: 8 bagged LGBM models fed into an FTRL model. Kaggle CPU had memory restrictions of 16GB. Even concatenating all the features together threw that horrible OOM error. I created features piece by piece and concatenated them into three different chunks to create the training data.
\nNow we have 100M data containing 38 Features. Training an LGB on this large data would still throw an OOM. So I decided to do bagging (which would still throw an OOM error if you didn't slowly convert the data into float numpy arrays yourself using loops). Now Bagging is not an option you'd go for if you wish to make more accurate predictions, but a discussion mentioned that training on larger dataset gave a score boost of ~0.01 and so I did.
\nTraining on the first chunk (25M) gave a CV of 0.791. After 7 such chunks, I got a CV of 0.792. Frustrated by seeing my LB ranks drop and to improve the predictive power a bit, I decided to add another model to the mix, a FTRL.
\nI was also motivated to use FTRL for two reasons:
\nFTRL by itself didn't perform very well on my CVs. So I decided to feed my LGB model predictions to it to help it make its predictions. If the FTRL sees the LGB making false predictions during inference, it can improve or alter its predictions accordingly. Doing this improved my LB from 0.792 to 0.793. I have no prior knowledge with FTRL models. Perhaps with better features FTRL might have worked even better.
\nI enjoyed participating in this competition. It gave me so much more than I imagined it would. To all the winners and participants of this competition: My heartiest congratulations and a Happy new year!
\nFirst and foremost I would like to thank the coordinators of RIIID for conducting this awesome competition and all the Kaggle Staffs who kept their constant vigilance for bugs and issues. Despite having data leaks and many such issues, the kaggle team handled the issue promptly and efficiently. The competition API was one of the most frustrating and yet necessary aspect of a competition such as this one.
\nThe list of people I would like to thank is numerous and large, a few notable mentions but not limited to are:
\nSo what did I do? My model idea is quite simple: 8 bagged LGBM models fed into an FTRL model. Kaggle CPU had memory restrictions of 16GB. Even concatenating all the features together threw that horrible OOM error. I created features piece by piece and concatenated them into three different chunks to create the training data.
\nNow we have 100M data containing 38 Features. Training an LGB on this large data would still throw an OOM. So I decided to do bagging (which would still throw an OOM error if you didn't slowly convert the data into float numpy arrays yourself using loops). Now Bagging is not an option you'd go for if you wish to make more accurate predictions, but a discussion mentioned that training on larger dataset gave a score boost of ~0.01 and so I did.
\nTraining on the first chunk (25M) gave a CV of 0.791. After 7 such chunks, I got a CV of 0.792. Frustrated by seeing my LB ranks drop and to improve the predictive power a bit, I decided to add another model to the mix, a FTRL.
\nI was also motivated to use FTRL for two reasons:
\nFTRL by itself didn't perform very well on my CVs. So I decided to feed my LGB model predictions to it to help it make its predictions. If the FTRL sees the LGB making false predictions during inference, it can improve or alter its predictions accordingly. Doing this improved my LB from 0.792 to 0.793. I have no prior knowledge with FTRL models. Perhaps with better features FTRL might have worked even better.
\nI enjoyed participating in this competition. It gave me so much more than I imagined it would. To all the winners and participants of this competition: My heartiest congratulations and a Happy new year!
\nNice Idea to use FTRLs!
", "rawMarkdown": "Nice Idea to use FTRLs!", "votes": null }, { "id": "1144381", "postDate": "01/08/2021 12:36:46", "content": "My notebook is now public. You can check them out here!
", "rawMarkdown": "My notebook is now public. You can check them out [here](https://www.kaggle.com/doctorkael/riiid-lgb-ftrl-data-gen-and-training-logic)!", "votes": null } ], "comments": [ { "id": 1143863, "author_name": "adityaecdrid", "author_url": "", "post_date": "01/08/2021 05:31:51", "content": "Nice Idea to use FTRLs!
", "votes": null, "replies": [] }, { "id": 1144381, "author_name": "doctorkael", "author_url": "", "post_date": "01/08/2021 12:36:46", "content": "My notebook is now public. You can check them out here!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1143738": "## Huge thanks to:\nFirst and foremost I would like to thank the coordinators of RIIID for conducting this awesome competition and all the Kaggle Staffs who kept their constant vigilance for bugs and issues. Despite having data leaks and many such issues, the kaggle team handled the issue promptly and efficiently. The competition API was one of the most frustrating and yet necessary aspect of a competition such as this one. \n\nThe list of people I would like to thank is numerous and large, a few notable mentions but not limited to are:\n1. https://www.kaggle.com/shoheiazuma/riiid-lgbm-starter: The dataset was huge and intimidating, @shoheiazuma notebook gave the idea to train on just the last 24 records. I improvised on this idea a lot. Using final 30M records for training felt like I was missing out some users. Training on final 250 records of each user gave me comparable results to training on the entire dataset. \n2. @vopani's classic notebooks: [this](https://www.kaggle.com/rohanrao/tutorial-on-reading-large-datasets) and [this](https://www.kaggle.com/rohanrao/riiid-ftrl-ftw): These tutorials Introduced me to my now favorite feather format. It also introduced me to DataTable FTRL and Dask. \n3. @adityaecdrid discussions on repeated user questions and his ideas to use Bit Arrays. He contributed a ton of ideas and intrigued me to come with cleverer features. \n4. https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/196009: Ditch pandas! All my user state updations were done in a loop. Easier on the pipeline and easier for my brain too :)\n5. https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/196942: A kaggle GM @silogram using LGBM to hit high ranks inspired me not to give up so soon when my score stuck at local optimas.\n6. @anuragtr discussion [here](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/198245) on how to prevent ram spikes when starting LGB training.\n\n## The Model Idea and difficulties faced:\nSo what did *I* do? My model idea is quite simple: **8 bagged LGBM models fed into an FTRL model**. Kaggle CPU had memory restrictions of 16GB. Even concatenating all the features together threw that horrible OOM error. I created features piece by piece and concatenated them into three different chunks to create the training data. \n\nNow we have 100M data containing 38 Features. Training an LGB on this large data would still throw an OOM. So I decided to do bagging (which would still throw an OOM error if you didn't slowly convert the data into float numpy arrays yourself using loops). Now Bagging is not an option you'd go for if you wish to make more accurate predictions, but a discussion mentioned that training on larger dataset gave a score boost of ~0.01 and so I did. \n\nTraining on the first chunk (25M) gave a CV of 0.791. After 7 such chunks, I got a CV of 0.792. Frustrated by seeing my LB ranks drop and to improve the predictive power a bit, I decided to add another model to the mix, a FTRL. \n\nI was also motivated to use FTRL for two reasons: \n\n1. I only had two days left before competition ends & FTRL was super fast!\n2. FTRL model is best suited for applications were the predictions need to dynamic and evolving. \n\nFTRL by itself didn't perform very well on my CVs. So I decided to feed my LGB model predictions to it to help it make its predictions. If the FTRL sees the LGB making false predictions during inference, it can improve or alter its predictions accordingly. Doing this improved my LB from 0.792 to 0.793. I have no prior knowledge with FTRL models. Perhaps with better features FTRL might have worked even better.\n\n## Some of my personal takeaways:\n\n- *Don't shy away from newer unfamiliar concepts!* With such a huge time frame given, I could have learned and implemented a transformer model. Instead I chose to stick with what I knew best: Brute forcing my way with hand coded features. Although this approach taught me how to make the most out of the limited given resources.\n- *Numba is awesome!* Ton of code speed up, all with a simple decorator! Numba although fast, may throw OOM if blindly used. My notebook (soon to be published) demonstrates how I leveraged this awesome library.\n- *LB ranks doesn't matter!* To be honest, It was painful to see my LB ranks drop from 60 all the way down to 158. But in the end, I managed to accept it and even laugh about it! See [this](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/208356) hilarious meme thread.\n\n***\nI enjoyed participating in this competition. It gave me so much more than I imagined it would. To all the winners and participants of this competition: *My heartiest congratulations and a Happy new year!*\n***", "1143863": "Nice Idea to use FTRLs!", "1144381": "My notebook is now public. You can check them out [here](https://www.kaggle.com/doctorkael/riiid-lgb-ftrl-data-gen-and-training-logic)!" }, "source": "meta" }