Looking at many public notebooks from this competition I didn't met a single one that process the data in a different way than in the host's example. Everywhere the same copy/paste blocks of codes or entire processing cells with classes based on Polars with just small differences between them. Or it's because it's still rare who knows Polars and for big data like this nobody bothers to use Pandas so people are just using what's working by copying the code?
\nJust from curiosity I was able to preprocess all the files in Pandas without exceeding the memory and got ~900 features. After this I re-wrote this in Polars just from curiosity to learn some Polars. It seams really strange for me and I don't think I'm the only crazy that wrote from scratch all the processing in an absolutely other way than the 'public template'. Maybe you saw other interesting notebooks with alternative ways and not just a variation of the 'template'?
Looking at many public notebooks from this competition I didn't met a single one that process the data in a different way than in the host's example. Everywhere the same copy/paste blocks of codes or entire processing cells with classes based on Polars with just small differences between them. Or it's because it's still rare who knows Polars and for big data like this nobody bothers to use Pandas so people are just using what's working by copying the code?
\nJust from curiosity I was able to preprocess all the files in Pandas without exceeding the memory and got ~900 features. After this I re-wrote this in Polars just from curiosity to learn some Polars. It seams really strange for me and I don't think I'm the only crazy that wrote from scratch all the processing in an absolutely other way than the 'public template'. Maybe you saw other interesting notebooks with alternative ways and not just a variation of the 'template'?
I did it myself, although I followed the idea of open source code and after integrating open source catboost, I was able to achieve 0.582.
\nhttps://www.kaggle.com/code/yunsuxiaozi/home-credit-lgbm-with-677-features-inference
", "rawMarkdown": "I did it myself, although I followed the idea of open source code and after integrating open source catboost, I was able to achieve 0.582.\n\nhttps://www.kaggle.com/code/yunsuxiaozi/home-credit-lgbm-with-677-features-inference", "votes": null }, { "id": "2773817", "postDate": "04/24/2024 23:31:09", "content": "Try to read from parquet - should be quicker and it reads the columns' dtypes as parquet holds this info
", "rawMarkdown": "Try to read from parquet - should be quicker and it reads the columns' dtypes as parquet holds this info", "votes": null } ], "comments": [ { "id": 2773815, "author_name": "yunsuxiaozi", "author_url": "", "post_date": "04/24/2024 23:24:57", "content": "I did it myself, although I followed the idea of open source code and after integrating open source catboost, I was able to achieve 0.582.
\nhttps://www.kaggle.com/code/yunsuxiaozi/home-credit-lgbm-with-677-features-inference
", "votes": null, "replies": [ { "id": 2773817, "author_name": "eu1234", "author_url": "", "post_date": "04/24/2024 23:31:09", "content": "Try to read from parquet - should be quicker and it reads the columns' dtypes as parquet holds this info
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2773782": "Looking at many public notebooks from this competition I didn't met a single one that process the data in a different way than in the host's example. Everywhere the same copy/paste blocks of codes or entire processing cells with classes based on Polars with just small differences between them. Or it's because it's still rare who knows Polars and for big data like this nobody bothers to use Pandas so people are just using what's working by copying the code?\nJust from curiosity I was able to preprocess all the files in Pandas without exceeding the memory and got ~900 features. After this I re-wrote this in Polars just from curiosity to learn some Polars. It seams really strange for me and I don't think I'm the only crazy that wrote from scratch all the processing in an absolutely other way than the 'public template'. Maybe you saw other interesting notebooks with alternative ways and not just a variation of the 'template'?", "2773815": "I did it myself, although I followed the idea of open source code and after integrating open source catboost, I was able to achieve 0.582.\n\nhttps://www.kaggle.com/code/yunsuxiaozi/home-credit-lgbm-with-677-features-inference", "2773817": "Try to read from parquet - should be quicker and it reads the columns' dtypes as parquet holds this info" }, "source": "meta" }