If you're using a LSTM like method for this competition such as LSTM Baseline by ZhaounBooty. Every time you want to try out a different preprocessing idea, you have to regenerate a large set of files. This is both time-consuming and takes up a lot of space if you're using your own machine.
\nI wanted a preprocessing setup with which I can quickly try out different preprocessing ideas without having to pre-cache the data.
\npolars for loading parquet files, way faster than pandasnp.zeros array instead of using np.padnumba. Here's the code, hope it helps.
\nhttps://www.kaggle.com/code/dipamc77/fast-preprocessing-with-polars-and-numba-1-5-s/notebook
If you're using a LSTM like method for this competition such as LSTM Baseline by ZhaounBooty. Every time you want to try out a different preprocessing idea, you have to regenerate a large set of files. This is both time-consuming and takes up a lot of space if you're using your own machine.
\nI wanted a preprocessing setup with which I can quickly try out different preprocessing ideas without having to pre-cache the data.
\npolars for loading parquet files, way faster than pandasnp.zeros array instead of using np.padnumba. Here's the code, hope it helps.
\nhttps://www.kaggle.com/code/dipamc77/fast-preprocessing-with-polars-and-numba-1-5-s/notebook
very good notebook! I like notebooks which are about pipeline optimization. This is important topic. Definitely good job @dipamc77 👍
", "rawMarkdown": "very good notebook! I like notebooks which are about pipeline optimization. This is important topic. Definitely good job @dipamc77 👍", "votes": null }, { "id": "2204358", "postDate": "03/31/2023 13:59:23", "content": "Thanks!
\nI guess the incentive to optimize is high when one is limited by compute. 😅
", "rawMarkdown": "Thanks! \n\nI guess the incentive to optimize is high when one is limited by compute. 😅", "votes": null }, { "id": "2205309", "postDate": "04/01/2023 12:20:33", "content": "Numba is amazing. If you don't do that already, try to parallelize your Numba loops with nb.prange (remember to add parallel=True in the jit decorator) for even faster processing!
", "rawMarkdown": "Numba is amazing. If you don't do that already, try to parallelize your Numba loops with nb.prange (remember to add parallel=True in the jit decorator) for even faster processing!", "votes": null }, { "id": "2205310", "postDate": "04/01/2023 12:22:01", "content": "Oh, I didn't know this, I check it out. Thanks a lot.
", "rawMarkdown": "Oh, I didn't know this, I check it out. Thanks a lot.", "votes": null } ], "comments": [ { "id": 2204248, "author_name": "remekkinas", "author_url": "", "post_date": "03/31/2023 12:45:51", "content": "very good notebook! I like notebooks which are about pipeline optimization. This is important topic. Definitely good job @dipamc77 👍
", "votes": null, "replies": [ { "id": 2204358, "author_name": "dipamc77", "author_url": "", "post_date": "03/31/2023 13:59:23", "content": "Thanks!
\nI guess the incentive to optimize is high when one is limited by compute. 😅
", "votes": null, "replies": [] } ] }, { "id": 2205309, "author_name": "shlomoron", "author_url": "", "post_date": "04/01/2023 12:20:33", "content": "Numba is amazing. If you don't do that already, try to parallelize your Numba loops with nb.prange (remember to add parallel=True in the jit decorator) for even faster processing!
", "votes": null, "replies": [ { "id": 2205310, "author_name": "dipamc77", "author_url": "", "post_date": "04/01/2023 12:22:01", "content": "Oh, I didn't know this, I check it out. Thanks a lot.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2203520": "# Fast preprocessing with Polars and Numba\n If you're using a LSTM like method for this competition such as [LSTM Baseline by ZhaounBooty](https://www.kaggle.com/code/seungmoklee/lstm-preprocessing-point-picker). Every time you want to try out a different preprocessing idea, you have to regenerate a large set of files. This is both time-consuming and takes up a lot of space if you're using your own machine.\n\nI wanted a preprocessing setup with which I can quickly try out different preprocessing ideas without having to pre-cache the data. \n\n### Timings for preprocessing one batch: \n- Kaggle - 7 seconds\n- i5 12400 Processor - 1.5 seconds (probably because Kaggle disk read is slow)\n\n### Key Insights\n1. Use `polars` for loading parquet files, way faster than `pandas`\n2. For padding, pre-assign the `np.zeros` array instead of using `np.pad`\n3. For loop based operations operations, such as sampling points or filtering based on particular data, use `numba`. \n\nHere's the code, hope it helps. \nhttps://www.kaggle.com/code/dipamc77/fast-preprocessing-with-polars-and-numba-1-5-s/notebook", "2204248": "very good notebook! I like notebooks which are about pipeline optimization. This is important topic. Definitely good job @dipamc77 👍", "2204358": "Thanks! \n\nI guess the incentive to optimize is high when one is limited by compute. 😅", "2205309": "Numba is amazing. If you don't do that already, try to parallelize your Numba loops with nb.prange (remember to add parallel=True in the jit decorator) for even faster processing!", "2205310": "Oh, I didn't know this, I check it out. Thanks a lot." }, "source": "meta" }