Hi everyone,
\nis anyone working on a graph-based approach and can share some insights on what type of graph and ML approach is feasible? I am having quite a difficult time to find a efficient solution of creating the graphs for all the train/test data. Here is my current progress Can a Graph Help Classify Player Data?
\nCheers
\nTom
Hi everyone,
\nis anyone working on a graph-based approach and can share some insights on what type of graph and ML approach is feasible? I am having quite a difficult time to find a efficient solution of creating the graphs for all the train/test data. Here is my current progress Can a Graph Help Classify Player Data?
\nCheers
\nTom
Hi @tompaulat, would really recommend profiling your code, and identifying where the bottleneck is. Check if the bottlenecked task is parallelizable, and more so if it can be parallelized on a GPU(my KNN model that took 40-45 minutes to run on CPU on a small 3k~ sample dataset took like 3 minutes, and Kaggle recently launched a 2x GPU option + 30 GB RAM!), or if there's other packages/methods that do that specific task faster.(I once got a 3x~ speedup simply from switching to the selectolax library with C-backend engine for HTML processing instead of BeautifulSoup).
\nMost of these packages intended as faster replacements usually have similar APIs so switching is easy. I recently found JAX, and switching to it in-place can be easy as changing import numpy as np to import jax.numpy as np. It's got good GPU+TPU support, and it's apparently faster than pytorch at basic number-crunching stuff. I've been planning to switch my KNN to JAX this weekend, and try JIT-ing as much as can be JIT-ed, and see how much faster I can make it go.
Thanks @aatiffraz,
\nI will look into that! Interesting. Much success with your approach! 👍
Also forgot to mention, you've linked this discussion in the post instead of the notebook 👀
", "rawMarkdown": "Also forgot to mention, you've linked this discussion in the post instead of the notebook 👀", "votes": null } ], "comments": [ { "id": 1998346, "author_name": "aatiffraz", "author_url": "", "post_date": "10/21/2022 14:41:14", "content": "Hi @tompaulat, would really recommend profiling your code, and identifying where the bottleneck is. Check if the bottlenecked task is parallelizable, and more so if it can be parallelized on a GPU(my KNN model that took 40-45 minutes to run on CPU on a small 3k~ sample dataset took like 3 minutes, and Kaggle recently launched a 2x GPU option + 30 GB RAM!), or if there's other packages/methods that do that specific task faster.(I once got a 3x~ speedup simply from switching to the selectolax library with C-backend engine for HTML processing instead of BeautifulSoup).
\nMost of these packages intended as faster replacements usually have similar APIs so switching is easy. I recently found JAX, and switching to it in-place can be easy as changing import numpy as np to import jax.numpy as np. It's got good GPU+TPU support, and it's apparently faster than pytorch at basic number-crunching stuff. I've been planning to switch my KNN to JAX this weekend, and try JIT-ing as much as can be JIT-ed, and see how much faster I can make it go.
Thanks @aatiffraz,
\nI will look into that! Interesting. Much success with your approach! 👍
Also forgot to mention, you've linked this discussion in the post instead of the notebook 👀
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1997972": "Hi everyone,\nis anyone working on a graph-based approach and can share some insights on what type of graph and ML approach is feasible? I am having quite a difficult time to find a efficient solution of creating the graphs for all the train/test data. Here is my current progress [Can a Graph Help Classify Player Data?](https://www.kaggle.com/code/tompaulat/can-a-graph-help-classify-player-data?scriptVersionId=108755522)\nCheers \nTom", "1998346": "Hi @tompaulat, would really recommend profiling your code, and identifying where the bottleneck is. Check if the bottlenecked task is parallelizable, and more so if it can be parallelized on a GPU(my KNN model that took 40-45 minutes to run on CPU on a small 3k~ sample dataset took like 3 minutes, and Kaggle recently launched a 2x GPU option + 30 GB RAM!), or if there's other packages/methods that do that specific task faster.(I once got a 3x~ speedup simply from switching to the selectolax library with C-backend engine for HTML processing instead of BeautifulSoup). \n\nMost of these packages intended as faster replacements usually have similar APIs so switching is easy. I recently found JAX, and switching to it in-place can be easy as changing `import numpy as np` to `import jax.numpy as np`. It's got good GPU+TPU support, and it's apparently faster than pytorch at basic number-crunching stuff. I've been planning to switch my KNN to JAX this weekend, and try JIT-ing as much as can be JIT-ed, and see how much faster I can make it go.", "1998355": "Thanks @aatiffraz, \nI will look into that! Interesting. Much success with your approach! 👍", "1998378": "Also forgot to mention, you've linked this discussion in the post instead of the notebook 👀" }, "source": "meta" }