![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F11676771%2Fb77fe43aef10f9c642e9323f56fb4d26%2FYIV_8_1866NG18MZMKU.png?generation=1715317890692229&alt=media\")
I looked at ahmedelfazouan's datasets, but I wanted to change the encoding, however, the speed is very slow (joblib 96 thread takes 48 hours). So I decided to optimize.
\nThe Tokenizer library is written in rust and is very fast.
\nI designed a tokenizer and did the encoding, which took about 6 hours.
\nI also implemented joblib locally, which reduced the time to less than 10 minutes!
Here's the notebook and the dataset.
\nhttps://www.kaggle.com/code/horikitasaku/tokenize-in-terms-of-chemical-principles/
\nhttps://www.kaggle.com/datasets/horikitasaku/belka-chemical-encoding/data
", "messages": [ { "id": 2804957, "postDate": "2024-05-10T09:22:09.823Z", "content": "I looked at ahmedelfazouan's datasets, but I wanted to change the encoding, however, the speed is very slow (joblib 96 thread takes 48 hours). So I decided to optimize.
\nThe Tokenizer library is written in rust and is very fast.
\nI designed a tokenizer and did the encoding, which took about 6 hours.
\nI also implemented joblib locally, which reduced the time to less than 10 minutes!
Here's the notebook and the dataset.
\nhttps://www.kaggle.com/code/horikitasaku/tokenize-in-terms-of-chemical-principles/
\nhttps://www.kaggle.com/datasets/horikitasaku/belka-chemical-encoding/data
", "rawMarkdown": "I looked at ahmedelfazouan's [datasets](https://www.kaggle.com/datasets/ahmedelfazouan/belka-enc-dataset), but I wanted to change the encoding, however, the speed is very slow (joblib 96 thread takes 48 hours). So I decided to optimize.\n\nThe ```Tokenizer``` library is written in rust and is very fast.\nI designed a tokenizer and did the encoding, which took about 6 hours.\nI also implemented joblib locally, which reduced the time to less than 10 minutes!\n\n\n\nHere's the notebook and the dataset.\n\nhttps://www.kaggle.com/code/horikitasaku/tokenize-in-terms-of-chemical-principles/\n\nhttps://www.kaggle.com/datasets/horikitasaku/belka-chemical-encoding/data\n", "votes": 24 } ], "comments": [], "raw_markdown_by_id": { "2804957": "I looked at ahmedelfazouan's [datasets](https://www.kaggle.com/datasets/ahmedelfazouan/belka-enc-dataset), but I wanted to change the encoding, however, the speed is very slow (joblib 96 thread takes 48 hours). So I decided to optimize.\n\nThe ```Tokenizer``` library is written in rust and is very fast.\nI designed a tokenizer and did the encoding, which took about 6 hours.\nI also implemented joblib locally, which reduced the time to less than 10 minutes!\n\n\n\nHere's the notebook and the dataset.\n\nhttps://www.kaggle.com/code/horikitasaku/tokenize-in-terms-of-chemical-principles/\n\nhttps://www.kaggle.com/datasets/horikitasaku/belka-chemical-encoding/data\n" } }