{ "id": 55328, "title": "How to deal with Russian text features?", "url": "/competitions/avito-demand-prediction/discussion/55328", "author_name": "", "post_date": "2018-04-25T08:22:46.759247100Z", "votes": 12, "comment_count": 23, "views": 0, "content": "

I'm not familiar with Russian, bellow is some open-source-code for deal with Russian:

\n\n\n\n

Would you please share some tools can deal with Russian ?

", "messages": [ { "id": "319105", "postDate": "04/25/2018 08:22:46", "content": "

I'm not familiar with Russian, bellow is some open-source-code for deal with Russian:

\n\n\n\n

Would you please share some tools can deal with Russian ?

", "rawMarkdown": "I'm not familiar with Russian, bellow is some open-source-code for deal with Russian:\n\n- [nltk4russian][1]\n- [redhenlab][2]\n- [word2vec skip-gram model for Russian language][3]\n- [fasttext-vectors wiki ru][4]\n\nWould you please share some tools can deal with Russian ?\n\n\n [1]: https://github.com/named-entity/nltk4russian\n [2]: http://www.redhenlab.org/home/the-cognitive-core-research-topics-in-red-hen/the-barnyard/russian-nlp\n [3]: https://github.com/nlpub/russe-evaluation/tree/master/russe/measures/word2vec\n [4]: https://s3-us-west-1.amazonaws.com/fasttext-vectors/wiki.ru.zip", "votes": null }, { "id": "319118", "postDate": "04/25/2018 08:47:33", "content": "

like this intuition : https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/48038

\n\n

you can use text blob to translate it to english and using other stuff to help you or just using the tool you mentioned like fasttext-vectors wiki ru. Another way is using the word embedding library or code to train the Russian word embedding by your self. (glove has the code on GitHub, fastext and word2vec has its own library)

", "rawMarkdown": "like this intuition : https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/48038\n\nyou can use text blob to translate it to english and using other stuff to help you or just using the tool you mentioned like fasttext-vectors wiki ru. Another way is using the word embedding library or code to train the Russian word embedding by your self. (glove has the code on GitHub, fastext and word2vec has its own library)", "votes": null }, { "id": "319377", "postDate": "04/25/2018 23:06:43", "content": "

Do you know something to help me here with R, and not python?

", "rawMarkdown": "Do you know something to help me here with R, and not python?", "votes": null }, { "id": "319392", "postDate": "04/26/2018 00:38:49", "content": "

There are a couple of approaches to this problem as far as I know. The first is to translate it to English and then use whatever word embeddings you want on your own, such as fasttext or word2vec. You could also train the Russian word embeddings yourself using the public source code for those word vectors. Additionally, you could simply try those things you mentioned for your word vectors; in my opinion, that would be the place to start. Further, you could try training your own Tf-idf on the train set (that would also be a good starting point, I think).

", "rawMarkdown": "There are a couple of approaches to this problem as far as I know. The first is to translate it to English and then use whatever word embeddings you want on your own, such as fasttext or word2vec. You could also train the Russian word embeddings yourself using the public source code for those word vectors. Additionally, you could simply try those things you mentioned for your word vectors; in my opinion, that would be the place to start. Further, you could try training your own Tf-idf on the train set (that would also be a good starting point, I think).", "votes": null }, { "id": "319483", "postDate": "04/26/2018 06:53:21", "content": "

Sklearn doesn't seem to have a countvectorizer / tfidf for Russian. Anybody know of an alternative?

", "rawMarkdown": "Sklearn doesn't seem to have a countvectorizer / tfidf for Russian. Anybody know of an alternative?", "votes": null }, { "id": "319495", "postDate": "04/26/2018 07:21:35", "content": "

I think the point is you have to know about the stop word in Russian, because tf-idf just filter out of some stop words and counting the frequency of word(here I just simply said, but the intuition is counting the term frequency and inverse document frequency). So if you kwon the stop word, it will work.

", "rawMarkdown": "I think the point is you have to know about the stop word in Russian, because tf-idf just filter out of some stop words and counting the frequency of word(here I just simply said, but the intuition is counting the term frequency and inverse document frequency). So if you kwon the stop word, it will work.", "votes": null }, { "id": "319675", "postDate": "04/26/2018 15:22:05", "content": "

Learn Russian? ;)

", "rawMarkdown": "Learn Russian? ;)", "votes": null }, { "id": "319676", "postDate": "04/26/2018 15:22:30", "content": "

Team with a Russian Kaggler?

", "rawMarkdown": "Team with a Russian Kaggler?", "votes": null }, { "id": "319706", "postDate": "04/26/2018 16:48:20", "content": "

You can get russian stopwords from nltk.

\n\n

from nltk.corpus import stopwords
\nstop = set(stopwords.words('russian'))

", "rawMarkdown": "You can get russian stopwords from nltk. \n\n`from nltk.corpus import stopwords \nstop = set(stopwords.words('russian'))`", "votes": null }, { "id": "319750", "postDate": "04/26/2018 18:59:55", "content": "

Mystem - best tool for text lemmatization https://github.com/nlpub/pymystem3

", "rawMarkdown": "Mystem - best tool for text lemmatization https://github.com/nlpub/pymystem3", "votes": null }, { "id": "319918", "postDate": "04/27/2018 05:22:48", "content": "

It's relative term. A site under same name will have both a website and a mobile app. Thanks You For Posting;

\n\n

https://tutuhelperx.com https://emus4u.co https://tutuappapk.vip http://cinemaboxapkdownloads.com

", "rawMarkdown": "It's relative term. A site under same name will have both a website and a mobile app. Thanks You For Posting;\n\n https://tutuhelperx.com https://emus4u.co https://tutuappapk.vip http://cinemaboxapkdownloads.com", "votes": null }, { "id": "320354", "postDate": "04/28/2018 12:18:55", "content": "

One can use transliterate which will at the least help to read and understand the russian text values better, might come in handy.

", "rawMarkdown": "One can use [transliterate][1] which will at the least help to read and understand the russian text values better, might come in handy.\n\n\n [1]: https://pypi.org/project/transliterate/", "votes": null }, { "id": "320429", "postDate": "04/28/2018 16:41:08", "content": "

Thanks , this is really helpful for mates.

", "rawMarkdown": "Thanks , this is really helpful for mates.", "votes": null }, { "id": "320572", "postDate": "04/29/2018 05:18:04", "content": "

thanks for sharing these tools.

", "rawMarkdown": "thanks for sharing these tools.", "votes": null }, { "id": "321360", "postDate": "05/01/2018 05:07:15", "content": "

I wrote a script to do the Russian to English conversion, code is here https://www.kaggle.com/gunnvant/russian-to-english-translate-with-progress-bar

", "rawMarkdown": "I wrote a script to do the Russian to English conversion, code is here https://www.kaggle.com/gunnvant/russian-to-english-translate-with-progress-bar", "votes": null }, { "id": "321704", "postDate": "05/01/2018 18:51:59", "content": "

You can even try using TextBlob package. It has lot of inbuilt functionalities including lemmatization and translation!

", "rawMarkdown": "You can even try using TextBlob package. It has lot of inbuilt functionalities including lemmatization and translation!", "votes": null }, { "id": "323032", "postDate": "05/04/2018 07:38:09", "content": "

@Kishore, is this available in kaggle kernels?

", "rawMarkdown": "Kishore, is this available in kaggle kernels?", "votes": null }, { "id": "323075", "postDate": "05/04/2018 10:03:00", "content": "

Hey Sumit, it doesn't come preloaded but you should be able to install it from \"Add a custom package\" in the settings tab of the kernel.

", "rawMarkdown": "Hey Sumit, it doesn't come preloaded but you should be able to install it from \"Add a custom package\" in the settings tab of the kernel.", "votes": null }, { "id": "323108", "postDate": "05/04/2018 12:00:52", "content": "

Thanks @Kishore.

", "rawMarkdown": "Thanks @Kishore.", "votes": null }, { "id": "324920", "postDate": "05/08/2018 01:40:10", "content": "

Good suggestion

", "rawMarkdown": "Good suggestion", "votes": null }, { "id": "324924", "postDate": "05/08/2018 01:41:46", "content": "

Thank you Pratik, let know the awesome package whic can translate ru to en

", "rawMarkdown": "Thank you Pratik, let know the awesome package whic can translate ru to en", "votes": null }, { "id": "324926", "postDate": "05/08/2018 01:42:52", "content": "

Really good kernel. Thanks for your sharing.

", "rawMarkdown": "Really good kernel. Thanks for your sharing.", "votes": null }, { "id": "325351", "postDate": "05/08/2018 10:24:47", "content": "

Alexander, I always use pymystem3 in my daily work. It's really the best tool for text lemmatization!

", "rawMarkdown": "Alexander, I always use pymystem3 in my daily work. It's really the best tool for text lemmatization!", "votes": null }, { "id": "770301", "postDate": "03/12/2020 19:50:53", "content": "

Hi, just wondering how could I deal with the \"HTTP Error 429: Too Many Requests\" when I use textblob translate() method. It seems that I'm being rate-limited due to too much call to Google Translate API

", "rawMarkdown": "Hi, just wondering how could I deal with the \"HTTP Error 429: Too Many Requests\" when I use textblob translate() method. It seems that I'm being rate-limited due to too much call to Google Translate API", "votes": null } ], "comments": [ { "id": 319118, "author_name": "ericlikedata", "author_url": "", "post_date": "04/25/2018 08:47:33", "content": "

like this intuition : https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/48038

\n\n

you can use text blob to translate it to english and using other stuff to help you or just using the tool you mentioned like fasttext-vectors wiki ru. Another way is using the word embedding library or code to train the Russian word embedding by your self. (glove has the code on GitHub, fastext and word2vec has its own library)

", "votes": null, "replies": [ { "id": 770301, "author_name": "shuai6", "author_url": "", "post_date": "03/12/2020 19:50:53", "content": "

Hi, just wondering how could I deal with the \"HTTP Error 429: Too Many Requests\" when I use textblob translate() method. It seems that I'm being rate-limited due to too much call to Google Translate API

", "votes": null, "replies": [] } ] }, { "id": 319377, "author_name": "rohitchauhan56itbhu", "author_url": "", "post_date": "04/25/2018 23:06:43", "content": "

Do you know something to help me here with R, and not python?

", "votes": null, "replies": [] }, { "id": 319392, "author_name": "matthewa313", "author_url": "", "post_date": "04/26/2018 00:38:49", "content": "

There are a couple of approaches to this problem as far as I know. The first is to translate it to English and then use whatever word embeddings you want on your own, such as fasttext or word2vec. You could also train the Russian word embeddings yourself using the public source code for those word vectors. Additionally, you could simply try those things you mentioned for your word vectors; in my opinion, that would be the place to start. Further, you could try training your own Tf-idf on the train set (that would also be a good starting point, I think).

", "votes": null, "replies": [] }, { "id": 319483, "author_name": "nicapotato", "author_url": "", "post_date": "04/26/2018 06:53:21", "content": "

Sklearn doesn't seem to have a countvectorizer / tfidf for Russian. Anybody know of an alternative?

", "votes": null, "replies": [ { "id": 319495, "author_name": "ericlikedata", "author_url": "", "post_date": "04/26/2018 07:21:35", "content": "

I think the point is you have to know about the stop word in Russian, because tf-idf just filter out of some stop words and counting the frequency of word(here I just simply said, but the intuition is counting the term frequency and inverse document frequency). So if you kwon the stop word, it will work.

", "votes": null, "replies": [] }, { "id": 319706, "author_name": "petrbolotin", "author_url": "", "post_date": "04/26/2018 16:48:20", "content": "

You can get russian stopwords from nltk.

\n\n

from nltk.corpus import stopwords
\nstop = set(stopwords.words('russian'))

", "votes": null, "replies": [] } ] }, { "id": 319675, "author_name": "cpmpml", "author_url": "", "post_date": "04/26/2018 15:22:05", "content": "

Learn Russian? ;)

", "votes": null, "replies": [] }, { "id": 319676, "author_name": "cpmpml", "author_url": "", "post_date": "04/26/2018 15:22:30", "content": "

Team with a Russian Kaggler?

", "votes": null, "replies": [ { "id": 324920, "author_name": "classtag", "author_url": "", "post_date": "05/08/2018 01:40:10", "content": "

Good suggestion

", "votes": null, "replies": [] } ] }, { "id": 319750, "author_name": "alxmamaev", "author_url": "", "post_date": "04/26/2018 18:59:55", "content": "

Mystem - best tool for text lemmatization https://github.com/nlpub/pymystem3

", "votes": null, "replies": [ { "id": 320429, "author_name": "classtag", "author_url": "", "post_date": "04/28/2018 16:41:08", "content": "

Thanks , this is really helpful for mates.

", "votes": null, "replies": [] }, { "id": 321704, "author_name": "pratikkgandhi", "author_url": "", "post_date": "05/01/2018 18:51:59", "content": "

You can even try using TextBlob package. It has lot of inbuilt functionalities including lemmatization and translation!

", "votes": null, "replies": [] }, { "id": 324924, "author_name": "classtag", "author_url": "", "post_date": "05/08/2018 01:41:46", "content": "

Thank you Pratik, let know the awesome package whic can translate ru to en

", "votes": null, "replies": [] }, { "id": 325351, "author_name": "petrbolotin", "author_url": "", "post_date": "05/08/2018 10:24:47", "content": "

Alexander, I always use pymystem3 in my daily work. It's really the best tool for text lemmatization!

", "votes": null, "replies": [] } ] }, { "id": 319918, "author_name": "josemark", "author_url": "", "post_date": "04/27/2018 05:22:48", "content": "

It's relative term. A site under same name will have both a website and a mobile app. Thanks You For Posting;

\n\n

https://tutuhelperx.com https://emus4u.co https://tutuappapk.vip http://cinemaboxapkdownloads.com

", "votes": null, "replies": [] }, { "id": 320354, "author_name": "kishupro", "author_url": "", "post_date": "04/28/2018 12:18:55", "content": "

One can use transliterate which will at the least help to read and understand the russian text values better, might come in handy.

", "votes": null, "replies": [ { "id": 323032, "author_name": "sumitsethy", "author_url": "", "post_date": "05/04/2018 07:38:09", "content": "

@Kishore, is this available in kaggle kernels?

", "votes": null, "replies": [] }, { "id": 323075, "author_name": "kishupro", "author_url": "", "post_date": "05/04/2018 10:03:00", "content": "

Hey Sumit, it doesn't come preloaded but you should be able to install it from \"Add a custom package\" in the settings tab of the kernel.

", "votes": null, "replies": [] }, { "id": 323108, "author_name": "sumitsethy", "author_url": "", "post_date": "05/04/2018 12:00:52", "content": "

Thanks @Kishore.

", "votes": null, "replies": [] } ] }, { "id": 320572, "author_name": "ayush0510", "author_url": "", "post_date": "04/29/2018 05:18:04", "content": "

thanks for sharing these tools.

", "votes": null, "replies": [] }, { "id": 321360, "author_name": "gunnvant", "author_url": "", "post_date": "05/01/2018 05:07:15", "content": "

I wrote a script to do the Russian to English conversion, code is here https://www.kaggle.com/gunnvant/russian-to-english-translate-with-progress-bar

", "votes": null, "replies": [ { "id": 324926, "author_name": "classtag", "author_url": "", "post_date": "05/08/2018 01:42:52", "content": "

Really good kernel. Thanks for your sharing.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "319105": "I'm not familiar with Russian, bellow is some open-source-code for deal with Russian:\n\n- [nltk4russian][1]\n- [redhenlab][2]\n- [word2vec skip-gram model for Russian language][3]\n- [fasttext-vectors wiki ru][4]\n\nWould you please share some tools can deal with Russian ?\n\n\n [1]: https://github.com/named-entity/nltk4russian\n [2]: http://www.redhenlab.org/home/the-cognitive-core-research-topics-in-red-hen/the-barnyard/russian-nlp\n [3]: https://github.com/nlpub/russe-evaluation/tree/master/russe/measures/word2vec\n [4]: https://s3-us-west-1.amazonaws.com/fasttext-vectors/wiki.ru.zip", "319118": "like this intuition : https://www.kaggle.com/c/jigsaw-toxic-comment-classification-challenge/discussion/48038\n\nyou can use text blob to translate it to english and using other stuff to help you or just using the tool you mentioned like fasttext-vectors wiki ru. Another way is using the word embedding library or code to train the Russian word embedding by your self. (glove has the code on GitHub, fastext and word2vec has its own library)", "319377": "Do you know something to help me here with R, and not python?", "319392": "There are a couple of approaches to this problem as far as I know. The first is to translate it to English and then use whatever word embeddings you want on your own, such as fasttext or word2vec. You could also train the Russian word embeddings yourself using the public source code for those word vectors. Additionally, you could simply try those things you mentioned for your word vectors; in my opinion, that would be the place to start. Further, you could try training your own Tf-idf on the train set (that would also be a good starting point, I think).", "319483": "Sklearn doesn't seem to have a countvectorizer / tfidf for Russian. Anybody know of an alternative?", "319495": "I think the point is you have to know about the stop word in Russian, because tf-idf just filter out of some stop words and counting the frequency of word(here I just simply said, but the intuition is counting the term frequency and inverse document frequency). So if you kwon the stop word, it will work.", "319675": "Learn Russian? ;)", "319676": "Team with a Russian Kaggler?", "319706": "You can get russian stopwords from nltk. \n\n`from nltk.corpus import stopwords \nstop = set(stopwords.words('russian'))`", "319750": "Mystem - best tool for text lemmatization https://github.com/nlpub/pymystem3", "319918": "It's relative term. A site under same name will have both a website and a mobile app. Thanks You For Posting;\n\n https://tutuhelperx.com https://emus4u.co https://tutuappapk.vip http://cinemaboxapkdownloads.com", "320354": "One can use [transliterate][1] which will at the least help to read and understand the russian text values better, might come in handy.\n\n\n [1]: https://pypi.org/project/transliterate/", "320429": "Thanks , this is really helpful for mates.", "320572": "thanks for sharing these tools.", "321360": "I wrote a script to do the Russian to English conversion, code is here https://www.kaggle.com/gunnvant/russian-to-english-translate-with-progress-bar", "321704": "You can even try using TextBlob package. It has lot of inbuilt functionalities including lemmatization and translation!", "323032": "Kishore, is this available in kaggle kernels?", "323075": "Hey Sumit, it doesn't come preloaded but you should be able to install it from \"Add a custom package\" in the settings tab of the kernel.", "323108": "Thanks @Kishore.", "324920": "Good suggestion", "324924": "Thank you Pratik, let know the awesome package whic can translate ru to en", "324926": "Really good kernel. Thanks for your sharing.", "325351": "Alexander, I always use pymystem3 in my daily work. It's really the best tool for text lemmatization!", "770301": "Hi, just wondering how could I deal with the \"HTTP Error 429: Too Many Requests\" when I use textblob translate() method. It seems that I'm being rate-limited due to too much call to Google Translate API" }, "source": "meta" }