I am listing down somethings that worked for me in my RNN network.
\n\nApproach for Word Processing:
\n\nCategorical Variables
\n\nLabel Encoding on Train+ Test ( I ended up combining train and test since I was getting errors \nduring the prediction phase that said new labels found )
Log or Log1p conversions for numerical values such as price
Embedding Approaches tried so far -
\n\nSelf Trained Word2Vec Embeddings - 300D\n Embedding and CV score improves with higher dimensions and more epochs used for training
Architecture - Adding more dense layers is dropping Local CV.
FastText for Russian - Best Score So far for the RNN network I used. 300D
Russian Glove - Lowest score. A very large portion of the vocab was found missing. I am guessing that weakened the model since I was simply replacing missing words with a zeros (300D)
Concatenation of FastText and Glove - 600 Vector embedding. \n Dropped the score achieved by Fast Text. Clearly Glove doesn't work for whatever reason.
Concatenation of FastText and Self Trained Word2vec
\n\nI think there is a bit of a slight bump is CV with this. In the process of generating 300D Vectors \n for Word2vec.
Will update as and when I try new things.
\n\nRegards\nShanth
", "messages": [ { "id": "331911", "postDate": "05/22/2018 05:06:33", "content": "I am listing down somethings that worked for me in my RNN network.
\n\nApproach for Word Processing:
\n\nCategorical Variables
\n\nLabel Encoding on Train+ Test ( I ended up combining train and test since I was getting errors \nduring the prediction phase that said new labels found )
Log or Log1p conversions for numerical values such as price
Embedding Approaches tried so far -
\n\nSelf Trained Word2Vec Embeddings - 300D\n Embedding and CV score improves with higher dimensions and more epochs used for training
Architecture - Adding more dense layers is dropping Local CV.
FastText for Russian - Best Score So far for the RNN network I used. 300D
Russian Glove - Lowest score. A very large portion of the vocab was found missing. I am guessing that weakened the model since I was simply replacing missing words with a zeros (300D)
Concatenation of FastText and Glove - 600 Vector embedding. \n Dropped the score achieved by Fast Text. Clearly Glove doesn't work for whatever reason.
Concatenation of FastText and Self Trained Word2vec
\n\nI think there is a bit of a slight bump is CV with this. In the process of generating 300D Vectors \n for Word2vec.
Will update as and when I try new things.
\n\nRegards\nShanth
", "rawMarkdown": "I am listing down somethings that worked for me in my RNN network. \n\nApproach for Word Processing: \n\n- Single Text vector combining title and description \n- Zero word processing \n- Vocab size is fairly large at 200K \n\nCategorical Variables \n\n- Label Encoding on Train+ Test ( I ended up combining train and test since I was getting errors \n during the prediction phase that said new labels found )\n\n- Log or Log1p conversions for numerical values such as price \n\nEmbedding Approaches tried so far - \n\n1. Self Trained Word2Vec Embeddings - 300D\n Embedding and CV score improves with higher dimensions and more epochs used for training \n\n2. Architecture - Adding more dense layers is dropping Local CV. \n\n3. FastText for Russian - Best Score So far for the RNN network I used. 300D\n\n4. Russian Glove - Lowest score. A very large portion of the vocab was found missing. I am guessing that weakened the model since I was simply replacing missing words with a zeros (300D)\n\n5. Concatenation of FastText and Glove - 600 Vector embedding. \n Dropped the score achieved by Fast Text. Clearly Glove doesn't work for whatever reason. \n\n6. Concatenation of FastText and Self Trained Word2vec \n\n I think there is a bit of a slight bump is CV with this. In the process of generating 300D Vectors \n for Word2vec. \n\nWill update as and when I try new things. \n\nRegards\nShanth", "votes": null }, { "id": "331977", "postDate": "05/22/2018 09:41:31", "content": "Thanks shanth for the summary
\n\n\n\n\nSingle Text vector combining title and description
\n
Did it improve your CV ( instead of using 2 text vectors separately ) ?
\n\n\n\n\nSelf Trained Word2Vec Embeddings - 300D Embedding and CV score improves with higher dimensions and more epochs used for training ...
\n
I've just tried 100D Self Trained . May be I will try higher dimension. However it gave me better CV and LB than Fasttext ( the one trained on wiki)
\n\n\n\n\nConcatenation of FastText and Glove - 600 Vector embedding. Dropped the score achieved by Fast Text. Clearly Glove doesn't work for whatever reason.
\n
I was using multi-embedding (Glove + fasttext) on Toxic competition but it was very time consumming for the training ( I used 2M rows though). Given the bigger dataset here. I didn't try it. Good to know it didn't improve.
", "rawMarkdown": "Thanks shanth for the summary \n\n> Single Text vector combining title and description\n\nDid it improve your CV ( instead of using 2 text vectors separately ) ? \n\n> Self Trained Word2Vec Embeddings - 300D Embedding and CV score improves with higher dimensions and more epochs used for training ...\n\nI've just tried 100D Self Trained . May be I will try higher dimension. However it gave me better CV and LB than Fasttext ( the one trained on wiki)\n\n\n>Concatenation of FastText and Glove - 600 Vector embedding. Dropped the score achieved by Fast Text. Clearly Glove doesn't work for whatever reason.\n\nI was using multi-embedding (Glove + fasttext) on Toxic competition but it was very time consumming for the training ( I used 2M rows though). Given the bigger dataset here. I didn't try it. Good to know it didn't improve.", "votes": null }, { "id": "331989", "postDate": "05/22/2018 10:02:30", "content": "@ Seringe
\n\nI never tried individual vectors. I have been doing title+description all along. \n100D self trained actually reduced the CV for me significantly. Mebbe I have made some mistakes. Will go back and check. Thanks for the udpate.
\n\nAlso, for your embedding did you use all of train_active and test_active?
", "rawMarkdown": "Seringe \n\nI never tried individual vectors. I have been doing title+description all along. \n100D self trained actually reduced the CV for me significantly. Mebbe I have made some mistakes. Will go back and check. Thanks for the udpate. \n\nAlso, for your embedding did you use all of train_active and test_active?", "votes": null }, { "id": "332145", "postDate": "05/22/2018 16:13:12", "content": "Just a word of caution on your point, \"( I ended up combining train and test since I was getting errors during the prediction phase that said new labels found )\". Since there are labels unique to the test set, those values should not be embedded they should be deleted and replaced with the \"missing\" value. If you embed them, they will be initialized but never trained where as the \"missing\" embedding will be trained. In fact you should treat the validation set the same as test in this respect for the same reason.
", "rawMarkdown": "Just a word of caution on your point, \"( I ended up combining train and test since I was getting errors during the prediction phase that said new labels found )\". Since there are labels unique to the test set, those values should not be embedded they should be deleted and replaced with the \"missing\" value. If you embed them, they will be initialized but never trained where as the \"missing\" embedding will be trained. In fact you should treat the validation set the same as test in this respect for the same reason.", "votes": null }, { "id": "332161", "postDate": "05/22/2018 16:28:15", "content": "@ Michael \nThanks for that note. Will look into it.
", "rawMarkdown": "Michael \nThanks for that note. Will look into it.", "votes": null }, { "id": "332256", "postDate": "05/22/2018 21:32:19", "content": "I also get significantly better results with self trained 100D vs Russian fastText 300D (note some of the text we have is Ukrainian I think...). Did someone try Cbow vs SkipGram ?
\n\nI'll train 300D during the night first.
", "rawMarkdown": "I also get significantly better results with self trained 100D vs Russian fastText 300D (note some of the text we have is Ukrainian I think...). Did someone try Cbow vs SkipGram ?\n\nI'll train 300D during the night first.", "votes": null }, { "id": "332717", "postDate": "05/23/2018 16:01:29", "content": "@ Seringe
\n\nRan a kernel for title and description as separate vectors and it converged better. But it take twice as much time though.
\n\nMy local CV had a boost of about 0.0005. Yet to submit.
\n\nRegards\nShanth
", "rawMarkdown": "Seringe \n\nRan a kernel for title and description as separate vectors and it converged better. But it take twice as much time though. \n\nMy local CV had a boost of about 0.0005. Yet to submit. \n\nRegards\nShanth", "votes": null }, { "id": "333091", "postDate": "05/24/2018 11:35:18", "content": "Self-trained embeddings with the dimension of 64 worked fine for me. I tried to use frozen fasttext vectors pretrained on wikipedia, but they seem to have quite a small overlap with the vocabulary from avito dataset, so the performance was not very good.\nNow I'm thinking about simultaneously using two sets of embeddings: pretrained fasttext vectors on known words and self-trained embeddings on unknown words.
\n\nAlso I have found params fields and the title field to be complementary to each other. Many ads miss some important information in the title field, while it can be found in the params. Say, a title may not contain the word \"автомобиль\" (a car) which seems to be extremely important for the prediction, while param_1 can represent the whole category (cars). So I tried to process these fields together (just merged all into 1 text field) and it improved my score.
", "rawMarkdown": "Self-trained embeddings with the dimension of 64 worked fine for me. I tried to use frozen fasttext vectors pretrained on wikipedia, but they seem to have quite a small overlap with the vocabulary from avito dataset, so the performance was not very good.\nNow I'm thinking about simultaneously using two sets of embeddings: pretrained fasttext vectors on known words and self-trained embeddings on unknown words. \n\nAlso I have found params fields and the title field to be complementary to each other. Many ads miss some important information in the title field, while it can be found in the params. Say, a title may not contain the word \"автомобиль\" (a car) which seems to be extremely important for the prediction, while param_1 can represent the whole category (cars). So I tried to process these fields together (just merged all into 1 text field) and it improved my score.", "votes": null }, { "id": "333098", "postDate": "05/24/2018 11:53:22", "content": "@ Dimitriy.
\n\nJoining title and param1 seems very intuitive. Good one. I will try it myself.\nThe other idea of using two vectors also sounds interesting.
\n\nRegards\nShanth
", "rawMarkdown": "Dimitriy.\n\nJoining title and param1 seems very intuitive. Good one. I will try it myself.\nThe other idea of using two vectors also sounds interesting. \n\nRegards\nShanth", "votes": null }, { "id": "333200", "postDate": "05/24/2018 16:13:51", "content": "I also trained some embeddings using image_top_1 as target. You can check an example in this kernel.
", "rawMarkdown": "I also trained some embeddings using image_top_1 as target. You can check an example in this [kernel][1].\n\n\n [1]: https://www.kaggle.com/christofhenkel/text2image-top-1", "votes": null }, { "id": "333216", "postDate": "05/24/2018 17:16:55", "content": "@ Dieter
\n\nThat's a great way to approach it. Will take a look at it. :)
", "rawMarkdown": "Dieter \n\nThat's a great way to approach it. Will take a look at it. :)", "votes": null } ], "comments": [ { "id": 331977, "author_name": "serigne", "author_url": "", "post_date": "05/22/2018 09:41:31", "content": "Thanks shanth for the summary
\n\n\n\n\nSingle Text vector combining title and description
\n
Did it improve your CV ( instead of using 2 text vectors separately ) ?
\n\n\n\n\nSelf Trained Word2Vec Embeddings - 300D Embedding and CV score improves with higher dimensions and more epochs used for training ...
\n
I've just tried 100D Self Trained . May be I will try higher dimension. However it gave me better CV and LB than Fasttext ( the one trained on wiki)
\n\n\n\n\nConcatenation of FastText and Glove - 600 Vector embedding. Dropped the score achieved by Fast Text. Clearly Glove doesn't work for whatever reason.
\n
I was using multi-embedding (Glove + fasttext) on Toxic competition but it was very time consumming for the training ( I used 2M rows though). Given the bigger dataset here. I didn't try it. Good to know it didn't improve.
", "votes": null, "replies": [ { "id": 331989, "author_name": "shanth84", "author_url": "", "post_date": "05/22/2018 10:02:30", "content": "@ Seringe
\n\nI never tried individual vectors. I have been doing title+description all along. \n100D self trained actually reduced the CV for me significantly. Mebbe I have made some mistakes. Will go back and check. Thanks for the udpate.
\n\nAlso, for your embedding did you use all of train_active and test_active?
", "votes": null, "replies": [] }, { "id": 332256, "author_name": "arroqc", "author_url": "", "post_date": "05/22/2018 21:32:19", "content": "I also get significantly better results with self trained 100D vs Russian fastText 300D (note some of the text we have is Ukrainian I think...). Did someone try Cbow vs SkipGram ?
\n\nI'll train 300D during the night first.
", "votes": null, "replies": [] }, { "id": 332717, "author_name": "shanth84", "author_url": "", "post_date": "05/23/2018 16:01:29", "content": "@ Seringe
\n\nRan a kernel for title and description as separate vectors and it converged better. But it take twice as much time though.
\n\nMy local CV had a boost of about 0.0005. Yet to submit.
\n\nRegards\nShanth
", "votes": null, "replies": [] } ] }, { "id": 332145, "author_name": "vannak", "author_url": "", "post_date": "05/22/2018 16:13:12", "content": "Just a word of caution on your point, \"( I ended up combining train and test since I was getting errors during the prediction phase that said new labels found )\". Since there are labels unique to the test set, those values should not be embedded they should be deleted and replaced with the \"missing\" value. If you embed them, they will be initialized but never trained where as the \"missing\" embedding will be trained. In fact you should treat the validation set the same as test in this respect for the same reason.
", "votes": null, "replies": [ { "id": 332161, "author_name": "shanth84", "author_url": "", "post_date": "05/22/2018 16:28:15", "content": "@ Michael \nThanks for that note. Will look into it.
", "votes": null, "replies": [] } ] }, { "id": 333091, "author_name": "ddanevskyi", "author_url": "", "post_date": "05/24/2018 11:35:18", "content": "Self-trained embeddings with the dimension of 64 worked fine for me. I tried to use frozen fasttext vectors pretrained on wikipedia, but they seem to have quite a small overlap with the vocabulary from avito dataset, so the performance was not very good.\nNow I'm thinking about simultaneously using two sets of embeddings: pretrained fasttext vectors on known words and self-trained embeddings on unknown words.
\n\nAlso I have found params fields and the title field to be complementary to each other. Many ads miss some important information in the title field, while it can be found in the params. Say, a title may not contain the word \"автомобиль\" (a car) which seems to be extremely important for the prediction, while param_1 can represent the whole category (cars). So I tried to process these fields together (just merged all into 1 text field) and it improved my score.
", "votes": null, "replies": [ { "id": 333098, "author_name": "shanth84", "author_url": "", "post_date": "05/24/2018 11:53:22", "content": "@ Dimitriy.
\n\nJoining title and param1 seems very intuitive. Good one. I will try it myself.\nThe other idea of using two vectors also sounds interesting.
\n\nRegards\nShanth
", "votes": null, "replies": [] } ] }, { "id": 333200, "author_name": "christofhenkel", "author_url": "", "post_date": "05/24/2018 16:13:51", "content": "I also trained some embeddings using image_top_1 as target. You can check an example in this kernel.
", "votes": null, "replies": [ { "id": 333216, "author_name": "shanth84", "author_url": "", "post_date": "05/24/2018 17:16:55", "content": "@ Dieter
\n\nThat's a great way to approach it. Will take a look at it. :)
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "331911": "I am listing down somethings that worked for me in my RNN network. \n\nApproach for Word Processing: \n\n- Single Text vector combining title and description \n- Zero word processing \n- Vocab size is fairly large at 200K \n\nCategorical Variables \n\n- Label Encoding on Train+ Test ( I ended up combining train and test since I was getting errors \n during the prediction phase that said new labels found )\n\n- Log or Log1p conversions for numerical values such as price \n\nEmbedding Approaches tried so far - \n\n1. Self Trained Word2Vec Embeddings - 300D\n Embedding and CV score improves with higher dimensions and more epochs used for training \n\n2. Architecture - Adding more dense layers is dropping Local CV. \n\n3. FastText for Russian - Best Score So far for the RNN network I used. 300D\n\n4. Russian Glove - Lowest score. A very large portion of the vocab was found missing. I am guessing that weakened the model since I was simply replacing missing words with a zeros (300D)\n\n5. Concatenation of FastText and Glove - 600 Vector embedding. \n Dropped the score achieved by Fast Text. Clearly Glove doesn't work for whatever reason. \n\n6. Concatenation of FastText and Self Trained Word2vec \n\n I think there is a bit of a slight bump is CV with this. In the process of generating 300D Vectors \n for Word2vec. \n\nWill update as and when I try new things. \n\nRegards\nShanth", "331977": "Thanks shanth for the summary \n\n> Single Text vector combining title and description\n\nDid it improve your CV ( instead of using 2 text vectors separately ) ? \n\n> Self Trained Word2Vec Embeddings - 300D Embedding and CV score improves with higher dimensions and more epochs used for training ...\n\nI've just tried 100D Self Trained . May be I will try higher dimension. However it gave me better CV and LB than Fasttext ( the one trained on wiki)\n\n\n>Concatenation of FastText and Glove - 600 Vector embedding. Dropped the score achieved by Fast Text. Clearly Glove doesn't work for whatever reason.\n\nI was using multi-embedding (Glove + fasttext) on Toxic competition but it was very time consumming for the training ( I used 2M rows though). Given the bigger dataset here. I didn't try it. Good to know it didn't improve.", "331989": "Seringe \n\nI never tried individual vectors. I have been doing title+description all along. \n100D self trained actually reduced the CV for me significantly. Mebbe I have made some mistakes. Will go back and check. Thanks for the udpate. \n\nAlso, for your embedding did you use all of train_active and test_active?", "332145": "Just a word of caution on your point, \"( I ended up combining train and test since I was getting errors during the prediction phase that said new labels found )\". Since there are labels unique to the test set, those values should not be embedded they should be deleted and replaced with the \"missing\" value. If you embed them, they will be initialized but never trained where as the \"missing\" embedding will be trained. In fact you should treat the validation set the same as test in this respect for the same reason.", "332161": "Michael \nThanks for that note. Will look into it.", "332256": "I also get significantly better results with self trained 100D vs Russian fastText 300D (note some of the text we have is Ukrainian I think...). Did someone try Cbow vs SkipGram ?\n\nI'll train 300D during the night first.", "332717": "Seringe \n\nRan a kernel for title and description as separate vectors and it converged better. But it take twice as much time though. \n\nMy local CV had a boost of about 0.0005. Yet to submit. \n\nRegards\nShanth", "333091": "Self-trained embeddings with the dimension of 64 worked fine for me. I tried to use frozen fasttext vectors pretrained on wikipedia, but they seem to have quite a small overlap with the vocabulary from avito dataset, so the performance was not very good.\nNow I'm thinking about simultaneously using two sets of embeddings: pretrained fasttext vectors on known words and self-trained embeddings on unknown words. \n\nAlso I have found params fields and the title field to be complementary to each other. Many ads miss some important information in the title field, while it can be found in the params. Say, a title may not contain the word \"автомобиль\" (a car) which seems to be extremely important for the prediction, while param_1 can represent the whole category (cars). So I tried to process these fields together (just merged all into 1 text field) and it improved my score.", "333098": "Dimitriy.\n\nJoining title and param1 seems very intuitive. Good one. I will try it myself.\nThe other idea of using two vectors also sounds interesting. \n\nRegards\nShanth", "333200": "I also trained some embeddings using image_top_1 as target. You can check an example in this [kernel][1].\n\n\n [1]: https://www.kaggle.com/christofhenkel/text2image-top-1", "333216": "Dieter \n\nThat's a great way to approach it. Will take a look at it. :)" }, "source": "meta" }