![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/173459/6298/f81e92c0-472b-4770-b808-1abdd9376edf-original.png\")
Stolen from @ryches who stole from an old post from @tunguz
\n\n
Stolen from @ryches who stole from an old post from @tunguz
\n\n
Anyway while we are on the matter of word to vec in this thread, What embedding preprocessing is working best for everyone? We have many options :
\n\n6 . Weighted average of 2+ embeds
\n\n7 . Attention mechanism which embed to use for each word individually (computationally expensive)
", "rawMarkdown": "6 . Weighted average of 2+ embeds\n\n7 . Attention mechanism which embed to use for each word individually (computationally expensive)", "votes": null }, { "id": "447215", "postDate": "12/29/2018 11:31:17", "content": "weighted embed is a nice Idea. Any idea why all the kernels are not using Fasttext embedding?
", "rawMarkdown": "weighted embed is a nice Idea. Any idea why all the kernels are not using Fasttext embedding?", "votes": null }, { "id": "447221", "postDate": "12/29/2018 11:44:01", "content": "@Dieter How do you decide on weights for weighted average? @mlwhiz: Trial and error I guess and mismatch is larger.
", "rawMarkdown": "Dieter How do you decide on weights for weighted average? @mlwhiz: Trial and error I guess and mismatch is larger.", "votes": null }, { "id": "447223", "postDate": "12/29/2018 11:47:31", "content": "In my experience, if one has to do weighted average between Glove and Paragram, more weightage to be given to glove. Somehow my model CV is higher with only using single glove embedding. Tanks on LB though.
", "rawMarkdown": "In my experience, if one has to do weighted average between Glove and Paragram, more weightage to be given to glove. Somehow my model CV is higher with only using single glove embedding. Tanks on LB though.", "votes": null }, { "id": "447224", "postDate": "12/29/2018 11:51:19", "content": "Haha! That's why I prefer Tea at new places. Less chances of overfitting and tea generalizes well :)
", "rawMarkdown": "Haha! That's why I prefer Tea at new places. Less chances of overfitting and tea generalizes well :)", "votes": null }, { "id": "447226", "postDate": "12/29/2018 11:57:22", "content": "@philippsinger I train a model which has a Dense(4) with softmax activation layer to combine embeddings and then check weights
", "rawMarkdown": "philippsinger I train a model which has a Dense(4) with softmax activation layer to combine embeddings and then check weights", "votes": null }, { "id": "447229", "postDate": "12/29/2018 12:03:20", "content": "@Dieter neat! Do you have that as part of your overall model or seperately? I guess one could also just do hyperparameter tuning on the weights. I have observed similar things as @mlwhiz though that the LB score is quite sensitive on that choice.
", "rawMarkdown": "Dieter neat! Do you have that as part of your overall model or seperately? I guess one could also just do hyperparameter tuning on the weights. I have observed similar things as @mlwhiz though that the LB score is quite sensitive on that choice.", "votes": null }, { "id": "447826", "postDate": "12/30/2018 16:52:12", "content": "So if you don't mind which one are you using in your current model - mean/concat/single glove. :)
", "rawMarkdown": "So if you don't mind which one are you using in your current model - mean/concat/single glove. :)", "votes": null }, { "id": "447945", "postDate": "12/30/2018 21:59:53", "content": "It's a tradeoff between precision and recall.
", "rawMarkdown": "It's a tradeoff between precision and recall.", "votes": null }, { "id": "447948", "postDate": "12/30/2018 22:11:28", "content": "dieter,\nCan I know what are the inputs to dense layer?\nMaking a dense layer 4(activation='softmax') means that the output of dense layer will contain weights of each embedding matrix?
\n\nThanks
", "rawMarkdown": "dieter,\nCan I know what are the inputs to dense layer?\nMaking a dense layer 4(activation='softmax') means that the output of dense layer will contain weights of each embedding matrix?\n\nThanks", "votes": null }, { "id": "448102", "postDate": "12/31/2018 08:46:33", "content": "haha
", "rawMarkdown": "haha", "votes": null }, { "id": "448273", "postDate": "12/31/2018 16:45:18", "content": "-8. Use cosine distance between embeddings for words as a feature (how are ambiguity and insincerity related?)\n-9. Train all embeddings for a couple of epochs then look at which word vectors changed the most and use this to help create other features\n-10. Train only the embeddings for words that are out of vocab (tricky to implement)\n-11. Train all embeddings for an epoch but don't actually update them - instead measure the magnitude of the attempted gradient updates for each word. Then train all words the model most wanted to update (i.e. above some gradient threshold) and freeze everything else.\n-12. Use the average of any synonyms word vectors for words that are out of vocab.
", "rawMarkdown": "8. Use cosine distance between embeddings for words as a feature (how are ambiguity and insincerity related?)\n-9. Train all embeddings for a couple of epochs then look at which word vectors changed the most and use this to help create other features\n-10. Train only the embeddings for words that are out of vocab (tricky to implement)\n-11. Train all embeddings for an epoch but don't actually update them - instead measure the magnitude of the attempted gradient updates for each word. Then train all words the model most wanted to update (i.e. above some gradient threshold) and freeze everything else.\n-12. Use the average of any synonyms word vectors for words that are out of vocab.", "votes": null }, { "id": "448563", "postDate": "01/01/2019 13:58:10", "content": "In response to the comic, I think espresso and cappuccino should be the considered same thing.
\n\n\"Why don't liberals realise that ....\" and \"Why don't feminists realise that ...\" tend to be insincere questions.
\n\nWe should focus on question form. We can do it by decontextualising every question. Replace \"liberals\" with any other group of people and the question should be judged as insincere.
\n\nQuora should use this approach as well since they do not want to be seen as biased and their dataset may be biased.
", "rawMarkdown": "In response to the comic, I think espresso and cappuccino should be the considered same thing.\n\n\"Why don't liberals realise that ....\" and \"Why don't feminists realise that ...\" tend to be insincere questions.\n\nWe should focus on question form. We can do it by decontextualising every question. Replace \"liberals\" with any other group of people and the question should be judged as insincere.\n\nQuora should use this approach as well since they do not want to be seen as biased and their dataset may be biased.", "votes": null }, { "id": "461711", "postDate": "01/26/2019 19:58:23", "content": "If you take mean of 3 embeddings, are they in equal position?
\n\nI suppose if we build embedding several times, with different random, we will not get same result, but each time different.
\n\nOne time word x1 might locate high positive value are and x2 other side of origo on negative. On other trained embedding they could be opposite: x1 negative and x2 positive - sort of rotated \"180 degrees\". Kind of symmetric, but whole embedding (the words) rotated.
\n\nSo if 2 embeddings would place word x1 on positive and 3d as negative, how does that work for taking a mean?
\n\nOr have people who pretrained glove, paragraph, fasttext somehow tried to make em compatible by trying to rotate words to similar areas in the space?
", "rawMarkdown": "If you take mean of 3 embeddings, are they in equal position?\n\nI suppose if we build embedding several times, with different random, we will not get same result, but each time different.\n\nOne time word x1 might locate high positive value are and x2 other side of origo on negative. On other trained embedding they could be opposite: x1 negative and x2 positive - sort of rotated \"180 degrees\". Kind of symmetric, but whole embedding (the words) rotated.\n\nSo if 2 embeddings would place word x1 on positive and 3d as negative, how does that work for taking a mean?\n\nOr have people who pretrained glove, paragraph, fasttext somehow tried to make em compatible by trying to rotate words to similar areas in the space?", "votes": null }, { "id": "687240", "postDate": "12/04/2019 05:34:58", "content": "👍
", "rawMarkdown": "👍", "votes": null }, { "id": "1077490", "postDate": "11/13/2020 16:33:46", "content": "Notebook : https://www.kaggle.com/naim99/text-clustering-with-word2vec
\nMy dataset : https://www.kaggle.com/naim99/ts-naim-mhedhbi
Check this kernel for Arabic Text Analysis . I hope you find it useful and informative.
", "rawMarkdown": "Notebook : https://www.kaggle.com/naim99/text-clustering-with-word2vec \nMy dataset : https://www.kaggle.com/naim99/ts-naim-mhedhbi\n\nCheck this kernel for Arabic Text Analysis . I hope you find it useful and informative.", "votes": null }, { "id": "2055628", "postDate": "12/05/2022 08:46:23", "content": "hahahahahahahah😂This is so fun!
", "rawMarkdown": "hahahahahahahah😂This is so fun!", "votes": null } ], "comments": [ { "id": 1077490, "author_name": "naim99", "author_url": "", "post_date": "11/13/2020 16:33:46", "content": "Notebook : https://www.kaggle.com/naim99/text-clustering-with-word2vec
\nMy dataset : https://www.kaggle.com/naim99/ts-naim-mhedhbi
Check this kernel for Arabic Text Analysis . I hope you find it useful and informative.
", "votes": null, "replies": [] }, { "id": 2055628, "author_name": "jiangyunyao", "author_url": "", "post_date": "12/05/2022 08:46:23", "content": "hahahahahahahah😂This is so fun!
", "votes": null, "replies": [] }, { "id": 447210, "author_name": "mlwhiz", "author_url": "", "post_date": "12/29/2018 11:21:31", "content": "Anyway while we are on the matter of word to vec in this thread, What embedding preprocessing is working best for everyone? We have many options :
\n\n6 . Weighted average of 2+ embeds
\n\n7 . Attention mechanism which embed to use for each word individually (computationally expensive)
", "votes": null, "replies": [] }, { "id": 447215, "author_name": "mlwhiz", "author_url": "", "post_date": "12/29/2018 11:31:17", "content": "weighted embed is a nice Idea. Any idea why all the kernels are not using Fasttext embedding?
", "votes": null, "replies": [] }, { "id": 447221, "author_name": "philippsinger", "author_url": "", "post_date": "12/29/2018 11:44:01", "content": "@Dieter How do you decide on weights for weighted average? @mlwhiz: Trial and error I guess and mismatch is larger.
", "votes": null, "replies": [] }, { "id": 447223, "author_name": "mlwhiz", "author_url": "", "post_date": "12/29/2018 11:47:31", "content": "In my experience, if one has to do weighted average between Glove and Paragram, more weightage to be given to glove. Somehow my model CV is higher with only using single glove embedding. Tanks on LB though.
", "votes": null, "replies": [] }, { "id": 447226, "author_name": "christofhenkel", "author_url": "", "post_date": "12/29/2018 11:57:22", "content": "@philippsinger I train a model which has a Dense(4) with softmax activation layer to combine embeddings and then check weights
", "votes": null, "replies": [] }, { "id": 447229, "author_name": "philippsinger", "author_url": "", "post_date": "12/29/2018 12:03:20", "content": "@Dieter neat! Do you have that as part of your overall model or seperately? I guess one could also just do hyperparameter tuning on the weights. I have observed similar things as @mlwhiz though that the LB score is quite sensitive on that choice.
", "votes": null, "replies": [] }, { "id": 447826, "author_name": "mlwhiz", "author_url": "", "post_date": "12/30/2018 16:52:12", "content": "So if you don't mind which one are you using in your current model - mean/concat/single glove. :)
", "votes": null, "replies": [] }, { "id": 447948, "author_name": "suchith0312", "author_url": "", "post_date": "12/30/2018 22:11:28", "content": "dieter,\nCan I know what are the inputs to dense layer?\nMaking a dense layer 4(activation='softmax') means that the output of dense layer will contain weights of each embedding matrix?
\n\nThanks
", "votes": null, "replies": [] }, { "id": 448273, "author_name": "maw501", "author_url": "", "post_date": "12/31/2018 16:45:18", "content": "-8. Use cosine distance between embeddings for words as a feature (how are ambiguity and insincerity related?)\n-9. Train all embeddings for a couple of epochs then look at which word vectors changed the most and use this to help create other features\n-10. Train only the embeddings for words that are out of vocab (tricky to implement)\n-11. Train all embeddings for an epoch but don't actually update them - instead measure the magnitude of the attempted gradient updates for each word. Then train all words the model most wanted to update (i.e. above some gradient threshold) and freeze everything else.\n-12. Use the average of any synonyms word vectors for words that are out of vocab.
", "votes": null, "replies": [] }, { "id": 461711, "author_name": "jannen", "author_url": "", "post_date": "01/26/2019 19:58:23", "content": "If you take mean of 3 embeddings, are they in equal position?
\n\nI suppose if we build embedding several times, with different random, we will not get same result, but each time different.
\n\nOne time word x1 might locate high positive value are and x2 other side of origo on negative. On other trained embedding they could be opposite: x1 negative and x2 positive - sort of rotated \"180 degrees\". Kind of symmetric, but whole embedding (the words) rotated.
\n\nSo if 2 embeddings would place word x1 on positive and 3d as negative, how does that work for taking a mean?
\n\nOr have people who pretrained glove, paragraph, fasttext somehow tried to make em compatible by trying to rotate words to similar areas in the space?
", "votes": null, "replies": [] } ] }, { "id": 447224, "author_name": "shaz13", "author_url": "", "post_date": "12/29/2018 11:51:19", "content": "Haha! That's why I prefer Tea at new places. Less chances of overfitting and tea generalizes well :)
", "votes": null, "replies": [] }, { "id": 447945, "author_name": "ma2rten", "author_url": "", "post_date": "12/30/2018 21:59:53", "content": "It's a tradeoff between precision and recall.
", "votes": null, "replies": [] }, { "id": 448102, "author_name": "kailashnath1998", "author_url": "", "post_date": "12/31/2018 08:46:33", "content": "haha
", "votes": null, "replies": [] }, { "id": 448563, "author_name": "huikang", "author_url": "", "post_date": "01/01/2019 13:58:10", "content": "In response to the comic, I think espresso and cappuccino should be the considered same thing.
\n\n\"Why don't liberals realise that ....\" and \"Why don't feminists realise that ...\" tend to be insincere questions.
\n\nWe should focus on question form. We can do it by decontextualising every question. Replace \"liberals\" with any other group of people and the question should be judged as insincere.
\n\nQuora should use this approach as well since they do not want to be seen as biased and their dataset may be biased.
", "votes": null, "replies": [] }, { "id": 687240, "author_name": "kaledonec", "author_url": "", "post_date": "12/04/2019 05:34:58", "content": "👍
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "447202": "Stolen from [@ryches][1] who stole from an old post from [@tunguz][2]\n\n![enter image description here][3]\n\n\n [1]: https://www.kaggle.com/ryches\n [2]: https://www.kaggle.com/tunguz\n [3]: https://storage.googleapis.com/kaggle-forum-message-attachments/173459/6298/f81e92c0-472b-4770-b808-1abdd9376edf-original.png", "447210": "Anyway while we are on the matter of word to vec in this thread, What embedding preprocessing is working best for everyone? We have many options :\n\n1. Concat of all 3 embeds\n2. Mean of all 3 embeds \n3. Concat/Mean of any 2 embeds\n4. Single embed\n5. train from scratch an embedding matrix as part of the network? Time consuming?", "447212": "6 . Weighted average of 2+ embeds\n\n7 . Attention mechanism which embed to use for each word individually (computationally expensive)", "447215": "weighted embed is a nice Idea. Any idea why all the kernels are not using Fasttext embedding?", "447221": "Dieter How do you decide on weights for weighted average? @mlwhiz: Trial and error I guess and mismatch is larger.", "447223": "In my experience, if one has to do weighted average between Glove and Paragram, more weightage to be given to glove. Somehow my model CV is higher with only using single glove embedding. Tanks on LB though.", "447224": "Haha! That's why I prefer Tea at new places. Less chances of overfitting and tea generalizes well :)", "447226": "philippsinger I train a model which has a Dense(4) with softmax activation layer to combine embeddings and then check weights", "447229": "Dieter neat! Do you have that as part of your overall model or seperately? I guess one could also just do hyperparameter tuning on the weights. I have observed similar things as @mlwhiz though that the LB score is quite sensitive on that choice.", "447826": "So if you don't mind which one are you using in your current model - mean/concat/single glove. :)", "447945": "It's a tradeoff between precision and recall.", "447948": "dieter,\nCan I know what are the inputs to dense layer?\nMaking a dense layer 4(activation='softmax') means that the output of dense layer will contain weights of each embedding matrix?\n\nThanks", "448102": "haha", "448273": "8. Use cosine distance between embeddings for words as a feature (how are ambiguity and insincerity related?)\n-9. Train all embeddings for a couple of epochs then look at which word vectors changed the most and use this to help create other features\n-10. Train only the embeddings for words that are out of vocab (tricky to implement)\n-11. Train all embeddings for an epoch but don't actually update them - instead measure the magnitude of the attempted gradient updates for each word. Then train all words the model most wanted to update (i.e. above some gradient threshold) and freeze everything else.\n-12. Use the average of any synonyms word vectors for words that are out of vocab.", "448563": "In response to the comic, I think espresso and cappuccino should be the considered same thing.\n\n\"Why don't liberals realise that ....\" and \"Why don't feminists realise that ...\" tend to be insincere questions.\n\nWe should focus on question form. We can do it by decontextualising every question. Replace \"liberals\" with any other group of people and the question should be judged as insincere.\n\nQuora should use this approach as well since they do not want to be seen as biased and their dataset may be biased.", "461711": "If you take mean of 3 embeddings, are they in equal position?\n\nI suppose if we build embedding several times, with different random, we will not get same result, but each time different.\n\nOne time word x1 might locate high positive value are and x2 other side of origo on negative. On other trained embedding they could be opposite: x1 negative and x2 positive - sort of rotated \"180 degrees\". Kind of symmetric, but whole embedding (the words) rotated.\n\nSo if 2 embeddings would place word x1 on positive and 3d as negative, how does that work for taking a mean?\n\nOr have people who pretrained glove, paragraph, fasttext somehow tried to make em compatible by trying to rotate words to similar areas in the space?", "687240": "👍", "1077490": "Notebook : https://www.kaggle.com/naim99/text-clustering-with-word2vec \nMy dataset : https://www.kaggle.com/naim99/ts-naim-mhedhbi\n\nCheck this kernel for Arabic Text Analysis . I hope you find it useful and informative.", "2055628": "hahahahahahahah😂This is so fun!" }, "source": "meta" }