My teammate mentioned averaging all 3 embeddings caused a lower score. I was wondering if anyone did anything different?
", "messages": [ { "id": "447399", "postDate": "12/29/2018 18:58:24", "content": "My teammate mentioned averaging all 3 embeddings caused a lower score. I was wondering if anyone did anything different?
", "rawMarkdown": "My teammate mentioned averaging all 3 embeddings caused a lower score. I was wondering if anyone did anything different?", "votes": null }, { "id": "447476", "postDate": "12/29/2018 22:34:53", "content": "A blending of different embeddings doesn't make sense. Just because they may share same words, it does not mean that they were placed on the same n-dimensional space. There are papers that explain this reasoning. There are a couple of libraries that attempt to handle multiple embeddings such as FastText and Flair.
", "rawMarkdown": "A blending of different embeddings doesn't make sense. Just because they may share same words, it does not mean that they were placed on the same n-dimensional space. There are papers that explain this reasoning. There are a couple of libraries that attempt to handle multiple embeddings such as FastText and Flair.", "votes": null }, { "id": "447506", "postDate": "12/30/2018 01:12:23", "content": "By blending, I mean concatenating or averaging two or more word embeddings. How does that not make sense?
", "rawMarkdown": "By blending, I mean concatenating or averaging two or more word embeddings. How does that not make sense?", "votes": null }, { "id": "447520", "postDate": "12/30/2018 01:58:56", "content": "You said “averaging” so I'm just commenting on that method and why it probably doesn't work as well - just as your teammate reported.. and google is your friend: https://stackoverflow.com/questions/49451160/how-to-combine-two-pre-trained-word2vec-models. There are other resources on the web about this and averaging... Maybe try some reading instead of relying on copying and pasting kernels because there tends to be a lot of misleading methods or they haven't been thoroughly researched/tested.
", "rawMarkdown": "You said “averaging” so I'm just commenting on that method and why it probably doesn't work as well - just as your teammate reported.. and google is your friend: https://stackoverflow.com/questions/49451160/how-to-combine-two-pre-trained-word2vec-models. There are other resources on the web about this and averaging... Maybe try some reading instead of relying on copying and pasting kernels because there tends to be a lot of misleading methods or they haven't been thoroughly researched/tested.", "votes": null }, { "id": "447528", "postDate": "12/30/2018 02:32:57", "content": "Actually 'averaging' (counter-intuitively since it's not a common embedding space) actually works surprisingly well. This is discussed and analyzed in this paper: http://aclweb.org/anthology/N18-2031
", "rawMarkdown": "Actually 'averaging' (counter-intuitively since it's not a common embedding space) actually works surprisingly well. This is discussed and analyzed in this paper: http://aclweb.org/anthology/N18-2031", "votes": null }, { "id": "447601", "postDate": "12/30/2018 06:44:21", "content": "I have done much research and read many research papers. A few weeks ago, I didn't know anything about NLP and now I have a decent amount of knowledge on this amazing branch of computer science and deep learning, but there will always be things I don't know. Don't get pissy for me asking a machine learning question on a machine learning discussion board.
", "rawMarkdown": "I have done much research and read many research papers. A few weeks ago, I didn't know anything about NLP and now I have a decent amount of knowledge on this amazing branch of computer science and deep learning, but there will always be things I don't know. Don't get pissy for me asking a machine learning question on a machine learning discussion board.", "votes": null }, { "id": "447603", "postDate": "12/30/2018 06:45:40", "content": "I've read that one Steve Draper. Fascinating stuff, especially when they showed the result of vector distances in average and concatenated meta-embeddings. I just was wonder why doing this with 3 embeddings causes the accuracy to decrease.
", "rawMarkdown": "I've read that one Steve Draper. Fascinating stuff, especially when they showed the result of vector distances in average and concatenated meta-embeddings. I just was wonder why doing this with 3 embeddings causes the accuracy to decrease.", "votes": null } ], "comments": [ { "id": 447476, "author_name": "learnmower", "author_url": "", "post_date": "12/29/2018 22:34:53", "content": "A blending of different embeddings doesn't make sense. Just because they may share same words, it does not mean that they were placed on the same n-dimensional space. There are papers that explain this reasoning. There are a couple of libraries that attempt to handle multiple embeddings such as FastText and Flair.
", "votes": null, "replies": [ { "id": 447506, "author_name": "pocketmad", "author_url": "", "post_date": "12/30/2018 01:12:23", "content": "By blending, I mean concatenating or averaging two or more word embeddings. How does that not make sense?
", "votes": null, "replies": [] }, { "id": 447520, "author_name": "learnmower", "author_url": "", "post_date": "12/30/2018 01:58:56", "content": "You said “averaging” so I'm just commenting on that method and why it probably doesn't work as well - just as your teammate reported.. and google is your friend: https://stackoverflow.com/questions/49451160/how-to-combine-two-pre-trained-word2vec-models. There are other resources on the web about this and averaging... Maybe try some reading instead of relying on copying and pasting kernels because there tends to be a lot of misleading methods or they haven't been thoroughly researched/tested.
", "votes": null, "replies": [] }, { "id": 447528, "author_name": "stevedraper", "author_url": "", "post_date": "12/30/2018 02:32:57", "content": "Actually 'averaging' (counter-intuitively since it's not a common embedding space) actually works surprisingly well. This is discussed and analyzed in this paper: http://aclweb.org/anthology/N18-2031
", "votes": null, "replies": [] }, { "id": 447601, "author_name": "pocketmad", "author_url": "", "post_date": "12/30/2018 06:44:21", "content": "I have done much research and read many research papers. A few weeks ago, I didn't know anything about NLP and now I have a decent amount of knowledge on this amazing branch of computer science and deep learning, but there will always be things I don't know. Don't get pissy for me asking a machine learning question on a machine learning discussion board.
", "votes": null, "replies": [] }, { "id": 447603, "author_name": "pocketmad", "author_url": "", "post_date": "12/30/2018 06:45:40", "content": "I've read that one Steve Draper. Fascinating stuff, especially when they showed the result of vector distances in average and concatenated meta-embeddings. I just was wonder why doing this with 3 embeddings causes the accuracy to decrease.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "447399": "My teammate mentioned averaging all 3 embeddings caused a lower score. I was wondering if anyone did anything different?", "447476": "A blending of different embeddings doesn't make sense. Just because they may share same words, it does not mean that they were placed on the same n-dimensional space. There are papers that explain this reasoning. There are a couple of libraries that attempt to handle multiple embeddings such as FastText and Flair.", "447506": "By blending, I mean concatenating or averaging two or more word embeddings. How does that not make sense?", "447520": "You said “averaging” so I'm just commenting on that method and why it probably doesn't work as well - just as your teammate reported.. and google is your friend: https://stackoverflow.com/questions/49451160/how-to-combine-two-pre-trained-word2vec-models. There are other resources on the web about this and averaging... Maybe try some reading instead of relying on copying and pasting kernels because there tends to be a lot of misleading methods or they haven't been thoroughly researched/tested.", "447528": "Actually 'averaging' (counter-intuitively since it's not a common embedding space) actually works surprisingly well. This is discussed and analyzed in this paper: http://aclweb.org/anthology/N18-2031", "447601": "I have done much research and read many research papers. A few weeks ago, I didn't know anything about NLP and now I have a decent amount of knowledge on this amazing branch of computer science and deep learning, but there will always be things I don't know. Don't get pissy for me asking a machine learning question on a machine learning discussion board.", "447603": "I've read that one Steve Draper. Fascinating stuff, especially when they showed the result of vector distances in average and concatenated meta-embeddings. I just was wonder why doing this with 3 embeddings causes the accuracy to decrease." }, "source": "meta" }