Do you know how training set was labeled? Among my misclassified questions I see many questions that were labeled as insincere but look completely ok for me. For example I just checked top 100 msclassified questions:
\n\nWhat is freight forwarder management system?\nHow can I register as a freight forwarder in international services?\nWhere can I find android app developers?\nWhy do you have to select painting based on your furniture and fabrics?\nHow do I choose an apartment in Edmonton?\nHow do students get an academic update?\nWhich is the best best TRUCK TIRE CHAINS?\nWhat’s the coolest place you have been to in Chicago?\nWhat are the names of woman athletes participating in the 2018 Olympics?\nWhich low-investment businesses are most profitable in the world?\nHow does Luminary Cream function?\nWhat did you think of the Berserk movies?\nWho were the Elves? Tolkien is supposed to be inspired by folklore but some real history may be hiding behind the myth!\nWhat star were you born under?\nWhat is the greatest amount of blood you can donate?\nWhere can I find best romantic shayaris?
\n\nAnd I didn't mentioned here many more questions that I would say are sincere, but have some doubts.
\n\nSuch questions can make model teach something wrong. But relabeling it could cause model train something else, not what Quora expects. So again, it would be interesting to know what is the policy regarding insincerity, how labeling is performed.
", "messages": [ { "id": 429411, "postDate": "2018-11-28T20:15:42.550Z", "content": "Do you know how training set was labeled? Among my misclassified questions I see many questions that were labeled as insincere but look completely ok for me. For example I just checked top 100 msclassified questions:
\n\nWhat is freight forwarder management system?\nHow can I register as a freight forwarder in international services?\nWhere can I find android app developers?\nWhy do you have to select painting based on your furniture and fabrics?\nHow do I choose an apartment in Edmonton?\nHow do students get an academic update?\nWhich is the best best TRUCK TIRE CHAINS?\nWhat’s the coolest place you have been to in Chicago?\nWhat are the names of woman athletes participating in the 2018 Olympics?\nWhich low-investment businesses are most profitable in the world?\nHow does Luminary Cream function?\nWhat did you think of the Berserk movies?\nWho were the Elves? Tolkien is supposed to be inspired by folklore but some real history may be hiding behind the myth!\nWhat star were you born under?\nWhat is the greatest amount of blood you can donate?\nWhere can I find best romantic shayaris?
\n\nAnd I didn't mentioned here many more questions that I would say are sincere, but have some doubts.
\n\nSuch questions can make model teach something wrong. But relabeling it could cause model train something else, not what Quora expects. So again, it would be interesting to know what is the policy regarding insincerity, how labeling is performed.
", "rawMarkdown": "Do you know how training set was labeled? Among my misclassified questions I see many questions that were labeled as insincere but look completely ok for me. For example I just checked top 100 msclassified questions: \n\nWhat is freight forwarder management system?\nHow can I register as a freight forwarder in international services?\nWhere can I find android app developers?\nWhy do you have to select painting based on your furniture and fabrics?\nHow do I choose an apartment in Edmonton?\nHow do students get an academic update?\nWhich is the best best TRUCK TIRE CHAINS?\nWhat’s the coolest place you have been to in Chicago?\nWhat are the names of woman athletes participating in the 2018 Olympics?\nWhich low-investment businesses are most profitable in the world?\nHow does Luminary Cream function?\nWhat did you think of the Berserk movies?\nWho were the Elves? Tolkien is supposed to be inspired by folklore but some real history may be hiding behind the myth!\nWhat star were you born under?\nWhat is the greatest amount of blood you can donate?\nWhere can I find best romantic shayaris?\n\nAnd I didn't mentioned here many more questions that I would say are sincere, but have some doubts.\n\nSuch questions can make model teach something wrong. But relabeling it could cause model train something else, not what Quora expects. So again, it would be interesting to know what is the policy regarding insincerity, how labeling is performed.", "votes": 18 }, { "id": 430050, "postDate": "2018-11-29T18:30:30.140Z", "content": "Apart from obvious reasons, moderators generally flag questions as insincere, if the answer can be found using a simple google search. The following questions from your list will fall under this category:
\n\nWhat is the greatest amount of blood you can donate?\nWhere can I find best romantic shayaris?\nWhat is freight forwarder management system?
\n\nQuestions are also marked insincere if they do not have a background or context set to it. For Example:\nHow can I register as a freight forwarder in international services?\nWhere can I find android app developers?\nWhy do you have to select painting based on your furniture and fabrics?\nHow do I choose an apartment in Edmonton?\nHow do students get an academic update?
\n\nHope this helps!
", "rawMarkdown": "Apart from obvious reasons, moderators generally flag questions as insincere, if the answer can be found using a simple google search. The following questions from your list will fall under this category:\n\nWhat is the greatest amount of blood you can donate?\nWhere can I find best romantic shayaris?\nWhat is freight forwarder management system?\n\nQuestions are also marked insincere if they do not have a background or context set to it. For Example:\nHow can I register as a freight forwarder in international services?\nWhere can I find android app developers?\nWhy do you have to select painting based on your furniture and fabrics?\nHow do I choose an apartment in Edmonton?\nHow do students get an academic update?\n\nHope this helps!\n", "votes": 10, "replies": [ { "id": 430367, "postDate": "2018-11-30T09:18:56.343Z", "content": "Thank you, very good point!
", "rawMarkdown": "Thank you, very good point!", "votes": 1 }, { "id": 430400, "postDate": "2018-11-30T10:25:34.163Z", "content": "Good point, but I don't think it explains dataset.\nI checked first 20 questions form train dataset that marked as not insincre:
\n\nHow did Otto von Guericke used the Magdeburg hemispheres? //Shouldn't be toxic, according to \"easy \ngoogled case\" because first result directs you exactly to Wiki page\nWhat can you say about feminism? //Could be easy googled, or this is a provocation?
\n\nNow again let's look into our dataset, just first 20 questions:
\n\nIs Gaza slowly becoming Auschwitz, Dachau or Treblinka for Palestinians? //Seems to me toxic and provocative, but marked as not toxic\nWhy does Quora automatically ban conservative opinions when reported, but does not do the same for liberal views? //100% toxic, question make statement\nHave you licked the skin of a corpse?\nWhat is the dumbest, yet possibly true explanation for Trump being elected?
\n\nI understand that first 20 aren't representative, but ratio of IMHO mislabeled questions are pretty high, actually comparable with current top competitor's error rate.
", "rawMarkdown": "Good point, but I don't think it explains dataset.\nI checked first 20 questions form train dataset that marked as not insincre:\n\nHow did Otto von Guericke used the Magdeburg hemispheres? //Shouldn't be toxic, according to \"easy \ngoogled case\" because first result directs you exactly to Wiki page\nWhat can you say about feminism? //Could be easy googled, or this is a provocation?\n\nNow again let's look into our dataset, just first 20 questions:\n\nIs Gaza slowly becoming Auschwitz, Dachau or Treblinka for Palestinians? //Seems to me toxic and provocative, but marked as not toxic\nWhy does Quora automatically ban conservative opinions when reported, but does not do the same for liberal views? //100% toxic, question make statement\nHave you licked the skin of a corpse?\nWhat is the dumbest, yet possibly true explanation for Trump being elected?\n\nI understand that first 20 aren't representative, but ratio of IMHO mislabeled questions are pretty high, actually comparable with current top competitor's error rate." } ] }, { "id": 429873, "postDate": "2018-11-29T13:43:54.033Z", "content": "Yes. They also missed out on a few blatantly insincere questions.
", "rawMarkdown": "Yes. They also missed out on a few blatantly insincere questions.", "votes": 3 }, { "id": 429757, "postDate": "2018-11-29T09:57:10.533Z", "content": "I agree there seems to be a lot of mislabeled questions. It was said in the introduction that they were selected by a mixed ML/manual method so as we see their current approach leads to many misclassifications.\nBut as long as we assume that the train/test split was random we should teach our models even this unattended behaviours of their current ones.
", "rawMarkdown": "I agree there seems to be a lot of mislabeled questions. It was said in the introduction that they were selected by a mixed ML/manual method so as we see their current approach leads to many misclassifications.\nBut as long as we assume that the train/test split was random we should teach our models even this unattended behaviours of their current ones.", "replies": [ { "id": 429768, "postDate": "2018-11-29T10:12:43.737Z", "content": "Eventually model that actually detects insincerity will loose to model that trained to reproduce \"mixed ML/manual method\" behavior because our test set doesn't represent insincerity good, it represent just model output. So basically we teach our model not to detect insincerity but behave like this \"mixed ML/manual method\" model.
", "rawMarkdown": "Eventually model that actually detects insincerity will loose to model that trained to reproduce \"mixed ML/manual method\" behavior because our test set doesn't represent insincerity good, it represent just model output. So basically we teach our model not to detect insincerity but behave like this \"mixed ML/manual method\" model. ", "votes": 5 }, { "id": 434241, "postDate": "2018-12-06T04:56:31.890Z", "rawMarkdown": "", "isDeleted": true }, { "id": 435026, "postDate": "2018-12-07T10:41:06.437Z", "content": "I believe we have treat it as it is, because test set is from same(mislabeled) distribution as train set.
", "rawMarkdown": "I believe we have treat it as it is, because test set is from same(mislabeled) distribution as train set." } ] }, { "id": 457412, "postDate": "2019-01-17T11:12:32.827Z", "rawMarkdown": "", "isDeleted": true }, { "id": 435541, "postDate": "2018-12-08T07:28:48.120Z", "rawMarkdown": "", "isDeleted": true }, { "id": 434240, "postDate": "2018-12-06T04:54:48.720Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 430050, "author_name": "VenkateshRadhakrishnan", "author_url": "", "post_date": "2018-11-29T18:30:30.140000", "content": "Apart from obvious reasons, moderators generally flag questions as insincere, if the answer can be found using a simple google search. The following questions from your list will fall under this category:
\n\nWhat is the greatest amount of blood you can donate?\nWhere can I find best romantic shayaris?\nWhat is freight forwarder management system?
\n\nQuestions are also marked insincere if they do not have a background or context set to it. For Example:\nHow can I register as a freight forwarder in international services?\nWhere can I find android app developers?\nWhy do you have to select painting based on your furniture and fabrics?\nHow do I choose an apartment in Edmonton?\nHow do students get an academic update?
\n\nHope this helps!
", "votes": 10, "replies": [ { "id": 430367, "author_name": "Andrzej Kuro", "author_url": "", "post_date": "2018-11-30T09:18:56.343000", "content": "Thank you, very good point!
", "votes": 1, "replies": [] }, { "id": 430400, "author_name": "Taras Petrytsyn", "author_url": "", "post_date": "2018-11-30T10:25:34.163000", "content": "Good point, but I don't think it explains dataset.\nI checked first 20 questions form train dataset that marked as not insincre:
\n\nHow did Otto von Guericke used the Magdeburg hemispheres? //Shouldn't be toxic, according to \"easy \ngoogled case\" because first result directs you exactly to Wiki page\nWhat can you say about feminism? //Could be easy googled, or this is a provocation?
\n\nNow again let's look into our dataset, just first 20 questions:
\n\nIs Gaza slowly becoming Auschwitz, Dachau or Treblinka for Palestinians? //Seems to me toxic and provocative, but marked as not toxic\nWhy does Quora automatically ban conservative opinions when reported, but does not do the same for liberal views? //100% toxic, question make statement\nHave you licked the skin of a corpse?\nWhat is the dumbest, yet possibly true explanation for Trump being elected?
\n\nI understand that first 20 aren't representative, but ratio of IMHO mislabeled questions are pretty high, actually comparable with current top competitor's error rate.
", "votes": 0, "replies": [] } ] }, { "id": 429873, "author_name": "Matthew Anderson", "author_url": "", "post_date": "2018-11-29T13:43:54.033000", "content": "Yes. They also missed out on a few blatantly insincere questions.
", "votes": 3, "replies": [] }, { "id": 429757, "author_name": "Andrzej Kuro", "author_url": "", "post_date": "2018-11-29T09:57:10.533000", "content": "I agree there seems to be a lot of mislabeled questions. It was said in the introduction that they were selected by a mixed ML/manual method so as we see their current approach leads to many misclassifications.\nBut as long as we assume that the train/test split was random we should teach our models even this unattended behaviours of their current ones.
", "votes": 0, "replies": [ { "id": 429768, "author_name": "Taras Petrytsyn", "author_url": "", "post_date": "2018-11-29T10:12:43.737000", "content": "Eventually model that actually detects insincerity will loose to model that trained to reproduce \"mixed ML/manual method\" behavior because our test set doesn't represent insincerity good, it represent just model output. So basically we teach our model not to detect insincerity but behave like this \"mixed ML/manual method\" model.
", "votes": 5, "replies": [] }, { "id": 434241, "author_name": "", "author_url": "", "post_date": "2018-12-06T04:56:31.890000", "content": "", "votes": 0, "replies": [] }, { "id": 435026, "author_name": "Taras Petrytsyn", "author_url": "", "post_date": "2018-12-07T10:41:06.437000", "content": "I believe we have treat it as it is, because test set is from same(mislabeled) distribution as train set.
", "votes": 0, "replies": [] } ] }, { "id": 457412, "author_name": "", "author_url": "", "post_date": "2019-01-17T11:12:32.827000", "content": "", "votes": 0, "replies": [] }, { "id": 435541, "author_name": "", "author_url": "", "post_date": "2018-12-08T07:28:48.120000", "content": "", "votes": 0, "replies": [] }, { "id": 434240, "author_name": "", "author_url": "", "post_date": "2018-12-06T04:54:48.720000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "429411": "Do you know how training set was labeled? Among my misclassified questions I see many questions that were labeled as insincere but look completely ok for me. For example I just checked top 100 msclassified questions: \n\nWhat is freight forwarder management system?\nHow can I register as a freight forwarder in international services?\nWhere can I find android app developers?\nWhy do you have to select painting based on your furniture and fabrics?\nHow do I choose an apartment in Edmonton?\nHow do students get an academic update?\nWhich is the best best TRUCK TIRE CHAINS?\nWhat’s the coolest place you have been to in Chicago?\nWhat are the names of woman athletes participating in the 2018 Olympics?\nWhich low-investment businesses are most profitable in the world?\nHow does Luminary Cream function?\nWhat did you think of the Berserk movies?\nWho were the Elves? Tolkien is supposed to be inspired by folklore but some real history may be hiding behind the myth!\nWhat star were you born under?\nWhat is the greatest amount of blood you can donate?\nWhere can I find best romantic shayaris?\n\nAnd I didn't mentioned here many more questions that I would say are sincere, but have some doubts.\n\nSuch questions can make model teach something wrong. But relabeling it could cause model train something else, not what Quora expects. So again, it would be interesting to know what is the policy regarding insincerity, how labeling is performed.", "430050": "Apart from obvious reasons, moderators generally flag questions as insincere, if the answer can be found using a simple google search. The following questions from your list will fall under this category:\n\nWhat is the greatest amount of blood you can donate?\nWhere can I find best romantic shayaris?\nWhat is freight forwarder management system?\n\nQuestions are also marked insincere if they do not have a background or context set to it. For Example:\nHow can I register as a freight forwarder in international services?\nWhere can I find android app developers?\nWhy do you have to select painting based on your furniture and fabrics?\nHow do I choose an apartment in Edmonton?\nHow do students get an academic update?\n\nHope this helps!\n", "429873": "Yes. They also missed out on a few blatantly insincere questions.", "429757": "I agree there seems to be a lot of mislabeled questions. It was said in the introduction that they were selected by a mixed ML/manual method so as we see their current approach leads to many misclassifications.\nBut as long as we assume that the train/test split was random we should teach our models even this unattended behaviours of their current ones.", "457412": "", "435541": "", "434240": "" } }