{
  "id": 147417,
  "title": "Ideas for multilingual sentence spell checking?",
  "url": "/competitions/jigsaw-multilingual-toxic-comment-classification/discussion/147417",
  "author_name": "",
  "post_date": "2020-04-30T15:57:05.651751400Z",
  "votes": 8,
  "comment_count": 9,
  "views": 0,
  "content": "<p>Any python implemntations for a reliable multilingual spell correcter for sentences?</p>",
  "messages": [
    {
      "id": "827889",
      "postDate": "04/30/2020 15:57:05",
      "content": "<p>Any python implemntations for a reliable multilingual spell correcter for sentences?</p>",
      "rawMarkdown": "Any python implemntations for a reliable multilingual spell correcter for sentences?",
      "votes": null
    },
    {
      "id": "827947",
      "postDate": "04/30/2020 16:51:23",
      "content": "<p>following : because this is really very very important for this competition</p>",
      "rawMarkdown": "following : because this is really very very important for this competition",
      "votes": null
    },
    {
      "id": "828069",
      "postDate": "04/30/2020 18:45:04",
      "content": "<p>One thing you can try is using a language detector. If the score is too low (below 0.8 for instance), then you remove the sentence from the dataset.\nSo that you are sure to have a neat dataset.</p>\n\n<p>Credits: <a href=\"https://www.kaggle.com/shonenkov/hack-with-parallel-corpus\">https://www.kaggle.com/shonenkov/hack-with-parallel-corpus</a></p>",
      "rawMarkdown": "One thing you can try is using a language detector. If the score is too low (below 0.8 for instance), then you remove the sentence from the dataset.\nSo that you are sure to have a neat dataset.\n\nCredits: https://www.kaggle.com/shonenkov/hack-with-parallel-corpus",
      "votes": null
    },
    {
      "id": "828078",
      "postDate": "04/30/2020 18:49:47",
      "content": "<p>I haven't used it before, but pyspellchecker (<a href=\"https://pypi.org/project/pyspellchecker/\">https://pypi.org/project/pyspellchecker/</a>) could be worth a try! It supports multiple languages. The package uses corpora from <a href=\"http://opus.nlpl.eu/OpenSubtitles2018.php\">http://opus.nlpl.eu/OpenSubtitles2018.php</a>. </p>",
      "rawMarkdown": "I haven't used it before, but pyspellchecker (https://pypi.org/project/pyspellchecker/) could be worth a try! It supports multiple languages. The package uses corpora from http://opus.nlpl.eu/OpenSubtitles2018.php.",
      "votes": null
    },
    {
      "id": "828199",
      "postDate": "04/30/2020 20:31:27",
      "content": "<p>Hello guys, I just wrote a short kernel where I experiment some cleaning and spell-checking techniques. \nIf you want to have a look: <a href=\"https://www.kaggle.com/rftexas/cleaning-and-removing-mis-spells-from-texts\">https://www.kaggle.com/rftexas/cleaning-and-removing-mis-spells-from-texts</a></p>",
      "rawMarkdown": "Hello guys, I just wrote a short kernel where I experiment some cleaning and spell-checking techniques. \nIf you want to have a look: https://www.kaggle.com/rftexas/cleaning-and-removing-mis-spells-from-texts",
      "votes": null
    },
    {
      "id": "828325",
      "postDate": "04/30/2020 22:45:20",
      "content": "<p>i think that works</p>",
      "rawMarkdown": "i think that works",
      "votes": null
    },
    {
      "id": "829139",
      "postDate": "05/01/2020 14:37:03",
      "content": "<p>why do you think this thing is important?</p>",
      "rawMarkdown": "why do you think this thing is important?",
      "votes": null
    },
    {
      "id": "829457",
      "postDate": "05/01/2020 19:37:41",
      "content": "<p>incorrectly spelled words are handled in a special manner by BERT models, please read here\n<a href=\"https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/discussion/92737\">https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/discussion/92737</a></p>\n\n<p>Also, translations are ignored for misspelled words.</p>",
      "rawMarkdown": "incorrectly spelled words are handled in a special manner by BERT models, please read here\nhttps://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/discussion/92737\n\nAlso, translations are ignored for misspelled words.",
      "votes": null
    },
    {
      "id": "829574",
      "postDate": "05/01/2020 23:19:54",
      "content": "<p>Hey mate, I have checked it and even tried both pycld2 and pycld3, it does not always return reliable results so for a sentence of 20 words, it does return high confidence (0.96+) even if two or 3 words are totally off</p>",
      "rawMarkdown": "Hey mate, I have checked it and even tried both pycld2 and pycld3, it does not always return reliable results so for a sentence of 20 words, it does return high confidence (0.96+) even if two or 3 words are totally off",
      "votes": null
    },
    {
      "id": "855522",
      "postDate": "05/21/2020 02:27:52",
      "content": "<p>Hi, I am also trying pyspellchecker, but did not find a way to switch language like do a checker for only Russian, did I miss anything? It would be great if you can share how to do that. Thanks!</p>",
      "rawMarkdown": "Hi, I am also trying pyspellchecker, but did not find a way to switch language like do a checker for only Russian, did I miss anything? It would be great if you can share how to do that. Thanks!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 827947,
      "author_name": "mobassir",
      "author_url": "",
      "post_date": "04/30/2020 16:51:23",
      "content": "<p>following : because this is really very very important for this competition</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 828069,
      "author_name": "rftexas",
      "author_url": "",
      "post_date": "04/30/2020 18:45:04",
      "content": "<p>One thing you can try is using a language detector. If the score is too low (below 0.8 for instance), then you remove the sentence from the dataset.\nSo that you are sure to have a neat dataset.</p>\n\n<p>Credits: <a href=\"https://www.kaggle.com/shonenkov/hack-with-parallel-corpus\">https://www.kaggle.com/shonenkov/hack-with-parallel-corpus</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 828078,
      "author_name": "davelo12",
      "author_url": "",
      "post_date": "04/30/2020 18:49:47",
      "content": "<p>I haven't used it before, but pyspellchecker (<a href=\"https://pypi.org/project/pyspellchecker/\">https://pypi.org/project/pyspellchecker/</a>) could be worth a try! It supports multiple languages. The package uses corpora from <a href=\"http://opus.nlpl.eu/OpenSubtitles2018.php\">http://opus.nlpl.eu/OpenSubtitles2018.php</a>. </p>",
      "votes": null,
      "replies": [
        {
          "id": 855522,
          "author_name": "",
          "author_url": "",
          "post_date": "05/21/2020 02:27:52",
          "content": "<p>Hi, I am also trying pyspellchecker, but did not find a way to switch language like do a checker for only Russian, did I miss anything? It would be great if you can share how to do that. Thanks!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 828199,
      "author_name": "rftexas",
      "author_url": "",
      "post_date": "04/30/2020 20:31:27",
      "content": "<p>Hello guys, I just wrote a short kernel where I experiment some cleaning and spell-checking techniques. \nIf you want to have a look: <a href=\"https://www.kaggle.com/rftexas/cleaning-and-removing-mis-spells-from-texts\">https://www.kaggle.com/rftexas/cleaning-and-removing-mis-spells-from-texts</a></p>",
      "votes": null,
      "replies": [
        {
          "id": 829574,
          "author_name": "ma7555",
          "author_url": "",
          "post_date": "05/01/2020 23:19:54",
          "content": "<p>Hey mate, I have checked it and even tried both pycld2 and pycld3, it does not always return reliable results so for a sentence of 20 words, it does return high confidence (0.96+) even if two or 3 words are totally off</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 828325,
      "author_name": "winneremeto",
      "author_url": "",
      "post_date": "04/30/2020 22:45:20",
      "content": "<p>i think that works</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 829139,
      "author_name": "mohamedlotfyelrefai",
      "author_url": "",
      "post_date": "05/01/2020 14:37:03",
      "content": "<p>why do you think this thing is important?</p>",
      "votes": null,
      "replies": [
        {
          "id": 829457,
          "author_name": "ma7555",
          "author_url": "",
          "post_date": "05/01/2020 19:37:41",
          "content": "<p>incorrectly spelled words are handled in a special manner by BERT models, please read here\n<a href=\"https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/discussion/92737\">https://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/discussion/92737</a></p>\n\n<p>Also, translations are ignored for misspelled words.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "827889": "Any python implemntations for a reliable multilingual spell correcter for sentences?",
    "827947": "following : because this is really very very important for this competition",
    "828069": "One thing you can try is using a language detector. If the score is too low (below 0.8 for instance), then you remove the sentence from the dataset.\nSo that you are sure to have a neat dataset.\n\nCredits: https://www.kaggle.com/shonenkov/hack-with-parallel-corpus",
    "828078": "I haven't used it before, but pyspellchecker (https://pypi.org/project/pyspellchecker/) could be worth a try! It supports multiple languages. The package uses corpora from http://opus.nlpl.eu/OpenSubtitles2018.php.",
    "828199": "Hello guys, I just wrote a short kernel where I experiment some cleaning and spell-checking techniques. \nIf you want to have a look: https://www.kaggle.com/rftexas/cleaning-and-removing-mis-spells-from-texts",
    "828325": "i think that works",
    "829139": "why do you think this thing is important?",
    "829457": "incorrectly spelled words are handled in a special manner by BERT models, please read here\nhttps://www.kaggle.com/c/jigsaw-unintended-bias-in-toxicity-classification/discussion/92737\n\nAlso, translations are ignored for misspelled words.",
    "829574": "Hey mate, I have checked it and even tried both pycld2 and pycld3, it does not always return reliable results so for a sentence of 20 words, it does return high confidence (0.96+) even if two or 3 words are totally off",
    "855522": "Hi, I am also trying pyspellchecker, but did not find a way to switch language like do a checker for only Russian, did I miss anything? It would be great if you can share how to do that. Thanks!"
  },
  "source": "meta"
}