I found that some of sentences in the train data whose split is 'valid' don't end with punctuations.
\nThe example I found is as following.
id:03ef414022ac
\nsentence:তিনি নলডাঙ্গা উপজেলার ব্যাপক উন্নতি করেছে
According to Google Translation, the sentence means \"He has greatly improved Naldanga upazila\".
\nI don't know Bengali well, but I think that this sentence should end with dari(।).
\nAre there any annotation in test data which don't end with dari(।) though dari(।) is needed grammatically?
\nI think test data should not contain such an annotation.
I found that some of sentences in the train data whose split is 'valid' don't end with punctuations.
\nThe example I found is as following.
id:03ef414022ac
\nsentence:তিনি নলডাঙ্গা উপজেলার ব্যাপক উন্নতি করেছে
According to Google Translation, the sentence means \"He has greatly improved Naldanga upazila\".
\nI don't know Bengali well, but I think that this sentence should end with dari(।).
\nAre there any annotation in test data which don't end with dari(।) though dari(।) is needed grammatically?
\nI think test data should not contain such an annotation.
Hi moto, great question!
\nFirst thing first, the test data is normalized using bnunicodenormalizer, so unicode specific errors and conflicts in punctuation types are more or less resolved by that. We didn't normalize the training data to provide training data in its rawest form.
\nSecondly, if there are punctuations in test data, it is very likely because a human annotator could hear its context in the audio. The test data was quite rigorously validated so we can expect label noise (including noise related to punctuation labels) should be very low.
", "rawMarkdown": "Hi moto, great question!\n\nFirst thing first, the test data is normalized using bnunicodenormalizer, so unicode specific errors and conflicts in punctuation types are more or less resolved by that. We didn't normalize the training data to provide training data in its rawest form.\n\nSecondly, if there are punctuations in test data, it is very likely because a human annotator could hear its context in the audio. The test data was quite rigorously validated so we can expect label noise (including noise related to punctuation labels) should be very low.", "votes": null }, { "id": "2410125", "postDate": "08/26/2023 16:41:37", "content": "Hi @yamashitamotokazu , great find! I have ran an analysis and here are my findings.
\n\n", "rawMarkdown": "Hi @yamashitamotokazu , great find! I have ran an analysis and here are my findings. \n> 2454 sentences in the train set and 214 sentences in the valid split don't contain any punctuations ( '।': '?', '!' ) in the end. \nMy hypothesis is either these are mistakes by the annotators or some of these are not complete sentences.", "votes": null } ], "comments": [ { "id": 2409566, "author_name": "imtiazprio", "author_url": "", "post_date": "08/26/2023 10:09:35", "content": "2454 sentences in the train set and 214 sentences in the valid split don't contain any punctuations ( '।': '?', '!' ) in the end.
\n
\n My hypothesis is either these are mistakes by the annotators or some of these are not complete sentences.
Hi moto, great question!
\nFirst thing first, the test data is normalized using bnunicodenormalizer, so unicode specific errors and conflicts in punctuation types are more or less resolved by that. We didn't normalize the training data to provide training data in its rawest form.
\nSecondly, if there are punctuations in test data, it is very likely because a human annotator could hear its context in the audio. The test data was quite rigorously validated so we can expect label noise (including noise related to punctuation labels) should be very low.
", "votes": null, "replies": [] }, { "id": 2410125, "author_name": "mbmmurad", "author_url": "", "post_date": "08/26/2023 16:41:37", "content": "Hi @yamashitamotokazu , great find! I have ran an analysis and here are my findings.
\n\n", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2409409": "I found that some of sentences in the train data whose split is 'valid' don't end with punctuations.\nThe example I found is as following.\n\nid:03ef414022ac\nsentence:তিনি নলডাঙ্গা উপজেলার ব্যাপক উন্নতি করেছে\n\nAccording to Google Translation, the sentence means \"He has greatly improved Naldanga upazila\".\n\nI don't know Bengali well, but I think that this sentence should end with dari(।).\nAre there any annotation in test data which don't end with dari(।) though dari(।) is needed grammatically?\nI think test data should not contain such an annotation.", "2409566": "Hi moto, great question!\n\nFirst thing first, the test data is normalized using bnunicodenormalizer, so unicode specific errors and conflicts in punctuation types are more or less resolved by that. We didn't normalize the training data to provide training data in its rawest form.\n\nSecondly, if there are punctuations in test data, it is very likely because a human annotator could hear its context in the audio. The test data was quite rigorously validated so we can expect label noise (including noise related to punctuation labels) should be very low.", "2410125": "Hi @yamashitamotokazu , great find! I have ran an analysis and here are my findings. \n> 2454 sentences in the train set and 214 sentences in the valid split don't contain any punctuations ( '।': '?', '!' ) in the end. \nMy hypothesis is either these are mistakes by the annotators or some of these are not complete sentences." }, "source": "meta" }2454 sentences in the train set and 214 sentences in the valid split don't contain any punctuations ( '।': '?', '!' ) in the end.
\n
\n My hypothesis is either these are mistakes by the annotators or some of these are not complete sentences.