![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F292868f6b063331aee01c773f277dffd%2FSelection_999(2853).png?generation=1691559896222461&alt=media\")
They are two popular normalizers available for Bangla text sentences ,
\n\n \n\n
It seems like bnunicodenomlaizer gives better LB result than unnormalized prediction and the other normalizer.
\nCan the organizers kindly confirm whether the ground truths for the hidden test set are normalized? If yes, then by which normalizer?
They are two popular normalizers available for Bangla text sentences ,
\n\n \n\n
It seems like bnunicodenomlaizer gives better LB result than unnormalized prediction and the other normalizer.
\nCan the organizers kindly confirm whether the ground truths for the hidden test set are normalized? If yes, then by which normalizer?
Hi Murad, we used bnunicodenormalizer.
", "rawMarkdown": "Hi Murad, we used bnunicodenormalizer.", "votes": 5, "replies": [ { "id": 2374492, "postDate": "2023-08-05T03:20:27.643Z", "content": "Thanks for the reply.
\nThis is very important infomation, it affects how the model should be trained and train data/prediction should be pre/post-processed.
This post should be pinned in the discussion or the evaluation page sould be upated.
", "rawMarkdown": "Thanks for the reply.\nThis is very important infomation, it affects how the model should be trained and train data/prediction should be pre/post-processed.\n\nThis post should be pinned in the discussion or the evaluation page sould be upated.", "votes": 3 }, { "id": 2374496, "postDate": "2023-08-05T03:27:13.967Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2374497, "postDate": "2023-08-05T03:28:02.243Z", "content": "@reasat bhai, thanks for confirming.
", "rawMarkdown": "@reasat bhai, thanks for confirming. " }, { "id": 2374651, "postDate": "2023-08-05T06:18:35.897Z", "content": "So what about labels in the train set, are they normalized?
", "rawMarkdown": "So what about labels in the train set, are they normalized?", "replies": [ { "id": 2374656, "postDate": "2023-08-05T06:21:21.723Z", "content": "No, neither of the train/validation split are normalized.
", "rawMarkdown": "No, neither of the train/validation split are normalized.", "votes": 3 } ] }, { "id": 2375383, "postDate": "2023-08-05T15:31:16.677Z", "content": "Hi, thanks for this information!
\nAnother question is that are the punctuation marks such as '।' (Bengali full stop DARI) kept in the ground-truth test-set sentences while calculating WER? Because I notice that adding full stop after each sentence will cause a lot difference in terms of WER.
Yeah, that's a critical issue. The punctuation should be kept in the GT sentences and then WER should be calculated. You can check whether they've done it by submitting a prediction with punctuation and another without punctuation. If the one with punctuations give better LB result, that means they've kept the punctuation in the GT's (which I think is the actual case here)
", "rawMarkdown": "Yeah, that's a critical issue. The punctuation should be kept in the GT sentences and then WER should be calculated. You can check whether they've done it by submitting a prediction with punctuation and another without punctuation. If the one with punctuations give better LB result, that means they've kept the punctuation in the GT's (which I think is the actual case here)" }, { "id": 2376600, "postDate": "2023-08-06T13:49:37.223Z", "content": "there are two choice:
\n1) the organizer can change the ground truth to only normalised text and remove any punctuation. This will be a pure asr problem (instead of transcription). here we just solve the task of convert wave to word.
2) you have to to predict punctuation, either by:
\na. post processing : wave --> character --> word + punctuation
\nb. treat punctuation as a character : wave --> character + punctuation --> word + punctuation
which methods work better depends on the data …. wether you have sufficient data and if the source/target data have domain shift, etc
", "rawMarkdown": "there are two choice:\n1) the organizer can change the ground truth to only normalised text and remove any punctuation. This will be a pure asr problem (instead of transcription). here we just solve the task of convert wave to word.\n\n2) you have to to predict punctuation, either by:\na. post processing : wave --> character --> word + punctuation\nb. treat punctuation as a character : wave --> character + punctuation --> word + punctuation\n\nwhich methods work better depends on the data .... wether you have sufficient data and if the source/target data have domain shift, etc", "votes": 1 }, { "id": 2377504, "postDate": "2023-08-07T07:16:16.150Z", "content": "Hi @revospeech @mbmmurad @hengck23!
\nPunctuations were left in because it is not completely disconnected from the ASR task. E.g., intonation changes based on the presence/absence of full stop. Since the sentences are human annotated and validated, there shouldn't be label noise wrt punctuations. Best of luck!
", "rawMarkdown": "Hi @revospeech @mbmmurad @hengck23!\n\nPunctuations were left in because it is not completely disconnected from the ASR task. E.g., intonation changes based on the presence/absence of full stop. Since the sentences are human annotated and validated, there shouldn't be label noise wrt punctuations. Best of luck!" }, { "id": 2377522, "postDate": "2023-08-07T07:28:07.010Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2377523, "postDate": "2023-08-07T07:29:48.457Z", "content": "@imtiazprio bhai, thanks for the clarification!
\nWe expected that punctuations would be left in. This now makes the competition much more interesting and challenging too.
\nAlso kudos for arranging such a nice competition on Bangla ASR!
thanks! also you guys are doing great congrats!
", "rawMarkdown": "thanks! also you guys are doing great congrats!", "votes": 1 } ] } ] }, { "id": 2381288, "postDate": "2023-08-09T05:45:24.600Z", "content": "some information at discord discussion
\n
there is a paper on the normaliser used in evalution:
\nhttps://arxiv.org/pdf/2306.01743.pdf
", "rawMarkdown": "some information at discord discussion\n\n.png?generation=1691559896222461&alt=media)\n\nthere is a paper on the normaliser used in evalution:\n\nhttps://arxiv.org/pdf/2306.01743.pdf", "votes": 2 }, { "id": 2379220, "postDate": "2023-08-08T04:46:28.763Z", "content": "Sorry I'm new to this, but do these normalizers delete punctuation?
", "rawMarkdown": "Sorry I'm new to this, but do these normalizers delete punctuation?", "replies": [ { "id": 2379240, "postDate": "2023-08-08T04:58:37.337Z", "content": "Nope they don't delete punctuations, but may remove extra spaces around punctuations if inappropriate!
", "rawMarkdown": "Nope they don't delete punctuations, but may remove extra spaces around punctuations if inappropriate!", "votes": 2, "replies": [ { "id": 2379273, "postDate": "2023-08-08T05:09:13.803Z", "content": "Thank you so much!
", "rawMarkdown": "Thank you so much!" } ] } ] } ], "comments": [ { "id": 2374380, "author_name": "Tahsin", "author_url": "", "post_date": "2023-08-04T23:02:01.830000", "content": "Hi Murad, we used bnunicodenormalizer.
", "votes": 5, "replies": [ { "id": 2374492, "author_name": "hengck23", "author_url": "", "post_date": "2023-08-05T03:20:27.643000", "content": "Thanks for the reply.
\nThis is very important infomation, it affects how the model should be trained and train data/prediction should be pre/post-processed.
This post should be pinned in the discussion or the evaluation page sould be upated.
", "votes": 3, "replies": [] }, { "id": 2374496, "author_name": "", "author_url": "", "post_date": "2023-08-05T03:27:13.967000", "content": "", "votes": 0, "replies": [] }, { "id": 2374497, "author_name": "Md Boktiar Mahbub Murad", "author_url": "", "post_date": "2023-08-05T03:28:02.243000", "content": "@reasat bhai, thanks for confirming.
", "votes": 0, "replies": [] }, { "id": 2374651, "author_name": "Aphysict", "author_url": "", "post_date": "2023-08-05T06:18:35.897000", "content": "So what about labels in the train set, are they normalized?
", "votes": 0, "replies": [ { "id": 2374656, "author_name": "Md Boktiar Mahbub Murad", "author_url": "", "post_date": "2023-08-05T06:21:21.723000", "content": "No, neither of the train/validation split are normalized.
", "votes": 3, "replies": [] } ] }, { "id": 2375383, "author_name": "Modifier", "author_url": "", "post_date": "2023-08-05T15:31:16.677000", "content": "Hi, thanks for this information!
\nAnother question is that are the punctuation marks such as '।' (Bengali full stop DARI) kept in the ground-truth test-set sentences while calculating WER? Because I notice that adding full stop after each sentence will cause a lot difference in terms of WER.
Yeah, that's a critical issue. The punctuation should be kept in the GT sentences and then WER should be calculated. You can check whether they've done it by submitting a prediction with punctuation and another without punctuation. If the one with punctuations give better LB result, that means they've kept the punctuation in the GT's (which I think is the actual case here)
", "votes": 0, "replies": [] }, { "id": 2376600, "author_name": "hengck23", "author_url": "", "post_date": "2023-08-06T13:49:37.223000", "content": "there are two choice:
\n1) the organizer can change the ground truth to only normalised text and remove any punctuation. This will be a pure asr problem (instead of transcription). here we just solve the task of convert wave to word.
2) you have to to predict punctuation, either by:
\na. post processing : wave --> character --> word + punctuation
\nb. treat punctuation as a character : wave --> character + punctuation --> word + punctuation
which methods work better depends on the data …. wether you have sufficient data and if the source/target data have domain shift, etc
", "votes": 1, "replies": [] }, { "id": 2377504, "author_name": "Ahmed Imtiaz Humayun", "author_url": "", "post_date": "2023-08-07T07:16:16.150000", "content": "Hi @revospeech @mbmmurad @hengck23!
\nPunctuations were left in because it is not completely disconnected from the ASR task. E.g., intonation changes based on the presence/absence of full stop. Since the sentences are human annotated and validated, there shouldn't be label noise wrt punctuations. Best of luck!
", "votes": 0, "replies": [] }, { "id": 2377522, "author_name": "", "author_url": "", "post_date": "2023-08-07T07:28:07.010000", "content": "", "votes": 0, "replies": [] }, { "id": 2377523, "author_name": "Md Boktiar Mahbub Murad", "author_url": "", "post_date": "2023-08-07T07:29:48.457000", "content": "@imtiazprio bhai, thanks for the clarification!
\nWe expected that punctuations would be left in. This now makes the competition much more interesting and challenging too.
\nAlso kudos for arranging such a nice competition on Bangla ASR!
thanks! also you guys are doing great congrats!
", "votes": 1, "replies": [] } ] } ] }, { "id": 2381288, "author_name": "hengck23", "author_url": "", "post_date": "2023-08-09T05:45:24.600000", "content": "some information at discord discussion
\n
there is a paper on the normaliser used in evalution:
\nhttps://arxiv.org/pdf/2306.01743.pdf
", "votes": 2, "replies": [] }, { "id": 2379220, "author_name": "Larry Gan", "author_url": "", "post_date": "2023-08-08T04:46:28.763000", "content": "Sorry I'm new to this, but do these normalizers delete punctuation?
", "votes": 0, "replies": [ { "id": 2379240, "author_name": "Ahmed Imtiaz Humayun", "author_url": "", "post_date": "2023-08-08T04:58:37.337000", "content": "Nope they don't delete punctuations, but may remove extra spaces around punctuations if inappropriate!
", "votes": 2, "replies": [ { "id": 2379273, "author_name": "Larry Gan", "author_url": "", "post_date": "2023-08-08T05:09:13.803000", "content": "Thank you so much!
", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2374068": "They are two popular normalizers available for Bangla text sentences , \n>[bnunicodenormalizer](https://pypi.org/project/bnunicodenormalizer)\n>[csebuetnlp/normalizer](https://github.com/csebuetnlp/normalizer). \n\nIt seems like **bnunicodenomlaizer gives better LB result** than unnormalized prediction and the other normalizer.\nCan the organizers kindly confirm **whether the ground truths for the hidden test set are normalized?** If yes, then **by which normalizer?**", "2374380": "Hi Murad, we used bnunicodenormalizer.", "2381288": "some information at discord discussion\n\n.png?generation=1691559896222461&alt=media)\n\nthere is a paper on the normaliser used in evalution:\n\nhttps://arxiv.org/pdf/2306.01743.pdf", "2379220": "Sorry I'm new to this, but do these normalizers delete punctuation?" } }