This is the notebook
\nhttps://www.kaggle.com/hengck23/help-why-submission-score-error
\nusing public huggingface/ai4bharat/indicwav2vec_v1_bengali model
\n(this model has good local CV of 0.39)
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F19ab286f9fc5c62bfac6fb18cdce5ffd%2FSelection_999(2762).png?generation=1689916005019800&alt=media\")
\n\" submission score error\" occurs when the evaluation script at the server uses your submission csv cannot complete due to error.
\ni find it strange:
assert (sample_submission_df==submit_df)\n to string\nsubmit_df = submit_df(lambda x: x (x)==str ) \nsubmission score error :
\n\"Your notebook generated a submission file with incorrect format. Some examples causing this are: wrong number of rows or columns, empty values, an incorrect data type for a value, or invalid submission values from what is expected. See more debugging tips\"
This is the notebook
\nhttps://www.kaggle.com/hengck23/help-why-submission-score-error
\nusing public huggingface/ai4bharat/indicwav2vec_v1_bengali model
\n(this model has good local CV of 0.39)
\n
\n\" submission score error\" occurs when the evaluation script at the server uses your submission csv cannot complete due to error.
\ni find it strange:
assert (sample_submission_df==submit_df)\n to string\nsubmit_df = submit_df(lambda x: x (x)==str ) \nsubmission score error :
\n\"Your notebook generated a submission file with incorrect format. Some examples causing this are: wrong number of rows or columns, empty values, an incorrect data type for a value, or invalid submission values from what is expected. See more debugging tips\"
Have same issue(
", "rawMarkdown": "Have same issue(", "votes": null }, { "id": "2352900", "postDate": "07/21/2023 10:33:06", "content": "", "rawMarkdown": "~~ ```\nif mode=='submit':\n mp3_dir = f'/kaggle/input/bengaliai-speech/test_mp3s'\n valid_df = pd.read_csv('/kaggle/input/bengaliai-speech/sample_submission.csv') \n```\n\n\nYou are loading sample_submission.csv and then infering on the ids of this dataframe. It contains only 3 values. But the actual test set has around 8k.\n\nWhat you need to do is iterate over the files of the 'test_mp3s' directory, store the filenames and prediction, and then create a new dataframe using these. \nPlease refer to the inference part of [this notebook ](https://www.kaggle.com/code/mbmmurad/lb-0-641-nemo-conformer-baseline-w-o-internet-in)~~", "votes": null }, { "id": "2352945", "postDate": "07/21/2023 10:59:27", "content": "Pseudo-code of a correct way of submission :
\nfilenames = \npredictions = \nfiles = os()\nbase_path = \n\n file files:\n filenames(file())\n = (base_path+file) function to load the file\n text = (audio) with your model\n (text)==:\n text = \n\n predictions(text)\n\ndf = pd({:filenames,:predictions})\ndf(,index=False)\nif you use a normalizer than make sure to check the texts after normalizing.
\ndef check():\n ()==:\n = \n \ndf. = df..apply(lambda x:normalizer(x)))\ndf. = df..apply(lambda x:check(x)))\ni think both test folder and sample_submission.csv are replaced during submission.
\nMy code work for other model, so i think it is not the reason.
besides, if i were only processing three files, the submission time should be very short
", "rawMarkdown": "i think both test folder and sample\\_submission.csv are replaced during submission.\nMy code work for other model, so i think it is not the reason.\n\nbesides, if i were only processing three files, the submission time should be very short", "votes": null }, { "id": "2353253", "postDate": "07/21/2023 14:34:54", "content": "the latest version still course error:
\n mode==:\n mp3_dir = \n glob_file = glob()\n = ([f[(mp3_dir)+:-] f glob_file])\n valid_df = pd.DataFrame({:,:})\ndef check_text():\n t = \n \n :\n jiwer.wer(t, [])\n except:\n = \n \n\n\nsubmit_df.loc[:,] = submit_df..apply(check_text)\nI successfully submitted using your notebook. The main problem is :
\n\n\nThe model is generating some empty predictions. It seems like you cant't submit empty strings in the dataframe. I replaced the empty strings with \",\" and it got submitted.
\n
filenames = \npredictions = \nfiles = os()\nbase_path = \n\n file files:\n filenames(file())\n = (base_path+file) function to load the file\n text = (audio) with your model\n (text)==:\n text = \n\n predictions(text)\n\ndf = pd({:filenames,:predictions})\ndf(,index=False)\nRefer to this notebook
", "rawMarkdown": "# Solved LB 0.52\n\nI successfully submitted using your notebook. The main problem is :\n> The model is generating some empty predictions. It seems like you cant't submit empty strings in the dataframe. I replaced the empty strings with \",\" and it got submitted. \n\n``` \nfilenames = []\npredictions = []\nfiles = os.listdir('/kaggle/input/bengaliai-speech/test_mp3s')\nbase_path = \"/kaggle/input/bengaliai-speech/test_mp3s/\"\n\nfor file in files:\n filenames.append(file.split(\".\")[0])\n audio = load_audio(base_path+file) #Some function to load the audio file\n text = model(audio) #Infer with your model\n if len(text)==0:\n text = \",\"\n \n predictions.append(text)\n\ndf = pd.DataFrame({'id':filenames,'sentence':predictions})\ndf.to_csv('submission.csv',index=False)\n```\n\nRefer to [this notebook](https://www.kaggle.com/mbmmurad/fork-of-fork-of-local-nemo-baseline-conformer)", "votes": null }, { "id": "2353306", "postDate": "07/21/2023 15:21:12", "content": "You're right. My mistake. Your pipeline worked. Just needed to handle the empty strings.
", "rawMarkdown": "You're right. My mistake. Your pipeline worked. Just needed to handle the empty strings.", "votes": null }, { "id": "2353310", "postDate": "07/21/2023 15:22:15", "content": "Just change to sentence = \",\" .
Thanks a lot!!!
", "rawMarkdown": "mbmmurad \n\nThanks a lot!!!", "votes": null }, { "id": "2353352", "postDate": "07/21/2023 15:45:00", "content": "by the way, this is wave2vec2-CTC without LM
", "rawMarkdown": "by the way, this is wave2vec2-CTC without LM", "votes": null }, { "id": "2353382", "postDate": "07/21/2023 16:02:55", "content": "more information about the model can be found here:
\nhttps://github.com/AI4Bharat/IndicWav2Vec/tree/main
i think the model is trained with author's own dataset.
\nHence we can conclude? OOD cause adroup of about 0.12
\n(since kaggle valid split has WER = 0.40, public LB 0.52)
IndicWav2Vec paper:
\nhttps://arxiv.org/pdf/2111.03945.pdf
\"Apart from YouTube, we also curated content
\nfrom newsonair3 which is a radio news channel
\nrun by the Govt. of India and broadcasts news in
\nmultiple Indian languages\"
there is CTC-LM model which gives better results but you need to convert from fairseq to hf yourself,
\nfollowing instructions from githut repo
\"3.5 Rescoring
\nNote that in the above decoding process, we use an
\nn-gram KenLM language model (Heafield, 2011).
\nHowever, recent advances in language modeling
\nhave shown that transformer based language models perform well. To get the best of both worlds,
\nwe optionally use an external transformer based
\nlanguage model to rescore the n-best hypothesis
\nproduced above (Synnaeve et al., 2019). Specifically, for each output in the n-best list, we compute a new score by combining the decoder logprobability, \"
the rescoring transformer can be used for out ensemble of different models
", "rawMarkdown": "more information about the model can be found here:\nhttps://github.com/AI4Bharat/IndicWav2Vec/tree/main\n\ni think the model is trained with author's own dataset.\nHence we can conclude? OOD cause adroup of about 0.12\n(since kaggle valid split has WER = 0.40, public LB 0.52)\n\n\n---\nIndicWav2Vec paper:\nhttps://arxiv.org/pdf/2111.03945.pdf\n\n\"Apart from YouTube, we also curated content\nfrom newsonair3 which is a radio news channel\nrun by the Govt. of India and broadcasts news in\nmultiple Indian languages\"\n\n---\nthere is CTC-LM model which gives better results but you need to convert from fairseq to hf yourself,\nfollowing instructions from githut repo\n\n\n---\n\n\"3.5 Rescoring\nNote that in the above decoding process, we use an\nn-gram KenLM language model (Heafield, 2011).\nHowever, recent advances in language modeling\nhave shown that transformer based language models perform well. To get the best of both worlds,\nwe optionally use an external transformer based\nlanguage model to rescore the n-best hypothesis\nproduced above (Synnaeve et al., 2019). Specifically, for each output in the n-best list, we compute a new score by combining the decoder logprobability, \"\n\nthe rescoring transformer can be used for out ensemble of different models", "votes": null }, { "id": "2353407", "postDate": "07/21/2023 16:17:50", "content": "Yes the model was trained on their own dataset. The model without LM scored WER 0.166 on their evaluation set and with LM performed 0.136.
\nSo I guess the model with LM might score better than this one. Agreed with the drop for the OOD part! But a thing to consider is the valid split contains 30k files, with comparatively shorter sentence lengths. The public LB is evaluated on around 4k samples. So it will be interesting to see how we make a decision from the LB-CV difference.
", "rawMarkdown": "Yes the model was trained on their own dataset. The model without LM scored WER 0.166 on their evaluation set and with LM performed 0.136.\n\nSo I guess the model with LM might score better than this one. Agreed with the drop for the OOD part! But a thing to consider is the valid split contains 30k files, with comparatively shorter sentence lengths. The public LB is evaluated on around 4k samples. So it will be interesting to see how we make a decision from the LB-CV difference.", "votes": null }, { "id": "2353451", "postDate": "07/21/2023 17:03:21", "content": "considering that indian language is close to bengali, this model is a good starting source. they clamied their training data is public … i am trying to find them now
", "rawMarkdown": "considering that indian language is close to bengali, this model is a good starting source. they clamied their training data is public ... i am trying to find them now", "votes": null }, { "id": "2353470", "postDate": "07/21/2023 17:11:55", "content": "Seems so. Good luck!
", "rawMarkdown": "Seems so. Good luck!", "votes": null }, { "id": "2353493", "postDate": "07/21/2023 17:26:22", "content": "maybe here!!!
\nhttps://ai4bharat.iitm.ac.in/dhwani
Dataset Format
\nThe audio files present in separate folders.
\nFor YouTubeThe audio filenames are named YouTube-ids and for Newsonair, the contatination of region name, bulletin timing makes the filename.
more here:
\nhttps://ai4bharat.iitm.ac.in/datasets
conformer model demo
\nhttps://models.ai4bharat.org/#/asr/conformer
Have the same issue. Is here a feasible solution for this error provided?
\nIs that
():\n t = \n \n :\n jiwer.wer(t, [sentence])\n :\n sentence = \n sentence\n\n\nsubmit_df.loc[:,] = submit_df.sentence.apply(check_text)\ndef check():\n ()==:\n = \n \ndf. = df..apply(lambda x: check(x))\nThis should work. The problem is your model can sometimes generate empty strings, and when you convert this empty strings to a dataframe, it appears as nan.
\nSo we're just handling the empty strings by this function, replacing them with a character so that no empty strings remain in the predictions
Is this scenario referring to that the whole prediction for this audio is empty, or there are spaces in the predicted sentences?
", "rawMarkdown": "Is this scenario referring to that the whole prediction for this audio is empty, or there are spaces in the predicted sentences?", "votes": null }, { "id": "2362394", "postDate": "07/28/2023 04:22:33", "content": "For the whole audio. There are some clips in the training/validation set that are 1-2 sec long, consisting of only 2-3 words. Now if the voice Isn't loud enough/audio contain more external noise, the model might fail to even extract a single word/character, thus returning an empty string for the whole audio.
", "rawMarkdown": "For the whole audio. There are some clips in the training/validation set that are 1-2 sec long, consisting of only 2-3 words. Now if the voice Isn't loud enough/audio contain more external noise, the model might fail to even extract a single word/character, thus returning an empty string for the whole audio.", "votes": null }, { "id": "2362398", "postDate": "07/28/2023 04:27:20", "content": "When I submit my notebook and prediction, will the model also predict for audios in training and validating dataset, or some private dataset unseen so that the problem could occur? (I'm somewhat novice, not understanding the competition mechanism)
", "rawMarkdown": "When I submit my notebook and prediction, will the model also predict for audios in training and validating dataset, or some private dataset unseen so that the problem could occur? (I'm somewhat novice, not understanding the competition mechanism)", "votes": null }, { "id": "2362403", "postDate": "07/28/2023 04:37:05", "content": "That's completely fine. You only need to predict for the 'test_mp3s' folder. This folder contains unseen audios (OOD audios), you can only see three of them now. But when you submit your notebook, your notebook will have access to all of the unseen audios( around 8000)
\nYou can have a look at my notebook or the notebook pinned in the code section to see how it works!
", "rawMarkdown": "That's completely fine. You only need to predict for the 'test_mp3s' folder. This folder contains unseen audios (OOD audios), you can only see three of them now. But when you submit your notebook, your notebook will have access to all of the unseen audios( around 8000) \n\nYou can have a look at [my notebook](https://www.kaggle.com/code/mbmmurad/lb-0-506-inference-w-previous-comp-winner-s-model) or the notebook pinned in the code section to see how it works!", "votes": null }, { "id": "2362410", "postDate": "07/28/2023 04:46:17", "content": "OK. Thanks a lot for your detailed explanation!🥰
", "rawMarkdown": "OK. Thanks a lot for your detailed explanation!🥰", "votes": null }, { "id": "2362559", "postDate": "07/28/2023 06:57:12", "content": "I'm newly in compitition i know simple basic Python,pandas and numpy knowledge so can you guide me this project?
", "rawMarkdown": "I'm newly in compitition i know simple basic Python,pandas and numpy knowledge so can you guide me this project?", "votes": null }, { "id": "2362737", "postDate": "07/28/2023 09:17:47", "content": "This may help (about this competition): https://www.kaggle.com/code/ayushs9020/understanding-the-competition-bengalai
\nAnd some lectures like Stanford CS230n may help!
@hengck23 could you give some pointer or example how rescoring transformer would work for ensembling model ?
", "rawMarkdown": "hengck23 could you give some pointer or example how rescoring transformer would work for ensembling model ?", "votes": null } ], "comments": [ { "id": 2352727, "author_name": "dimonovez", "author_url": "", "post_date": "07/21/2023 08:16:51", "content": "Have same issue(
", "votes": null, "replies": [] }, { "id": 2352900, "author_name": "mbmmurad", "author_url": "", "post_date": "07/21/2023 10:33:06", "content": "", "votes": null, "replies": [ { "id": 2353004, "author_name": "hengck23", "author_url": "", "post_date": "07/21/2023 11:46:54", "content": "i think both test folder and sample_submission.csv are replaced during submission.
\nMy code work for other model, so i think it is not the reason.
besides, if i were only processing three files, the submission time should be very short
", "votes": null, "replies": [ { "id": 2353306, "author_name": "mbmmurad", "author_url": "", "post_date": "07/21/2023 15:21:12", "content": "You're right. My mistake. Your pipeline worked. Just needed to handle the empty strings.
", "votes": null, "replies": [] } ] } ] }, { "id": 2352945, "author_name": "mbmmurad", "author_url": "", "post_date": "07/21/2023 10:59:27", "content": "Pseudo-code of a correct way of submission :
\nfilenames = \npredictions = \nfiles = os()\nbase_path = \n\n file files:\n filenames(file())\n = (base_path+file) function to load the file\n text = (audio) with your model\n (text)==:\n text = \n\n predictions(text)\n\ndf = pd({:filenames,:predictions})\ndf(,index=False)\nif you use a normalizer than make sure to check the texts after normalizing.
\ndef check():\n ()==:\n = \n \ndf. = df..apply(lambda x:normalizer(x)))\ndf. = df..apply(lambda x:check(x)))\nthe latest version still course error:
\n mode==:\n mp3_dir = \n glob_file = glob()\n = ([f[(mp3_dir)+:-] f glob_file])\n valid_df = pd.DataFrame({:,:})\ndef check_text():\n t = \n \n :\n jiwer.wer(t, [])\n except:\n = \n \n\n\nsubmit_df.loc[:,] = submit_df..apply(check_text)\nJust change to sentence = \",\" .
I successfully submitted using your notebook. The main problem is :
\n\n\nThe model is generating some empty predictions. It seems like you cant't submit empty strings in the dataframe. I replaced the empty strings with \",\" and it got submitted.
\n
filenames = \npredictions = \nfiles = os()\nbase_path = \n\n file files:\n filenames(file())\n = (base_path+file) function to load the file\n text = (audio) with your model\n (text)==:\n text = \n\n predictions(text)\n\ndf = pd({:filenames,:predictions})\ndf(,index=False)\nRefer to this notebook
", "votes": null, "replies": [ { "id": 2353348, "author_name": "hengck23", "author_url": "", "post_date": "07/21/2023 15:43:29", "content": "\nThanks a lot!!!
", "votes": null, "replies": [ { "id": 2353352, "author_name": "hengck23", "author_url": "", "post_date": "07/21/2023 15:45:00", "content": "by the way, this is wave2vec2-CTC without LM
", "votes": null, "replies": [ { "id": 2353382, "author_name": "hengck23", "author_url": "", "post_date": "07/21/2023 16:02:55", "content": "more information about the model can be found here:
\nhttps://github.com/AI4Bharat/IndicWav2Vec/tree/main
i think the model is trained with author's own dataset.
\nHence we can conclude? OOD cause adroup of about 0.12
\n(since kaggle valid split has WER = 0.40, public LB 0.52)
IndicWav2Vec paper:
\nhttps://arxiv.org/pdf/2111.03945.pdf
\"Apart from YouTube, we also curated content
\nfrom newsonair3 which is a radio news channel
\nrun by the Govt. of India and broadcasts news in
\nmultiple Indian languages\"
there is CTC-LM model which gives better results but you need to convert from fairseq to hf yourself,
\nfollowing instructions from githut repo
\"3.5 Rescoring
\nNote that in the above decoding process, we use an
\nn-gram KenLM language model (Heafield, 2011).
\nHowever, recent advances in language modeling
\nhave shown that transformer based language models perform well. To get the best of both worlds,
\nwe optionally use an external transformer based
\nlanguage model to rescore the n-best hypothesis
\nproduced above (Synnaeve et al., 2019). Specifically, for each output in the n-best list, we compute a new score by combining the decoder logprobability, \"
the rescoring transformer can be used for out ensemble of different models
", "votes": null, "replies": [ { "id": 2353407, "author_name": "mbmmurad", "author_url": "", "post_date": "07/21/2023 16:17:50", "content": "Yes the model was trained on their own dataset. The model without LM scored WER 0.166 on their evaluation set and with LM performed 0.136.
\nSo I guess the model with LM might score better than this one. Agreed with the drop for the OOD part! But a thing to consider is the valid split contains 30k files, with comparatively shorter sentence lengths. The public LB is evaluated on around 4k samples. So it will be interesting to see how we make a decision from the LB-CV difference.
", "votes": null, "replies": [ { "id": 2353451, "author_name": "hengck23", "author_url": "", "post_date": "07/21/2023 17:03:21", "content": "considering that indian language is close to bengali, this model is a good starting source. they clamied their training data is public … i am trying to find them now
", "votes": null, "replies": [ { "id": 2353470, "author_name": "mbmmurad", "author_url": "", "post_date": "07/21/2023 17:11:55", "content": "Seems so. Good luck!
", "votes": null, "replies": [ { "id": 2353493, "author_name": "hengck23", "author_url": "", "post_date": "07/21/2023 17:26:22", "content": "maybe here!!!
\nhttps://ai4bharat.iitm.ac.in/dhwani
Dataset Format
\nThe audio files present in separate folders.
\nFor YouTubeThe audio filenames are named YouTube-ids and for Newsonair, the contatination of region name, bulletin timing makes the filename.
more here:
\nhttps://ai4bharat.iitm.ac.in/datasets
conformer model demo
\nhttps://models.ai4bharat.org/#/asr/conformer
@hengck23 could you give some pointer or example how rescoring transformer would work for ensembling model ?
", "votes": null, "replies": [] } ] } ] } ] } ] }, { "id": 2362376, "author_name": "nisshokuitsuki", "author_url": "", "post_date": "07/28/2023 04:01:56", "content": "Have the same issue. Is here a feasible solution for this error provided?
\nIs that
():\n t = \n \n :\n jiwer.wer(t, [sentence])\n :\n sentence = \n sentence\n\n\nsubmit_df.loc[:,] = submit_df.sentence.apply(check_text)\ndef check():\n ()==:\n = \n \ndf. = df..apply(lambda x: check(x))\nThis should work. The problem is your model can sometimes generate empty strings, and when you convert this empty strings to a dataframe, it appears as nan.
\nSo we're just handling the empty strings by this function, replacing them with a character so that no empty strings remain in the predictions
Is this scenario referring to that the whole prediction for this audio is empty, or there are spaces in the predicted sentences?
", "votes": null, "replies": [ { "id": 2362394, "author_name": "mbmmurad", "author_url": "", "post_date": "07/28/2023 04:22:33", "content": "For the whole audio. There are some clips in the training/validation set that are 1-2 sec long, consisting of only 2-3 words. Now if the voice Isn't loud enough/audio contain more external noise, the model might fail to even extract a single word/character, thus returning an empty string for the whole audio.
", "votes": null, "replies": [ { "id": 2362398, "author_name": "nisshokuitsuki", "author_url": "", "post_date": "07/28/2023 04:27:20", "content": "When I submit my notebook and prediction, will the model also predict for audios in training and validating dataset, or some private dataset unseen so that the problem could occur? (I'm somewhat novice, not understanding the competition mechanism)
", "votes": null, "replies": [ { "id": 2362403, "author_name": "mbmmurad", "author_url": "", "post_date": "07/28/2023 04:37:05", "content": "That's completely fine. You only need to predict for the 'test_mp3s' folder. This folder contains unseen audios (OOD audios), you can only see three of them now. But when you submit your notebook, your notebook will have access to all of the unseen audios( around 8000)
\nYou can have a look at my notebook or the notebook pinned in the code section to see how it works!
", "votes": null, "replies": [ { "id": 2362410, "author_name": "nisshokuitsuki", "author_url": "", "post_date": "07/28/2023 04:46:17", "content": "OK. Thanks a lot for your detailed explanation!🥰
", "votes": null, "replies": [] } ] } ] }, { "id": 2362559, "author_name": "sonalisul", "author_url": "", "post_date": "07/28/2023 06:57:12", "content": "I'm newly in compitition i know simple basic Python,pandas and numpy knowledge so can you guide me this project?
", "votes": null, "replies": [ { "id": 2362737, "author_name": "nisshokuitsuki", "author_url": "", "post_date": "07/28/2023 09:17:47", "content": "This may help (about this competition): https://www.kaggle.com/code/ayushs9020/understanding-the-competition-bengalai
\nAnd some lectures like Stanford CS230n may help!