![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F390e29106b1d53147e9a0868da08bfa7%2FSelection_999(2801).png?generation=1690204289638959&alt=media\")
How is your CV compare to LB?
\nDo you use the \"default\" train/val split?
How is your CV compare to LB?
\nDo you use the \"default\" train/val split?
see
\nhttps://www.kaggle.com/competitions/bengaliai-speech/discussion/425496#2356341
it may be more importnat to check this
\n
That's neat! Thank you!
\nSo someone has transcribed the given examples I guess?
Yeah, I have done it here. It can be used as a pseudo-label, but don't completely rely on the numerical results generated using these transcriptions.
", "rawMarkdown": "Yeah, I have done it [here](https://www.kaggle.com/competitions/bengaliai-speech/discussion/425932). It can be used as a pseudo-label, but don't completely rely on the numerical results generated using these transcriptions.", "votes": null }, { "id": "2357979", "postDate": "07/25/2023 08:38:10", "content": "Of course, it's very low number of transcripts, but it's much much better than nothing, it can tell you if something is very off.
\nSo thank you very much for this!:)
Well, thank you too. Hope this helps!
", "rawMarkdown": "Well, thank you too. Hope this helps!", "votes": null }, { "id": "2358147", "postDate": "07/25/2023 10:10:40", "content": "anyone has good results (better than yellow-king model) using this kaggle training set?
", "rawMarkdown": "anyone has good results (better than yellow-king model) using this kaggle training set?", "votes": null }, { "id": "2358470", "postDate": "07/25/2023 14:23:21", "content": "@hengck23 How long does it take for you to train an epoch?
", "rawMarkdown": "hengck23 How long does it take for you to train an epoch?", "votes": null }, { "id": "2360893", "postDate": "07/27/2023 04:45:40", "content": "Since this competition is aim to test mdel's out-of-distribution performance, I believe we should use GroupKFold, however, there's no domain column in the dataset
", "rawMarkdown": "Since this competition is aim to test mdel's out-of-distribution performance, I believe we should use GroupKFold, however, there's no domain column in the dataset", "votes": null }, { "id": "2361110", "postDate": "07/27/2023 07:29:13", "content": "I've only listened a couple of the samples, they all seem like someone reading a text or something like that.
\nIt would be good to see though, how well CV correlates with LB.
Yeah. They collected most of the audios using this process, where given a sentence, a person would read this out.
\nThese are in the train fold.
The test set contains audios from TV channels, songs, slangs etc.
", "rawMarkdown": "Yeah. They collected most of the audios using this process, where given a sentence, a person would read this out. \nThese are in the train fold.\n\nThe test set contains audios from TV channels, songs, slangs etc.", "votes": null }, { "id": "2361341", "postDate": "07/27/2023 10:53:39", "content": "Sounds like that we need to collect the validation set according to the examples by ourselves:)
", "rawMarkdown": "Sounds like that we need to collect the validation set according to the examples by ourselves:)", "votes": null }, { "id": "2361587", "postDate": "07/27/2023 13:10:13", "content": "\n\n\n\nSounds like that we need to collect the validation set according to the examples by ourselves:)
\n
This is what i recommend too.
\n(but collecting data for ASR is time consuming)
You should also read papers that discuss about out-of-domain ASR.
\ne.g.
this is from the dataset paper:
\nWe machine-validate the un-validate test dataset using Google ASR model and another model (wav2vec 2.0 [13]) and exclude blank recordings. We manually validated samples that had at least one full word difference between wav2vec 2.0 and ground truth. After this we finalize the machine-validated data.
\nwav2vec 2.0 model[13] refers to yellow-king model.
\nso if your model consistently perform better wav2vec 2.0 model[13] and Google ASR , you can be sure that there will be no shakeup in hidden private test set (even though it is OOD). THis is the trick in winning in this competition
", "rawMarkdown": "this is from the dataset paper:\n\nWe machine-validate the un-validate test dataset using Google ASR model and another model (wav2vec 2.0 [13]) and exclude blank recordings. We manually validated samples that had at least one full word difference between wav2vec 2.0 and ground truth. After this we finalize the machine-validated data.\n\n---\n\nwav2vec 2.0 model[13] refers to yellow-king model.\n\nso if your model consistently perform better wav2vec 2.0 model[13] and Google ASR , you can be sure that there will be no shakeup in hidden private test set (even though it is OOD). THis is the trick in winning in this competition", "votes": null }, { "id": "2362759", "postDate": "07/28/2023 09:39:57", "content": "@nofreewill Please refer this article, I find out this useful
", "rawMarkdown": "nofreewill Please refer this article, I find out this useful", "votes": null }, { "id": "2363849", "postDate": "07/29/2023 02:21:51", "content": "Thanks for sharing. @hengck23, let me check the resources that you mentioned
", "rawMarkdown": "Thanks for sharing. @hengck23, let me check the resources that you mentioned", "votes": null }, { "id": "2363854", "postDate": "07/29/2023 02:30:36", "content": "I finetuned the YellowKing-model with 30k new data and 10k validation.
\nCV : 0.561 on the new 10k validation set.
\nLB : 0.487
@mbmmurad
\nThanks for the information.
I have been studying wave2vec paper,
\nhttps://arxiv.org/abs/2006.11477
i think Table 9 can be a good guide on performance versus amount of data.
\n(but do note that there is language difference, bn verus en and domain difference)
\n\"Table 9: WER on the Librispeech dev/test sets when training on the Libri-light low-resource labeled
\ndata setups (cf. Table 1).\"
for domain experiments, i recommend this: https://arxiv.org/abs/2104.01027
", "rawMarkdown": "mbmmurad \nThanks for the information.\n\nI have been studying wave2vec paper,\nhttps://arxiv.org/abs/2006.11477\n\ni think Table 9 can be a good guide on performance versus amount of data.\n(but do note that there is language difference, bn verus en and domain difference)\n\"Table 9: WER on the Librispeech dev/test sets when training on the Libri-light low-resource labeled\ndata setups (cf. Table 1).\"\n\nfor domain experiments, i recommend this: https://arxiv.org/abs/2104.01027", "votes": null }, { "id": "2403238", "postDate": "08/22/2023 15:15:24", "content": "2023-08-22:
\nCV: 0.284
\nLB: 0.470
\nStill need lots of work.
see
\nhttps://www.kaggle.com/competitions/bengaliai-speech/discussion/425496#2356341
it may be more importnat to check this
\n
That's neat! Thank you!
\nSo someone has transcribed the given examples I guess?
Yeah, I have done it here. It can be used as a pseudo-label, but don't completely rely on the numerical results generated using these transcriptions.
", "votes": null, "replies": [ { "id": 2357979, "author_name": "nofreewill", "author_url": "", "post_date": "07/25/2023 08:38:10", "content": "Of course, it's very low number of transcripts, but it's much much better than nothing, it can tell you if something is very off.
\nSo thank you very much for this!:)
Well, thank you too. Hope this helps!
", "votes": null, "replies": [] } ] } ] } ] } ] } ] }, { "id": 2358147, "author_name": "hengck23", "author_url": "", "post_date": "07/25/2023 10:10:40", "content": "anyone has good results (better than yellow-king model) using this kaggle training set?
", "votes": null, "replies": [] }, { "id": 2358470, "author_name": "arunodhayan", "author_url": "", "post_date": "07/25/2023 14:23:21", "content": "@hengck23 How long does it take for you to train an epoch?
", "votes": null, "replies": [] }, { "id": 2360893, "author_name": "wuwenmin", "author_url": "", "post_date": "07/27/2023 04:45:40", "content": "Since this competition is aim to test mdel's out-of-distribution performance, I believe we should use GroupKFold, however, there's no domain column in the dataset
", "votes": null, "replies": [ { "id": 2361110, "author_name": "nofreewill", "author_url": "", "post_date": "07/27/2023 07:29:13", "content": "I've only listened a couple of the samples, they all seem like someone reading a text or something like that.
\nIt would be good to see though, how well CV correlates with LB.
Yeah. They collected most of the audios using this process, where given a sentence, a person would read this out.
\nThese are in the train fold.
The test set contains audios from TV channels, songs, slangs etc.
", "votes": null, "replies": [ { "id": 2361341, "author_name": "wuwenmin", "author_url": "", "post_date": "07/27/2023 10:53:39", "content": "Sounds like that we need to collect the validation set according to the examples by ourselves:)
", "votes": null, "replies": [ { "id": 2361587, "author_name": "hengck23", "author_url": "", "post_date": "07/27/2023 13:10:13", "content": "\n\n\n\nSounds like that we need to collect the validation set according to the examples by ourselves:)
\n
This is what i recommend too.
\n(but collecting data for ASR is time consuming)
You should also read papers that discuss about out-of-domain ASR.
\ne.g.
this is from the dataset paper:
\nWe machine-validate the un-validate test dataset using Google ASR model and another model (wav2vec 2.0 [13]) and exclude blank recordings. We manually validated samples that had at least one full word difference between wav2vec 2.0 and ground truth. After this we finalize the machine-validated data.
\nwav2vec 2.0 model[13] refers to yellow-king model.
\nso if your model consistently perform better wav2vec 2.0 model[13] and Google ASR , you can be sure that there will be no shakeup in hidden private test set (even though it is OOD). THis is the trick in winning in this competition
", "votes": null, "replies": [] }, { "id": 2363849, "author_name": "wuwenmin", "author_url": "", "post_date": "07/29/2023 02:21:51", "content": "Thanks for sharing. @hengck23, let me check the resources that you mentioned
", "votes": null, "replies": [] } ] } ] } ] } ] } ] }, { "id": 2362759, "author_name": "dhirajmwagh1111", "author_url": "", "post_date": "07/28/2023 09:39:57", "content": "@nofreewill Please refer this article, I find out this useful
", "votes": null, "replies": [] }, { "id": 2363854, "author_name": "mbmmurad", "author_url": "", "post_date": "07/29/2023 02:30:36", "content": "I finetuned the YellowKing-model with 30k new data and 10k validation.
\nCV : 0.561 on the new 10k validation set.
\nLB : 0.487
@mbmmurad
\nThanks for the information.
I have been studying wave2vec paper,
\nhttps://arxiv.org/abs/2006.11477
i think Table 9 can be a good guide on performance versus amount of data.
\n(but do note that there is language difference, bn verus en and domain difference)
\n\"Table 9: WER on the Librispeech dev/test sets when training on the Libri-light low-resource labeled
\ndata setups (cf. Table 1).\"
for domain experiments, i recommend this: https://arxiv.org/abs/2104.01027
", "votes": null, "replies": [] } ] }, { "id": 2403238, "author_name": "berserker408", "author_url": "", "post_date": "08/22/2023 15:15:24", "content": "2023-08-22:
\nCV: 0.284
\nLB: 0.470
\nStill need lots of work.