I got sometime today to clean up my code and write the solution doc.
\n\nPushed them here https://github.com/xiaozhouwang/tensorflow_speech_recognition_solution
\n\nMy best single model was only 0.87 on public LB and ensemble of them could only barely enter 0.89. The boost came from two semi supervised learning techniques. You can refer to the code and doc but I also conveniently posted the key part below:
\n\nModels
\n\n10 main models were trained
\n\nResnet with 9 layers on mfcc and mel (input reshaped to 128 * 128)
\n\nSenet18 on mfcc and mel (input reshaped to 128 * 128)
\n\nDensenet121 on mfcc and mel (input reshaped to 128 * 128)
\n\nVGG (with GlobalMaxPooling+GlobalAveragePooling in the end) on mfcc and mel (input reshaped to 128 * 128)
\n\nVGG (with GlobalMaxPooling+GlobalAveragePooling in the end) on raw input
\n\nVGG (with fully connected layers in the end) on mel where it only convs along the time axis
\n\nSemi Supervised Learning
\n\nWeighted average of previously trained 10 models was used as the “ground truth” labels for test data. Two semi supervised learning techniques were used:
\n\n100% of training data + 20%-35% of test data selected per epoch were used as the new training data. Resnet (mfcc and mel), Senet (mfcc and mel) and VGG (mel and raw) were trained this way.
\n\n100% of test data were used to train a “pretrained” model. And then the weights of the “pretrained” model were loaded as initial weights to train the same model with only training data (fine tuning with training data). It turned out that fine tuning with only one epoch was enough to “correct” the overfitted “pretrained” model.
\n\nBoth techniques were used with the hope that they could learn/approximate the test distribution better (instead of “overfitting” training data distribution). First one does it on the data level and second one does it on the model level.
\n\nThank you Kaggle and Google Brain team for this great competition and congrats to all top teams! This surely wasn't an easy competition and I had quite a lot of fun doing it!
\n\nUpdate:\nuploaded my best private sub with pred scores (slightly better than my selected one, 0.91084).
", "messages": [ { "id": "270256", "postDate": "01/18/2018 00:48:49", "content": "I got sometime today to clean up my code and write the solution doc.
\n\nPushed them here https://github.com/xiaozhouwang/tensorflow_speech_recognition_solution
\n\nMy best single model was only 0.87 on public LB and ensemble of them could only barely enter 0.89. The boost came from two semi supervised learning techniques. You can refer to the code and doc but I also conveniently posted the key part below:
\n\nModels
\n\n10 main models were trained
\n\nResnet with 9 layers on mfcc and mel (input reshaped to 128 * 128)
\n\nSenet18 on mfcc and mel (input reshaped to 128 * 128)
\n\nDensenet121 on mfcc and mel (input reshaped to 128 * 128)
\n\nVGG (with GlobalMaxPooling+GlobalAveragePooling in the end) on mfcc and mel (input reshaped to 128 * 128)
\n\nVGG (with GlobalMaxPooling+GlobalAveragePooling in the end) on raw input
\n\nVGG (with fully connected layers in the end) on mel where it only convs along the time axis
\n\nSemi Supervised Learning
\n\nWeighted average of previously trained 10 models was used as the “ground truth” labels for test data. Two semi supervised learning techniques were used:
\n\n100% of training data + 20%-35% of test data selected per epoch were used as the new training data. Resnet (mfcc and mel), Senet (mfcc and mel) and VGG (mel and raw) were trained this way.
\n\n100% of test data were used to train a “pretrained” model. And then the weights of the “pretrained” model were loaded as initial weights to train the same model with only training data (fine tuning with training data). It turned out that fine tuning with only one epoch was enough to “correct” the overfitted “pretrained” model.
\n\nBoth techniques were used with the hope that they could learn/approximate the test distribution better (instead of “overfitting” training data distribution). First one does it on the data level and second one does it on the model level.
\n\nThank you Kaggle and Google Brain team for this great competition and congrats to all top teams! This surely wasn't an easy competition and I had quite a lot of fun doing it!
\n\nUpdate:\nuploaded my best private sub with pred scores (slightly better than my selected one, 0.91084).
", "rawMarkdown": "I got sometime today to clean up my code and write the solution doc. \n\nPushed them here https://github.com/xiaozhouwang/tensorflow_speech_recognition_solution\n\nMy best single model was only 0.87 on public LB and ensemble of them could only barely enter 0.89. The boost came from two semi supervised learning techniques. You can refer to the code and doc but I also conveniently posted the key part below:\n\n\n**Models**\n\n10 main models were trained\n\nResnet with 9 layers on mfcc and mel (input reshaped to 128 * 128)\n\nSenet18 on mfcc and mel (input reshaped to 128 * 128)\n\nDensenet121 on mfcc and mel (input reshaped to 128 * 128)\n\nVGG (with GlobalMaxPooling+GlobalAveragePooling in the end) on mfcc and mel (input reshaped to 128 * 128)\n\nVGG (with GlobalMaxPooling+GlobalAveragePooling in the end) on raw input\n\nVGG (with fully connected layers in the end) on mel where it only convs along the time axis\n\n**Semi Supervised Learning**\n\nWeighted average of previously trained 10 models was used as the “ground truth” labels for test data. Two semi supervised learning techniques were used:\n\n100% of training data + 20%-35% of test data selected per epoch were used as the new training data. Resnet (mfcc and mel), Senet (mfcc and mel) and VGG (mel and raw) were trained this way.\n\n100% of test data were used to train a “pretrained” model. And then the weights of the “pretrained” model were loaded as initial weights to train the same model with only training data (fine tuning with training data). It turned out that fine tuning with only one epoch was enough to “correct” the overfitted “pretrained” model.\n\nBoth techniques were used with the hope that they could learn/approximate the test distribution better (instead of “overfitting” training data distribution). First one does it on the data level and second one does it on the model level. \n\nThank you Kaggle and Google Brain team for this great competition and congrats to all top teams! This surely wasn't an easy competition and I had quite a lot of fun doing it!\n\n\nUpdate:\nuploaded my best private sub with pred scores (slightly better than my selected one, 0.91084).", "votes": null }, { "id": "270257", "postDate": "01/18/2018 00:51:43", "content": "Thank you for sharing， 感谢舟神分享
", "rawMarkdown": "Thank you for sharing， 感谢舟神分享", "votes": null }, { "id": "270258", "postDate": "01/18/2018 00:52:36", "content": "Thank you for sharing， 感谢舟神分享 +1
", "rawMarkdown": "Thank you for sharing， 感谢舟神分享 +1", "votes": null }, { "id": "270267", "postDate": "01/18/2018 01:17:08", "content": "great work!
", "rawMarkdown": "great work!", "votes": null }, { "id": "270276", "postDate": "01/18/2018 01:53:58", "content": "Hi Xiao Zhou,
\n\nSo for the test set pretrained model, you used the training data to train a model, and with the test set results, you used that to retrain a model and then use the trained weights as the initialization weights to retrain the training data?
\n\nDoes that help with the training model to converge faster or what is the point of that?
\n\nThanks.
\n\nBest.
", "rawMarkdown": "Hi Xiao Zhou,\n\nSo for the test set pretrained model, you used the training data to train a model, and with the test set results, you used that to retrain a model and then use the trained weights as the initialization weights to retrain the training data?\n\nDoes that help with the training model to converge faster or what is the point of that?\n\nThanks.\n\nBest.", "votes": null }, { "id": "270279", "postDate": "01/18/2018 01:59:54", "content": "sorry if I was not clear. So say, you have a model that scores 0.89 on LB.
\n\nYou take the submission of that model, and use that as the labels for test data. Now this test data with 0.89 accurate labels become the new training data for the pretrained model. You train your pretrained model with only test data.
\n\nAnd then you use the weights from the pretrained model to initialize one same model, and now, instead of training on test data, you train on training data. and then do prediction with this fine tuned model for test data.
\n\nHope it is clearer now.
", "rawMarkdown": "sorry if I was not clear. So say, you have a model that scores 0.89 on LB.\n\nYou take the submission of that model, and use that as the labels for test data. Now this test data with 0.89 accurate labels become the new training data for the pretrained model. You train your pretrained model with only test data.\n\nAnd then you use the weights from the pretrained model to initialize one same model, and now, instead of training on test data, you train on training data. and then do prediction with this fine tuned model for test data.\n\nHope it is clearer now.", "votes": null }, { "id": "270283", "postDate": "01/18/2018 02:11:18", "content": "Thank you for sharing， 感谢舟神分享+2
", "rawMarkdown": "Thank you for sharing， 感谢舟神分享+2", "votes": null }, { "id": "270286", "postDate": "01/18/2018 02:27:19", "content": "Congratulation and Thanks for your great sharing. Would you please explain a little more on the ensemble weight? Specifically, how did you come out 0.75, 0.25 and 0.4 (vgg1d_raw.as_matrix() * 0.75 + vgg1d_mel.as_matrix() * 0.25) * 0.4? I understand the formula is base on the public LB, just curious the detail steps.
", "rawMarkdown": "Congratulation and Thanks for your great sharing. Would you please explain a little more on the ensemble weight? Specifically, how did you come out 0.75, 0.25 and 0.4 (vgg1d_raw.as_matrix() * 0.75 + vgg1d_mel.as_matrix() * 0.25) * 0.4? I understand the formula is base on the public LB, just curious the detail steps.", "votes": null }, { "id": "270296", "postDate": "01/18/2018 03:03:13", "content": "sure. I just look at the correlation of the submission probabilities of each class, and then use the difference of the model/inputs as well as model score on public LB and then decide a weight and then submit, if it is better than any of the single model score, I just use that. I might try 2 - 3 times and then settle with the weights.
", "rawMarkdown": "sure. I just look at the correlation of the submission probabilities of each class, and then use the difference of the model/inputs as well as model score on public LB and then decide a weight and then submit, if it is better than any of the single model score, I just use that. I might try 2 - 3 times and then settle with the weights.", "votes": null }, { "id": "270298", "postDate": "01/18/2018 03:04:49", "content": "Thank you for sharing， 感谢舟神分享+3
", "rawMarkdown": "Thank you for sharing， 感谢舟神分享+3", "votes": null }, { "id": "270300", "postDate": "01/18/2018 03:07:15", "content": "Thanks mate o/
", "rawMarkdown": "Thanks mate o/", "votes": null }, { "id": "270306", "postDate": "01/18/2018 03:25:51", "content": "Thank you for sharing， 感谢舟神分享
", "rawMarkdown": "Thank you for sharing， 感谢舟神分享", "votes": null }, { "id": "270332", "postDate": "01/18/2018 05:18:33", "content": "Thanks for sharing! Semi supervised learning works!
\n\nDo you mind posting your submission CSV file along with the raw probability values here?\nI can make a comparison between your results and mine. I posted mine at: https://www.kaggle.com/c/tensorflow-speech-recognition-challenge/discussion/47728
", "rawMarkdown": "Thanks for sharing! Semi supervised learning works! \n\nDo you mind posting your submission CSV file along with the raw probability values here?\nI can make a comparison between your results and mine. I posted mine at: https://www.kaggle.com/c/tensorflow-speech-recognition-challenge/discussion/47728", "votes": null }, { "id": "270337", "postDate": "01/18/2018 05:28:03", "content": "For sure but somehow it seems there is a limit for uploading files. So I could only attach one with predicted probabilities. The one attached has slightly better score than my selected one. (You can find the file in the main post)
\n\nAlso congrats!
", "rawMarkdown": "For sure but somehow it seems there is a limit for uploading files. So I could only attach one with predicted probabilities. The one attached has slightly better score than my selected one. (You can find the file in the main post)\n\n\nAlso congrats!", "votes": null }, { "id": "270526", "postDate": "01/18/2018 12:25:05", "content": "Great job Little Boat and thanks so much for sharing. Brilliant use of semi-supervised and congrats on a top finish!
", "rawMarkdown": "Great job Little Boat and thanks so much for sharing. Brilliant use of semi-supervised and congrats on a top finish!", "votes": null }, { "id": "270779", "postDate": "01/18/2018 22:37:43", "content": "给舟神疯狂打call ：D
", "rawMarkdown": "给舟神疯狂打call ：D", "votes": null }, { "id": "271364", "postDate": "01/20/2018 09:35:52", "content": "Congratulations !
\n\nDo you think you could turn your prediction .csv files into Kaggle datasets (so that we can use it from kernels) ?
", "rawMarkdown": "Congratulations !\n\nDo you think you could turn your prediction .csv files into Kaggle datasets (so that we can use it from kernels) ?", "votes": null }, { "id": "302859", "postDate": "03/24/2018 21:19:19", "content": "First time seeing someone pretrain a model using semi-supervised learning on the full test set, thanks for sharing!
\n\nDid you see significant improvement over supervised learning without pretraining ?
", "rawMarkdown": "First time seeing someone pretrain a model using semi-supervised learning on the full test set, thanks for sharing!\n\nDid you see significant improvement over supervised learning without pretraining ?", "votes": null }, { "id": "3000208", "postDate": "09/27/2024 11:52:08", "content": "Thanks for the idea. I used it for other ASR competition, it worked very well !
", "rawMarkdown": "Thanks for the idea. I used it for other ASR competition, it worked very well !", "votes": null } ], "comments": [ { "id": 3000208, "author_name": "daonguyenduong", "author_url": "", "post_date": "09/27/2024 11:52:08", "content": "Thanks for the idea. I used it for other ASR competition, it worked very well !
", "votes": null, "replies": [] }, { "id": 270257, "author_name": "ryanzhang", "author_url": "", "post_date": "01/18/2018 00:51:43", "content": "Thank you for sharing， 感谢舟神分享
", "votes": null, "replies": [] }, { "id": 270258, "author_name": "shujian", "author_url": "", "post_date": "01/18/2018 00:52:36", "content": "Thank you for sharing， 感谢舟神分享 +1
", "votes": null, "replies": [] }, { "id": 270267, "author_name": "danielyang1009", "author_url": "", "post_date": "01/18/2018 01:17:08", "content": "great work!
", "votes": null, "replies": [] }, { "id": 270276, "author_name": "bopengiowa", "author_url": "", "post_date": "01/18/2018 01:53:58", "content": "Hi Xiao Zhou,
\n\nSo for the test set pretrained model, you used the training data to train a model, and with the test set results, you used that to retrain a model and then use the trained weights as the initialization weights to retrain the training data?
\n\nDoes that help with the training model to converge faster or what is the point of that?
\n\nThanks.
\n\nBest.
", "votes": null, "replies": [ { "id": 270279, "author_name": "xiaozhouwang", "author_url": "", "post_date": "01/18/2018 01:59:54", "content": "sorry if I was not clear. So say, you have a model that scores 0.89 on LB.
\n\nYou take the submission of that model, and use that as the labels for test data. Now this test data with 0.89 accurate labels become the new training data for the pretrained model. You train your pretrained model with only test data.
\n\nAnd then you use the weights from the pretrained model to initialize one same model, and now, instead of training on test data, you train on training data. and then do prediction with this fine tuned model for test data.
\n\nHope it is clearer now.
", "votes": null, "replies": [] } ] }, { "id": 270283, "author_name": "kueipo", "author_url": "", "post_date": "01/18/2018 02:11:18", "content": "Thank you for sharing， 感谢舟神分享+2
", "votes": null, "replies": [] }, { "id": 270286, "author_name": "johnsondata", "author_url": "", "post_date": "01/18/2018 02:27:19", "content": "Congratulation and Thanks for your great sharing. Would you please explain a little more on the ensemble weight? Specifically, how did you come out 0.75, 0.25 and 0.4 (vgg1d_raw.as_matrix() * 0.75 + vgg1d_mel.as_matrix() * 0.25) * 0.4? I understand the formula is base on the public LB, just curious the detail steps.
", "votes": null, "replies": [ { "id": 270296, "author_name": "xiaozhouwang", "author_url": "", "post_date": "01/18/2018 03:03:13", "content": "sure. I just look at the correlation of the submission probabilities of each class, and then use the difference of the model/inputs as well as model score on public LB and then decide a weight and then submit, if it is better than any of the single model score, I just use that. I might try 2 - 3 times and then settle with the weights.
", "votes": null, "replies": [] } ] }, { "id": 270298, "author_name": "taka152", "author_url": "", "post_date": "01/18/2018 03:04:49", "content": "Thank you for sharing， 感谢舟神分享+3
", "votes": null, "replies": [] }, { "id": 270300, "author_name": "edreams", "author_url": "", "post_date": "01/18/2018 03:07:15", "content": "Thanks mate o/
", "votes": null, "replies": [] }, { "id": 270306, "author_name": "yyll008", "author_url": "", "post_date": "01/18/2018 03:25:51", "content": "Thank you for sharing， 感谢舟神分享
", "votes": null, "replies": [] }, { "id": 270332, "author_name": "hengck23", "author_url": "", "post_date": "01/18/2018 05:18:33", "content": "Thanks for sharing! Semi supervised learning works!
\n\nDo you mind posting your submission CSV file along with the raw probability values here?\nI can make a comparison between your results and mine. I posted mine at: https://www.kaggle.com/c/tensorflow-speech-recognition-challenge/discussion/47728
", "votes": null, "replies": [ { "id": 270337, "author_name": "xiaozhouwang", "author_url": "", "post_date": "01/18/2018 05:28:03", "content": "For sure but somehow it seems there is a limit for uploading files. So I could only attach one with predicted probabilities. The one attached has slightly better score than my selected one. (You can find the file in the main post)
\n\nAlso congrats!
", "votes": null, "replies": [] } ] }, { "id": 270526, "author_name": "sasrdw", "author_url": "", "post_date": "01/18/2018 12:25:05", "content": "Great job Little Boat and thanks so much for sharing. Brilliant use of semi-supervised and congrats on a top finish!
", "votes": null, "replies": [] }, { "id": 270779, "author_name": "jiweiliu", "author_url": "", "post_date": "01/18/2018 22:37:43", "content": "给舟神疯狂打call ：D
", "votes": null, "replies": [] }, { "id": 271364, "author_name": "holzner", "author_url": "", "post_date": "01/20/2018 09:35:52", "content": "Congratulations !
\n\nDo you think you could turn your prediction .csv files into Kaggle datasets (so that we can use it from kernels) ?
", "votes": null, "replies": [] }, { "id": 302859, "author_name": "marvinl", "author_url": "", "post_date": "03/24/2018 21:19:19", "content": "First time seeing someone pretrain a model using semi-supervised learning on the full test set, thanks for sharing!
\n\nDid you see significant improvement over supervised learning without pretraining ?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "270256": "I got sometime today to clean up my code and write the solution doc. \n\nPushed them here https://github.com/xiaozhouwang/tensorflow_speech_recognition_solution\n\nMy best single model was only 0.87 on public LB and ensemble of them could only barely enter 0.89. The boost came from two semi supervised learning techniques. You can refer to the code and doc but I also conveniently posted the key part below:\n\n\n**Models**\n\n10 main models were trained\n\nResnet with 9 layers on mfcc and mel (input reshaped to 128 * 128)\n\nSenet18 on mfcc and mel (input reshaped to 128 * 128)\n\nDensenet121 on mfcc and mel (input reshaped to 128 * 128)\n\nVGG (with GlobalMaxPooling+GlobalAveragePooling in the end) on mfcc and mel (input reshaped to 128 * 128)\n\nVGG (with GlobalMaxPooling+GlobalAveragePooling in the end) on raw input\n\nVGG (with fully connected layers in the end) on mel where it only convs along the time axis\n\n**Semi Supervised Learning**\n\nWeighted average of previously trained 10 models was used as the “ground truth” labels for test data. Two semi supervised learning techniques were used:\n\n100% of training data + 20%-35% of test data selected per epoch were used as the new training data. Resnet (mfcc and mel), Senet (mfcc and mel) and VGG (mel and raw) were trained this way.\n\n100% of test data were used to train a “pretrained” model. And then the weights of the “pretrained” model were loaded as initial weights to train the same model with only training data (fine tuning with training data). It turned out that fine tuning with only one epoch was enough to “correct” the overfitted “pretrained” model.\n\nBoth techniques were used with the hope that they could learn/approximate the test distribution better (instead of “overfitting” training data distribution). First one does it on the data level and second one does it on the model level. \n\nThank you Kaggle and Google Brain team for this great competition and congrats to all top teams! This surely wasn't an easy competition and I had quite a lot of fun doing it!\n\n\nUpdate:\nuploaded my best private sub with pred scores (slightly better than my selected one, 0.91084).", "270257": "Thank you for sharing， 感谢舟神分享", "270258": "Thank you for sharing， 感谢舟神分享 +1", "270267": "great work!", "270276": "Hi Xiao Zhou,\n\nSo for the test set pretrained model, you used the training data to train a model, and with the test set results, you used that to retrain a model and then use the trained weights as the initialization weights to retrain the training data?\n\nDoes that help with the training model to converge faster or what is the point of that?\n\nThanks.\n\nBest.", "270279": "sorry if I was not clear. So say, you have a model that scores 0.89 on LB.\n\nYou take the submission of that model, and use that as the labels for test data. Now this test data with 0.89 accurate labels become the new training data for the pretrained model. You train your pretrained model with only test data.\n\nAnd then you use the weights from the pretrained model to initialize one same model, and now, instead of training on test data, you train on training data. and then do prediction with this fine tuned model for test data.\n\nHope it is clearer now.", "270283": "Thank you for sharing， 感谢舟神分享+2", "270286": "Congratulation and Thanks for your great sharing. Would you please explain a little more on the ensemble weight? Specifically, how did you come out 0.75, 0.25 and 0.4 (vgg1d_raw.as_matrix() * 0.75 + vgg1d_mel.as_matrix() * 0.25) * 0.4? I understand the formula is base on the public LB, just curious the detail steps.", "270296": "sure. I just look at the correlation of the submission probabilities of each class, and then use the difference of the model/inputs as well as model score on public LB and then decide a weight and then submit, if it is better than any of the single model score, I just use that. I might try 2 - 3 times and then settle with the weights.", "270298": "Thank you for sharing， 感谢舟神分享+3", "270300": "Thanks mate o/", "270306": "Thank you for sharing， 感谢舟神分享", "270332": "Thanks for sharing! Semi supervised learning works! \n\nDo you mind posting your submission CSV file along with the raw probability values here?\nI can make a comparison between your results and mine. I posted mine at: https://www.kaggle.com/c/tensorflow-speech-recognition-challenge/discussion/47728", "270337": "For sure but somehow it seems there is a limit for uploading files. So I could only attach one with predicted probabilities. The one attached has slightly better score than my selected one. (You can find the file in the main post)\n\n\nAlso congrats!", "270526": "Great job Little Boat and thanks so much for sharing. Brilliant use of semi-supervised and congrats on a top finish!", "270779": "给舟神疯狂打call ：D", "271364": "Congratulations !\n\nDo you think you could turn your prediction .csv files into Kaggle datasets (so that we can use it from kernels) ?", "302859": "First time seeing someone pretrain a model using semi-supervised learning on the full test set, thanks for sharing!\n\nDid you see significant improvement over supervised learning without pretraining ?", "3000208": "Thanks for the idea. I used it for other ASR competition, it worked very well !" }, "source": "meta" }