![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1483555%2F5dead633baa5d42311d1b982e18fcc40%2F2020-08-07%209.53.35.png?generation=1596761781428144&alt=media\")
I saw this discussion, and I try to remove noise by using Sound Envelope.
\n\nThe result of denoise looks good, but score is bad.(fold-0: 0.6319398546 / LB: 0.543.)
\n\nI think the reason is that I apply denoise to only training.\nHowever, if I apply denoise in test, submission send \"Notebook Exceeded Allowed Compute\"
\n\n
Are you have idea to apply denoise in test time?
\n\nThank you.
", "messages": [ { "id": "961168", "postDate": "08/07/2020 00:59:08", "content": "I saw this discussion, and I try to remove noise by using Sound Envelope.
\n\nThe result of denoise looks good, but score is bad.(fold-0: 0.6319398546 / LB: 0.543.)
\n\nI think the reason is that I apply denoise to only training.\nHowever, if I apply denoise in test, submission send \"Notebook Exceeded Allowed Compute\"
\n\n
Are you have idea to apply denoise in test time?
\n\nThank you.
", "rawMarkdown": "I saw [this discussion](https://www.kaggle.com/c/birdsong-recognition/discussion/169582#946072), and I try to remove noise by using Sound Envelope.\n\n- nontebook: https://www.kaggle.com/takamichitoda/birdcall-noise-reduction\n- dataset: https://www.kaggle.com/takamichitoda/birdcall-spectrogram-images\n\nThe result of denoise looks good, but score is bad.(fold-0: 0.6319398546 / LB: 0.543.)\n\nI think the reason is that I apply denoise to only training.\nHowever, if I apply denoise in test, submission send \"Notebook Exceeded Allowed Compute\"\n\n\n\nAre you have idea to apply denoise in test time?\n\nThank you.", "votes": null }, { "id": "961410", "postDate": "08/07/2020 07:08:22", "content": "In test time do you apply denoise for each row id or for the audio id only one time for site 1 and 2? These should have row ids for each 5 seconds so one audio can have many rows. Site 3 should be just one audio per row but longer so you probably break it down yourself.
\nFrom the data page the hidden test set should be 150 clips of 10 minutes. Maybe checking times for denoise on the example_test_audio can help you determine how long they take, how you can speed things up, etc.
Thank you comment.
\n\nI apply denoise for each row id.
\n\nI will try to apply denoise to example test audio, and check processing time.
", "rawMarkdown": "Thank you comment.\n\nI apply denoise for each row id.\n\nI will try to apply denoise to example test audio, and check processing time.", "votes": null }, { "id": "961538", "postDate": "08/07/2020 09:15:29", "content": "Note that Notebook Exceeded Allowed Compute happens when your notebook use up all the memory. For running time exceeding the limit, you will get Notebook Timeout, so this is not about the running time.
Oh, I didn't know that. Thank you tell me.\nSo, should I give up apply to denoise processing in test data?
", "rawMarkdown": "Oh, I didn't know that. Thank you tell me.\nSo, should I give up apply to denoise processing in test data?", "votes": null }, { "id": "961575", "postDate": "08/07/2020 09:58:41", "content": "Similar problem is reported here \nhttps://www.kaggle.com/c/birdsong-recognition/discussion/172346
\n\nIt seems the reporter also got memory error when applying denoising method and cause may be in pandas.DataFrame.roll. Did you get that error using the original denoising method introduced in https://www.kaggle.com/c/birdsong-recognition/discussion/169582#946072 or did you get that using noisereduce?
\n\n\nSo, should I give up apply to denoise processing in test data?
\n
I'm also interested in denoising as a preprocess, so let me help on this problem.
", "rawMarkdown": "Similar problem is reported here \nhttps://www.kaggle.com/c/birdsong-recognition/discussion/172346\n\nIt seems the reporter also got memory error when applying denoising method and cause may be in `pandas.DataFrame.roll`. Did you get that error using the original denoising method introduced in https://www.kaggle.com/c/birdsong-recognition/discussion/169582#946072 or did you get that using `noisereduce`?\n\n> So, should I give up apply to denoise processing in test data?\n\nI'm also interested in denoising as a preprocess, so let me help on this problem.", "votes": null }, { "id": "962037", "postDate": "08/07/2020 18:31:52", "content": "Kaggle seems to have pandas 1.0.3 in the notebook. In this version, pandas.rolling.max() has a memory leak issue and will cause your notebook to run out of memory. More details here: https://github.com/pandas-dev/pandas/issues/32266\nI suggest either install the most recent version of pandas or use scipy.ndimage.maximum_filter1d instead.
", "rawMarkdown": "Kaggle seems to have pandas 1.0.3 in the notebook. In this version, pandas.rolling.max() has a memory leak issue and will cause your notebook to run out of memory. More details here: https://github.com/pandas-dev/pandas/issues/32266\nI suggest either install the most recent version of pandas or use scipy.ndimage.maximum_filter1d instead.", "votes": null }, { "id": "966173", "postDate": "08/11/2020 08:04:37", "content": "Thank you share important information.
\n\nI have tried it and got same denoise result.\nhttps://www.kaggle.com/takamichitoda/birdcall-noise-reduction?scriptVersionId=40539957
\n\nI will try submission.
", "rawMarkdown": "Thank you share important information.\n\nI have tried it and got same denoise result.\nhttps://www.kaggle.com/takamichitoda/birdcall-noise-reduction?scriptVersionId=40539957\n\nI will try submission.", "votes": null }, { "id": "966356", "postDate": "08/11/2020 11:09:13", "content": "Thank you very much.
\n\nI used scipy.ndimage.maximum_filter1d instead of pandas.DataFrame.roll to denoise.\nNotebook Exceeded Allowed Compute has been resolved, but Notebook Timeout Error has occurred.
\n\nI continue trying other denoise method and sharing knowledge.
", "rawMarkdown": "Thank you very much.\n\nI used scipy.ndimage.maximum_filter1d instead of pandas.DataFrame.roll to denoise.\nNotebook Exceeded Allowed Compute has been resolved, but Notebook Timeout Error has occurred.\n\nI continue trying other denoise method and sharing knowledge.", "votes": null } ], "comments": [ { "id": 961410, "author_name": "something4kag", "author_url": "", "post_date": "08/07/2020 07:08:22", "content": "In test time do you apply denoise for each row id or for the audio id only one time for site 1 and 2? These should have row ids for each 5 seconds so one audio can have many rows. Site 3 should be just one audio per row but longer so you probably break it down yourself.
\nFrom the data page the hidden test set should be 150 clips of 10 minutes. Maybe checking times for denoise on the example_test_audio can help you determine how long they take, how you can speed things up, etc.
Thank you comment.
\n\nI apply denoise for each row id.
\n\nI will try to apply denoise to example test audio, and check processing time.
", "votes": null, "replies": [] }, { "id": 961538, "author_name": "hidehisaarai1213", "author_url": "", "post_date": "08/07/2020 09:15:29", "content": "Note that Notebook Exceeded Allowed Compute happens when your notebook use up all the memory. For running time exceeding the limit, you will get Notebook Timeout, so this is not about the running time.
Oh, I didn't know that. Thank you tell me.\nSo, should I give up apply to denoise processing in test data?
", "votes": null, "replies": [] }, { "id": 961575, "author_name": "hidehisaarai1213", "author_url": "", "post_date": "08/07/2020 09:58:41", "content": "Similar problem is reported here \nhttps://www.kaggle.com/c/birdsong-recognition/discussion/172346
\n\nIt seems the reporter also got memory error when applying denoising method and cause may be in pandas.DataFrame.roll. Did you get that error using the original denoising method introduced in https://www.kaggle.com/c/birdsong-recognition/discussion/169582#946072 or did you get that using noisereduce?
\n\n\nSo, should I give up apply to denoise processing in test data?
\n
I'm also interested in denoising as a preprocess, so let me help on this problem.
", "votes": null, "replies": [] }, { "id": 966356, "author_name": "takamichitoda", "author_url": "", "post_date": "08/11/2020 11:09:13", "content": "Thank you very much.
\n\nI used scipy.ndimage.maximum_filter1d instead of pandas.DataFrame.roll to denoise.\nNotebook Exceeded Allowed Compute has been resolved, but Notebook Timeout Error has occurred.
\n\nI continue trying other denoise method and sharing knowledge.
", "votes": null, "replies": [] } ] }, { "id": 962037, "author_name": "qiuosier", "author_url": "", "post_date": "08/07/2020 18:31:52", "content": "Kaggle seems to have pandas 1.0.3 in the notebook. In this version, pandas.rolling.max() has a memory leak issue and will cause your notebook to run out of memory. More details here: https://github.com/pandas-dev/pandas/issues/32266\nI suggest either install the most recent version of pandas or use scipy.ndimage.maximum_filter1d instead.
", "votes": null, "replies": [ { "id": 966173, "author_name": "takamichitoda", "author_url": "", "post_date": "08/11/2020 08:04:37", "content": "Thank you share important information.
\n\nI have tried it and got same denoise result.\nhttps://www.kaggle.com/takamichitoda/birdcall-noise-reduction?scriptVersionId=40539957
\n\nI will try submission.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "961168": "I saw [this discussion](https://www.kaggle.com/c/birdsong-recognition/discussion/169582#946072), and I try to remove noise by using Sound Envelope.\n\n- nontebook: https://www.kaggle.com/takamichitoda/birdcall-noise-reduction\n- dataset: https://www.kaggle.com/takamichitoda/birdcall-spectrogram-images\n\nThe result of denoise looks good, but score is bad.(fold-0: 0.6319398546 / LB: 0.543.)\n\nI think the reason is that I apply denoise to only training.\nHowever, if I apply denoise in test, submission send \"Notebook Exceeded Allowed Compute\"\n\n\n\nAre you have idea to apply denoise in test time?\n\nThank you.", "961410": "In test time do you apply denoise for each row id or for the audio id only one time for site 1 and 2? These should have row ids for each 5 seconds so one audio can have many rows. Site 3 should be just one audio per row but longer so you probably break it down yourself. \nFrom the data page the hidden test set should be 150 clips of 10 minutes. Maybe checking times for denoise on the example_test_audio can help you determine how long they take, how you can speed things up, etc.", "961536": "Thank you comment.\n\nI apply denoise for each row id.\n\nI will try to apply denoise to example test audio, and check processing time.", "961538": "Note that `Notebook Exceeded Allowed Compute` happens when your notebook use up all the memory. For running time exceeding the limit, you will get `Notebook Timeout`, so this is not about the running time.", "961572": "Oh, I didn't know that. Thank you tell me.\nSo, should I give up apply to denoise processing in test data?", "961575": "Similar problem is reported here \nhttps://www.kaggle.com/c/birdsong-recognition/discussion/172346\n\nIt seems the reporter also got memory error when applying denoising method and cause may be in `pandas.DataFrame.roll`. Did you get that error using the original denoising method introduced in https://www.kaggle.com/c/birdsong-recognition/discussion/169582#946072 or did you get that using `noisereduce`?\n\n> So, should I give up apply to denoise processing in test data?\n\nI'm also interested in denoising as a preprocess, so let me help on this problem.", "962037": "Kaggle seems to have pandas 1.0.3 in the notebook. In this version, pandas.rolling.max() has a memory leak issue and will cause your notebook to run out of memory. More details here: https://github.com/pandas-dev/pandas/issues/32266\nI suggest either install the most recent version of pandas or use scipy.ndimage.maximum_filter1d instead.", "966173": "Thank you share important information.\n\nI have tried it and got same denoise result.\nhttps://www.kaggle.com/takamichitoda/birdcall-noise-reduction?scriptVersionId=40539957\n\nI will try submission.", "966356": "Thank you very much.\n\nI used scipy.ndimage.maximum_filter1d instead of pandas.DataFrame.roll to denoise.\nNotebook Exceeded Allowed Compute has been resolved, but Notebook Timeout Error has occurred.\n\nI continue trying other denoise method and sharing knowledge." }, "source": "meta" }