I have recently started out on using PANN architecture for this competition and have followed closely on the discussions and notebooks relating to PANNs. I would like to thank the community for sharing their knowledge and work! However, i have been trying to wrap my head around the issue of a huge deviation between my Validation & LB score, and in particular, i am not very sure of the inference process. I have used 10s crop on the test set by doing the following:
\n1. test_batch = [y[i:i+effective_length].astype(np.float32) for i in range(i, 60*sr, effective_length)]
\n 2. test_batch = torch.FloatTensor(np.stack(test_batch))
\nwhich generates 6 consecutive 10s crop for each test example with shape (6,480000). I set batch_size = 1, thus, resulting in model output of (6xTx24) for each test example. By using max(output, dim=T), i get the max probabilities for each 'crop' of the test example => (6,24). It would then make sense for me to sum them up along the 0th axis to get the final output for 1 single test example.
framewise_output for my validation metric, as suggested in this post, and trained with weak labels. I don't seem to be able to narrow the gap between my validation and LB (difference of ~0.12)I have recently started out on using PANN architecture for this competition and have followed closely on the discussions and notebooks relating to PANNs. I would like to thank the community for sharing their knowledge and work! However, i have been trying to wrap my head around the issue of a huge deviation between my Validation & LB score, and in particular, i am not very sure of the inference process. I have used 10s crop on the test set by doing the following:
\n1. test_batch = [y[i:i+effective_length].astype(np.float32) for i in range(i, 60*sr, effective_length)]
\n 2. test_batch = torch.FloatTensor(np.stack(test_batch))
\nwhich generates 6 consecutive 10s crop for each test example with shape (6,480000). I set batch_size = 1, thus, resulting in model output of (6xTx24) for each test example. By using max(output, dim=T), i get the max probabilities for each 'crop' of the test example => (6,24). It would then make sense for me to sum them up along the 0th axis to get the final output for 1 single test example.
framewise_output for my validation metric, as suggested in this post, and trained with weak labels. I don't seem to be able to narrow the gap between my validation and LB (difference of ~0.12)IMHO ~0.12 diffierence between CV and LB is quite normal, as the train dataset is small. Your inference crop is fine as most of discussion are using the same way, and as many discussion indicated that the key for this competition is how to crop the training data. If you generate training data smarter, and use more folds or models to ensemble your prediction, you may see stable CV and LB correlation.
", "rawMarkdown": "IMHO ~0.12 diffierence between CV and LB is quite normal, as the train dataset is small. Your inference crop is fine as most of discussion are using the same way, and as many discussion indicated that the key for this competition is how to crop the training data. If you generate training data smarter, and use more folds or models to ensemble your prediction, you may see stable CV and LB correlation.", "votes": null }, { "id": "1183762", "postDate": "02/03/2021 07:51:26", "content": "\n\nhuge deviation between my Validation & LB score
\n
This is not necessarily an issue as long as they are correlated. If your LB increases when you CV increases then the gap between the two isn't relevant.
\nIn this competition it is hard to get a great CV setting because we don't have ground truth for training data. Indeed, we don't have recoding level info on the presence of all species.
\nTL;DR Dont' worry about CV LB gap as long as you have some correlation between the two.
", "rawMarkdown": "> huge deviation between my Validation & LB score\n\nThis is not necessarily an issue as long as they are correlated. If your LB increases when you CV increases then the gap between the two isn't relevant.\n\nIn this competition it is hard to get a great CV setting because we don't have ground truth for training data. Indeed, we don't have recoding level info on the presence of all species. \n\nTL;DR Dont' worry about CV LB gap as long as you have some correlation between the two.", "votes": null }, { "id": "1183776", "postDate": "02/03/2021 07:58:56", "content": "Thank you for the advice! I guess i will have to change my approach :)
", "rawMarkdown": "Thank you for the advice! I guess i will have to change my approach :)", "votes": null }, { "id": "1183779", "postDate": "02/03/2021 08:00:35", "content": "Thank you for the explanation! :) With this i will not have to spend unnecessary amount of time pondering about it and focus on my approach instead!
", "rawMarkdown": "Thank you for the explanation! :) With this i will not have to spend unnecessary amount of time pondering about it and focus on my approach instead!", "votes": null } ], "comments": [ { "id": 1183467, "author_name": "superchenhao", "author_url": "", "post_date": "02/03/2021 02:58:04", "content": "IMHO ~0.12 diffierence between CV and LB is quite normal, as the train dataset is small. Your inference crop is fine as most of discussion are using the same way, and as many discussion indicated that the key for this competition is how to crop the training data. If you generate training data smarter, and use more folds or models to ensemble your prediction, you may see stable CV and LB correlation.
", "votes": null, "replies": [ { "id": 1183776, "author_name": "angqx95", "author_url": "", "post_date": "02/03/2021 07:58:56", "content": "Thank you for the advice! I guess i will have to change my approach :)
", "votes": null, "replies": [] } ] }, { "id": 1183762, "author_name": "cpmpml", "author_url": "", "post_date": "02/03/2021 07:51:26", "content": "\n\nhuge deviation between my Validation & LB score
\n
This is not necessarily an issue as long as they are correlated. If your LB increases when you CV increases then the gap between the two isn't relevant.
\nIn this competition it is hard to get a great CV setting because we don't have ground truth for training data. Indeed, we don't have recoding level info on the presence of all species.
\nTL;DR Dont' worry about CV LB gap as long as you have some correlation between the two.
", "votes": null, "replies": [ { "id": 1183779, "author_name": "angqx95", "author_url": "", "post_date": "02/03/2021 08:00:35", "content": "Thank you for the explanation! :) With this i will not have to spend unnecessary amount of time pondering about it and focus on my approach instead!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1164851": "I have recently started out on using PANN architecture for this competition and have followed closely on the discussions and notebooks relating to PANNs. I would like to thank the community for sharing their knowledge and work! However, i have been trying to wrap my head around the issue of a huge deviation between my Validation & LB score, and in particular, i am not very sure of the inference process. I have used 10s crop on the test set by doing the following:\n`1. test_batch = [y[i:i+effective_length].astype(np.float32) for i in range(i, 60*sr, effective_length)] ` \n `2. test_batch = torch.FloatTensor(np.stack(test_batch))`\nwhich generates 6 consecutive 10s crop for each test example with shape *(6,480000)*. I set *batch_size = 1*, thus, resulting in model output of *(6xTx24)* for each test example. By using *max(output, dim=T)*, i get the max probabilities for each 'crop' of the test example => *(6,24)*. It would then make sense for me to sum them up along the 0th axis to get the final output for 1 single test example. \n1. Is this way of cropping the right way to get prediction for my test? \n2. I am using the `framewise_output` for my validation metric, as suggested in [this post](https://www.kaggle.com/c/rfcx-species-audio-detection/discussion/209684), and trained with weak labels. I don't seem to be able to narrow the gap between my validation and LB (difference of ~0.12)\nI would really appreciate any advice on this:)", "1183467": "IMHO ~0.12 diffierence between CV and LB is quite normal, as the train dataset is small. Your inference crop is fine as most of discussion are using the same way, and as many discussion indicated that the key for this competition is how to crop the training data. If you generate training data smarter, and use more folds or models to ensemble your prediction, you may see stable CV and LB correlation.", "1183762": "> huge deviation between my Validation & LB score\n\nThis is not necessarily an issue as long as they are correlated. If your LB increases when you CV increases then the gap between the two isn't relevant.\n\nIn this competition it is hard to get a great CV setting because we don't have ground truth for training data. Indeed, we don't have recoding level info on the presence of all species. \n\nTL;DR Dont' worry about CV LB gap as long as you have some correlation between the two.", "1183776": "Thank you for the advice! I guess i will have to change my approach :)", "1183779": "Thank you for the explanation! :) With this i will not have to spend unnecessary amount of time pondering about it and focus on my approach instead!" }, "source": "meta" }