Hi! What could be the reason why lwlrap too low on valid set? \nLwLRAP 0.0708 for valid set, but LB score is 0.359.\nI used CNN with sigmoid activation at the last layer and nn.BCELoss\nUsed calculate_overall_lwlrap_sklearn to calc LwLRAP.
", "messages": [ { "id": "519758", "postDate": "04/19/2019 15:43:25", "content": "Hi! What could be the reason why lwlrap too low on valid set? \nLwLRAP 0.0708 for valid set, but LB score is 0.359.\nI used CNN with sigmoid activation at the last layer and nn.BCELoss\nUsed calculate_overall_lwlrap_sklearn to calc LwLRAP.
", "rawMarkdown": "Hi! What could be the reason why lwlrap too low on valid set? \nLwLRAP 0.0708 for valid set, but LB score is 0.359.\nI used CNN with sigmoid activation at the last layer and nn.BCELoss\nUsed calculate_overall_lwlrap_sklearn to calc LwLRAP.", "votes": null }, { "id": "519805", "postDate": "04/19/2019 16:56:36", "content": "a bug in your code?
", "rawMarkdown": "a bug in your code?", "votes": null }, { "id": "520209", "postDate": "04/20/2019 12:26:34", "content": "I meet the same problem too.I think there are some bugs in calculateoveralllwlrap_sklearn implement.
", "rawMarkdown": "I meet the same problem too.I think there are some bugs in calculateoveralllwlrap_sklearn implement.", "votes": null }, { "id": "520866", "postDate": "04/21/2019 21:59:30", "content": "Hi there!
\n\nAre you averaging your lwlrap scores over all your validation samples or batches? The *calculate_overall_lwlrap_sklearn* function computes the lwlrap for all the scores in a batch. In my case, I was wrongly dividing the sum of all the batches lwlrap scores by the total number of samples in my validation, which of course led to a lower result. Could this be happening to you?
\n\nSince my validation set is not too large, I have finally passed the whole validation set as a single batch, and then plugged the predicted and true labels to *calculate_overall_lwlrap_sklearn_*, getting a reasonable score. In case of splitting the validation set in smaller batches, you might be better off using the *lwlrap_accumulator* implementation.
\n\nHope it helps, cheers! :)
", "rawMarkdown": "Hi there! \n\nAre you averaging your lwlrap scores over all your validation samples or batches? The *calculate_overall_lwlrap_sklearn* function computes the lwlrap for all the scores in a batch. In my case, I was wrongly dividing the sum of all the batches lwlrap scores by the total number of samples in my validation, which of course led to a lower result. Could this be happening to you?\n\nSince my validation set is not too large, I have finally passed the whole validation set as a single batch, and then plugged the predicted and true labels to *calculate_overall_lwlrap_sklearn_*, getting a reasonable score. In case of splitting the validation set in smaller batches, you might be better off using the *lwlrap_accumulator* implementation.\n\nHope it helps, cheers! :)", "votes": null }, { "id": "522326", "postDate": "04/24/2019 08:38:40", "content": "There're no bug in code, unfortunetly.\nAs I see in https://www.kaggle.com/mhiro2/simple-2d-cnn-classifier-with-pytorch, there are also low lwlrap on valid set almost all of the training. \nProbaply seems normal to see LwLRAP 0.0708 for my simple 5 layers CNN and 20 epoch on curated dataset....\nBut I cant figure out why I have so high public score....?
", "rawMarkdown": "There're no bug in code, unfortunetly.\nAs I see in https://www.kaggle.com/mhiro2/simple-2d-cnn-classifier-with-pytorch, there are also low lwlrap on valid set almost all of the training. \nProbaply seems normal to see LwLRAP 0.0708 for my simple 5 layers CNN and 20 epoch on curated dataset....\nBut I cant figure out why I have so high public score....?", "votes": null }, { "id": "522527", "postDate": "04/24/2019 15:03:14", "content": "If you look carefully at the kernel that you linked, in the cell after calling 'train_model', the 'result' printed by the training shows a best validation lwlrap of 0.631 after 72 epochs, compared to the publis score of 0.610.
", "rawMarkdown": "If you look carefully at the kernel that you linked, in the cell after calling 'train_model', the 'result' printed by the training shows a best validation lwlrap of 0.631 after 72 epochs, compared to the publis score of 0.610.", "votes": null }, { "id": "527430", "postDate": "05/05/2019 13:28:50", "content": "Hi!
\n\nPersonnaly, I've encountered the same issue (lwlrap = 0.71 after training, and 0.071 on valid set). \nIt appeared that it was a typo in my code. I loaded the wrong file. I wrote:
\n\nX_test = load_pkl('../input/fat2019/fat2019_prep_mels1/mels_trn_noisy_best50s.pkl')\ninstead of\nX_test = load_pkl('../input/fat2019/fat2019_prep_mels1/mels_test.pkl')
I hope it can help somebody. :-)
", "rawMarkdown": "Hi!\n\nPersonnaly, I've encountered the same issue (lwlrap = 0.71 after training, and 0.071 on valid set). \nIt appeared that it was a typo in my code. I loaded the wrong file. I wrote:\n\n`X_test = load_pkl('../input/fat2019/fat2019_prep_mels1/mels_trn_noisy_best50s.pkl')`\ninstead of\n`X_test = load_pkl('../input/fat2019/fat2019_prep_mels1/mels_test.pkl')`\n\nI hope it can help somebody. :-)", "votes": null } ], "comments": [ { "id": 519805, "author_name": "titericz", "author_url": "", "post_date": "04/19/2019 16:56:36", "content": "a bug in your code?
", "votes": null, "replies": [] }, { "id": 520209, "author_name": "gmhost", "author_url": "", "post_date": "04/20/2019 12:26:34", "content": "I meet the same problem too.I think there are some bugs in calculateoveralllwlrap_sklearn implement.
", "votes": null, "replies": [ { "id": 522326, "author_name": "serk00", "author_url": "", "post_date": "04/24/2019 08:38:40", "content": "There're no bug in code, unfortunetly.\nAs I see in https://www.kaggle.com/mhiro2/simple-2d-cnn-classifier-with-pytorch, there are also low lwlrap on valid set almost all of the training. \nProbaply seems normal to see LwLRAP 0.0708 for my simple 5 layers CNN and 20 epoch on curated dataset....\nBut I cant figure out why I have so high public score....?
", "votes": null, "replies": [] }, { "id": 522527, "author_name": "plakal", "author_url": "", "post_date": "04/24/2019 15:03:14", "content": "If you look carefully at the kernel that you linked, in the cell after calling 'train_model', the 'result' printed by the training shows a best validation lwlrap of 0.631 after 72 epochs, compared to the publis score of 0.610.
", "votes": null, "replies": [] } ] }, { "id": 520866, "author_name": "gcambara", "author_url": "", "post_date": "04/21/2019 21:59:30", "content": "Hi there!
\n\nAre you averaging your lwlrap scores over all your validation samples or batches? The *calculate_overall_lwlrap_sklearn* function computes the lwlrap for all the scores in a batch. In my case, I was wrongly dividing the sum of all the batches lwlrap scores by the total number of samples in my validation, which of course led to a lower result. Could this be happening to you?
\n\nSince my validation set is not too large, I have finally passed the whole validation set as a single batch, and then plugged the predicted and true labels to *calculate_overall_lwlrap_sklearn_*, getting a reasonable score. In case of splitting the validation set in smaller batches, you might be better off using the *lwlrap_accumulator* implementation.
\n\nHope it helps, cheers! :)
", "votes": null, "replies": [] }, { "id": 527430, "author_name": "toldo171", "author_url": "", "post_date": "05/05/2019 13:28:50", "content": "Hi!
\n\nPersonnaly, I've encountered the same issue (lwlrap = 0.71 after training, and 0.071 on valid set). \nIt appeared that it was a typo in my code. I loaded the wrong file. I wrote:
\n\nX_test = load_pkl('../input/fat2019/fat2019_prep_mels1/mels_trn_noisy_best50s.pkl')\ninstead of\nX_test = load_pkl('../input/fat2019/fat2019_prep_mels1/mels_test.pkl')
I hope it can help somebody. :-)
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "519758": "Hi! What could be the reason why lwlrap too low on valid set? \nLwLRAP 0.0708 for valid set, but LB score is 0.359.\nI used CNN with sigmoid activation at the last layer and nn.BCELoss\nUsed calculate_overall_lwlrap_sklearn to calc LwLRAP.", "519805": "a bug in your code?", "520209": "I meet the same problem too.I think there are some bugs in calculateoveralllwlrap_sklearn implement.", "520866": "Hi there! \n\nAre you averaging your lwlrap scores over all your validation samples or batches? The *calculate_overall_lwlrap_sklearn* function computes the lwlrap for all the scores in a batch. In my case, I was wrongly dividing the sum of all the batches lwlrap scores by the total number of samples in my validation, which of course led to a lower result. Could this be happening to you?\n\nSince my validation set is not too large, I have finally passed the whole validation set as a single batch, and then plugged the predicted and true labels to *calculate_overall_lwlrap_sklearn_*, getting a reasonable score. In case of splitting the validation set in smaller batches, you might be better off using the *lwlrap_accumulator* implementation.\n\nHope it helps, cheers! :)", "522326": "There're no bug in code, unfortunetly.\nAs I see in https://www.kaggle.com/mhiro2/simple-2d-cnn-classifier-with-pytorch, there are also low lwlrap on valid set almost all of the training. \nProbaply seems normal to see LwLRAP 0.0708 for my simple 5 layers CNN and 20 epoch on curated dataset....\nBut I cant figure out why I have so high public score....?", "522527": "If you look carefully at the kernel that you linked, in the cell after calling 'train_model', the 'result' printed by the training shows a best validation lwlrap of 0.631 after 72 epochs, compared to the publis score of 0.610.", "527430": "Hi!\n\nPersonnaly, I've encountered the same issue (lwlrap = 0.71 after training, and 0.071 on valid set). \nIt appeared that it was a typo in my code. I loaded the wrong file. I wrote:\n\n`X_test = load_pkl('../input/fat2019/fat2019_prep_mels1/mels_trn_noisy_best50s.pkl')`\ninstead of\n`X_test = load_pkl('../input/fat2019/fat2019_prep_mels1/mels_test.pkl')`\n\nI hope it can help somebody. :-)" }, "source": "meta" }