This is my implementation of @cdeotte EfficientNetB0 Starter - [LB 0.43]. I tried to resemble it as closely as possible.
\nYou can find the code here:
\n\nThis notebook has approximately same CV and public LB scores as the version of Chirs. It achieves 0.60 in CV score and 0.44 score in public LB.
\nHope you like it 👍🏼
", "messages": [ { "id": "2627656", "postDate": "01/30/2024 19:46:58", "content": "This is my implementation of @cdeotte EfficientNetB0 Starter - [LB 0.43]. I tried to resemble it as closely as possible.
\nYou can find the code here:
\n\nThis notebook has approximately same CV and public LB scores as the version of Chirs. It achieves 0.60 in CV score and 0.44 score in public LB.
\nHope you like it 👍🏼
", "rawMarkdown": "This is my implementation of @cdeotte [EfficientNetB0 Starter - [LB 0.43]](https://www.kaggle.com/code/cdeotte/efficientnetb0-starter-lb-0-43). I tried to resemble it as closely as possible. \n\nYou can find the code here:\n- [HMS | EfficientNetB0 PyTorch [Train]](https://www.kaggle.com/code/alejopaullier/hms-efficientnetb0-pytorch-train)\n- [HMS | EfficientNetB0 PyTorch [Inference]](https://www.kaggle.com/code/alejopaullier/hms-efficientnetb0-pytorch-inference)\n\nThis notebook has approximately same CV and public LB scores as the version of Chirs. It achieves 0.60 in CV score and 0.44 score in public LB.\n\nHope you like it 👍🏼", "votes": null }, { "id": "2628016", "postDate": "01/31/2024 03:47:33", "content": "great work! pytorch is where I know my stuff
", "rawMarkdown": "great work! pytorch is where I know my stuff", "votes": null }, { "id": "2628324", "postDate": "01/31/2024 08:21:28", "content": "Great work!!!!!!!
", "rawMarkdown": "Great work!!!!!!!", "votes": null }, { "id": "2629615", "postDate": "01/31/2024 22:41:24", "content": "How many Epoch to tune the model?
", "rawMarkdown": "How many Epoch to tune the model?", "votes": null }, { "id": "2631020", "postDate": "02/01/2024 15:16:34", "content": "I used 4 epochs. You can check it in the config class under Configuration section
Good post! I noticed the spectrograms looked different from Chris's. Did you modify them for plotting? Thanks for the well-organized notebooks.
", "rawMarkdown": "Good post! I noticed the spectrograms looked different from Chris's. Did you modify them for plotting? Thanks for the well-organized notebooks.", "votes": null }, { "id": "2631237", "postDate": "02/01/2024 17:14:02", "content": "Somehow I am struggling with reproducing the result of this notebook . I trained locally the same notebook and I got a CV of around .640 and somehow the inference didnt work (May be some mistake , which I cant figure) LB was around .61 , it should be around .4xx or max early .5xx . Anyone else faced the same issue ?
\n@alejopaullier should I run the training notebook as is to reproduce the result or should I need to change something ?
", "rawMarkdown": "Somehow I am struggling with reproducing the result of this notebook . I trained locally the same notebook and I got a CV of around .640 and somehow the inference didnt work (May be some mistake , which I cant figure) LB was around .61 , it should be around .4xx or max early .5xx . Anyone else faced the same issue ?\n\n@alejopaullier should I run the training notebook as is to reproduce the result or should I need to change something ?", "votes": null }, { "id": "2631263", "postDate": "02/01/2024 17:22:00", "content": "@phoenix9032 have you ran it without any change?
", "rawMarkdown": "phoenix9032 have you ran it without any change?", "votes": null }, { "id": "2631329", "postDate": "02/01/2024 17:55:53", "content": "yep , no change except for pointing it to the spectrograms path in my local . I have downloaded the chris created spectrograms fresh as well just to be sure ..
", "rawMarkdown": "yep , no change except for pointing it to the spectrograms path in my local . I have downloaded the chris created spectrograms fresh as well just to be sure ..", "votes": null }, { "id": "2631935", "postDate": "02/02/2024 03:38:51", "content": "I think somehow I messed up the spectrograms or there were some differences in the way my spectrograms and Chris's one was generated . I carefully re-downloaded the latest version of spectrogram and got CV around .604 ,this amount of deviation is understandable…
", "rawMarkdown": "I think somehow I messed up the spectrograms or there were some differences in the way my spectrograms and Chris's one was generated . I carefully re-downloaded the latest version of spectrogram and got CV around .604 ,this amount of deviation is understandable...", "votes": null }, { "id": "2632020", "postDate": "02/02/2024 05:05:28", "content": "I reproduced the code with LB around 0.44
", "rawMarkdown": "I reproduced the code with LB around 0.44", "votes": null }, { "id": "2632396", "postDate": "02/02/2024 10:35:46", "content": "FYI.For the locally trained model, the LB was .45, so it seems that there is not much difference.
\nRegarding CV, it is in the dataset log file, and there seems to be no big difference.
\n \n \n \n \n \n \n \n \n \n \n\n\nTorch：1.12.1\nCUDA：10.02\nGPU ：V100(Train Batch size has not changed)\nHello my friend @hideyukizushi … Thanks . I am now able to replicate it too..LB was .45 for me and CV .601
", "rawMarkdown": "Hello my friend @hideyukizushi ... Thanks . I am now able to replicate it too..LB was .45 for me and CV .601", "votes": null }, { "id": "2632764", "postDate": "02/02/2024 16:03:26", "content": "Gotcha! good luck!
", "rawMarkdown": "Gotcha! good luck!", "votes": null }, { "id": "2634879", "postDate": "02/04/2024 04:37:51", "content": "This notebook might be useful for me. Thank you so much.
\nNow I have one question.
When I try to use this notebook, I get result below.
\n========== Fold: training ==========\n.\n.\n.\nEpoch - avg_train_loss: avg_val_loss: time: 214s\nEpoch - Save Best Loss: Model\n========== Fold result: ==========\n========== CV: ========\nMy question is why the scores of Fold X result & CV is too higher than avg_train_loss ?
\nThe Fold 4 result & CV is calculated by get_result function.
\nAre there any problem?
In addition,
\nI try to use
sys\nsys.path.append()\n kaggle_kl_div score\n\noof = pd.DataFrame(oof_df[target_preds].copy())\noof.columns = label_cols\noof[] = np.arange((oof))\n\ntrue = pd.DataFrame(oof_df[label_cols].copy())\ntrue[] = np.arange((true))\n\ncv = score(solution=true, submission=oof, row_id_column_name=)\n(,cv)\nThe result is below. It seems no problem.
\nCV Score KL-Div for EfficientNetB2 = 0.6780376070130062
I changed the get_result function similar to the above and it works .. that way ..
", "rawMarkdown": "I changed the get_result function similar to the above and it works .. that way ..", "votes": null }, { "id": "2635353", "postDate": "02/04/2024 10:49:11", "content": "really helpful
", "rawMarkdown": "really helpful", "votes": null }, { "id": "2641501", "postDate": "02/07/2024 14:20:49", "content": "Hi @alejopaullier! I thought of the same and tried implementing Chris's notebook in pytorch. However I couldn't get the CV score to go below 1.4 no matter what I try. I tried to replicate his result but I don't know whats's going wrong. Could you please take a look at it and point me where I'm wrong. Thanks!
\nHere's the notebook : HMS-Starter-Pytorch (Train)
", "rawMarkdown": "Hi @alejopaullier! I thought of the same and tried implementing Chris's notebook in pytorch. However I couldn't get the CV score to go below 1.4 no matter what I try. I tried to replicate his result but I don't know whats's going wrong. Could you please take a look at it and point me where I'm wrong. Thanks!\n\nHere's the notebook : [HMS-Starter-Pytorch (Train)](https://www.kaggle.com/code/sunny7712/hms-starter-pytorch-train)", "votes": null }, { "id": "2641509", "postDate": "02/07/2024 14:25:56", "content": "Your model has a softmax layer at the end, while mine doesn't. Check that you are correctly passing the predictions to the criterion in the same way as I did. Also, I use a different scheduler which is kind of similar to Chris'. Try debugging the code piece by piece. @sunny7712
", "rawMarkdown": "Your model has a softmax layer at the end, while mine doesn't. Check that you are correctly passing the predictions to the criterion in the same way as I did. Also, I use a different scheduler which is kind of similar to Chris'. Try debugging the code piece by piece. @sunny7712", "votes": null }, { "id": "2641729", "postDate": "02/07/2024 16:29:55", "content": "I tried the same thing but apparently it is taking too much time to train like on gpu it is taking 10 minutes per iteration for resnet50
\nis it normal
I also failed to achieve the same result as Chris's TensorFlow version using the same df splits.
\n: CV Score KL-Div for EfficientNetB2 = .\n: CV Score KL-Div for EfficientNetB2 = .\nThanks for nice notebook.
\nWhy do you use following scheduler ?
\nIs it normal when training pre-trained model ?
torch.optim.lr_scheduler OneCycleLR\nAs in Chris` notebook, I used following scheduler and scheduler.step in every epoch but got worse result, CV:0.65.
\n torch.optim.lr_scheduler LambdaLR\n\nLR_START = \nLR_MAX = \nLR_RAMPUP_EPOCHS = \nLR_SUSTAIN_EPOCHS = \nLR_STEP_DECAY = \nEVERY = \n\n ():\n epoch < LR_RAMPUP_EPOCHS:\n lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n lr = LR_MAX\n :\n lr = LR_MAX * LR_STEP_DECAY**((epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS)//EVERY)\n lr\n\n scheduler = LambdaLR(optimizer, lr_lambda=lrfn)\nI changed optimizer and scheduler like chris notebook that are difference between moth and chris, but got worse result, CV:0.649
\noptimizer = torch.optim.Adam(model.parameters(), lr=)\n\n torch.optim.lr_scheduler LambdaLR\nLR_START = \nLR_MAX = \nLR_RAMPUP_EPOCHS = \nLR_SUSTAIN_EPOCHS = \nLR_STEP_DECAY = \nEVERY = \n\n ():\n epoch < LR_RAMPUP_EPOCHS:\n lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n lr = LR_MAX\n :\n lr = LR_MAX * LR_STEP_DECAY**((epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS)//EVERY)\n lr\n\n scheduler = LambdaLR(optimizer, lr_lambda=lrfn)\nGreat job! Upvoted!
\nI like Pytorch, so using this source as a baseline, I was able to get very good results with a single model.
In my case, I got CV0.60 and LB0.44 using this notebook without any change.
\nDo you use some training trick like 2 stage learning ?
\nhttps://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477461
\n\nDo you use some training trick like 2 stage learning ?
\n
No, I haven't used it.No major changes were made. As a pipeline, I use a little trick with CustomModel and CustomDataset.
\nLike the link you wrote.
\nI haven't worked on it yet, but I think it's necessary to deal with the problem of large dispersion occurring on a fold-by-fold basis.
Just Started Pytorch a month a ago really liked your notebook
", "rawMarkdown": "Just Started Pytorch a month a ago really liked your notebook", "votes": null } ], "comments": [ { "id": 2628016, "author_name": "chenboluo", "author_url": "", "post_date": "01/31/2024 03:47:33", "content": "great work! pytorch is where I know my stuff
", "votes": null, "replies": [] }, { "id": 2628324, "author_name": "saurabh2019", "author_url": "", "post_date": "01/31/2024 08:21:28", "content": "Great work!!!!!!!
", "votes": null, "replies": [] }, { "id": 2629615, "author_name": "odigallm", "author_url": "", "post_date": "01/31/2024 22:41:24", "content": "How many Epoch to tune the model?
", "votes": null, "replies": [ { "id": 2631020, "author_name": "alejopaullier", "author_url": "", "post_date": "02/01/2024 15:16:34", "content": "I used 4 epochs. You can check it in the config class under Configuration section
Good post! I noticed the spectrograms looked different from Chris's. Did you modify them for plotting? Thanks for the well-organized notebooks.
", "votes": null, "replies": [] }, { "id": 2631237, "author_name": "phoenix9032", "author_url": "", "post_date": "02/01/2024 17:14:02", "content": "Somehow I am struggling with reproducing the result of this notebook . I trained locally the same notebook and I got a CV of around .640 and somehow the inference didnt work (May be some mistake , which I cant figure) LB was around .61 , it should be around .4xx or max early .5xx . Anyone else faced the same issue ?
\n@alejopaullier should I run the training notebook as is to reproduce the result or should I need to change something ?
", "votes": null, "replies": [ { "id": 2631263, "author_name": "alejopaullier", "author_url": "", "post_date": "02/01/2024 17:22:00", "content": "@phoenix9032 have you ran it without any change?
", "votes": null, "replies": [ { "id": 2631329, "author_name": "phoenix9032", "author_url": "", "post_date": "02/01/2024 17:55:53", "content": "yep , no change except for pointing it to the spectrograms path in my local . I have downloaded the chris created spectrograms fresh as well just to be sure ..
", "votes": null, "replies": [ { "id": 2631935, "author_name": "phoenix9032", "author_url": "", "post_date": "02/02/2024 03:38:51", "content": "I think somehow I messed up the spectrograms or there were some differences in the way my spectrograms and Chris's one was generated . I carefully re-downloaded the latest version of spectrogram and got CV around .604 ,this amount of deviation is understandable…
", "votes": null, "replies": [] } ] } ] }, { "id": 2632020, "author_name": "chenboluo", "author_url": "", "post_date": "02/02/2024 05:05:28", "content": "I reproduced the code with LB around 0.44
", "votes": null, "replies": [] }, { "id": 2632396, "author_name": "hideyukizushi", "author_url": "", "post_date": "02/02/2024 10:35:46", "content": "FYI.For the locally trained model, the LB was .45, so it seems that there is not much difference.
\nRegarding CV, it is in the dataset log file, and there seems to be no big difference.
\n \n \n \n \n \n \n \n \n \n \n\n\nTorch：1.12.1\nCUDA：10.02\nGPU ：V100(Train Batch size has not changed)\nHello my friend @hideyukizushi … Thanks . I am now able to replicate it too..LB was .45 for me and CV .601
", "votes": null, "replies": [ { "id": 2632764, "author_name": "hideyukizushi", "author_url": "", "post_date": "02/02/2024 16:03:26", "content": "Gotcha! good luck!
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2634879, "author_name": "hidebu", "author_url": "", "post_date": "02/04/2024 04:37:51", "content": "This notebook might be useful for me. Thank you so much.
\nNow I have one question.
When I try to use this notebook, I get result below.
\n========== Fold: training ==========\n.\n.\n.\nEpoch - avg_train_loss: avg_val_loss: time: 214s\nEpoch - Save Best Loss: Model\n========== Fold result: ==========\n========== CV: ========\nMy question is why the scores of Fold X result & CV is too higher than avg_train_loss ?
\nThe Fold 4 result & CV is calculated by get_result function.
\nAre there any problem?
In addition,
\nI try to use
sys\nsys.path.append()\n kaggle_kl_div score\n\noof = pd.DataFrame(oof_df[target_preds].copy())\noof.columns = label_cols\noof[] = np.arange((oof))\n\ntrue = pd.DataFrame(oof_df[label_cols].copy())\ntrue[] = np.arange((true))\n\ncv = score(solution=true, submission=oof, row_id_column_name=)\n(,cv)\nThe result is below. It seems no problem.
\nCV Score KL-Div for EfficientNetB2 = 0.6780376070130062
I changed the get_result function similar to the above and it works .. that way ..
", "votes": null, "replies": [] }, { "id": 2642735, "author_name": "wuwenmin", "author_url": "", "post_date": "02/08/2024 11:24:13", "content": "I also failed to achieve the same result as Chris's TensorFlow version using the same df splits.
\n: CV Score KL-Div for EfficientNetB2 = .\n: CV Score KL-Div for EfficientNetB2 = .\nI changed optimizer and scheduler like chris notebook that are difference between moth and chris, but got worse result, CV:0.649
\noptimizer = torch.optim.Adam(model.parameters(), lr=)\n\n torch.optim.lr_scheduler LambdaLR\nLR_START = \nLR_MAX = \nLR_RAMPUP_EPOCHS = \nLR_SUSTAIN_EPOCHS = \nLR_STEP_DECAY = \nEVERY = \n\n ():\n epoch < LR_RAMPUP_EPOCHS:\n lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n lr = LR_MAX\n :\n lr = LR_MAX * LR_STEP_DECAY**((epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS)//EVERY)\n lr\n\n scheduler = LambdaLR(optimizer, lr_lambda=lrfn)\nreally helpful
", "votes": null, "replies": [] }, { "id": 2641501, "author_name": "sunny7712", "author_url": "", "post_date": "02/07/2024 14:20:49", "content": "Hi @alejopaullier! I thought of the same and tried implementing Chris's notebook in pytorch. However I couldn't get the CV score to go below 1.4 no matter what I try. I tried to replicate his result but I don't know whats's going wrong. Could you please take a look at it and point me where I'm wrong. Thanks!
\nHere's the notebook : HMS-Starter-Pytorch (Train)
", "votes": null, "replies": [ { "id": 2641509, "author_name": "alejopaullier", "author_url": "", "post_date": "02/07/2024 14:25:56", "content": "Your model has a softmax layer at the end, while mine doesn't. Check that you are correctly passing the predictions to the criterion in the same way as I did. Also, I use a different scheduler which is kind of similar to Chris'. Try debugging the code piece by piece. @sunny7712
", "votes": null, "replies": [] } ] }, { "id": 2641729, "author_name": "ghatotkachh", "author_url": "", "post_date": "02/07/2024 16:29:55", "content": "I tried the same thing but apparently it is taking too much time to train like on gpu it is taking 10 minutes per iteration for resnet50
\nis it normal
Thanks for nice notebook.
\nWhy do you use following scheduler ?
\nIs it normal when training pre-trained model ?
torch.optim.lr_scheduler OneCycleLR\nAs in Chris` notebook, I used following scheduler and scheduler.step in every epoch but got worse result, CV:0.65.
\n torch.optim.lr_scheduler LambdaLR\n\nLR_START = \nLR_MAX = \nLR_RAMPUP_EPOCHS = \nLR_SUSTAIN_EPOCHS = \nLR_STEP_DECAY = \nEVERY = \n\n ():\n epoch < LR_RAMPUP_EPOCHS:\n lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n lr = LR_MAX\n :\n lr = LR_MAX * LR_STEP_DECAY**((epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS)//EVERY)\n lr\n\n scheduler = LambdaLR(optimizer, lr_lambda=lrfn)\nGreat job! Upvoted!
\nI like Pytorch, so using this source as a baseline, I was able to get very good results with a single model.
In my case, I got CV0.60 and LB0.44 using this notebook without any change.
\nDo you use some training trick like 2 stage learning ?
\nhttps://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477461
\n\nDo you use some training trick like 2 stage learning ?
\n
No, I haven't used it.No major changes were made. As a pipeline, I use a little trick with CustomModel and CustomDataset.
\nLike the link you wrote.
\nI haven't worked on it yet, but I think it's necessary to deal with the problem of large dispersion occurring on a fold-by-fold basis.
Just Started Pytorch a month a ago really liked your notebook
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2627656": "This is my implementation of @cdeotte [EfficientNetB0 Starter - [LB 0.43]](https://www.kaggle.com/code/cdeotte/efficientnetb0-starter-lb-0-43). I tried to resemble it as closely as possible. \n\nYou can find the code here:\n- [HMS | EfficientNetB0 PyTorch [Train]](https://www.kaggle.com/code/alejopaullier/hms-efficientnetb0-pytorch-train)\n- [HMS | EfficientNetB0 PyTorch [Inference]](https://www.kaggle.com/code/alejopaullier/hms-efficientnetb0-pytorch-inference)\n\nThis notebook has approximately same CV and public LB scores as the version of Chirs. It achieves 0.60 in CV score and 0.44 score in public LB.\n\nHope you like it 👍🏼", "2628016": "great work! pytorch is where I know my stuff", "2628324": "Great work!!!!!!!", "2629615": "How many Epoch to tune the model?", "2631020": "I used 4 epochs. You can check it in the `config` class under Configuration section", "2631133": "Good post! I noticed the spectrograms looked different from Chris's. Did you modify them for plotting? Thanks for the well-organized notebooks.", "2631237": "Somehow I am struggling with reproducing the result of this notebook . I trained locally the same notebook and I got a CV of around .640 and somehow the inference didnt work (May be some mistake , which I cant figure) LB was around .61 , it should be around .4xx or max early .5xx . Anyone else faced the same issue ?\n\n@alejopaullier should I run the training notebook as is to reproduce the result or should I need to change something ?", "2631263": "phoenix9032 have you ran it without any change?", "2631329": "yep , no change except for pointing it to the spectrograms path in my local . I have downloaded the chris created spectrograms fresh as well just to be sure ..", "2631935": "I think somehow I messed up the spectrograms or there were some differences in the way my spectrograms and Chris's one was generated . I carefully re-downloaded the latest version of spectrogram and got CV around .604 ,this amount of deviation is understandable...", "2632020": "I reproduced the code with LB around 0.44", "2632396": "FYI.For the locally trained model, the LB was .45, so it seems that there is not much difference.\nRegarding CV, it is in the dataset log file, and there seems to be no big difference.\n\n```\n# My Local Train logs\n========== Fold: 0 training ==========\nEpoch 4 - Save Best Loss: 0.6434 Model\n========== Fold: 1 training ==========\nEpoch 4 - Save Best Loss: 0.6151 Model\n========== Fold: 2 training ==========\nEpoch 4 - Save Best Loss: 0.5551 Model\n========== Fold: 3 training ==========\nEpoch 4 - Save Best Loss: 0.6258 Model\n========== Fold: 4 training ==========\nEpoch 4 - Save Best Loss: 0.5609 Model\n\n# My Local Env\nTorch：1.12.1\nCUDA：10.02\nGPU ：V100(Train Batch size has not changed)\n```", "2632714": "Hello my friend @hideyukizushi ... Thanks . I am now able to replicate it too..LB was .45 for me and CV .601", "2632764": "Gotcha! good luck!", "2634879": "This notebook might be useful for me. Thank you so much.\nNow I have one question.\n\nWhen I try to use this notebook, I get result below.\n```python\n\n========== Fold: 4 training ==========\n.\n.\n.\nEpoch 4 - avg_train_loss: 0.3205 avg_val_loss: 0.5700 time: 214s\nEpoch 4 - Save Best Loss: 0.5700 Model\n========== Fold 4 result: 1.0911863897146958 ==========\n========== CV: 1.0901223557220237 ========\n\n\n```\nMy question is why the scores of Fold X result & CV is too higher than avg_train_loss ?\nThe Fold 4 result & CV is calculated by get_result function. \nAre there any problem?", "2634880": "In addition,\nI try to use \n\n```python\nimport sys\nsys.path.append('../input')\nfrom kaggle_kl_div import score\n\noof = pd.DataFrame(oof_df[target_preds].copy())\noof.columns = label_cols\noof['id'] = np.arange(len(oof))\n\ntrue = pd.DataFrame(oof_df[label_cols].copy())\ntrue['id'] = np.arange(len(true))\n\ncv = score(solution=true, submission=oof, row_id_column_name='id')\nprint('CV Score KL-Div for EfficientNetB2 =',cv)\n```\n\nThe result is below. It seems no problem.\nCV Score KL-Div for EfficientNetB2 = 0.6780376070130062", "2634900": "I changed the get_result function similar to the above and it works .. that way ..", "2635353": "really helpful", "2641501": "Hi @alejopaullier! I thought of the same and tried implementing Chris's notebook in pytorch. However I couldn't get the CV score to go below 1.4 no matter what I try. I tried to replicate his result but I don't know whats's going wrong. Could you please take a look at it and point me where I'm wrong. Thanks!\n\nHere's the notebook : [HMS-Starter-Pytorch (Train)](https://www.kaggle.com/code/sunny7712/hms-starter-pytorch-train)", "2641509": "Your model has a softmax layer at the end, while mine doesn't. Check that you are correctly passing the predictions to the criterion in the same way as I did. Also, I use a different scheduler which is kind of similar to Chris'. Try debugging the code piece by piece. @sunny7712", "2641729": "I tried the same thing but apparently it is taking too much time to train like on gpu it is taking 10 minutes per iteration for resnet50\nis it normal", "2642735": "I also failed to achieve the same result as Chris's TensorFlow version using the same df splits.\n```\nPytorch: CV Score KL-Div for EfficientNetB2 = 0.6172535691816387\nTensorflow: CV Score KL-Div for EfficientNetB2 = 0.5986079200120783\n```", "2650359": "Thanks for nice notebook.\nWhy do you use following scheduler ?\nIs it normal when training pre-trained model ?\n\n``` python\nfrom torch.optim.lr_scheduler import OneCycleLR\n```\n\nAs in Chris` notebook, I used following scheduler and scheduler.step in every epoch but got worse result, CV:0.65.\n\n\n```python\nfrom torch.optim.lr_scheduler import LambdaLR\n\nLR_START = 1e-4\nLR_MAX = 1e-3\nLR_RAMPUP_EPOCHS = 0\nLR_SUSTAIN_EPOCHS = 1\nLR_STEP_DECAY = 0.1\nEVERY = 1\n\ndef lrfn(epoch):\n if epoch < LR_RAMPUP_EPOCHS:\n lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n lr = LR_MAX\n else:\n lr = LR_MAX * LR_STEP_DECAY**((epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS)//EVERY)\n return lr\n\n scheduler = LambdaLR(optimizer, lr_lambda=lrfn)\n```", "2650363": "I changed optimizer and scheduler like chris notebook that are difference between moth and chris, but got worse result, CV:0.649\n\n``` python\n\noptimizer = torch.optim.Adam(model.parameters(), lr=0.01)\n\nfrom torch.optim.lr_scheduler import LambdaLR\nLR_START = 1e-4\nLR_MAX = 1e-3\nLR_RAMPUP_EPOCHS = 0\nLR_SUSTAIN_EPOCHS = 1\nLR_STEP_DECAY = 0.1\nEVERY = 1\n\ndef lrfn(epoch):\n if epoch < LR_RAMPUP_EPOCHS:\n lr = (LR_MAX - LR_START) / LR_RAMPUP_EPOCHS * epoch + LR_START\n elif epoch < LR_RAMPUP_EPOCHS + LR_SUSTAIN_EPOCHS:\n lr = LR_MAX\n else:\n lr = LR_MAX * LR_STEP_DECAY**((epoch - LR_RAMPUP_EPOCHS - LR_SUSTAIN_EPOCHS)//EVERY)\n return lr\n\n scheduler = LambdaLR(optimizer, lr_lambda=lrfn)\n\n\n```", "2658237": "Great job! Upvoted!\nI like Pytorch, so using this source as a baseline, I was able to get very good results with a single model.\n* **CV：0.5994416155018669(NoAug)**\n* **LB：0.38**", "2659210": "In my case, I got CV0.60 and LB0.44 using this notebook without any change.\nDo you use some training trick like 2 stage learning ?\nhttps://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477461", "2659520": "> Do you use some training trick like 2 stage learning ?\n\nNo, I haven't used it.No major changes were made. As a pipeline, I use a little trick with CustomModel and CustomDataset.\n\nLike the link you wrote.\nI haven't worked on it yet, but I think it's necessary to deal with the problem of large dispersion occurring on a fold-by-fold basis.", "3180392": "Just Started Pytorch a month a ago really liked your notebook" }, "source": "meta" }