\nHere is screenshot:
\n
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4495635%2Feaf8a656d546ba1043df67ccb170b3d8%2FScreenshot%20from%202024-03-04%2017-07-11.png?generation=1709561273435620&alt=media\")
Hello Fellow Kagglers,
\nExciting news! Our Features+Head Starter project is ready to level up, but your help is needed to take it to the next level. Looking for enthusiastic volunteers who have extra GPU time and are eager to collaborate on running experiments and accelerating the release cycle for new versions.
\nBy joining this effort, you'll have the opportunity to dive into experiments and be among the first to contribute to the Leaderboard. While not every experiment may succeed, your valuable contributions will drive progress in our project.
\nIf you're interested in being part of this exciting journey, please let me know in the comments below. You will have the code for training, and together, we'll make strides towards our shared goals.
\nLooking forward to collaborating!
", "messages": [ { "id": 2677753, "postDate": "2024-03-02T12:05:54.043Z", "content": "Hi，Thanks for sharing ！I have extra GPU time in Kaggle at present.Could join your project as a volunteer and dive into experiments with hope to drive progress in the project.And I also have a server about two 4090，but i need some time to configure the server environment
", "rawMarkdown": "Hi，Thanks for sharing ！I have extra GPU time in Kaggle at present.Could join your project as a volunteer and dive into experiments with hope to drive progress in the project.And I also have a server about two 4090，but i need some time to configure the server environment", "votes": 1, "replies": [ { "id": 2677881, "postDate": "2024-03-02T13:22:24.637Z", "content": "Hello, you are welcome! @pr0ark
\nWe need to conduct the following experiment on Kaggle with multiple GPUs. The original experiment resulted in a leaderboard (LB) score of 0.37. With the addition of MixUp, I anticipate the score to improve to 0.36 LB. Please share the notebook later for review.
\n", "rawMarkdown": "Hello, you are welcome! @pr0ark \n\nWe need to conduct the following experiment on Kaggle with multiple GPUs. The original experiment resulted in a leaderboard (LB) score of 0.37. With the addition of MixUp, I anticipate the score to improve to 0.36 LB. Please share the notebook later for review.\n\n[Version 204 notebook](https://www.kaggle.com/code/nartaa/training-features-head-starter-testbed-003)", "votes": -1, "replies": [ { "id": 2678220, "postDate": "2024-03-02T17:28:53.153Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2680077, "postDate": "2024-03-04T00:58:23.097Z", "content": "Hi, Danial! I have trained your notebook. It took about 3 hours for each of models 'both', 'eeg' and 'kaggle' and 2 hours for
\nf model 'raw'.I use v204 model to infer
\nv205LB0.38
\nv204LB0.38
\nboth cv0.572
\neeg cv0.640
\nkaggle cv0.645
\nraw cv0.768
\nI think maybe there is alittle mass about vers version of training notebook，what s should wo do next？Is there anything to do to help you？
Hello @pr0ark this indicates the way MixUp is setup didn't improve LB.
\nI updated Features+Head starter, still 0.34 with little boost, this is the latest.
\nYou could fork it, and based on it train 'both', 'kaggle' and 'eeg' on EffecientNetB0, then we ensemble on 7 models.
You should create a spreedsheet to keep track of experiments. It can get overwhelming.
\nHere is screenshot:
\n
Hello !
\nI have seen this discussion, which is very useful to me. Data augmentation has only had a negative impact.
And this discussion prove this method .
", "rawMarkdown": "Hello ! \nI have seen this [discussion](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/480775#2675818), which is very useful to me. Data augmentation has only had a negative impact.\n\nAnd this [discussion](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/481436#2679779) prove this method .", "votes": 1 } ] } ] }, { "id": 2672737, "postDate": "2024-02-28T09:01:00.790Z", "content": "Hi！I have extract GPU time on Kaggle，also I have my own server （RTX 4090）but just one GPU.
", "rawMarkdown": "Hi！I have extract GPU time on Kaggle，also I have my own server （RTX 4090）but just one GPU.", "votes": 1, "replies": [ { "id": 2672894, "postDate": "2024-02-28T11:02:48.973Z", "content": "Hello,
\nYou could run this notebook version 202.
\nIt needs to be trained on multiple 2xGPUs, it will take around 3 hours.
\nFor submission, you should set submission = True and update '/kaggle/input/training-features-head-starter-lb-0-35/'
\nLet me know how it works out.
How fast are RTX 4090 when compared with the p100 or T4 that are on kaggle ?
", "rawMarkdown": "How fast are RTX 4090 when compared with the p100 or T4 that are on kaggle ?", "replies": [ { "id": 2678675, "postDate": "2024-03-03T00:57:03.437Z", "content": "I never train any model on Kaggle😂
", "rawMarkdown": "I never train any model on Kaggle😂" }, { "id": 2679592, "postDate": "2024-03-03T16:08:06.993Z", "content": "haha, advantages of having a powerful GPU, but I guess its troublesome setting up all the data sources offline, and on kaggle its taking around 3 hours to train the 5 fold EfficientNet model
", "rawMarkdown": "haha, advantages of having a powerful GPU, but I guess its troublesome setting up all the data sources offline, and on kaggle its taking around 3 hours to train the 5 fold EfficientNet model " } ] } ] }, { "id": 2672324, "postDate": "2024-02-28T04:12:22.447Z", "content": "Hi I move the code(Chris's efficientnet) from jupyter to .py and it can reduce a lot of time to training(1 epoch cost 5 mins)
", "rawMarkdown": "Hi I move the code(Chris's efficientnet) from jupyter to .py and it can reduce a lot of time to training(1 epoch cost 5 mins)", "votes": 1, "replies": [ { "id": 2672875, "postDate": "2024-02-28T10:57:08.727Z", "content": "Hi,
\nAppreciated, I'll keep it in mind.
Hi, Danial!
\nThanks for sharing your enthusiasm!
\nI have extra GPU time in Kaggle at present.
\nCould join your project as a volunteer and dive into experiments with hope to drive progress in the project.
Hello @andrey67
\nGreat :)
\nThis notebook version 202 needs to be trained on multiple GPUs, it will take around 3 hours.
\nFor submission, you should set submission = True and update '/kaggle/input/training-features-head-starter-lb-0-35/'
\nLet me know how it goes.
Hi, Danial!
\nI trained your notebook. I haven't done any experiments with it yet.
\nIt took approximately 3 hours for each of models 'both', 'eeg' and 'kaggle' and approximately 2 hours for model 'raw'.
\nWeights from trained models are here: [https://www.kaggle.com/datasets/andrey67/f-h-effnetb2-weights-v45]
\nCV score for 'both' ~ 0.57
\nCV score for 'eeg' ~ 0.63
\nCV score for 'kaggle' ~ 0.62
\nCV score for 'raw' ~ 0.72
\nFinal submission LB = 0.35 (for ensemble with LBs = [0.39,0.41,0.43,0.41]).
\nSubmission LB can be slightly improved (but it remains = 0.35) if you set other weights in the LBs.
Hi Andrey,
\nI am not sure why you are getting worst CV and LB, have you made any changes? The training is sensitive to the batch size, for 'both', 'kaggle', and 'eeg', it should be 2x8(2 GPUs, 8 batch size), and for 'raw' it should be 1x8(single GPU, 8 batch size), I did modify the learning rate for 'both' but I still got 0.37 LB.
Could you share the notebook you've used for training, I'll try to figure out why you got worse results.
\nI released a new version with [0.37, 0.39, 0.41, 0.41] LBs and ensemble 0.34 LB
", "rawMarkdown": "Hi Andrey,\nI am not sure why you are getting worst CV and LB, have you made any changes? The training is sensitive to the batch size, for 'both', 'kaggle', and 'eeg', it should be 2x8(2 GPUs, 8 batch size), and for 'raw' it should be 1x8(single GPU, 8 batch size), I did modify the learning rate for 'both' but I still got 0.37 LB.\n\nCould you share the notebook you've used for training, I'll try to figure out why you got worse results.\n\nI released a [new version](https://www.kaggle.com/code/nartaa/features-head-starter) with [0.37, 0.39, 0.41, 0.41] LBs and ensemble 0.34 LB", "votes": -1 }, { "id": 2674797, "postDate": "2024-02-29T14:46:22.530Z", "content": "Hi, Danial.
\nThank you for the feedback.
\nYes, I used 2 GPUs for 'raw' - maybe this is the reason.
\nAnd I didn’t change anything in the notebook - absolutely the same notebook to which you provided the link.
\nI'll try to train 'raw' again with single GPU, 8 batch size and will write about the results later, maybe tomorrow.
Hi, Danial.
\nI trained the 'raw' model again with single GPU, 8 batch size.
\nWeights for re-trained 'raw' model are here: F + H EffNetB2 weights V45
\nNext, I conducted A/B testing to determine the impact of each of the updated versions of models' weights on the LB.
\nIt turned out that only the updated version of weights for “kaggle” has an effect on the deterioration of LB.
\nNamely:
I see, so you are saying all models V45 for 'both', 'eeg' + 'raw' + V43 for 'kaggle' gets us 0.34
\nI modified the learning rate when I trained 'both', so that could be the reason.
I am going to rerun 'kaggle', we shouldn't move forward without knowing what's going on.
", "rawMarkdown": "I see, so you are saying all models V45 for 'both', 'eeg' + 'raw' + V43 for 'kaggle' gets us 0.34\nI modified the learning rate when I trained 'both', so that could be the reason.\n\nI am going to rerun 'kaggle', we shouldn't move forward without knowing what's going on.", "votes": -1 }, { "id": 2679708, "postDate": "2024-03-03T17:29:25.003Z", "content": "Hi @andrey67 Good catch!!
\nThe A/B you've done helped boost the LB a little, but still 0.34.
\nI reverted the learning rate back, and added the trained model 47 to notebook version 22
Hello Fellow Kagglers,
\nExciting news! Our Features+Head Starter project is ready to level up, but your help is needed to take it to the next level. Looking for enthusiastic volunteers who have extra GPU time and are eager to collaborate on running experiments and accelerating the release cycle for new versions.
\nBy joining this effort, you'll have the opportunity to dive into experiments and be among the first to contribute to the Leaderboard. While not every experiment may succeed, your valuable contributions will drive progress in our project.
\nIf you're interested in being part of this exciting journey, please let me know in the comments below. You will have the code for training, and together, we'll make strides towards our shared goals.
\nLooking forward to collaborating!
", "rawMarkdown": "Hello Fellow Kagglers,\n\nExciting news! Our[ Features+Head Starter](https://www.kaggle.com/code/nartaa/features-head-starter) project is ready to level up, but your help is needed to take it to the next level. Looking for enthusiastic volunteers who have extra GPU time and are eager to collaborate on running experiments and accelerating the release cycle for new versions.\n\nBy joining this effort, you'll have the opportunity to dive into experiments and be among the first to contribute to the Leaderboard. While not every experiment may succeed, your valuable contributions will drive progress in our project.\n\nIf you're interested in being part of this exciting journey, please let me know in the comments below. You will have the code for training, and together, we'll make strides towards our shared goals.\n\nLooking forward to collaborating!", "votes": 1 } ], "comments": [ { "id": 2677753, "author_name": "Pr04Ark", "author_url": "", "post_date": "2024-03-02T12:05:54.043000", "content": "Hi，Thanks for sharing ！I have extra GPU time in Kaggle at present.Could join your project as a volunteer and dive into experiments with hope to drive progress in the project.And I also have a server about two 4090，but i need some time to configure the server environment
", "votes": 1, "replies": [ { "id": 2677881, "author_name": "Danial Zakaria", "author_url": "", "post_date": "2024-03-02T13:22:24.637000", "content": "Hello, you are welcome! @pr0ark
\nWe need to conduct the following experiment on Kaggle with multiple GPUs. The original experiment resulted in a leaderboard (LB) score of 0.37. With the addition of MixUp, I anticipate the score to improve to 0.36 LB. Please share the notebook later for review.
\n", "votes": -1, "replies": [ { "id": 2678220, "author_name": "", "author_url": "", "post_date": "2024-03-02T17:28:53.153000", "content": "", "votes": 0, "replies": [] }, { "id": 2680077, "author_name": "Pr04Ark", "author_url": "", "post_date": "2024-03-04T00:58:23.097000", "content": "Hi, Danial! I have trained your notebook. It took about 3 hours for each of models 'both', 'eeg' and 'kaggle' and 2 hours for
\nf model 'raw'.I use v204 model to infer
\nv205LB0.38
\nv204LB0.38
\nboth cv0.572
\neeg cv0.640
\nkaggle cv0.645
\nraw cv0.768
\nI think maybe there is alittle mass about vers version of training notebook，what s should wo do next？Is there anything to do to help you？
Hello @pr0ark this indicates the way MixUp is setup didn't improve LB.
\nI updated Features+Head starter, still 0.34 with little boost, this is the latest.
\nYou could fork it, and based on it train 'both', 'kaggle' and 'eeg' on EffecientNetB0, then we ensemble on 7 models.
You should create a spreedsheet to keep track of experiments. It can get overwhelming.
\nHere is screenshot:
\n
Hello !
\nI have seen this discussion, which is very useful to me. Data augmentation has only had a negative impact.
And this discussion prove this method .
", "votes": 1, "replies": [] } ] } ] }, { "id": 2672737, "author_name": "Seeing Times", "author_url": "", "post_date": "2024-02-28T09:01:00.790000", "content": "Hi！I have extract GPU time on Kaggle，also I have my own server （RTX 4090）but just one GPU.
", "votes": 1, "replies": [ { "id": 2672894, "author_name": "Danial Zakaria", "author_url": "", "post_date": "2024-02-28T11:02:48.973000", "content": "Hello,
\nYou could run this notebook version 202.
\nIt needs to be trained on multiple 2xGPUs, it will take around 3 hours.
\nFor submission, you should set submission = True and update '/kaggle/input/training-features-head-starter-lb-0-35/'
\nLet me know how it works out.
How fast are RTX 4090 when compared with the p100 or T4 that are on kaggle ?
", "votes": 0, "replies": [ { "id": 2678675, "author_name": "Seeing Times", "author_url": "", "post_date": "2024-03-03T00:57:03.437000", "content": "I never train any model on Kaggle😂
", "votes": 0, "replies": [] }, { "id": 2679592, "author_name": "Arun", "author_url": "", "post_date": "2024-03-03T16:08:06.993000", "content": "haha, advantages of having a powerful GPU, but I guess its troublesome setting up all the data sources offline, and on kaggle its taking around 3 hours to train the 5 fold EfficientNet model
", "votes": 0, "replies": [] } ] } ] }, { "id": 2672324, "author_name": "kerry sun", "author_url": "", "post_date": "2024-02-28T04:12:22.447000", "content": "Hi I move the code(Chris's efficientnet) from jupyter to .py and it can reduce a lot of time to training(1 epoch cost 5 mins)
", "votes": 1, "replies": [ { "id": 2672875, "author_name": "Danial Zakaria", "author_url": "", "post_date": "2024-02-28T10:57:08.727000", "content": "Hi,
\nAppreciated, I'll keep it in mind.
Hi, Danial!
\nThanks for sharing your enthusiasm!
\nI have extra GPU time in Kaggle at present.
\nCould join your project as a volunteer and dive into experiments with hope to drive progress in the project.
Hello @andrey67
\nGreat :)
\nThis notebook version 202 needs to be trained on multiple GPUs, it will take around 3 hours.
\nFor submission, you should set submission = True and update '/kaggle/input/training-features-head-starter-lb-0-35/'
\nLet me know how it goes.
Hi, Danial!
\nI trained your notebook. I haven't done any experiments with it yet.
\nIt took approximately 3 hours for each of models 'both', 'eeg' and 'kaggle' and approximately 2 hours for model 'raw'.
\nWeights from trained models are here: [https://www.kaggle.com/datasets/andrey67/f-h-effnetb2-weights-v45]
\nCV score for 'both' ~ 0.57
\nCV score for 'eeg' ~ 0.63
\nCV score for 'kaggle' ~ 0.62
\nCV score for 'raw' ~ 0.72
\nFinal submission LB = 0.35 (for ensemble with LBs = [0.39,0.41,0.43,0.41]).
\nSubmission LB can be slightly improved (but it remains = 0.35) if you set other weights in the LBs.
Hi Andrey,
\nI am not sure why you are getting worst CV and LB, have you made any changes? The training is sensitive to the batch size, for 'both', 'kaggle', and 'eeg', it should be 2x8(2 GPUs, 8 batch size), and for 'raw' it should be 1x8(single GPU, 8 batch size), I did modify the learning rate for 'both' but I still got 0.37 LB.
Could you share the notebook you've used for training, I'll try to figure out why you got worse results.
\nI released a new version with [0.37, 0.39, 0.41, 0.41] LBs and ensemble 0.34 LB
", "votes": -1, "replies": [] }, { "id": 2674797, "author_name": "Andrey", "author_url": "", "post_date": "2024-02-29T14:46:22.530000", "content": "Hi, Danial.
\nThank you for the feedback.
\nYes, I used 2 GPUs for 'raw' - maybe this is the reason.
\nAnd I didn’t change anything in the notebook - absolutely the same notebook to which you provided the link.
\nI'll try to train 'raw' again with single GPU, 8 batch size and will write about the results later, maybe tomorrow.
Hi, Danial.
\nI trained the 'raw' model again with single GPU, 8 batch size.
\nWeights for re-trained 'raw' model are here: F + H EffNetB2 weights V45
\nNext, I conducted A/B testing to determine the impact of each of the updated versions of models' weights on the LB.
\nIt turned out that only the updated version of weights for “kaggle” has an effect on the deterioration of LB.
\nNamely:
I see, so you are saying all models V45 for 'both', 'eeg' + 'raw' + V43 for 'kaggle' gets us 0.34
\nI modified the learning rate when I trained 'both', so that could be the reason.
I am going to rerun 'kaggle', we shouldn't move forward without knowing what's going on.
", "votes": -1, "replies": [] }, { "id": 2679708, "author_name": "Danial Zakaria", "author_url": "", "post_date": "2024-03-03T17:29:25.003000", "content": "Hi @andrey67 Good catch!!
\nThe A/B you've done helped boost the LB a little, but still 0.34.
\nI reverted the learning rate back, and added the trained model 47 to notebook version 22