Hello everybody!
\nI wonder who has the best models!
\nHere are my best single models for 5 folds:
\nModel based on Chris's pipeline, with modifications:
\nCV 0.5418, LB 0.43
\nModel based on a combination of my pipeline and Chris' pipeline, with modifications:
\nCV 0.555, LB 0.44
\nThe size of one spectrogram is 128,256,3, just like Chris's.
\nGood luck to everyone and a peaceful sky above your head!
Hello everybody!
\nI wonder who has the best models!
\nHere are my best single models for 5 folds:
\nModel based on Chris's pipeline, with modifications:
\nCV 0.5418, LB 0.43
\nModel based on a combination of my pipeline and Chris' pipeline, with modifications:
\nCV 0.555, LB 0.44
\nThe size of one spectrogram is 128,256,3, just like Chris's.
\nGood luck to everyone and a peaceful sky above your head!
CV 0.519 and LB 0.37 for a single 5-fold model
", "rawMarkdown": "CV 0.519 and LB 0.37 for a single 5-fold model", "votes": 12, "replies": [ { "id": 2641544, "postDate": "2024-02-07T14:51:13.023Z", "content": "Wow, If you can share: this result on spectrogram sizes [128,256,3] or larger.
", "rawMarkdown": "Wow, If you can share: this result on spectrogram sizes [128,256,3] or larger.", "votes": 1, "replies": [ { "id": 2641564, "postDate": "2024-02-07T15:01:32.853Z", "content": "Cant reveal anything for obvious reasons
", "rawMarkdown": "Cant reveal anything for obvious reasons", "votes": 3 }, { "id": 2641575, "postDate": "2024-02-07T15:05:27.380Z", "content": "Ok, it's actually an incentive for me to work harder to understand what I'm doing wrong. Good luck
", "rawMarkdown": "Ok, it's actually an incentive for me to work harder to understand what I'm doing wrong. Good luck", "votes": 2 } ] }, { "id": 2642824, "postDate": "2024-02-08T12:51:39.153Z", "content": "Are you evaluating your cv on non-overlapping spectrograms based samples?
", "rawMarkdown": "Are you evaluating your cv on non-overlapping spectrograms based samples?", "votes": 4, "replies": [ { "id": 2647591, "postDate": "2024-02-11T16:49:57.860Z", "content": "sgkf on patient_id
Mine is 4 models: all Chris's
\nEffecintNetB2 Kaggle spectrograms 0.47
\nEffecintNetB2 EEG spectrograms 0.44
\nEffecintNetB2 both spectrograms 0.43
\nWavenet raw EEG 0.52
Weighted Ensemble 0.38
\nThe notebook code is available in this link
best single CV: 0.56, LB: 0.38
", "rawMarkdown": "best single CV: 0.56, LB: 0.38", "votes": 6, "replies": [ { "id": 2641175, "postDate": "2024-02-07T10:17:39.930Z", "content": "Great result!
", "rawMarkdown": "Great result! ", "votes": 1 } ] }, { "id": 2652656, "postDate": "2024-02-14T23:01:27.910Z", "content": "I have a model with CV 0.68 and LB 0.39.
\nI split the training data into two populations, then train on it in two stages (once for each population). I calculated the CV score on all OOF data from both populations, but predicted using only the final model. The first training population has a higher KL-Divergence for the CV than the second population. This was done purposely, as discussed in the notebook, to bias samples with total votes >= 10 with the goal of training on more granular vote distributions.
", "rawMarkdown": "I have a [model](https://www.kaggle.com/code/seanbearden/effnetb0-2-pop-model-train-twice-lb-0-39) with CV 0.68 and LB 0.39. \n\nI split the training data into two populations, then train on it in two stages (once for each population). I calculated the CV score on all OOF data from both populations, but predicted using only the final model. The first training population has a higher KL-Divergence for the CV than the second population. This was done purposely, as discussed in the notebook, to bias samples with total votes >= 10 with the goal of training on more granular vote distributions.", "votes": 3, "replies": [ { "id": 2652962, "postDate": "2024-02-15T05:39:27.887Z", "content": "I did a similar experiment yesterday but didn't finish, my result was 0.42
", "rawMarkdown": "I did a similar experiment yesterday but didn't finish, my result was 0.42", "votes": 1 } ] }, { "id": 2641718, "postDate": "2024-02-07T16:23:29.480Z", "content": "My case, using three models and ensembling them together:
\nEfficientNetB0: best CV score 0.55 and LB 0.42
\nEfficientNetB1: best CV score 0.53 and LB 0.46
\nResNet34d: best CV score 0.56 and LB 0.45
Weighted ensembler (inference): LB 0.41
", "rawMarkdown": "My case, using three models and ensembling them together:\nEfficientNetB0: best CV score 0.55 and LB 0.42\nEfficientNetB1: best CV score 0.53 and LB 0.46\nResNet34d: best CV score 0.56 and LB 0.45\n\nWeighted ensembler (inference): LB 0.41", "votes": 3, "replies": [ { "id": 2641741, "postDate": "2024-02-07T16:35:19.117Z", "content": "Yes, the ensemble will improve the results, but right now it's important for me to improve the single model as much as possible
", "rawMarkdown": "Yes, the ensemble will improve the results, but right now it's important for me to improve the single model as much as possible", "votes": 3, "replies": [ { "id": 2641744, "postDate": "2024-02-07T16:38:12.410Z", "content": "@aikhmelnytskyy
\nI don't fancy asking questions about how to improve my models, but… what image size are you using?
128,256.3 - This is the size of one spectrogram, I tried doubling the size of the spectrograms, but it doesn't seem to have any effect on CV and LB in my case
", "rawMarkdown": "128,256.3 - This is the size of one spectrogram, I tried doubling the size of the spectrograms, but it doesn't seem to have any effect on CV and LB in my case", "votes": 1 }, { "id": 2641814, "postDate": "2024-02-07T17:23:01.367Z", "content": "I'm asking because I've been using 512x512 (when resizing). You think I should lower the size to 256x256?
", "rawMarkdown": "I'm asking because I've been using 512x512 (when resizing). You think I should lower the size to 256x256?", "votes": 2 }, { "id": 2641879, "postDate": "2024-02-07T18:19:39.757Z", "content": "I think it's worth trying different sizes and seeing the results. By the way, don't forget that you can build an ensemble on images of different sizes, it works because the models focus on different features.
", "rawMarkdown": "I think it's worth trying different sizes and seeing the results. By the way, don't forget that you can build an ensemble on images of different sizes, it works because the models focus on different features.", "votes": 3 } ] } ] }, { "id": 2641209, "postDate": "2024-02-07T10:51:17.003Z", "content": "single model at present
\nCV：0.591 LB：0.40
\nThere is under improvement. Good news is that CV score is stable in this competition.
", "rawMarkdown": "**single model at present**\n\nCV：0.591 LB：0.40\n\nThere is under improvement. Good news is that CV score is stable in this competition.", "votes": 3, "replies": [ { "id": 2641289, "postDate": "2024-02-07T11:53:49.677Z", "content": "For me, it is a problem that LB is at the same time slightly worse than train loss and significantly better than CV. It looks like the model is overfitting, and the result on the private dataset will be poorly predicted, although the training CV may be affected by the fact that we have a different number of spectrograms and eegs: essentially, the spectrograms are repeated during training
", "rawMarkdown": "For me, it is a problem that LB is at the same time slightly worse than train loss and significantly better than CV. It looks like the model is overfitting, and the result on the private dataset will be poorly predicted, although the training CV may be affected by the fact that we have a different number of spectrograms and eegs: essentially, the spectrograms are repeated during training", "votes": 2 } ] }, { "id": 2641061, "postDate": "2024-02-07T08:50:00.470Z", "content": "Mine is CV: 0.58, LB: 0.41
", "rawMarkdown": "Mine is CV: 0.58, LB: 0.41", "votes": 3, "replies": [ { "id": 2641068, "postDate": "2024-02-07T08:58:55.027Z", "content": "Great, I just saw a similar discussion and from what I see people with worse CV's have better LB's than mine.
", "rawMarkdown": "Great, I just saw a similar discussion and from what I see people with worse CV's have better LB's than mine.", "votes": 3 } ] }, { "id": 2641097, "postDate": "2024-02-07T09:38:36.420Z", "content": "In my case, CV:0.569 LB:0.43
\nemsemble: LB:0.40
The result is similar to mine, I tried to choose different ranges for eeg spectrograms during training, this approach obviously worsens LB, it definitely worsens if we take into account during training that some eegs in different ranges have different labels.
", "rawMarkdown": "The result is similar to mine, I tried to choose different ranges for eeg spectrograms during training, this approach obviously worsens LB, it definitely worsens if we take into account during training that some eegs in different ranges have different labels.", "votes": 2, "replies": [ { "id": 2641210, "postDate": "2024-02-07T10:52:32.807Z", "content": "Really it it similar.
\nI tried to create different type of spectrograms from eeg(eg. Hz range, denoise, CWT), CV is improving but LB has not changed.
\nDid you try tree model and ensemble of tree model and NN model?
Yes, I also tried different type of spectrograms from eeg (eg. Hz range, denoise, CWT). In the end, I found an option for preparing spectrograms that improves CV, but does not affect LB… I've tried building separate models for each type of spectrogram, but haven't been able to get them to work yet
", "rawMarkdown": "Yes, I also tried different type of spectrograms from eeg (eg. Hz range, denoise, CWT). In the end, I found an option for preparing spectrograms that improves CV, but does not affect LB... I've tried building separate models for each type of spectrogram, but haven't been able to get them to work yet", "votes": 2 }, { "id": 2641286, "postDate": "2024-02-07T11:53:20.057Z", "content": "This is super similar situation.
\nMy CV was improved, but does not affect LB…
\nIt seems necessary to think of ways to improve the LB (Leaderboard) from different methods.
I just experienced this as well. A very nice CV increase by 0.05 and the LB went in the opposite direction 🤷♂️
", "rawMarkdown": "I just experienced this as well. A very nice CV increase by 0.05 and the LB went in the opposite direction 🤷♂️", "votes": 5 }, { "id": 2645466, "postDate": "2024-02-10T09:03:53.500Z", "content": "I haven't experienced it yet, can you explain what you did?
", "rawMarkdown": "I haven't experienced it yet, can you explain what you did? " } ] } ] }, { "id": 2641949, "postDate": "2024-02-07T19:21:01.023Z", "content": "I think the power lies in effectively ensembling your best deep learning models. But taking the mean of my predictions didn't improve my score.
", "rawMarkdown": "I think the power lies in effectively ensembling your best deep learning models. But taking the mean of my predictions didn't improve my score.", "votes": 2, "replies": [ { "id": 2641962, "postDate": "2024-02-07T19:32:56.680Z", "content": "It is important that the models add new information to the ensemble. If you make an ensemble from similar models, or models built on similar features, they will give a smaller increase than models built on different pipelines.
", "rawMarkdown": "It is important that the models add new information to the ensemble. If you make an ensemble from similar models, or models built on similar features, they will give a smaller increase than models built on different pipelines.", "votes": 4, "replies": [ { "id": 2641968, "postDate": "2024-02-07T19:39:44.987Z", "content": "Hmm currently utilizing 4 different transfer learning models (densenet, exception,..), all scoring between 0.57 and 0.63. Do you think those are not able to catch very different kind of features?
", "rawMarkdown": "Hmm currently utilizing 4 different transfer learning models (densenet, exception,..), all scoring between 0.57 and 0.63. Do you think those are not able to catch very different kind of features?", "votes": 2 }, { "id": 2641979, "postDate": "2024-02-07T19:52:07.797Z", "content": "I don't know what the problem is, but I have two hypotheses: 1. It might be worth trying to build a weighted ensemble, one of the models might worsen your results. 2. You have already extracted all the information from your data, then the ensemble does not work (more precisely, it works worse). I mean, the problem might be in the preprocessing, to improve the results you need to feed the models with different data. I met a similar story a few months ago, at the end of the day the disappointment was great, and the reason was that I failed the preprocessing step. But I am not the leader in this competition and my hypotheses may be wrong
", "rawMarkdown": "I don't know what the problem is, but I have two hypotheses: 1. It might be worth trying to build a weighted ensemble, one of the models might worsen your results. 2. You have already extracted all the information from your data, then the ensemble does not work (more precisely, it works worse). I mean, the problem might be in the preprocessing, to improve the results you need to feed the models with different data. I met a similar story a few months ago, at the end of the day the disappointment was great, and the reason was that I failed the preprocessing step. But I am not the leader in this competition and my hypotheses may be wrong", "votes": 1 }, { "id": 2642008, "postDate": "2024-02-07T20:27:56.940Z", "content": "Yes, my plan was to find an effective way of ensembling the models, maybe I can instead use a dense layer to predict the value or 6 regression models. Wanted to make sure to find an effective ensembling method before retraining my models with more augmentations or epochs.
", "rawMarkdown": "Yes, my plan was to find an effective way of ensembling the models, maybe I can instead use a dense layer to predict the value or 6 regression models. Wanted to make sure to find an effective ensembling method before retraining my models with more augmentations or epochs.", "votes": 1 }, { "id": 2642013, "postDate": "2024-02-07T20:40:18.853Z", "content": "You can try, but keep in mind that with this procedure there's a risk of further overfitting the model
", "rawMarkdown": "You can try, but keep in mind that with this procedure there's a risk of further overfitting the model", "votes": 1 }, { "id": 2642038, "postDate": "2024-02-07T21:18:11.213Z", "content": "Yes true, need to be careful with that but luckily there seems to be a reliable cv/lb correlation.
", "rawMarkdown": "Yes true, need to be careful with that but luckily there seems to be a reliable cv/lb correlation.", "votes": 1 }, { "id": 2706372, "postDate": "2024-03-19T20:44:41.683Z", "content": "@aikhmelnytskyy To go back to this discussion. The combination of different models only improved behind the kaggle digits. With a reliable cv. When doing the same by evaluating 1000 weighted ensembles next to the cv score the ensemble was way worse.
", "rawMarkdown": "@aikhmelnytskyy To go back to this discussion. The combination of different models only improved behind the kaggle digits. With a reliable cv. When doing the same by evaluating 1000 weighted ensembles next to the cv score the ensemble was way worse. ", "votes": 1 }, { "id": 2706374, "postDate": "2024-03-19T20:46:48.320Z", "content": "Now I'm back to improving one model, but I think I'm to late for this if I look to the top scores.
", "rawMarkdown": "Now I'm back to improving one model, but I think I'm to late for this if I look to the top scores.\n", "votes": 1 }, { "id": 2706483, "postDate": "2024-03-19T22:33:22.883Z", "content": "My best model shows lb 0.32 this is one model. Read the comments, all tops have single models at the level of lb 0.25-0.28. It's not about the ensemble, it's about something else
", "rawMarkdown": "My best model shows lb 0.32 this is one model. Read the comments, all tops have single models at the level of lb 0.25-0.28. It's not about the ensemble, it's about something else" }, { "id": 2706488, "postDate": "2024-03-19T22:51:04.570Z", "content": "If that's the case I'll have a look into optimizing the model or preprocessing the data. Got albumentations to work correctly and improve my cv score.
", "rawMarkdown": "If that's the case I'll have a look into optimizing the model or preprocessing the data. Got albumentations to work correctly and improve my cv score." } ] } ] }, { "id": 2641727, "postDate": "2024-02-07T16:29:03.980Z", "content": "Mine is 0.61 in CV and 0.42 lb using pythorch, I stopped trying to improve the model because I think it's overfitting, thinking about different ways to improve, better EEGs might be the solution
", "rawMarkdown": "Mine is 0.61 in CV and 0.42 lb using pythorch, I stopped trying to improve the model because I think it's overfitting, thinking about different ways to improve, better EEGs might be the solution", "votes": 2, "replies": [ { "id": 2641739, "postDate": "2024-02-07T16:33:15.590Z", "content": "Yes, you are right, the problem here is not in the models, the problem is in preprocessing
", "rawMarkdown": "Yes, you are right, the problem here is not in the models, the problem is in preprocessing", "votes": 3, "replies": [ { "id": 2641820, "postDate": "2024-02-07T17:26:35.757Z", "content": "Perfect, you said everything that matters, I'm not an expert in this area, so I'm studying the subject a lot, it's quite challenging to say the least, I want to invest all my efforts in preprocessing
", "rawMarkdown": "Perfect, you said everything that matters, I'm not an expert in this area, so I'm studying the subject a lot, it's quite challenging to say the least, I want to invest all my efforts in preprocessing", "votes": 1 } ] } ] }, { "id": 2651665, "postDate": "2024-02-14T10:08:56.470Z", "content": "Hi, I'm new here on kaggle and i was wondering what 'CV' and 'LB' stands for, kind regards
", "rawMarkdown": "Hi, I'm new here on kaggle and i was wondering what 'CV' and 'LB' stands for, kind regards ", "replies": [ { "id": 2651704, "postDate": "2024-02-14T10:33:06.650Z", "content": "Hello! СV is the result of your cross-validation, in this case the average value of the metrics obtained on the validation data. LB is your indicator on the public test data, Do not forget that there are also three-week test data.
", "rawMarkdown": "Hello! СV is the result of your cross-validation, in this case the average value of the metrics obtained on the validation data. LB is your indicator on the public test data, Do not forget that there are also three-week test data.", "votes": 2, "replies": [ { "id": 2653142, "postDate": "2024-02-15T08:34:30.883Z", "content": "Ahh thanks you so much!
", "rawMarkdown": "Ahh thanks you so much!" } ] } ] }, { "id": 2651280, "postDate": "2024-02-14T04:26:03.493Z", "content": "Helpful!!!!
", "rawMarkdown": "Helpful!!!!" }, { "id": 2642838, "postDate": "2024-02-08T12:58:01.650Z", "rawMarkdown": "", "isDeleted": true }, { "id": 2642941, "postDate": "2024-02-08T14:12:51.197Z", "content": "helpful , thanks
", "rawMarkdown": "helpful , thanks " } ], "comments": [ { "id": 2641474, "author_name": "Yurnero", "author_url": "", "post_date": "2024-02-07T13:59:10.163000", "content": "CV 0.519 and LB 0.37 for a single 5-fold model
", "votes": 12, "replies": [ { "id": 2641544, "author_name": "Andrij", "author_url": "", "post_date": "2024-02-07T14:51:13.023000", "content": "Wow, If you can share: this result on spectrogram sizes [128,256,3] or larger.
", "votes": 1, "replies": [ { "id": 2641564, "author_name": "Yurnero", "author_url": "", "post_date": "2024-02-07T15:01:32.853000", "content": "Cant reveal anything for obvious reasons
", "votes": 3, "replies": [] }, { "id": 2641575, "author_name": "Andrij", "author_url": "", "post_date": "2024-02-07T15:05:27.380000", "content": "Ok, it's actually an incentive for me to work harder to understand what I'm doing wrong. Good luck
", "votes": 2, "replies": [] } ] }, { "id": 2642824, "author_name": "Priyanshu Chaudhary", "author_url": "", "post_date": "2024-02-08T12:51:39.153000", "content": "Are you evaluating your cv on non-overlapping spectrograms based samples?
", "votes": 4, "replies": [ { "id": 2647591, "author_name": "Yurnero", "author_url": "", "post_date": "2024-02-11T16:49:57.860000", "content": "sgkf on patient_id
Mine is 4 models: all Chris's
\nEffecintNetB2 Kaggle spectrograms 0.47
\nEffecintNetB2 EEG spectrograms 0.44
\nEffecintNetB2 both spectrograms 0.43
\nWavenet raw EEG 0.52
Weighted Ensemble 0.38
\nThe notebook code is available in this link
best single CV: 0.56, LB: 0.38
", "votes": 6, "replies": [ { "id": 2641175, "author_name": "Andrij", "author_url": "", "post_date": "2024-02-07T10:17:39.930000", "content": "Great result!
", "votes": 1, "replies": [] } ] }, { "id": 2652656, "author_name": "Sean R.B. Bearden, Ph.D.", "author_url": "", "post_date": "2024-02-14T23:01:27.910000", "content": "I have a model with CV 0.68 and LB 0.39.
\nI split the training data into two populations, then train on it in two stages (once for each population). I calculated the CV score on all OOF data from both populations, but predicted using only the final model. The first training population has a higher KL-Divergence for the CV than the second population. This was done purposely, as discussed in the notebook, to bias samples with total votes >= 10 with the goal of training on more granular vote distributions.
", "votes": 3, "replies": [ { "id": 2652962, "author_name": "Andrij", "author_url": "", "post_date": "2024-02-15T05:39:27.887000", "content": "I did a similar experiment yesterday but didn't finish, my result was 0.42
", "votes": 1, "replies": [] } ] }, { "id": 2641718, "author_name": "Andreas Bisiadis", "author_url": "", "post_date": "2024-02-07T16:23:29.480000", "content": "My case, using three models and ensembling them together:
\nEfficientNetB0: best CV score 0.55 and LB 0.42
\nEfficientNetB1: best CV score 0.53 and LB 0.46
\nResNet34d: best CV score 0.56 and LB 0.45
Weighted ensembler (inference): LB 0.41
", "votes": 3, "replies": [ { "id": 2641741, "author_name": "Andrij", "author_url": "", "post_date": "2024-02-07T16:35:19.117000", "content": "Yes, the ensemble will improve the results, but right now it's important for me to improve the single model as much as possible
", "votes": 3, "replies": [ { "id": 2641744, "author_name": "Andreas Bisiadis", "author_url": "", "post_date": "2024-02-07T16:38:12.410000", "content": "@aikhmelnytskyy
\nI don't fancy asking questions about how to improve my models, but… what image size are you using?
128,256.3 - This is the size of one spectrogram, I tried doubling the size of the spectrograms, but it doesn't seem to have any effect on CV and LB in my case
", "votes": 1, "replies": [] }, { "id": 2641814, "author_name": "Andreas Bisiadis", "author_url": "", "post_date": "2024-02-07T17:23:01.367000", "content": "I'm asking because I've been using 512x512 (when resizing). You think I should lower the size to 256x256?
", "votes": 2, "replies": [] }, { "id": 2641879, "author_name": "Andrij", "author_url": "", "post_date": "2024-02-07T18:19:39.757000", "content": "I think it's worth trying different sizes and seeing the results. By the way, don't forget that you can build an ensemble on images of different sizes, it works because the models focus on different features.
", "votes": 3, "replies": [] } ] } ] }, { "id": 2641209, "author_name": "Timmy Juicehouse", "author_url": "", "post_date": "2024-02-07T10:51:17.003000", "content": "single model at present
\nCV：0.591 LB：0.40
\nThere is under improvement. Good news is that CV score is stable in this competition.
", "votes": 3, "replies": [ { "id": 2641289, "author_name": "Andrij", "author_url": "", "post_date": "2024-02-07T11:53:49.677000", "content": "For me, it is a problem that LB is at the same time slightly worse than train loss and significantly better than CV. It looks like the model is overfitting, and the result on the private dataset will be poorly predicted, although the training CV may be affected by the fact that we have a different number of spectrograms and eegs: essentially, the spectrograms are repeated during training
", "votes": 2, "replies": [] } ] }, { "id": 2641061, "author_name": "Dracarys", "author_url": "", "post_date": "2024-02-07T08:50:00.470000", "content": "Mine is CV: 0.58, LB: 0.41
", "votes": 3, "replies": [ { "id": 2641068, "author_name": "Andrij", "author_url": "", "post_date": "2024-02-07T08:58:55.027000", "content": "Great, I just saw a similar discussion and from what I see people with worse CV's have better LB's than mine.
", "votes": 3, "replies": [] } ] }, { "id": 2641097, "author_name": "Haru", "author_url": "", "post_date": "2024-02-07T09:38:36.420000", "content": "In my case, CV:0.569 LB:0.43
\nemsemble: LB:0.40
The result is similar to mine, I tried to choose different ranges for eeg spectrograms during training, this approach obviously worsens LB, it definitely worsens if we take into account during training that some eegs in different ranges have different labels.
", "votes": 2, "replies": [ { "id": 2641210, "author_name": "Haru", "author_url": "", "post_date": "2024-02-07T10:52:32.807000", "content": "Really it it similar.
\nI tried to create different type of spectrograms from eeg(eg. Hz range, denoise, CWT), CV is improving but LB has not changed.
\nDid you try tree model and ensemble of tree model and NN model?
Yes, I also tried different type of spectrograms from eeg (eg. Hz range, denoise, CWT). In the end, I found an option for preparing spectrograms that improves CV, but does not affect LB… I've tried building separate models for each type of spectrogram, but haven't been able to get them to work yet
", "votes": 2, "replies": [] }, { "id": 2641286, "author_name": "Haru", "author_url": "", "post_date": "2024-02-07T11:53:20.057000", "content": "This is super similar situation.
\nMy CV was improved, but does not affect LB…
\nIt seems necessary to think of ways to improve the LB (Leaderboard) from different methods.
I just experienced this as well. A very nice CV increase by 0.05 and the LB went in the opposite direction 🤷♂️
", "votes": 5, "replies": [] }, { "id": 2645466, "author_name": "stefanoclss", "author_url": "", "post_date": "2024-02-10T09:03:53.500000", "content": "I haven't experienced it yet, can you explain what you did?
", "votes": 0, "replies": [] } ] } ] }, { "id": 2641949, "author_name": "stefanoclss", "author_url": "", "post_date": "2024-02-07T19:21:01.023000", "content": "I think the power lies in effectively ensembling your best deep learning models. But taking the mean of my predictions didn't improve my score.
", "votes": 2, "replies": [ { "id": 2641962, "author_name": "Andrij", "author_url": "", "post_date": "2024-02-07T19:32:56.680000", "content": "It is important that the models add new information to the ensemble. If you make an ensemble from similar models, or models built on similar features, they will give a smaller increase than models built on different pipelines.
", "votes": 4, "replies": [ { "id": 2641968, "author_name": "stefanoclss", "author_url": "", "post_date": "2024-02-07T19:39:44.987000", "content": "Hmm currently utilizing 4 different transfer learning models (densenet, exception,..), all scoring between 0.57 and 0.63. Do you think those are not able to catch very different kind of features?
", "votes": 2, "replies": [] }, { "id": 2641979, "author_name": "Andrij", "author_url": "", "post_date": "2024-02-07T19:52:07.797000", "content": "I don't know what the problem is, but I have two hypotheses: 1. It might be worth trying to build a weighted ensemble, one of the models might worsen your results. 2. You have already extracted all the information from your data, then the ensemble does not work (more precisely, it works worse). I mean, the problem might be in the preprocessing, to improve the results you need to feed the models with different data. I met a similar story a few months ago, at the end of the day the disappointment was great, and the reason was that I failed the preprocessing step. But I am not the leader in this competition and my hypotheses may be wrong
", "votes": 1, "replies": [] }, { "id": 2642008, "author_name": "stefanoclss", "author_url": "", "post_date": "2024-02-07T20:27:56.940000", "content": "Yes, my plan was to find an effective way of ensembling the models, maybe I can instead use a dense layer to predict the value or 6 regression models. Wanted to make sure to find an effective ensembling method before retraining my models with more augmentations or epochs.
", "votes": 1, "replies": [] }, { "id": 2642013, "author_name": "Andrij", "author_url": "", "post_date": "2024-02-07T20:40:18.853000", "content": "You can try, but keep in mind that with this procedure there's a risk of further overfitting the model
", "votes": 1, "replies": [] }, { "id": 2642038, "author_name": "stefanoclss", "author_url": "", "post_date": "2024-02-07T21:18:11.213000", "content": "Yes true, need to be careful with that but luckily there seems to be a reliable cv/lb correlation.
", "votes": 1, "replies": [] }, { "id": 2706372, "author_name": "stefanoclss", "author_url": "", "post_date": "2024-03-19T20:44:41.683000", "content": "@aikhmelnytskyy To go back to this discussion. The combination of different models only improved behind the kaggle digits. With a reliable cv. When doing the same by evaluating 1000 weighted ensembles next to the cv score the ensemble was way worse.
", "votes": 1, "replies": [] }, { "id": 2706374, "author_name": "stefanoclss", "author_url": "", "post_date": "2024-03-19T20:46:48.320000", "content": "Now I'm back to improving one model, but I think I'm to late for this if I look to the top scores.
", "votes": 1, "replies": [] }, { "id": 2706483, "author_name": "Andrij", "author_url": "", "post_date": "2024-03-19T22:33:22.883000", "content": "My best model shows lb 0.32 this is one model. Read the comments, all tops have single models at the level of lb 0.25-0.28. It's not about the ensemble, it's about something else
", "votes": 0, "replies": [] }, { "id": 2706488, "author_name": "stefanoclss", "author_url": "", "post_date": "2024-03-19T22:51:04.570000", "content": "If that's the case I'll have a look into optimizing the model or preprocessing the data. Got albumentations to work correctly and improve my cv score.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2641727, "author_name": "Rafael Zimmermann", "author_url": "", "post_date": "2024-02-07T16:29:03.980000", "content": "Mine is 0.61 in CV and 0.42 lb using pythorch, I stopped trying to improve the model because I think it's overfitting, thinking about different ways to improve, better EEGs might be the solution
", "votes": 2, "replies": [ { "id": 2641739, "author_name": "Andrij", "author_url": "", "post_date": "2024-02-07T16:33:15.590000", "content": "Yes, you are right, the problem here is not in the models, the problem is in preprocessing
", "votes": 3, "replies": [ { "id": 2641820, "author_name": "Rafael Zimmermann", "author_url": "", "post_date": "2024-02-07T17:26:35.757000", "content": "Perfect, you said everything that matters, I'm not an expert in this area, so I'm studying the subject a lot, it's quite challenging to say the least, I want to invest all my efforts in preprocessing
", "votes": 1, "replies": [] } ] } ] }, { "id": 2651665, "author_name": "ewo", "author_url": "", "post_date": "2024-02-14T10:08:56.470000", "content": "Hi, I'm new here on kaggle and i was wondering what 'CV' and 'LB' stands for, kind regards
", "votes": 0, "replies": [ { "id": 2651704, "author_name": "Andrij", "author_url": "", "post_date": "2024-02-14T10:33:06.650000", "content": "Hello! СV is the result of your cross-validation, in this case the average value of the metrics obtained on the validation data. LB is your indicator on the public test data, Do not forget that there are also three-week test data.
", "votes": 2, "replies": [ { "id": 2653142, "author_name": "ewo", "author_url": "", "post_date": "2024-02-15T08:34:30.883000", "content": "Ahh thanks you so much!
", "votes": 0, "replies": [] } ] } ] }, { "id": 2651280, "author_name": "vibuitruong", "author_url": "", "post_date": "2024-02-14T04:26:03.493000", "content": "Helpful!!!!
", "votes": 0, "replies": [] }, { "id": 2642838, "author_name": "", "author_url": "", "post_date": "2024-02-08T12:58:01.650000", "content": "", "votes": 0, "replies": [] }, { "id": 2642941, "author_name": "", "author_url": "", "post_date": "2024-02-08T14:12:51.197000", "content": "helpful , thanks
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2641031": "Hello everybody!\nI wonder who has the best models!\nHere are my best single models for 5 folds:\nModel based on Chris's pipeline, with modifications:\nCV 0.5418, LB 0.43\nModel based on a combination of my pipeline and Chris' pipeline, with modifications:\nCV 0.555, LB 0.44\nThe size of one spectrogram is 128,256,3, just like Chris's.\nGood luck to everyone and a peaceful sky above your head!", "2641474": "CV 0.519 and LB 0.37 for a single 5-fold model", "2649500": "Mine is 4 models: all Chris's\nEffecintNetB2 Kaggle spectrograms 0.47\nEffecintNetB2 EEG spectrograms 0.44\nEffecintNetB2 both spectrograms 0.43\nWavenet raw EEG 0.52\n\nWeighted Ensemble 0.38\nThe notebook code is available in [this link](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/469666)", "2641074": "best single CV: 0.56, LB: 0.38", "2652656": "I have a [model](https://www.kaggle.com/code/seanbearden/effnetb0-2-pop-model-train-twice-lb-0-39) with CV 0.68 and LB 0.39. \n\nI split the training data into two populations, then train on it in two stages (once for each population). I calculated the CV score on all OOF data from both populations, but predicted using only the final model. The first training population has a higher KL-Divergence for the CV than the second population. This was done purposely, as discussed in the notebook, to bias samples with total votes >= 10 with the goal of training on more granular vote distributions.", "2641718": "My case, using three models and ensembling them together:\nEfficientNetB0: best CV score 0.55 and LB 0.42\nEfficientNetB1: best CV score 0.53 and LB 0.46\nResNet34d: best CV score 0.56 and LB 0.45\n\nWeighted ensembler (inference): LB 0.41", "2641209": "**single model at present**\n\nCV：0.591 LB：0.40\n\nThere is under improvement. Good news is that CV score is stable in this competition.", "2641061": "Mine is CV: 0.58, LB: 0.41", "2641097": "In my case, CV:0.569 LB:0.43\nemsemble: LB:0.40", "2641949": "I think the power lies in effectively ensembling your best deep learning models. But taking the mean of my predictions didn't improve my score.", "2641727": "Mine is 0.61 in CV and 0.42 lb using pythorch, I stopped trying to improve the model because I think it's overfitting, thinking about different ways to improve, better EEGs might be the solution", "2651665": "Hi, I'm new here on kaggle and i was wondering what 'CV' and 'LB' stands for, kind regards ", "2651280": "Helpful!!!!", "2642838": "", "2642941": "helpful , thanks " } }