I originally got little gain from augmentations and started to double their usefulness for this competition. After some further experimentation, this is absolutely not the case. I thought I could share some of my research results here.
\nWhat worked:
\nWhat didn't work:
\nConclusion: It may be due to a knowledge gap in EEG and spectrogram data but I was really surprised by some of the results. I really expected value in horizontal flips and none with vertical flips. However, after extensive testing that just wasn't the case. I also believe some are focusing too much on LB, which is common of course. So I would like to remind people to pay attention to and trust the CV a bit!
", "messages": [ { "id": "2668827", "postDate": "02/25/2024 23:59:43", "content": "I originally got little gain from augmentations and started to double their usefulness for this competition. After some further experimentation, this is absolutely not the case. I thought I could share some of my research results here.
\nWhat worked:
\nWhat didn't work:
\nConclusion: It may be due to a knowledge gap in EEG and spectrogram data but I was really surprised by some of the results. I really expected value in horizontal flips and none with vertical flips. However, after extensive testing that just wasn't the case. I also believe some are focusing too much on LB, which is common of course. So I would like to remind people to pay attention to and trust the CV a bit!
", "rawMarkdown": "I originally got little gain from augmentations and started to double their usefulness for this competition. After some further experimentation, this is absolutely not the case. I thought I could share some of my research results here. \n\nWhat worked:\n* A few augmentations that worked was the mentioned XYMask which is described in detail with an example notebook here\nhttps://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477825\n* Vertical flips\n\nWhat didn't work:\n* Horizontal flips\n* Brightness contrast\n* rotation\n* cropping\n\nConclusion: It may be due to a knowledge gap in EEG and spectrogram data but I was really surprised by some of the results. I really expected value in horizontal flips and none with vertical flips. However, after extensive testing that just wasn't the case. I also believe some are focusing too much on LB, which is common of course. So I would like to remind people to pay attention to and trust the CV a bit!", "votes": null }, { "id": "2668856", "postDate": "02/26/2024 00:37:07", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!", "votes": null }, { "id": "2668906", "postDate": "02/26/2024 01:33:01", "content": "Sure! I used some augmentation and improved my CV score, but my LB score unchanged.
", "rawMarkdown": "Sure! I used some augmentation and improved my CV score, but my LB score unchanged.", "votes": null }, { "id": "2668912", "postDate": "02/26/2024 02:09:30", "content": "Yes I have a feeling LB will only mean so much in this competition I have suspicions that many of the top LB scores are using data that is not available in test. And usually in competition CV is king
", "rawMarkdown": "Yes I have a feeling LB will only mean so much in this competition I have suspicions that many of the top LB scores are using data that is not available in test. And usually in competition CV is king", "votes": null }, { "id": "2669595", "postDate": "02/26/2024 11:07:42", "content": "I am surprised about vertical flips. It would change the place of every frequency of spectrogram. Some of the patterns are frequency specific (lateralized rhythmic delta activity (LRDA), generalized rhythmic delta activity (GRDA). My intuition is that this would worsen the classification of these patterns.
", "rawMarkdown": "I am surprised about vertical flips. It would change the place of every frequency of spectrogram. Some of the patterns are frequency specific (lateralized rhythmic **delta** activity (LRDA), generalized rhythmic **delta** activity (GRDA). My intuition is that this would worsen the classification of these patterns.", "votes": null }, { "id": "2670307", "postDate": "02/26/2024 19:04:08", "content": "I think the same way on this, perhaps it’s overfitting but it does improve my CV score so it’s tough to tell if it is legitimate or not
", "rawMarkdown": "I think the same way on this, perhaps it’s overfitting but it does improve my CV score so it’s tough to tell if it is legitimate or not", "votes": null }, { "id": "2670370", "postDate": "02/26/2024 20:16:16", "content": "XYMasking is working well for me, but it's quite sensitive to parameter choices, I find that generally multiple smaller masks work better than a few larger ones.
", "rawMarkdown": "XYMasking is working well for me, but it's quite sensitive to parameter choices, I find that generally multiple smaller masks work better than a few larger ones.", "votes": null }, { "id": "2671166", "postDate": "02/27/2024 11:10:59", "content": "It seems that finetuning on samples with a higher number of votes will boost lb but will make cv worse, this what probably explains the jump in the two stage approach (0.4* -> 0.36) .
\nThis won't be overfitting if the private test has the same distribution as the public one.
\nI can think of a good reason for this to be the case : simply the organisers thought that samples with more votes are more trusthworthy and they should be used in both public and private test.
\nUnfortunately, if this is true, we should redefine our cv or finetune all our models
what are your experiences on mixup? I have seen both positive and negative responses about mixup, it didn't work for me. can anyone share their experiences with mixup?
", "rawMarkdown": "what are your experiences on mixup? I have seen both positive and negative responses about mixup, it didn't work for me. can anyone share their experiences with mixup?", "votes": null }, { "id": "2676033", "postDate": "03/01/2024 08:51:35", "content": "For me, my CV has boost ~0.02 when using mix up, but LB has not changes
", "rawMarkdown": "For me, my CV has boost ~0.02 when using mix up, but LB has not changes", "votes": null }, { "id": "2676150", "postDate": "03/01/2024 10:12:30", "content": "Thank you, same with me, my LB had no changes with mixup so far, but I am trying some few tricks with mixup after that I will move on to other augs
", "rawMarkdown": "Thank you, same with me, my LB had no changes with mixup so far, but I am trying some few tricks with mixup after that I will move on to other augs", "votes": null }, { "id": "2676311", "postDate": "03/01/2024 12:12:22", "content": "My experience:
\nIt's a true @cody11null, various transpose techniques making key to proper argumentation and to good CV score.
", "rawMarkdown": "It's a true @cody11null, various transpose techniques making key to proper argumentation and to good CV score.", "votes": null }, { "id": "2688342", "postDate": "03/09/2024 07:06:04", "content": "Hello, I have two inquiries:
\nCould you explain the integration of both EEG data and spectrogram into the processes of model training?
Is it imperative to apply filtering, denoising, or normalization techniques to the provided readings?
Thanks! Interesting to know. I should have discovered this post before uploading my data augmentation package haha
", "rawMarkdown": "Thanks! Interesting to know. I should have discovered this post before uploading my data augmentation package haha", "votes": null } ], "comments": [ { "id": 2668856, "author_name": "akulvaishnavi", "author_url": "", "post_date": "02/26/2024 00:37:07", "content": "Thanks for sharing!
", "votes": null, "replies": [] }, { "id": 2668906, "author_name": "sweetyheehee", "author_url": "", "post_date": "02/26/2024 01:33:01", "content": "Sure! I used some augmentation and improved my CV score, but my LB score unchanged.
", "votes": null, "replies": [ { "id": 2668912, "author_name": "cody11null", "author_url": "", "post_date": "02/26/2024 02:09:30", "content": "Yes I have a feeling LB will only mean so much in this competition I have suspicions that many of the top LB scores are using data that is not available in test. And usually in competition CV is king
", "votes": null, "replies": [ { "id": 2671166, "author_name": "ahmedelfazouan", "author_url": "", "post_date": "02/27/2024 11:10:59", "content": "It seems that finetuning on samples with a higher number of votes will boost lb but will make cv worse, this what probably explains the jump in the two stage approach (0.4* -> 0.36) .
\nThis won't be overfitting if the private test has the same distribution as the public one.
\nI can think of a good reason for this to be the case : simply the organisers thought that samples with more votes are more trusthworthy and they should be used in both public and private test.
\nUnfortunately, if this is true, we should redefine our cv or finetune all our models
I am surprised about vertical flips. It would change the place of every frequency of spectrogram. Some of the patterns are frequency specific (lateralized rhythmic delta activity (LRDA), generalized rhythmic delta activity (GRDA). My intuition is that this would worsen the classification of these patterns.
", "votes": null, "replies": [ { "id": 2670307, "author_name": "cody11null", "author_url": "", "post_date": "02/26/2024 19:04:08", "content": "I think the same way on this, perhaps it’s overfitting but it does improve my CV score so it’s tough to tell if it is legitimate or not
", "votes": null, "replies": [] } ] }, { "id": 2670370, "author_name": "kionce", "author_url": "", "post_date": "02/26/2024 20:16:16", "content": "XYMasking is working well for me, but it's quite sensitive to parameter choices, I find that generally multiple smaller masks work better than a few larger ones.
", "votes": null, "replies": [] }, { "id": 2675822, "author_name": "arunsensei", "author_url": "", "post_date": "03/01/2024 05:53:50", "content": "what are your experiences on mixup? I have seen both positive and negative responses about mixup, it didn't work for me. can anyone share their experiences with mixup?
", "votes": null, "replies": [ { "id": 2676033, "author_name": "haruki741", "author_url": "", "post_date": "03/01/2024 08:51:35", "content": "For me, my CV has boost ~0.02 when using mix up, but LB has not changes
", "votes": null, "replies": [ { "id": 2676150, "author_name": "arunsensei", "author_url": "", "post_date": "03/01/2024 10:12:30", "content": "Thank you, same with me, my LB had no changes with mixup so far, but I am trying some few tricks with mixup after that I will move on to other augs
", "votes": null, "replies": [] } ] }, { "id": 2676311, "author_name": "symz00", "author_url": "", "post_date": "03/01/2024 12:12:22", "content": "My experience:
\nIt's a true @cody11null, various transpose techniques making key to proper argumentation and to good CV score.
", "votes": null, "replies": [] }, { "id": 2688342, "author_name": "farwa99", "author_url": "", "post_date": "03/09/2024 07:06:04", "content": "Hello, I have two inquiries:
\nCould you explain the integration of both EEG data and spectrogram into the processes of model training?
Is it imperative to apply filtering, denoising, or normalization techniques to the provided readings?
Thanks! Interesting to know. I should have discovered this post before uploading my data augmentation package haha
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2668827": "I originally got little gain from augmentations and started to double their usefulness for this competition. After some further experimentation, this is absolutely not the case. I thought I could share some of my research results here. \n\nWhat worked:\n* A few augmentations that worked was the mentioned XYMask which is described in detail with an example notebook here\nhttps://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/477825\n* Vertical flips\n\nWhat didn't work:\n* Horizontal flips\n* Brightness contrast\n* rotation\n* cropping\n\nConclusion: It may be due to a knowledge gap in EEG and spectrogram data but I was really surprised by some of the results. I really expected value in horizontal flips and none with vertical flips. However, after extensive testing that just wasn't the case. I also believe some are focusing too much on LB, which is common of course. So I would like to remind people to pay attention to and trust the CV a bit!", "2668856": "Thanks for sharing!", "2668906": "Sure! I used some augmentation and improved my CV score, but my LB score unchanged.", "2668912": "Yes I have a feeling LB will only mean so much in this competition I have suspicions that many of the top LB scores are using data that is not available in test. And usually in competition CV is king", "2669595": "I am surprised about vertical flips. It would change the place of every frequency of spectrogram. Some of the patterns are frequency specific (lateralized rhythmic **delta** activity (LRDA), generalized rhythmic **delta** activity (GRDA). My intuition is that this would worsen the classification of these patterns.", "2670307": "I think the same way on this, perhaps it’s overfitting but it does improve my CV score so it’s tough to tell if it is legitimate or not", "2670370": "XYMasking is working well for me, but it's quite sensitive to parameter choices, I find that generally multiple smaller masks work better than a few larger ones.", "2671166": "It seems that finetuning on samples with a higher number of votes will boost lb but will make cv worse, this what probably explains the jump in the two stage approach (0.4* -> 0.36) .\nThis won't be overfitting if the private test has the same distribution as the public one.\nI can think of a good reason for this to be the case : simply the organisers thought that samples with more votes are more trusthworthy and they should be used in both public and private test.\nUnfortunately, if this is true, we should redefine our cv or finetune all our models", "2675822": "what are your experiences on mixup? I have seen both positive and negative responses about mixup, it didn't work for me. can anyone share their experiences with mixup?", "2676033": "For me, my CV has boost ~0.02 when using mix up, but LB has not changes", "2676150": "Thank you, same with me, my LB had no changes with mixup so far, but I am trying some few tricks with mixup after that I will move on to other augs", "2676311": "My experience:\n- The parameters (alpha and probability) should be tuned. Optimal parameters seem to differ between kaggle spectrograms and eeg spectrograms. \n- Using different augmentation methods together with mixup possibly make mixup effects invisible. For example, when I used both horizontal-flip and mixup, which can reduce LB 0.01 respectively, I got only 0.01 reduction (not 0.01+0.01=0.02). I tried to tune probabilities, but I couldn't find the way to make both methods effective. \n - Note: My experiments were based on kaggle spectrograms. An effective combination of mixup and other methods is reported in a different situation (https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/480674), so additional experiments still seem to be valuable.\n- Mixup effect was 0.02 at most.", "2677534": "It's a true @cody11null, various transpose techniques making key to proper argumentation and to good CV score.", "2688342": "Hello, I have two inquiries:\n\n1. Could you explain the integration of both EEG data and spectrogram into the processes of model training?\n\n2. Is it imperative to apply filtering, denoising, or normalization techniques to the provided readings?", "3159311": "Thanks! Interesting to know. I should have discovered this post before uploading my data augmentation package haha" }, "source": "meta" }