![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F761268%2Ff126aa616eceb9485d67b41aa1f5286a%2Fdownload.png?generation=1705008260627596&alt=media\")
\nsub Eegs sequences are of 50secs where as sub spectrogram sequences are 10 minute long
Spectrograms can have many annotations, for example spectogram_id=1254544437 has 440 annotations with a total spectrogram duration of 8851 seconds.
How to select the proper spectrogram window based on the spectrogram_label_offset_seconds?
1) How to determine the window size in seconds the annotation applies to?
\n2) Does the spectrogram_label_offset_seconds correspond to the center, start or end of the window?
In short, what strategy should be applied to select the correct spectrogram window?
", "messages": [ { "id": "2597744", "postDate": "01/11/2024 20:45:36", "content": "Spectrograms can have many annotations, for example spectogram_id=1254544437 has 440 annotations with a total spectrogram duration of 8851 seconds.
How to select the proper spectrogram window based on the spectrogram_label_offset_seconds?
1) How to determine the window size in seconds the annotation applies to?
\n2) Does the spectrogram_label_offset_seconds correspond to the center, start or end of the window?
In short, what strategy should be applied to select the correct spectrogram window?
", "rawMarkdown": "Spectrograms can have many annotations, for example `spectogram_id=1254544437` has 440 annotations with a total spectrogram duration of 8851 seconds.\n\nHow to select the proper spectrogram window based on the `spectrogram_label_offset_seconds`?\n\n1) How to determine the window size in seconds the annotation applies to?\n2) Does the `spectrogram_label_offset_seconds` correspond to the center, start or end of the window?\n\nIn short, what strategy should be applied to select the correct spectrogram window?", "votes": null }, { "id": "2597767", "postDate": "01/11/2024 21:26:01", "content": "@markwijkhuizen I didn't spend time on analysing spectrogram but I felt it is more similar to eeg_label_offset_seconds
\n
\nsub Eegs sequences are of 50secs where as sub spectrogram sequences are 10 minute long
Notebook - Eegs Target Analysis - Correct way to merge target
\n\n\nHow to determine the window size in seconds the annotation applies to?
\n
\n 10mins ( from Data tab )
\n Does the spectrogram_label_offset_seconds correspond to the center, start or end of the window?
\n Start ( similar to egg analysis based )
Check - Discussion - Correct way to merge targets for better understanding about sub-sequence
", "rawMarkdown": "markwijkhuizen I didn't spend time on analysing spectrogram but I felt it is more similar to eeg_label_offset_seconds \n\n\nsub Eegs sequences are of 50secs where as sub spectrogram sequences are 10 minute long\n\n[Notebook - Eegs Target Analysis - Correct way to merge target](https://www.kaggle.com/code/seshurajup/eegs-target-analysis-correct-way-to-merge-target)\n\n\n> How to determine the window size in seconds the annotation applies to?\n**10mins** ( from Data tab )\n> Does the spectrogram_label_offset_seconds correspond to the center, start or end of the window?\n**Start** ( similar to egg analysis based )\n\nCheck - [Discussion - Correct way to merge targets](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467127#2597451) for better understanding about sub-sequence", "votes": null }, { "id": "2597780", "postDate": "01/11/2024 21:49:44", "content": "Here is code to get the spectrogram:
\n train = pd.read_csv()\n = train.iloc[]\n spectrogram = pd.read_parquet(+str(.spectrogram_id)+)\n start = (.spectrogram_label_offset_seconds)\n %==: += \n end = + \n spectrogram = spectrogram.loc[(spectrogram.time>=)&(spectrogram.time<=)]\nAfter running this code you have the spectrogram for train row ROW in the dataframe named spectrogram. It has shape (300,401).
Understood [ spectrogram.time>=start, spectrogram.time<=end] including start and end bounds too
", "rawMarkdown": "cdeotte ~~any specific reason for window size as **598** instead **600** as 10 mins~~\n\nUnderstood **[ spectrogram.time>=start, spectrogram.time<=end]** including start and end bounds too", "votes": null }, { "id": "2598219", "postDate": "01/12/2024 08:04:03", "content": "Thanks for the clear code.
\nThe 10 minutes(600s) window would imply a lot of overlap in the labels, as shown below for spectogram_id=1254544437.
\nEven in the first couple of rows, a 600 seconds window would make dozens of different labels overlap.
Shouldn't we take the 10 seconds in the center, thus spectrogram_label_offset_seconds+295:spectrogram_label_offset_seconds+305 ?
\n\nThe expert annotators reviewed 50 second long EEG samples plus matched spectrograms covering 10 a minute window centered at the same time and labeled the central 10 seconds.
\n
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F4433335%2Ffc76ec4f519a6ce65a396fed426b43c1%2Fmhs_screenshot.png?generation=1705046326071606&alt=media\")
Thanks for the example. Note that the same results are provided by the same number of annotators. Here it is interesting how many annotators made decisions here in total, there were 16 or more of them. I see at least two main options here: First, the annotators themselves made different decisions in different areas, then we can assume that the identifiers are hidden in parts that do not intersect. The second option is that different annotators provided different results, that is, there were more than 16 annotators, then part of the data will be contradictory both in the training sample and in the public and private test sample. The second option means a shakeup (. I wonder what the organizers will say about it.
", "rawMarkdown": "Thanks for the example. Note that the same results are provided by the same number of annotators. Here it is interesting how many annotators made decisions here in total, there were 16 or more of them. I see at least two main options here: First, the annotators themselves made different decisions in different areas, then we can assume that the identifiers are hidden in parts that do not intersect. The second option is that different annotators provided different results, that is, there were more than 16 annotators, then part of the data will be contradictory both in the training sample and in the public and private test sample. The second option means a shakeup (. I wonder what the organizers will say about it.", "votes": null }, { "id": "2599248", "postDate": "01/12/2024 18:21:50", "content": "We can use the overlap as data augmentation. We can create a data loader which randomly crops 10 minute spectrogram together with corresponding 50 second eeg. Then we feed this into our model with the overall target.
", "rawMarkdown": "We can use the overlap as data augmentation. We can create a data loader which randomly crops 10 minute spectrogram together with corresponding 50 second eeg. Then we feed this into our model with the overall target.", "votes": null }, { "id": "2599299", "postDate": "01/12/2024 18:48:43", "content": "Yes, I was thinking of doing just that. By the way, I got LB 0.62 exclusively on full spectrograms, without any augmentation with minimal preprocessing. I think that the selection of the final models at the end of the competition will be of increased complexity (in comparison, of course). But it's too early to talk about it, now I'm more concerned about the issue of anomalous values in the data, how to find them and how to remove/replace them
", "rawMarkdown": "Yes, I was thinking of doing just that. By the way, I got LB 0.62 exclusively on full spectrograms, without any augmentation with minimal preprocessing. I think that the selection of the final models at the end of the competition will be of increased complexity (in comparison, of course). But it's too early to talk about it, now I'm more concerned about the issue of anomalous values in the data, how to find them and how to remove/replace them", "votes": null }, { "id": "2599335", "postDate": "01/12/2024 19:04:28", "content": "\n\nI got LB 0.62 exclusively on full spectrograms, without any augmentation with minimal preprocessing
\n
Thanks for sharing this. I was curious where you found signal. For the past two days, I have been building WaveNet, Transformer, and GRU models using exclusively EEG. However, my local CV is only the same CV score as using \"non-overlapping eeg id means\" (i.e. the public LB 0.97 technique). So my model is only learning means and not finding additional signal.
\nSo either my code has a bug, or my model isn't tuned correctly, or it is difficult to predict targets using only EEG. I will continue to improve my EEG models and I will try spectrogram models soon.
", "rawMarkdown": ">I got LB 0.62 exclusively on full spectrograms, without any augmentation with minimal preprocessing\n\nThanks for sharing this. I was curious where you found signal. For the past two days, I have been building WaveNet, Transformer, and GRU models using exclusively EEG. However, my local CV is only the same CV score as using \"non-overlapping eeg id means\" (i.e. the public LB 0.97 technique). So my model is only learning means and not finding additional signal.\n\nSo either my code has a bug, or my model isn't tuned correctly, or it is difficult to predict targets using only EEG. I will continue to improve my EEG models and I will try spectrogram models soon.", "votes": null }, { "id": "2599362", "postDate": "01/12/2024 19:12:52", "content": "For features, I tried many things including creating the \"double banana\" differences, i.e. Fp1-F7, F7-T3, etc (described here). I also tried raw features and groups from correlation denograms. I also tried various ways to create targets and process outliers.
", "rawMarkdown": "For features, I tried many things including creating the \"double banana\" differences, i.e. Fp1-F7, F7-T3, etc (described [here][1]). I also tried raw features and groups from correlation denograms. I also tried various ways to create targets and process outliers.\n\n[1]: https://www.learningeeg.com/montages-and-technical-components", "votes": null }, { "id": "2599394", "postDate": "01/12/2024 19:39:32", "content": "Yes, I tried to build the eeg model. The result was probably the same as yours, I even stopped training the model. Thanks for the information, you have already done a great deal of work!
", "rawMarkdown": "Yes, I tried to build the eeg model. The result was probably the same as yours, I even stopped training the model. Thanks for the information, you have already done a great deal of work!", "votes": null } ], "comments": [ { "id": 2597767, "author_name": "seshurajup", "author_url": "", "post_date": "01/11/2024 21:26:01", "content": "@markwijkhuizen I didn't spend time on analysing spectrogram but I felt it is more similar to eeg_label_offset_seconds
\n
\nsub Eegs sequences are of 50secs where as sub spectrogram sequences are 10 minute long
Notebook - Eegs Target Analysis - Correct way to merge target
\n\n\nHow to determine the window size in seconds the annotation applies to?
\n
\n 10mins ( from Data tab )
\n Does the spectrogram_label_offset_seconds correspond to the center, start or end of the window?
\n Start ( similar to egg analysis based )
Check - Discussion - Correct way to merge targets for better understanding about sub-sequence
", "votes": null, "replies": [] }, { "id": 2597780, "author_name": "cdeotte", "author_url": "", "post_date": "01/11/2024 21:49:44", "content": "Here is code to get the spectrogram:
\n train = pd.read_csv()\n = train.iloc[]\n spectrogram = pd.read_parquet(+str(.spectrogram_id)+)\n start = (.spectrogram_label_offset_seconds)\n %==: += \n end = + \n spectrogram = spectrogram.loc[(spectrogram.time>=)&(spectrogram.time<=)]\nAfter running this code you have the spectrogram for train row ROW in the dataframe named spectrogram. It has shape (300,401).
Understood [ spectrogram.time>=start, spectrogram.time<=end] including start and end bounds too
", "votes": null, "replies": [] }, { "id": 2598219, "author_name": "markwijkhuizen", "author_url": "", "post_date": "01/12/2024 08:04:03", "content": "Thanks for the clear code.
\nThe 10 minutes(600s) window would imply a lot of overlap in the labels, as shown below for spectogram_id=1254544437.
\nEven in the first couple of rows, a 600 seconds window would make dozens of different labels overlap.
Shouldn't we take the 10 seconds in the center, thus spectrogram_label_offset_seconds+295:spectrogram_label_offset_seconds+305 ?
\n\nThe expert annotators reviewed 50 second long EEG samples plus matched spectrograms covering 10 a minute window centered at the same time and labeled the central 10 seconds.
\n
Thanks for the example. Note that the same results are provided by the same number of annotators. Here it is interesting how many annotators made decisions here in total, there were 16 or more of them. I see at least two main options here: First, the annotators themselves made different decisions in different areas, then we can assume that the identifiers are hidden in parts that do not intersect. The second option is that different annotators provided different results, that is, there were more than 16 annotators, then part of the data will be contradictory both in the training sample and in the public and private test sample. The second option means a shakeup (. I wonder what the organizers will say about it.
", "votes": null, "replies": [] }, { "id": 2599248, "author_name": "cdeotte", "author_url": "", "post_date": "01/12/2024 18:21:50", "content": "We can use the overlap as data augmentation. We can create a data loader which randomly crops 10 minute spectrogram together with corresponding 50 second eeg. Then we feed this into our model with the overall target.
", "votes": null, "replies": [ { "id": 2599299, "author_name": "aikhmelnytskyy", "author_url": "", "post_date": "01/12/2024 18:48:43", "content": "Yes, I was thinking of doing just that. By the way, I got LB 0.62 exclusively on full spectrograms, without any augmentation with minimal preprocessing. I think that the selection of the final models at the end of the competition will be of increased complexity (in comparison, of course). But it's too early to talk about it, now I'm more concerned about the issue of anomalous values in the data, how to find them and how to remove/replace them
", "votes": null, "replies": [ { "id": 2599335, "author_name": "cdeotte", "author_url": "", "post_date": "01/12/2024 19:04:28", "content": "\n\nI got LB 0.62 exclusively on full spectrograms, without any augmentation with minimal preprocessing
\n
Thanks for sharing this. I was curious where you found signal. For the past two days, I have been building WaveNet, Transformer, and GRU models using exclusively EEG. However, my local CV is only the same CV score as using \"non-overlapping eeg id means\" (i.e. the public LB 0.97 technique). So my model is only learning means and not finding additional signal.
\nSo either my code has a bug, or my model isn't tuned correctly, or it is difficult to predict targets using only EEG. I will continue to improve my EEG models and I will try spectrogram models soon.
", "votes": null, "replies": [] }, { "id": 2599362, "author_name": "cdeotte", "author_url": "", "post_date": "01/12/2024 19:12:52", "content": "For features, I tried many things including creating the \"double banana\" differences, i.e. Fp1-F7, F7-T3, etc (described here). I also tried raw features and groups from correlation denograms. I also tried various ways to create targets and process outliers.
", "votes": null, "replies": [ { "id": 2599394, "author_name": "aikhmelnytskyy", "author_url": "", "post_date": "01/12/2024 19:39:32", "content": "Yes, I tried to build the eeg model. The result was probably the same as yours, I even stopped training the model. Thanks for the information, you have already done a great deal of work!
", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "2597744": "Spectrograms can have many annotations, for example `spectogram_id=1254544437` has 440 annotations with a total spectrogram duration of 8851 seconds.\n\nHow to select the proper spectrogram window based on the `spectrogram_label_offset_seconds`?\n\n1) How to determine the window size in seconds the annotation applies to?\n2) Does the `spectrogram_label_offset_seconds` correspond to the center, start or end of the window?\n\nIn short, what strategy should be applied to select the correct spectrogram window?", "2597767": "markwijkhuizen I didn't spend time on analysing spectrogram but I felt it is more similar to eeg_label_offset_seconds \n\n\nsub Eegs sequences are of 50secs where as sub spectrogram sequences are 10 minute long\n\n[Notebook - Eegs Target Analysis - Correct way to merge target](https://www.kaggle.com/code/seshurajup/eegs-target-analysis-correct-way-to-merge-target)\n\n\n> How to determine the window size in seconds the annotation applies to?\n**10mins** ( from Data tab )\n> Does the spectrogram_label_offset_seconds correspond to the center, start or end of the window?\n**Start** ( similar to egg analysis based )\n\nCheck - [Discussion - Correct way to merge targets](https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467127#2597451) for better understanding about sub-sequence", "2597780": "Here is code to get the spectrogram:\n\n train = pd.read_csv('train.csv')\n row = train.iloc[ROW]\n spectrogram = pd.read_parquet(PATH+str(row.spectrogram_id)+'.parquet')\n start = int(row.spectrogram_label_offset_seconds)\n if start%2==0: start += 1\n end = start + 598\n spectrogram = spectrogram.loc[(spectrogram.time>=start)&(spectrogram.time<=end)]\n\nAfter running this code you have the spectrogram for train row `ROW` in the dataframe named `spectrogram`. It has shape `(300,401)`.", "2597783": "cdeotte ~~any specific reason for window size as **598** instead **600** as 10 mins~~\n\nUnderstood **[ spectrogram.time>=start, spectrogram.time<=end]** including start and end bounds too", "2598219": "Thanks for the clear code.\n\nThe 10 minutes(600s) window would imply a lot of overlap in the labels, as shown below for `spectogram_id=1254544437`.\nEven in the first couple of rows, a 600 seconds window would make dozens of different labels overlap.\n\nShouldn't we take the 10 seconds in the center, thus `spectrogram_label_offset_seconds+295:spectrogram_label_offset_seconds+305` ?\n\n>The expert annotators reviewed 50 second long EEG samples plus matched spectrograms covering 10 a minute window centered at the same time and labeled the central 10 seconds.\n\n", "2599242": "Thanks for the example. Note that the same results are provided by the same number of annotators. Here it is interesting how many annotators made decisions here in total, there were 16 or more of them. I see at least two main options here: First, the annotators themselves made different decisions in different areas, then we can assume that the identifiers are hidden in parts that do not intersect. The second option is that different annotators provided different results, that is, there were more than 16 annotators, then part of the data will be contradictory both in the training sample and in the public and private test sample. The second option means a shakeup (. I wonder what the organizers will say about it.", "2599248": "We can use the overlap as data augmentation. We can create a data loader which randomly crops 10 minute spectrogram together with corresponding 50 second eeg. Then we feed this into our model with the overall target.", "2599299": "Yes, I was thinking of doing just that. By the way, I got LB 0.62 exclusively on full spectrograms, without any augmentation with minimal preprocessing. I think that the selection of the final models at the end of the competition will be of increased complexity (in comparison, of course). But it's too early to talk about it, now I'm more concerned about the issue of anomalous values in the data, how to find them and how to remove/replace them", "2599335": ">I got LB 0.62 exclusively on full spectrograms, without any augmentation with minimal preprocessing\n\nThanks for sharing this. I was curious where you found signal. For the past two days, I have been building WaveNet, Transformer, and GRU models using exclusively EEG. However, my local CV is only the same CV score as using \"non-overlapping eeg id means\" (i.e. the public LB 0.97 technique). So my model is only learning means and not finding additional signal.\n\nSo either my code has a bug, or my model isn't tuned correctly, or it is difficult to predict targets using only EEG. I will continue to improve my EEG models and I will try spectrogram models soon.", "2599362": "For features, I tried many things including creating the \"double banana\" differences, i.e. Fp1-F7, F7-T3, etc (described [here][1]). I also tried raw features and groups from correlation denograms. I also tried various ways to create targets and process outliers.\n\n[1]: https://www.learningeeg.com/montages-and-technical-components", "2599394": "Yes, I tried to build the eeg model. The result was probably the same as yours, I even stopped training the model. Thanks for the information, you have already done a great deal of work!" }, "source": "meta" }