\n
![](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Feb-2024/ml-dl.png\")
This competition is a great competition to use both Machine Learning (ML) and Deep Learning (DL). We can use insights from our machine learning models to improve our deep learning models. And we can use insights from our deep learning models to improve our machine learning models. This competition's winning solution will most likely be an ensemble of machine learning and deep learning!
\nIt is interesting to note that the classifier in both machine learning and deep learning is the same. With binary cross entropy loss, both are logistic regression. In other scenarios, both are just ML models. The two approaches differ in how features are engineered.
\nFor machine learning (like SVC, KNN, LogReg, GBT), the human converts the raw features into useful \"smart\" features via feature engineering. For deep learning (like CNN, RNN, Transformer), deep hidden layers create features from the raw features.
\nIn both methods, the \"smart\" features are inputted into a final model to perform classification. And both final classification models are basically the same.
\n
For machine learning, we can view feature importance and/or feature weights easily (like GBT feature importance or LogReg coefficients). Thus it is easy to see what the machine learning model is doing. For deep learning, it is more difficult to understand what features the model is making. One technique to locate where the deep learning model is creating features from is called Grad Cam
\nWith grad cam, given a specific OOF (out of fold) train sample, we can view both a model's prediction and where it looked to make this prediction. In the plots below we display the image that was fed into our image model, in my popular EfficientNet starter notebook here, there are 8 spectrograms that have been tiled into one 1 input image.
\nOn the left we have the 4 Kaggle spectrograms where each is 10 minutes long. Each represents one of the 4 montages LL, RL, LP, RP. (Montages explained here) On the right, we have the 4 EEG spectrograms where each is 50 seconds long. The EEG spectrograms are made from the Magic Formula here. The spectrograms are each 128x256x1, so the final concatenation is 512x512x1.
\n![](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Feb-2024/key2.png\")
Here is an example for a train sample with ground truth 76% Seizure and 24% LPD. The model predicts 88% Seizure and 6.5% LPD. Below we see where the model looked to make this decision. The left image is just the Grad Cam. Larger values are shown in yellow and indicate where the model pays attention to more (when making this specific prediction from this specific input image).
\nThe middle image is the contours of the Grad Cam's 10% largest values superimposed over the image that we fed into our model. The right image is also Grad Cam contour superimposed over image but we add an emboss filter to the image to make the details more visible to humans. We observe
\n![](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Feb-2024/seizure.png\")
Here is an example of ground truth LPD at 100%. The model predicted LPD at 93%.
\n![](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Feb-2024/lpd.png\")
Here is an example of ground truth LRDA at 100%. The model predicted LRDA at 68%.
\n![](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Feb-2024/lrda.png\")
Here is an example of ground truth GRDA at 100%. The model predicted GRDA at 79%.
\n![](\"https://raw.githubusercontent.com/cdeotte/Kaggle_Images/main/Feb-2024/grda.png\")
I published a starter notebook here which loads the trained models from my EfficientNet starter notebook here and performs Grad Cam
", "messages": [ { "id": 2633286, "postDate": "2024-02-03T00:59:32.163Z", "content": "This competition is a great competition to use both Machine Learning (ML) and Deep Learning (DL). We can use insights from our machine learning models to improve our deep learning models. And we can use insights from our deep learning models to improve our machine learning models. This competition's winning solution will most likely be an ensemble of machine learning and deep learning!
\nIt is interesting to note that the classifier in both machine learning and deep learning is the same. With binary cross entropy loss, both are logistic regression. In other scenarios, both are just ML models. The two approaches differ in how features are engineered.
\nFor machine learning (like SVC, KNN, LogReg, GBT), the human converts the raw features into useful \"smart\" features via feature engineering. For deep learning (like CNN, RNN, Transformer), deep hidden layers create features from the raw features.
\nIn both methods, the \"smart\" features are inputted into a final model to perform classification. And both final classification models are basically the same.
\n
For machine learning, we can view feature importance and/or feature weights easily (like GBT feature importance or LogReg coefficients). Thus it is easy to see what the machine learning model is doing. For deep learning, it is more difficult to understand what features the model is making. One technique to locate where the deep learning model is creating features from is called Grad Cam
\nWith grad cam, given a specific OOF (out of fold) train sample, we can view both a model's prediction and where it looked to make this prediction. In the plots below we display the image that was fed into our image model, in my popular EfficientNet starter notebook here, there are 8 spectrograms that have been tiled into one 1 input image.
\nOn the left we have the 4 Kaggle spectrograms where each is 10 minutes long. Each represents one of the 4 montages LL, RL, LP, RP. (Montages explained here) On the right, we have the 4 EEG spectrograms where each is 50 seconds long. The EEG spectrograms are made from the Magic Formula here. The spectrograms are each 128x256x1, so the final concatenation is 512x512x1.
\n
Here is an example for a train sample with ground truth 76% Seizure and 24% LPD. The model predicts 88% Seizure and 6.5% LPD. Below we see where the model looked to make this decision. The left image is just the Grad Cam. Larger values are shown in yellow and indicate where the model pays attention to more (when making this specific prediction from this specific input image).
\nThe middle image is the contours of the Grad Cam's 10% largest values superimposed over the image that we fed into our model. The right image is also Grad Cam contour superimposed over image but we add an emboss filter to the image to make the details more visible to humans. We observe
\nHere is an example of ground truth LPD at 100%. The model predicted LPD at 93%.
\nHere is an example of ground truth LRDA at 100%. The model predicted LRDA at 68%.
\nHere is an example of ground truth GRDA at 100%. The model predicted GRDA at 79%.
\nI published a starter notebook here which loads the trained models from my EfficientNet starter notebook here and performs Grad Cam
", "rawMarkdown": "This competition is a great competition to use both Machine Learning (ML) and Deep Learning (DL). We can use insights from our machine learning models to improve our deep learning models. And we can use insights from our deep learning models to improve our machine learning models. This competition's winning solution will most likely be an ensemble of machine learning and deep learning!\n\n# Machine Learning versus Deep Learning\nIt is interesting to note that the classifier in both machine learning and deep learning is the same. With binary cross entropy loss, both are logistic regression. In other scenarios, both are just ML models. The two approaches differ in how features are engineered. \n\nFor machine learning (like SVC, KNN, LogReg, GBT), the human converts the raw features into useful \"smart\" features via feature engineering. For deep learning (like CNN, RNN, Transformer), deep hidden layers create features from the raw features.\n\nIn both methods, the \"smart\" features are inputted into a final model to perform classification. And both final classification models are basically the same.\n\n\n# Explainability\nFor machine learning, we can view feature importance and/or feature weights easily (like GBT feature importance or LogReg coefficients). Thus it is easy to see what the machine learning model is doing. For deep learning, it is more difficult to understand what features the model is making. One technique to locate where the deep learning model is creating features from is called Grad Cam\n\n# Grad Cam\nWith grad cam, given a specific OOF (out of fold) train sample, we can view both a model's prediction and where it looked to make this prediction. In the plots below we display the image that was fed into our image model, in my popular EfficientNet starter notebook [here][4], there are 8 spectrograms that have been tiled into one 1 input image. \n\nOn the left we have the 4 Kaggle spectrograms where each is 10 minutes long. Each represents one of the 4 montages LL, RL, LP, RP. (Montages explained [here][1]) On the right, we have the 4 EEG spectrograms where each is 50 seconds long. The EEG spectrograms are made from the Magic Formula [here][2]. The spectrograms are each `128x256x1`, so the final concatenation is `512x512x1`.\n\n## Seizure\nHere is an example for a train sample with ground truth 76% Seizure and 24% LPD. The model predicts 88% Seizure and 6.5% LPD. Below we see where the model looked to make this decision. The left image is just the Grad Cam. Larger values are shown in yellow and indicate where the model pays attention to more (when making this specific prediction from this specific input image).\n\nThe middle image is the contours of the Grad Cam's 10% largest values superimposed over the image that we fed into our model. The right image is also Grad Cam contour superimposed over image but we add an emboss filter to the image to make the details more visible to humans. We observe\n* **Observation:** The model looked at the middle 10 seconds of the RL and RP montage spectograms to decide Seizure\n\n\n## LPD\nHere is an example of ground truth LPD at 100%. The model predicted LPD at 93%. \n* **Observation** The model looked at the beginning and ending of the 50 second spectrogram (but not directly in the middle) to decide LPD.\n\n\n## LRDA\nHere is an example of ground truth LRDA at 100%. The model predicted LRDA at 68%.\n* **Observation** The model looked at the center 40 seconds (not just middle 10 seconds) to decide LRDA\n\n\n## GRDA\nHere is an example of ground truth GRDA at 100%. The model predicted GRDA at 79%.\n* **Observation** The model looked at many spots and mostly beginning and end to determine GRDA\n\n\n# Starter Notebook\nI published a starter notebook [here][3] which loads the trained models from my EfficientNet starter notebook [here][4] and performs Grad Cam\n\n[1]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467877\n[2]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/469760\n[3]: https://www.kaggle.com/code/cdeotte/grad-cam-what-is-important-in-spectrograms\n[4]: https://www.kaggle.com/code/cdeotte/efficientnetb0-starter-lb-0-43", "votes": 174 }, { "id": 2648271, "postDate": "2024-02-12T07:23:37.940Z", "content": "I noticed that LPD and LRDA cases , the model in this case didnt focus on Kaggle Spectrogram at all . Ideally it should have found the regions from both Kaggle and EEG spectrograms to focus on in case of LPD and LRDA . Is this correct observation ?
", "rawMarkdown": "I noticed that LPD and LRDA cases , the model in this case didnt focus on Kaggle Spectrogram at all . Ideally it should have found the regions from both Kaggle and EEG spectrograms to focus on in case of LPD and LRDA . Is this correct observation ?", "votes": 9, "replies": [ { "id": 2648955, "postDate": "2024-02-12T14:27:52.627Z", "content": "Nice observation. The best (and most reliable) way to make conclusions about the different targets is to create the Grad Cam for every OOF ground truth for a given target. Then take the average of all the Grad Cams images (then convert the result to contour lines if preferred). This will show us what is important for each target.
", "rawMarkdown": "Nice observation. The best (and most reliable) way to make conclusions about the different targets is to create the Grad Cam for **every** OOF ground truth for a given target. Then take the average of all the Grad Cams images (then convert the result to contour lines if preferred). This will show us what is important for each target.", "votes": 10 } ] }, { "id": 2637252, "postDate": "2024-02-05T15:55:56.580Z", "content": "Thanks for sharing your insights on Grad Cam and its application in understanding model decisions. For someone new to this technique, could you provide some practical tips or best practices on how to effectively use Grad Cam in a machine learning or deep learning project? Any specific scenarios or types of models where Grad Cam has proven particularly useful?
", "rawMarkdown": "Thanks for sharing your insights on Grad Cam and its application in understanding model decisions. For someone new to this technique, could you provide some practical tips or best practices on how to effectively use Grad Cam in a machine learning or deep learning project? Any specific scenarios or types of models where Grad Cam has proven particularly useful?", "votes": 4, "replies": [ { "id": 2637285, "postDate": "2024-02-05T16:19:47.790Z", "content": "Great question. Below are how to improve our deep learning model using Grad Cam insights. Alternatively, we can use Grad Cam to see where the deep learning model is looking and then convert this information into machine learning features for ML model such as CatBoost. (For example if model looks at 15Hz row of image, then make features about 15Hz for CatBoost).
\n==========
\nHere is one example (of Grad Cam improving deep learning) that happened in a previous competition. I describe it as cat or dog, but the actually comp was classifying something else.
\nImagine a competition where we must classify an image as cat or dog. Imagine that our CV score is 0.900 AUC but our LB score is worse 0.600. Grad Cam will help explain what is going on. If we apply Grad Cam to this, we see that every cat sits on a car, and every dog sits on a boat. When we look at Grad Cam, we see that the model ignores the animal and only looks at the car or boat. In the LB images, all animals sit on a plane. Hence the model learned to recognize the vehicle and not the animal. And then the vehicle changed in the test data.
\nWe solve the above scenario by training with cutmix or mixup. This starts to make new images where cats sit on boats and dogs sit on cars. After training a new model, we see that the model starts to look at the animal and ignore the vehicle. We submit to LB and get 0.900 AUC!
\n=====
\nThe above example is a case where Grad Cam shows us what the model is doing wrong. Below is an example where Grad Cam shows us what the model is doing correct.
Imagine that Grad Cam shows us that the model is frequently looking at 10Hz row. Or the model is looking at column of image at 15 seconds. We can then make new spectrograms that enlarge these areas of interest to give the model more info on the area it is looking. Furthermore if we notice that the model never looks somewhere in the image, we can remove the wasted space and put something more important in that space.
", "rawMarkdown": "Great question. Below are how to improve our deep learning model using Grad Cam insights. Alternatively, we can use Grad Cam to see where the deep learning model is looking and then convert this information into machine learning features for ML model such as CatBoost. (For example if model looks at 15Hz row of image, then make features about 15Hz for CatBoost).\n\n==========\n\nHere is one example (of Grad Cam improving deep learning) that happened in a previous competition. I describe it as cat or dog, but the actually comp was classifying something else.\n\nImagine a competition where we must classify an image as cat or dog. Imagine that our CV score is 0.900 AUC but our LB score is worse 0.600. Grad Cam will help explain what is going on. If we apply Grad Cam to this, we see that every cat sits on a car, and every dog sits on a boat. When we look at Grad Cam, we see that the model ignores the animal and only looks at the car or boat. In the LB images, all animals sit on a plane. Hence the model learned to recognize the vehicle and not the animal. And then the vehicle changed in the test data.\n\nWe solve the above scenario by training with cutmix or mixup. This starts to make new images where cats sit on boats and dogs sit on cars. After training a new model, we see that the model starts to look at the animal and ignore the vehicle. We submit to LB and get 0.900 AUC!\n\n=====\nThe above example is a case where Grad Cam shows us what the model is doing wrong. Below is an example where Grad Cam shows us what the model is doing correct.\n\nImagine that Grad Cam shows us that the model is frequently looking at 10Hz row. Or the model is looking at column of image at 15 seconds. We can then make new spectrograms that enlarge these areas of interest to give the model more info on the area it is looking. Furthermore if we notice that the model never looks somewhere in the image, we can remove the wasted space and put something more important in that space.", "votes": 32, "replies": [ { "id": 2646922, "postDate": "2024-02-11T09:40:06.937Z", "content": "Great answer and explanation, thanks Chris, opened up new horizons for me.
", "rawMarkdown": "Great answer and explanation, thanks Chris, opened up new horizons for me.", "votes": 1 } ] } ] }, { "id": 2721642, "postDate": "2024-03-29T06:07:06.160Z", "content": "Hi @cdeotte ,
\nThanks for the insight.
\nFar as I know, the intepretation of DL models has been controversy. There are many different ways with each has its good-looking cases. I am wondering why do you choose Grad Cam?
\nThanks
", "rawMarkdown": "Hi @cdeotte ,\n\nThanks for the insight.\n\nFar as I know, the intepretation of DL models has been controversy. There are many different ways with each has its good-looking cases. I am wondering why do you choose Grad Cam?\n\nThanks", "votes": 1, "replies": [ { "id": 2721778, "postDate": "2024-03-29T07:43:06.663Z", "content": "Grad Cam shows us what is important inside each spectrogram. This helps us create better spectrograms.
", "rawMarkdown": "Grad Cam shows us what is important inside each spectrogram. This helps us create better spectrograms.", "votes": 1 } ] }, { "id": 2680978, "postDate": "2024-03-04T12:17:54.037Z", "content": "Hello,
\nI am new to this competition and while reading the data description, I encountered a statement that I'm having trouble understanding: \"The expert annotators reviewed 50-second long EEG samples plus matched spectrograms covering a 10-minute window centered at the same time and labeled the central 10 seconds. Many of these samples overlapped and have been consolidated.\"
\nCould someone please explain what this means? I would greatly appreciate it. Thank you.
", "rawMarkdown": "Hello,\n\nI am new to this competition and while reading the data description, I encountered a statement that I'm having trouble understanding: \"The expert annotators reviewed 50-second long EEG samples plus matched spectrograms covering a 10-minute window centered at the same time and labeled the central 10 seconds. Many of these samples overlapped and have been consolidated.\"\n\nCould someone please explain what this means? I would greatly appreciate it. Thank you.", "votes": 1, "replies": [ { "id": 2681038, "postDate": "2024-03-04T13:06:18.610Z", "content": "Hi welcome. The data is explained in my discussion here
", "rawMarkdown": "Hi welcome. The data is explained in my discussion [here][1]\n\n[1]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/468010", "replies": [ { "id": 2682347, "postDate": "2024-03-05T09:00:28.017Z", "content": "Thanks alot for your assistance
", "rawMarkdown": "Thanks alot for your assistance", "votes": 1 } ] } ] }, { "id": 2640530, "postDate": "2024-02-07T01:14:32.243Z", "content": "I will soon start working on this competition, and i found it very useful in advance. I would also like to know if there are any other indispensable methods to inspect my model's inner workings (in a spectrogram context). Thank you very much
", "rawMarkdown": "I will soon start working on this competition, and i found it very useful in advance. I would also like to know if there are any other indispensable methods to inspect my model's inner workings (in a spectrogram context). Thank you very much", "votes": 1 }, { "id": 2638215, "postDate": "2024-02-06T06:22:26.300Z", "content": "good work explaining grad cam! would anyone care to share some light on how much computational resource this competition requires? I'm just starting on Kaggle, and I 'd like to pick a competition with relatively lower requirement on the computing side. Thanks!
", "rawMarkdown": "good work explaining grad cam! would anyone care to share some light on how much computational resource this competition requires? I'm just starting on Kaggle, and I 'd like to pick a competition with relatively lower requirement on the computing side. Thanks!", "votes": 1, "replies": [ { "id": 2638724, "postDate": "2024-02-06T12:59:43.913Z", "content": "This comp does not require too much compute. There are many starter notebooks that train in Kaggle notebooks and achieve good CV and LB score. Furthermore all starter notebooks can be improved to achieve better CV and LB scores using only Kaggle compute.
", "rawMarkdown": "This comp does not require too much compute. There are many starter notebooks that train in Kaggle notebooks and achieve good CV and LB score. Furthermore all starter notebooks can be improved to achieve better CV and LB scores using only Kaggle compute.", "votes": 2, "replies": [ { "id": 2638739, "postDate": "2024-02-06T13:09:17.517Z", "content": "The 30 hours GPU time are sufficient but I can imagine I would have been able to speed up the processing if I had access to more GPU time or better GPU's. Nevertheless I think surely you can get in the gold with just the kaggle GPU's but they'll be a bottleneck in the later stages of the competition imo.
", "rawMarkdown": "The 30 hours GPU time are sufficient but I can imagine I would have been able to speed up the processing if I had access to more GPU time or better GPU's. Nevertheless I think surely you can get in the gold with just the kaggle GPU's but they'll be a bottleneck in the later stages of the competition imo. ", "votes": 2 } ] } ] }, { "id": 2637368, "postDate": "2024-02-05T17:10:35.487Z", "content": "Thanks sir for sharing your views on Grad Cam
", "rawMarkdown": "Thanks sir for sharing your views on Grad Cam", "votes": 1 }, { "id": 2633436, "postDate": "2024-02-03T04:37:57.697Z", "content": "Thanks for sharing! for now, I've learned about Grad Cam😊
", "rawMarkdown": "Thanks for sharing! for now, I've learned about Grad Cam😊", "votes": 1 }, { "id": 2647269, "postDate": "2024-02-11T13:00:00.330Z", "content": "Thx, the feeling of learning a little new knowledge every day is really nice 😂
", "rawMarkdown": "Thx, the feeling of learning a little new knowledge every day is really nice 😂\n\n", "votes": 2 }, { "id": 2643731, "postDate": "2024-02-09T04:04:39.337Z", "content": "The difficulty in this competition is that too many raw features can be digged out. I'm struggling to find the ground truth of each pattern.
", "rawMarkdown": "The difficulty in this competition is that too many raw features can be digged out. I'm struggling to find the ground truth of each pattern. ", "votes": 2 }, { "id": 2634042, "postDate": "2024-02-03T13:06:57.510Z", "content": "Hi Chris, again a great learning experience with you, the grad cam will really help you a lot to create new and better spectrograms, thanks for sharing.
\nSpeaking of them, I have a doubt regarding personalized spectrograms, I noticed that you are using the frequency from 0 to 20, but apparently there seems to be an important region also in the area close to 59, was there any special reason for choosing 0 to 20, do you think that Is it worth creating spectrograms with other frequencies?
", "rawMarkdown": "Hi Chris, again a great learning experience with you, the grad cam will really help you a lot to create new and better spectrograms, thanks for sharing.\n\nSpeaking of them, I have a doubt regarding personalized spectrograms, I noticed that you are using the frequency from 0 to 20, but apparently there seems to be an important region also in the area close to 59, was there any special reason for choosing 0 to 20, do you think that Is it worth creating spectrograms with other frequencies?", "votes": 2, "replies": [ { "id": 2634235, "postDate": "2024-02-03T15:51:17.197Z", "content": "I did 0 to 20 because Kaggle provided spectrograms are 0 to 20. I'm not sure what the best range to do is. We need to try different ranges and evaluate CV score and LB score.
", "rawMarkdown": "I did 0 to 20 because Kaggle provided spectrograms are 0 to 20. I'm not sure what the best range to do is. We need to try different ranges and evaluate CV score and LB score.", "votes": 5, "replies": [ { "id": 2634344, "postDate": "2024-02-03T17:41:54.527Z", "content": "Chris, thanks for the answer, I also think the best way to do this is to do the test and test different types of ranges for the specs, I did an optimization using the MLP that you provided, your idea was fantastic, really using it to do The experiments help a lot, especially the fact that you don't need to use the GPU…
\nThe only problem I'm facing is that apparently my experiments are not 100% correlated with the code I'm using to submit using efficientnet_b2, even so, below is the frequency experiment I did using 40% of the training data.
\n-20Hz: ,\n-30Hz: ,\n-40Hz: ,\n-50Hz: ,\n-60Hz: ,\n-70Hz: ,\n-80Hz: \nI'm studying a new way of doing experiments with a small model more similar to efficientnet to try to have a greater correlation without the need to use a GPU.
\nStill studying, but the model below is looking promising.
():\n inp = tf.keras.Input(shape=input_shape)\n\n \n x = layers.Conv2D(, (, ), activation=)(inp)\n x = layers.MaxPooling2D((, ))(x)\n x = layers.Conv2D(, (, ), activation=)(x)\n x = layers.MaxPooling2D((, ))(x)\n x = layers.Conv2D(, (, ), activation=)(x)\n\n \n x = layers.Flatten() (x)\n x = layers.Dense(, activation=)(x)\n x = layers.Dense(num_classes, activation=)(x)\n\n \n model = tf.keras.Model(inputs=inp, outputs=x)\n opt = tf.keras.optimizers.Adam(learning_rate=)\n loss = tf.keras.losses.KLDivergence()\n\n model.(optimizer=opt, loss=loss, metrics=[])\n\n model\n59 Hz is probably just interference from AC power.
", "rawMarkdown": "59 Hz is probably just interference from AC power.", "votes": 2, "replies": [ { "id": 2645074, "postDate": "2024-02-09T23:23:15.743Z", "content": "Yes, it's true, after a while I discovered this, it really is a strong noise, it is possible to treat it using filters, such as high pass, low pass or notch, in my case it made an improvement by doing this.
", "rawMarkdown": "Yes, it's true, after a while I discovered this, it really is a strong noise, it is possible to treat it using filters, such as high pass, low pass or notch, in my case it made an improvement by doing this." } ] } ] }, { "id": 2633879, "postDate": "2024-02-03T10:42:24.203Z", "content": "One question in the spectrogram concatenation image, why Kaggle RL spectrogram concatented with EEG LP spectrogram not EEG LP spectrogram? And also for Kaggle LP spectrogram?
", "rawMarkdown": "One question in the spectrogram concatenation image, why Kaggle RL spectrogram concatented with EEG LP spectrogram not EEG LP spectrogram? And also for Kaggle LP spectrogram?", "votes": 2, "replies": [ { "id": 2633960, "postDate": "2024-02-03T11:59:44.013Z", "content": "That's just the way it is. My notebook that makes EEG spectrograms puts them in the order LL, LP, RL, RP. Then my EfficientNet notebook starter just concatenates them vertically as LL, LP, RL, RP. And Kaggle puts them in order LL, RL, LP, RP. In my EfficientNet starter, you can try rearranging so that my EEG spectrograms are also LL, RL, LP, RP. We can experiment to see if that improve CV score and LB score.
We could just randomly place them using augmentations. When I have published my ensemble I'll have a look if I can further improve the models like this.
", "rawMarkdown": "We could just randomly place them using augmentations. When I have published my ensemble I'll have a look if I can further improve the models like this.", "votes": 1 }, { "id": 2649138, "postDate": "2024-02-12T16:27:47.553Z", "content": "On my phone so not able to find all the links to host. But there are three suggested standard montages that are recommended to standardize the way eeg’s are looked at by the medical folks. Probably would be useful to the hosts if we used standard layouts.
", "rawMarkdown": "On my phone so not able to find all the links to host. But there are three suggested standard montages that are recommended to standardize the way eeg’s are looked at by the medical folks. Probably would be useful to the hosts if we used standard layouts. " } ] } ] }, { "id": 2633728, "postDate": "2024-02-03T08:45:30.783Z", "content": "I haven't looked that much into the metric of this competition yet in very detail. It compares our predicted distribution against the ground truth prediction distribution. But since these samples are not totally perfectly annotated is it possible that a better model performs worse on the LB?
", "rawMarkdown": "I haven't looked that much into the metric of this competition yet in very detail. It compares our predicted distribution against the ground truth prediction distribution. But since these samples are not totally perfectly annotated is it possible that a better model performs worse on the LB?", "votes": 2, "replies": [ { "id": 2633781, "postDate": "2024-02-03T09:21:46.230Z", "content": "It is possible, but unlikely
", "rawMarkdown": "It is possible, but unlikely", "votes": 1 } ] }, { "id": 2871885, "postDate": "2024-06-14T12:52:11.077Z", "content": "Great insight and starter notebooks. Thank you.
", "rawMarkdown": " Great insight and starter notebooks. Thank you." }, { "id": 2861872, "postDate": "2024-06-08T12:09:39.003Z", "content": "I would like to express late gratitude for this insight, it was very helpful during the competition.
", "rawMarkdown": "I would like to express late gratitude for this insight, it was very helpful during the competition." }, { "id": 2642798, "postDate": "2024-02-08T12:26:25.710Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 2637225, "postDate": "2024-02-05T15:51:45.600Z", "rawMarkdown": "", "votes": 2, "isDeleted": true }, { "id": 2642728, "postDate": "2024-02-08T11:11:05.837Z", "content": "Thank you for sharing!
", "rawMarkdown": "Thank you for sharing!", "votes": 1 }, { "id": 2640905, "postDate": "2024-02-07T06:35:43.423Z", "content": "thanks for sharing
", "rawMarkdown": "thanks for sharing", "votes": 1 }, { "id": 2636920, "postDate": "2024-02-05T11:15:57.457Z", "content": "thanks for sharing
", "rawMarkdown": "thanks for sharing", "votes": 1 }, { "id": 2635356, "postDate": "2024-02-04T10:50:43.843Z", "content": "thanks for sharing Chris sir.
", "rawMarkdown": "thanks for sharing Chris sir.\n \n", "votes": 1 } ], "comments": [ { "id": 2648271, "author_name": "Nirjhar Roy", "author_url": "", "post_date": "2024-02-12T07:23:37.940000", "content": "I noticed that LPD and LRDA cases , the model in this case didnt focus on Kaggle Spectrogram at all . Ideally it should have found the regions from both Kaggle and EEG spectrograms to focus on in case of LPD and LRDA . Is this correct observation ?
", "votes": 9, "replies": [ { "id": 2648955, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-02-12T14:27:52.627000", "content": "Nice observation. The best (and most reliable) way to make conclusions about the different targets is to create the Grad Cam for every OOF ground truth for a given target. Then take the average of all the Grad Cams images (then convert the result to contour lines if preferred). This will show us what is important for each target.
", "votes": 10, "replies": [] } ] }, { "id": 2637252, "author_name": "Kaushal Surana", "author_url": "", "post_date": "2024-02-05T15:55:56.580000", "content": "Thanks for sharing your insights on Grad Cam and its application in understanding model decisions. For someone new to this technique, could you provide some practical tips or best practices on how to effectively use Grad Cam in a machine learning or deep learning project? Any specific scenarios or types of models where Grad Cam has proven particularly useful?
", "votes": 4, "replies": [ { "id": 2637285, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-02-05T16:19:47.790000", "content": "Great question. Below are how to improve our deep learning model using Grad Cam insights. Alternatively, we can use Grad Cam to see where the deep learning model is looking and then convert this information into machine learning features for ML model such as CatBoost. (For example if model looks at 15Hz row of image, then make features about 15Hz for CatBoost).
\n==========
\nHere is one example (of Grad Cam improving deep learning) that happened in a previous competition. I describe it as cat or dog, but the actually comp was classifying something else.
\nImagine a competition where we must classify an image as cat or dog. Imagine that our CV score is 0.900 AUC but our LB score is worse 0.600. Grad Cam will help explain what is going on. If we apply Grad Cam to this, we see that every cat sits on a car, and every dog sits on a boat. When we look at Grad Cam, we see that the model ignores the animal and only looks at the car or boat. In the LB images, all animals sit on a plane. Hence the model learned to recognize the vehicle and not the animal. And then the vehicle changed in the test data.
\nWe solve the above scenario by training with cutmix or mixup. This starts to make new images where cats sit on boats and dogs sit on cars. After training a new model, we see that the model starts to look at the animal and ignore the vehicle. We submit to LB and get 0.900 AUC!
\n=====
\nThe above example is a case where Grad Cam shows us what the model is doing wrong. Below is an example where Grad Cam shows us what the model is doing correct.
Imagine that Grad Cam shows us that the model is frequently looking at 10Hz row. Or the model is looking at column of image at 15 seconds. We can then make new spectrograms that enlarge these areas of interest to give the model more info on the area it is looking. Furthermore if we notice that the model never looks somewhere in the image, we can remove the wasted space and put something more important in that space.
", "votes": 32, "replies": [ { "id": 2646922, "author_name": "sys-f", "author_url": "", "post_date": "2024-02-11T09:40:06.937000", "content": "Great answer and explanation, thanks Chris, opened up new horizons for me.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2721642, "author_name": "YillusionBrain", "author_url": "", "post_date": "2024-03-29T06:07:06.160000", "content": "Hi @cdeotte ,
\nThanks for the insight.
\nFar as I know, the intepretation of DL models has been controversy. There are many different ways with each has its good-looking cases. I am wondering why do you choose Grad Cam?
\nThanks
", "votes": 1, "replies": [ { "id": 2721778, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-03-29T07:43:06.663000", "content": "Grad Cam shows us what is important inside each spectrogram. This helps us create better spectrograms.
", "votes": 1, "replies": [] } ] }, { "id": 2680978, "author_name": "FARWA99", "author_url": "", "post_date": "2024-03-04T12:17:54.037000", "content": "Hello,
\nI am new to this competition and while reading the data description, I encountered a statement that I'm having trouble understanding: \"The expert annotators reviewed 50-second long EEG samples plus matched spectrograms covering a 10-minute window centered at the same time and labeled the central 10 seconds. Many of these samples overlapped and have been consolidated.\"
\nCould someone please explain what this means? I would greatly appreciate it. Thank you.
", "votes": 1, "replies": [ { "id": 2681038, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-03-04T13:06:18.610000", "content": "Hi welcome. The data is explained in my discussion here
", "votes": 0, "replies": [ { "id": 2682347, "author_name": "FARWA99", "author_url": "", "post_date": "2024-03-05T09:00:28.017000", "content": "Thanks alot for your assistance
", "votes": 1, "replies": [] } ] } ] }, { "id": 2640530, "author_name": "Gabriel Freddi", "author_url": "", "post_date": "2024-02-07T01:14:32.243000", "content": "I will soon start working on this competition, and i found it very useful in advance. I would also like to know if there are any other indispensable methods to inspect my model's inner workings (in a spectrogram context). Thank you very much
", "votes": 1, "replies": [] }, { "id": 2638215, "author_name": "sol lxr", "author_url": "", "post_date": "2024-02-06T06:22:26.300000", "content": "good work explaining grad cam! would anyone care to share some light on how much computational resource this competition requires? I'm just starting on Kaggle, and I 'd like to pick a competition with relatively lower requirement on the computing side. Thanks!
", "votes": 1, "replies": [ { "id": 2638724, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-02-06T12:59:43.913000", "content": "This comp does not require too much compute. There are many starter notebooks that train in Kaggle notebooks and achieve good CV and LB score. Furthermore all starter notebooks can be improved to achieve better CV and LB scores using only Kaggle compute.
", "votes": 2, "replies": [ { "id": 2638739, "author_name": "stefanoclss", "author_url": "", "post_date": "2024-02-06T13:09:17.517000", "content": "The 30 hours GPU time are sufficient but I can imagine I would have been able to speed up the processing if I had access to more GPU time or better GPU's. Nevertheless I think surely you can get in the gold with just the kaggle GPU's but they'll be a bottleneck in the later stages of the competition imo.
", "votes": 2, "replies": [] } ] } ] }, { "id": 2637368, "author_name": "Ganesh Talwar", "author_url": "", "post_date": "2024-02-05T17:10:35.487000", "content": "Thanks sir for sharing your views on Grad Cam
", "votes": 1, "replies": [] }, { "id": 2633436, "author_name": "Peter", "author_url": "", "post_date": "2024-02-03T04:37:57.697000", "content": "Thanks for sharing! for now, I've learned about Grad Cam😊
", "votes": 1, "replies": [] }, { "id": 2647269, "author_name": "RogerOcean", "author_url": "", "post_date": "2024-02-11T13:00:00.330000", "content": "Thx, the feeling of learning a little new knowledge every day is really nice 😂
", "votes": 2, "replies": [] }, { "id": 2643731, "author_name": "Timmy Juicehouse", "author_url": "", "post_date": "2024-02-09T04:04:39.337000", "content": "The difficulty in this competition is that too many raw features can be digged out. I'm struggling to find the ground truth of each pattern.
", "votes": 2, "replies": [] }, { "id": 2634042, "author_name": "Rafael Zimmermann", "author_url": "", "post_date": "2024-02-03T13:06:57.510000", "content": "Hi Chris, again a great learning experience with you, the grad cam will really help you a lot to create new and better spectrograms, thanks for sharing.
\nSpeaking of them, I have a doubt regarding personalized spectrograms, I noticed that you are using the frequency from 0 to 20, but apparently there seems to be an important region also in the area close to 59, was there any special reason for choosing 0 to 20, do you think that Is it worth creating spectrograms with other frequencies?
", "votes": 2, "replies": [ { "id": 2634235, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-02-03T15:51:17.197000", "content": "I did 0 to 20 because Kaggle provided spectrograms are 0 to 20. I'm not sure what the best range to do is. We need to try different ranges and evaluate CV score and LB score.
", "votes": 5, "replies": [ { "id": 2634344, "author_name": "Rafael Zimmermann", "author_url": "", "post_date": "2024-02-03T17:41:54.527000", "content": "Chris, thanks for the answer, I also think the best way to do this is to do the test and test different types of ranges for the specs, I did an optimization using the MLP that you provided, your idea was fantastic, really using it to do The experiments help a lot, especially the fact that you don't need to use the GPU…
\nThe only problem I'm facing is that apparently my experiments are not 100% correlated with the code I'm using to submit using efficientnet_b2, even so, below is the frequency experiment I did using 40% of the training data.
\n-20Hz: ,\n-30Hz: ,\n-40Hz: ,\n-50Hz: ,\n-60Hz: ,\n-70Hz: ,\n-80Hz: \nI'm studying a new way of doing experiments with a small model more similar to efficientnet to try to have a greater correlation without the need to use a GPU.
\nStill studying, but the model below is looking promising.
():\n inp = tf.keras.Input(shape=input_shape)\n\n \n x = layers.Conv2D(, (, ), activation=)(inp)\n x = layers.MaxPooling2D((, ))(x)\n x = layers.Conv2D(, (, ), activation=)(x)\n x = layers.MaxPooling2D((, ))(x)\n x = layers.Conv2D(, (, ), activation=)(x)\n\n \n x = layers.Flatten() (x)\n x = layers.Dense(, activation=)(x)\n x = layers.Dense(num_classes, activation=)(x)\n\n \n model = tf.keras.Model(inputs=inp, outputs=x)\n opt = tf.keras.optimizers.Adam(learning_rate=)\n loss = tf.keras.losses.KLDivergence()\n\n model.(optimizer=opt, loss=loss, metrics=[])\n\n model\n59 Hz is probably just interference from AC power.
", "votes": 2, "replies": [ { "id": 2645074, "author_name": "Rafael Zimmermann", "author_url": "", "post_date": "2024-02-09T23:23:15.743000", "content": "Yes, it's true, after a while I discovered this, it really is a strong noise, it is possible to treat it using filters, such as high pass, low pass or notch, in my case it made an improvement by doing this.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2633879, "author_name": "Pritam Sinha", "author_url": "", "post_date": "2024-02-03T10:42:24.203000", "content": "One question in the spectrogram concatenation image, why Kaggle RL spectrogram concatented with EEG LP spectrogram not EEG LP spectrogram? And also for Kaggle LP spectrogram?
", "votes": 2, "replies": [ { "id": 2633960, "author_name": "Chris Deotte", "author_url": "", "post_date": "2024-02-03T11:59:44.013000", "content": "That's just the way it is. My notebook that makes EEG spectrograms puts them in the order LL, LP, RL, RP. Then my EfficientNet notebook starter just concatenates them vertically as LL, LP, RL, RP. And Kaggle puts them in order LL, RL, LP, RP. In my EfficientNet starter, you can try rearranging so that my EEG spectrograms are also LL, RL, LP, RP. We can experiment to see if that improve CV score and LB score.
We could just randomly place them using augmentations. When I have published my ensemble I'll have a look if I can further improve the models like this.
", "votes": 1, "replies": [] }, { "id": 2649138, "author_name": "PC Jimmmy", "author_url": "", "post_date": "2024-02-12T16:27:47.553000", "content": "On my phone so not able to find all the links to host. But there are three suggested standard montages that are recommended to standardize the way eeg’s are looked at by the medical folks. Probably would be useful to the hosts if we used standard layouts.
", "votes": 0, "replies": [] } ] } ] }, { "id": 2633728, "author_name": "stefanoclss", "author_url": "", "post_date": "2024-02-03T08:45:30.783000", "content": "I haven't looked that much into the metric of this competition yet in very detail. It compares our predicted distribution against the ground truth prediction distribution. But since these samples are not totally perfectly annotated is it possible that a better model performs worse on the LB?
", "votes": 2, "replies": [ { "id": 2633781, "author_name": "Yurnero", "author_url": "", "post_date": "2024-02-03T09:21:46.230000", "content": "It is possible, but unlikely
", "votes": 1, "replies": [] } ] }, { "id": 2871885, "author_name": "Phil Fosso", "author_url": "", "post_date": "2024-06-14T12:52:11.077000", "content": "Great insight and starter notebooks. Thank you.
", "votes": 0, "replies": [] }, { "id": 2861872, "author_name": "lead_n_validate", "author_url": "", "post_date": "2024-06-08T12:09:39.003000", "content": "I would like to express late gratitude for this insight, it was very helpful during the competition.
", "votes": 0, "replies": [] }, { "id": 2642798, "author_name": "", "author_url": "", "post_date": "2024-02-08T12:26:25.710000", "content": "", "votes": 1, "replies": [] }, { "id": 2637225, "author_name": "", "author_url": "", "post_date": "2024-02-05T15:51:45.600000", "content": "", "votes": 2, "replies": [] }, { "id": 2642728, "author_name": "A-K", "author_url": "", "post_date": "2024-02-08T11:11:05.837000", "content": "Thank you for sharing!
", "votes": 1, "replies": [] }, { "id": 2640905, "author_name": "npusrc", "author_url": "", "post_date": "2024-02-07T06:35:43.423000", "content": "thanks for sharing
", "votes": 1, "replies": [] }, { "id": 2636920, "author_name": "parksb_skku", "author_url": "", "post_date": "2024-02-05T11:15:57.457000", "content": "thanks for sharing
", "votes": 1, "replies": [] }, { "id": 2635356, "author_name": "Ganesh Talwar", "author_url": "", "post_date": "2024-02-04T10:50:43.843000", "content": "thanks for sharing Chris sir.
", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "2633286": "This competition is a great competition to use both Machine Learning (ML) and Deep Learning (DL). We can use insights from our machine learning models to improve our deep learning models. And we can use insights from our deep learning models to improve our machine learning models. This competition's winning solution will most likely be an ensemble of machine learning and deep learning!\n\n# Machine Learning versus Deep Learning\nIt is interesting to note that the classifier in both machine learning and deep learning is the same. With binary cross entropy loss, both are logistic regression. In other scenarios, both are just ML models. The two approaches differ in how features are engineered. \n\nFor machine learning (like SVC, KNN, LogReg, GBT), the human converts the raw features into useful \"smart\" features via feature engineering. For deep learning (like CNN, RNN, Transformer), deep hidden layers create features from the raw features.\n\nIn both methods, the \"smart\" features are inputted into a final model to perform classification. And both final classification models are basically the same.\n\n\n# Explainability\nFor machine learning, we can view feature importance and/or feature weights easily (like GBT feature importance or LogReg coefficients). Thus it is easy to see what the machine learning model is doing. For deep learning, it is more difficult to understand what features the model is making. One technique to locate where the deep learning model is creating features from is called Grad Cam\n\n# Grad Cam\nWith grad cam, given a specific OOF (out of fold) train sample, we can view both a model's prediction and where it looked to make this prediction. In the plots below we display the image that was fed into our image model, in my popular EfficientNet starter notebook [here][4], there are 8 spectrograms that have been tiled into one 1 input image. \n\nOn the left we have the 4 Kaggle spectrograms where each is 10 minutes long. Each represents one of the 4 montages LL, RL, LP, RP. (Montages explained [here][1]) On the right, we have the 4 EEG spectrograms where each is 50 seconds long. The EEG spectrograms are made from the Magic Formula [here][2]. The spectrograms are each `128x256x1`, so the final concatenation is `512x512x1`.\n\n## Seizure\nHere is an example for a train sample with ground truth 76% Seizure and 24% LPD. The model predicts 88% Seizure and 6.5% LPD. Below we see where the model looked to make this decision. The left image is just the Grad Cam. Larger values are shown in yellow and indicate where the model pays attention to more (when making this specific prediction from this specific input image).\n\nThe middle image is the contours of the Grad Cam's 10% largest values superimposed over the image that we fed into our model. The right image is also Grad Cam contour superimposed over image but we add an emboss filter to the image to make the details more visible to humans. We observe\n* **Observation:** The model looked at the middle 10 seconds of the RL and RP montage spectograms to decide Seizure\n\n\n## LPD\nHere is an example of ground truth LPD at 100%. The model predicted LPD at 93%. \n* **Observation** The model looked at the beginning and ending of the 50 second spectrogram (but not directly in the middle) to decide LPD.\n\n\n## LRDA\nHere is an example of ground truth LRDA at 100%. The model predicted LRDA at 68%.\n* **Observation** The model looked at the center 40 seconds (not just middle 10 seconds) to decide LRDA\n\n\n## GRDA\nHere is an example of ground truth GRDA at 100%. The model predicted GRDA at 79%.\n* **Observation** The model looked at many spots and mostly beginning and end to determine GRDA\n\n\n# Starter Notebook\nI published a starter notebook [here][3] which loads the trained models from my EfficientNet starter notebook [here][4] and performs Grad Cam\n\n[1]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/467877\n[2]: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/469760\n[3]: https://www.kaggle.com/code/cdeotte/grad-cam-what-is-important-in-spectrograms\n[4]: https://www.kaggle.com/code/cdeotte/efficientnetb0-starter-lb-0-43", "2648271": "I noticed that LPD and LRDA cases , the model in this case didnt focus on Kaggle Spectrogram at all . Ideally it should have found the regions from both Kaggle and EEG spectrograms to focus on in case of LPD and LRDA . Is this correct observation ?", "2637252": "Thanks for sharing your insights on Grad Cam and its application in understanding model decisions. For someone new to this technique, could you provide some practical tips or best practices on how to effectively use Grad Cam in a machine learning or deep learning project? Any specific scenarios or types of models where Grad Cam has proven particularly useful?", "2721642": "Hi @cdeotte ,\n\nThanks for the insight.\n\nFar as I know, the intepretation of DL models has been controversy. There are many different ways with each has its good-looking cases. I am wondering why do you choose Grad Cam?\n\nThanks", "2680978": "Hello,\n\nI am new to this competition and while reading the data description, I encountered a statement that I'm having trouble understanding: \"The expert annotators reviewed 50-second long EEG samples plus matched spectrograms covering a 10-minute window centered at the same time and labeled the central 10 seconds. Many of these samples overlapped and have been consolidated.\"\n\nCould someone please explain what this means? I would greatly appreciate it. Thank you.", "2640530": "I will soon start working on this competition, and i found it very useful in advance. I would also like to know if there are any other indispensable methods to inspect my model's inner workings (in a spectrogram context). Thank you very much", "2638215": "good work explaining grad cam! would anyone care to share some light on how much computational resource this competition requires? I'm just starting on Kaggle, and I 'd like to pick a competition with relatively lower requirement on the computing side. Thanks!", "2637368": "Thanks sir for sharing your views on Grad Cam", "2633436": "Thanks for sharing! for now, I've learned about Grad Cam😊", "2647269": "Thx, the feeling of learning a little new knowledge every day is really nice 😂\n\n", "2643731": "The difficulty in this competition is that too many raw features can be digged out. I'm struggling to find the ground truth of each pattern. ", "2634042": "Hi Chris, again a great learning experience with you, the grad cam will really help you a lot to create new and better spectrograms, thanks for sharing.\n\nSpeaking of them, I have a doubt regarding personalized spectrograms, I noticed that you are using the frequency from 0 to 20, but apparently there seems to be an important region also in the area close to 59, was there any special reason for choosing 0 to 20, do you think that Is it worth creating spectrograms with other frequencies?", "2633879": "One question in the spectrogram concatenation image, why Kaggle RL spectrogram concatented with EEG LP spectrogram not EEG LP spectrogram? And also for Kaggle LP spectrogram?", "2633728": "I haven't looked that much into the metric of this competition yet in very detail. It compares our predicted distribution against the ground truth prediction distribution. But since these samples are not totally perfectly annotated is it possible that a better model performs worse on the LB?", "2871885": " Great insight and starter notebooks. Thank you.", "2861872": "I would like to express late gratitude for this insight, it was very helpful during the competition.", "2642798": "", "2637225": "", "2642728": "Thank you for sharing!", "2640905": "thanks for sharing", "2636920": "thanks for sharing", "2635356": "thanks for sharing Chris sir.\n \n" } }