![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F761268%2F5108b87498942a37058ab1164bf8a8de%2FScreenshot%202024-01-11%20at%202.38.42AM.png?generation=1704920935509556&alt=media\")
\nWell balanced targets
\nWell balanced targets
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F761268%2Fbef0e17513d4ea1c81654664b7df1720%2FScreenshot%202024-01-11%20at%202.36.39AM.png?generation=1704920849053906&alt=media\")
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F761268%2F5098c5d109726d95a7ff26a8fd6456a9%2FScreenshot%202024-01-11%20at%202.59.21AM.png?generation=1704922187122299&alt=media\")
\nSignificant correlation between seizure_vote, other vote and grda_vote types ( all 3 targets are frequent targets )
![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F761268%2F20d7d0dfbc933068f13abc37b3881443%2FScreenshot%202024-01-11%20at%202.52.19AM.png?generation=1704921765311082&alt=media\")
\nis < 1000 offset is the major information of the targets except for gpd target ?
with Equal probability 1/6: LB 1.09
\nWith Overlapping LB: 1.1
\nseizure_vote ,\ngpd_vote ,\nlrda_vote ,\nother_vote ,\ngrda_vote ,\nlpd_vote \nwith non-overlapping based on spectrogram id
\nLB: 1.0
seizure_vote \nlpd_vote \ngpd_vote \nlrda_vote \ngrda_vote \nother_vote \nwith non-overlapping based on eeg id
\nLB: 0.97
seizure_vote \nlpd_vote \ngpd_vote \nlrda_vote \ngrda_vote \nother_vote \nwith non-overlapping based on p id
\nLB: 1.28
seizure_vote \nlpd_vote \ngpd_vote \nlrda_vote \ngrda_vote \nother_vote \n
\nWell balanced targets
\n
\nSignificant correlation between seizure_vote, other vote and grda_vote types ( all 3 targets are frequent targets )
\nis < 1000 offset is the major information of the targets except for gpd target ?
with Equal probability 1/6: LB 1.09
\nWith Overlapping LB: 1.1
\nseizure_vote ,\ngpd_vote ,\nlrda_vote ,\nother_vote ,\ngrda_vote ,\nlpd_vote \nwith non-overlapping based on spectrogram id
\nLB: 1.0
seizure_vote \nlpd_vote \ngpd_vote \nlrda_vote \ngrda_vote \nother_vote \nwith non-overlapping based on eeg id
\nLB: 0.97
seizure_vote \nlpd_vote \ngpd_vote \nlrda_vote \ngrda_vote \nother_vote \nwith non-overlapping based on p id
\nLB: 1.28
seizure_vote \nlpd_vote \ngpd_vote \nlrda_vote \ngrda_vote \nother_vote \nThanks for EDA. Have you tried submitting the train means as test predictions? I'm curious if that beats the sample submission 1/6 predictions.
", "rawMarkdown": "Thanks for EDA. Have you tried submitting the train means as test predictions? I'm curious if that beats the sample submission 1/6 predictions.", "votes": null }, { "id": "2596119", "postDate": "01/10/2024 22:21:06", "content": "@cdeotte Thank you. Yes mean with overlaps and mean without overlaps
\n@seshurajup Have you tried submitting the means of non-overlaps spectrograms? (as shown in code below)
\n = train.groupby('spectrogram_id')[targets].sum().sum(axis=)\n = train.groupby('spectrogram_id')[targets].sum().div(total_votes_per_spec, axis=)\n = normalized_votes.mean()\n( mean_vote_ratio )\n\n .\n .\n .\n .\n .\n .\nThis is slightly different than non-overlaps eeg
\n .\n .\n .\n .\n .\n .\nI'm curious which is better. This will give us insight about the best way to train our models.
", "rawMarkdown": "seshurajup Have you tried submitting the means of non-overlaps spectrograms? (as shown in code below)\n\n total_votes_per_spec = train.groupby('spectrogram_id')[targets].sum().sum(axis=1)\n normalized_votes = train.groupby('spectrogram_id')[targets].sum().div(total_votes_per_spec, axis=0)\n mean_vote_ratio = normalized_votes.mean()\n print( mean_vote_ratio )\n\n seizure_vote 0.174031\n lpd_vote 0.112700\n gpd_vote 0.090854\n lrda_vote 0.071484\n grda_vote 0.136408\n other_vote 0.414523\n\nThis is slightly different than non-overlaps eeg\n\n seizure_vote 0.152810\n lpd_vote 0.142456\n gpd_vote 0.104062\n lrda_vote 0.065407\n grda_vote 0.114851\n other_vote 0.420414\n\nI'm curious which is better. This will give us insight about the best way to train our models.", "votes": null }, { "id": "2598973", "postDate": "01/12/2024 15:32:16", "content": "What is your definition of overlap here? Overlapping 50 second entire subsample or central 10 seconds?
", "rawMarkdown": "What is your definition of overlap here? Overlapping 50 second entire subsample or central 10 seconds?", "votes": null }, { "id": "2598983", "postDate": "01/12/2024 15:40:44", "content": "@cdeotte I didn't submitted spectrograms based, only did with egg_id based. Curious how it help insight about the best way to train our models ?
", "rawMarkdown": "cdeotte I didn't submitted spectrograms based, only did with egg_id based. Curious how it help **insight about the best way to train our models** ?", "votes": null }, { "id": "2598988", "postDate": "01/12/2024 15:41:44", "content": "Overlap = overlapping spectrogram ids
\nThe best public notebook of 0.97 (and LGB/XGB 0.96) use \"non-overlapping eeg\". We take the average of the train targets from the 17089 unique eeg_id in train.csv.
To submit \"non-overlapping spectrogram\" we can take the average of the train targets from the 11138 unique spectrogram_id in train.csv. I posted the code to do this above.
@gunesevitan for submissions I treated overlaps for eegs is 50secs
", "rawMarkdown": "gunesevitan for submissions I treated overlaps for eegs is 50secs", "votes": null }, { "id": "2598994", "postDate": "01/12/2024 15:45:05", "content": "Whichever has the best LB score will tell us how to set our train sample weights during training. (And inform us how to compute CV score).
\nFor example, we can give each row of train.csv weight 1 during training. Or we can give each unique eeg_id weight 1 during training. You already submitted these two cases and using unique eeg_id weight is better.
If unique spectrogram_id is better, then we can give each spectrogram_id weight 1 during training.
@cdeotte if I understood correctly as follow
\nusing these scores to sample the train dataset to build cv?
", "rawMarkdown": "cdeotte if I understood correctly as follow \n1. Average non-overlapping eeg based score -> 0.97\n2. Average non-overlapped spectrogram based score -> x\n3. LGB/XGB non-overlapped egg based score -> 0.96\n\nusing these scores to sample the train dataset to build cv?", "votes": null }, { "id": "2599014", "postDate": "01/12/2024 15:54:40", "content": "Understanding the test data will help us both train and validate our models more accurately.
\nThe metric KL-Div is very sensitive to global means.
", "rawMarkdown": "Understanding the test data will help us both train and validate our models more accurately. \n\nThe metric KL-Div is very sensitive to global means.", "votes": null }, { "id": "2599017", "postDate": "01/12/2024 15:55:23", "content": "Thanks, its new learning for me @cdeotte
", "rawMarkdown": "Thanks, its new learning for me @cdeotte", "votes": null }, { "id": "2599036", "postDate": "01/12/2024 16:08:40", "content": "LB 1.0 ( updated on topic too ) @cdeotte
\n\n", "rawMarkdown": "**LB 1.0** ( updated on topic too ) @cdeotte \n>with overlapping --> 1.1\nwithout overlapping based spectrogram --> 1.0\n**without overlapping based eeg --> 0.97**", "votes": null }, { "id": "2599038", "postDate": "01/12/2024 16:11:17", "content": "with overlapping --> 1.1
\n
\n without overlapping based spectrogram --> 1.0
\n without overlapping based eeg --> 0.97
Thanks for checking @seshurajup . I will continue to train and validate my models giving each unique eeg_id weight 1.
@cdeotte Fixed thanks
", "rawMarkdown": "cdeotte Fixed thanks", "votes": null }, { "id": "2599058", "postDate": "01/12/2024 16:29:37", "content": "These are good LB probing experiments. Based on what you submitted so far, it seems that the train data has \"overlapping eeg\" whereas the test data has \"non-overlapping eeg\". This is said in the competition description, but it is always good to verify ourselves.
\n\n\ntest.csv Metadata for the test set. As there are no overlapping samples in the test set, many columns in the train metadata don't apply.
\n
The next question i'm curious about is if the same patient appears multiple times in the test data. In the train data, the same patient does appear multiple times. For example if we compute the mean from train with \"non-overlapping patients\" we get much different means (than \"overlapping patients\"):
\n = train.groupby('patient_id')[targets].sum().sum(axis=)\n = train.groupby('patient_id')[targets].sum().div(total_votes_per_pat, axis=)\n = normalized_votes.mean()\n( mean_vote_ratio )\n\n .\n .\n .\n .\n .\n .\nIf these means produce a better LB score (than 0.97), then the test data is most likely unique patients (i.e. \"non overlapping patients\"). If these means do not produce a better LB score, then the test data is most likely repeat patients with \"non overlapping eeg\". (Of course we can also submit an \"if-else\" test to LB. Our submit code can check if a test patient repeats, then create submission.csv with 1/6 preds else create submission.csv with \"non-overlapp eeg_id\" preds. But probing LB score with means is more fun).
\nIf the test data has repeated patients, then perhaps we can use one row of test data to help us predict another row of test data.
", "rawMarkdown": "These are good LB probing experiments. Based on what you submitted so far, it seems that the train data has \"overlapping eeg\" whereas the test data has \"non-overlapping eeg\". This is said in the competition description, but it is always good to verify ourselves.\n\n>test.csv Metadata for the test set. As there are no overlapping samples in the test set, many columns in the train metadata don't apply.\n\nThe next question i'm curious about is if the same patient appears multiple times in the test data. In the train data, the same patient does appear multiple times. For example if we compute the mean from train with \"non-overlapping patients\" we get much different means (than \"overlapping patients\"):\n\n total_votes_per_pat = train.groupby('patient_id')[targets].sum().sum(axis=1)\n normalized_votes = train.groupby('patient_id')[targets].sum().div(total_votes_per_pat, axis=0)\n mean_vote_ratio = normalized_votes.mean()\n print( mean_vote_ratio )\n\n seizure_vote 0.310718\n lpd_vote 0.046279\n gpd_vote 0.051885\n lrda_vote 0.081796\n grda_vote 0.231471\n other_vote 0.277851\n\nIf these means produce a better LB score (than 0.97), then the test data is most likely unique patients (i.e. \"non overlapping patients\"). If these means do not produce a better LB score, then the test data is most likely repeat patients with \"non overlapping eeg\". (Of course we can also submit an \"if-else\" test to LB. Our submit code can check if a test patient repeats, then create submission.csv with 1/6 preds else create submission.csv with \"non-overlapp eeg_id\" preds. But probing LB score with means is more fun).\n\nIf the test data has repeated patients, then perhaps we can use one row of test data to help us predict another row of test data.", "votes": null }, { "id": "2599250", "postDate": "01/12/2024 18:22:46", "content": "Its great learning with you (@cdeotte) and @hengck23 (other competitions). Its motivate me and learn new things every time i see the knowledge share from last 4 years. Thanks for it ( Today lesson - Power of Mean )
\nLB: 1.28
\nseizure_vote 0.310718
\nlpd_vote 0.046279
\ngpd_vote 0.051885
\nlrda_vote 0.081796
\ngrda_vote 0.231471
\nother_vote 0.277851
conclusion - repeat patients with \"non overlapping eeg\".
", "rawMarkdown": "Its great learning with you (@cdeotte) and @hengck23 (other competitions). Its motivate me and learn new things every time i see the knowledge share from last 4 years. Thanks for it ( Today lesson - **Power of Mean** )\n\n**LB: 1.28**\nseizure_vote 0.310718\nlpd_vote 0.046279\ngpd_vote 0.051885\nlrda_vote 0.081796\ngrda_vote 0.231471\nother_vote 0.277851\n\n**conclusion** - repeat patients with \"non overlapping eeg\".", "votes": null }, { "id": "2599328", "postDate": "01/12/2024 18:59:17", "content": "Thanks for checking this. If test has multiple patient_id, then we can use patient_id count z_score as a feature. From train data, it appears when patient id count z_score is small, then patient is more likely seizure or grda. And when patient count z_score is large, target is more likely gpd or lpd.
We should be able to train model using only this one feature patient_id count z_score and beat LB 0.97. Note that we will need to standardize the count feature so that it generalizes from train to test. For example, the feature will be the z_score = (count - count_mean) / count_std. So during train count_mean = total patients in train and during infer count_mean = total patients in test. Likewise count_std is computed once during train and once during infer.
@cdeotte i have train and test the using CatBoost Classifier using feature patient_id count z_score and i got following results:
\nCV : 1.185
\nLB : 1.01
Here is the feature calculation:
\ntmp = df.groupby().size()\neeg_id_count_per_patient_id = pd.DataFrame({:tmp.index,:tmp.values})\nmean_value = np.mean(eeg_id_count_per_patient_id[])\nstd_value = np.std(eeg_id_count_per_patient_id[])\n\neeg_id_count_per_patient_id[] = (eeg_id_count_per_patient_id[] - mean_value)/std_value\neeg_id_count_per_patient_id = eeg_id_count_per_patient_id.drop(columns=[])\nSo, this LB is not able to beat LB 0.97.
\nNotebook Link https://www.kaggle.com/code/deepaksingh47/catboost-starter-with-patient-eeg-count-z-score#CV-Score-for-CatBoost
", "rawMarkdown": "cdeotte i have train and test the using CatBoost Classifier using feature `patient_id count z_score` and i got following results:\nCV : 1.185\nLB : 1.01\n\nHere is the feature calculation:\n```python\ntmp = df.groupby('patient_id').size()\neeg_id_count_per_patient_id = pd.DataFrame({'patient_id':tmp.index,'eeg_count':tmp.values})\nmean_value = np.mean(eeg_id_count_per_patient_id[\"eeg_count\"])\nstd_value = np.std(eeg_id_count_per_patient_id[\"eeg_count\"])\n\neeg_id_count_per_patient_id[\"eeg_count_z_score\"] = (eeg_id_count_per_patient_id[\"eeg_count\"] - mean_value)/std_value\neeg_id_count_per_patient_id = eeg_id_count_per_patient_id.drop(columns=[\"eeg_count\"])\n```\n\nSo, this LB is not able to beat LB 0.97. \n\nNotebook Link [https://www.kaggle.com/code/deepaksingh47/catboost-starter-with-patient-eeg-count-z-score#CV-Score-for-CatBoost](url)", "votes": null }, { "id": "2618149", "postDate": "01/24/2024 15:31:51", "content": "Thanks for doing the test. I also performed a test with my best model. Including patient z score improved CV score but hurt LB score.
", "rawMarkdown": "Thanks for doing the test. I also performed a test with my best model. Including patient z score improved CV score but hurt LB score.", "votes": null }, { "id": "2618154", "postDate": "01/24/2024 15:35:19", "content": "It is interesting that if we include z_score along with other features, then z_score becomes the most important feature. Unfortuntately it doesn't seem to work on test.
\n@deepaksingh47 Note that your notebook has a leak. We cannot compute the z_score once. We must compute the z_score each fold. In the folds for-loop. We compute z_score for the train folds. And then we compute z_score for the valid folds. Then for fold 2, we delete previous z_score and compute z_score again. (In total we compute z_score on 10 dataframes for 5 KFold)
@cdeotte, thanks for pointing that out. i forget about that.
", "rawMarkdown": "cdeotte, thanks for pointing that out. i forget about that.", "votes": null }, { "id": "2619937", "postDate": "01/25/2024 19:47:57", "content": "Hi, Many thanks for your discussion post and the eda nbs as they provide many insights into the data. Could you pls share how you calculated these particular values (the ones you call \"with overlapping LB: 1.1\". While I am able to understand many of the other calculations based on per eeg_id, per spectrogram_id, per row etc, I am unable to grasp how these numbers are calculated.
\nwith overlapping LB: 1.1
seizure_vote 0.196002,
\ngpd_vote 0.156386,
\nlrda_vote 0.155805,
\nother_vote 0.17610,
\ngrda_vote 0.17660,
\nlpd_vote 0.139101
Thanks.
", "rawMarkdown": "Hi, Many thanks for your discussion post and the eda nbs as they provide many insights into the data. Could you pls share how you calculated these particular values (the ones you call \"with overlapping LB: 1.1\". While I am able to understand many of the other calculations based on per eeg_id, per spectrogram_id, per row etc, I am unable to grasp how these numbers are calculated.\nwith overlapping LB: 1.1\n\nseizure_vote 0.196002,\ngpd_vote 0.156386,\nlrda_vote 0.155805,\nother_vote 0.17610,\ngrda_vote 0.17660,\nlpd_vote 0.139101\n\nThanks.", "votes": null }, { "id": "2619980", "postDate": "01/25/2024 20:17:53", "content": "@skr1125 from Notebook v16 - EDA train.csv
\n collections Counter\ntarget_votes = Counter((train[]))\ntarget_votes = {:v k,v target_votes.items()}\ntotal_votes = ([v _,v target_votes.items()])\nmean_vote_ratio = {k:(target_votes[k]/total_votes) k,target target_votes.items()}\nok, understand
", "rawMarkdown": "ok, understand", "votes": null }, { "id": "2641812", "postDate": "02/07/2024 17:19:42", "content": "Thanks for providing this, and thanks to all the commentators. Insightful information.
", "rawMarkdown": "Thanks for providing this, and thanks to all the commentators. Insightful information.", "votes": null }, { "id": "2704278", "postDate": "03/18/2024 17:25:00", "content": "Could anyone be kind enough to explain the first conclusion along with what the author means by
\n\"with Equal probability 1/6: LB 1.09
With Overlapping LB: 1.1\"
\nI tried to get the same numbers but only in this scenario I failed. I am assuming equal probability means author put all targets value as 0.17 and submitted it. I can't figure out what he means by with overlapping.
\nAlso what does getting better score by equal probability mean? In the other 2 conclusions I understand that the case in which better score is achieved we can assume then test case has similar data. But for 1st conclusion I am having a hard time understanding if anyone could explain it I would be very grateful. Thanks in advance.
same i also stuck with this
", "rawMarkdown": "same i also stuck with this", "votes": null } ], "comments": [ { "id": 2596092, "author_name": "cdeotte", "author_url": "", "post_date": "01/10/2024 21:31:05", "content": "Thanks for EDA. Have you tried submitting the train means as test predictions? I'm curious if that beats the sample submission 1/6 predictions.
", "votes": null, "replies": [ { "id": 2596119, "author_name": "seshurajup", "author_url": "", "post_date": "01/10/2024 22:21:06", "content": "@cdeotte Thank you. Yes mean with overlaps and mean without overlaps
\n@seshurajup Have you tried submitting the means of non-overlaps spectrograms? (as shown in code below)
\n = train.groupby('spectrogram_id')[targets].sum().sum(axis=)\n = train.groupby('spectrogram_id')[targets].sum().div(total_votes_per_spec, axis=)\n = normalized_votes.mean()\n( mean_vote_ratio )\n\n .\n .\n .\n .\n .\n .\nThis is slightly different than non-overlaps eeg
\n .\n .\n .\n .\n .\n .\nI'm curious which is better. This will give us insight about the best way to train our models.
", "votes": null, "replies": [ { "id": 2598973, "author_name": "gunesevitan", "author_url": "", "post_date": "01/12/2024 15:32:16", "content": "What is your definition of overlap here? Overlapping 50 second entire subsample or central 10 seconds?
", "votes": null, "replies": [ { "id": 2598988, "author_name": "cdeotte", "author_url": "", "post_date": "01/12/2024 15:41:44", "content": "Overlap = overlapping spectrogram ids
\nThe best public notebook of 0.97 (and LGB/XGB 0.96) use \"non-overlapping eeg\". We take the average of the train targets from the 17089 unique eeg_id in train.csv.
To submit \"non-overlapping spectrogram\" we can take the average of the train targets from the 11138 unique spectrogram_id in train.csv. I posted the code to do this above.
@cdeotte if I understood correctly as follow
\nusing these scores to sample the train dataset to build cv?
", "votes": null, "replies": [ { "id": 2599014, "author_name": "cdeotte", "author_url": "", "post_date": "01/12/2024 15:54:40", "content": "Understanding the test data will help us both train and validate our models more accurately.
\nThe metric KL-Div is very sensitive to global means.
", "votes": null, "replies": [] } ] } ] }, { "id": 2598991, "author_name": "seshurajup", "author_url": "", "post_date": "01/12/2024 15:43:21", "content": "@gunesevitan for submissions I treated overlaps for eegs is 50secs
", "votes": null, "replies": [] } ] }, { "id": 2598983, "author_name": "seshurajup", "author_url": "", "post_date": "01/12/2024 15:40:44", "content": "@cdeotte I didn't submitted spectrograms based, only did with egg_id based. Curious how it help insight about the best way to train our models ?
", "votes": null, "replies": [ { "id": 2598994, "author_name": "cdeotte", "author_url": "", "post_date": "01/12/2024 15:45:05", "content": "Whichever has the best LB score will tell us how to set our train sample weights during training. (And inform us how to compute CV score).
\nFor example, we can give each row of train.csv weight 1 during training. Or we can give each unique eeg_id weight 1 during training. You already submitted these two cases and using unique eeg_id weight is better.
If unique spectrogram_id is better, then we can give each spectrogram_id weight 1 during training.
Thanks, its new learning for me @cdeotte
", "votes": null, "replies": [] } ] } ] }, { "id": 2599036, "author_name": "seshurajup", "author_url": "", "post_date": "01/12/2024 16:08:40", "content": "LB 1.0 ( updated on topic too ) @cdeotte
\n\n", "votes": null, "replies": [ { "id": 2599038, "author_name": "cdeotte", "author_url": "", "post_date": "01/12/2024 16:11:17", "content": "with overlapping --> 1.1
\n
\n without overlapping based spectrogram --> 1.0
\n without overlapping based eeg --> 0.97
Thanks for checking @seshurajup . I will continue to train and validate my models giving each unique eeg_id weight 1.
@cdeotte Fixed thanks
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2599058, "author_name": "cdeotte", "author_url": "", "post_date": "01/12/2024 16:29:37", "content": "These are good LB probing experiments. Based on what you submitted so far, it seems that the train data has \"overlapping eeg\" whereas the test data has \"non-overlapping eeg\". This is said in the competition description, but it is always good to verify ourselves.
\n\n\ntest.csv Metadata for the test set. As there are no overlapping samples in the test set, many columns in the train metadata don't apply.
\n
The next question i'm curious about is if the same patient appears multiple times in the test data. In the train data, the same patient does appear multiple times. For example if we compute the mean from train with \"non-overlapping patients\" we get much different means (than \"overlapping patients\"):
\n = train.groupby('patient_id')[targets].sum().sum(axis=)\n = train.groupby('patient_id')[targets].sum().div(total_votes_per_pat, axis=)\n = normalized_votes.mean()\n( mean_vote_ratio )\n\n .\n .\n .\n .\n .\n .\nIf these means produce a better LB score (than 0.97), then the test data is most likely unique patients (i.e. \"non overlapping patients\"). If these means do not produce a better LB score, then the test data is most likely repeat patients with \"non overlapping eeg\". (Of course we can also submit an \"if-else\" test to LB. Our submit code can check if a test patient repeats, then create submission.csv with 1/6 preds else create submission.csv with \"non-overlapp eeg_id\" preds. But probing LB score with means is more fun).
\nIf the test data has repeated patients, then perhaps we can use one row of test data to help us predict another row of test data.
", "votes": null, "replies": [ { "id": 2599250, "author_name": "seshurajup", "author_url": "", "post_date": "01/12/2024 18:22:46", "content": "Its great learning with you (@cdeotte) and @hengck23 (other competitions). Its motivate me and learn new things every time i see the knowledge share from last 4 years. Thanks for it ( Today lesson - Power of Mean )
\nLB: 1.28
\nseizure_vote 0.310718
\nlpd_vote 0.046279
\ngpd_vote 0.051885
\nlrda_vote 0.081796
\ngrda_vote 0.231471
\nother_vote 0.277851
conclusion - repeat patients with \"non overlapping eeg\".
", "votes": null, "replies": [ { "id": 2599328, "author_name": "cdeotte", "author_url": "", "post_date": "01/12/2024 18:59:17", "content": "Thanks for checking this. If test has multiple patient_id, then we can use patient_id count z_score as a feature. From train data, it appears when patient id count z_score is small, then patient is more likely seizure or grda. And when patient count z_score is large, target is more likely gpd or lpd.
We should be able to train model using only this one feature patient_id count z_score and beat LB 0.97. Note that we will need to standardize the count feature so that it generalizes from train to test. For example, the feature will be the z_score = (count - count_mean) / count_std. So during train count_mean = total patients in train and during infer count_mean = total patients in test. Likewise count_std is computed once during train and once during infer.
@cdeotte i have train and test the using CatBoost Classifier using feature patient_id count z_score and i got following results:
\nCV : 1.185
\nLB : 1.01
Here is the feature calculation:
\ntmp = df.groupby().size()\neeg_id_count_per_patient_id = pd.DataFrame({:tmp.index,:tmp.values})\nmean_value = np.mean(eeg_id_count_per_patient_id[])\nstd_value = np.std(eeg_id_count_per_patient_id[])\n\neeg_id_count_per_patient_id[] = (eeg_id_count_per_patient_id[] - mean_value)/std_value\neeg_id_count_per_patient_id = eeg_id_count_per_patient_id.drop(columns=[])\nSo, this LB is not able to beat LB 0.97.
\nNotebook Link https://www.kaggle.com/code/deepaksingh47/catboost-starter-with-patient-eeg-count-z-score#CV-Score-for-CatBoost
", "votes": null, "replies": [ { "id": 2618149, "author_name": "cdeotte", "author_url": "", "post_date": "01/24/2024 15:31:51", "content": "Thanks for doing the test. I also performed a test with my best model. Including patient z score improved CV score but hurt LB score.
", "votes": null, "replies": [] }, { "id": 2618154, "author_name": "cdeotte", "author_url": "", "post_date": "01/24/2024 15:35:19", "content": "It is interesting that if we include z_score along with other features, then z_score becomes the most important feature. Unfortuntately it doesn't seem to work on test.
\n@deepaksingh47 Note that your notebook has a leak. We cannot compute the z_score once. We must compute the z_score each fold. In the folds for-loop. We compute z_score for the train folds. And then we compute z_score for the valid folds. Then for fold 2, we delete previous z_score and compute z_score again. (In total we compute z_score on 10 dataframes for 5 KFold)
@cdeotte, thanks for pointing that out. i forget about that.
", "votes": null, "replies": [] } ] } ] } ] } ] } ] }, { "id": 2619937, "author_name": "skr1125", "author_url": "", "post_date": "01/25/2024 19:47:57", "content": "Hi, Many thanks for your discussion post and the eda nbs as they provide many insights into the data. Could you pls share how you calculated these particular values (the ones you call \"with overlapping LB: 1.1\". While I am able to understand many of the other calculations based on per eeg_id, per spectrogram_id, per row etc, I am unable to grasp how these numbers are calculated.
\nwith overlapping LB: 1.1
seizure_vote 0.196002,
\ngpd_vote 0.156386,
\nlrda_vote 0.155805,
\nother_vote 0.17610,
\ngrda_vote 0.17660,
\nlpd_vote 0.139101
Thanks.
", "votes": null, "replies": [ { "id": 2619980, "author_name": "seshurajup", "author_url": "", "post_date": "01/25/2024 20:17:53", "content": "@skr1125 from Notebook v16 - EDA train.csv
\n collections Counter\ntarget_votes = Counter((train[]))\ntarget_votes = {:v k,v target_votes.items()}\ntotal_votes = ([v _,v target_votes.items()])\nmean_vote_ratio = {k:(target_votes[k]/total_votes) k,target target_votes.items()}\nok, understand
", "votes": null, "replies": [] } ] } ] }, { "id": 2641812, "author_name": "marioandrs", "author_url": "", "post_date": "02/07/2024 17:19:42", "content": "Thanks for providing this, and thanks to all the commentators. Insightful information.
", "votes": null, "replies": [] }, { "id": 2704278, "author_name": "otutsukihyuuga", "author_url": "", "post_date": "03/18/2024 17:25:00", "content": "Could anyone be kind enough to explain the first conclusion along with what the author means by
\n\"with Equal probability 1/6: LB 1.09
With Overlapping LB: 1.1\"
\nI tried to get the same numbers but only in this scenario I failed. I am assuming equal probability means author put all targets value as 0.17 and submitted it. I can't figure out what he means by with overlapping.
\nAlso what does getting better score by equal probability mean? In the other 2 conclusions I understand that the case in which better score is achieved we can assume then test case has similar data. But for 1st conclusion I am having a hard time understanding if anyone could explain it I would be very grateful. Thanks in advance.
same i also stuck with this
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2596082": "\n**Well balanced targets**\n\n\n\n**Significant correlation between seizure_vote, other vote and grda_vote types ( all 3 targets are frequent targets )**\n\n\n**is < 1000 offset is the major information of the targets except for gpd target ?**\n\n\nwith Equal probability 1/6: **LB 1.09**\n\nWith Overlapping **LB: 1.1**\n```python\nseizure_vote 0.196002,\ngpd_vote 0.156386,\nlrda_vote 0.155805,\nother_vote 0.17610,\ngrda_vote 0.17660,\nlpd_vote 0.139101\n```\n## Conclusion: Test set have means closer to Train Set\n\nwith non-overlapping based on spectrogram id\n**LB: 1.0**\n```python\nseizure_vote 0.174031\nlpd_vote 0.112700\ngpd_vote 0.090854\nlrda_vote 0.071484\ngrda_vote 0.136408\nother_vote 0.414523\n```\nwith non-overlapping based on eeg id\n**LB: 0.97**\n```python\nseizure_vote 0.152810\nlpd_vote 0.142456\ngpd_vote 0.104062\nlrda_vote 0.065407\ngrda_vote 0.114851\nother_vote 0.420414\n```\n\n## Conclusion: Test set have non-overlapping egg based sequences\nwith non-overlapping based on p id\n**LB: 1.28**\n```python\nseizure_vote 0.310718\nlpd_vote 0.046279\ngpd_vote 0.051885\nlrda_vote 0.081796\ngrda_vote 0.231471\nother_vote 0.277851\n```\n\n## Conclusion: Test set have multiple patient_id sequences\n\n# Overall : Test set have repeat patients with \"non overlapping eeg\".\n\n# [Notebook EDA train.csv](https://www.kaggle.com/code/seshurajup/eda-train-csv)", "2596092": "Thanks for EDA. Have you tried submitting the train means as test predictions? I'm curious if that beats the sample submission 1/6 predictions.", "2596119": "cdeotte Thank you. Yes mean with overlaps and mean without overlaps\n\n#### **with Overlaps mean: 1.1**\n### **without Overlaps mean: 0.97 (beats the sample submission)**", "2598950": "seshurajup Have you tried submitting the means of non-overlaps spectrograms? (as shown in code below)\n\n total_votes_per_spec = train.groupby('spectrogram_id')[targets].sum().sum(axis=1)\n normalized_votes = train.groupby('spectrogram_id')[targets].sum().div(total_votes_per_spec, axis=0)\n mean_vote_ratio = normalized_votes.mean()\n print( mean_vote_ratio )\n\n seizure_vote 0.174031\n lpd_vote 0.112700\n gpd_vote 0.090854\n lrda_vote 0.071484\n grda_vote 0.136408\n other_vote 0.414523\n\nThis is slightly different than non-overlaps eeg\n\n seizure_vote 0.152810\n lpd_vote 0.142456\n gpd_vote 0.104062\n lrda_vote 0.065407\n grda_vote 0.114851\n other_vote 0.420414\n\nI'm curious which is better. This will give us insight about the best way to train our models.", "2598973": "What is your definition of overlap here? Overlapping 50 second entire subsample or central 10 seconds?", "2598983": "cdeotte I didn't submitted spectrograms based, only did with egg_id based. Curious how it help **insight about the best way to train our models** ?", "2598988": "Overlap = overlapping spectrogram ids\n\nThe best public notebook of 0.97 (and LGB/XGB 0.96) use \"non-overlapping eeg\". We take the average of the train targets from the 17089 unique `eeg_id` in train.csv.\n\nTo submit \"non-overlapping spectrogram\" we can take the average of the train targets from the 11138 unique `spectrogram_id` in train.csv. I posted the code to do this above.", "2598991": "gunesevitan for submissions I treated overlaps for eegs is 50secs", "2598994": "Whichever has the best LB score will tell us how to set our train sample weights during training. (And inform us how to compute CV score).\n\nFor example, we can give each row of `train.csv` weight 1 during training. Or we can give each unique `eeg_id` weight 1 during training. You already submitted these two cases and using unique `eeg_id` weight is better.\n\nIf unique `spectrogram_id` is better, then we can give each `spectrogram_id` weight 1 during training.", "2599005": "cdeotte if I understood correctly as follow \n1. Average non-overlapping eeg based score -> 0.97\n2. Average non-overlapped spectrogram based score -> x\n3. LGB/XGB non-overlapped egg based score -> 0.96\n\nusing these scores to sample the train dataset to build cv?", "2599014": "Understanding the test data will help us both train and validate our models more accurately. \n\nThe metric KL-Div is very sensitive to global means.", "2599017": "Thanks, its new learning for me @cdeotte", "2599036": "**LB 1.0** ( updated on topic too ) @cdeotte \n>with overlapping --> 1.1\nwithout overlapping based spectrogram --> 1.0\n**without overlapping based eeg --> 0.97**", "2599038": "Thanks for checking @seshurajup . I will continue to train and validate my models giving each unique `eeg_id` weight 1.\n\n~~Note in your discussion post, you list the means of eeg_id for both eeg_id and spectrogram_id.~~", "2599041": "cdeotte Fixed thanks", "2599058": "These are good LB probing experiments. Based on what you submitted so far, it seems that the train data has \"overlapping eeg\" whereas the test data has \"non-overlapping eeg\". This is said in the competition description, but it is always good to verify ourselves.\n\n>test.csv Metadata for the test set. As there are no overlapping samples in the test set, many columns in the train metadata don't apply.\n\nThe next question i'm curious about is if the same patient appears multiple times in the test data. In the train data, the same patient does appear multiple times. For example if we compute the mean from train with \"non-overlapping patients\" we get much different means (than \"overlapping patients\"):\n\n total_votes_per_pat = train.groupby('patient_id')[targets].sum().sum(axis=1)\n normalized_votes = train.groupby('patient_id')[targets].sum().div(total_votes_per_pat, axis=0)\n mean_vote_ratio = normalized_votes.mean()\n print( mean_vote_ratio )\n\n seizure_vote 0.310718\n lpd_vote 0.046279\n gpd_vote 0.051885\n lrda_vote 0.081796\n grda_vote 0.231471\n other_vote 0.277851\n\nIf these means produce a better LB score (than 0.97), then the test data is most likely unique patients (i.e. \"non overlapping patients\"). If these means do not produce a better LB score, then the test data is most likely repeat patients with \"non overlapping eeg\". (Of course we can also submit an \"if-else\" test to LB. Our submit code can check if a test patient repeats, then create submission.csv with 1/6 preds else create submission.csv with \"non-overlapp eeg_id\" preds. But probing LB score with means is more fun).\n\nIf the test data has repeated patients, then perhaps we can use one row of test data to help us predict another row of test data.", "2599250": "Its great learning with you (@cdeotte) and @hengck23 (other competitions). Its motivate me and learn new things every time i see the knowledge share from last 4 years. Thanks for it ( Today lesson - **Power of Mean** )\n\n**LB: 1.28**\nseizure_vote 0.310718\nlpd_vote 0.046279\ngpd_vote 0.051885\nlrda_vote 0.081796\ngrda_vote 0.231471\nother_vote 0.277851\n\n**conclusion** - repeat patients with \"non overlapping eeg\".", "2599328": "Thanks for checking this. If test has multiple patient_id, then we can use `patient_id count z_score` as a feature. From train data, it appears when patient id count z_score is small, then patient is more likely `seizure` or `grda`. And when patient count z_score is large, target is more likely `gpd` or `lpd`.\n\nWe should be able to train model using only this one feature `patient_id count z_score` and beat LB 0.97. Note that we will need to standardize the count feature so that it generalizes from train to test. For example, the feature will be the `z_score = (count - count_mean) / count_std`. So during train `count_mean = total patients in train` and during infer `count_mean = total patients in test`. Likewise `count_std` is computed once during train and once during infer.", "2618121": "cdeotte i have train and test the using CatBoost Classifier using feature `patient_id count z_score` and i got following results:\nCV : 1.185\nLB : 1.01\n\nHere is the feature calculation:\n```python\ntmp = df.groupby('patient_id').size()\neeg_id_count_per_patient_id = pd.DataFrame({'patient_id':tmp.index,'eeg_count':tmp.values})\nmean_value = np.mean(eeg_id_count_per_patient_id[\"eeg_count\"])\nstd_value = np.std(eeg_id_count_per_patient_id[\"eeg_count\"])\n\neeg_id_count_per_patient_id[\"eeg_count_z_score\"] = (eeg_id_count_per_patient_id[\"eeg_count\"] - mean_value)/std_value\neeg_id_count_per_patient_id = eeg_id_count_per_patient_id.drop(columns=[\"eeg_count\"])\n```\n\nSo, this LB is not able to beat LB 0.97. \n\nNotebook Link [https://www.kaggle.com/code/deepaksingh47/catboost-starter-with-patient-eeg-count-z-score#CV-Score-for-CatBoost](url)", "2618149": "Thanks for doing the test. I also performed a test with my best model. Including patient z score improved CV score but hurt LB score.", "2618154": "It is interesting that if we include z_score along with other features, then z_score becomes the most important feature. Unfortuntately it doesn't seem to work on test.\n\n@deepaksingh47 Note that your notebook has a leak. We cannot compute the z_score once. We must compute the z_score each fold. In the `folds for-loop`. We compute z_score for the train folds. And then we compute z_score for the valid folds. Then for fold 2, we delete previous z_score and compute z_score again. (In total we compute z_score on 10 dataframes for 5 KFold)", "2618814": "cdeotte, thanks for pointing that out. i forget about that.", "2619937": "Hi, Many thanks for your discussion post and the eda nbs as they provide many insights into the data. Could you pls share how you calculated these particular values (the ones you call \"with overlapping LB: 1.1\". While I am able to understand many of the other calculations based on per eeg_id, per spectrogram_id, per row etc, I am unable to grasp how these numbers are calculated.\nwith overlapping LB: 1.1\n\nseizure_vote 0.196002,\ngpd_vote 0.156386,\nlrda_vote 0.155805,\nother_vote 0.17610,\ngrda_vote 0.17660,\nlpd_vote 0.139101\n\nThanks.", "2619980": "skr1125 from [Notebook v16 - EDA train.csv](https://www.kaggle.com/code/seshurajup/eda-train-csv?scriptVersionId=158496297)\n```python\n from collections import Counter\ntarget_votes = Counter(list(train['expert_consensus']))\ntarget_votes = {f\"{k.lower()}_vote\":v for k,v in target_votes.items()}\ntotal_votes = sum([v for _,v in target_votes.items()])\nmean_vote_ratio = {k:(target_votes[k]/total_votes) for k,target in target_votes.items()}\n```", "2633353": "ok, understand", "2641812": "Thanks for providing this, and thanks to all the commentators. Insightful information.", "2704278": "Could anyone be kind enough to explain the first conclusion along with what the author means by \n\"with Equal probability 1/6: LB 1.09\n\nWith Overlapping LB: 1.1\"\n\nI tried to get the same numbers but only in this scenario I failed. I am assuming equal probability means author put all targets value as 0.17 and submitted it. I can't figure out what he means by with overlapping. \nAlso what does getting better score by equal probability mean? In the other 2 conclusions I understand that the case in which better score is achieved we can assume then test case has similar data. But for 1st conclusion I am having a hard time understanding if anyone could explain it I would be very grateful. Thanks in advance.", "2705719": "same i also stuck with this" }, "source": "meta" }