{ "id": 486805, "title": "Knowledge Distillation", "url": "/competitions/hms-harmful-brain-activity-classification/discussion/486805", "author_name": "", "post_date": "2024-03-26T11:51:08.019998800Z", "votes": 8, "comment_count": 14, "views": 0, "content": "

Hello Everyone,

\n

Knowledge distillation seems to be giving good improvments, (i.e single model from 0.37 to 0.34 LB)
\nI have shared this distilled knowledge dataset that is based on 0.3 LB ensemble predictions.

\n

To find out how to use it, you can check out the code in the Features+Head Starter

", "messages": [ { "id": "2717129", "postDate": "03/26/2024 11:51:08", "content": "

Hello Everyone,

\n

Knowledge distillation seems to be giving good improvments, (i.e single model from 0.37 to 0.34 LB)
\nI have shared this distilled knowledge dataset that is based on 0.3 LB ensemble predictions.

\n

To find out how to use it, you can check out the code in the Features+Head Starter

", "rawMarkdown": "Hello Everyone,\n\nKnowledge distillation seems to be giving good improvments, (i.e single model from 0.37 to 0.34 LB) \nI have shared [this distilled knowledge dataset](https://www.kaggle.com/datasets/nartaa/knowledge-distillation/data) that is based on 0.3 LB ensemble predictions.\n\nTo find out how to use it, you can check out the code in the [Features+Head Starter] (https://www.kaggle.com/code/nartaa/features-head-starter)", "votes": null }, { "id": "2717377", "postDate": "03/26/2024 14:40:16", "content": "

Thanks for providing the dataset. Are these OOF predictions from the ensembled models or just train in sample?

", "rawMarkdown": "Thanks for providing the dataset. Are these OOF predictions from the ensembled models or just train in sample?", "votes": null }, { "id": "2717402", "postDate": "03/26/2024 14:50:18", "content": "

You are welcome.
\nNot OOF, every fold predicts on all training samples.

", "rawMarkdown": "You are welcome.\nNot OOF, every fold predicts on all training samples.", "votes": null }, { "id": "2717527", "postDate": "03/26/2024 16:10:14", "content": "

Does it have any effect on your integration?

", "rawMarkdown": "Does it have any effect on your integration?", "votes": null }, { "id": "2717560", "postDate": "03/26/2024 16:37:45", "content": "

I ran out of GPU time this week, so I couldn't say, but it should

", "rawMarkdown": "I ran out of GPU time this week, so I couldn't say, but it should", "votes": null }, { "id": "2718176", "postDate": "03/27/2024 02:45:35", "content": "

What do you mean by KD here in this context? Are you feeding a model the ensemble's predictions on train.csv as a ground truth? In essence, the smaller/simplier model will just learn to emulate the ensemble - don't really see the benefit of it when the simpler model is then placed as part of that ensemble

", "rawMarkdown": "What do you mean by KD here in this context? Are you feeding a model the ensemble's predictions on train.csv as a ground truth? In essence, the smaller/simplier model will just learn to emulate the ensemble - don't really see the benefit of it when the simpler model is then placed as part of that ensemble", "votes": null }, { "id": "2718237", "postDate": "03/27/2024 03:22:18", "content": "

Thanks for sharing!
\nDoes the cv also improve?

", "rawMarkdown": "Thanks for sharing!\nDoes the cv also improve?", "votes": null }, { "id": "2718694", "postDate": "03/27/2024 07:49:40", "content": "

You are welcome!
\nYes, CV and LB both improve nicely.

", "rawMarkdown": "You are welcome!\nYes, CV and LB both improve nicely.", "votes": null }, { "id": "2718725", "postDate": "03/27/2024 08:01:11", "content": "
\n

What do you mean by KD here in this context?

\n
\n

Probably in this context it would mean training on pseudo labeling based on best model's predictions, the term knowledge distillation includes true labels in the loss function, but both methods distill knowledge from pseudo labels.

\n
\n

Are you feeding a model the ensemble's predictions on train.csv as a ground truth?

\n
\n

Yes.

\n
\n

don't really see the benefit of it when the simpler model is then placed as part of that ensemble

\n
\n

I have similliar intuition, but knowledge distillation is a phenomena that works against intuition, just like ensemble. I've looked at previous competitions, they utilize this method. I will conclude next week if it would actually give better LB on ensemble.

", "rawMarkdown": "> What do you mean by KD here in this context? \n\nProbably in this context it would mean training on pseudo labeling based on best model's predictions, the term knowledge distillation includes true labels in the loss function, but both methods distill knowledge from pseudo labels.\n\n> Are you feeding a model the ensemble's predictions on train.csv as a ground truth?\n\nYes.\n\n> don't really see the benefit of it when the simpler model is then placed as part of that ensemble\n\nI have similliar intuition, but knowledge distillation is a phenomena that works against intuition, just like ensemble. I've looked at previous competitions, they utilize this method. I will conclude next week if it would actually give better LB on ensemble.", "votes": null }, { "id": "2719734", "postDate": "03/27/2024 22:16:07", "content": "

Thank you for the reply, eager to see what happens (also out of GPU allocation this week) - If it improves I'd be keen to understand why!

", "rawMarkdown": "Thank you for the reply, eager to see what happens (also out of GPU allocation this week) - If it improves I'd be keen to understand why!", "votes": null }, { "id": "2722397", "postDate": "03/29/2024 15:13:48", "content": "

I retrained the KER with KD and got a single model 0.33 LB
\nhttps://www.kaggle.com/code/jimmyisme1/features-head-ker-only-lb-0-33

", "rawMarkdown": "I retrained the KER with KD and got a single model 0.33 LB\nhttps://www.kaggle.com/code/jimmyisme1/features-head-ker-only-lb-0-33", "votes": null }, { "id": "2722691", "postDate": "03/29/2024 18:06:33", "content": "

Nicely done!

", "rawMarkdown": "Nicely done!", "votes": null }, { "id": "2723259", "postDate": "03/30/2024 04:55:11", "content": "

Would it be better if I trained more models with KD and then fused them together？

", "rawMarkdown": "Would it be better if I trained more models with KD and then fused them together？", "votes": null }, { "id": "2723390", "postDate": "03/30/2024 07:02:45", "content": "

I think so, that would be my next move.

", "rawMarkdown": "I think so, that would be my next move.", "votes": null }, { "id": "2725208", "postDate": "03/31/2024 12:13:38", "content": "

how about your cv?

", "rawMarkdown": "how about your cv?", "votes": null } ], "comments": [ { "id": 2717377, "author_name": "julianmukaj", "author_url": "", "post_date": "03/26/2024 14:40:16", "content": "

Thanks for providing the dataset. Are these OOF predictions from the ensembled models or just train in sample?

", "votes": null, "replies": [ { "id": 2717402, "author_name": "nartaa", "author_url": "", "post_date": "03/26/2024 14:50:18", "content": "

You are welcome.
\nNot OOF, every fold predicts on all training samples.

", "votes": null, "replies": [] } ] }, { "id": 2717527, "author_name": "gentlezdh", "author_url": "", "post_date": "03/26/2024 16:10:14", "content": "

Does it have any effect on your integration?

", "votes": null, "replies": [ { "id": 2717560, "author_name": "nartaa", "author_url": "", "post_date": "03/26/2024 16:37:45", "content": "

I ran out of GPU time this week, so I couldn't say, but it should

", "votes": null, "replies": [] } ] }, { "id": 2718176, "author_name": "exjustice", "author_url": "", "post_date": "03/27/2024 02:45:35", "content": "

What do you mean by KD here in this context? Are you feeding a model the ensemble's predictions on train.csv as a ground truth? In essence, the smaller/simplier model will just learn to emulate the ensemble - don't really see the benefit of it when the simpler model is then placed as part of that ensemble

", "votes": null, "replies": [ { "id": 2718725, "author_name": "nartaa", "author_url": "", "post_date": "03/27/2024 08:01:11", "content": "
\n

What do you mean by KD here in this context?

\n
\n

Probably in this context it would mean training on pseudo labeling based on best model's predictions, the term knowledge distillation includes true labels in the loss function, but both methods distill knowledge from pseudo labels.

\n
\n

Are you feeding a model the ensemble's predictions on train.csv as a ground truth?

\n
\n

Yes.

\n
\n

don't really see the benefit of it when the simpler model is then placed as part of that ensemble

\n
\n

I have similliar intuition, but knowledge distillation is a phenomena that works against intuition, just like ensemble. I've looked at previous competitions, they utilize this method. I will conclude next week if it would actually give better LB on ensemble.

", "votes": null, "replies": [ { "id": 2719734, "author_name": "exjustice", "author_url": "", "post_date": "03/27/2024 22:16:07", "content": "

Thank you for the reply, eager to see what happens (also out of GPU allocation this week) - If it improves I'd be keen to understand why!

", "votes": null, "replies": [] } ] } ] }, { "id": 2718237, "author_name": "johnlemon3", "author_url": "", "post_date": "03/27/2024 03:22:18", "content": "

Thanks for sharing!
\nDoes the cv also improve?

", "votes": null, "replies": [ { "id": 2718694, "author_name": "nartaa", "author_url": "", "post_date": "03/27/2024 07:49:40", "content": "

You are welcome!
\nYes, CV and LB both improve nicely.

", "votes": null, "replies": [] } ] }, { "id": 2722397, "author_name": "jimmyisme1", "author_url": "", "post_date": "03/29/2024 15:13:48", "content": "

I retrained the KER with KD and got a single model 0.33 LB
\nhttps://www.kaggle.com/code/jimmyisme1/features-head-ker-only-lb-0-33

", "votes": null, "replies": [ { "id": 2722691, "author_name": "nartaa", "author_url": "", "post_date": "03/29/2024 18:06:33", "content": "

Nicely done!

", "votes": null, "replies": [] }, { "id": 2723259, "author_name": "majiaqi111", "author_url": "", "post_date": "03/30/2024 04:55:11", "content": "

Would it be better if I trained more models with KD and then fused them together？

", "votes": null, "replies": [ { "id": 2723390, "author_name": "nartaa", "author_url": "", "post_date": "03/30/2024 07:02:45", "content": "

I think so, that would be my next move.

", "votes": null, "replies": [] } ] }, { "id": 2725208, "author_name": "gentlezdh", "author_url": "", "post_date": "03/31/2024 12:13:38", "content": "

how about your cv?

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2717129": "Hello Everyone,\n\nKnowledge distillation seems to be giving good improvments, (i.e single model from 0.37 to 0.34 LB) \nI have shared [this distilled knowledge dataset](https://www.kaggle.com/datasets/nartaa/knowledge-distillation/data) that is based on 0.3 LB ensemble predictions.\n\nTo find out how to use it, you can check out the code in the [Features+Head Starter] (https://www.kaggle.com/code/nartaa/features-head-starter)", "2717377": "Thanks for providing the dataset. Are these OOF predictions from the ensembled models or just train in sample?", "2717402": "You are welcome.\nNot OOF, every fold predicts on all training samples.", "2717527": "Does it have any effect on your integration?", "2717560": "I ran out of GPU time this week, so I couldn't say, but it should", "2718176": "What do you mean by KD here in this context? Are you feeding a model the ensemble's predictions on train.csv as a ground truth? In essence, the smaller/simplier model will just learn to emulate the ensemble - don't really see the benefit of it when the simpler model is then placed as part of that ensemble", "2718237": "Thanks for sharing!\nDoes the cv also improve?", "2718694": "You are welcome!\nYes, CV and LB both improve nicely.", "2718725": "> What do you mean by KD here in this context? \n\nProbably in this context it would mean training on pseudo labeling based on best model's predictions, the term knowledge distillation includes true labels in the loss function, but both methods distill knowledge from pseudo labels.\n\n> Are you feeding a model the ensemble's predictions on train.csv as a ground truth?\n\nYes.\n\n> don't really see the benefit of it when the simpler model is then placed as part of that ensemble\n\nI have similliar intuition, but knowledge distillation is a phenomena that works against intuition, just like ensemble. I've looked at previous competitions, they utilize this method. I will conclude next week if it would actually give better LB on ensemble.", "2719734": "Thank you for the reply, eager to see what happens (also out of GPU allocation this week) - If it improves I'd be keen to understand why!", "2722397": "I retrained the KER with KD and got a single model 0.33 LB\nhttps://www.kaggle.com/code/jimmyisme1/features-head-ker-only-lb-0-33", "2722691": "Nicely done!", "2723259": "Would it be better if I trained more models with KD and then fused them together？", "2723390": "I think so, that would be my next move.", "2725208": "how about your cv?" }, "source": "meta" }