{ "id": 207577, "title": "3-stage training with additional annotation [CV:0.95x]", "url": "/competitions/ranzcr-clip-catheter-line-classification/discussion/207577", "author_name": "Y.Nakama", "post_date": "2020-12-30T11:29:47.003000", "votes": 149, "comment_count": 26, "views": 0, "content": "

The simplest way to use additional annotation is already discussed in this thread by @hengck23.
\nI implemented this kind of approach on kaggle notebooks (you can select GPU/TPU) and local validation score can get 0.95x. In my experience, there is 0.01 improvement with this approach.

\n

Training strategy

\n\n

Hope this helps, happy kaggling :)

", "messages": [ { "id": 1132394, "postDate": "2020-12-30T11:29:47.003Z", "content": "

The simplest way to use additional annotation is already discussed in this thread by @hengck23.
\nI implemented this kind of approach on kaggle notebooks (you can select GPU/TPU) and local validation score can get 0.95x. In my experience, there is 0.01 improvement with this approach.

\n

Training strategy

\n\n

Hope this helps, happy kaggling :)

", "rawMarkdown": "The simplest way to use additional annotation is already discussed in [this thread](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/205243) by @hengck23.\nI implemented this kind of approach on kaggle notebooks (you can select GPU/TPU) and local validation score can get 0.95x. In my experience, there is 0.01 improvement with this approach.\n\n## Training strategy\n- [1st-stage training](https://www.kaggle.com/yasufuminakama/ranzcr-resnet200d-3-stage-training-step1)\n - teacher model training for annotated image\n - data: annotated data\n - pretrained weight: imagenet weight\n - `BCEWithLogitsLoss(y_preds, labels)`\n - `y_preds: teacher model predictions for annotated image`\n- [2nd-stage training](https://www.kaggle.com/yasufuminakama/ranzcr-resnet200d-3-stage-training-step2)\n - student model training with teacher model features\n - data: annotated data\n - student model pretrained weight: imagenet weight\n - teacher model pretrained weight: 1st-stage weight\n - `BCEWithLogitsLoss(y_preds, labels) + w * MSELoss(student_features, teacher_features)`\n - `y_preds: student model predictions for normal image`\n - `student_features: student model features for normal image`\n - `teacher_features: teacher model features for annotated image`\n- [3rd-stage training](https://www.kaggle.com/yasufuminakama/ranzcr-resnet200d-3-stage-training-step3)\n - model training\n - data: all data\n - pretrained weight: 2nd-stage weight\n - `BCEWithLogitsLoss(y_preds, labels)`\n - `y_preds: student model predictions for normal image`\n- [inference notebook](https://www.kaggle.com/yasufuminakama/ranzcr-resnet200d-3-stage-training-sub)\n\nHope this helps, happy kaggling :)", "votes": 149 }, { "id": 1132504, "postDate": "2020-12-30T12:58:31.920Z", "content": "

When I first joined Kaggle few years ago, I was amazed to see a brilliant 15 or (16) y.o guy @anokas who used to wrote quickly very robust starter kernels at almost every competition.

\n

It seems now @xhlulu and you have brilliantly taken it over for TF and Pytorch respectively.

\n

Even though I have my own workflow and Pipeline. Your kernels are always a great inspiration to adapt mines on new competitions.

\n

Many thanks and keep up the great work !

\n

And of course huge thanks to @hengck23 for giving too much inspirations and ideas to all of us.

", "rawMarkdown": "When I first joined Kaggle few years ago, I was amazed to see a brilliant 15 or (16) y.o guy @anokas who used to wrote quickly very robust starter kernels at almost every competition. \n\nIt seems now @xhlulu and you have brilliantly taken it over for TF and Pytorch respectively. \n\nEven though I have my own workflow and Pipeline. Your kernels are always a great inspiration to adapt mines on new competitions. \n\nMany thanks and keep up the great work ! \n\nAnd of course huge thanks to @hengck23 for giving too much inspirations and ideas to all of us. ", "votes": 7, "replies": [ { "id": 1132511, "postDate": "2020-12-30T13:05:14Z", "content": "

Thank you! :)

", "rawMarkdown": "Thank you! :)" } ] }, { "id": 1132554, "postDate": "2020-12-30T13:44:41.333Z", "content": "

@yasufuminakama

\n

thanks for the results and experiments.
\n I only verify that it would work based on my experiments. but I haven't implemented and submitted it yet.
\nbased on my experiments, the potential increase can be about 1%.

\n

you can check the CAM map difference before and after the approach.

", "rawMarkdown": "@yasufuminakama \n\nthanks for the results and experiments.\n I only verify that it would work based on my experiments. but I haven't implemented and submitted it yet.\nbased on my experiments, the potential increase can be about 1%.\n\nyou can check the CAM map difference before and after the approach.\n\n", "votes": 5 }, { "id": 1135016, "postDate": "2021-01-01T19:45:37.923Z", "content": "

It's always amazing how ideas turn into reality. :)

\n

Thanks @hengck23 @yasufuminakama.

", "rawMarkdown": "It's always amazing how ideas turn into reality. :)\n\nThanks @hengck23 @yasufuminakama.", "votes": 1 }, { "id": 1133846, "postDate": "2020-12-31T15:26:01.370Z", "content": "

Is this Knowledge Distillation? or another technique?

", "rawMarkdown": "Is this **Knowledge Distillation**? or another technique?", "votes": 2 }, { "id": 1132438, "postDate": "2020-12-30T12:08:28.443Z", "content": "

Unfortunately can upvote only once all these kernels :( hats off @yasufuminakama !

", "rawMarkdown": "Unfortunately can upvote only once all these kernels :( hats off @yasufuminakama !", "votes": 2, "replies": [ { "id": 1132441, "postDate": "2020-12-30T12:13:12.513Z", "content": "

Thanks for your upvote @atanasova :)

", "rawMarkdown": "Thanks for your upvote @atanasova :)" } ] }, { "id": 1230264, "postDate": "2021-03-08T00:22:59.380Z", "content": "

Hey @yasufuminakama, thanks for this. What is 'w' in the 2nd stage loss. A constant?

", "rawMarkdown": "Hey @yasufuminakama, thanks for this. What is 'w' in the 2nd stage loss. A constant?", "replies": [ { "id": 1236032, "postDate": "2021-03-12T17:31:20.207Z", "content": "

Yes, this gives weightage to MSELoss.
\nFor me 1 for 'BCEWithLogitsLoss' and 0.5 for 'MSELoss' worked best.

", "rawMarkdown": "Yes, this gives weightage to MSELoss. \nFor me 1 for 'BCEWithLogitsLoss' and 0.5 for 'MSELoss' worked best." } ] }, { "id": 1225263, "postDate": "2021-03-03T13:38:29.990Z", "content": "

Great work, thanx

", "rawMarkdown": "Great work, thanx" }, { "id": 1224982, "postDate": "2021-03-03T08:09:17.320Z", "content": "

Thanks, Y.Nakama for your insightful and helpful posts.

\n

May I ask a newbie's question?

\n

How do you do training in stages 1 and 2? I mean, In understand that for stage 3 a conventional CV Stratified Group K-fold strategy can be applied, but for stages 1 and 2 I'm not sure. Maybe in stage 1 train the teacher model with all the data, without validation, and for stage 2 reserving a holdout set for validation for training the student model?

\n

Thanks, and sorry for my dumb question.

", "rawMarkdown": "Thanks, Y.Nakama for your insightful and helpful posts.\n\nMay I ask a newbie's question?\n\nHow do you do training in stages 1 and 2? I mean, In understand that for stage 3 a conventional CV Stratified Group K-fold strategy can be applied, but for stages 1 and 2 I'm not sure. Maybe in stage 1 train the teacher model with all the data, without validation, and for stage 2 reserving a holdout set for validation for training the student model?\n\nThanks, and sorry for my dumb question." }, { "id": 1148861, "postDate": "2021-01-11T12:40:52.253Z", "content": "

Hi,I have a question.How do we adjust the params 'weights' in step2 training,you gaved is [0.5,1.0].Thanks!

", "rawMarkdown": "Hi,I have a question.How do we adjust the params 'weights' in step2 training,you gaved is [0.5,1.0].Thanks!" }, { "id": 1148629, "postDate": "2021-01-11T09:19:17.293Z", "content": "

Thanks for sharing such great work!
\nCould you tell me why the COLOR MAP trick works on this project? I mean it is likely that the teacher model would be supervised by the COLOR signals rather than the semantic information of the X-ray.

", "rawMarkdown": "Thanks for sharing such great work!\nCould you tell me why the COLOR MAP trick works on this project? I mean it is likely that the teacher model would be supervised by the COLOR signals rather than the semantic information of the X-ray. ", "replies": [ { "id": 1148705, "postDate": "2021-01-11T10:29:54.037Z", "content": "

in summary:
\n1) human input supervision by color
\n2)teacher convert supervision to feature map
\n3)student must find real signals to produce the same feature map as the teacher
\n(i.e. find the image features that is equivalent as if the image are colored)

\n

the role of the teacher is to convert human supervision to feature map supervision

", "rawMarkdown": "in summary:\n1) human input supervision by color\n2)teacher convert supervision to feature map\n3)student must find real signals to produce the same feature map as the teacher\n(i.e. find the image features that is equivalent as if the image are colored)\n\nthe role of the teacher is to convert human supervision to feature map supervision", "votes": 11 } ] }, { "id": 1147692, "postDate": "2021-01-10T16:43:04.007Z", "content": "

Hi @yasufuminakama thanks for sharing an excellent series of notebooks, I've learned a lot from them. In stage 2, have you tried other losses here? Seems like KL divergence would make a good fit.

", "rawMarkdown": "Hi @yasufuminakama thanks for sharing an excellent series of notebooks, I've learned a lot from them. In stage 2, have you tried other losses here? Seems like KL divergence would make a good fit.", "replies": [ { "id": 1148696, "postDate": "2021-01-11T10:20:45.480Z", "content": "

You're welcome :) I haven't tried other losses yet.

", "rawMarkdown": "You're welcome :) I haven't tried other losses yet." }, { "id": 1158524, "postDate": "2021-01-18T16:02:48.667Z", "content": "

Hi,did you test the KL loss,I test it ,but I get a lower score in stage2.It is only get 0.776.

", "rawMarkdown": "Hi,did you test the KL loss,I test it ,but I get a lower score in stage2.It is only get 0.776." } ] }, { "id": 1141302, "postDate": "2021-01-06T15:59:36.410Z", "content": "

Hey, great work and thank you for ideas and the code. Side question though. I noticed you are training on TPU with nproc=1. Have you seen speed or batch size improvement comparing pytorch xla on TPU and regular Pytorch on GPU? Have you managed to make it work on all 8 cores of TPU accelerator?

\n

Thanks in advance)
\nCheers,
\nAlexey

", "rawMarkdown": "Hey, great work and thank you for ideas and the code. Side question though. I noticed you are training on TPU with `nproc=1`. Have you seen speed or batch size improvement comparing pytorch xla on TPU and regular Pytorch on GPU? Have you managed to make it work on all 8 cores of TPU accelerator?\n\nThanks in advance)\nCheers,\nAlexey", "replies": [ { "id": 1148702, "postDate": "2021-01-11T10:27:29.253Z", "content": "

In my experiment, GPU was faster than single core of TPU.

\n
\n

Have you managed to make it work on all 8 cores of TPU accelerator?

\n
\n

No, I'm not sure but maybe something is wrong…

", "rawMarkdown": "In my experiment, GPU was faster than single core of TPU.\n\n> Have you managed to make it work on all 8 cores of TPU accelerator?\n\nNo, I'm not sure but maybe something is wrong..." } ] }, { "id": 1135664, "postDate": "2021-01-02T12:28:50.667Z", "content": "

Can someone please explain the rationale of this 3 step process? What added benefit it has over the normal training?
\nAlso, what is a Teacher model and Student model - is this in context to Transfer learning?

", "rawMarkdown": "Can someone please explain the rationale of this 3 step process? What added benefit it has over the normal training?\nAlso, what is a Teacher model and Student model - is this in context to Transfer learning? \n ", "replies": [ { "id": 1135666, "postDate": "2021-01-02T12:30:04.460Z", "content": "

@yasufuminakama, @piantic, @hiramcho please shed some light.

", "rawMarkdown": " @yasufuminakama, @piantic, @hiramcho please shed some light." }, { "id": 1135667, "postDate": "2021-01-02T12:31:01.287Z", "content": "

Also, goes without saying, @yasufuminakama you for sharing your approach.

", "rawMarkdown": "Also, goes without saying, @yasufuminakama you for sharing your approach.", "votes": -2 }, { "id": 1135932, "postDate": "2021-01-02T15:53:13.033Z", "content": "

Please check this thread https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/205243

", "rawMarkdown": "Please check this thread https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/205243", "votes": 1 }, { "id": 1136579, "postDate": "2021-01-03T08:10:50.673Z", "content": "

@yasufuminakama Thank you!

", "rawMarkdown": "@yasufuminakama Thank you!", "votes": 1 } ] }, { "id": 1244437, "postDate": "2021-03-19T01:58:29.510Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1132903, "postDate": "2020-12-30T19:15:52.580Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1132504, "author_name": "Serigne ", "author_url": "", "post_date": "2020-12-30T12:58:31.920000", "content": "

When I first joined Kaggle few years ago, I was amazed to see a brilliant 15 or (16) y.o guy @anokas who used to wrote quickly very robust starter kernels at almost every competition.

\n

It seems now @xhlulu and you have brilliantly taken it over for TF and Pytorch respectively.

\n

Even though I have my own workflow and Pipeline. Your kernels are always a great inspiration to adapt mines on new competitions.

\n

Many thanks and keep up the great work !

\n

And of course huge thanks to @hengck23 for giving too much inspirations and ideas to all of us.

", "votes": 7, "replies": [ { "id": 1132511, "author_name": "Y.Nakama", "author_url": "", "post_date": "2020-12-30T13:05:14", "content": "

Thank you! :)

", "votes": 0, "replies": [] } ] }, { "id": 1132554, "author_name": "hengck23", "author_url": "", "post_date": "2020-12-30T13:44:41.333000", "content": "

@yasufuminakama

\n

thanks for the results and experiments.
\n I only verify that it would work based on my experiments. but I haven't implemented and submitted it yet.
\nbased on my experiments, the potential increase can be about 1%.

\n

you can check the CAM map difference before and after the approach.

", "votes": 5, "replies": [] }, { "id": 1135016, "author_name": "Heroseo", "author_url": "", "post_date": "2021-01-01T19:45:37.923000", "content": "

It's always amazing how ideas turn into reality. :)

\n

Thanks @hengck23 @yasufuminakama.

", "votes": 1, "replies": [] }, { "id": 1133846, "author_name": "Hiram Coria 🧬", "author_url": "", "post_date": "2020-12-31T15:26:01.370000", "content": "

Is this Knowledge Distillation? or another technique?

", "votes": 2, "replies": [] }, { "id": 1132438, "author_name": "Atanas Atanasov", "author_url": "", "post_date": "2020-12-30T12:08:28.443000", "content": "

Unfortunately can upvote only once all these kernels :( hats off @yasufuminakama !

", "votes": 2, "replies": [ { "id": 1132441, "author_name": "Y.Nakama", "author_url": "", "post_date": "2020-12-30T12:13:12.513000", "content": "

Thanks for your upvote @atanasova :)

", "votes": 0, "replies": [] } ] }, { "id": 1230264, "author_name": "James Condon", "author_url": "", "post_date": "2021-03-08T00:22:59.380000", "content": "

Hey @yasufuminakama, thanks for this. What is 'w' in the 2nd stage loss. A constant?

", "votes": 0, "replies": [ { "id": 1236032, "author_name": "Aman Deep Gupta", "author_url": "", "post_date": "2021-03-12T17:31:20.207000", "content": "

Yes, this gives weightage to MSELoss.
\nFor me 1 for 'BCEWithLogitsLoss' and 0.5 for 'MSELoss' worked best.

", "votes": 0, "replies": [] } ] }, { "id": 1225263, "author_name": "Evgeny Sidorov", "author_url": "", "post_date": "2021-03-03T13:38:29.990000", "content": "

Great work, thanx

", "votes": 0, "replies": [] }, { "id": 1224982, "author_name": "jcesquivel", "author_url": "", "post_date": "2021-03-03T08:09:17.320000", "content": "

Thanks, Y.Nakama for your insightful and helpful posts.

\n

May I ask a newbie's question?

\n

How do you do training in stages 1 and 2? I mean, In understand that for stage 3 a conventional CV Stratified Group K-fold strategy can be applied, but for stages 1 and 2 I'm not sure. Maybe in stage 1 train the teacher model with all the data, without validation, and for stage 2 reserving a holdout set for validation for training the student model?

\n

Thanks, and sorry for my dumb question.

", "votes": 0, "replies": [] }, { "id": 1148861, "author_name": "Bcw93", "author_url": "", "post_date": "2021-01-11T12:40:52.253000", "content": "

Hi,I have a question.How do we adjust the params 'weights' in step2 training,you gaved is [0.5,1.0].Thanks!

", "votes": 0, "replies": [] }, { "id": 1148629, "author_name": "BIRDAD", "author_url": "", "post_date": "2021-01-11T09:19:17.293000", "content": "

Thanks for sharing such great work!
\nCould you tell me why the COLOR MAP trick works on this project? I mean it is likely that the teacher model would be supervised by the COLOR signals rather than the semantic information of the X-ray.

", "votes": 0, "replies": [ { "id": 1148705, "author_name": "hengck23", "author_url": "", "post_date": "2021-01-11T10:29:54.037000", "content": "

in summary:
\n1) human input supervision by color
\n2)teacher convert supervision to feature map
\n3)student must find real signals to produce the same feature map as the teacher
\n(i.e. find the image features that is equivalent as if the image are colored)

\n

the role of the teacher is to convert human supervision to feature map supervision

", "votes": 11, "replies": [] } ] }, { "id": 1147692, "author_name": "datasaurus", "author_url": "", "post_date": "2021-01-10T16:43:04.007000", "content": "

Hi @yasufuminakama thanks for sharing an excellent series of notebooks, I've learned a lot from them. In stage 2, have you tried other losses here? Seems like KL divergence would make a good fit.

", "votes": 0, "replies": [ { "id": 1148696, "author_name": "Y.Nakama", "author_url": "", "post_date": "2021-01-11T10:20:45.480000", "content": "

You're welcome :) I haven't tried other losses yet.

", "votes": 0, "replies": [] }, { "id": 1158524, "author_name": "Bcw93", "author_url": "", "post_date": "2021-01-18T16:02:48.667000", "content": "

Hi,did you test the KL loss,I test it ,but I get a lower score in stage2.It is only get 0.776.

", "votes": 0, "replies": [] } ] }, { "id": 1141302, "author_name": "A.Demyanchuk", "author_url": "", "post_date": "2021-01-06T15:59:36.410000", "content": "

Hey, great work and thank you for ideas and the code. Side question though. I noticed you are training on TPU with nproc=1. Have you seen speed or batch size improvement comparing pytorch xla on TPU and regular Pytorch on GPU? Have you managed to make it work on all 8 cores of TPU accelerator?

\n

Thanks in advance)
\nCheers,
\nAlexey

", "votes": 0, "replies": [ { "id": 1148702, "author_name": "Y.Nakama", "author_url": "", "post_date": "2021-01-11T10:27:29.253000", "content": "

In my experiment, GPU was faster than single core of TPU.

\n
\n

Have you managed to make it work on all 8 cores of TPU accelerator?

\n
\n

No, I'm not sure but maybe something is wrong…

", "votes": 0, "replies": [] } ] }, { "id": 1135664, "author_name": "Dr. Amritpal Singh", "author_url": "", "post_date": "2021-01-02T12:28:50.667000", "content": "

Can someone please explain the rationale of this 3 step process? What added benefit it has over the normal training?
\nAlso, what is a Teacher model and Student model - is this in context to Transfer learning?

", "votes": 0, "replies": [ { "id": 1135666, "author_name": "Dr. Amritpal Singh", "author_url": "", "post_date": "2021-01-02T12:30:04.460000", "content": "

@yasufuminakama, @piantic, @hiramcho please shed some light.

", "votes": 0, "replies": [] }, { "id": 1135667, "author_name": "Dr. Amritpal Singh", "author_url": "", "post_date": "2021-01-02T12:31:01.287000", "content": "

Also, goes without saying, @yasufuminakama you for sharing your approach.

", "votes": -2, "replies": [] }, { "id": 1135932, "author_name": "Y.Nakama", "author_url": "", "post_date": "2021-01-02T15:53:13.033000", "content": "

Please check this thread https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/205243

", "votes": 1, "replies": [] }, { "id": 1136579, "author_name": "Dr. Amritpal Singh", "author_url": "", "post_date": "2021-01-03T08:10:50.673000", "content": "

@yasufuminakama Thank you!

", "votes": 1, "replies": [] } ] }, { "id": 1244437, "author_name": "", "author_url": "", "post_date": "2021-03-19T01:58:29.510000", "content": "", "votes": 0, "replies": [] }, { "id": 1132903, "author_name": "", "author_url": "", "post_date": "2020-12-30T19:15:52.580000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1132394": "The simplest way to use additional annotation is already discussed in [this thread](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/205243) by @hengck23.\nI implemented this kind of approach on kaggle notebooks (you can select GPU/TPU) and local validation score can get 0.95x. In my experience, there is 0.01 improvement with this approach.\n\n## Training strategy\n- [1st-stage training](https://www.kaggle.com/yasufuminakama/ranzcr-resnet200d-3-stage-training-step1)\n - teacher model training for annotated image\n - data: annotated data\n - pretrained weight: imagenet weight\n - `BCEWithLogitsLoss(y_preds, labels)`\n - `y_preds: teacher model predictions for annotated image`\n- [2nd-stage training](https://www.kaggle.com/yasufuminakama/ranzcr-resnet200d-3-stage-training-step2)\n - student model training with teacher model features\n - data: annotated data\n - student model pretrained weight: imagenet weight\n - teacher model pretrained weight: 1st-stage weight\n - `BCEWithLogitsLoss(y_preds, labels) + w * MSELoss(student_features, teacher_features)`\n - `y_preds: student model predictions for normal image`\n - `student_features: student model features for normal image`\n - `teacher_features: teacher model features for annotated image`\n- [3rd-stage training](https://www.kaggle.com/yasufuminakama/ranzcr-resnet200d-3-stage-training-step3)\n - model training\n - data: all data\n - pretrained weight: 2nd-stage weight\n - `BCEWithLogitsLoss(y_preds, labels)`\n - `y_preds: student model predictions for normal image`\n- [inference notebook](https://www.kaggle.com/yasufuminakama/ranzcr-resnet200d-3-stage-training-sub)\n\nHope this helps, happy kaggling :)", "1132504": "When I first joined Kaggle few years ago, I was amazed to see a brilliant 15 or (16) y.o guy @anokas who used to wrote quickly very robust starter kernels at almost every competition. \n\nIt seems now @xhlulu and you have brilliantly taken it over for TF and Pytorch respectively. \n\nEven though I have my own workflow and Pipeline. Your kernels are always a great inspiration to adapt mines on new competitions. \n\nMany thanks and keep up the great work ! \n\nAnd of course huge thanks to @hengck23 for giving too much inspirations and ideas to all of us. ", "1132554": "@yasufuminakama \n\nthanks for the results and experiments.\n I only verify that it would work based on my experiments. but I haven't implemented and submitted it yet.\nbased on my experiments, the potential increase can be about 1%.\n\nyou can check the CAM map difference before and after the approach.\n\n", "1135016": "It's always amazing how ideas turn into reality. :)\n\nThanks @hengck23 @yasufuminakama.", "1133846": "Is this **Knowledge Distillation**? or another technique?", "1132438": "Unfortunately can upvote only once all these kernels :( hats off @yasufuminakama !", "1230264": "Hey @yasufuminakama, thanks for this. What is 'w' in the 2nd stage loss. A constant?", "1225263": "Great work, thanx", "1224982": "Thanks, Y.Nakama for your insightful and helpful posts.\n\nMay I ask a newbie's question?\n\nHow do you do training in stages 1 and 2? I mean, In understand that for stage 3 a conventional CV Stratified Group K-fold strategy can be applied, but for stages 1 and 2 I'm not sure. Maybe in stage 1 train the teacher model with all the data, without validation, and for stage 2 reserving a holdout set for validation for training the student model?\n\nThanks, and sorry for my dumb question.", "1148861": "Hi,I have a question.How do we adjust the params 'weights' in step2 training,you gaved is [0.5,1.0].Thanks!", "1148629": "Thanks for sharing such great work!\nCould you tell me why the COLOR MAP trick works on this project? I mean it is likely that the teacher model would be supervised by the COLOR signals rather than the semantic information of the X-ray. ", "1147692": "Hi @yasufuminakama thanks for sharing an excellent series of notebooks, I've learned a lot from them. In stage 2, have you tried other losses here? Seems like KL divergence would make a good fit.", "1141302": "Hey, great work and thank you for ideas and the code. Side question though. I noticed you are training on TPU with `nproc=1`. Have you seen speed or batch size improvement comparing pytorch xla on TPU and regular Pytorch on GPU? Have you managed to make it work on all 8 cores of TPU accelerator?\n\nThanks in advance)\nCheers,\nAlexey", "1135664": "Can someone please explain the rationale of this 3 step process? What added benefit it has over the normal training?\nAlso, what is a Teacher model and Student model - is this in context to Transfer learning? \n ", "1244437": "", "1132903": "" } }