![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1035002%2F7a8a2b326d0e6101a6959668da52a96e%2F__results___16_1.png?generation=1611090568639975&alt=media\")
One of the highest scoring public notebooks using an interesting application of the annotations and a teacher student process to instill some extra knowledge in the final network. This process doesnt entirely make logical sense to me and decided to try a slightly different direction with the annotations that looks reasonably promising.
\nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/207577
\nMy gripe with the annotation usage in the teacher student setup is that the annotations are applied to the image and painted on in different colors for the varying classifications. Then an initial model is trained to use these inputs to classify. With the colors marking the catheter/line and their classification, the model simply needs to classify by color. \"If pixel value == (128, 0, 128) then x condition\", but this is not useful information because the colors will not be available on future classifications.
\n
You could argue that in the second stage of this training process when the student model is given the unannotated inputs and the teacher is given annotated and then their embeddings are pushed together that it is trying to inject that knowledge into the student network, but it still seems a bit obtuse and overcomplicated to me.
\nMy proposed method is instead of using the annotations as input and then do the student-teacher process, just directly teach the initial model to predict the catheter/line annotations. In this scenario using a model like a U-net we can have the model predict the 11 different labels out of the bottleneck of the U and then also output a segmentation mask showing the path of the catheter.
\nMaking some modifications to the code used for the teacher-student, adding linear interpolation between points, we can make the model predict something like this:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1035002%2F12fab9ce99226218d93b8ee2c8bc0062%2Fdownload.png?generation=1611091855826936&alt=media\")
This functions as additional supervision through the backbone of the network forcing it to learn characteristics related to the catheter and preventing it from overfitting on irrelevant features.
\nOnce again we have to deal with the fact that annotations are not available for all x-rays. We can remedy this by masking off the segmentation loss when no annotation was present so the model is not penalized for predictions we dont have the label for.
", "messages": [ { "id": 1160380, "postDate": "2021-01-19T21:37:16.623Z", "content": "One of the highest scoring public notebooks using an interesting application of the annotations and a teacher student process to instill some extra knowledge in the final network. This process doesnt entirely make logical sense to me and decided to try a slightly different direction with the annotations that looks reasonably promising.
\nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/207577
\nMy gripe with the annotation usage in the teacher student setup is that the annotations are applied to the image and painted on in different colors for the varying classifications. Then an initial model is trained to use these inputs to classify. With the colors marking the catheter/line and their classification, the model simply needs to classify by color. \"If pixel value == (128, 0, 128) then x condition\", but this is not useful information because the colors will not be available on future classifications.
\n
You could argue that in the second stage of this training process when the student model is given the unannotated inputs and the teacher is given annotated and then their embeddings are pushed together that it is trying to inject that knowledge into the student network, but it still seems a bit obtuse and overcomplicated to me.
\nMy proposed method is instead of using the annotations as input and then do the student-teacher process, just directly teach the initial model to predict the catheter/line annotations. In this scenario using a model like a U-net we can have the model predict the 11 different labels out of the bottleneck of the U and then also output a segmentation mask showing the path of the catheter.
\nMaking some modifications to the code used for the teacher-student, adding linear interpolation between points, we can make the model predict something like this:
\n
This functions as additional supervision through the backbone of the network forcing it to learn characteristics related to the catheter and preventing it from overfitting on irrelevant features.
\nOnce again we have to deal with the fact that annotations are not available for all x-rays. We can remedy this by masking off the segmentation loss when no annotation was present so the model is not penalized for predictions we dont have the label for.
", "rawMarkdown": "One of the highest scoring public notebooks using an interesting application of the annotations and a teacher student process to instill some extra knowledge in the final network. This process doesnt entirely make logical sense to me and decided to try a slightly different direction with the annotations that looks reasonably promising.\n\n\nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/207577\n\nMy gripe with the annotation usage in the teacher student setup is that the annotations are applied to the image and painted on in different colors for the varying classifications. Then an initial model is trained to use these inputs to classify. With the colors marking the catheter/line and their classification, the model simply needs to classify by color. \"If pixel value == (128, 0, 128) then x condition\", but this is not useful information because the colors will not be available on future classifications. \n\n\n\nYou could argue that in the second stage of this training process when the student model is given the unannotated inputs and the teacher is given annotated and then their embeddings are pushed together that it is trying to inject that knowledge into the student network, but it still seems a bit obtuse and overcomplicated to me. \n\nMy proposed method is instead of using the annotations as input and then do the student-teacher process, just directly teach the initial model to predict the catheter/line annotations. In this scenario using a model like a U-net we can have the model predict the 11 different labels out of the bottleneck of the U and then also output a segmentation mask showing the path of the catheter. \n\nMaking some modifications to the code used for the teacher-student, adding linear interpolation between points, we can make the model predict something like this: \n\n\nThis functions as additional supervision through the backbone of the network forcing it to learn characteristics related to the catheter and preventing it from overfitting on irrelevant features.\n\nOnce again we have to deal with the fact that annotations are not available for all x-rays. We can remedy this by masking off the segmentation loss when no annotation was present so the model is not penalized for predictions we dont have the label for. ", "votes": 23 }, { "id": 1172471, "postDate": "2021-01-27T11:52:08.173Z", "content": "Thank you very much. I have the same question with Veljko Kovac. How do you connect the annotation dots and build the segmentation label?
", "rawMarkdown": "Thank you very much. I have the same question with Veljko Kovac. How do you connect the annotation dots and build the segmentation label?", "replies": [ { "id": 1172527, "postDate": "2021-01-27T12:14:06.063Z", "content": "Hello Chuanyang. I have contacted you on LinkedIn. To connect the dots a simple linear interpolation is done.
", "rawMarkdown": "Hello Chuanyang. I have contacted you on LinkedIn. To connect the dots a simple linear interpolation is done.", "votes": 1 }, { "id": 1172545, "postDate": "2021-01-27T12:22:57.060Z", "content": "Thank you very much for your reply!
", "rawMarkdown": "Thank you very much for your reply!" } ] }, { "id": 1166274, "postDate": "2021-01-23T13:49:45.253Z", "content": "Your segmentation looks quite accurate. What loss did you use?
\nWhen connecting the annotations dots, the annotations are not completely accurate. Do you think that this does not affect a lot your segmentation loss when you backpropagate?
The visualizations I show are the labels, not the model predictions.
\nThis is what the model predictions look like
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1035002%2Fcc6462658828d9135ab8962da175f659%2Fimage%20(27).png?generation=1611599737838548&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1035002%2F5f73908ea796f89d2bbaec9c6586c327%2Fimage%20(28).png?generation=1611599752852557&alt=media\")
Did you use a UNet for that?
", "rawMarkdown": "Did you use a UNet for that?" }, { "id": 1170331, "postDate": "2021-01-26T06:31:12.320Z", "content": "Yes, I used the unet implementation from https://github.com/qubvel/segmentation_models.pytorch
", "rawMarkdown": "Yes, I used the unet implementation from https://github.com/qubvel/segmentation_models.pytorch" } ] }, { "id": 1165474, "postDate": "2021-01-23T02:11:12.303Z", "content": "Did you check the breakdown of the loss? When I tried this method, the segmentation loss very quickly has no effect on regularizing the model because the proportion of annotated data is just too little in comparison.
\nI could be completely over-fitting but I'm curious if you ran into that problem and how you solve it.
", "rawMarkdown": "Did you check the breakdown of the loss? When I tried this method, the segmentation loss very quickly has no effect on regularizing the model because the proportion of annotated data is just too little in comparison. \n\nI could be completely over-fitting but I'm curious if you ran into that problem and how you solve it.", "replies": [ { "id": 1165481, "postDate": "2021-01-23T02:22:16.190Z", "content": "Yes I print them separately. The validation segmentation loss continues to Improve fairly deep into training. Hard to say if it really has any impact in the result of the classification though. Seems small if any
", "rawMarkdown": "Yes I print them separately. The validation segmentation loss continues to Improve fairly deep into training. Hard to say if it really has any impact in the result of the classification though. Seems small if any", "votes": 2 }, { "id": 1165645, "postDate": "2021-01-23T06:37:13.357Z", "content": "Looking my training log, I think I definitely has too large of a learning rate…
", "rawMarkdown": "Looking my training log, I think I definitely has too large of a learning rate..." } ] }, { "id": 1163004, "postDate": "2021-01-21T12:53:01.007Z", "content": "@ryches Sorry but I didn't get it, are you proposing to train a single model or two models?
", "rawMarkdown": "@ryches Sorry but I didn't get it, are you proposing to train a single model or two models?", "replies": [ { "id": 1163790, "postDate": "2021-01-21T23:56:59.890Z", "content": "it is a single model that has two different outputs, one is the classification and the other is segmentation marking where the catheter got annotated by doctors
", "rawMarkdown": "it is a single model that has two different outputs, one is the classification and the other is segmentation marking where the catheter got annotated by doctors", "votes": 1 }, { "id": 1164266, "postDate": "2021-01-22T08:46:28.517Z", "content": "Then don't you think learning the segmentation mask together with classification would be a little bit challenging because we only have annotations of about 9095 unique images and 17999 label annotations available. While there are 30083 unique images and 30083 x 3(average) labels possible?
", "rawMarkdown": "Then don't you think learning the segmentation mask together with classification would be a little bit challenging because we only have annotations of about 9095 unique images and 17999 label annotations available. While there are 30083 unique images and 30083 x 3(average) labels possible?" }, { "id": 1165246, "postDate": "2021-01-22T19:55:16.800Z", "content": "Yes, it is a difficult task, but 9k images is enough to generally learn from and the whole point is to regularize and guide learning with this auxiliary task to prevent overfitting
", "rawMarkdown": "Yes, it is a difficult task, but 9k images is enough to generally learn from and the whole point is to regularize and guide learning with this auxiliary task to prevent overfitting", "votes": 1 } ] }, { "id": 1160696, "postDate": "2021-01-20T04:35:39.113Z", "content": "so,did you let it work and test it?Thanks!
", "rawMarkdown": "so,did you let it work and test it?Thanks!", "replies": [ { "id": 1160863, "postDate": "2021-01-20T07:23:59.090Z", "content": "Ran first experiment with a resnet18 based unet and was able to get .93. My base resnet18 without segmentation has not completed training yet, but looks marginally worse. Hard to judge because it converged faster without the auxiliary segmentation loss, but validation BCE seems to be a little worse without segmnetation.
", "rawMarkdown": "Ran first experiment with a resnet18 based unet and was able to get .93. My base resnet18 without segmentation has not completed training yet, but looks marginally worse. Hard to judge because it converged faster without the auxiliary segmentation loss, but validation BCE seems to be a little worse without segmnetation. ", "votes": 1 }, { "id": 1161882, "postDate": "2021-01-20T19:44:22.910Z", "content": "Here are results for with segmentation
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1035002%2Fe0b5e50d11b7836e503b5f0232df7a49%2FScreenshot%20from%202021-01-20%2011-42-03.png?generation=1611171770603298&alt=media\")
Without segmentation
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F1035002%2F94b541f0908dabde8f01df43bc6dcb82%2FScreenshot%20from%202021-01-20%2011-42-22.png?generation=1611171831078910&alt=media\")
Thank you very much. I have the same question with Veljko Kovac. How do you connect the annotation dots and build the segmentation label?
", "votes": 0, "replies": [ { "id": 1172527, "author_name": "Veljko Kovac", "author_url": "", "post_date": "2021-01-27T12:14:06.063000", "content": "Hello Chuanyang. I have contacted you on LinkedIn. To connect the dots a simple linear interpolation is done.
", "votes": 1, "replies": [] }, { "id": 1172545, "author_name": "Chuanyang ZHENG", "author_url": "", "post_date": "2021-01-27T12:22:57.060000", "content": "Thank you very much for your reply!
", "votes": 0, "replies": [] } ] }, { "id": 1166274, "author_name": "Veljko Kovac", "author_url": "", "post_date": "2021-01-23T13:49:45.253000", "content": "Your segmentation looks quite accurate. What loss did you use?
\nWhen connecting the annotations dots, the annotations are not completely accurate. Do you think that this does not affect a lot your segmentation loss when you backpropagate?
The visualizations I show are the labels, not the model predictions.
\nThis is what the model predictions look like
\n
Did you use a UNet for that?
", "votes": 0, "replies": [] }, { "id": 1170331, "author_name": "ryches", "author_url": "", "post_date": "2021-01-26T06:31:12.320000", "content": "Yes, I used the unet implementation from https://github.com/qubvel/segmentation_models.pytorch
", "votes": 0, "replies": [] } ] }, { "id": 1165474, "author_name": "JunYong Tong", "author_url": "", "post_date": "2021-01-23T02:11:12.303000", "content": "Did you check the breakdown of the loss? When I tried this method, the segmentation loss very quickly has no effect on regularizing the model because the proportion of annotated data is just too little in comparison.
\nI could be completely over-fitting but I'm curious if you ran into that problem and how you solve it.
", "votes": 0, "replies": [ { "id": 1165481, "author_name": "ryches", "author_url": "", "post_date": "2021-01-23T02:22:16.190000", "content": "Yes I print them separately. The validation segmentation loss continues to Improve fairly deep into training. Hard to say if it really has any impact in the result of the classification though. Seems small if any
", "votes": 2, "replies": [] }, { "id": 1165645, "author_name": "JunYong Tong", "author_url": "", "post_date": "2021-01-23T06:37:13.357000", "content": "Looking my training log, I think I definitely has too large of a learning rate…
", "votes": 0, "replies": [] } ] }, { "id": 1163004, "author_name": "Izzy Adesanya", "author_url": "", "post_date": "2021-01-21T12:53:01.007000", "content": "@ryches Sorry but I didn't get it, are you proposing to train a single model or two models?
", "votes": 0, "replies": [ { "id": 1163790, "author_name": "ryches", "author_url": "", "post_date": "2021-01-21T23:56:59.890000", "content": "it is a single model that has two different outputs, one is the classification and the other is segmentation marking where the catheter got annotated by doctors
", "votes": 1, "replies": [] }, { "id": 1164266, "author_name": "Izzy Adesanya", "author_url": "", "post_date": "2021-01-22T08:46:28.517000", "content": "Then don't you think learning the segmentation mask together with classification would be a little bit challenging because we only have annotations of about 9095 unique images and 17999 label annotations available. While there are 30083 unique images and 30083 x 3(average) labels possible?
", "votes": 0, "replies": [] }, { "id": 1165246, "author_name": "ryches", "author_url": "", "post_date": "2021-01-22T19:55:16.800000", "content": "Yes, it is a difficult task, but 9k images is enough to generally learn from and the whole point is to regularize and guide learning with this auxiliary task to prevent overfitting
", "votes": 1, "replies": [] } ] }, { "id": 1160696, "author_name": "Bcw93", "author_url": "", "post_date": "2021-01-20T04:35:39.113000", "content": "so,did you let it work and test it?Thanks!
", "votes": 0, "replies": [ { "id": 1160863, "author_name": "ryches", "author_url": "", "post_date": "2021-01-20T07:23:59.090000", "content": "Ran first experiment with a resnet18 based unet and was able to get .93. My base resnet18 without segmentation has not completed training yet, but looks marginally worse. Hard to judge because it converged faster without the auxiliary segmentation loss, but validation BCE seems to be a little worse without segmnetation.
", "votes": 1, "replies": [] }, { "id": 1161882, "author_name": "ryches", "author_url": "", "post_date": "2021-01-20T19:44:22.910000", "content": "Here are results for with segmentation
\n
Without segmentation
\n