{ "id": 225445, "title": "Fine-Tuning: NIH Chest X-rays vs ImageNet", "url": "/competitions/ranzcr-clip-catheter-line-classification/discussion/225445", "author_name": "Tawara", "post_date": "2021-03-12T10:08:57.264000", "votes": 16, "comment_count": 23, "views": 0, "content": "

NIH Chest X-rays seems worth using for this competition.
\nOne of usages is training a model by multi-label (14 classes) classification task of this data and fine-tuning the model using this competition data.
\n(Other usage is a teacher-student training method shared in this topic: 4 stage training )

\n

I tried this approach but got lower CV score than fine-tuning a model pretrained on ImageNet.

\n\n

Has anyone have a similar experience?

", "messages": [ { "id": 1235621, "postDate": "2021-03-12T10:08:57.263Z", "content": "

NIH Chest X-rays seems worth using for this competition.
\nOne of usages is training a model by multi-label (14 classes) classification task of this data and fine-tuning the model using this competition data.
\n(Other usage is a teacher-student training method shared in this topic: 4 stage training )

\n

I tried this approach but got lower CV score than fine-tuning a model pretrained on ImageNet.

\n\n

Has anyone have a similar experience?

", "rawMarkdown": "[NIH Chest X-rays](https://www.kaggle.com/nih-chest-xrays/data) seems worth using for this competition. \nOne of usages is training a model by multi-label (14 classes) classification task of this data and fine-tuning the model using this competition data.\n(Other usage is a teacher-student training method shared in this topic: [4 stage training](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/215910) )\n\nI tried this approach but got lower CV score than fine-tuning a model pretrained on ImageNet.\n\n* Model: resnest50d_1s4x24d\n* Image Size: 640x640\n* Val Score (single fold)\n * fine-tuning chest X-rays pretrained model : 0.9385\n * fine-tuning ImageNet pretrained model 　: 0.9464\n* NOTE: Training settings are exactly same. \n\nHas anyone have a similar experience? ", "votes": 16 }, { "id": 1236040, "postDate": "2021-03-12T17:40:29.827Z", "content": "

What worked for me was using a trained model on competition data as teacher model for pretraining a student model on NIH dataset. This way you dont learn bad features from diseases! Boosted my single model CV to 0.967+

", "rawMarkdown": "What worked for me was using a trained model on competition data as teacher model for pretraining a student model on NIH dataset. This way you dont learn bad features from diseases! Boosted my single model CV to 0.967+", "votes": 4, "replies": [ { "id": 1236647, "postDate": "2021-03-13T10:33:38.407Z", "content": "

@yannmajewski can you elaborate a little. What trained model on which data exactly.

", "rawMarkdown": "@yannmajewski can you elaborate a little. What trained model on which data exactly." }, { "id": 1236831, "postDate": "2021-03-13T13:48:27.117Z", "content": "
    \n
  1. Train model on competition data
    2. \n
  2. Use that model’s features to teach an other model on external data (NIH dataset)
    3. \n
  3. Fine tune the student model on competition data
    \nVoila!
    4. \n
", "rawMarkdown": "1. Train model on competition data\n2. Use that model’s features to teach an other model on external data (NIH dataset)\n3. Fine tune the student model on competition data\nVoila!", "votes": 2 }, { "id": 1236842, "postDate": "2021-03-13T13:57:39.530Z", "content": "

thanks, @yannmajewski, Good Luck

", "rawMarkdown": "thanks, @yannmajewski, Good Luck" }, { "id": 1237189, "postDate": "2021-03-13T21:14:36.650Z", "content": "

Thnak you @yannmajewski
\nWhat is the best way to deal with the fact that RANZCR CLiP (11 outputs ) but NIH is 15 outputs?

", "rawMarkdown": "Thnak you @yannmajewski\nWhat is the best way to deal with the fact that RANZCR CLiP (11 outputs ) but NIH is 15 outputs?\n" } ] }, { "id": 1236030, "postDate": "2021-03-12T17:27:59.130Z", "content": "

@ttahara thanks a lot for sharing your experience. I have observed similar results in my experiments and was wondering whether I am doing something wrong. This discussion topic made me realize that I am not the only one who faces challenges with trying to adapt the pretrained NIH model to the competition data 🙂

", "rawMarkdown": "@ttahara thanks a lot for sharing your experience. I have observed similar results in my experiments and was wondering whether I am doing something wrong. This discussion topic made me realize that I am not the only one who faces challenges with trying to adapt the pretrained NIH model to the competition data 🙂", "votes": 1, "replies": [ { "id": 1236053, "postDate": "2021-03-12T17:53:42.753Z", "content": "

You're welcome 👍

", "rawMarkdown": "You're welcome 👍" } ] }, { "id": 1235814, "postDate": "2021-03-12T13:44:41.627Z", "content": "

Based on this post from the competition host -
\nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/221808#1216234
\n\"As per the acknowledgements page - this dataset was created by relabelling the publicly available CXR14
\ndataset from NIH, therefore there will be 100% overlap in image data. These labels (concerning lines and tubes
\nplacement) are not part of the original dataset.\"

\n

The topic author included a link to a dataset of duplicates RANZCR and Chest-X based on imagehash ~28K images.
\nPerhaps using a subset of the duplicates may yield better results. Since there are various numbers of follow-ups, it may be a larger number in Chest-X without any of the lines and tubes in this dataset. Based on checks only around 24 images in RANZCR had none. Also the finding label 14 classes may have some correlation to the competition labels and again a subset for those might be more useful.

", "rawMarkdown": "Based on this post from the competition host -\nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/221808#1216234\n\"As per the acknowledgements page - this dataset was created by relabelling the publicly available CXR14 \ndataset from NIH, therefore there will be 100% overlap in image data. These labels (concerning lines and tubes \nplacement) are not part of the original dataset.\"\n\nThe topic author included a link to a dataset of duplicates RANZCR and Chest-X based on imagehash ~28K images.\nPerhaps using a subset of the duplicates may yield better results. Since there are various numbers of follow-ups, it may be a larger number in Chest-X without any of the lines and tubes in this dataset. Based on checks only around 24 images in RANZCR had none. Also the finding label 14 classes may have some correlation to the competition labels and again a subset for those might be more useful. ", "votes": 1, "replies": [ { "id": 1235887, "postDate": "2021-03-12T14:58:50.887Z", "content": "
\n

Also the finding label 14 classes may have some correlation to the competition labels

\n
\n

Do you check this in some way?
\nAbout 45% of images which exist in both Chest-X and RANZCR have No Finding label.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
IsNoFinding \\ IsCompDataFalseTrue
False3618115578
True4791112450
", "rawMarkdown": "> Also the finding label 14 classes may have some correlation to the competition labels\n\nDo you check this in some way?\nAbout 45% of images which exist in both Chest-X and RANZCR have `No Finding` label.\n\n| IsNoFinding \\ IsCompData | False | True | \n|:---:|:---:|:----:|\n| False | 36181 | 15578 |\n| True | 47911 | **12450** |" }, { "id": 1235912, "postDate": "2021-03-12T15:25:53.273Z", "content": "

An example discussed here
\nPatientID 8849382d0
\nStudyInstanceUID 1.2.826.0.1.3680043.8.498.11693509889426445054876979814173446281
\nbased on imagehash is duplicate of 00001836_119.png so follow up# 119 for patient 1836 that has over 130 images in that dataset Chest-X. 119 has finding label Atelectasis|Edema. So other examples in Chest-X for Atelectasis or for patient 1836 e.g. could be more relevant to the competition so just a subset might be better. Kind of in the same way that annotations are not available for all of train.

", "rawMarkdown": "An example discussed [here](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/222626 )\nPatientID 8849382d0 \nStudyInstanceUID 1.2.826.0.1.3680043.8.498.11693509889426445054876979814173446281\nbased on imagehash is duplicate of 00001836_119.png so follow up# 119 for patient 1836 that has over 130 images in that dataset Chest-X. 119 has finding label Atelectasis|Edema. So other examples in Chest-X for Atelectasis or for patient 1836 e.g. could be more relevant to the competition so just a subset might be better. Kind of in the same way that annotations are not available for all of train. " } ] }, { "id": 1235678, "postDate": "2021-03-12T11:26:16.517Z", "content": "

Thank you for a really nice post!
\nI am also interested in pretraining using NIH dataset.

\n

We did the similar experiments with 4 stages training by @ammarali32 .
\nThe CVs(all ResNet200D, 4 stages training, 640x640) are,

\n

ImageNet: 0.9511 (only this was trained with 512x512 and 3 stages)
\nNIH Chest X-rays 15 labels(original 14 labels + PA or AP): 0.9496
\nNIH Chest X-rays 4 labels(ETT, CVC, Swan-Ganz, NGT present or not): 0.9497
\nAmmarali32's pretrained weights: 0.9594

", "rawMarkdown": "Thank you for a really nice post!\nI am also interested in pretraining using NIH dataset.\n \nWe did the similar experiments with 4 stages training by @ammarali32 .\nThe CVs(all ResNet200D, 4 stages training, 640x640) are,\n\nImageNet: 0.9511 (only this was trained with 512x512 and 3 stages)\nNIH Chest X-rays 15 labels(original 14 labels + PA or AP): 0.9496\nNIH Chest X-rays 4 labels(ETT, CVC, Swan-Ganz, NGT present or not): 0.9497\nAmmarali32's pretrained weights: 0.9594", "votes": 1, "replies": [ { "id": 1235688, "postDate": "2021-03-12T11:36:41.177Z", "content": "

Thanks for sharing your experiment results.

\n

It is unexpected for me that the model pretrained by NIH Chest X-rays 4 labels (similar task with this competition) is lower than ImageNet.

", "rawMarkdown": "Thanks for sharing your experiment results.\n\nIt is unexpected for me that the model pretrained by NIH Chest X-rays 4 labels (similar task with this competition) is lower than ImageNet. ", "votes": 1 }, { "id": 1235699, "postDate": "2021-03-12T11:45:35.040Z", "content": "

I noticed 4 stage training is not that straightforward either. There are some things to tune: label (soft or hard), loss function etc.

", "rawMarkdown": "I noticed 4 stage training is not that straightforward either. There are some things to tune: label (soft or hard), loss function etc.", "votes": 1 }, { "id": 1235735, "postDate": "2021-03-12T12:24:01.090Z", "content": "

@yosukeyama
\n\"NIH Chest X-rays 4 labels(ETT, CVC, Swan-Ganz, NGT present or not)\"

\n

how you get the 4 labels of NIH Chest X-rays ?
\ni thought the Data_Entry_2017.csv file only have:

\n
Image Index,Finding Labels,Follow-up #,Patient ID,Patient Age,Patient Gender,View Position,OriginalImage[Width,Height],OriginalImagePixelSpacing[x,y],\n
", "rawMarkdown": "@yosukeyama \n\"NIH Chest X-rays 4 labels(ETT, CVC, Swan-Ganz, NGT present or not)\"\n\nhow you get the 4 labels of NIH Chest X-rays ?\ni thought the Data\\_Entry\\_2017.csv file only have:\n\n```\nImage Index,Finding Labels,Follow-up #,Patient ID,Patient Age,Patient Gender,View Position,OriginalImage[Width,Height],OriginalImagePixelSpacing[x,y],\n\n\n```", "votes": 1 }, { "id": 1235807, "postDate": "2021-03-12T13:35:02.907Z", "content": "

@hengck23
\nThank you for your question. I always look forward to your posts!

\n

First, I used RANZCR's train.csv to create a new train.csv, which only have 4 labels(the presence or absence of catheters).
\nThen I created a model using the RANZCR dataset based on the csv file and labeled the NIH dataset by the model.
\nCV of that model was 0.9920 [ETT: 0.9987, NGT: 0.9974, CVC: 0.9726, Swan-Ganz: 0.9993].

", "rawMarkdown": "@hengck23 \nThank you for your question. I always look forward to your posts!\n\nFirst, I used RANZCR's train.csv to create a new train.csv, which only have 4 labels(the presence or absence of catheters).\nThen I created a model using the RANZCR dataset based on the csv file and labeled the NIH dataset by the model.\nCV of that model was 0.9920 [ETT: 0.9987, NGT: 0.9974, CVC: 0.9726, Swan-Ganz: 0.9993]." } ] }, { "id": 1235640, "postDate": "2021-03-12T10:43:41.423Z", "content": "

I had similar results. This might be explained by the difference in dataset size (i.e. variety of patterns), and the difference in task characteristic (i.e. patterns to classify diseases are different from those to classify catheter position).
\nAfter all, 4 stage training works better in this competition.

", "rawMarkdown": "I had similar results. This might be explained by the difference in dataset size (i.e. variety of patterns), and the difference in task characteristic (i.e. patterns to classify diseases are different from those to classify catheter position). \nAfter all, [4 stage training](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/215910) works better in this competition.", "votes": 1, "replies": [ { "id": 1235651, "postDate": "2021-03-12T11:09:03.653Z", "content": "

Thanks. Your explanation makes a lot of sense for me.
\nImageNet Pretrained models are great.

\n

Teacher-student training on external data is one of things I learned in this competition 😃

", "rawMarkdown": "Thanks. Your explanation makes a lot of sense for me.\nImageNet Pretrained models are great.\n\nTeacher-student training on external data is one of things I learned in this competition 😃" } ] }, { "id": 1235729, "postDate": "2021-03-12T12:17:32.230Z", "content": "

\"I tried this approach but got lower CV score than fine-tuning a model pretrained on ImageNet.\"

\n

this is possible because features for xray diseases is not the same for line/tube features.

\n

you can compare images that are wrong and right. check their CAM heatmap too

", "rawMarkdown": "\"I tried this approach but got lower CV score than fine-tuning a model pretrained on ImageNet.\"\n\nthis is possible because features for xray diseases is not the same for line/tube features.\n\nyou can compare images that are wrong and right. check their CAM heatmap too\n", "votes": 2, "replies": [ { "id": 1235771, "postDate": "2021-03-12T13:13:18.867Z", "content": "

Thanks.
\nYes. I think difference of tasks has a big impact.

\n

Pre-training in unsupervised or self-supervised manner may work. (this is just an idea)

", "rawMarkdown": "Thanks.\nYes. I think difference of tasks has a big impact.\n\nPre-training in unsupervised or self-supervised manner may work. (this is just an idea)" } ] }, { "id": 1235689, "postDate": "2021-03-12T11:36:45.800Z", "content": "

Hi @ttahara :)

\n

with weights='ImageNet', I am getting better modes than weights='noisy-student.

\n

but about using other Chest X-rays, I have an idea since in this competition we have 24 X-rays without labels ( i mean 0 for all 11 outputs)

\n

Here the list:

\n

1.2.826.0.1.3680043.8.498.58959974007194881910883633900120894133
\n1.2.826.0.1.3680043.8.498.62015824992988632734732028510878678058
\n1.2.826.0.1.3680043.8.498.11745816736251186885492299187567575889
\n1.2.826.0.1.3680043.8.498.59000837536872259040577141592245957659
\n1.2.826.0.1.3680043.8.498.10031131575745854404126410511882847279
\n1.2.826.0.1.3680043.8.498.23463192819909852867890848692185962628
\n1.2.826.0.1.3680043.8.498.72598527577197250033387270423700920577
\n1.2.826.0.1.3680043.8.498.76873981093833909666100955199059328203
\n1.2.826.0.1.3680043.8.498.22681438984244725251540286783146340207
\n1.2.826.0.1.3680043.8.498.94546335208494362824048256394230982330
\n1.2.826.0.1.3680043.8.498.27735788065706318427834428727058767495
\n1.2.826.0.1.3680043.8.498.70556928637314908190043753509080639346
\n1.2.826.0.1.3680043.8.498.29594566253802547412666405573794509996
\n1.2.826.0.1.3680043.8.498.29120924921129086021386997558046643281
\n1.2.826.0.1.3680043.8.498.12117973685813682040906233799430905529
\n1.2.826.0.1.3680043.8.498.99722513002839474309194956069139195175
\n1.2.826.0.1.3680043.8.498.85532799680825387534302013447068585944
\n1.2.826.0.1.3680043.8.498.12223483707517344235132707229554872826
\n1.2.826.0.1.3680043.8.498.87060544214618912677358800066372917759
\n1.2.826.0.1.3680043.8.498.74692334349977474246279475957036146226
\n1.2.826.0.1.3680043.8.498.77617913658664951057379050596538324265
\n1.2.826.0.1.3680043.8.498.16289885199975270507878278461180774853
\n1.2.826.0.1.3680043.8.498.32419559375493685668844762644430268742
\n1.2.826.0.1.3680043.8.498.57553766290478117689691227338973153148

\n

The idea is to use this dataset (3836 X-Ray Images for Normal People)
\nhttps://www.kaggle.com/anaselmasry/normalxray

\n

and label them with Zeros for all 11 outputs. This might lead to improvement …

", "rawMarkdown": "Hi @ttahara :)\n\nwith weights='ImageNet', I am getting better modes than weights='noisy-student. \n\nbut about using other Chest X-rays, I have an idea since in this competition we have 24 X-rays without labels ( i mean 0 for all 11 outputs) \n\nHere the list:\n\n1.2.826.0.1.3680043.8.498.58959974007194881910883633900120894133\n1.2.826.0.1.3680043.8.498.62015824992988632734732028510878678058\n1.2.826.0.1.3680043.8.498.11745816736251186885492299187567575889\n1.2.826.0.1.3680043.8.498.59000837536872259040577141592245957659\n1.2.826.0.1.3680043.8.498.10031131575745854404126410511882847279\n1.2.826.0.1.3680043.8.498.23463192819909852867890848692185962628\n1.2.826.0.1.3680043.8.498.72598527577197250033387270423700920577\n1.2.826.0.1.3680043.8.498.76873981093833909666100955199059328203\n1.2.826.0.1.3680043.8.498.22681438984244725251540286783146340207\n1.2.826.0.1.3680043.8.498.94546335208494362824048256394230982330\n1.2.826.0.1.3680043.8.498.27735788065706318427834428727058767495\n1.2.826.0.1.3680043.8.498.70556928637314908190043753509080639346\n1.2.826.0.1.3680043.8.498.29594566253802547412666405573794509996\n1.2.826.0.1.3680043.8.498.29120924921129086021386997558046643281\n1.2.826.0.1.3680043.8.498.12117973685813682040906233799430905529\n1.2.826.0.1.3680043.8.498.99722513002839474309194956069139195175\n1.2.826.0.1.3680043.8.498.85532799680825387534302013447068585944\n1.2.826.0.1.3680043.8.498.12223483707517344235132707229554872826\n1.2.826.0.1.3680043.8.498.87060544214618912677358800066372917759\n1.2.826.0.1.3680043.8.498.74692334349977474246279475957036146226\n1.2.826.0.1.3680043.8.498.77617913658664951057379050596538324265\n1.2.826.0.1.3680043.8.498.16289885199975270507878278461180774853\n1.2.826.0.1.3680043.8.498.32419559375493685668844762644430268742\n1.2.826.0.1.3680043.8.498.57553766290478117689691227338973153148\n\nThe idea is to use this dataset (3836 X-Ray Images for Normal People)\nhttps://www.kaggle.com/anaselmasry/normalxray\n\n and label them with Zeros for all 11 outputs. This might lead to improvement ...\n\n", "replies": [ { "id": 1235702, "postDate": "2021-03-12T11:47:58.613Z", "content": "

Thanks. Did you try it?

\n

I don't know whether extra all 0 data leads to improvement.

", "rawMarkdown": "Thanks. Did you try it?\n\nI don't know whether extra all 0 data leads to improvement." }, { "id": 1235723, "postDate": "2021-03-12T12:12:13.763Z", "content": "

I will give it a try tomorrow.

", "rawMarkdown": "I will give it a try tomorrow. ", "votes": 1 } ] }, { "id": 1235755, "postDate": "2021-03-12T13:00:43.867Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 1236040, "author_name": "Yann Majewski", "author_url": "", "post_date": "2021-03-12T17:40:29.827000", "content": "

What worked for me was using a trained model on competition data as teacher model for pretraining a student model on NIH dataset. This way you dont learn bad features from diseases! Boosted my single model CV to 0.967+

", "votes": 4, "replies": [ { "id": 1236647, "author_name": "Sanchit Vijay", "author_url": "", "post_date": "2021-03-13T10:33:38.407000", "content": "

@yannmajewski can you elaborate a little. What trained model on which data exactly.

", "votes": 0, "replies": [] }, { "id": 1236831, "author_name": "Yann Majewski", "author_url": "", "post_date": "2021-03-13T13:48:27.117000", "content": "
    \n
  1. Train model on competition data
    2. \n
  2. Use that model’s features to teach an other model on external data (NIH dataset)
    3. \n
  3. Fine tune the student model on competition data
    \nVoila!
    4. \n
", "votes": 2, "replies": [] }, { "id": 1236842, "author_name": "Sanchit Vijay", "author_url": "", "post_date": "2021-03-13T13:57:39.530000", "content": "

thanks, @yannmajewski, Good Luck

", "votes": 0, "replies": [] }, { "id": 1237189, "author_name": "Faisal Alsrheed", "author_url": "", "post_date": "2021-03-13T21:14:36.650000", "content": "

Thnak you @yannmajewski
\nWhat is the best way to deal with the fact that RANZCR CLiP (11 outputs ) but NIH is 15 outputs?

", "votes": 0, "replies": [] } ] }, { "id": 1236030, "author_name": "Nikita Kozodoi", "author_url": "", "post_date": "2021-03-12T17:27:59.130000", "content": "

@ttahara thanks a lot for sharing your experience. I have observed similar results in my experiments and was wondering whether I am doing something wrong. This discussion topic made me realize that I am not the only one who faces challenges with trying to adapt the pretrained NIH model to the competition data 🙂

", "votes": 1, "replies": [ { "id": 1236053, "author_name": "Tawara", "author_url": "", "post_date": "2021-03-12T17:53:42.753000", "content": "

You're welcome 👍

", "votes": 0, "replies": [] } ] }, { "id": 1235814, "author_name": "something4kag", "author_url": "", "post_date": "2021-03-12T13:44:41.627000", "content": "

Based on this post from the competition host -
\nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/221808#1216234
\n\"As per the acknowledgements page - this dataset was created by relabelling the publicly available CXR14
\ndataset from NIH, therefore there will be 100% overlap in image data. These labels (concerning lines and tubes
\nplacement) are not part of the original dataset.\"

\n

The topic author included a link to a dataset of duplicates RANZCR and Chest-X based on imagehash ~28K images.
\nPerhaps using a subset of the duplicates may yield better results. Since there are various numbers of follow-ups, it may be a larger number in Chest-X without any of the lines and tubes in this dataset. Based on checks only around 24 images in RANZCR had none. Also the finding label 14 classes may have some correlation to the competition labels and again a subset for those might be more useful.

", "votes": 1, "replies": [ { "id": 1235887, "author_name": "Tawara", "author_url": "", "post_date": "2021-03-12T14:58:50.887000", "content": "
\n

Also the finding label 14 classes may have some correlation to the competition labels

\n
\n

Do you check this in some way?
\nAbout 45% of images which exist in both Chest-X and RANZCR have No Finding label.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
IsNoFinding \\ IsCompDataFalseTrue
False3618115578
True4791112450
", "votes": 0, "replies": [] }, { "id": 1235912, "author_name": "something4kag", "author_url": "", "post_date": "2021-03-12T15:25:53.273000", "content": "

An example discussed here
\nPatientID 8849382d0
\nStudyInstanceUID 1.2.826.0.1.3680043.8.498.11693509889426445054876979814173446281
\nbased on imagehash is duplicate of 00001836_119.png so follow up# 119 for patient 1836 that has over 130 images in that dataset Chest-X. 119 has finding label Atelectasis|Edema. So other examples in Chest-X for Atelectasis or for patient 1836 e.g. could be more relevant to the competition so just a subset might be better. Kind of in the same way that annotations are not available for all of train.

", "votes": 0, "replies": [] } ] }, { "id": 1235678, "author_name": "YYama", "author_url": "", "post_date": "2021-03-12T11:26:16.517000", "content": "

Thank you for a really nice post!
\nI am also interested in pretraining using NIH dataset.

\n

We did the similar experiments with 4 stages training by @ammarali32 .
\nThe CVs(all ResNet200D, 4 stages training, 640x640) are,

\n

ImageNet: 0.9511 (only this was trained with 512x512 and 3 stages)
\nNIH Chest X-rays 15 labels(original 14 labels + PA or AP): 0.9496
\nNIH Chest X-rays 4 labels(ETT, CVC, Swan-Ganz, NGT present or not): 0.9497
\nAmmarali32's pretrained weights: 0.9594

", "votes": 1, "replies": [ { "id": 1235688, "author_name": "Tawara", "author_url": "", "post_date": "2021-03-12T11:36:41.177000", "content": "

Thanks for sharing your experiment results.

\n

It is unexpected for me that the model pretrained by NIH Chest X-rays 4 labels (similar task with this competition) is lower than ImageNet.

", "votes": 1, "replies": [] }, { "id": 1235699, "author_name": "RabotniKuma", "author_url": "", "post_date": "2021-03-12T11:45:35.040000", "content": "

I noticed 4 stage training is not that straightforward either. There are some things to tune: label (soft or hard), loss function etc.

", "votes": 1, "replies": [] }, { "id": 1235735, "author_name": "hengck23", "author_url": "", "post_date": "2021-03-12T12:24:01.090000", "content": "

@yosukeyama
\n\"NIH Chest X-rays 4 labels(ETT, CVC, Swan-Ganz, NGT present or not)\"

\n

how you get the 4 labels of NIH Chest X-rays ?
\ni thought the Data_Entry_2017.csv file only have:

\n
Image Index,Finding Labels,Follow-up #,Patient ID,Patient Age,Patient Gender,View Position,OriginalImage[Width,Height],OriginalImagePixelSpacing[x,y],\n
", "votes": 1, "replies": [] }, { "id": 1235807, "author_name": "YYama", "author_url": "", "post_date": "2021-03-12T13:35:02.907000", "content": "

@hengck23
\nThank you for your question. I always look forward to your posts!

\n

First, I used RANZCR's train.csv to create a new train.csv, which only have 4 labels(the presence or absence of catheters).
\nThen I created a model using the RANZCR dataset based on the csv file and labeled the NIH dataset by the model.
\nCV of that model was 0.9920 [ETT: 0.9987, NGT: 0.9974, CVC: 0.9726, Swan-Ganz: 0.9993].

", "votes": 0, "replies": [] } ] }, { "id": 1235640, "author_name": "RabotniKuma", "author_url": "", "post_date": "2021-03-12T10:43:41.423000", "content": "

I had similar results. This might be explained by the difference in dataset size (i.e. variety of patterns), and the difference in task characteristic (i.e. patterns to classify diseases are different from those to classify catheter position).
\nAfter all, 4 stage training works better in this competition.

", "votes": 1, "replies": [ { "id": 1235651, "author_name": "Tawara", "author_url": "", "post_date": "2021-03-12T11:09:03.653000", "content": "

Thanks. Your explanation makes a lot of sense for me.
\nImageNet Pretrained models are great.

\n

Teacher-student training on external data is one of things I learned in this competition 😃

", "votes": 0, "replies": [] } ] }, { "id": 1235729, "author_name": "hengck23", "author_url": "", "post_date": "2021-03-12T12:17:32.230000", "content": "

\"I tried this approach but got lower CV score than fine-tuning a model pretrained on ImageNet.\"

\n

this is possible because features for xray diseases is not the same for line/tube features.

\n

you can compare images that are wrong and right. check their CAM heatmap too

", "votes": 2, "replies": [ { "id": 1235771, "author_name": "Tawara", "author_url": "", "post_date": "2021-03-12T13:13:18.867000", "content": "

Thanks.
\nYes. I think difference of tasks has a big impact.

\n

Pre-training in unsupervised or self-supervised manner may work. (this is just an idea)

", "votes": 0, "replies": [] } ] }, { "id": 1235689, "author_name": "Faisal Alsrheed", "author_url": "", "post_date": "2021-03-12T11:36:45.800000", "content": "

Hi @ttahara :)

\n

with weights='ImageNet', I am getting better modes than weights='noisy-student.

\n

but about using other Chest X-rays, I have an idea since in this competition we have 24 X-rays without labels ( i mean 0 for all 11 outputs)

\n

Here the list:

\n

1.2.826.0.1.3680043.8.498.58959974007194881910883633900120894133
\n1.2.826.0.1.3680043.8.498.62015824992988632734732028510878678058
\n1.2.826.0.1.3680043.8.498.11745816736251186885492299187567575889
\n1.2.826.0.1.3680043.8.498.59000837536872259040577141592245957659
\n1.2.826.0.1.3680043.8.498.10031131575745854404126410511882847279
\n1.2.826.0.1.3680043.8.498.23463192819909852867890848692185962628
\n1.2.826.0.1.3680043.8.498.72598527577197250033387270423700920577
\n1.2.826.0.1.3680043.8.498.76873981093833909666100955199059328203
\n1.2.826.0.1.3680043.8.498.22681438984244725251540286783146340207
\n1.2.826.0.1.3680043.8.498.94546335208494362824048256394230982330
\n1.2.826.0.1.3680043.8.498.27735788065706318427834428727058767495
\n1.2.826.0.1.3680043.8.498.70556928637314908190043753509080639346
\n1.2.826.0.1.3680043.8.498.29594566253802547412666405573794509996
\n1.2.826.0.1.3680043.8.498.29120924921129086021386997558046643281
\n1.2.826.0.1.3680043.8.498.12117973685813682040906233799430905529
\n1.2.826.0.1.3680043.8.498.99722513002839474309194956069139195175
\n1.2.826.0.1.3680043.8.498.85532799680825387534302013447068585944
\n1.2.826.0.1.3680043.8.498.12223483707517344235132707229554872826
\n1.2.826.0.1.3680043.8.498.87060544214618912677358800066372917759
\n1.2.826.0.1.3680043.8.498.74692334349977474246279475957036146226
\n1.2.826.0.1.3680043.8.498.77617913658664951057379050596538324265
\n1.2.826.0.1.3680043.8.498.16289885199975270507878278461180774853
\n1.2.826.0.1.3680043.8.498.32419559375493685668844762644430268742
\n1.2.826.0.1.3680043.8.498.57553766290478117689691227338973153148

\n

The idea is to use this dataset (3836 X-Ray Images for Normal People)
\nhttps://www.kaggle.com/anaselmasry/normalxray

\n

and label them with Zeros for all 11 outputs. This might lead to improvement …

", "votes": 0, "replies": [ { "id": 1235702, "author_name": "Tawara", "author_url": "", "post_date": "2021-03-12T11:47:58.613000", "content": "

Thanks. Did you try it?

\n

I don't know whether extra all 0 data leads to improvement.

", "votes": 0, "replies": [] }, { "id": 1235723, "author_name": "Faisal Alsrheed", "author_url": "", "post_date": "2021-03-12T12:12:13.763000", "content": "

I will give it a try tomorrow.

", "votes": 1, "replies": [] } ] }, { "id": 1235755, "author_name": "", "author_url": "", "post_date": "2021-03-12T13:00:43.867000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1235621": "[NIH Chest X-rays](https://www.kaggle.com/nih-chest-xrays/data) seems worth using for this competition. \nOne of usages is training a model by multi-label (14 classes) classification task of this data and fine-tuning the model using this competition data.\n(Other usage is a teacher-student training method shared in this topic: [4 stage training](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/215910) )\n\nI tried this approach but got lower CV score than fine-tuning a model pretrained on ImageNet.\n\n* Model: resnest50d_1s4x24d\n* Image Size: 640x640\n* Val Score (single fold)\n * fine-tuning chest X-rays pretrained model : 0.9385\n * fine-tuning ImageNet pretrained model 　: 0.9464\n* NOTE: Training settings are exactly same. \n\nHas anyone have a similar experience? ", "1236040": "What worked for me was using a trained model on competition data as teacher model for pretraining a student model on NIH dataset. This way you dont learn bad features from diseases! Boosted my single model CV to 0.967+", "1236030": "@ttahara thanks a lot for sharing your experience. I have observed similar results in my experiments and was wondering whether I am doing something wrong. This discussion topic made me realize that I am not the only one who faces challenges with trying to adapt the pretrained NIH model to the competition data 🙂", "1235814": "Based on this post from the competition host -\nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/221808#1216234\n\"As per the acknowledgements page - this dataset was created by relabelling the publicly available CXR14 \ndataset from NIH, therefore there will be 100% overlap in image data. These labels (concerning lines and tubes \nplacement) are not part of the original dataset.\"\n\nThe topic author included a link to a dataset of duplicates RANZCR and Chest-X based on imagehash ~28K images.\nPerhaps using a subset of the duplicates may yield better results. Since there are various numbers of follow-ups, it may be a larger number in Chest-X without any of the lines and tubes in this dataset. Based on checks only around 24 images in RANZCR had none. Also the finding label 14 classes may have some correlation to the competition labels and again a subset for those might be more useful. ", "1235678": "Thank you for a really nice post!\nI am also interested in pretraining using NIH dataset.\n \nWe did the similar experiments with 4 stages training by @ammarali32 .\nThe CVs(all ResNet200D, 4 stages training, 640x640) are,\n\nImageNet: 0.9511 (only this was trained with 512x512 and 3 stages)\nNIH Chest X-rays 15 labels(original 14 labels + PA or AP): 0.9496\nNIH Chest X-rays 4 labels(ETT, CVC, Swan-Ganz, NGT present or not): 0.9497\nAmmarali32's pretrained weights: 0.9594", "1235640": "I had similar results. This might be explained by the difference in dataset size (i.e. variety of patterns), and the difference in task characteristic (i.e. patterns to classify diseases are different from those to classify catheter position). \nAfter all, [4 stage training](https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/215910) works better in this competition.", "1235729": "\"I tried this approach but got lower CV score than fine-tuning a model pretrained on ImageNet.\"\n\nthis is possible because features for xray diseases is not the same for line/tube features.\n\nyou can compare images that are wrong and right. check their CAM heatmap too\n", "1235689": "Hi @ttahara :)\n\nwith weights='ImageNet', I am getting better modes than weights='noisy-student. \n\nbut about using other Chest X-rays, I have an idea since in this competition we have 24 X-rays without labels ( i mean 0 for all 11 outputs) \n\nHere the list:\n\n1.2.826.0.1.3680043.8.498.58959974007194881910883633900120894133\n1.2.826.0.1.3680043.8.498.62015824992988632734732028510878678058\n1.2.826.0.1.3680043.8.498.11745816736251186885492299187567575889\n1.2.826.0.1.3680043.8.498.59000837536872259040577141592245957659\n1.2.826.0.1.3680043.8.498.10031131575745854404126410511882847279\n1.2.826.0.1.3680043.8.498.23463192819909852867890848692185962628\n1.2.826.0.1.3680043.8.498.72598527577197250033387270423700920577\n1.2.826.0.1.3680043.8.498.76873981093833909666100955199059328203\n1.2.826.0.1.3680043.8.498.22681438984244725251540286783146340207\n1.2.826.0.1.3680043.8.498.94546335208494362824048256394230982330\n1.2.826.0.1.3680043.8.498.27735788065706318427834428727058767495\n1.2.826.0.1.3680043.8.498.70556928637314908190043753509080639346\n1.2.826.0.1.3680043.8.498.29594566253802547412666405573794509996\n1.2.826.0.1.3680043.8.498.29120924921129086021386997558046643281\n1.2.826.0.1.3680043.8.498.12117973685813682040906233799430905529\n1.2.826.0.1.3680043.8.498.99722513002839474309194956069139195175\n1.2.826.0.1.3680043.8.498.85532799680825387534302013447068585944\n1.2.826.0.1.3680043.8.498.12223483707517344235132707229554872826\n1.2.826.0.1.3680043.8.498.87060544214618912677358800066372917759\n1.2.826.0.1.3680043.8.498.74692334349977474246279475957036146226\n1.2.826.0.1.3680043.8.498.77617913658664951057379050596538324265\n1.2.826.0.1.3680043.8.498.16289885199975270507878278461180774853\n1.2.826.0.1.3680043.8.498.32419559375493685668844762644430268742\n1.2.826.0.1.3680043.8.498.57553766290478117689691227338973153148\n\nThe idea is to use this dataset (3836 X-Ray Images for Normal People)\nhttps://www.kaggle.com/anaselmasry/normalxray\n\n and label them with Zeros for all 11 outputs. This might lead to improvement ...\n\n", "1235755": "" } }