{
  "id": 204835,
  "title": " ‎‏‏‎‎[placeholder] LB-0.956 - 5xFold 2xTTA xception-512",
  "url": "/competitions/ranzcr-clip-catheter-line-classification/discussion/204835",
  "author_name": "hengck23",
  "post_date": "2020-12-17T05:25:25.335000",
  "votes": 77,
  "comment_count": 31,
  "views": 0,
  "content": "<p>I estimate i can get LB &gt; 0.985 at the end of 2 months.</p>\n<p>I am working towards the following:</p>\n<ul>\n<li>generate synthetic GAN data/augment data/external data/test/data and treat them as noisy data</li>\n<li>use self/weak supervised learning to make clean labels or weigh samples</li>\n<li>use segmentation information for aux loss</li>\n<li>transformer to model relationship (the distance of tubes from chest parts)</li>\n<li>etc …</li>\n</ul>\n<hr>\n<p>if you are interested in my work and progress, you can follow my work here. it will be updated when I am free to work and make progress.</p>\n<p>all code/model/result/log file can be downloaded via:<br>\n<a href=\"https://drive.google.com/drive/folders/1jza6k-wHFVrJUaRgjvuR5_Mpwk7ClNVd?usp=sharing\" target=\"_blank\">https://drive.google.com/drive/folders/1jza6k-wHFVrJUaRgjvuR5_Mpwk7ClNVd?usp=sharing</a></p>\n<p>here is the first version:</p>\n<ol>\n<li>2020-12-17</li>\n</ol>\n<ul>\n<li>just a dummy baseline to check the problem and data</li>\n<li>5xfold exception 512x512 without augmentation: LB =0.951 (ensemble), LB=0.939 (fold-2)</li>\n</ul>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F9a4dc6b9904ac9962b24c6a2397dce6c%2FSelection_058.png?generation=1608182442517494&amp;alt=media\" alt=\"\"></p>\n<ol>\n<li>2020-12-18</li>\n</ol>\n<ul>\n<li>LB-0.956 - 5xFold 2xTTA xception-512</li>\n<li>Implement simple augmentation and TTA. better results by using lower learning rate </li>\n</ul>\n<hr>\n<p>version 2 (plan)</p>\n<ul>\n<li>add more augmentation, preprocessing</li>\n<li>structured ouput</li>\n<li>add visualization and interpretability of model (e.g. CAM heatmap)</li>\n<li>how to probe for AUC kaggle (I need to find out the number of instances per class for the test set … some class has rare occurrence … it is possible to recover the positive ground truth)</li>\n<li>I need  \"Sharpness-Aware Minimization optimizer.\"!!!</li>\n</ul>\n<hr>\n<p>version 3 (plan)</p>\n<ul>\n<li>add segmentation/attention/ as aux loss</li>\n<li>think of how to use transformer</li>\n</ul>\n<hr>\n<p>version 4 (plan)</p>\n<ul>\n<li>how to generate synthetic data and use self-supervised learning</li>\n</ul>",
  "messages": [
    {
      "id": 1116357,
      "postDate": "2020-12-17T05:25:25.337Z",
      "content": "<p>I estimate i can get LB &gt; 0.985 at the end of 2 months.</p>\n<p>I am working towards the following:</p>\n<ul>\n<li>generate synthetic GAN data/augment data/external data/test/data and treat them as noisy data</li>\n<li>use self/weak supervised learning to make clean labels or weigh samples</li>\n<li>use segmentation information for aux loss</li>\n<li>transformer to model relationship (the distance of tubes from chest parts)</li>\n<li>etc …</li>\n</ul>\n<hr>\n<p>if you are interested in my work and progress, you can follow my work here. it will be updated when I am free to work and make progress.</p>\n<p>all code/model/result/log file can be downloaded via:<br>\n<a href=\"https://drive.google.com/drive/folders/1jza6k-wHFVrJUaRgjvuR5_Mpwk7ClNVd?usp=sharing\" target=\"_blank\">https://drive.google.com/drive/folders/1jza6k-wHFVrJUaRgjvuR5_Mpwk7ClNVd?usp=sharing</a></p>\n<p>here is the first version:</p>\n<ol>\n<li>2020-12-17</li>\n</ol>\n<ul>\n<li>just a dummy baseline to check the problem and data</li>\n<li>5xfold exception 512x512 without augmentation: LB =0.951 (ensemble), LB=0.939 (fold-2)</li>\n</ul>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F9a4dc6b9904ac9962b24c6a2397dce6c%2FSelection_058.png?generation=1608182442517494&amp;alt=media\" alt=\"\"></p>\n<ol>\n<li>2020-12-18</li>\n</ol>\n<ul>\n<li>LB-0.956 - 5xFold 2xTTA xception-512</li>\n<li>Implement simple augmentation and TTA. better results by using lower learning rate </li>\n</ul>\n<hr>\n<p>version 2 (plan)</p>\n<ul>\n<li>add more augmentation, preprocessing</li>\n<li>structured ouput</li>\n<li>add visualization and interpretability of model (e.g. CAM heatmap)</li>\n<li>how to probe for AUC kaggle (I need to find out the number of instances per class for the test set … some class has rare occurrence … it is possible to recover the positive ground truth)</li>\n<li>I need  \"Sharpness-Aware Minimization optimizer.\"!!!</li>\n</ul>\n<hr>\n<p>version 3 (plan)</p>\n<ul>\n<li>add segmentation/attention/ as aux loss</li>\n<li>think of how to use transformer</li>\n</ul>\n<hr>\n<p>version 4 (plan)</p>\n<ul>\n<li>how to generate synthetic data and use self-supervised learning</li>\n</ul>",
      "rawMarkdown": "I estimate i can get LB > 0.985 at the end of 2 months.\n\nI am working towards the following:\n- generate synthetic GAN data/augment data/external data/test/data and treat them as noisy data\n- use self/weak supervised learning to make clean labels or weigh samples\n- use segmentation information for aux loss\n- transformer to model relationship (the distance of tubes from chest parts)\n- etc ...\n\n----\nif you are interested in my work and progress, you can follow my work here. it will be updated when I am free to work and make progress.\n\nall code/model/result/log file can be downloaded via:\nhttps://drive.google.com/drive/folders/1jza6k-wHFVrJUaRgjvuR5_Mpwk7ClNVd?usp=sharing\n\nhere is the first version:\n1. 2020-12-17\n- just a dummy baseline to check the problem and data\n- 5xfold exception 512x512 without augmentation: LB =0.951 (ensemble), LB=0.939 (fold-2)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F9a4dc6b9904ac9962b24c6a2397dce6c%2FSelection_058.png?generation=1608182442517494&alt=media)\n\n2. 2020-12-18\n- LB-0.956 - 5xFold 2xTTA xception-512\n- Implement simple augmentation and TTA. better results by using lower learning rate \n\n---\n\nversion 2 (plan)\n- add more augmentation, preprocessing\n- structured ouput\n- add visualization and interpretability of model (e.g. CAM heatmap)\n- how to probe for AUC kaggle (I need to find out the number of instances per class for the test set ... some class has rare occurrence ... it is possible to recover the positive ground truth)\n- I need  \"Sharpness-Aware Minimization optimizer.\"!!!\n\n---\n\nversion 3 (plan)\n- add segmentation/attention/ as aux loss\n- think of how to use transformer\n\n---\n\nversion 4 (plan)\n- how to generate synthetic data and use self-supervised learning",
      "votes": 77
    },
    {
      "id": 1152370,
      "postDate": "2021-01-14T04:37:29.163Z",
      "content": "<p>i can see the future of kaggle, pseudo label 2.0:</p>\n<ul>\n<li>generate many data</li>\n<li>some weak/self/semi supervised labeling</li>\n</ul>\n<p>Meta Pseudo Labels - google paper<br>\n<a href=\"https://arxiv.org/abs/2003.10580\" target=\"_blank\">https://arxiv.org/abs/2003.10580</a></p>\n<p>\" new state-of-the-art top-1 accuracy of 90.2% on ImageNet, which is 1.6% better than the existing state-of-the-art\"</p>\n<p>\"Like Pseudo Labels, Meta Pseudo Labels has a teacher network to generate pseudo labels on unlabeled data to teach a student network. However, unlike Pseudo Labels where the teacher is fixed, the teacher in Meta Pseudo Labels is constantly adapted by the feedback of the student's performance on the labeled dataset. \"</p>\n<p><img src=\"https://pbs.twimg.com/media/ErovYVNU0AAyACb?format=jpg&amp;name=medium\" alt=\"\"></p>",
      "rawMarkdown": "i can see the future of kaggle, pseudo label 2.0:\n- generate many data\n- some weak/self/semi supervised labeling\n\nMeta Pseudo Labels - google paper\nhttps://arxiv.org/abs/2003.10580\n\n\" new state-of-the-art top-1 accuracy of 90.2% on ImageNet, which is 1.6% better than the existing state-of-the-art\"\n\n\"Like Pseudo Labels, Meta Pseudo Labels has a teacher network to generate pseudo labels on unlabeled data to teach a student network. However, unlike Pseudo Labels where the teacher is fixed, the teacher in Meta Pseudo Labels is constantly adapted by the feedback of the student's performance on the labeled dataset. \"\n\n\n![](https://pbs.twimg.com/media/ErovYVNU0AAyACb?format=jpg&name=medium)",
      "votes": 6,
      "replies": [
        {
          "id": 1186573,
          "postDate": "2021-02-04T22:45:42.583Z",
          "content": "<p>hey <a href=\"https://www.kaggle.com/hengck23\" target=\"_blank\">@hengck23</a> , can you kindly illustrate a bit the procedure for updating teacher model parameters? I kinda got lost understanding the pseudocode provided in the paper.</p>",
          "rawMarkdown": "hey @hengck23 , can you kindly illustrate a bit the procedure for updating teacher model parameters? I kinda got lost understanding the pseudocode provided in the paper."
        },
        {
          "id": 1187989,
          "postDate": "2021-02-05T20:51:14.683Z",
          "content": "<p>Hold on a sec, let me spin up that 2048 tpu cores.</p>",
          "rawMarkdown": "Hold on a sec, let me spin up that 2048 tpu cores."
        }
      ]
    },
    {
      "id": 1122715,
      "postDate": "2020-12-22T16:43:33.890Z",
      "content": "<p>MOCO PRETRAINING IMPROVES REPRESENTATION AND TRANSFERABILITY OF CHEST X-RAY MODELS - ICLR 2021<br>\n<a href=\"https://openreview.net/pdf?id=kmN6SQIjk-r\" target=\"_blank\">https://openreview.net/pdf?id=kmN6SQIjk-r</a></p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F0c068aa465dc9b24257407e289b0c078%2FSelection_128.png?generation=1608655394232898&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "MOCO PRETRAINING IMPROVES REPRESENTATION AND TRANSFERABILITY OF CHEST X-RAY MODELS - ICLR 2021\nhttps://openreview.net/pdf?id=kmN6SQIjk-r\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F0c068aa465dc9b24257407e289b0c078%2FSelection_128.png?generation=1608655394232898&alt=media)",
      "votes": 5
    },
    {
      "id": 1117051,
      "postDate": "2020-12-17T17:16:49.280Z",
      "content": "<p>does the detection of catheters and lines help in the grading?<br>\nto prove this, one can:</p>\n<ol>\n<li>use the training annotation and draw the lines onto the images (e.g. different colors for ETT,CVC,NGT)</li>\n<li>train your model on these fake images</li>\n</ol>\n<p>this represents the best results you can get since you have perfect detection. if it helps, then we can think of how to detect them, e.g. using segmentation, pre-processing to enhance the lines, add attention branch to focus attention on lines, etc</p>",
      "rawMarkdown": "does the detection of catheters and lines help in the grading?\nto prove this, one can:\n1. use the training annotation and draw the lines onto the images (e.g. different colors for ETT,CVC,NGT)\n2. train your model on these fake images\n\nthis represents the best results you can get since you have perfect detection. if it helps, then we can think of how to detect them, e.g. using segmentation, pre-processing to enhance the lines, add attention branch to focus attention on lines, etc\n\n",
      "votes": 3
    },
    {
      "id": 1148712,
      "postDate": "2021-01-11T10:35:19.340Z",
      "content": "<p>Tubular Shape Aware Data Generation for Semantic Segmentation in Medical Imaging<br>\n<a href=\"https://arxiv.org/pdf/2010.00907v1.pdf\" target=\"_blank\">https://arxiv.org/pdf/2010.00907v1.pdf</a></p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F96000972a03fda4d0e022d8ac04d80ac%2FSelection_032.png?generation=1610361306658819&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "Tubular Shape Aware Data Generation for Semantic Segmentation in Medical Imaging\nhttps://arxiv.org/pdf/2010.00907v1.pdf\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F96000972a03fda4d0e022d8ac04d80ac%2FSelection_032.png?generation=1610361306658819&alt=media)\n\n",
      "votes": 4
    },
    {
      "id": 1116386,
      "postDate": "2020-12-17T06:05:34.623Z",
      "content": "<p>a typical local validation results</p>\n<pre><code>valid_dataset : \n    len  = 6017\n    df   = (6017, 14)\n    label\n          0                 ETT - Abnormal =     18  (0.003)\n          1               ETT - Borderline =    214  (0.036)\n          2                   ETT - Normal =   1439  (0.239)\n          3                 NGT - Abnormal =     46  (0.008)\n          4               NGT - Borderline =    140  (0.023)\n          5      NGT - Incompletely Imaged =    438  (0.073)\n          6                   NGT - Normal =   1082  (0.180)\n          7                 CVC - Abnormal =    535  (0.089)\n          8               CVC - Borderline =   1595  (0.265)\n          9                   CVC - Normal =   4337  (0.721)\n         10     Swan Ganz Catheter Present =    168  (0.028)\n\nsubmit_dir : /root/share1/kaggle/2020/ranzcr/result/xception/512/fold2/valid/local-00006000_model\ninitial_checkpoint : /root/share1/kaggle/2020/ranzcr/result/xception/512/fold2/checkpoint/00006000_model.pth\n\nloss : 0.173312\nauc_flat : 0.964769\nauc      : [0.9226, 0.9439, 0.9883, 0.8992, 0.9172, 0.9676, 0.9769, 0.8743, 0.7853, 0.8602, 0.9943]\nauc mean : 0.920883\n</code></pre>\n<p>class 7,8,9 are the most difficult<br>\nCentral Venous Catheters (CVC)</p>\n<p>both  ETT - Abnormal, NGT - Abnormal  have very few test cases</p>",
      "rawMarkdown": "a typical local validation results\n```\nvalid_dataset : \n\tlen  = 6017\n\tdf   = (6017, 14)\n\tlabel\n\t\t  0                 ETT - Abnormal =     18  (0.003)\n\t\t  1               ETT - Borderline =    214  (0.036)\n\t\t  2                   ETT - Normal =   1439  (0.239)\n\t\t  3                 NGT - Abnormal =     46  (0.008)\n\t\t  4               NGT - Borderline =    140  (0.023)\n\t\t  5      NGT - Incompletely Imaged =    438  (0.073)\n\t\t  6                   NGT - Normal =   1082  (0.180)\n\t\t  7                 CVC - Abnormal =    535  (0.089)\n\t\t  8               CVC - Borderline =   1595  (0.265)\n\t\t  9                   CVC - Normal =   4337  (0.721)\n\t\t 10     Swan Ganz Catheter Present =    168  (0.028)\n\nsubmit_dir : /root/share1/kaggle/2020/ranzcr/result/xception/512/fold2/valid/local-00006000_model\ninitial_checkpoint : /root/share1/kaggle/2020/ranzcr/result/xception/512/fold2/checkpoint/00006000_model.pth\n\nloss : 0.173312\nauc_flat : 0.964769\nauc      : [0.9226, 0.9439, 0.9883, 0.8992, 0.9172, 0.9676, 0.9769, 0.8743, 0.7853, 0.8602, 0.9943]\nauc mean : 0.920883\n\n```\n\nclass 7,8,9 are the most difficult\nCentral Venous Catheters (CVC)\n\nboth  ETT - Abnormal, NGT - Abnormal  have very few test cases",
      "votes": 4
    },
    {
      "id": 1117716,
      "postDate": "2020-12-18T11:04:03.637Z",
      "content": "<p>here is a piece of code to render the tubes only the x-ray images<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F80bffb71bb65142cb1f22dcd997ad8aa%2F1.2.826.0.1.3680043.8.498.10334972209383771821443362353408232478.png?generation=1608289439209100&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "here is a piece of code to render the tubes only the x-ray images\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F80bffb71bb65142cb1f22dcd997ad8aa%2F1.2.826.0.1.3680043.8.498.10334972209383771821443362353408232478.png?generation=1608289439209100&alt=media)",
      "votes": 1,
      "replies": [
        {
          "id": 1117816,
          "postDate": "2020-12-18T13:32:09.807Z",
          "content": "<p>some interesting observations (you can use this to post process your results):</p>\n<ul>\n<li>there can only be one ETT in an image</li>\n<li>\"Incompletely Imaged\" may be related to image size? </li>\n</ul>",
          "rawMarkdown": "some interesting observations (you can use this to post process your results):\n- there can only be one ETT in an image\n- \"Incompletely Imaged\" may be related to image size? "
        },
        {
          "id": 1117974,
          "postDate": "2020-12-18T16:19:46.210Z",
          "content": "<p>visualizing CAM activation maps. The network seems to understand the image correctly. … hmm</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fbb87b9a5772cf7136569d6e70f804988%2FSelection_064.png?generation=1608308383423871&amp;alt=media\" alt=\"\"></p>",
          "rawMarkdown": "visualizing CAM activation maps. The network seems to understand the image correctly. ... hmm\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fbb87b9a5772cf7136569d6e70f804988%2FSelection_064.png?generation=1608308383423871&alt=media)",
          "votes": 2
        },
        {
          "id": 1117975,
          "postDate": "2020-12-18T16:19:46.210Z",
          "content": "<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F4afe454ffbefea9700371564e4f6b267%2FSelection_084.png?generation=1608309424898357&amp;alt=media\" alt=\"‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ \"></p>",
          "rawMarkdown": "![‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F4afe454ffbefea9700371564e4f6b267%2FSelection_084.png?generation=1608309424898357&alt=media)",
          "votes": 4
        },
        {
          "id": 1117980,
          "postDate": "2020-12-18T16:26:15.453Z",
          "content": "<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F583e433355f93e89bb2ce4822125713e%2FSelection_071.png?generation=1608308772974624&amp;alt=media\" alt=\"\"></p>",
          "rawMarkdown": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F583e433355f93e89bb2ce4822125713e%2FSelection_071.png?generation=1608308772974624&alt=media)",
          "votes": 3
        }
      ]
    },
    {
      "id": 1116594,
      "postDate": "2020-12-17T09:59:15.520Z",
      "content": "<p>how to decide when to stop training:</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fe304798dab39c074e3951607b03d1b32%2FSelection_025.png?generation=1608199150061614&amp;alt=media\" alt=\"\"></p>\n<p>from this graph, it is probably safe to stop when train loss reaches about 0.050.<br>\nbest validation results probably lies when train loss is between 0.05to 0.15</p>",
      "rawMarkdown": "how to decide when to stop training:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fe304798dab39c074e3951607b03d1b32%2FSelection_025.png?generation=1608199150061614&alt=media)\n\nfrom this graph, it is probably safe to stop when train loss reaches about 0.050.\nbest validation results probably lies when train loss is between 0.05to 0.15\n\n ",
      "votes": 2,
      "replies": [
        {
          "id": 1116964,
          "postDate": "2020-12-17T15:42:32.687Z",
          "content": "<p></p>\n<p>this is included in version 2020-12-17</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fe89abaa281b0e3d02bf3a9a0a15bf18e%2FSelection_029.png?generation=1608219647053034&amp;alt=media\" alt=\"\"></p>\n<p>see attached files</p>",
          "rawMarkdown": "~~Nightly built version: augmentation and preliminary results (expected to add +0.01 in mean AUC)~~\n\nthis is included in version 2020-12-17\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fe89abaa281b0e3d02bf3a9a0a15bf18e%2FSelection_029.png?generation=1608219647053034&alt=media)\n\nsee attached files"
        }
      ]
    },
    {
      "id": 1126353,
      "postDate": "2020-12-25T14:38:54.320Z",
      "content": "<p>It is an honor for every kaggler that you can participate in kaggle…Thanks,hengck</p>",
      "rawMarkdown": "It is an honor for every kaggler that you can participate in kaggle...Thanks,hengck",
      "votes": 1
    },
    {
      "id": 1188914,
      "postDate": "2021-02-06T15:23:06.377Z",
      "content": "<p>See you again <a href=\"https://www.kaggle.com/hengck23\" target=\"_blank\">@hengck23</a> , thanks for sharing👍</p>",
      "rawMarkdown": "See you again @hengck23 , thanks for sharing👍"
    },
    {
      "id": 1140430,
      "postDate": "2021-01-06T02:20:53.137Z",
      "content": "<p>i haven't tried the below yet. it is just a thought experiment:</p>\n<p>would image alignment helps? how to align the images in the first place (aka image registration to some canonical mean)?</p>\n<ul>\n<li>train a model</li>\n<li>do TTA, we want to collect samples of the form: (image,score, transformed_image,score, transformed_score).<br>\nwe are interested in poorly scored images that will do better if you have correctly align the image (e.g. geometric or intensity align)</li>\n<li>now we train a network:<ul>\n<li>input = image, transform</li>\n<li>target = predict score improvement</li></ul></li>\n</ul>",
      "rawMarkdown": "i haven't tried the below yet. it is just a thought experiment:\n\nwould image alignment helps? how to align the images in the first place (aka image registration to some canonical mean)?\n- train a model\n- do TTA, we want to collect samples of the form: (image,score, transformed\\_image,score, transformed\\_score).\nwe are interested in poorly scored images that will do better if you have correctly align the image (e.g. geometric or intensity align)\n- now we train a network:\n    - input = image, transform\n    - target = predict score improvement"
    },
    {
      "id": 1139242,
      "postDate": "2021-01-05T09:10:12.137Z",
      "content": "<p>Radiographic Assessment of Tubes, Lines, Drains, and Other Devices - Normal Placement, Positioning Errors, Complications, and Indications for Radiological Evaluation<br>\n<a href=\"https://www.uhhospitals.org/-/media/Files/Medical-Education/im-radiology-lines-and-tubes-kevin-kalisz.pdf?la=en&amp;hash=C4D108DD80921CD9A81E60ACFFFE782CD3900FA7\" target=\"_blank\">https://www.uhhospitals.org/-/media/Files/Medical-Education/im-radiology-lines-and-tubes-kevin-kalisz.pdf?la=en&amp;hash=C4D108DD80921CD9A81E60ACFFFE782CD3900FA7</a></p>\n<p>wrong placement can lead to complications, can detection of complications help ????</p>",
      "rawMarkdown": "Radiographic Assessment of Tubes, Lines, Drains, and Other Devices - Normal Placement, Positioning Errors, Complications, and Indications for Radiological Evaluation\nhttps://www.uhhospitals.org/-/media/Files/Medical-Education/im-radiology-lines-and-tubes-kevin-kalisz.pdf?la=en&hash=C4D108DD80921CD9A81E60ACFFFE782CD3900FA7\n\nwrong placement can lead to complications, can detection of complications help ????\n",
      "replies": [
        {
          "id": 1139561,
          "postDate": "2021-01-05T13:45:16.557Z",
          "content": "<blockquote>\n  <p>wrong placement can lead to complications, can detection of complications help ????</p>\n</blockquote>\n<p>Maybe you mean <strong>predict</strong>, because once a complication happens, doctors are very able to detect it. The main objective of this competitions is to predict if a tube is wrong placed, and therefore, predict a complication.</p>",
          "rawMarkdown": "> wrong placement can lead to complications, can detection of complications help ????\n\nMaybe you mean **predict**, because once a complication happens, doctors are very able to detect it. The main objective of this competitions is to predict if a tube is wrong placed, and therefore, predict a complication."
        }
      ]
    },
    {
      "id": 1139238,
      "postDate": "2021-01-05T09:05:49.030Z",
      "content": "<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F3930b8da0a10d7878e81c6aeb947b0f8%2FSelection_037.png?generation=1609837547170508&amp;alt=media\" alt=\"\"></p>",
      "rawMarkdown": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F3930b8da0a10d7878e81c6aeb947b0f8%2FSelection_037.png?generation=1609837547170508&alt=media)",
      "replies": [
        {
          "id": 1141509,
          "postDate": "2021-01-06T18:33:50.403Z",
          "rawMarkdown": "",
          "isDeleted": true
        }
      ]
    },
    {
      "id": 1122607,
      "postDate": "2020-12-22T14:59:35.363Z",
      "content": "<p>Hello,how do you adjust your lr in training?I am not see the code in your code.Thanks!</p>",
      "rawMarkdown": "Hello,how do you adjust your lr in training?I am not see the code in your code.Thanks!"
    },
    {
      "id": 1120811,
      "postDate": "2020-12-21T06:05:41.333Z",
      "content": "<p>You are excellent</p>",
      "rawMarkdown": "You are excellent"
    },
    {
      "id": 1120702,
      "postDate": "2020-12-21T04:09:18.013Z",
      "content": "<p>It's always a great job. Thank you for sharing. I will follow you.</p>",
      "rawMarkdown": "It's always a great job. Thank you for sharing. I will follow you."
    },
    {
      "id": 1117825,
      "postDate": "2020-12-18T13:42:51.087Z",
      "content": "<p>I saw another's work which also has a good LB score and use <strong>Xception</strong>, any ideas of why this model has a good performance on this competition?</p>\n<p>BTW:</p>\n<blockquote>\n  <p>version 3 (plan)</p>\n  <ul>\n  <li>add segmentation/attention/ as aux loss</li>\n  <li>think of how to use transformer</li>\n  </ul>\n</blockquote>\n<p>Why don't use <strong>visual transformer</strong>?</p>",
      "rawMarkdown": "I saw another's work which also has a good LB score and use **Xception**, any ideas of why this model has a good performance on this competition?\n\nBTW:\n\n>version 3 (plan)\n>- add segmentation/attention/ as aux loss\n>- think of how to use transformer\n\nWhy don't use **visual transformer**?",
      "replies": [
        {
          "id": 1118001,
          "postDate": "2020-12-18T16:46:43.267Z",
          "content": "<p>i don't think Xception performs better. others network can also get the same result if they are trained properly.</p>",
          "rawMarkdown": "i don't think Xception performs better. others network can also get the same result if they are trained properly.",
          "votes": 2
        }
      ]
    },
    {
      "id": 1117699,
      "postDate": "2020-12-18T10:51:55.457Z",
      "content": "<p>Nice job!</p>\n<p><a href=\"https://www.kaggle.com/hengck23\" target=\"_blank\">@hengck23</a> I've seen in your logs that you don't have the same number of iterations/epochs per fold.<br>\nBut, at the same time, you haven't implemented early stopping in your code.</p>\n<p>Are you supervising the training and stopping it manually?</p>",
      "rawMarkdown": "Nice job!\n\n@hengck23 I've seen in your logs that you don't have the same number of iterations/epochs per fold.\nBut, at the same time, you haven't implemented early stopping in your code.\n\nAre you supervising the training and stopping it manually?",
      "replies": [
        {
          "id": 1117712,
          "postDate": "2020-12-18T11:02:18.260Z",
          "content": "<p>currently, i did it manually. it may become automatic later after I know what hyper-parameters to use.</p>\n<p>you can refer to the paper: <a href=\"https://www.youtube.com/watch?v=DiNzQP7kK-s&amp;feature=youtu.be\" target=\"_blank\">https://www.youtube.com/watch?v=DiNzQP7kK-s&amp;feature=youtu.be</a><br>\noptimizer taken out of the box often does not performs better than hand-tuned one.</p>",
          "rawMarkdown": "currently, i did it manually. it may become automatic later after I know what hyper-parameters to use.\n\nyou can refer to the paper: https://www.youtube.com/watch?v=DiNzQP7kK-s&feature=youtu.be\noptimizer taken out of the box often does not performs better than hand-tuned one.\n\n",
          "votes": 1
        },
        {
          "id": 1117727,
          "postDate": "2020-12-18T11:15:01.570Z",
          "content": "<p>thanks a lot!</p>",
          "rawMarkdown": "thanks a lot!"
        }
      ]
    },
    {
      "id": 1117107,
      "postDate": "2020-12-17T18:25:23.190Z",
      "content": "<p>‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎</p>",
      "rawMarkdown": " ‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎"
    },
    {
      "id": 1116588,
      "postDate": "2020-12-17T09:50:44.003Z",
      "content": "<p>Nice job!Thanks for you share!</p>",
      "rawMarkdown": "Nice job!Thanks for you share!"
    }
  ],
  "comments": [
    {
      "id": 1152370,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2021-01-14T04:37:29.163000",
      "content": "<p>i can see the future of kaggle, pseudo label 2.0:</p>\n<ul>\n<li>generate many data</li>\n<li>some weak/self/semi supervised labeling</li>\n</ul>\n<p>Meta Pseudo Labels - google paper<br>\n<a href=\"https://arxiv.org/abs/2003.10580\" target=\"_blank\">https://arxiv.org/abs/2003.10580</a></p>\n<p>\" new state-of-the-art top-1 accuracy of 90.2% on ImageNet, which is 1.6% better than the existing state-of-the-art\"</p>\n<p>\"Like Pseudo Labels, Meta Pseudo Labels has a teacher network to generate pseudo labels on unlabeled data to teach a student network. However, unlike Pseudo Labels where the teacher is fixed, the teacher in Meta Pseudo Labels is constantly adapted by the feedback of the student's performance on the labeled dataset. \"</p>\n<p><img src=\"https://pbs.twimg.com/media/ErovYVNU0AAyACb?format=jpg&amp;name=medium\" alt=\"\"></p>",
      "votes": 6,
      "replies": [
        {
          "id": 1186573,
          "author_name": "Zaber Ibn Abdul Hakim",
          "author_url": "",
          "post_date": "2021-02-04T22:45:42.583000",
          "content": "<p>hey <a href=\"https://www.kaggle.com/hengck23\" target=\"_blank\">@hengck23</a> , can you kindly illustrate a bit the procedure for updating teacher model parameters? I kinda got lost understanding the pseudocode provided in the paper.</p>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1187989,
          "author_name": "JunYong Tong",
          "author_url": "",
          "post_date": "2021-02-05T20:51:14.683000",
          "content": "<p>Hold on a sec, let me spin up that 2048 tpu cores.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1122715,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2020-12-22T16:43:33.890000",
      "content": "<p>MOCO PRETRAINING IMPROVES REPRESENTATION AND TRANSFERABILITY OF CHEST X-RAY MODELS - ICLR 2021<br>\n<a href=\"https://openreview.net/pdf?id=kmN6SQIjk-r\" target=\"_blank\">https://openreview.net/pdf?id=kmN6SQIjk-r</a></p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F0c068aa465dc9b24257407e289b0c078%2FSelection_128.png?generation=1608655394232898&amp;alt=media\" alt=\"\"></p>",
      "votes": 5,
      "replies": []
    },
    {
      "id": 1117051,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2020-12-17T17:16:49.280000",
      "content": "<p>does the detection of catheters and lines help in the grading?<br>\nto prove this, one can:</p>\n<ol>\n<li>use the training annotation and draw the lines onto the images (e.g. different colors for ETT,CVC,NGT)</li>\n<li>train your model on these fake images</li>\n</ol>\n<p>this represents the best results you can get since you have perfect detection. if it helps, then we can think of how to detect them, e.g. using segmentation, pre-processing to enhance the lines, add attention branch to focus attention on lines, etc</p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 1148712,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2021-01-11T10:35:19.340000",
      "content": "<p>Tubular Shape Aware Data Generation for Semantic Segmentation in Medical Imaging<br>\n<a href=\"https://arxiv.org/pdf/2010.00907v1.pdf\" target=\"_blank\">https://arxiv.org/pdf/2010.00907v1.pdf</a></p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F96000972a03fda4d0e022d8ac04d80ac%2FSelection_032.png?generation=1610361306658819&amp;alt=media\" alt=\"\"></p>",
      "votes": 4,
      "replies": []
    },
    {
      "id": 1116386,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2020-12-17T06:05:34.623000",
      "content": "<p>a typical local validation results</p>\n<pre><code>valid_dataset : \n    len  = 6017\n    df   = (6017, 14)\n    label\n          0                 ETT - Abnormal =     18  (0.003)\n          1               ETT - Borderline =    214  (0.036)\n          2                   ETT - Normal =   1439  (0.239)\n          3                 NGT - Abnormal =     46  (0.008)\n          4               NGT - Borderline =    140  (0.023)\n          5      NGT - Incompletely Imaged =    438  (0.073)\n          6                   NGT - Normal =   1082  (0.180)\n          7                 CVC - Abnormal =    535  (0.089)\n          8               CVC - Borderline =   1595  (0.265)\n          9                   CVC - Normal =   4337  (0.721)\n         10     Swan Ganz Catheter Present =    168  (0.028)\n\nsubmit_dir : /root/share1/kaggle/2020/ranzcr/result/xception/512/fold2/valid/local-00006000_model\ninitial_checkpoint : /root/share1/kaggle/2020/ranzcr/result/xception/512/fold2/checkpoint/00006000_model.pth\n\nloss : 0.173312\nauc_flat : 0.964769\nauc      : [0.9226, 0.9439, 0.9883, 0.8992, 0.9172, 0.9676, 0.9769, 0.8743, 0.7853, 0.8602, 0.9943]\nauc mean : 0.920883\n</code></pre>\n<p>class 7,8,9 are the most difficult<br>\nCentral Venous Catheters (CVC)</p>\n<p>both  ETT - Abnormal, NGT - Abnormal  have very few test cases</p>",
      "votes": 4,
      "replies": []
    },
    {
      "id": 1117716,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2020-12-18T11:04:03.637000",
      "content": "<p>here is a piece of code to render the tubes only the x-ray images<br>\n<img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F80bffb71bb65142cb1f22dcd997ad8aa%2F1.2.826.0.1.3680043.8.498.10334972209383771821443362353408232478.png?generation=1608289439209100&amp;alt=media\" alt=\"\"></p>",
      "votes": 1,
      "replies": [
        {
          "id": 1117816,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2020-12-18T13:32:09.807000",
          "content": "<p>some interesting observations (you can use this to post process your results):</p>\n<ul>\n<li>there can only be one ETT in an image</li>\n<li>\"Incompletely Imaged\" may be related to image size? </li>\n</ul>",
          "votes": 0,
          "replies": []
        },
        {
          "id": 1117974,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2020-12-18T16:19:46.210000",
          "content": "<p>visualizing CAM activation maps. The network seems to understand the image correctly. … hmm</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fbb87b9a5772cf7136569d6e70f804988%2FSelection_064.png?generation=1608308383423871&amp;alt=media\" alt=\"\"></p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 1117975,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2020-12-18T16:19:46.210000",
          "content": "<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F4afe454ffbefea9700371564e4f6b267%2FSelection_084.png?generation=1608309424898357&amp;alt=media\" alt=\"‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ \"></p>",
          "votes": 4,
          "replies": []
        },
        {
          "id": 1117980,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2020-12-18T16:26:15.453000",
          "content": "<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F583e433355f93e89bb2ce4822125713e%2FSelection_071.png?generation=1608308772974624&amp;alt=media\" alt=\"\"></p>",
          "votes": 3,
          "replies": []
        }
      ]
    },
    {
      "id": 1116594,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2020-12-17T09:59:15.520000",
      "content": "<p>how to decide when to stop training:</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fe304798dab39c074e3951607b03d1b32%2FSelection_025.png?generation=1608199150061614&amp;alt=media\" alt=\"\"></p>\n<p>from this graph, it is probably safe to stop when train loss reaches about 0.050.<br>\nbest validation results probably lies when train loss is between 0.05to 0.15</p>",
      "votes": 2,
      "replies": [
        {
          "id": 1116964,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2020-12-17T15:42:32.687000",
          "content": "<p></p>\n<p>this is included in version 2020-12-17</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fe89abaa281b0e3d02bf3a9a0a15bf18e%2FSelection_029.png?generation=1608219647053034&amp;alt=media\" alt=\"\"></p>\n<p>see attached files</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1126353,
      "author_name": "Dapeng",
      "author_url": "",
      "post_date": "2020-12-25T14:38:54.320000",
      "content": "<p>It is an honor for every kaggler that you can participate in kaggle…Thanks,hengck</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 1188914,
      "author_name": "cswwp",
      "author_url": "",
      "post_date": "2021-02-06T15:23:06.377000",
      "content": "<p>See you again <a href=\"https://www.kaggle.com/hengck23\" target=\"_blank\">@hengck23</a> , thanks for sharing👍</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1140430,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2021-01-06T02:20:53.137000",
      "content": "<p>i haven't tried the below yet. it is just a thought experiment:</p>\n<p>would image alignment helps? how to align the images in the first place (aka image registration to some canonical mean)?</p>\n<ul>\n<li>train a model</li>\n<li>do TTA, we want to collect samples of the form: (image,score, transformed_image,score, transformed_score).<br>\nwe are interested in poorly scored images that will do better if you have correctly align the image (e.g. geometric or intensity align)</li>\n<li>now we train a network:<ul>\n<li>input = image, transform</li>\n<li>target = predict score improvement</li></ul></li>\n</ul>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1139242,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2021-01-05T09:10:12.137000",
      "content": "<p>Radiographic Assessment of Tubes, Lines, Drains, and Other Devices - Normal Placement, Positioning Errors, Complications, and Indications for Radiological Evaluation<br>\n<a href=\"https://www.uhhospitals.org/-/media/Files/Medical-Education/im-radiology-lines-and-tubes-kevin-kalisz.pdf?la=en&amp;hash=C4D108DD80921CD9A81E60ACFFFE782CD3900FA7\" target=\"_blank\">https://www.uhhospitals.org/-/media/Files/Medical-Education/im-radiology-lines-and-tubes-kevin-kalisz.pdf?la=en&amp;hash=C4D108DD80921CD9A81E60ACFFFE782CD3900FA7</a></p>\n<p>wrong placement can lead to complications, can detection of complications help ????</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1139561,
          "author_name": "Hiram Coria 🧬",
          "author_url": "",
          "post_date": "2021-01-05T13:45:16.557000",
          "content": "<blockquote>\n  <p>wrong placement can lead to complications, can detection of complications help ????</p>\n</blockquote>\n<p>Maybe you mean <strong>predict</strong>, because once a complication happens, doctors are very able to detect it. The main objective of this competitions is to predict if a tube is wrong placed, and therefore, predict a complication.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1139238,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2021-01-05T09:05:49.030000",
      "content": "<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F3930b8da0a10d7878e81c6aeb947b0f8%2FSelection_037.png?generation=1609837547170508&amp;alt=media\" alt=\"\"></p>",
      "votes": 0,
      "replies": [
        {
          "id": 1141509,
          "author_name": "",
          "author_url": "",
          "post_date": "2021-01-06T18:33:50.403000",
          "content": "",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1122607,
      "author_name": "Bcw93",
      "author_url": "",
      "post_date": "2020-12-22T14:59:35.363000",
      "content": "<p>Hello,how do you adjust your lr in training?I am not see the code in your code.Thanks!</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1120811,
      "author_name": "guochangZhang",
      "author_url": "",
      "post_date": "2020-12-21T06:05:41.333000",
      "content": "<p>You are excellent</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1120702,
      "author_name": "Respect",
      "author_url": "",
      "post_date": "2020-12-21T04:09:18.013000",
      "content": "<p>It's always a great job. Thank you for sharing. I will follow you.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1117825,
      "author_name": "Hiram Coria 🧬",
      "author_url": "",
      "post_date": "2020-12-18T13:42:51.087000",
      "content": "<p>I saw another's work which also has a good LB score and use <strong>Xception</strong>, any ideas of why this model has a good performance on this competition?</p>\n<p>BTW:</p>\n<blockquote>\n  <p>version 3 (plan)</p>\n  <ul>\n  <li>add segmentation/attention/ as aux loss</li>\n  <li>think of how to use transformer</li>\n  </ul>\n</blockquote>\n<p>Why don't use <strong>visual transformer</strong>?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1118001,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2020-12-18T16:46:43.267000",
          "content": "<p>i don't think Xception performs better. others network can also get the same result if they are trained properly.</p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 1117699,
      "author_name": "Virilo Tejedor Aguilera",
      "author_url": "",
      "post_date": "2020-12-18T10:51:55.457000",
      "content": "<p>Nice job!</p>\n<p><a href=\"https://www.kaggle.com/hengck23\" target=\"_blank\">@hengck23</a> I've seen in your logs that you don't have the same number of iterations/epochs per fold.<br>\nBut, at the same time, you haven't implemented early stopping in your code.</p>\n<p>Are you supervising the training and stopping it manually?</p>",
      "votes": 0,
      "replies": [
        {
          "id": 1117712,
          "author_name": "hengck23",
          "author_url": "",
          "post_date": "2020-12-18T11:02:18.260000",
          "content": "<p>currently, i did it manually. it may become automatic later after I know what hyper-parameters to use.</p>\n<p>you can refer to the paper: <a href=\"https://www.youtube.com/watch?v=DiNzQP7kK-s&amp;feature=youtu.be\" target=\"_blank\">https://www.youtube.com/watch?v=DiNzQP7kK-s&amp;feature=youtu.be</a><br>\noptimizer taken out of the box often does not performs better than hand-tuned one.</p>",
          "votes": 1,
          "replies": []
        },
        {
          "id": 1117727,
          "author_name": "Virilo Tejedor Aguilera",
          "author_url": "",
          "post_date": "2020-12-18T11:15:01.570000",
          "content": "<p>thanks a lot!</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 1117107,
      "author_name": "hengck23",
      "author_url": "",
      "post_date": "2020-12-17T18:25:23.190000",
      "content": "<p>‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 1116588,
      "author_name": "Bcw93",
      "author_url": "",
      "post_date": "2020-12-17T09:50:44.003000",
      "content": "<p>Nice job!Thanks for you share!</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1116357": "I estimate i can get LB > 0.985 at the end of 2 months.\n\nI am working towards the following:\n- generate synthetic GAN data/augment data/external data/test/data and treat them as noisy data\n- use self/weak supervised learning to make clean labels or weigh samples\n- use segmentation information for aux loss\n- transformer to model relationship (the distance of tubes from chest parts)\n- etc ...\n\n----\nif you are interested in my work and progress, you can follow my work here. it will be updated when I am free to work and make progress.\n\nall code/model/result/log file can be downloaded via:\nhttps://drive.google.com/drive/folders/1jza6k-wHFVrJUaRgjvuR5_Mpwk7ClNVd?usp=sharing\n\nhere is the first version:\n1. 2020-12-17\n- just a dummy baseline to check the problem and data\n- 5xfold exception 512x512 without augmentation: LB =0.951 (ensemble), LB=0.939 (fold-2)\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F9a4dc6b9904ac9962b24c6a2397dce6c%2FSelection_058.png?generation=1608182442517494&alt=media)\n\n2. 2020-12-18\n- LB-0.956 - 5xFold 2xTTA xception-512\n- Implement simple augmentation and TTA. better results by using lower learning rate \n\n---\n\nversion 2 (plan)\n- add more augmentation, preprocessing\n- structured ouput\n- add visualization and interpretability of model (e.g. CAM heatmap)\n- how to probe for AUC kaggle (I need to find out the number of instances per class for the test set ... some class has rare occurrence ... it is possible to recover the positive ground truth)\n- I need  \"Sharpness-Aware Minimization optimizer.\"!!!\n\n---\n\nversion 3 (plan)\n- add segmentation/attention/ as aux loss\n- think of how to use transformer\n\n---\n\nversion 4 (plan)\n- how to generate synthetic data and use self-supervised learning",
    "1152370": "i can see the future of kaggle, pseudo label 2.0:\n- generate many data\n- some weak/self/semi supervised labeling\n\nMeta Pseudo Labels - google paper\nhttps://arxiv.org/abs/2003.10580\n\n\" new state-of-the-art top-1 accuracy of 90.2% on ImageNet, which is 1.6% better than the existing state-of-the-art\"\n\n\"Like Pseudo Labels, Meta Pseudo Labels has a teacher network to generate pseudo labels on unlabeled data to teach a student network. However, unlike Pseudo Labels where the teacher is fixed, the teacher in Meta Pseudo Labels is constantly adapted by the feedback of the student's performance on the labeled dataset. \"\n\n\n![](https://pbs.twimg.com/media/ErovYVNU0AAyACb?format=jpg&name=medium)",
    "1122715": "MOCO PRETRAINING IMPROVES REPRESENTATION AND TRANSFERABILITY OF CHEST X-RAY MODELS - ICLR 2021\nhttps://openreview.net/pdf?id=kmN6SQIjk-r\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F0c068aa465dc9b24257407e289b0c078%2FSelection_128.png?generation=1608655394232898&alt=media)",
    "1117051": "does the detection of catheters and lines help in the grading?\nto prove this, one can:\n1. use the training annotation and draw the lines onto the images (e.g. different colors for ETT,CVC,NGT)\n2. train your model on these fake images\n\nthis represents the best results you can get since you have perfect detection. if it helps, then we can think of how to detect them, e.g. using segmentation, pre-processing to enhance the lines, add attention branch to focus attention on lines, etc\n\n",
    "1148712": "Tubular Shape Aware Data Generation for Semantic Segmentation in Medical Imaging\nhttps://arxiv.org/pdf/2010.00907v1.pdf\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F96000972a03fda4d0e022d8ac04d80ac%2FSelection_032.png?generation=1610361306658819&alt=media)\n\n",
    "1116386": "a typical local validation results\n```\nvalid_dataset : \n\tlen  = 6017\n\tdf   = (6017, 14)\n\tlabel\n\t\t  0                 ETT - Abnormal =     18  (0.003)\n\t\t  1               ETT - Borderline =    214  (0.036)\n\t\t  2                   ETT - Normal =   1439  (0.239)\n\t\t  3                 NGT - Abnormal =     46  (0.008)\n\t\t  4               NGT - Borderline =    140  (0.023)\n\t\t  5      NGT - Incompletely Imaged =    438  (0.073)\n\t\t  6                   NGT - Normal =   1082  (0.180)\n\t\t  7                 CVC - Abnormal =    535  (0.089)\n\t\t  8               CVC - Borderline =   1595  (0.265)\n\t\t  9                   CVC - Normal =   4337  (0.721)\n\t\t 10     Swan Ganz Catheter Present =    168  (0.028)\n\nsubmit_dir : /root/share1/kaggle/2020/ranzcr/result/xception/512/fold2/valid/local-00006000_model\ninitial_checkpoint : /root/share1/kaggle/2020/ranzcr/result/xception/512/fold2/checkpoint/00006000_model.pth\n\nloss : 0.173312\nauc_flat : 0.964769\nauc      : [0.9226, 0.9439, 0.9883, 0.8992, 0.9172, 0.9676, 0.9769, 0.8743, 0.7853, 0.8602, 0.9943]\nauc mean : 0.920883\n\n```\n\nclass 7,8,9 are the most difficult\nCentral Venous Catheters (CVC)\n\nboth  ETT - Abnormal, NGT - Abnormal  have very few test cases",
    "1117716": "here is a piece of code to render the tubes only the x-ray images\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F80bffb71bb65142cb1f22dcd997ad8aa%2F1.2.826.0.1.3680043.8.498.10334972209383771821443362353408232478.png?generation=1608289439209100&alt=media)",
    "1116594": "how to decide when to stop training:\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fe304798dab39c074e3951607b03d1b32%2FSelection_025.png?generation=1608199150061614&alt=media)\n\nfrom this graph, it is probably safe to stop when train loss reaches about 0.050.\nbest validation results probably lies when train loss is between 0.05to 0.15\n\n ",
    "1126353": "It is an honor for every kaggler that you can participate in kaggle...Thanks,hengck",
    "1188914": "See you again @hengck23 , thanks for sharing👍",
    "1140430": "i haven't tried the below yet. it is just a thought experiment:\n\nwould image alignment helps? how to align the images in the first place (aka image registration to some canonical mean)?\n- train a model\n- do TTA, we want to collect samples of the form: (image,score, transformed\\_image,score, transformed\\_score).\nwe are interested in poorly scored images that will do better if you have correctly align the image (e.g. geometric or intensity align)\n- now we train a network:\n    - input = image, transform\n    - target = predict score improvement",
    "1139242": "Radiographic Assessment of Tubes, Lines, Drains, and Other Devices - Normal Placement, Positioning Errors, Complications, and Indications for Radiological Evaluation\nhttps://www.uhhospitals.org/-/media/Files/Medical-Education/im-radiology-lines-and-tubes-kevin-kalisz.pdf?la=en&hash=C4D108DD80921CD9A81E60ACFFFE782CD3900FA7\n\nwrong placement can lead to complications, can detection of complications help ????\n",
    "1139238": "![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2F3930b8da0a10d7878e81c6aeb947b0f8%2FSelection_037.png?generation=1609837547170508&alt=media)",
    "1122607": "Hello,how do you adjust your lr in training?I am not see the code in your code.Thanks!",
    "1120811": "You are excellent",
    "1120702": "It's always a great job. Thank you for sharing. I will follow you.",
    "1117825": "I saw another's work which also has a good LB score and use **Xception**, any ideas of why this model has a good performance on this competition?\n\nBTW:\n\n>version 3 (plan)\n>- add segmentation/attention/ as aux loss\n>- think of how to use transformer\n\nWhy don't use **visual transformer**?",
    "1117699": "Nice job!\n\n@hengck23 I've seen in your logs that you don't have the same number of iterations/epochs per fold.\nBut, at the same time, you haven't implemented early stopping in your code.\n\nAre you supervising the training and stopping it manually?",
    "1117107": " ‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎‏‏‎ ‎",
    "1116588": "Nice job!Thanks for you share!"
  }
}