![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fd8e62984343f73764b13346d5d16aa71%2FSelection_089.png?generation=1608359259306689&alt=media\")
the key idea is to modify the activation (activation of the feature map in this case) by changing the input so that the network can pick up extra supervision and decide if they want to use them or not.
\nThere are many ways to implement this. if you want to mix those with and without annotations, you can add a flag \"is additional annotation available\" as input at training. this is probably the easiest \"human in the loop for training\"
\nnote: if the kaggle annotation is not enough, add your own annotation, etc. part of lung, etc
\nin my experiment, I estimated the addition annotation supervision signals will cause an improvement of about 2.0% to 2.5% for the AUC metric
", "messages": [ { "id": 1118515, "postDate": "2020-12-19T06:27:46.523Z", "content": "
the key idea is to modify the activation (activation of the feature map in this case) by changing the input so that the network can pick up extra supervision and decide if they want to use them or not.
\nThere are many ways to implement this. if you want to mix those with and without annotations, you can add a flag \"is additional annotation available\" as input at training. this is probably the easiest \"human in the loop for training\"
\nnote: if the kaggle annotation is not enough, add your own annotation, etc. part of lung, etc
\nin my experiment, I estimated the addition annotation supervision signals will cause an improvement of about 2.0% to 2.5% for the AUC metric
", "rawMarkdown": "\n\nthe key idea is to modify the activation (activation of the feature map in this case) by changing the input so that the network can pick up extra supervision and decide if they want to use them or not.\n\nThere are many ways to implement this. if you want to mix those with and without annotations, you can add a flag \"is additional annotation available\" as input at training. this is probably the easiest \"human in the loop for training\"\n\nnote: if the kaggle annotation is not enough, add your own annotation, etc. part of lung, etc\n\nin my experiment, I estimated the addition annotation supervision signals will cause an improvement of about 2.0% to 2.5% for the AUC metric", "votes": 169 }, { "id": 1213323, "postDate": "2021-02-22T03:56:17.227Z", "content": "check these as well
\nEmbedding Human Knowledge into Deep Neural Network via Attention Map
\nhttps://arxiv.org/abs/1905.03540
https://github.com/machine-perception-robotics-group/attention_branch_network/
\n![\"\"](\"https://storage.googleapis.com/groundai-web-prod/media%2Fusers%2Fuser_233037%2Fproject_360896%2Fimages%2Fx1.png\")
Human-in-the-loop!
", "rawMarkdown": "Human-in-the-loop!" } ] }, { "id": 1226219, "postDate": "2021-03-04T10:49:16.970Z", "content": "reference code and model is here!
\nhttps://drive.google.com/drive/folders/1JRqCpV3NMGFeyd5ws6_pyXwQ5x3n8CHF
\nsee folder \"2021-03-04\" and refer to the readme.pptx
![\"\"](\"https://i.ibb.co/cvNVfSy/Selection-035.png\")
i further compare my implementation and those by public kernels:
\nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/215910
\nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/207577
i confirm a third stage will improve results:
\nstage3 : remove teacher consistency loss and finetune with only bce
this is because teacher consistency loss can limit performance at fine tunning. L2 loss is higher than BCE.
\nL2 loss acts as regularsier in the early stage, but hinders improvement at fine tunning.
", "rawMarkdown": "i further compare my implementation and those by public kernels:\nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/215910\nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/207577\n\ni confirm a third stage will improve results:\nstage3 : remove teacher consistency loss and finetune with only bce\n\nthis is because teacher consistency loss can limit performance at fine tunning. L2 loss is higher than BCE.\n\n L2 loss acts as regularsier in the early stage, but hinders improvement at fine tunning." } ] }, { "id": 1118777, "postDate": "2020-12-19T11:46:47.753Z", "content": "if one wants to be fanciful (typo error: should have been x = (1+a)*x
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fd5d7d39813aa6095732c9df13900f9fc%2FSelection_092.png?generation=1608378425203989&alt=media\")
even more fanciful
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fdfd1a32b29050b169e0ab8965afd6c02%2FSelection_094.png?generation=1608378725312235&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F113660%2Fc7d613c6a38945ecd88249728b1c997a%2FSelection_099.png?generation=1608428351163583&alt=media\")
\n\nI estimated the addition annotation supervision signals will cause an improvement of about 2.0% to 2.5% for the AUC metric
\n
Is your mean your CV increase from x to x+0.02?
\nThanks for sharing your ideas. These are really very impressive.
i haven't tried this yet, but it is an idea: contrastive attention for self-supervised learning
\nthe idea is:
\nHello, are we allowed to use both normal images dataset and annotated images dataset? If I understand this competition correctly? Haha sorry, I joined late.
", "rawMarkdown": "Hello, are we allowed to use both normal images dataset and annotated images dataset? If I understand this competition correctly? Haha sorry, I joined late." }, { "id": 1220918, "postDate": "2021-02-28T13:45:03.440Z", "content": "Hi @h3309089 can you please give us a intuition on what contrastive learning is , I am very new to this topic , it will help me a lot
\nThanks
", "rawMarkdown": "Hi @h3309089 can you please give us a intuition on what contrastive learning is , I am very new to this topic , it will help me a lot\n\nThanks", "votes": -1 }, { "id": 1221994, "postDate": "2021-03-01T13:10:31.807Z", "content": "search for MoCo or SimCLR. Note it requires a lot of computation and sometimes huge batch sizes
", "rawMarkdown": "search for MoCo or SimCLR. Note it requires a lot of computation and sometimes huge batch sizes" }, { "id": 1224362, "postDate": "2021-03-02T16:52:56.470Z", "content": "added a few interesting ideas
\nhttps://ibb.co/2NhCgKT
\nhttps://ibb.co/CMc22pW
\nhttps://ibb.co/S09dJ3x
\nhttps://ibb.co/C7mC5F1
extreme augmentation with teacher attention distillation improves my LB from 0.959 to 0.960 for single fold resnet50d 640 input.
\nThese extreme augmentations failed without teacher distillation
", "rawMarkdown": "added a few interesting ideas\nhttps://ibb.co/2NhCgKT\nhttps://ibb.co/CMc22pW\nhttps://ibb.co/S09dJ3x\nhttps://ibb.co/C7mC5F1\n\nextreme augmentation with teacher attention distillation improves my LB from 0.959 to 0.960 for single fold resnet50d 640 input.\n\nThese extreme augmentations failed without teacher distillation", "votes": 1 }, { "id": 1224366, "postDate": "2021-03-02T16:54:11.363Z", "content": "\"search for MoCo or SimCLR. Note it requires a lot of computation and sometimes huge batch sizes\"
\ni believe strong supervision reduce batch size. maybe someone can write a paper on it …
", "rawMarkdown": "\"search for MoCo or SimCLR. Note it requires a lot of computation and sometimes huge batch sizes\"\n\ni believe strong supervision reduce batch size. maybe someone can write a paper on it ... ", "votes": -1 }, { "id": 1225777, "postDate": "2021-03-03T23:13:24.487Z", "content": "multi task teacher is also giving higher LB score (LB 0.960 for single fold resnet50d).
\nit is better to have a separate feature map teacher for each task.
Could you please share some work or paper that implements the presented concepts using TF2.0. Thanks
", "rawMarkdown": "Could you please share some work or paper that implements the presented concepts using TF2.0. Thanks", "votes": 1 }, { "id": 1118743, "postDate": "2020-12-19T11:04:21.597Z", "content": "Thank you for sharing your idea! I'm not familiar with this concept. So I would like to ask some questions if you don't mind. What do you use for consistency loss? Do you train the teacher first and then distill the teacher to the student?
", "rawMarkdown": "Thank you for sharing your idea! I'm not familiar with this concept. So I would like to ask some questions if you don't mind. What do you use for consistency loss? Do you train the teacher first and then distill the teacher to the student?\n", "votes": 1, "replies": [ { "id": 1118765, "postDate": "2020-12-19T11:32:40.230Z", "content": "\"What do you use for consistency loss?\"
\nI think this is not so important. i think normal L2 loss is good enough
\"Do you train the teacher first and then distill the teacher to the student?\"
\nmy plan is to train the teacher first and freeze it. a model trained on rendered image can achieve about 96% in local CV
Thank you!
", "rawMarkdown": "Thank you!" }, { "id": 1136613, "postDate": "2021-01-03T08:49:09.620Z", "content": "Got great insights from this discussion. Thank you @hengck23
", "rawMarkdown": "Got great insights from this discussion. Thank you @hengck23 " }, { "id": 1212293, "postDate": "2021-02-21T04:39:34.167Z", "content": "Will teacher better than student?
", "rawMarkdown": "Will teacher better than student?" } ] }, { "id": 1239780, "postDate": "2021-03-16T02:25:41.553Z", "content": "i find a new home for this method. it think it will shine there
\nhttps://www.kaggle.com/c/bms-molecular-translation
![\"\"](\"https://i.ibb.co/L1S3cW0/Selection-352.png\")
some results.
\ni use two-stage (instead of three-stage in some public code)
\n![\"\"](\"https://i.ibb.co/nmJBJNx/Selection-191.png\")
\n![\"\"](\"https://i.ibb.co/4YfY3xG/Selection-193.png\")
batch_size = len(batch['index'])\n df = batch['df']\n\n target = batch['target'].cuda()\n image = batch['image'].cuda()\n render = batch['render'].cuda()\n is_annotate = batch['is_annotate'].cuda()\n\n with torch.no_grad():\n teacher.eval()\n logit_t, f_t = teacher(render)\n\n #----\n net.train()\n optimizer.zero_grad()\n\n def loss_consistency(f,truth):\n f = f[is_annotate==1]\n t = truth[is_annotate==1]\n loss = F.mse_loss(f,t)\n return loss\n\n if is_mixed_precision: \n image = image.half()\n with amp.autocast():\n logit,f = net(image)\n loss0 = binary_cross_entropy_loss(logit, target)\n loss1 = loss_consistency(f, f_t)\n\n scaler.scale(loss0+loss1).backward()\n scaler.step(optimizer)\n scaler.update()\nsince the score on the \"my submissions\" page can be ranked, I can see:
\nonly one fold:
\nfor 640x640 input, tta= origina+flip:
\nresnet200d + teacher attention : local cv 0.956805, lb 0.959 (best)
\nresnet50d + teacher attention : local cv 0.956385, lb 0.959
\nresnet200d : local cv 0.955946, lb 0.959 (worst)
\n(https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/204950)
it is seen that now resnet50d is as good as resnet200d with extra supervision
", "rawMarkdown": "since the score on the \"my submissions\" page can be ranked, I can see:\n\nonly one fold:\nfor 640x640 input, tta= origina+flip:\nresnet200d + teacher attention : local cv 0.956805, lb 0.959 (best)\nresnet50d + teacher attention : local cv 0.956385, lb 0.959 \nresnet200d : local cv 0.955946, lb 0.959 (worst)\n(https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/204950)\n\n\nit is seen that now resnet50d is as good as resnet200d with extra supervision", "votes": 1 }, { "id": 1220422, "postDate": "2021-02-28T01:26:45.613Z", "content": "i will not be surprised that more hand-annotation will be the key to winning this challenge.
\nwe will see who can do it \"smartly\"
![\"\"](\"https://i.ibb.co/pd14X6t/Selection-215.png\")
replacing line annotation with endpoint annotation. how is teacher distillation affected?
\nNote this is results for CVC only. the others like ETT are not used at all
![\"\"](\"https://i.ibb.co/Y3j63HP/Selection-221.png\")
be creative!
\nmulti scale annotation, find pretain model that segment lung,heart, etc …. maybe they would work?
", "rawMarkdown": "be creative!\n\nmulti scale annotation, find pretain model that segment lung,heart, etc .... maybe they would work?", "votes": 2 }, { "id": 1220873, "postDate": "2021-02-28T12:39:29.817Z", "content": "@hengck23 you tried normfree nets? how does it work for you?
", "rawMarkdown": "@hengck23 you tried normfree nets? how does it work for you?", "votes": 1 } ] }, { "id": 1151143, "postDate": "2021-01-13T06:53:54.077Z", "content": "i would like to combine my idea with this google paper:
\nConcept Bottleneck Models
\nhttps://arxiv.org/pdf/2007.04612.pdf
\"By construction, we can intervene on these concept bottleneck models by editing their predicted concept values and propagating these changes to the final prediction\"
\ni want to intervene by sketching on the image
", "rawMarkdown": "i would like to combine my idea with this google paper:\nConcept Bottleneck Models\nhttps://arxiv.org/pdf/2007.04612.pdf\n\n\"By construction, we can intervene on these concept bottleneck models by editing their predicted concept values and propagating these changes to the final prediction\"\n\ni want to intervene by sketching on the image", "votes": 1 }, { "id": 1214113, "postDate": "2021-02-22T15:43:44.077Z", "content": "Great discussion thread. Can anyone share an example of the Teacher-Student model example kernel?
", "rawMarkdown": "Great discussion thread. Can anyone share an example of the Teacher-Student model example kernel? " }, { "id": 1133438, "postDate": "2020-12-31T08:08:51.367Z", "content": "Sir how you ensemble with annotation without annotations can you explain? And what part of the images you annotated ? Thanks in advance.
", "rawMarkdown": "Sir how you ensemble with annotation without annotations can you explain? And what part of the images you annotated ? Thanks in advance. " }, { "id": 1121948, "postDate": "2020-12-22T03:40:55.427Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1121735, "postDate": "2020-12-21T21:34:26.697Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 1118630, "postDate": "2020-12-19T08:55:24.363Z", "content": "Thanks for insights!
", "rawMarkdown": "Thanks for insights!" } ], "comments": [ { "id": 1213323, "author_name": "hengck23", "author_url": "", "post_date": "2021-02-22T03:56:17.227000", "content": "check these as well
\nEmbedding Human Knowledge into Deep Neural Network via Attention Map
\nhttps://arxiv.org/abs/1905.03540
https://github.com/machine-perception-robotics-group/attention_branch_network/
\n
Human-in-the-loop!
", "votes": 0, "replies": [] } ] }, { "id": 1226219, "author_name": "hengck23", "author_url": "", "post_date": "2021-03-04T10:49:16.970000", "content": "reference code and model is here!
\nhttps://drive.google.com/drive/folders/1JRqCpV3NMGFeyd5ws6_pyXwQ5x3n8CHF
\nsee folder \"2021-03-04\" and refer to the readme.pptx
i further compare my implementation and those by public kernels:
\nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/215910
\nhttps://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/207577
i confirm a third stage will improve results:
\nstage3 : remove teacher consistency loss and finetune with only bce
this is because teacher consistency loss can limit performance at fine tunning. L2 loss is higher than BCE.
\nL2 loss acts as regularsier in the early stage, but hinders improvement at fine tunning.
", "votes": 0, "replies": [] } ] }, { "id": 1118777, "author_name": "hengck23", "author_url": "", "post_date": "2020-12-19T11:46:47.753000", "content": "if one wants to be fanciful (typo error: should have been x = (1+a)*x
\n
even more fanciful
\n
\n\nI estimated the addition annotation supervision signals will cause an improvement of about 2.0% to 2.5% for the AUC metric
\n
Is your mean your CV increase from x to x+0.02?
\nThanks for sharing your ideas. These are really very impressive.
i haven't tried this yet, but it is an idea: contrastive attention for self-supervised learning
\nthe idea is:
\nHello, are we allowed to use both normal images dataset and annotated images dataset? If I understand this competition correctly? Haha sorry, I joined late.
", "votes": 0, "replies": [] }, { "id": 1220918, "author_name": "Mr_KnowNothing", "author_url": "", "post_date": "2021-02-28T13:45:03.440000", "content": "Hi @h3309089 can you please give us a intuition on what contrastive learning is , I am very new to this topic , it will help me a lot
\nThanks
", "votes": -1, "replies": [] }, { "id": 1221994, "author_name": "Miroslav Valan", "author_url": "", "post_date": "2021-03-01T13:10:31.807000", "content": "search for MoCo or SimCLR. Note it requires a lot of computation and sometimes huge batch sizes
", "votes": 0, "replies": [] }, { "id": 1224362, "author_name": "hengck23", "author_url": "", "post_date": "2021-03-02T16:52:56.470000", "content": "added a few interesting ideas
\nhttps://ibb.co/2NhCgKT
\nhttps://ibb.co/CMc22pW
\nhttps://ibb.co/S09dJ3x
\nhttps://ibb.co/C7mC5F1
extreme augmentation with teacher attention distillation improves my LB from 0.959 to 0.960 for single fold resnet50d 640 input.
\nThese extreme augmentations failed without teacher distillation
", "votes": 1, "replies": [] }, { "id": 1224366, "author_name": "hengck23", "author_url": "", "post_date": "2021-03-02T16:54:11.363000", "content": "\"search for MoCo or SimCLR. Note it requires a lot of computation and sometimes huge batch sizes\"
\ni believe strong supervision reduce batch size. maybe someone can write a paper on it …
", "votes": -1, "replies": [] }, { "id": 1225777, "author_name": "hengck23", "author_url": "", "post_date": "2021-03-03T23:13:24.487000", "content": "multi task teacher is also giving higher LB score (LB 0.960 for single fold resnet50d).
\nit is better to have a separate feature map teacher for each task.
Could you please share some work or paper that implements the presented concepts using TF2.0. Thanks
", "votes": 1, "replies": [] }, { "id": 1118743, "author_name": "Tolga", "author_url": "", "post_date": "2020-12-19T11:04:21.597000", "content": "Thank you for sharing your idea! I'm not familiar with this concept. So I would like to ask some questions if you don't mind. What do you use for consistency loss? Do you train the teacher first and then distill the teacher to the student?
", "votes": 1, "replies": [ { "id": 1118765, "author_name": "hengck23", "author_url": "", "post_date": "2020-12-19T11:32:40.230000", "content": "\"What do you use for consistency loss?\"
\nI think this is not so important. i think normal L2 loss is good enough
\"Do you train the teacher first and then distill the teacher to the student?\"
\nmy plan is to train the teacher first and freeze it. a model trained on rendered image can achieve about 96% in local CV
Thank you!
", "votes": 0, "replies": [] }, { "id": 1136613, "author_name": "Dr. Amritpal Singh", "author_url": "", "post_date": "2021-01-03T08:49:09.620000", "content": "Got great insights from this discussion. Thank you @hengck23
", "votes": 0, "replies": [] }, { "id": 1212293, "author_name": "Zekun", "author_url": "", "post_date": "2021-02-21T04:39:34.167000", "content": "Will teacher better than student?
", "votes": 0, "replies": [] } ] }, { "id": 1239780, "author_name": "hengck23", "author_url": "", "post_date": "2021-03-16T02:25:41.553000", "content": "i find a new home for this method. it think it will shine there
\nhttps://www.kaggle.com/c/bms-molecular-translation
some results.
\ni use two-stage (instead of three-stage in some public code)
\n
\n
batch_size = len(batch['index'])\n df = batch['df']\n\n target = batch['target'].cuda()\n image = batch['image'].cuda()\n render = batch['render'].cuda()\n is_annotate = batch['is_annotate'].cuda()\n\n with torch.no_grad():\n teacher.eval()\n logit_t, f_t = teacher(render)\n\n #----\n net.train()\n optimizer.zero_grad()\n\n def loss_consistency(f,truth):\n f = f[is_annotate==1]\n t = truth[is_annotate==1]\n loss = F.mse_loss(f,t)\n return loss\n\n if is_mixed_precision: \n image = image.half()\n with amp.autocast():\n logit,f = net(image)\n loss0 = binary_cross_entropy_loss(logit, target)\n loss1 = loss_consistency(f, f_t)\n\n scaler.scale(loss0+loss1).backward()\n scaler.step(optimizer)\n scaler.update()\nsince the score on the \"my submissions\" page can be ranked, I can see:
\nonly one fold:
\nfor 640x640 input, tta= origina+flip:
\nresnet200d + teacher attention : local cv 0.956805, lb 0.959 (best)
\nresnet50d + teacher attention : local cv 0.956385, lb 0.959
\nresnet200d : local cv 0.955946, lb 0.959 (worst)
\n(https://www.kaggle.com/c/ranzcr-clip-catheter-line-classification/discussion/204950)
it is seen that now resnet50d is as good as resnet200d with extra supervision
", "votes": 1, "replies": [] }, { "id": 1220422, "author_name": "hengck23", "author_url": "", "post_date": "2021-02-28T01:26:45.613000", "content": "i will not be surprised that more hand-annotation will be the key to winning this challenge.
\nwe will see who can do it \"smartly\"
replacing line annotation with endpoint annotation. how is teacher distillation affected?
\nNote this is results for CVC only. the others like ETT are not used at all
be creative!
\nmulti scale annotation, find pretain model that segment lung,heart, etc …. maybe they would work?
", "votes": 2, "replies": [] }, { "id": 1220873, "author_name": "Mobassir", "author_url": "", "post_date": "2021-02-28T12:39:29.817000", "content": "@hengck23 you tried normfree nets? how does it work for you?
", "votes": 1, "replies": [] } ] }, { "id": 1151143, "author_name": "hengck23", "author_url": "", "post_date": "2021-01-13T06:53:54.077000", "content": "i would like to combine my idea with this google paper:
\nConcept Bottleneck Models
\nhttps://arxiv.org/pdf/2007.04612.pdf
\"By construction, we can intervene on these concept bottleneck models by editing their predicted concept values and propagating these changes to the final prediction\"
\ni want to intervene by sketching on the image
", "votes": 1, "replies": [] }, { "id": 1214113, "author_name": "Md. Masud Rana", "author_url": "", "post_date": "2021-02-22T15:43:44.077000", "content": "Great discussion thread. Can anyone share an example of the Teacher-Student model example kernel?
", "votes": 0, "replies": [] }, { "id": 1133438, "author_name": "AIFahim", "author_url": "", "post_date": "2020-12-31T08:08:51.367000", "content": "Sir how you ensemble with annotation without annotations can you explain? And what part of the images you annotated ? Thanks in advance.
", "votes": 0, "replies": [] }, { "id": 1121948, "author_name": "", "author_url": "", "post_date": "2020-12-22T03:40:55.427000", "content": "", "votes": 0, "replies": [] }, { "id": 1121735, "author_name": "", "author_url": "", "post_date": "2020-12-21T21:34:26.697000", "content": "", "votes": 1, "replies": [] }, { "id": 1118630, "author_name": "Saurabh Shahane", "author_url": "", "post_date": "2020-12-19T08:55:24.363000", "content": "Thanks for insights!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "1118515": "\n\nthe key idea is to modify the activation (activation of the feature map in this case) by changing the input so that the network can pick up extra supervision and decide if they want to use them or not.\n\nThere are many ways to implement this. if you want to mix those with and without annotations, you can add a flag \"is additional annotation available\" as input at training. this is probably the easiest \"human in the loop for training\"\n\nnote: if the kaggle annotation is not enough, add your own annotation, etc. part of lung, etc\n\nin my experiment, I estimated the addition annotation supervision signals will cause an improvement of about 2.0% to 2.5% for the AUC metric", "1213323": "check these as well\n\nEmbedding Human Knowledge into Deep Neural Network via Attention Map\nhttps://arxiv.org/abs/1905.03540\n\nhttps://github.com/machine-perception-robotics-group/attention_branch_network/\n\n\n", "1226219": "reference code and model is here!\nhttps://drive.google.com/drive/folders/1JRqCpV3NMGFeyd5ws6_pyXwQ5x3n8CHF\nsee folder \"2021-03-04\" and refer to the readme.pptx\n\n", "1118777": "if one wants to be fanciful (typo error: should have been x = (1+a)*x\n\n", "1221979": "> I estimated the addition annotation supervision signals will cause an improvement of about 2.0% to 2.5% for the AUC metric\n\nIs your mean your CV increase from *x* to *x+0.02*?\nThanks for sharing your ideas. These are really very impressive. \n\n\n\n\n", "1218383": "i haven't tried this yet, but it is an idea: contrastive attention for self-supervised learning\n\nthe idea is:\n- use all images (annotated and non annotated) for contrastive learning:\n attention\\_loss (image, augment\\_image) \n- then finetune with annotated images", "1136710": "Could you please share some work or paper that implements the presented concepts using TF2.0. Thanks", "1118743": "Thank you for sharing your idea! I'm not familiar with this concept. So I would like to ask some questions if you don't mind. What do you use for consistency loss? Do you train the teacher first and then distill the teacher to the student?\n", "1239780": "i find a new home for this method. it think it will shine there\nhttps://www.kaggle.com/c/bms-molecular-translation\n\n", "1217634": "some results.\n\ni use two-stage (instead of three-stage in some public code)\n\n\n", "1151143": "i would like to combine my idea with this google paper:\nConcept Bottleneck Models\nhttps://arxiv.org/pdf/2007.04612.pdf\n\n\"By construction, we can intervene on these concept bottleneck models by editing their predicted concept values and propagating these changes to the final prediction\"\n\ni want to intervene by sketching on the image", "1214113": "Great discussion thread. Can anyone share an example of the Teacher-Student model example kernel? ", "1133438": "Sir how you ensemble with annotation without annotations can you explain? And what part of the images you annotated ? Thanks in advance. ", "1121948": "", "1121735": "", "1118630": "Thanks for insights!" } }