![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F1528571%2F40053af5a0e08698729d5bdd124624ef%2Fheng.png?generation=1574229021225455&alt=media\")
https://github.com/pudae/kaggle-understanding-clouds
\n\nCongrats to all the winners and survivors of the shake-up.\nThanks to Kaggle and the hosting team for the interesting competition.
\n\nExcept for some tricks, improvements almost have been made by using ensemble. So, in this post, I will briefly describe the track of scores in the last week. The details will be shared as codes.
\n\nTypes of networks\n- Model A: UNet with classification head\n- Model B: FPN or UNet, no classification head
\n\nBackbones\n- resnet34, efficientnet-b1, resnext101_32x8d_wsl, resnext101_32x16d_wsl
\n\nDataSet\n- split: train vs val = 9 vs 2\n- Model A: All labels\n- Model B: non-empty labels
\n\nLoss\n- classification part: BCE\n- segmentation part: BCE * 0.75 + DICE * 0.25
\n\nOptimizer\n- AdamW, weight decay 0.01\n- encoder learning rate 0.000025\n- decoder learning rate 0.00025\n- OneCycle scheduler, shallow models 30 epochs, deep models 15 epochs
\n\nAugmentation\n- Common: hflip, vflip, shift/scale/rotate, grid distortion, channel shuffle, invert, to gray\n- Model A: random crop, size 384\n- Model B: full-size, size 384, 544, 576, 768
\n\ntrain single model\nAt first, I’d tried to train a good single network. I’d struggled to improve and stabilize the LB scores for 2 weeks, but I’d failed. \n- TTA3: CV 0.6517 / Public LB 0..66951 / Private LB 0.65828
\n\nadd segmentation models\nI thought the reason for the unstable LB score was because of poor segmentation performance. If we can have a more powerful segmentation model, the effect of poor classification performance can be reduced.
\n\nSo, I began trying to train good segmentation only model. Because I could filter out negative predictions using the classification model, only positive labels were needed to train.
\n\nFrom this time, CV and LB were correlated well.\nI trained several segmentation models with different backbone, image size, etc.\n- TTA4, 1 seg with cls + 1 seg: CV 0.6560, Public LB 0.67395, Private LB 0.66495\n- TTA4, 1 seg with cls + 3 seg: CV 0.6582, Public LB 0.67482, Private LB 0.66501\n- TTA4, 1 seg with cls + 4 seg: CV 0.6587, Public LB 0.67551, Private LB 0.66604\n- TTA4, 1 seg with cls + 7 seg: CV 0.6594, Public LB 0.67596, Private LB 0.66663
\n\nadd more models with classification head\nNow, the segmentation part became enough good. so, I added two more models with classification head.\n- TTA4, 3 seg with cls + 7 seg: CV 0.6625, Public LB 0.67678, Private LB 0.66746
\n\nuse segmentation models as a classifier\nTo take advantage of the performance of the segmentation models, I used a mean of top K pixel probabilities as a classification probability. \n\ncls_probabilities = np.sort(mask_probabilities.reshape(4, -1), axis=1)\ncls_probabilities = np.mean(cls_probabilities[:,-17500:], axis=1)\n
use max probability as a positive prediction\nAll images in the train set have at least one type of cloud, so I treated the label of max probability in each image as a positive prediction. \n\ncls_probabilities[np.argmax(cls_probabilities)] = 1\n\n- TTA4, 3 seg with cls + 8 seg: CV 0.6640, Public LB 0.68031, Private LB 0.67170
use exponential moving average\nFinally, I changed the averaging weights method to the exponential moving average. Before that, the average of the last 5 weights was used.\n- TTA4, 3 seg with cls + 8 seg: CV 0.6636, Public LB 0.68130, Private LB 0.67126\n- TTA4, 3 seg with cls + 9 seg: CV 0.6637, Public LB 0.68185, Private LB 0.67175 (Final Submission)
", "messages": [ { "id": 676764, "postDate": "2019-11-19T13:41:18.153Z", "content": "https://github.com/pudae/kaggle-understanding-clouds
\n\nCongrats to all the winners and survivors of the shake-up.\nThanks to Kaggle and the hosting team for the interesting competition.
\n\nExcept for some tricks, improvements almost have been made by using ensemble. So, in this post, I will briefly describe the track of scores in the last week. The details will be shared as codes.
\n\nTypes of networks\n- Model A: UNet with classification head\n- Model B: FPN or UNet, no classification head
\n\nBackbones\n- resnet34, efficientnet-b1, resnext101_32x8d_wsl, resnext101_32x16d_wsl
\n\nDataSet\n- split: train vs val = 9 vs 2\n- Model A: All labels\n- Model B: non-empty labels
\n\nLoss\n- classification part: BCE\n- segmentation part: BCE * 0.75 + DICE * 0.25
\n\nOptimizer\n- AdamW, weight decay 0.01\n- encoder learning rate 0.000025\n- decoder learning rate 0.00025\n- OneCycle scheduler, shallow models 30 epochs, deep models 15 epochs
\n\nAugmentation\n- Common: hflip, vflip, shift/scale/rotate, grid distortion, channel shuffle, invert, to gray\n- Model A: random crop, size 384\n- Model B: full-size, size 384, 544, 576, 768
\n\ntrain single model\nAt first, I’d tried to train a good single network. I’d struggled to improve and stabilize the LB scores for 2 weeks, but I’d failed. \n- TTA3: CV 0.6517 / Public LB 0..66951 / Private LB 0.65828
\n\nadd segmentation models\nI thought the reason for the unstable LB score was because of poor segmentation performance. If we can have a more powerful segmentation model, the effect of poor classification performance can be reduced.
\n\nSo, I began trying to train good segmentation only model. Because I could filter out negative predictions using the classification model, only positive labels were needed to train.
\n\nFrom this time, CV and LB were correlated well.\nI trained several segmentation models with different backbone, image size, etc.\n- TTA4, 1 seg with cls + 1 seg: CV 0.6560, Public LB 0.67395, Private LB 0.66495\n- TTA4, 1 seg with cls + 3 seg: CV 0.6582, Public LB 0.67482, Private LB 0.66501\n- TTA4, 1 seg with cls + 4 seg: CV 0.6587, Public LB 0.67551, Private LB 0.66604\n- TTA4, 1 seg with cls + 7 seg: CV 0.6594, Public LB 0.67596, Private LB 0.66663
\n\nadd more models with classification head\nNow, the segmentation part became enough good. so, I added two more models with classification head.\n- TTA4, 3 seg with cls + 7 seg: CV 0.6625, Public LB 0.67678, Private LB 0.66746
\n\nuse segmentation models as a classifier\nTo take advantage of the performance of the segmentation models, I used a mean of top K pixel probabilities as a classification probability. \n\ncls_probabilities = np.sort(mask_probabilities.reshape(4, -1), axis=1)\ncls_probabilities = np.mean(cls_probabilities[:,-17500:], axis=1)\n
use max probability as a positive prediction\nAll images in the train set have at least one type of cloud, so I treated the label of max probability in each image as a positive prediction. \n\ncls_probabilities[np.argmax(cls_probabilities)] = 1\n\n- TTA4, 3 seg with cls + 8 seg: CV 0.6640, Public LB 0.68031, Private LB 0.67170
use exponential moving average\nFinally, I changed the averaging weights method to the exponential moving average. Before that, the average of the last 5 weights was used.\n- TTA4, 3 seg with cls + 8 seg: CV 0.6636, Public LB 0.68130, Private LB 0.67126\n- TTA4, 3 seg with cls + 9 seg: CV 0.6637, Public LB 0.68185, Private LB 0.67175 (Final Submission)
", "rawMarkdown": "## UPDATE: code available on github\nhttps://github.com/pudae/kaggle-understanding-clouds\n\n---\n\nCongrats to all the winners and survivors of the shake-up.\nThanks to Kaggle and the hosting team for the interesting competition.\n\nExcept for some tricks, improvements almost have been made by using ensemble. So, in this post, I will briefly describe the track of scores in the last week. The details will be shared as codes.\n\n### Common Settings\n**Types of networks**\n- Model A: UNet with classification head\n- Model B: FPN or UNet, no classification head\n\n**Backbones**\n- resnet34, efficientnet-b1, resnext101_32x8d_wsl, resnext101_32x16d_wsl\n\n**DataSet**\n- split: train vs val = 9 vs 2\n- Model A: All labels\n- Model B: non-empty labels\n\n**Loss**\n- classification part: BCE\n- segmentation part: BCE * 0.75 + DICE * 0.25\n\n**Optimizer**\n- AdamW, weight decay 0.01\n- encoder learning rate 0.000025\n- decoder learning rate 0.00025\n- OneCycle scheduler, shallow models 30 epochs, deep models 15 epochs\n\n**Augmentation**\n- Common: hflip, vflip, shift/scale/rotate, grid distortion, channel shuffle, invert, to gray\n- Model A: random crop, size 384\n- Model B: full-size, size 384, 544, 576, 768\n\n### The track of scores\n**train single model**\nAt first, I’d tried to train a good single network. I’d struggled to improve and stabilize the LB scores for 2 weeks, but I’d failed. \n- TTA3: CV 0.6517 / Public LB 0..66951 / Private LB 0.65828\n\n**add segmentation models**\nI thought the reason for the unstable LB score was because of poor segmentation performance. If we can have a more powerful segmentation model, the effect of poor classification performance can be reduced.\n\nSo, I began trying to train good segmentation only model. Because I could filter out negative predictions using the classification model, only positive labels were needed to train.\n\nFrom this time, CV and LB were correlated well.\nI trained several segmentation models with different backbone, image size, etc.\n- TTA4, 1 seg with cls + 1 seg: CV 0.6560, Public LB 0.67395, Private LB 0.66495\n- TTA4, 1 seg with cls + 3 seg: CV 0.6582, Public LB 0.67482, Private LB 0.66501\n- TTA4, 1 seg with cls + 4 seg: CV 0.6587, Public LB 0.67551, Private LB 0.66604\n- TTA4, 1 seg with cls + 7 seg: CV 0.6594, Public LB 0.67596, Private LB 0.66663\n\n**add more models with classification head**\nNow, the segmentation part became enough good. so, I added two more models with classification head.\n- TTA4, 3 seg with cls + 7 seg: CV 0.6625, Public LB 0.67678, Private LB 0.66746\n\n**use segmentation models as a classifier**\nTo take advantage of the performance of the segmentation models, I used a mean of top K pixel probabilities as a classification probability. \n```\ncls_probabilities = np.sort(mask_probabilities.reshape(4, -1), axis=1)\ncls_probabilities = np.mean(cls_probabilities[:,-17500:], axis=1)\n```\n\n- TTA4, 3 seg with cls + 7 seg: 0.6629, 0.67822, 0.67046\n- TTA4, 3 seg with cls + 8 seg: 0.6635, 0.67906, 0.67117\n\n**use max probability as a positive prediction**\nAll images in the train set have at least one type of cloud, so I treated the label of max probability in each image as a positive prediction. \n```\ncls_probabilities[np.argmax(cls_probabilities)] = 1\n```\n- TTA4, 3 seg with cls + 8 seg: CV 0.6640, Public LB 0.68031, Private LB 0.67170\n\n**use exponential moving average**\nFinally, I changed the averaging weights method to the exponential moving average. Before that, the average of the last 5 weights was used.\n- TTA4, 3 seg with cls + 8 seg: CV 0.6636, Public LB 0.68130, Private LB 0.67126\n- TTA4, 3 seg with cls + 9 seg: CV 0.6637, Public LB 0.68185, Private LB 0.67175 (**Final Submission**)\n", "votes": 160 }, { "id": 677390, "postDate": "2019-11-20T05:09:54.620Z", "content": "Congratulations for winning this competition and thanks for sharing your solution.
\nI have one question:
\nWhat is classification head in Model A(UNet)?
maybe something like this?\n
@bibek777 is right!! :)
", "rawMarkdown": "@bibek777 is right!! :)", "votes": 2 } ] }, { "id": 676956, "postDate": "2019-11-19T16:45:53.897Z", "content": "Congratulations, great job. Using segmentation models as classifiers is smart. In my experiments, I found that segmentation models could predict empty masks more accurately than a classification model trained on labels only. Doing \"use max probability as a positive prediction\" is a great trick.
\n\nHow did you train on non-empty masks only? Since one image may be positive for Fish, Flower and negative for Gravel and Sugar, when the model sees that image isn't it training on the empty Gravel and Sugar? Do you adjust the loss function to avoid using those?
", "rawMarkdown": "Congratulations, great job. Using segmentation models as classifiers is smart. In my experiments, I found that segmentation models could predict empty masks more accurately than a classification model trained on labels only. Doing \"use max probability as a positive prediction\" is a great trick.\n \nHow did you train on non-empty masks only? Since one image may be positive for Fish, Flower and negative for Gravel and Sugar, when the model sees that image isn't it training on the empty Gravel and Sugar? Do you adjust the loss function to avoid using those?", "votes": 5, "replies": [ { "id": 677364, "postDate": "2019-11-20T04:29:38.133Z", "content": "I simply multiplied mask logits by class labels.\n\ncls_labels = labels.view(B,C,-1)\ncls_labels = torch.sum(cls_labels, dim=2, keepdims=True)\ncls_labels = (cls_labels > 0).float()\nloss = loss_fn(input=logits.view(B,C,-1)*cls_labels , target=labels.view(B,C,-1)) \n
That's really great. Thanks for sharing.
", "rawMarkdown": "That's really great. Thanks for sharing.", "votes": 1 }, { "id": 676896, "postDate": "2019-11-19T15:29:45.323Z", "content": "Congratulations on your result and thanks for the report.
\n\nI have a question about this line:\nOneCycle scheduler, shallow models 30 epochs, deep models 15 epochs\nShouldn't we train deep models with more epochs and lower LR to avoid abrupt changes?
In Self-training with Noisy Student improves ImageNet classification, the authors train deeper models for less epochs than smaller ones.
\n\n\n", "rawMarkdown": "In Self-training with Noisy Student improves ImageNet classification, the authors train deeper models for less epochs than smaller ones.\n\n> Specifically, we train the student model for 350 epochs for models larger than EfficientNet-B4, including EfficientNet-L0, L1 and L2 and train the student model for 700 epochs for smaller models.", "votes": 1 }, { "id": 677255, "postDate": "2019-11-20T00:59:31.930Z", "content": "Specifically, we train the student model for 350 epochs for models larger than EfficientNet-B4, including EfficientNet-L0, L1 and L2 and train the student model for 700 epochs for smaller models.
\n
Interesting to see this also on research, @sidhanthholalkere on the paper is there an explanation why this works?
", "rawMarkdown": "Interesting to see this also on research, @sidhanthholalkere on the paper is there an explanation why this works?" }, { "id": 677376, "postDate": "2019-11-20T04:42:21.727Z", "content": "In my case, deeper models were overfitted easily if I trained more than 15 epochs.\nSo, I'd scheduled the learning rate to train models within around 15 epochs.
", "rawMarkdown": "In my case, deeper models were overfitted easily if I trained more than 15 epochs.\nSo, I'd scheduled the learning rate to train models within around 15 epochs.", "votes": 2 }, { "id": 677697, "postDate": "2019-11-20T14:03:51.380Z", "content": "Nice, thanks @pudae81
", "rawMarkdown": "Nice, thanks @pudae81 " } ] }, { "id": 677006, "postDate": "2019-11-19T17:50:25.360Z", "content": "thanks for the nice work!
\n\n\"use max probability as a positive prediction\nAll images in the train set have at least one type of cloud, so I treated the label of max probability in each image as a positive prediction.\"
\n\ni note that in the training, we do not actually force this condition to be true in training. \ni wonder if the following will give better results?
\n\n```\n... in training iteration ...\nprob_mask = net(intput)\nmax_pixel_prob = ....
\n\nmask_loss = loss_mask(prob_mask, truth_mask)\nlabel_loss = loss_label(max_pixel_prob, truth_label)
\n\n... backpropagte mask_loss+label_loss ...\n```
", "rawMarkdown": "thanks for the nice work!\n\n\"use max probability as a positive prediction\nAll images in the train set have at least one type of cloud, so I treated the label of max probability in each image as a positive prediction.\"\n\ni note that in the training, we do not actually force this condition to be true in training. \ni wonder if the following will give better results?\n\n```\n... in training iteration ...\nprob_mask = net(intput)\nmax_pixel_prob = ....\n\nmask_loss = loss_mask(prob_mask, truth_mask)\nlabel_loss = loss_label(max_pixel_prob, truth_label)\n\n ... backpropagte mask_loss+label_loss ...\n```", "votes": 2, "replies": [ { "id": 677018, "postDate": "2019-11-19T18:00:06.753Z", "content": "can i also confirm that you are using max pixel over all four prediction class mask?
\n\ne.g
\n\ninput = [batch_size,C,H,W]\npredict = [batch_size,4,H,W]
\n\nmax_class = convert_to_class_index( predict.reshape(batch_size, -1).argmax(-1) )
\n\nmax_class must be 1 seen we are told at least one label
", "rawMarkdown": "can i also confirm that you are using max pixel over all four prediction class mask?\n\ne.g\n\ninput = [batch\\_size,C,H,W]\npredict = [batch\\_size,4,H,W]\n\nmax_class = convert\\_to\\_class\\_index( predict.reshape(batch\\_size, -1).argmax(-1) )\n\nmax_class must be 1 seen we are told at least one label" }, { "id": 677359, "postDate": "2019-11-20T04:20:39.443Z", "content": "The above code snippet is for a single image.\nFor batch version will be ...\n\ncls_probabilities = np.sort(mask_probabilities.reshape(B, 4, -1), axis=-1)\ncls_probabilities = np.mean(cls_probabilities[:,:,-17500:], axis=-1)\n
Congratulations for being the solo winner, and thanks for sharing!
", "rawMarkdown": "Congratulations for being the solo winner, and thanks for sharing!", "replies": [ { "id": 677368, "postDate": "2019-11-20T04:31:11.847Z", "content": "Thanks for your congratulations! 😃
", "rawMarkdown": "Thanks for your congratulations! 😃 " } ] }, { "id": 1127935, "postDate": "2020-12-27T03:57:57.160Z", "content": "Excuse me. I can't understand \"I could filter out negative predictions using the classification model\".
\nI want to know the struct of the classification model and how to train and use the classification model.
\nThank you!
", "rawMarkdown": "Excuse me. I can't understand \"I could filter out negative predictions using the classification model\".\n\nI want to know the struct of the classification model and how to train and use the classification model. \n\nThank you!" }, { "id": 964473, "postDate": "2020-08-09T23:09:34.977Z", "content": "Hi. I've been looking through your solution's GitHub code to learn from it. It seems like you write from tensorboardX import SummaryWriter in your kvt\\apis but never create a SummaryWriter object. Why? Additionally you use sacred but don't use any functions that log training metrics like ex.log_scalar. How are you monitoring training?
\nencoder learning rate 0.000025\ndecoder learning rate 0.00025\n
What is the reason for this setting? and Are there any learning materials that can be used for reference
", "rawMarkdown": "```\nencoder learning rate 0.000025\ndecoder learning rate 0.00025\n```\n\nWhat is the reason for this setting? and Are there any learning materials that can be used for reference" }, { "id": 753327, "postDate": "2020-02-22T03:27:41.750Z", "content": "Congratulations.
", "rawMarkdown": "Congratulations. " }, { "id": 698679, "postDate": "2019-12-19T15:06:20.263Z", "content": "Hey there,
\n\nFirst of all, congratulations on winning this competition! We have a school assignment for which we need to find a dataset and use different classification measures on it. Our intention is to use your solution and use the output of the last hidden layer of the network as input for our classification measures. We were wondering if you could help us explain how to get this output as input.
\n\nKind regards,\nJoost
", "rawMarkdown": "Hey there,\n\nFirst of all, congratulations on winning this competition! We have a school assignment for which we need to find a dataset and use different classification measures on it. Our intention is to use your solution and use the output of the last hidden layer of the network as input for our classification measures. We were wondering if you could help us explain how to get this output as input.\n\nKind regards,\nJoost" }, { "id": 696909, "postDate": "2019-12-17T08:13:10.370Z", "content": "how can I export pth to model format, etc .pb.
", "rawMarkdown": "how can I export pth to model format, etc .pb. " }, { "id": 678849, "postDate": "2019-11-22T01:11:03.593Z", "content": "\n\n\nCode for 1st place solution in Kaggle Humpback Whale Identification Challange.
\n
😄 😄
", "rawMarkdown": "> Code for 1st place solution in Kaggle Humpback Whale Identification Challange.\n\n😄 😄 ", "replies": [ { "id": 678986, "postDate": "2019-11-22T06:03:52.857Z", "content": "😨 😭 😏 😁 👍
", "rawMarkdown": "😨 😭 😏 😁 👍 " } ] }, { "id": 677756, "postDate": "2019-11-20T14:56:48.697Z", "content": "Its great work! Congratulations :)
", "rawMarkdown": "Its great work! Congratulations :)", "replies": [ { "id": 678854, "postDate": "2019-11-22T01:17:02.937Z", "content": "Thanks~!
", "rawMarkdown": "Thanks~!" } ] }, { "id": 677611, "postDate": "2019-11-20T11:46:43.083Z", "content": "Congratulations! Thank you from a beginner for sharing :)
", "rawMarkdown": "Congratulations! Thank you from a beginner for sharing :) ", "replies": [ { "id": 678852, "postDate": "2019-11-22T01:16:48.227Z", "content": "Thanks~
", "rawMarkdown": "Thanks~" } ] }, { "id": 677592, "postDate": "2019-11-20T11:19:16.543Z", "content": "Congratulations!!!
", "rawMarkdown": "Congratulations!!!", "replies": [ { "id": 678851, "postDate": "2019-11-22T01:16:18.277Z", "content": "Thanks!!
", "rawMarkdown": "Thanks!! " } ] }, { "id": 677414, "postDate": "2019-11-20T05:41:36.410Z", "content": "Congratulations! How do you implement exponential moving average?
", "rawMarkdown": "Congratulations! How do you implement exponential moving average? ", "replies": [ { "id": 677417, "postDate": "2019-11-20T05:46:22.187Z", "content": "\n\n\nHow do you implement exponential moving average?
\n
Wait for his code....he plans to release it 😄
", "rawMarkdown": "> How do you implement exponential moving average?\n\nWait for his code....he plans to release it 😄 ", "votes": -1 }, { "id": 678850, "postDate": "2019-11-22T01:16:04.563Z", "content": "Thanks for your congratulations. I released my codes for this competition. I hope it will be helpful for you. :)
", "rawMarkdown": "Thanks for your congratulations. I released my codes for this competition. I hope it will be helpful for you. :)" }, { "id": 680980, "postDate": "2019-11-25T13:31:43.223Z", "content": "Thanks for your released code, in your code ,what mean of kvt, it's a open-sourse library or just achieve by yourself
", "rawMarkdown": "Thanks for your released code, in your code ,what mean of kvt, it's a open-sourse library or just achieve by yourself" }, { "id": 686564, "postDate": "2019-12-03T09:47:29.827Z", "content": "Congrats on the result! I am also curious about kvt, is it your own high level framework surch as fast.ai or was it built for this competition ? Do you have futur plans about it ? Thanks !
", "rawMarkdown": "Congrats on the result! I am also curious about kvt, is it your own high level framework surch as fast.ai or was it built for this competition ? Do you have futur plans about it ? Thanks !" }, { "id": 687108, "postDate": "2019-12-03T23:43:27.930Z", "content": "It's just my trying to separate common codes from competition codes.\nCurrently, I have no plan. I'll just use it in the next competitions with continous improvement.
", "rawMarkdown": "It's just my trying to separate common codes from competition codes.\nCurrently, I have no plan. I'll just use it in the next competitions with continous improvement.\n", "votes": 1 } ] }, { "id": 677311, "postDate": "2019-11-20T03:14:08.347Z", "content": "Congratulations and Thank you for sharing !!
", "rawMarkdown": "Congratulations and Thank you for sharing !!", "replies": [ { "id": 677366, "postDate": "2019-11-20T04:31:01.493Z", "content": "Thanks for your congratulations! 😃
", "rawMarkdown": "Thanks for your congratulations! 😃 " } ] }, { "id": 677262, "postDate": "2019-11-20T01:19:45.600Z", "content": "Congratulations on your result.
", "rawMarkdown": "Congratulations on your result.\n\n", "replies": [ { "id": 677367, "postDate": "2019-11-20T04:31:06.797Z", "content": "Thanks for your congratulations! 😃
", "rawMarkdown": "Thanks for your congratulations! 😃 ", "votes": 1 } ] }, { "id": 677202, "postDate": "2019-11-19T23:13:21.560Z", "content": "Congrats! Thanks for sharing.
", "rawMarkdown": "Congrats! Thanks for sharing.", "replies": [ { "id": 677369, "postDate": "2019-11-20T04:31:18.753Z", "content": "Thanks for your congratulations!
", "rawMarkdown": "Thanks for your congratulations!" } ] }, { "id": 677200, "postDate": "2019-11-19T23:10:44.673Z", "content": "Congratulations on winning the first place!
", "rawMarkdown": "Congratulations on winning the first place!", "replies": [ { "id": 677370, "postDate": "2019-11-20T04:31:24.960Z", "content": "Thanks for your congratulations! 😃
", "rawMarkdown": "Thanks for your congratulations! 😃 " } ] }, { "id": 677180, "postDate": "2019-11-19T22:22:27.637Z", "content": "Congrats! Thank you for sharing.
", "rawMarkdown": "Congrats! Thank you for sharing.", "replies": [ { "id": 677371, "postDate": "2019-11-20T04:31:29.067Z", "content": "Thanks for your congratulations! 😃
", "rawMarkdown": "Thanks for your congratulations! 😃 " } ] }, { "id": 676880, "postDate": "2019-11-19T15:11:22.600Z", "content": "Congratulations, thanks for sharing, impressive solution
", "rawMarkdown": "Congratulations, thanks for sharing, impressive solution", "replies": [ { "id": 677377, "postDate": "2019-11-20T04:42:35.273Z", "content": "Thanks~ :)
", "rawMarkdown": "Thanks~ :)" } ] }, { "id": 676867, "postDate": "2019-11-19T15:01:32.060Z", "content": "Congratulations!! There are so many things to learn. I can't wait to see your code!
", "rawMarkdown": "Congratulations!! There are so many things to learn. I can't wait to see your code!", "replies": [ { "id": 677378, "postDate": "2019-11-20T04:43:06.317Z", "content": "Thanks for your congrats!
", "rawMarkdown": "Thanks for your congrats!" } ] }, { "id": 676802, "postDate": "2019-11-19T14:11:34.910Z", "content": "congratulations.\nthe trick of exponential moving average is impressived.
", "rawMarkdown": "congratulations.\nthe trick of exponential moving average is impressived.", "replies": [ { "id": 677380, "postDate": "2019-11-20T04:43:16.770Z", "content": "Thanks for your congrats!
", "rawMarkdown": "Thanks for your congrats!" } ] }, { "id": 676789, "postDate": "2019-11-19T13:58:22.717Z", "content": "Congrats and Thank you for sharing
", "rawMarkdown": "Congrats and Thank you for sharing", "replies": [ { "id": 677381, "postDate": "2019-11-20T04:43:19.277Z", "content": "Thanks for your congrats!
", "rawMarkdown": "Thanks for your congrats!" } ] }, { "id": 689509, "postDate": "2019-12-07T01:45:48.100Z", "rawMarkdown": "", "isDeleted": true }, { "id": 676846, "postDate": "2019-11-19T14:44:34.897Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 677379, "postDate": "2019-11-20T04:43:10.913Z", "content": "Thanks for your congrats!
", "rawMarkdown": "Thanks for your congrats!" } ] }, { "id": 690436, "postDate": "2019-12-08T15:30:34.353Z", "content": "Great work! Thanks for sharing!
", "rawMarkdown": "Great work! Thanks for sharing!" } ], "comments": [ { "id": 677390, "author_name": "Raghawendra Singh", "author_url": "", "post_date": "2019-11-20T05:09:54.620000", "content": "Congratulations for winning this competition and thanks for sharing your solution.
\nI have one question:
\nWhat is classification head in Model A(UNet)?
maybe something like this?\n
@bibek777 is right!! :)
", "votes": 2, "replies": [] } ] }, { "id": 676956, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-11-19T16:45:53.897000", "content": "Congratulations, great job. Using segmentation models as classifiers is smart. In my experiments, I found that segmentation models could predict empty masks more accurately than a classification model trained on labels only. Doing \"use max probability as a positive prediction\" is a great trick.
\n\nHow did you train on non-empty masks only? Since one image may be positive for Fish, Flower and negative for Gravel and Sugar, when the model sees that image isn't it training on the empty Gravel and Sugar? Do you adjust the loss function to avoid using those?
", "votes": 5, "replies": [ { "id": 677364, "author_name": "pudae", "author_url": "", "post_date": "2019-11-20T04:29:38.133000", "content": "I simply multiplied mask logits by class labels.\n\ncls_labels = labels.view(B,C,-1)\ncls_labels = torch.sum(cls_labels, dim=2, keepdims=True)\ncls_labels = (cls_labels > 0).float()\nloss = loss_fn(input=logits.view(B,C,-1)*cls_labels , target=labels.view(B,C,-1)) \n
That's really great. Thanks for sharing.
", "votes": 1, "replies": [] }, { "id": 676896, "author_name": "DimitreOliveira", "author_url": "", "post_date": "2019-11-19T15:29:45.323000", "content": "Congratulations on your result and thanks for the report.
\n\nI have a question about this line:\nOneCycle scheduler, shallow models 30 epochs, deep models 15 epochs\nShouldn't we train deep models with more epochs and lower LR to avoid abrupt changes?
In Self-training with Noisy Student improves ImageNet classification, the authors train deeper models for less epochs than smaller ones.
\n\n\n", "votes": 1, "replies": [] }, { "id": 677255, "author_name": "DimitreOliveira", "author_url": "", "post_date": "2019-11-20T00:59:31.930000", "content": "Specifically, we train the student model for 350 epochs for models larger than EfficientNet-B4, including EfficientNet-L0, L1 and L2 and train the student model for 700 epochs for smaller models.
\n
Interesting to see this also on research, @sidhanthholalkere on the paper is there an explanation why this works?
", "votes": 0, "replies": [] }, { "id": 677376, "author_name": "pudae", "author_url": "", "post_date": "2019-11-20T04:42:21.727000", "content": "In my case, deeper models were overfitted easily if I trained more than 15 epochs.\nSo, I'd scheduled the learning rate to train models within around 15 epochs.
", "votes": 2, "replies": [] }, { "id": 677697, "author_name": "DimitreOliveira", "author_url": "", "post_date": "2019-11-20T14:03:51.380000", "content": "Nice, thanks @pudae81
", "votes": 0, "replies": [] } ] }, { "id": 677006, "author_name": "hengck23", "author_url": "", "post_date": "2019-11-19T17:50:25.360000", "content": "thanks for the nice work!
\n\n\"use max probability as a positive prediction\nAll images in the train set have at least one type of cloud, so I treated the label of max probability in each image as a positive prediction.\"
\n\ni note that in the training, we do not actually force this condition to be true in training. \ni wonder if the following will give better results?
\n\n```\n... in training iteration ...\nprob_mask = net(intput)\nmax_pixel_prob = ....
\n\nmask_loss = loss_mask(prob_mask, truth_mask)\nlabel_loss = loss_label(max_pixel_prob, truth_label)
\n\n... backpropagte mask_loss+label_loss ...\n```
", "votes": 2, "replies": [ { "id": 677018, "author_name": "hengck23", "author_url": "", "post_date": "2019-11-19T18:00:06.753000", "content": "can i also confirm that you are using max pixel over all four prediction class mask?
\n\ne.g
\n\ninput = [batch_size,C,H,W]\npredict = [batch_size,4,H,W]
\n\nmax_class = convert_to_class_index( predict.reshape(batch_size, -1).argmax(-1) )
\n\nmax_class must be 1 seen we are told at least one label
", "votes": 0, "replies": [] }, { "id": 677359, "author_name": "pudae", "author_url": "", "post_date": "2019-11-20T04:20:39.443000", "content": "The above code snippet is for a single image.\nFor batch version will be ...\n\ncls_probabilities = np.sort(mask_probabilities.reshape(B, 4, -1), axis=-1)\ncls_probabilities = np.mean(cls_probabilities[:,:,-17500:], axis=-1)\n
Congratulations for being the solo winner, and thanks for sharing!
", "votes": 0, "replies": [ { "id": 677368, "author_name": "pudae", "author_url": "", "post_date": "2019-11-20T04:31:11.847000", "content": "Thanks for your congratulations! 😃
", "votes": 0, "replies": [] } ] }, { "id": 1127935, "author_name": "lifengnan", "author_url": "", "post_date": "2020-12-27T03:57:57.160000", "content": "Excuse me. I can't understand \"I could filter out negative predictions using the classification model\".
\nI want to know the struct of the classification model and how to train and use the classification model.
\nThank you!
", "votes": 0, "replies": [] }, { "id": 964473, "author_name": "Sam Lin", "author_url": "", "post_date": "2020-08-09T23:09:34.977000", "content": "Hi. I've been looking through your solution's GitHub code to learn from it. It seems like you write from tensorboardX import SummaryWriter in your kvt\\apis but never create a SummaryWriter object. Why? Additionally you use sacred but don't use any functions that log training metrics like ex.log_scalar. How are you monitoring training?
\nencoder learning rate 0.000025\ndecoder learning rate 0.00025\n
What is the reason for this setting? and Are there any learning materials that can be used for reference
", "votes": 0, "replies": [] }, { "id": 753327, "author_name": "Sedat", "author_url": "", "post_date": "2020-02-22T03:27:41.750000", "content": "Congratulations.
", "votes": 0, "replies": [] }, { "id": 698679, "author_name": "Joost Vossers", "author_url": "", "post_date": "2019-12-19T15:06:20.263000", "content": "Hey there,
\n\nFirst of all, congratulations on winning this competition! We have a school assignment for which we need to find a dataset and use different classification measures on it. Our intention is to use your solution and use the output of the last hidden layer of the network as input for our classification measures. We were wondering if you could help us explain how to get this output as input.
\n\nKind regards,\nJoost
", "votes": 0, "replies": [] }, { "id": 696909, "author_name": "luffysup", "author_url": "", "post_date": "2019-12-17T08:13:10.370000", "content": "how can I export pth to model format, etc .pb.
", "votes": 0, "replies": [] }, { "id": 678849, "author_name": "Bibek", "author_url": "", "post_date": "2019-11-22T01:11:03.593000", "content": "\n\n\nCode for 1st place solution in Kaggle Humpback Whale Identification Challange.
\n
😄 😄
", "votes": 0, "replies": [ { "id": 678986, "author_name": "pudae", "author_url": "", "post_date": "2019-11-22T06:03:52.857000", "content": "😨 😭 😏 😁 👍
", "votes": 0, "replies": [] } ] }, { "id": 677756, "author_name": "Sri Harish", "author_url": "", "post_date": "2019-11-20T14:56:48.697000", "content": "Its great work! Congratulations :)
", "votes": 0, "replies": [ { "id": 678854, "author_name": "pudae", "author_url": "", "post_date": "2019-11-22T01:17:02.937000", "content": "Thanks~!
", "votes": 0, "replies": [] } ] }, { "id": 677611, "author_name": "Ivan Mikhnenkov", "author_url": "", "post_date": "2019-11-20T11:46:43.083000", "content": "Congratulations! Thank you from a beginner for sharing :)
", "votes": 0, "replies": [ { "id": 678852, "author_name": "pudae", "author_url": "", "post_date": "2019-11-22T01:16:48.227000", "content": "Thanks~
", "votes": 0, "replies": [] } ] }, { "id": 677592, "author_name": "Rahul Misal", "author_url": "", "post_date": "2019-11-20T11:19:16.543000", "content": "Congratulations!!!
", "votes": 0, "replies": [ { "id": 678851, "author_name": "pudae", "author_url": "", "post_date": "2019-11-22T01:16:18.277000", "content": "Thanks!!
", "votes": 0, "replies": [] } ] }, { "id": 677414, "author_name": "Morphy", "author_url": "", "post_date": "2019-11-20T05:41:36.410000", "content": "Congratulations! How do you implement exponential moving average?
", "votes": 0, "replies": [ { "id": 677417, "author_name": "Bibek", "author_url": "", "post_date": "2019-11-20T05:46:22.187000", "content": "\n\n\nHow do you implement exponential moving average?
\n
Wait for his code....he plans to release it 😄
", "votes": -1, "replies": [] }, { "id": 678850, "author_name": "pudae", "author_url": "", "post_date": "2019-11-22T01:16:04.563000", "content": "Thanks for your congratulations. I released my codes for this competition. I hope it will be helpful for you. :)
", "votes": 0, "replies": [] }, { "id": 680980, "author_name": "kevin", "author_url": "", "post_date": "2019-11-25T13:31:43.223000", "content": "Thanks for your released code, in your code ,what mean of kvt, it's a open-sourse library or just achieve by yourself
", "votes": 0, "replies": [] }, { "id": 686564, "author_name": "fl2o", "author_url": "", "post_date": "2019-12-03T09:47:29.827000", "content": "Congrats on the result! I am also curious about kvt, is it your own high level framework surch as fast.ai or was it built for this competition ? Do you have futur plans about it ? Thanks !
", "votes": 0, "replies": [] }, { "id": 687108, "author_name": "pudae", "author_url": "", "post_date": "2019-12-03T23:43:27.930000", "content": "It's just my trying to separate common codes from competition codes.\nCurrently, I have no plan. I'll just use it in the next competitions with continous improvement.
", "votes": 1, "replies": [] } ] }, { "id": 677311, "author_name": "AKLDF", "author_url": "", "post_date": "2019-11-20T03:14:08.347000", "content": "Congratulations and Thank you for sharing !!
", "votes": 0, "replies": [ { "id": 677366, "author_name": "pudae", "author_url": "", "post_date": "2019-11-20T04:31:01.493000", "content": "Thanks for your congratulations! 😃
", "votes": 0, "replies": [] } ] }, { "id": 677262, "author_name": "sabari nathan", "author_url": "", "post_date": "2019-11-20T01:19:45.600000", "content": "Congratulations on your result.
", "votes": 0, "replies": [ { "id": 677367, "author_name": "pudae", "author_url": "", "post_date": "2019-11-20T04:31:06.797000", "content": "Thanks for your congratulations! 😃
", "votes": 1, "replies": [] } ] }, { "id": 677202, "author_name": "corochann", "author_url": "", "post_date": "2019-11-19T23:13:21.560000", "content": "Congrats! Thanks for sharing.
", "votes": 0, "replies": [ { "id": 677369, "author_name": "pudae", "author_url": "", "post_date": "2019-11-20T04:31:18.753000", "content": "Thanks for your congratulations!
", "votes": 0, "replies": [] } ] }, { "id": 677200, "author_name": "Taemyung Heo", "author_url": "", "post_date": "2019-11-19T23:10:44.673000", "content": "Congratulations on winning the first place!
", "votes": 0, "replies": [ { "id": 677370, "author_name": "pudae", "author_url": "", "post_date": "2019-11-20T04:31:24.960000", "content": "Thanks for your congratulations! 😃
", "votes": 0, "replies": [] } ] }, { "id": 677180, "author_name": "Laevatein", "author_url": "", "post_date": "2019-11-19T22:22:27.637000", "content": "Congrats! Thank you for sharing.
", "votes": 0, "replies": [ { "id": 677371, "author_name": "pudae", "author_url": "", "post_date": "2019-11-20T04:31:29.067000", "content": "Thanks for your congratulations! 😃
", "votes": 0, "replies": [] } ] }, { "id": 676880, "author_name": "liuze", "author_url": "", "post_date": "2019-11-19T15:11:22.600000", "content": "Congratulations, thanks for sharing, impressive solution
", "votes": 0, "replies": [ { "id": 677377, "author_name": "pudae", "author_url": "", "post_date": "2019-11-20T04:42:35.273000", "content": "Thanks~ :)
", "votes": 0, "replies": [] } ] }, { "id": 676867, "author_name": "Camaro", "author_url": "", "post_date": "2019-11-19T15:01:32.060000", "content": "Congratulations!! There are so many things to learn. I can't wait to see your code!
", "votes": 0, "replies": [ { "id": 677378, "author_name": "pudae", "author_url": "", "post_date": "2019-11-20T04:43:06.317000", "content": "Thanks for your congrats!
", "votes": 0, "replies": [] } ] }, { "id": 676802, "author_name": "Gary", "author_url": "", "post_date": "2019-11-19T14:11:34.910000", "content": "congratulations.\nthe trick of exponential moving average is impressived.
", "votes": 0, "replies": [ { "id": 677380, "author_name": "pudae", "author_url": "", "post_date": "2019-11-20T04:43:16.770000", "content": "Thanks for your congrats!
", "votes": 0, "replies": [] } ] }, { "id": 676789, "author_name": "SeshuRaju 🧘♂️", "author_url": "", "post_date": "2019-11-19T13:58:22.717000", "content": "Congrats and Thank you for sharing
", "votes": 0, "replies": [ { "id": 677381, "author_name": "pudae", "author_url": "", "post_date": "2019-11-20T04:43:19.277000", "content": "Thanks for your congrats!
", "votes": 0, "replies": [] } ] }, { "id": 689509, "author_name": "", "author_url": "", "post_date": "2019-12-07T01:45:48.100000", "content": "", "votes": 0, "replies": [] }, { "id": 676846, "author_name": "", "author_url": "", "post_date": "2019-11-19T14:44:34.897000", "content": "", "votes": 0, "replies": [ { "id": 677379, "author_name": "pudae", "author_url": "", "post_date": "2019-11-20T04:43:10.913000", "content": "Thanks for your congrats!
", "votes": 0, "replies": [] } ] }, { "id": 690436, "author_name": "Kranti Kumar", "author_url": "", "post_date": "2019-12-08T15:30:34.353000", "content": "Great work! Thanks for sharing!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "676764": "## UPDATE: code available on github\nhttps://github.com/pudae/kaggle-understanding-clouds\n\n---\n\nCongrats to all the winners and survivors of the shake-up.\nThanks to Kaggle and the hosting team for the interesting competition.\n\nExcept for some tricks, improvements almost have been made by using ensemble. So, in this post, I will briefly describe the track of scores in the last week. The details will be shared as codes.\n\n### Common Settings\n**Types of networks**\n- Model A: UNet with classification head\n- Model B: FPN or UNet, no classification head\n\n**Backbones**\n- resnet34, efficientnet-b1, resnext101_32x8d_wsl, resnext101_32x16d_wsl\n\n**DataSet**\n- split: train vs val = 9 vs 2\n- Model A: All labels\n- Model B: non-empty labels\n\n**Loss**\n- classification part: BCE\n- segmentation part: BCE * 0.75 + DICE * 0.25\n\n**Optimizer**\n- AdamW, weight decay 0.01\n- encoder learning rate 0.000025\n- decoder learning rate 0.00025\n- OneCycle scheduler, shallow models 30 epochs, deep models 15 epochs\n\n**Augmentation**\n- Common: hflip, vflip, shift/scale/rotate, grid distortion, channel shuffle, invert, to gray\n- Model A: random crop, size 384\n- Model B: full-size, size 384, 544, 576, 768\n\n### The track of scores\n**train single model**\nAt first, I’d tried to train a good single network. I’d struggled to improve and stabilize the LB scores for 2 weeks, but I’d failed. \n- TTA3: CV 0.6517 / Public LB 0..66951 / Private LB 0.65828\n\n**add segmentation models**\nI thought the reason for the unstable LB score was because of poor segmentation performance. If we can have a more powerful segmentation model, the effect of poor classification performance can be reduced.\n\nSo, I began trying to train good segmentation only model. Because I could filter out negative predictions using the classification model, only positive labels were needed to train.\n\nFrom this time, CV and LB were correlated well.\nI trained several segmentation models with different backbone, image size, etc.\n- TTA4, 1 seg with cls + 1 seg: CV 0.6560, Public LB 0.67395, Private LB 0.66495\n- TTA4, 1 seg with cls + 3 seg: CV 0.6582, Public LB 0.67482, Private LB 0.66501\n- TTA4, 1 seg with cls + 4 seg: CV 0.6587, Public LB 0.67551, Private LB 0.66604\n- TTA4, 1 seg with cls + 7 seg: CV 0.6594, Public LB 0.67596, Private LB 0.66663\n\n**add more models with classification head**\nNow, the segmentation part became enough good. so, I added two more models with classification head.\n- TTA4, 3 seg with cls + 7 seg: CV 0.6625, Public LB 0.67678, Private LB 0.66746\n\n**use segmentation models as a classifier**\nTo take advantage of the performance of the segmentation models, I used a mean of top K pixel probabilities as a classification probability. \n```\ncls_probabilities = np.sort(mask_probabilities.reshape(4, -1), axis=1)\ncls_probabilities = np.mean(cls_probabilities[:,-17500:], axis=1)\n```\n\n- TTA4, 3 seg with cls + 7 seg: 0.6629, 0.67822, 0.67046\n- TTA4, 3 seg with cls + 8 seg: 0.6635, 0.67906, 0.67117\n\n**use max probability as a positive prediction**\nAll images in the train set have at least one type of cloud, so I treated the label of max probability in each image as a positive prediction. \n```\ncls_probabilities[np.argmax(cls_probabilities)] = 1\n```\n- TTA4, 3 seg with cls + 8 seg: CV 0.6640, Public LB 0.68031, Private LB 0.67170\n\n**use exponential moving average**\nFinally, I changed the averaging weights method to the exponential moving average. Before that, the average of the last 5 weights was used.\n- TTA4, 3 seg with cls + 8 seg: CV 0.6636, Public LB 0.68130, Private LB 0.67126\n- TTA4, 3 seg with cls + 9 seg: CV 0.6637, Public LB 0.68185, Private LB 0.67175 (**Final Submission**)\n", "677390": "Congratulations for winning this competition and thanks for sharing your solution.\nI have one question:\nWhat is classification head in Model A(UNet)?", "676956": "Congratulations, great job. Using segmentation models as classifiers is smart. In my experiments, I found that segmentation models could predict empty masks more accurately than a classification model trained on labels only. Doing \"use max probability as a positive prediction\" is a great trick.\n \nHow did you train on non-empty masks only? Since one image may be positive for Fish, Flower and negative for Gravel and Sugar, when the model sees that image isn't it training on the empty Gravel and Sugar? Do you adjust the loss function to avoid using those?", "679820": "That's really great. Thanks for sharing.", "676896": "Congratulations on your result and thanks for the report.\n\nI have a question about this line:\n`OneCycle scheduler, shallow models 30 epochs, deep models 15 epochs`\nShouldn't we train deep models with more epochs and lower LR to avoid abrupt changes?", "677006": "thanks for the nice work!\n\n\"use max probability as a positive prediction\nAll images in the train set have at least one type of cloud, so I treated the label of max probability in each image as a positive prediction.\"\n\ni note that in the training, we do not actually force this condition to be true in training. \ni wonder if the following will give better results?\n\n```\n... in training iteration ...\nprob_mask = net(intput)\nmax_pixel_prob = ....\n\nmask_loss = loss_mask(prob_mask, truth_mask)\nlabel_loss = loss_label(max_pixel_prob, truth_label)\n\n ... backpropagte mask_loss+label_loss ...\n```", "677258": "Congratulations for being the solo winner, and thanks for sharing!", "1127935": "Excuse me. I can't understand \"I could filter out negative predictions using the classification model\".\n\nI want to know the struct of the classification model and how to train and use the classification model. \n\nThank you!", "964473": "Hi. I've been looking through your solution's GitHub code to learn from it. It seems like you write `from tensorboardX import SummaryWriter` in your kvt\\apis but never create a SummaryWriter object. Why? Additionally you use sacred but don't use any functions that log training metrics like ex.log_scalar. How are you monitoring training?", "816089": "```\nencoder learning rate 0.000025\ndecoder learning rate 0.00025\n```\n\nWhat is the reason for this setting? and Are there any learning materials that can be used for reference", "753327": "Congratulations. ", "698679": "Hey there,\n\nFirst of all, congratulations on winning this competition! We have a school assignment for which we need to find a dataset and use different classification measures on it. Our intention is to use your solution and use the output of the last hidden layer of the network as input for our classification measures. We were wondering if you could help us explain how to get this output as input.\n\nKind regards,\nJoost", "696909": "how can I export pth to model format, etc .pb. ", "678849": "> Code for 1st place solution in Kaggle Humpback Whale Identification Challange.\n\n😄 😄 ", "677756": "Its great work! Congratulations :)", "677611": "Congratulations! Thank you from a beginner for sharing :) ", "677592": "Congratulations!!!", "677414": "Congratulations! How do you implement exponential moving average? ", "677311": "Congratulations and Thank you for sharing !!", "677262": "Congratulations on your result.\n\n", "677202": "Congrats! Thanks for sharing.", "677200": "Congratulations on winning the first place!", "677180": "Congrats! Thank you for sharing.", "676880": "Congratulations, thanks for sharing, impressive solution", "676867": "Congratulations!! There are so many things to learn. I can't wait to see your code!", "676802": "congratulations.\nthe trick of exponential moving average is impressived.", "676789": "Congrats and Thank you for sharing", "689509": "", "676846": "", "690436": "Great work! Thanks for sharing!" } }