Hi, everyone, I am curious what is your best single model on public LB without any leak?\nPlease post your results here if you would like to share.\nFor me:\nLB 0.580 for self-designed resnet18 with external data (single fold result)
", "messages": [ { "id": 444589, "postDate": "2018-12-24T10:35:54.437Z", "content": "Hi, everyone, I am curious what is your best single model on public LB without any leak?\nPlease post your results here if you would like to share.\nFor me:\nLB 0.580 for self-designed resnet18 with external data (single fold result)
", "rawMarkdown": "Hi, everyone, I am curious what is your best single model on public LB without any leak?\nPlease post your results here if you would like to share.\nFor me:\nLB 0.580 for self-designed resnet18 with external data (single fold result)", "votes": 29 }, { "id": 445783, "postDate": "2018-12-27T03:50:09.140Z", "content": "my weird results on 512x512 images. All results below is single fold without TTA, without ensemble
\n\nresnet18 on train = kaggle+external(HPAv18RBGY_wodpl):
\n\nlocal lb : 0.755 (F1@0.5)
\n\npublic lb: 0.551 (F1@0.3) without leak
\n\npublic lb: 0.550 (F1@0.3) overwrite prediction with leak259
\n\ninception-bn on train = kaggle+external(HPAv18RBGY_wodpl):
\n\nlocal lb : 0.785 (F1@0.5)
\n\npublic lb: 0.530 (F1@0.3) without leak
\n\npublic lb: 0.530 (F1@0.3) overwrite prediction with leak259
\n\nresnet34 on train = kaggle+external(HPAv18RBGY_wodpl):
\n\nlocal lb : 0.795 (F1@0.5)
\n\npublic lb: 0.511 (F1@0.3) overwrite prediction with leak259
", "rawMarkdown": "my weird results on 512x512 images. All results below is single fold without TTA, without ensemble\n\n\n\n\nresnet18 on train = kaggle+external(HPAv18RBGY_wodpl):\n\nlocal lb : 0.755 (F1@0.5)\n\npublic lb: 0.551 (F1@0.3) without leak\n\npublic lb: 0.550 (F1@0.3) overwrite prediction with leak259\n\n\n---\n\n\ninception-bn on train = kaggle+external(HPAv18RBGY_wodpl):\n\nlocal lb : 0.785 (F1@0.5)\n\npublic lb: 0.530 (F1@0.3) without leak\n\npublic lb: 0.530 (F1@0.3) overwrite prediction with leak259\n\n\n---\n\n\nresnet34 on train = kaggle+external(HPAv18RBGY_wodpl):\n\nlocal lb : 0.795 (F1@0.5)\n\npublic lb: 0.511 (F1@0.3) overwrite prediction with leak259\n\n \n", "votes": 11, "replies": [ { "id": 445792, "postDate": "2018-12-27T04:15:22.420Z", "content": "Thank you for sharing your result. Did you find the explanation to the decrease of public LB score with increase of the val score? I see exactly the same trend, and my model with 0.80 val score perform worse on public LB than models having 0.75?
", "rawMarkdown": "Thank you for sharing your result. Did you find the explanation to the decrease of public LB score with increase of the val score? I see exactly the same trend, and my model with 0.80 val score perform worse on public LB than models having 0.75?", "votes": 1 }, { "id": 445816, "postDate": "2018-12-27T04:59:57.613Z", "content": "Far from being expert but my guess would be that the model became better at predicting the abundant classes, hence the val_loss improvement (which is not class avg) but it caused the f1 macro score (which is class avg) to drop. Very hard to avoid this, alas.... I actually over-sampled only in my training set and thinking back it might not be the correct strategy.
", "rawMarkdown": "Far from being expert but my guess would be that the model became better at predicting the abundant classes, hence the val_loss improvement (which is not class avg) but it caused the f1 macro score (which is class avg) to drop. Very hard to avoid this, alas.... I actually over-sampled only in my training set and thinking back it might not be the correct strategy. \n" }, { "id": 445849, "postDate": "2018-12-27T05:49:14.833Z", "content": "Hi Heng CherKeng. For \"F1@0.5\", what's the meaning of @?
", "rawMarkdown": "Hi Heng CherKeng. For \"F1@0.5\", what's the meaning of @?" }, { "id": 445851, "postDate": "2018-12-27T05:57:39.483Z", "content": "F1 computed with threshold 0.5
", "rawMarkdown": "F1 computed with threshold 0.5" }, { "id": 445872, "postDate": "2018-12-27T06:36:47.107Z", "content": "Thank you~
", "rawMarkdown": "Thank you~" }, { "id": 449629, "postDate": "2019-01-03T13:18:20.417Z", "content": "How did you split data into train and val?
", "rawMarkdown": "How did you split data into train and val?" }, { "id": 453154, "postDate": "2019-01-09T18:45:12.310Z", "content": "@hengck23 and @iafos just started to use external data and I am observing exactly the same F1 0.75-0.80 with resnet18 scores 0.53-0.51 on LB. Any advice?
", "rawMarkdown": "@hengck23 and @iafos just started to use external data and I am observing exactly the same F1 0.75-0.80 with resnet18 scores 0.53-0.51 on LB. Any advice?" } ] }, { "id": 449127, "postDate": "2019-01-02T17:35:35.170Z", "content": "I have two quite different architectures both LB 0.6+
\n\n@trangle1302 If I would run in kernels I guess I could reach 0.55+ with them
", "rawMarkdown": "I have two quite different architectures both LB 0.6+\n\n- 5-fold\n- rgb\n- 512x512\n\n@trangle1302 If I would run in kernels I guess I could reach 0.55+ with them", "votes": 8, "replies": [ { "id": 449545, "postDate": "2019-01-03T10:32:44.677Z", "rawMarkdown": "", "isDeleted": true }, { "id": 451713, "postDate": "2019-01-07T14:32:08.343Z", "content": "How to preprocess the images with 4 channels, when using pre-trained models. With 5-fold CV and 512 image size, and oversampling and using adam, I am not able to get past 0.50 :(
", "rawMarkdown": "How to preprocess the images with 4 channels, when using pre-trained models. With 5-fold CV and 512 image size, and oversampling and using adam, I am not able to get past 0.50 :(" }, { "id": 451799, "postDate": "2019-01-07T17:17:02.440Z", "content": "I also used external data for training but not the leak file. @sourajmishra I did not use 4 channels, only 3
", "rawMarkdown": "I also used external data for training but not the leak file. @sourajmishra I did not use 4 channels, only 3", "votes": 1 }, { "id": 451809, "postDate": "2019-01-07T17:55:33.340Z", "content": "Thanks. Do you process to compress RBGY to 3 channel somehow, or just use RGB and ignore Y?
", "rawMarkdown": "Thanks. Do you process to compress RBGY to 3 channel somehow, or just use RGB and ignore Y?", "votes": 2 }, { "id": 453137, "postDate": "2019-01-09T18:17:36.133Z", "content": "ignore Y
", "rawMarkdown": "ignore Y" }, { "id": 454221, "postDate": "2019-01-11T09:10:19.143Z", "content": "i wrote a quick summary here:
\n\nhttps://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/77300
", "rawMarkdown": "i wrote a quick summary here:\n\nhttps://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/77300", "votes": 1 } ] }, { "id": 444721, "postDate": "2018-12-24T16:34:49.177Z", "content": "self-designed InceptionV3 (external data + image size 512x512): 0.577 single fold, 0.599 average of 3 folds
", "rawMarkdown": "self-designed InceptionV3 (external data + image size 512x512): 0.577 single fold, 0.599 average of 3 folds", "votes": 6, "replies": [ { "id": 444732, "postDate": "2018-12-24T16:59:56.207Z", "content": "Great work! May I ask that did you use green channel mask attention?
", "rawMarkdown": "Great work! May I ask that did you use green channel mask attention?" }, { "id": 444765, "postDate": "2018-12-24T18:51:09.973Z", "content": "Very impressive. Can you drop some hints on your approach?
", "rawMarkdown": "Very impressive. Can you drop some hints on your approach? " }, { "id": 444850, "postDate": "2018-12-25T01:31:23.903Z", "content": "I did not use green channel mask attention. Instead, I design a new method to make the network attend on the rare labels.
", "rawMarkdown": "I did not use green channel mask attention. Instead, I design a new method to make the network attend on the rare labels.", "votes": 6 }, { "id": 444866, "postDate": "2018-12-25T02:41:34.843Z", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!" }, { "id": 445307, "postDate": "2018-12-26T07:28:35.357Z", "content": "May I ask which loss function you used for this report?
", "rawMarkdown": "May I ask which loss function you used for this report?" }, { "id": 445347, "postDate": "2018-12-26T08:49:05.817Z", "content": "Hi Chris, I'm using binary cross-entropy. I tried to combine it with focal loss and f1 loss but pure binary cross-entropy gave me the best result.
", "rawMarkdown": "Hi Chris, I'm using binary cross-entropy. I tried to combine it with focal loss and f1 loss but pure binary cross-entropy gave me the best result.", "votes": 1 }, { "id": 447099, "postDate": "2018-12-29T06:23:28.703Z", "content": "That's just the opposite of my observation. I'm using self-designed resnet50 with focal loss got 0.574 single fold.
", "rawMarkdown": "That's just the opposite of my observation. I'm using self-designed resnet50 with focal loss got 0.574 single fold." }, { "id": 447102, "postDate": "2018-12-29T06:32:33.840Z", "content": "Did you tune the gamma and alpha value? I used the default set up, gamma=2, alpha=0.25
", "rawMarkdown": "Did you tune the gamma and alpha value? I used the default set up, gamma=2, alpha=0.25", "votes": 1 }, { "id": 447183, "postDate": "2018-12-29T10:18:20.943Z", "content": "I have tried different settings for gamma and alpha, found that gamma=2, alpha=0.25 gives the best performance.
", "rawMarkdown": "I have tried different settings for gamma and alpha, found that gamma=2, alpha=0.25 gives the best performance.", "votes": 1 }, { "id": 450964, "postDate": "2019-01-06T07:00:29.613Z", "content": "Hi, @Chris J.Liu, by using normal ResNet50, how much LB do you get for single fold with focal loss?
", "rawMarkdown": "Hi, @Chris J.Liu, by using normal ResNet50, how much LB do you get for single fold with focal loss?" } ] }, { "id": 444609, "postDate": "2018-12-24T11:15:39.450Z", "content": "Could you give me some hints about your loss function??
", "rawMarkdown": "Could you give me some hints about your loss function??", "votes": 1, "replies": [ { "id": 444613, "postDate": "2018-12-24T11:49:42.517Z", "content": "Check this excellent kernel: https://www.kaggle.com/rejpalcz/best-loss-function-for-f1-score-metric. For me, soft f1 loss works well, some combination may help, on experiment.
", "rawMarkdown": "Check this excellent kernel: https://www.kaggle.com/rejpalcz/best-loss-function-for-f1-score-metric. For me, soft f1 loss works well, some combination may help, on experiment.", "votes": 4 }, { "id": 444814, "postDate": "2018-12-24T21:56:24.160Z", "content": "I was under the impression that F1, being a macro metrics, works ok only on very large batches. What is the size of your batch?
", "rawMarkdown": "I was under the impression that F1, being a macro metrics, works ok only on very large batches. What is the size of your batch?" }, { "id": 444882, "postDate": "2018-12-25T04:09:05.297Z", "content": "I have the same experience. On 256x256 with batches 64 and 128 F1 loss works. However, when I tried to move to 512x512 images the performance has degraded, even if I mixed focal and F1 loss. With batch size 16 training didn't really work.
", "rawMarkdown": "I have the same experience. On 256x256 with batches 64 and 128 F1 loss works. However, when I tried to move to 512x512 images the performance has degraded, even if I mixed focal and F1 loss. With batch size 16 training didn't really work. ", "votes": 1 }, { "id": 444896, "postDate": "2018-12-25T05:00:12.140Z", "content": "Now my batch size is 128, the performance better than 64.
", "rawMarkdown": "Now my batch size is 128, the performance better than 64." }, { "id": 448892, "postDate": "2019-01-02T10:26:51.957Z", "content": "definitely larger batch could represent the true distribution of whole dataset, but I wonder the image size u r using now, is it possible to get LB 0.58 + with size 256 * 256 ? \nhope for ur answer hh
", "rawMarkdown": "definitely larger batch could represent the true distribution of whole dataset, but I wonder the image size u r using now, is it possible to get LB 0.58 + with size 256 * 256 ? \nhope for ur answer hh " } ] }, { "id": 449449, "postDate": "2019-01-03T06:47:03.997Z", "content": "I have a Resnet50, 512 x 512 image size, external data, and some test data leak, gives a LB 0.569
", "rawMarkdown": "I have a Resnet50, 512 x 512 image size, external data, and some test data leak, gives a LB 0.569", "votes": 2, "replies": [ { "id": 451811, "postDate": "2019-01-07T17:57:29.420Z", "content": "Do you get this score with Binary cross entropy, or with some custom loss?
", "rawMarkdown": "Do you get this score with Binary cross entropy, or with some custom loss?", "votes": 1 }, { "id": 451867, "postDate": "2019-01-07T19:51:45.377Z", "content": "With BCE, I got higher validation score, but poor LB score, but with focal loss, I got a much better LB score
", "rawMarkdown": "With BCE, I got higher validation score, but poor LB score, but with focal loss, I got a much better LB score" }, { "id": 452012, "postDate": "2019-01-08T03:07:08.417Z", "content": "Thank You. With the focal loss, I have tried Adam with staring lr 0.001 and then decreasing it every 10 epochs with gamma 0.1 to 0.5. But it gets stuck on around 0.3 on LB. Can you suggest some strategy for this.
", "rawMarkdown": "Thank You. With the focal loss, I have tried Adam with staring lr 0.001 and then decreasing it every 10 epochs with gamma 0.1 to 0.5. But it gets stuck on around 0.3 on LB. Can you suggest some strategy for this." }, { "id": 452593, "postDate": "2019-01-08T23:34:53.607Z", "content": "I started with 2e-5, and decreasing every 10 epochs, trained with total 30 epochs.
", "rawMarkdown": "I started with 2e-5, and decreasing every 10 epochs, trained with total 30 epochs. " } ] }, { "id": 444755, "postDate": "2018-12-24T18:24:37.677Z", "content": "0.564, 512x512, resnet34 with HPAv18 external data, single fold. I'm running a 5-fold experiment, but it's taking forever to finish with the external data.
", "rawMarkdown": "0.564, 512x512, resnet34 with HPAv18 external data, single fold. I'm running a 5-fold experiment, but it's taking forever to finish with the external data.", "votes": 2, "replies": [ { "id": 444890, "postDate": "2018-12-25T04:41:52.730Z", "content": "Good work!
", "rawMarkdown": "Good work!" }, { "id": 444913, "postDate": "2018-12-25T06:27:33.850Z", "content": "\n\n\n\n\n0.564, 512x512, resnet34 with HPAv18 external data, single fold
\n
Is this with TTA or just on original input images? With is the threshold you are using?
", "rawMarkdown": ">> 0.564, 512x512, resnet34 with HPAv18 external data, single fold\n\nIs this with TTA or just on original input images? With is the threshold you are using?" }, { "id": 445093, "postDate": "2018-12-25T15:54:00.843Z", "content": "With TTA and adjusted thresholds fitted on the validation set.
", "rawMarkdown": "With TTA and adjusted thresholds fitted on the validation set. ", "votes": 1 } ] }, { "id": 446094, "postDate": "2018-12-27T13:35:16.253Z", "content": "Hi, @Kevin Zheng, Do you train from scratch? Can you share more details? Like loss function, image size, optimizer, init lr, etc.
", "rawMarkdown": "Hi, @Kevin Zheng, Do you train from scratch? Can you share more details? Like loss function, image size, optimizer, init lr, etc.", "votes": 1 }, { "id": 445841, "postDate": "2018-12-27T05:35:15.313Z", "content": "Thanks, can you share the lr schedule?
", "rawMarkdown": "Thanks, can you share the lr schedule?", "votes": -1, "replies": [ { "id": 445914, "postDate": "2018-12-27T07:48:58.197Z", "content": "SGD, lr start with 0.1, reduce lr by 0.2 when the loss stop decrease in 8 epochs.
", "rawMarkdown": "SGD, lr start with 0.1, reduce lr by 0.2 when the loss stop decrease in 8 epochs.", "votes": 2 }, { "id": 446710, "postDate": "2018-12-28T14:53:35.627Z", "content": "Wow!\nHow many epochs does it take?
", "rawMarkdown": "Wow!\nHow many epochs does it take?" }, { "id": 447254, "postDate": "2018-12-29T13:37:57.153Z", "content": "80~100
", "rawMarkdown": "80~100", "votes": 1 } ] }, { "id": 453564, "postDate": "2019-01-10T11:28:18.270Z", "content": "I am curious about what type of resnet you are using?
", "rawMarkdown": "I am curious about what type of resnet you are using?", "replies": [ { "id": 453581, "postDate": "2019-01-10T12:25:17.020Z", "content": "Same here. People often state they are using self-defined well-known architecture but I have seen cases where this stands for changing the output layer to nets that are just inspired by well known architectures (residual, inception block, etc.). I assume head is self defined but none of the earlier layers are changed (perhaps in conv1 3=>4 channels).
", "rawMarkdown": "Same here. People often state they are using self-defined well-known architecture but I have seen cases where this stands for changing the output layer to nets that are just inspired by well known architectures (residual, inception block, etc.). I assume head is self defined but none of the earlier layers are changed (perhaps in conv1 3=>4 channels). " } ] }, { "id": 451791, "postDate": "2019-01-07T16:51:27.943Z", "content": "I use a Resnet34, 512 x 512 image size, RGBY, external data, no leaked data, single fold with data augmentation and TTA, upsample of the less frequent classes and it gives a LB 0.568
", "rawMarkdown": "I use a Resnet34, 512 x 512 image size, RGBY, external data, no leaked data, single fold with data augmentation and TTA, upsample of the less frequent classes and it gives a LB 0.568" }, { "id": 451337, "postDate": "2019-01-06T22:15:54.170Z", "content": "0.577
\n\nResnet50, 512x512 rgb images, oversampling rare classes, simple data augmentation, external data, 5-fold with TTA, BCE loss
\n\nThe leak boosts me to 0.588. Oversampling dramatically reduced the boost I was getting from the leak.
", "rawMarkdown": "0.577\n\nResnet50, 512x512 rgb images, oversampling rare classes, simple data augmentation, external data, 5-fold with TTA, BCE loss\n\nThe leak boosts me to 0.588. Oversampling dramatically reduced the boost I was getting from the leak.", "replies": [ { "id": 451397, "postDate": "2019-01-07T02:43:52.157Z", "content": "cool, thx for ur sharing. Do u use leak126 ?
", "rawMarkdown": "cool, thx for ur sharing. Do u use leak126 ?" }, { "id": 451406, "postDate": "2019-01-07T03:01:58.177Z", "content": "Thanks for sharing, May I ask did you try rgby? what's the result?
", "rawMarkdown": "Thanks for sharing, May I ask did you try rgby? what's the result?" }, { "id": 451416, "postDate": "2019-01-07T03:47:40.093Z", "content": "I've been using leak259
", "rawMarkdown": "I've been using leak259" }, { "id": 451417, "postDate": "2019-01-07T03:49:21.777Z", "content": "I've tried rgby several times with slightly worse results than rgb. I don't have a good understanding why that is.
", "rawMarkdown": "I've tried rgby several times with slightly worse results than rgb. I don't have a good understanding why that is." }, { "id": 451533, "postDate": "2019-01-07T08:34:41.310Z", "content": "Hi, Do you use same threshold for all classes? or choose threshold from validation set?
", "rawMarkdown": "Hi, Do you use same threshold for all classes? or choose threshold from validation set?" } ] }, { "id": 448773, "postDate": "2019-01-02T04:18:18.413Z", "content": "Can you guys tell me the way of split the trainval? my val is not robust, the score on lb didn't increase even thought local increases by a large margin
", "rawMarkdown": "Can you guys tell me the way of split the trainval? my val is not robust, the score on lb didn't increase even thought local increases by a large margin" }, { "id": 447844, "postDate": "2018-12-30T18:02:19.110Z", "content": "ResNet18 is overfitting even after using external data and dropout(0.5). Can Anyone give any suggestion to avoid this problem??
", "rawMarkdown": "ResNet18 is overfitting even after using external data and dropout(0.5). Can Anyone give any suggestion to avoid this problem??", "replies": [ { "id": 447881, "postDate": "2018-12-30T18:56:13.693Z", "content": "Few things to deal with overfitting:
\n\nMost robust way is to use a validation set that is not very similar to train set but still represents test set. Usually, when your training loss is getting lower significantly compared to validation loss, it's a sign of overfitting. You may want to use early stopping or use an earlier checkpoint in that case when validation loss starts increasing. However, sometimes maximising competition validation metric helps, even if validation loss increases. But for this competition, there are similar images in train set. So your validation metric(e.g. F-1) may increase with your training, but it is because your validation set contains similar images like train, and you will not know if your model is overfitting.
Be careful when using Dropout. If there are certain features present in your data that doesn't represent general feature but overfitting feature, dropout may force the network to learn overfitting features to decrease loss. (you can think of an overfitting feature something that is not protein in an image, but something irrelevant present in a given class). Other regularisation techniques help in complementary to dropout.
Fun fact: Neural networks are like babies. You can spoil them really easily if you force to teach them everything or leave them alone to learn as they will. Teach them slowly and patiently.
\n\nHave fun!!
", "rawMarkdown": "Few things to deal with overfitting:\n\n1. Most robust way is to use a validation set that is not very similar to train set but still represents test set. Usually, when your training loss is getting lower significantly compared to validation loss, it's a sign of overfitting. You may want to use early stopping or use an earlier checkpoint in that case when validation loss starts increasing. However, sometimes maximising competition validation metric helps, even if validation loss increases. But for this competition, there are similar images in train set. So your validation metric(e.g. F-1) may increase with your training, but it is because your validation set contains similar images like train, and you will not know if your model is overfitting.\n\n2. Be careful when using Dropout. If there are certain features present in your data that doesn't represent general feature but overfitting feature, dropout may force the network to learn overfitting features to decrease loss. (you can think of an overfitting feature something that is not protein in an image, but something irrelevant present in a given class). Other regularisation techniques help in complementary to dropout. \n\nFun fact: Neural networks are like babies. You can spoil them really easily if you force to teach them everything or leave them alone to learn as they will. Teach them slowly and patiently.\n\nHave fun!!", "votes": 9 }, { "id": 447883, "postDate": "2018-12-30T19:07:39.980Z", "content": "Augmentation is your friend! Try flipping and rotating for start. The more augmentation you use the more robust your network will be, as it won't see \"the same\" images all the time. Make sure the original images still show once in a while...
", "rawMarkdown": "Augmentation is your friend! Try flipping and rotating for start. The more augmentation you use the more robust your network will be, as it won't see \"the same\" images all the time. Make sure the original images still show once in a while... ", "votes": 2 }, { "id": 447886, "postDate": "2018-12-30T19:09:22.767Z", "content": "Actually it's the other way around. Babies are like neural networks :)
", "rawMarkdown": "Actually it's the other way around. Babies are like neural networks :) ", "votes": 7 }, { "id": 448544, "postDate": "2019-01-01T13:13:33.023Z", "content": "Actually, too wild augmentation can hurt you!
", "rawMarkdown": "Actually, too wild augmentation can hurt you!\n" }, { "id": 449034, "postDate": "2019-01-02T15:08:52.120Z", "content": "Hi @sgalib,
\n\nYour 1st point is actually really interesting... How can you make sure that the model is not overfitting if you can't use early stopping ? and can't really trust your local CV F1 score and loss ? Is the solution to manually design this val set to avoid being too similar to train ? Or are you able to completely avoid overfitting with regularization ?
", "rawMarkdown": "Hi @sgalib,\n\nYour 1st point is actually really interesting... How can you make sure that the model is not overfitting if you can't use early stopping ? and can't really trust your local CV F1 score and loss ? Is the solution to manually design this val set to avoid being too similar to train ? Or are you able to completely avoid overfitting with regularization ? " }, { "id": 449228, "postDate": "2019-01-02T20:27:48.487Z", "content": "Interesting question. I hope I understood your question correctly.
\n\nIf you can't use early stopping: I suppose every neural network will overfit with favorable hyper-parameter selection if you train them for very long period of time. This is particularly true if the dataset is small and your neural network architecture is very deep/complex.So you have to stop somewhere depending on your training and validation metric.
If you can't trust local validation metric: I guess validation metric is your only friend while training. The more effort you put to construct a validation set, the more reward you get. Making a good validation set can be effort-some. You can explore some clustering algorithms to divide your dataset, so that same clusters from same class are not present in both of your tain-val sets. This is one approach, there are tons of others. Fold-wise cross-validation will also give you a hint on how much varience you have between folds, and potentially in test set also. So yes, it's a semi-manual process.
In this specific competition, dataset is rather small even with external data. State-of-the art imagenet architectures overfit pretty easily. Regularization reduces overfit but it also suffers from underfitting. So there is a trade-off. I have mostly played with lighter architectures which are less likely to overfit. And early-stopping as well.
Last but not the least, suppose you have constructed a good validation set, played to reduce overfit but still not satisfied with your local to LB performance, then check the the ghost in some other places. Study competition metric and have a sense how it might be affecting, because your validation set still may not represent Public LB set!
Hope this helps!
", "rawMarkdown": "Interesting question. I hope I understood your question correctly.\n\n1. If you can't use early stopping: I suppose every neural network will overfit with favorable hyper-parameter selection if you train them for very long period of time. This is particularly true if the dataset is small and your neural network architecture is very deep/complex.So you have to stop somewhere depending on your training and validation metric.\n\n2. If you can't trust local validation metric: I guess validation metric is your only friend while training. The more effort you put to construct a validation set, the more reward you get. Making a good validation set can be effort-some. You can explore some clustering algorithms to divide your dataset, so that same clusters from same class are not present in both of your tain-val sets. This is one approach, there are tons of others. Fold-wise cross-validation will also give you a hint on how much varience you have between folds, and potentially in test set also. So yes, it's a semi-manual process.\n\n3. In this specific competition, dataset is rather small even with external data. State-of-the art imagenet architectures overfit pretty easily. Regularization reduces overfit but it also suffers from underfitting. So there is a trade-off. I have mostly played with lighter architectures which are less likely to overfit. And early-stopping as well.\n\n4. Last but not the least, suppose you have constructed a good validation set, played to reduce overfit but still not satisfied with your local to LB performance, then check the the ghost in some other places. Study competition metric and have a sense how it might be affecting, because your validation set still may not represent Public LB set!\n\nHope this helps!", "votes": 5 }, { "id": 449326, "postDate": "2019-01-03T01:02:38.440Z", "content": "Thanks a lot for your answer @sgalib !
\n\nYes I think you got my point. As you said \"there are similar images in train set.\" so when creating your validation you might end up with similar data in train and val, and so even though you overfit, your val loss might still decrease and you won't be able to use early stopping. At least that's the problem I ran into with one of my models. So I wanted to know if you had any advice for this particular case. I guess I should have spent more time on making a proper CV earlier in the competition. I agree with using light models. Where you able to get a stable LB / CV F1 difference ?
", "rawMarkdown": "Thanks a lot for your answer @sgalib !\n\nYes I think you got my point. As you said \"there are similar images in train set.\" so when creating your validation you might end up with similar data in train and val, and so even though you overfit, your val loss might still decrease and you won't be able to use early stopping. At least that's the problem I ran into with one of my models. So I wanted to know if you had any advice for this particular case. I guess I should have spent more time on making a proper CV earlier in the competition. I agree with using light models. Where you able to get a stable LB / CV F1 difference ? ", "votes": 1 }, { "id": 449802, "postDate": "2019-01-03T19:17:22.337Z", "content": "Aha... I got your point. If I do not have time to do CV, a quick fix in that case I would do is train a few models with random validation split, average/ensemble predictions. Usually you won't be disappointed! :)
", "rawMarkdown": "Aha... I got your point. If I do not have time to do CV, a quick fix in that case I would do is train a few models with random validation split, average/ensemble predictions. Usually you won't be disappointed! :)", "votes": 1 } ] }, { "id": 445226, "postDate": "2018-12-26T02:22:41.770Z", "content": "I got 0.532 without specifically treating the leak as per the posted leak csv. But for this, I did use the HPA data which had the leak data in it. I did not specifically exclude this. \nAdding the leak csv to this boosted the score a bunch.
\n\nHave you tried adding leak info to your submission? If so, what boost did you get?
", "rawMarkdown": "I got 0.532 without specifically treating the leak as per the posted leak csv. But for this, I did use the HPA data which had the leak data in it. I did not specifically exclude this. \nAdding the leak csv to this boosted the score a bunch.\n\nHave you tried adding leak info to your submission? If so, what boost did you get?", "replies": [ { "id": 445260, "postDate": "2018-12-26T04:20:48.353Z", "content": "The difference is smaller and smaller as my models improve. Currently its 0.555->0.566
", "rawMarkdown": "The difference is smaller and smaller as my models improve. Currently its 0.555->0.566" }, { "id": 445281, "postDate": "2018-12-26T05:58:17.600Z", "content": "boost 0.006 with brain's 126 leak
", "rawMarkdown": "boost 0.006 with brain's 126 leak", "votes": 1 }, { "id": 445343, "postDate": "2018-12-26T08:45:38.897Z", "content": "Amazing. Looks like you got a very good answer for those two classes without help, just by using HPA. To be honest, I feel that a lot of my score is due to random boosts and that randomness is playing a big part here.
", "rawMarkdown": "Amazing. Looks like you got a very good answer for those two classes without help, just by using HPA. To be honest, I feel that a lot of my score is due to random boosts and that randomness is playing a big part here.", "votes": -1 }, { "id": 445425, "postDate": "2018-12-26T12:30:32.073Z", "content": "@Kevin Zheng
\n\nthere are two leak:
\n\nhttps://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/73395
\n\nhttps://storage.googleapis.com/kaggle-forum-message-attachments/433234/10820/overlap.py
\n\nhttps://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/72534
\n\nTestEtraMatchingUnder_259_R14_G12_B10.csv (18.92 KB)
\n\nfor the same input test image, leak126 and leak256 sometimes disagree on the labels. i think leak259 is more correct
", "rawMarkdown": "@Kevin Zheng\n \nthere are two leak:\n\n\n1. leak 126 \n\nhttps://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/73395\n\nhttps://storage.googleapis.com/kaggle-forum-message-attachments/433234/10820/overlap.py\n\n\n2. leak 259\n\nhttps://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/72534\n\nTestEtraMatchingUnder_259_R14_G12_B10.csv (18.92 KB)\n\nfor the same input test image, leak126 and leak256 sometimes disagree on the labels. i think leak259 is more correct \n\n" }, { "id": 445429, "postDate": "2018-12-26T12:44:14.290Z", "content": "Thanks, Heng. Learn a lot from you. Happy to see you in the challenge!
", "rawMarkdown": "Thanks, Heng. Learn a lot from you. Happy to see you in the challenge!" }, { "id": 449348, "postDate": "2019-01-03T02:16:21.337Z", "content": "Thanks @hengck23. For the leaked data, I downloaded \"TestEtraMatchingUnder259R14G12B10.csv\" but don't see any labels. Is that left as an exercise for the reader?
", "rawMarkdown": "Thanks @hengck23. For the leaked data, I downloaded \"TestEtraMatchingUnder259R14G12B10.csv\" but don't see any labels. Is that left as an exercise for the reader?" } ] }, { "id": 444883, "postDate": "2018-12-25T04:10:01.817Z", "content": "0.576, 512 image size, BNInception, with external data.
", "rawMarkdown": "0.576, 512 image size, BNInception, with external data.", "replies": [ { "id": 444897, "postDate": "2018-12-25T05:07:39.547Z", "content": "Sounds like Inception help capture multi-scale information?
", "rawMarkdown": "Sounds like Inception help capture multi-scale information?" }, { "id": 444898, "postDate": "2018-12-25T05:17:59.457Z", "content": "Not sure, I used several models (inception or non-inception), they have similar performance.
", "rawMarkdown": "Not sure, I used several models (inception or non-inception), they have similar performance." }, { "id": 445318, "postDate": "2018-12-26T07:48:13.873Z", "content": "Can you share more details? Thx!
", "rawMarkdown": "Can you share more details? Thx!" }, { "id": 445432, "postDate": "2018-12-26T12:53:51.743Z", "content": "@ManyFoldCV
\n\n\" I used several models (inception or non-inception), they have similar performance.\"
\n\nIn my experiments, they are same in F1 score. But inception has more correct multi-label images and more wrong false positive.
\n\nresnet has less correct multi-label images and more less false positive.
", "rawMarkdown": "@ManyFoldCV\n \n\" I used several models (inception or non-inception), they have similar performance.\"\n\nIn my experiments, they are same in F1 score. But inception has more correct multi-label images and more wrong false positive.\n\nresnet has less correct multi-label images and more less false positive.", "votes": 1 }, { "id": 446105, "postDate": "2018-12-27T13:59:35.807Z", "content": "Could you please share which implementation of BN-Inception do you use? Is it from Cadene repo with PyTorch pretrained models?
", "rawMarkdown": "Could you please share which implementation of BN-Inception do you use? Is it from Cadene repo with PyTorch pretrained models?", "votes": 1 }, { "id": 446121, "postDate": "2018-12-27T14:42:26.973Z", "content": "Yes, i am using pretrain model from Cadene repo
", "rawMarkdown": "Yes, i am using pretrain model from Cadene repo", "votes": 2 }, { "id": 446163, "postDate": "2018-12-27T16:27:29.610Z", "rawMarkdown": "", "isDeleted": true }, { "id": 446164, "postDate": "2018-12-27T16:28:13.957Z", "content": "The one from Cadene repo @hokmund
", "rawMarkdown": "The one from Cadene repo @hokmund", "votes": 1 }, { "id": 446165, "postDate": "2018-12-27T16:28:34.050Z", "content": "Check my discussion post :D @chrisluu
", "rawMarkdown": "Check my discussion post :D @chrisluu" }, { "id": 447888, "postDate": "2018-12-30T19:17:43.150Z", "content": "@hengck23\n'In my experiments, they are same in F1 score. But inception has more correct multi-label images and more wrong false positive.\nresnet has less correct multi-label images and more less false positive.'
\n\nI think this is an important observation. I reached the same conclusion. I am using mostly resnets.
", "rawMarkdown": "@hengck23\n'In my experiments, they are same in F1 score. But inception has more correct multi-label images and more wrong false positive.\nresnet has less correct multi-label images and more less false positive.'\n\nI think this is an important observation. I reached the same conclusion. I am using mostly resnets.\n", "votes": 2 }, { "id": 447957, "postDate": "2018-12-30T22:38:42.107Z", "content": "Hi @arnaurm, Do you have any ideas about the reason?
", "rawMarkdown": "Hi @arnaurm, Do you have any ideas about the reason?" }, { "id": 448151, "postDate": "2018-12-31T10:56:14.340Z", "content": "No, not really
", "rawMarkdown": "No, not really" }, { "id": 448429, "postDate": "2019-01-01T06:14:30.673Z", "content": "Another observation:
\n\nI compute f1 for single label and multi label for validation during the metering in the training iterations. If i sample more multi label train images in a batch, i can see both validation f1 falls.
\n\nHowever, public lb is significantly worsen.
", "rawMarkdown": "Another observation:\n\nI compute f1 for single label and multi label for validation during the metering in the training iterations. If i sample more multi label train images in a batch, i can see both validation f1 falls.\n\nHowever, public lb is significantly worsen." } ] }, { "id": 444737, "postDate": "2018-12-24T17:13:13.520Z", "content": "0.545 resnet34 with naive 0.5 thresholds
", "rawMarkdown": "0.545 resnet34 with naive 0.5 thresholds", "replies": [ { "id": 444865, "postDate": "2018-12-25T02:41:00.983Z", "content": "Thanks for reply. Now I use multi thresholds searching from val dataset. I will try single threshold.
", "rawMarkdown": "Thanks for reply. Now I use multi thresholds searching from val dataset. I will try single threshold." }, { "id": 444914, "postDate": "2018-12-25T06:29:12.733Z", "content": "@Florian Muellerklein
\n\nlower threshold (e.g. 0.3 for my case) seems to give better results. Did you compare results for other threshold values or adaptive thresholds for different classes?
", "rawMarkdown": "@Florian Muellerklein\n\nlower threshold (e.g. 0.3 for my case) seems to give better results. Did you compare results for other threshold values or adaptive thresholds for different classes?" }, { "id": 445072, "postDate": "2018-12-25T14:51:20.017Z", "content": "I don't want to choose thresholds just based only on the public leaderboard performance. I've done some local threshold search with my validation sets but they do worse on the public leaderboard.
", "rawMarkdown": "I don't want to choose thresholds just based only on the public leaderboard performance. I've done some local threshold search with my validation sets but they do worse on the public leaderboard. ", "votes": 2 } ] }, { "id": 444654, "postDate": "2018-12-24T13:50:08.520Z", "content": "How image size do you use?\nI used self-designed seresnet50 with 512x512, Public LB: 0.539
", "rawMarkdown": "How image size do you use?\nI used self-designed seresnet50 with 512x512, Public LB: 0.539", "replies": [ { "id": 444710, "postDate": "2018-12-24T16:09:29.317Z", "content": "256x256
", "rawMarkdown": "256x256", "votes": 1 } ] }, { "id": 444598, "postDate": "2018-12-24T10:48:19.673Z", "content": "I only got 0.517 without leak and additional data using backbone resnet50
", "rawMarkdown": "I only got 0.517 without leak and additional data using backbone resnet50", "replies": [ { "id": 444608, "postDate": "2018-12-24T11:08:36.167Z", "content": "Thanks for reply. Focus on your loss function design maybe help. Good luck.
", "rawMarkdown": "Thanks for reply. Focus on your loss function design maybe help. Good luck.", "votes": 3 }, { "id": 444659, "postDate": "2018-12-24T14:12:04.997Z", "content": "I am curious about your image size. I think the reason why using resnet18 is to fit the larger image ?
", "rawMarkdown": "I am curious about your image size. I think the reason why using resnet18 is to fit the larger image ?" }, { "id": 444709, "postDate": "2018-12-24T16:09:12.723Z", "content": "256x256
", "rawMarkdown": "256x256", "votes": 2 }, { "id": 444712, "postDate": "2018-12-24T16:15:08.663Z", "content": "Amazing performance ! Do you use external data ?
", "rawMarkdown": "Amazing performance ! Do you use external data ?" }, { "id": 444716, "postDate": "2018-12-24T16:25:12.693Z", "content": "Yes, I use HPAv18 external data.
", "rawMarkdown": "Yes, I use HPAv18 external data." }, { "id": 444722, "postDate": "2018-12-24T16:45:16.100Z", "content": "Just found out the author of this kernel can achieve 0.577 with external data (inception V3)
", "rawMarkdown": "Just found out the author of this [kernel][1] can achieve 0.577 with external data (inception V3)\n\n\n [1]: https://www.kaggle.com/mathormad/inceptionv3-baseline-lb-0-379 \"kernel\"", "votes": 2 }, { "id": 444733, "postDate": "2018-12-24T17:00:18.223Z", "content": "Thanks!
", "rawMarkdown": "Thanks!" }, { "id": 445312, "postDate": "2018-12-26T07:35:04.487Z", "content": "That's amazing! Do you try a larger image size? What about the results? For my observation, the larger image size will help improve LB score.
", "rawMarkdown": "That's amazing! Do you try a larger image size? What about the results? For my observation, the larger image size will help improve LB score. " }, { "id": 445387, "postDate": "2018-12-26T10:59:02.143Z", "content": "I will try. GPU is my bottleneck now...
", "rawMarkdown": "I will try. GPU is my bottleneck now..." }, { "id": 448210, "postDate": "2018-12-31T13:18:58.480Z", "content": "@tkuanlun350
\n\n\"I only got 0.517 without leak and additional data using backbone resnet50\"
\n\n0.517 is without hpa external data?
", "rawMarkdown": "@tkuanlun350\n\n\"I only got 0.517 without leak and additional data using backbone resnet50\"\n\n 0.517 is without hpa external data?" }, { "id": 448230, "postDate": "2018-12-31T14:42:55.323Z", "content": "Yes, however 0.517 for single fold without external and TTA is just one-time luck (cannot get external data work at that time). I cannot reproduce the performance for the same setting. After some tuning and bug fixing, my current single fold model with external data ~0.58( with TTA )
", "rawMarkdown": "Yes, however 0.517 for single fold without external and TTA is just one-time luck (cannot get external data work at that time). I cannot reproduce the performance for the same setting. After some tuning and bug fixing, my current single fold model with external data ~0.58( with TTA )" }, { "id": 448418, "postDate": "2019-01-01T05:15:12.977Z", "content": "Same as you. Now my single fold model with external data get lb 0.580. May I ask you how you choose threshold?
", "rawMarkdown": "Same as you. Now my single fold model with external data get lb 0.580. May I ask you how you choose threshold?" }, { "id": 449160, "postDate": "2019-01-02T18:19:31.660Z", "content": "You can decide on different threshold for each class which can be deduced from the validation set.
", "rawMarkdown": "You can decide on different threshold for each class which can be deduced from the validation set." }, { "id": 449699, "postDate": "2019-01-03T16:08:38.990Z", "content": "@hdzheng\nI am just using the same threshold for every class (based on LB). Threshold selection is not working very well in my current setting :(
", "rawMarkdown": "@hdzheng\nI am just using the same threshold for every class (based on LB). Threshold selection is not working very well in my current setting :(" }, { "id": 449709, "postDate": "2019-01-03T16:25:00.563Z", "content": "@tkuanlun350,\nHow much did you improve the LB by fine-tuning the thresholds?\nmy best single model can get 0.59 on LB with the optimal threshold from 5-fold validation.\nSo I am wondering if I should change the threshold or not.
", "rawMarkdown": "@tkuanlun350,\nHow much did you improve the LB by fine-tuning the thresholds?\nmy best single model can get 0.59 on LB with the optimal threshold from 5-fold validation.\nSo I am wondering if I should change the threshold or not.\n" } ] }, { "id": 448186, "postDate": "2018-12-31T11:58:12.313Z", "rawMarkdown": "", "votes": 2, "isDeleted": true } ], "comments": [ { "id": 445783, "author_name": "hengck23", "author_url": "", "post_date": "2018-12-27T03:50:09.140000", "content": "my weird results on 512x512 images. All results below is single fold without TTA, without ensemble
\n\nresnet18 on train = kaggle+external(HPAv18RBGY_wodpl):
\n\nlocal lb : 0.755 (F1@0.5)
\n\npublic lb: 0.551 (F1@0.3) without leak
\n\npublic lb: 0.550 (F1@0.3) overwrite prediction with leak259
\n\ninception-bn on train = kaggle+external(HPAv18RBGY_wodpl):
\n\nlocal lb : 0.785 (F1@0.5)
\n\npublic lb: 0.530 (F1@0.3) without leak
\n\npublic lb: 0.530 (F1@0.3) overwrite prediction with leak259
\n\nresnet34 on train = kaggle+external(HPAv18RBGY_wodpl):
\n\nlocal lb : 0.795 (F1@0.5)
\n\npublic lb: 0.511 (F1@0.3) overwrite prediction with leak259
", "votes": 11, "replies": [ { "id": 445792, "author_name": "Iafoss", "author_url": "", "post_date": "2018-12-27T04:15:22.420000", "content": "Thank you for sharing your result. Did you find the explanation to the decrease of public LB score with increase of the val score? I see exactly the same trend, and my model with 0.80 val score perform worse on public LB than models having 0.75?
", "votes": 1, "replies": [] }, { "id": 445816, "author_name": "Moshel", "author_url": "", "post_date": "2018-12-27T04:59:57.613000", "content": "Far from being expert but my guess would be that the model became better at predicting the abundant classes, hence the val_loss improvement (which is not class avg) but it caused the f1 macro score (which is class avg) to drop. Very hard to avoid this, alas.... I actually over-sampled only in my training set and thinking back it might not be the correct strategy.
", "votes": 0, "replies": [] }, { "id": 445849, "author_name": "Wang Xinliang", "author_url": "", "post_date": "2018-12-27T05:49:14.833000", "content": "Hi Heng CherKeng. For \"F1@0.5\", what's the meaning of @?
", "votes": 0, "replies": [] }, { "id": 445851, "author_name": "hengck23", "author_url": "", "post_date": "2018-12-27T05:57:39.483000", "content": "F1 computed with threshold 0.5
", "votes": 0, "replies": [] }, { "id": 445872, "author_name": "Wang Xinliang", "author_url": "", "post_date": "2018-12-27T06:36:47.107000", "content": "Thank you~
", "votes": 0, "replies": [] }, { "id": 449629, "author_name": "Peterzhang", "author_url": "", "post_date": "2019-01-03T13:18:20.417000", "content": "How did you split data into train and val?
", "votes": 0, "replies": [] }, { "id": 453154, "author_name": "Miroslav Valan", "author_url": "", "post_date": "2019-01-09T18:45:12.310000", "content": "@hengck23 and @iafos just started to use external data and I am observing exactly the same F1 0.75-0.80 with resnet18 scores 0.53-0.51 on LB. Any advice?
", "votes": 0, "replies": [] } ] }, { "id": 449127, "author_name": "Dieter", "author_url": "", "post_date": "2019-01-02T17:35:35.170000", "content": "I have two quite different architectures both LB 0.6+
\n\n@trangle1302 If I would run in kernels I guess I could reach 0.55+ with them
", "votes": 8, "replies": [ { "id": 449545, "author_name": "", "author_url": "", "post_date": "2019-01-03T10:32:44.677000", "content": "", "votes": 0, "replies": [] }, { "id": 451713, "author_name": "souraj", "author_url": "", "post_date": "2019-01-07T14:32:08.343000", "content": "How to preprocess the images with 4 channels, when using pre-trained models. With 5-fold CV and 512 image size, and oversampling and using adam, I am not able to get past 0.50 :(
", "votes": 0, "replies": [] }, { "id": 451799, "author_name": "Dieter", "author_url": "", "post_date": "2019-01-07T17:17:02.440000", "content": "I also used external data for training but not the leak file. @sourajmishra I did not use 4 channels, only 3
", "votes": 1, "replies": [] }, { "id": 451809, "author_name": "souraj", "author_url": "", "post_date": "2019-01-07T17:55:33.340000", "content": "Thanks. Do you process to compress RBGY to 3 channel somehow, or just use RGB and ignore Y?
", "votes": 2, "replies": [] }, { "id": 453137, "author_name": "Dieter", "author_url": "", "post_date": "2019-01-09T18:17:36.133000", "content": "ignore Y
", "votes": 0, "replies": [] }, { "id": 454221, "author_name": "Dieter", "author_url": "", "post_date": "2019-01-11T09:10:19.143000", "content": "i wrote a quick summary here:
\n\nhttps://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/77300
", "votes": 1, "replies": [] } ] }, { "id": 444721, "author_name": "Duc Nguyen", "author_url": "", "post_date": "2018-12-24T16:34:49.177000", "content": "self-designed InceptionV3 (external data + image size 512x512): 0.577 single fold, 0.599 average of 3 folds
", "votes": 6, "replies": [ { "id": 444732, "author_name": "Kevin Zheng", "author_url": "", "post_date": "2018-12-24T16:59:56.207000", "content": "Great work! May I ask that did you use green channel mask attention?
", "votes": 0, "replies": [] }, { "id": 444765, "author_name": "Ashish Lal", "author_url": "", "post_date": "2018-12-24T18:51:09.973000", "content": "Very impressive. Can you drop some hints on your approach?
", "votes": 0, "replies": [] }, { "id": 444850, "author_name": "Duc Nguyen", "author_url": "", "post_date": "2018-12-25T01:31:23.903000", "content": "I did not use green channel mask attention. Instead, I design a new method to make the network attend on the rare labels.
", "votes": 6, "replies": [] }, { "id": 444866, "author_name": "Kevin Zheng", "author_url": "", "post_date": "2018-12-25T02:41:34.843000", "content": "Thanks for sharing!
", "votes": 0, "replies": [] }, { "id": 445307, "author_name": "Jun Liu", "author_url": "", "post_date": "2018-12-26T07:28:35.357000", "content": "May I ask which loss function you used for this report?
", "votes": 0, "replies": [] }, { "id": 445347, "author_name": "Duc Nguyen", "author_url": "", "post_date": "2018-12-26T08:49:05.817000", "content": "Hi Chris, I'm using binary cross-entropy. I tried to combine it with focal loss and f1 loss but pure binary cross-entropy gave me the best result.
", "votes": 1, "replies": [] }, { "id": 447099, "author_name": "Jun Liu", "author_url": "", "post_date": "2018-12-29T06:23:28.703000", "content": "That's just the opposite of my observation. I'm using self-designed resnet50 with focal loss got 0.574 single fold.
", "votes": 0, "replies": [] }, { "id": 447102, "author_name": "Duc Nguyen", "author_url": "", "post_date": "2018-12-29T06:32:33.840000", "content": "Did you tune the gamma and alpha value? I used the default set up, gamma=2, alpha=0.25
", "votes": 1, "replies": [] }, { "id": 447183, "author_name": "Jun Liu", "author_url": "", "post_date": "2018-12-29T10:18:20.943000", "content": "I have tried different settings for gamma and alpha, found that gamma=2, alpha=0.25 gives the best performance.
", "votes": 1, "replies": [] }, { "id": 450964, "author_name": "good good study", "author_url": "", "post_date": "2019-01-06T07:00:29.613000", "content": "Hi, @Chris J.Liu, by using normal ResNet50, how much LB do you get for single fold with focal loss?
", "votes": 0, "replies": [] } ] }, { "id": 444609, "author_name": "Jsato", "author_url": "", "post_date": "2018-12-24T11:15:39.450000", "content": "Could you give me some hints about your loss function??
", "votes": 1, "replies": [ { "id": 444613, "author_name": "Kevin Zheng", "author_url": "", "post_date": "2018-12-24T11:49:42.517000", "content": "Check this excellent kernel: https://www.kaggle.com/rejpalcz/best-loss-function-for-f1-score-metric. For me, soft f1 loss works well, some combination may help, on experiment.
", "votes": 4, "replies": [] }, { "id": 444814, "author_name": "Moshel", "author_url": "", "post_date": "2018-12-24T21:56:24.160000", "content": "I was under the impression that F1, being a macro metrics, works ok only on very large batches. What is the size of your batch?
", "votes": 0, "replies": [] }, { "id": 444882, "author_name": "Iafoss", "author_url": "", "post_date": "2018-12-25T04:09:05.297000", "content": "I have the same experience. On 256x256 with batches 64 and 128 F1 loss works. However, when I tried to move to 512x512 images the performance has degraded, even if I mixed focal and F1 loss. With batch size 16 training didn't really work.
", "votes": 1, "replies": [] }, { "id": 444896, "author_name": "Kevin Zheng", "author_url": "", "post_date": "2018-12-25T05:00:12.140000", "content": "Now my batch size is 128, the performance better than 64.
", "votes": 0, "replies": [] }, { "id": 448892, "author_name": "GhMa", "author_url": "", "post_date": "2019-01-02T10:26:51.957000", "content": "definitely larger batch could represent the true distribution of whole dataset, but I wonder the image size u r using now, is it possible to get LB 0.58 + with size 256 * 256 ? \nhope for ur answer hh
", "votes": 0, "replies": [] } ] }, { "id": 449449, "author_name": "sx318", "author_url": "", "post_date": "2019-01-03T06:47:03.997000", "content": "I have a Resnet50, 512 x 512 image size, external data, and some test data leak, gives a LB 0.569
", "votes": 2, "replies": [ { "id": 451811, "author_name": "souraj", "author_url": "", "post_date": "2019-01-07T17:57:29.420000", "content": "Do you get this score with Binary cross entropy, or with some custom loss?
", "votes": 1, "replies": [] }, { "id": 451867, "author_name": "sx318", "author_url": "", "post_date": "2019-01-07T19:51:45.377000", "content": "With BCE, I got higher validation score, but poor LB score, but with focal loss, I got a much better LB score
", "votes": 0, "replies": [] }, { "id": 452012, "author_name": "souraj", "author_url": "", "post_date": "2019-01-08T03:07:08.417000", "content": "Thank You. With the focal loss, I have tried Adam with staring lr 0.001 and then decreasing it every 10 epochs with gamma 0.1 to 0.5. But it gets stuck on around 0.3 on LB. Can you suggest some strategy for this.
", "votes": 0, "replies": [] }, { "id": 452593, "author_name": "sx318", "author_url": "", "post_date": "2019-01-08T23:34:53.607000", "content": "I started with 2e-5, and decreasing every 10 epochs, trained with total 30 epochs.
", "votes": 0, "replies": [] } ] }, { "id": 444755, "author_name": "YALICKJ", "author_url": "", "post_date": "2018-12-24T18:24:37.677000", "content": "0.564, 512x512, resnet34 with HPAv18 external data, single fold. I'm running a 5-fold experiment, but it's taking forever to finish with the external data.
", "votes": 2, "replies": [ { "id": 444890, "author_name": "Kevin Zheng", "author_url": "", "post_date": "2018-12-25T04:41:52.730000", "content": "Good work!
", "votes": 0, "replies": [] }, { "id": 444913, "author_name": "hengck23", "author_url": "", "post_date": "2018-12-25T06:27:33.850000", "content": "\n\n\n\n\n0.564, 512x512, resnet34 with HPAv18 external data, single fold
\n
Is this with TTA or just on original input images? With is the threshold you are using?
", "votes": 0, "replies": [] }, { "id": 445093, "author_name": "YALICKJ", "author_url": "", "post_date": "2018-12-25T15:54:00.843000", "content": "With TTA and adjusted thresholds fitted on the validation set.
", "votes": 1, "replies": [] } ] }, { "id": 446094, "author_name": "Jun Liu", "author_url": "", "post_date": "2018-12-27T13:35:16.253000", "content": "Hi, @Kevin Zheng, Do you train from scratch? Can you share more details? Like loss function, image size, optimizer, init lr, etc.
", "votes": 1, "replies": [] }, { "id": 445841, "author_name": "Wang Xinliang", "author_url": "", "post_date": "2018-12-27T05:35:15.313000", "content": "Thanks, can you share the lr schedule?
", "votes": -1, "replies": [ { "id": 445914, "author_name": "Kevin Zheng", "author_url": "", "post_date": "2018-12-27T07:48:58.197000", "content": "SGD, lr start with 0.1, reduce lr by 0.2 when the loss stop decrease in 8 epochs.
", "votes": 2, "replies": [] }, { "id": 446710, "author_name": "Borys Tymchenko", "author_url": "", "post_date": "2018-12-28T14:53:35.627000", "content": "Wow!\nHow many epochs does it take?
", "votes": 0, "replies": [] }, { "id": 447254, "author_name": "Kevin Zheng", "author_url": "", "post_date": "2018-12-29T13:37:57.153000", "content": "80~100
", "votes": 1, "replies": [] } ] }, { "id": 453564, "author_name": "Cape", "author_url": "", "post_date": "2019-01-10T11:28:18.270000", "content": "I am curious about what type of resnet you are using?
", "votes": 0, "replies": [ { "id": 453581, "author_name": "Miroslav Valan", "author_url": "", "post_date": "2019-01-10T12:25:17.020000", "content": "Same here. People often state they are using self-defined well-known architecture but I have seen cases where this stands for changing the output layer to nets that are just inspired by well known architectures (residual, inception block, etc.). I assume head is self defined but none of the earlier layers are changed (perhaps in conv1 3=>4 channels).
", "votes": 0, "replies": [] } ] }, { "id": 451791, "author_name": "Gian12", "author_url": "", "post_date": "2019-01-07T16:51:27.943000", "content": "I use a Resnet34, 512 x 512 image size, RGBY, external data, no leaked data, single fold with data augmentation and TTA, upsample of the less frequent classes and it gives a LB 0.568
", "votes": 0, "replies": [] }, { "id": 451337, "author_name": "Phil Butcher", "author_url": "", "post_date": "2019-01-06T22:15:54.170000", "content": "0.577
\n\nResnet50, 512x512 rgb images, oversampling rare classes, simple data augmentation, external data, 5-fold with TTA, BCE loss
\n\nThe leak boosts me to 0.588. Oversampling dramatically reduced the boost I was getting from the leak.
", "votes": 0, "replies": [ { "id": 451397, "author_name": "Jun Liu", "author_url": "", "post_date": "2019-01-07T02:43:52.157000", "content": "cool, thx for ur sharing. Do u use leak126 ?
", "votes": 0, "replies": [] }, { "id": 451406, "author_name": "Peterzhang", "author_url": "", "post_date": "2019-01-07T03:01:58.177000", "content": "Thanks for sharing, May I ask did you try rgby? what's the result?
", "votes": 0, "replies": [] }, { "id": 451416, "author_name": "Phil Butcher", "author_url": "", "post_date": "2019-01-07T03:47:40.093000", "content": "I've been using leak259
", "votes": 0, "replies": [] }, { "id": 451417, "author_name": "Phil Butcher", "author_url": "", "post_date": "2019-01-07T03:49:21.777000", "content": "I've tried rgby several times with slightly worse results than rgb. I don't have a good understanding why that is.
", "votes": 0, "replies": [] }, { "id": 451533, "author_name": "good good study", "author_url": "", "post_date": "2019-01-07T08:34:41.310000", "content": "Hi, Do you use same threshold for all classes? or choose threshold from validation set?
", "votes": 0, "replies": [] } ] }, { "id": 448773, "author_name": "Peterzhang", "author_url": "", "post_date": "2019-01-02T04:18:18.413000", "content": "Can you guys tell me the way of split the trainval? my val is not robust, the score on lb didn't increase even thought local increases by a large margin
", "votes": 0, "replies": [] }, { "id": 447844, "author_name": "Sabbir Ahmed", "author_url": "", "post_date": "2018-12-30T18:02:19.110000", "content": "ResNet18 is overfitting even after using external data and dropout(0.5). Can Anyone give any suggestion to avoid this problem??
", "votes": 0, "replies": [ { "id": 447881, "author_name": "Shai", "author_url": "", "post_date": "2018-12-30T18:56:13.693000", "content": "Few things to deal with overfitting:
\n\nMost robust way is to use a validation set that is not very similar to train set but still represents test set. Usually, when your training loss is getting lower significantly compared to validation loss, it's a sign of overfitting. You may want to use early stopping or use an earlier checkpoint in that case when validation loss starts increasing. However, sometimes maximising competition validation metric helps, even if validation loss increases. But for this competition, there are similar images in train set. So your validation metric(e.g. F-1) may increase with your training, but it is because your validation set contains similar images like train, and you will not know if your model is overfitting.
Be careful when using Dropout. If there are certain features present in your data that doesn't represent general feature but overfitting feature, dropout may force the network to learn overfitting features to decrease loss. (you can think of an overfitting feature something that is not protein in an image, but something irrelevant present in a given class). Other regularisation techniques help in complementary to dropout.
Fun fact: Neural networks are like babies. You can spoil them really easily if you force to teach them everything or leave them alone to learn as they will. Teach them slowly and patiently.
\n\nHave fun!!
", "votes": 9, "replies": [] }, { "id": 447883, "author_name": "Moshel", "author_url": "", "post_date": "2018-12-30T19:07:39.980000", "content": "Augmentation is your friend! Try flipping and rotating for start. The more augmentation you use the more robust your network will be, as it won't see \"the same\" images all the time. Make sure the original images still show once in a while...
", "votes": 2, "replies": [] }, { "id": 447886, "author_name": "Moshel", "author_url": "", "post_date": "2018-12-30T19:09:22.767000", "content": "Actually it's the other way around. Babies are like neural networks :)
", "votes": 7, "replies": [] }, { "id": 448544, "author_name": "Borys Tymchenko", "author_url": "", "post_date": "2019-01-01T13:13:33.023000", "content": "Actually, too wild augmentation can hurt you!
", "votes": 0, "replies": [] }, { "id": 449034, "author_name": "Antoine", "author_url": "", "post_date": "2019-01-02T15:08:52.120000", "content": "Hi @sgalib,
\n\nYour 1st point is actually really interesting... How can you make sure that the model is not overfitting if you can't use early stopping ? and can't really trust your local CV F1 score and loss ? Is the solution to manually design this val set to avoid being too similar to train ? Or are you able to completely avoid overfitting with regularization ?
", "votes": 0, "replies": [] }, { "id": 449228, "author_name": "Shai", "author_url": "", "post_date": "2019-01-02T20:27:48.487000", "content": "Interesting question. I hope I understood your question correctly.
\n\nIf you can't use early stopping: I suppose every neural network will overfit with favorable hyper-parameter selection if you train them for very long period of time. This is particularly true if the dataset is small and your neural network architecture is very deep/complex.So you have to stop somewhere depending on your training and validation metric.
If you can't trust local validation metric: I guess validation metric is your only friend while training. The more effort you put to construct a validation set, the more reward you get. Making a good validation set can be effort-some. You can explore some clustering algorithms to divide your dataset, so that same clusters from same class are not present in both of your tain-val sets. This is one approach, there are tons of others. Fold-wise cross-validation will also give you a hint on how much varience you have between folds, and potentially in test set also. So yes, it's a semi-manual process.
In this specific competition, dataset is rather small even with external data. State-of-the art imagenet architectures overfit pretty easily. Regularization reduces overfit but it also suffers from underfitting. So there is a trade-off. I have mostly played with lighter architectures which are less likely to overfit. And early-stopping as well.
Last but not the least, suppose you have constructed a good validation set, played to reduce overfit but still not satisfied with your local to LB performance, then check the the ghost in some other places. Study competition metric and have a sense how it might be affecting, because your validation set still may not represent Public LB set!
Hope this helps!
", "votes": 5, "replies": [] }, { "id": 449326, "author_name": "Antoine", "author_url": "", "post_date": "2019-01-03T01:02:38.440000", "content": "Thanks a lot for your answer @sgalib !
\n\nYes I think you got my point. As you said \"there are similar images in train set.\" so when creating your validation you might end up with similar data in train and val, and so even though you overfit, your val loss might still decrease and you won't be able to use early stopping. At least that's the problem I ran into with one of my models. So I wanted to know if you had any advice for this particular case. I guess I should have spent more time on making a proper CV earlier in the competition. I agree with using light models. Where you able to get a stable LB / CV F1 difference ?
", "votes": 1, "replies": [] }, { "id": 449802, "author_name": "Shai", "author_url": "", "post_date": "2019-01-03T19:17:22.337000", "content": "Aha... I got your point. If I do not have time to do CV, a quick fix in that case I would do is train a few models with random validation split, average/ensemble predictions. Usually you won't be disappointed! :)
", "votes": 1, "replies": [] } ] }, { "id": 445226, "author_name": "pete", "author_url": "", "post_date": "2018-12-26T02:22:41.770000", "content": "I got 0.532 without specifically treating the leak as per the posted leak csv. But for this, I did use the HPA data which had the leak data in it. I did not specifically exclude this. \nAdding the leak csv to this boosted the score a bunch.
\n\nHave you tried adding leak info to your submission? If so, what boost did you get?
", "votes": 0, "replies": [ { "id": 445260, "author_name": "Moshel", "author_url": "", "post_date": "2018-12-26T04:20:48.353000", "content": "The difference is smaller and smaller as my models improve. Currently its 0.555->0.566
", "votes": 0, "replies": [] }, { "id": 445281, "author_name": "Kevin Zheng", "author_url": "", "post_date": "2018-12-26T05:58:17.600000", "content": "boost 0.006 with brain's 126 leak
", "votes": 1, "replies": [] }, { "id": 445343, "author_name": "pete", "author_url": "", "post_date": "2018-12-26T08:45:38.897000", "content": "Amazing. Looks like you got a very good answer for those two classes without help, just by using HPA. To be honest, I feel that a lot of my score is due to random boosts and that randomness is playing a big part here.
", "votes": -1, "replies": [] }, { "id": 445425, "author_name": "hengck23", "author_url": "", "post_date": "2018-12-26T12:30:32.073000", "content": "@Kevin Zheng
\n\nthere are two leak:
\n\nhttps://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/73395
\n\nhttps://storage.googleapis.com/kaggle-forum-message-attachments/433234/10820/overlap.py
\n\nhttps://www.kaggle.com/c/human-protein-atlas-image-classification/discussion/72534
\n\nTestEtraMatchingUnder_259_R14_G12_B10.csv (18.92 KB)
\n\nfor the same input test image, leak126 and leak256 sometimes disagree on the labels. i think leak259 is more correct
", "votes": 0, "replies": [] }, { "id": 445429, "author_name": "Kevin Zheng", "author_url": "", "post_date": "2018-12-26T12:44:14.290000", "content": "Thanks, Heng. Learn a lot from you. Happy to see you in the challenge!
", "votes": 0, "replies": [] }, { "id": 449348, "author_name": "Breck", "author_url": "", "post_date": "2019-01-03T02:16:21.337000", "content": "Thanks @hengck23. For the leaked data, I downloaded \"TestEtraMatchingUnder259R14G12B10.csv\" but don't see any labels. Is that left as an exercise for the reader?
", "votes": 0, "replies": [] } ] }, { "id": 444883, "author_name": "Kulbear", "author_url": "", "post_date": "2018-12-25T04:10:01.817000", "content": "0.576, 512 image size, BNInception, with external data.
", "votes": 0, "replies": [ { "id": 444897, "author_name": "Kevin Zheng", "author_url": "", "post_date": "2018-12-25T05:07:39.547000", "content": "Sounds like Inception help capture multi-scale information?
", "votes": 0, "replies": [] }, { "id": 444898, "author_name": "Kulbear", "author_url": "", "post_date": "2018-12-25T05:17:59.457000", "content": "Not sure, I used several models (inception or non-inception), they have similar performance.
", "votes": 0, "replies": [] }, { "id": 445318, "author_name": "Jun Liu", "author_url": "", "post_date": "2018-12-26T07:48:13.873000", "content": "Can you share more details? Thx!
", "votes": 0, "replies": [] }, { "id": 445432, "author_name": "hengck23", "author_url": "", "post_date": "2018-12-26T12:53:51.743000", "content": "@ManyFoldCV
\n\n\" I used several models (inception or non-inception), they have similar performance.\"
\n\nIn my experiments, they are same in F1 score. But inception has more correct multi-label images and more wrong false positive.
\n\nresnet has less correct multi-label images and more less false positive.
", "votes": 1, "replies": [] }, { "id": 446105, "author_name": "Dmytro Panchenko", "author_url": "", "post_date": "2018-12-27T13:59:35.807000", "content": "Could you please share which implementation of BN-Inception do you use? Is it from Cadene repo with PyTorch pretrained models?
", "votes": 1, "replies": [] }, { "id": 446121, "author_name": "hengck23", "author_url": "", "post_date": "2018-12-27T14:42:26.973000", "content": "Yes, i am using pretrain model from Cadene repo
", "votes": 2, "replies": [] }, { "id": 446163, "author_name": "", "author_url": "", "post_date": "2018-12-27T16:27:29.610000", "content": "", "votes": 0, "replies": [] }, { "id": 446164, "author_name": "Kulbear", "author_url": "", "post_date": "2018-12-27T16:28:13.957000", "content": "The one from Cadene repo @hokmund
", "votes": 1, "replies": [] }, { "id": 446165, "author_name": "Kulbear", "author_url": "", "post_date": "2018-12-27T16:28:34.050000", "content": "Check my discussion post :D @chrisluu
", "votes": 0, "replies": [] }, { "id": 447888, "author_name": "FlYM", "author_url": "", "post_date": "2018-12-30T19:17:43.150000", "content": "@hengck23\n'In my experiments, they are same in F1 score. But inception has more correct multi-label images and more wrong false positive.\nresnet has less correct multi-label images and more less false positive.'
\n\nI think this is an important observation. I reached the same conclusion. I am using mostly resnets.
", "votes": 2, "replies": [] }, { "id": 447957, "author_name": "Jun Liu", "author_url": "", "post_date": "2018-12-30T22:38:42.107000", "content": "Hi @arnaurm, Do you have any ideas about the reason?
", "votes": 0, "replies": [] }, { "id": 448151, "author_name": "FlYM", "author_url": "", "post_date": "2018-12-31T10:56:14.340000", "content": "No, not really
", "votes": 0, "replies": [] }, { "id": 448429, "author_name": "hengck23", "author_url": "", "post_date": "2019-01-01T06:14:30.673000", "content": "Another observation:
\n\nI compute f1 for single label and multi label for validation during the metering in the training iterations. If i sample more multi label train images in a batch, i can see both validation f1 falls.
\n\nHowever, public lb is significantly worsen.
", "votes": 0, "replies": [] } ] }, { "id": 444737, "author_name": "Florian Muellerklein", "author_url": "", "post_date": "2018-12-24T17:13:13.520000", "content": "0.545 resnet34 with naive 0.5 thresholds
", "votes": 0, "replies": [ { "id": 444865, "author_name": "Kevin Zheng", "author_url": "", "post_date": "2018-12-25T02:41:00.983000", "content": "Thanks for reply. Now I use multi thresholds searching from val dataset. I will try single threshold.
", "votes": 0, "replies": [] }, { "id": 444914, "author_name": "hengck23", "author_url": "", "post_date": "2018-12-25T06:29:12.733000", "content": "@Florian Muellerklein
\n\nlower threshold (e.g. 0.3 for my case) seems to give better results. Did you compare results for other threshold values or adaptive thresholds for different classes?
", "votes": 0, "replies": [] }, { "id": 445072, "author_name": "Florian Muellerklein", "author_url": "", "post_date": "2018-12-25T14:51:20.017000", "content": "I don't want to choose thresholds just based only on the public leaderboard performance. I've done some local threshold search with my validation sets but they do worse on the public leaderboard.
", "votes": 2, "replies": [] } ] }, { "id": 444654, "author_name": "owruby", "author_url": "", "post_date": "2018-12-24T13:50:08.520000", "content": "How image size do you use?\nI used self-designed seresnet50 with 512x512, Public LB: 0.539
", "votes": 0, "replies": [ { "id": 444710, "author_name": "Kevin Zheng", "author_url": "", "post_date": "2018-12-24T16:09:29.317000", "content": "256x256
", "votes": 1, "replies": [] } ] }, { "id": 444598, "author_name": "tkuanlun350", "author_url": "", "post_date": "2018-12-24T10:48:19.673000", "content": "I only got 0.517 without leak and additional data using backbone resnet50
", "votes": 0, "replies": [ { "id": 444608, "author_name": "Kevin Zheng", "author_url": "", "post_date": "2018-12-24T11:08:36.167000", "content": "Thanks for reply. Focus on your loss function design maybe help. Good luck.
", "votes": 3, "replies": [] }, { "id": 444659, "author_name": "tkuanlun350", "author_url": "", "post_date": "2018-12-24T14:12:04.997000", "content": "I am curious about your image size. I think the reason why using resnet18 is to fit the larger image ?
", "votes": 0, "replies": [] }, { "id": 444709, "author_name": "Kevin Zheng", "author_url": "", "post_date": "2018-12-24T16:09:12.723000", "content": "256x256
", "votes": 2, "replies": [] }, { "id": 444712, "author_name": "tkuanlun350", "author_url": "", "post_date": "2018-12-24T16:15:08.663000", "content": "Amazing performance ! Do you use external data ?
", "votes": 0, "replies": [] }, { "id": 444716, "author_name": "Kevin Zheng", "author_url": "", "post_date": "2018-12-24T16:25:12.693000", "content": "Yes, I use HPAv18 external data.
", "votes": 0, "replies": [] }, { "id": 444722, "author_name": "tkuanlun350", "author_url": "", "post_date": "2018-12-24T16:45:16.100000", "content": "Just found out the author of this kernel can achieve 0.577 with external data (inception V3)
", "votes": 2, "replies": [] }, { "id": 444733, "author_name": "Kevin Zheng", "author_url": "", "post_date": "2018-12-24T17:00:18.223000", "content": "Thanks!
", "votes": 0, "replies": [] }, { "id": 445312, "author_name": "Jun Liu", "author_url": "", "post_date": "2018-12-26T07:35:04.487000", "content": "That's amazing! Do you try a larger image size? What about the results? For my observation, the larger image size will help improve LB score.
", "votes": 0, "replies": [] }, { "id": 445387, "author_name": "Kevin Zheng", "author_url": "", "post_date": "2018-12-26T10:59:02.143000", "content": "I will try. GPU is my bottleneck now...
", "votes": 0, "replies": [] }, { "id": 448210, "author_name": "hengck23", "author_url": "", "post_date": "2018-12-31T13:18:58.480000", "content": "@tkuanlun350
\n\n\"I only got 0.517 without leak and additional data using backbone resnet50\"
\n\n0.517 is without hpa external data?
", "votes": 0, "replies": [] }, { "id": 448230, "author_name": "tkuanlun350", "author_url": "", "post_date": "2018-12-31T14:42:55.323000", "content": "Yes, however 0.517 for single fold without external and TTA is just one-time luck (cannot get external data work at that time). I cannot reproduce the performance for the same setting. After some tuning and bug fixing, my current single fold model with external data ~0.58( with TTA )
", "votes": 0, "replies": [] }, { "id": 448418, "author_name": "Kevin Zheng", "author_url": "", "post_date": "2019-01-01T05:15:12.977000", "content": "Same as you. Now my single fold model with external data get lb 0.580. May I ask you how you choose threshold?
", "votes": 0, "replies": [] }, { "id": 449160, "author_name": "Abhilash Awasthi", "author_url": "", "post_date": "2019-01-02T18:19:31.660000", "content": "You can decide on different threshold for each class which can be deduced from the validation set.
", "votes": 0, "replies": [] }, { "id": 449699, "author_name": "tkuanlun350", "author_url": "", "post_date": "2019-01-03T16:08:38.990000", "content": "@hdzheng\nI am just using the same threshold for every class (based on LB). Threshold selection is not working very well in my current setting :(
", "votes": 0, "replies": [] }, { "id": 449709, "author_name": "Zhijian Li", "author_url": "", "post_date": "2019-01-03T16:25:00.563000", "content": "@tkuanlun350,\nHow much did you improve the LB by fine-tuning the thresholds?\nmy best single model can get 0.59 on LB with the optimal threshold from 5-fold validation.\nSo I am wondering if I should change the threshold or not.
", "votes": 0, "replies": [] } ] }, { "id": 448186, "author_name": "", "author_url": "", "post_date": "2018-12-31T11:58:12.313000", "content": "", "votes": 2, "replies": [] } ], "raw_markdown_by_id": { "444589": "Hi, everyone, I am curious what is your best single model on public LB without any leak?\nPlease post your results here if you would like to share.\nFor me:\nLB 0.580 for self-designed resnet18 with external data (single fold result)", "445783": "my weird results on 512x512 images. All results below is single fold without TTA, without ensemble\n\n\n\n\nresnet18 on train = kaggle+external(HPAv18RBGY_wodpl):\n\nlocal lb : 0.755 (F1@0.5)\n\npublic lb: 0.551 (F1@0.3) without leak\n\npublic lb: 0.550 (F1@0.3) overwrite prediction with leak259\n\n\n---\n\n\ninception-bn on train = kaggle+external(HPAv18RBGY_wodpl):\n\nlocal lb : 0.785 (F1@0.5)\n\npublic lb: 0.530 (F1@0.3) without leak\n\npublic lb: 0.530 (F1@0.3) overwrite prediction with leak259\n\n\n---\n\n\nresnet34 on train = kaggle+external(HPAv18RBGY_wodpl):\n\nlocal lb : 0.795 (F1@0.5)\n\npublic lb: 0.511 (F1@0.3) overwrite prediction with leak259\n\n \n", "449127": "I have two quite different architectures both LB 0.6+\n\n- 5-fold\n- rgb\n- 512x512\n\n@trangle1302 If I would run in kernels I guess I could reach 0.55+ with them", "444721": "self-designed InceptionV3 (external data + image size 512x512): 0.577 single fold, 0.599 average of 3 folds", "444609": "Could you give me some hints about your loss function??", "449449": "I have a Resnet50, 512 x 512 image size, external data, and some test data leak, gives a LB 0.569", "444755": "0.564, 512x512, resnet34 with HPAv18 external data, single fold. I'm running a 5-fold experiment, but it's taking forever to finish with the external data.", "446094": "Hi, @Kevin Zheng, Do you train from scratch? Can you share more details? Like loss function, image size, optimizer, init lr, etc.", "445841": "Thanks, can you share the lr schedule?", "453564": "I am curious about what type of resnet you are using?", "451791": "I use a Resnet34, 512 x 512 image size, RGBY, external data, no leaked data, single fold with data augmentation and TTA, upsample of the less frequent classes and it gives a LB 0.568", "451337": "0.577\n\nResnet50, 512x512 rgb images, oversampling rare classes, simple data augmentation, external data, 5-fold with TTA, BCE loss\n\nThe leak boosts me to 0.588. Oversampling dramatically reduced the boost I was getting from the leak.", "448773": "Can you guys tell me the way of split the trainval? my val is not robust, the score on lb didn't increase even thought local increases by a large margin", "447844": "ResNet18 is overfitting even after using external data and dropout(0.5). Can Anyone give any suggestion to avoid this problem??", "445226": "I got 0.532 without specifically treating the leak as per the posted leak csv. But for this, I did use the HPA data which had the leak data in it. I did not specifically exclude this. \nAdding the leak csv to this boosted the score a bunch.\n\nHave you tried adding leak info to your submission? If so, what boost did you get?", "444883": "0.576, 512 image size, BNInception, with external data.", "444737": "0.545 resnet34 with naive 0.5 thresholds", "444654": "How image size do you use?\nI used self-designed seresnet50 with 512x512, Public LB: 0.539", "444598": "I only got 0.517 without leak and additional data using backbone resnet50", "448186": "" } }