Hello ! \nSince this is definitely an odd problem I am very curious about what approaches are other kagglers using to solve it.
\n\nI'll start with myself : I am using an average of multiple architectures of deep CNNs ( like the kernel I shared ) applied to small crops of the training images, though I am not really sure what the models are learning exactly and I have a huge score mismatch between Validation and LB that I can't explain.
\n\nAre there people using other types of methods ?
", "messages": [ { "id": 262984, "postDate": "2017-12-28T14:40:10.330Z", "content": "Hello ! \nSince this is definitely an odd problem I am very curious about what approaches are other kagglers using to solve it.
\n\nI'll start with myself : I am using an average of multiple architectures of deep CNNs ( like the kernel I shared ) applied to small crops of the training images, though I am not really sure what the models are learning exactly and I have a huge score mismatch between Validation and LB that I can't explain.
\n\nAre there people using other types of methods ?
", "rawMarkdown": "Hello ! \nSince this is definitely an odd problem I am very curious about what approaches are other kagglers using to solve it.\n\nI'll start with myself : I am using an average of multiple architectures of deep CNNs ( like the kernel I shared ) applied to small crops of the training images, though I am not really sure what the models are learning exactly and I have a huge score mismatch between Validation and LB that I can't explain.\n\nAre there people using other types of methods ? \n", "votes": 10 }, { "id": 263078, "postDate": "2017-12-28T21:37:47.893Z", "content": "I am seeing similar things. (validation 97, Test 80) My theory is that the modified images in the test set is dragging down the total score. I am using a variety of Xfr learning and averaging. I found that center patches seem to work the best.
", "rawMarkdown": "I am seeing similar things. (validation 97, Test 80) My theory is that the modified images in the test set is dragging down the total score. I am using a variety of Xfr learning and averaging. I found that center patches seem to work the best.", "votes": 1, "replies": [ { "id": 263093, "postDate": "2017-12-28T22:04:40.120Z", "content": "What is Xfr learning ?
", "rawMarkdown": "What is Xfr learning ?", "votes": 1 }, { "id": 263096, "postDate": "2017-12-28T22:12:20.993Z", "content": "Transfer learning. You take a pretrained deep network (ResNet, Inception) and attach your own end layer. https://machinelearningmastery.com/transfer-learning-for-deep-learning/. Useful when you don't have a lot of training data.
", "rawMarkdown": " Transfer learning. You take a pretrained deep network (ResNet, Inception) and attach your own end layer. https://machinelearningmastery.com/transfer-learning-for-deep-learning/. Useful when you don't have a lot of training data.", "votes": -3 }, { "id": 263099, "postDate": "2017-12-28T22:17:07.580Z", "content": "Oh Yeah I get it ! I am doing the same
", "rawMarkdown": "Oh Yeah I get it ! I am doing the same " }, { "id": 268023, "postDate": "2018-01-13T02:54:53.910Z", "content": "does it work in this task? i thought that it only works for similiar tasks. - -|
", "rawMarkdown": "does it work in this task? i thought that it only works for similiar tasks. - -|", "votes": -1 } ] }, { "id": 263495, "postDate": "2017-12-30T13:21:20.403Z", "content": "I split all training images into 512x512 then resized all to 256x256 (similar approach used to feed test into the model). Getting > 90% validation but on test goes almost half. I am assuming it's due to altered images... I tried setting all altered images to one category and the score goes down just a little.
", "rawMarkdown": "I split all training images into 512x512 then resized all to 256x256 (similar approach used to feed test into the model). Getting > 90% validation but on test goes almost half. I am assuming it's due to altered images... I tried setting all altered images to one category and the score goes down just a little.", "votes": 1 }, { "id": 263384, "postDate": "2017-12-30T00:27:15.927Z", "content": "CVxTz in your approach How can you be sure that your model isn't just detecting objects that are same in training data and testing data...
", "rawMarkdown": "CVxTz in your approach How can you be sure that your model isn't just detecting objects that are same in training data and testing data... ", "votes": 1, "replies": [ { "id": 263486, "postDate": "2017-12-30T12:29:57.480Z", "content": "Images in the train are not in the validation set.
", "rawMarkdown": "Images in the train are not in the validation set." }, { "id": 263493, "postDate": "2017-12-30T13:17:43.923Z", "content": "For this to be the case, there would have to be objects specific to photos taken on a single phone model. I.e. All Samsung Galaxy S6 photos having dogs in them.
", "rawMarkdown": "For this to be the case, there would have to be objects specific to photos taken on a single phone model. I.e. All Samsung Galaxy S6 photos having dogs in them." } ] }, { "id": 262992, "postDate": "2017-12-28T15:03:28.643Z", "content": "Similar approach but no ensembl model yet.
", "rawMarkdown": "Similar approach but no ensembl model yet. ", "votes": 1 }, { "id": 263235, "postDate": "2017-12-29T11:30:53.453Z", "content": "We have the same issue. There might be another case, which is, in the test dataset, all the images are captured from another devices (but the models are same). So there is a high possibility that deep cnns are learning the device specific PRNU noises.
", "rawMarkdown": "We have the same issue. There might be another case, which is, in the test dataset, all the images are captured from another devices (but the models are same). So there is a high possibility that deep cnns are learning the device specific PRNU noises.\n ", "votes": 2, "replies": [ { "id": 263238, "postDate": "2017-12-29T11:36:00Z", "content": "I had the same doubts so I asked this question in the Presentation thread but the organizers did not answer
", "rawMarkdown": "I had the same doubts so I asked this question in the Presentation thread but the organizers did not answer " }, { "id": 265300, "postDate": "2018-01-05T04:47:09.330Z", "content": "Have any thoughts about improving this situation?
", "rawMarkdown": "Have any thoughts about improving this situation?" }, { "id": 265336, "postDate": "2018-01-05T08:39:46.087Z", "rawMarkdown": "", "isDeleted": true }, { "id": 265354, "postDate": "2018-01-05T09:57:29.353Z", "content": "I don't think purely ensembling might help a lot. The variance between test set and validation set is too large.
", "rawMarkdown": "I don't think purely ensembling might help a lot. The variance between test set and validation set is too large." } ] }, { "id": 263789, "postDate": "2017-12-31T22:35:45.113Z", "content": "I've been running some tests on data \"out in the wild\" (images from the web). Even though my current model has a 99% validation accuracy, these get correctly identified around 20-30% of the time.
\n\nAs another test, I sent 10 images from the training set through \"google hangouts\" and none of the resulting images get correctly classified. Looking at the jpeg headers for these failing images, I notice that the quality is reduced to 90%. However, if I perform the same operation manually (Image.save), I still can identify the resulting images.
\n\nI have an old Note 3 and about 90% of these images get correctly classified unless they are sent through google hangout.
\n\nI suspect that we probably need to augment the training set with additional images from other sources or reprocessed training files. too bad ImageDataGenerator doesn't have a \"Google Hangouts\" operator ;)
", "rawMarkdown": "I've been running some tests on data \"out in the wild\" (images from the web). Even though my current model has a 99% validation accuracy, these get correctly identified around 20-30% of the time. \n\nAs another test, I sent 10 images from the training set through \"google hangouts\" and none of the resulting images get correctly classified. Looking at the jpeg headers for these failing images, I notice that the quality is reduced to 90%. However, if I perform the same operation manually (Image.save), I still can identify the resulting images.\n\nI have an old Note 3 and about 90% of these images get correctly classified unless they are sent through google hangout.\n\nI suspect that we probably need to augment the training set with additional images from other sources or reprocessed training files. too bad ImageDataGenerator doesn't have a \"Google Hangouts\" operator ;)", "votes": -2 }, { "id": 263301, "postDate": "2017-12-29T17:19:41.973Z", "content": "Does anyone have any advice for processing the JPGs in R to get them to a 'model ready' state?
", "rawMarkdown": "Does anyone have any advice for processing the JPGs in R to get them to a 'model ready' state?", "votes": -1 }, { "id": 265695, "postDate": "2018-01-06T10:10:50.987Z", "content": "Hi All,\nI am doing transfer learning for RESNET50. I have modified FC layers-
\n\nx = resnet_model.output\nx = Flatten()(x)\nx = Dropout(0.3)(x)\nx = Dense(1024,name='FCL1_1024')(x)\n#x = BatchNormalization()(x)\nx = Dropout(0.3)(x)\nx = Dense(num_class,activation ='sigmoid',name='Class_out')(x)\nI have my own patch generation code and random_transform. But even after 20 epochs I am not seeing any improvement in val_loss/val_acc --see in the image-
\n\nPlz suggest, what could be the reason for this.
", "rawMarkdown": "Hi All,\nI am doing transfer learning for RESNET50. I have modified FC layers-\n\n x = resnet_model.output\n\tx = Flatten()(x)\n\tx = Dropout(0.3)(x)\n\tx = Dense(1024,name='FCL1_1024')(x)\n\t#x = BatchNormalization()(x)\n\tx = Dropout(0.3)(x)\n\tx = Dense(num_class,activation ='sigmoid',name='Class_out')(x)\n\nI have my own patch generation code and random_transform. But even after 20 epochs I am not seeing any improvement in val_loss/val_acc --see in the image-\n\nPlz suggest, what could be the reason for this.\n", "replies": [ { "id": 265709, "postDate": "2018-01-06T11:42:46.963Z", "rawMarkdown": "", "isDeleted": true }, { "id": 265721, "postDate": "2018-01-06T12:23:58.950Z", "content": "Just guessing as your splitting code is not there.. Have you shuffled the data?
", "rawMarkdown": "Just guessing as your splitting code is not there.. Have you shuffled the data?" }, { "id": 265731, "postDate": "2018-01-06T13:17:50.327Z", "content": "You're using sigmoid, which is good for binary classification, not multiclass. Change to:
\n\nx = Dense(num_class,activation =\"softmax\",name=\"Class_out\")(x)
I'm using a mix of features + a random forest classifier, which has the same mismatch between validation and LB, but at a much lower score. Just to get a better idea of approach preferences, I'm curious if anyone in the top 25 is not using a deep learning model?
", "rawMarkdown": "I'm using a mix of features + a random forest classifier, which has the same mismatch between validation and LB, but at a much lower score. Just to get a better idea of approach preferences, I'm curious if anyone in the top 25 is not using a deep learning model?", "replies": [ { "id": 264514, "postDate": "2018-01-03T09:22:59.630Z", "content": "\n\n\nI'm curious if anyone in the top 25 is not using a deep learning model?
\n
Yes, almost all of them.
\n\n\n\n\nnot
\n
Oh. :D
", "rawMarkdown": "> I'm curious if anyone in the top 25 is not using a deep learning model?\n\nYes, almost all of them. \n\n> not \n\nOh. :D", "votes": -1 } ] }, { "id": 263864, "postDate": "2018-01-01T11:55:20.510Z", "content": "Hi All,\nI am facing an issue, I have started training RESNET18 from scratch. I have written my own data augmentation which randomly picks an image from given Camera model and randomly extract a patch (64x64x3,224x224x3).\nMy training details for resnet18:\npatch size- 64x64x3\nBatch_size = 32![enter image description here][1]\nbatch_per_epoch = 512\nOptimizer- sdg,lr= 0.01
\n\nEven after 30 epochs, I am seeing validation loss = 1.90 (started from 3.3) and accuracy- 30% ( started from 9%). After I am not seeing any improvement. Can you suggest what is the issue? With this model, my score is 0.209.
\n\nAlso, I am trying transfer learning with RESNET 50 which 2 change- Num class =10 and making 'Activation_43' to FC trainable and other layers untrainable.
\n\nDetails of training-\npatch size- 224x224x3\nBatch_size = 128\nbatch_per_epoch = 15\nOptimizer- adadelta,
\n\nEven with this, I am seeing very poor validation accuracy even though traing accuracy is jumping up but in very oscilating manner. Plz see the graphs of tensorbaords.
\n\nIf you guys have any suggestion then plz suggest it.
", "rawMarkdown": "Hi All,\nI am facing an issue, I have started training RESNET18 from scratch. I have written my own data augmentation which randomly picks an image from given Camera model and randomly extract a patch (64x64x3,224x224x3).\nMy training details for resnet18:\npatch size- 64x64x3\nBatch_size = 32![enter image description here][1]\nbatch_per_epoch = 512\nOptimizer- sdg,lr= 0.01\n\nEven after 30 epochs, I am seeing validation loss = 1.90 (started from 3.3) and accuracy- 30% ( started from 9%). After I am not seeing any improvement. Can you suggest what is the issue? With this model, my score is 0.209.\n\nAlso, I am trying transfer learning with RESNET 50 which 2 change- Num class =10 and making 'Activation_43' to FC trainable and other layers untrainable.\n\nDetails of training-\npatch size- 224x224x3\nBatch_size = 128\nbatch_per_epoch = 15\nOptimizer- adadelta,\n\nEven with this, I am seeing very poor validation accuracy even though traing accuracy is jumping up but in very oscilating manner. Plz see the graphs of tensorbaords.\n\nIf you guys have any suggestion then plz suggest it.\n\n\n\n\n ", "replies": [ { "id": 263911, "postDate": "2018-01-01T15:30:00.887Z", "content": "What do your final layers look like? Are you taking the output of the ResNet directly into the softmax layer? Often I will put an intermediate layer between the output layer. I then add dropout as needed to reduce overfitting. \n Here is some psudo Keras code:
\n\nx = ResNet.output \nx = GlobalAverage2D()(x)\nx = Dense(128) (x)\nx = BatchNormalization(x)\nx = Dropout(0.1)(x)\nx = Dense(10)(x)\nx = Dropout(0.1)(x)\npredictions = Activation('softmax')(x)\nThat's a pretty high learning rate. When I do transfer learning, I first turn off all the training in the model I am using (In your case ResNet) and train at an LR of 1e-3 (sometimes I will do annealing). Once I am happy with the results (I find val_acc at around 60%) I will open up the other layers and start training at LR of 1 e-4.
", "rawMarkdown": "What do your final layers look like? Are you taking the output of the ResNet directly into the softmax layer? Often I will put an intermediate layer between the output layer. I then add dropout as needed to reduce overfitting. \n Here is some psudo Keras code:\n\n x = ResNet.output \n x = GlobalAverage2D()(x)\n x = Dense(128) (x)\n x = BatchNormalization(x)\n x = Dropout(0.1)(x)\n x = Dense(10)(x)\n x = Dropout(0.1)(x)\n predictions = Activation('softmax')(x)\n \n\nThat's a pretty high learning rate. When I do transfer learning, I first turn off all the training in the model I am using (In your case ResNet) and train at an LR of 1e-3 (sometimes I will do annealing). Once I am happy with the results (I find val_acc at around 60%) I will open up the other layers and start training at LR of 1 e-4.\n", "votes": -3 }, { "id": 263954, "postDate": "2018-01-01T19:52:25.193Z", "content": "Hi, \nI am follwoing exactly same approch what you have suggested. However there is mis-match in model.\nThe very first time I have strated with adding 2 drop-out layers as follows-
\n\n x = resnet_model.output #after Avg pooling layer\nx = Flatten()(x) #2048 vector-RESNET 50\nx = Dense(1024,name='FCL1_1024')(x)\nx = BatchNormalization()(x)\nx = Dropout(0.3)(x)\nx = Dense(num_class,activation ='sigmoid',name='Class_out')(x)\nI was training on this layers and all others were fixed. I have seen that even after 15 epochs error and accuracy for both -Training/Validation- was not improving much - Training Accuracy-12%, Valid Accu-10%. So, I have stopped training process.
\n\nWith the latest model, which is currently being trained I have used only-
\n\n x = resnet_model.output #after Avg pooling layer\nx = Flatten()(x) #2048 vector-RESNET 50\nx = Dense(num_class,activation ='sigmoid',name='Class_out')(x)\nTrained this for 15 epochs- Train_Acc=20-22%, Valid_Accu=10%, I have stopped this and loaded model with this step and made top-2 convolution block trainable.
\n\nCurrent status aftre 18epochs-{128-batch size, 15_batch_per epoch} Training accuracy-40% but validation Accuarcy-10% (same).
\n\nThe pattern of training accuracy plot follows same as I have sent in the last image..it is periodic..not monotonically increasing/decreasing.
\n\nBtw, why are you adding drop-out to the final output of 10 class-
\n\nx = Dense(10)(x)\n **x = Dropout(0.1)(x)**\n predictions = Activation('softmax')(x)\nSuggest if you have any clue on this. Btw are you training resnet18 or 50 ?
", "rawMarkdown": "Hi, \nI am follwoing exactly same approch what you have suggested. However there is mis-match in model.\nThe very first time I have strated with adding 2 drop-out layers as follows-\n\n x = resnet_model.output #after Avg pooling layer\n\tx = Flatten()(x) #2048 vector-RESNET 50\n\tx = Dense(1024,name='FCL1_1024')(x)\n\tx = BatchNormalization()(x)\n\tx = Dropout(0.3)(x)\n\tx = Dense(num_class,activation ='sigmoid',name='Class_out')(x)\nI was training on this layers and all others were fixed. I have seen that even after 15 epochs error and accuracy for both -Training/Validation- was not improving much - Training Accuracy-12%, Valid Accu-10%. So, I have stopped training process.\n\nWith the latest model, which is currently being trained I have used only-\n\n x = resnet_model.output #after Avg pooling layer\n\tx = Flatten()(x) #2048 vector-RESNET 50\n\tx = Dense(num_class,activation ='sigmoid',name='Class_out')(x)\n\nTrained this for 15 epochs- Train_Acc=20-22%, Valid_Accu=10%, I have stopped this and loaded model with this step and made top-2 convolution block trainable. \n\nCurrent status aftre 18epochs-{128-batch size, 15_batch_per epoch} Training accuracy-40% but validation Accuarcy-10% (same).\n\nThe pattern of training accuracy plot follows same as I have sent in the last image..it is periodic..not monotonically increasing/decreasing.\n\nBtw, why are you adding drop-out to the final output of 10 class-\n\n x = Dense(10)(x)\n **x = Dropout(0.1)(x)**\n predictions = Activation('softmax')(x)\n\nSuggest if you have any clue on this. Btw are you training resnet18 or 50 ?\n\n" }, { "id": 263961, "postDate": "2018-01-01T20:22:16.650Z", "content": "I used ResNet 50. My last run with ResNet (256x256) was:\nTraining loss 0.0366\nTraining accuracy: 0.991\nVal loss: 0.355\nVal acc: 0.909
\n\nLeaderboard submission: 0.742 (Not one of my best models)
\n\nI use multiple layers at the end to have better control overfitting. ResNet has a tendency to overfit compared to other models. So I almost always use dropout, even at the softmax layer.\nGenerally, I train these models for quite a while (100 + epochs) with fine tuning LR as low as 1e-5.
\n\nAre you normalizing your data?
", "rawMarkdown": "I used ResNet 50. My last run with ResNet (256x256) was:\nTraining loss 0.0366\nTraining accuracy: 0.991\nVal loss: 0.355\nVal acc: 0.909\n\nLeaderboard submission: 0.742 (Not one of my best models)\n\nI use multiple layers at the end to have better control overfitting. ResNet has a tendency to overfit compared to other models. So I almost always use dropout, even at the softmax layer.\nGenerally, I train these models for quite a while (100 + epochs) with fine tuning LR as low as 1e-5.\n\nAre you normalizing your data? \n", "votes": -6 }, { "id": 263967, "postDate": "2018-01-01T20:47:32.450Z", "content": "I wouldn't recommend using Dropout at the softmax layer.
", "rawMarkdown": "I wouldn't recommend using Dropout at the softmax layer. ", "votes": 7 }, { "id": 263970, "postDate": "2018-01-01T20:57:13.907Z", "content": "Thank you for your comment, but I know what I am doing.
", "rawMarkdown": "Thank you for your comment, but I know what I am doing. ", "votes": -22 }, { "id": 263973, "postDate": "2018-01-01T21:08:40.180Z", "content": "Sure. Using dropout before the softmax layer will simply discard x% of the information used to predict the labels (where x is the dropout probability/rate).
\n\nMultiple sources to attest this:
\n\nhttps://stats.stackexchange.com/questions/299292/dropout-makes-performance-worse
\n\nhttps://stackoverflow.com/questions/40426737/where-to-add-dropout-in-neural-network
", "rawMarkdown": "Sure. Using dropout before the softmax layer will simply discard x% of the information used to predict the labels (where x is the dropout probability/rate).\n\nMultiple sources to attest this:\n\nhttps://stats.stackexchange.com/questions/299292/dropout-makes-performance-worse\n\nhttps://stackoverflow.com/questions/40426737/where-to-add-dropout-in-neural-network\n", "votes": 12 }, { "id": 263975, "postDate": "2018-01-01T21:29:56.747Z", "content": "Thanks again for your comments. I generally don't like to be harassed in Kaggle, but when I do, I prefer to be harassed by someone out-performing me in the competition.
", "rawMarkdown": "Thanks again for your comments. I generally don't like to be harassed in Kaggle, but when I do, I prefer to be harassed by someone out-performing me in the competition.", "votes": -35 }, { "id": 263979, "postDate": "2018-01-01T21:59:42.883Z", "content": "Ah :). The age when sharing advice and expertise is 'harassment'. We've reached peak Trump in 2018 already..
", "rawMarkdown": "Ah :). The age when sharing advice and expertise is 'harassment'. We've reached peak Trump in 2018 already..", "votes": 9 }, { "id": 263984, "postDate": "2018-01-01T22:36:36.510Z", "content": "Thank you for your reply. Your comments are noted and will be given the appropriate amount of attention.
", "rawMarkdown": "Thank you for your reply. Your comments are noted and will be given the appropriate amount of attention. ", "votes": -28 }, { "id": 264057, "postDate": "2018-01-02T05:30:30.023Z", "content": "random data augmentation/transformation- flip/gamma/rot
Patch = patch - { avg of RGB across all the training data} channel wise
This patch goes to model...so No normalization in term of bringing data between 0-1.
", "rawMarkdown": " 1. My steps for dats pre processing-\n 2. randomly pic an image for a given camear model and randomly pic patch size of 224x224\n 3. random data augmentation/transformation- flip/gamma/rot\n \n 4. Patch = patch - { avg of RGB across all the training data} channel wise\n 5. scale it , patch = patch/255 ( -1 to 1 - data range )\n\nThis patch goes to model...so No normalization in term of bringing data between 0-1." }, { "id": 268107, "postDate": "2018-01-13T11:04:38.183Z", "content": "Don't normalise/scale the images - I also fell into that trap... If you are using ResNet in Keras, it expects the images \"as they are\"
", "rawMarkdown": "Don't normalise/scale the images - I also fell into that trap... If you are using ResNet in Keras, it expects the images \"as they are\"", "votes": 1 }, { "id": 268215, "postDate": "2018-01-13T19:44:53.860Z", "content": "Out of curiosity (not using ResNet), where did you find this information ?
", "rawMarkdown": "Out of curiosity (not using ResNet), where did you find this information ?" }, { "id": 268216, "postDate": "2018-01-13T20:04:27.793Z", "content": "trial and error + a hint in this keras thread where someone said \"Probably uint8 rgb images unless specified otherwise.\"
\n\nhttps://github.com/keras-team/keras/issues/5154
\n\nbut yes, mostly trial and error...
", "rawMarkdown": "trial and error + a hint in this keras thread where someone said \"Probably uint8 rgb images unless specified otherwise.\"\n\nhttps://github.com/keras-team/keras/issues/5154\n\nbut yes, mostly trial and error...", "votes": 1 }, { "id": 275676, "postDate": "2018-01-29T19:17:41.320Z", "content": "The source code for pretrained models is actually available on github. You can check it for yourself instead of making guesses. ResNet50 model in keras was converted from weights trained using caffee and its input has to be normalized. Every model in keras.applications supplies a preprocess_input function which adjusts your images according to the demands of a particular model.
Share some of my experiments here.
\n\nI have no clue why there is a huge gap between validation and LB. :(
", "rawMarkdown": "Share some of my experiments here.\n\n 1. I implemented all the manipulations on the cropped images and used them for training.\n 2. Validation split 0.2, Simple CNN trained 100 epochs had 26% on LB. (Validation acc = 67%)\n 3. Transfer learning with ResNet50, trained 100 epochs had 33% on LB. (Validation acc = 96%)\n\nI have no clue why there is a huge gap between validation and LB. :(", "replies": [ { "id": 263682, "postDate": "2017-12-31T11:16:44Z", "content": "I implemented the manipulations on training images and still processing to see if this improves results. I am wondering how you're cropping because from what I see, it makes a big difference in LB score.
", "rawMarkdown": "I implemented the manipulations on training images and still processing to see if this improves results. I am wondering how you're cropping because from what I see, it makes a big difference in LB score.", "votes": 1 }, { "id": 263767, "postDate": "2017-12-31T20:12:42.847Z", "content": "I cropped from the center of the image as described on the data. I tried cropping all patches from the image, but it didn't seem to be effective for LB score.
", "rawMarkdown": "I cropped from the center of the image as described on the data. I tried cropping all patches from the image, but it didn't seem to be effective for LB score." } ] }, { "id": 263035, "postDate": "2017-12-28T18:55:15.053Z", "content": "When you say huge mismatch between validation and LB, which one is higher/lower?\nI'm getting on LB about half of what I get on my validation...\n(and yes, I know the test set includes image manipulations)
", "rawMarkdown": "When you say huge mismatch between validation and LB, which one is higher/lower?\nI'm getting on LB about half of what I get on my validation...\n(and yes, I know the test set includes image manipulations)", "replies": [ { "id": 263039, "postDate": "2017-12-28T19:03:25.867Z", "content": "Validation 97% acc, LB 82%
", "rawMarkdown": "Validation 97% acc, LB 82% ", "votes": 1 }, { "id": 263040, "postDate": "2017-12-28T19:08:16.730Z", "content": "That's not bad at all... hope to get there some day ;-)\nMy last submission was ~0.65 on validation, ~0.35 on LB.
", "rawMarkdown": "That's not bad at all... hope to get there some day ;-)\nMy last submission was ~0.65 on validation, ~0.35 on LB.", "votes": 2 } ] }, { "id": 263188, "postDate": "2017-12-29T07:38:57.107Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 263231, "postDate": "2017-12-29T11:03:24.617Z", "content": "I assume this is to get chunks of images ?
\n\nI cast the pil image object as a numpy array and then use array slicing to get chunk\nYou can use pil fromarray if you wanna save it afterwords
", "rawMarkdown": "I assume this is to get chunks of images ? \n\nI cast the pil image object as a numpy array and then use array slicing to get chunk\nYou can use pil fromarray if you wanna save it afterwords" } ] } ], "comments": [ { "id": 263078, "author_name": "Bruner Consulting", "author_url": "", "post_date": "2017-12-28T21:37:47.893000", "content": "I am seeing similar things. (validation 97, Test 80) My theory is that the modified images in the test set is dragging down the total score. I am using a variety of Xfr learning and averaging. I found that center patches seem to work the best.
", "votes": 1, "replies": [ { "id": 263093, "author_name": "CVxTz", "author_url": "", "post_date": "2017-12-28T22:04:40.120000", "content": "What is Xfr learning ?
", "votes": 1, "replies": [] }, { "id": 263096, "author_name": "Bruner Consulting", "author_url": "", "post_date": "2017-12-28T22:12:20.993000", "content": "Transfer learning. You take a pretrained deep network (ResNet, Inception) and attach your own end layer. https://machinelearningmastery.com/transfer-learning-for-deep-learning/. Useful when you don't have a lot of training data.
", "votes": -3, "replies": [] }, { "id": 263099, "author_name": "CVxTz", "author_url": "", "post_date": "2017-12-28T22:17:07.580000", "content": "Oh Yeah I get it ! I am doing the same
", "votes": 0, "replies": [] }, { "id": 268023, "author_name": "newlifer", "author_url": "", "post_date": "2018-01-13T02:54:53.910000", "content": "does it work in this task? i thought that it only works for similiar tasks. - -|
", "votes": -1, "replies": [] } ] }, { "id": 263495, "author_name": "Ali Sharaf", "author_url": "", "post_date": "2017-12-30T13:21:20.403000", "content": "I split all training images into 512x512 then resized all to 256x256 (similar approach used to feed test into the model). Getting > 90% validation but on test goes almost half. I am assuming it's due to altered images... I tried setting all altered images to one category and the score goes down just a little.
", "votes": 1, "replies": [] }, { "id": 263384, "author_name": "Abdul Muntakim Rafi", "author_url": "", "post_date": "2017-12-30T00:27:15.927000", "content": "CVxTz in your approach How can you be sure that your model isn't just detecting objects that are same in training data and testing data...
", "votes": 1, "replies": [ { "id": 263486, "author_name": "CVxTz", "author_url": "", "post_date": "2017-12-30T12:29:57.480000", "content": "Images in the train are not in the validation set.
", "votes": 0, "replies": [] }, { "id": 263493, "author_name": "Craig Glastonbury", "author_url": "", "post_date": "2017-12-30T13:17:43.923000", "content": "For this to be the case, there would have to be objects specific to photos taken on a single phone model. I.e. All Samsung Galaxy S6 photos having dogs in them.
", "votes": 0, "replies": [] } ] }, { "id": 262992, "author_name": "Craig Glastonbury", "author_url": "", "post_date": "2017-12-28T15:03:28.643000", "content": "Similar approach but no ensembl model yet.
", "votes": 1, "replies": [] }, { "id": 263235, "author_name": "Uday Kamal", "author_url": "", "post_date": "2017-12-29T11:30:53.453000", "content": "We have the same issue. There might be another case, which is, in the test dataset, all the images are captured from another devices (but the models are same). So there is a high possibility that deep cnns are learning the device specific PRNU noises.
", "votes": 2, "replies": [ { "id": 263238, "author_name": "CVxTz", "author_url": "", "post_date": "2017-12-29T11:36:00", "content": "I had the same doubts so I asked this question in the Presentation thread but the organizers did not answer
", "votes": 0, "replies": [] }, { "id": 265300, "author_name": "Xiangyi Yan", "author_url": "", "post_date": "2018-01-05T04:47:09.330000", "content": "Have any thoughts about improving this situation?
", "votes": 0, "replies": [] }, { "id": 265336, "author_name": "", "author_url": "", "post_date": "2018-01-05T08:39:46.087000", "content": "", "votes": 0, "replies": [] }, { "id": 265354, "author_name": "Xiangyi Yan", "author_url": "", "post_date": "2018-01-05T09:57:29.353000", "content": "I don't think purely ensembling might help a lot. The variance between test set and validation set is too large.
", "votes": 0, "replies": [] } ] }, { "id": 263789, "author_name": "Bruner Consulting", "author_url": "", "post_date": "2017-12-31T22:35:45.113000", "content": "I've been running some tests on data \"out in the wild\" (images from the web). Even though my current model has a 99% validation accuracy, these get correctly identified around 20-30% of the time.
\n\nAs another test, I sent 10 images from the training set through \"google hangouts\" and none of the resulting images get correctly classified. Looking at the jpeg headers for these failing images, I notice that the quality is reduced to 90%. However, if I perform the same operation manually (Image.save), I still can identify the resulting images.
\n\nI have an old Note 3 and about 90% of these images get correctly classified unless they are sent through google hangout.
\n\nI suspect that we probably need to augment the training set with additional images from other sources or reprocessed training files. too bad ImageDataGenerator doesn't have a \"Google Hangouts\" operator ;)
", "votes": -2, "replies": [] }, { "id": 263301, "author_name": "moonforsun", "author_url": "", "post_date": "2017-12-29T17:19:41.973000", "content": "Does anyone have any advice for processing the JPGs in R to get them to a 'model ready' state?
", "votes": -1, "replies": [] }, { "id": 265695, "author_name": "SumitJha", "author_url": "", "post_date": "2018-01-06T10:10:50.987000", "content": "Hi All,\nI am doing transfer learning for RESNET50. I have modified FC layers-
\n\nx = resnet_model.output\nx = Flatten()(x)\nx = Dropout(0.3)(x)\nx = Dense(1024,name='FCL1_1024')(x)\n#x = BatchNormalization()(x)\nx = Dropout(0.3)(x)\nx = Dense(num_class,activation ='sigmoid',name='Class_out')(x)\nI have my own patch generation code and random_transform. But even after 20 epochs I am not seeing any improvement in val_loss/val_acc --see in the image-
\n\nPlz suggest, what could be the reason for this.
", "votes": 0, "replies": [ { "id": 265709, "author_name": "", "author_url": "", "post_date": "2018-01-06T11:42:46.963000", "content": "", "votes": 0, "replies": [] }, { "id": 265721, "author_name": "", "author_url": "", "post_date": "2018-01-06T12:23:58.950000", "content": "Just guessing as your splitting code is not there.. Have you shuffled the data?
", "votes": 0, "replies": [] }, { "id": 265731, "author_name": "Craig Glastonbury", "author_url": "", "post_date": "2018-01-06T13:17:50.327000", "content": "You're using sigmoid, which is good for binary classification, not multiclass. Change to:
\n\nx = Dense(num_class,activation =\"softmax\",name=\"Class_out\")(x)
I'm using a mix of features + a random forest classifier, which has the same mismatch between validation and LB, but at a much lower score. Just to get a better idea of approach preferences, I'm curious if anyone in the top 25 is not using a deep learning model?
", "votes": 0, "replies": [ { "id": 264514, "author_name": "", "author_url": "", "post_date": "2018-01-03T09:22:59.630000", "content": "\n\n\nI'm curious if anyone in the top 25 is not using a deep learning model?
\n
Yes, almost all of them.
\n\n\n\n\nnot
\n
Oh. :D
", "votes": -1, "replies": [] } ] }, { "id": 263864, "author_name": "SumitJha", "author_url": "", "post_date": "2018-01-01T11:55:20.510000", "content": "Hi All,\nI am facing an issue, I have started training RESNET18 from scratch. I have written my own data augmentation which randomly picks an image from given Camera model and randomly extract a patch (64x64x3,224x224x3).\nMy training details for resnet18:\npatch size- 64x64x3\nBatch_size = 32![enter image description here][1]\nbatch_per_epoch = 512\nOptimizer- sdg,lr= 0.01
\n\nEven after 30 epochs, I am seeing validation loss = 1.90 (started from 3.3) and accuracy- 30% ( started from 9%). After I am not seeing any improvement. Can you suggest what is the issue? With this model, my score is 0.209.
\n\nAlso, I am trying transfer learning with RESNET 50 which 2 change- Num class =10 and making 'Activation_43' to FC trainable and other layers untrainable.
\n\nDetails of training-\npatch size- 224x224x3\nBatch_size = 128\nbatch_per_epoch = 15\nOptimizer- adadelta,
\n\nEven with this, I am seeing very poor validation accuracy even though traing accuracy is jumping up but in very oscilating manner. Plz see the graphs of tensorbaords.
\n\nIf you guys have any suggestion then plz suggest it.
", "votes": 0, "replies": [ { "id": 263911, "author_name": "Bruner Consulting", "author_url": "", "post_date": "2018-01-01T15:30:00.887000", "content": "What do your final layers look like? Are you taking the output of the ResNet directly into the softmax layer? Often I will put an intermediate layer between the output layer. I then add dropout as needed to reduce overfitting. \n Here is some psudo Keras code:
\n\nx = ResNet.output \nx = GlobalAverage2D()(x)\nx = Dense(128) (x)\nx = BatchNormalization(x)\nx = Dropout(0.1)(x)\nx = Dense(10)(x)\nx = Dropout(0.1)(x)\npredictions = Activation('softmax')(x)\nThat's a pretty high learning rate. When I do transfer learning, I first turn off all the training in the model I am using (In your case ResNet) and train at an LR of 1e-3 (sometimes I will do annealing). Once I am happy with the results (I find val_acc at around 60%) I will open up the other layers and start training at LR of 1 e-4.
", "votes": -3, "replies": [] }, { "id": 263954, "author_name": "SumitJha", "author_url": "", "post_date": "2018-01-01T19:52:25.193000", "content": "Hi, \nI am follwoing exactly same approch what you have suggested. However there is mis-match in model.\nThe very first time I have strated with adding 2 drop-out layers as follows-
\n\n x = resnet_model.output #after Avg pooling layer\nx = Flatten()(x) #2048 vector-RESNET 50\nx = Dense(1024,name='FCL1_1024')(x)\nx = BatchNormalization()(x)\nx = Dropout(0.3)(x)\nx = Dense(num_class,activation ='sigmoid',name='Class_out')(x)\nI was training on this layers and all others were fixed. I have seen that even after 15 epochs error and accuracy for both -Training/Validation- was not improving much - Training Accuracy-12%, Valid Accu-10%. So, I have stopped training process.
\n\nWith the latest model, which is currently being trained I have used only-
\n\n x = resnet_model.output #after Avg pooling layer\nx = Flatten()(x) #2048 vector-RESNET 50\nx = Dense(num_class,activation ='sigmoid',name='Class_out')(x)\nTrained this for 15 epochs- Train_Acc=20-22%, Valid_Accu=10%, I have stopped this and loaded model with this step and made top-2 convolution block trainable.
\n\nCurrent status aftre 18epochs-{128-batch size, 15_batch_per epoch} Training accuracy-40% but validation Accuarcy-10% (same).
\n\nThe pattern of training accuracy plot follows same as I have sent in the last image..it is periodic..not monotonically increasing/decreasing.
\n\nBtw, why are you adding drop-out to the final output of 10 class-
\n\nx = Dense(10)(x)\n **x = Dropout(0.1)(x)**\n predictions = Activation('softmax')(x)\nSuggest if you have any clue on this. Btw are you training resnet18 or 50 ?
", "votes": 0, "replies": [] }, { "id": 263961, "author_name": "Bruner Consulting", "author_url": "", "post_date": "2018-01-01T20:22:16.650000", "content": "I used ResNet 50. My last run with ResNet (256x256) was:\nTraining loss 0.0366\nTraining accuracy: 0.991\nVal loss: 0.355\nVal acc: 0.909
\n\nLeaderboard submission: 0.742 (Not one of my best models)
\n\nI use multiple layers at the end to have better control overfitting. ResNet has a tendency to overfit compared to other models. So I almost always use dropout, even at the softmax layer.\nGenerally, I train these models for quite a while (100 + epochs) with fine tuning LR as low as 1e-5.
\n\nAre you normalizing your data?
", "votes": -6, "replies": [] }, { "id": 263967, "author_name": "Craig Glastonbury", "author_url": "", "post_date": "2018-01-01T20:47:32.450000", "content": "I wouldn't recommend using Dropout at the softmax layer.
", "votes": 7, "replies": [] }, { "id": 263970, "author_name": "Bruner Consulting", "author_url": "", "post_date": "2018-01-01T20:57:13.907000", "content": "Thank you for your comment, but I know what I am doing.
", "votes": -22, "replies": [] }, { "id": 263973, "author_name": "Craig Glastonbury", "author_url": "", "post_date": "2018-01-01T21:08:40.180000", "content": "Sure. Using dropout before the softmax layer will simply discard x% of the information used to predict the labels (where x is the dropout probability/rate).
\n\nMultiple sources to attest this:
\n\nhttps://stats.stackexchange.com/questions/299292/dropout-makes-performance-worse
\n\nhttps://stackoverflow.com/questions/40426737/where-to-add-dropout-in-neural-network
", "votes": 12, "replies": [] }, { "id": 263975, "author_name": "Bruner Consulting", "author_url": "", "post_date": "2018-01-01T21:29:56.747000", "content": "Thanks again for your comments. I generally don't like to be harassed in Kaggle, but when I do, I prefer to be harassed by someone out-performing me in the competition.
", "votes": -35, "replies": [] }, { "id": 263979, "author_name": "Craig Glastonbury", "author_url": "", "post_date": "2018-01-01T21:59:42.883000", "content": "Ah :). The age when sharing advice and expertise is 'harassment'. We've reached peak Trump in 2018 already..
", "votes": 9, "replies": [] }, { "id": 263984, "author_name": "Bruner Consulting", "author_url": "", "post_date": "2018-01-01T22:36:36.510000", "content": "Thank you for your reply. Your comments are noted and will be given the appropriate amount of attention.
", "votes": -28, "replies": [] }, { "id": 264057, "author_name": "SumitJha", "author_url": "", "post_date": "2018-01-02T05:30:30.023000", "content": "random data augmentation/transformation- flip/gamma/rot
Patch = patch - { avg of RGB across all the training data} channel wise
This patch goes to model...so No normalization in term of bringing data between 0-1.
", "votes": 0, "replies": [] }, { "id": 268107, "author_name": "nkroher", "author_url": "", "post_date": "2018-01-13T11:04:38.183000", "content": "Don't normalise/scale the images - I also fell into that trap... If you are using ResNet in Keras, it expects the images \"as they are\"
", "votes": 1, "replies": [] }, { "id": 268215, "author_name": "Max", "author_url": "", "post_date": "2018-01-13T19:44:53.860000", "content": "Out of curiosity (not using ResNet), where did you find this information ?
", "votes": 0, "replies": [] }, { "id": 268216, "author_name": "nkroher", "author_url": "", "post_date": "2018-01-13T20:04:27.793000", "content": "trial and error + a hint in this keras thread where someone said \"Probably uint8 rgb images unless specified otherwise.\"
\n\nhttps://github.com/keras-team/keras/issues/5154
\n\nbut yes, mostly trial and error...
", "votes": 1, "replies": [] }, { "id": 275676, "author_name": "Maksym Pyrozhok", "author_url": "", "post_date": "2018-01-29T19:17:41.320000", "content": "The source code for pretrained models is actually available on github. You can check it for yourself instead of making guesses. ResNet50 model in keras was converted from weights trained using caffee and its input has to be normalized. Every model in keras.applications supplies a preprocess_input function which adjusts your images according to the demands of a particular model.
Share some of my experiments here.
\n\nI have no clue why there is a huge gap between validation and LB. :(
", "votes": 0, "replies": [ { "id": 263682, "author_name": "Ali Sharaf", "author_url": "", "post_date": "2017-12-31T11:16:44", "content": "I implemented the manipulations on training images and still processing to see if this improves results. I am wondering how you're cropping because from what I see, it makes a big difference in LB score.
", "votes": 1, "replies": [] }, { "id": 263767, "author_name": "Shunjia Ding", "author_url": "", "post_date": "2017-12-31T20:12:42.847000", "content": "I cropped from the center of the image as described on the data. I tried cropping all patches from the image, but it didn't seem to be effective for LB score.
", "votes": 0, "replies": [] } ] }, { "id": 263035, "author_name": "Diogo R. Ferreira", "author_url": "", "post_date": "2017-12-28T18:55:15.053000", "content": "When you say huge mismatch between validation and LB, which one is higher/lower?\nI'm getting on LB about half of what I get on my validation...\n(and yes, I know the test set includes image manipulations)
", "votes": 0, "replies": [ { "id": 263039, "author_name": "CVxTz", "author_url": "", "post_date": "2017-12-28T19:03:25.867000", "content": "Validation 97% acc, LB 82%
", "votes": 1, "replies": [] }, { "id": 263040, "author_name": "Diogo R. Ferreira", "author_url": "", "post_date": "2017-12-28T19:08:16.730000", "content": "That's not bad at all... hope to get there some day ;-)\nMy last submission was ~0.65 on validation, ~0.35 on LB.
", "votes": 2, "replies": [] } ] }, { "id": 263188, "author_name": "", "author_url": "", "post_date": "2017-12-29T07:38:57.107000", "content": "", "votes": 0, "replies": [ { "id": 263231, "author_name": "CVxTz", "author_url": "", "post_date": "2017-12-29T11:03:24.617000", "content": "I assume this is to get chunks of images ?
\n\nI cast the pil image object as a numpy array and then use array slicing to get chunk\nYou can use pil fromarray if you wanna save it afterwords
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "262984": "Hello ! \nSince this is definitely an odd problem I am very curious about what approaches are other kagglers using to solve it.\n\nI'll start with myself : I am using an average of multiple architectures of deep CNNs ( like the kernel I shared ) applied to small crops of the training images, though I am not really sure what the models are learning exactly and I have a huge score mismatch between Validation and LB that I can't explain.\n\nAre there people using other types of methods ? \n", "263078": "I am seeing similar things. (validation 97, Test 80) My theory is that the modified images in the test set is dragging down the total score. I am using a variety of Xfr learning and averaging. I found that center patches seem to work the best.", "263495": "I split all training images into 512x512 then resized all to 256x256 (similar approach used to feed test into the model). Getting > 90% validation but on test goes almost half. I am assuming it's due to altered images... I tried setting all altered images to one category and the score goes down just a little.", "263384": "CVxTz in your approach How can you be sure that your model isn't just detecting objects that are same in training data and testing data... ", "262992": "Similar approach but no ensembl model yet. ", "263235": "We have the same issue. There might be another case, which is, in the test dataset, all the images are captured from another devices (but the models are same). So there is a high possibility that deep cnns are learning the device specific PRNU noises.\n ", "263789": "I've been running some tests on data \"out in the wild\" (images from the web). Even though my current model has a 99% validation accuracy, these get correctly identified around 20-30% of the time. \n\nAs another test, I sent 10 images from the training set through \"google hangouts\" and none of the resulting images get correctly classified. Looking at the jpeg headers for these failing images, I notice that the quality is reduced to 90%. However, if I perform the same operation manually (Image.save), I still can identify the resulting images.\n\nI have an old Note 3 and about 90% of these images get correctly classified unless they are sent through google hangout.\n\nI suspect that we probably need to augment the training set with additional images from other sources or reprocessed training files. too bad ImageDataGenerator doesn't have a \"Google Hangouts\" operator ;)", "263301": "Does anyone have any advice for processing the JPGs in R to get them to a 'model ready' state?", "265695": "Hi All,\nI am doing transfer learning for RESNET50. I have modified FC layers-\n\n x = resnet_model.output\n\tx = Flatten()(x)\n\tx = Dropout(0.3)(x)\n\tx = Dense(1024,name='FCL1_1024')(x)\n\t#x = BatchNormalization()(x)\n\tx = Dropout(0.3)(x)\n\tx = Dense(num_class,activation ='sigmoid',name='Class_out')(x)\n\nI have my own patch generation code and random_transform. But even after 20 epochs I am not seeing any improvement in val_loss/val_acc --see in the image-\n\nPlz suggest, what could be the reason for this.\n", "264299": "I'm using a mix of features + a random forest classifier, which has the same mismatch between validation and LB, but at a much lower score. Just to get a better idea of approach preferences, I'm curious if anyone in the top 25 is not using a deep learning model?", "263864": "Hi All,\nI am facing an issue, I have started training RESNET18 from scratch. I have written my own data augmentation which randomly picks an image from given Camera model and randomly extract a patch (64x64x3,224x224x3).\nMy training details for resnet18:\npatch size- 64x64x3\nBatch_size = 32![enter image description here][1]\nbatch_per_epoch = 512\nOptimizer- sdg,lr= 0.01\n\nEven after 30 epochs, I am seeing validation loss = 1.90 (started from 3.3) and accuracy- 30% ( started from 9%). After I am not seeing any improvement. Can you suggest what is the issue? With this model, my score is 0.209.\n\nAlso, I am trying transfer learning with RESNET 50 which 2 change- Num class =10 and making 'Activation_43' to FC trainable and other layers untrainable.\n\nDetails of training-\npatch size- 224x224x3\nBatch_size = 128\nbatch_per_epoch = 15\nOptimizer- adadelta,\n\nEven with this, I am seeing very poor validation accuracy even though traing accuracy is jumping up but in very oscilating manner. Plz see the graphs of tensorbaords.\n\nIf you guys have any suggestion then plz suggest it.\n\n\n\n\n ", "263652": "Share some of my experiments here.\n\n 1. I implemented all the manipulations on the cropped images and used them for training.\n 2. Validation split 0.2, Simple CNN trained 100 epochs had 26% on LB. (Validation acc = 67%)\n 3. Transfer learning with ResNet50, trained 100 epochs had 33% on LB. (Validation acc = 96%)\n\nI have no clue why there is a huge gap between validation and LB. :(", "263035": "When you say huge mismatch between validation and LB, which one is higher/lower?\nI'm getting on LB about half of what I get on my validation...\n(and yes, I know the test set includes image manipulations)", "263188": "" } }