Has anyone built a reliable local cross-validation strategy? For me the gap is massive, up to 12 on local vs 24 on LB.
\n\nI'm leaving about 250 random images for the model that predicts counts (so the CNN does not see them at all). Then I'm doing a 5-fold cross-validation on this 250 images using a couple of models such as XGBoost and Lasso.
\n\nWhat could explain such a huge gap? It could obviously be a bug in my code, or very different images in test vs. train (I didn't check predictions on test but the images look similar). Apart from that, I don't see any obvious reasons, or ways I'm leaking train into my validation.
", "messages": [ { "id": "179595", "postDate": "05/02/2017 07:06:41", "content": "Has anyone built a reliable local cross-validation strategy? For me the gap is massive, up to 12 on local vs 24 on LB.
\n\nI'm leaving about 250 random images for the model that predicts counts (so the CNN does not see them at all). Then I'm doing a 5-fold cross-validation on this 250 images using a couple of models such as XGBoost and Lasso.
\n\nWhat could explain such a huge gap? It could obviously be a bug in my code, or very different images in test vs. train (I didn't check predictions on test but the images look similar). Apart from that, I don't see any obvious reasons, or ways I'm leaking train into my validation.
", "rawMarkdown": "Has anyone built a reliable local cross-validation strategy? For me the gap is massive, up to 12 on local vs 24 on LB.\n\nI'm leaving about 250 random images for the model that predicts counts (so the CNN does not see them at all). Then I'm doing a 5-fold cross-validation on this 250 images using a couple of models such as XGBoost and Lasso.\n\nWhat could explain such a huge gap? It could obviously be a bug in my code, or very different images in test vs. train (I didn't check predictions on test but the images look similar). Apart from that, I don't see any obvious reasons, or ways I'm leaking train into my validation.", "votes": null }, { "id": "179651", "postDate": "05/02/2017 12:15:08", "content": "Konstantin, do you train a CNN on first ~750 images and than tune classifiers on hold-out 250 ?
", "rawMarkdown": "Konstantin, do you train a CNN on first ~750 images and than tune classifiers on hold-out 250 ?", "votes": null }, { "id": "179657", "postDate": "05/02/2017 12:27:35", "content": "Yes - actually, I'm training a CNN on about 650 images, and use about 100 for validation of the CNN, and then tune regression models that take CNN predictions as input on the hold-out 250.
\n\nEdit: and I'm also excluding images from MismatchedTrainImages.txt + 941, 200, 491, 912
", "rawMarkdown": "Yes - actually, I'm training a CNN on about 650 images, and use about 100 for validation of the CNN, and then tune regression models that take CNN predictions as input on the hold-out 250.\n\nEdit: and I'm also excluding images from MismatchedTrainImages.txt + 941, 200, 491, 912", "votes": null }, { "id": "179688", "postDate": "05/02/2017 14:11:23", "content": "My current approach make massive gap too.
\nIn my opinion, we should not believe public LB.
\n\n\nIn my opinion, we should not believe public LB.
\n
Do you think that the private LB will be better? Do you have any hypothesis about that gap between local CV and public LB?
", "rawMarkdown": "> In my opinion, we should not believe public LB.\n\nDo you think that the private LB will be better? Do you have any hypothesis about that gap between local CV and public LB?", "votes": null }, { "id": "179906", "postDate": "05/03/2017 08:14:47", "content": "Some competitors point out that number of sea-lions in train.csv don't consistent real number. This implies train.csv includes much noise. I don't know that private LB will be better, but I hope our final submission evaluated with real number.
By the way, admin shared MismatchedTrainImages.txt after starting competition. I wonder if there is MismatchedTestImages.txt potentially or not.
Hello.\nHave the same problem =(\nI tried to submit all zeros, and it gave me the score 29.
", "rawMarkdown": "Hello.\nHave the same problem =(\nI tried to submit all zeros, and it gave me the score 29.", "votes": null }, { "id": "182912", "postDate": "05/16/2017 09:44:20", "content": "Just to understand your approach, and I apologize in advance if I misunderstood something; couldn't you also validate your model on the test set? That will only allow you to validate the counts, not the predicted coordinates of the animals, but wouldn't that be enough?
", "rawMarkdown": "Just to understand your approach, and I apologize in advance if I misunderstood something; couldn't you also validate your model on the test set? That will only allow you to validate the counts, not the predicted coordinates of the animals, but wouldn't that be enough?", "votes": null }, { "id": "182914", "postDate": "05/16/2017 09:47:48", "content": "@erde not sure I understand what you mean - I can't use the test set for validation because we don't have the counts for the test set - we need to predict them.
", "rawMarkdown": "erde not sure I understand what you mean - I can't use the test set for validation because we don't have the counts for the test set - we need to predict them.", "votes": null }, { "id": "182988", "postDate": "05/16/2017 16:19:24", "content": "I think that the discrepancy between local validation score and public leaderboard may arise from difference in calculation of RMSE. The metric for the competition is average columnwise RMSE, the metric used for example by Keras is rowwise RMSE.
\n\nFor a 3x5 table \n[1 2 3 4 5]\n[6 7 8 9 10]\n[11 12 13 14 15]\nwith ground truth of 0
\n\nRMSE columnwise = 9.1\nRMS rowwise = 15.8
\n\nOn the bright side they move in the same direction, so if your rowwise RMSE is going down, that means your columnwise RMSE will go down too.
", "rawMarkdown": "I think that the discrepancy between local validation score and public leaderboard may arise from difference in calculation of RMSE. The metric for the competition is average columnwise RMSE, the metric used for example by Keras is rowwise RMSE. \n\nFor a 3x5 table \n[1 2 3 4 5]\n[6 7 8 9 10]\n[11 12 13 14 15]\nwith ground truth of 0\n\nRMSE columnwise = 9.1\nRMS rowwise = 15.8\n\nOn the bright side they move in the same direction, so if your rowwise RMSE is going down, that means your columnwise RMSE will go down too.", "votes": null }, { "id": "182990", "postDate": "05/16/2017 16:29:58", "content": "This gap could be explained by the metric. RMSE between different datasets can only be compared if the average counts are similar. In case you did the train/validation split blindly you could end up with a validation set with lower counts (when counts are lower the same model scores better).
", "rawMarkdown": "This gap could be explained by the metric. RMSE between different datasets can only be compared if the average counts are similar. In case you did the train/validation split blindly you could end up with a validation set with lower counts (when counts are lower the same model scores better).", "votes": null }, { "id": "182994", "postDate": "05/16/2017 16:46:42", "content": "Yes, maybe this is the reason, because I'm getting completely different values.\nFor your example (assuming you meant columns of length 3 and rows of length 5), my mean column-wise RMSE is 9.0052, and my mean row-wise RMSE is 8.1724.
\n\nCan someone else try calculating it? :)
", "rawMarkdown": "Yes, maybe this is the reason, because I'm getting completely different values.\nFor your example (assuming you meant columns of length 3 and rows of length 5), my mean column-wise RMSE is 9.0052, and my mean row-wise RMSE is 8.1724.\n\nCan someone else try calculating it? :)", "votes": null }, { "id": "182997", "postDate": "05/16/2017 16:57:45", "content": "Yes, some difference (e.g. scale) between train and test set is my leading hypothesis at the moment, assuming I implemented RMSE calculation correctly. I don't think it's explained just by a bad split though.
", "rawMarkdown": "Yes, some difference (e.g. scale) between train and test set is my leading hypothesis at the moment, assuming I implemented RMSE calculation correctly. I don't think it's explained just by a bad split though.", "votes": null }, { "id": "183136", "postDate": "05/17/2017 02:58:46", "content": "I got a huge gap, too. I think row-wise RMSE is lower than column-wise in this competition.\nYou can check your submission's column-wise RMS (RMSE to all zero) vs. 29 (public LB RMS).\nThe public LB score always worse than abs(submission's RMS - 29).\nIf model isn't good enough, the public LB score will worse than abs(submission's RMS - 29) + local validation score.
", "rawMarkdown": "I got a huge gap, too. I think row-wise RMSE is lower than column-wise in this competition.\nYou can check your submission's column-wise RMS (RMSE to all zero) vs. 29 (public LB RMS).\nThe public LB score always worse than abs(submission's RMS - 29).\nIf model isn't good enough, the public LB score will worse than abs(submission's RMS - 29) + local validation score.", "votes": null }, { "id": "183150", "postDate": "05/17/2017 05:16:35", "content": "Thanks for the clarification. I misunderstood the structure of the dataset. But you could automate uploading the test set predictions to Kaggle to get a score, or would that evaluation be too noisy/slow?
", "rawMarkdown": "Thanks for the clarification. I misunderstood the structure of the dataset. But you could automate uploading the test set predictions to Kaggle to get a score, or would that evaluation be too noisy/slow?", "votes": null }, { "id": "183278", "postDate": "05/17/2017 16:20:58", "content": "@erde that's very dangerous, as you can overfit to the public LB and then fail on the private LB. So it's better to have a working local CV. Also, you can have only 5 submissions per day, which is not a lot.
", "rawMarkdown": "erde that's very dangerous, as you can overfit to the public LB and then fail on the private LB. So it's better to have a working local CV. Also, you can have only 5 submissions per day, which is not a lot.", "votes": null }, { "id": "183442", "postDate": "05/18/2017 08:54:47", "content": "Public and private LB are different subsets of the files in input/Test?
Same values for me: 9.00520 column-wise, 8.17245 row-wise.
", "rawMarkdown": "Same values for me: 9.00520 column-wise, 8.17245 row-wise.", "votes": null }, { "id": "183578", "postDate": "05/18/2017 16:41:22", "content": "whew, thanks Lowik!
", "rawMarkdown": "whew, thanks Lowik!", "votes": null }, { "id": "183580", "postDate": "05/18/2017 16:50:14", "content": "> Public and private LB are different subsets of the files in input/Test?
\n\nRight, if you go to the Leaderboard page, it says: \"This leaderboard is calculated with approximately 50% of the test data. The final results will be based on the other 50%, so the final standings may be different.\"
\n\nBesides that, only a small portion of the \"input/Test\" is labeled (no one knows, but probably on the order of 1000 images), so probably the public LB is about 500-1000 images, and the private LB is the same.
", "rawMarkdown": "> Public and private LB are different subsets of the files in input/Test?\n\nRight, if you go to the Leaderboard page, it says: \"This leaderboard is calculated with approximately 50% of the test data. The final results will be based on the other 50%, so the final standings may be different.\"\n\nBesides that, only a small portion of the \"input/Test\" is labeled (no one knows, but probably on the order of 1000 images), so probably the public LB is about 500-1000 images, and the private LB is the same.", "votes": null }, { "id": "189006", "postDate": "06/04/2017 20:02:00", "content": "I have some problems calculating RMSE:
\n\nFirst of all seems new version of Keras doesn't have RMSE (only MSE):
\n\nBut it's not hard to implement:
\n\ndef mean_squared_error(y_true, y_pred):\n \"\"\"\n MSE loss function\n \"\"\"\n return K.mean(K.square(y_pred - y_true), axis=-1)\n\ndef root_mean_squared_error(y_true, y_pred):\n \"\"\"\n RMSE loss function\n \"\"\"\n return K.sqrt(K.mean(K.square(y_pred - y_true), axis=-1))\nThan I have tried to calculate it by hand and it doesn't the same as keras output:
\n\ndef compute_RMSE(y_true, y_pred):\n RMSE= np.sqrt(np.mean((y_true-y_pred)**2))\n\n return RMSE\n\ndef evalute_on_val(model,X_val,y_val):\n y_pred= model.predict(X_val)\n RMSE= compute_RMSE(y_val, y_pred)\n\n print 'RMSE:', RMSE\n\n\nmodel.fit(X_train, Y_train, validation_data=(X_val, Y_val), batch_size=batch_size, epochs=1)\nprint model.evaluate(X_val, Y_val)\nevalute_on_val(model, X_val, Y_val)\n\n\n\nEpoch 1/1 32/32 [==============================] - 4s - loss: 4.0545 -\n val_loss: 1.9832 224/228 [============================>.] - ETA:\n 0s1.98315653048 RMSE: 2.14734170033 Train on 32 samples, validate on\n 228 samples Epoch 1/1 32/32 [==============================] - 2s -\n loss: 3.0376 - val_loss: 1.9279 224/228\n [============================>.] - ETA: 0s1.92787208892 RMSE:\n 2.08691443128
\n
So it's 1.98 vs 2.08
\n\nAlso here is seems related issue: https://github.com/fchollet/keras/issues/1170
", "rawMarkdown": "I have some problems calculating RMSE:\n\nFirst of all seems new version of Keras doesn't have RMSE (only MSE):\n\nBut it's not hard to implement:\n\n def mean_squared_error(y_true, y_pred):\n \t\"\"\"\n \tMSE loss function\n \t\"\"\"\n \treturn K.mean(K.square(y_pred - y_true), axis=-1)\n \n def root_mean_squared_error(y_true, y_pred):\n \t\"\"\"\n \tRMSE loss function\n \t\"\"\"\n \treturn K.sqrt(K.mean(K.square(y_pred - y_true), axis=-1))\n\nThan I have tried to calculate it by hand and it doesn't the same as keras output:\n\n def compute_RMSE(y_true, y_pred):\n \tRMSE= np.sqrt(np.mean((y_true-y_pred)**2))\n \t\n \treturn RMSE\n \n def evalute_on_val(model,X_val,y_val):\n \ty_pred= model.predict(X_val)\n \tRMSE= compute_RMSE(y_val, y_pred)\n \n \tprint 'RMSE:', RMSE\n\n\n model.fit(X_train, Y_train, validation_data=(X_val, Y_val), batch_size=batch_size, epochs=1)\n print model.evaluate(X_val, Y_val)\n evalute_on_val(model, X_val, Y_val)\n\n> Epoch 1/1 32/32 [==============================] - 4s - loss: 4.0545 -\n> val_loss: 1.9832 224/228 [============================>.] - ETA:\n> 0s1.98315653048 RMSE: 2.14734170033 Train on 32 samples, validate on\n> 228 samples Epoch 1/1 32/32 [==============================] - 2s -\n> loss: 3.0376 - val_loss: 1.9279 224/228\n> [============================>.] - ETA: 0s1.92787208892 RMSE:\n> 2.08691443128\n\nSo it's 1.98 vs 2.08\n\nAlso here is seems related issue: https://github.com/fchollet/keras/issues/1170", "votes": null }, { "id": "192393", "postDate": "06/13/2017 12:08:36", "content": "BTW Leaderboard is 'Mean Columnwise Root Mean Squared Error'
", "rawMarkdown": "BTW Leaderboard is 'Mean Columnwise Root Mean Squared Error'", "votes": null }, { "id": "192513", "postDate": "06/13/2017 20:53:29", "content": "Hmm, but I still wonder what is row-wise rmse?
\n\n#This is like in sklearn\ndef compute_rmse(y_true, y_pred):\n RMSE= np.sqrt(np.mean((y_true-y_pred)**2))\n\n return RMSE\n\ndef test_rmse():\n y_arr= np.array([[1,2,3,4,5],[6,7,8,9,10],[11,12,13,14,15]])\n\n print 'RMSE:', compute_rmse(y_arr, np.zeros(y_arr.shape))\n\n from sklearn.metrics import mean_squared_error\n print 'RMSE sklearn:', mean_squared_error(y_arr, np.zeros(y_arr.shape))**0.5\n\n", "rawMarkdown": "Hmm, but I still wonder what is row-wise rmse?\n\n #This is like in sklearn\n def compute_rmse(y_true, y_pred):\n \tRMSE= np.sqrt(np.mean((y_true-y_pred)**2))\n \t\n \treturn RMSE\n \n def test_rmse():\n \ty_arr= np.array([[1,2,3,4,5],[6,7,8,9,10],[11,12,13,14,15]])\n \n \tprint 'RMSE:', compute_rmse(y_arr, np.zeros(y_arr.shape))\n \t\n \tfrom sklearn.metrics import mean_squared_error\n \tprint 'RMSE sklearn:', mean_squared_error(y_arr, np.zeros(y_arr.shape))**0.5\n\n> RMSE: 9.09212113132 RMSE sklearn: 9.09212113132", "votes": null }, { "id": "195734", "postDate": "06/24/2017 19:43:10", "content": "RMSE: 9.09212113132 RMSE sklearn: 9.09212113132
\n
By \"local\" you refer what exactly? The corrected SeaLionData tid's with the masks from TrainDotted applied to the images from Train?
", "rawMarkdown": "By \"local\" you refer what exactly? The corrected SeaLionData tid's with the masks from TrainDotted applied to the images from Train?", "votes": null }, { "id": "195744", "postDate": "06/24/2017 21:17:04", "content": "\n\n\nBy \"local\" you refer what exactly?
\n
Run against validation set that I constructed from a part of Train. Here \"local\" means local as opposed to the public leaderboard which is run on the Kaggle servers.
", "rawMarkdown": "> By \"local\" you refer what exactly?\n\nRun against validation set that I constructed from a part of Train. Here \"local\" means local as opposed to the public leaderboard which is run on the Kaggle servers.", "votes": null }, { "id": "195745", "postDate": "06/24/2017 21:20:05", "content": "I see, Thanks! But I guess you did add the masks to them from TrainDotted? By the way, what RMSE do you now get on the local validation set?
", "rawMarkdown": "I see, Thanks! But I guess you did add the masks to them from TrainDotted? By the way, what RMSE do you now get on the local validation set?", "votes": null }, { "id": "195838", "postDate": "06/25/2017 08:21:32", "content": "\n\n\nBut I guess you did add the masks to them from TrainDotted?
\n
Yes, for training
\n\n\n\n\nBy the way, what RMSE do you now get on the local validation set?
\n
10 -- 16
", "rawMarkdown": "> But I guess you did add the masks to them from TrainDotted?\n\nYes, for training\n\n> By the way, what RMSE do you now get on the local validation set?\n\n10 -- 16", "votes": null } ], "comments": [ { "id": 179651, "author_name": "asanakoev", "author_url": "", "post_date": "05/02/2017 12:15:08", "content": "Konstantin, do you train a CNN on first ~750 images and than tune classifiers on hold-out 250 ?
", "votes": null, "replies": [ { "id": 179657, "author_name": "lopuhin", "author_url": "", "post_date": "05/02/2017 12:27:35", "content": "Yes - actually, I'm training a CNN on about 650 images, and use about 100 for validation of the CNN, and then tune regression models that take CNN predictions as input on the hold-out 250.
\n\nEdit: and I'm also excluding images from MismatchedTrainImages.txt + 941, 200, 491, 912
", "votes": null, "replies": [] }, { "id": 182912, "author_name": "traces", "author_url": "", "post_date": "05/16/2017 09:44:20", "content": "Just to understand your approach, and I apologize in advance if I misunderstood something; couldn't you also validate your model on the test set? That will only allow you to validate the counts, not the predicted coordinates of the animals, but wouldn't that be enough?
", "votes": null, "replies": [] }, { "id": 182914, "author_name": "lopuhin", "author_url": "", "post_date": "05/16/2017 09:47:48", "content": "@erde not sure I understand what you mean - I can't use the test set for validation because we don't have the counts for the test set - we need to predict them.
", "votes": null, "replies": [] }, { "id": 183150, "author_name": "traces", "author_url": "", "post_date": "05/17/2017 05:16:35", "content": "Thanks for the clarification. I misunderstood the structure of the dataset. But you could automate uploading the test set predictions to Kaggle to get a score, or would that evaluation be too noisy/slow?
", "votes": null, "replies": [] }, { "id": 183278, "author_name": "lopuhin", "author_url": "", "post_date": "05/17/2017 16:20:58", "content": "@erde that's very dangerous, as you can overfit to the public LB and then fail on the private LB. So it's better to have a working local CV. Also, you can have only 5 submissions per day, which is not a lot.
", "votes": null, "replies": [] }, { "id": 183442, "author_name": "traces", "author_url": "", "post_date": "05/18/2017 08:54:47", "content": "Public and private LB are different subsets of the files in input/Test?
> Public and private LB are different subsets of the files in input/Test?
\n\nRight, if you go to the Leaderboard page, it says: \"This leaderboard is calculated with approximately 50% of the test data. The final results will be based on the other 50%, so the final standings may be different.\"
\n\nBesides that, only a small portion of the \"input/Test\" is labeled (no one knows, but probably on the order of 1000 images), so probably the public LB is about 500-1000 images, and the private LB is the same.
", "votes": null, "replies": [] } ] }, { "id": 179688, "author_name": "toshik", "author_url": "", "post_date": "05/02/2017 14:11:23", "content": "My current approach make massive gap too.
\nIn my opinion, we should not believe public LB.
\n\n\nIn my opinion, we should not believe public LB.
\n
Do you think that the private LB will be better? Do you have any hypothesis about that gap between local CV and public LB?
", "votes": null, "replies": [] }, { "id": 179906, "author_name": "toshik", "author_url": "", "post_date": "05/03/2017 08:14:47", "content": "Some competitors point out that number of sea-lions in train.csv don't consistent real number. This implies train.csv includes much noise. I don't know that private LB will be better, but I hope our final submission evaluated with real number.
By the way, admin shared MismatchedTrainImages.txt after starting competition. I wonder if there is MismatchedTestImages.txt potentially or not.
Hello.\nHave the same problem =(\nI tried to submit all zeros, and it gave me the score 29.
", "votes": null, "replies": [] }, { "id": 182988, "author_name": "mitiau", "author_url": "", "post_date": "05/16/2017 16:19:24", "content": "I think that the discrepancy between local validation score and public leaderboard may arise from difference in calculation of RMSE. The metric for the competition is average columnwise RMSE, the metric used for example by Keras is rowwise RMSE.
\n\nFor a 3x5 table \n[1 2 3 4 5]\n[6 7 8 9 10]\n[11 12 13 14 15]\nwith ground truth of 0
\n\nRMSE columnwise = 9.1\nRMS rowwise = 15.8
\n\nOn the bright side they move in the same direction, so if your rowwise RMSE is going down, that means your columnwise RMSE will go down too.
", "votes": null, "replies": [ { "id": 182994, "author_name": "lopuhin", "author_url": "", "post_date": "05/16/2017 16:46:42", "content": "Yes, maybe this is the reason, because I'm getting completely different values.\nFor your example (assuming you meant columns of length 3 and rows of length 5), my mean column-wise RMSE is 9.0052, and my mean row-wise RMSE is 8.1724.
\n\nCan someone else try calculating it? :)
", "votes": null, "replies": [] }, { "id": 183446, "author_name": "nzeuwik", "author_url": "", "post_date": "05/18/2017 09:14:09", "content": "Same values for me: 9.00520 column-wise, 8.17245 row-wise.
", "votes": null, "replies": [] }, { "id": 183578, "author_name": "lopuhin", "author_url": "", "post_date": "05/18/2017 16:41:22", "content": "whew, thanks Lowik!
", "votes": null, "replies": [] }, { "id": 192393, "author_name": "mrgloom", "author_url": "", "post_date": "06/13/2017 12:08:36", "content": "BTW Leaderboard is 'Mean Columnwise Root Mean Squared Error'
", "votes": null, "replies": [] }, { "id": 192513, "author_name": "mrgloom", "author_url": "", "post_date": "06/13/2017 20:53:29", "content": "Hmm, but I still wonder what is row-wise rmse?
\n\n#This is like in sklearn\ndef compute_rmse(y_true, y_pred):\n RMSE= np.sqrt(np.mean((y_true-y_pred)**2))\n\n return RMSE\n\ndef test_rmse():\n y_arr= np.array([[1,2,3,4,5],[6,7,8,9,10],[11,12,13,14,15]])\n\n print 'RMSE:', compute_rmse(y_arr, np.zeros(y_arr.shape))\n\n from sklearn.metrics import mean_squared_error\n print 'RMSE sklearn:', mean_squared_error(y_arr, np.zeros(y_arr.shape))**0.5\n\n", "votes": null, "replies": [] } ] }, { "id": 182990, "author_name": "aamaia", "author_url": "", "post_date": "05/16/2017 16:29:58", "content": "RMSE: 9.09212113132 RMSE sklearn: 9.09212113132
\n
This gap could be explained by the metric. RMSE between different datasets can only be compared if the average counts are similar. In case you did the train/validation split blindly you could end up with a validation set with lower counts (when counts are lower the same model scores better).
", "votes": null, "replies": [ { "id": 182997, "author_name": "lopuhin", "author_url": "", "post_date": "05/16/2017 16:57:45", "content": "Yes, some difference (e.g. scale) between train and test set is my leading hypothesis at the moment, assuming I implemented RMSE calculation correctly. I don't think it's explained just by a bad split though.
", "votes": null, "replies": [] } ] }, { "id": 183136, "author_name": "outrunner", "author_url": "", "post_date": "05/17/2017 02:58:46", "content": "I got a huge gap, too. I think row-wise RMSE is lower than column-wise in this competition.\nYou can check your submission's column-wise RMS (RMSE to all zero) vs. 29 (public LB RMS).\nThe public LB score always worse than abs(submission's RMS - 29).\nIf model isn't good enough, the public LB score will worse than abs(submission's RMS - 29) + local validation score.
", "votes": null, "replies": [] }, { "id": 189006, "author_name": "mrgloom", "author_url": "", "post_date": "06/04/2017 20:02:00", "content": "I have some problems calculating RMSE:
\n\nFirst of all seems new version of Keras doesn't have RMSE (only MSE):
\n\nBut it's not hard to implement:
\n\ndef mean_squared_error(y_true, y_pred):\n \"\"\"\n MSE loss function\n \"\"\"\n return K.mean(K.square(y_pred - y_true), axis=-1)\n\ndef root_mean_squared_error(y_true, y_pred):\n \"\"\"\n RMSE loss function\n \"\"\"\n return K.sqrt(K.mean(K.square(y_pred - y_true), axis=-1))\nThan I have tried to calculate it by hand and it doesn't the same as keras output:
\n\ndef compute_RMSE(y_true, y_pred):\n RMSE= np.sqrt(np.mean((y_true-y_pred)**2))\n\n return RMSE\n\ndef evalute_on_val(model,X_val,y_val):\n y_pred= model.predict(X_val)\n RMSE= compute_RMSE(y_val, y_pred)\n\n print 'RMSE:', RMSE\n\n\nmodel.fit(X_train, Y_train, validation_data=(X_val, Y_val), batch_size=batch_size, epochs=1)\nprint model.evaluate(X_val, Y_val)\nevalute_on_val(model, X_val, Y_val)\n\n\n\nEpoch 1/1 32/32 [==============================] - 4s - loss: 4.0545 -\n val_loss: 1.9832 224/228 [============================>.] - ETA:\n 0s1.98315653048 RMSE: 2.14734170033 Train on 32 samples, validate on\n 228 samples Epoch 1/1 32/32 [==============================] - 2s -\n loss: 3.0376 - val_loss: 1.9279 224/228\n [============================>.] - ETA: 0s1.92787208892 RMSE:\n 2.08691443128
\n
So it's 1.98 vs 2.08
\n\nAlso here is seems related issue: https://github.com/fchollet/keras/issues/1170
", "votes": null, "replies": [] }, { "id": 195734, "author_name": "traces", "author_url": "", "post_date": "06/24/2017 19:43:10", "content": "By \"local\" you refer what exactly? The corrected SeaLionData tid's with the masks from TrainDotted applied to the images from Train?
", "votes": null, "replies": [ { "id": 195744, "author_name": "lopuhin", "author_url": "", "post_date": "06/24/2017 21:17:04", "content": "\n\n\nBy \"local\" you refer what exactly?
\n
Run against validation set that I constructed from a part of Train. Here \"local\" means local as opposed to the public leaderboard which is run on the Kaggle servers.
", "votes": null, "replies": [] }, { "id": 195745, "author_name": "traces", "author_url": "", "post_date": "06/24/2017 21:20:05", "content": "I see, Thanks! But I guess you did add the masks to them from TrainDotted? By the way, what RMSE do you now get on the local validation set?
", "votes": null, "replies": [] }, { "id": 195838, "author_name": "lopuhin", "author_url": "", "post_date": "06/25/2017 08:21:32", "content": "\n\n\nBut I guess you did add the masks to them from TrainDotted?
\n
Yes, for training
\n\n\n\n\nBy the way, what RMSE do you now get on the local validation set?
\n
10 -- 16
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "179595": "Has anyone built a reliable local cross-validation strategy? For me the gap is massive, up to 12 on local vs 24 on LB.\n\nI'm leaving about 250 random images for the model that predicts counts (so the CNN does not see them at all). Then I'm doing a 5-fold cross-validation on this 250 images using a couple of models such as XGBoost and Lasso.\n\nWhat could explain such a huge gap? It could obviously be a bug in my code, or very different images in test vs. train (I didn't check predictions on test but the images look similar). Apart from that, I don't see any obvious reasons, or ways I'm leaking train into my validation.", "179651": "Konstantin, do you train a CNN on first ~750 images and than tune classifiers on hold-out 250 ?", "179657": "Yes - actually, I'm training a CNN on about 650 images, and use about 100 for validation of the CNN, and then tune regression models that take CNN predictions as input on the hold-out 250.\n\nEdit: and I'm also excluding images from MismatchedTrainImages.txt + 941, 200, 491, 912", "179688": "My current approach make massive gap too.