![\"Simulating](\"https://www.kaggle.io/svf/319705/219bd839a1c0f1f479e0ae4a4fadbf9e/Rplot001.png\")
Congratulations to winners!
\n\nCompetition is over, and now it is time to share ideas.
\n\nMy biggest problem here was that I did not figure out how to do reliable cross-validation for this competition and as a result, I just overfitted Public Leaderboard.
\n\nI am assuming that people that had the same result at Public and Private Leaderboards have some sacred knowledge about cross-validation techniques that allowed them to validate their scores locally.
\n\nCan anyone share what they did?
", "messages": [ { "id": "129723", "postDate": "08/02/2016 00:09:50", "content": "Congratulations to winners!
\n\nCompetition is over, and now it is time to share ideas.
\n\nMy biggest problem here was that I did not figure out how to do reliable cross-validation for this competition and as a result, I just overfitted Public Leaderboard.
\n\nI am assuming that people that had the same result at Public and Private Leaderboards have some sacred knowledge about cross-validation techniques that allowed them to validate their scores locally.
\n\nCan anyone share what they did?
", "rawMarkdown": "Congratulations to winners!\r\n\r\nCompetition is over, and now it is time to share ideas. \r\n\r\nMy biggest problem here was that I did not figure out how to do reliable cross-validation for this competition and as a result, I just overfitted Public Leaderboard.\r\n\r\n\r\nI am assuming that people that had the same result at Public and Private Leaderboards have some sacred knowledge about cross-validation techniques that allowed them to validate their scores locally.\r\n\r\nCan anyone share what they did?", "votes": null }, { "id": "129879", "postDate": "08/02/2016 17:45:46", "content": "hi, here is my results:\npublic = 0.16820 (rank 37), private = 0.16522 (rank 20)
\n\ncross-validation dataset: divide train image by {driver,action} clusters. Use 70 to 80% of clusters for training and the rest for validation.
\n\nregularisation: this is very important. I try to make training difficult by synthesizing hard train samples.\nhard train sample = image + "cut and paste" of some region of another image of another class
\n\nensemble: 25 CNNs of googlenet_v1/v2, vgg_16/19, resnet50
", "rawMarkdown": "hi, here is my results:\r\npublic = 0.16820 (rank 37), private = 0.16522 (rank 20)\r\n\r\ncross-validation dataset: divide train image by {driver,action} clusters. Use 70 to 80% of clusters for training and the rest for validation.\r\n\r\nregularisation: this is very important. I try to make training difficult by synthesizing hard train samples.\r\nhard train sample = image + \"cut and paste\" of some region of another image of another class\r\n\r\nensemble: 25 CNNs of googlenet_v1/v2, vgg_16/19, resnet50", "votes": null }, { "id": "129880", "postDate": "08/02/2016 17:49:06", "content": "To do a reliable CV you just need to be sure that the same drivers you are using for training are not in the validation (holdout) fold.
", "rawMarkdown": "To do a reliable CV you just need to be sure that the same drivers you are using for training are not in the validation (holdout) fold.", "votes": null }, { "id": "129891", "postDate": "08/02/2016 18:42:14", "content": "The inconsistency between CV and LB results have been bothering me too. I used 26-fold (based on drivers) CV, and only found out that I need to training longer than was suggested by the CV results to obtain a decent LB score. So I ended up using all the training data to build models and solely relied on the public LB for validation. To alleviate overfitting to the public LB, I trained 150+ models, as I believe the large ensemble is less likely to overfit.
\n\nPublic LB: 0.15592
\n\nPrivate LB: 0.15985
", "rawMarkdown": "The inconsistency between CV and LB results have been bothering me too. I used 26-fold (based on drivers) CV, and only found out that I need to training longer than was suggested by the CV results to obtain a decent LB score. So I ended up using all the training data to build models and solely relied on the public LB for validation. To alleviate overfitting to the public LB, I trained 150+ models, as I believe the large ensemble is less likely to overfit. \r\n\r\nPublic LB: 0.15592\r\n\r\nPrivate LB: 0.15985", "votes": null }, { "id": "129928", "postDate": "08/02/2016 22:12:45", "content": "I started by "leave one driver" out (or 26 folds based on driver id) and cannot get reliable CV. (validation loss ranged from 0.001 to 0.7, some models stop at 6 epochs and some only have 1 epochs ).
\n\nSo I think I should increase the size of hold out ,like use 5 fold or even 3 folds based on driver id.
\n\nAnyway I didn't have time to try that so I end up using 5 fold based on random split. With ensemble ,the public LB is 0.20979 and private LB: 0.20367
\n\nI suppose for this dataset the optimum parameters of tuning a VGG19 is more or less the same no matter CV by driver_id or by random split.
", "rawMarkdown": "I started by \"leave one driver\" out (or 26 folds based on driver id) and cannot get reliable CV. (validation loss ranged from 0.001 to 0.7, some models stop at 6 epochs and some only have 1 epochs ). \r\n\r\nSo I think I should increase the size of hold out ,like use 5 fold or even 3 folds based on driver id.\r\n\r\nAnyway I didn't have time to try that so I end up using 5 fold based on random split. With ensemble ,the public LB is 0.20979 and private LB: 0.20367\r\n\r\n\r\nI suppose for this dataset the optimum parameters of tuning a VGG19 is more or less the same no matter CV by driver_id or by random split.", "votes": null }, { "id": "129934", "postDate": "08/02/2016 23:10:55", "content": "The right way to do CV for this competition is to split by driver. However, remember that CV is for model selection to reduce variance (so you get the same results with any new data). After you get a stable model, you should train it with the entire data set. Many people combine these two steps. This saves computing time, but can lead to increased bias (lower accuracy) because you are using subsets of the data to train. This competition was particularly difficult because we really only had 26 unique drivers. It seems that Heng CherKeng got around this by simulating new cases with cut-and-paste. Many people ended up randomly selecting from the images and tuning based on the public LB. This worked to some degree, but you definitely have the risk of over-fitting to the LB.
\n\nHere is an R script and plot that demonstrates the effect of small sample size. Feel free to run yourself with different seeds.
\n\nHere is a link to the Kernel
\n\n## Simulating Small Data Set Bias and Variance\n# Jeff Hebert 8/2/2016\n\ncnt <- 26\nset.seed(123)\nx <- runif(cnt,0,10)\ny <- x + rnorm(cnt, mean = 0, sd = 1)\nfolds <- as.integer(cut(1:cnt, 4))\n\nplot(x,y, col = folds, pch = 19)\n\nmodel <- lm(y ~ x)\nabline(model, lwd = 3)\n\nfor(fold in folds){\n idx <- folds == fold\n x_tr <- x[idx]\n y_tr <- y[idx]\n x_te <- x[-idx]\n y_te <- y[-idx]\n\n model <- lm(y_tr ~ x_tr)\n abline(model, col = fold)\n}\n
My takeaway was that overffiting is OK as long as models overfitted on different examples and then averaged. I used 8-fold CV by drivers. Difference between 0.33 on single fold model and 0.20 must mean overfitting was completely removed by avereging.
", "rawMarkdown": "My takeaway was that overffiting is OK as long as models overfitted on different examples and then averaged. I used 8-fold CV by drivers. Difference between 0.33 on single fold model and 0.20 must mean overfitting was completely removed by avereging.", "votes": null }, { "id": "129940", "postDate": "08/03/2016 00:25:24", "content": "[quote=kyv;129936]
\n\nMy takeaway was that overffiting is OK as long as models overfitted on different examples and then averaged. I used 8-fold CV by drivers. Difference between 0.33 on single fold model and 0.20 must mean overfitting was completely removed by avereging.
\n\n[/quote]\nI had hope for the same assumption(Basically we mimic how Random Forest works) and it affected me in a way:
\n\n0.18 Public => 0.2 Private
", "rawMarkdown": "[quote=kyv;129936]\r\n\r\nMy takeaway was that overffiting is OK as long as models overfitted on different examples and then averaged. I used 8-fold CV by drivers. Difference between 0.33 on single fold model and 0.20 must mean overfitting was completely removed by avereging.\r\n\r\n[/quote]\r\nI had hope for the same assumption(Basically we mimic how Random Forest works) and it affected me in a way:\r\n\r\n0.18 Public => 0.2 Private", "votes": null }, { "id": "129942", "postDate": "08/03/2016 00:35:12", "content": "[quote=JeffH;129934]
\n\nThe right way to do CV for this competition is to split by driver. However, remember that CV is for model selection to reduce variance (so you get the same results with any new data). After you get a stable model, you should train it with the entire data set. Many people combine these two steps. This saves computing time, but can lead to increased bias (lower accuracy) because you are using subsets of the data to train. This competition was particularly difficult because we really only had 26 unique drivers. It seems that Heng CherKeng got around this by simulating new cases with cut-and-paste. Many people ended up randomly selecting from the images and tuning based on the public LB. This worked to some degree, but you definitely have the risk of over-fitting to the LB.
\n\nHere is an R script and plot that demonstrates the effect of small sample size. Feel free to run yourself with different seeds.
\n\nHere is a link to the Kernel
\n\n## Simulating Small Data Set Bias and Variance\n# Jeff Hebert 8/2/2016\n\ncnt <- 26\nset.seed(123)\nx <- runif(cnt,0,10)\ny <- x + rnorm(cnt, mean = 0, sd = 1)\nfolds <- as.integer(cut(1:cnt, 4))\n\nplot(x,y, col = folds, pch = 19)\n\nmodel <- lm(y ~ x)\nabline(model, lwd = 3)\n\nfor(fold in folds){\n idx <- folds == fold\n x_tr <- x[idx]\n y_tr <- y[idx]\n x_te <- x[-idx]\n y_te <- y[-idx]\n\n model <- lm(y_tr ~ x_tr)\n abline(model, col = fold)\n}\n
[/quote]
\n\nI have some different opinion.
\n\nhi, here is my results:\npublic = 0.16820 (rank 37), private = 0.16522 (rank 20)
\n\ncross-validation dataset: divide train image by {driver,action} clusters. Use 70 to 80% of clusters for training and the rest for validation.
\n\nregularisation: this is very important. I try to make training difficult by synthesizing hard train samples.\nhard train sample = image + "cut and paste" of some region of another image of another class
\n\nensemble: 25 CNNs of googlenet_v1/v2, vgg_16/19, resnet50
", "votes": null, "replies": [] }, { "id": 129880, "author_name": "titericz", "author_url": "", "post_date": "08/02/2016 17:49:06", "content": "To do a reliable CV you just need to be sure that the same drivers you are using for training are not in the validation (holdout) fold.
", "votes": null, "replies": [] }, { "id": 129891, "author_name": "wowfattie", "author_url": "", "post_date": "08/02/2016 18:42:14", "content": "The inconsistency between CV and LB results have been bothering me too. I used 26-fold (based on drivers) CV, and only found out that I need to training longer than was suggested by the CV results to obtain a decent LB score. So I ended up using all the training data to build models and solely relied on the public LB for validation. To alleviate overfitting to the public LB, I trained 150+ models, as I believe the large ensemble is less likely to overfit.
\n\nPublic LB: 0.15592
\n\nPrivate LB: 0.15985
", "votes": null, "replies": [] }, { "id": 129928, "author_name": "stevendu", "author_url": "", "post_date": "08/02/2016 22:12:45", "content": "I started by "leave one driver" out (or 26 folds based on driver id) and cannot get reliable CV. (validation loss ranged from 0.001 to 0.7, some models stop at 6 epochs and some only have 1 epochs ).
\n\nSo I think I should increase the size of hold out ,like use 5 fold or even 3 folds based on driver id.
\n\nAnyway I didn't have time to try that so I end up using 5 fold based on random split. With ensemble ,the public LB is 0.20979 and private LB: 0.20367
\n\nI suppose for this dataset the optimum parameters of tuning a VGG19 is more or less the same no matter CV by driver_id or by random split.
", "votes": null, "replies": [] }, { "id": 129934, "author_name": "jeffhebert", "author_url": "", "post_date": "08/02/2016 23:10:55", "content": "The right way to do CV for this competition is to split by driver. However, remember that CV is for model selection to reduce variance (so you get the same results with any new data). After you get a stable model, you should train it with the entire data set. Many people combine these two steps. This saves computing time, but can lead to increased bias (lower accuracy) because you are using subsets of the data to train. This competition was particularly difficult because we really only had 26 unique drivers. It seems that Heng CherKeng got around this by simulating new cases with cut-and-paste. Many people ended up randomly selecting from the images and tuning based on the public LB. This worked to some degree, but you definitely have the risk of over-fitting to the LB.
\n\nHere is an R script and plot that demonstrates the effect of small sample size. Feel free to run yourself with different seeds.
\n\nHere is a link to the Kernel
\n\n## Simulating Small Data Set Bias and Variance\n# Jeff Hebert 8/2/2016\n\ncnt <- 26\nset.seed(123)\nx <- runif(cnt,0,10)\ny <- x + rnorm(cnt, mean = 0, sd = 1)\nfolds <- as.integer(cut(1:cnt, 4))\n\nplot(x,y, col = folds, pch = 19)\n\nmodel <- lm(y ~ x)\nabline(model, lwd = 3)\n\nfor(fold in folds){\n idx <- folds == fold\n x_tr <- x[idx]\n y_tr <- y[idx]\n x_te <- x[-idx]\n y_te <- y[-idx]\n\n model <- lm(y_tr ~ x_tr)\n abline(model, col = fold)\n}\n
My takeaway was that overffiting is OK as long as models overfitted on different examples and then averaged. I used 8-fold CV by drivers. Difference between 0.33 on single fold model and 0.20 must mean overfitting was completely removed by avereging.
", "votes": null, "replies": [] }, { "id": 129940, "author_name": "iglovikov", "author_url": "", "post_date": "08/03/2016 00:25:24", "content": "[quote=kyv;129936]
\n\nMy takeaway was that overffiting is OK as long as models overfitted on different examples and then averaged. I used 8-fold CV by drivers. Difference between 0.33 on single fold model and 0.20 must mean overfitting was completely removed by avereging.
\n\n[/quote]\nI had hope for the same assumption(Basically we mimic how Random Forest works) and it affected me in a way:
\n\n0.18 Public => 0.2 Private
", "votes": null, "replies": [] }, { "id": 129942, "author_name": "wowfattie", "author_url": "", "post_date": "08/03/2016 00:35:12", "content": "[quote=JeffH;129934]
\n\nThe right way to do CV for this competition is to split by driver. However, remember that CV is for model selection to reduce variance (so you get the same results with any new data). After you get a stable model, you should train it with the entire data set. Many people combine these two steps. This saves computing time, but can lead to increased bias (lower accuracy) because you are using subsets of the data to train. This competition was particularly difficult because we really only had 26 unique drivers. It seems that Heng CherKeng got around this by simulating new cases with cut-and-paste. Many people ended up randomly selecting from the images and tuning based on the public LB. This worked to some degree, but you definitely have the risk of over-fitting to the LB.
\n\nHere is an R script and plot that demonstrates the effect of small sample size. Feel free to run yourself with different seeds.
\n\nHere is a link to the Kernel
\n\n## Simulating Small Data Set Bias and Variance\n# Jeff Hebert 8/2/2016\n\ncnt <- 26\nset.seed(123)\nx <- runif(cnt,0,10)\ny <- x + rnorm(cnt, mean = 0, sd = 1)\nfolds <- as.integer(cut(1:cnt, 4))\n\nplot(x,y, col = folds, pch = 19)\n\nmodel <- lm(y ~ x)\nabline(model, lwd = 3)\n\nfor(fold in folds){\n idx <- folds == fold\n x_tr <- x[idx]\n y_tr <- y[idx]\n x_te <- x[-idx]\n y_te <- y[-idx]\n\n model <- lm(y_tr ~ x_tr)\n abline(model, col = fold)\n}\n
[/quote]
\n\nI have some different opinion.
\n\n