i think noisy data may be be a norm to future competition. it will no longer about just getting good models and good hyperparameters.
\n\nwe need to re-think about how to survive in the new world of uncertainty, especially you are given both public and private test data!
", "messages": [ { "id": "676081", "postDate": "11/19/2019 00:30:09", "content": "i think noisy data may be be a norm to future competition. it will no longer about just getting good models and good hyperparameters.
\n\nwe need to re-think about how to survive in the new world of uncertainty, especially you are given both public and private test data!
", "rawMarkdown": "i think noisy data may be be a norm to future competition. it will no longer about just getting good models and good hyperparameters.\n\nwe need to re-think about how to survive in the new world of uncertainty, especially you are given both public and private test data!", "votes": null }, { "id": "676090", "postDate": "11/19/2019 00:36:48", "content": "We used 11 folds. 1 as hold out. I guess it was too few :)
", "rawMarkdown": "We used 11 folds. 1 as hold out. I guess it was too few :)", "votes": null }, { "id": "676098", "postDate": "11/19/2019 00:44:00", "content": "i don't think using fold-out method would work. i think it is about how to reduce variance rather then estimating performance (e.g. via hold out or other methods)
\n\nunfortunately, i have no ideas how to do that yet
", "rawMarkdown": "i don't think using fold-out method would work. i think it is about how to reduce variance rather then estimating performance (e.g. via hold out or other methods)\n\nunfortunately, i have no ideas how to do that yet", "votes": null }, { "id": "676110", "postDate": "11/19/2019 01:00:13", "content": "Hi Heng , first of all thanks for showing the light at the end of the tunnel and giving good ideas .\nWhat do you think about this write up by Bestfitting regarding shakeup and I think he touched upon a little bit on noisy labels as well .\nhttps://www.kaggle.com/c/planet-understanding-the-amazon-from-space/discussion/36809
", "rawMarkdown": "Hi Heng , first of all thanks for showing the light at the end of the tunnel and giving good ideas .\nWhat do you think about this write up by Bestfitting regarding shakeup and I think he touched upon a little bit on noisy labels as well .\nhttps://www.kaggle.com/c/planet-understanding-the-amazon-from-space/discussion/36809", "votes": null }, { "id": "676113", "postDate": "11/19/2019 01:05:41", "content": "I think adversarial validation in building model with structual data which has a different distribution is quite a good method,but I don't know is there any possibility to do the same validation tactic in CV comps
", "rawMarkdown": "I think adversarial validation in building model with structual data which has a different distribution is quite a good method,but I don't know is there any possibility to do the same validation tactic in CV comps", "votes": null }, { "id": "676117", "postDate": "11/19/2019 01:10:51", "content": "i normally uses bestfitting results as a benchmark, because he don't overfit.
\n\nalso,many will note that their best private score is not their best public score. so it is not about getting the best model, but how to get least-variance model. so the output of our model should be LB score +/- std. now we need to estimate std
\n\ni normally choose the one closest to bestfitting results
", "rawMarkdown": "i normally uses bestfitting results as a benchmark, because he don't overfit.\n\nalso,many will note that their best private score is not their best public score. so it is not about getting the best model, but how to get least-variance model. so the output of our model should be LB score +/- std. now we need to estimate std\n\ni normally choose the one closest to bestfitting results", "votes": null }, { "id": "676119", "postDate": "11/19/2019 01:12:31", "content": "thanks for the suggestion, i will try in post submission.
", "rawMarkdown": "thanks for the suggestion, i will try in post submission.", "votes": null }, { "id": "676120", "postDate": "11/19/2019 01:15:19", "content": "However, I see in this comp our teams CV and LB was very correlated .. for most of our submissions starting from .58 range to .66 range . May be because Train, Public and Private data were similarly noisy and there were no hidden data , so verification by manual eye-on-glass check is always possible .
", "rawMarkdown": "However, I see in this comp our teams CV and LB was very correlated .. for most of our submissions starting from .58 range to .66 range . May be because Train, Public and Private data were similarly noisy and there were no hidden data , so verification by manual eye-on-glass check is always possible .", "votes": null }, { "id": "676123", "postDate": "11/19/2019 01:17:22", "content": "My solution was training every single model on a different set of data rather than doing the typical k-fold setup. So each of my 20 model ensemble was trained on a different 80% of the data and then validated on that 20%. This made it difficult to do any post-hoc cleaning and ensemble tuning because I didnt have full OOF sets of predictions, but I believe it is more robust than keeping structured folds.
", "rawMarkdown": "My solution was training every single model on a different set of data rather than doing the typical k-fold setup. So each of my 20 model ensemble was trained on a different 80% of the data and then validated on that 20%. This made it difficult to do any post-hoc cleaning and ensemble tuning because I didnt have full OOF sets of predictions, but I believe it is more robust than keeping structured folds.", "votes": null }, { "id": "676129", "postDate": "11/19/2019 01:24:57", "content": "\" every single model on a different set of data rather\"
\n\nthanks for the suggestion. i try that too, but i find it difficult to compare experiment results if the input data is different. i would need to train the same model with different dataset (to get metric mean and variance) to compare with another model
", "rawMarkdown": "\" every single model on a different set of data rather\"\n\nthanks for the suggestion. i try that too, but i find it difficult to compare experiment results if the input data is different. i would need to train the same model with different dataset (to get metric mean and variance) to compare with another model", "votes": null }, { "id": "676130", "postDate": "11/19/2019 01:27:29", "content": "This greatly hurt my visibility of results but aided in robustness I believe. I would not recommend doing it again. At the beginning, I was doing a single fold and iterating all of my models to a single fold. Then I gave up on trying to improve anything and just spammed models on random 80% of the data and it seemed to have paid off.
\n\nOperating without any real consistent validation is not good though.
", "rawMarkdown": "This greatly hurt my visibility of results but aided in robustness I believe. I would not recommend doing it again. At the beginning, I was doing a single fold and iterating all of my models to a single fold. Then I gave up on trying to improve anything and just spammed models on random 80% of the data and it seemed to have paid off. \n\nOperating without any real consistent validation is not good though.", "votes": null }, { "id": "676204", "postDate": "11/19/2019 02:47:09", "content": "great work! And you resized dataset and kernel help me a lot👍
", "rawMarkdown": "great work! And you resized dataset and kernel help me a lot👍", "votes": null } ], "comments": [ { "id": 676090, "author_name": "tugstugi", "author_url": "", "post_date": "11/19/2019 00:36:48", "content": "We used 11 folds. 1 as hold out. I guess it was too few :)
", "votes": null, "replies": [ { "id": 676098, "author_name": "hengck23", "author_url": "", "post_date": "11/19/2019 00:44:00", "content": "i don't think using fold-out method would work. i think it is about how to reduce variance rather then estimating performance (e.g. via hold out or other methods)
\n\nunfortunately, i have no ideas how to do that yet
", "votes": null, "replies": [] }, { "id": 676110, "author_name": "phoenix9032", "author_url": "", "post_date": "11/19/2019 01:00:13", "content": "Hi Heng , first of all thanks for showing the light at the end of the tunnel and giving good ideas .\nWhat do you think about this write up by Bestfitting regarding shakeup and I think he touched upon a little bit on noisy labels as well .\nhttps://www.kaggle.com/c/planet-understanding-the-amazon-from-space/discussion/36809
", "votes": null, "replies": [] }, { "id": 676117, "author_name": "hengck23", "author_url": "", "post_date": "11/19/2019 01:10:51", "content": "i normally uses bestfitting results as a benchmark, because he don't overfit.
\n\nalso,many will note that their best private score is not their best public score. so it is not about getting the best model, but how to get least-variance model. so the output of our model should be LB score +/- std. now we need to estimate std
\n\ni normally choose the one closest to bestfitting results
", "votes": null, "replies": [] }, { "id": 676120, "author_name": "phoenix9032", "author_url": "", "post_date": "11/19/2019 01:15:19", "content": "However, I see in this comp our teams CV and LB was very correlated .. for most of our submissions starting from .58 range to .66 range . May be because Train, Public and Private data were similarly noisy and there were no hidden data , so verification by manual eye-on-glass check is always possible .
", "votes": null, "replies": [] } ] }, { "id": 676113, "author_name": "sj626591833", "author_url": "", "post_date": "11/19/2019 01:05:41", "content": "I think adversarial validation in building model with structual data which has a different distribution is quite a good method,but I don't know is there any possibility to do the same validation tactic in CV comps
", "votes": null, "replies": [ { "id": 676119, "author_name": "hengck23", "author_url": "", "post_date": "11/19/2019 01:12:31", "content": "thanks for the suggestion, i will try in post submission.
", "votes": null, "replies": [] } ] }, { "id": 676123, "author_name": "ryches", "author_url": "", "post_date": "11/19/2019 01:17:22", "content": "My solution was training every single model on a different set of data rather than doing the typical k-fold setup. So each of my 20 model ensemble was trained on a different 80% of the data and then validated on that 20%. This made it difficult to do any post-hoc cleaning and ensemble tuning because I didnt have full OOF sets of predictions, but I believe it is more robust than keeping structured folds.
", "votes": null, "replies": [ { "id": 676129, "author_name": "hengck23", "author_url": "", "post_date": "11/19/2019 01:24:57", "content": "\" every single model on a different set of data rather\"
\n\nthanks for the suggestion. i try that too, but i find it difficult to compare experiment results if the input data is different. i would need to train the same model with different dataset (to get metric mean and variance) to compare with another model
", "votes": null, "replies": [] }, { "id": 676130, "author_name": "ryches", "author_url": "", "post_date": "11/19/2019 01:27:29", "content": "This greatly hurt my visibility of results but aided in robustness I believe. I would not recommend doing it again. At the beginning, I was doing a single fold and iterating all of my models to a single fold. Then I gave up on trying to improve anything and just spammed models on random 80% of the data and it seemed to have paid off.
\n\nOperating without any real consistent validation is not good though.
", "votes": null, "replies": [] }, { "id": 676204, "author_name": "dandingclam", "author_url": "", "post_date": "11/19/2019 02:47:09", "content": "great work! And you resized dataset and kernel help me a lot👍
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "676081": "i think noisy data may be be a norm to future competition. it will no longer about just getting good models and good hyperparameters.\n\nwe need to re-think about how to survive in the new world of uncertainty, especially you are given both public and private test data!", "676090": "We used 11 folds. 1 as hold out. I guess it was too few :)", "676098": "i don't think using fold-out method would work. i think it is about how to reduce variance rather then estimating performance (e.g. via hold out or other methods)\n\nunfortunately, i have no ideas how to do that yet", "676110": "Hi Heng , first of all thanks for showing the light at the end of the tunnel and giving good ideas .\nWhat do you think about this write up by Bestfitting regarding shakeup and I think he touched upon a little bit on noisy labels as well .\nhttps://www.kaggle.com/c/planet-understanding-the-amazon-from-space/discussion/36809", "676113": "I think adversarial validation in building model with structual data which has a different distribution is quite a good method,but I don't know is there any possibility to do the same validation tactic in CV comps", "676117": "i normally uses bestfitting results as a benchmark, because he don't overfit.\n\nalso,many will note that their best private score is not their best public score. so it is not about getting the best model, but how to get least-variance model. so the output of our model should be LB score +/- std. now we need to estimate std\n\ni normally choose the one closest to bestfitting results", "676119": "thanks for the suggestion, i will try in post submission.", "676120": "However, I see in this comp our teams CV and LB was very correlated .. for most of our submissions starting from .58 range to .66 range . May be because Train, Public and Private data were similarly noisy and there were no hidden data , so verification by manual eye-on-glass check is always possible .", "676123": "My solution was training every single model on a different set of data rather than doing the typical k-fold setup. So each of my 20 model ensemble was trained on a different 80% of the data and then validated on that 20%. This made it difficult to do any post-hoc cleaning and ensemble tuning because I didnt have full OOF sets of predictions, but I believe it is more robust than keeping structured folds.", "676129": "\" every single model on a different set of data rather\"\n\nthanks for the suggestion. i try that too, but i find it difficult to compare experiment results if the input data is different. i would need to train the same model with different dataset (to get metric mean and variance) to compare with another model", "676130": "This greatly hurt my visibility of results but aided in robustness I believe. I would not recommend doing it again. At the beginning, I was doing a single fold and iterating all of my models to a single fold. Then I gave up on trying to improve anything and just spammed models on random 80% of the data and it seemed to have paid off. \n\nOperating without any real consistent validation is not good though.", "676204": "great work! And you resized dataset and kernel help me a lot👍" }, "source": "meta" }