![\"\"](\"https://habrastorage.org/webt/vd/go/zs/vdgozs9pbyoroboyjfgup-rw5ju.jpeg\")
![\"\"](\"https://storage.googleapis.com/kaggle-forum-message-attachments/452209/10982/image-3.png\")
![\"\"](\"https://habrastorage.org/webt/ei/ii/tb/eiiitbu69eui3wlvamrya6jpsny.png\")
At least there is a private LB. In real world, how many times did I see people not validating properly their models? At best they use one train/test split, at worst they validate on training data.
\n\nAnd this is true in academia too: for how long did people use imagenet with the same train/validation split?
", "rawMarkdown": "At least there is a private LB. In real world, how many times did I see people not validating properly their models? At best they use one train/test split, at worst they validate on training data.\n\nAnd this is true in academia too: for how long did people use imagenet with the same train/validation split?", "votes": 19 }, { "id": 536324, "postDate": "2019-05-24T09:02:18.907Z", "content": "Sadly most academic papers are trash and contain too much intimidating equations or phrases. Practical and good papers are extremely rare, which I deem only 1/10 of those pupblished in top conferences...
", "rawMarkdown": "Sadly most academic papers are trash and contain too much intimidating equations or phrases. Practical and good papers are extremely rare, which I deem only 1/10 of those pupblished in top conferences...", "votes": 6 }, { "id": 536342, "postDate": "2019-05-24T09:35:45.187Z", "content": "@cpmp - I looked at https://public.lanl.gov/geophysics/geophysics/nonlinear/2017/Rouet-LeducGRL17.pdf again and couldn't find the word validation at all. \"RF model are highly accurate, with an R2 value of 0.89\" - this is meaningless without error std deviations related to folds. As you have stated in the past a lot Machine Learning papers are terrible! Every machine learning paper should come with a code and data repository - there is no excuse! I am amazed that ML papers can be published if the peer reviewers / readers cannot reproduce anything it is a disservice to science.
", "rawMarkdown": "@cpmp - I looked at https://public.lanl.gov/geophysics/geophysics/nonlinear/2017/Rouet-LeducGRL17.pdf again and couldn't find the word validation at all. \"RF model are highly accurate, with an R2 value of 0.89\" - this is meaningless without error std deviations related to folds. As you have stated in the past a lot Machine Learning papers are terrible! Every machine learning paper should come with a code and data repository - there is no excuse! I am amazed that ML papers can be published if the peer reviewers / readers cannot reproduce anything it is a disservice to science.", "votes": 8 }, { "id": 536350, "postDate": "2019-05-24T09:48:49.497Z", "content": "\n\n\ncouldn't find the word validation
\n
Why am I not surprised?
\n\nBut to be fair, they use a train/test split. Which is the bare minimum.
", "rawMarkdown": "> couldn't find the word validation \n\nWhy am I not surprised?\n\nBut to be fair, they use a train/test split. Which is the bare minimum.\n", "votes": 4 }, { "id": 538956, "postDate": "2019-05-29T10:31:43.063Z", "content": "During train/test split though one may get mutually exclusive datasets but end of the day one ignores the fact that they are drawn from the same distributions.
\n\nIn science it will be a huge achievement if we could focus on distribution types other than normal/bell curve. Almost all the algorithms assume that the data follows a normal curve. This limits the application of the solution to only one of the two cases in which the population was actually a normal curve but what about the other half?
\n\nThen there is PAC very few care about it.\n![\"\"](\"https://image.slidesharecdn.com/alpaydin-chapter-22167/95/alpaydin-chapter-2-11-728.jpg?cb=1272283279\")
Only dropping 145 places? - wow, you must have a great model ;)
", "rawMarkdown": "Only dropping 145 places? - wow, you must have a great model ;)", "votes": 8, "replies": [ { "id": 536423, "postDate": "2019-05-24T12:56:57.977Z", "content": "Well ofcourse its great- since its only blend of your GP-s ;)
", "rawMarkdown": "Well ofcourse its great- since its only blend of your GP-s ;)", "votes": 6 }, { "id": 536433, "postDate": "2019-05-24T13:09:07.460Z", "content": "I believe a good score will be 1.9 with a great score being 1.8 on private - we shall see! Good luck!
", "rawMarkdown": "I believe a good score will be 1.9 with a great score being 1.8 on private - we shall see! Good luck!", "votes": 4 }, { "id": 536449, "postDate": "2019-05-24T13:55:41.683Z", "content": "@scirpus, do you mean CV of 1.8 or at least 1.9?
\n\nI am asking because, I could not get my CV scores to go below 1.94+ with LB=1.555 and 1.99+ with LB = 1.441 on the single model. So I believe the shakeup is definately coming. I am waiting to be proven wrong by all these low sub 1.2/ 1.3 LB scores.
\n\nUpdate: I meant CV 1.94 and 1.99, I did not mean 0.94/0.99 so I have corrected it above.
", "rawMarkdown": "@scirpus, do you mean CV of 1.8 or at least 1.9? \n\nI am asking because, I could not get my CV scores to go below 1.94+ with LB=1.555 and 1.99+ with LB = 1.441 on the single model. So I believe the shakeup is definately coming. I am waiting to be proven wrong by all these low sub 1.2/ 1.3 LB scores.\n\nUpdate: I meant CV 1.94 and 1.99, I did not mean 0.94/0.99 so I have corrected it above.", "votes": 1 }, { "id": 536452, "postDate": "2019-05-24T13:59:37.767Z", "content": "Private LB scores
", "rawMarkdown": "Private LB scores", "votes": 1 }, { "id": 536467, "postDate": "2019-05-24T14:17:34.137Z", "content": "To me, private LB will be mostly above 2, with top 10 around 1.9. Exceptional 1.8 might be just a very few.
", "rawMarkdown": "To me, private LB will be mostly above 2, with top 10 around 1.9. Exceptional 1.8 might be just a very few.", "votes": 2 }, { "id": 536477, "postDate": "2019-05-24T14:31:53.100Z", "content": "@khahuras - I hope we will have between 1.8 & 2 otherwise both Public & Private LBs won't be representative of the training data!
", "rawMarkdown": "@khahuras - I hope we will have between 1.8 & 2 otherwise both Public & Private LBs won't be representative of the training data!", "votes": 1 }, { "id": 536506, "postDate": "2019-05-24T15:15:02.343Z", "content": "I even believe 2.1 should be more popular and no one can get 1.8
", "rawMarkdown": "I even believe 2.1 should be more popular and no one can get 1.8", "votes": 1 }, { "id": 536509, "postDate": "2019-05-24T15:17:58.237Z", "content": "I can almost guarantee my best submission will be in my first 20 submissions - I am such a Leaderboard chaser - I just can't help myself - maybe I need some therapy! ;)
", "rawMarkdown": "I can almost guarantee my best submission will be in my first 20 submissions - I am such a Leaderboard chaser - I just can't help myself - maybe I need some therapy! ;)", "votes": 5 }, { "id": 536525, "postDate": "2019-05-24T16:04:17.610Z", "content": "@scirpus, sorry I meant CV 1.94 and 1.99, I did not mean 0.94/0.99 so I have corrected it above.
", "rawMarkdown": "@scirpus, sorry I meant CV 1.94 and 1.99, I did not mean 0.94/0.99 so I have corrected it above.", "votes": 1 }, { "id": 536530, "postDate": "2019-05-24T16:21:11.697Z", "content": "@sheriytm - those numbers are more in line with mine
", "rawMarkdown": "@sheriytm - those numbers are more in line with mine ", "votes": 1 }, { "id": 536594, "postDate": "2019-05-24T18:51:09.667Z", "content": "\n\n\nWell ofcourse its great- since its only blend of your GP-s ;)
\n
By the way! Scirpus, thanks for your GP kernel. I use it in the blend. :)
", "rawMarkdown": "> Well ofcourse its great- since its only blend of your GP-s ;)\n\nBy the way! Scirpus, thanks for your GP kernel. I use it in the blend. :)", "votes": 1 }, { "id": 536600, "postDate": "2019-05-24T19:14:09.143Z", "content": "@sergeyzlobin - if it crashes and burns don't blame me please as I deliberately made them overfit (just a little bit) - I am still shocked by people hammering @cpmp after the last competition!
\n\nWhat I know:\nWe have only 16 earthquakes in sequence\nThere will be a huge shake up\nThe best thing to take away from this competition is what you have learned.\nAll the best and good luck!
", "rawMarkdown": "@sergeyzlobin - if it crashes and burns don't blame me please as I deliberately made them overfit (just a little bit) - I am still shocked by people hammering @cpmp after the last competition!\n\nWhat I know:\nWe have only 16 earthquakes in sequence\nThere **will** be a huge shake up\nThe best thing to take away from this competition is what you have learned.\nAll the best and good luck!\n\n", "votes": 4 }, { "id": 536623, "postDate": "2019-05-24T20:59:52.483Z", "content": "There's actually only 15 full EQs if you ignore the first and last bits
", "rawMarkdown": "There's actually only 15 full EQs if you ignore the first and last bits" }, { "id": 536631, "postDate": "2019-05-24T21:36:35.773Z", "content": "\n\n\nI am still shocked by people hammering @cpmp after the last competition!
\n
Thanks. I am past this, but it was a very unpleasant surprise indeed.
", "rawMarkdown": "> I am still shocked by people hammering @cpmp after the last competition!\n\nThanks. I am past this, but it was a very unpleasant surprise indeed.", "votes": 1 }, { "id": 537706, "postDate": "2019-05-27T13:43:23.223Z", "content": "Rereading this, i'd be interested to know how the private LB estimate were done here.
", "rawMarkdown": "Rereading this, i'd be interested to know how the private LB estimate were done here.\n" }, { "id": 537720, "postDate": "2019-05-27T13:57:39.540Z", "content": "you may ask, but I won't answer until the competition ends ;)
", "rawMarkdown": "you may ask, but I won't answer until the competition ends ;)", "votes": 4 }, { "id": 538284, "postDate": "2019-05-28T11:07:32.033Z", "content": "I hope you will answer after competition end ...
", "rawMarkdown": "I hope you will answer after competition end ...", "votes": 2 }, { "id": 538302, "postDate": "2019-05-28T11:34:37.270Z", "content": "I will only answer if I finish well (say, gold medal), otherwise my estimation is crap...
", "rawMarkdown": "I will only answer if I finish well (say, gold medal), otherwise my estimation is crap...", "votes": 2 } ] }, { "id": 538381, "postDate": "2019-05-28T13:59:13.893Z", "content": "so far I have found this competition to be a good platform to build on my kaggle codebase, and also a very good testing bed for various validation settings. I found this dataset to be one that you can set up quite realistic local validation.
\n\nMy LB/CV isn't good at all, at least compared to what has been shared by the teams near the top of the LB, whatever happens ranking wise I will be quite happy of what I take away from this π
\n\nAside from this, I take quite a lot of pleasure from working on a scientific dataset.
", "rawMarkdown": "so far I have found this competition to be a good platform to build on my kaggle codebase, and also a very good testing bed for various validation settings. I found this dataset to be one that you can set up quite realistic local validation. \n\nMy LB/CV isn't good at all, at least compared to what has been shared by the teams near the top of the LB, whatever happens ranking wise I will be quite happy of what I take away from this π \n\nAside from this, I take quite a lot of pleasure from working on a scientific dataset.", "votes": 6 }, { "id": 538373, "postDate": "2019-05-28T13:46:56.950Z", "content": "whoops, 67... It's my current position ;)
", "rawMarkdown": "whoops, 67... It's my current position ;)", "votes": 3, "replies": [ { "id": 539126, "postDate": "2019-05-29T15:00:53.760Z", "content": "you're on 69th right now ;) i'll say thats better haha
", "rawMarkdown": "you're on 69th right now ;) i'll say thats better haha", "votes": 1 }, { "id": 539332, "postDate": "2019-05-29T23:29:48.553Z", "content": "It looks much better now ;)\nBut I was on 67th... this picture may come true :)
", "rawMarkdown": "It looks much better now ;)\nBut I was on 67th... this picture may come true :)" } ] }, { "id": 538418, "postDate": "2019-05-28T14:46:10.140Z", "content": "I really locking forward to the results. This is my first kaggle competition and i already learned so much about data and feature selection.
", "rawMarkdown": "I really locking forward to the results. This is my first kaggle competition and i already learned so much about data and feature selection. ", "votes": 1 }, { "id": 536761, "postDate": "2019-05-25T07:45:38.610Z", "content": "Yeah.... Shakeup time is close.... Everyone will eagerly waiting for private LB score. Let's have some interesting ideas will get from different kagglers...... π
", "rawMarkdown": "Yeah.... Shakeup time is close.... Everyone will eagerly waiting for private LB score. Let's have some interesting ideas will get from different kagglers...... π", "votes": 1 }, { "id": 538436, "postDate": "2019-05-28T15:04:38.817Z", "content": "The real shakeup is the friends we made along the way π
", "rawMarkdown": "The real shakeup is the friends we made along the way π ", "votes": 2 }, { "id": 537455, "postDate": "2019-05-27T03:45:05.233Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 537514, "postDate": "2019-05-27T06:47:34.787Z", "content": "A score that will score bronze.
", "rawMarkdown": "A score that will score bronze." } ] } ], "comments": [ { "id": 536312, "author_name": "Noah Weber", "author_url": "", "post_date": "2019-05-24T08:33:52.823000", "content": "
At least there is a private LB. In real world, how many times did I see people not validating properly their models? At best they use one train/test split, at worst they validate on training data.
\n\nAnd this is true in academia too: for how long did people use imagenet with the same train/validation split?
", "votes": 19, "replies": [] }, { "id": 536324, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-05-24T09:02:18.907000", "content": "Sadly most academic papers are trash and contain too much intimidating equations or phrases. Practical and good papers are extremely rare, which I deem only 1/10 of those pupblished in top conferences...
", "votes": 6, "replies": [] }, { "id": 536342, "author_name": "Scirpus", "author_url": "", "post_date": "2019-05-24T09:35:45.187000", "content": "@cpmp - I looked at https://public.lanl.gov/geophysics/geophysics/nonlinear/2017/Rouet-LeducGRL17.pdf again and couldn't find the word validation at all. \"RF model are highly accurate, with an R2 value of 0.89\" - this is meaningless without error std deviations related to folds. As you have stated in the past a lot Machine Learning papers are terrible! Every machine learning paper should come with a code and data repository - there is no excuse! I am amazed that ML papers can be published if the peer reviewers / readers cannot reproduce anything it is a disservice to science.
", "votes": 8, "replies": [] }, { "id": 536350, "author_name": "CPMP", "author_url": "", "post_date": "2019-05-24T09:48:49.497000", "content": "\n\n\ncouldn't find the word validation
\n
Why am I not surprised?
\n\nBut to be fair, they use a train/test split. Which is the bare minimum.
", "votes": 4, "replies": [] }, { "id": 538956, "author_name": "cyberia", "author_url": "", "post_date": "2019-05-29T10:31:43.063000", "content": "During train/test split though one may get mutually exclusive datasets but end of the day one ignores the fact that they are drawn from the same distributions.
\n\nIn science it will be a huge achievement if we could focus on distribution types other than normal/bell curve. Almost all the algorithms assume that the data follows a normal curve. This limits the application of the solution to only one of the two cases in which the population was actually a normal curve but what about the other half?
\n\nThen there is PAC very few care about it.\n
Only dropping 145 places? - wow, you must have a great model ;)
", "votes": 8, "replies": [ { "id": 536423, "author_name": "Noah Weber", "author_url": "", "post_date": "2019-05-24T12:56:57.977000", "content": "Well ofcourse its great- since its only blend of your GP-s ;)
", "votes": 6, "replies": [] }, { "id": 536433, "author_name": "Scirpus", "author_url": "", "post_date": "2019-05-24T13:09:07.460000", "content": "I believe a good score will be 1.9 with a great score being 1.8 on private - we shall see! Good luck!
", "votes": 4, "replies": [] }, { "id": 536449, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-05-24T13:55:41.683000", "content": "@scirpus, do you mean CV of 1.8 or at least 1.9?
\n\nI am asking because, I could not get my CV scores to go below 1.94+ with LB=1.555 and 1.99+ with LB = 1.441 on the single model. So I believe the shakeup is definately coming. I am waiting to be proven wrong by all these low sub 1.2/ 1.3 LB scores.
\n\nUpdate: I meant CV 1.94 and 1.99, I did not mean 0.94/0.99 so I have corrected it above.
", "votes": 1, "replies": [] }, { "id": 536452, "author_name": "Scirpus", "author_url": "", "post_date": "2019-05-24T13:59:37.767000", "content": "Private LB scores
", "votes": 1, "replies": [] }, { "id": 536467, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-05-24T14:17:34.137000", "content": "To me, private LB will be mostly above 2, with top 10 around 1.9. Exceptional 1.8 might be just a very few.
", "votes": 2, "replies": [] }, { "id": 536477, "author_name": "Scirpus", "author_url": "", "post_date": "2019-05-24T14:31:53.100000", "content": "@khahuras - I hope we will have between 1.8 & 2 otherwise both Public & Private LBs won't be representative of the training data!
", "votes": 1, "replies": [] }, { "id": 536506, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-05-24T15:15:02.343000", "content": "I even believe 2.1 should be more popular and no one can get 1.8
", "votes": 1, "replies": [] }, { "id": 536509, "author_name": "Scirpus", "author_url": "", "post_date": "2019-05-24T15:17:58.237000", "content": "I can almost guarantee my best submission will be in my first 20 submissions - I am such a Leaderboard chaser - I just can't help myself - maybe I need some therapy! ;)
", "votes": 5, "replies": [] }, { "id": 536525, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-05-24T16:04:17.610000", "content": "@scirpus, sorry I meant CV 1.94 and 1.99, I did not mean 0.94/0.99 so I have corrected it above.
", "votes": 1, "replies": [] }, { "id": 536530, "author_name": "Scirpus", "author_url": "", "post_date": "2019-05-24T16:21:11.697000", "content": "@sheriytm - those numbers are more in line with mine
", "votes": 1, "replies": [] }, { "id": 536594, "author_name": "Sergey Zlobin", "author_url": "", "post_date": "2019-05-24T18:51:09.667000", "content": "\n\n\nWell ofcourse its great- since its only blend of your GP-s ;)
\n
By the way! Scirpus, thanks for your GP kernel. I use it in the blend. :)
", "votes": 1, "replies": [] }, { "id": 536600, "author_name": "Scirpus", "author_url": "", "post_date": "2019-05-24T19:14:09.143000", "content": "@sergeyzlobin - if it crashes and burns don't blame me please as I deliberately made them overfit (just a little bit) - I am still shocked by people hammering @cpmp after the last competition!
\n\nWhat I know:\nWe have only 16 earthquakes in sequence\nThere will be a huge shake up\nThe best thing to take away from this competition is what you have learned.\nAll the best and good luck!
", "votes": 4, "replies": [] }, { "id": 536623, "author_name": "Tim Yee", "author_url": "", "post_date": "2019-05-24T20:59:52.483000", "content": "There's actually only 15 full EQs if you ignore the first and last bits
", "votes": 0, "replies": [] }, { "id": 536631, "author_name": "CPMP", "author_url": "", "post_date": "2019-05-24T21:36:35.773000", "content": "\n\n\nI am still shocked by people hammering @cpmp after the last competition!
\n
Thanks. I am past this, but it was a very unpleasant surprise indeed.
", "votes": 1, "replies": [] }, { "id": 537706, "author_name": "CPMP", "author_url": "", "post_date": "2019-05-27T13:43:23.223000", "content": "Rereading this, i'd be interested to know how the private LB estimate were done here.
", "votes": 0, "replies": [] }, { "id": 537720, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-05-27T13:57:39.540000", "content": "you may ask, but I won't answer until the competition ends ;)
", "votes": 4, "replies": [] }, { "id": 538284, "author_name": "CPMP", "author_url": "", "post_date": "2019-05-28T11:07:32.033000", "content": "I hope you will answer after competition end ...
", "votes": 2, "replies": [] }, { "id": 538302, "author_name": "Kha Vo", "author_url": "", "post_date": "2019-05-28T11:34:37.270000", "content": "I will only answer if I finish well (say, gold medal), otherwise my estimation is crap...
", "votes": 2, "replies": [] } ] }, { "id": 538381, "author_name": "Yifan Xie", "author_url": "", "post_date": "2019-05-28T13:59:13.893000", "content": "so far I have found this competition to be a good platform to build on my kaggle codebase, and also a very good testing bed for various validation settings. I found this dataset to be one that you can set up quite realistic local validation.
\n\nMy LB/CV isn't good at all, at least compared to what has been shared by the teams near the top of the LB, whatever happens ranking wise I will be quite happy of what I take away from this π
\n\nAside from this, I take quite a lot of pleasure from working on a scientific dataset.
", "votes": 6, "replies": [] }, { "id": 538373, "author_name": "MichaΕ Franczak", "author_url": "", "post_date": "2019-05-28T13:46:56.950000", "content": "whoops, 67... It's my current position ;)
", "votes": 3, "replies": [ { "id": 539126, "author_name": "Tanmay Pandey", "author_url": "", "post_date": "2019-05-29T15:00:53.760000", "content": "you're on 69th right now ;) i'll say thats better haha
", "votes": 1, "replies": [] }, { "id": 539332, "author_name": "MichaΕ Franczak", "author_url": "", "post_date": "2019-05-29T23:29:48.553000", "content": "It looks much better now ;)\nBut I was on 67th... this picture may come true :)
", "votes": 0, "replies": [] } ] }, { "id": 538418, "author_name": "SJ", "author_url": "", "post_date": "2019-05-28T14:46:10.140000", "content": "I really locking forward to the results. This is my first kaggle competition and i already learned so much about data and feature selection.
", "votes": 1, "replies": [] }, { "id": 536761, "author_name": "Himanshu Soni", "author_url": "", "post_date": "2019-05-25T07:45:38.610000", "content": "Yeah.... Shakeup time is close.... Everyone will eagerly waiting for private LB score. Let's have some interesting ideas will get from different kagglers...... π
", "votes": 1, "replies": [] }, { "id": 538436, "author_name": "CΓ©sar Parra Rojas", "author_url": "", "post_date": "2019-05-28T15:04:38.817000", "content": "The real shakeup is the friends we made along the way π
", "votes": 2, "replies": [] }, { "id": 537455, "author_name": "", "author_url": "", "post_date": "2019-05-27T03:45:05.233000", "content": "", "votes": 0, "replies": [ { "id": 537514, "author_name": "Psi", "author_url": "", "post_date": "2019-05-27T06:47:34.787000", "content": "A score that will score bronze.
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "536311": "\n\n\n", "536312": "\n\n[\nCheck other out](https://www.kaggle.com/general/76963#latest-533927)", "536385": "Only dropping 145 places? - wow, you must have a great model ;)", "538381": "so far I have found this competition to be a good platform to build on my kaggle codebase, and also a very good testing bed for various validation settings. I found this dataset to be one that you can set up quite realistic local validation. \n\nMy LB/CV isn't good at all, at least compared to what has been shared by the teams near the top of the LB, whatever happens ranking wise I will be quite happy of what I take away from this π \n\nAside from this, I take quite a lot of pleasure from working on a scientific dataset.", "538373": "whoops, 67... It's my current position ;)", "538418": "I really locking forward to the results. This is my first kaggle competition and i already learned so much about data and feature selection. ", "536761": "Yeah.... Shakeup time is close.... Everyone will eagerly waiting for private LB score. Let's have some interesting ideas will get from different kagglers...... π", "538436": "The real shakeup is the friends we made along the way π ", "537455": "" } }