Since the updated data includes May, if we want to use it for training, CV is our only choice. However, when I use July as validation, I found my CV mae score abnormally low ~0.83. Still cannot find out why after a while of debugging. If anyone is suffering from the same issue, please feel free to share your thoughts.
", "messages": [ { "id": 1396986, "postDate": "2021-07-22T16:49:59.823Z", "content": "Since the updated data includes May, if we want to use it for training, CV is our only choice. However, when I use July as validation, I found my CV mae score abnormally low ~0.83. Still cannot find out why after a while of debugging. If anyone is suffering from the same issue, please feel free to share your thoughts.
", "rawMarkdown": "Since the updated data includes May, if we want to use it for training, CV is our only choice. However, when I use July as validation, I found my CV mae score abnormally low ~0.83. Still cannot find out why after a while of debugging. If anyone is suffering from the same issue, please feel free to share your thoughts." }, { "id": 1397687, "postDate": "2021-07-23T12:21:39.907Z", "content": "Does your training set include May? If it includes, your LB is meaningless.
", "rawMarkdown": "Does your training set include May? If it includes, your LB is meaningless. ", "replies": [ { "id": 1397898, "postDate": "2021-07-23T15:41:24.430Z", "content": "Hey @lhagiimn Thanks for the heads up! We are aware that if we use the updated dataset, there will be a \"leak\" in our LB score and hence why we are making sure that our CV is not messed up. Because CV scores are relatively better compared to the old data CV, we are just making sure that we did not introduce a leak in our CV (which will make gauging our model a little difficult :-:). Since we are on this topic, would you be willing to share how your CV scores have generally changed when shifting to the new data? Thx
", "rawMarkdown": "Hey @lhagiimn Thanks for the heads up! We are aware that if we use the updated dataset, there will be a \"leak\" in our LB score and hence why we are making sure that our CV is not messed up. Because CV scores are relatively better compared to the old data CV, we are just making sure that we did not introduce a leak in our CV (which will make gauging our model a little difficult :-:). Since we are on this topic, would you be willing to share how your CV scores have generally changed when shifting to the new data? Thx" }, { "id": 1398146, "postDate": "2021-07-23T19:09:39.853Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 1397298, "postDate": "2021-07-23T03:46:08.973Z", "content": "I'm also seeing pretty low MAE when only using July, but the MAE goes up when expanding the validation set to include June
", "rawMarkdown": "I'm also seeing pretty low MAE when only using July, but the MAE goes up when expanding the validation set to include June", "replies": [ { "id": 1397306, "postDate": "2021-07-23T04:14:54.867Z", "rawMarkdown": "", "isDeleted": true } ] }, { "id": 1397108, "postDate": "2021-07-22T19:20:03.343Z", "content": "0.83 for 17 days of July does not seem to be an abnormal difference (for time series) from April's 0.95. Interesting to compare what were mae scores for May and June taken as validatiion months.
\nAnother factor might be \"more training data - better is the predictive power of the model - better is the score\"
Hi @isakev , thanks for the suggestion. But I'm not exactly sure why mae will decrease with less test data. Isn't it averaged over the number of data points?
", "rawMarkdown": "Hi @isakev , thanks for the suggestion. But I'm not exactly sure why mae will decrease with less test data. Isn't it averaged over the number of data points?" }, { "id": 1397291, "postDate": "2021-07-23T03:30:07.740Z", "content": "Hi, @zacchaeus, it is averaged, but usually smaller is the validation set, higher is the variability of the validation score. That's why it would be interesting to see some other months validation scores. I did not see any except for April in discussions or notebooks. And I have not done yet any training myself.
", "rawMarkdown": "Hi, @zacchaeus, it is averaged, but usually smaller is the validation set, higher is the variability of the validation score. That's why it would be interesting to see some other months validation scores. I did not see any except for April in discussions or notebooks. And I have not done yet any training myself. " } ] } ], "comments": [ { "id": 1397687, "author_name": "lhagiimn", "author_url": "", "post_date": "2021-07-23T12:21:39.907000", "content": "Does your training set include May? If it includes, your LB is meaningless.
", "votes": 0, "replies": [ { "id": 1397898, "author_name": "maze508", "author_url": "", "post_date": "2021-07-23T15:41:24.430000", "content": "Hey @lhagiimn Thanks for the heads up! We are aware that if we use the updated dataset, there will be a \"leak\" in our LB score and hence why we are making sure that our CV is not messed up. Because CV scores are relatively better compared to the old data CV, we are just making sure that we did not introduce a leak in our CV (which will make gauging our model a little difficult :-:). Since we are on this topic, would you be willing to share how your CV scores have generally changed when shifting to the new data? Thx
", "votes": 0, "replies": [] }, { "id": 1398146, "author_name": "", "author_url": "", "post_date": "2021-07-23T19:09:39.853000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1397298, "author_name": "Lu Bin Liu", "author_url": "", "post_date": "2021-07-23T03:46:08.973000", "content": "I'm also seeing pretty low MAE when only using July, but the MAE goes up when expanding the validation set to include June
", "votes": 0, "replies": [ { "id": 1397306, "author_name": "", "author_url": "", "post_date": "2021-07-23T04:14:54.867000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1397108, "author_name": "Issagali Konysbayev [dsmlkz]", "author_url": "", "post_date": "2021-07-22T19:20:03.343000", "content": "0.83 for 17 days of July does not seem to be an abnormal difference (for time series) from April's 0.95. Interesting to compare what were mae scores for May and June taken as validatiion months.
\nAnother factor might be \"more training data - better is the predictive power of the model - better is the score\"
Hi @isakev , thanks for the suggestion. But I'm not exactly sure why mae will decrease with less test data. Isn't it averaged over the number of data points?
", "votes": 0, "replies": [] }, { "id": 1397291, "author_name": "Issagali Konysbayev [dsmlkz]", "author_url": "", "post_date": "2021-07-23T03:30:07.740000", "content": "Hi, @zacchaeus, it is averaged, but usually smaller is the validation set, higher is the variability of the validation score. That's why it would be interesting to see some other months validation scores. I did not see any except for April in discussions or notebooks. And I have not done yet any training myself.
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "1396986": "Since the updated data includes May, if we want to use it for training, CV is our only choice. However, when I use July as validation, I found my CV mae score abnormally low ~0.83. Still cannot find out why after a while of debugging. If anyone is suffering from the same issue, please feel free to share your thoughts.", "1397687": "Does your training set include May? If it includes, your LB is meaningless. ", "1397298": "I'm also seeing pretty low MAE when only using July, but the MAE goes up when expanding the validation set to include June", "1397108": "0.83 for 17 days of July does not seem to be an abnormal difference (for time series) from April's 0.95. Interesting to compare what were mae scores for May and June taken as validatiion months.\nAnother factor might be \"more training data - better is the predictive power of the model - better is the score\"" } }