![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F21287471%2F62a1ca64d234b66569783fd3324e6aa0%2Fonline%20ensemble.png?generation=1735913116467045&alt=media\")
Me: check the public code and see the model got 0.0079 has XGB and trained with no special setting other than CV and lesser epochs. So I trained with entire dataset with more epochs and click submit, but failed miserably and getting score 0.0068š
\nOthers: put models together and getting higher and higher scoreā¦.like 0.0076 to 0.0079
\nhow do people getting better scoreā¦? same strategy just try many times? this is so overfittingā¦.will it work in the end?
", "messages": [ { "id": 3089621, "postDate": "2025-01-06T10:13:22.063Z", "content": "As @shiyili said, you should have faith in your own way. I remember in last year's optiver competition, I was overwhelmed by the improved score of public kernel in the last 3 weeks dropping from around top10 to 100+ in public leaderboard. I even almost gave up that competition (stopped it for almost 2 weeks and back 2 days before its deadline). But because of my last 2 days work (adding online learning strategy), even though my public ranking is still 100+, but quickly jumped into top10 after the first private update, while people trusting the highest scoring public kernel (which used random split for train/val, can you believe that??) quickly fell behind. So trust your own strategy and never give up. Best luck!
", "rawMarkdown": "As @shiyili said, you should have faith in your own way. I remember in last year's optiver competition, I was overwhelmed by the improved score of public kernel in the last 3 weeks dropping from around top10 to 100+ in public leaderboard. I even almost gave up that competition (stopped it for almost 2 weeks and back 2 days before its deadline). But because of my last 2 days work (adding online learning strategy), even though my public ranking is still 100+, but quickly jumped into top10 after the first private update, while people trusting the highest scoring public kernel (which used random split for train/val, can you believe that??) quickly fell behind. So trust your own strategy and never give up. Best luck!", "votes": 12 }, { "id": 3086376, "postDate": "2025-01-02T09:16:07.833Z", "content": "Have faith in your own way :) Naively ensemble public models is too risky. Not saying this won't work, but it is like rolling a dice.
", "rawMarkdown": "Have faith in your own way :) Naively ensemble public models is too risky. Not saying this won't work, but it is like rolling a dice. ", "votes": 7 }, { "id": 3086407, "postDate": "2025-01-02T09:41:51.360Z", "content": "I suggest you start with CV. If you tried to validate your method of 'trainimg on all the data for more epoch' on the last 4.5M rows, you would see that it perform worse also in validation.
\nOverfitting won't work, but 'put models together' a.k.a. ensembling definitely willā¦
\nAnd there are a lot more strategies that top scores using and you would not see in the code section. Features engineering, RNN/GRU/LSTM, Transformersā¦I suggest you read the top place solution (of @hydantess) in Enefit and Optiver conpetitions from about a year ago. It's eye opening. (BTW, he participate it this competition too, lurking in the shadows, and I'm willing to bet he will finish in top 3 and probably 1st š¤£š¤£š¤£)
great suggestions, thanks
", "rawMarkdown": "great suggestions, thanks" }, { "id": 3087069, "postDate": "2025-01-03T04:06:05.907Z", "content": "yes, one of the biggest mystery is why hyd is not top 10 on the lb yet lol
", "rawMarkdown": "yes, one of the biggest mystery is why hyd is not top 10 on the lb yet lol ", "votes": 2, "replies": [ { "id": 3087175, "postDate": "2025-01-03T07:24:31.003Z", "content": "He is ē„ēåč²ing
", "rawMarkdown": "He is ē„ēåč²ing", "votes": 1 }, { "id": 3087310, "postDate": "2025-01-03T11:10:51.187Z", "content": "\n\nyes, one of the biggest mystery is why hyd is not top 10 on the lb yet lol
\n
Is not a mystery, he keeps his public score low intentionally. In LEAP he submitted only in the last day and finished in top 20.
", "rawMarkdown": ">yes, one of the biggest mystery is why hyd is not top 10 on the lb yet lol \n\nIs not a mystery, he keeps his public score low intentionally. In LEAP he submitted only in the last day and finished in top 20.", "votes": 1 }, { "id": 3087316, "postDate": "2025-01-03T11:15:23.383Z", "content": "\n\nIs not a mystery, he keeps his public score low intentionally. In LEAP he submitted only in the last day and finished in top 20.
\n
Do you follow the same strategy?
", "rawMarkdown": ">Is not a mystery, he keeps his public score low intentionally. In LEAP he submitted only in the last day and finished in top 20.\n\nDo you follow the same strategy?" }, { "id": 3087327, "postDate": "2025-01-03T11:30:41.670Z", "content": "No š¤£ in my case I joined this competition in the middle and it's my first time series competitionā¦well I still hope to finish somewhere in gold but nothing is intentional š¤£
", "rawMarkdown": "No š¤£ in my case I joined this competition in the middle and it's my first time series competition...well I still hope to finish somewhere in gold but nothing is intentional š¤£ ", "votes": 2 }, { "id": 3087336, "postDate": "2025-01-03T11:39:48.950Z", "content": "There is some secret in this competition that immediately gives people a place in the 0.0100+ zone. it can be seen from the current difference between 9th and 10th places. It seems that this secret has nothing like that with architectures, so one insight can pull in gold for now.
", "rawMarkdown": "There is some secret in this competition that immediately gives people a place in the 0.0100+ zone. it can be seen from the current difference between 9th and 10th places. It seems that this secret has nothing like that with architectures, so one insight can pull in gold for now.", "votes": 2 }, { "id": 3087360, "postDate": "2025-01-03T12:03:27.367Z", "content": "Judging by top2, there is more than one secret š¤£
", "rawMarkdown": "Judging by top2, there is more than one secret š¤£", "votes": 1 }, { "id": 3087363, "postDate": "2025-01-03T12:09:19.757Z", "content": "I share the same thoughts. It seems like people who works in the quant industry has some special tricks or prior knowledge that bring them quite some edge. Itās like the secret ingredient that only the chief knows - and he never tells š¤£
", "rawMarkdown": "I share the same thoughts. It seems like people who works in the quant industry has some special tricks or prior knowledge that bring them quite some edge. Itās like the secret ingredient that only the chief knows - and he never tells š¤£", "votes": 2 }, { "id": 3087410, "postDate": "2025-01-03T13:14:09.430Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3087411, "postDate": "2025-01-03T13:14:28.980Z", "content": "\n\n\n\nyes, one of the biggest mystery is why hyd is not top 10 on the lb yet lol
\nIs not a mystery, he keeps his public score low intentionally. In LEAP he submitted only in the last day and finished in top 20.
\n
i rmb he tweeted that he was challenging himself to just submit once or twice for LEAP š
", "rawMarkdown": "> >yes, one of the biggest mystery is why hyd is not top 10 on the lb yet lol \n> \n> Is not a mystery, he keeps his public score low intentionally. In LEAP he submitted only in the last day and finished in top 20.\n\ni rmb he tweeted that he was challenging himself to just submit once or twice for LEAP š" } ] }, { "id": 3087321, "postDate": "2025-01-03T11:18:20.030Z", "content": "I tried GRU, LSTM and Transformers. I even tried and ensemble of all of them and got nowhere near the leaderboard. Best was like 0.002 or something after weeks of hyper parameter tuning. Best NN performer I got was a very small, very shallow NN with a simple embedding layer for symbol_id and it scores like 0.0022
\nA very small LightGBM model feeding in raw data got 0.005 and still is the best submission Iāve been able to get to date
", "rawMarkdown": "I tried GRU, LSTM and Transformers. I even tried and ensemble of all of them and got nowhere near the leaderboard. Best was like 0.002 or something after weeks of hyper parameter tuning. Best NN performer I got was a very small, very shallow NN with a simple embedding layer for symbol_id and it scores like 0.0022\n\nA very small LightGBM model feeding in raw data got 0.005 and still is the best submission Iāve been able to get to date \n", "replies": [ { "id": 3087418, "postDate": "2025-01-03T13:23:27.380Z", "content": "Keep trying, you can get 0.006 at least with simple feed forward NN
", "rawMarkdown": "Keep trying, you can get 0.006 at least with simple feed forward NN" }, { "id": 3087445, "postDate": "2025-01-03T13:51:18.883Z", "content": "What kind of structure? My models get worse with size - especially number of layers. The best performing NN I have found only has 4 embedding dims and 16 hidden dims with a simple linear layer -> input norm -> output linear layer.
\nIf I add too much normalization or layers things get bad pretty quick. Daily R2 tracks ok for about half the days but it has negative R2 for many days as well
\nI tried many kinds of activation functions, batch/layer norm. LSTM, GRU etc but everything made performance worse
", "rawMarkdown": "What kind of structure? My models get worse with size - especially number of layers. The best performing NN I have found only has 4 embedding dims and 16 hidden dims with a simple linear layer -> input norm -> output linear layer.\n\nIf I add too much normalization or layers things get bad pretty quick. Daily R2 tracks ok for about half the days but it has negative R2 for many days as well\n\nI tried many kinds of activation functions, batch/layer norm. LSTM, GRU etc but everything made performance worse" }, { "id": 3087456, "postDate": "2025-01-03T14:04:31.613Z", "content": "organize your batch the same way as how the api feed data.
", "rawMarkdown": "organize your batch the same way as how the api feed data. " }, { "id": 3087460, "postDate": "2025-01-03T14:06:48.853Z", "content": "You can see a plot of 20 day moving average of daily R2 scores here for my model ensemble. Even though the entire history here is trained online (I initialise untrained models and then train every day on lags before predicting) performance in the holdout set is only 0.0036 for this ensemble.
\n
But there is still a week to go and it feels like there is probably only one or two little breakthroughs left that will unlock a score around the 0.008+ range which maybe isn't a bad effort for a first timer.
", "rawMarkdown": "But there is still a week to go and it feels like there is probably only one or two little breakthroughs left that will unlock a score around the 0.008+ range which maybe isn't a bad effort for a first timer." }, { "id": 3087506, "postDate": "2025-01-03T14:51:56.653Z", "content": "Oh wow I just saw there are a bunch of public notebooks with scores of roughly 0.008. I didn't realise people had shared these and I can see some things people do differently. Gives me some experiments to run. Two key ones seem to be
\nyeahā¦. i mean we trained with same stuffā¦ how d they getting better scores? but it look overfitting.
", "rawMarkdown": "yeah.... i mean we trained with same stuff... how d they getting better scores? but it look overfitting." } ] }, { "id": 3089606, "postDate": "2025-01-06T09:44:31.357Z", "content": "Do you have the link to the Enefit and Optiver competitions solutions by @hydantess?
", "rawMarkdown": "Do you have the link to the Enefit and Optiver competitions solutions by @hydantess?", "replies": [ { "id": 3089718, "postDate": "2025-01-06T12:47:29.330Z", "content": "https://www.kaggle.com/competitions/optiver-trading-at-the-close/discussion/487446
\nhttps://www.kaggle.com/competitions/predict-energy-behavior-of-prosumers/discussion/472793
Me: check the public code and see the model got 0.0079 has XGB and trained with no special setting other than CV and lesser epochs. So I trained with entire dataset with more epochs and click submit, but failed miserably and getting score 0.0068š
\nOthers: put models together and getting higher and higher scoreā¦.like 0.0076 to 0.0079
\nhow do people getting better scoreā¦? same strategy just try many times? this is so overfittingā¦.will it work in the end?
", "rawMarkdown": "Me: check the public code and see the model got 0.0079 has XGB and trained with no special setting other than CV and lesser epochs. So I trained with entire dataset with more epochs and click submit, but failed miserably and getting score 0.0068š\n\nOthers: put models together and getting higher and higher score....like 0.0076 to 0.0079\n\n\nhow do people getting better score...? same strategy just try many times? this is so overfitting....will it work in the end?", "votes": 1 }, { "id": 3095321, "postDate": "2025-01-13T08:07:55.380Z", "content": "I feel like the \"problem\" might be using the entire dataset. When I compared 2 models trained on partition range 6-9 and 0-5, the model with the latest partition dataset performed much better on lb. So don't feel so bad, your model can be more robust as the testing time expands.
", "rawMarkdown": "I feel like the \"problem\" might be using the entire dataset. When I compared 2 models trained on partition range 6-9 and 0-5, the model with the latest partition dataset performed much better on lb. So don't feel so bad, your model can be more robust as the testing time expands.", "replies": [ { "id": 3095701, "postDate": "2025-01-13T15:52:58.693Z", "content": "agree exactly, most likely anyone who scored > 0.008 had some sort of online learning implemented, can't wait to see the top solution after the competition ends
", "rawMarkdown": "agree exactly, most likely anyone who scored > 0.008 had some sort of online learning implemented, can't wait to see the top solution after the competition ends", "replies": [ { "id": 3095949, "postDate": "2025-01-13T23:57:59.247Z", "content": "I did not.
", "rawMarkdown": "I did not. " } ] } ] }, { "id": 3090630, "postDate": "2025-01-07T14:05:22.410Z", "content": "I feel like this time the given train_data are very prone to overfitting locally, we might have to do online fine-tuning at inferencing time
", "rawMarkdown": "I feel like this time the given train_data are very prone to overfitting locally, we might have to do online fine-tuning at inferencing time" } ], "comments": [ { "id": 3089621, "author_name": "HAO", "author_url": "", "post_date": "2025-01-06T10:13:22.063000", "content": "As @shiyili said, you should have faith in your own way. I remember in last year's optiver competition, I was overwhelmed by the improved score of public kernel in the last 3 weeks dropping from around top10 to 100+ in public leaderboard. I even almost gave up that competition (stopped it for almost 2 weeks and back 2 days before its deadline). But because of my last 2 days work (adding online learning strategy), even though my public ranking is still 100+, but quickly jumped into top10 after the first private update, while people trusting the highest scoring public kernel (which used random split for train/val, can you believe that??) quickly fell behind. So trust your own strategy and never give up. Best luck!
", "votes": 12, "replies": [] }, { "id": 3086376, "author_name": "SLi", "author_url": "", "post_date": "2025-01-02T09:16:07.833000", "content": "Have faith in your own way :) Naively ensemble public models is too risky. Not saying this won't work, but it is like rolling a dice.
", "votes": 7, "replies": [] }, { "id": 3086407, "author_name": "greySnow", "author_url": "", "post_date": "2025-01-02T09:41:51.360000", "content": "I suggest you start with CV. If you tried to validate your method of 'trainimg on all the data for more epoch' on the last 4.5M rows, you would see that it perform worse also in validation.
\nOverfitting won't work, but 'put models together' a.k.a. ensembling definitely willā¦
\nAnd there are a lot more strategies that top scores using and you would not see in the code section. Features engineering, RNN/GRU/LSTM, Transformersā¦I suggest you read the top place solution (of @hydantess) in Enefit and Optiver conpetitions from about a year ago. It's eye opening. (BTW, he participate it this competition too, lurking in the shadows, and I'm willing to bet he will finish in top 3 and probably 1st š¤£š¤£š¤£)
great suggestions, thanks
", "votes": 0, "replies": [] }, { "id": 3087069, "author_name": "yu", "author_url": "", "post_date": "2025-01-03T04:06:05.907000", "content": "yes, one of the biggest mystery is why hyd is not top 10 on the lb yet lol
", "votes": 2, "replies": [ { "id": 3087175, "author_name": "Timmy Juicehouse", "author_url": "", "post_date": "2025-01-03T07:24:31.003000", "content": "He is ē„ēåč²ing
", "votes": 1, "replies": [] }, { "id": 3087310, "author_name": "greySnow", "author_url": "", "post_date": "2025-01-03T11:10:51.187000", "content": "\n\nyes, one of the biggest mystery is why hyd is not top 10 on the lb yet lol
\n
Is not a mystery, he keeps his public score low intentionally. In LEAP he submitted only in the last day and finished in top 20.
", "votes": 1, "replies": [] }, { "id": 3087316, "author_name": "Sergei Fironov", "author_url": "", "post_date": "2025-01-03T11:15:23.383000", "content": "\n\nIs not a mystery, he keeps his public score low intentionally. In LEAP he submitted only in the last day and finished in top 20.
\n
Do you follow the same strategy?
", "votes": 0, "replies": [] }, { "id": 3087327, "author_name": "greySnow", "author_url": "", "post_date": "2025-01-03T11:30:41.670000", "content": "No š¤£ in my case I joined this competition in the middle and it's my first time series competitionā¦well I still hope to finish somewhere in gold but nothing is intentional š¤£
", "votes": 2, "replies": [] }, { "id": 3087336, "author_name": "Sergei Fironov", "author_url": "", "post_date": "2025-01-03T11:39:48.950000", "content": "There is some secret in this competition that immediately gives people a place in the 0.0100+ zone. it can be seen from the current difference between 9th and 10th places. It seems that this secret has nothing like that with architectures, so one insight can pull in gold for now.
", "votes": 2, "replies": [] }, { "id": 3087360, "author_name": "greySnow", "author_url": "", "post_date": "2025-01-03T12:03:27.367000", "content": "Judging by top2, there is more than one secret š¤£
", "votes": 1, "replies": [] }, { "id": 3087363, "author_name": "SLi", "author_url": "", "post_date": "2025-01-03T12:09:19.757000", "content": "I share the same thoughts. It seems like people who works in the quant industry has some special tricks or prior knowledge that bring them quite some edge. Itās like the secret ingredient that only the chief knows - and he never tells š¤£
", "votes": 2, "replies": [] }, { "id": 3087410, "author_name": "", "author_url": "", "post_date": "2025-01-03T13:14:09.430000", "content": "", "votes": 0, "replies": [] }, { "id": 3087411, "author_name": "yu", "author_url": "", "post_date": "2025-01-03T13:14:28.980000", "content": "\n\n\n\nyes, one of the biggest mystery is why hyd is not top 10 on the lb yet lol
\nIs not a mystery, he keeps his public score low intentionally. In LEAP he submitted only in the last day and finished in top 20.
\n
i rmb he tweeted that he was challenging himself to just submit once or twice for LEAP š
", "votes": 0, "replies": [] } ] }, { "id": 3087321, "author_name": "Michael Timbs", "author_url": "", "post_date": "2025-01-03T11:18:20.030000", "content": "I tried GRU, LSTM and Transformers. I even tried and ensemble of all of them and got nowhere near the leaderboard. Best was like 0.002 or something after weeks of hyper parameter tuning. Best NN performer I got was a very small, very shallow NN with a simple embedding layer for symbol_id and it scores like 0.0022
\nA very small LightGBM model feeding in raw data got 0.005 and still is the best submission Iāve been able to get to date
", "votes": 0, "replies": [ { "id": 3087418, "author_name": "skrrydg", "author_url": "", "post_date": "2025-01-03T13:23:27.380000", "content": "Keep trying, you can get 0.006 at least with simple feed forward NN
", "votes": 0, "replies": [] }, { "id": 3087445, "author_name": "Michael Timbs", "author_url": "", "post_date": "2025-01-03T13:51:18.883000", "content": "What kind of structure? My models get worse with size - especially number of layers. The best performing NN I have found only has 4 embedding dims and 16 hidden dims with a simple linear layer -> input norm -> output linear layer.
\nIf I add too much normalization or layers things get bad pretty quick. Daily R2 tracks ok for about half the days but it has negative R2 for many days as well
\nI tried many kinds of activation functions, batch/layer norm. LSTM, GRU etc but everything made performance worse
", "votes": 0, "replies": [] }, { "id": 3087456, "author_name": "SLi", "author_url": "", "post_date": "2025-01-03T14:04:31.613000", "content": "organize your batch the same way as how the api feed data.
", "votes": 0, "replies": [] }, { "id": 3087460, "author_name": "Michael Timbs", "author_url": "", "post_date": "2025-01-03T14:06:48.853000", "content": "You can see a plot of 20 day moving average of daily R2 scores here for my model ensemble. Even though the entire history here is trained online (I initialise untrained models and then train every day on lags before predicting) performance in the holdout set is only 0.0036 for this ensemble.
\n
But there is still a week to go and it feels like there is probably only one or two little breakthroughs left that will unlock a score around the 0.008+ range which maybe isn't a bad effort for a first timer.
", "votes": 0, "replies": [] }, { "id": 3087506, "author_name": "Michael Timbs", "author_url": "", "post_date": "2025-01-03T14:51:56.653000", "content": "Oh wow I just saw there are a bunch of public notebooks with scores of roughly 0.008. I didn't realise people had shared these and I can see some things people do differently. Gives me some experiments to run. Two key ones seem to be
\nyeahā¦. i mean we trained with same stuffā¦ how d they getting better scores? but it look overfitting.
", "votes": 0, "replies": [] } ] }, { "id": 3089606, "author_name": "aleg", "author_url": "", "post_date": "2025-01-06T09:44:31.357000", "content": "Do you have the link to the Enefit and Optiver competitions solutions by @hydantess?
", "votes": 0, "replies": [ { "id": 3089718, "author_name": "hyd", "author_url": "", "post_date": "2025-01-06T12:47:29.330000", "content": "https://www.kaggle.com/competitions/optiver-trading-at-the-close/discussion/487446
\nhttps://www.kaggle.com/competitions/predict-energy-behavior-of-prosumers/discussion/472793
I feel like the \"problem\" might be using the entire dataset. When I compared 2 models trained on partition range 6-9 and 0-5, the model with the latest partition dataset performed much better on lb. So don't feel so bad, your model can be more robust as the testing time expands.
", "votes": 0, "replies": [ { "id": 3095701, "author_name": "Creative-Ataraxia", "author_url": "", "post_date": "2025-01-13T15:52:58.693000", "content": "agree exactly, most likely anyone who scored > 0.008 had some sort of online learning implemented, can't wait to see the top solution after the competition ends
", "votes": 0, "replies": [ { "id": 3095949, "author_name": "LordofSauce", "author_url": "", "post_date": "2025-01-13T23:57:59.247000", "content": "I did not.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3090630, "author_name": "Creative-Ataraxia", "author_url": "", "post_date": "2025-01-07T14:05:22.410000", "content": "I feel like this time the given train_data are very prone to overfitting locally, we might have to do online fine-tuning at inferencing time
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3089621": "As @shiyili said, you should have faith in your own way. I remember in last year's optiver competition, I was overwhelmed by the improved score of public kernel in the last 3 weeks dropping from around top10 to 100+ in public leaderboard. I even almost gave up that competition (stopped it for almost 2 weeks and back 2 days before its deadline). But because of my last 2 days work (adding online learning strategy), even though my public ranking is still 100+, but quickly jumped into top10 after the first private update, while people trusting the highest scoring public kernel (which used random split for train/val, can you believe that??) quickly fell behind. So trust your own strategy and never give up. Best luck!", "3086376": "Have faith in your own way :) Naively ensemble public models is too risky. Not saying this won't work, but it is like rolling a dice. ", "3086407": "I suggest you start with CV. If you tried to validate your method of 'trainimg on all the data for more epoch' on the last 4.5M rows, you would see that it perform worse also in validation. \nOverfitting won't work, but 'put models together' a.k.a. ensembling definitely will...\nAnd there are a lot more strategies that top scores using and you would not see in the code section. Features engineering, RNN/GRU/LSTM, Transformers...I suggest you read the top place solution (of @hydantess) in Enefit and Optiver conpetitions from about a year ago. It's eye opening. (BTW, he participate it this competition too, lurking in the shadows, and I'm willing to bet he will finish in top 3 and probably 1st š¤£š¤£š¤£)", "3086368": "Me: check the public code and see the model got 0.0079 has XGB and trained with no special setting other than CV and lesser epochs. So I trained with entire dataset with more epochs and click submit, but failed miserably and getting score 0.0068š\n\nOthers: put models together and getting higher and higher score....like 0.0076 to 0.0079\n\n\nhow do people getting better score...? same strategy just try many times? this is so overfitting....will it work in the end?", "3095321": "I feel like the \"problem\" might be using the entire dataset. When I compared 2 models trained on partition range 6-9 and 0-5, the model with the latest partition dataset performed much better on lb. So don't feel so bad, your model can be more robust as the testing time expands.", "3090630": "I feel like this time the given train_data are very prone to overfitting locally, we might have to do online fine-tuning at inferencing time" } }