![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F175917%2Fc87d62311c86e6746c1004f85282e4fe%2Fdownload%20(4).png?generation=1729947254771306&alt=media\")
Hello fellow participants,
\nPlease find this thread for CV and leaderboard relations for starter models. I shall commence with my models insofar from the below public work-
\nAll of these are single LGBM models with simple train-test split CV scheme
\n| Version | \nDate | \nCV score | \nLB score | \n
|---|---|---|---|
| LGBMV1_1 | \n15Oct2024 | \n0.007555 | \n0.0043 | \n
| LGBMV1_2 | \n15Oct2024 | \n0.007569 | \n0.0044 | \n
| LGBMV1_4 | \n15Oct2024 | \n0.007882 | \n0.0043 | \n
| LGBMV1_5 | \n15Oct2024 | \n0.007667 | \n0.0041 | \n
| CBV1_2 | \n16Oct2024 | \n0.007734 | \n0.0039 | \n
Thoughts? Comments?
Hello fellow participants,
\nPlease find this thread for CV and leaderboard relations for starter models. I shall commence with my models insofar from the below public work-
\nAll of these are single LGBM models with simple train-test split CV scheme
\n| Version | \nDate | \nCV score | \nLB score | \n
|---|---|---|---|
| LGBMV1_1 | \n15Oct2024 | \n0.007555 | \n0.0043 | \n
| LGBMV1_2 | \n15Oct2024 | \n0.007569 | \n0.0044 | \n
| LGBMV1_4 | \n15Oct2024 | \n0.007882 | \n0.0043 | \n
| LGBMV1_5 | \n15Oct2024 | \n0.007667 | \n0.0041 | \n
| CBV1_2 | \n16Oct2024 | \n0.007734 | \n0.0039 | \n
Thoughts? Comments?
A couple of stupidly simple baseline CVs (based on the last 100 days in the training data):
\nThis last one surprised me enough to plot out the metric at different bucket sizes (time_id has factors [1, 2, 4, 8, 11, 22, 44, 88, 121, 242, 484, 968])
\n
An untuned catboost, has a CV of ~0.0075 (at least it's positive).
", "rawMarkdown": "A couple of stupidly simple baseline CVs (based on the last 100 days in the training data):\n\n1. guess 0 for everything, R^2 = 0\n2. use the previous day's responder_6 mean, R^2 = -0.02\n3. use the previous day's responder_6 observation (at each time_id). R^2 = -0.99\n\nThis last one surprised me enough to plot out the metric at different bucket sizes *(time_id has factors [1, 2, 4, 8, 11, 22, 44, 88, 121, 242, 484, 968])*\n\n.png?generation=1729947254771306&alt=media)\n\n- Previous values are anti-predictive. \n- Simple lag-based features based on the target will probably not be useful\n- if in doubt, guess 0 🤣\n\nAn untuned catboost, has a CV of ~0.0075 (at least it's positive).", "votes": 6, "replies": [ { "id": 3028791, "postDate": "2024-10-26T14:09:33.887Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3030397, "postDate": "2024-10-28T13:42:02.447Z", "content": "I have also tried it, and I thought autoregression would be great, but the result was unexpected.
", "rawMarkdown": "I have also tried it, and I thought autoregression would be great, but the result was unexpected.", "votes": 1 } ] }, { "id": 3049088, "postDate": "2024-11-18T17:30:33.517Z", "content": "I'm using a single validation split by time right now. Date ids after 1515 (inclusive) are selected as validation so it makes exactly 183 date ids.
\n| validation | \nleaderboard | \n
|---|---|
| 0.009278876873841768 | \n0.0043 | \n
| 0.009420489539111121 | \n0.0040 | \n
| 0.009872482483289868 | \n0.0045 | \n
| 0.009936407757916044 | \n0.0044 | \n
| 0.010493038314628556 | \n0.0045 | \n
| 0.01055438183200752 | \n0.0046 | \n
| 0.01062276941316631 | \n0.0046 | \n
| 0.0107125793311309 | \n0.0047 | \n
| 0.01079937669003117 | \n0.0047 | \n
| 0.010810543767857728 | \n0.0049 | \n
| 0.010920114827567384 | \n0.0048 | \n
| 0.010962342625704502 | \n0.0049 | \n
| 0.011066736623069118 | \n0.0048 | \n
I can say that my validation scores are correlated with leaderboard so far.
\nCorrelation Pearson: 0.908436, Spearman: 0.946081
Thanks for sharing! Are the submitted models retrained with all date_ids using the best iterations found from validation data or the last 183 date_ids are not used (only as validation)
", "rawMarkdown": "Thanks for sharing! Are the submitted models retrained with all date_ids using the best iterations found from validation data or the last 183 date_ids are not used (only as validation)", "replies": [ { "id": 3049117, "postDate": "2024-11-18T18:09:15.467Z", "content": "Yeah, I retrain on 47M samples after validating, but I don't find best iterations through early stopping. I'm using fixed number of iterations.
", "rawMarkdown": "Yeah, I retrain on 47M samples after validating, but I don't find best iterations through early stopping. I'm using fixed number of iterations." } ] } ] }, { "id": 3030421, "postDate": "2024-10-28T14:14:22.277Z", "content": "| model | \ncv | \nlb | \n
|---|---|---|
| lgb | \n0.007869 | \n0.0048 | \n
| lgb | \n0.006137 | \n0.0031 | \n
| lgb | \n0.005106 | \n0.0032 | \n
| lgb | \n0.004907 | \n0.0033 | \n
| lgb | \n0.005246 | \n0.0036 | \n
| lgb+xgb | \n0.007840 | \n0.0052 | \n
| ridge | \n0.003261 | \n0.0023 | \n
Thanks for sharing @shiyili . Is it you hyper-parameter tunning with lgb for different lb?
", "rawMarkdown": "Thanks for sharing @shiyili . Is it you hyper-parameter tunning with lgb for different lb?", "votes": 1, "replies": [ { "id": 3030456, "postDate": "2024-10-28T14:37:26.473Z", "content": "There are different tricks (inlucding hp tuning) for each model. But the validation set are more-or-less the same.
", "rawMarkdown": "There are different tricks (inlucding hp tuning) for each model. But the validation set are more-or-less the same. ", "votes": 2 } ] }, { "id": 3030461, "postDate": "2024-10-28T14:39:43.687Z", "content": "Mind sharing your cv scheme? I only tried using > 1577 time id as validation set but don’t seem correlated well
", "rawMarkdown": "Mind sharing your cv scheme? I only tried using > 1577 time id as validation set but don’t seem correlated well", "votes": 1, "replies": [ { "id": 3030464, "postDate": "2024-10-28T14:43:10.363Z", "content": "I used partition 6-10, and kept the last 100 dates as validation. 5-folds are splited based on date_id (e.g. i % n_fold == 0 )
Are you early stopping on the last 100 date_id? Or just using it as a holdout?
", "rawMarkdown": "Are you early stopping on the last 100 date_id? Or just using it as a holdout?", "replies": [ { "id": 3037576, "postDate": "2024-11-05T20:45:10.923Z", "content": "holdout the last 100 date_id as OOF
", "rawMarkdown": "holdout the last 100 date_id as OOF " }, { "id": 3037632, "postDate": "2024-11-05T22:28:06.943Z", "content": "Could you please check which date_id's your xgb+lgb (lb 0.0052) did better on relative to your solo lgb (lb 0.0048)? I'd love to know.
", "rawMarkdown": "Could you please check which date_id's your xgb+lgb (lb 0.0052) did better on relative to your solo lgb (lb 0.0048)? I'd love to know. " }, { "id": 3037836, "postDate": "2024-11-06T08:20:27.993Z", "content": "Do you mean check it in the validation set? Why does it matter?
", "rawMarkdown": "Do you mean check it in the validation set? Why does it matter?" }, { "id": 3037954, "postDate": "2024-11-06T12:06:33.500Z", "content": "Hmm also using it as a holdout and 5 fold within train set only, holdout score is 0.0085+ but LB stuck at half that..
", "rawMarkdown": "Hmm also using it as a holdout and 5 fold within train set only, holdout score is 0.0085+ but LB stuck at half that..", "votes": 1 }, { "id": 3038073, "postDate": "2024-11-06T14:59:02.403Z", "content": "Yeah, it performed worse on validation but better on leaderboard - I was wondering what dates the predictions really differed on from the solo lgb model. I'm willing to sacrifice submissions to experiment with validation sets that better correlate with leaderboard.
", "rawMarkdown": "Yeah, it performed worse on validation but better on leaderboard - I was wondering what dates the predictions really differed on from the solo lgb model. I'm willing to sacrifice submissions to experiment with validation sets that better correlate with leaderboard. " } ] } ] }, { "id": 3019235, "postDate": "2024-10-16T11:25:28.623Z", "content": "Continuous updates
\n| date | \nmodel | \nCV_method | \nCV_score | \nLB_score | \n
|---|---|---|---|---|
| 10/16 | \nRidge | \ndata 9train_test_split(8:2) | \n0.0036 | \n0.0026 | \n
| 10/23 | \nRidge | \ndata 7,8,9 train_test_split(8:2) | \n0.00489 | \n0.0033 | \n
Hi @ravi20076 Great work and thanks for sharing your results.
\nI noticed that you used symbol_id as feature for model input. I was wondering if it will make a difference when new symbol id is added in the test?
", "rawMarkdown": "Hi @ravi20076 Great work and thanks for sharing your results.\n\nI noticed that you used symbol_id as feature for model input. I was wondering if it will make a difference when new symbol id is added in the test?", "votes": 1 }, { "id": 3022196, "postDate": "2024-10-19T11:38:33.493Z", "content": "| Model | \nTest R² | \nLeaderboard (LB) Score | \n
|---|---|---|
| Single LGBM | \n0.0433 | \n0.0033 | \n
can you share your training notebook? im curious what if anything was wrong with your setup
", "rawMarkdown": "can you share your training notebook? im curious what if anything was wrong with your setup", "votes": 1, "replies": [ { "id": 3037535, "postDate": "2024-11-05T19:42:29.583Z", "content": "This is my Training Code , @dc260123.
\n%%time\n\ndef weighted_r2_score(y_true, y_pred, sample_weight):\n numerator = np.sum(sample_weight * (y_true - y_pred) ** 2)\n denominator = np.sum(sample_weight * (y_true) ** 2)\n r2 = 1 - (numerator / denominator)\n return r2\n\ndef train_and_evaluate_kfold(model, X, y, sample_weights, =5, =42, =):\n\n kf = KFold(=n_splits, =, =SEED)\n train_r2_scores = []\n test_r2_scores = []\n models = [] \n\n fold, (train_index, test_index) enumerate(tqdm(kf.split(X), =n_splits, =)):\n X_train, X_test = X.iloc[train_index], X.iloc[test_index]\n y_train, y_test = y.iloc[train_index], y.iloc[test_index]\n weights_train, weights_test = sample_weights[train_index], sample_weights[test_index]\n\n model_clone = model.__class__(**model.get_params())\n\n model_clone.fit(X_train, y_train)\n\n models.append(model_clone)\n\n y_train_pred = model_clone.predict(X_train)\n y_test_pred = model_clone.predict(X_test)\n\n train_r2 = weighted_r2_score(y_train, y_train_pred, weights_train)\n test_r2 = weighted_r2_score(y_test, y_test_pred, weights_test)\n\n train_r2_scores.append(train_r2)\n test_r2_scores.append(test_r2)\n\n (f)\n\n time.sleep(1)\n\n clear_output(=)\n\n with open(pickle_file, ) as f:\n pickle.dump(models, f)\n\n (f)\n (f)\n\n return models\n\nX = train[fe] \ny = train[] \nsample_weights = train[].values \n\nmodel = LGBMRegressor(=-1, =42, =)\n\nLightModels = train_and_evaluate_kfold(model, X, y, sample_weights)\nI dont think they are taking the mean of each folds losses. Can you try to save the prediction and targets and calculate the 'overall' score for each fold?
", "rawMarkdown": "I dont think they are taking the mean of each folds losses. Can you try to save the prediction and targets and calculate the 'overall' score for each fold?" }, { "id": 3042267, "postDate": "2024-11-11T11:38:22.923Z", "content": "@dc260123 may i ask whats your cv - lb and validation scheme looks like?
", "rawMarkdown": "@dc260123 may i ask whats your cv - lb and validation scheme looks like? " }, { "id": 3042465, "postDate": "2024-11-11T14:35:09.163Z", "content": "CV 0.0082
\nlb 0.006
but my best score is a blend of two models and with online learning so take it with a grain of salt
\nLooking at other people's numbers it seems like a .002~0.003 spread seems to be reasonable
How'd you implement online learning? Do you know of any good example notebooks? I've failed at every attempt of it 😞
", "rawMarkdown": "How'd you implement online learning? Do you know of any good example notebooks? I've failed at every attempt of it 😞" }, { "id": 3042510, "postDate": "2024-11-11T15:09:19.043Z", "content": "I dont think they are changing the 1 minute restriction and the callback api makes it difficult too.
\nMy code is in a hobby project state with a lot of comments and todos so I can't share the whole thing without investing a lot of time making a fork and maintaining the public fork: you will most likely run into other problems anyway…
\nIf you haven't done it already I would suggest to do what the others have suggested too to set up some sort of test calling your predict callback and debug the callback directly instead of through the grpc server or modify their server locally to forward server side error properly back to client.(start with removing the wrapper exception GatewayRuntimeError)
\nAnd no I just brute-forced through it without reference any notebook… Most of my submission are for troubleshooting this. At one point I had to rely on trying to understand if I made some progress if I see \"Submission Scoring Error\" or \"Notebook Threw Exception\" : )
\nI understand that to some extend they want to prevent private test set leakage but forwarding an exception type back to us with a limit on the exception's length should be safe and helpful for troubleshooting.
", "rawMarkdown": "I dont think they are changing the 1 minute restriction and the callback api makes it difficult too. \n\nMy code is in a hobby project state with a lot of comments and todos so I can't share the whole thing without investing a lot of time making a fork and maintaining the public fork: you will most likely run into other problems anyway...\n\nIf you haven't done it already I would suggest to do what the others have suggested too to set up some sort of test calling your predict callback and debug the callback directly instead of through the grpc server or modify their server locally to forward server side error properly back to client.(start with removing the wrapper exception GatewayRuntimeError)\n\nAnd no I just brute-forced through it without reference any notebook... Most of my submission are for troubleshooting this. At one point I had to rely on trying to understand if I made some progress if I see \"Submission Scoring Error\" or \"Notebook Threw Exception\" : )\n\nI understand that to some extend they want to prevent private test set leakage but forwarding an exception type back to us with a limit on the exception's length should be safe and helpful for troubleshooting.", "votes": 1 }, { "id": 3042523, "postDate": "2024-11-11T15:13:09.453Z", "content": "I don't get errors. When I say I've failed, I mean my score drops significantly. I'm attempting to incrementally train an xgboost model by specifying the model parameter in the fit method.
", "rawMarkdown": "I don't get errors. When I say I've failed, I mean my score drops significantly. I'm attempting to incrementally train an xgboost model by specifying the model parameter in the fit method." }, { "id": 3042541, "postDate": "2024-11-11T15:26:50.433Z", "content": "try run it with online learning on your validation set and print a result each day comparing it with without online learning?
\nI only have a theoretical understanding of gradient boosting and am fairly new to kaggle + practical use of boosting libraries but I dont think GB models are good candidates for online learning because they are not trained with SGD.
\nThe good empirical results we see are generally from training them with the provided algorithm(some sort of full batch gradient descent with GOSS/bagging). If I have to guess I think you can either try to fit the model again with the new data only or a mix of old and new data.
\nWith only the new data the model does not have enough sample to be refitted properly and if you mix new data with some old data to stabilize the gradient's direction you might overfit the model
", "rawMarkdown": "try run it with online learning on your validation set and print a result each day comparing it with without online learning? \n\nI only have a theoretical understanding of gradient boosting and am fairly new to kaggle + practical use of boosting libraries but I dont think GB models are good candidates for online learning because they are not trained with SGD. \n\nThe good empirical results we see are generally from training them with the provided algorithm(some sort of full batch gradient descent with GOSS/bagging). If I have to guess I think you can either try to fit the model again with the new data only or a mix of old and new data. \n\nWith only the new data the model does not have enough sample to be refitted properly and if you mix new data with some old data to stabilize the gradient's direction you might overfit the model " }, { "id": 3042599, "postDate": "2024-11-11T16:03:20.743Z", "content": "https://www.kaggle.com/competitions/optiver-trading-at-the-close/discussion/487446
\nhttps://www.kaggle.com/competitions/optiver-trading-at-the-close/discussion/486169
\nBoth notebooks mention online learning/training for GB models.
I'll try adjusting the kaggle submission code locally to test OL methods and print with/without OL. Still working on feature engineering anyways. If you don't mind, what dates are you using for local validation? Is your reported CV with your ensemble and OL? Or ensemble without OL?
\n\"CV 0.0082
\nlb 0.006\"
validation period : date_id more over 1577
\ncv : 0.006210 , lb : 0.0039
Hello, do you have any new ideas about the updated LB scores and validation strategy?
", "rawMarkdown": "Hello, do you have any new ideas about the updated LB scores and validation strategy?" }, { "id": 3037678, "postDate": "2024-11-06T01:58:04.957Z", "rawMarkdown": "", "votes": 2, "isDeleted": true }, { "id": 3037677, "postDate": "2024-11-06T01:57:16.337Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3037162, "postDate": "2024-11-05T12:40:55.367Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 3028716, "author_name": "paddykb", "author_url": "", "post_date": "2024-10-26T13:01:21.280000", "content": "A couple of stupidly simple baseline CVs (based on the last 100 days in the training data):
\nThis last one surprised me enough to plot out the metric at different bucket sizes (time_id has factors [1, 2, 4, 8, 11, 22, 44, 88, 121, 242, 484, 968])
\n
An untuned catboost, has a CV of ~0.0075 (at least it's positive).
", "votes": 6, "replies": [ { "id": 3028791, "author_name": "", "author_url": "", "post_date": "2024-10-26T14:09:33.887000", "content": "", "votes": 0, "replies": [] }, { "id": 3030397, "author_name": "yunsuxiaozi", "author_url": "", "post_date": "2024-10-28T13:42:02.447000", "content": "I have also tried it, and I thought autoregression would be great, but the result was unexpected.
", "votes": 1, "replies": [] } ] }, { "id": 3049088, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2024-11-18T17:30:33.517000", "content": "I'm using a single validation split by time right now. Date ids after 1515 (inclusive) are selected as validation so it makes exactly 183 date ids.
\n| validation | \nleaderboard | \n
|---|---|
| 0.009278876873841768 | \n0.0043 | \n
| 0.009420489539111121 | \n0.0040 | \n
| 0.009872482483289868 | \n0.0045 | \n
| 0.009936407757916044 | \n0.0044 | \n
| 0.010493038314628556 | \n0.0045 | \n
| 0.01055438183200752 | \n0.0046 | \n
| 0.01062276941316631 | \n0.0046 | \n
| 0.0107125793311309 | \n0.0047 | \n
| 0.01079937669003117 | \n0.0047 | \n
| 0.010810543767857728 | \n0.0049 | \n
| 0.010920114827567384 | \n0.0048 | \n
| 0.010962342625704502 | \n0.0049 | \n
| 0.011066736623069118 | \n0.0048 | \n
I can say that my validation scores are correlated with leaderboard so far.
\nCorrelation Pearson: 0.908436, Spearman: 0.946081
Thanks for sharing! Are the submitted models retrained with all date_ids using the best iterations found from validation data or the last 183 date_ids are not used (only as validation)
", "votes": 0, "replies": [ { "id": 3049117, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2024-11-18T18:09:15.467000", "content": "Yeah, I retrain on 47M samples after validating, but I don't find best iterations through early stopping. I'm using fixed number of iterations.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3030421, "author_name": "SLi", "author_url": "", "post_date": "2024-10-28T14:14:22.277000", "content": "| model | \ncv | \nlb | \n
|---|---|---|
| lgb | \n0.007869 | \n0.0048 | \n
| lgb | \n0.006137 | \n0.0031 | \n
| lgb | \n0.005106 | \n0.0032 | \n
| lgb | \n0.004907 | \n0.0033 | \n
| lgb | \n0.005246 | \n0.0036 | \n
| lgb+xgb | \n0.007840 | \n0.0052 | \n
| ridge | \n0.003261 | \n0.0023 | \n
Thanks for sharing @shiyili . Is it you hyper-parameter tunning with lgb for different lb?
", "votes": 1, "replies": [ { "id": 3030456, "author_name": "SLi", "author_url": "", "post_date": "2024-10-28T14:37:26.473000", "content": "There are different tricks (inlucding hp tuning) for each model. But the validation set are more-or-less the same.
", "votes": 2, "replies": [] } ] }, { "id": 3030461, "author_name": "yu", "author_url": "", "post_date": "2024-10-28T14:39:43.687000", "content": "Mind sharing your cv scheme? I only tried using > 1577 time id as validation set but don’t seem correlated well
", "votes": 1, "replies": [ { "id": 3030464, "author_name": "SLi", "author_url": "", "post_date": "2024-10-28T14:43:10.363000", "content": "I used partition 6-10, and kept the last 100 dates as validation. 5-folds are splited based on date_id (e.g. i % n_fold == 0 )
Are you early stopping on the last 100 date_id? Or just using it as a holdout?
", "votes": 0, "replies": [ { "id": 3037576, "author_name": "SLi", "author_url": "", "post_date": "2024-11-05T20:45:10.923000", "content": "holdout the last 100 date_id as OOF
", "votes": 0, "replies": [] }, { "id": 3037632, "author_name": "Jack", "author_url": "", "post_date": "2024-11-05T22:28:06.943000", "content": "Could you please check which date_id's your xgb+lgb (lb 0.0052) did better on relative to your solo lgb (lb 0.0048)? I'd love to know.
", "votes": 0, "replies": [] }, { "id": 3037836, "author_name": "SLi", "author_url": "", "post_date": "2024-11-06T08:20:27.993000", "content": "Do you mean check it in the validation set? Why does it matter?
", "votes": 0, "replies": [] }, { "id": 3037954, "author_name": "JM", "author_url": "", "post_date": "2024-11-06T12:06:33.500000", "content": "Hmm also using it as a holdout and 5 fold within train set only, holdout score is 0.0085+ but LB stuck at half that..
", "votes": 1, "replies": [] }, { "id": 3038073, "author_name": "Jack", "author_url": "", "post_date": "2024-11-06T14:59:02.403000", "content": "Yeah, it performed worse on validation but better on leaderboard - I was wondering what dates the predictions really differed on from the solo lgb model. I'm willing to sacrifice submissions to experiment with validation sets that better correlate with leaderboard.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3019235, "author_name": "yunsuxiaozi", "author_url": "", "post_date": "2024-10-16T11:25:28.623000", "content": "Continuous updates
\n| date | \nmodel | \nCV_method | \nCV_score | \nLB_score | \n
|---|---|---|---|---|
| 10/16 | \nRidge | \ndata 9train_test_split(8:2) | \n0.0036 | \n0.0026 | \n
| 10/23 | \nRidge | \ndata 7,8,9 train_test_split(8:2) | \n0.00489 | \n0.0033 | \n
Hi @ravi20076 Great work and thanks for sharing your results.
\nI noticed that you used symbol_id as feature for model input. I was wondering if it will make a difference when new symbol id is added in the test?
", "votes": 1, "replies": [] }, { "id": 3022196, "author_name": "Sheikh Muhammad Abdullah", "author_url": "", "post_date": "2024-10-19T11:38:33.493000", "content": "| Model | \nTest R² | \nLeaderboard (LB) Score | \n
|---|---|---|
| Single LGBM | \n0.0433 | \n0.0033 | \n
can you share your training notebook? im curious what if anything was wrong with your setup
", "votes": 1, "replies": [ { "id": 3037535, "author_name": "Sheikh Muhammad Abdullah", "author_url": "", "post_date": "2024-11-05T19:42:29.583000", "content": "This is my Training Code , @dc260123.
\n%%time\n\ndef weighted_r2_score(y_true, y_pred, sample_weight):\n numerator = np.sum(sample_weight * (y_true - y_pred) ** 2)\n denominator = np.sum(sample_weight * (y_true) ** 2)\n r2 = 1 - (numerator / denominator)\n return r2\n\ndef train_and_evaluate_kfold(model, X, y, sample_weights, =5, =42, =):\n\n kf = KFold(=n_splits, =, =SEED)\n train_r2_scores = []\n test_r2_scores = []\n models = [] \n\n fold, (train_index, test_index) enumerate(tqdm(kf.split(X), =n_splits, =)):\n X_train, X_test = X.iloc[train_index], X.iloc[test_index]\n y_train, y_test = y.iloc[train_index], y.iloc[test_index]\n weights_train, weights_test = sample_weights[train_index], sample_weights[test_index]\n\n model_clone = model.__class__(**model.get_params())\n\n model_clone.fit(X_train, y_train)\n\n models.append(model_clone)\n\n y_train_pred = model_clone.predict(X_train)\n y_test_pred = model_clone.predict(X_test)\n\n train_r2 = weighted_r2_score(y_train, y_train_pred, weights_train)\n test_r2 = weighted_r2_score(y_test, y_test_pred, weights_test)\n\n train_r2_scores.append(train_r2)\n test_r2_scores.append(test_r2)\n\n (f)\n\n time.sleep(1)\n\n clear_output(=)\n\n with open(pickle_file, ) as f:\n pickle.dump(models, f)\n\n (f)\n (f)\n\n return models\n\nX = train[fe] \ny = train[] \nsample_weights = train[].values \n\nmodel = LGBMRegressor(=-1, =42, =)\n\nLightModels = train_and_evaluate_kfold(model, X, y, sample_weights)\nI dont think they are taking the mean of each folds losses. Can you try to save the prediction and targets and calculate the 'overall' score for each fold?
", "votes": 0, "replies": [] }, { "id": 3042267, "author_name": "yu", "author_url": "", "post_date": "2024-11-11T11:38:22.923000", "content": "@dc260123 may i ask whats your cv - lb and validation scheme looks like?
", "votes": 0, "replies": [] }, { "id": 3042465, "author_name": "dc260123", "author_url": "", "post_date": "2024-11-11T14:35:09.163000", "content": "CV 0.0082
\nlb 0.006
but my best score is a blend of two models and with online learning so take it with a grain of salt
\nLooking at other people's numbers it seems like a .002~0.003 spread seems to be reasonable
How'd you implement online learning? Do you know of any good example notebooks? I've failed at every attempt of it 😞
", "votes": 0, "replies": [] }, { "id": 3042510, "author_name": "dc260123", "author_url": "", "post_date": "2024-11-11T15:09:19.043000", "content": "I dont think they are changing the 1 minute restriction and the callback api makes it difficult too.
\nMy code is in a hobby project state with a lot of comments and todos so I can't share the whole thing without investing a lot of time making a fork and maintaining the public fork: you will most likely run into other problems anyway…
\nIf you haven't done it already I would suggest to do what the others have suggested too to set up some sort of test calling your predict callback and debug the callback directly instead of through the grpc server or modify their server locally to forward server side error properly back to client.(start with removing the wrapper exception GatewayRuntimeError)
\nAnd no I just brute-forced through it without reference any notebook… Most of my submission are for troubleshooting this. At one point I had to rely on trying to understand if I made some progress if I see \"Submission Scoring Error\" or \"Notebook Threw Exception\" : )
\nI understand that to some extend they want to prevent private test set leakage but forwarding an exception type back to us with a limit on the exception's length should be safe and helpful for troubleshooting.
", "votes": 1, "replies": [] }, { "id": 3042523, "author_name": "Jack", "author_url": "", "post_date": "2024-11-11T15:13:09.453000", "content": "I don't get errors. When I say I've failed, I mean my score drops significantly. I'm attempting to incrementally train an xgboost model by specifying the model parameter in the fit method.
", "votes": 0, "replies": [] }, { "id": 3042541, "author_name": "dc260123", "author_url": "", "post_date": "2024-11-11T15:26:50.433000", "content": "try run it with online learning on your validation set and print a result each day comparing it with without online learning?
\nI only have a theoretical understanding of gradient boosting and am fairly new to kaggle + practical use of boosting libraries but I dont think GB models are good candidates for online learning because they are not trained with SGD.
\nThe good empirical results we see are generally from training them with the provided algorithm(some sort of full batch gradient descent with GOSS/bagging). If I have to guess I think you can either try to fit the model again with the new data only or a mix of old and new data.
\nWith only the new data the model does not have enough sample to be refitted properly and if you mix new data with some old data to stabilize the gradient's direction you might overfit the model
", "votes": 0, "replies": [] }, { "id": 3042599, "author_name": "Jack", "author_url": "", "post_date": "2024-11-11T16:03:20.743000", "content": "https://www.kaggle.com/competitions/optiver-trading-at-the-close/discussion/487446
\nhttps://www.kaggle.com/competitions/optiver-trading-at-the-close/discussion/486169
\nBoth notebooks mention online learning/training for GB models.
I'll try adjusting the kaggle submission code locally to test OL methods and print with/without OL. Still working on feature engineering anyways. If you don't mind, what dates are you using for local validation? Is your reported CV with your ensemble and OL? Or ensemble without OL?
\n\"CV 0.0082
\nlb 0.006\"
validation period : date_id more over 1577
\ncv : 0.006210 , lb : 0.0039
Hello, do you have any new ideas about the updated LB scores and validation strategy?
", "votes": 0, "replies": [] }, { "id": 3037678, "author_name": "", "author_url": "", "post_date": "2024-11-06T01:58:04.957000", "content": "", "votes": 2, "replies": [] }, { "id": 3037677, "author_name": "", "author_url": "", "post_date": "2024-11-06T01:57:16.337000", "content": "", "votes": 0, "replies": [] }, { "id": 3037162, "author_name": "", "author_url": "", "post_date": "2024-11-05T12:40:55.367000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "3019116": "Hello fellow participants,\n\nPlease find this thread for CV and leaderboard relations for starter models. I shall commence with my models insofar from the below public work-\n1. https://www.kaggle.com/code/ravi20076/janestreet2024-baseline-submission-v1\n2. https://www.kaggle.com/datasets/ravi20076/janestreetpublicv1\n3. https://www.kaggle.com/code/ravi20076/janestreet2024-baseline-train-v1 \n\nAll of these are single LGBM models with simple train-test split CV scheme\n\n|Version |Date |CV score | LB score|\n| --- | --- | -------------- | ---------------|\n|LGBMV1_1 |15Oct2024|0.007555| 0.0043 |\n|LGBMV1_2|15Oct2024|0.007569| 0.0044 |\n|LGBMV1_4|15Oct2024|0.007882| 0.0043 |\n|LGBMV1_5|15Oct2024|0.007667| 0.0041 |\n|CBV1_2|16Oct2024|0.007734| 0.0039 |\n\n