{ "id": 546625, "title": "Dear Jane Street: My Cross-Validation Looks More Suspicious Than My Cousin's Business Cards ๐Ÿคฃ", "url": "/competitions/jane-street-real-time-market-data-forecasting/discussion/546625", "author_name": "Khaled Alghaish", "post_date": "2024-11-16T23:17:01.783000", "votes": 5, "comment_count": 15, "views": 0, "content": "

I've been staring at this competition like it's a suspicious package. My head? Spinning worse than when my cousin tried to explain cryptocurrency ๐Ÿ˜†

\n

I hate to pretend. Most of the ML terms are like gibberish to me! but backing down from a challenge is not me.
\nI'm taking this puzzle piece by piece!
\nRight now, I'm stuck on this validation thing. Local validation, cross-validationโ€ฆ All these terms fly around like a duck in the park.

\n

But you know what? @cdeotte (good guy, solid guy) I understood from one of his YT videos that if you can't get local validation right, you're just spinning your wheels like my cryptocurrency cousin's \"imported\" Mercedes ๐Ÿ˜‚

\n

So here's my 100th plan:

\n

Get that validation running
\nMake sure the computation metric is solid (good luck with this one ๐Ÿ˜†)
\nDo the EDA
\nBuild a baseline model
\nThen tinker

\n

I will fkng learn ML on Kaggle like I learned anything else. One step at a time.
\nBtw, you can make jokes on me or even better help if you are a Kaggle PRACTITIONER!

\n

I luuuv it, LFG ๐Ÿš€

", "messages": [ { "id": 3048000, "postDate": "2024-11-17T12:27:23.927Z", "content": "

A simple validation split that I found is correlating decent with LB is:

\n

`Training | Gap | Validation | Gap | Test

\n

Only early stop on validation if you need to, increase in test score generally correlated with LB going up too.. Leave at least 10 date_ids gap and at least 50 date_id for validation/test parts..

", "rawMarkdown": "A simple validation split that I found is correlating decent with LB is:\n\n`Training | Gap | Validation | Gap | Test \n\nOnly early stop on validation if you need to, increase in test score generally correlated with LB going up too.. Leave at least 10 date_ids gap and at least 50 date_id for validation/test parts..", "votes": 8, "replies": [ { "id": 3048416, "postDate": "2024-11-17T22:05:56.893Z", "content": "

@julianmukaj - we talking time series splits like:
\nTrain โž [10d gap] โž Val โž [10d gap] โž Test
\nSorry, I am a rookie, is my understanding correct?
\nand why, is it to try to avoid noise and generalize better?
\nBut why ๐Ÿคฃ
\nMuch respectโ€ฆ

", "rawMarkdown": "@julianmukaj - we talking time series splits like:\nTrain โž [10d gap] โž Val โž [10d gap] โž Test\nSorry, I am a rookie, is my understanding correct? \nand why, is it to try to avoid noise and generalize better? \nBut why ๐Ÿคฃ\nMuch respect... \n", "votes": 1, "replies": [ { "id": 3049084, "postDate": "2024-11-18T17:24:16.067Z", "content": "

Yes your understanding is correct, its just to simulate a more realistic time series setup to help you avoid over fitting. There is no past data validation or future data training, and there is gaps to help stop exaggerated scores from temporal patterns closely linked to the end of your training data, and the last test set should appear more independent and realistic in score compared to your train/validation.

", "rawMarkdown": "Yes your understanding is correct, its just to simulate a more realistic time series setup to help you avoid over fitting. There is no past data validation or future data training, and there is gaps to help stop exaggerated scores from temporal patterns closely linked to the end of your training data, and the last test set should appear more independent and realistic in score compared to your train/validation.", "votes": 2 } ] } ] }, { "id": 3047595, "postDate": "2024-11-16T23:17:01.783Z", "content": "

I've been staring at this competition like it's a suspicious package. My head? Spinning worse than when my cousin tried to explain cryptocurrency ๐Ÿ˜†

\n

I hate to pretend. Most of the ML terms are like gibberish to me! but backing down from a challenge is not me.
\nI'm taking this puzzle piece by piece!
\nRight now, I'm stuck on this validation thing. Local validation, cross-validationโ€ฆ All these terms fly around like a duck in the park.

\n

But you know what? @cdeotte (good guy, solid guy) I understood from one of his YT videos that if you can't get local validation right, you're just spinning your wheels like my cryptocurrency cousin's \"imported\" Mercedes ๐Ÿ˜‚

\n

So here's my 100th plan:

\n

Get that validation running
\nMake sure the computation metric is solid (good luck with this one ๐Ÿ˜†)
\nDo the EDA
\nBuild a baseline model
\nThen tinker

\n

I will fkng learn ML on Kaggle like I learned anything else. One step at a time.
\nBtw, you can make jokes on me or even better help if you are a Kaggle PRACTITIONER!

\n

I luuuv it, LFG ๐Ÿš€

", "rawMarkdown": "I've been staring at this competition like it's a suspicious package. My head? Spinning worse than when my cousin tried to explain cryptocurrency ๐Ÿ˜†\n\nI hate to pretend. Most of the ML terms are like gibberish to me! but backing down from a challenge is not me. \nI'm taking this puzzle piece by piece! \nRight now, I'm stuck on this validation thing. Local validation, cross-validation... All these terms fly around like a duck in the park.\n\nBut you know what? @cdeotte (good guy, solid guy) I understood from one of his YT videos that if you can't get local validation right, you're just spinning your wheels like my cryptocurrency cousin's \"imported\" Mercedes ๐Ÿ˜‚\n\nSo here's my 100th plan:\n\nGet that validation running\nMake sure the computation metric is solid (good luck with this one ๐Ÿ˜†)\nDo the EDA\nBuild a baseline model\nThen tinker \n\nI will fkng learn ML on Kaggle like I learned anything else. One step at a time.\nBtw, you can make jokes on me or even better help if you are a Kaggle PRACTITIONER!\n\nI luuuv it, LFG ๐Ÿš€", "votes": 5 }, { "id": 3047614, "postDate": "2024-11-17T00:18:34.787Z", "content": "

This competition is hard. Time series is hard in general, and here, we also have very weak signal and high volatility. I recommend the MCTS or Child Institute competitions to learn the basics. Here, half the challenge is building a good local validation and finding a good signal; thus, you won't have good resources in the forum/code sections. But in MCTS/Child Institute, the basics are much easier. (which would maybe make them harder for winning gold/price. But you are here to learn the basics, so it should not be a concern yet, I think?)
\nIf you insist on this competition, start from this notebook. Run it and all related notebooks, try to understand what is going on there, submit, try to tinker a bit and improve your LB score even by a single point. This process would force you to understand everything much faster. Then, it will be much easier for you to build your own model and local validation. The most important thing is moving fast. Good luck.

", "rawMarkdown": "This competition is hard. Time series is hard in general, and here, we also have very weak signal and high volatility. I recommend the MCTS or Child Institute competitions to learn the basics. Here, half the challenge is building a good local validation and finding a good signal; thus, you won't have good resources in the forum/code sections. But in MCTS/Child Institute, the basics are much easier. (which would maybe make them harder for winning gold/price. But you are here to learn the basics, so it should not be a concern yet, I think?)\nIf you insist on this competition, start from [this notebook](https://www.kaggle.com/code/ravi20076/janestreet2024-baseline-submission-v1). Run it and all related notebooks, try to understand what is going on there, submit, try to tinker a bit and improve your LB score even by a single point. This process would force you to understand everything much faster. Then, it will be much easier for you to build your own model and local validation. The most important thing is moving fast. Good luck.", "votes": 4, "replies": [ { "id": 3047726, "postDate": "2024-11-17T05:40:38.267Z", "content": "

@shlomoron thanks for the mention. I hope my work helps others like him who are looking for resources.

\n

@tinkerturbo you can ask me any doubts on the kernel. This is a 3 part series - imports, training and submission, you need to run all of them 1-by-1. Just follow them and you are set.

", "rawMarkdown": "@shlomoron thanks for the mention. I hope my work helps others like him who are looking for resources. \n\n@tinkerturbo you can ask me any doubts on the kernel. This is a 3 part series - imports, training and submission, you need to run all of them 1-by-1. Just follow them and you are set. ", "votes": 2, "replies": [ { "id": 3047960, "postDate": "2024-11-17T11:51:35.943Z", "content": "

@ravi20076 I see you are the guy who is incentivized by sharing the love. Your help is everywhere โค๏ธ
\nI'll go through the 3 and make sure to understand what matters.
\nRegarding the computation metric (Rยฒ), with fat tails and non-stationary data, and those weights, I want to understand your thought process to come up with your validation.

\n

Much appreciated in advance.

", "rawMarkdown": "@ravi20076 I see you are the guy who is incentivized by sharing the love. Your help is everywhere โค๏ธ\nI'll go through the 3 and make sure to understand what matters.\nRegarding the computation metric (Rยฒ), with fat tails and non-stationary data, and those weights, I want to understand your thought process to come up with your validation.\n\nMuch appreciated in advance.\n ", "votes": 1 } ] }, { "id": 3047906, "postDate": "2024-11-17T10:41:25.350Z", "rawMarkdown": "", "isDeleted": true }, { "id": 3047907, "postDate": "2024-11-17T10:41:47.220Z", "content": "

@shlomoron - respect for the advice chief, and I'm like a dog with a bone on this one.
\nI'm gonna run that notebook you linked like it's my grandma's Baklava๐Ÿคฃ. Understand every ingredient, every step. Then maybe push that LB score up just a point, like you said.
\nThen later on I will have higher odds of building my model and local validation. Fast execution's in the family DNA ๐Ÿ˜‰

\n

Your guidance is much appreciated ๐Ÿ™

", "rawMarkdown": "@shlomoron - respect for the advice chief, and I'm like a dog with a bone on this one.\nI'm gonna run that notebook you linked like it's my grandma's Baklava๐Ÿคฃ. Understand every ingredient, every step. Then maybe push that LB score up just a point, like you said.\nThen later on I will have higher odds of building my model and local validation. Fast execution's in the family DNA ๐Ÿ˜‰\n\nYour guidance is much appreciated ๐Ÿ™", "votes": 2, "replies": [ { "id": 3047982, "postDate": "2024-11-17T12:12:01.973Z", "content": "

This is a usual validation scheme with time series, though not the best always @tinkerturbo

", "rawMarkdown": "This is a usual validation scheme with time series, though not the best always @tinkerturbo ", "votes": 1 }, { "id": 3047997, "postDate": "2024-11-17T12:20:10.240Z", "content": "

Alright, I'll check what you shared and get back to you, much respect.
\n-K

", "rawMarkdown": "Alright, I'll check what you shared and get back to you, much respect.\n-K" }, { "id": 3048035, "postDate": "2024-11-17T13:35:57.887Z", "content": "

@tinkerturbo On the contrary. You don't need to understand every ingredient and step. Just enough to understand what happend there. Moving fast is better than understanding every small detail (and a lot of times if you try to understand everything you will start to ask yourself why the author made this and this, when in reality it's just a bug or the author missed something etc.). Try to understand it just enough to be able to tinker with it enough to achieve an improvement of one point.
\nAnd then another oneโ€ฆ

", "rawMarkdown": "@tinkerturbo On the contrary. You don't need to understand every ingredient and step. Just enough to understand what happend there. Moving fast is better than understanding every small detail (and a lot of times if you try to understand *everything* you will start to ask yourself why the author made this and this, when in reality it's just a bug or the author missed something etc.). Try to understand it just enough to be able to tinker with it enough to achieve an improvement of one point.\nAnd then another one...", "votes": 1 }, { "id": 3048399, "postDate": "2024-11-17T20:53:50.603Z", "content": "

@shlomoron Love how you cut through the BS straight to what matters. Got it.
\nFeeling good knowing I can tap your wisdom when I hit a wall.
\nMuch respect ๐Ÿ™

", "rawMarkdown": "@shlomoron Love how you cut through the BS straight to what matters. Got it.\nFeeling good knowing I can tap your wisdom when I hit a wall.\nMuch respect ๐Ÿ™\n", "votes": 1 }, { "id": 3055262, "postDate": "2024-11-25T16:54:18.483Z", "content": "

@ravi20076 Noticed your Kaggle rank dropped from ~20 to ~260. What changed? Competition's fierce out there - hardware, tinkering velocity, or out of luck? Would love to learn from your experience. ๐Ÿค

", "rawMarkdown": "@ravi20076 Noticed your Kaggle rank dropped from ~20 to ~260. What changed? Competition's fierce out there - hardware, tinkering velocity, or out of luck? Would love to learn from your experience. ๐Ÿค", "votes": 1 }, { "id": 3055301, "postDate": "2024-11-25T17:14:24.190Z", "content": "

I am currently focusing on another competition and aiming for a hold there @tinkerturbo
\nI will attend to this one a bit later

", "rawMarkdown": "I am currently focusing on another competition and aiming for a hold there @tinkerturbo \nI will attend to this one a bit later " }, { "id": 3055615, "postDate": "2024-11-25T23:15:01.703Z", "content": "

Good luck.

", "rawMarkdown": "Good luck. " } ] } ] } ], "comments": [ { "id": 3048000, "author_name": "JM", "author_url": "", "post_date": "2024-11-17T12:27:23.927000", "content": "

A simple validation split that I found is correlating decent with LB is:

\n

`Training | Gap | Validation | Gap | Test

\n

Only early stop on validation if you need to, increase in test score generally correlated with LB going up too.. Leave at least 10 date_ids gap and at least 50 date_id for validation/test parts..

", "votes": 8, "replies": [ { "id": 3048416, "author_name": "Khaled Alghaish", "author_url": "", "post_date": "2024-11-17T22:05:56.893000", "content": "

@julianmukaj - we talking time series splits like:
\nTrain โž [10d gap] โž Val โž [10d gap] โž Test
\nSorry, I am a rookie, is my understanding correct?
\nand why, is it to try to avoid noise and generalize better?
\nBut why ๐Ÿคฃ
\nMuch respectโ€ฆ

", "votes": 1, "replies": [ { "id": 3049084, "author_name": "JM", "author_url": "", "post_date": "2024-11-18T17:24:16.067000", "content": "

Yes your understanding is correct, its just to simulate a more realistic time series setup to help you avoid over fitting. There is no past data validation or future data training, and there is gaps to help stop exaggerated scores from temporal patterns closely linked to the end of your training data, and the last test set should appear more independent and realistic in score compared to your train/validation.

", "votes": 2, "replies": [] } ] } ] }, { "id": 3047614, "author_name": "greySnow", "author_url": "", "post_date": "2024-11-17T00:18:34.787000", "content": "

This competition is hard. Time series is hard in general, and here, we also have very weak signal and high volatility. I recommend the MCTS or Child Institute competitions to learn the basics. Here, half the challenge is building a good local validation and finding a good signal; thus, you won't have good resources in the forum/code sections. But in MCTS/Child Institute, the basics are much easier. (which would maybe make them harder for winning gold/price. But you are here to learn the basics, so it should not be a concern yet, I think?)
\nIf you insist on this competition, start from this notebook. Run it and all related notebooks, try to understand what is going on there, submit, try to tinker a bit and improve your LB score even by a single point. This process would force you to understand everything much faster. Then, it will be much easier for you to build your own model and local validation. The most important thing is moving fast. Good luck.

", "votes": 4, "replies": [ { "id": 3047726, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2024-11-17T05:40:38.267000", "content": "

@shlomoron thanks for the mention. I hope my work helps others like him who are looking for resources.

\n

@tinkerturbo you can ask me any doubts on the kernel. This is a 3 part series - imports, training and submission, you need to run all of them 1-by-1. Just follow them and you are set.

", "votes": 2, "replies": [ { "id": 3047960, "author_name": "Khaled Alghaish", "author_url": "", "post_date": "2024-11-17T11:51:35.943000", "content": "

@ravi20076 I see you are the guy who is incentivized by sharing the love. Your help is everywhere โค๏ธ
\nI'll go through the 3 and make sure to understand what matters.
\nRegarding the computation metric (Rยฒ), with fat tails and non-stationary data, and those weights, I want to understand your thought process to come up with your validation.

\n

Much appreciated in advance.

", "votes": 1, "replies": [] } ] }, { "id": 3047906, "author_name": "", "author_url": "", "post_date": "2024-11-17T10:41:25.350000", "content": "", "votes": 0, "replies": [] }, { "id": 3047907, "author_name": "Khaled Alghaish", "author_url": "", "post_date": "2024-11-17T10:41:47.220000", "content": "

@shlomoron - respect for the advice chief, and I'm like a dog with a bone on this one.
\nI'm gonna run that notebook you linked like it's my grandma's Baklava๐Ÿคฃ. Understand every ingredient, every step. Then maybe push that LB score up just a point, like you said.
\nThen later on I will have higher odds of building my model and local validation. Fast execution's in the family DNA ๐Ÿ˜‰

\n

Your guidance is much appreciated ๐Ÿ™

", "votes": 2, "replies": [ { "id": 3047982, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2024-11-17T12:12:01.973000", "content": "

This is a usual validation scheme with time series, though not the best always @tinkerturbo

", "votes": 1, "replies": [] }, { "id": 3047997, "author_name": "Khaled Alghaish", "author_url": "", "post_date": "2024-11-17T12:20:10.240000", "content": "

Alright, I'll check what you shared and get back to you, much respect.
\n-K

", "votes": 0, "replies": [] }, { "id": 3048035, "author_name": "greySnow", "author_url": "", "post_date": "2024-11-17T13:35:57.887000", "content": "

@tinkerturbo On the contrary. You don't need to understand every ingredient and step. Just enough to understand what happend there. Moving fast is better than understanding every small detail (and a lot of times if you try to understand everything you will start to ask yourself why the author made this and this, when in reality it's just a bug or the author missed something etc.). Try to understand it just enough to be able to tinker with it enough to achieve an improvement of one point.
\nAnd then another oneโ€ฆ

", "votes": 1, "replies": [] }, { "id": 3048399, "author_name": "Khaled Alghaish", "author_url": "", "post_date": "2024-11-17T20:53:50.603000", "content": "

@shlomoron Love how you cut through the BS straight to what matters. Got it.
\nFeeling good knowing I can tap your wisdom when I hit a wall.
\nMuch respect ๐Ÿ™

", "votes": 1, "replies": [] }, { "id": 3055262, "author_name": "Khaled Alghaish", "author_url": "", "post_date": "2024-11-25T16:54:18.483000", "content": "

@ravi20076 Noticed your Kaggle rank dropped from ~20 to ~260. What changed? Competition's fierce out there - hardware, tinkering velocity, or out of luck? Would love to learn from your experience. ๐Ÿค

", "votes": 1, "replies": [] }, { "id": 3055301, "author_name": "Ravi Ramakrishnan", "author_url": "", "post_date": "2024-11-25T17:14:24.190000", "content": "

I am currently focusing on another competition and aiming for a hold there @tinkerturbo
\nI will attend to this one a bit later

", "votes": 0, "replies": [] }, { "id": 3055615, "author_name": "Khaled Alghaish", "author_url": "", "post_date": "2024-11-25T23:15:01.703000", "content": "

Good luck.

", "votes": 0, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3048000": "A simple validation split that I found is correlating decent with LB is:\n\n`Training | Gap | Validation | Gap | Test \n\nOnly early stop on validation if you need to, increase in test score generally correlated with LB going up too.. Leave at least 10 date_ids gap and at least 50 date_id for validation/test parts..", "3047595": "I've been staring at this competition like it's a suspicious package. My head? Spinning worse than when my cousin tried to explain cryptocurrency ๐Ÿ˜†\n\nI hate to pretend. Most of the ML terms are like gibberish to me! but backing down from a challenge is not me. \nI'm taking this puzzle piece by piece! \nRight now, I'm stuck on this validation thing. Local validation, cross-validation... All these terms fly around like a duck in the park.\n\nBut you know what? @cdeotte (good guy, solid guy) I understood from one of his YT videos that if you can't get local validation right, you're just spinning your wheels like my cryptocurrency cousin's \"imported\" Mercedes ๐Ÿ˜‚\n\nSo here's my 100th plan:\n\nGet that validation running\nMake sure the computation metric is solid (good luck with this one ๐Ÿ˜†)\nDo the EDA\nBuild a baseline model\nThen tinker \n\nI will fkng learn ML on Kaggle like I learned anything else. One step at a time.\nBtw, you can make jokes on me or even better help if you are a Kaggle PRACTITIONER!\n\nI luuuv it, LFG ๐Ÿš€", "3047614": "This competition is hard. Time series is hard in general, and here, we also have very weak signal and high volatility. I recommend the MCTS or Child Institute competitions to learn the basics. Here, half the challenge is building a good local validation and finding a good signal; thus, you won't have good resources in the forum/code sections. But in MCTS/Child Institute, the basics are much easier. (which would maybe make them harder for winning gold/price. But you are here to learn the basics, so it should not be a concern yet, I think?)\nIf you insist on this competition, start from [this notebook](https://www.kaggle.com/code/ravi20076/janestreet2024-baseline-submission-v1). Run it and all related notebooks, try to understand what is going on there, submit, try to tinker a bit and improve your LB score even by a single point. This process would force you to understand everything much faster. Then, it will be much easier for you to build your own model and local validation. The most important thing is moving fast. Good luck." } }