\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2051677%2Fce2f767ac69da293d3b20655f9fa8b6d%2Fpacfacf.png?generation=1750622050282378&alt=media\")
Hi everyone, I spent some time trying to \"reverse-engineer\" the target (inspired by this brilliant post, tbh).
\nCan't say I discovered a big insight, but maybe someone will find it helpful.
\nFirst, I decided to check the autocorrelation properties of the target time series:
\n
At this point, I started to suspect the target series might be an AR(1) process, since the main correlation sits in lag 1.
\nFitting ARIMA(1,0,0) yields coef=0.9812, const=0.036, and residuals std=0.195.
A Student-t distribution fits the residuals pretty well, unlike the normal distribution (probably due to heavy tails):
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2051677%2F866e45f31a0553355737c70d02862634%2Fqqplot.png?generation=1750622746253530&alt=media\")
Finally, I was really curious to check if the target's spectrum matches the theoretical one—the power spectral density for AR(1):
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F2051677%2F900a60a9e40753ed496a96e700f5d604%2Fpsd.png?generation=1750623709638675&alt=media\")
The AR(1) PSD roughly fits the FFT spectrum normalized by the series length.
\nIf the label is some price functional like pct_change, I would expect it to behave like a noise process with little lag 1 correlation. But we see that an AR(1)-like process models the target pretty well. Is it even possible to observe such a phenomenon at minute frequencies in financial markets?
Hi everyone, I spent some time trying to \"reverse-engineer\" the target (inspired by this brilliant post, tbh).
\nCan't say I discovered a big insight, but maybe someone will find it helpful.
\nFirst, I decided to check the autocorrelation properties of the target time series:
\n
At this point, I started to suspect the target series might be an AR(1) process, since the main correlation sits in lag 1.
\nFitting ARIMA(1,0,0) yields coef=0.9812, const=0.036, and residuals std=0.195.
A Student-t distribution fits the residuals pretty well, unlike the normal distribution (probably due to heavy tails):
\n
Finally, I was really curious to check if the target's spectrum matches the theoretical one—the power spectral density for AR(1):
\n
The AR(1) PSD roughly fits the FFT spectrum normalized by the series length.
\nIf the label is some price functional like pct_change, I would expect it to behave like a noise process with little lag 1 correlation. But we see that an AR(1)-like process models the target pretty well. Is it even possible to observe such a phenomenon at minute frequencies in financial markets?
The label seems to be a long-term pct_change() — likely covering a period of over 1 hour — rather than a simply 1-minute return. That could explain the AR1 pattern.
", "rawMarkdown": "The label seems to be a long-term pct_change() — likely covering a period of over 1 hour — rather than a simply 1-minute return. That could explain the AR1 pattern.", "votes": null }, { "id": "3231220", "postDate": "06/24/2025 05:06:15", "content": "It is an ema of minute return of the underlying
", "rawMarkdown": "It is an ema of minute return of the underlying", "votes": null } ], "comments": [ { "id": 3230442, "author_name": "melodyd", "author_url": "", "post_date": "06/23/2025 03:46:14", "content": "The label seems to be a long-term pct_change() — likely covering a period of over 1 hour — rather than a simply 1-minute return. That could explain the AR1 pattern.
", "votes": null, "replies": [] }, { "id": 3231220, "author_name": "tony271ynot", "author_url": "", "post_date": "06/24/2025 05:06:15", "content": "It is an ema of minute return of the underlying
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3230360": "Hi everyone, I spent some time trying to \"reverse-engineer\" the target (inspired by this brilliant [post](https://www.kaggle.com/competitions/jane-street-real-time-market-data-forecasting/discussion/555562), tbh). \n\nCan't say I discovered a big insight, but maybe someone will find it helpful. \n\n### **PACF and ACF** \nFirst, I decided to check the autocorrelation properties of the target time series: \n \n\nAt this point, I started to suspect the target series might be an AR(1) process, since the main correlation sits in lag 1. \n\n### **ARIMA residuals** \nFitting ARIMA(1,0,0) yields `coef=0.9812`, `const=0.036`, and residuals `std=0.195`. \n\nA Student-t distribution fits the residuals pretty well, unlike the normal distribution (probably due to heavy tails): \n \n\n### **Fourier transform spectrum** \nFinally, I was really curious to check if the target's spectrum matches the theoretical one—the power spectral density for AR(1): \n \n\nThe AR(1) PSD roughly fits the FFT spectrum normalized by the series length. \n\n### **How is this possible?** \nIf the label is some price functional like `pct_change`, I would expect it to behave like a noise process with little lag 1 correlation. But we see that an AR(1)-like process models the target pretty well. Is it even possible to observe such a phenomenon at minute frequencies in financial markets?", "3230442": "The label seems to be a long-term pct_change() — likely covering a period of over 1 hour — rather than a simply 1-minute return. That could explain the AR1 pattern.", "3231220": "It is an ema of minute return of the underlying" }, "source": "meta" }