![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F347463%2Fb573b32f6cb38894b1dc3d32341b08e4%2Fdate_1600_scaled.webp?generation=1736896107733795&alt=media\")
This competition was a fun one, with clean and well-organized data in an accessible format - big thanks to Kaggle and Jane Street for making it possible!
\nMy goal was to explore different neural network architectures, and especially try out attention mechanisms, with the hope that correlations between symbols would help out in some way. Until the last day my models were all based on a simple PyTorch LSTM, using regular features, with categorical embeddings based on features 9, 10, 11, and diffs between features for a symbol and the median over symbols at that time. I used batches of (Symbol, Time, Feature), one batch per day.
\nYesterday I got my symbol cross-attention layer to work, enabling each symbol in the batch to consider a weighted combination of features from the other symbols. This operated within a single timestep; the output was fed into the LSTM alongside the original features, which unfortunately showed no improvement. Adding in on top the original features minus the attention output (so the LSTM then saw 3x as many inputs as before), finally saw a validation uplift! (And eventually an LB 0.0086 → 0.0094 jump.)
\nOne vital trick was: to ensure the model was learning meaningful patterns, I saved and visualized the attention softmax outputs using plt.imshow().
\nI realised after the deadline it would be simple to adapt this to save all days during a training run and turn them into webp animations, one per day. Here's an example, day 1600 (scaled up for forum):
\n
I added two rows of pixels at the bottom - the last row is a playback time marker (i.e. 0..967).
\nThey'd take a while to watch, so why not watch in parallel? Loading days 1301 to 1698 in one webpage, now the attention layers fight for my attention - this is an interesting way to do anomaly detection, your eye cannot help but be drawn to the days with most changes at any time. There's apparent synchronicity at the start & end of days and the middle.
\nThis is just two frames of screen capture, to demonstrate:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F347463%2F0a7584813e84620668d7c27d90c6b614%2Ftwo-frames.gif?generation=1736895526044593&alt=media\")
Landmarks: Symbols 0, 1, 7, 9, 16 and 33 all have the same feature_09, feature_10 and feature_11 values and are often very correlated. Columns 0 and 1 are active a lot of the time meaning most symbols are paying attention to them. (That group often has correlated responder_6 values.)
\nOther columns require a bit of counting to find them!
\nThe webp files are big, it's better to view them using local file:/// so I've uploaded them as a dataset that also includes some html files to view them in. (Made with a Kaggle notebook with 500 output file limit so starts at date 1201).
Code to reproduce:
\n date_id, attention_logits stored.():\n attention_weights = F.softmax(attention_logits, dim=).numpy()\n T, B, _ = attention_weights.shape\n imageio.get_writer(f, mode=, loop=) writer:\n i range(attention_weights.shape[]):\n \n = np.vstack((attention_weights[i]**, np.zeros(B), np.zeros(B)))\n pos = (B - , int(np.((i / T) * B)))\n \n [][pos] = \n writer.append_data( * )\nThe bigger your computer & screen, the more history you can see unfold in parallel! I'd be interested to hear if watching these animations triggers any insights or recollections about the competition :)
", "messages": [ { "id": "3097000", "postDate": "01/14/2025 23:29:43", "content": "This competition was a fun one, with clean and well-organized data in an accessible format - big thanks to Kaggle and Jane Street for making it possible!
\nMy goal was to explore different neural network architectures, and especially try out attention mechanisms, with the hope that correlations between symbols would help out in some way. Until the last day my models were all based on a simple PyTorch LSTM, using regular features, with categorical embeddings based on features 9, 10, 11, and diffs between features for a symbol and the median over symbols at that time. I used batches of (Symbol, Time, Feature), one batch per day.
\nYesterday I got my symbol cross-attention layer to work, enabling each symbol in the batch to consider a weighted combination of features from the other symbols. This operated within a single timestep; the output was fed into the LSTM alongside the original features, which unfortunately showed no improvement. Adding in on top the original features minus the attention output (so the LSTM then saw 3x as many inputs as before), finally saw a validation uplift! (And eventually an LB 0.0086 → 0.0094 jump.)
\nOne vital trick was: to ensure the model was learning meaningful patterns, I saved and visualized the attention softmax outputs using plt.imshow().
\nI realised after the deadline it would be simple to adapt this to save all days during a training run and turn them into webp animations, one per day. Here's an example, day 1600 (scaled up for forum):
\n
I added two rows of pixels at the bottom - the last row is a playback time marker (i.e. 0..967).
\nThey'd take a while to watch, so why not watch in parallel? Loading days 1301 to 1698 in one webpage, now the attention layers fight for my attention - this is an interesting way to do anomaly detection, your eye cannot help but be drawn to the days with most changes at any time. There's apparent synchronicity at the start & end of days and the middle.
\nThis is just two frames of screen capture, to demonstrate:
\n
Landmarks: Symbols 0, 1, 7, 9, 16 and 33 all have the same feature_09, feature_10 and feature_11 values and are often very correlated. Columns 0 and 1 are active a lot of the time meaning most symbols are paying attention to them. (That group often has correlated responder_6 values.)
\nOther columns require a bit of counting to find them!
\nThe webp files are big, it's better to view them using local file:/// so I've uploaded them as a dataset that also includes some html files to view them in. (Made with a Kaggle notebook with 500 output file limit so starts at date 1201).
Code to reproduce:
\n date_id, attention_logits stored.():\n attention_weights = F.softmax(attention_logits, dim=).numpy()\n T, B, _ = attention_weights.shape\n imageio.get_writer(f, mode=, loop=) writer:\n i range(attention_weights.shape[]):\n \n = np.vstack((attention_weights[i]**, np.zeros(B), np.zeros(B)))\n pos = (B - , int(np.((i / T) * B)))\n \n [][pos] = \n writer.append_data( * )\nThe bigger your computer & screen, the more history you can see unfold in parallel! I'd be interested to hear if watching these animations triggers any insights or recollections about the competition :)
", "rawMarkdown": "This competition was a fun one, with clean and well-organized data in an accessible format - big thanks to Kaggle and Jane Street for making it possible!\n\nMy goal was to explore different neural network architectures, and especially try out attention mechanisms, with the hope that correlations between symbols would help out in some way. Until the last day my models were all based on a simple PyTorch LSTM, using regular features, with categorical embeddings based on features 9, 10, 11, and diffs between features for a symbol and the median over symbols at that time. I used batches of (Symbol, Time, Feature), one batch per day.\n\nYesterday I got my symbol cross-attention layer to work, enabling each symbol in the batch to consider a weighted combination of features from the other symbols. This operated within a single timestep; the output was fed into the LSTM alongside the original features, which unfortunately showed no improvement. Adding in on top the original features *minus* the attention output (so the LSTM then saw 3x as many inputs as before), finally saw a validation uplift! (And eventually an LB 0.0086 → 0.0094 jump.)\n\nOne vital trick was: to ensure the model was learning meaningful patterns, I saved and visualized the attention softmax outputs using plt.imshow().\n\nI realised after the deadline it would be simple to adapt this to save **all** days during a training run and turn them into webp animations, one per day. Here's an example, day 1600 (scaled up for forum):\n\n\n\n**I added two rows of pixels at the bottom - the last row is a playback time marker (i.e. 0..967).**\n\nThey'd take a while to watch, so why not watch in parallel? Loading days 1301 to 1698 in one webpage, now the attention layers fight for *my* attention - this is an interesting way to do anomaly detection, your eye cannot help but be drawn to the days with most changes at any time. There's apparent synchronicity at the start & end of days and the middle.\n\nThis is just two frames of screen capture, to demonstrate:\n\n\n\n\n**Landmarks**: Symbols 0, 1, 7, 9, 16 and 33 all have the same feature_09, feature_10 and feature_11 values and are often very correlated. Columns 0 and 1 are active a lot of the time meaning most symbols are *paying attention* to them. (That group often has correlated responder_6 values.)\n\nOther columns require a bit of counting to find them!\n\nThe webp files are big, it's better to view them using local `file:///` so I've uploaded them as [a dataset](https://www.kaggle.com/datasets/jtrotman/jane-street-symbol-cross-attention-animations) that also includes some html files to view them in. (Made with a Kaggle notebook with 500 output file limit so starts at date 1201).\n\nCode to reproduce:\n\n```\nfor date_id, attention_logits in stored.items():\n attention_weights = F.softmax(attention_logits, dim=-1).numpy()\n T, B, _ = attention_weights.shape\n with imageio.get_writer(f'date_{date_id:04d}.webp', mode='I', loop=0) as writer:\n for i in range(attention_weights.shape[0]):\n # nonlinear sqrt to make pixels brighter on average\n a = np.vstack((attention_weights[i]**.5, np.zeros(B), np.zeros(B)))\n pos = min(B - 1, int(np.round((i / T) * B)))\n # playback marker\n a[-1][pos] = 1\n writer.append_data(a * 255.)\n```\n\nThe bigger your computer & screen, the more history you can see unfold in parallel! I'd be interested to hear if watching these animations triggers any insights or recollections about the competition :)", "votes": null }, { "id": "3097055", "postDate": "01/15/2025 02:09:57", "content": "Thank you for sharing your very interesting symbol cross-attention layer to work. And congratulations for your excellent (long time that I don't use that word) position on Jane Street Competition Trotman.
", "rawMarkdown": "Thank you for sharing your very interesting symbol cross-attention layer to work. And congratulations for your excellent (long time that I don't use that word) position on Jane Street Competition Trotman.", "votes": null }, { "id": "3097112", "postDate": "01/15/2025 04:00:06", "content": "Your experiment with the symbol cross-attention layer sounds really interesting! I noticed you mentioned adding the original features minus the attention output, which eventually showed a validation uplift. Did you happen to try other operations as well, like using the mean, standard deviation, or something else, instead of just subtracting? Just curious if those might have worked too!
", "rawMarkdown": "Your experiment with the symbol cross-attention layer sounds really interesting! I noticed you mentioned adding the original features minus the attention output, which eventually showed a validation uplift. Did you happen to try other operations as well, like using the mean, standard deviation, or something else, instead of just subtracting? Just curious if those might have worked too!", "votes": null }, { "id": "3097418", "postDate": "01/15/2025 10:46:53", "content": "I'm not sure which axis you'd apply mean or standard deviation to?
\nI was already doing a kind of lightweight version of this by using these features:
\ndate_df = date_df()()(date_df, axis=)\nThat definitely helped early on, so the idea to subtract from attention features was mirroring that. My original intent was to replace those features with attention, but I ended up leaving those median features in, so there could be some kind of hierarchical effect.
", "rawMarkdown": "I'm not sure which axis you'd apply mean or standard deviation to?\n\nI was already doing a kind of lightweight version of this by using these features:\n```\ndate_df[extra] = date_df.groupby('time_id')[feats].transform('median').subtract(date_df[feats], axis=0)\n```\nThat definitely helped early on, so the idea to subtract from attention features was mirroring that. My original intent was to replace those features with attention, but I ended up leaving those median features in, so there could be some kind of hierarchical effect.", "votes": null }, { "id": "3098617", "postDate": "01/16/2025 17:59:27", "content": "James, (LB neighbor), congrats on the competition. I love the idea of the cross-symbol attention. We played around with this and even had an embedded network to create cross-symbol feature values, but ended up not using it in the end.
\nIn the cross-symbol attention, do you use any positional encoding? \"Positions\" among the symbols don't seem to make much sense here, so wondering how you approached this. Also, how do you deal with extra symbols?
\nGreat visualizations. Thanks for sharing.
", "rawMarkdown": "James, (LB neighbor), congrats on the competition. I love the idea of the cross-symbol attention. We played around with this and even had an embedded network to create cross-symbol feature values, but ended up not using it in the end. \n\nIn the cross-symbol attention, do you use any positional encoding? \"Positions\" among the symbols don't seem to make much sense here, so wondering how you approached this. Also, how do you deal with extra symbols? \n\nGreat visualizations. Thanks for sharing.", "votes": null }, { "id": "3098767", "postDate": "01/16/2025 21:39:10", "content": "Thanks Maciej, congrats on your public LB placing too!
\nI agree symbol positions make no sense, I've done something that is invariant to the order of the symbols in the batch - it is using two small single-hidden-layer MLPs to compute key & query vectors from each symbol's features, then a dot product (and scaling) of those vectors to create the B*B matrix. I tried simple linear projections (shallower) and also more layers in the MLPs (deeper), and using the same MLP for keys & queries, but separate networks with a single hidden layer worked best. I've added the code below:
\n (nn.Module):\n ():\n (SymbolCrossAttention, ).__init__()\n .feature_dim = feature_dim\n .projection_dim = projection_dim\n \n mlp_hidden_dim = \n \n layers = [nn.Linear(feature_dim, mlp_hidden_dim), nn.ReLU()]\n _ (): \n layers.extend([nn.Linear(mlp_hidden_dim, mlp_hidden_dim), nn.ReLU()])\n layers.append(nn.Linear(mlp_hidden_dim, projection_dim, bias=))\n .query_proj = nn.Sequential(*layers)\n\n \n layers = [nn.Linear(feature_dim, mlp_hidden_dim), nn.ReLU()]\n _ (): \n layers.extend([nn.Linear(mlp_hidden_dim, mlp_hidden_dim), nn.ReLU()])\n layers.append(nn.Linear(mlp_hidden_dim, projection_dim, bias=))\n .key_proj = nn.Sequential(*layers)\n\n ():\n \n B, L, FT = x.shape\n FT == .feature_dim, \n \n queries = .query_proj(x) \n keys = .key_proj(x) \n \n \n queries = queries.permute(, , ) \n keys = keys.permute(, , ) \n \n similarity = torch.bmm(queries, keys.transpose(, )) \n \n similarity = similarity / (.projection_dim ** ) \n \n attention_weights = F.softmax(similarity, dim=-) \n \n \n x = x.permute(, , ) \n result = torch.bmm(attention_weights, x) \n \n result = result.permute(, , ) \n result\n\n\n\n .batch_attention = SymbolCrossAttention(feature_dim, projection_dim)\n\n\n attention_result = .batch_attention(features)\n features = torch.cat([\n features,\n attention_result,\n features - attention_result,\n ], dim=-)\nI used projection_dim=48, and the 'features' above means:
\nNow I'm wondering about ways to find out which feature interactions are really helping…
\nFor new symbols, I used embeddings on features 9, 10 & 11, but clipping them to between zero (lower) and 100, 100, 1000 respectively. Hopefully if new symbols appear they may share some values with other symbols, but if not I think new embedding vectors should be learnt quite quickly. (Note that they go into the LSTM as they are, and also influence the attention projections.) That part of it is a gamble, symbols only appear gradually in the training set, I hope they don't add 100 new symbols in one day… or maybe I do, I don't know how my models would handle it relative to other teams!
", "rawMarkdown": "Thanks Maciej, congrats on your public LB placing too!\n\nI agree symbol positions make no sense, I've done something that is invariant to the order of the symbols in the batch - it is using two small single-hidden-layer MLPs to compute key & query vectors from each symbol's features, then a dot product (and scaling) of those vectors to create the B*B matrix. I tried simple linear projections (shallower) and also more layers in the MLPs (deeper), and using the same MLP for keys & queries, but separate networks with a single hidden layer worked best. I've added the code below:\n\n```python\nclass SymbolCrossAttention(nn.Module):\n def __init__(self, feature_dim, projection_dim):\n super(SymbolCrossAttention, self).__init__()\n self.feature_dim = feature_dim\n self.projection_dim = projection_dim\n # Learnable projections for computing similarity\n mlp_hidden_dim = 256\n # Define MLP for queries\n layers = [nn.Linear(feature_dim, mlp_hidden_dim), nn.ReLU()]\n for _ in range(0): # Add hidden layers\n layers.extend([nn.Linear(mlp_hidden_dim, mlp_hidden_dim), nn.ReLU()])\n layers.append(nn.Linear(mlp_hidden_dim, projection_dim, bias=False))\n self.query_proj = nn.Sequential(*layers)\n \n # Define MLP for keys (can share weights with queries if desired)\n layers = [nn.Linear(feature_dim, mlp_hidden_dim), nn.ReLU()]\n for _ in range(0): # Add hidden layers\n layers.extend([nn.Linear(mlp_hidden_dim, mlp_hidden_dim), nn.ReLU()])\n layers.append(nn.Linear(mlp_hidden_dim, projection_dim, bias=False))\n self.key_proj = nn.Sequential(*layers)\n\n def forward(self, x):\n # Input shape: (B, L, FT)\n B, L, FT = x.shape\n assert FT == self.feature_dim, \"Feature dimension mismatch\"\n # Project features into the high-dimensional space\n queries = self.query_proj(x) # Shape: (B, L, P)\n keys = self.key_proj(x) # Shape: (B, L, P)\n # Compute similarity scores for all timesteps in parallel\n # Step 1: Reshape for batch-level interaction\n queries = queries.permute(1, 0, 2) # Shape: (L, B, P)\n keys = keys.permute(1, 0, 2) # Shape: (L, B, P)\n # Step 2: Compute similarity matrix: (L, B, B)\n similarity = torch.bmm(queries, keys.transpose(1, 2)) # Dot product: (L, B, B)\n # Step 3: Scale the similarity scores by 1 / sqrt(P)\n similarity = similarity / (self.projection_dim ** 0.5) # Scale by sqrt(P)\n # Step 3: Normalize with softmax over rows\n attention_weights = F.softmax(similarity, dim=-1) # Shape: (L, B, B)\n # Step 4: Compute weighted sum of features\n # Reshape original features for batch-level interaction\n x = x.permute(1, 0, 2) # Shape: (L, B, FT)\n result = torch.bmm(attention_weights, x) # Weighted sum: (L, B, FT)\n # Step 5: Reshape back to original shape\n result = result.permute(1, 0, 2) # Shape: (B, L, FT)\n return result\n\n\n# Main model: __init__\n self.batch_attention = SymbolCrossAttention(feature_dim, projection_dim)\n\n# Main model: in forward()\n attention_result = self.batch_attention(features)\n features = torch.cat([\n features,\n attention_result,\n features - attention_result,\n ], dim=-1)\n\n```\n\nI used projection_dim=48, and the 'features' above means:\n - live_features (those that vary per time_id)\n - live_features - median_live_features_for_time_id\n - day_features (those that vary per date_id, 20 to 31?)\n - feature_09_embed # 8 features\n - feature_10_embed # 12 features\n - feature_11_embed # 12 features\n - time_sin\n - time_cos\n \nNow I'm wondering about ways to find out which feature interactions are really helping...\n\nFor new symbols, I used embeddings on features 9, 10 & 11, but clipping them to between zero (lower) and 100, 100, 1000 respectively. Hopefully if new symbols appear they may share some values with other symbols, but if not I *think* new embedding vectors should be learnt quite quickly. (Note that they go into the LSTM as they are, and also influence the attention projections.) That part of it is a gamble, symbols only appear gradually in the training set, I hope they don't add 100 new symbols in one day... or maybe I do, I don't know how my models would handle it relative to other teams!", "votes": null }, { "id": "3098786", "postDate": "01/16/2025 22:37:53", "content": "Hi James, congratulations on your impressive performance in the competition! I found your post on Symbol Cross-Attention Animations and your model architecture incredibly insightful.
\nI have a question regarding the inference phase of your SymbolCrossAttention model. Specifically, does your approach explain how predictions are made when each batch during inference contains features for only one time point across all symbol_ids? Or do you incorporate lag features for context?
", "rawMarkdown": "Hi James, congratulations on your impressive performance in the competition! I found your post on Symbol Cross-Attention Animations and your model architecture incredibly insightful.\n\nI have a question regarding the inference phase of your SymbolCrossAttention model. Specifically, does your approach explain how predictions are made when each batch during inference contains features for only one time point across all symbol_ids? Or do you incorporate lag features for context?", "votes": null }, { "id": "3098804", "postDate": "01/16/2025 23:44:31", "content": "Thanks @sumenzhang - I don't use any lags or rolling statistics at all, so I can just format the test dataframe we get with a single time_id into the right format and pass it to the model, along with the hidden states from the LSTM (that were returned from the previous timestep). The LSTM is remembering some state from previous timesteps, and it keeps the feature preparation very simple. I only really went that way because the first LSTM model I submitted used a slow pandas groupby operation to format the batches, which turned out to be too slow for the submission time limits, so that biased me to thinking more feature preparation would have taken too long. I switched to polars quite late, and realised much much more feature engineering would have been possible, but too late to really explore it.
@jtrotman Thanks for sharing. This is very nice. I'll have to play around with it. Theoretically, we should be adding the attention_result, right? But, I agree that in the context of this problem, taking the difference makes more sense -- just because many of the feature values are correlated across symbols and thus you'd want to know how individual symbols differ and the attention_result is a more robust form of the cross-symbol median that you were using or the cross-symbol mean that we were using.
", "rawMarkdown": "jtrotman Thanks for sharing. This is very nice. I'll have to play around with it. Theoretically, we should be adding the attention_result, right? But, I agree that in the context of this problem, taking the difference makes more sense -- just because many of the feature values are correlated across symbols and thus you'd want to know how individual symbols differ and the attention_result is a more robust form of the cross-symbol median that you were using or the cross-symbol mean that we were using.", "votes": null }, { "id": "3099567", "postDate": "01/17/2025 23:07:30", "content": "A huge congratulations on your incredible performance in the competition!
", "rawMarkdown": "A huge congratulations on your incredible performance in the competition!", "votes": null }, { "id": "3102932", "postDate": "01/22/2025 20:23:13", "content": "@jtrotman I've finally gotten around to experimenting with cross-symbol attention. You posted a nice and clean piece of code :) I noticed that you use an a hidden layer when projecting features to Keys and Queries -- did this yield better results than simple linear projections? BTW, I've been having a very difficult time boosting validation performance with Attention, have you seen something similar? I use the last 126 days of data as my \"validation\". I mention this because my best model in LB (0.96) is NOT the best scoring model in validation -- that model only scores 0.91 in LB.
", "rawMarkdown": "jtrotman I've finally gotten around to experimenting with cross-symbol attention. You posted a nice and clean piece of code :) I noticed that you use an a hidden layer when projecting features to Keys and Queries -- did this yield better results than simple linear projections? BTW, I've been having a very difficult time boosting validation performance with Attention, have you seen something similar? I use the last 126 days of data as my \"validation\". I mention this because my best model in LB (0.96) is NOT the best scoring model in validation -- that model only scores 0.91 in LB.", "votes": null }, { "id": "3103099", "postDate": "01/23/2025 01:49:56", "content": "I did try simple linear projections for key/query but don't have a log of that run, I think I truncated it early, so maybe it would work. I'm more confident that anything deeper than 1 hidden layer did not work.
\nFor validation, I was doing a walk-forward process of predicting on one day, recording the wr2 score (numerator and denominator), then doing the optimizer step (weight update) for that day. I then used a sliding 100 day window of the performance (summing numerators and denominators), but with particular focus on the last 100 days of train.
\nHere's a plot:
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F347463%2Fd29602e8cc8bb02bedd415a30c324b54%2Fjane-st-training-log.png?generation=1737596669637104&alt=media\")
(Days 1487 and 1488 are wild, with scores 0.015+, which bumps the 100 day average; when they leave the 100 day window it's an even more dramatic drop.)
\nI'd got the final 100 day rolling average from 0.006 to 0.007, 0.008, then 0.010 for a long time without much progress. Adding the cross attention got it to 0.011, although I did not submit it on its own, so the 0.0086 → 0.0094 LB was from adding it to my existing submission with two other LSTM based models (giving the new model a higher weight). If/when late submissions come back I will get some individual scores…
", "rawMarkdown": "I did try simple linear projections for key/query but don't have a log of that run, I think I truncated it early, so maybe it would work. I'm more confident that anything deeper than 1 hidden layer did not work.\n\nFor validation, I was doing a walk-forward process of predicting on one day, recording the wr2 score (numerator and denominator), then doing the optimizer step (weight update) for that day. I then used a sliding 100 day window of the performance (summing numerators and denominators), but with particular focus on the last 100 days of train.\n\nHere's a plot:\n\n\n\n - blue is daily wr2 score\n - green is rolling 100 day average\n - orange is global average (i.e. *expanding* window)\n - red just highlights the zero line\n\n(Days 1487 and 1488 are wild, with scores 0.015+, which bumps the 100 day average; when they leave the 100 day window it's an even more dramatic drop.)\n\nI'd got the final 100 day rolling average from 0.006 to 0.007, 0.008, then 0.010 for a long time without much progress. Adding the cross attention got it to 0.011, although I did not submit it on its own, so the 0.0086 → 0.0094 LB was from adding it to my existing submission with two other LSTM based models (giving the new model a higher weight). If/when late submissions come back I will get some individual scores...", "votes": null } ], "comments": [ { "id": 3097055, "author_name": "mpwolke", "author_url": "", "post_date": "01/15/2025 02:09:57", "content": "Thank you for sharing your very interesting symbol cross-attention layer to work. And congratulations for your excellent (long time that I don't use that word) position on Jane Street Competition Trotman.
", "votes": null, "replies": [] }, { "id": 3097112, "author_name": "byunjins", "author_url": "", "post_date": "01/15/2025 04:00:06", "content": "Your experiment with the symbol cross-attention layer sounds really interesting! I noticed you mentioned adding the original features minus the attention output, which eventually showed a validation uplift. Did you happen to try other operations as well, like using the mean, standard deviation, or something else, instead of just subtracting? Just curious if those might have worked too!
", "votes": null, "replies": [ { "id": 3097418, "author_name": "jtrotman", "author_url": "", "post_date": "01/15/2025 10:46:53", "content": "I'm not sure which axis you'd apply mean or standard deviation to?
\nI was already doing a kind of lightweight version of this by using these features:
\ndate_df = date_df()()(date_df, axis=)\nThat definitely helped early on, so the idea to subtract from attention features was mirroring that. My original intent was to replace those features with attention, but I ended up leaving those median features in, so there could be some kind of hierarchical effect.
", "votes": null, "replies": [] } ] }, { "id": 3098617, "author_name": "maciejzawadzki", "author_url": "", "post_date": "01/16/2025 17:59:27", "content": "James, (LB neighbor), congrats on the competition. I love the idea of the cross-symbol attention. We played around with this and even had an embedded network to create cross-symbol feature values, but ended up not using it in the end.
\nIn the cross-symbol attention, do you use any positional encoding? \"Positions\" among the symbols don't seem to make much sense here, so wondering how you approached this. Also, how do you deal with extra symbols?
\nGreat visualizations. Thanks for sharing.
", "votes": null, "replies": [ { "id": 3098767, "author_name": "jtrotman", "author_url": "", "post_date": "01/16/2025 21:39:10", "content": "Thanks Maciej, congrats on your public LB placing too!
\nI agree symbol positions make no sense, I've done something that is invariant to the order of the symbols in the batch - it is using two small single-hidden-layer MLPs to compute key & query vectors from each symbol's features, then a dot product (and scaling) of those vectors to create the B*B matrix. I tried simple linear projections (shallower) and also more layers in the MLPs (deeper), and using the same MLP for keys & queries, but separate networks with a single hidden layer worked best. I've added the code below:
\n (nn.Module):\n ():\n (SymbolCrossAttention, ).__init__()\n .feature_dim = feature_dim\n .projection_dim = projection_dim\n \n mlp_hidden_dim = \n \n layers = [nn.Linear(feature_dim, mlp_hidden_dim), nn.ReLU()]\n _ (): \n layers.extend([nn.Linear(mlp_hidden_dim, mlp_hidden_dim), nn.ReLU()])\n layers.append(nn.Linear(mlp_hidden_dim, projection_dim, bias=))\n .query_proj = nn.Sequential(*layers)\n\n \n layers = [nn.Linear(feature_dim, mlp_hidden_dim), nn.ReLU()]\n _ (): \n layers.extend([nn.Linear(mlp_hidden_dim, mlp_hidden_dim), nn.ReLU()])\n layers.append(nn.Linear(mlp_hidden_dim, projection_dim, bias=))\n .key_proj = nn.Sequential(*layers)\n\n ():\n \n B, L, FT = x.shape\n FT == .feature_dim, \n \n queries = .query_proj(x) \n keys = .key_proj(x) \n \n \n queries = queries.permute(, , ) \n keys = keys.permute(, , ) \n \n similarity = torch.bmm(queries, keys.transpose(, )) \n \n similarity = similarity / (.projection_dim ** ) \n \n attention_weights = F.softmax(similarity, dim=-) \n \n \n x = x.permute(, , ) \n result = torch.bmm(attention_weights, x) \n \n result = result.permute(, , ) \n result\n\n\n\n .batch_attention = SymbolCrossAttention(feature_dim, projection_dim)\n\n\n attention_result = .batch_attention(features)\n features = torch.cat([\n features,\n attention_result,\n features - attention_result,\n ], dim=-)\nI used projection_dim=48, and the 'features' above means:
\nNow I'm wondering about ways to find out which feature interactions are really helping…
\nFor new symbols, I used embeddings on features 9, 10 & 11, but clipping them to between zero (lower) and 100, 100, 1000 respectively. Hopefully if new symbols appear they may share some values with other symbols, but if not I think new embedding vectors should be learnt quite quickly. (Note that they go into the LSTM as they are, and also influence the attention projections.) That part of it is a gamble, symbols only appear gradually in the training set, I hope they don't add 100 new symbols in one day… or maybe I do, I don't know how my models would handle it relative to other teams!
", "votes": null, "replies": [ { "id": 3098986, "author_name": "maciejzawadzki", "author_url": "", "post_date": "01/17/2025 06:39:08", "content": "@jtrotman Thanks for sharing. This is very nice. I'll have to play around with it. Theoretically, we should be adding the attention_result, right? But, I agree that in the context of this problem, taking the difference makes more sense -- just because many of the feature values are correlated across symbols and thus you'd want to know how individual symbols differ and the attention_result is a more robust form of the cross-symbol median that you were using or the cross-symbol mean that we were using.
", "votes": null, "replies": [] }, { "id": 3102932, "author_name": "maciejzawadzki", "author_url": "", "post_date": "01/22/2025 20:23:13", "content": "@jtrotman I've finally gotten around to experimenting with cross-symbol attention. You posted a nice and clean piece of code :) I noticed that you use an a hidden layer when projecting features to Keys and Queries -- did this yield better results than simple linear projections? BTW, I've been having a very difficult time boosting validation performance with Attention, have you seen something similar? I use the last 126 days of data as my \"validation\". I mention this because my best model in LB (0.96) is NOT the best scoring model in validation -- that model only scores 0.91 in LB.
", "votes": null, "replies": [ { "id": 3103099, "author_name": "jtrotman", "author_url": "", "post_date": "01/23/2025 01:49:56", "content": "I did try simple linear projections for key/query but don't have a log of that run, I think I truncated it early, so maybe it would work. I'm more confident that anything deeper than 1 hidden layer did not work.
\nFor validation, I was doing a walk-forward process of predicting on one day, recording the wr2 score (numerator and denominator), then doing the optimizer step (weight update) for that day. I then used a sliding 100 day window of the performance (summing numerators and denominators), but with particular focus on the last 100 days of train.
\nHere's a plot:
\n
(Days 1487 and 1488 are wild, with scores 0.015+, which bumps the 100 day average; when they leave the 100 day window it's an even more dramatic drop.)
\nI'd got the final 100 day rolling average from 0.006 to 0.007, 0.008, then 0.010 for a long time without much progress. Adding the cross attention got it to 0.011, although I did not submit it on its own, so the 0.0086 → 0.0094 LB was from adding it to my existing submission with two other LSTM based models (giving the new model a higher weight). If/when late submissions come back I will get some individual scores…
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 3098786, "author_name": "sumenzhang", "author_url": "", "post_date": "01/16/2025 22:37:53", "content": "Hi James, congratulations on your impressive performance in the competition! I found your post on Symbol Cross-Attention Animations and your model architecture incredibly insightful.
\nI have a question regarding the inference phase of your SymbolCrossAttention model. Specifically, does your approach explain how predictions are made when each batch during inference contains features for only one time point across all symbol_ids? Or do you incorporate lag features for context?
", "votes": null, "replies": [ { "id": 3098804, "author_name": "jtrotman", "author_url": "", "post_date": "01/16/2025 23:44:31", "content": "Thanks @sumenzhang - I don't use any lags or rolling statistics at all, so I can just format the test dataframe we get with a single time_id into the right format and pass it to the model, along with the hidden states from the LSTM (that were returned from the previous timestep). The LSTM is remembering some state from previous timesteps, and it keeps the feature preparation very simple. I only really went that way because the first LSTM model I submitted used a slow pandas groupby operation to format the batches, which turned out to be too slow for the submission time limits, so that biased me to thinking more feature preparation would have taken too long. I switched to polars quite late, and realised much much more feature engineering would have been possible, but too late to really explore it.
A huge congratulations on your incredible performance in the competition!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3097000": "This competition was a fun one, with clean and well-organized data in an accessible format - big thanks to Kaggle and Jane Street for making it possible!\n\nMy goal was to explore different neural network architectures, and especially try out attention mechanisms, with the hope that correlations between symbols would help out in some way. Until the last day my models were all based on a simple PyTorch LSTM, using regular features, with categorical embeddings based on features 9, 10, 11, and diffs between features for a symbol and the median over symbols at that time. I used batches of (Symbol, Time, Feature), one batch per day.\n\nYesterday I got my symbol cross-attention layer to work, enabling each symbol in the batch to consider a weighted combination of features from the other symbols. This operated within a single timestep; the output was fed into the LSTM alongside the original features, which unfortunately showed no improvement. Adding in on top the original features *minus* the attention output (so the LSTM then saw 3x as many inputs as before), finally saw a validation uplift! (And eventually an LB 0.0086 → 0.0094 jump.)\n\nOne vital trick was: to ensure the model was learning meaningful patterns, I saved and visualized the attention softmax outputs using plt.imshow().\n\nI realised after the deadline it would be simple to adapt this to save **all** days during a training run and turn them into webp animations, one per day. Here's an example, day 1600 (scaled up for forum):\n\n\n\n**I added two rows of pixels at the bottom - the last row is a playback time marker (i.e. 0..967).**\n\nThey'd take a while to watch, so why not watch in parallel? Loading days 1301 to 1698 in one webpage, now the attention layers fight for *my* attention - this is an interesting way to do anomaly detection, your eye cannot help but be drawn to the days with most changes at any time. There's apparent synchronicity at the start & end of days and the middle.\n\nThis is just two frames of screen capture, to demonstrate:\n\n\n\n\n**Landmarks**: Symbols 0, 1, 7, 9, 16 and 33 all have the same feature_09, feature_10 and feature_11 values and are often very correlated. Columns 0 and 1 are active a lot of the time meaning most symbols are *paying attention* to them. (That group often has correlated responder_6 values.)\n\nOther columns require a bit of counting to find them!\n\nThe webp files are big, it's better to view them using local `file:///` so I've uploaded them as [a dataset](https://www.kaggle.com/datasets/jtrotman/jane-street-symbol-cross-attention-animations) that also includes some html files to view them in. (Made with a Kaggle notebook with 500 output file limit so starts at date 1201).\n\nCode to reproduce:\n\n```\nfor date_id, attention_logits in stored.items():\n attention_weights = F.softmax(attention_logits, dim=-1).numpy()\n T, B, _ = attention_weights.shape\n with imageio.get_writer(f'date_{date_id:04d}.webp', mode='I', loop=0) as writer:\n for i in range(attention_weights.shape[0]):\n # nonlinear sqrt to make pixels brighter on average\n a = np.vstack((attention_weights[i]**.5, np.zeros(B), np.zeros(B)))\n pos = min(B - 1, int(np.round((i / T) * B)))\n # playback marker\n a[-1][pos] = 1\n writer.append_data(a * 255.)\n```\n\nThe bigger your computer & screen, the more history you can see unfold in parallel! I'd be interested to hear if watching these animations triggers any insights or recollections about the competition :)", "3097055": "Thank you for sharing your very interesting symbol cross-attention layer to work. And congratulations for your excellent (long time that I don't use that word) position on Jane Street Competition Trotman.", "3097112": "Your experiment with the symbol cross-attention layer sounds really interesting! I noticed you mentioned adding the original features minus the attention output, which eventually showed a validation uplift. Did you happen to try other operations as well, like using the mean, standard deviation, or something else, instead of just subtracting? Just curious if those might have worked too!", "3097418": "I'm not sure which axis you'd apply mean or standard deviation to?\n\nI was already doing a kind of lightweight version of this by using these features:\n```\ndate_df[extra] = date_df.groupby('time_id')[feats].transform('median').subtract(date_df[feats], axis=0)\n```\nThat definitely helped early on, so the idea to subtract from attention features was mirroring that. My original intent was to replace those features with attention, but I ended up leaving those median features in, so there could be some kind of hierarchical effect.", "3098617": "James, (LB neighbor), congrats on the competition. I love the idea of the cross-symbol attention. We played around with this and even had an embedded network to create cross-symbol feature values, but ended up not using it in the end. \n\nIn the cross-symbol attention, do you use any positional encoding? \"Positions\" among the symbols don't seem to make much sense here, so wondering how you approached this. Also, how do you deal with extra symbols? \n\nGreat visualizations. Thanks for sharing.", "3098767": "Thanks Maciej, congrats on your public LB placing too!\n\nI agree symbol positions make no sense, I've done something that is invariant to the order of the symbols in the batch - it is using two small single-hidden-layer MLPs to compute key & query vectors from each symbol's features, then a dot product (and scaling) of those vectors to create the B*B matrix. I tried simple linear projections (shallower) and also more layers in the MLPs (deeper), and using the same MLP for keys & queries, but separate networks with a single hidden layer worked best. I've added the code below:\n\n```python\nclass SymbolCrossAttention(nn.Module):\n def __init__(self, feature_dim, projection_dim):\n super(SymbolCrossAttention, self).__init__()\n self.feature_dim = feature_dim\n self.projection_dim = projection_dim\n # Learnable projections for computing similarity\n mlp_hidden_dim = 256\n # Define MLP for queries\n layers = [nn.Linear(feature_dim, mlp_hidden_dim), nn.ReLU()]\n for _ in range(0): # Add hidden layers\n layers.extend([nn.Linear(mlp_hidden_dim, mlp_hidden_dim), nn.ReLU()])\n layers.append(nn.Linear(mlp_hidden_dim, projection_dim, bias=False))\n self.query_proj = nn.Sequential(*layers)\n \n # Define MLP for keys (can share weights with queries if desired)\n layers = [nn.Linear(feature_dim, mlp_hidden_dim), nn.ReLU()]\n for _ in range(0): # Add hidden layers\n layers.extend([nn.Linear(mlp_hidden_dim, mlp_hidden_dim), nn.ReLU()])\n layers.append(nn.Linear(mlp_hidden_dim, projection_dim, bias=False))\n self.key_proj = nn.Sequential(*layers)\n\n def forward(self, x):\n # Input shape: (B, L, FT)\n B, L, FT = x.shape\n assert FT == self.feature_dim, \"Feature dimension mismatch\"\n # Project features into the high-dimensional space\n queries = self.query_proj(x) # Shape: (B, L, P)\n keys = self.key_proj(x) # Shape: (B, L, P)\n # Compute similarity scores for all timesteps in parallel\n # Step 1: Reshape for batch-level interaction\n queries = queries.permute(1, 0, 2) # Shape: (L, B, P)\n keys = keys.permute(1, 0, 2) # Shape: (L, B, P)\n # Step 2: Compute similarity matrix: (L, B, B)\n similarity = torch.bmm(queries, keys.transpose(1, 2)) # Dot product: (L, B, B)\n # Step 3: Scale the similarity scores by 1 / sqrt(P)\n similarity = similarity / (self.projection_dim ** 0.5) # Scale by sqrt(P)\n # Step 3: Normalize with softmax over rows\n attention_weights = F.softmax(similarity, dim=-1) # Shape: (L, B, B)\n # Step 4: Compute weighted sum of features\n # Reshape original features for batch-level interaction\n x = x.permute(1, 0, 2) # Shape: (L, B, FT)\n result = torch.bmm(attention_weights, x) # Weighted sum: (L, B, FT)\n # Step 5: Reshape back to original shape\n result = result.permute(1, 0, 2) # Shape: (B, L, FT)\n return result\n\n\n# Main model: __init__\n self.batch_attention = SymbolCrossAttention(feature_dim, projection_dim)\n\n# Main model: in forward()\n attention_result = self.batch_attention(features)\n features = torch.cat([\n features,\n attention_result,\n features - attention_result,\n ], dim=-1)\n\n```\n\nI used projection_dim=48, and the 'features' above means:\n - live_features (those that vary per time_id)\n - live_features - median_live_features_for_time_id\n - day_features (those that vary per date_id, 20 to 31?)\n - feature_09_embed # 8 features\n - feature_10_embed # 12 features\n - feature_11_embed # 12 features\n - time_sin\n - time_cos\n \nNow I'm wondering about ways to find out which feature interactions are really helping...\n\nFor new symbols, I used embeddings on features 9, 10 & 11, but clipping them to between zero (lower) and 100, 100, 1000 respectively. Hopefully if new symbols appear they may share some values with other symbols, but if not I *think* new embedding vectors should be learnt quite quickly. (Note that they go into the LSTM as they are, and also influence the attention projections.) That part of it is a gamble, symbols only appear gradually in the training set, I hope they don't add 100 new symbols in one day... or maybe I do, I don't know how my models would handle it relative to other teams!", "3098786": "Hi James, congratulations on your impressive performance in the competition! I found your post on Symbol Cross-Attention Animations and your model architecture incredibly insightful.\n\nI have a question regarding the inference phase of your SymbolCrossAttention model. Specifically, does your approach explain how predictions are made when each batch during inference contains features for only one time point across all symbol_ids? Or do you incorporate lag features for context?", "3098804": "Thanks @sumenzhang - I don't use any lags or rolling statistics at all, so I can just format the `test` dataframe we get with a single time_id into the right format and pass it to the model, along with the hidden states from the LSTM (that were returned from the previous timestep). The LSTM is remembering some state from previous timesteps, and it keeps the feature preparation very simple. I only really went that way because the first LSTM model I submitted used a slow pandas `groupby` operation to format the batches, which turned out to be too slow for the submission time limits, so that biased me to thinking more feature preparation would have taken too long. I switched to polars quite late, and realised much much more feature engineering would have been possible, but too late to really explore it.", "3098986": "jtrotman Thanks for sharing. This is very nice. I'll have to play around with it. Theoretically, we should be adding the attention_result, right? But, I agree that in the context of this problem, taking the difference makes more sense -- just because many of the feature values are correlated across symbols and thus you'd want to know how individual symbols differ and the attention_result is a more robust form of the cross-symbol median that you were using or the cross-symbol mean that we were using.", "3099567": "A huge congratulations on your incredible performance in the competition!", "3102932": "jtrotman I've finally gotten around to experimenting with cross-symbol attention. You posted a nice and clean piece of code :) I noticed that you use an a hidden layer when projecting features to Keys and Queries -- did this yield better results than simple linear projections? BTW, I've been having a very difficult time boosting validation performance with Attention, have you seen something similar? I use the last 126 days of data as my \"validation\". I mention this because my best model in LB (0.96) is NOT the best scoring model in validation -- that model only scores 0.91 in LB.", "3103099": "I did try simple linear projections for key/query but don't have a log of that run, I think I truncated it early, so maybe it would work. I'm more confident that anything deeper than 1 hidden layer did not work.\n\nFor validation, I was doing a walk-forward process of predicting on one day, recording the wr2 score (numerator and denominator), then doing the optimizer step (weight update) for that day. I then used a sliding 100 day window of the performance (summing numerators and denominators), but with particular focus on the last 100 days of train.\n\nHere's a plot:\n\n\n\n - blue is daily wr2 score\n - green is rolling 100 day average\n - orange is global average (i.e. *expanding* window)\n - red just highlights the zero line\n\n(Days 1487 and 1488 are wild, with scores 0.015+, which bumps the 100 day average; when they leave the 100 day window it's an even more dramatic drop.)\n\nI'd got the final 100 day rolling average from 0.006 to 0.007, 0.008, then 0.010 for a long time without much progress. Adding the cross attention got it to 0.011, although I did not submit it on its own, so the 0.0086 → 0.0094 LB was from adding it to my existing submission with two other LSTM based models (giving the new model a higher weight). If/when late submissions come back I will get some individual scores..." }, "source": "meta" }