{
  "id": 553231,
  "title": "Ragged tensors or padding?",
  "url": "/competitions/jane-street-real-time-market-data-forecasting/discussion/553231",
  "author_name": "",
  "post_date": "2024-12-24T14:36:34.885233600Z",
  "votes": 4,
  "comment_count": 3,
  "views": 0,
  "content": "<p>It’s possible to approach this problem as a sequence, but in reality, we have a variable number of symbol_ids. If we define our seq_len as either symbol_id or time_id, we will end up with different sequence lengths over time. In this case, we can either pad our data or use ragged tensors (tensors with variable sequence lengths).</p>\n<p>I tried building a ragged model with Keras, but Keras only supports ragged tensors starting from version 2. Therefore, we need to update the package. While updating itself isn’t an issue, I encountered some bugs on Kaggle related to this.</p>\n<p>It’s not clear to me if PyTorch supports ragged tensors. I believe in PyTorch, they are referred to as jagged/nested tensors.</p>\n<p>The main question is: should we use ragged tensors, or is padding sufficient? I tested this offline, and ragged tensors gave me better results, but I’m not sure if the extra complexity in the NN structure is worth it.</p>",
  "messages": [
    {
      "id": "3080039",
      "postDate": "12/24/2024 14:36:34",
      "content": "<p>It’s possible to approach this problem as a sequence, but in reality, we have a variable number of symbol_ids. If we define our seq_len as either symbol_id or time_id, we will end up with different sequence lengths over time. In this case, we can either pad our data or use ragged tensors (tensors with variable sequence lengths).</p>\n<p>I tried building a ragged model with Keras, but Keras only supports ragged tensors starting from version 2. Therefore, we need to update the package. While updating itself isn’t an issue, I encountered some bugs on Kaggle related to this.</p>\n<p>It’s not clear to me if PyTorch supports ragged tensors. I believe in PyTorch, they are referred to as jagged/nested tensors.</p>\n<p>The main question is: should we use ragged tensors, or is padding sufficient? I tested this offline, and ragged tensors gave me better results, but I’m not sure if the extra complexity in the NN structure is worth it.</p>",
      "rawMarkdown": "It’s possible to approach this problem as a sequence, but in reality, we have a variable number of symbol_ids. If we define our seq_len as either symbol_id or time_id, we will end up with different sequence lengths over time. In this case, we can either pad our data or use ragged tensors (tensors with variable sequence lengths).\n\nI tried building a ragged model with Keras, but Keras only supports ragged tensors starting from version 2. Therefore, we need to update the package. While updating itself isn’t an issue, I encountered some bugs on Kaggle related to this.\n\nIt’s not clear to me if PyTorch supports ragged tensors. I believe in PyTorch, they are referred to as jagged/nested tensors.\n\nThe main question is: should we use ragged tensors, or is padding sufficient? I tested this offline, and ragged tensors gave me better results, but I’m not sure if the extra complexity in the NN structure is worth it.",
      "votes": null
    },
    {
      "id": "3080178",
      "postDate": "12/24/2024 20:17:16",
      "content": "<p>For me, padding with torch is sufficient. </p>",
      "rawMarkdown": "For me, padding with torch is sufficient.",
      "votes": null
    },
    {
      "id": "3080190",
      "postDate": "12/24/2024 20:41:51",
      "content": "<p>How are you padding? <br>\nBecause the seq_len is symbol_id * time_id, and symbol_id doesn't have a maximum size.</p>",
      "rawMarkdown": "How are you padding? \nBecause the seq_len is symbol_id * time_id, and symbol_id doesn't have a maximum size.",
      "votes": null
    },
    {
      "id": "3080195",
      "postDate": "12/24/2024 20:51:06",
      "content": "<p>I assume symbol_id does not exceed a certain big number.</p>",
      "rawMarkdown": "I assume symbol_id does not exceed a certain big number.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3080178,
      "author_name": "shiyili",
      "author_url": "",
      "post_date": "12/24/2024 20:17:16",
      "content": "<p>For me, padding with torch is sufficient. </p>",
      "votes": null,
      "replies": [
        {
          "id": 3080190,
          "author_name": "nandodmelo",
          "author_url": "",
          "post_date": "12/24/2024 20:41:51",
          "content": "<p>How are you padding? <br>\nBecause the seq_len is symbol_id * time_id, and symbol_id doesn't have a maximum size.</p>",
          "votes": null,
          "replies": [
            {
              "id": 3080195,
              "author_name": "shiyili",
              "author_url": "",
              "post_date": "12/24/2024 20:51:06",
              "content": "<p>I assume symbol_id does not exceed a certain big number.</p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "3080039": "It’s possible to approach this problem as a sequence, but in reality, we have a variable number of symbol_ids. If we define our seq_len as either symbol_id or time_id, we will end up with different sequence lengths over time. In this case, we can either pad our data or use ragged tensors (tensors with variable sequence lengths).\n\nI tried building a ragged model with Keras, but Keras only supports ragged tensors starting from version 2. Therefore, we need to update the package. While updating itself isn’t an issue, I encountered some bugs on Kaggle related to this.\n\nIt’s not clear to me if PyTorch supports ragged tensors. I believe in PyTorch, they are referred to as jagged/nested tensors.\n\nThe main question is: should we use ragged tensors, or is padding sufficient? I tested this offline, and ragged tensors gave me better results, but I’m not sure if the extra complexity in the NN structure is worth it.",
    "3080178": "For me, padding with torch is sufficient.",
    "3080190": "How are you padding? \nBecause the seq_len is symbol_id * time_id, and symbol_id doesn't have a maximum size.",
    "3080195": "I assume symbol_id does not exceed a certain big number."
  },
  "source": "meta"
}