{
  "id": 551085,
  "title": "Neural Net Performance",
  "url": "/competitions/jane-street-real-time-market-data-forecasting/discussion/551085",
  "author_name": "",
  "post_date": "2024-12-11T06:14:22.278961700Z",
  "votes": 2,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Has anyone been able to train a neural net to result in any positive R2 at all? I either get ones that collapse to predicting 0 (or near zero) for everything or ones that cant escape a negative R2.  I've tried everything from a simple Linear layer to a complex ensemble model with embedding spaces and attention heads and nothing seems to be working.</p>\n<p>Starting to wonder if I have a bug in my training harness……..</p>",
  "messages": [
    {
      "id": "3069162",
      "postDate": "12/11/2024 06:14:22",
      "content": "<p>Has anyone been able to train a neural net to result in any positive R2 at all? I either get ones that collapse to predicting 0 (or near zero) for everything or ones that cant escape a negative R2.  I've tried everything from a simple Linear layer to a complex ensemble model with embedding spaces and attention heads and nothing seems to be working.</p>\n<p>Starting to wonder if I have a bug in my training harness……..</p>",
      "rawMarkdown": "Has anyone been able to train a neural net to result in any positive R2 at all? I either get ones that collapse to predicting 0 (or near zero) for everything or ones that cant escape a negative R2.  I've tried everything from a simple Linear layer to a complex ensemble model with embedding spaces and attention heads and nothing seems to be working.\n\nStarting to wonder if I have a bug in my training harness........",
      "votes": null
    },
    {
      "id": "3069295",
      "postDate": "12/11/2024 09:42:02",
      "content": "<p>I think there is a bug in your setup. That was the case with me at least, I also had this issue, but don't anymore, after cleaning up the setup.</p>",
      "rawMarkdown": "I think there is a bug in your setup. That was the case with me at least, I also had this issue, but don't anymore, after cleaning up the setup.",
      "votes": null
    },
    {
      "id": "3069347",
      "postDate": "12/11/2024 11:20:04",
      "content": "<p>Shuffle the training set, and some noise to data before training. This started to give me +ve r2's</p>",
      "rawMarkdown": "Shuffle the training set, and some noise to data before training. This started to give me +ve r2's",
      "votes": null
    },
    {
      "id": "3069419",
      "postDate": "12/11/2024 13:21:43",
      "content": "<p>OK finally found an architecture that gives +ve R2. Had to remove dropout, regularization/normalization, remove a few layers, significantly reduce embedding dimensions, decrease batch size and 5x the training epochs.</p>\n<p>Now lets hope when I move this from MLX to Pytorch I get similar results </p>",
      "rawMarkdown": "OK finally found an architecture that gives +ve R2. Had to remove dropout, regularization/normalization, remove a few layers, significantly reduce embedding dimensions, decrease batch size and 5x the training epochs.\n\nNow lets hope when I move this from MLX to Pytorch I get similar results",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 3069295,
      "author_name": "reeeeeeeeeeeeeee",
      "author_url": "",
      "post_date": "12/11/2024 09:42:02",
      "content": "<p>I think there is a bug in your setup. That was the case with me at least, I also had this issue, but don't anymore, after cleaning up the setup.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3069347,
      "author_name": "zeroday19",
      "author_url": "",
      "post_date": "12/11/2024 11:20:04",
      "content": "<p>Shuffle the training set, and some noise to data before training. This started to give me +ve r2's</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 3069419,
      "author_name": "michaeltimbs",
      "author_url": "",
      "post_date": "12/11/2024 13:21:43",
      "content": "<p>OK finally found an architecture that gives +ve R2. Had to remove dropout, regularization/normalization, remove a few layers, significantly reduce embedding dimensions, decrease batch size and 5x the training epochs.</p>\n<p>Now lets hope when I move this from MLX to Pytorch I get similar results </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "3069162": "Has anyone been able to train a neural net to result in any positive R2 at all? I either get ones that collapse to predicting 0 (or near zero) for everything or ones that cant escape a negative R2.  I've tried everything from a simple Linear layer to a complex ensemble model with embedding spaces and attention heads and nothing seems to be working.\n\nStarting to wonder if I have a bug in my training harness........",
    "3069295": "I think there is a bug in your setup. That was the case with me at least, I also had this issue, but don't anymore, after cleaning up the setup.",
    "3069347": "Shuffle the training set, and some noise to data before training. This started to give me +ve r2's",
    "3069419": "OK finally found an architecture that gives +ve R2. Had to remove dropout, regularization/normalization, remove a few layers, significantly reduce embedding dimensions, decrease batch size and 5x the training epochs.\n\nNow lets hope when I move this from MLX to Pytorch I get similar results"
  },
  "source": "meta"
}