{
  "id": 486558,
  "title": "Loading data into dictionary in inference of private LB scoring ",
  "url": "/competitions/hms-harmful-brain-activity-classification/discussion/486558",
  "author_name": "",
  "post_date": "2024-03-25T14:17:40.332933600Z",
  "votes": 2,
  "comment_count": 3,
  "views": 0,
  "content": "<p>This is my first notebook competition, and I wonder whether loading data (spec/eeg) into a dictionary could reach the memory limit in the private LB scoring. In my test, loading eeg data into a dictionary in the public LB scoring still takes a few minutes, and I don't know how much memory is consumed.   The private data is nearly twice as much as the public one.  Would it be safe to read the data on-the-fly inside the dataloader in a submission for scoring private LB?</p>",
  "messages": [
    {
      "id": "2715475",
      "postDate": "03/25/2024 14:17:40",
      "content": "<p>This is my first notebook competition, and I wonder whether loading data (spec/eeg) into a dictionary could reach the memory limit in the private LB scoring. In my test, loading eeg data into a dictionary in the public LB scoring still takes a few minutes, and I don't know how much memory is consumed.   The private data is nearly twice as much as the public one.  Would it be safe to read the data on-the-fly inside the dataloader in a submission for scoring private LB?</p>",
      "rawMarkdown": "This is my first notebook competition, and I wonder whether loading data (spec/eeg) into a dictionary could reach the memory limit in the private LB scoring. In my test, loading eeg data into a dictionary in the public LB scoring still takes a few minutes, and I don't know how much memory is consumed.   The private data is nearly twice as much as the public one.  Would it be safe to read the data on-the-fly inside the dataloader in a submission for scoring private LB?",
      "votes": null
    },
    {
      "id": "2715603",
      "postDate": "03/25/2024 15:25:48",
      "content": "<p>You can't distinguish public from private. Kaggle shows score for public. But your code processes all data. What I mean is that your code already allocated private data on memmory.</p>",
      "rawMarkdown": "You can't distinguish public from private. Kaggle shows score for public. But your code processes all data. What I mean is that your code already allocated private data on memmory.",
      "votes": null
    },
    {
      "id": "2715768",
      "postDate": "03/25/2024 17:12:27",
      "content": "<p>I see - thank you very much for the clarification.</p>",
      "rawMarkdown": "I see - thank you very much for the clarification.",
      "votes": null
    },
    {
      "id": "2715779",
      "postDate": "03/25/2024 17:24:18",
      "content": "<p>When in an interactive session, you can view usage metrics by clicking on <code>Draft Session</code>. You could use this to estimate your memory requirements for submission.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5570735%2F6fdd2c88f480493b145e43d884170336%2Fa.JPG?generation=1711387289272568&amp;alt=media\"></p>",
      "rawMarkdown": "When in an interactive session, you can view usage metrics by clicking on `Draft Session`. You could use this to estimate your memory requirements for submission.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5570735%2F6fdd2c88f480493b145e43d884170336%2Fa.JPG?generation=1711387289272568&alt=media)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2715603,
      "author_name": "sacuscreed",
      "author_url": "",
      "post_date": "03/25/2024 15:25:48",
      "content": "<p>You can't distinguish public from private. Kaggle shows score for public. But your code processes all data. What I mean is that your code already allocated private data on memmory.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2715768,
          "author_name": "makio323",
          "author_url": "",
          "post_date": "03/25/2024 17:12:27",
          "content": "<p>I see - thank you very much for the clarification.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 2715779,
      "author_name": "brendanartley",
      "author_url": "",
      "post_date": "03/25/2024 17:24:18",
      "content": "<p>When in an interactive session, you can view usage metrics by clicking on <code>Draft Session</code>. You could use this to estimate your memory requirements for submission.</p>\n<p><img src=\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5570735%2F6fdd2c88f480493b145e43d884170336%2Fa.JPG?generation=1711387289272568&amp;alt=media\"></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2715475": "This is my first notebook competition, and I wonder whether loading data (spec/eeg) into a dictionary could reach the memory limit in the private LB scoring. In my test, loading eeg data into a dictionary in the public LB scoring still takes a few minutes, and I don't know how much memory is consumed.   The private data is nearly twice as much as the public one.  Would it be safe to read the data on-the-fly inside the dataloader in a submission for scoring private LB?",
    "2715603": "You can't distinguish public from private. Kaggle shows score for public. But your code processes all data. What I mean is that your code already allocated private data on memmory.",
    "2715768": "I see - thank you very much for the clarification.",
    "2715779": "When in an interactive session, you can view usage metrics by clicking on `Draft Session`. You could use this to estimate your memory requirements for submission.\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F5570735%2F6fdd2c88f480493b145e43d884170336%2Fa.JPG?generation=1711387289272568&alt=media)"
  },
  "source": "meta"
}