{
  "id": 264504,
  "title": "Understand Train Dir Structure",
  "url": "/competitions/g2net-gravitational-wave-detection/discussion/264504",
  "author_name": "",
  "post_date": "2021-08-12T09:34:30.971564300Z",
  "votes": -1,
  "comment_count": 4,
  "views": 0,
  "content": "<p>I am trying to undestand data structure.<br>\nPlease help me understand why do we have so many directories. </p>\n<pre><code>a-f , \n  a-&gt;0-9 \n           0-&gt;0-9,a-&gt;e etc.\n</code></pre>\n<p>I believe we have labels for each leaf file it means they are independent then why we have directories.</p>\n<p>Each clip is collection of 3 source detectors but we have single label.</p>",
  "messages": [
    {
      "id": "1468169",
      "postDate": "08/12/2021 09:34:30",
      "content": "<p>I am trying to undestand data structure.<br>\nPlease help me understand why do we have so many directories. </p>\n<pre><code>a-f , \n  a-&gt;0-9 \n           0-&gt;0-9,a-&gt;e etc.\n</code></pre>\n<p>I believe we have labels for each leaf file it means they are independent then why we have directories.</p>\n<p>Each clip is collection of 3 source detectors but we have single label.</p>",
      "rawMarkdown": "I am trying to undestand data structure.\nPlease help me understand why do we have so many directories. \n```\na-f , \n  a->0-9 \n           0->0-9,a->e etc.\n```\nI believe we have labels for each leaf file it means they are independent then why we have directories.\n\nEach clip is collection of 3 source detectors but we have single label.",
      "votes": null
    },
    {
      "id": "1470891",
      "postDate": "08/13/2021 19:13:04",
      "content": "<p>I don't think there is any particular meaning behind the folders' structure.</p>\n<p>It seems that this structure was useful for those that have created the dataset, probably made by automated software.</p>\n<p>The only \"interesting\" insight (if we can call it that) is that the <strong>first 3 letters of each file name are the 3 nested folder</strong> names. So for instance the file <code>00000e74ad.npy</code> is located under the folders <strong>0/0/0</strong>. </p>\n<p>Using this information, you can write a function that reads the <code>training_labels.csv</code> file and finds where each .npy file is located.</p>",
      "rawMarkdown": "I don't think there is any particular meaning behind the folders' structure.\n\nIt seems that this structure was useful for those that have created the dataset, probably made by automated software.\n\nThe only \"interesting\" insight (if we can call it that) is that the **first 3 letters of each file name are the 3 nested folder** names. So for instance the file `00000e74ad.npy` is located under the folders **0/0/0**. \n\nUsing this information, you can write a function that reads the `training_labels.csv` file and finds where each .npy file is located.",
      "votes": null
    },
    {
      "id": "1471025",
      "postDate": "08/13/2021 21:39:29",
      "content": "<blockquote>\n  <p>Please help me understand why do we have so many directories. </p>\n</blockquote>\n<p>If you have 560000 files in one folder, then in some situations accessing that folder becomes very slow. By having nested subdirectories, which are encoded in filename itself - you avoid that potential headache, and it costs you just a single extra line of code.</p>",
      "rawMarkdown": "> Please help me understand why do we have so many directories. \n\nIf you have 560000 files in one folder, then in some situations accessing that folder becomes very slow. By having nested subdirectories, which are encoded in filename itself - you avoid that potential headache, and it costs you just a single extra line of code.",
      "votes": null
    },
    {
      "id": "1471531",
      "postDate": "08/14/2021 09:17:28",
      "content": "<p>That's also a very good reason (especially if you are using a GUI). For CLI, the number of files in a single folder shouldn't be an issue though.</p>",
      "rawMarkdown": "That's also a very good reason (especially if you are using a GUI). For CLI, the number of files in a single folder shouldn't be an issue though.",
      "votes": null
    },
    {
      "id": "1561211",
      "postDate": "10/27/2021 12:20:24",
      "content": "<p>Hey All,</p>\n<p>Thank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey <a href=\"https://forms.gle/QP9L16niPexozyhu5\" target=\"_blank\">https://forms.gle/QP9L16niPexozyhu5</a>.</p>\n<p>Thank you all,</p>\n<p>Regards,<br>\nChris</p>",
      "rawMarkdown": "Hey All,\n\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.\n\nThank you all,\n\nRegards,\nChris",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1470891,
      "author_name": "yassinealouini",
      "author_url": "",
      "post_date": "08/13/2021 19:13:04",
      "content": "<p>I don't think there is any particular meaning behind the folders' structure.</p>\n<p>It seems that this structure was useful for those that have created the dataset, probably made by automated software.</p>\n<p>The only \"interesting\" insight (if we can call it that) is that the <strong>first 3 letters of each file name are the 3 nested folder</strong> names. So for instance the file <code>00000e74ad.npy</code> is located under the folders <strong>0/0/0</strong>. </p>\n<p>Using this information, you can write a function that reads the <code>training_labels.csv</code> file and finds where each .npy file is located.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1471025,
      "author_name": "fffrrt",
      "author_url": "",
      "post_date": "08/13/2021 21:39:29",
      "content": "<blockquote>\n  <p>Please help me understand why do we have so many directories. </p>\n</blockquote>\n<p>If you have 560000 files in one folder, then in some situations accessing that folder becomes very slow. By having nested subdirectories, which are encoded in filename itself - you avoid that potential headache, and it costs you just a single extra line of code.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1471531,
          "author_name": "yassinealouini",
          "author_url": "",
          "post_date": "08/14/2021 09:17:28",
          "content": "<p>That's also a very good reason (especially if you are using a GUI). For CLI, the number of files in a single folder shouldn't be an issue though.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1561211,
      "author_name": "zerafachris",
      "author_url": "",
      "post_date": "10/27/2021 12:20:24",
      "content": "<p>Hey All,</p>\n<p>Thank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey <a href=\"https://forms.gle/QP9L16niPexozyhu5\" target=\"_blank\">https://forms.gle/QP9L16niPexozyhu5</a>.</p>\n<p>Thank you all,</p>\n<p>Regards,<br>\nChris</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1468169": "I am trying to undestand data structure.\nPlease help me understand why do we have so many directories. \n```\na-f , \n  a->0-9 \n           0->0-9,a->e etc.\n```\nI believe we have labels for each leaf file it means they are independent then why we have directories.\n\nEach clip is collection of 3 source detectors but we have single label.",
    "1470891": "I don't think there is any particular meaning behind the folders' structure.\n\nIt seems that this structure was useful for those that have created the dataset, probably made by automated software.\n\nThe only \"interesting\" insight (if we can call it that) is that the **first 3 letters of each file name are the 3 nested folder** names. So for instance the file `00000e74ad.npy` is located under the folders **0/0/0**. \n\nUsing this information, you can write a function that reads the `training_labels.csv` file and finds where each .npy file is located.",
    "1471025": "> Please help me understand why do we have so many directories. \n\nIf you have 560000 files in one folder, then in some situations accessing that folder becomes very slow. By having nested subdirectories, which are encoded in filename itself - you avoid that potential headache, and it costs you just a single extra line of code.",
    "1471531": "That's also a very good reason (especially if you are using a GUI). For CLI, the number of files in a single folder shouldn't be an issue though.",
    "1561211": "Hey All,\n\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.\n\nThank you all,\n\nRegards,\nChris"
  },
  "source": "meta"
}