{
  "id": 316907,
  "title": "How do I test ideas without using this huge dataset?",
  "url": "/competitions/happy-whale-and-dolphin/discussion/316907",
  "author_name": "",
  "post_date": "2022-04-04T15:22:00.519986600Z",
  "votes": 10,
  "comment_count": 6,
  "views": 0,
  "content": "<p>The competition dataset is huge (62 GB), with 50K images, and my usual process of using multiple folds is severely limited because it takes me 1.5 days to check one idea or change. Is there a way of cutting down this data or a shortcut to training (besides using multiple GPU/TPU) that would allow me to do 5-10 folds and test out 2-3 ideas per day? Please let me know about your own experiences and solutions to big data as well.</p>",
  "messages": [
    {
      "id": "1745079",
      "postDate": "04/04/2022 15:22:00",
      "content": "<p>The competition dataset is huge (62 GB), with 50K images, and my usual process of using multiple folds is severely limited because it takes me 1.5 days to check one idea or change. Is there a way of cutting down this data or a shortcut to training (besides using multiple GPU/TPU) that would allow me to do 5-10 folds and test out 2-3 ideas per day? Please let me know about your own experiences and solutions to big data as well.</p>",
      "rawMarkdown": "The competition dataset is huge (62 GB), with 50K images, and my usual process of using multiple folds is severely limited because it takes me 1.5 days to check one idea or change. Is there a way of cutting down this data or a shortcut to training (besides using multiple GPU/TPU) that would allow me to do 5-10 folds and test out 2-3 ideas per day? Please let me know about your own experiences and solutions to big data as well.",
      "votes": null
    },
    {
      "id": "1745692",
      "postDate": "04/05/2022 07:31:06",
      "content": "<p>that is abc of deep learning.  simply with small sample.</p>",
      "rawMarkdown": "that is abc of deep learning.  simply with small sample.",
      "votes": null
    },
    {
      "id": "1745763",
      "postDate": "04/05/2022 08:35:52",
      "content": "<p>You can try the tiny version of your backbone to estimate and compare ideas</p>",
      "rawMarkdown": "You can try the tiny version of your backbone to estimate and compare ideas",
      "votes": null
    },
    {
      "id": "1745766",
      "postDate": "04/05/2022 08:38:50",
      "content": "<p>With a large dataset like this, I personally think using Kfold is very expensive (if you're using 10 folds, running 10 models to test an idea is a nightmare) I simply use 8:2 split or only test for single fold (after splitting to 5 or 10 folds)</p>",
      "rawMarkdown": "With a large dataset like this, I personally think using Kfold is very expensive (if you're using 10 folds, running 10 models to test an idea is a nightmare) I simply use 8:2 split or only test for single fold (after splitting to 5 or 10 folds)",
      "votes": null
    },
    {
      "id": "1746281",
      "postDate": "04/05/2022 16:15:22",
      "content": "<p>I just wanted to ask a similar question, but you beat me to it ! xD</p>",
      "rawMarkdown": "I just wanted to ask a similar question, but you beat me to it ! xD",
      "votes": null
    },
    {
      "id": "1746545",
      "postDate": "04/05/2022 22:57:25",
      "content": "<p>+1 start with a small sample and smaller number of classes and test your ideas there and then scale slowly. </p>",
      "rawMarkdown": "1 start with a small sample and smaller number of classes and test your ideas there and then scale slowly.",
      "votes": null
    },
    {
      "id": "1748637",
      "postDate": "04/07/2022 20:05:23",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/anandparthiban\" target=\"_blank\">@anandparthiban</a>, </p>\n<p>you can start modelling with smaller image sizes. This will speed up training. </p>\n<p><a href=\"https://www.kaggle.com/rdizzl3\" target=\"_blank\">@rdizzl3</a> rescaled the images and shared his links <a href=\"https://www.kaggle.com/competitions/happy-whale-and-dolphin/discussion/304686\" target=\"_blank\">here</a>.</p>",
      "rawMarkdown": "Hi @anandparthiban, \n\nyou can start modelling with smaller image sizes. This will speed up training. \n\n@rdizzl3 rescaled the images and shared his links [here](https://www.kaggle.com/competitions/happy-whale-and-dolphin/discussion/304686).",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1745692,
      "author_name": "dragonzhang",
      "author_url": "",
      "post_date": "04/05/2022 07:31:06",
      "content": "<p>that is abc of deep learning.  simply with small sample.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1745763,
      "author_name": "minhtu123",
      "author_url": "",
      "post_date": "04/05/2022 08:35:52",
      "content": "<p>You can try the tiny version of your backbone to estimate and compare ideas</p>",
      "votes": null,
      "replies": [
        {
          "id": 1746545,
          "author_name": "init27",
          "author_url": "",
          "post_date": "04/05/2022 22:57:25",
          "content": "<p>+1 start with a small sample and smaller number of classes and test your ideas there and then scale slowly. </p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1745766,
      "author_name": "ptran1203",
      "author_url": "",
      "post_date": "04/05/2022 08:38:50",
      "content": "<p>With a large dataset like this, I personally think using Kfold is very expensive (if you're using 10 folds, running 10 models to test an idea is a nightmare) I simply use 8:2 split or only test for single fold (after splitting to 5 or 10 folds)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1746281,
      "author_name": "dkurbatovv",
      "author_url": "",
      "post_date": "04/05/2022 16:15:22",
      "content": "<p>I just wanted to ask a similar question, but you beat me to it ! xD</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1748637,
      "author_name": "joatom",
      "author_url": "",
      "post_date": "04/07/2022 20:05:23",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/anandparthiban\" target=\"_blank\">@anandparthiban</a>, </p>\n<p>you can start modelling with smaller image sizes. This will speed up training. </p>\n<p><a href=\"https://www.kaggle.com/rdizzl3\" target=\"_blank\">@rdizzl3</a> rescaled the images and shared his links <a href=\"https://www.kaggle.com/competitions/happy-whale-and-dolphin/discussion/304686\" target=\"_blank\">here</a>.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1745079": "The competition dataset is huge (62 GB), with 50K images, and my usual process of using multiple folds is severely limited because it takes me 1.5 days to check one idea or change. Is there a way of cutting down this data or a shortcut to training (besides using multiple GPU/TPU) that would allow me to do 5-10 folds and test out 2-3 ideas per day? Please let me know about your own experiences and solutions to big data as well.",
    "1745692": "that is abc of deep learning.  simply with small sample.",
    "1745763": "You can try the tiny version of your backbone to estimate and compare ideas",
    "1745766": "With a large dataset like this, I personally think using Kfold is very expensive (if you're using 10 folds, running 10 models to test an idea is a nightmare) I simply use 8:2 split or only test for single fold (after splitting to 5 or 10 folds)",
    "1746281": "I just wanted to ask a similar question, but you beat me to it ! xD",
    "1746545": "1 start with a small sample and smaller number of classes and test your ideas there and then scale slowly.",
    "1748637": "Hi @anandparthiban, \n\nyou can start modelling with smaller image sizes. This will speed up training. \n\n@rdizzl3 rescaled the images and shared his links [here](https://www.kaggle.com/competitions/happy-whale-and-dolphin/discussion/304686)."
  },
  "source": "meta"
}