{
  "id": 212508,
  "title": "Reducing training times using npy dataset",
  "url": "/competitions/cassava-leaf-disease-classification/discussion/212508",
  "author_name": "",
  "post_date": "2021-01-19T06:35:20.747468100Z",
  "votes": 2,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Hi,</p>\n<p>I wanted to reduce training time and so created a npy version of the original dataset - <a href=\"https://www.kaggle.com/suryajrrafl/cassava-npy-train-images\" target=\"_blank\">Cassava npy images dataset</a>. I used <a href=\"https://www.kaggle.com/pestipeti\" target=\"_blank\">@pestipeti</a>'s <a href=\"https://www.kaggle.com/pestipeti/cassava-pytorch-starter-train\" target=\"_blank\">wonderful notebook</a> for comparison (2-fold 1 epoch resnet18 baseline)</p>\n<p>These are the results:</p>\n<p><strong>cv2 2-fold training</strong></p>\n<ul>\n<li>fold0 - train_set(4:03 mins), val_set(2:01 mins)</li>\n<li>fold1 - train_set(3:47 mins), val_set(1:51 mins)</li>\n</ul>\n<p><strong>npy 2-fold training</strong></p>\n<ul>\n<li>fold0 - train_set(1:28 mins), val_set(1:55 mins)</li>\n<li>fold1 - train_set(1:05 mins), val_set(1:09 mins)</li>\n</ul>\n<p><a href=\"https://www.kaggle.com/suryajrrafl/cassava-cv2-npy-comparison?scriptVersionId=52199380\" target=\"_blank\">Notebook comparing cv2 vs npy images</a>. Any feedback on the method used and inference made is most welcome. </p>",
  "messages": [
    {
      "id": "1159265",
      "postDate": "01/19/2021 06:35:20",
      "content": "<p>Hi,</p>\n<p>I wanted to reduce training time and so created a npy version of the original dataset - <a href=\"https://www.kaggle.com/suryajrrafl/cassava-npy-train-images\" target=\"_blank\">Cassava npy images dataset</a>. I used <a href=\"https://www.kaggle.com/pestipeti\" target=\"_blank\">@pestipeti</a>'s <a href=\"https://www.kaggle.com/pestipeti/cassava-pytorch-starter-train\" target=\"_blank\">wonderful notebook</a> for comparison (2-fold 1 epoch resnet18 baseline)</p>\n<p>These are the results:</p>\n<p><strong>cv2 2-fold training</strong></p>\n<ul>\n<li>fold0 - train_set(4:03 mins), val_set(2:01 mins)</li>\n<li>fold1 - train_set(3:47 mins), val_set(1:51 mins)</li>\n</ul>\n<p><strong>npy 2-fold training</strong></p>\n<ul>\n<li>fold0 - train_set(1:28 mins), val_set(1:55 mins)</li>\n<li>fold1 - train_set(1:05 mins), val_set(1:09 mins)</li>\n</ul>\n<p><a href=\"https://www.kaggle.com/suryajrrafl/cassava-cv2-npy-comparison?scriptVersionId=52199380\" target=\"_blank\">Notebook comparing cv2 vs npy images</a>. Any feedback on the method used and inference made is most welcome. </p>",
      "rawMarkdown": "Hi,\n\nI wanted to reduce training time and so created a npy version of the original dataset - [Cassava npy images dataset](https://www.kaggle.com/suryajrrafl/cassava-npy-train-images). I used @pestipeti's [wonderful notebook](https://www.kaggle.com/pestipeti/cassava-pytorch-starter-train) for comparison (2-fold 1 epoch resnet18 baseline)\n\n\n\nThese are the results:\n\n**cv2 2-fold training**\n- fold0 - train_set(4:03 mins), val_set(2:01 mins)\n- fold1 - train_set(3:47 mins), val_set(1:51 mins)\n\n**npy 2-fold training**\n- fold0 - train_set(1:28 mins), val_set(1:55 mins)\n- fold1 - train_set(1:05 mins), val_set(1:09 mins)\n\n[Notebook comparing cv2 vs npy images](https://www.kaggle.com/suryajrrafl/cassava-cv2-npy-comparison?scriptVersionId=52199380). Any feedback on the method used and inference made is most welcome.",
      "votes": null
    },
    {
      "id": "1159488",
      "postDate": "01/19/2021 09:57:18",
      "content": "<p>Are the npy images in the dataset RGB or BGR?</p>",
      "rawMarkdown": "Are the npy images in the dataset RGB or BGR?",
      "votes": null
    },
    {
      "id": "1159520",
      "postDate": "01/19/2021 10:28:42",
      "content": "<p>RGB format. This is the code I used to convert to npy images</p>\n<pre><code>for idx in range(0, len(train_csv)):\n    image_src = f'{DIR_INPUT}/train_images/{train_csv.loc[idx, \"image_id\"]}'\n    image = cv2.imread(image_src, cv2.IMREAD_COLOR)\n    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n    np.save(f'{NPY_FOLDER}/{train_csv.loc[idx, \"npy_image_id\"]}', image)\n</code></pre>",
      "rawMarkdown": "RGB format. This is the code I used to convert to npy images\n\n```\nfor idx in range(0, len(train_csv)):\n    image_src = f'{DIR_INPUT}/train_images/{train_csv.loc[idx, \"image_id\"]}'\n    image = cv2.imread(image_src, cv2.IMREAD_COLOR)\n    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n    np.save(f'{NPY_FOLDER}/{train_csv.loc[idx, \"npy_image_id\"]}', image)\n```",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1159488,
      "author_name": "tmhrkt",
      "author_url": "",
      "post_date": "01/19/2021 09:57:18",
      "content": "<p>Are the npy images in the dataset RGB or BGR?</p>",
      "votes": null,
      "replies": [
        {
          "id": 1159520,
          "author_name": "suryajrrafl",
          "author_url": "",
          "post_date": "01/19/2021 10:28:42",
          "content": "<p>RGB format. This is the code I used to convert to npy images</p>\n<pre><code>for idx in range(0, len(train_csv)):\n    image_src = f'{DIR_INPUT}/train_images/{train_csv.loc[idx, \"image_id\"]}'\n    image = cv2.imread(image_src, cv2.IMREAD_COLOR)\n    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n    np.save(f'{NPY_FOLDER}/{train_csv.loc[idx, \"npy_image_id\"]}', image)\n</code></pre>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1159265": "Hi,\n\nI wanted to reduce training time and so created a npy version of the original dataset - [Cassava npy images dataset](https://www.kaggle.com/suryajrrafl/cassava-npy-train-images). I used @pestipeti's [wonderful notebook](https://www.kaggle.com/pestipeti/cassava-pytorch-starter-train) for comparison (2-fold 1 epoch resnet18 baseline)\n\n\n\nThese are the results:\n\n**cv2 2-fold training**\n- fold0 - train_set(4:03 mins), val_set(2:01 mins)\n- fold1 - train_set(3:47 mins), val_set(1:51 mins)\n\n**npy 2-fold training**\n- fold0 - train_set(1:28 mins), val_set(1:55 mins)\n- fold1 - train_set(1:05 mins), val_set(1:09 mins)\n\n[Notebook comparing cv2 vs npy images](https://www.kaggle.com/suryajrrafl/cassava-cv2-npy-comparison?scriptVersionId=52199380). Any feedback on the method used and inference made is most welcome.",
    "1159488": "Are the npy images in the dataset RGB or BGR?",
    "1159520": "RGB format. This is the code I used to convert to npy images\n\n```\nfor idx in range(0, len(train_csv)):\n    image_src = f'{DIR_INPUT}/train_images/{train_csv.loc[idx, \"image_id\"]}'\n    image = cv2.imread(image_src, cv2.IMREAD_COLOR)\n    image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n    np.save(f'{NPY_FOLDER}/{train_csv.loc[idx, \"npy_image_id\"]}', image)\n```"
  },
  "source": "meta"
}