{
  "id": 274883,
  "title": "Code out of memory on private dataset?",
  "url": "/competitions/landmark-retrieval-2021/discussion/274883",
  "author_name": "",
  "post_date": "2021-09-28T01:51:17.403427100Z",
  "votes": 1,
  "comment_count": 11,
  "views": 0,
  "content": "<p>I found baseline code ----- efficientnet influence fork</p>\n<p>When the picture is changed to 324, the calculation is exceeded. However, the private data set is twice that of the test data set. Will even 256 * 256 exceed the calculation on the private data set</p>",
  "messages": [
    {
      "id": "1526268",
      "postDate": "09/28/2021 01:51:17",
      "content": "<p>I found baseline code ----- efficientnet influence fork</p>\n<p>When the picture is changed to 324, the calculation is exceeded. However, the private data set is twice that of the test data set. Will even 256 * 256 exceed the calculation on the private data set</p>",
      "rawMarkdown": "I found baseline code ----- efficientnet influence fork\n\nWhen the picture is changed to 324, the calculation is exceeded. However, the private data set is twice that of the test data set. Will even 256 * 256 exceed the calculation on the private data set",
      "votes": null
    },
    {
      "id": "1526651",
      "postDate": "09/28/2021 09:07:15",
      "content": "<p>So as far as I understood you are trying to submit a public kernel and changing the image size from 256 to 324 right?</p>",
      "rawMarkdown": "So as far as I understood you are trying to submit a public kernel and changing the image size from 256 to 324 right?",
      "votes": null
    },
    {
      "id": "1526860",
      "postDate": "09/28/2021 11:40:59",
      "content": "<p>You understand very well. But my notebook is beyond the calculation.</p>",
      "rawMarkdown": "You understand very well. But my notebook is beyond the calculation.",
      "votes": null
    },
    {
      "id": "1526861",
      "postDate": "09/28/2021 11:41:06",
      "content": "<p>I think so, do you have any idea about solving this problem?</p>",
      "rawMarkdown": "I think so, do you have any idea about solving this problem?",
      "votes": null
    },
    {
      "id": "1526885",
      "postDate": "09/28/2021 11:54:06",
      "content": "<p>Change the n_chunks in the inference notebook<br>\nReducing it should solve your problem</p>",
      "rawMarkdown": "Change the n_chunks in the inference notebook\nReducing it should solve your problem",
      "votes": null
    },
    {
      "id": "1526932",
      "postDate": "09/28/2021 12:25:15",
      "content": "<p>Changing the n_chunks in that notebook to lower values should do the trick. </p>",
      "rawMarkdown": "Changing the n_chunks in that notebook to lower values should do the trick.",
      "votes": null
    },
    {
      "id": "1527040",
      "postDate": "09/28/2021 13:38:07",
      "content": "<p>n_chunks = len(image_paths) // chunk_size</p>\n<p>n_chunks  is determined by chunk_size. directly reducing n_chunks  will result in fewer pictures. Should I increase chunk_size and reduce n_chunks  ?</p>",
      "rawMarkdown": "n_chunks = len(image_paths) // chunk_size\n\nn_chunks  is determined by chunk_size. directly reducing n_chunks  will result in fewer pictures. Should I increase chunk_size and reduce n_chunks  ?",
      "votes": null
    },
    {
      "id": "1527042",
      "postDate": "09/28/2021 13:38:19",
      "content": "<p>n_chunks = len(image_paths) // chunk_size</p>\n<p>n_chunks is determined by chunk_size. directly reducing n_chunks will result in fewer pictures. Should I increase chunk_size and reduce n_chunks ?</p>",
      "rawMarkdown": "n_chunks = len(image_paths) // chunk_size\n\nn_chunks is determined by chunk_size. directly reducing n_chunks will result in fewer pictures. Should I increase chunk_size and reduce n_chunks ?",
      "votes": null
    },
    {
      "id": "1527056",
      "postDate": "09/28/2021 13:42:45",
      "content": "<p>Yeah, I meant the chunk_size<br>\nReduce the chunk_size</p>",
      "rawMarkdown": "Yeah, I meant the chunk_size\nReduce the chunk_size",
      "votes": null
    },
    {
      "id": "1527091",
      "postDate": "09/28/2021 14:10:45",
      "content": "<p>I changed a from 512 to 256, but it's still notebook exceeded allowed compute！</p>\n<p>image_paths = np.array(image_paths)<br>\n    chunk_size = 256</p>\n<pre><code>n_chunks = len(image_paths) // chunk_size\nif len(image_paths) % chunk_size != 0:\n    n_chunks += 1\n\nfor n in range(n_models):\n    print(f\"Getting Embedding for fold{n} model.\")\n    model = create_model_for_inference(f\"../input/glret21-eftb7-baseline-training/fold{n}.h5\")\n    for i in tqdm(range(n_chunks)):\n        files = image_paths[i * chunk_size:(i + 1) * chunk_size]\n        batch = create_batch(files)\n        embedding_tensor = model.predict(batch)\n        embeddings[i * chunk_size:(i + 1) * chunk_size] += embedding_tensor / n_models\n    del model\n    gc.collect()\n    tf.keras.backend.clear_session()**\n</code></pre>",
      "rawMarkdown": "I changed a from 512 to 256, but it's still notebook exceeded allowed compute！\n\n\n\n image_paths = np.array(image_paths)\n    chunk_size = 256\n    \n    n_chunks = len(image_paths) // chunk_size\n    if len(image_paths) % chunk_size != 0:\n        n_chunks += 1\n\n    for n in range(n_models):\n        print(f\"Getting Embedding for fold{n} model.\")\n        model = create_model_for_inference(f\"../input/glret21-eftb7-baseline-training/fold{n}.h5\")\n        for i in tqdm(range(n_chunks)):\n            files = image_paths[i * chunk_size:(i + 1) * chunk_size]\n            batch = create_batch(files)\n            embedding_tensor = model.predict(batch)\n            embeddings[i * chunk_size:(i + 1) * chunk_size] += embedding_tensor / n_models\n        del model\n        gc.collect()\n        tf.keras.backend.clear_session()**",
      "votes": null
    },
    {
      "id": "1527185",
      "postDate": "09/28/2021 15:04:17",
      "content": "<p>If the rest of the code is the same, only chunk_size can cause a memory issue<br>\nYou can maybe try 128 or 64 and see.<br>\nAlso since you are working with b7 which takes more memory than b4</p>",
      "rawMarkdown": "If the rest of the code is the same, only chunk_size can cause a memory issue\nYou can maybe try 128 or 64 and see.\nAlso since you are working with b7 which takes more memory than b4",
      "votes": null
    },
    {
      "id": "1527193",
      "postDate": "09/28/2021 15:12:18",
      "content": "<p>That's OK. Thank you very much! I'll try it right away</p>",
      "rawMarkdown": "That's OK. Thank you very much! I'll try it right away",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1526651,
      "author_name": "debarshichanda",
      "author_url": "",
      "post_date": "09/28/2021 09:07:15",
      "content": "<p>So as far as I understood you are trying to submit a public kernel and changing the image size from 256 to 324 right?</p>",
      "votes": null,
      "replies": [
        {
          "id": 1526860,
          "author_name": "henini",
          "author_url": "",
          "post_date": "09/28/2021 11:40:59",
          "content": "<p>You understand very well. But my notebook is beyond the calculation.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1526861,
          "author_name": "nin7a1",
          "author_url": "",
          "post_date": "09/28/2021 11:41:06",
          "content": "<p>I think so, do you have any idea about solving this problem?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1526885,
          "author_name": "debarshichanda",
          "author_url": "",
          "post_date": "09/28/2021 11:54:06",
          "content": "<p>Change the n_chunks in the inference notebook<br>\nReducing it should solve your problem</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1527040,
          "author_name": "henini",
          "author_url": "",
          "post_date": "09/28/2021 13:38:07",
          "content": "<p>n_chunks = len(image_paths) // chunk_size</p>\n<p>n_chunks  is determined by chunk_size. directly reducing n_chunks  will result in fewer pictures. Should I increase chunk_size and reduce n_chunks  ?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1527056,
          "author_name": "debarshichanda",
          "author_url": "",
          "post_date": "09/28/2021 13:42:45",
          "content": "<p>Yeah, I meant the chunk_size<br>\nReduce the chunk_size</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1527091,
          "author_name": "henini",
          "author_url": "",
          "post_date": "09/28/2021 14:10:45",
          "content": "<p>I changed a from 512 to 256, but it's still notebook exceeded allowed compute！</p>\n<p>image_paths = np.array(image_paths)<br>\n    chunk_size = 256</p>\n<pre><code>n_chunks = len(image_paths) // chunk_size\nif len(image_paths) % chunk_size != 0:\n    n_chunks += 1\n\nfor n in range(n_models):\n    print(f\"Getting Embedding for fold{n} model.\")\n    model = create_model_for_inference(f\"../input/glret21-eftb7-baseline-training/fold{n}.h5\")\n    for i in tqdm(range(n_chunks)):\n        files = image_paths[i * chunk_size:(i + 1) * chunk_size]\n        batch = create_batch(files)\n        embedding_tensor = model.predict(batch)\n        embeddings[i * chunk_size:(i + 1) * chunk_size] += embedding_tensor / n_models\n    del model\n    gc.collect()\n    tf.keras.backend.clear_session()**\n</code></pre>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1527185,
          "author_name": "debarshichanda",
          "author_url": "",
          "post_date": "09/28/2021 15:04:17",
          "content": "<p>If the rest of the code is the same, only chunk_size can cause a memory issue<br>\nYou can maybe try 128 or 64 and see.<br>\nAlso since you are working with b7 which takes more memory than b4</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1527193,
          "author_name": "henini",
          "author_url": "",
          "post_date": "09/28/2021 15:12:18",
          "content": "<p>That's OK. Thank you very much! I'll try it right away</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1526932,
      "author_name": "sandy1112",
      "author_url": "",
      "post_date": "09/28/2021 12:25:15",
      "content": "<p>Changing the n_chunks in that notebook to lower values should do the trick. </p>",
      "votes": null,
      "replies": [
        {
          "id": 1527042,
          "author_name": "henini",
          "author_url": "",
          "post_date": "09/28/2021 13:38:19",
          "content": "<p>n_chunks = len(image_paths) // chunk_size</p>\n<p>n_chunks is determined by chunk_size. directly reducing n_chunks will result in fewer pictures. Should I increase chunk_size and reduce n_chunks ?</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1526268": "I found baseline code ----- efficientnet influence fork\n\nWhen the picture is changed to 324, the calculation is exceeded. However, the private data set is twice that of the test data set. Will even 256 * 256 exceed the calculation on the private data set",
    "1526651": "So as far as I understood you are trying to submit a public kernel and changing the image size from 256 to 324 right?",
    "1526860": "You understand very well. But my notebook is beyond the calculation.",
    "1526861": "I think so, do you have any idea about solving this problem?",
    "1526885": "Change the n_chunks in the inference notebook\nReducing it should solve your problem",
    "1526932": "Changing the n_chunks in that notebook to lower values should do the trick.",
    "1527040": "n_chunks = len(image_paths) // chunk_size\n\nn_chunks  is determined by chunk_size. directly reducing n_chunks  will result in fewer pictures. Should I increase chunk_size and reduce n_chunks  ?",
    "1527042": "n_chunks = len(image_paths) // chunk_size\n\nn_chunks is determined by chunk_size. directly reducing n_chunks will result in fewer pictures. Should I increase chunk_size and reduce n_chunks ?",
    "1527056": "Yeah, I meant the chunk_size\nReduce the chunk_size",
    "1527091": "I changed a from 512 to 256, but it's still notebook exceeded allowed compute！\n\n\n\n image_paths = np.array(image_paths)\n    chunk_size = 256\n    \n    n_chunks = len(image_paths) // chunk_size\n    if len(image_paths) % chunk_size != 0:\n        n_chunks += 1\n\n    for n in range(n_models):\n        print(f\"Getting Embedding for fold{n} model.\")\n        model = create_model_for_inference(f\"../input/glret21-eftb7-baseline-training/fold{n}.h5\")\n        for i in tqdm(range(n_chunks)):\n            files = image_paths[i * chunk_size:(i + 1) * chunk_size]\n            batch = create_batch(files)\n            embedding_tensor = model.predict(batch)\n            embeddings[i * chunk_size:(i + 1) * chunk_size] += embedding_tensor / n_models\n        del model\n        gc.collect()\n        tf.keras.backend.clear_session()**",
    "1527185": "If the rest of the code is the same, only chunk_size can cause a memory issue\nYou can maybe try 128 or 64 and see.\nAlso since you are working with b7 which takes more memory than b4",
    "1527193": "That's OK. Thank you very much! I'll try it right away"
  },
  "source": "meta"
}