{
  "id": 164481,
  "title": "TPU Training not working",
  "url": "/competitions/siim-isic-melanoma-classification/discussion/164481",
  "author_name": "",
  "post_date": "2020-07-06T12:47:47.134512700Z",
  "votes": null,
  "comment_count": 8,
  "views": 0,
  "content": "<p>I'm trying to implement a solution using a TPU but the fit function isn't training beyond the first epoch. After the first epoch the notebook says that it is executing but the whole 3 hours get used and the second epoch doesn't start. Does anyone know what's going wrong?</p>\n\n<p>Link to my Code: <a href=\"https://www.kaggle.com/aryanpandey1109/melanoma-tpu-attempt\">https://www.kaggle.com/aryanpandey1109/melanoma-tpu-attempt</a></p>",
  "messages": [
    {
      "id": "917363",
      "postDate": "07/06/2020 12:47:47",
      "content": "<p>I'm trying to implement a solution using a TPU but the fit function isn't training beyond the first epoch. After the first epoch the notebook says that it is executing but the whole 3 hours get used and the second epoch doesn't start. Does anyone know what's going wrong?</p>\n\n<p>Link to my Code: <a href=\"https://www.kaggle.com/aryanpandey1109/melanoma-tpu-attempt\">https://www.kaggle.com/aryanpandey1109/melanoma-tpu-attempt</a></p>",
      "rawMarkdown": "I'm trying to implement a solution using a TPU but the fit function isn't training beyond the first epoch. After the first epoch the notebook says that it is executing but the whole 3 hours get used and the second epoch doesn't start. Does anyone know what's going wrong?\n\nLink to my Code: https://www.kaggle.com/aryanpandey1109/melanoma-tpu-attempt",
      "votes": null
    },
    {
      "id": "917399",
      "postDate": "07/06/2020 13:23:48",
      "content": "<p>Are you using one of the public kernels?\nPerhaps share your notebook so that others can provide feedback.</p>",
      "rawMarkdown": "Are you using one of the public kernels?\nPerhaps share your notebook so that others can provide feedback.",
      "votes": null
    },
    {
      "id": "917434",
      "postDate": "07/06/2020 13:59:18",
      "content": "<p>Something similar happened to me a couple of days and fixed it by setting the validation_steps=validation_size//batch_size</p>\n\n<p>Hope this helps.</p>",
      "rawMarkdown": "Something similar happened to me a couple of days and fixed it by setting the validation_steps=validation_size//batch_size\n\nHope this helps.",
      "votes": null
    },
    {
      "id": "917469",
      "postDate": "07/06/2020 14:35:56",
      "content": "<p><a href=\"https://www.kaggle.com/aryanpandey1109/melanoma-tpu-attempt\">https://www.kaggle.com/aryanpandey1109/melanoma-tpu-attempt</a></p>\n\n<p>Hi,\nThis is the code that I have been doing the training on\nAs you can see the training doesn't go beyond the first epoch</p>",
      "rawMarkdown": "https://www.kaggle.com/aryanpandey1109/melanoma-tpu-attempt\n\nHi,\nThis is the code that I have been doing the training on\nAs you can see the training doesn't go beyond the first epoch",
      "votes": null
    },
    {
      "id": "917471",
      "postDate": "07/06/2020 14:37:50",
      "content": "<p>Thank you for taking your time to reply to this thread\nUnfortunately I had already included that statement in my code</p>",
      "rawMarkdown": "Thank you for taking your time to reply to this thread\nUnfortunately I had already included that statement in my code",
      "votes": null
    },
    {
      "id": "917578",
      "postDate": "07/06/2020 16:06:22",
      "content": "<p>Hum, looking at your code actually the same thing is happening to me when using EfficientNetB7 with 512x512 images. If you change the images to a smaller size the model is going to run.</p>\n\n<p>I'm investigating the same issue, because if I try EfficientNetB6 with 512x512 it runs smoothly.</p>",
      "rawMarkdown": "Hum, looking at your code actually the same thing is happening to me when using EfficientNetB7 with 512x512 images. If you change the images to a smaller size the model is going to run.\n\nI'm investigating the same issue, because if I try EfficientNetB6 with 512x512 it runs smoothly.",
      "votes": null
    },
    {
      "id": "917889",
      "postDate": "07/06/2020 19:29:48",
      "content": "<p>Yeah it works with B6\nThank you so much</p>",
      "rawMarkdown": "Yeah it works with B6\nThank you so much",
      "votes": null
    },
    {
      "id": "918323",
      "postDate": "07/07/2020 07:13:28",
      "content": "<p>Seems like you got it worked now?</p>",
      "rawMarkdown": "Seems like you got it worked now?",
      "votes": null
    },
    {
      "id": "918414",
      "postDate": "07/07/2020 08:37:12",
      "content": "<p>It still doesn't work with EfficientnetB7 (Check Version 2 of my code). I am able to implement it smoothly with EfficientnetB6. Any help would be highly appreciated</p>",
      "rawMarkdown": "It still doesn't work with EfficientnetB7 (Check Version 2 of my code). I am able to implement it smoothly with EfficientnetB6. Any help would be highly appreciated",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 917399,
      "author_name": "sirishks",
      "author_url": "",
      "post_date": "07/06/2020 13:23:48",
      "content": "<p>Are you using one of the public kernels?\nPerhaps share your notebook so that others can provide feedback.</p>",
      "votes": null,
      "replies": [
        {
          "id": 917469,
          "author_name": "aryanpandey1109",
          "author_url": "",
          "post_date": "07/06/2020 14:35:56",
          "content": "<p><a href=\"https://www.kaggle.com/aryanpandey1109/melanoma-tpu-attempt\">https://www.kaggle.com/aryanpandey1109/melanoma-tpu-attempt</a></p>\n\n<p>Hi,\nThis is the code that I have been doing the training on\nAs you can see the training doesn't go beyond the first epoch</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 917434,
      "author_name": "santiviquez",
      "author_url": "",
      "post_date": "07/06/2020 13:59:18",
      "content": "<p>Something similar happened to me a couple of days and fixed it by setting the validation_steps=validation_size//batch_size</p>\n\n<p>Hope this helps.</p>",
      "votes": null,
      "replies": [
        {
          "id": 917471,
          "author_name": "aryanpandey1109",
          "author_url": "",
          "post_date": "07/06/2020 14:37:50",
          "content": "<p>Thank you for taking your time to reply to this thread\nUnfortunately I had already included that statement in my code</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 917578,
          "author_name": "santiviquez",
          "author_url": "",
          "post_date": "07/06/2020 16:06:22",
          "content": "<p>Hum, looking at your code actually the same thing is happening to me when using EfficientNetB7 with 512x512 images. If you change the images to a smaller size the model is going to run.</p>\n\n<p>I'm investigating the same issue, because if I try EfficientNetB6 with 512x512 it runs smoothly.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 917889,
          "author_name": "aryanpandey1109",
          "author_url": "",
          "post_date": "07/06/2020 19:29:48",
          "content": "<p>Yeah it works with B6\nThank you so much</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 918323,
      "author_name": "lpachuong",
      "author_url": "",
      "post_date": "07/07/2020 07:13:28",
      "content": "<p>Seems like you got it worked now?</p>",
      "votes": null,
      "replies": [
        {
          "id": 918414,
          "author_name": "aryanpandey1109",
          "author_url": "",
          "post_date": "07/07/2020 08:37:12",
          "content": "<p>It still doesn't work with EfficientnetB7 (Check Version 2 of my code). I am able to implement it smoothly with EfficientnetB6. Any help would be highly appreciated</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "917363": "I'm trying to implement a solution using a TPU but the fit function isn't training beyond the first epoch. After the first epoch the notebook says that it is executing but the whole 3 hours get used and the second epoch doesn't start. Does anyone know what's going wrong?\n\nLink to my Code: https://www.kaggle.com/aryanpandey1109/melanoma-tpu-attempt",
    "917399": "Are you using one of the public kernels?\nPerhaps share your notebook so that others can provide feedback.",
    "917434": "Something similar happened to me a couple of days and fixed it by setting the validation_steps=validation_size//batch_size\n\nHope this helps.",
    "917469": "https://www.kaggle.com/aryanpandey1109/melanoma-tpu-attempt\n\nHi,\nThis is the code that I have been doing the training on\nAs you can see the training doesn't go beyond the first epoch",
    "917471": "Thank you for taking your time to reply to this thread\nUnfortunately I had already included that statement in my code",
    "917578": "Hum, looking at your code actually the same thing is happening to me when using EfficientNetB7 with 512x512 images. If you change the images to a smaller size the model is going to run.\n\nI'm investigating the same issue, because if I try EfficientNetB6 with 512x512 it runs smoothly.",
    "917889": "Yeah it works with B6\nThank you so much",
    "918323": "Seems like you got it worked now?",
    "918414": "It still doesn't work with EfficientnetB7 (Check Version 2 of my code). I am able to implement it smoothly with EfficientnetB6. Any help would be highly appreciated"
  },
  "source": "meta"
}