{
  "id": 172252,
  "title": "filesystem access in tpu sessions seems slow since last night",
  "url": "/competitions/siim-isic-melanoma-classification/discussion/172252",
  "author_name": "Roman Weilguny",
  "post_date": "2020-08-04T09:24:30.210000",
  "votes": 7,
  "comment_count": 10,
  "views": 0,
  "content": "<p>calls like</p>\n\n<p><code>files_malignant = tf.io.gfile.glob(MALIGNANT)</code></p>\n\n<p>take much longer. Probably only in tpu sessions. </p>\n\n<p>Did anyone experience this too? Ideas?</p>\n\n<p>thx</p>",
  "messages": [
    {
      "id": 957405,
      "postDate": "2020-08-04T09:24:30.210Z",
      "content": "<p>calls like</p>\n\n<p><code>files_malignant = tf.io.gfile.glob(MALIGNANT)</code></p>\n\n<p>take much longer. Probably only in tpu sessions. </p>\n\n<p>Did anyone experience this too? Ideas?</p>\n\n<p>thx</p>",
      "rawMarkdown": "calls like\n\n`files_malignant = tf.io.gfile.glob(MALIGNANT)`\n\ntake much longer. Probably only in tpu sessions. \n\nDid anyone experience this too? Ideas?\n\nthx",
      "votes": 7
    },
    {
      "id": 959438,
      "postDate": "2020-08-05T15:27:41.400Z",
      "content": "<p><a href=\"/romanweilguny\">@romanweilguny</a> we have identified an issue with networking, and the fix should already be rolled out. </p>",
      "rawMarkdown": "@romanweilguny we have identified an issue with networking, and the fix should already be rolled out. ",
      "votes": 1
    },
    {
      "id": 957664,
      "postDate": "2020-08-04T13:31:07.487Z",
      "content": "<p>Yes i am also getting this now..\n<a href=\"/juliaelliott\">@juliaelliott</a> Could you please check on this issue.\nIt would be helpfull in the last stage of this competiton and the precious amounts of TPU time.</p>",
      "rawMarkdown": "Yes i am also getting this now..\n@juliaelliott Could you please check on this issue.\nIt would be helpfull in the last stage of this competiton and the precious amounts of TPU time.",
      "votes": 1
    },
    {
      "id": 957630,
      "postDate": "2020-08-04T13:00:35.897Z",
      "content": "<p>I can confirm this. <code>get_gcs_path()</code> and <code>tf.io.gfile.glob()</code> takes a lot longer than before. </p>",
      "rawMarkdown": "I can confirm this. `get_gcs_path()` and `tf.io.gfile.glob()` takes a lot longer than before. ",
      "votes": 2
    },
    {
      "id": 957594,
      "postDate": "2020-08-04T12:33:13.030Z",
      "content": "<p>Maybe this is the reason why my 3 commit sessions could not complete, of course giving me \"Time Limit exceeded error\", seems like I wasted about 9 hrs. of my quota due to this. :/</p>",
      "rawMarkdown": "Maybe this is the reason why my 3 commit sessions could not complete, of course giving me \"Time Limit exceeded error\", seems like I wasted about 9 hrs. of my quota due to this. :/",
      "votes": 2
    },
    {
      "id": 957409,
      "postDate": "2020-08-04T09:29:08.977Z",
      "content": "<p>I'm having some weird log messages which may be related: </p>\n\n<p>2020-08-04 09:05:: I tensorflow/core/platform/cloud/google_auth_provider.cc:180] Attempting an empty bearer token since no token was retrieved from files, and GCE metadata check was skipped.</p>\n\n<p>and running scripts keep disconnecting from my neptune experiment manager. </p>\n\n<p>Can't comment on if this is slowing things down.</p>",
      "rawMarkdown": "I'm having some weird log messages which may be related: \n\n2020-08-04 09:05:: I tensorflow/core/platform/cloud/google_auth_provider.cc:180] Attempting an empty bearer token since no token was retrieved from files, and GCE metadata check was skipped.\n\nand running scripts keep disconnecting from my neptune experiment manager. \n\nCan't comment on if this is slowing things down.",
      "votes": 2,
      "replies": [
        {
          "id": 957886,
          "postDate": "2020-08-04T16:02:13.373Z",
          "content": "<p><a href=\"/fchmiel\">@fchmiel</a> this is harmless info log message form Tensorflow indicating that you are accessing GCS anonymously. So, this by itself, shouldn't cause slowness.</p>",
          "rawMarkdown": "@fchmiel this is harmless info log message form Tensorflow indicating that you are accessing GCS anonymously. So, this by itself, shouldn't cause slowness.",
          "votes": 2
        },
        {
          "id": 957892,
          "postDate": "2020-08-04T16:06:03.017Z",
          "content": "<p>Regarding the slowness, I'm not aware of any changes to that code. So, it is it possible that GCS itself was experiencing something (though I don't see any issues reported <a href=\"https://status.cloud.google.com/\">here</a>). </p>",
          "rawMarkdown": "Regarding the slowness, I'm not aware of any changes to that code. So, it is it possible that GCS itself was experiencing something (though I don't see any issues reported [here](https://status.cloud.google.com/)). "
        }
      ]
    },
    {
      "id": 957672,
      "postDate": "2020-08-04T13:36:57.383Z",
      "content": "<p>also summon <a href=\"/herbison\">@herbison</a> </p>",
      "rawMarkdown": "also summon @herbison ",
      "replies": [
        {
          "id": 957690,
          "postDate": "2020-08-04T13:48:25.100Z",
          "content": "<p><a href=\"/ifigotin\">@ifigotin</a> Any idea on this? I'm not super familiar with expected timing for these calls.</p>",
          "rawMarkdown": "@ifigotin Any idea on this? I'm not super familiar with expected timing for these calls."
        }
      ]
    },
    {
      "id": 958122,
      "postDate": "2020-08-04T19:35:51.947Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 959438,
      "author_name": "Ilya Figotin",
      "author_url": "",
      "post_date": "2020-08-05T15:27:41.400000",
      "content": "<p><a href=\"/romanweilguny\">@romanweilguny</a> we have identified an issue with networking, and the fix should already be rolled out. </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 957664,
      "author_name": "MhdSharuk",
      "author_url": "",
      "post_date": "2020-08-04T13:31:07.487000",
      "content": "<p>Yes i am also getting this now..\n<a href=\"/juliaelliott\">@juliaelliott</a> Could you please check on this issue.\nIt would be helpfull in the last stage of this competiton and the precious amounts of TPU time.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 957630,
      "author_name": "Stanislav Blinov",
      "author_url": "",
      "post_date": "2020-08-04T13:00:35.897000",
      "content": "<p>I can confirm this. <code>get_gcs_path()</code> and <code>tf.io.gfile.glob()</code> takes a lot longer than before. </p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 957594,
      "author_name": "Gajendra Saraswat",
      "author_url": "",
      "post_date": "2020-08-04T12:33:13.030000",
      "content": "<p>Maybe this is the reason why my 3 commit sessions could not complete, of course giving me \"Time Limit exceeded error\", seems like I wasted about 9 hrs. of my quota due to this. :/</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 957409,
      "author_name": "FChmiel",
      "author_url": "",
      "post_date": "2020-08-04T09:29:08.977000",
      "content": "<p>I'm having some weird log messages which may be related: </p>\n\n<p>2020-08-04 09:05:: I tensorflow/core/platform/cloud/google_auth_provider.cc:180] Attempting an empty bearer token since no token was retrieved from files, and GCE metadata check was skipped.</p>\n\n<p>and running scripts keep disconnecting from my neptune experiment manager. </p>\n\n<p>Can't comment on if this is slowing things down.</p>",
      "votes": 2,
      "replies": [
        {
          "id": 957886,
          "author_name": "Ilya Figotin",
          "author_url": "",
          "post_date": "2020-08-04T16:02:13.373000",
          "content": "<p><a href=\"/fchmiel\">@fchmiel</a> this is harmless info log message form Tensorflow indicating that you are accessing GCS anonymously. So, this by itself, shouldn't cause slowness.</p>",
          "votes": 2,
          "replies": []
        },
        {
          "id": 957892,
          "author_name": "Ilya Figotin",
          "author_url": "",
          "post_date": "2020-08-04T16:06:03.017000",
          "content": "<p>Regarding the slowness, I'm not aware of any changes to that code. So, it is it possible that GCS itself was experiencing something (though I don't see any issues reported <a href=\"https://status.cloud.google.com/\">here</a>). </p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 957672,
      "author_name": "Stanislav Blinov",
      "author_url": "",
      "post_date": "2020-08-04T13:36:57.383000",
      "content": "<p>also summon <a href=\"/herbison\">@herbison</a> </p>",
      "votes": 0,
      "replies": [
        {
          "id": 957690,
          "author_name": "Dustin",
          "author_url": "",
          "post_date": "2020-08-04T13:48:25.100000",
          "content": "<p><a href=\"/ifigotin\">@ifigotin</a> Any idea on this? I'm not super familiar with expected timing for these calls.</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 958122,
      "author_name": "",
      "author_url": "",
      "post_date": "2020-08-04T19:35:51.947000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "957405": "calls like\n\n`files_malignant = tf.io.gfile.glob(MALIGNANT)`\n\ntake much longer. Probably only in tpu sessions. \n\nDid anyone experience this too? Ideas?\n\nthx",
    "959438": "@romanweilguny we have identified an issue with networking, and the fix should already be rolled out. ",
    "957664": "Yes i am also getting this now..\n@juliaelliott Could you please check on this issue.\nIt would be helpfull in the last stage of this competiton and the precious amounts of TPU time.",
    "957630": "I can confirm this. `get_gcs_path()` and `tf.io.gfile.glob()` takes a lot longer than before. ",
    "957594": "Maybe this is the reason why my 3 commit sessions could not complete, of course giving me \"Time Limit exceeded error\", seems like I wasted about 9 hrs. of my quota due to this. :/",
    "957409": "I'm having some weird log messages which may be related: \n\n2020-08-04 09:05:: I tensorflow/core/platform/cloud/google_auth_provider.cc:180] Attempting an empty bearer token since no token was retrieved from files, and GCE metadata check was skipped.\n\nand running scripts keep disconnecting from my neptune experiment manager. \n\nCan't comment on if this is slowing things down.",
    "957672": "also summon @herbison ",
    "958122": ""
  }
}