{
  "id": 271552,
  "title": "Colab TPU inference error?",
  "url": "/competitions/g2net-gravitational-wave-detection/discussion/271552",
  "author_name": "",
  "post_date": "2021-09-11T06:14:01.063507700Z",
  "votes": 6,
  "comment_count": 11,
  "views": 0,
  "content": "<p>I tried to infer using colab tpu, but some output is nan.<br>\nKaggle TPU doesn't happen that error using same codes.</p>\n<p>What is cause?<br>\nDoes anyone know?</p>",
  "messages": [
    {
      "id": "1509322",
      "postDate": "09/11/2021 06:14:01",
      "content": "<p>I tried to infer using colab tpu, but some output is nan.<br>\nKaggle TPU doesn't happen that error using same codes.</p>\n<p>What is cause?<br>\nDoes anyone know?</p>",
      "rawMarkdown": "I tried to infer using colab tpu, but some output is nan.\nKaggle TPU doesn't happen that error using same codes.\n\nWhat is cause?\nDoes anyone know?",
      "votes": null
    },
    {
      "id": "1509610",
      "postDate": "09/11/2021 13:09:45",
      "content": "<p>Maybe your GCS path is set incorrectly in Colab?</p>",
      "rawMarkdown": "Maybe your GCS path is set incorrectly in Colab?",
      "votes": null
    },
    {
      "id": "1509906",
      "postDate": "09/11/2021 19:14:55",
      "content": "<p>GCS path changes every week as far as I know. Did you change the paths ?</p>",
      "rawMarkdown": "GCS path changes every week as far as I know. Did you change the paths ?",
      "votes": null
    },
    {
      "id": "1510017",
      "postDate": "09/12/2021 00:11:06",
      "content": "<p>Thank you for comment.<br>\nI think GCS path looks right, because almost predicsion are not nan.</p>\n<p>However, prediction value other than nan is slightly different between Colab and Kaggle environment.</p>",
      "rawMarkdown": "Thank you for comment.\nI think GCS path looks right, because almost predicsion are not nan.\n\nHowever, prediction value other than nan is slightly different between Colab and Kaggle environment.",
      "votes": null
    },
    {
      "id": "1510022",
      "postDate": "09/12/2021 00:41:30",
      "content": "<p>Did you check the tensorflow version of Kaggle and Colab?</p>",
      "rawMarkdown": "Did you check the tensorflow version of Kaggle and Colab?",
      "votes": null
    },
    {
      "id": "1511056",
      "postDate": "09/13/2021 04:41:43",
      "content": "<p>Thank you. I forgot to check it.</p>\n<p>I tried to reinstall same version tensorflow using another model codes.<br>\nThe prediction result does not containt nan, but there are subtle difference beween colab tpu inference and kaggle tpu inference.</p>\n<p>I think this is from tpu.</p>",
      "rawMarkdown": "Thank you. I forgot to check it.\n\nI tried to reinstall same version tensorflow using another model codes.\nThe prediction result does not containt nan, but there are subtle difference beween colab tpu inference and kaggle tpu inference.\n\nI think this is from tpu.",
      "votes": null
    },
    {
      "id": "1511127",
      "postDate": "09/13/2021 06:34:57",
      "content": "<p>Do you get the exact same prediction when you run your Kaggle tpu inference code again?</p>\n<p>It may have changed. but when I was using it TensorFlow behavior was not deterministic, even on the same machine. </p>",
      "rawMarkdown": "Do you get the exact same prediction when you run your Kaggle tpu inference code again?\n\nIt may have changed. but when I was using it TensorFlow behavior was not deterministic, even on the same machine.",
      "votes": null
    },
    {
      "id": "1511322",
      "postDate": "09/13/2021 10:28:15",
      "content": "<p>Thanks!</p>\n<blockquote>\n  <p>It may have changed. but when I was using it TensorFlow behavior was not deterministic, even on the same machine.</p>\n</blockquote>\n<p>Do you mean tensorflow is not deterministic even if inferance?<br>\nMy code doesn't contain randomness, but any library may contain.<br>\nI don't know critical cause(tensorflow randomness? hardware problem?).<br>\nI will try </p>\n<p><code>tf.random.set_seed(0)</code></p>\n<p>ex) Below is one example of prediction result<br>\n0.16135386 and 0.16138226<br>\nI think this subtle difference is not critical, but I want to solve this problem for my studing.</p>",
      "rawMarkdown": "Thanks!\n\n> It may have changed. but when I was using it TensorFlow behavior was not deterministic, even on the same machine.\n\nDo you mean tensorflow is not deterministic even if inferance?\nMy code doesn't contain randomness, but any library may contain.\nI don't know critical cause(tensorflow randomness? hardware problem?).\nI will try \n\n`tf.random.set_seed(0)`\n\nex) Below is one example of prediction result\n0.16135386 and 0.16138226\nI think this subtle difference is not critical, but I want to solve this problem for my studing.",
      "votes": null
    },
    {
      "id": "1511365",
      "postDate": "09/13/2021 11:14:06",
      "content": "<p>Yes, TF is not deterministic.  Or at least it was not.  That's one of the reasons I switched to Pytorch.  It is not perfect either, but way more deterministic.</p>",
      "rawMarkdown": "Yes, TF is not deterministic.  Or at least it was not.  That's one of the reasons I switched to Pytorch.  It is not perfect either, but way more deterministic.",
      "votes": null
    },
    {
      "id": "1511391",
      "postDate": "09/13/2021 11:40:45",
      "content": "<p>I see.<br>\nYou helped me a lot! Thank you!</p>",
      "rawMarkdown": "I see.\nYou helped me a lot! Thank you!",
      "votes": null
    },
    {
      "id": "1519337",
      "postDate": "09/21/2021 14:37:28",
      "content": "<p>Not perfect, I fixed this error.</p>\n<p>I install same tensorflow version, then nan output disappeared.<br>\nThere are small difference with a tiny decimal point, but this is not critical.</p>",
      "rawMarkdown": "Not perfect, I fixed this error.\n\nI install same tensorflow version, then nan output disappeared.\nThere are small difference with a tiny decimal point, but this is not critical.",
      "votes": null
    },
    {
      "id": "1559728",
      "postDate": "10/27/2021 07:09:49",
      "content": "<p>Hey All,</p>\n<p>Thank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey <a href=\"https://forms.gle/QP9L16niPexozyhu5\" target=\"_blank\">https://forms.gle/QP9L16niPexozyhu5</a>.</p>\n<p>Thank you all,</p>\n<p>Regards,<br>\nChris</p>",
      "rawMarkdown": "Hey All,\n\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.\n\nThank you all,\n\nRegards,\nChris",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1509610,
      "author_name": "atamazian",
      "author_url": "",
      "post_date": "09/11/2021 13:09:45",
      "content": "<p>Maybe your GCS path is set incorrectly in Colab?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1509906,
      "author_name": "zarif98sjs",
      "author_url": "",
      "post_date": "09/11/2021 19:14:55",
      "content": "<p>GCS path changes every week as far as I know. Did you change the paths ?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1510017,
      "author_name": "yoshito",
      "author_url": "",
      "post_date": "09/12/2021 00:11:06",
      "content": "<p>Thank you for comment.<br>\nI think GCS path looks right, because almost predicsion are not nan.</p>\n<p>However, prediction value other than nan is slightly different between Colab and Kaggle environment.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1510022,
      "author_name": "wuliaokaola",
      "author_url": "",
      "post_date": "09/12/2021 00:41:30",
      "content": "<p>Did you check the tensorflow version of Kaggle and Colab?</p>",
      "votes": null,
      "replies": [
        {
          "id": 1511056,
          "author_name": "yoshito",
          "author_url": "",
          "post_date": "09/13/2021 04:41:43",
          "content": "<p>Thank you. I forgot to check it.</p>\n<p>I tried to reinstall same version tensorflow using another model codes.<br>\nThe prediction result does not containt nan, but there are subtle difference beween colab tpu inference and kaggle tpu inference.</p>\n<p>I think this is from tpu.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1511127,
          "author_name": "cpmpml",
          "author_url": "",
          "post_date": "09/13/2021 06:34:57",
          "content": "<p>Do you get the exact same prediction when you run your Kaggle tpu inference code again?</p>\n<p>It may have changed. but when I was using it TensorFlow behavior was not deterministic, even on the same machine. </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1511322,
          "author_name": "yoshito",
          "author_url": "",
          "post_date": "09/13/2021 10:28:15",
          "content": "<p>Thanks!</p>\n<blockquote>\n  <p>It may have changed. but when I was using it TensorFlow behavior was not deterministic, even on the same machine.</p>\n</blockquote>\n<p>Do you mean tensorflow is not deterministic even if inferance?<br>\nMy code doesn't contain randomness, but any library may contain.<br>\nI don't know critical cause(tensorflow randomness? hardware problem?).<br>\nI will try </p>\n<p><code>tf.random.set_seed(0)</code></p>\n<p>ex) Below is one example of prediction result<br>\n0.16135386 and 0.16138226<br>\nI think this subtle difference is not critical, but I want to solve this problem for my studing.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1511365,
          "author_name": "cpmpml",
          "author_url": "",
          "post_date": "09/13/2021 11:14:06",
          "content": "<p>Yes, TF is not deterministic.  Or at least it was not.  That's one of the reasons I switched to Pytorch.  It is not perfect either, but way more deterministic.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1511391,
          "author_name": "yoshito",
          "author_url": "",
          "post_date": "09/13/2021 11:40:45",
          "content": "<p>I see.<br>\nYou helped me a lot! Thank you!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1519337,
      "author_name": "yoshito",
      "author_url": "",
      "post_date": "09/21/2021 14:37:28",
      "content": "<p>Not perfect, I fixed this error.</p>\n<p>I install same tensorflow version, then nan output disappeared.<br>\nThere are small difference with a tiny decimal point, but this is not critical.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1559728,
      "author_name": "zerafachris",
      "author_url": "",
      "post_date": "10/27/2021 07:09:49",
      "content": "<p>Hey All,</p>\n<p>Thank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey <a href=\"https://forms.gle/QP9L16niPexozyhu5\" target=\"_blank\">https://forms.gle/QP9L16niPexozyhu5</a>.</p>\n<p>Thank you all,</p>\n<p>Regards,<br>\nChris</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1509322": "I tried to infer using colab tpu, but some output is nan.\nKaggle TPU doesn't happen that error using same codes.\n\nWhat is cause?\nDoes anyone know?",
    "1509610": "Maybe your GCS path is set incorrectly in Colab?",
    "1509906": "GCS path changes every week as far as I know. Did you change the paths ?",
    "1510017": "Thank you for comment.\nI think GCS path looks right, because almost predicsion are not nan.\n\nHowever, prediction value other than nan is slightly different between Colab and Kaggle environment.",
    "1510022": "Did you check the tensorflow version of Kaggle and Colab?",
    "1511056": "Thank you. I forgot to check it.\n\nI tried to reinstall same version tensorflow using another model codes.\nThe prediction result does not containt nan, but there are subtle difference beween colab tpu inference and kaggle tpu inference.\n\nI think this is from tpu.",
    "1511127": "Do you get the exact same prediction when you run your Kaggle tpu inference code again?\n\nIt may have changed. but when I was using it TensorFlow behavior was not deterministic, even on the same machine.",
    "1511322": "Thanks!\n\n> It may have changed. but when I was using it TensorFlow behavior was not deterministic, even on the same machine.\n\nDo you mean tensorflow is not deterministic even if inferance?\nMy code doesn't contain randomness, but any library may contain.\nI don't know critical cause(tensorflow randomness? hardware problem?).\nI will try \n\n`tf.random.set_seed(0)`\n\nex) Below is one example of prediction result\n0.16135386 and 0.16138226\nI think this subtle difference is not critical, but I want to solve this problem for my studing.",
    "1511365": "Yes, TF is not deterministic.  Or at least it was not.  That's one of the reasons I switched to Pytorch.  It is not perfect either, but way more deterministic.",
    "1511391": "I see.\nYou helped me a lot! Thank you!",
    "1519337": "Not perfect, I fixed this error.\n\nI install same tensorflow version, then nan output disappeared.\nThere are small difference with a tiny decimal point, but this is not critical.",
    "1559728": "Hey All,\n\nThank you all for taking part in our competition. The participation has been overwhelmingly positive. We are currently conducting a survey to gauge the demographic and outreach achieved. Kindly spare 2min and fill in this survey https://forms.gle/QP9L16niPexozyhu5.\n\nThank you all,\n\nRegards,\nChris"
  },
  "source": "meta"
}