{
  "id": 141873,
  "title": "New video: what I've learned about TPU hardware",
  "url": "/competitions/flower-classification-with-tpus/discussion/141873",
  "author_name": "",
  "post_date": "2020-04-07T19:46:51.483606900Z",
  "votes": 35,
  "comment_count": 15,
  "views": 0,
  "content": "<p>Hi y'all! I'm Jesse Mostipak, and I'm a new Community Advocate here at Kaggle. I've spent my first six weeks on the job learning all about systolic arrays and bfloat16 number formats, and how these two components of TPUs help reduce deep learning model training times. I've created <a href=\"https://www.youtube.com/watch?v=JC84GCU7zqA\">a video on the topic</a>, and would love to hear more about what resonated with you, and what questions you might have about TPUs!</p>\n\n<p>Your forum discussions have also been a wonderful resource for me to learn more, and I'm excited to be a part of this learning community.</p>",
  "messages": [
    {
      "id": "800850",
      "postDate": "04/07/2020 19:46:51",
      "content": "<p>Hi y'all! I'm Jesse Mostipak, and I'm a new Community Advocate here at Kaggle. I've spent my first six weeks on the job learning all about systolic arrays and bfloat16 number formats, and how these two components of TPUs help reduce deep learning model training times. I've created <a href=\"https://www.youtube.com/watch?v=JC84GCU7zqA\">a video on the topic</a>, and would love to hear more about what resonated with you, and what questions you might have about TPUs!</p>\n\n<p>Your forum discussions have also been a wonderful resource for me to learn more, and I'm excited to be a part of this learning community.</p>",
      "rawMarkdown": "Hi y'all! I'm Jesse Mostipak, and I'm a new Community Advocate here at Kaggle. I've spent my first six weeks on the job learning all about systolic arrays and bfloat16 number formats, and how these two components of TPUs help reduce deep learning model training times. I've created [a video on the topic](https://www.youtube.com/watch?v=JC84GCU7zqA), and would love to hear more about what resonated with you, and what questions you might have about TPUs!\n\nYour forum discussions have also been a wonderful resource for me to learn more, and I'm excited to be a part of this learning community.",
      "votes": null
    },
    {
      "id": "800926",
      "postDate": "04/07/2020 22:12:27",
      "content": "<p>Welcome Jesse. I enjoyed watching your animated legos teach matrix multiplication. Nice video!</p>\n\n<p>For those that like numbers, here are example numbers to accompany the lego animation. Each number is one lego:</p>\n\n<pre><code>A = [[1, 2],      B = [[5, 6],\n     [3, 4]]           [7, 8]]\n</code></pre>\n\n<p>The resultant product <code>C = A * B</code> equals:</p>\n\n<pre><code>C = [[1*5 + 2*7,   1*6 + 2*8],\n     [3*5 + 4*7,   3*6 + 4*8]\n</code></pre>\n\n<p>In the video a transposed flipped version of <code>A</code> moves in from the left. Matrix <code>B</code> is on the right.</p>",
      "rawMarkdown": "Welcome Jesse. I enjoyed watching your animated legos teach matrix multiplication. Nice video!\n\nFor those that like numbers, here are example numbers to accompany the lego animation. Each number is one lego:\n\n    A = [[1, 2],      B = [[5, 6],\n         [3, 4]]           [7, 8]]\n\nThe resultant product `C = A * B` equals:\n\n    C = [[1*5 + 2*7,   1*6 + 2*8],\n         [3*5 + 4*7,   3*6 + 4*8]\n\nIn the video a transposed flipped version of `A` moves in from the left. Matrix `B` is on the right.",
      "votes": null
    },
    {
      "id": "801142",
      "postDate": "04/08/2020 05:41:24",
      "content": "<p>Welcome <a href=\"/jessemostipak\">@jessemostipak</a> Great video.  </p>",
      "rawMarkdown": "Welcome @jessemostipak Great video.",
      "votes": null
    },
    {
      "id": "801309",
      "postDate": "04/08/2020 10:39:16",
      "content": "<p>Nice</p>",
      "rawMarkdown": "Nice",
      "votes": null
    },
    {
      "id": "801518",
      "postDate": "04/08/2020 15:03:57",
      "content": "<p>Thank you for the warm welcome, Chris! And a huge thank you for creating this overview of the matrix - I really appreciate it 😊 </p>",
      "rawMarkdown": "Thank you for the warm welcome, Chris! And a huge thank you for creating this overview of the matrix - I really appreciate it 😊",
      "votes": null
    },
    {
      "id": "801540",
      "postDate": "04/08/2020 15:16:46",
      "content": "<p>Yup, LEGOs have always inspired me to build things. Greatly explained TPUs. I almost exhaust 30hrs!</p>",
      "rawMarkdown": "Yup, LEGOs have always inspired me to build things. Greatly explained TPUs. I almost exhaust 30hrs!",
      "votes": null
    },
    {
      "id": "801591",
      "postDate": "04/08/2020 15:54:05",
      "content": "<p>welcome Jesse</p>",
      "rawMarkdown": "welcome Jesse",
      "votes": null
    },
    {
      "id": "802068",
      "postDate": "04/09/2020 05:39:14",
      "content": "<p>great video</p>",
      "rawMarkdown": "great video",
      "votes": null
    },
    {
      "id": "804427",
      "postDate": "04/11/2020 15:16:15",
      "content": "<p>welcome</p>",
      "rawMarkdown": "welcome",
      "votes": null
    },
    {
      "id": "809440",
      "postDate": "04/16/2020 07:41:55",
      "content": "<p>Thanks for the video <a href=\"/jessemostipak\">@jessemostipak</a> , I have mentioned about this in my <a href=\"https://www.kaggle.com/kurianbenoy/introduction-kernel-what-s-efficientnet/\">Introduction Kaggle Notebook</a></p>",
      "rawMarkdown": "Thanks for the video @jessemostipak , I have mentioned about this in my [Introduction Kaggle Notebook](https://www.kaggle.com/kurianbenoy/introduction-kernel-what-s-efficientnet/)",
      "votes": null
    },
    {
      "id": "836298",
      "postDate": "05/06/2020 21:46:40",
      "content": "<p>can you please help me with this : <a href=\"https://www.kaggle.com/mobassir/pytorch-tpu-transfer-learning-baseline?scriptVersionId=33452080\"> Pytorch TPU Transfer Learning Baseline</a>\nyou can see there that 1 epoch of my tpu training takes 883.1570 seconds then why i can't train same model for 8 epochs?\nfor 8 epoch i get error(kernel doesn't finish commit within 3 hours tpu limit)\nwill you please help me solving this issue? <a href=\"/jessemostipak\">@jessemostipak</a> \n1 epoch takes 15 minutes then why i can't train that same model for 8 epoch? i lost my 3 hours tpu quota twice for that :(</p>",
      "rawMarkdown": "can you please help me with this : [ Pytorch TPU Transfer Learning Baseline](https://www.kaggle.com/mobassir/pytorch-tpu-transfer-learning-baseline?scriptVersionId=33452080)\nyou can see there that 1 epoch of my tpu training takes 883.1570 seconds then why i can't train same model for 8 epochs?\nfor 8 epoch i get error(kernel doesn't finish commit within 3 hours tpu limit)\nwill you please help me solving this issue? @jessemostipak \n1 epoch takes 15 minutes then why i can't train that same model for 8 epoch? i lost my 3 hours tpu quota twice for that :(",
      "votes": null
    },
    {
      "id": "842992",
      "postDate": "05/11/2020 18:35:51",
      "content": "<p>hi <a href=\"/mobassir\">@mobassir</a> - 8 epochs at 20 minutes each is pretty close to the TPU runtime 3 hour limit - it might be that you're doing something before (or after) training that then pushes you above the 3 hour limit. </p>\n\n<p>to troubleshoot this, drop your epochs down to 4 and see if that works. if it does, then try increasing to 6, then 7.</p>",
      "rawMarkdown": "hi @mobassir - 8 epochs at 20 minutes each is pretty close to the TPU runtime 3 hour limit - it might be that you're doing something before (or after) training that then pushes you above the 3 hour limit. \n\nto troubleshoot this, drop your epochs down to 4 and see if that works. if it does, then try increasing to 6, then 7.",
      "votes": null
    },
    {
      "id": "843009",
      "postDate": "05/11/2020 18:49:21",
      "content": "<p><a href=\"/jessemostipak\">@jessemostipak</a>  for 7 it works, actually validation loader was using 1 core and hence  it was slow,i just solved it and  thank you for your comment</p>",
      "rawMarkdown": "jessemostipak  for 7 it works, actually validation loader was using 1 core and hence  it was slow,i just solved it and  thank you for your comment",
      "votes": null
    },
    {
      "id": "843069",
      "postDate": "05/11/2020 19:59:31",
      "content": "<p>any time - I'm glad you got it to work! </p>",
      "rawMarkdown": "any time - I'm glad you got it to work!",
      "votes": null
    },
    {
      "id": "844352",
      "postDate": "05/12/2020 15:12:15",
      "content": "<p><a href=\"/jessemostipak\">@jessemostipak</a>  i again need your help,if you  check the latest version(version 22) of this kernel : <a href=\"https://www.kaggle.com/mobassir/pytorch-tpu-transfer-learning-baseline?scriptVersionId=33841130\">https://www.kaggle.com/mobassir/pytorch-tpu-transfer-learning-baseline?scriptVersionId=33841130</a>\nyou can see that training finished after 2 hours but with error,i can see code of version 22,see output but it says <strong>You are viewing the last run of the kernel, which had an error. Click here to see the last successful run.</strong></p>",
      "rawMarkdown": "jessemostipak  i again need your help,if you  check the latest version(version 22) of this kernel : https://www.kaggle.com/mobassir/pytorch-tpu-transfer-learning-baseline?scriptVersionId=33841130\nyou can see that training finished after 2 hours but with error,i can see code of version 22,see output but it says **You are viewing the last run of the kernel, which had an error. Click here to see the last successful run.**",
      "votes": null
    },
    {
      "id": "846390",
      "postDate": "05/13/2020 18:18:22",
      "content": "<p>I'm not really sure what's happening with your notebook, recommend asking over on Kaggle's Product Feedback forum. Our team monitors questions there regularly :)</p>",
      "rawMarkdown": "I'm not really sure what's happening with your notebook, recommend asking over on Kaggle's Product Feedback forum. Our team monitors questions there regularly :)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 800926,
      "author_name": "cdeotte",
      "author_url": "",
      "post_date": "04/07/2020 22:12:27",
      "content": "<p>Welcome Jesse. I enjoyed watching your animated legos teach matrix multiplication. Nice video!</p>\n\n<p>For those that like numbers, here are example numbers to accompany the lego animation. Each number is one lego:</p>\n\n<pre><code>A = [[1, 2],      B = [[5, 6],\n     [3, 4]]           [7, 8]]\n</code></pre>\n\n<p>The resultant product <code>C = A * B</code> equals:</p>\n\n<pre><code>C = [[1*5 + 2*7,   1*6 + 2*8],\n     [3*5 + 4*7,   3*6 + 4*8]\n</code></pre>\n\n<p>In the video a transposed flipped version of <code>A</code> moves in from the left. Matrix <code>B</code> is on the right.</p>",
      "votes": null,
      "replies": [
        {
          "id": 801518,
          "author_name": "jessemostipak",
          "author_url": "",
          "post_date": "04/08/2020 15:03:57",
          "content": "<p>Thank you for the warm welcome, Chris! And a huge thank you for creating this overview of the matrix - I really appreciate it 😊 </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 801540,
          "author_name": "parmarsuraj99",
          "author_url": "",
          "post_date": "04/08/2020 15:16:46",
          "content": "<p>Yup, LEGOs have always inspired me to build things. Greatly explained TPUs. I almost exhaust 30hrs!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 801142,
      "author_name": "anshumoudgil",
      "author_url": "",
      "post_date": "04/08/2020 05:41:24",
      "content": "<p>Welcome <a href=\"/jessemostipak\">@jessemostipak</a> Great video.  </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 801309,
      "author_name": "ashweenmankash",
      "author_url": "",
      "post_date": "04/08/2020 10:39:16",
      "content": "<p>Nice</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 801591,
      "author_name": "ogenesomto",
      "author_url": "",
      "post_date": "04/08/2020 15:54:05",
      "content": "<p>welcome Jesse</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 802068,
      "author_name": "piyushnegi97",
      "author_url": "",
      "post_date": "04/09/2020 05:39:14",
      "content": "<p>great video</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 804427,
      "author_name": "rakeshahg",
      "author_url": "",
      "post_date": "04/11/2020 15:16:15",
      "content": "<p>welcome</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 809440,
      "author_name": "kurianbenoy",
      "author_url": "",
      "post_date": "04/16/2020 07:41:55",
      "content": "<p>Thanks for the video <a href=\"/jessemostipak\">@jessemostipak</a> , I have mentioned about this in my <a href=\"https://www.kaggle.com/kurianbenoy/introduction-kernel-what-s-efficientnet/\">Introduction Kaggle Notebook</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 836298,
      "author_name": "mobassir",
      "author_url": "",
      "post_date": "05/06/2020 21:46:40",
      "content": "<p>can you please help me with this : <a href=\"https://www.kaggle.com/mobassir/pytorch-tpu-transfer-learning-baseline?scriptVersionId=33452080\"> Pytorch TPU Transfer Learning Baseline</a>\nyou can see there that 1 epoch of my tpu training takes 883.1570 seconds then why i can't train same model for 8 epochs?\nfor 8 epoch i get error(kernel doesn't finish commit within 3 hours tpu limit)\nwill you please help me solving this issue? <a href=\"/jessemostipak\">@jessemostipak</a> \n1 epoch takes 15 minutes then why i can't train that same model for 8 epoch? i lost my 3 hours tpu quota twice for that :(</p>",
      "votes": null,
      "replies": [
        {
          "id": 842992,
          "author_name": "jessemostipak",
          "author_url": "",
          "post_date": "05/11/2020 18:35:51",
          "content": "<p>hi <a href=\"/mobassir\">@mobassir</a> - 8 epochs at 20 minutes each is pretty close to the TPU runtime 3 hour limit - it might be that you're doing something before (or after) training that then pushes you above the 3 hour limit. </p>\n\n<p>to troubleshoot this, drop your epochs down to 4 and see if that works. if it does, then try increasing to 6, then 7.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 843009,
          "author_name": "mobassir",
          "author_url": "",
          "post_date": "05/11/2020 18:49:21",
          "content": "<p><a href=\"/jessemostipak\">@jessemostipak</a>  for 7 it works, actually validation loader was using 1 core and hence  it was slow,i just solved it and  thank you for your comment</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 843069,
          "author_name": "jessemostipak",
          "author_url": "",
          "post_date": "05/11/2020 19:59:31",
          "content": "<p>any time - I'm glad you got it to work! </p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 844352,
          "author_name": "mobassir",
          "author_url": "",
          "post_date": "05/12/2020 15:12:15",
          "content": "<p><a href=\"/jessemostipak\">@jessemostipak</a>  i again need your help,if you  check the latest version(version 22) of this kernel : <a href=\"https://www.kaggle.com/mobassir/pytorch-tpu-transfer-learning-baseline?scriptVersionId=33841130\">https://www.kaggle.com/mobassir/pytorch-tpu-transfer-learning-baseline?scriptVersionId=33841130</a>\nyou can see that training finished after 2 hours but with error,i can see code of version 22,see output but it says <strong>You are viewing the last run of the kernel, which had an error. Click here to see the last successful run.</strong></p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 846390,
          "author_name": "jessemostipak",
          "author_url": "",
          "post_date": "05/13/2020 18:18:22",
          "content": "<p>I'm not really sure what's happening with your notebook, recommend asking over on Kaggle's Product Feedback forum. Our team monitors questions there regularly :)</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "800850": "Hi y'all! I'm Jesse Mostipak, and I'm a new Community Advocate here at Kaggle. I've spent my first six weeks on the job learning all about systolic arrays and bfloat16 number formats, and how these two components of TPUs help reduce deep learning model training times. I've created [a video on the topic](https://www.youtube.com/watch?v=JC84GCU7zqA), and would love to hear more about what resonated with you, and what questions you might have about TPUs!\n\nYour forum discussions have also been a wonderful resource for me to learn more, and I'm excited to be a part of this learning community.",
    "800926": "Welcome Jesse. I enjoyed watching your animated legos teach matrix multiplication. Nice video!\n\nFor those that like numbers, here are example numbers to accompany the lego animation. Each number is one lego:\n\n    A = [[1, 2],      B = [[5, 6],\n         [3, 4]]           [7, 8]]\n\nThe resultant product `C = A * B` equals:\n\n    C = [[1*5 + 2*7,   1*6 + 2*8],\n         [3*5 + 4*7,   3*6 + 4*8]\n\nIn the video a transposed flipped version of `A` moves in from the left. Matrix `B` is on the right.",
    "801142": "Welcome @jessemostipak Great video.",
    "801309": "Nice",
    "801518": "Thank you for the warm welcome, Chris! And a huge thank you for creating this overview of the matrix - I really appreciate it 😊",
    "801540": "Yup, LEGOs have always inspired me to build things. Greatly explained TPUs. I almost exhaust 30hrs!",
    "801591": "welcome Jesse",
    "802068": "great video",
    "804427": "welcome",
    "809440": "Thanks for the video @jessemostipak , I have mentioned about this in my [Introduction Kaggle Notebook](https://www.kaggle.com/kurianbenoy/introduction-kernel-what-s-efficientnet/)",
    "836298": "can you please help me with this : [ Pytorch TPU Transfer Learning Baseline](https://www.kaggle.com/mobassir/pytorch-tpu-transfer-learning-baseline?scriptVersionId=33452080)\nyou can see there that 1 epoch of my tpu training takes 883.1570 seconds then why i can't train same model for 8 epochs?\nfor 8 epoch i get error(kernel doesn't finish commit within 3 hours tpu limit)\nwill you please help me solving this issue? @jessemostipak \n1 epoch takes 15 minutes then why i can't train that same model for 8 epoch? i lost my 3 hours tpu quota twice for that :(",
    "842992": "hi @mobassir - 8 epochs at 20 minutes each is pretty close to the TPU runtime 3 hour limit - it might be that you're doing something before (or after) training that then pushes you above the 3 hour limit. \n\nto troubleshoot this, drop your epochs down to 4 and see if that works. if it does, then try increasing to 6, then 7.",
    "843009": "jessemostipak  for 7 it works, actually validation loader was using 1 core and hence  it was slow,i just solved it and  thank you for your comment",
    "843069": "any time - I'm glad you got it to work!",
    "844352": "jessemostipak  i again need your help,if you  check the latest version(version 22) of this kernel : https://www.kaggle.com/mobassir/pytorch-tpu-transfer-learning-baseline?scriptVersionId=33841130\nyou can see that training finished after 2 hours but with error,i can see code of version 22,see output but it says **You are viewing the last run of the kernel, which had an error. Click here to see the last successful run.**",
    "846390": "I'm not really sure what's happening with your notebook, recommend asking over on Kaggle's Product Feedback forum. Our team monitors questions there regularly :)"
  },
  "source": "meta"
}