{
  "id": 476550,
  "title": "Using dilated convolution decreases pytorch performance.",
  "url": "/competitions/hms-harmful-brain-activity-classification/discussion/476550",
  "author_name": "",
  "post_date": "2024-02-12T18:28:14.646405300Z",
  "votes": 10,
  "comment_count": 5,
  "views": 0,
  "content": "<p>I am trying to replicate wavenet notebook in torch. However one epoch takes 15 mins to train. I tried to debug it and found out that it's the dilated convolutions that are slowing down pytorch performance. Has anyone experienced the same?<br>\nI tried to change <code>torch.backends.cudnn.benchmark</code> to True/ False as indicated in many forums on google. However, it didn't help. Does anyone else experiences the same issue. <br>\nI checked the total parameters in both keras notebook by Chris and my implementation and both are equal.</p>",
  "messages": [
    {
      "id": "2649308",
      "postDate": "02/12/2024 18:28:14",
      "content": "<p>I am trying to replicate wavenet notebook in torch. However one epoch takes 15 mins to train. I tried to debug it and found out that it's the dilated convolutions that are slowing down pytorch performance. Has anyone experienced the same?<br>\nI tried to change <code>torch.backends.cudnn.benchmark</code> to True/ False as indicated in many forums on google. However, it didn't help. Does anyone else experiences the same issue. <br>\nI checked the total parameters in both keras notebook by Chris and my implementation and both are equal.</p>",
      "rawMarkdown": "I am trying to replicate wavenet notebook in torch. However one epoch takes 15 mins to train. I tried to debug it and found out that it's the dilated convolutions that are slowing down pytorch performance. Has anyone experienced the same?\nI tried to change `torch.backends.cudnn.benchmark` to True/ False as indicated in many forums on google. However, it didn't help. Does anyone else experiences the same issue. \nI checked the total parameters in both keras notebook by Chris and my implementation and both are equal.",
      "votes": null
    },
    {
      "id": "2649356",
      "postDate": "02/12/2024 19:21:13",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/chaudharypriyanshu\" target=\"_blank\">@chaudharypriyanshu</a>,</p>\n<p>Try to disable <code>deterministic</code> using,</p>\n<pre><code> = \n</code></pre>\n<p>I solve the runtime performance problem by this setting. Note that it'll affect the reproducibility of your experiment. Hope this helps.<br>\nBtw, I find it difficult to reproduce the performance (KL divergence) of WaveNet using torch. So far, I can only reach CV around 0.9 with exactly the same experimental setting as Chirs'.</p>",
      "rawMarkdown": "Hi @chaudharypriyanshu,\n\nTry to disable `deterministic` using,\n\n```\ntorch.backends.cudnn.deterministic = False\n```\n\nI solve the runtime performance problem by this setting. Note that it'll affect the reproducibility of your experiment. Hope this helps.\nBtw, I find it difficult to reproduce the performance (KL divergence) of WaveNet using torch. So far, I can only reach CV around 0.9 with exactly the same experimental setting as Chirs'.",
      "votes": null
    },
    {
      "id": "2649873",
      "postDate": "02/13/2024 05:31:51",
      "content": "<p>Do you think it is the issue due to <code>torch.backends.cudnn.deterministic = False</code>, have you tried running your code by keeping it true?</p>",
      "rawMarkdown": "Do you think it is the issue due to `torch.backends.cudnn.deterministic = False`, have you tried running your code by keeping it true?",
      "votes": null
    },
    {
      "id": "2650376",
      "postDate": "02/13/2024 12:28:23",
      "content": "<blockquote>\n  <p>Btw, I find it difficult to reproduce the performance (KL divergence) of WaveNet using torch. So far, I can only reach CV around 0.9 with exactly the same experimental setting as Chirs'.</p>\n</blockquote>\n<p>Interesting. Do you have any intuition that explains why ?</p>",
      "rawMarkdown": "> Btw, I find it difficult to reproduce the performance (KL divergence) of WaveNet using torch. So far, I can only reach CV around 0.9 with exactly the same experimental setting as Chirs'.\n\nInteresting. Do you have any intuition that explains why ?",
      "votes": null
    },
    {
      "id": "2650549",
      "postDate": "02/13/2024 14:22:28",
      "content": "<p>Try initialising the conv blocks with <code>nn.init.xavier_uniform_</code>. This discussion might be of good help: <a href=\"https://www.kaggle.com/competitions/liverpool-ion-switching/discussion/145256\" target=\"_blank\">discussion</a></p>",
      "rawMarkdown": "Try initialising the conv blocks with `nn.init.xavier_uniform_ `. This discussion might be of good help: [discussion](https://www.kaggle.com/competitions/liverpool-ion-switching/discussion/145256)",
      "votes": null
    },
    {
      "id": "2651124",
      "postDate": "02/14/2024 00:26:11",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/chaudharypriyanshu\" target=\"_blank\">@chaudharypriyanshu</a>,</p>\n<p>Thanks for your advice, I finally succeeded to reproduce the performance in torch by changing weight initialization.</p>",
      "rawMarkdown": "Hi @chaudharypriyanshu,\n\nThanks for your advice, I finally succeeded to reproduce the performance in torch by changing weight initialization.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2649356,
      "author_name": "abaojiang",
      "author_url": "",
      "post_date": "02/12/2024 19:21:13",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/chaudharypriyanshu\" target=\"_blank\">@chaudharypriyanshu</a>,</p>\n<p>Try to disable <code>deterministic</code> using,</p>\n<pre><code> = \n</code></pre>\n<p>I solve the runtime performance problem by this setting. Note that it'll affect the reproducibility of your experiment. Hope this helps.<br>\nBtw, I find it difficult to reproduce the performance (KL divergence) of WaveNet using torch. So far, I can only reach CV around 0.9 with exactly the same experimental setting as Chirs'.</p>",
      "votes": null,
      "replies": [
        {
          "id": 2649873,
          "author_name": "chaudharypriyanshu",
          "author_url": "",
          "post_date": "02/13/2024 05:31:51",
          "content": "<p>Do you think it is the issue due to <code>torch.backends.cudnn.deterministic = False</code>, have you tried running your code by keeping it true?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 2650376,
          "author_name": "serjhenrique",
          "author_url": "",
          "post_date": "02/13/2024 12:28:23",
          "content": "<blockquote>\n  <p>Btw, I find it difficult to reproduce the performance (KL divergence) of WaveNet using torch. So far, I can only reach CV around 0.9 with exactly the same experimental setting as Chirs'.</p>\n</blockquote>\n<p>Interesting. Do you have any intuition that explains why ?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 2650549,
          "author_name": "chaudharypriyanshu",
          "author_url": "",
          "post_date": "02/13/2024 14:22:28",
          "content": "<p>Try initialising the conv blocks with <code>nn.init.xavier_uniform_</code>. This discussion might be of good help: <a href=\"https://www.kaggle.com/competitions/liverpool-ion-switching/discussion/145256\" target=\"_blank\">discussion</a></p>",
          "votes": null,
          "replies": [
            {
              "id": 2651124,
              "author_name": "abaojiang",
              "author_url": "",
              "post_date": "02/14/2024 00:26:11",
              "content": "<p>Hi <a href=\"https://www.kaggle.com/chaudharypriyanshu\" target=\"_blank\">@chaudharypriyanshu</a>,</p>\n<p>Thanks for your advice, I finally succeeded to reproduce the performance in torch by changing weight initialization.</p>",
              "votes": null,
              "replies": []
            }
          ]
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "2649308": "I am trying to replicate wavenet notebook in torch. However one epoch takes 15 mins to train. I tried to debug it and found out that it's the dilated convolutions that are slowing down pytorch performance. Has anyone experienced the same?\nI tried to change `torch.backends.cudnn.benchmark` to True/ False as indicated in many forums on google. However, it didn't help. Does anyone else experiences the same issue. \nI checked the total parameters in both keras notebook by Chris and my implementation and both are equal.",
    "2649356": "Hi @chaudharypriyanshu,\n\nTry to disable `deterministic` using,\n\n```\ntorch.backends.cudnn.deterministic = False\n```\n\nI solve the runtime performance problem by this setting. Note that it'll affect the reproducibility of your experiment. Hope this helps.\nBtw, I find it difficult to reproduce the performance (KL divergence) of WaveNet using torch. So far, I can only reach CV around 0.9 with exactly the same experimental setting as Chirs'.",
    "2649873": "Do you think it is the issue due to `torch.backends.cudnn.deterministic = False`, have you tried running your code by keeping it true?",
    "2650376": "> Btw, I find it difficult to reproduce the performance (KL divergence) of WaveNet using torch. So far, I can only reach CV around 0.9 with exactly the same experimental setting as Chirs'.\n\nInteresting. Do you have any intuition that explains why ?",
    "2650549": "Try initialising the conv blocks with `nn.init.xavier_uniform_ `. This discussion might be of good help: [discussion](https://www.kaggle.com/competitions/liverpool-ion-switching/discussion/145256)",
    "2651124": "Hi @chaudharypriyanshu,\n\nThanks for your advice, I finally succeeded to reproduce the performance in torch by changing weight initialization."
  },
  "source": "meta"
}