{
  "id": 382773,
  "title": "pip install torch-sparse takes for ages, help",
  "url": "/competitions/icecube-neutrinos-in-deep-ice/discussion/382773",
  "author_name": "Luigi Stf",
  "post_date": "2023-02-01T00:19:16.147000",
  "votes": 3,
  "comment_count": 8,
  "views": 0,
  "content": "<p>Have you ever faced any super slow pip installs on Kaggle ? Installing torch dependencies as torch-sparse and torch-scatter takes around 20min on Kaggle in my notebook, against a 45 sec on Colab. Any ideas why it is so slow ? Any suggestions ?<br>\nA guy fed up with waiting for his first cell to complete ^^</p>",
  "messages": [
    {
      "id": 2124337,
      "postDate": "2023-02-01T00:19:16.147Z",
      "content": "<p>Have you ever faced any super slow pip installs on Kaggle ? Installing torch dependencies as torch-sparse and torch-scatter takes around 20min on Kaggle in my notebook, against a 45 sec on Colab. Any ideas why it is so slow ? Any suggestions ?<br>\nA guy fed up with waiting for his first cell to complete ^^</p>",
      "rawMarkdown": "Have you ever faced any super slow pip installs on Kaggle ? Installing torch dependencies as torch-sparse and torch-scatter takes around 20min on Kaggle in my notebook, against a 45 sec on Colab. Any ideas why it is so slow ? Any suggestions ?\nA guy fed up with waiting for his first cell to complete ^^",
      "votes": 3
    },
    {
      "id": 2191064,
      "postDate": "2023-03-21T17:08:00.387Z",
      "content": "<p>This only takes a few minutes in Kaggle if you're running with a GPU. I had a similar problem installing sparse and cluster on Colab. It was taking around 45 minutes and I assume it was to do with version incompatibility.</p>\n<pre><code>!pip install -U torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116\n!pip install pyg_lib torch_scatter torch_sparse torch_cluster torch_spline_conv torch_geometric -f https://data.pyg.org/whl/torch-+cu116.html\n</code></pre>",
      "rawMarkdown": "This only takes a few minutes in Kaggle if you're running with a GPU. I had a similar problem installing sparse and cluster on Colab. It was taking around 45 minutes and I assume it was to do with version incompatibility.\n\n```python\n!pip install -U torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116\n!pip install pyg_lib torch_scatter torch_sparse torch_cluster torch_spline_conv torch_geometric -f https://data.pyg.org/whl/torch-1.13.0+cu116.html\n```",
      "votes": 2,
      "replies": [
        {
          "id": 2191108,
          "postDate": "2023-03-21T17:51:41.760Z",
          "content": "<p>Note that the kernels are using Cuda 11.3</p>\n<pre><code>!nvcc --version\nnvcc: NVIDIA (R) Cuda compiler driver\nCopyright (c) 2005-2021 NVIDIA Corporation\nBuilt on Mon_May__3_19:15:13_PDT_2021\nCuda compilation tools, release 11.3, V11.3.109\nBuild cuda_11.3.r11.3/compiler.29920130_0\n</code></pre>",
          "rawMarkdown": "Note that the kernels are using Cuda 11.3\n\n```\n!nvcc --version\nnvcc: NVIDIA (R) Cuda compiler driver\nCopyright (c) 2005-2021 NVIDIA Corporation\nBuilt on Mon_May__3_19:15:13_PDT_2021\nCuda compilation tools, release 11.3, V11.3.109\nBuild cuda_11.3.r11.3/compiler.29920130_0\n```",
          "votes": 2,
          "replies": [
            {
              "id": 2209136,
              "postDate": "2023-04-04T13:42:22.493Z",
              "content": "<p>Thx, also helped me👍</p>",
              "rawMarkdown": "Thx, also helped me👍",
              "votes": 1
            }
          ]
        },
        {
          "id": 2191960,
          "postDate": "2023-03-22T10:36:27.033Z",
          "content": "<p>Hi <a href=\"https://www.kaggle.com/michaelbarrett\" target=\"_blank\">@michaelbarrett</a> , thank you for your reply. I will try it as well👍</p>",
          "rawMarkdown": "Hi @michaelbarrett , thank you for your reply. I will try it as well👍"
        }
      ]
    },
    {
      "id": 2190680,
      "postDate": "2023-03-21T12:18:52.907Z",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/louisstefanuto\" target=\"_blank\">@louisstefanuto</a> , have you found the solution for that slow installing torch dependencies? I am facing same problem now :(</p>",
      "rawMarkdown": "Hi @louisstefanuto , have you found the solution for that slow installing torch dependencies? I am facing same problem now :(\n",
      "replies": [
        {
          "id": 2190774,
          "postDate": "2023-03-21T13:26:51.420Z",
          "content": "<p>Here are the commands I used to download the packages. Later on the competition, you will have to use notebooks without internet, so you can check existing kernels for how they install packages offline (and it might be even better than using the snippets I provide here, but you may run into some version conflicts - I did hehe).</p>\n<pre><code># if on CPU\ncommand = 'torch==1.12.1+cpu torchvision==0.13.1+cpu torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cpu'\n!pip install --force-reinstall {command}\n\ncommand = 'pyg-lib torch-scatter torch-sparse torch-cluster torch-spline-conv torch-geometric -f https://data.pyg.org/whl/torch-1.12.0+cpu.html'\n!pip install {command}\n</code></pre>\n<pre><code># if on GPU\n\ncommand = 'torch==1.12.1+cu113 torchvision==0.13.1+cu113 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu113'\n!pip install --force-reinstall {command}\n\ncommand = 'pyg_lib torch_scatter torch_sparse torch_cluster torch_spline_conv torch_geometric -f https://data.pyg.org/whl/torch-1.12.0+cu113.html'\n!pip install {command}\n</code></pre>\n<p>It takes 3-6min to install.</p>",
          "rawMarkdown": "Here are the commands I used to download the packages. Later on the competition, you will have to use notebooks without internet, so you can check existing kernels for how they install packages offline (and it might be even better than using the snippets I provide here, but you may run into some version conflicts - I did hehe).\n\n```\n# if on CPU\ncommand = 'torch==1.12.1+cpu torchvision==0.13.1+cpu torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cpu'\n!pip install --force-reinstall {command}\n\ncommand = 'pyg-lib torch-scatter torch-sparse torch-cluster torch-spline-conv torch-geometric -f https://data.pyg.org/whl/torch-1.12.0+cpu.html'\n!pip install {command}\n```\n\n```\n# if on GPU\n\ncommand = 'torch==1.12.1+cu113 torchvision==0.13.1+cu113 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu113'\n!pip install --force-reinstall {command}\n\ncommand = 'pyg_lib torch_scatter torch_sparse torch_cluster torch_spline_conv torch_geometric -f https://data.pyg.org/whl/torch-1.12.0+cu113.html'\n!pip install {command}\n```\n\nIt takes 3-6min to install.",
          "votes": 1,
          "replies": [
            {
              "id": 2191956,
              "postDate": "2023-03-22T10:35:33.617Z",
              "content": "<p>Hello <a href=\"https://www.kaggle.com/araraonline\" target=\"_blank\">@araraonline</a>, thank you very much for your reply🙏</p>",
              "rawMarkdown": "Hello @araraonline, thank you very much for your reply🙏",
              "votes": 1
            }
          ]
        }
      ]
    },
    {
      "id": 2125111,
      "postDate": "2023-02-01T12:35:06.480Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 2191064,
      "author_name": "Michael Barrett",
      "author_url": "",
      "post_date": "2023-03-21T17:08:00.387000",
      "content": "<p>This only takes a few minutes in Kaggle if you're running with a GPU. I had a similar problem installing sparse and cluster on Colab. It was taking around 45 minutes and I assume it was to do with version incompatibility.</p>\n<pre><code>!pip install -U torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116\n!pip install pyg_lib torch_scatter torch_sparse torch_cluster torch_spline_conv torch_geometric -f https://data.pyg.org/whl/torch-+cu116.html\n</code></pre>",
      "votes": 2,
      "replies": [
        {
          "id": 2191108,
          "author_name": "lowerthansound",
          "author_url": "",
          "post_date": "2023-03-21T17:51:41.760000",
          "content": "<p>Note that the kernels are using Cuda 11.3</p>\n<pre><code>!nvcc --version\nnvcc: NVIDIA (R) Cuda compiler driver\nCopyright (c) 2005-2021 NVIDIA Corporation\nBuilt on Mon_May__3_19:15:13_PDT_2021\nCuda compilation tools, release 11.3, V11.3.109\nBuild cuda_11.3.r11.3/compiler.29920130_0\n</code></pre>",
          "votes": 2,
          "replies": [
            {
              "id": 2209136,
              "author_name": "DDD",
              "author_url": "",
              "post_date": "2023-04-04T13:42:22.493000",
              "content": "<p>Thx, also helped me👍</p>",
              "votes": 1,
              "replies": []
            }
          ]
        },
        {
          "id": 2191960,
          "author_name": "HechtJP",
          "author_url": "",
          "post_date": "2023-03-22T10:36:27.033000",
          "content": "<p>Hi <a href=\"https://www.kaggle.com/michaelbarrett\" target=\"_blank\">@michaelbarrett</a> , thank you for your reply. I will try it as well👍</p>",
          "votes": 0,
          "replies": []
        }
      ]
    },
    {
      "id": 2190680,
      "author_name": "HechtJP",
      "author_url": "",
      "post_date": "2023-03-21T12:18:52.907000",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/louisstefanuto\" target=\"_blank\">@louisstefanuto</a> , have you found the solution for that slow installing torch dependencies? I am facing same problem now :(</p>",
      "votes": 0,
      "replies": [
        {
          "id": 2190774,
          "author_name": "lowerthansound",
          "author_url": "",
          "post_date": "2023-03-21T13:26:51.420000",
          "content": "<p>Here are the commands I used to download the packages. Later on the competition, you will have to use notebooks without internet, so you can check existing kernels for how they install packages offline (and it might be even better than using the snippets I provide here, but you may run into some version conflicts - I did hehe).</p>\n<pre><code># if on CPU\ncommand = 'torch==1.12.1+cpu torchvision==0.13.1+cpu torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cpu'\n!pip install --force-reinstall {command}\n\ncommand = 'pyg-lib torch-scatter torch-sparse torch-cluster torch-spline-conv torch-geometric -f https://data.pyg.org/whl/torch-1.12.0+cpu.html'\n!pip install {command}\n</code></pre>\n<pre><code># if on GPU\n\ncommand = 'torch==1.12.1+cu113 torchvision==0.13.1+cu113 torchaudio==0.12.1 --extra-index-url https://download.pytorch.org/whl/cu113'\n!pip install --force-reinstall {command}\n\ncommand = 'pyg_lib torch_scatter torch_sparse torch_cluster torch_spline_conv torch_geometric -f https://data.pyg.org/whl/torch-1.12.0+cu113.html'\n!pip install {command}\n</code></pre>\n<p>It takes 3-6min to install.</p>",
          "votes": 1,
          "replies": [
            {
              "id": 2191956,
              "author_name": "HechtJP",
              "author_url": "",
              "post_date": "2023-03-22T10:35:33.617000",
              "content": "<p>Hello <a href=\"https://www.kaggle.com/araraonline\" target=\"_blank\">@araraonline</a>, thank you very much for your reply🙏</p>",
              "votes": 1,
              "replies": []
            }
          ]
        }
      ]
    },
    {
      "id": 2125111,
      "author_name": "",
      "author_url": "",
      "post_date": "2023-02-01T12:35:06.480000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2124337": "Have you ever faced any super slow pip installs on Kaggle ? Installing torch dependencies as torch-sparse and torch-scatter takes around 20min on Kaggle in my notebook, against a 45 sec on Colab. Any ideas why it is so slow ? Any suggestions ?\nA guy fed up with waiting for his first cell to complete ^^",
    "2191064": "This only takes a few minutes in Kaggle if you're running with a GPU. I had a similar problem installing sparse and cluster on Colab. It was taking around 45 minutes and I assume it was to do with version incompatibility.\n\n```python\n!pip install -U torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116\n!pip install pyg_lib torch_scatter torch_sparse torch_cluster torch_spline_conv torch_geometric -f https://data.pyg.org/whl/torch-1.13.0+cu116.html\n```",
    "2190680": "Hi @louisstefanuto , have you found the solution for that slow installing torch dependencies? I am facing same problem now :(\n",
    "2125111": ""
  }
}