{
  "id": 123895,
  "title": "Issues with Parquet files and R",
  "url": "/competitions/bengaliai-cv19/discussion/123895",
  "author_name": "",
  "post_date": "2019-12-31T09:19:46.357907300Z",
  "votes": 1,
  "comment_count": 7,
  "views": 0,
  "content": "<p>Hello,</p>\n\n<p>I can't open with R the Parquet files provided for this competition.</p>\n\n<p>If I load the arrow package I get an error about installing the arrow C++ library with install_arrow(). If I try, a new message about more instructions from Apache appears, but it's about compiling code from source files. A reinstall of the arrow package also fails since it's not found for the current version of R.</p>\n\n<p>I have also tried to use the reticulate package and importing Pandas, but when I call read_parquet() an error rises indicating that the engine pyarrow or fastparquet are not found and I have no idea how to fix this. Once again, maybe it's about some limitation with our access to the environment or system to make some changes.</p>\n\n<p>I think that some update or maintenance task is needed for R users. Can anyone help, please?</p>\n\n<p>Thank you in advance\nJesus</p>",
  "messages": [
    {
      "id": "707119",
      "postDate": "12/31/2019 09:19:46",
      "content": "<p>Hello,</p>\n\n<p>I can't open with R the Parquet files provided for this competition.</p>\n\n<p>If I load the arrow package I get an error about installing the arrow C++ library with install_arrow(). If I try, a new message about more instructions from Apache appears, but it's about compiling code from source files. A reinstall of the arrow package also fails since it's not found for the current version of R.</p>\n\n<p>I have also tried to use the reticulate package and importing Pandas, but when I call read_parquet() an error rises indicating that the engine pyarrow or fastparquet are not found and I have no idea how to fix this. Once again, maybe it's about some limitation with our access to the environment or system to make some changes.</p>\n\n<p>I think that some update or maintenance task is needed for R users. Can anyone help, please?</p>\n\n<p>Thank you in advance\nJesus</p>",
      "rawMarkdown": "Hello,\n\nI can't open with R the Parquet files provided for this competition.\n\nIf I load the arrow package I get an error about installing the arrow C++ library with install_arrow(). If I try, a new message about more instructions from Apache appears, but it's about compiling code from source files. A reinstall of the arrow package also fails since it's not found for the current version of R.\n\nI have also tried to use the reticulate package and importing Pandas, but when I call read_parquet() an error rises indicating that the engine pyarrow or fastparquet are not found and I have no idea how to fix this. Once again, maybe it's about some limitation with our access to the environment or system to make some changes.\n\nI think that some update or maintenance task is needed for R users. Can anyone help, please?\n\nThank you in advance\nJesus",
      "votes": null
    },
    {
      "id": "707608",
      "postDate": "01/01/2020 06:33:10",
      "content": "<p>Why don't you use the method from public kernels? Use pandas to load parquet </p>",
      "rawMarkdown": "Why don't you use the method from public kernels? Use pandas to load parquet",
      "votes": null
    },
    {
      "id": "707639",
      "postDate": "01/01/2020 07:53:09",
      "content": "<p>Hello <a href=\"/bibek777\">@bibek777</a>,</p>\n\n<p>That was one of my choices as I already commented. I tried the following:</p>\n\n<p><code>Sys.setenv(RETICULATE_PYTHON=\"/opt/conda/envs/py36/bin/python\", required=TRUE)</code></p>\n\n<p><code>library(reticulate)</code></p>\n\n<p><code>pandas &amp;lt;- import(\"pandas\")</code></p>\n\n<p><code>test_1 &amp;lt;- pandas$read_parquet(\"../input/bengaliai-cv19/test_image_data_1.parquet\")</code></p>\n\n<p>The last line returns an error asking for pyarrow or fastparquet engine. What am I missing?</p>\n\n<p>Thank you\nJesús</p>",
      "rawMarkdown": "Hello @bibek777,\n\nThat was one of my choices as I already commented. I tried the following:\n\n`Sys.setenv(RETICULATE_PYTHON=\"/opt/conda/envs/py36/bin/python\", required=TRUE)`\n\n`library(reticulate)`\n\n`pandas &lt;- import(\"pandas\")`\n\n`test_1 &lt;- pandas$read_parquet(\"../input/bengaliai-cv19/test_image_data_1.parquet\")`\n\nThe last line returns an error asking for pyarrow or fastparquet engine. What am I missing?\n\nThank you\nJesús",
      "votes": null
    },
    {
      "id": "707676",
      "postDate": "01/01/2020 09:30:50",
      "content": "<p>You need to install pyarrow as a dependency for python.</p>\n\n<p>The parquet files were written with pyarrow v 0.10.0 for compatibility with kaggle notebooks; you might not be able to read them with other versions of arrow/parquet.</p>",
      "rawMarkdown": "You need to install pyarrow as a dependency for python.\n\nThe parquet files were written with pyarrow v 0.10.0 for compatibility with kaggle notebooks; you might not be able to read them with other versions of arrow/parquet.",
      "votes": null
    },
    {
      "id": "707683",
      "postDate": "01/01/2020 09:43:08",
      "content": "<p>And how can I do that? Can we install such dependencies by ourselves?</p>",
      "rawMarkdown": "And how can I do that? Can we install such dependencies by ourselves?",
      "votes": null
    },
    {
      "id": "733827",
      "postDate": "01/31/2020 14:59:43",
      "content": "<p>Thank you for the discussion.\nI wrote <a href=\"https://www.kaggle.com/uriushi/an-introduction-to-bengali-by-visualization\">a note</a> to read parquet with arrow.\nAs a result, reinstalling the arrow will fix the problem. </p>\n\n<p><code>r\ninstall.packages(\"arrow\", repos = \"https://dl.bintray.com/ursalabs/arrow-r\")\n</code></p>\n\n<p>(But it's a temporary measure, so we hope you get official support.)</p>",
      "rawMarkdown": "Thank you for the discussion.\nI wrote [a note](https://www.kaggle.com/uriushi/an-introduction-to-bengali-by-visualization) to read parquet with arrow.\nAs a result, reinstalling the arrow will fix the problem. \n\n```r\ninstall.packages(\"arrow\", repos = \"https://dl.bintray.com/ursalabs/arrow-r\")\n```\n\n\n(But it's a temporary measure, so we hope you get official support.)",
      "votes": null
    },
    {
      "id": "734348",
      "postDate": "02/01/2020 10:24:51",
      "content": "<p>This is awesome <a href=\"/uriushi\">@uriushi</a>. I still have to try since I saw that the installation may take many minutes according to your notebook execution.</p>\n\n<p>I missed a month because of this issue. I hope to be in time to make more progress and submit some result. Thank you very much.</p>",
      "rawMarkdown": "This is awesome @uriushi. I still have to try since I saw that the installation may take many minutes according to your notebook execution.\n\nI missed a month because of this issue. I hope to be in time to make more progress and submit some result. Thank you very much.",
      "votes": null
    },
    {
      "id": "739300",
      "postDate": "02/07/2020 17:00:31",
      "content": "<p>Hello <a href=\"/imtiazprio\">@imtiazprio</a>, since you are a competition host I write to you again about this subject because I didn't get an official solution to my issue. I only got a temporary fix in this thread thanks to another user. This works by reinstalling a package from the internet but one of the requirements says \"No internet access enabled\".</p>\n\n<p>Do I still have any chance to work in my submission? How can R users participate in this competition? I opened an issue in the Kaggle's GitHub about this subject on 2 January, but I didn't get any answer yet.</p>\n\n<p>Thank you.</p>",
      "rawMarkdown": "Hello @imtiazprio, since you are a competition host I write to you again about this subject because I didn't get an official solution to my issue. I only got a temporary fix in this thread thanks to another user. This works by reinstalling a package from the internet but one of the requirements says \"No internet access enabled\".\n\nDo I still have any chance to work in my submission? How can R users participate in this competition? I opened an issue in the Kaggle's GitHub about this subject on 2 January, but I didn't get any answer yet.\n\nThank you.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 707608,
      "author_name": "bibek777",
      "author_url": "",
      "post_date": "01/01/2020 06:33:10",
      "content": "<p>Why don't you use the method from public kernels? Use pandas to load parquet </p>",
      "votes": null,
      "replies": [
        {
          "id": 707639,
          "author_name": "jmartindelasierra",
          "author_url": "",
          "post_date": "01/01/2020 07:53:09",
          "content": "<p>Hello <a href=\"/bibek777\">@bibek777</a>,</p>\n\n<p>That was one of my choices as I already commented. I tried the following:</p>\n\n<p><code>Sys.setenv(RETICULATE_PYTHON=\"/opt/conda/envs/py36/bin/python\", required=TRUE)</code></p>\n\n<p><code>library(reticulate)</code></p>\n\n<p><code>pandas &amp;lt;- import(\"pandas\")</code></p>\n\n<p><code>test_1 &amp;lt;- pandas$read_parquet(\"../input/bengaliai-cv19/test_image_data_1.parquet\")</code></p>\n\n<p>The last line returns an error asking for pyarrow or fastparquet engine. What am I missing?</p>\n\n<p>Thank you\nJesús</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 707676,
          "author_name": "imtiazprio",
          "author_url": "",
          "post_date": "01/01/2020 09:30:50",
          "content": "<p>You need to install pyarrow as a dependency for python.</p>\n\n<p>The parquet files were written with pyarrow v 0.10.0 for compatibility with kaggle notebooks; you might not be able to read them with other versions of arrow/parquet.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 707683,
          "author_name": "jmartindelasierra",
          "author_url": "",
          "post_date": "01/01/2020 09:43:08",
          "content": "<p>And how can I do that? Can we install such dependencies by ourselves?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 739300,
          "author_name": "jmartindelasierra",
          "author_url": "",
          "post_date": "02/07/2020 17:00:31",
          "content": "<p>Hello <a href=\"/imtiazprio\">@imtiazprio</a>, since you are a competition host I write to you again about this subject because I didn't get an official solution to my issue. I only got a temporary fix in this thread thanks to another user. This works by reinstalling a package from the internet but one of the requirements says \"No internet access enabled\".</p>\n\n<p>Do I still have any chance to work in my submission? How can R users participate in this competition? I opened an issue in the Kaggle's GitHub about this subject on 2 January, but I didn't get any answer yet.</p>\n\n<p>Thank you.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 733827,
      "author_name": "uriushi",
      "author_url": "",
      "post_date": "01/31/2020 14:59:43",
      "content": "<p>Thank you for the discussion.\nI wrote <a href=\"https://www.kaggle.com/uriushi/an-introduction-to-bengali-by-visualization\">a note</a> to read parquet with arrow.\nAs a result, reinstalling the arrow will fix the problem. </p>\n\n<p><code>r\ninstall.packages(\"arrow\", repos = \"https://dl.bintray.com/ursalabs/arrow-r\")\n</code></p>\n\n<p>(But it's a temporary measure, so we hope you get official support.)</p>",
      "votes": null,
      "replies": [
        {
          "id": 734348,
          "author_name": "jmartindelasierra",
          "author_url": "",
          "post_date": "02/01/2020 10:24:51",
          "content": "<p>This is awesome <a href=\"/uriushi\">@uriushi</a>. I still have to try since I saw that the installation may take many minutes according to your notebook execution.</p>\n\n<p>I missed a month because of this issue. I hope to be in time to make more progress and submit some result. Thank you very much.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "707119": "Hello,\n\nI can't open with R the Parquet files provided for this competition.\n\nIf I load the arrow package I get an error about installing the arrow C++ library with install_arrow(). If I try, a new message about more instructions from Apache appears, but it's about compiling code from source files. A reinstall of the arrow package also fails since it's not found for the current version of R.\n\nI have also tried to use the reticulate package and importing Pandas, but when I call read_parquet() an error rises indicating that the engine pyarrow or fastparquet are not found and I have no idea how to fix this. Once again, maybe it's about some limitation with our access to the environment or system to make some changes.\n\nI think that some update or maintenance task is needed for R users. Can anyone help, please?\n\nThank you in advance\nJesus",
    "707608": "Why don't you use the method from public kernels? Use pandas to load parquet",
    "707639": "Hello @bibek777,\n\nThat was one of my choices as I already commented. I tried the following:\n\n`Sys.setenv(RETICULATE_PYTHON=\"/opt/conda/envs/py36/bin/python\", required=TRUE)`\n\n`library(reticulate)`\n\n`pandas &lt;- import(\"pandas\")`\n\n`test_1 &lt;- pandas$read_parquet(\"../input/bengaliai-cv19/test_image_data_1.parquet\")`\n\nThe last line returns an error asking for pyarrow or fastparquet engine. What am I missing?\n\nThank you\nJesús",
    "707676": "You need to install pyarrow as a dependency for python.\n\nThe parquet files were written with pyarrow v 0.10.0 for compatibility with kaggle notebooks; you might not be able to read them with other versions of arrow/parquet.",
    "707683": "And how can I do that? Can we install such dependencies by ourselves?",
    "733827": "Thank you for the discussion.\nI wrote [a note](https://www.kaggle.com/uriushi/an-introduction-to-bengali-by-visualization) to read parquet with arrow.\nAs a result, reinstalling the arrow will fix the problem. \n\n```r\ninstall.packages(\"arrow\", repos = \"https://dl.bintray.com/ursalabs/arrow-r\")\n```\n\n\n(But it's a temporary measure, so we hope you get official support.)",
    "734348": "This is awesome @uriushi. I still have to try since I saw that the installation may take many minutes according to your notebook execution.\n\nI missed a month because of this issue. I hope to be in time to make more progress and submit some result. Thank you very much.",
    "739300": "Hello @imtiazprio, since you are a competition host I write to you again about this subject because I didn't get an official solution to my issue. I only got a temporary fix in this thread thanks to another user. This works by reinstalling a package from the internet but one of the requirements says \"No internet access enabled\".\n\nDo I still have any chance to work in my submission? How can R users participate in this competition? I opened an issue in the Kaggle's GitHub about this subject on 2 January, but I didn't get any answer yet.\n\nThank you."
  },
  "source": "meta"
}