{
  "id": 327965,
  "title": "Segmentation Fault 😱 with PyArrow & Dask",
  "url": "/competitions/amex-default-prediction/discussion/327965",
  "author_name": "Eric",
  "post_date": "2022-05-30T08:38:32.061000",
  "votes": 1,
  "comment_count": 0,
  "views": 0,
  "content": "<p>This may be very specific but I wish I had known this sooner. So just in case this could help someone save time. </p>\n<p>If you are <strong>not using a kaggle kernel</strong>, but <strong>run your own environment and pip / poetry</strong> to manage it, you may encounter a segmentation fault turning the csvs to parquet files using dask.</p>\n<p>A possible solution for this is to use conda instead (or a different engine than pyarrow, haven't tried). After setting up a <a href=\"https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-with-commands\" target=\"_blank\">conda environment</a> run</p>\n<pre><code>conda install -c conda-forge pyarrow\nconda install -c conda-forge dask[complete]\n</code></pre>\n<p>Then something like</p>\n<pre><code>import dask.dataframe as dd\nfrom pathlib import Path\n\ndef parse_func(source:Path, target:Path):\n    print(f\"reading from {source}\")\n    df = dd.read_csv(source, blocksize=\"100MB\", sep=\",\") \n\n    print(f\"writing to {target}\")\n    df.to_parquet(target)\n\npath = Path(\"./data\")\n\n# train_data\ncsv = path / \"train_data.csv\"\nparquet = path / \"train.parquet\"\nparse_func(csv, parquet)\n</code></pre>\n<p>Should work.</p>",
  "messages": [
    {
      "id": 1805495,
      "postDate": "2022-05-30T08:38:32.063Z",
      "content": "<p>This may be very specific but I wish I had known this sooner. So just in case this could help someone save time. </p>\n<p>If you are <strong>not using a kaggle kernel</strong>, but <strong>run your own environment and pip / poetry</strong> to manage it, you may encounter a segmentation fault turning the csvs to parquet files using dask.</p>\n<p>A possible solution for this is to use conda instead (or a different engine than pyarrow, haven't tried). After setting up a <a href=\"https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-with-commands\" target=\"_blank\">conda environment</a> run</p>\n<pre><code>conda install -c conda-forge pyarrow\nconda install -c conda-forge dask[complete]\n</code></pre>\n<p>Then something like</p>\n<pre><code>import dask.dataframe as dd\nfrom pathlib import Path\n\ndef parse_func(source:Path, target:Path):\n    print(f\"reading from {source}\")\n    df = dd.read_csv(source, blocksize=\"100MB\", sep=\",\") \n\n    print(f\"writing to {target}\")\n    df.to_parquet(target)\n\npath = Path(\"./data\")\n\n# train_data\ncsv = path / \"train_data.csv\"\nparquet = path / \"train.parquet\"\nparse_func(csv, parquet)\n</code></pre>\n<p>Should work.</p>",
      "rawMarkdown": "This may be very specific but I wish I had known this sooner. So just in case this could help someone save time. \n\nIf you are **not using a kaggle kernel**, but **run your own environment and pip / poetry** to manage it, you may encounter a segmentation fault turning the csvs to parquet files using dask.\n\nA possible solution for this is to use conda instead (or a different engine than pyarrow, haven't tried). After setting up a [conda environment](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-with-commands) run\n\n```shell\nconda install -c conda-forge pyarrow\nconda install -c conda-forge dask[complete]\n```\n\nThen something like\n```python\nimport dask.dataframe as dd\nfrom pathlib import Path\n\ndef parse_func(source:Path, target:Path):\n    print(f\"reading from {source}\")\n    df = dd.read_csv(source, blocksize=\"100MB\", sep=\",\") \n\n    print(f\"writing to {target}\")\n    df.to_parquet(target)\n\npath = Path(\"./data\")\n\n# train_data\ncsv = path / \"train_data.csv\"\nparquet = path / \"train.parquet\"\nparse_func(csv, parquet)\n```\n\nShould work.",
      "votes": 1
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "1805495": "This may be very specific but I wish I had known this sooner. So just in case this could help someone save time. \n\nIf you are **not using a kaggle kernel**, but **run your own environment and pip / poetry** to manage it, you may encounter a segmentation fault turning the csvs to parquet files using dask.\n\nA possible solution for this is to use conda instead (or a different engine than pyarrow, haven't tried). After setting up a [conda environment](https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#creating-an-environment-with-commands) run\n\n```shell\nconda install -c conda-forge pyarrow\nconda install -c conda-forge dask[complete]\n```\n\nThen something like\n```python\nimport dask.dataframe as dd\nfrom pathlib import Path\n\ndef parse_func(source:Path, target:Path):\n    print(f\"reading from {source}\")\n    df = dd.read_csv(source, blocksize=\"100MB\", sep=\",\") \n\n    print(f\"writing to {target}\")\n    df.to_parquet(target)\n\npath = Path(\"./data\")\n\n# train_data\ncsv = path / \"train_data.csv\"\nparquet = path / \"train.parquet\"\nparse_func(csv, parquet)\n```\n\nShould work."
  }
}