{
  "id": 356769,
  "title": "⚡⚡Loading compressed data faster with Feather ⚡⚡",
  "url": "/competitions/tabular-playground-series-oct-2022/discussion/356769",
  "author_name": "Ayush Bihani",
  "post_date": "2022-10-01T19:53:21.088000",
  "votes": 3,
  "comment_count": 0,
  "views": 0,
  "content": "<ul>\n<li>Feather and parquet file formats can compress the data quite well so I used that.</li>\n<li>Also, parquet files were having some issues with <code>float16</code> values so I removed those and instead used <code>float32</code>. </li>\n<li><strong>Feather</strong> achieved a compression of <strong>~77%</strong> and <strong>parquet</strong> of <strong>~57%</strong>.</li>\n<li>All the data can be found <a href=\"https://www.kaggle.com/code/hsuyab/fast-loading-high-compression-with-feather/data\" target=\"_blank\">here</a>.</li>\n<li>I have created a notebook discussing the walkthrough for creating the above dataset <a href=\"https://www.kaggle.com/code/hsuyab/fast-loading-high-compression-with-feather/data\" target=\"_blank\">here</a>.</li>\n<li>Other helpful resources are <a href=\"https://www.kaggle.com/competitions/tabular-playground-series-oct-2022/discussion/356540\" target=\"_blank\">1</a> and  <a href=\"https://www.kaggle.com/code/reymaster/compress-files-parquet-7x-loading-speedup\" target=\"_blank\">2</a>.</li>\n<li>Feel free to share other ideas as well.</li>\n</ul>",
  "messages": [
    {
      "id": 1966252,
      "postDate": "2022-10-01T19:53:21.090Z",
      "content": "<ul>\n<li>Feather and parquet file formats can compress the data quite well so I used that.</li>\n<li>Also, parquet files were having some issues with <code>float16</code> values so I removed those and instead used <code>float32</code>. </li>\n<li><strong>Feather</strong> achieved a compression of <strong>~77%</strong> and <strong>parquet</strong> of <strong>~57%</strong>.</li>\n<li>All the data can be found <a href=\"https://www.kaggle.com/code/hsuyab/fast-loading-high-compression-with-feather/data\" target=\"_blank\">here</a>.</li>\n<li>I have created a notebook discussing the walkthrough for creating the above dataset <a href=\"https://www.kaggle.com/code/hsuyab/fast-loading-high-compression-with-feather/data\" target=\"_blank\">here</a>.</li>\n<li>Other helpful resources are <a href=\"https://www.kaggle.com/competitions/tabular-playground-series-oct-2022/discussion/356540\" target=\"_blank\">1</a> and  <a href=\"https://www.kaggle.com/code/reymaster/compress-files-parquet-7x-loading-speedup\" target=\"_blank\">2</a>.</li>\n<li>Feel free to share other ideas as well.</li>\n</ul>",
      "rawMarkdown": "- Feather and parquet file formats can compress the data quite well so I used that.\n- Also, parquet files were having some issues with `float16` values so I removed those and instead used `float32`. \n- **Feather** achieved a compression of **~77%** and **parquet** of **~57%**.\n- All the data can be found [here](https://www.kaggle.com/code/hsuyab/fast-loading-high-compression-with-feather/data).\n- I have created a notebook discussing the walkthrough for creating the above dataset [here](https://www.kaggle.com/code/hsuyab/fast-loading-high-compression-with-feather/data).\n- Other helpful resources are [1](https://www.kaggle.com/competitions/tabular-playground-series-oct-2022/discussion/356540) and  [2](https://www.kaggle.com/code/reymaster/compress-files-parquet-7x-loading-speedup).\n- Feel free to share other ideas as well.",
      "votes": 3
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "1966252": "- Feather and parquet file formats can compress the data quite well so I used that.\n- Also, parquet files were having some issues with `float16` values so I removed those and instead used `float32`. \n- **Feather** achieved a compression of **~77%** and **parquet** of **~57%**.\n- All the data can be found [here](https://www.kaggle.com/code/hsuyab/fast-loading-high-compression-with-feather/data).\n- I have created a notebook discussing the walkthrough for creating the above dataset [here](https://www.kaggle.com/code/hsuyab/fast-loading-high-compression-with-feather/data).\n- Other helpful resources are [1](https://www.kaggle.com/competitions/tabular-playground-series-oct-2022/discussion/356540) and  [2](https://www.kaggle.com/code/reymaster/compress-files-parquet-7x-loading-speedup).\n- Feel free to share other ideas as well."
  }
}