{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Problem we want to address\n\nHello Kagglers! I hope everyone is busy with the competitions. This competition is very interesting (and a bit hard TBH). One of the things that I found while looking at the data is that the `TSV` files are huge. If you try to read the file in `pandas`, then it is highly likely that your kernel memory will blow up, and the kernel will eventually crash.\n\nSo, we will be doing something clever here to extract the relevant information from these TSVs, but without blowing up the memory. Let's start!","metadata":{}},{"cell_type":"markdown","source":"# Import the required libraries","metadata":{}},{"cell_type":"code","source":"import os\nimport gc\nimport cv2\nimport time\nimport requests\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nfrom pathlib import Path\nimport matplotlib.pyplot as plt\n\nimport dask\nfrom dask import delayed\nimport dask.dataframe as dd\n\nsns.set()\nseed = 1234\nnp.random.seed(seed)\n\n%config IPCompleter.use_jedi = False","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2021-09-14T15:42:31.710488Z","iopub.execute_input":"2021-09-14T15:42:31.710840Z","iopub.status.idle":"2021-09-14T15:42:31.729949Z","shell.execute_reply.started":"2021-09-14T15:42:31.710799Z","shell.execute_reply":"2021-09-14T15:42:31.728929Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Dataset","metadata":{}},{"cell_type":"code","source":"# Path to the directory where data is stored\ndata_path = Path(\"../input/wikipedia-image-caption/\")\n\n# Selecting only a subset of columns as there\n# many columns that we don't need.\ncolumns_to_select = [\"language\",\n                     \"page_url\",\n                     \"image_url\",\n                     \"caption_title_and_reference_description\"\n                    ]\n\n# Get the list of all tsv files we need to read for training\ntsvs = sorted(list(data_path.glob(\"*.tsv\")))\n\n# Remove test tsv file as we don't need it for now\ntsvs.remove(data_path / \"test.tsv\")\n\nprint(\"Number of TSV files found: \", len(tsvs))","metadata":{"execution":{"iopub.status.busy":"2021-09-14T15:21:51.088449Z","iopub.execute_input":"2021-09-14T15:21:51.089446Z","iopub.status.idle":"2021-09-14T15:21:51.097494Z","shell.execute_reply.started":"2021-09-14T15:21:51.089400Z","shell.execute_reply":"2021-09-14T15:21:51.096858Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Data Processing\n\nAs said earlier, we will be using Dask for reading the data. Why?\n1. Dask Dataframe can read data in chunks/partitions. When you read data with Dask, you will get a `Delayed` object. This makes it easier to read data that doesn't fit into RAM\n2. The files are tab-separated. So, we need to pass this info while reading the file. Bonus point: Dask Dataframe almost has the same API as pandas, so whatever parameter you pass in pandas, you can pass it here as well. There are some differences though because of the parallel stuff that Dask does. You can read about it in detail [here](https://docs.dask.org/en/latest/dataframe.html)\n3. I am selecting only a few columns that seem relevant to me but you can add/remove more columns if you want\n4. I will be dropping the null values as well. Again, just a choice. It's up to you how you want to deal with the missing data\n5. We will convert the `dask dataframe` to `pandas dataframe` afterward and save it in `feather` format for the future use case. Why feather? Because it is much faster to read from feather as compared to csv/tsv, and in most cases, it consumes less disk space as well","metadata":{}},{"cell_type":"code","source":"for tsv in tsvs:\n    # We will record the time taken to process\n    # each file to convert it into desired format\n    start_time = time.time()\n    \n    # 1. Name of the file to read\n    name = tsv.name\n    \n    # 2. Use Dask dataframe\n    df = dd.read_csv((data_path / name), \n                       sep=\"\\t\",\n                       quoting=3,\n                       escapechar=\"\\n\",\n                       usecols=columns_to_select,\n                       on_bad_lines=\"skip\",\n                       dtype=\"string\"\n                    )\n    # 3. Dropping the null values for now\n    df = df.dropna()\n    \n    # 4. Convert to pandas dataframe now\n    df = df.compute()\n    \n    # 5. Reset the index\n    df = df.reset_index(drop=True)\n    \n    # 6. Save to feather format for future use\n    df.to_feather(name.split(\".tsv\")[0])\n    \n    # 7. Del the dataframe and do garbage collection\n    del df\n    gc.collect()\n    \n    print(f\"File {name} processed and saved successfully in feather format\", end=\" =>\")\n    print(f\"Time taken: {time.time() - start_time:.2f} seconds\")\n    print(\"\")\n    print(\"=\"*50)","metadata":{"execution":{"iopub.status.busy":"2021-09-14T15:49:40.499890Z","iopub.execute_input":"2021-09-14T15:49:40.500169Z","iopub.status.idle":"2021-09-14T15:51:17.293780Z","shell.execute_reply.started":"2021-09-14T15:49:40.500141Z","shell.execute_reply":"2021-09-14T15:51:17.278841Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Sanity-check\n\nLet's load one of the processed files to see how much time and memory it takes while reading. Also, we will be looking at a few data entries as well","metadata":{}},{"cell_type":"code","source":"# File path\nfilepath = \"train-00000-of-00005\"\n\n# Read using pandas now\nstart_time = time.time()\ndf = pd.read_feather(filepath)\nprint(f\"Time taken to read {filepath} in feather format: {time.time()-start_time:.2f} seconds\")\nprint(\"\")\n\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2021-09-14T15:48:52.343917Z","iopub.execute_input":"2021-09-14T15:48:52.344189Z","iopub.status.idle":"2021-09-14T15:49:00.359603Z","shell.execute_reply.started":"2021-09-14T15:48:52.344161Z","shell.execute_reply":"2021-09-14T15:49:00.358716Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Hooray! So, we just have to read these processed files now without waiting for too long. Hope you enjoyed this simple kernel. Moaarrrr coming soon! ","metadata":{}}]}