{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Peek\n\nThis notebook is here to just unify the dataset into one. I will perform further analysis and the Deep Learning algorithm in a future kernel. If you like this kernel, or forked this version, please upvote.\n\nFirst step, we peek at the data paths:"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        if filename.endswith('.jpg'):\n            break\n        print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since there's a lot of images included there, we only checked non-image files and got the three above. Next, we will load the sample submission and check."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"sample_sub = pd.read_csv('../input/herbarium-2020-fgvc7/sample_submission.csv')\ndisplay(sample_sub)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For the `*.json` files, we cannot load them to a DataFrame as there's two items that prevents this: `license` and `info`. So, I manually read the `*.json` files as follows:"},{"metadata":{"trusted":true},"cell_type":"code","source":"import json, codecs\nwith codecs.open(\"../input/herbarium-2020-fgvc7/nybg2020/train/metadata.json\", 'r',\n                 encoding='utf-8', errors='ignore') as f:\n    train_meta = json.load(f)\n    \nwith codecs.open(\"../input/herbarium-2020-fgvc7/nybg2020/test/metadata.json\", 'r',\n                 encoding='utf-8', errors='ignore') as f:\n    test_meta = json.load(f)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"display(train_meta.keys())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now, we will be unifying the metadata from the `*.json` files. We will first work with the `train` data."},{"metadata":{},"cell_type":"markdown","source":"First, we access the `annotations` list and convert it to a df."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = pd.DataFrame(train_meta['annotations'])\ndisplay(train_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Next is for `plant categories`:"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_cat = pd.DataFrame(train_meta['categories'])\ntrain_cat.columns = ['family', 'genus', 'category_id', 'categort_name']\ndisplay(train_cat)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Followed by the `image properties`:"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_img = pd.DataFrame(train_meta['images'])\ntrain_img.columns = ['file_name', 'height', 'image_id', 'license', 'width']\ndisplay(train_img)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And lastly, the `region`:"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_reg = pd.DataFrame(train_meta['regions'])\ntrain_reg.columns = ['region_id', 'region_name']\ndisplay(train_reg)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Then, we will merge all the DataFrames and see what we got:"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_df = train_df.merge(train_cat, on='category_id', how='outer')\ntrain_df = train_df.merge(train_img, on='image_id', how='outer')\ntrain_df = train_df.merge(train_reg, on='region_id', how='outer')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(train_df.info())\n\ndisplay(train_df)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looking closer, there's a line with `NaN` values there. We need to remove rows with `NaN`s so we proceed to the next line:"},{"metadata":{"trusted":true},"cell_type":"code","source":"na = train_df.file_name.isna()\nkeep = [x for x in range(train_df.shape[0]) if not na[x]]\ntrain_df = train_df.iloc[keep]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"After selecting the `non-NaN` items, we now reiterate on their file types. We need to save on memory, as we reached `102+ MB` for this DataFrame Only."},{"metadata":{"trusted":true},"cell_type":"code","source":"dtypes = ['int32', 'int32', 'int32', 'int32', 'object', 'object', 'object', 'object', 'int32', 'int32', 'int32', 'object']\nfor n, col in enumerate(train_df.columns):\n    train_df[col] = train_df[col].astype(dtypes[n])\nprint(train_df.info())\ndisplay(train_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Finally, for our `test` dataset. Since it only contains one key, `images`:"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_df = pd.DataFrame(test_meta['images'])\ntest_df.columns = ['file_name', 'height', 'image_id', 'license', 'width']\nprint(test_df.info())\ndisplay(test_df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Perfect!\n\nNow, we can go ahead and save this dataframe as a `*.csv` file for future use!"},{"metadata":{"trusted":true},"cell_type":"code","source":"#train_df.to_csv('full_train_data.csv', index=False)\n#test_df.to_csv('full_test_data.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Last Steps\n\nBefore we end this kernel, let's check the total number of targets for this dataset:"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(len(train_df.category_id.unique()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It's a shocking `32,093` unique targets! I can't think of how to approach this to simplify the data so for now, let's end it here!"},{"metadata":{},"cell_type":"markdown","source":"# Submission\n\nLet's create a submission file to check the format! I'll be using a random number generator for the targets!"},{"metadata":{"trusted":true},"cell_type":"code","source":"sub = pd.DataFrame()\nsub['Id'] = test_df.image_id\nsub['Predicted'] = list(map(int, np.random.randint(1, 32093, (test_df.shape[0]))))\ndisplay(sub)\nsub.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}