{"cells":[{"metadata":{},"cell_type":"markdown","source":"\nThis kernel is an attempt at understanding what we have. This is the first of the series of kernels. \n\n1. [Exploring csv files](#Exploring-CSV-files)\n    - [train.csv](#train.csv)\n    - [test.csv](#test.csv)\n    - [class_maps.csv](#class_maps.csv)\n    - [sample_submission.csv](#sample_submission.csv)\n2. [Exploring parquet files](#Exploring-parquet-files)\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport gc # garbage collector\n\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"project_dir = '/kaggle/input/bengaliai-cv19/'","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import glob\ncsv_files = [file for file in glob.glob(project_dir+\"*.csv\")]\ntrain_parquet_files =  [file for file in glob.glob(project_dir+\"train*.parquet\")]\ntest_parquet_files =  [file for file in glob.glob(project_dir+\"test*.parquet\")]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"---\n\n## Exploring CSV files\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"csv_files","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submission = '/kaggle/input/bengaliai-cv19/sample_submission.csv'\nclass_maps = '/kaggle/input/bengaliai-cv19/class_map.csv'\ntest = '/kaggle/input/bengaliai-cv19/test.csv'\ntrain = '/kaggle/input/bengaliai-cv19/train.csv'\n\n# For some strange reason, when I commit csv_files order is changing\n\n# sample_submission = csv_files[0]\n# class_maps = csv_files[1]\n# test = csv_files[2]\n# train = csv_files[3]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def csv_overview(csv_file, name='', head=3, tail=3, columns=False, describe=False, info=True):\n    print('file :', csv_file)\n    df = pd.read_csv(csv_file)\n    print('{} Shape : '.format(name),df.shape)\n    print('-'*36)\n    if columns:\n        print('{} Columns : '.format(name),df.columns)\n        print('-'*36)\n    if describe:\n        print('{} Distribution :\\n'.format(name),df.describe().T)\n        print('-'*36)\n    if info:\n        print('{} Summary :\\n'.format(name))\n#         print(df.info())\n        df.info()\n        print('-'*36)\n    print('{} Unique values :\\n'.format(name),df.nunique())\n    print('-'*36)\n    print('Sample data')\n    print('-'*12)\n    print('head')\n    print(df.head(head))\n    print('-'*12)\n    print('tail')\n    print(df.tail(tail))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### train.csv"},{"metadata":{"trusted":true},"cell_type":"code","source":"csv_overview(train, 'train_df', columns=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Overview :\nThere are total of 200840 (~200K images) and each image has \n- grapheme_root\n- vowel_diacritic\n- consonant_diacritic\n- and grapheme (which is combination of above three), is the bengali character in image\n\n> Task at hand is to identify the compoments which makeup the grapheme\n\n#### Summary :\n\nFrom train.csv, we can observe\n\n-  168 unique grapheme_root \n-  11 unique vowel_diacritic\n-  7 unique consonant_diacritic\n-  1295 unique graphemes \n\nImages are identified using image_id col(Train_image-no) which are likely to be foreign keys in paraquet files\n\n> There are doesn't seem to be any null values\n\n\n---"},{"metadata":{},"cell_type":"markdown","source":"### test.csv"},{"metadata":{"trusted":true},"cell_type":"code","source":"csv_overview(test, 'test_df', head=5, tail=5, columns=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Each images seems to have 3 dedicated rows to it each for different type of component.\n\nGiven test.csv only have test data for 12 images compared to train.csv with over ~200K images. These shall be used only for sample submission.\n\n> Goal would be to predict the component given its corresponding row_id and/or image_id\n\n---\n"},{"metadata":{},"cell_type":"markdown","source":"### class_maps.csv"},{"metadata":{"trusted":true},"cell_type":"code","source":"csv_overview(class_maps, 'Class Maps', columns=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Each component type (represented by a unique label) is mapped to the font/visualization\n\nwe have a total of 168+11+7 -> 186 components\n\n---"},{"metadata":{},"cell_type":"markdown","source":"### sample_submission.csv"},{"metadata":{"trusted":true},"cell_type":"code","source":"csv_overview(sample_submission, 'Sample Submissions', head=5, columns=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Submission should have only 2 columns, one for image and component identification and the other for its label\n\n---"},{"metadata":{},"cell_type":"markdown","source":"## Exploring parquet files"},{"metadata":{"trusted":true},"cell_type":"code","source":"def explore_parquet(file, name='', head=3, tail=3, columns=False, describe=False, unique=False, info=True):\n    print('file : {}'.format(file))\n    df = pd.read_parquet(file)\n    print('{} Shape : '.format(name),df.shape)\n    print('-'*36)\n    if columns:\n        print('{} Columns : '.format(name),df.columns)\n        print('-'*36)\n    if describe:\n        print('{} Distribution :\\n'.format(name),df.describe().T)\n        print('-'*36)\n    if info:\n        print('{} Summary :\\n'.format(name))\n#         print(df.info())\n        df.info()\n        print('-'*36)\n    if unique:\n        print('{} Unique values :\\n'.format(name),df.nunique())\n        print('-'*36)\n    print('Sample data')\n    print('-'*12)\n    print('head')\n    print(df.head(head))\n    print('-'*12)\n    print('tail')\n    print(df.tail(tail))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def visualize_parquet(file, shape=(137, 236), cmap=None):\n    # shape - (height, width)\n    df = pd.read_parquet(file)\n    df1 = df.head(25)\n    labels, images = df1.iloc[:, 0], df1.iloc[:, 1:].values.reshape(-1, *shape) \n    \n    f, ax = plt.subplots(5, 4, figsize=(20, 20))\n    ax = ax.flatten()\n    f.suptitle(file) #super title\n    \n    for i in range(20):\n        ax[i].set_title(labels[i])\n        ax[i].imshow(images[i], cmap=cmap)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Train parquet files"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_parquet_files","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for file in train_parquet_files:\n    explore_parquet(file)\n    print('==='*18)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that each parquet file has ~50k images of the total 200K train images.\n\nOut of 32333 columns\n\n- First column is the Foriegn key for the image from train.csv\n- The next 32332 columns are pixels of the flattend image of size 137x236"},{"metadata":{"trusted":true},"cell_type":"code","source":"for file in train_parquet_files:\n    visualize_parquet(file, cmap='Blues')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Test parquet files"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_parquet_files","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for file in test_parquet_files:\n    explore_parquet(file)\n    print('==='*18)\n        ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Each test parquet file has 3 images - flattened along with the image_id**"},{"metadata":{"trusted":true},"cell_type":"code","source":"shape=(137, 236)\nfor file in test_parquet_files:\n    print('file', file)\n    df = pd.read_parquet(file)\n    labels, images = df.iloc[:, 0], df.iloc[:, 1:].values.reshape(-1, *shape) \n\n    f, ax = plt.subplots(1, 3, figsize=(10, 10))\n    ax = ax.flatten()\n#     f.suptitle(file) #super title\n\n    for i in range(3):\n        ax[i].set_title(labels[i])\n        ax[i].imshow(images[i], cmap='Blues')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}