{"cells":[{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"# 1. Introduction\n\nBengali is the 5th most spoken language in the world with hundreds of million of speakers. It’s the official language of Bangladesh and the second most spoken language in India. Considering its reach, there’s significant business and educational interest in developing AI that can optically recognize images of the language handwritten. This challenge hopes to improve on approaches to Bengali recognition.\n\nOptical character recognition is particularly challenging for Bengali. While Bengali has 49 letters (to be more specific 11 vowels and 38 consonants) in its alphabet, there are also 18 potential diacritics, or accents. This means that there are many more graphemes, or the smallest units in a written language. The added complexity results in ~13,000 different grapheme variations (compared to English’s 250 graphemic units).\n\nBangladesh-based non-profit Bengali.AI is focused on helping to solve this problem. They build and release crowdsourced, metadata-rich datasets and open source them through research competitions. Through this work, Bengali.AI hopes to democratize and accelerate research in Bengali language technologies and to promote machine learning education.\n\nFor this competition, you’re given the image of a handwritten Bengali grapheme and are challenged to separately classify three constituent elements in the image: grapheme root, vowel diacritics, and consonant diacritics.\n\n \n"},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# 2. Data Description\n\nThis dataset contains images of individual hand-written Bengali characters. Bengali characters (graphemes) are written by combining three components: a grapheme_root, vowel_diacritic, and consonant_diacritic. Your challenge is to classify the components of the grapheme in each image. There are roughly 10,000 possible graphemes, of which roughly 1,000 are represented in the training set. The test set includes some graphemes that do not exist in train but has no new grapheme components. It takes a lot of volunteers filling out sheets like this to generate a useful amount of real data; focusing the problem on the grapheme components rather than on recognizing whole graphemes should make it possible to assemble a Bengali OCR system without handwriting samples for all 10,000 graphemes.\n\n## 2.1 Files\n\n**train.csv**\n\n* `image_id`: the foreign key for the parquet files\n* `grapheme_root`: the first of the three target classes\n* `vowel_diacritic`: the second target class\n* `consonant_diacritic`: the third target class\n* `grapheme`: the complete character. Provided for informational purposes only, you should not need to use this.\n\n**test.csv**\n\nEvery image in the test set will require three rows of predictions, one for each component. This csv specifies the exact order for you to provide your labels. - `row_id`: foreign key to the sample submission - `image_id`: foreign key to the parquet file - `component`: the required target class for the row (grapheme_root, vowel_diacritic, or consonant_diacritic)\n\n**sample_submission.csv**\n\n`row_id`: foreign key to test.csv\n`target`: the target column\n\n**(train/test).parquet**\n\nEach parquet file contains tens of thousands of 137x236 grayscale images. The images have been provided in the parquet format for I/O and space efficiency. Each row in the parquet files contains an `image_id` column, and the flattened image.\n\n**class_map.csv**\n\nMaps the class labels to the actual Bengali grapheme components."},{"metadata":{},"cell_type":"markdown","source":"# 3. Peek to the input Folder"},{"metadata":{"trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. Fetching Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('/kaggle/input/bengaliai-cv19/train.csv')\ntest = pd.read_csv('/kaggle/input/bengaliai-cv19/test.csv')\nsample = pd.read_csv('/kaggle/input/bengaliai-cv19/sample_submission.csv')\nclass_map = pd.read_csv('/kaggle/input/bengaliai-cv19/class_map.csv')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print('Size of train data', train.shape)\nprint('Size of test data', test.shape)\nprint('Size of sample submission', sample.shape)\nprint('Size of Class Map: ', class_map.shape)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.2  Peek at the data\n\n### Train Dataframe"},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.columns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Test Dataframe"},{"metadata":{"trusted":true},"cell_type":"code","source":"test.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.columns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### samble_submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"sample.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Class Map"},{"metadata":{"trusted":true},"cell_type":"code","source":"class_map.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Image Data\n\nImage Data in in parquet files and contrain grayscale images of below mentioned dimentions. If you want to read more about this file format then try: https://acadgild.com/blog/parquet-file-format-hadoop Note that the file it self conatins values of all the 32332 pixels (137*236) in each row coresponding to a image.\n\n`Image Height = 137`\n\n`Image Width = 236`\n\n\n### Image Utils"},{"metadata":{"trusted":true},"cell_type":"code","source":"HEIGHT = 137\nWIDTH = 236\n\ndef load_images(file):\n    df = pd.read_parquet(file)\n    return df.iloc[:, 1:].values.reshape(-1, HEIGHT, WIDTH)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"## loading one of the parquest file for analysis\ndummy_images = load_images('/kaggle/input/bengaliai-cv19/train_image_data_0.parquet')\nprint(\"Shape of loaded files: \", dummy_images.shape)\nprint(\"Number of images in loaded files: \", dummy_images.shape[0])\nprint(\"Shape of first loaded image: \", dummy_images[0].shape)\nprint(\"\\n\\nFirst image looks like:\\n\\n\", dummy_images[0])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Plotting image"},{"metadata":{"trusted":true},"cell_type":"code","source":"import seaborn as sb\nimport matplotlib.pyplot as plt\n\n## View the pixel values as image\nplt.imshow(dummy_images[10], cmap='Greys')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### plotting more images for better intution"},{"metadata":{"trusted":true},"cell_type":"code","source":"f, ax = plt.subplots(6, 6, figsize=(16, 10))\n\nfor i in range(6):\n    for j in range(6):\n        ax[i][j].imshow(dummy_images[i*6+j], cmap='Greys')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.3. Checking for Null Values"},{"metadata":{"trusted":true},"cell_type":"code","source":"train.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"class_map.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Luckily there are no null values in this competition, Bye! Bye! Imputation"},{"metadata":{},"cell_type":"markdown","source":"## 4.4 Checking for class distribution"},{"metadata":{"trusted":true},"cell_type":"code","source":"import seaborn as sns\n\n\nsns.catplot(x='vowel_diacritic',data=train,kind=\"count\", height=8.27, aspect=11.7/8.27)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.catplot(x='consonant_diacritic',data=train,kind=\"count\", height=8.27, aspect=11.7/8.27)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.catplot(x='grapheme_root',data=train,kind=\"count\", height=8.27, aspect=30/8.27)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Unique Grapheme-Root in train data: \", train.grapheme_root.nunique())\nprint(\"Unique Vowel-Diacritic in train data: \", train.vowel_diacritic.nunique())\nprint(\"Unique Consonant-Diacritic in train data: \", train.consonant_diacritic.nunique())\nprint(\"Unique Grapheme (Combination of three) in train data: \", train.grapheme.nunique())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Since I don't want this kernel notebook to get heavy, I am experimenting model related tasks in another notebook.\n\n[Please visit this kernel for modelling ](https://www.kaggle.com/rohitsingh9990/bengaliai-starter-eda-multi-output-densenet/edit)"},{"metadata":{},"cell_type":"markdown","source":"### If you find this kernel usefull, Do upvote.\n"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}