{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Unicode Visualization of the Bengali Alphabet\n\nThis notebook attempts to extend upon this to visualize the Bengali Alphabet, and builds on my previous [Bengali AI Dataset - EDA Grapheme Combinations](https://www.kaggle.com/jamesmcguigan/bengali-ai-dataset-eda-grapheme-combinations/)\n\nUnicode itself is encoded as a multibyte string, using a lower level of base_graphemes than root/vowel/consonant diacritics. Some Benglai Graphemes have multiple renderings for the same root/vowel/consonant combination, which is implemented in unicode by allowing duplicate base_graphemes within the encoding. \n\n* This potentually opens up another datasource for investigation, which is to explore the full range of diacritic combinations within the unicode specification. The paper [Fonts-2-Handwriting: A Seed-Augment-Train framework for universal digit classification](https://arxiv.org/pdf/1905.08633.pdf) also makes the suggestion that it may be possible to generate synethetic data for handwriting recognition by rendering each of the unicode graphemes using various Bengali fonts"},{"metadata":{},"cell_type":"markdown","source":"## Imports"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport seaborn as sns\nfrom sklearn.metrics import confusion_matrix\nfrom IPython.display import Markdown, HTML\nfrom collections import Counter\nfrom itertools import chain\nfrom functools import reduce\n# from src.jupyter import grid_df_display, combination_matrix\n\npd.set_option('display.max_columns',   500)\npd.set_option('display.max_colwidth',   -1)\n\n%load_ext autoreload\n%autoreload 2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Source: https://github.com/JamesMcGuigan/kaggle-digit-recognizer/blob/master/src/utils/confusion_matrix.py\nfrom typing import Union\n\nimport pandas as pd\nfrom pandas.io.formats.style import Styler\n\n\ndef combination_matrix(dataset: pd.DataFrame, x: str, y: str, z: str,\n                       format=None, unique=True) -> Union[pd.DataFrame, Styler]:\n    \"\"\"\n    Returns a combination matrix, showing all valid combinations between three DataFrame columns.\n    Sort of like a heatmap, but returning lists of (optionally) unique values\n\n    :param dataset: The dataframe to create a combination_matrx from\n    :param x: column name to use for the X axis\n    :param y: column name to use for the Y axis\n    :param z: column name to use for the Z axis (values that appear in the cells)\n    :param format: '', ', '-', ', '\\n'    = format value lists as \"\".join() string\n                    str, bool, int, float = cast value lists\n    :param unique:  whether to return only unique values or not - eg: combination_matrix(unique=False).applymap(sum)\n    :return: returns nothing\n    \"\"\"\n    unique_y = sorted(dataset[y].unique())\n    combinations = pd.DataFrame({\n        n: dataset.where(lambda df: df[y] == n)\n            .groupby(x)[z]\n            .pipe(lambda df: df.unique() if unique else df )\n            .apply(list)\n            .apply(sorted)\n        for n in unique_y\n    }).T\n\n    if isinstance(format, str):\n        combinations = combinations.applymap(\n            lambda cell: f\"{format}\".join([str(value) for value in list(cell) ])\n            if isinstance(cell, list) else cell\n        )\n    if format == str:   combinations = combinations.applymap(lambda cell: str(cell)      if isinstance(cell, list) and len(cell) > 0 else ''     )\n    if format == bool:  combinations = combinations.applymap(lambda cell: True           if isinstance(cell, list) and len(cell) > 0 else False  )\n    if format == int:   combinations = combinations.applymap(lambda cell: int(cell[0])   if isinstance(cell, list) and len(cell)     else ''     )\n    if format == float: combinations = combinations.applymap(lambda cell: float(cell[0]) if isinstance(cell, list) and len(cell)     else ''     )\n\n    combinations.index.rename(y, inplace=True)\n    combinations.fillna('', inplace=True)\n    if format == '\\n':\n        return combinations.style.set_properties(**{'white-space': 'pre-wrap'})  # needed for display\n    else:\n        return combinations  # Allows for subsequent .applymap()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Statistics"},{"metadata":{},"cell_type":"markdown","source":"We can decode the graphemes into their constituant base_graphemes using `list()` "},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"dataset = pd.read_csv('../input/bengaliai-cv19/train.csv'); \ndataset['base_graphemes'] = dataset['grapheme'].apply(list)\ndataset.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are 62 base_graphemes in the unicode encoding, with a median of 4 and maximum of 8 symbols required to encode each grapheme. \n\nWe can also see the percentage frequency for each root diacritic."},{"metadata":{"trusted":true},"cell_type":"code","source":"base_diacritics_unique = sorted(set(chain(*dataset['base_graphemes'].values)))\nbase_diacritics_stats  = {\n    \"mean\":   round( dataset['base_graphemes'].apply(len).mean(), 2),\n    \"median\": np.median( dataset['base_graphemes'].apply(len) ),\n    \"min\":    dataset['base_graphemes'].apply(len).min(),\n    \"max\":    dataset['base_graphemes'].apply(len).max(),\n    \"std\":    dataset['base_graphemes'].apply(len).std(),    \n    \"unique\": len( set(chain(*dataset['base_graphemes'].values))),\n    \"count\":  len(list(chain(*dataset['base_graphemes'].values))),\n    \"mean_duplicated_bases\":  dataset['base_graphemes'].apply(lambda value: (len(value) - len(set(value)))).mean(),\n    \"max_duplicated_bases\":   dataset['base_graphemes'].apply(lambda value: (len(value) - len(set(value)))).max(),    \n    \"count_duplicated_bases\": dataset['base_graphemes'].apply(lambda value: (len(value) - len(set(value))) != 0).sum(),        \n}\nbase_diacritics_counter = dict( \n    sum(dataset['base_graphemes'].apply(Counter), Counter()).most_common()\n)\n\ndisplay( pd.DataFrame([base_diacritics_counter]) / base_diacritics_stats['count'] )\ndisplay( \" \".join(base_diacritics_unique) )\ndisplay( base_diacritics_stats )","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Base Graphemes\n\nAs there are fewer base_graphemes than root_graphemes in the Bengali alphabet. Thus we can perform a set analyis to determine how the grapheme_roots are themselves decomposed into a lower level of root diacritic combinations."},{"metadata":{"trusted":true},"cell_type":"code","source":"base_diacritic_sets = {\n    key: dataset.groupby(key)['base_graphemes']\n                .apply(lambda group: reduce(lambda a,b: set(a) & set(b), group)) \n                .apply(sorted)     \n    for key in [ 'vowel_diacritic', 'consonant_diacritic', 'grapheme_root' ]\n}\ndisplay(\n    pd.DataFrame(base_diacritic_sets)\n        .applymap(lambda x: x if x is not np.nan else set())        \n        .applymap(lambda group: \"\\n\".join(group))\n        .T\n        .style.set_properties(**{'white-space': 'pre-wrap'})\n)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"If we extend the dataset with the information, we can display a combination matrix in pure Bengali"},{"metadata":{"trusted":true},"cell_type":"code","source":"for key in [ 'vowel_diacritic', 'consonant_diacritic', 'grapheme_root' ]:\n    base_key = key.split('_')[0] + '_base'\n    zfill = 3 if key == 'grapheme_root' else 2\n    dataset[base_key] = (\n        dataset[key]\n            .apply(lambda value: [ str(value).zfill(zfill)] + sorted(base_diacritic_sets[key][value]))\n            .apply(lambda value: \" \".join(value))            \n            .fillna('')\n    )\n# Make numeric strings sortable\ndataset.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualization of Grapheme Combinations in Bengali Alphabet\n\n# Vowel/Consonant Combinations Table"},{"metadata":{"trusted":true},"cell_type":"code","source":"combination_matrix(dataset, x='consonant_base', y='vowel_base', z='grapheme', format=' ')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Base/Vowel Combinations Table"},{"metadata":{"trusted":true},"cell_type":"code","source":"combination_matrix(dataset, x='grapheme_base', y='vowel_base', z='grapheme', format='\\n')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Base/Consonant Combinations Table"},{"metadata":{"trusted":true},"cell_type":"code","source":"combination_matrix(dataset, x='grapheme_base', y='consonant_base', z='grapheme', format='\\n')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Full Combination Table"},{"metadata":{"trusted":true},"cell_type":"code","source":"combination_matrix(dataset, x=['vowel_base','consonant_base'], y='grapheme_base', z='grapheme', format=' ').T","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"combination_matrix(dataset, x=['vowel_base','consonant_base'], y='grapheme_base', z='grapheme', format=' ')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}