{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Exploratory Data Analysis - Bengali AI Dataset\n# Grapheme Combinations "},{"metadata":{},"cell_type":"markdown","source":"This dataset contains images of individual hand-written [Bengali characters](https://en.wikipedia.org/wiki/Bengali_alphabet). \nBengali characters (graphemes) are written by combining three components: a grapheme_root\n, vowel_diacritic, and consonant_diacritic. Your challenge is to classify the components of the grapheme in each\nimage. There are roughly 10,000 possible graphemes, of which roughly 1,000 are represented in the training set. The\ntest set includes some graphemes that do not exist in train but has no new grapheme components. It takes a lot of\nvolunteers filling out [sheets like this](https://github.com/BengaliAI/graphemePrepare/blob/master/collection/A4/form_1.jpg)\nto generate a useful amount of real data; focusing the problem on the grapheme components rather than on recognizing\nwhole graphemes should make it possible to assemble a Bengali OCR system without handwriting samples for all 10,000\ngraphemes.\n\nI have extended this EDA by doing a full [Unicode Visualization of the Bengali Alphabet](https://www.kaggle.com/jamesmcguigan/unicode-visualization-of-the-bengali-alphabet/)"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport seaborn as sns\nfrom sklearn.metrics import confusion_matrix\nfrom IPython.display import Markdown, HTML\nfrom itertools import chain, product\n# from src.jupyter import grid_df_display, combination_matrix\n\npd.set_option('display.max_columns',   500)\npd.set_option('display.max_colwidth',   -1)\n\n%load_ext autoreload\n%autoreload 2","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Custom Library Functions"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Source: https://stackoverflow.com/questions/38783027/jupyter-notebook-display-two-pandas-tables-side-by-side/50899244#50899244\nimport pandas as pd\nfrom IPython.display import display,HTML\n\ndef grid_df_display(list_dfs, rows = 2, cols=3, fill = 'cols'):\n    if fill not in ['rows', 'cols']: print(\"grid_df_display() - fill must be one of: 'rows', 'cols'\")\n\n    html_table = \"<table style='width:100%; border:0px'>{content}</table>\"\n    html_row   = \"<tr style='border:0px'>{content}</tr>\"\n    html_cell  = \"<td style='width:{width}%;vertical-align:top;border:0px'>{{content}}</td>\"\n    html_cell  = html_cell.format(width=100/cols)\n\n    cells = [ html_cell.format(content=df.to_html()) for df in list_dfs[:rows*cols] ]\n    cells += cols * [html_cell.format(content=\"\")] # pad\n\n    if fill == 'rows':   # fill in rows first (first row: 0,1,2,... col-1)\n        grid = [ html_row.format(content=\"\".join(cells[i:i+cols])) for i in range(0,rows*cols,cols)]\n    elif fill == 'cols': # fill columns first (first column: 0,1,2,..., rows-1)\n        grid = [ html_row.format(content=\"\".join(cells[i:rows*cols:rows])) for i in range(0,rows)]\n    else:\n        grid = []\n\n    # noinspection PyTypeChecker\n    display(HTML(html_table.format(content=\"\".join(grid))))\n\n    # add extra dfs to bottom\n    [display(list_dfs[i]) for i in range(rows*cols,len(list_dfs))]\n\n\nif __name__ == \"main\":\n    list_dfs = []\n    list_dfs.extend((pd.DataFrame(2*[{\"x\":\"hello\"}]),\n                     pd.DataFrame(2*[{\"x\":\"world\"}]),\n                     pd.DataFrame(2*[{\"x\":\"gdbye\"}])))\n\n    grid_df_display(3*list_dfs)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Source: https://github.com/JamesMcGuigan/kaggle-digit-recognizer/blob/master/src/utils/confusion_matrix.py\nfrom typing import Union\n\nimport pandas as pd\nfrom pandas.io.formats.style import Styler\n\n\ndef combination_matrix(dataset: pd.DataFrame, x: str, y: str, z: str,\n                       format=None, unique=True) -> Union[pd.DataFrame, Styler]:\n    \"\"\"\n    Returns a combination matrix, showing all valid combinations between three DataFrame columns.\n    Sort of like a heatmap, but returning lists of (optionally) unique values\n\n    :param dataset: The dataframe to create a combination_matrx from\n    :param x: column name to use for the X axis\n    :param y: column name to use for the Y axis\n    :param z: column name to use for the Z axis (values that appear in the cells)\n    :param format: '', ', '-', ', '\\n'    = format value lists as \"\".join() string\n                    str, bool, int, float = cast value lists\n    :param unique:  whether to return only unique values or not - eg: combination_matrix(unique=False).applymap(sum)\n    :return: returns nothing\n    \"\"\"\n    unique_y = sorted(dataset[y].unique())\n    combinations = pd.DataFrame({\n        n: dataset.where(lambda df: df[y] == n)\n            .groupby(x)[z]\n            .pipe(lambda df: df.unique() if unique else df )\n            .apply(list)\n            .apply(sorted)\n        for n in unique_y\n    }).T\n\n    if isinstance(format, str):\n        combinations = combinations.applymap(\n            lambda cell: f\"{format}\".join([str(value) for value in list(cell) ])\n            if isinstance(cell, list) else cell\n        )\n    if format == str:   combinations = combinations.applymap(lambda cell: str(cell)      if isinstance(cell, list) and len(cell) > 0 else ''     )\n    if format == bool:  combinations = combinations.applymap(lambda cell: True           if isinstance(cell, list) and len(cell) > 0 else False  )\n    if format == int:   combinations = combinations.applymap(lambda cell: int(cell[0])   if isinstance(cell, list) and len(cell)     else ''     )\n    if format == float: combinations = combinations.applymap(lambda cell: float(cell[0]) if isinstance(cell, list) and len(cell)     else ''     )\n\n    combinations.index.rename(y, inplace=True)\n    combinations.fillna('', inplace=True)\n    if format == '\\n':\n        return combinations.style.set_properties(**{'white-space': 'pre-wrap'})  # needed for display\n    else:\n        return combinations  # Allows for subsequent .applymap()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Inspect Raw Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls ../input/bengaliai-cv19/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset = pd.read_csv('../input/bengaliai-cv19/train.csv'); \n# for key in ['grapheme_root','vowel_diacritic','consonant_diacritic','grapheme']:\n#     dataset[key] = dataset[key].astype('category')  # ensures groupby().count() shows zeros\ndataset['graphemes'] = dataset['grapheme'].apply(tuple)\ndataset.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Question: How many unique graphemes are there?\n\nThere are 168 grapheme roots, 11 vowel diacritics, 7 consonant diacritics, and 1295 unique graphemes within the 20k training dataset. "},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"unique = dataset.apply(lambda col: col.nunique()); unique","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Question: Can all diacritics be used with any grapheme?\n\n- Documentation claims 10,000+ possible graphemes, which is indeed `168 * 11 * 7 = 12936`\n\n- Assuming that the training dataset is representative of common usage, \n  then certian combinations may never (or rarely) be used in practice).\n\n- Unconfirmed Theory: the physics of the human mouth may make such combinations unpronouncable.\n\n- Conclusion: it may be able infer excluded combinations using simple logical rules"},{"metadata":{},"cell_type":"markdown","source":"### Vowel / Consonant Combinations:\n- Vowel #0 and Consonant #0 combine with everything\n- Vowels #3, #5, #6, #8 have limited combinations with Consonants \n- Consonant #3 is never combined except with Vowel #0\n- Consonant #6 only combineds with Vowels #0 and #1"},{"metadata":{"trusted":true},"cell_type":"code","source":"combination_matrix(dataset, x='consonant_diacritic', y='vowel_diacritic', z='consonant_diacritic', unique=False).applymap(len)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Grapheme Root Combinations:\n- Vowel #0 and Consonant #0 combine with (nearly) everything\n- ALL Roots combine with some Consonant #0\n- Several Roots do NOT combine with Vowel #0 = [26, 28, 33, 34, 73, 82, 108, 114, 126, 152, 157, 158, 163]\n- Several Roots do combine ALL Vowels = [13, 23, 64, 72, 79, 81, 96, 107, 113, 115, 133, 147]}\n- Only Root #107 combines with ALL Consonants"},{"metadata":{"trusted":true},"cell_type":"code","source":"root_vowels            = dataset.groupby('grapheme_root')['vowel_diacritic'].unique().apply(sorted).to_frame().T\nroot_consonants        = dataset.groupby('grapheme_root')['consonant_diacritic'].unique().apply(sorted).to_frame().T\nroot_vowels_values     = root_vowels.applymap(len).values.flatten()\nroot_consonants_values = root_consonants.applymap(len).values.flatten()\n\ndisplay(root_vowels)\ndisplay({\n    \"mean\":   root_vowels_values.mean(),\n    \"median\": np.median( root_vowels_values ),\n    \"min\":    root_vowels_values.min(),\n    \"max\":    root_vowels_values.max(),\n    \"unique_vowels\":    unique['vowel_diacritic'],\n    \"root_combine_0\":   sum([ 0 in lst for lst in root_vowels.values.flatten() ]),\n    \"unique_roots\":     unique['grapheme_root'],\n    \"root_combine_not_0\": str([ index for index, lst in enumerate(root_vowels.values.flatten()) if 0 not in lst ]),    \n    \"root_combine_all\":       [ index for index, lst in enumerate(root_vowels.values.flatten()) if len(lst) == unique['vowel_diacritic'] ],\n})\n# print('--------------------')\ndisplay(root_consonants)\ndisplay({\n    \"mean\":   root_consonants_values.mean(),\n    \"median\": np.median( root_consonants_values ),\n    \"min\":    root_consonants_values.min(),\n    \"max\":    root_consonants_values.max(),\n    \"unique_consonants\":  unique['consonant_diacritic'],\n    \"root_combine_0\": sum([ 0 in lst for lst in root_consonants.values.flatten() ]),\n    \"unique_roots\":   unique['grapheme_root'],\n    \"root_combine_not_0\": str([ index for index, lst in enumerate(root_consonants.values.flatten()) if 0 not in lst ]),        \n    \"root_combine_all\":       [ index for index, lst in enumerate(root_consonants.values.flatten()) if len(lst) == unique['consonant_diacritic'] ],\n})","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Combination Matrices\n\nThis is the full list of which Grapheme Roots combine with which Vowels and Consonant Diacritics"},{"metadata":{"trusted":true},"cell_type":"code","source":"combination_matrix(dataset, x='consonant_diacritic', y='vowel_diacritic', z='grapheme_root', format=', ')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Visualizing Bengali"},{"metadata":{"trusted":true},"cell_type":"code","source":"from collections import Counter\n\ndef filter_pairs_diacritics(pairs, diacritics, key=None):\n    previous_diacritics = set(chain(*[ diacritics[k] for k,v in diacritics.items() if k != key ]))\n    return [ pair for pair in pairs if pair[0] not in previous_diacritics ]\n\ndef print_conflicts(pairs, diacritics, key):\n    valid = filter_pairs_diacritics(pairs, diacritics, key)\n    if len(valid) == 0:\n        conflict_key   = [ k for k,v in diacritics.items() if pairs[0][0] in diacritics[k].values ][0]\n        conflict_dict  = { v:k for k,v in diacritics[conflict_key].items() }\n        display({\n            \"source\":   ( key, pairs[:4] ),\n            \"conflict\": ( conflict_key, conflict_dict[pairs[0][0]], pairs[0][0] ),\n        })\n    return pairs","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"diacritics_raw = {\n    \"vowel_diacritic\":     None,\n    \"consonant_diacritic\": None,\n    \"grapheme_root\":       None\n}\nfor key in [ 'vowel_diacritic', 'consonant_diacritic', 'grapheme_root' ]:\n    diacritics_raw[key] = (\n        dataset.groupby(key)\n            .apply(lambda group:   sum(group['graphemes'].apply(set).apply(Counter), Counter()) )   # -> Counter()\n            .apply(lambda counter: counter.most_common() )                                          # -> [ tuple(symbol, count), ]\n            .apply(lambda pairs:   pairs[0][0] if len(pairs) else '?' )\n    )\n\n    \n### Hardcode conflict resolution and deduplicate - TODO: Verify correctness        \ndiacritics_resolutions = {\n    \"vowel_diacritic\":     pd.Series({ 0: '্', 1: 'া', 2: 'ি' }),\n    \"consonant_diacritic\": pd.Series({ 0: '্' }),\n    \"grapheme_root\":       pd.Series({ 4: 'য' })\n}\ndiacritics = { k:v.copy() for k,v in diacritics_resolutions.items() }\nfor key in [ 'vowel_diacritic', 'consonant_diacritic', 'grapheme_root' ]:\n    diacritics[key] = (\n        dataset.groupby(key)\n            ### NOTE: group['graphemes'].apply(set) removes duplicate unicode diacritics       \n            .apply(lambda group:   sum(group['graphemes'].apply(set).apply(Counter), Counter()) )   # -> Counter()\n            .apply(lambda counter: counter.most_common() )                                          # -> [ tuple(symbol, count), ]\n            .apply(lambda pairs:   print_conflicts(pairs, diacritics, key) ) \n            .apply(lambda pairs:   filter_pairs_diacritics(pairs, diacritics, key)) \n            .apply(lambda pairs:   pairs[0][0] if len(pairs) else '?' )\n    )\n    for index, symbol in diacritics_resolutions[key].items():\n        diacritics[key][index] = symbol\n    \ndisplay(\"Before Conflict Resolution\")\ndisplay( pd.DataFrame(diacritics_raw).fillna('').T )\n\ndisplay(\"Deduplicated\")\ndisplay( pd.DataFrame(diacritics).fillna('').T )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"combination_matrix(dataset, x='consonant_diacritic', y='vowel_diacritic', z='grapheme', format=' ')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"combination_matrix(dataset, x='grapheme_root', y='vowel_diacritic', z='grapheme', format=' ')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"combination_matrix(dataset, x='grapheme_root', y='consonant_diacritic', z='grapheme', format=' ')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Sanity Checking = Found Dataset BUG!"},{"metadata":{},"cell_type":"markdown","source":"This combination_matrix lists 1292 unique grapheme combinations, which is 3 less than the 1295 unique graphemes listed in the training dataset. Something is WRONG!\n\nFound a discrepency BUG is in the dataset. The following root/vowel/consonant keys have multiple unicode graphemes renderings! \n\n{'64-3-2': ['র্তী', 'র্ত্রী'],\n '64-7-2': ['র্তে', 'র্ত্রে'],\n '72-0-2': ['র্দ্র', 'র্দ']}"},{"metadata":{"trusted":true},"cell_type":"code","source":"from itertools import chain\n{\n    \"combinations\": len(list(chain( \n        *combination_matrix(dataset, x='consonant_diacritic', y='vowel_diacritic', z='grapheme_root')\n        .values.flatten() \n    ))),\n    \"unique_graphemes\": unique['grapheme']\n}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Confirm that there are no null or NaN values in the dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset.apply(lambda row: row.isnull()).sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"- Found the BUG! It is in the dataset!\n- There is THREE sets of unique root/vowel/consonant keys that have multiple unicode renderings "},{"metadata":{"trusted":true},"cell_type":"code","source":"( \n    dataset\n    .groupby(['grapheme_root', 'vowel_diacritic', 'consonant_diacritic'])\n    .nunique(dropna=False) > 1 \n).sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"( \n    dataset\n    .groupby(['grapheme_root', 'vowel_diacritic', 'consonant_diacritic'])\n    .nunique(dropna=False) > 1\n).query(\"grapheme != False\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"multilabled_graphemes = {\n    \"64-3-2\": dataset.query(\"grapheme_root == 64 & vowel_diacritic == 3 & consonant_diacritic == 2\")['grapheme'].unique().tolist(),\n    \"64-7-2\": dataset.query(\"grapheme_root == 64 & vowel_diacritic == 7 & consonant_diacritic == 2\")['grapheme'].unique().tolist(),\n    \"72-0-2\": dataset.query(\"grapheme_root == 72 & vowel_diacritic == 0 & consonant_diacritic == 2\")['grapheme'].unique().tolist(),\n}\nmultilabled_graphemes","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As discovered by @ren4yu[[1](https://www.kaggle.com/c/bengaliai-cv19/discussion/133735#763562)] the unicode itself is encoded using the root/vowel/consonant diacritics as a multibyte string, and the unicode consortium have implemented multiple renderings of the same grapheme by allowing duplicate diacritics within the unicode. \n\nThis potentually opens up another datasource for investigation, which is to explore the full range of diacritic combinations within the unicode specification.\n\nThe paper [Fonts-2-Handwriting: A Seed-Augment-Train framework for universal digit classification](https://arxiv.org/pdf/1905.08633.pdf) also makes the suggestion that it may be possible to generate synethetic data for handwriting recognition by rendering each of the unicode graphemes using various Bengali fonts"},{"metadata":{"trusted":true},"cell_type":"code","source":"multilabled_grapheme_list   = list(chain(*multilabled_graphemes.values())); multilabled_grapheme_list\nmultilabled_grapheme_dict   = { grapheme: list(grapheme) for grapheme in multilabled_grapheme_list }\ndisplay(multilabled_grapheme_list)\ndisplay(multilabled_grapheme_dict)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This simply counts how many times each of these multi-keyed unicode graphemes is listed in the database"},{"metadata":{"trusted":true},"cell_type":"code","source":"dataset[ dataset['grapheme'].isin(multilabled_grapheme_list) ].groupby(['grapheme']).count()['image_id'].to_dict()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}