{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Imports and Constants"},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"DATA_DIR = '/kaggle/input/bengaliai-cv19/'\n\npath_train_data = DATA_DIR + 'train.csv'","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = pd.read_csv(path_train_data)\ndf_train.drop('grapheme', inplace=True, axis=1)\ndf_train.sample(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# EDA"},{"metadata":{"trusted":true},"cell_type":"code","source":"import seaborn as sns\nimport matplotlib.pyplot as plt\n\nsns.set()\nsns.set_palette(sns.dark_palette('purple'))\n\nbar_palette = lambda n: sns.hls_palette(n, l=.4)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_frequency(data):\n    if isinstance(data, str):\n        counts = df_train[data].value_counts()\n        name = data\n        \n    else:\n        counts = df_train.groupby(data).size().sort_values(ascending=False)\n        name = '-'.join(data)\n\n    counts /= len(df_train)\n    n = len(counts)\n    x = np.arange(n)\n    y = counts.cumsum()\n    \n    plt.figure(figsize=(20, 5))\n    if n <= 100:\n        plt.subplot(1, 2, 1)\n    sns.barplot(x=counts.index, y=counts, order=counts.index, palette=bar_palette(n))\n    plt.title('Frequency of each %s class [%d]' % (name, n))\n    plt.ylabel('Frequency')\n    \n    if n > 40:\n        plt.xticks([])\n    else:\n        plt.xlabel('Class label')\n    \n    if n > 100:\n        plt.show()\n        plt.figure(figsize=(20, 5))\n       \n    else:\n        plt.subplot(1, 2, 2)\n        \n    plt.fill_between(x, y, step='post', alpha=0.4)\n    plt.step(x, y, where='post')\n    plt.ylim(0, 1.05)\n    plt.title('Cumulative frequency of each %s class' % name)\n    plt.xlabel('Number of classes')\n    plt.ylabel('Cumulative frequency')\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Single Class Frequencies"},{"metadata":{},"cell_type":"markdown","source":"### Consonant Diacritic"},{"metadata":{},"cell_type":"markdown","source":"7 consonant diacritics. About 60% of all data has class 1. Classes 3 and 6 are extremely rare."},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_frequency('consonant_diacritic')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Vowel Diacritic"},{"metadata":{},"cell_type":"markdown","source":"11 Vowels diacritics. Classes 0 and 1 are common. Classes 5, 6, 8 and 10 are rare."},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_frequency('vowel_diacritic')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Grapheme Root"},{"metadata":{},"cell_type":"markdown","source":"168 grapheme roots. Classes are quite unbalanced. First 10 classes are 25% of all data. First 30 classes are 50% of all data."},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_frequency('grapheme_root')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Double Class Frequencies"},{"metadata":{},"cell_type":"markdown","source":"### Consonant Diacritic - Vowel Diacritic"},{"metadata":{},"cell_type":"markdown","source":"About 62% of all possible combinations appear in the training set. Classes are quite unbalanced. First 5 classes are about half of all data."},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_frequency(['consonant_diacritic', 'vowel_diacritic'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Consonant Diacritic - Grapheme Root"},{"metadata":{},"cell_type":"markdown","source":"About 28% of all possible combinations appear in the training set. Classes are relatively more balanced than \"consonant diacritic - vowel diacritic\"."},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_frequency(['consonant_diacritic', 'grapheme_root'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Vowel Diacritic - Grapheme Root"},{"metadata":{},"cell_type":"markdown","source":"About 44% of all possible combinations appear in the training set."},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_frequency(['vowel_diacritic', 'grapheme_root'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Total Frequencies"},{"metadata":{},"cell_type":"markdown","source":"About 10% of all possible combinations appear in the training set. Surprisingly all classes have very similar frequencies."},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_frequency(['consonant_diacritic', 'vowel_diacritic', 'grapheme_root'])","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.6"}},"nbformat":4,"nbformat_minor":1}