{"cells":[{"metadata":{},"cell_type":"markdown","source":"Sorry I can't speak and write well.\n\nFirst write in Japanese, then translate into English and append."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import re\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom pathlib import Path\ninput_path = Path(\"../input\")\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Load Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"# reference\n# https://www.kaggle.com/anokas/kuzushiji-visualisation\ndf_train = pd.read_csv( input_path / 'train.csv')\nunicode_trans = pd.read_csv( input_path / 'unicode_translation.csv')\ntrain_image_path = input_path / \"train_images\"\ntest_image_path = input_path / \"test_images\"\nunicode_map = {codepoint: char for codepoint, char in unicode_trans.values}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Train.csv include in Nan Labels\n\nこちらのdiscussionでも指摘されている通り、**train.csv**にnanデータが含まれている\n\n- https://www.kaggle.com/c/kuzushiji-recognition/discussion/100748#latest-580727"},{"metadata":{"trusted":true},"cell_type":"code","source":"# nan count per column\ndf_train.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Show NaN column\nnan_df = df_train[df_train.isnull()[\"labels\"] == True]\nnan_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Check Unique Label\n**train.csv**内のユニークなラベルを取り出します。取り出す際の処理は以下のカーネルを参考にしています。\n\n- https://www.kaggle.com/wakamezake/kuzushiji-pytorch-data-preprocessing"},{"metadata":{"trusted":true},"cell_type":"code","source":"# get unique labels\nlength = 5\nlabels = []\nfor label in df_train[\"labels\"]:\n    # skip nan\n    if type(label) == str:\n        split_label = label.split()[::length]\n        labels += split_label","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**unicode_translation.csv**には存在するが**train.csv**に存在しないラベルが**569**個ある"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Number of unique_label: {}\".format(len(set(labels))))\nprint(\"Number of unique_label(unicode_translation.csv): {}\".format(unicode_trans.shape[0]))\nprint(\"diff: {}\".format(abs(len(set(labels)) - unicode_trans.shape[0])))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"差集合を使って**unicode_translation.csv**にのみ存在するラベルを取り出したが569個と数が合わない？"},{"metadata":{"trusted":true},"cell_type":"code","source":"unicode_trans_only_labels = set(unicode_trans[\"Unicode\"]) - set(labels)\nprint(\"Number of unicode_trans_only_label: {}\".format(len(unicode_trans_only_labels)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"unicode_trans_only_df = pd.DataFrame({\"Unicode\": list(unicode_trans_only_labels)})\nunicode_trans_only_df[\"string\"] = unicode_trans_only_df[\"Unicode\"].map(unicode_map)\nunicode_trans_only_df.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Label Apperance Bar Plot\nラベルの出現頻度を可視化しました。全てのラベルの出現頻度を描画するととても時間がかかるため、Top10とBottom10のみ表示しています。また図中の表示に日本語の文字表示をすることができなかったため、図の下にUnicodeと文字の対応を載せています。"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df_labels = pd.DataFrame({\"label\": labels})\n# df_labels[\"str\"] = df_labels[\"label\"].map(unicode_map)\nlabel_count  = df_labels[\"label\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# Reference\n# https://www.kaggle.com/tejainece/seaborn-barplot-and-pandas-value-counts\n# label_count  = df_labels[\"label\"].value_counts()\n# plt.figure(figsize=(16,10))\n# sns.barplot(label_count.index, label_count.values, alpha=0.8)\n# plt.title('all unicode counts')\n# plt.ylabel('Number of count', fontsize=12)\n# plt.xlabel('unicode', fontsize=12)\n# plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Reference\n# https://www.kaggle.com/tejainece/seaborn-barplot-and-pandas-value-counts\nplt.figure(figsize=(10,5))\nsns.barplot(label_count[:10,].index, label_count[:10,].values, alpha=0.8)\nplt.title('top 10 unicode counts')\nplt.ylabel('Number of count', fontsize=12)\nplt.xlabel('unicode', fontsize=12)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# convert unicode to str\nd = {\"unicode\": label_count[:10].index.values,\n     \"str\": [unicode_map[l] for l in label_count[:10].index.values]}\npd.DataFrame(d)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,5))\nsns.barplot(label_count[-11:-1,].index, label_count[-11:-1,].values, alpha=0.8)\nplt.title('bottom 10 unicode counts')\nplt.ylabel('Number of count', fontsize=12)\nplt.xlabel('unicode', fontsize=12)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# convert unicode to str\nd = {\"unicode\": label_count[-11:-1,].index.values,\n     \"str\": [unicode_map[l] for l in label_count[-11:-1,].index.values]}\npd.DataFrame(d)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"hirakanaの出現頻度が高いことがわかりました、では全体の何割程度がhirakanaなのか調べてみます。"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Reference\n# https://ja.wikipedia.org/wiki/%E5%B9%B3%E4%BB%AE%E5%90%8D_(Unicode%E3%81%AE%E3%83%96%E3%83%AD%E3%83%83%E3%82%AF)\n# > U+3040..U+309F\nhirakana = re.compile(r\"U\\+30[4-9][0-9a-fA-F]\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sum(label_count)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# The appearance rate of Hirakana accounts for about 60% of the whole.\nhirakana_counts = df_labels[\"label\"].str.match(hirakana).value_counts()\nhirakana_counts / sum(hirakana_counts) * 100","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"hirakanaの中でもどのhirakanaの出現頻度が高いのか気になります、top10に関してはさきほどと同じ結果になりますので省略します。"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_labels[\"is_hirakana\"] = df_labels[\"label\"].str.match(hirakana)\nlabel_count = df_labels[df_labels[\"is_hirakana\"] == True][\"label\"].value_counts()\nplt.figure(figsize=(10,5))\nsns.barplot(label_count[-11:-1,].index, label_count[-11:-1,].values, alpha=0.8)\nplt.title('hirakana bottom 10 unicode counts')\nplt.ylabel('Number of count', fontsize=12)\nplt.xlabel('unicode', fontsize=12)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# convert unicode to str\nd = {\"unicode\": label_count[-11:-1,].index.values,\n     \"str\": [unicode_map[l] for l in label_count[-11:-1,].index.values]}\npd.DataFrame(d)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"逆にhirakana以外の出現頻度はどうなっているか気になります。"},{"metadata":{"trusted":true},"cell_type":"code","source":"label_count = df_labels[df_labels[\"is_hirakana\"] == False][\"label\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,5))\nsns.barplot(label_count[:10,].index, label_count[:10,].values, alpha=0.8)\nplt.title('top 10 unicode counts')\nplt.ylabel('Number of count', fontsize=12)\nplt.xlabel('unicode', fontsize=12)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# convert unicode to str\nd = {\"unicode\": label_count[:10,].index.values,\n     \"str\": [unicode_map[l] for l in label_count[:10,].index.values]}\npd.DataFrame(d)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,5))\nsns.barplot(label_count[-11:-1,].index, label_count[-11:-1,].values, alpha=0.8)\nplt.title('hirakana bottom 10 unicode counts')\nplt.ylabel('Number of count', fontsize=12)\nplt.xlabel('unicode', fontsize=12)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# convert unicode to str\nd = {\"unicode\": label_count[-11:-1,].index.values,\n     \"str\": [unicode_map[l] for l in label_count[-11:-1,].index.values]}\npd.DataFrame(d)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}