{"cells":[{"metadata":{"_uuid":"f60b6918bd1cb74f2c86071a926d2f5610b576bd"},"cell_type":"markdown","source":"# メモ\n自分自身の理解のためにも、メモとして日本語訳を随時更新していれていきます。"},{"metadata":{"_uuid":"3b5c9dc5eedea5af75e07cada3328992f7e7dc84"},"cell_type":"markdown","source":"## General information（概要）\n\n- This kernel is dedicated to EDA of PetFinder.my Adoption Prediction challenge as well as feature engineering and modelling.\n- このカーネルはPetFinder.my Adoption Predictionの特徴量エンジニアリングとモデル作成について記載しています。\n\n![](https://i.imgur.com/rvSWCYO.png)\n(a screenshot of the PetFinder.my site)\n\n- In this dataset we have lots of information: tabular data, texts and even images! This gives a lot of possibilties for feature engineering and modelling. The only limiting factor is the fact that the competition is kernel-only. On the other hand this will ensure everyone has the same computational resources.\n- このデータセットには多くの情報が存在しており、表形式、文字列、そして画像データが存在する。これは特徴量エンジニアリングとモデルに多くの可能性が存在することを示唆している。制限事項はカーネルのみであること。一方でこれはすべての参加者が同一のマシンリソースの利用ということを意味している。\n\n- In this kernel I want to pay attention to several things:\n- このカーネルでは以下の事項に注意を置いている。\n* comparing distribution of features in train and test data;\n* exploring features and their interactions;\n* trying various types of feature engineering;\n* trying various models without neural nets (for now);\n\n* トレーニングデータとテストデータの分散の比較\n* 特徴量と相互作用の探索\n* 種類の異なる特徴量エンジニアリングの試み\n* ニューラルネットワークを使わない複数モデルでの試み\n\n- It is important to remember that this competition has stage 2, so our models will run against unseen data.\n* - このコンペティションは2ステージ存在し、参加者が作成したモデルは非公開のデータに対して最終的に実行される\n\n*Work still in progress*"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"#libraries\nimport numpy as np \nimport pandas as pd \nimport os\nimport json\nimport seaborn as sns \nimport matplotlib.pyplot as plt\n%matplotlib inline\nplt.style.use('ggplot')\nimport lightgbm as lgb\nimport xgboost as xgb\nimport time\nimport datetime\nfrom PIL import Image\nfrom wordcloud import WordCloud\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.model_selection import StratifiedKFold, KFold\nfrom sklearn.metrics import mean_squared_error, roc_auc_score\nfrom sklearn.linear_model import LogisticRegression, LogisticRegressionCV\nimport gc\nfrom catboost import CatBoostClassifier\nfrom tqdm import tqdm_notebook\nimport plotly.offline as py\npy.init_notebook_mode(connected=True)\nimport plotly.graph_objs as go\nimport plotly.tools as tls\nimport random\nimport warnings\nwarnings.filterwarnings(\"ignore\")\nfrom functools import partial\npd.set_option('max_colwidth', 500)\npd.set_option('max_columns', 500)\npd.set_option('max_rows', 100)\nimport os\nimport scipy as sp\nfrom math import sqrt\nfrom collections import Counter\nfrom sklearn.metrics import confusion_matrix as sk_cmatrix\n\nfrom sklearn.feature_extraction.text import TfidfVectorizer\nfrom nltk.tokenize import TweetTokenizer\nfrom sklearn.ensemble import RandomForestClassifier\nimport langdetect\nimport eli5\nfrom IPython.display import display \n\nfrom sklearn.metrics import cohen_kappa_score\ndef kappa(y_true, y_pred):\n    return cohen_kappa_score(y_true, y_pred, weights='quadratic')","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"breeds = pd.read_csv('../input/breed_labels.csv')\ncolors = pd.read_csv('../input/color_labels.csv')\nstates = pd.read_csv('../input/state_labels.csv')\n\ntrain = pd.read_csv('../input/train/train.csv')\ntest = pd.read_csv('../input/test/test.csv')\nsub = pd.read_csv('../input/test/sample_submission.csv')\n\ntrain['dataset_type'] = 'train'\ntest['dataset_type'] = 'test'\nall_data = pd.concat([train, test])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"64c7b97d92588795fe9968d0d4d0ada723fe3a9d"},"cell_type":"markdown","source":"## Data overview（データ概要）\n\n- Let's have a quick look at the data first!\n- 初めに簡単にデータを見てみる"},{"metadata":{"trusted":true,"_uuid":"f636d2b0986b026e2f9e05b1e088ea3e8f0f2e3f"},"cell_type":"code","source":"print(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"aeae0a045c4e8bc12ec95b954666cf57442fdfac"},"cell_type":"code","source":"train.drop('Description', axis=1).head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9faa1ff39a080f5ab524bc9fecbccb2777628139","scrolled":true},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0a9533822d60c67f1efd4d536ffd781e84d8678e"},"cell_type":"markdown","source":"* We have almost 15 thousands dogs and cats in the dataset;\n* 約15000の犬と猫のデータセットがある\n* Main dataset contains all important information about pets: age, breed, color, some characteristics and other things;\n* メインのデータセット（train.csv）にはペットの重要な情報がすべて格納されている。（年齢、ブリード、色、特徴、その他）\n* Desctiptions were analyzed using Google's Natural Language API providing sentiments and entities. I suppose we could do a similar thing ourselves;\n* 概要はGoogleのNatural Language APIにて解析されており、思い（ネガポジの話？）と実体を表している。参加者も自身で同様のことができる。\n* There are photos of some pets;\n* いくつかのペットには写真が存在する。\n* Some meta-information was extracted from images and we can use it;\n* いくつかのメタ情報は画像から抽出でき、参加者は利用できる。\n* There are separate files with labels for breeds, colors and states;\n* ブリード、色、状態のラベルは別ファイルに記載されている。\n\n- Let's start with the main dataset.\n- インデータから確認を始める。\n\n- I have also created a full dataset by combining train and test data. This is done purely for more convenient visualization. Column \"dataset_type\" shows which dataset the data belongs to.\n- トレーニングデータとテストデータの両方を組み合わせたデータセットを作成。より簡易にビジュアライズするため純粋にこのデータ作成は行っている。カラムの「dataset_type」はトレーニング、テストデータのどちらに属するかを表している。"},{"metadata":{"trusted":true,"_uuid":"8542f37472712951dc614bd5d4fce2403b2efff2"},"cell_type":"markdown","source":"## Main data exploration（メインデータの探索）"},{"metadata":{"trusted":true,"_uuid":"d2352291c16f881f9dbd851e2dbed3cfe1e0b15c"},"cell_type":"markdown","source":"### Target: Adoption speed（目的変数の里親が見つかる速度）\n\n* 0 - Pet was adopted on the same day as it was listed.\n* 1 - Pet was adopted between 1 and 7 days (1st week) after being listed.\n* 2 - Pet was adopted between 8 and 30 days (1st month) after being listed.\n* 3 - Pet was adopted between 31 and 90 days (2nd & 3rd month) after being listed.\n* 4 - No adoption after 100 days of being listed. (There are no pets in this dataset that waited between 90 and 100 days). \n\n* 0 - リストに掲載された日に里親が見つかる\n* 1 - リストに掲載された日から1～7日の間に里親が見つかる\n* 2 - リストに掲載された日から8～30日の間に里親が見つかる\n* 3 - リストに掲載された日から31～90日の間に里親が見つかる\n* 4 - リストに掲載された日から100日後も里親が見つからない"},{"metadata":{"trusted":true,"_uuid":"14c64e761f2f5df96b90bc82d20d57c67004e38d","_kg_hide-input":true},"cell_type":"code","source":"train['AdoptionSpeed'].value_counts().sort_index().plot('barh', color='teal');\nplt.title('Adoption speed classes counts');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"605f45c4f67fe32115799cf269c6a232e2eef615"},"cell_type":"markdown","source":"- A small note on how annotating works:\n- seaborn利用時のメモ\n* When I use seaborn countplot, I assign the figure to a variable - this allows to change its attributes and go deeper into its parameters;\n* seabornのcountplotを利用する際、図は可変にしている。これは項目の変更ができるためよりパラメータを深く考察できるようになるためである\n* Figure has `Axes` - bars - which contain information about color, transparency and other parameters;\n* 図には軸が存在する。バーは色、透過、その他パラメータの情報を保持する\n* And `patches` in `Axes` contain this information;\n* 軸中のpatchesは情報を持つ\n* So we can take information from 'patches`, for example width and height of each bar, and plot correct text in correct places\n* そのため我々はpatchesから情報が得られる。例えば、バーの太さ、高さ、そして期待する位置に期待する文字列を追加できる。\n\nhttps://matplotlib.org/users/artists.html"},{"metadata":{"_kg_hide-output":true,"trusted":true,"_uuid":"a02eef2ccad681aa0103cbd22a8e8032af74489e"},"cell_type":"code","source":"plt.figure(figsize=(14, 6));\ng = sns.countplot(x='AdoptionSpeed', data=all_data.loc[all_data['dataset_type'] == 'train']);\nplt.title('Adoption speed classes rates');\nax=g.axes","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d6da75eddb7b7938b1accf9812ebae34f6cf2e61"},"cell_type":"code","source":"#Axes\nax","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f62c09b5bb3f0b75fb98b919efdcbc8a5a84aaae"},"cell_type":"code","source":"# patches\nax.patches","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e25d2efe8c0fd2d3aa7ad00e80fccca41cf13fd9"},"cell_type":"code","source":"# example of info in patches\nax.patches[0].get_x()","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"0529ee4fa802f9c0575b8f6e88533bc4a94746af"},"cell_type":"code","source":"plt.figure(figsize=(14, 6));\ng = sns.countplot(x='AdoptionSpeed', data=all_data.loc[all_data['dataset_type'] == 'train'])\nplt.title('Adoption speed classes rates');\nax=g.axes\nfor p in ax.patches:\n     ax.annotate(f\"{p.get_height() * 100 / train.shape[0]:.2f}%\", (p.get_x() + p.get_width() / 2., p.get_height()),\n         ha='center', va='center', fontsize=11, color='gray', rotation=0, xytext=(0, 10),\n         textcoords='offset points')  ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"16b66de28b50c41c8fcc2c2ea5d77e0e722f6658"},"cell_type":"markdown","source":"- We can see that some pets were adopted immediately, but these are rare cases: maybe someone wanted to adopt any pet, or the pet was lucky to be seen by person, who wanted a similar pet.\n- 幾つかのペットはすぐに里親が見つかるが、それはとてもまれなケースであることが分かる。ペットが欲しい人がいたか、もしくは運よく欲しかったペットに似たペットがいたかというケースが考えられる。\n- A lot of pets aren't adopted at all, which is quite sad :( I hope our models and analysis will help them to find their home!\n- 多くのペットは全く里親が見つからない。\n\n- It is nice that a lot of pets are adopted within a first week of being listed!\n- 幸運なことにリストに掲載された初めの週に多くの動物に里親が見つかっている。\n\n- One more interesting thing is that the classes have a linear relationship - the higher the number, the worse situation is. So it could be possible to build not only multiclass classification, but also regression.\n- 興味深いこととして、このクラス（AdoptionSpeed）は線形の関係を持っていることだ。高い数値は、悪い状況を示す。つまり多クラス分類ではなく、回帰でモデルを作れる可能性がある。"},{"metadata":{"_uuid":"ea232a5cf54c3220f266863e29b9f8517fda6b3a"},"cell_type":"markdown","source":"### Type（タイプ）\n1 - Dog, 2 - Cat"},{"metadata":{"trusted":true,"_uuid":"af73348f3044d637a5f11bc4645a31d725b6670e","_kg_hide-input":true},"cell_type":"code","source":"all_data['Type'] = all_data['Type'].apply(lambda x: 'Dog' if x == 1 else 'Cat')\nplt.figure(figsize=(10, 6));\nsns.countplot(x='dataset_type', data=all_data, hue='Type');\nplt.title('Number of cats and dogs in train and test data');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0765c6567da19a6bd83789421675e29338bbb02b"},"cell_type":"markdown","source":"- We can see that the rate of dogs in train dataset is higher that in test set. But I don't think the difference is seriuos.\n- トレーニングデータにはテストデータよりも犬の割合が多くあるとわかる。この違いは深刻である。"},{"metadata":{"_uuid":"6ec4fe56f1e6126b833549a954c27a8a74a7e572"},"cell_type":"markdown","source":"#### Comparison of rates（比率の比較）\n\n- From here on I'll compare not only counts of pets in different categories, but also compate adoption speed rates with base ones.\n- 異なるカテゴリのペットの数の比較だけでなく里親が見つかる早さについても比較していく。\n\n- This is how it works:\n- どのようにアプローチしたかは、以下の通りである。\n* As we saw earlier the base rate of pets with Adoption speed 0 is 410 / 14993 = 0.027;\n* Adoption speed0のペット比率は0.027だった。\n* Now look at the next graph: there are 6861 cats in train dataset and 240 of them have Adoption Speed 0. So the rate is 240 / 6861 = 0.035;\n* 次のグラフを参照いただきたい。6861の猫の内240はAdoption Speed 0になっている。つまり比率は0.035\n* 0.035 / 0027 = 1.2792 so cats have 27, 92% more chances to have Adoption Speed 0;\n* 0.035 / 0.025 = 1.2792なので、猫の方が27.92%Adoption Speed 0 になる可能性がある。"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"5ac22d86fb4266d9ba39645a910bcbf0ff6a8db7"},"cell_type":"code","source":"main_count = train['AdoptionSpeed'].value_counts(normalize=True).sort_index()\ndef prepare_plot_dict(df, col, main_count):\n    \"\"\"\n    Preparing dictionary with data for plotting.\n    \n    I want to show how much higher/lower are the rates of Adoption speed for the current column comparing to base values (as described higher),\n    At first I calculate base rates, then for each category in the column I calculate rates of Adoption speed and find difference with the base rates.\n    \n    \"\"\"\n    main_count = dict(main_count)\n    plot_dict = {}\n    for i in df[col].unique():\n        val_count = dict(df.loc[df[col] == i, 'AdoptionSpeed'].value_counts().sort_index())\n\n        for k, v in main_count.items():\n            if k in val_count:\n                plot_dict[val_count[k]] = ((val_count[k] / sum(val_count.values())) / main_count[k]) * 100 - 100\n            else:\n                plot_dict[0] = 0\n\n    return plot_dict\n\ndef make_count_plot(df, x, hue='AdoptionSpeed', title='', main_count=main_count):\n    \"\"\"\n    Plotting countplot with correct annotations.\n    \"\"\"\n    g = sns.countplot(x=x, data=df, hue=hue);\n    plt.title(f'AdoptionSpeed {title}');\n    ax = g.axes\n\n    plot_dict = prepare_plot_dict(df, x, main_count)\n\n    for p in ax.patches:\n        h = p.get_height() if str(p.get_height()) != 'nan' else 0\n        text = f\"{plot_dict[h]:.0f}%\" if plot_dict[h] < 0 else f\"+{plot_dict[h]:.0f}%\"\n        ax.annotate(text, (p.get_x() + p.get_width() / 2., h),\n             ha='center', va='center', fontsize=11, color='green' if plot_dict[h] > 0 else 'red', rotation=0, xytext=(0, 10),\n             textcoords='offset points')  ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e45a3d16efb767b4de299f5d28527e76a493e118"},"cell_type":"code","source":"plt.figure(figsize=(18, 8));\nmake_count_plot(df=all_data.loc[all_data['dataset_type'] == 'train'], x='Type', title='by pet Type')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"479f2dd53a07fc87e05579b54cda20e4f9a9844c"},"cell_type":"markdown","source":"- We can see that cats are more likely to be adopted early than dogs and overall the percentage of not adopted cats is lower. \n- 猫の方が犬に比べ早く里親が見つかる可能性が高く、猫の里親が見つからなかった可能性は犬に比べて低い\n- Does this mean people prefer cats? Or maybe this dataset is small and could contain bias.\n- これは人々は猫を好むことを示すのだろうか。またはこのデータセットは小さく、バイアスが含まれているのだろうか。\n- On the other hand more dogs are adopted after several months.\n- 一方でより多くの犬は数か月後に里親が見つかる"},{"metadata":{"_uuid":"c33e9eb0b3a712e5d9c2d46591055377ebcd1e34"},"cell_type":"markdown","source":"### Name（名前）\n- I don't really think that names are important in adoption, but let's see.\n- 私は名前は里親が見つかる上で重要であると考える。\n\n- At first let's look at most common names.\n- 初めに最も一般的な名前を観察していく。"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"aba94b4f396cac1c5151c5984318ed26e1b3dc64"},"cell_type":"code","source":"fig, ax = plt.subplots(figsize = (16, 12))\nplt.subplot(1, 2, 1)\ntext_cat = ' '.join(all_data.loc[all_data['Type'] == 'Cat', 'Name'].fillna('').values)\nwordcloud = WordCloud(max_font_size=None, background_color='white',\n                      width=1200, height=1000).generate(text_cat)\nplt.imshow(wordcloud)\nplt.title('Top cat names')\nplt.axis(\"off\")\n\nplt.subplot(1, 2, 2)\ntext_dog = ' '.join(all_data.loc[all_data['Type'] == 'Dog', 'Name'].fillna('').values)\nwordcloud = WordCloud(max_font_size=None, background_color='white',\n                      width=1200, height=1000).generate(text_dog)\nplt.imshow(wordcloud)\nplt.title('Top dog names')\nplt.axis(\"off\")\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"82d458fa1b622dba68d3e244dd0b8d97191d8c66"},"cell_type":"markdown","source":"- Cute names! :) I like some of them!\n\n- It is worth noticing some things:\n- 以下のことに気が付いた\n* Often we see normal pet names like \"Mimi\", \"Angel\" and so on;\n* MimiやAngelといった一般的な名前を見つけることができる\n* Quite often people write simply who is there for adoption: \"Kitten\", \"Puppies\";\n* 頻繁に里親探しのためKittenやPuppiesといった何かがわかる名前にしている\n* Vety often the color of pet is written, sometimes gender;\n* また動物の色も名前になる頻度は高い。性別で名前を付けられることもある\n* And it seems that sometimes names can be strange or there is some info written instead of the name;\n* 名前は変な場合もあるし、また名前の変わりに情報が書かれている場合もある\n\n- One more thing to notice is that some pets don't have names. Let's see whether this is important\n- もう一つ気が付いたことは、いくつかのペットは名前を持たないということである。これが重要であるか確認する"},{"metadata":{"trusted":true,"_uuid":"4c3b1a154876b224f9de77af9103bfc0686ffb87"},"cell_type":"code","source":"print('Most popular pet names and AdoptionSpeed')\nfor n in train['Name'].value_counts().index[:5]:\n    print(n)\n    print(train.loc[train['Name'] == n, 'AdoptionSpeed'].value_counts().sort_index())\n    print('')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e50b97ac02bb49c8445dd52916d0c2c318aa8638"},"cell_type":"markdown","source":"#### No name（名なし）"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"7673019ef3d7ed0f03db97a770c88343b1429d7b"},"cell_type":"code","source":"train['Name'] = train['Name'].fillna('Unnamed')\ntest['Name'] = test['Name'].fillna('Unnamed')\nall_data['Name'] = all_data['Name'].fillna('Unnamed')\n\ntrain['No_name'] = 0\ntrain.loc[train['Name'] == 'Unnamed', 'No_name'] = 1\ntest['No_name'] = 0\ntest.loc[test['Name'] == 'Unnamed', 'No_name'] = 1\nall_data['No_name'] = 0\nall_data.loc[all_data['Name'] == 'Unnamed', 'No_name'] = 1\n\nprint(f\"Rate of unnamed pets in train data: {train['No_name'].sum() * 100 / train['No_name'].shape[0]:.4f}%.\")\nprint(f\"Rate of unnamed pets in test data: {test['No_name'].sum() * 100 / test['No_name'].shape[0]:.4f}%.\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c0d12229204c3173d61d9fd50725be7d347e60bb"},"cell_type":"code","source":"pd.crosstab(train['No_name'], train['AdoptionSpeed'], normalize='index')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9ccbfd235c0f1f1c49a28a59380a12ce4bba80bb"},"cell_type":"markdown","source":"- Less than 10% of pets don't have names, but they have a higher possibility of not being adopted.\n- 10%以下のペットは名前を持っていない。しかし名前のないペットたちは、高い確率で里親が見つからない。"},{"metadata":{"trusted":true,"_uuid":"1d91e8d3b3233dd60f7d4f2edd284bc24df2d63d","_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize=(18, 8));\nmake_count_plot(df=all_data.loc[all_data['dataset_type'] == 'train'], x='No_name', title='and having a name')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"441b5440d73f116986cffe7700e2954ce5116b51"},"cell_type":"markdown","source":"#### \"Bad\" names（悪い名前）\n\n- I have noticed that shorter names tend to be meaningless. Here is an example of some names with 3 characters.\n- 短い名前は意味がない傾向にあることに気が付いた。3文字の名前の霊をここにあげる"},{"metadata":{"trusted":true,"_uuid":"ec8345055a0a8c0b8f1cdbecfeee07d4fdf3b4ba"},"cell_type":"code","source":"all_data[all_data['Name'].apply(lambda x: len(str(x))) == 3]['Name'].value_counts().tail()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"63cb97e8bb01902129db7bddb9e9207a7f8e9a7b"},"cell_type":"markdown","source":"- And here are names with 1 or 2 characters...\n- 1から2文字の名前の霊は以下である"},{"metadata":{"trusted":true,"_uuid":"c62f444899bc2a956d582de306b1c93c447b0293"},"cell_type":"code","source":"all_data[all_data['Name'].apply(lambda x: len(str(x))) < 3]['Name'].unique()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"74fc6b4f2832b522601d1cd78fbeb99924eec4fb"},"cell_type":"markdown","source":"- I think that we could create a new feature, showing that name is meaningless - pets with these names could have less success in adoption.\n- 名前の意味のなさを表す特徴量を作れる可能性がある。意味のない名前を持つペットは里親が見つかる可能性が低いためだ。"},{"metadata":{"_uuid":"89eb4bc124932ffc778bb954b1045b2878a56cd9"},"cell_type":"markdown","source":"### Age（年齢）"},{"metadata":{"trusted":true,"_uuid":"b2bf602e384fefb192a91f2ded29e902df5b1483","_kg_hide-input":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize = (16, 6))\nplt.subplot(1, 2, 1)\nplt.title('Distribution of pets age');\ntrain['Age'].plot('hist', label='train');\ntest['Age'].plot('hist', label='test');\nplt.legend();\n\nplt.subplot(1, 2, 2)\nplt.title('Distribution of pets age (log)');\nnp.log1p(train['Age']).plot('hist', label='train');\nnp.log1p(test['Age']).plot('hist', label='test');\nplt.legend();","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f931e84ad40d1acdf65b469a0eac3c7dbc279b79"},"cell_type":"code","source":"train['Age'].value_counts().head(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f34bcab946dfbed17f83358055e2a2f141129b8e"},"cell_type":"markdown","source":"- We can see that most pets are young - maybe after the birth. Also there a lot of pets with an age equal to multiples of 12 - I think than owners didn't bother with the exact age.\n- ほとんどのペットは若いことがわかる。産まれてすぐ、または12の倍数ののペットも多く存在する。里親は正確な年齢は気にしないと考える。"},{"metadata":{"trusted":true,"_uuid":"0882a131ea8dc53772aaa77a32ae39eaebad4f7b","_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize=(10, 6));\nsns.violinplot(x=\"AdoptionSpeed\", y=\"Age\", hue=\"Type\", data=train);\nplt.title('AdoptionSpeed by Type and age');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8aeb4bcc7295e6e5e47b12b6da135ea2e81b295f","_kg_hide-input":true},"cell_type":"code","source":"data = []\nfor a in range(5):\n    df = train.loc[train['AdoptionSpeed'] == a]\n\n    data.append(go.Scatter(\n        x = df['Age'].value_counts().sort_index().index,\n        y = df['Age'].value_counts().sort_index().values,\n        name = str(a)\n    ))\n    \nlayout = go.Layout(dict(title = \"AdoptionSpeed trends by Age\",\n                  xaxis = dict(title = 'Age (months)'),\n                  yaxis = dict(title = 'Counts'),\n                  )\n                  )\npy.iplot(dict(data=data, layout=layout), filename='basic-line')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b5dd32edf666aded299493fe341d2bf387fd15ef"},"cell_type":"markdown","source":"* We can see that young pets are adopted quite fast and most of them are adopted;\n* 若いペットはすぐに、且つ高い確率で里親が見つかる\n* Most pets are less than 4 months old with a huge spike at 2 months;\n* ほとんどのペットは生後4ヶ月以下で2ヶ月目に大きなスパイクがある\n* It seems that a lot of people don't input exact age and write age in years (or multiples of 12);\n* ほとんどの人は正確な齢を入力しておらず、年齢を入れている場合もある\n* It could make sense to create some binary variables based on age;\n* 年齢に関するバイナリの変数を作成する必要がありそうである。"},{"metadata":{"_uuid":"0a98ebc0ed3511b8661d01f385af1500830dec97"},"cell_type":"markdown","source":"### Breeds（品種）\n- There is a main breed of the pet and secondary if relevant.\n- 主となる品種と二次品種がある\n\n- At first let's see whether having secondary breed influences adoption speed.\n- はじめに二次品種を持つことが「里親が見つかること」に影響するのかを見ていく。"},{"metadata":{"trusted":true,"_uuid":"d881bcc7cb65c714ca61e55b70ee1c9fa2bb4b54","_kg_hide-input":true},"cell_type":"code","source":"train['Pure_breed'] = 0\ntrain.loc[train['Breed2'] == 0, 'Pure_breed'] = 1\ntest['Pure_breed'] = 0\ntest.loc[test['Breed2'] == 0, 'Pure_breed'] = 1\nall_data['Pure_breed'] = 0\nall_data.loc[all_data['Breed2'] == 0, 'Pure_breed'] = 1\n\nprint(f\"Rate of pure breed pets in train data: {train['Pure_breed'].sum() * 100 / train['Pure_breed'].shape[0]:.4f}%.\")\nprint(f\"Rate of pure breed pets in test data: {test['Pure_breed'].sum() * 100 / test['Pure_breed'].shape[0]:.4f}%.\")","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"3a4d0fbca9b371b56e940ff4576bc9a5fa55c089"},"cell_type":"code","source":"def plot_four_graphs(col='', main_title='', dataset_title=''):\n    \"\"\"\n    Plotting four graphs:\n    - adoption speed by variable;\n    - counts of categories in the variable in train and test;\n    - adoption speed by variable for dogs;\n    - adoption speed by variable for cats;    \n    \"\"\"\n    plt.figure(figsize=(20, 12));\n    plt.subplot(2, 2, 1)\n    make_count_plot(df=train, x=col, title=f'and {main_title}')\n\n    plt.subplot(2, 2, 2)\n    sns.countplot(x='dataset_type', data=all_data, hue=col);\n    plt.title(dataset_title);\n\n    plt.subplot(2, 2, 3)\n    make_count_plot(df=train.loc[train['Type'] == 1], x=col, title=f'and {main_title} for dogs')\n\n    plt.subplot(2, 2, 4)\n    make_count_plot(df=train.loc[train['Type'] == 2], x=col, title=f'and {main_title} for cats')\n    \nplot_four_graphs(col='Pure_breed', main_title='having pure breed', dataset_title='Number of pets by pure/not-pure breed in train and test data')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"958ca3062f38c5ef33ec6b01bb39d0a4c91e810d"},"cell_type":"markdown","source":"- It seems that non-pure breed pets tend to be adopted more and faster, especially cats.\n- 純血のペットの方がより多く、且つ早く里親が見つかる傾向にあり、それは特に猫に置いて顕著に出ている\n- Let's look at the breeds themselves\n- 品種そのものについて観察していく"},{"metadata":{"trusted":true,"_uuid":"a1e930d8f1bb50ec1735adbd05c1f22f3d13ab51"},"cell_type":"code","source":"breeds_dict = {k: v for k, v in zip(breeds['BreedID'], breeds['BreedName'])}","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"637edff87ab2b77e500782e779b3732a109009c5"},"cell_type":"code","source":"train['Breed1_name'] = train['Breed1'].apply(lambda x: '_'.join(breeds_dict[x].split()) if x in breeds_dict else 'Unknown')\ntrain['Breed2_name'] = train['Breed2'].apply(lambda x: '_'.join(breeds_dict[x]) if x in breeds_dict else '-')\n\ntest['Breed1_name'] = test['Breed1'].apply(lambda x: '_'.join(breeds_dict[x].split()) if x in breeds_dict else 'Unknown')\ntest['Breed2_name'] = test['Breed2'].apply(lambda x: '_'.join(breeds_dict[x].split()) if x in breeds_dict else '-')\n\nall_data['Breed1_name'] = all_data['Breed1'].apply(lambda x: '_'.join(breeds_dict[x].split()) if x in breeds_dict else 'Unknown')\nall_data['Breed2_name'] = all_data['Breed2'].apply(lambda x: '_'.join(breeds_dict[x].split()) if x in breeds_dict else '-')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dd739c34bf9311ca357159ec978a7201a6b436dd","_kg_hide-input":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize = (20, 18))\nplt.subplot(2, 2, 1)\ntext_cat1 = ' '.join(all_data.loc[all_data['Type'] == 'Cat', 'Breed1_name'].fillna('').values)\nwordcloud = WordCloud(max_font_size=None, background_color='black', collocations=False,\n                      width=1200, height=1000).generate(text_cat1)\nplt.imshow(wordcloud)\nplt.title('Top cat breed1')\nplt.axis(\"off\")\n\nplt.subplot(2, 2, 2)\ntext_dog1 = ' '.join(all_data.loc[all_data['Type'] == 'Dog', 'Breed1_name'].fillna('').values)\nwordcloud = WordCloud(max_font_size=None, background_color='black', collocations=False,\n                      width=1200, height=1000).generate(text_dog1)\nplt.imshow(wordcloud)\nplt.title('Top dog breed1')\nplt.axis(\"off\")\n\nplt.subplot(2, 2, 3)\ntext_cat2 = ' '.join(all_data.loc[all_data['Type'] == 'Cat', 'Breed2_name'].fillna('').values)\nwordcloud = WordCloud(max_font_size=None, background_color='black', collocations=False,\n                      width=1200, height=1000).generate(text_cat2)\nplt.imshow(wordcloud)\nplt.title('Top cat breed2')\nplt.axis(\"off\")\n\nplt.subplot(2, 2, 4)\ntext_dog2 = ' '.join(all_data.loc[all_data['Type'] == 'Dog', 'Breed2_name'].fillna('').values)\nwordcloud = WordCloud(max_font_size=None, background_color='black', collocations=False,\n                      width=1200, height=1000).generate(text_dog2)\nplt.imshow(wordcloud)\nplt.title('Top dog breed2')\nplt.axis(\"off\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e7c9ec2abb3d60a8ec682063052f62acd762f463"},"cell_type":"markdown","source":"- It seems that not all values of these features are really breeds. Sometimes people simply write that the dogs has a mixed breed, cats often are described as domestic with certain hair length.\n- これらの特徴量の値がすべて本当の品種を表しているわけではない。時々、単純に雑種と書いている人もいるし、猫の場合は、家の長さとdomesticと表現されていることがある。\n\n- Now let's have a look at the combinations of breed names.\n- 品種名の組み合わせについて見ていく。"},{"metadata":{"trusted":true,"_uuid":"b21482f03c2bf3bb2eba684147dda7d777013762"},"cell_type":"code","source":"(all_data['Breed1_name'] + '__' + all_data['Breed2_name']).value_counts().head(15)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aff14ca84a79dddda64b202e628e8bb7dbb32207"},"cell_type":"markdown","source":"- It seems that most dogs aren't pure breeds, but mixed breeds! My first assumption was wrong.\n- ほとんどの犬は純血ではなく、雑種である。私のはじめの仮説は間違っていた。\n- Sometimes people write \"mixed breed\" in the first fiels, sometimes in both, and sometimes main breed is in the first field and is marked as mixed breed in the second field.\n- 初めの欄に雑種、もしくは両方の欄\n\nI think we can create new features based on this information. And later we can verify the hair length of pets."},{"metadata":{"_uuid":"56bdade2a8b9c3ea1d282f23a5801df80add345e"},"cell_type":"markdown","source":"### Gender（性別）\n - 1 = Male, 2 = Female, 3 = Mixed, if profile represents group of pets\n - 1 = オス、2 = メス、3 = 混合、プロファイルはペットのグループを表している。"},{"metadata":{"trusted":true,"_uuid":"181f0075c215c25c0b6346e4468dd47b1ccb7d72","_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize=(18, 6));\nplt.subplot(1, 2, 1)\nmake_count_plot(df=train, x='Gender', title='and gender')\n\nplt.subplot(1, 2, 2)\nsns.countplot(x='dataset_type', data=all_data, hue='Gender');\nplt.title('Number of pets by gender in train and test data');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f70e78d4cef95fe9058b7de6ae30093d39559bbd","_kg_hide-input":true},"cell_type":"code","source":"sns.factorplot('Type', col='Gender', data=all_data, kind='count', hue='dataset_type');\nplt.subplots_adjust(top=0.8)\nplt.suptitle('Count of cats and dogs in train and test set by gender');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a82b61384813d4208c8cbf9e68e79b58074ec9ee"},"cell_type":"markdown","source":"- It seems that male pets are adopted faster than female. Having no information about the gender really decreases chances.\n- オスはメスよりも早く里親が見つかると思われる。性別情報がない場合は可能性が減少する。"},{"metadata":{"trusted":true,"_uuid":"53fa658b6224b3dfcd7836fe866ac7866f0dad9f"},"cell_type":"markdown","source":"### Colors（色）"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"5183e7f289e686e17c05d6f91f2d7e4529640684"},"cell_type":"code","source":"colors_dict = {k: v for k, v in zip(colors['ColorID'], colors['ColorName'])}\ntrain['Color1_name'] = train['Color1'].apply(lambda x: colors_dict[x] if x in colors_dict else '')\ntrain['Color2_name'] = train['Color2'].apply(lambda x: colors_dict[x] if x in colors_dict else '')\ntrain['Color3_name'] = train['Color3'].apply(lambda x: colors_dict[x] if x in colors_dict else '')\n\ntest['Color1_name'] = test['Color1'].apply(lambda x: colors_dict[x] if x in colors_dict else '')\ntest['Color2_name'] = test['Color2'].apply(lambda x: colors_dict[x] if x in colors_dict else '')\ntest['Color3_name'] = test['Color3'].apply(lambda x: colors_dict[x] if x in colors_dict else '')\n\nall_data['Color1_name'] = all_data['Color1'].apply(lambda x: colors_dict[x] if x in colors_dict else '')\nall_data['Color2_name'] = all_data['Color2'].apply(lambda x: colors_dict[x] if x in colors_dict else '')\nall_data['Color3_name'] = all_data['Color3'].apply(lambda x: colors_dict[x] if x in colors_dict else '')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"ed0156512e6f8b8d22d02a74bc46849761b6c4ca"},"cell_type":"code","source":"def make_factor_plot(df, x, col, title, main_count=main_count, hue=None, ann=True, col_wrap=4):\n    \"\"\"\n    Plotting countplot.\n    Making annotations is a bit more complicated, because we need to iterate over axes.\n    \"\"\"\n    if hue:\n        g = sns.factorplot(col, col=x, data=df, kind='count', col_wrap=col_wrap, hue=hue);\n    else:\n        g = sns.factorplot(col, col=x, data=df, kind='count', col_wrap=col_wrap);\n    plt.subplots_adjust(top=0.9);\n    plt.suptitle(title);\n    ax = g.axes\n    plot_dict = prepare_plot_dict(df, x, main_count)\n    if ann:\n        for a in ax:\n            for p in a.patches:\n                text = f\"{plot_dict[p.get_height()]:.0f}%\" if plot_dict[p.get_height()] < 0 else f\"+{plot_dict[p.get_height()]:.0f}%\"\n                a.annotate(text, (p.get_x() + p.get_width() / 2., p.get_height()),\n                     ha='center', va='center', fontsize=11, color='green' if plot_dict[p.get_height()] > 0 else 'red', rotation=0, xytext=(0, 10),\n                     textcoords='offset points')  ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fe07002920afa569d6a1339be28601aa09297ee4","_kg_hide-input":true},"cell_type":"code","source":"sns.factorplot('dataset_type', col='Type', data=all_data, kind='count', hue='Color1_name', palette=['Black', 'Brown', '#FFFDD0', 'Gray', 'Gold', 'White', 'Yellow']);\nplt.subplots_adjust(top=0.8)\nplt.suptitle('Counts of pets in datasets by main color');","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"55903a9cd5c54cfab6b8dfe7452b4e0456b26cb8"},"cell_type":"markdown","source":"- We can see that most common colors are black and brown. Interesting to notice that there are almost no gray or yellow dogs :)\n- 黒と茶色が最も頻度の高い色となっている。灰色や黄色の犬はほとんど存在しないことも興味深い点だ。\n\n- Now let's see whether colors influence adoption speed\n- どの色が里親が見つかる速さに影響があるのか調べる。"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"0b9879b5c9832d002c4d7472cb3989a12a2a1a40"},"cell_type":"code","source":"make_factor_plot(df=train, x='Color1_name', col='AdoptionSpeed', title='Counts of pets by main color and Adoption Speed')","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"fd0a21adc084dad9d53865f18bedc844a7d21539"},"cell_type":"code","source":"train['full_color'] = (train['Color1_name'] + '__' + train['Color2_name'] + '__' + train['Color3_name']).str.replace('__', '')\ntest['full_color'] = (test['Color1_name'] + '__' + test['Color2_name'] + '__' + test['Color3_name']).str.replace('__', '')\nall_data['full_color'] = (all_data['Color1_name'] + '__' + all_data['Color2_name'] + '__' + all_data['Color3_name']).str.replace('__', '')\n\nmake_factor_plot(df=train.loc[train['full_color'].isin(list(train['full_color'].value_counts().index)[:12])], x='full_color', col='AdoptionSpeed', title='Counts of pets by color and Adoption Speed')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"10e2273417de407e4fb104cd10a813ba0f5429a9"},"cell_type":"markdown","source":"- We can see that there are some differences based on color, but the number of pets in most colors isn't very high, so this could be due to randomness.\n- 色による違いはいくつかある、しかし大抵の色のペット数は少ないため、これはたまたまであると考える。"},{"metadata":{"trusted":true,"_uuid":"7fba74b81cd002cf2b867b586c7d2f69c6cdbc45"},"cell_type":"code","source":"gender_dict = {1: 'Male', 2: 'Female', 3: 'Mixed'}\nfor i in all_data['Type'].unique():\n    for j in all_data['Gender'].unique():\n        df = all_data.loc[(all_data['Type'] == i) & (all_data['Gender'] == j)]\n        top_colors = list(df['full_color'].value_counts().index)[:5]\n        j = gender_dict[j]\n        print(f\"Most popular colors of {j} {i}s: {' '.join(top_colors)}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"783219e4add952b649a21f60129561594dd881e4"},"cell_type":"markdown","source":"### MatiritySize（成熟サイズ）\n- Size at maturity (1 = Small, 2 = Medium, 3 = Large, 4 = Extra Large, 0 = Not Specified)\n- 大きさ"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"4177da0415552a52ecc1a26a9f6811d11bbdda4a"},"cell_type":"code","source":"plot_four_graphs(col='MaturitySize', main_title='MaturitySize', dataset_title='Number of pets by MaturitySize in train and test data')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3b11d2a8afbb3bfd89643e38b3e29b06f90c1edf"},"cell_type":"code","source":"make_factor_plot(df=all_data, x='MaturitySize', col='Type', title='Count of cats and dogs in train and test set by MaturitySize', hue='dataset_type', ann=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"31159a85ecb67be76bd9955b1c8fa39288720d5a","_kg_hide-input":true},"cell_type":"code","source":"images = [i.split('-')[0] for i in os.listdir('../input/train_images/')]\nsize_dict = {1: 'Small', 2: 'Medium', 3: 'Large', 4: 'Extra Large'}\nfor t in all_data['Type'].unique():\n    for m in all_data['MaturitySize'].unique():\n        df = all_data.loc[(all_data['Type'] == t) & (all_data['MaturitySize'] == m)]\n        top_breeds = list(df['Breed1_name'].value_counts().index)[:5]\n        m = size_dict[m]\n        print(f\"Most common Breeds of {m} {t}s:\")\n        \n        fig = plt.figure(figsize=(25, 4))\n        \n        for i, breed in enumerate(top_breeds):\n            # excluding pets without pictures\n            b_df = df.loc[(df['Breed1_name'] == breed) & (df['PetID'].isin(images)), 'PetID']\n            if len(b_df) > 1:\n                pet_id = b_df.values[1]\n            else:\n                pet_id = b_df.values[0]\n            ax = fig.add_subplot(1, 5, i+1, xticks=[], yticks=[])\n\n            im = Image.open(\"../input/train_images/\" + pet_id + '-1.jpg')\n            plt.imshow(im)\n            ax.set_title(f'Breed: {breed}')\n        plt.show();","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7d9e4759c640f24c6135387acf8e8742a6d97499"},"cell_type":"markdown","source":"Quite interesting:\n* We can see that maturity size isn't very important. Medium sized pets are most common and they have slightly more chances to be not adopted;\n* There are almost no Extra Large pets. I hope it means that their owners like them and there is no need for them to be adopted :)\n* I wanted to gave a look at different pets, so I showed examples of pictures of most common breeds for each maturity size of cats and dogs;\n* I think not all data is entirely correct: sometimes short haired cats have breed with \"medium hair\", not sure that all breeds are entirely correct. Some photoes have bad quality;"},{"metadata":{"trusted":true,"_uuid":"fa5fada835211e4ee296af3f4d62a38cf9bb398a"},"cell_type":"markdown","source":"### FurLength\n\n (1 = Short, 2 = Medium, 3 = Long, 0 = Not Specified)"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"0c1d309ef612f2b821a444fee1e58c17feb95cc5"},"cell_type":"code","source":"plot_four_graphs(col='FurLength', main_title='FurLength', dataset_title='Number of pets by FurLength in train and test data')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"054f00a5304987efd024fd88da21a5d9d12ff4ec"},"cell_type":"markdown","source":"* We can see that most of the pets have short fur and long fur is the least common;\n* Pets with long hair tend to have a higher chance of being adopted. Though it could be because of randomness due to low count;\n\nAs I said earlier, some breed have hair length in the text, let's check these values!"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"9b8382cc28ae0069adea06df4eca3bf11fb91c07"},"cell_type":"code","source":"fig, ax = plt.subplots(figsize = (20, 18))\nplt.subplot(2, 2, 1)\ntext_cat1 = ' '.join(all_data.loc[(all_data['FurLength'] == 1) & (all_data['Type'] == 'Cat'), 'Breed1_name'].fillna('').values)\nwordcloud = WordCloud(max_font_size=None, background_color='white', collocations=False,\n                      width=1200, height=1000).generate(text_cat1)\nplt.imshow(wordcloud)\nplt.title('Top cat breed1 with short fur')\nplt.axis(\"off\")\n\nplt.subplot(2, 2, 2)\ntext_dog1 = ' '.join(all_data.loc[(all_data['FurLength'] == 1) & (all_data['Type'] == 'Dog'), 'Breed1_name'].fillna('').values)\nwordcloud = WordCloud(max_font_size=None, background_color='white', collocations=False,\n                      width=1200, height=1000).generate(text_dog1)\nplt.imshow(wordcloud)\nplt.title('Top dog breed1 with short fur')\nplt.axis(\"off\")\n\nplt.subplot(2, 2, 3)\ntext_cat2 = ' '.join(all_data.loc[(all_data['FurLength'] == 2) & (all_data['Type'] == 'Cat'), 'Breed1_name'].fillna('').values)\nwordcloud = WordCloud(max_font_size=None, background_color='white', collocations=False,\n                      width=1200, height=1000).generate(text_cat2)\nplt.imshow(wordcloud)\nplt.title('Top cat breed1 with medium fur')\nplt.axis(\"off\")\n\nplt.subplot(2, 2, 4)\ntext_dog2 = ' '.join(all_data.loc[(all_data['FurLength'] == 2) & (all_data['Type'] == 'Dog'), 'Breed1_name'].fillna('').values)\nwordcloud = WordCloud(max_font_size=None, background_color='white', collocations=False,\n                      width=1200, height=1000).generate(text_dog2)\nplt.imshow(wordcloud)\nplt.title('Top dog breed2 with medium fur')\nplt.axis(\"off\")\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d7217972bb283763ab900f27d24b64a78ac37e20"},"cell_type":"code","source":"c = 0\nstrange_pets = []\nfor i, row in all_data[all_data['Breed1_name'].str.contains('air')].iterrows():\n    if 'Short' in row['Breed1_name'] and row['FurLength'] == 1:\n        pass\n    elif 'Medium' in row['Breed1_name'] and row['FurLength'] == 2:\n        pass\n    elif 'Long' in row['Breed1_name'] and row['FurLength'] == 3:\n        pass\n    else:\n        c += 1\n        strange_pets.append((row['PetID'], row['Breed1_name'], row['FurLength']))\n        \nprint(f\"There are {c} pets whose breed and fur length don't match\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e6564e8ec0301db1b3e43d228eaae6b578a64c9c"},"cell_type":"markdown","source":"It seems that almost one thousand pets have mismatch in breeds and fur lengths. Let's see!"},{"metadata":{"_kg_hide-output":false,"trusted":true,"_uuid":"7ac3b519ffb4abeb0d2648ad1639700cdbe26254","_kg_hide-input":true},"cell_type":"code","source":"strange_pets = [p for p in strange_pets if p[0] in images]\nfig = plt.figure(figsize=(25, 12))\nfur_dict = {1: 'Short', 2: 'Medium', 3: 'long'}\nfor i, s in enumerate(random.sample(strange_pets, 12)):\n    ax = fig.add_subplot(3, 4, i+1, xticks=[], yticks=[])\n\n    im = Image.open(\"../input/train_images/\" + s[0] + '-1.jpg')\n    plt.imshow(im)\n    ax.set_title(f'Breed: {s[1]} \\n Fur length: {fur_dict[s[2]]}')\nplt.show();","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"348e99f3ba2817ff981e1a21ee7d491089ec5fd0"},"cell_type":"markdown","source":"Everybody lies!\n\nSometimes breed is more correct, sometimes fur length... I suppose we could create a feature showing whether breed and fur length match."},{"metadata":{"_uuid":"906ae8c4cc259edca8130d320bd179b409d56469"},"cell_type":"markdown","source":"### Health\n\nThere are four features showing health of the pets:\n\n* Vaccinated - Pet has been vaccinated (1 = Yes, 2 = No, 3 = Not Sure)\n* Dewormed - Pet has been dewormed (1 = Yes, 2 = No, 3 = Not Sure)\n* Sterilized - Pet has been spayed / neutered (1 = Yes, 2 = No, 3 = Not Sure)\n* Health - Health Condition (1 = Healthy, 2 = Minor Injury, 3 = Serious Injury, 0 = Not Specified)\n\nI think that these features are very important - most people would prefer a healthy pet. While sterilization isn't the main concern, having healty and dewormed pet should have a great importance. Let's see whether I'm right!"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"6b367f78ca433579f61d06411e5b361c41342cf5"},"cell_type":"code","source":"plt.figure(figsize=(20, 12));\nplt.subplot(2, 2, 1)\nmake_count_plot(df=train, x='Vaccinated', title='Vaccinated')\nplt.xticks([0, 1, 2], ['Yes', 'No', 'Not sure']);\nplt.title('AdoptionSpeed and Vaccinated');\n\nplt.subplot(2, 2, 2)\nmake_count_plot(df=train, x='Dewormed', title='Dewormed')\nplt.xticks([0, 1, 2], ['Yes', 'No', 'Not sure']);\nplt.title('AdoptionSpeed and Dewormed');\n\nplt.subplot(2, 2, 3)\nmake_count_plot(df=train, x='Sterilized', title='Sterilized')\nplt.xticks([0, 1, 2], ['Yes', 'No', 'Not sure']);\nplt.title('AdoptionSpeed and Sterilized');\n\nplt.subplot(2, 2, 4)\nmake_count_plot(df=train, x='Health', title='Health')\nplt.xticks([0, 1, 2], ['Healthy', 'Minor Injury', 'Serious Injury']);\nplt.title('AdoptionSpeed and Health');\n\nplt.suptitle('Adoption Speed and health conditions');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"91d081672c5db05488a1ef84871834528ab1d6ea"},"cell_type":"markdown","source":"* Almost all pets are healthy! Pets with minor injuries are rare and sadly they aren't adopted well. Number of pets with serious injuries is negligible.\n* It is interesting that people prefer non-vaccinated pets. Maybe they want to bring pets to vets themselves...\n* People also prefer non-sterilized pets! Maybe they want puppies/kittens :)\n* Quite important is the fact that when there is no information about health condition, the probability of not being adopted is much higher;\n\nLet's have a look at most popular health conditions."},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"299e53c93240a700e706e8ea7b4ae4b103874490","_kg_hide-input":true},"cell_type":"code","source":"train['health'] = train['Vaccinated'].astype(str) + '_' + train['Dewormed'].astype(str) + '_' + train['Sterilized'].astype(str) + '_' + train['Health'].astype(str)\ntest['health'] = test['Vaccinated'].astype(str) + '_' + test['Dewormed'].astype(str) + '_' + test['Sterilized'].astype(str) + '_' + test['Health'].astype(str)\n\n\nmake_factor_plot(df=train.loc[train['health'].isin(list(train.health.value_counts().index[:5]))], x='health', col='AdoptionSpeed', title='Counts of pets by main health conditions and Adoption Speed')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a78d5e1ac3afc054c096c46114eb8fd08e8d410f"},"cell_type":"markdown","source":"* Healthy, dewormed and non-sterilized pets tend to be adopted faster!\n* Completely healthy pets are... more likely to be not adopted! I suppose that means that a lot of people pay attention to other characteristics;\n* And healthy pets with no information (not sure value) also tend to be adopted less frequently. Maybe people prefer having information, even if it is negative;"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"c10a37f98966145d1780897159bdaf8349d66c86"},"cell_type":"code","source":"plt.figure(figsize=(20, 16))\nplt.subplot(3, 2, 1)\nsns.violinplot(x=\"AdoptionSpeed\", y=\"Age\", data=train);\nplt.title('Age distribution by Age');\nplt.subplot(3, 2, 3)\nsns.violinplot(x=\"AdoptionSpeed\", y=\"Age\", hue=\"Vaccinated\", data=train);\nplt.title('Age distribution by Age and Vaccinated');\nplt.subplot(3, 2, 4)\nsns.violinplot(x=\"AdoptionSpeed\", y=\"Age\", hue=\"Dewormed\", data=train);\nplt.title('Age distribution by Age and Dewormed');\nplt.subplot(3, 2, 5)\nsns.violinplot(x=\"AdoptionSpeed\", y=\"Age\", hue=\"Sterilized\", data=train);\nplt.title('Age distribution by Age and Sterilized');\nplt.subplot(3, 2, 6)\nsns.violinplot(x=\"AdoptionSpeed\", y=\"Age\", hue=\"Health\", data=train);\nplt.title('Age distribution by Age and Health');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c07691860afd4035a7c33b3e45ed1c0c21d59011"},"cell_type":"markdown","source":"### Quantity\nSometimes there are several pets in one advertisement."},{"metadata":{"trusted":true,"_uuid":"1cfb828f5854393030b87974f7e6f2d24bdd24d5"},"cell_type":"code","source":"train.loc[train['Quantity'] > 11][['Name', 'Description', 'Quantity', 'AdoptionSpeed']].head(10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b0ea6c320c15bd2c0dcbd17e1a4d9f08c3cf7654"},"cell_type":"code","source":"train['Quantity'].value_counts().head(10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"01f5574b61acf106b19c2855099483e5f711228f"},"cell_type":"markdown","source":"Sometimes there is a huge amount of pets in some advertisements! But at the same time sometimes text and the quantity don't match. For example:\n\n    Pancho and Tita are 2 adorable, playful kittens. They can be shy at first but once they get to know you they are the sweetest pets anyone could ask for. Available for adoption now. They are very, very close so we are looking for someone who can take them both.\n    \n Obvously there are only two kittens, but the quantity is 12 for some reason.\n \n One thing worth noticing that sometimes all these pet are adopted which is great!\n \n For the sake of plotting I'll create a new variable, where 6 pets in one advertizement will the the max amount."},{"metadata":{"trusted":true,"_uuid":"e9b21e72515a61989ed3d4411ad3d95b62e21719","_kg_hide-input":true},"cell_type":"code","source":"train['Quantity_short'] = train['Quantity'].apply(lambda x: x if x <= 5 else 6)\ntest['Quantity_short'] = test['Quantity'].apply(lambda x: x if x <= 5 else 6)\nall_data['Quantity_short'] = all_data['Quantity'].apply(lambda x: x if x <= 5 else 6)\nplot_four_graphs(col='Quantity_short', main_title='Quantity_short', dataset_title='Number of pets by Quantity_short in train and test data')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b992a49cb96e8d93e1bce3308bbbb4a59b9bcfe1"},"cell_type":"markdown","source":"It seems that quantity has little to do with adoption speed. This is good, it means that abandoned cats/dogs with kittens/puppies have chances of being adopted! Though it seems that single cats have somewhat higher chances that single dogs."},{"metadata":{"trusted":true,"_uuid":"9a3b8c45c502a6b777214583b268176a77cbd8b8"},"cell_type":"markdown","source":"### Fee\nOne of interesting features is adoption fee. Some pets can be gotten for free, adopting some required paying a certain amount."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"e7591ff01f25232af38c73d0d96e53a99bfeb34f"},"cell_type":"code","source":"train['Free'] = train['Fee'].apply(lambda x: 'Free' if x == 0 else 'Not Free')\ntest['Free'] = test['Fee'].apply(lambda x: 'Free' if x == 0 else 'Not Free')\nall_data['Free'] = all_data['Fee'].apply(lambda x: 'Free' if x == 0 else 'Not Free')\nplot_four_graphs(col='Free', main_title='Free', dataset_title='Number of pets by Free in train and test data')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"88b34d852bff73eb5a72ab3c74acb4ac7cec08ae"},"cell_type":"markdown","source":"Most pets are free and it seems that asking for a fee slightly desreased the chance of adoption. Also free cats are adopted faster than free dogs"},{"metadata":{"trusted":true,"_uuid":"fc2836121b091a830569858250d013f1c0600676"},"cell_type":"code","source":"all_data.sort_values('Fee', ascending=False)[['Name', 'Description', 'Fee', 'AdoptionSpeed', 'dataset_type']].head(10)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"264d23cd4e7108893523d028c86333287024a347"},"cell_type":"code","source":"plt.figure(figsize=(16, 6));\nplt.subplot(1, 2, 1)\nplt.hist(train.loc[train['Fee'] < 400, 'Fee']);\nplt.title('Distribution of fees lower than 400');\n\nplt.subplot(1, 2, 2)\nsns.violinplot(x=\"AdoptionSpeed\", y=\"Fee\", hue=\"Type\", data=train);\nplt.title('AdoptionSpeed by Type and Fee');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dca7d0cc1eb204ae1c62b105c45b4bbf09722f04"},"cell_type":"markdown","source":"* It is interesting that pets with high fee tend to be adopted quite fast! Maybe people prefer to pay for \"better\" pets: healthy, trained and so on;\n* Most pets are given for free and fees are usually lower than 100 $;\n* Fees for dogs tend to be higher, though these are rare cases anyway."},{"metadata":{"trusted":true,"_uuid":"42f2447ed4197ca1e950b348846b2383cef7be0f"},"cell_type":"code","source":"plt.figure(figsize=(16, 10));\nsns.scatterplot(x=\"Fee\", y=\"Quantity\", hue=\"Type\",data=all_data);\nplt.title('Quantity of pets and Fee');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"97b35254458779abbad1f104ddca741a80322017"},"cell_type":"markdown","source":"It seems that fees and pet quantity have inversely proportional relationship. The less pets, the higher is the fee. I suppose these single pets are better trained and prepared than most others."},{"metadata":{"_uuid":"e69b241bbbe4e1d5ff929c698819deff4dfa1e0e"},"cell_type":"markdown","source":"### State"},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"71c41bfeb3acc076ba41307617f89788f98faddb"},"cell_type":"code","source":"states_dict = {k: v for k, v in zip(states['StateID'], states['StateName'])}\ntrain['State_name'] = train['State'].apply(lambda x: '_'.join(states_dict[x].split()) if x in states_dict else 'Unknown')\ntest['State_name'] = test['State'].apply(lambda x: '_'.join(states_dict[x].split()) if x in states_dict else 'Unknown')\nall_data['State_name'] = all_data['State'].apply(lambda x: '_'.join(states_dict[x].split()) if x in states_dict else 'Unknown')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b2962de32eacf5c3f1c292cfd60ce62797f34586"},"cell_type":"code","source":"all_data['State_name'].value_counts(normalize=True).head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"40fff8f5d21c513fcd46bea5b5c9d7e45ae1a8f4"},"cell_type":"markdown","source":"Sadly I don't know anything about Malaysia’s states, so I can only say that top three states account for ~90% of ads. Let's have a look at them."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"1ebe6886e3b92c62c71ff816125af2600d695b1f"},"cell_type":"code","source":"make_factor_plot(df=train.loc[train['State_name'].isin(list(train.State_name.value_counts().index[:3]))], x='State_name', col='AdoptionSpeed', title='Counts of pets by states and Adoption Speed')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6c26bbd4e4365406d2581d2ab37d86dd097e2890"},"cell_type":"markdown","source":"Intetestingly top-2 and top-3 states have lower rates of adoption."},{"metadata":{"_uuid":"057d283ff616d2b563c75cdf3f291362b632cfa8"},"cell_type":"markdown","source":"### Rescuer\nWe have unique hashes for resquers."},{"metadata":{"trusted":true,"_uuid":"483574c727e7f65d68e695617a821c833057e978"},"cell_type":"code","source":"all_data['RescuerID'].value_counts().head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a2cfc9049c68cdb8c67be8b949d1b607fb5ca166"},"cell_type":"markdown","source":"Top-5 resquers managed a lot of pets!\nI wonder whether these are individual people or organizations. Let's have a look at them."},{"metadata":{"trusted":true,"_uuid":"141b33aa83e333b3aa4f908c9248854ae439994c"},"cell_type":"code","source":"make_factor_plot(df=train.loc[train['RescuerID'].isin(list(train.RescuerID.value_counts().index[:5]))], x='RescuerID', col='AdoptionSpeed', title='Counts of pets by rescuers and Adoption Speed', col_wrap=5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"04062628d1a72209c30400d35f363e812eaedbcd"},"cell_type":"markdown","source":"Wow! The resquer with the highest amount of resqued pets has the best adoption rate! On the other hand the third one has the worst rate :("},{"metadata":{"_uuid":"bd666d9ba51eec84858a71847fa2a0e3bde99838"},"cell_type":"markdown","source":"### VideoAmt"},{"metadata":{"trusted":true,"_uuid":"89e1bbe4f8e738f2ade6fed5ed0c3c82d1b040d5"},"cell_type":"code","source":"train['VideoAmt'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"176bc84b64a5d61b9eb98b0fbdc299a8471381f2"},"cell_type":"markdown","source":"Hm. In most cases there are no videos at all. Sometimes there is one video, more than one video is quite rare. We don't have videos and considering a huge disbalance in values I'm not sure this variable will be useful."},{"metadata":{"_uuid":"e8836c571f74b3989d9a1ae5bfc15ed550d57990"},"cell_type":"markdown","source":"### PhotoAmt"},{"metadata":{"trusted":true,"scrolled":true,"_uuid":"f8badaeb2e79f0050d72f90c94397c7a8f1842c4"},"cell_type":"code","source":"print(F'Maximum amount of photos in {train[\"PhotoAmt\"].max()}')\ntrain['PhotoAmt'].value_counts().head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"435b3cca2715fff5d00d4b43a1e613a33d5d29e6"},"cell_type":"code","source":"make_factor_plot(df=train.loc[train['PhotoAmt'].isin(list(train.PhotoAmt.value_counts().index[:5]))], x='PhotoAmt', col='AdoptionSpeed', title='Counts of pets by PhotoAmt and Adoption Speed', col_wrap=5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"15828acbcc3497661de2cfaa21900ceca2d3ff34"},"cell_type":"code","source":"plt.figure(figsize=(16, 6));\nplt.subplot(1, 2, 1)\nplt.hist(train['PhotoAmt']);\nplt.title('Distribution of PhotoAmt');\n\nplt.subplot(1, 2, 2)\nsns.violinplot(x=\"AdoptionSpeed\", y=\"PhotoAmt\", hue=\"Type\", data=train);\nplt.title('AdoptionSpeed by Type and PhotoAmt');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4c16b3a846d088db84e435fa200091ecfbf102ac"},"cell_type":"markdown","source":"Pets can have up to 30 photos! That's a lot! But I'm not convinced that amount of photoes has any real influence."},{"metadata":{"_uuid":"7d4f53076823b79dbcd682966791df6b97e34be2"},"cell_type":"markdown","source":"### Description\n\nDescription contains a lot of important information, let' analyze it!"},{"metadata":{"trusted":true,"_uuid":"e3e6878f01fa17bf368219e89df0d9199b99bd03"},"cell_type":"code","source":"fig, ax = plt.subplots(figsize = (12, 8))\ntext_cat = ' '.join(all_data['Description'].fillna('').values)\nwordcloud = WordCloud(max_font_size=None, background_color='white',\n                      width=1200, height=1000).generate(text_cat)\nplt.imshow(wordcloud)\nplt.title('Top words in description');\nplt.axis(\"off\");","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"995a87dd48d98da789414b19074470d4bb8151a5"},"cell_type":"markdown","source":"There are too many similar general words like \"cat\". We need to go deeper.\n\nLet's use ELI5 library for prediction explanation. I'll fit a basic vectorizer on desctriptions and build a simple Random Forest model. Then we will look at words which caused certain labels to be predicted."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"e2cc8749825baef6989ca7bc3113b61dcdcbe2d1"},"cell_type":"code","source":"tokenizer = TweetTokenizer()\nvectorizer = TfidfVectorizer(ngram_range=(1, 2), tokenizer=tokenizer.tokenize)\n\nvectorizer.fit(all_data['Description'].fillna('').values)\nX_train = vectorizer.transform(train['Description'].fillna(''))\n\nrf = RandomForestClassifier(n_estimators=20)\nrf.fit(X_train, train['AdoptionSpeed'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"53be7de4dba82fb3ac7f14498a713cf1bfc6b2e3"},"cell_type":"code","source":"for i in range(5):\n    print(f'Example of Adoption speed {i}')\n    text = train.loc[train['AdoptionSpeed'] == i, 'Description'].values[0]\n    print(text)\n    display(eli5.show_prediction(rf, doc=text, vec=vectorizer, top=10))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"44718d6d4468a88ce36af396fc1a5b590c5806fa"},"cell_type":"markdown","source":"Some words/phrases seem to be useful, but it seems that different adoption speed classes could have similar important words..."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"dedb813860df7549409694e0de219488a58c6de4"},"cell_type":"code","source":"train['Description'] = train['Description'].fillna('')\ntest['Description'] = test['Description'].fillna('')\nall_data['Description'] = all_data['Description'].fillna('')\n\ntrain['desc_length'] = train['Description'].apply(lambda x: len(x))\ntrain['desc_words'] = train['Description'].apply(lambda x: len(x.split()))\n\ntest['desc_length'] = test['Description'].apply(lambda x: len(x))\ntest['desc_words'] = test['Description'].apply(lambda x: len(x.split()))\n\nall_data['desc_length'] = all_data['Description'].apply(lambda x: len(x))\nall_data['desc_words'] = all_data['Description'].apply(lambda x: len(x.split()))","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"3bb458e11f583ac254f919543f63c714a81b9c45"},"cell_type":"code","source":"plt.figure(figsize=(16, 6));\nplt.subplot(1, 2, 1)\nsns.violinplot(x=\"AdoptionSpeed\", y=\"desc_length\", hue=\"Type\", data=train);\nplt.title('AdoptionSpeed by Type and description length');\n\nplt.subplot(1, 2, 2)\nsns.violinplot(x=\"AdoptionSpeed\", y=\"desc_words\", hue=\"Type\", data=train);\nplt.title('AdoptionSpeed by Type and count of words in description');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6cfaa461767bd287786cc51fe157f840e6b0bcd7"},"cell_type":"markdown","source":"Interestingly pets with short text in ads are adopted quickly. Or maybe longer descriptions mean more problems in the pets, therefore adoption speed is lower?"},{"metadata":{"_uuid":"43a1670ae93a7a704437fb8468b597bc68fe11a4"},"cell_type":"markdown","source":"### Sentiment\nWe have run each pet profile's description through Google's Natural Language API, providing analysis on sentiment and key entities. You may optionally utilize this supplementary information for your pet description analysis. There are some descriptions that the API could not analyze. As such, there are fewer sentiment files than there are rows in the dataset. "},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"7da27e65952a4b19d9e63de56facb17c6c74add7"},"cell_type":"code","source":"sentiment_dict = {}\nfor filename in os.listdir('../input/train_sentiment/'):\n    with open('../input/train_sentiment/' + filename, 'r') as f:\n        sentiment = json.load(f)\n    pet_id = filename.split('.')[0]\n    sentiment_dict[pet_id] = {}\n    sentiment_dict[pet_id]['magnitude'] = sentiment['documentSentiment']['magnitude']\n    sentiment_dict[pet_id]['score'] = sentiment['documentSentiment']['score']\n    sentiment_dict[pet_id]['language'] = sentiment['language']\n\nfor filename in os.listdir('../input/test_sentiment/'):\n    with open('../input/test_sentiment/' + filename, 'r') as f:\n        sentiment = json.load(f)\n    pet_id = filename.split('.')[0]\n    sentiment_dict[pet_id] = {}\n    sentiment_dict[pet_id]['magnitude'] = sentiment['documentSentiment']['magnitude']\n    sentiment_dict[pet_id]['score'] = sentiment['documentSentiment']['score']\n    sentiment_dict[pet_id]['language'] = sentiment['language']","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"409a1b79f29012e37b6d4e9bbe15fba1715ce6ee"},"cell_type":"code","source":"train['lang'] = train['PetID'].apply(lambda x: sentiment_dict[x]['language'] if x in sentiment_dict else 'no')\ntrain['magnitude'] = train['PetID'].apply(lambda x: sentiment_dict[x]['magnitude'] if x in sentiment_dict else 0)\ntrain['score'] = train['PetID'].apply(lambda x: sentiment_dict[x]['score'] if x in sentiment_dict else 0)\n\ntest['lang'] = test['PetID'].apply(lambda x: sentiment_dict[x]['language'] if x in sentiment_dict else 'no')\ntest['magnitude'] = test['PetID'].apply(lambda x: sentiment_dict[x]['magnitude'] if x in sentiment_dict else 0)\ntest['score'] = test['PetID'].apply(lambda x: sentiment_dict[x]['score'] if x in sentiment_dict else 0)\n\nall_data['lang'] = all_data['PetID'].apply(lambda x: sentiment_dict[x]['language'] if x in sentiment_dict else 'no')\nall_data['magnitude'] = all_data['PetID'].apply(lambda x: sentiment_dict[x]['magnitude'] if x in sentiment_dict else 0)\nall_data['score'] = all_data['PetID'].apply(lambda x: sentiment_dict[x]['score'] if x in sentiment_dict else 0)","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"657f352d865016a7d5fa9220d59fb1520ecca563"},"cell_type":"code","source":"plot_four_graphs(col='lang', main_title='lang', dataset_title='Number of pets by lang in train and test data')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2a4a735a9403cc409e081b77591bcd02bd29f9b6"},"cell_type":"markdown","source":"Well, English is the most common language by far, so language feature will hardly help."},{"metadata":{"trusted":true,"_uuid":"7e79ac98ddad0776a1f3d21538abd31a020ff5ad"},"cell_type":"code","source":"plt.figure(figsize=(16, 6));\nplt.subplot(1, 2, 1)\nsns.violinplot(x=\"AdoptionSpeed\", y=\"score\", hue=\"Type\", data=train);\nplt.title('AdoptionSpeed by Type and score');\n\nplt.subplot(1, 2, 2)\nsns.violinplot(x=\"AdoptionSpeed\", y=\"magnitude\", hue=\"Type\", data=train);\nplt.title('AdoptionSpeed by Type and magnitude of sentiment');","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b554e95c200d22a00a6f2c2d7c6c5ba52dc76e65"},"cell_type":"markdown","source":"It seems that the lower is the magnitude of score, the faster pets are adopted."},{"metadata":{"_uuid":"c90159a95d5952c4b077bac75d450b89010a4737"},"cell_type":"markdown","source":"### Basic model\n\nThere are much more interesting things in the dataset and I'm going to explore them, but for now let's build a simple model as a baseline."},{"metadata":{"trusted":true,"_uuid":"20b467c6f19c9eac301e233acbc2d5a0f3c32f09"},"cell_type":"code","source":"cols_to_use = ['Type', 'Age', 'Breed1', 'Breed2', 'Gender', 'Color1', 'Color2',\n       'Color3', 'MaturitySize', 'FurLength', 'Vaccinated', 'Dewormed',\n       'Sterilized', 'Health', 'Quantity', 'Fee', 'State', 'RescuerID',\n       'VideoAmt', 'PhotoAmt', 'AdoptionSpeed', 'No_name', 'Pure_breed', 'health', 'Free', 'desc_length', 'desc_words', 'score', 'magnitude']\ntrain = train[[col for col in cols_to_use if col in train.columns]]\ntest = test[[col for col in cols_to_use if col in test.columns]]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a4841f7e73af45347802752d59556ea3e9d4523f"},"cell_type":"code","source":"cat_cols = ['Type', 'Breed1', 'Breed2', 'Gender', 'Color1', 'Color2',\n       'Color3', 'MaturitySize', 'FurLength', 'Vaccinated', 'Dewormed',\n       'Sterilized', 'Health', 'State', 'RescuerID',\n       'No_name', 'Pure_breed', 'health', 'Free']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d136fe07e55e9b008f9856898e3d4a26e0f941ba"},"cell_type":"code","source":"more_cols = []\nfor col1 in cat_cols:\n    for col2 in cat_cols:\n        if col1 != col2 and col1 not in ['RescuerID', 'State'] and col2 not in ['RescuerID', 'State']:\n            train[col1 + '_' + col2] = train[col1].astype(str) + '_' + train[col2].astype(str)\n            test[col1 + '_' + col2] = test[col1].astype(str) + '_' + test[col2].astype(str)\n            more_cols.append(col1 + '_' + col2)\n            \ncat_cols = cat_cols + more_cols","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d5d341bb001c3ae0009d1362773840fe85b9609c"},"cell_type":"code","source":"%%time\nindexer = {}\nfor col in cat_cols:\n    # print(col)\n    _, indexer[col] = pd.factorize(train[col].astype(str))\n    \nfor col in tqdm_notebook(cat_cols):\n    # print(col)\n    train[col] = indexer[col].get_indexer(train[col].astype(str))\n    test[col] = indexer[col].get_indexer(test[col].astype(str))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"26d823fffcdd7830e9224a5c22de58b51502b9b8"},"cell_type":"code","source":"y = train['AdoptionSpeed']\ntrain = train.drop(['AdoptionSpeed'], axis=1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ccb79707784bf821af808fcf820611649078cce9"},"cell_type":"markdown","source":"## Naive multiclass LGB"},{"metadata":{"trusted":true,"_uuid":"9ac32335b35a7aa014fd87a1e9d0ac932f372459"},"cell_type":"code","source":"n_fold = 5\nfolds = StratifiedKFold(n_splits=n_fold, shuffle=True, random_state=15)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"64e95e47a595b47083bd6dd3d99ffebb53fc6443","_kg_hide-input":true},"cell_type":"code","source":"def train_model(X=train, X_test=test, y=y, params=None, folds=folds, model_type='lgb', plot_feature_importance=False, averaging='usual', make_oof=False):\n    result_dict = {}\n    if make_oof:\n        oof = np.zeros(len(X))\n    prediction = np.zeros(len(X_test))\n    scores = []\n    feature_importance = pd.DataFrame()\n    for fold_n, (train_index, valid_index) in enumerate(folds.split(X, y)):\n        gc.collect()\n        print('Fold', fold_n + 1, 'started at', time.ctime())\n        X_train, X_valid = X.iloc[train_index], X.iloc[valid_index]\n        y_train, y_valid = y.iloc[train_index], y.iloc[valid_index]\n        \n        \n        if model_type == 'lgb':\n            train_data = lgb.Dataset(X_train, label=y_train, categorical_feature = cat_cols)\n            valid_data = lgb.Dataset(X_valid, label=y_valid, categorical_feature = cat_cols)\n            \n            model = lgb.train(params,\n                    train_data,\n                    num_boost_round=2000,\n                    valid_sets = [train_data, valid_data],\n                    verbose_eval=100,\n                    early_stopping_rounds = 200)\n\n            del train_data, valid_data\n            \n            y_pred_valid = model.predict(X_valid, num_iteration=model.best_iteration).argmax(1)\n            del X_valid\n            gc.collect()\n            y_pred = model.predict(X_test, num_iteration=model.best_iteration).argmax(1)\n            \n        if model_type == 'xgb':\n            train_data = xgb.DMatrix(data=X_train, label=y_train)\n            valid_data = xgb.DMatrix(data=X_valid, label=y_valid)\n\n            watchlist = [(train_data, 'train'), (valid_data, 'valid_data')]\n            model = xgb.train(dtrain=train_data, num_boost_round=20000, evals=watchlist, early_stopping_rounds=200, verbose_eval=500, params=params)\n            y_pred_valid = model.predict(xgb.DMatrix(X_valid), ntree_limit=model.best_ntree_limit)\n            y_pred = model.predict(xgb.DMatrix(X_test), ntree_limit=model.best_ntree_limit)\n            \n        if model_type == 'lcv':\n            model = LogisticRegressionCV(scoring='neg_log_loss', cv=3, multi_class='multinomial')\n            model.fit(X_train, y_train)\n\n            y_pred_valid = model.predict(X_valid)\n            y_pred = model.predict(X_test)\n            \n        if model_type == 'cat':\n            model = CatBoostClassifier(iterations=20000,  loss_function='MultiClass', **params)\n            model.fit(X_train, y_train, eval_set=(X_valid, y_valid), cat_features=[], use_best_model=True, verbose=False)\n\n            y_pred_valid = model.predict(X_valid)\n            y_pred = model.predict(X_test).reshape(-1,)\n        \n        if make_oof:\n            oof[valid_index] = y_pred_valid.reshape(-1,)\n            \n        scores.append(kappa(y_valid, y_pred_valid))\n        print('Fold kappa:', kappa(y_valid, y_pred_valid))\n        print('')\n        \n        if averaging == 'usual':\n            prediction += y_pred\n        elif averaging == 'rank':\n            prediction += pd.Series(y_pred).rank().values\n        \n        if model_type == 'lgb':\n            # feature importance\n            fold_importance = pd.DataFrame()\n            fold_importance[\"feature\"] = X.columns\n            fold_importance[\"importance\"] = model.feature_importance()\n            fold_importance[\"fold\"] = fold_n + 1\n            feature_importance = pd.concat([feature_importance, fold_importance], axis=0)\n\n    prediction /= n_fold\n    \n    print('CV mean score: {0:.4f}, std: {1:.4f}.'.format(np.mean(scores), np.std(scores)))\n    \n    if model_type == 'lgb':\n        \n        if plot_feature_importance:\n            feature_importance[\"importance\"] /= n_fold\n            cols = feature_importance[[\"feature\", \"importance\"]].groupby(\"feature\").mean().sort_values(\n                by=\"importance\", ascending=False)[:50].index\n\n            best_features = feature_importance.loc[feature_importance.feature.isin(cols)]\n\n            plt.figure(figsize=(16, 12));\n            sns.barplot(x=\"importance\", y=\"feature\", data=best_features.sort_values(by=\"importance\", ascending=False));\n            plt.title('LGB Features (avg over folds)');\n            \n            result_dict['feature_importance'] = feature_importance\n            \n    result_dict['prediction'] = prediction\n    if make_oof:\n        result_dict['oof'] = oof\n    \n    return result_dict","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5af7d16419babb3af2c4b10f429aa65bc1bdd2b1"},"cell_type":"code","source":"params = {'num_leaves': 128,\n        #  'min_data_in_leaf': 60,\n         'objective': 'multiclass',\n         'max_depth': -1,\n         'learning_rate': 0.05,\n         \"boosting\": \"gbdt\",\n         \"feature_fraction\": 0.9,\n         \"bagging_freq\": 3,\n         \"bagging_fraction\": 0.9,\n         \"bagging_seed\": 11,\n        #  \"lambda_l1\": 0.1,\n         # \"lambda_l2\": 0.1,\n         \"random_state\": 42,          \n         \"verbosity\": -1,\n         \"num_class\": 5}","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"85db5cb12bad162e2c3a0f08c640382a1228d73d","scrolled":true},"cell_type":"code","source":"result_dict_lgb = train_model(X=train, X_test=test, y=y, params=params, model_type='lgb', plot_feature_importance=True, make_oof=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"73325a3247f9cafd3af95323de9d4933a82e4d0b"},"cell_type":"code","source":"xgb_params = {'eta': 0.01, 'max_depth': 10, 'subsample': 0.9, 'colsample_bytree': 0.9, \n          'objective': 'multi:softmax', 'eval_metric': 'merror', 'silent': True, 'nthread': 4, 'num_class': 5}\nresult_dict_xgb = train_model(params=xgb_params, model_type='xgb', make_oof=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e5fcff4424ad30b5dafd6fda1eaac811a69aef5b"},"cell_type":"code","source":"prediction = (result_dict_lgb['prediction'] + result_dict_xgb['prediction']) / 2\nsubmission = pd.DataFrame({'PetID': sub.PetID, 'AdoptionSpeed': [int(i) for i in prediction]})\nsubmission.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5f4dd1fbd5fa4e6ed3f4fa7ea140cb9197bf6f94"},"cell_type":"code","source":"submission.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}