{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"\nimport pandas as pd #Pandas is the most popular library for manipulating data. Think of it as an Excel but a million times faster and more practical.\nimport numpy as np # This library allows to easily carry out simple and complex mathematical operations.\nimport matplotlib.pyplot as plt #Allows us to plot data, create graphs and visualize data. Perfect for your Powerpoint slides ;)\nimport sklearn #The one and only. This amazing library holds all the secrets. Containing powerful algorithms packed in a single line of code, this is where the magic will happen.\nimport sklearn.model_selection # more of sklearn. It is a big library, but trust me it is worth it.\nimport sklearn.preprocessing \nfrom sklearn.preprocessing import LabelEncoder,OneHotEncoder\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import r2_score, explained_variance_score,mean_absolute_error,mean_squared_error,precision_score,recall_score, accuracy_score,f1_score\nfrom sklearn.utils import shuffle\nimport os\n\n\nimport random # Allows us to call random numbers, occasionally very useful.\nimport pprint#Allows us to neatly display text\nfrom collections import OrderedDict\n\nimport seaborn as sns\n","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"#Introduction\nHi everyone, I hope the title of this Kernel made caused you to clickbait here. The main discussion point of this kernel are Feature creating and Model speration. These allowed me to get slightly better results and perhaps someone with more experience ( i am but a beginner) would be able to make better use of these"},{"metadata":{"_uuid":"7c5ae3a39e8f5cff49804915408315d0be70b5f1"},"cell_type":"markdown","source":"I'm importing the Quadratic Weighted Kappa function that [Aman Arora was so kind to share for all of us](https://www.kaggle.com/aroraaman/quadratic-kappa-metric-explained-in-5-simple-steps):"},{"metadata":{"trusted":true,"_uuid":"3eedbb86bf2029d472f06321998d786430ffa596"},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom sklearn.metrics.scorer import make_scorer\nfrom sklearn.metrics import confusion_matrix\n\ndef quadratic_kappa(y_true, y_pred):\n    \"\"\"This function calculates the Quadratic Kappa Metric used for Evaluation in the PetFinder competition\n    at Kaggle. It returns the Quadratic Weighted Kappa metric score between the actual and the predicted values \n    of adoption rating.\"\"\"\n    w = np.zeros((5,5))\n    O = confusion_matrix(y_true, y_pred)\n    for i in range(len(w)): \n        for j in range(len(w)):\n            w[i][j] = float(((i-j)**2)/(5-1)**2)\n    \n    act_hist=np.zeros([5])\n    for item in y_true: \n        act_hist[item]+=1\n    \n    pred_hist=np.zeros([5])\n    for item in y_pred: \n        pred_hist[item]+=1\n                         \n    E = np.outer(act_hist, pred_hist);\n    E = E/E.sum();\n    O = O/O.sum();\n    \n    num=0\n    den=0\n    for i in range(len(w)):\n        for j in range(len(w)):\n            num+=w[i][j]*O[i][j]\n            den+=w[i][j]*E[i][j]\n    return (1 - (num/den))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2345bcad219c32953b22590857c2eace6fe03570"},"cell_type":"markdown","source":"I always found this awesome correlation matrix function provided by [Enrique Herroreos in his kernel](https://www.kaggle.com/kikexclusive/curiosity-didn-t-kill-the-cat-all-in-one). Big up!"},{"metadata":{"trusted":true,"_uuid":"1f1c37d33782b89229f3c2a9f8ecb8683ebadc15"},"cell_type":"code","source":"def plot_correlation_matrix(df):\n    corr = df.corr()\n\n    # Generate a mask for the upper triangle\n    mask = np.zeros_like(corr, dtype=np.bool)\n    mask[np.triu_indices_from(mask)] = True\n\n    # Set up the matplotlib figure\n    f, ax = plt.subplots(figsize=(20, 9))\n\n    # Generate a custom diverging colormap\n    cmap = sns.diverging_palette(220, 10, as_cmap=True)\n\n    # Draw the heatmap with the mask and correct aspect ratio\n    sns.heatmap(corr, mask=mask, cmap=cmap, vmax=.3, center=0,\n                square=True, linewidths=.5, cbar_kws={\"shrink\": .5})","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a9cf288dbf80adadc3d68cc8a0a5e6c3575549e9"},"cell_type":"markdown","source":"# Part I - Data Cleaning"},{"metadata":{"trusted":true,"_uuid":"4165d6fb9e414fccca9f1a46c3e71c0f5c57d3c8"},"cell_type":"code","source":"#Let's load our data\nprint(os.listdir(\"../input\"))\n\ndata = pd.read_csv(\"../input/train/train.csv\")\ndata.columns","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a143b8a5d33ee6a2febc07b5efe50d41a8a7eadf"},"cell_type":"markdown","source":"### Step 1 - Data Dropping"},{"metadata":{"trusted":true,"_uuid":"5dbd2c9cff6957523da4ae21923bffa5cb50fe1d"},"cell_type":"code","source":"data.set_index('PetID', inplace=True)\n#I'm not going to use NLP in this Kernel, soz!\ndata.drop(columns=['RescuerID','Name','Description'], inplace=True)\n\n#Shuffling data... Just in case\ndata= data.sample(frac=1, random_state=85)\n\ndata.head() ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a70a001f30a4107c9c4087818ab46d1902fc0bcf"},"cell_type":"markdown","source":"### Step 2 - Feature Creation\n\nSo here is where the fun begins. There were some features I weren't too happy with, namely the \"States\", \"Breeds\" and \"Colours\".\n\nFor the Breeds an States, I though One-Hot-Encoding would be the best idea because I dataset is relatively small. So what I wanted to find was data that would be numerical and hierchacical to substitute with. The question became, what information does the State and Breed provide us with?\n\n* **For the States** the following two properties would have a direct impact on the Adoption Speed : Population Size and Internet Penetration . So I added them to the state_labels.csv. ( I couldn't find a dataset on the internet penetration rate per state so I used urbanization rate as the closest proxy). You can find t[he complete database here](https://raw.githubusercontent.com/busyML/Petfinder.com-Adoption-Time-Predictor/master/state_labels.csv)\n\n\n\n"},{"metadata":{"trusted":true,"_uuid":"f4da1d1b3594fd96874f9c35af581645c9aa6c1e"},"cell_type":"code","source":"states = pd.read_csv('../input/state_labels.csv')\n\n\nstates.at[0, \"State Urbanisation\"] = 71.9\nstates.at[1, \"State Urbanisation\"] = 64.6\nstates.at[2, \"State Urbanisation\"] = 42.4\nstates.at[3, \"State Urbanisation\"] = 100\nstates.at[4, \"State Urbanisation\"] = 82.3\nstates.at[5, \"State Urbanisation\"] = 86.5\nstates.at[6, \"State Urbanisation\"] = 66.5\nstates.at[7, \"State Urbanisation\"] = 50.5\nstates.at[8, \"State Urbanisation\"] = 69.7\nstates.at[9, \"State Urbanisation\"] = 51.4\nstates.at[10, \"State Urbanisation\"] = 90.8\nstates.at[11, \"State Urbanisation\"] = 54\nstates.at[12, \"State Urbanisation\"] = 53.8\nstates.at[13, \"State Urbanisation\"] = 91.4\nstates.at[14, \"State Urbanisation\"] = 59.1\n\nstates.at[0, \"State Population\"] = 3348283\nstates.at[1, \"State Population\"] = 1890098\nstates.at[2, \"State Population\"] = 1459994\nstates.at[3, \"State Population\"] = 1627172\nstates.at[4, \"State Population\"] = 86908\nstates.at[5, \"State Population\"] = 788706\nstates.at[6, \"State Population\"] = 997071\nstates.at[7, \"State Population\"] = 1443365\nstates.at[8, \"State Population\"] = 2258428\nstates.at[9, \"State Population\"] = 227025\nstates.at[10, \"State Population\"] = 1520143\nstates.at[11, \"State Population\"] = 3117405\nstates.at[12, \"State Population\"] = 72420009\nstates.at[13, \"State Population\"] = 5411324\nstates.at[14, \"State Population\"] = 1015776.9\n\n\n\n\nstates","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6d0594297724fdd60088a32ffca3d8c2c9e68c21"},"cell_type":"code","source":"#We convert the label dataframe into a dictionary that we then map onto our train dataframe:\nstates_ubran_dict=states.set_index('StateID')['State Urbanisation'].to_dict()\nstates_pop_dict=states.set_index('StateID')['State Population'].to_dict()\n\ndata[\"State Urbanisation\"]=data[\"State\"].map(states_ubran_dict)\ndata[\"State Population\"]=data[\"State\"].map(states_pop_dict)\ndata.drop(columns=\"State\", inplace= True)\n\n\ndata.head()\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"472ab5bbccd9fe6988b01232bcffa9fbbcf3b3d1"},"cell_type":"markdown","source":""},{"metadata":{"_uuid":"fee115f04b14cbc7c992189f70f880234696f5f9"},"cell_type":"markdown","source":"I had a similar feeling about the breed feature, I wanted a way to be able to rank each breed. So I took the breed_labels file and added to it a ranked popularity based on [this dataset of breed popularity ](https://www.akc.org/expert-advice/news/most-popular-dog-breeds-full-ranking-list/). Here, the smaller the number, the greater the popularity."},{"metadata":{"trusted":true,"_uuid":"8fa47b7fbb5820d80793da667875e53aa1af5374"},"cell_type":"code","source":"breeds=states = pd.read_csv('../input/breed_labels.csv')\n\nbreeds.at[0,\"Ranked Popularity\"]= 147\n#Let's speed this up. Here is a list of the ranked popularity with the same index as the breed labels.\nbreeds_popularity_list=[147,93,55,190,47,59,5,118,126,190,148,86,190,56,190,17,137,84,39,6,125,141,151,120,190,44,190,25,46,130,1,190,116,50,190,132,124,190,38,92,102,21,85,11,98,127,26,94,60,51,69,175,37,190,190,190,24,19,43,32,79,190,179,1,76,184,140,190,29,40,130,15,81,171,13,63,182,16,190,67,190,5,52,178,113,95,70,27,134,152,118,190,145,190,173,177,96,99,187,4,156,136,2,10,190,62,80,158,3,104,14,66,75,156,183,23,160,190,12,165,189,72,114,164,73,74,111,78,119,190,190,190,190,87,129,190,190,163,157,190,1,138,190,100,77,169,33,70,190,28,190,71,190,152,190,107,190,36,128,170,91,190,105,89,70,190,53,190,88,190,160,174,190,166,159,113,176,22,7,54,190,31,142,162,97,139,96,41,8,48,123,57,190,110,90,168,58,150,104,64,140,70,24,45,20,12,106,172,188,122,190,90,177,60,7,161,149,107,190,153,121,101,190,112,131,30,34,15,133,109,42,49,61,2,99,65,70,143,1,9,200,10,22,6,42,12,28,18,16,34,31,3,20,39,5,50,36,41,50,13,50,9,5,5,50,50,7,25,1,50,38,42,24,28,37,42,6,29,50,50,14,21,11,11,11,4,6,26,2,17,7,23,12,15,50,30,50,27,8,50,50,19,50,10,33,41,45]\ni=0\n#With a simple \"for loop\", we can add the whole list the breed_label dataframe\nfor i in range (len(breeds_popularity_list)):\n        breeds.at[i,\"Ranked Popularity\"]= breeds_popularity_list[i]\nbreeds","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fc23b688b05286fa4f718108ada55dbd57a0a16e"},"cell_type":"code","source":"breeds_dictionary=breeds.set_index('BreedID')['Ranked Popularity'].to_dict()\n\n#We map the dictionary onto the train dataframe\ndata[\"Breed1\"]=data[\"Breed1\"].map(breeds_dictionary)\ndata[\"Breed2\"]=data[\"Breed2\"].map(breeds_dictionary)\n\n#if we  only have one breed for both columns, then we'll repeat the bread, indicating that this is a pure breed\ndata[\"Breed2\"].fillna(data[\"Breed1\"],inplace=True)\ndata[\"Breed1\"].fillna(data[\"Breed2\"],inplace=True)\n\n\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c2249df6a16c4597dff9575112e2aa7d73d24c4a"},"cell_type":"markdown","source":"And now we need to do the same with colors. At first I tried to use one hot encoding and create 7 new columns but that matrix was too sparse for my liking. So what I decided to do was assign a intensity value between 0 and 1 ( 0 being \"white\" and 1 being \"Black\") and mapped the colour value that way. The model seemed to perform better this way:"},{"metadata":{"trusted":true,"_uuid":"8edf78a2c4dab1e4160fdd378d844e2316729de4"},"cell_type":"code","source":"colors = pd.read_csv('../input/color_labels.csv')\n\n\ncolors.at[0, \"Color Intensity\"] = 1\ncolors.at[1, \"Color Intensity\"] = 0.883\ncolors.at[2, \"Color Intensity\"] = 0.667\ncolors.at[3, \"Color Intensity\"] = 0.333\ncolors.at[4, \"Color Intensity\"] = 0.167\ncolors.at[5, \"Color Intensity\"] = 0.5\ncolors.at[6, \"Color Intensity\"] = 0\n\ncolors","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4c5e1ec4899d9abda7409cea731c963d8d3c6a50"},"cell_type":"code","source":"color_dictionary=colors.set_index('ColorID')['Color Intensity'].to_dict()\n\n\ndata[\"Color2\"]= data[\"Color2\"].apply(lambda x: None if x==0 else x)\ndata[\"Color3\"]= data[\"Color3\"].apply(lambda x: None if x==0 else x)\n\n\ndata[\"Color1\"]=data[\"Color1\"].map(color_dictionary)\ndata[\"Color2\"]=data[\"Color2\"].map(color_dictionary)\ndata[\"Color3\"]=data[\"Color3\"].map(color_dictionary)\n\n#This always to standerize the colors, if color1=color2=color3, then the pet only has one solid color\ndata[\"Color2\"].fillna(data[\"Color1\"],inplace=True)\ndata[\"Color3\"].fillna(data[\"Color2\"],inplace=True)\n\n#for the second row (id=29ffe21fe), we can see that that the pet is black and white\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ee42afc09df61a48116432d87672b4b46932e33e"},"cell_type":"markdown","source":"### Step 2- Feature Standarization\n\nI'm only going to to use some pretty basic Label Encoding here..."},{"metadata":{"trusted":true,"_uuid":"bd8c757aac090bdddf9184311565ec5f2be683dc"},"cell_type":"code","source":"data[\"Vaccinated\"]=data[\"Vaccinated\"].apply(lambda x:1 if x==1 else 0)\ndata[\"Dewormed\"]=data[\"Dewormed\"].apply(lambda x:1 if x==1 else 0)\ndata[\"Sterilized\"]=data[\"Sterilized\"].apply(lambda x:1 if x==1 else 0)\n#data[\"Gender\"]=data[\"Gender\"].apply(lambda x:1 if x==2 else 0)\n\n\ndata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a62c2fcecbb9d98e9d78b9fce47c7084f95fcf48"},"cell_type":"code","source":"#Let's just save our formatted data just in case.\n\ndata.to_csv(\"formated_data_petfinder.csv\")\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0d5c62836ef7561ff2353b8480ed9c2f68b12063"},"cell_type":"markdown","source":"### Step 3- Feature Selection\n\nSo let's start doing so analysis on our data. First, let's check what the most important features seem to be:"},{"metadata":{"trusted":true,"_uuid":"72ab8396ed52f011363bb149b449213e3bab16c7"},"cell_type":"code","source":"#creating our x datasheet, which contains all our data except for the ['AdoptionSpeed'] column.\nx=data.drop(columns='AdoptionSpeed')\n\n\n#The ['AdoptionSpeed'] column extracted.\ny=data['AdoptionSpeed']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8c1016de9c787eb39db3503894e42d4fea69c0ca"},"cell_type":"code","source":"#Checking the features that are correlated to the target feature (Adoption speed)\nfor columnname in x.columns: \n    if abs(x[columnname].corr(y)) >0.095 :    \n      print('The Correlation Between',columnname,'and Adoption Speed is significant:', abs(x[columnname].corr(y)))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"62bbddd9644856e1d4d41770231292b12f5b6225"},"cell_type":"markdown","source":"From this we can learn that the Breed, Age, Vaccination and Sterilization seem to be key predictors ( there might be others of course.)\n\nLet's now use a Decision Tree to keeping viewing the Feature Importance."},{"metadata":{"trusted":true,"_uuid":"10e61954ac42d2d0b78bb117051c44052d8901c8"},"cell_type":"code","source":"from sklearn.ensemble import ExtraTreesClassifier\nfeature_importance_indicator=ExtraTreesClassifier(n_estimators = 100)\n\n#Then we ask this algorithm, now dubbed as \"feature_importance_indicator\", to learn from our data x ( the indicators) and y( the Adoption Speed outcome):\n\nfeature_importance_indicator.fit(x,y)\n\n#We then ask the model politely to create a list of which columns it learnt the most from and which columns didn't help it at all, for this we use the \"feature_importances_\" command:\n\nimportance_dict= dict(zip(x.columns, (feature_importance_indicator.feature_importances_)))\n\n#We sort the list in descending order so that it is quicker to analyse.\n\nimportance_dict_sorted= OrderedDict(sorted(importance_dict.items(), key=lambda x:x[1],reverse=1))\n\n#The \"pprint\" commnad allows us to print things out nicely.\n\npprint.pprint(importance_dict_sorted)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c4002ddad3b39943b4738155a4f213aba85a14c3"},"cell_type":"code","source":"features = x.columns\nimportances = feature_importance_indicator.feature_importances_\nindices = np.argsort(importances)\n\nplt.title('Feature Importances')\nplt.barh(range(len(indices)), importances[indices], color='r', align='center')\nplt.yticks(range(len(indices)), [features[i] for i in indices])\nplt.xlabel('Importance Level')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3b633c68af4400782e47d2c275afb0b5d5f81795"},"cell_type":"markdown","source":"##For the key part of this kernel!!:\n\nNow, in my mind there was something that was bugging me about the dataset, and that was the fact that dataset  was including dogs and cats together. Now I know a thing or two about pet adoption, and for me it is pretty obvious that the criteria a person uses when accepting a cat is rather different to that of adopting a dog. People simply take different things into account. Ultimately, different features would have different importance based whether it was a cat or a dog. \nBecause i'm using some very narrow M.L here (just SciKit Learn Algorithms) was afraid that this distinction wouldn't be weighted strong enough in the model that tries to predict for both Cats and Dogs. \n\nSo I decided to visualize the data to see how significant the difference was between cat and dog feature, to see if I could validate by theory or not. "},{"metadata":{"trusted":true,"_uuid":"1f929842110eaeda0346c8ad280a64e086358886"},"cell_type":"code","source":"plot_correlation_matrix(data[data.Type==1].drop(columns=[\"Type\"]))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1b8d3b931d73b64fcee5ed37cdea6651b605b211"},"cell_type":"code","source":"plot_correlation_matrix(data[data.Type==2].drop(columns=[\"Type\"]))\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cc97a7e8232035a4b471017365270183e0c783ff"},"cell_type":"code","source":"rad_viz = pd.plotting.radviz(x, 'Type')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b145cf9583095ab6ea475e1b80c665e713a24c08"},"cell_type":"markdown","source":"In my opinion, the Correlation Matrixes show us that the two datasets are rather different indeed, different features seem to be correlated differently to one another depending on the Animal type.\nFuthermore, if we have take a look at the Adoption Speed line, we can see some difference in the correlations between the two the matrixes.\nFinally, the Radviz allows us to see that the cat data is a lot more spread out whereas the dog data is far more uniform. \n\nTherefore, I will try to seperate this dataset into two subdatasets ( one of cats, one of dogs) and train two different models on them."},{"metadata":{"trusted":true,"_uuid":"8ba8faff5255e0ecc45911986c85daa9af9f959e"},"cell_type":"code","source":"\n#we create two datasets, one for dogs and one for cats to train seperate data on them.\n\ndogdata = data[data.Type==1]\ncatdata = data[data.Type==2]\n\n\ndog_x=dogdata.drop(columns=[\"AdoptionSpeed\"])\ndog_y=dogdata[\"AdoptionSpeed\"]\n\ncat_x=catdata.drop(columns=[\"AdoptionSpeed\"])\ncat_y=catdata[\"AdoptionSpeed\"]\n\n\ndog_x.columns","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"101485fa8bba99f97330743b65fd47c16dc36912"},"cell_type":"markdown","source":"And finally, if we look at the Feature Importance for the dog dataset and the cat dataset, we can see some subtle differences such as the importance of the fur lenght or the size.\n\nThis in my opinion, validates to try and seperate the two datasets."},{"metadata":{"trusted":true,"_uuid":"537b124b0e54c3f4aca1a53313f96a3036384307"},"cell_type":"code","source":"feature_importance_indicator.fit(dog_x,dog_y)\nfeatures = dog_x.columns\nimportances = feature_importance_indicator.feature_importances_\nindices = np.argsort(importances)\n\nplt.title('Dog Feature Importances')\nplt.barh(range(len(indices)), importances[indices], color='r', align='center')\nplt.yticks(range(len(indices)), [features[i] for i in indices])\nplt.xlabel('Importance Level')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3e50648e6e7363557a882d8f2ea3f42d073e54c4"},"cell_type":"code","source":"feature_importance_indicator.fit(cat_x,cat_y)\nfeatures = cat_x.columns\nimportances = feature_importance_indicator.feature_importances_\nindices = np.argsort(importances)\n\nplt.title('Cat Feature Importances')\nplt.barh(range(len(indices)), importances[indices], color='r', align='center')\nplt.yticks(range(len(indices)), [features[i] for i in indices])\nplt.xlabel('Importance Level')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"847384640a4b5f9245e996d38f338b883ce4b7c7"},"cell_type":"markdown","source":"\n# Part 2- Data Learning\n\nWe'll now train our two datasets on different models. To find the optimal model for each dataset, I used the highly recommendable[ TPOT api](https://epistasislab.github.io/tpot/api/). TPOT uses genetic searching to give you the Sci kit learn model with the optimal hyperparameters. However, this process took roughly 8 hours overall. The models and hyperparameters used here below here the optimal models that the TPOT alogorithm found."},{"metadata":{"trusted":true,"_uuid":"1942c28a64beb86ddefca6ac04e92a6789616869"},"cell_type":"code","source":"\n#splitting between train and test for the dog data\ndog_x_training,dog_x_testing, dog_y_training, dog_y_testing = train_test_split(dog_x, dog_y, test_size=0.1, random_state=85)\n\n#splitting between train and test for the cat data\ncat_x_training,cat_x_testing,  cat_y_training, cat_y_testing = train_test_split(cat_x, cat_y, test_size=0.1, random_state=85)\n\ndog_x_training.drop(columns=\"Type\", inplace=True)\ndog_x_testing.drop(columns=\"Type\",inplace=True)\ncat_x_training.drop(columns=\"Type\",inplace=True)\ncat_x_testing.drop(columns=\"Type\",inplace=True)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dd3b2eb14b520d67b6db779c00217aed610b64d1"},"cell_type":"markdown","source":"Here is the best model found for the dogdata, GradientBoosting Classifier:"},{"metadata":{"trusted":true,"_uuid":"08143d669e61b733d7aa64c18876fe9be8ae17b9"},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom sklearn.ensemble import GradientBoostingClassifier\nfrom sklearn.model_selection import train_test_split\nimport numpy as np\nimport pandas as pd\nfrom sklearn.ensemble import GradientBoostingClassifier\nfrom sklearn.feature_selection import VarianceThreshold\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.pipeline import make_pipeline\n\n\n# Average CV score on the training set was:0.368919785590269\n\ndog_best_model = make_pipeline(\n    VarianceThreshold(threshold=0.0001),\n    GradientBoostingClassifier(learning_rate=0.1, max_depth=5, max_features=0.5, min_samples_leaf=4, min_samples_split=17, n_estimators=100, subsample=1.0)\n)\n\n\ndog_best_model.fit(dog_x_training, dog_y_training)\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dea913d6738b5ca55d5d026aeeb8a8522ec5d74e"},"cell_type":"markdown","source":"Now we can use our Quadratic Weighted Kappa (QWK) function from earlier to test it on our test data"},{"metadata":{"trusted":true,"_uuid":"fbbb60778839211b68678cbc5aada35d9358feb4"},"cell_type":"code","source":"y_true_train= np.array(dog_y_training)\ny_pred_train = np.array(dog_best_model.predict(dog_x_training))\n\ny_true_test= np.array(dog_y_testing)\ny_pred_test = np.array(dog_best_model.predict(dog_x_testing))\n\n\nprint(\"On training data QWK:...\",quadratic_kappa(y_true_train,y_pred_train),\"on testing data QWK...\",quadratic_kappa(y_true_test,y_pred_test))\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d61c80c56647d3b27a08811eb3880c64e85c3171"},"cell_type":"markdown","source":"Now for the cat data. Here the best model was found to be a Scaler along with XGBoost. The fact that the optimal models found for each dataset reinforces by point of view that cat data and dog data should be treated differently."},{"metadata":{"trusted":true,"_uuid":"1126617d733cd9563c74a4c1f59d69fc14d9f33b"},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.pipeline import make_pipeline\nfrom sklearn.preprocessing import StandardScaler\nfrom xgboost import XGBClassifier\n\n\n# Average CV score on the training set was:0.31768177281003007\ncat_best_model = make_pipeline(\n    StandardScaler(),\n    XGBClassifier(learning_rate=0.1, max_depth=4, min_child_weight=13, n_estimators=100, nthread=1, subsample=0.7500000000000001)\n)\n\ncat_best_model.fit(cat_x_training,cat_y_training)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"959120d17582327fd3adf27683c27bf0ccf251c6"},"cell_type":"code","source":"y_true_train= np.array(cat_y_training)\ny_pred_train = np.array(cat_best_model.predict(cat_x_training))\n\ny_true_test= np.array(cat_y_testing)\ny_pred_test = np.array(cat_best_model.predict(cat_x_testing))\n\nprint(\"On training data...\",quadratic_kappa(y_true_train,y_pred_train),\"on testing data...\",quadratic_kappa(y_true_test,y_pred_test))\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bfb9bbadf2b5546321f2e4173b4f02b08086bd0d"},"cell_type":"markdown","source":"On a final note, I also ran TPOT on the whole data (cats on dogs put together), here the best QWK I was able to get on the test data was 0.31...  In my opinion, seperating the catdata from the dogdata is the right way to go in order to increase the performance of our models. I encourage others to try it with more complex models that use the image dataset and NLP."},{"metadata":{"_uuid":"1d065d65b997e5b63a97f0e0235d88d4965a90b3"},"cell_type":"markdown","source":"# Part 3- Data Predicting\n\nLet's know use these two models to generate our answers for the submision:"},{"metadata":{"trusted":true,"_uuid":"f4af90b4d848e5ba7e01ce97542d8a1f9ee77f37"},"cell_type":"code","source":"#Let's first retrain our models on the complete dataset\ndog_best_model.fit(dog_x, dog_y)\ncat_best_model.fit(cat_x,cat_y)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4520032c0ce14b9f0928d48ccfd90ec7ea33e129"},"cell_type":"code","source":"#Loading the test data provided by Kaggle\ntest_data=pd.read_csv(\"../input/test/test.csv\")\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dc7a42cd681f14cc98e357c9d79190808df43086"},"cell_type":"code","source":"#Formatting the test data just like we did with the train data...\ntest_data.drop(columns=['RescuerID','Name','Description','PetID'], inplace=True)\n\ntest_data[\"State urbanisation\"]=test_data[\"State\"].map(states_ubran_dict)\ntest_data[\"State population\"]=test_data[\"State\"].map(states_pop_dict)\ntest_data.drop(columns=\"State\", inplace= True)\n\ntest_data[\"Color2\"]= test_data[\"Color2\"].apply(lambda x: None if x==0 else x)\ntest_data[\"Color3\"]= test_data[\"Color3\"].apply(lambda x: None if x==0 else x)\n\n\ntest_data[\"Color1\"]=test_data[\"Color1\"].map(color_dictionary)\ntest_data[\"Color2\"]=test_data[\"Color2\"].map(color_dictionary)\ntest_data[\"Color3\"]=test_data[\"Color3\"].map(color_dictionary)\n\ntest_data[\"Color2\"].fillna(test_data[\"Color1\"],inplace=True)\ntest_data[\"Color3\"].fillna(test_data[\"Color2\"],inplace=True)\n\ntest_data[\"Breed1\"]=test_data[\"Breed1\"].map(breeds_dictionary)\ntest_data[\"Breed2\"]=test_data[\"Breed2\"].map(breeds_dictionary)\n\ntest_data[\"Breed2\"].fillna(test_data[\"Breed1\"],inplace=True)\ntest_data[\"Breed1\"].fillna(test_data[\"Breed2\"],inplace=True)\n\ntest_data[\"Vaccinated\"]=test_data[\"Vaccinated\"].apply(lambda x:1 if x==1 else 0)\ntest_data[\"Dewormed\"]=test_data[\"Dewormed\"].apply(lambda x:1 if x==1 else 0)\ntest_data[\"Sterilized\"]=test_data[\"Sterilized\"].apply(lambda x:1 if x==1 else 0)\n\n\n\n\ntest_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c81bc4b5a553625c0d59385fc1e672e4287ebe2b"},"cell_type":"code","source":"#Making sure the formatting was done correctly\nif len(test_data.columns)==len(dog_x.columns):\n  print(\"Formatting done correctly\")\nelse:\n    print(\"H we have a problem, double check previous cell\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"bd2401494ae229bf30decf0cc2db78d9282ab8c8"},"cell_type":"markdown","source":"In order to apply the models correspondingly, I built the function below that discriminates between dogs and cats in the Test data and uses the correct model on that specific row."},{"metadata":{"trusted":true,"_uuid":"bf7a5d270dc2643283f5899d71688060f645d48c"},"cell_type":"code","source":"#We'll store our answers in this dictionary so that we can map them onto the submission file\nanswers_dictionary={}\n\ndef test_predicitions(test_data):\n    assert type(test_data)==type(data)\n    for i in range (len(test_data)):\n        if int(test_data[\"Type\"].iloc[[i,]])==1:\n            test_data.iloc[[i,]].drop(columns=\"Type\")\n            answers_dictionary[test_data.index[i]]=int(dog_best_model.predict(test_data.iloc[[i,]]))\n        else:\n            test_data.iloc[[i,]].drop(columns=\"Type\")\n            answers_dictionary[test_data.index[i]]=int(cat_best_model.predict(test_data.iloc[[i,]]))\n\ntest_predicitions(test_data)     \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"34387ca3064401f203a2dbc14556c332fffc6d61"},"cell_type":"code","source":"submission_answer=pd.read_csv(\"../input/test/sample_submission.csv\")\n\nsubmission_answer.head()\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f7e53c003c5205b91d8339d70073132c1a2c27e9"},"cell_type":"code","source":"#we map our asnwers from our two different models onto the submission file\nsubmission_answer[\"AdoptionSpeed\"]= submission_answer.index.map(answers_dictionary.get)\n\nsubmission_answer.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3acddae4deb73d82ea151ae6f527c235d5a2f873"},"cell_type":"code","source":"submission_answer.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"eb4bd1426640f53aca716020311f1438c184c076"},"cell_type":"markdown","source":"# Conclusion\n\nMy conclusion is that I think this problem needs to be treated as one with two independent sub-sets of data. From my own experience, when I seperated the cat of the dogs, my models, as simple as they were, improved significantly. I am therefore exicted to see if others take this approach and can get even better results by seperating the two datasrts with the use of DL, Computer vision and Sentiment Analysis."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}