{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Chapter: Introduction","metadata":{}},{"cell_type":"markdown","source":"## Purpose of this Notebook\nLearn machine learning techniques and use of kaggle through the Titanic project.\n## Policy\n* The purpose of this project is to acquire skills, not to improve the ranking. In this project, it has been pointed out that there is a discrepancy in the distribution between the training data and the test data for public scoring, and it seems that the improvement in technique and the improvement in public score do not always match. Furthermore, it has been pointed out that the scores of the higher ranks may be due to fraud, and the ranking may not be directly linked to the ability.(however, I'm not sure if this point is correct.)\n* A cross-validation method is used to evaluate model performance. Inference for submission is performed with a model learned by inputting the entire training data to the model judged to be the best by cross-validation.\n* At the data analysis stage, the entire training data is used. However, at the time of cross-validation, the hold for evaluation is not included in the fit target of data preprocessing. This process prevents the adverse effects of target leaks. This is achieved by building a pipeline.\n* XGBoost is used as the estimator. We build two patterns, a model using fixed hyperparameters and a model using hyperparameters adjusted by grid search. Also, We beuild multiple patterns of pipelines. therefore, We produce multiple outputs(submission.csv) as many as the number of combinations of models and pipelines.\n* Since the Titanic project is the task limited to the freight that have encountered an accident, the family information (family name, ticket No, barked port, and so on) of passengers will be extracted from the training data and used as clues for inference.\n \n ","metadata":{}},{"cell_type":"markdown","source":"# Chapter: Preparation","metadata":{}},{"cell_type":"markdown","source":"## Set parameters for platform operation","metadata":{"id":"1eX71lFxLRg6"}},{"cell_type":"markdown","source":"Environmental adjustments to absorb platform differences","metadata":{}},{"cell_type":"code","source":"EXTERNAL       = False          # kaggle-> False, External platforms -> True\nINPUT_PATH     = \"input\"        # required if executing this notebook on the external platform.\nAUTO_SUBMIT    = False          # True if auto submit mode at the last cell on the external platform.\nSUBMIT_COMMENT = \"submitted from google colab.\"    # comment string of submission.\nPROJECT_NAME   = \"titanic\"      # competition project name.\n\nif EXTERNAL:\n    print(\"Set the 'kaggle.json' at working directory before running subsequent cells.\")\nelse:\n    print(\"Excuting this notebook on Kaggle platform.\")","metadata":{"id":"3xUq7hXwLlEn","executionInfo":{"status":"ok","timestamp":1656775748512,"user_tz":-540,"elapsed":4,"user":{"displayName":"浜田光利","userId":"16018729205021549769"}},"outputId":"a7ea5956-d420-497d-9484-fcca60387065","execution":{"iopub.status.busy":"2022-07-06T04:11:53.686381Z","iopub.execute_input":"2022-07-06T04:11:53.687587Z","iopub.status.idle":"2022-07-06T04:11:53.720919Z","shell.execute_reply.started":"2022-07-06T04:11:53.687533Z","shell.execute_reply":"2022-07-06T04:11:53.719155Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Additional processing on platforms other than Kaggle","metadata":{"id":"WD6n7Fm-Lv-F"}},{"cell_type":"markdown","source":"* Setting up kaggle.json for using Kaggle API.\n* Installing libraries needed to run this notebook, those is preinstalled on Kaggle.(Those depends on the environment)","metadata":{}},{"cell_type":"code","source":"%%bash -s $EXTERNAL $PROJECT_NAME $INPUT_PATH\nif [ $1 = \"True\" ]; then\n    echo \"external mode.\"\n    # Put kaggle.json in place.\n    mkdir -p ~/.kaggle\n    cp kaggle.json ~/.kaggle/\n    chmod 600 ~/.kaggle/kaggle.json\n\n    # Download and extract zip file.\n    pip install -q kaggle\n    kaggle competitions download -c $2 -p $3\n    unzip $3/$2.zip -d $3/$2\n\n\n    # Install libraries\n    pip install -q category_encoders\n\n    echo \"END\"\nelse\n    echo \"not external mode.\"\nfi\n","metadata":{"id":"W1b105xUMLt3","executionInfo":{"status":"ok","timestamp":1656775760876,"user_tz":-540,"elapsed":12367,"user":{"displayName":"浜田光利","userId":"16018729205021549769"}},"outputId":"d8c27a6b-dc65-4ab6-9fb4-5f4619411703","execution":{"iopub.status.busy":"2022-07-06T04:11:58.534037Z","iopub.execute_input":"2022-07-06T04:11:58.5351Z","iopub.status.idle":"2022-07-06T04:11:58.560033Z","shell.execute_reply.started":"2022-07-06T04:11:58.535053Z","shell.execute_reply":"2022-07-06T04:11:58.558189Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Import libraries","metadata":{"id":"Bo5q6cHbMRqm"}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\nimport re\n\nfrom sklearn.pipeline import Pipeline\nfrom sklearn.impute import SimpleImputer\nfrom sklearn.compose import ColumnTransformer\n#from sklearn.preprocessing import OneHotEncoder\nimport category_encoders as ce\nfrom sklearn.preprocessing import StandardScaler\nfrom sklearn.decomposition import PCA\nfrom sklearn.base import BaseEstimator, TransformerMixin\nfrom sklearn.feature_selection import RFECV\nfrom sklearn.feature_selection import SelectFromModel\nfrom sklearn.ensemble import RandomForestClassifier\nfrom xgboost import XGBClassifier\n\nfrom xgboost import XGBClassifier\nfrom sklearn.model_selection import cross_val_score\nfrom sklearn.model_selection import GridSearchCV\nfrom sklearn import set_config\n","metadata":{"id":"LOL82tw0MV_v","executionInfo":{"status":"ok","timestamp":1656775762754,"user_tz":-540,"elapsed":1884,"user":{"displayName":"浜田光利","userId":"16018729205021549769"}},"outputId":"b225258c-44e6-430f-e158-1f3dba6c41e1","execution":{"iopub.status.busy":"2022-07-06T04:12:02.679511Z","iopub.execute_input":"2022-07-06T04:12:02.68007Z","iopub.status.idle":"2022-07-06T04:12:04.344235Z","shell.execute_reply.started":"2022-07-06T04:12:02.680033Z","shell.execute_reply":"2022-07-06T04:12:04.342687Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk(INPUT_PATH if EXTERNAL else '/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":0.027629,"end_time":"2022-07-02T10:40:43.444266","exception":false,"start_time":"2022-07-02T10:40:43.416637","status":"completed"},"tags":[],"id":"fed823e6","executionInfo":{"status":"ok","timestamp":1656775762755,"user_tz":-540,"elapsed":10,"user":{"displayName":"浜田光利","userId":"16018729205021549769"}},"outputId":"15bb205f-0a06-4c7d-f4c1-054396b02d37","execution":{"iopub.status.busy":"2022-07-06T04:12:05.695426Z","iopub.execute_input":"2022-07-06T04:12:05.695905Z","iopub.status.idle":"2022-07-06T04:12:05.707152Z","shell.execute_reply.started":"2022-07-06T04:12:05.695866Z","shell.execute_reply":"2022-07-06T04:12:05.706257Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Chapter: Data Analysis","metadata":{}},{"cell_type":"markdown","source":"## Load and preliminary exploring the train dataset","metadata":{"papermill":{"duration":0.005944,"end_time":"2022-07-02T10:40:43.456626","exception":false,"start_time":"2022-07-02T10:40:43.450682","status":"completed"},"tags":[],"id":"3e341447"}},{"cell_type":"code","source":"X_full=pd.read_csv(os.path.join(INPUT_PATH if EXTERNAL else '/kaggle/input',\"titanic/train.csv\"),index_col=\"PassengerId\")\ny=X_full[[\"Survived\"]]\nX=X_full.drop([\"Survived\"],axis=1)\n\nprint(\"X.shape=\",X.shape)\ndisplay(X.head())\ndisplay(X.info())\ndisplay(X.describe())\nprint()\nprint(\"y.shape=\",y.shape)\ndisplay(y.head())\ndisplay(y.info())\ndisplay(y.describe())\n","metadata":{"papermill":{"duration":0.12333,"end_time":"2022-07-02T10:40:43.586186","exception":false,"start_time":"2022-07-02T10:40:43.462856","status":"completed"},"tags":[],"id":"2dd5c885","executionInfo":{"status":"ok","timestamp":1656775762755,"user_tz":-540,"elapsed":8,"user":{"displayName":"浜田光利","userId":"16018729205021549769"}},"outputId":"04c99236-3b7f-4371-c406-7129e1eb6bc6","execution":{"iopub.status.busy":"2022-07-06T04:12:09.992464Z","iopub.execute_input":"2022-07-06T04:12:09.992935Z","iopub.status.idle":"2022-07-06T04:12:10.130773Z","shell.execute_reply.started":"2022-07-06T04:12:09.992897Z","shell.execute_reply":"2022-07-06T04:12:10.129429Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Missing values","metadata":{"papermill":{"duration":0.007019,"end_time":"2022-07-02T10:40:43.600665","exception":false,"start_time":"2022-07-02T10:40:43.593646","status":"completed"},"tags":[],"id":"748cdb2d"}},{"cell_type":"code","source":"\nfig,ax = plt.subplots(figsize=(25,10))\np=sns.heatmap(X.isnull(),cbar=False)\nplt.show()\n","metadata":{"papermill":{"duration":1.900983,"end_time":"2022-07-02T10:40:45.509299","exception":false,"start_time":"2022-07-02T10:40:43.608316","status":"completed"},"tags":[],"id":"c9d577f8","executionInfo":{"status":"ok","timestamp":1656775763896,"user_tz":-540,"elapsed":1145,"user":{"displayName":"浜田光利","userId":"16018729205021549769"}},"outputId":"b7f67294-cfcb-461e-9f47-086176f498e6","execution":{"iopub.status.busy":"2022-07-06T04:12:17.386362Z","iopub.execute_input":"2022-07-06T04:12:17.387532Z","iopub.status.idle":"2022-07-06T04:12:18.264963Z","shell.execute_reply.started":"2022-07-06T04:12:17.387457Z","shell.execute_reply":"2022-07-06T04:12:18.263694Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(\"### X missing values.\")\ntmp=X.isnull().sum()\ndisplay(tmp[tmp>0])\nprint()\nprint(\"### X missing rates.\")\ntmp=X.isnull().sum()/X.shape[0]\ndisplay(tmp[tmp>0])\nprint()\nprint(\"### y missing values.\")\ntmp=y.isnull().sum()\ndisplay(tmp[tmp>0])\nprint()\nprint(\"### y missing rates.\")\ntmp=y.isnull().sum()/y.shape[0]\ndisplay(tmp[tmp>0])\nprint()\n# Drop high missing rate features.\ntmp=X.isnull().sum()/X.shape[0]\nhigh_missing_rate_cols = list(tmp[tmp>0.4].index)\nprint(\"high_missing_rate_cols=\",high_missing_rate_cols)","metadata":{"papermill":{"duration":0.04796,"end_time":"2022-07-02T10:40:45.565302","exception":false,"start_time":"2022-07-02T10:40:45.517342","status":"completed"},"tags":[],"id":"58b3b5f4","executionInfo":{"status":"ok","timestamp":1656775763897,"user_tz":-540,"elapsed":25,"user":{"displayName":"浜田光利","userId":"16018729205021549769"}},"outputId":"1901fb81-78c5-40cd-b7ff-0a443537d2a5","execution":{"iopub.status.busy":"2022-07-06T04:12:21.958894Z","iopub.execute_input":"2022-07-06T04:12:21.959412Z","iopub.status.idle":"2022-07-06T04:12:21.999753Z","shell.execute_reply.started":"2022-07-06T04:12:21.959367Z","shell.execute_reply":"2022-07-06T04:12:21.99862Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Cardinality","metadata":{"papermill":{"duration":0.00848,"end_time":"2022-07-02T10:40:45.582505","exception":false,"start_time":"2022-07-02T10:40:45.574025","status":"completed"},"tags":[],"id":"048bbcc9"}},{"cell_type":"code","source":"tmp=X.select_dtypes(\"object\").nunique()\ndisplay(tmp)\nprint()\nhigh_cardinality_cols=list(tmp[tmp>10].index)\nprint(\"high_cardinality_cols=\",high_cardinality_cols)","metadata":{"papermill":{"duration":0.026531,"end_time":"2022-07-02T10:40:45.61773","exception":false,"start_time":"2022-07-02T10:40:45.591199","status":"completed"},"tags":[],"id":"98ca8b3e","executionInfo":{"status":"ok","timestamp":1656775763898,"user_tz":-540,"elapsed":21,"user":{"displayName":"浜田光利","userId":"16018729205021549769"}},"outputId":"97b2ec8a-57ad-48b9-f7a9-73e81e8dde6b","execution":{"iopub.status.busy":"2022-07-06T04:12:38.039764Z","iopub.execute_input":"2022-07-06T04:12:38.040297Z","iopub.status.idle":"2022-07-06T04:12:38.056512Z","shell.execute_reply.started":"2022-07-06T04:12:38.04024Z","shell.execute_reply":"2022-07-06T04:12:38.055683Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Extract the title from Name","metadata":{"papermill":{"duration":0.008349,"end_time":"2022-07-02T10:40:45.634897","exception":false,"start_time":"2022-07-02T10:40:45.626548","status":"completed"},"tags":[],"id":"c1c6f78f"}},{"cell_type":"code","source":"words={}\nfor name in X[\"Name\"]:\n    name = name.lower()\n    name_words = re.split(\"[,.\\s]\",name)\n    for w in name_words:\n        if w !=\"\":\n            if w not in words:\n                words[w]=0\n            words[w]+=1\nsorted_words=dict(sorted(words.items(),key = lambda item:item[1],reverse=True))\nfor k,v in sorted_words.items():\n    if v>10:\n        print(f\"{k}:{v}\")","metadata":{"papermill":{"duration":0.02568,"end_time":"2022-07-02T10:40:45.668943","exception":false,"start_time":"2022-07-02T10:40:45.643263","status":"completed"},"tags":[],"id":"e6a9f135","executionInfo":{"status":"ok","timestamp":1656775763899,"user_tz":-540,"elapsed":21,"user":{"displayName":"浜田光利","userId":"16018729205021549769"}},"outputId":"59bdeddd-1cfa-4091-81d0-7fce4ee3ab75","execution":{"iopub.status.busy":"2022-07-06T04:12:41.531623Z","iopub.execute_input":"2022-07-06T04:12:41.532798Z","iopub.status.idle":"2022-07-06T04:12:41.552248Z","shell.execute_reply.started":"2022-07-06T04:12:41.532731Z","shell.execute_reply":"2022-07-06T04:12:41.550909Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"title_list=[\"Mr\",\"Mrs\",\"Miss\",\"Ms\",\"Mx\",\"Ma'am\",\"Gentleman\",\"Lady\",\"Dad\",\"Sir\",\"Master\",\"reverend\",\"rev\",\"Dame\",\"Lord\",\"Majesty\",\"Highness\",\"Excellency\",\"Lordship\",\"Honourable\",\"Honor\",\"Dr\",\"Doctor\",\"Prof\",\"Professor\",\"Dean\",\"General\",\"Colonel\",\"Col\",\"Major\",\"Captain\",\"capt\",\"Sergeant\",\"Officer\",\"Lieutenant\",\"President\",\"Minister\",\"Secretary\",\"Senator\",\"Representative\",\"Governor\",\"Mayor\",\"Ambassador\",\"Pope\",\"Cardinal\",\"Archbishop\",\"Bishop\",\"Priest\",\"Brother\",\"Sister\",\"Count\",\"Countess\",\"Eorl\",\"Earl\",\"Earless\",\"Baron\",\"Marquess\",\"Viscount\"]\ntitle_list=list(set(title_list))\ntitle_list=[w.lower() for w in title_list]\ndisplay(\"title_list=\",title_list)\n\n\ndef func_title(x):\n    lower_x=x.lower()\n    words=[]\n    name_words = re.split(\"[,.\\s\\\"\\']\",lower_x)\n    for w in name_words:\n        if w in title_list:\n            words.append(w)\n    return \",\".join(words) if words else None\n\ntmp=X[[\"Name\"]].copy()\ntmp[\"Title\"]=X[[\"Name\"]].applymap(func_title)\ndisplay(tmp)\ndisplay(tmp.describe())\ndisplay(tmp[tmp.Title.isnull()])\n\nnew_X=pd.merge(X,tmp[[\"Title\"]],left_index=True,right_index=True)\ntitle_target_cols = [\"Name\"]\nnew_X.head()","metadata":{"papermill":{"duration":0.077522,"end_time":"2022-07-02T10:40:45.755581","exception":false,"start_time":"2022-07-02T10:40:45.678059","status":"completed"},"tags":[],"id":"9f8247d2","executionInfo":{"status":"ok","timestamp":1656775763899,"user_tz":-540,"elapsed":16,"user":{"displayName":"浜田光利","userId":"16018729205021549769"}},"outputId":"e0864f33-96ba-4b0a-ccd5-882f7e453fc5","execution":{"iopub.status.busy":"2022-07-06T04:12:45.180188Z","iopub.execute_input":"2022-07-06T04:12:45.181414Z","iopub.status.idle":"2022-07-06T04:12:45.261608Z","shell.execute_reply.started":"2022-07-06T04:12:45.181373Z","shell.execute_reply":"2022-07-06T04:12:45.260748Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Extract Family features from several features","metadata":{}},{"cell_type":"markdown","source":"* FamilySize : SibSp + Parch + 1  ('+1' is a person of record)\n* FamilyName : the first half of Name splitted by \",\"\n* FamilyInfo : 'Ticket / Fare / Embarked / FamilyName / FamilySize'  (concateneted as string features)","metadata":{}},{"cell_type":"code","source":"source_family_features=[\"SibSp\",\"Parch\",\"Name\",\"Ticket\",\"Fare\",\"Embarked\"]\n\ntmp=X[source_family_features].copy()\nprint(\"### missing values of \" + (\",\".join(source_family_features)))\ndisplay(tmp.isnull().sum())\ntmp[\"FamilySize\"] = tmp[\"SibSp\"] + tmp[\"Parch\"] + 1\ntmp[\"FamilyName\"] = tmp[\"Name\"].str.split(\",\",expand=True)[0]\ntmp[\"FamilyInfo\"] = tmp[\"Ticket\"]+\"/\"+tmp[\"Fare\"].astype(str)+\"/\"+tmp[\"Embarked\"].fillna(\"\")+\"/\"+tmp[\"FamilyName\"]+\"/\"+tmp[\"FamilySize\"].astype(str)\n\nprint(\"### created features\")\ndisplay(tmp.head().style.applymap(lambda x:\"background-color:skyblue\", subset=[\"FamilySize\",\"FamilyName\",\"FamilyInfo\"]))\n\n\n\nfamily_info_source_features=[\"SibSp\",\"Parch\",\"Name\",\"Ticket\",\"Fare\",\"Embarked\"]\nfamily_size_source_features=[\"SibSp\",\"Parch\"]\nfamily_name_source_features=[\"Name\"]\n\n\nprint(\"### relevant features cols list\")\nprint(\"family_info_source_features = \",family_info_source_features)\nprint(\"family_size_source_features = \",family_size_source_features)\nprint(\"family_name_source_features = \",family_name_source_features)","metadata":{"execution":{"iopub.status.busy":"2022-07-06T04:15:05.955077Z","iopub.execute_input":"2022-07-06T04:15:05.956308Z","iopub.status.idle":"2022-07-06T04:15:06.003538Z","shell.execute_reply.started":"2022-07-06T04:15:05.956257Z","shell.execute_reply":"2022-07-06T04:15:06.002235Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Visualize the relationship between each features and the objective variable","metadata":{"papermill":{"duration":0.009214,"end_time":"2022-07-02T10:40:45.84736","exception":false,"start_time":"2022-07-02T10:40:45.838146","status":"completed"},"tags":[],"id":"381a55a2"}},{"cell_type":"code","source":"dict_survived={0:\"Dead\",1:\"Survived\"}\n\ndef arrange_bar(ax,sr):\n    ax.set_xticklabels(labels=ax.get_xticklabels(),rotation=30, horizontalalignment=\"center\")\n    ax.grid(axis=\"y\", linestyle=\"dotted\")\n    [ ax.text(i,count,count,horizontalalignment=\"center\") for i,count in enumerate(sr)]\n\nsr_survived=X_full[\"Survived\"].value_counts().rename(dict_survived)\n\nfig,axes = plt.subplots(nrows=1,ncols=2,figsize=(8,3))\nfig.subplots_adjust(wspace=0.5,hspace=0.5)\nsr_survived.plot.pie(autopct=\"%1.1f%%\",ax=axes[0])\nsr_survived.plot.bar(ax=axes[1])\n\narrange_bar(axes[1],sr_survived)\nplt.show()","metadata":{"papermill":{"duration":0.258331,"end_time":"2022-07-02T10:40:46.11508","exception":false,"start_time":"2022-07-02T10:40:45.856749","status":"completed"},"tags":[],"id":"0777cc0b","executionInfo":{"status":"ok","timestamp":1656775764211,"user_tz":-540,"elapsed":326,"user":{"displayName":"浜田光利","userId":"16018729205021549769"}},"outputId":"a65056dd-a6d3-4a77-d2bb-2d3f3ab8224f","execution":{"iopub.status.busy":"2022-07-06T04:15:23.053282Z","iopub.execute_input":"2022-07-06T04:15:23.053892Z","iopub.status.idle":"2022-07-06T04:15:23.350395Z","shell.execute_reply.started":"2022-07-06T04:15:23.053843Z","shell.execute_reply":"2022-07-06T04:15:23.348765Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dict_survived={0:\"Dead\",1:\"Survived\"}\n\ndef output_bars(df, key_feature, index={}):\n    def arrange_stack_bar(ax):\n        ax.set_xticklabels(labels=ax.get_xticklabels(), rotation=30, horizontalalignment=\"center\")\n        ax.grid(axis=\"y\", linestyle=\"dotted\")\n\n    fig,axes = plt.subplots(nrows=2, ncols=2, figsize=(12,8))\n    fig.subplots_adjust(wspace=0.5,hspace=0.5)\n    \n    df_1 = df.groupby([key_feature])[\"Survived\"].value_counts(sort=False).unstack()\n    df_2 = df[key_feature].value_counts()\n    df_3 = df.groupby([key_feature])[\"Survived\"].value_counts(sort=False,normalize=True).unstack()\n    df_1 = df_1.rename(columns=dict_survived)\n    df_3 = df_3.rename(columns=dict_survived)\n    if len(index)!=0:\n        df_1 = df_1.rename(index=index)\n        df_2 = df_2.rename(index=index)\n        df_3 = df_3.rename(index=index)\n        \n    \n    df_2.plot.pie(ax=axes[0,0],autopct=\"%1.1f%%\")\n    df_1.plot.bar(ax=axes[0,1],stacked=True)\n    df_1.plot.bar(ax=axes[1,0])\n    df_3.plot.bar(ax=axes[1,1],stacked=True)\n    \n\n    for i, (_,item) in enumerate(df_1.iterrows()):\n        axes[0,1].text(i, item.sum(), item.sum(),horizontalalignment=\"center\")  \n\n    for rect in axes[1,0].patches:\n        x,y=rect.xy\n        w = rect.get_width()\n        h = rect.get_height()\n        x = x+w/2\n        axes[1,0].text(x,y+h,h,horizontalalignment=\"center\")\n\n    arrange_stack_bar(axes[0,1])\n    arrange_stack_bar(axes[1,0])\n    arrange_stack_bar(axes[1,1])\n    \n    plt.show()\n\n\ndef output_box_hist(df, feature, bins=20, query=None):\n    if query == None:\n        fig,axes = plt.subplots(nrows=2,ncols=2,figsize=(12,8))\n    else:\n        fig,axes = plt.subplots(nrows=3,ncols=2,figsize=(12,12))\n        df.query(query)[feature].hist(ax=axes[2,0],bins=bins)\n        df.query(query).groupby(\"Survived\")[feature].plot.hist(ax=axes[2,1],bins=bins,alpha=0.5,legend=True,grid=True)\n        axes[2,1].legend(labels=[ dict_survived[int(text.get_text())] for text in axes[2,1].get_legend().get_texts()])\n        axes[2,1].set_title(f\"{feature}({query})\")\n    \n    fig.subplots_adjust(wspace=0.5,hspace=0.5)\n    \n    df.boxplot(ax=axes[0,0],column=[feature])\n    df.boxplot(ax=axes[0,1],column=[feature], by=\"Survived\")\n    axes[0,1].set_xticklabels([dict_survived[int(xticklabel.get_text())]  for xticklabel in axes[0,1].get_xticklabels() ])\n    \n    df[feature].hist(ax=axes[1,0],bins=bins)\n    df.groupby(\"Survived\")[feature].plot.hist(ax=axes[1,1],bins=bins, alpha=0.5, grid=True,legend=True)\n    axes[1,1].legend(labels=[ dict_survived[int(text.get_text())] for text in axes[1,1].get_legend().get_texts()] )\n    axes[1,1].set_title(feature)\n    plt.show()\n\n\ndict_pclass = {1:\"1: 1st(Upper)\", 2:\"2: 2nd(Middle)\", 3:\"3: 3rd(Lower)\"}\noutput_bars(X_full, \"Pclass\", dict_pclass)\noutput_bars(X_full, \"Sex\")\ndict_embark = {\"C\":\"Cherbourg\", \"Q\":\"Queenstown\", \"S\":\"Southampton\"}\noutput_bars(X_full,\"Embarked\",dict_embark)\noutput_bars(X_full,\"SibSp\")\noutput_bars(X_full,\"Parch\")\noutput_box_hist(X_full,\"Age\")\noutput_box_hist(X_full,\"Fare\",20,\"Fare < 200\")\n","metadata":{"papermill":{"duration":2.994319,"end_time":"2022-07-02T10:40:49.119712","exception":false,"start_time":"2022-07-02T10:40:46.125393","status":"completed"},"tags":[],"id":"9d48e675","executionInfo":{"status":"ok","timestamp":1656775764211,"user_tz":-540,"elapsed":4,"user":{"displayName":"浜田光利","userId":"16018729205021549769"}},"execution":{"iopub.status.busy":"2022-07-06T04:16:10.343268Z","iopub.execute_input":"2022-07-06T04:16:10.344433Z","iopub.status.idle":"2022-07-06T04:16:15.838245Z","shell.execute_reply.started":"2022-07-06T04:16:10.344376Z","shell.execute_reply":"2022-07-06T04:16:15.83682Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Correlations of quantitative features","metadata":{}},{"cell_type":"code","source":"X_full.loc[:,[\"Survived\",\"Pclass\",\"Age\",\"SibSp\",\"Parch\",\"Fare\"]].corr().style.background_gradient(axis=None)\nsns.pairplot(X_full)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-07-06T04:18:08.624134Z","iopub.execute_input":"2022-07-06T04:18:08.62504Z","iopub.status.idle":"2022-07-06T04:18:16.10762Z","shell.execute_reply.started":"2022-07-06T04:18:08.624993Z","shell.execute_reply":"2022-07-06T04:18:16.106713Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Classification features by data type","metadata":{"papermill":{"duration":0.015952,"end_time":"2022-07-02T10:40:50.922606","exception":false,"start_time":"2022-07-02T10:40:50.906654","status":"completed"},"tags":[],"id":"acf98ca7"}},{"cell_type":"code","source":"non_numerical_cols =[\"Pclass\"]\n\ndisplay(X.select_dtypes([\"float64\",\"int64\"]))\ndisplay(X.select_dtypes([\"object\"]))\n\nnumerical_cols = list(set(list(X.select_dtypes([\"float64\",\"int64\"]).columns))-set(non_numerical_cols))\nprint(\"numerical_cols=\",numerical_cols)\nnon_numerical_cols = non_numerical_cols+list(X.select_dtypes(\"object\").columns)\nprint(\"non_numerical_cols=\",non_numerical_cols)\n","metadata":{"papermill":{"duration":0.059453,"end_time":"2022-07-02T10:40:50.998441","exception":false,"start_time":"2022-07-02T10:40:50.938988","status":"completed"},"tags":[],"id":"04da0bd6","executionInfo":{"status":"ok","timestamp":1656775770222,"user_tz":-540,"elapsed":561,"user":{"displayName":"浜田光利","userId":"16018729205021549769"}},"outputId":"6fffef57-f259-4db4-991b-70886f3b062c","execution":{"iopub.status.busy":"2022-07-06T04:18:55.580816Z","iopub.execute_input":"2022-07-06T04:18:55.581254Z","iopub.status.idle":"2022-07-06T04:18:55.63249Z","shell.execute_reply.started":"2022-07-06T04:18:55.581219Z","shell.execute_reply":"2022-07-06T04:18:55.631305Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Exclude features that are not suitable for inputting to models","metadata":{"papermill":{"duration":0.016679,"end_time":"2022-07-02T10:40:51.033571","exception":false,"start_time":"2022-07-02T10:40:51.016892","status":"completed"},"tags":[],"id":"b763727c"}},{"cell_type":"code","source":"\ndef subtract(x,y):\n    return list(set(x)-set(y))\n\n\n# subtract high_missing_rate_cols\nnumerical_cols         = subtract(numerical_cols,        high_missing_rate_cols)\nnon_numerical_cols     = subtract(non_numerical_cols,    high_missing_rate_cols)\n\n# subtract high_cardinality_cols\nnumerical_cols         = subtract(numerical_cols,        high_cardinality_cols)\nnon_numerical_cols     = subtract(non_numerical_cols,    high_cardinality_cols)\n\n\nprint(\"numerical_cols=\",numerical_cols)\nprint(\"non_numerical_cols=\",non_numerical_cols)\n","metadata":{"papermill":{"duration":0.033996,"end_time":"2022-07-02T10:40:51.085225","exception":false,"start_time":"2022-07-02T10:40:51.051229","status":"completed"},"tags":[],"id":"a129fa55","executionInfo":{"status":"ok","timestamp":1656775770224,"user_tz":-540,"elapsed":39,"user":{"displayName":"浜田光利","userId":"16018729205021549769"}},"outputId":"623e5aed-4c0f-49a2-eb0b-2b261f35508a","execution":{"iopub.status.busy":"2022-07-06T04:19:07.601209Z","iopub.execute_input":"2022-07-06T04:19:07.601714Z","iopub.status.idle":"2022-07-06T04:19:07.60982Z","shell.execute_reply.started":"2022-07-06T04:19:07.601669Z","shell.execute_reply":"2022-07-06T04:19:07.608629Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Features classification summary","metadata":{"papermill":{"duration":0.022743,"end_time":"2022-07-02T10:40:51.127328","exception":false,"start_time":"2022-07-02T10:40:51.104585","status":"completed"},"tags":[],"id":"cce9ff90"}},{"cell_type":"code","source":"print(\"high_missing_rate_cols=\",high_missing_rate_cols)\nprint(\"high_cardinality_cols=\",high_cardinality_cols)\nprint()\nprint(\"numerical_cols=\",numerical_cols)\nprint(\"non_numerical_cols=\",non_numerical_cols)\nprint()\nprint(\"title_target_cols = \",title_target_cols)\nprint(\"family_info_source_features = \",family_info_source_features)\nprint(\"family_size_source_features = \",family_size_source_features)\nprint(\"family_name_source_features = \",family_name_source_features)","metadata":{"papermill":{"duration":0.029425,"end_time":"2022-07-02T10:40:51.175994","exception":false,"start_time":"2022-07-02T10:40:51.146569","status":"completed"},"tags":[],"id":"a6560c6d","executionInfo":{"status":"ok","timestamp":1656775770226,"user_tz":-540,"elapsed":38,"user":{"displayName":"浜田光利","userId":"16018729205021549769"}},"outputId":"7f985c15-3ab8-42c2-bbe0-94f154f800c0","execution":{"iopub.status.busy":"2022-07-06T04:19:17.647013Z","iopub.execute_input":"2022-07-06T04:19:17.64751Z","iopub.status.idle":"2022-07-06T04:19:17.655627Z","shell.execute_reply.started":"2022-07-06T04:19:17.647472Z","shell.execute_reply":"2022-07-06T04:19:17.654505Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Chapter: Building Models and Pipelines","metadata":{}},{"cell_type":"markdown","source":"## Preprocessing pipeline","metadata":{"papermill":{"duration":0.016225,"end_time":"2022-07-02T10:40:51.210015","exception":false,"start_time":"2022-07-02T10:40:51.19379","status":"completed"},"tags":[],"id":"6bc95147"}},{"cell_type":"code","source":"\ndef make_pipline_set(encoder_desc,lc_encoder_class,lc_encoder_params,hc_encoder,hc_encoder_params,inner_search_X=None,inner_search_y=None,inner_model_params={}):\n    \n    class ExtractTitlePipeline(BaseEstimator, TransformerMixin):\n        def __init__(self,title_list,target_col_name,output_col_name):\n            super().__init__()\n            self.title_list=title_list\n            self.target_col_name=target_col_name\n            self.output_col_name=output_col_name\n\n        def fit(self, X, y=None):\n            return self\n\n        def transform(self, X):\n            if self.target_col_name not in X.columns:\n                return X\n\n            def func_title(x):\n                lower_x=x.lower()\n                words=[]\n                name_words = re.split(\"[,.\\s\\\"\\']\",lower_x)\n                for w in name_words:\n                    if w in self.title_list:\n                        words.append(w)\n                return \",\".join(words) if words else None\n\n            tmp=X[[self.target_col_name]].copy()\n            tmp[self.output_col_name]=X[[self.target_col_name]].applymap(func_title)\n            return tmp[[self.output_col_name]]\n\n    class ExtractFamilyInfoPipeline(BaseEstimator, TransformerMixin):\n        def __init__(self):\n            super().__init__()\n\n        def fit(self, X, y=None):\n            return self\n\n        def transform(self, X):\n            tmp = X[[\"SibSp\",\"Parch\",\"Name\",\"Ticket\",\"Fare\",\"Embarked\"]].copy()\n            tmp[\"FamilySize\"] = tmp[\"SibSp\"] + tmp[\"Parch\"] + 1\n            tmp[\"FamilyName\"] = tmp[\"Name\"].str.split(\",\",expand=True)[0]\n            tmp[\"FamilyInfo\"] = tmp[\"Ticket\"]+\"/\"+tmp[\"Fare\"].astype(str)+\"/\"+tmp[\"Embarked\"].fillna(\"\")+\"/\"+tmp[\"FamilyName\"]+\"/\"+tmp[\"FamilySize\"].astype(str)\n            return tmp[[\"FamilyInfo\"]]\n\n\n\n    class ExtractFamilySizePipeline(BaseEstimator, TransformerMixin):\n        def __init__(self):\n            super().__init__()\n\n        def fit(self, X, y=None):\n            return self\n\n        def transform(self, X):\n            tmp = X[[\"SibSp\",\"Parch\"]].copy()\n            tmp[\"FamilySize\"] = tmp[\"SibSp\"] + tmp[\"Parch\"] + 1\n            return tmp[[\"FamilySize\"]]\n\n    \n    class ExtractFamilyNamePipeline(BaseEstimator, TransformerMixin):\n        def __init__(self):\n            super().__init__()\n\n        def fit(self, X, y=None):\n            return self\n\n        def transform(self, X):\n            tmp = X[[\"Name\"]].copy()\n            tmp[\"FamilyName\"] = tmp[\"Name\"].str.split(\",\",expand=True)[0]\n            return tmp[[\"FamilyName\"]]\n\n    \n        \n    def get_inner_model_params(inner_preprocessor,inner_search_X,inner_search_y):\n        inner_model = RandomForestClassifier(random_state=1)\n        inner_search_pipe = Pipeline(steps=[(\"preprocess\",inner_preprocessor),(\"inner_model\",inner_model)])\n        parameters=[\n            {'inner_model__n_estimators': [1000,2000],\n             'inner_model__max_depth' : [5,10],\n             'inner_model__min_samples_leaf': [5,10],\n             'inner_model__min_samples_split': [5,10],}\n        ]\n        inner_grid_search = GridSearchCV(inner_search_pipe,parameters, scoring=\"accuracy\", cv=5, n_jobs=-1, verbose=0)\n        inner_grid_search.fit(inner_search_X,inner_search_y)\n        best_params={k.replace(\"inner_model__\",\"\"):v for k,v in inner_grid_search.best_params_.items()}\n        return best_params\n\n        \n        \n    numerical_pipe = Pipeline(steps=[\n        (\"impute\",SimpleImputer(strategy=\"median\"))\n    ])\n\n    categorical_pipe = Pipeline(steps=[\n        (\"low cardinality encoder\",lc_encoder_class(**lc_encoder_params))\n    ])\n\n    title_pipe = Pipeline(steps=[\n        (\"title\", ExtractTitlePipeline(title_list=title_list,target_col_name=\"Name\",output_col_name=\"Title\")),\n        (\"low cardinality encoder\",lc_encoder_class(**lc_encoder_params))\n    ])\n\n    family_info_pipe = Pipeline(steps=[\n        (\"family_info\", ExtractFamilyInfoPipeline()),\n        (\"high cardinality encoder\", hc_encoder_class(**hc_encoder_params))\n    ])\n\n    family_name_pipe = Pipeline(steps=[\n        (\"family_name\", ExtractFamilyNamePipeline()),\n        (\"high cardinality encoder\", hc_encoder_class(**hc_encoder_params))\n    ])\n\n    family_size_pipe = Pipeline(steps=[\n        (\"family_size\", ExtractFamilySizePipeline()),\n        (\"impute\",SimpleImputer(strategy=\"median\"))\n    ])\n\n    \n    features_pipeline_complex= ColumnTransformer(transformers=[\n        (\"numerical\",   numerical_pipe,     ['Parch','Fare','Age','SibSp']),\n        (\"categorical\", categorical_pipe,   ['Sex','Embarked','Pclass']),\n        (\"title\",       title_pipe,         [\"Name\"]),\n        (\"familyName\",  family_name_pipe,   [\"Name\"]),\n        (\"familySize\",  family_size_pipe,   [\"SibSp\",\"Parch\"]),\n        (\"familyInfo\",  family_info_pipe,   [\"SibSp\",\"Parch\",\"Name\",\"Ticket\",\"Fare\",\"Embarked\"]),\n    ])\n\n\n    features_pipeline_simple= ColumnTransformer(transformers=[\n        (\"numerical\",   numerical_pipe,     ['Parch','Fare','Age','SibSp']),\n        (\"categorical\", categorical_pipe,   ['Sex','Embarked','Pclass']),\n        (\"title\",       title_pipe,         [\"Name\"]),\n    ])\n\n\n    features_pipeline_c_pca = Pipeline(steps=[\n        (\"transform\", features_pipeline_complex),\n        (\"scale\",     StandardScaler()),\n        (\"pca\",       PCA())\n    ])\n\n    features_pipeline_s_pca = Pipeline(steps=[\n        (\"transform\", features_pipeline_simple),\n        (\"scale\",     StandardScaler()),\n        (\"pca\",       PCA())\n    ])\n\n    \n    if len(inner_model_params) != 0:\n        inner_model_params_for_s =inner_model_params\n        inner_model_params_for_c =inner_model_params\n    elif inner_search_X is None:\n        inner_model_params_for_s = {\"n_estimators\":1000,\"max_depth\":10}\n        inner_model_params_for_c = {\"n_estimators\":1000,\"max_depth\":10}\n        print(f\"'{encoder_desc}' inner model params (default)= {inner_model_params}\")\n    elif inner_search_X is not None:\n        inner_model_params_for_s = get_inner_model_params(features_pipeline_simple,inner_search_X,inner_search_y.to_numpy().reshape(-1))\n        inner_model_params_for_c = get_inner_model_params(features_pipeline_complex,inner_search_X,inner_search_y.to_numpy().reshape(-1))\n        print(f\"'{encoder_desc}' inner model params for simple (grid searched)= {inner_model_params_for_s}\")\n        print(f\"'{encoder_desc}' inner model params for complex (grid searched)= {inner_model_params_for_c}\")\n    \n\n    features_pipeline_c_sfm = Pipeline(steps=[\n        (\"process\", features_pipeline_complex),\n        (\"sfm\",     SelectFromModel(RandomForestClassifier(random_state=1,**inner_model_params_for_c)))\n    ])\n\n    features_pipeline_s_sfm = Pipeline(steps=[\n        (\"process\", features_pipeline_simple),\n        (\"sfm\",     SelectFromModel(RandomForestClassifier(random_state=1,**inner_model_params_for_s)))\n    ])\n\n    features_pipeline_c_rfe = Pipeline(steps=[\n        (\"preprocess\",features_pipeline_complex),\n        (\"model\",RFECV(RandomForestClassifier(random_state=1,**inner_model_params), min_features_to_select=3,cv=5,verbose=0))\n    ])\n\n    features_pipeline_s_rfe = Pipeline(steps=[\n        (\"preprocess\",features_pipeline_simple),\n        (\"model\",RFECV(RandomForestClassifier(random_state=1,**inner_model_params), min_features_to_select=3,cv=5,verbose=0))\n    ])\n\n    \n    pipe_dict={\n        f\"{encoder_desc}_01_s\"         : features_pipeline_simple,\n        f\"{encoder_desc}_02_s_pca\"     : features_pipeline_s_pca,\n        f\"{encoder_desc}_03_s_sfm\"     : features_pipeline_s_sfm,\n        f\"{encoder_desc}_04_s_rfe\"     : features_pipeline_s_rfe,\n        f\"{encoder_desc}_05_c\"         : features_pipeline_complex,\n        f\"{encoder_desc}_06_c_pca\"     : features_pipeline_c_pca,\n        f\"{encoder_desc}_07_c_sfm\"     : features_pipeline_c_sfm,\n        # f\"{encoder_desc}_08_c_rfe\"     : features_pipeline_c_rfe,   # Omitted this line to finish within the execution time limit on kaggle.\n    }\n    \n    return pipe_dict\n\n# List of tuples. The format of tuple is (encoder_desc,lc_encoder_class,lc_encoder_params,hc_encoder_class,hc_encoder_params).\nencoders=[\n    (\"od-loo\",  ce.OrdinalEncoder,    {\"handle_unknown\":\"value\",\"handle_missing\":\"value\"}, ce.LeaveOneOutEncoder, {\"handle_unknown\":\"value\",\"handle_missing\":\"value\"}),\n    (\"oh-loo\",  ce.OneHotEncoder,     {\"handle_unknown\":\"value\",\"handle_missing\":\"value\"}, ce.LeaveOneOutEncoder, {\"handle_unknown\":\"value\",\"handle_missing\":\"value\"}),\n    (\"oh-oh\",   ce.OneHotEncoder,     {\"handle_unknown\":\"value\",\"handle_missing\":\"value\"}, ce.OneHotEncoder,      {\"handle_unknown\":\"value\",\"handle_missing\":\"value\"}),\n]\n\npipe_dict={}\nfor encoder_desc,lc_encoder_class,lc_encoder_params,hc_encoder_class,hc_encoder_params in encoders:\n    tmp_dict = make_pipline_set(encoder_desc,lc_encoder_class,lc_encoder_params,hc_encoder_class,hc_encoder_params,X,y)\n    pipe_dict.update(tmp_dict)\n\nprint(\"### builded pipelines\")\nset_config(display=\"diagram\")\nfor pipe_name,pipe in pipe_dict.items():\n    print(pipe_name)\n    display(pipe)\n    print()\n\n","metadata":{"papermill":{"duration":0.917738,"end_time":"2022-07-02T10:40:52.144201","exception":false,"start_time":"2022-07-02T10:40:51.226463","status":"completed"},"tags":[],"id":"f6184ab3","executionInfo":{"status":"ok","timestamp":1656775770227,"user_tz":-540,"elapsed":32,"user":{"displayName":"浜田光利","userId":"16018729205021549769"}},"outputId":"bcc85a3c-64bd-48e2-de33-c2ce53dc0062","execution":{"iopub.status.busy":"2022-07-06T04:23:11.930932Z","iopub.execute_input":"2022-07-06T04:23:11.931398Z","iopub.status.idle":"2022-07-06T04:28:42.137083Z","shell.execute_reply.started":"2022-07-06T04:23:11.931357Z","shell.execute_reply":"2022-07-06T04:28:42.134363Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Helpers of building and testing models","metadata":{"papermill":{"duration":0.016356,"end_time":"2022-07-02T10:40:52.17699","exception":false,"start_time":"2022-07-02T10:40:52.160634","status":"completed"},"tags":[],"id":"a9e5825b"}},{"cell_type":"code","source":"\n# Helper of Prediction for test data\ndef predict_test_dataset(estimator,output_file_name,display_detail=False):\n    # Load the test data set.\n    test_X=pd.read_csv(os.path.join(INPUT_PATH if EXTERNAL else '/kaggle/input',\"titanic/test.csv\"),index_col=\"PassengerId\")\n    if display_detail:\n        print(\"test_X.shape :\",test_X.shape)\n        display(test_X.head())\n        display(test_X.info())\n        display(test_X.describe())\n\n    # Prediction\n    test_preds = estimator.predict(test_X)\n    output = pd.DataFrame({'PassengerId': test_X.index,'Survived': test_preds})\n    output.to_csv(output_file_name, index=False)\n    print(\"submission file       :\",output_file_name)\n\n    \n# Helper of XGBM + cross_val_score\ndef xgbm_cv(pipe,pipe_name,X,y):\n    model = XGBClassifier(n_estimators=1000,max_depth=10,random_state=1)\n    estimator = Pipeline(steps=[(\"preprocess\",pipe),(\"model\",model)])\n    score = cross_val_score(estimator,X,y,scoring=\"accuracy\",cv=5,verbose=0,error_score='raise',n_jobs=-1)\n    total_score= score.mean()\n    print(\"xgbm_cv mean_score    :\",total_score)\n    estimator.fit(X,y)\n    predict_test_dataset(estimator,f\"{pipe_name}_xgbm_cv_submission.csv\")\n    return total_score\n\n\n# Helper of XGBM + GridSearchCV\ndef xgbm_gridsearchcv(pipe,pipe_name,X,y):\n    param_grid={\n        \"model__n_estimators\": [1000,2000],\n        \"model__max_depth\": [5,10,20]\n    }\n\n    model = XGBClassifier(random_state=1)\n    estimator = Pipeline(steps=[(\"preprocess\",pipe),(\"model\",model)])\n    grid = GridSearchCV(estimator,param_grid, verbose=0,scoring=\"accuracy\",n_jobs=-1)\n    grid.fit(X, y)\n    print(\"gridsearch best score :\",grid.best_score_)\n    print(\"gridsearch best params:\",grid.best_params_)\n    grid.best_estimator_.fit(X,y)\n    predict_test_dataset(grid.best_estimator_,f\"{pipe_name}_xgbm_grdcv_submission.csv\")\n    return grid.best_score_","metadata":{"papermill":{"duration":0.0323,"end_time":"2022-07-02T10:40:52.22598","exception":false,"start_time":"2022-07-02T10:40:52.19368","status":"completed"},"tags":[],"id":"376ea280","executionInfo":{"status":"ok","timestamp":1656775770229,"user_tz":-540,"elapsed":29,"user":{"displayName":"浜田光利","userId":"16018729205021549769"}},"execution":{"iopub.status.busy":"2022-07-06T04:28:42.139003Z","iopub.status.idle":"2022-07-06T04:28:42.139549Z","shell.execute_reply.started":"2022-07-06T04:28:42.139312Z","shell.execute_reply":"2022-07-06T04:28:42.139346Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Execute all patterns","metadata":{"papermill":{"duration":0.016799,"end_time":"2022-07-02T10:40:52.259112","exception":false,"start_time":"2022-07-02T10:40:52.242313","status":"completed"},"tags":[],"id":"76d1c5c0"}},{"cell_type":"code","source":"reshaped_y = y.to_numpy().reshape(-1)\n\nresult_xgbm_cv=[]\nresult_xgbm_grdcv=[]\nfor pipe_name,pipe in pipe_dict.items():\n    print(f\"### {pipe_name} #################\")\n    result_xgbm_cv.append( xgbm_cv(pipe,pipe_name,X,reshaped_y) )\n    print()\n    result_xgbm_grdcv.append( xgbm_gridsearchcv(pipe,pipe_name,X,reshaped_y) )\n    print()\n\ndf_result=pd.DataFrame({\"xgbm_cv\":result_xgbm_cv,\"xgbm_grdcv\":result_xgbm_grdcv},index=pipe_dict.keys())\ndisplay(df_result.style.highlight_max(color='lightblue'))\nprint()\nprint(\"### best model and submission file #############\")\ncsv_modelscore_dict={}\nfor colname in df_result:\n    for row_index,value in df_result[colname].items():\n        submission_csv = f\"{row_index}_{colname}_submission.csv\"\n        csv_modelscore_dict[submission_csv]=df_result.loc[row_index,colname]\n\nbest_csv,best_model_score=sorted(csv_modelscore_dict.items(),key=lambda x:x[1],reverse=True)[0]\nprint(\"best model submission csv =\",best_csv)\nprint(\"best model score          =\",best_model_score)\nprint()\nprint(\"### submission files (listing *_submission.csv on working directory.) #############\")\n!ls -la *_submission.csv\n","metadata":{"papermill":{"duration":1375.73824,"end_time":"2022-07-02T11:03:48.013592","exception":false,"start_time":"2022-07-02T10:40:52.275352","status":"completed"},"tags":[],"id":"7cf47546","executionInfo":{"status":"ok","timestamp":1656776379050,"user_tz":-540,"elapsed":608848,"user":{"displayName":"浜田光利","userId":"16018729205021549769"}},"outputId":"d173313a-a901-4db5-8fad-f38d354e1158","execution":{"iopub.status.busy":"2022-07-05T06:26:42.19597Z","iopub.execute_input":"2022-07-05T06:26:42.196567Z","iopub.status.idle":"2022-07-05T06:52:41.436808Z","shell.execute_reply.started":"2022-07-05T06:26:42.196534Z","shell.execute_reply":"2022-07-05T06:52:41.435406Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Submit the file predicted by best pattern (pipeline + model).","metadata":{"id":"QwQWbC0gNvko"}},{"cell_type":"code","source":"\nif EXTERNAL and AUTO_SUBMIT:\n    !kaggle competitions submit -c $PROJECT_NAME -f $best_csv -m \"$SUBMIT_COMMENT\"\n","metadata":{"papermill":{"duration":0.019321,"end_time":"2022-07-02T11:03:48.052579","exception":false,"start_time":"2022-07-02T11:03:48.033258","status":"completed"},"tags":[],"id":"14ef5e21","executionInfo":{"status":"ok","timestamp":1656776379051,"user_tz":-540,"elapsed":8,"user":{"displayName":"浜田光利","userId":"16018729205021549769"}},"execution":{"iopub.status.busy":"2022-07-05T06:52:41.438933Z","iopub.execute_input":"2022-07-05T06:52:41.439311Z","iopub.status.idle":"2022-07-05T06:52:41.445541Z","shell.execute_reply.started":"2022-07-05T06:52:41.439273Z","shell.execute_reply":"2022-07-05T06:52:41.444314Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}