{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## 目的\n\n今回のコンペの目的は、failuerが0となるか1となるかを予測すること。評価はAUCROCによるため、failuerが1となる確率を求めればよい。(正確にいえば、failuerの大小関係のみで判定するため、0～1の間に収める必要はない)\n\n<https://www.kaggle.com/competitions/tabular-playground-series-aug-2022/overview/evaluation>\n\nAUCROCについては次の素晴らしいノートブックが参考になる\n\n<https://www.kaggle.com/code/mehrankazeminia/tpsmay22-auc-ensembling>\n\n与えられるデータは次のとおり\n\n* product_code: 製品を区分するコード\n* loading: 負荷。この量の液体を吸収した後故障するかどうか判定する\n* その他: その製品に対する計測値","metadata":{}},{"cell_type":"markdown","source":"## ライブラリのインポート&データ読み込み","metadata":{}},{"cell_type":"code","source":"# 何かと使うやつ.\nimport os\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nfrom collections import Counter\nimport time, gc, string, math\nfrom tqdm.notebook import tqdm\nimport warnings\nimport shutil\nfrom collections import defaultdict\nimport heapq\nimport datetime\nimport random\nfrom collections import OrderedDict\nimport glob\nimport copy\n\n# sklearn\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.model_selection import KFold\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.metrics import roc_auc_score\n\n# LGBM\nfrom lightgbm import LGBMClassifier\nimport lightgbm as lgb","metadata":{"execution":{"iopub.status.busy":"2022-08-01T03:42:39.849165Z","iopub.execute_input":"2022-08-01T03:42:39.849557Z","iopub.status.idle":"2022-08-01T03:42:39.858804Z","shell.execute_reply.started":"2022-08-01T03:42:39.849525Z","shell.execute_reply":"2022-08-01T03:42:39.857724Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pd.set_option('display.max_columns', 200)\npd.set_option('display.max_rows', 200)","metadata":{"execution":{"iopub.status.busy":"2022-08-01T03:40:44.999296Z","iopub.execute_input":"2022-08-01T03:40:44.999698Z","iopub.status.idle":"2022-08-01T03:40:45.004988Z","shell.execute_reply.started":"2022-08-01T03:40:44.999667Z","shell.execute_reply":"2022-08-01T03:40:45.003756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = pd.read_csv('../input/tabular-playground-series-aug-2022/train.csv', index_col='id')\ntest = pd.read_csv('../input/tabular-playground-series-aug-2022/test.csv', index_col='id')","metadata":{"execution":{"iopub.status.busy":"2022-08-01T03:40:45.107156Z","iopub.execute_input":"2022-08-01T03:40:45.107976Z","iopub.status.idle":"2022-08-01T03:40:45.335045Z","shell.execute_reply.started":"2022-08-01T03:40:45.107921Z","shell.execute_reply":"2022-08-01T03:40:45.334147Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## EDA: データの傾向をつかむ\n\n* どういうデータが入っているのか\n* 各データはどういう傾向なのか","metadata":{}},{"cell_type":"code","source":"# product_code, attribute_0, attribute_1は文字列なので、数値への置換が必要となる\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-01T03:40:45.337431Z","iopub.execute_input":"2022-08-01T03:40:45.338208Z","iopub.status.idle":"2022-08-01T03:40:45.373116Z","shell.execute_reply.started":"2022-08-01T03:40:45.338158Z","shell.execute_reply":"2022-08-01T03:40:45.372294Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# それ以外は、すべて整数若しくは浮動小数点である\ntrain.dtypes","metadata":{"execution":{"iopub.status.busy":"2022-08-01T03:40:45.374673Z","iopub.execute_input":"2022-08-01T03:40:45.375046Z","iopub.status.idle":"2022-08-01T03:40:45.383683Z","shell.execute_reply.started":"2022-08-01T03:40:45.375016Z","shell.execute_reply":"2022-08-01T03:40:45.382718Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 整数について、ハイカーディナリのものは存在しない。\ntrain.nunique()","metadata":{"execution":{"iopub.status.busy":"2022-08-01T03:40:45.388513Z","iopub.execute_input":"2022-08-01T03:40:45.388959Z","iopub.status.idle":"2022-08-01T03:40:45.421072Z","shell.execute_reply.started":"2022-08-01T03:40:45.388924Z","shell.execute_reply":"2022-08-01T03:40:45.419955Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# ほとんどのデータで、平均が非常に大きくなっている。\n# LGBMのような勾配ブースティング木モデルでは影響はないが、ニューラルネット(NN)モデル等を使用する際には、スケーリングが必須となる\n# (NNにとって、5とか10とかいう数字はあまりに大きすぎるため、壊れてしまう)\ntrain.describe()","metadata":{"execution":{"iopub.status.busy":"2022-08-01T03:40:45.491661Z","iopub.execute_input":"2022-08-01T03:40:45.492502Z","iopub.status.idle":"2022-08-01T03:40:45.606938Z","shell.execute_reply.started":"2022-08-01T03:40:45.492452Z","shell.execute_reply":"2022-08-01T03:40:45.605771Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# いくつか欠損値がある。欠損値埋めが重要となる\n# TPS2206(欠損値埋めのコンペ)での手法が役に立つかもしれない\ntrain.isna().sum()","metadata":{"execution":{"iopub.status.busy":"2022-08-01T03:40:45.609322Z","iopub.execute_input":"2022-08-01T03:40:45.610062Z","iopub.status.idle":"2022-08-01T03:40:45.630466Z","shell.execute_reply.started":"2022-08-01T03:40:45.610005Z","shell.execute_reply":"2022-08-01T03:40:45.629579Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# TODO: いくつかのグラフを作って比較する","metadata":{"execution":{"iopub.status.busy":"2022-08-01T03:40:45.647417Z","iopub.execute_input":"2022-08-01T03:40:45.648118Z","iopub.status.idle":"2022-08-01T03:40:45.652975Z","shell.execute_reply.started":"2022-08-01T03:40:45.648068Z","shell.execute_reply":"2022-08-01T03:40:45.652013Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## シンプルなLGBMモデル\n\n* 欠損値があっても動作するLBGMを使って、簡単なモデルを作成してみる。","metadata":{}},{"cell_type":"code","source":"# 文字列を整数にする\nall_df = pd.concat([train, test])\n\nfor col in all_df.columns:\n    if all_df[col].dtype != 'O':\n        continue\n    le = LabelEncoder()\n    all_df[col] = le.fit_transform(all_df[col])","metadata":{"execution":{"iopub.status.busy":"2022-08-01T03:40:45.698980Z","iopub.execute_input":"2022-08-01T03:40:45.700014Z","iopub.status.idle":"2022-08-01T03:40:45.768999Z","shell.execute_reply.started":"2022-08-01T03:40:45.699975Z","shell.execute_reply":"2022-08-01T03:40:45.767826Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 学習対象のcolumnを取得する\ntrain_col_list = []\nfor col in all_df.columns:\n    if col != 'failure':\n        train_col_list.append(col)","metadata":{"execution":{"iopub.status.busy":"2022-08-01T03:40:56.536986Z","iopub.execute_input":"2022-08-01T03:40:56.537431Z","iopub.status.idle":"2022-08-01T03:40:56.548152Z","shell.execute_reply.started":"2022-08-01T03:40:56.537393Z","shell.execute_reply":"2022-08-01T03:40:56.547008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train = all_df[:train.shape[0]]\ntest = all_df[train.shape[0]:]","metadata":{"execution":{"iopub.status.busy":"2022-08-01T03:40:58.366984Z","iopub.execute_input":"2022-08-01T03:40:58.367841Z","iopub.status.idle":"2022-08-01T03:40:58.373615Z","shell.execute_reply.started":"2022-08-01T03:40:58.367798Z","shell.execute_reply":"2022-08-01T03:40:58.372581Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# クロスバリデーションの設定をする\nn_splits = 5\nskf = StratifiedKFold(n_splits=n_splits, shuffle=True, random_state=42)\nfor n, (train_index, val_index) in enumerate(skf.split(train, train['failure'])):\n    train.loc[val_index, 'fold'] = int(n)","metadata":{"execution":{"iopub.status.busy":"2022-08-01T03:41:00.230952Z","iopub.execute_input":"2022-08-01T03:41:00.231369Z","iopub.status.idle":"2022-08-01T03:41:00.254891Z","shell.execute_reply.started":"2022-08-01T03:41:00.231333Z","shell.execute_reply":"2022-08-01T03:41:00.253672Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"train.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-01T03:41:02.668276Z","iopub.execute_input":"2022-08-01T03:41:02.669325Z","iopub.status.idle":"2022-08-01T03:41:02.703034Z","shell.execute_reply.started":"2022-08-01T03:41:02.669283Z","shell.execute_reply":"2022-08-01T03:41:02.701840Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for fold in range(n_splits):\n    print(\"=\"*10, fold, \"=\"*10)\n    train_folds = train[train['fold']!=fold]\n    valid_folds = train[train['fold']==fold]\n    X_train = train_folds[train_col_list]\n    y_train = train_folds['failure']\n    X_valid = valid_folds[train_col_list]\n    y_valid = valid_folds['failure']\n    X_test = test[train_col_list]\n\n    model = LGBMClassifier(random_state=42)\n    model.fit(X_train, y_train, eval_metric=\"loss\", eval_set=[(X_valid, y_valid)], callbacks=[lgb.early_stopping(stopping_rounds=10)], verbose=128)\n    y_pred = model.predict_proba(X_valid)[:, 1]\n\n    auc = roc_auc_score(y_valid, y_pred)\n    print(\"auc: \", auc)\n    lgb.plot_importance(model, max_num_features=20, title=fold)\n    plt.show()\n\n    y_test = model.predict_proba(X_test)[:, 1]\n    test[fold] = y_test","metadata":{"execution":{"iopub.status.busy":"2022-08-01T03:49:27.031683Z","iopub.execute_input":"2022-08-01T03:49:27.032200Z","iopub.status.idle":"2022-08-01T03:49:33.899380Z","shell.execute_reply.started":"2022-08-01T03:49:27.032157Z","shell.execute_reply":"2022-08-01T03:49:33.898313Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"AUCが約0.58と非常に低い(下限は0.5のため、ほとんど予測できていない)","metadata":{}},{"cell_type":"code","source":"test.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-01T03:44:33.861336Z","iopub.execute_input":"2022-08-01T03:44:33.862514Z","iopub.status.idle":"2022-08-01T03:44:33.900630Z","shell.execute_reply.started":"2022-08-01T03:44:33.862474Z","shell.execute_reply":"2022-08-01T03:44:33.899810Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub = pd.read_csv('../input/tabular-playground-series-aug-2022/sample_submission.csv')","metadata":{"execution":{"iopub.status.busy":"2022-08-01T03:46:01.257282Z","iopub.execute_input":"2022-08-01T03:46:01.257742Z","iopub.status.idle":"2022-08-01T03:46:01.287745Z","shell.execute_reply.started":"2022-08-01T03:46:01.257707Z","shell.execute_reply":"2022-08-01T03:46:01.286687Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub['failure'] = test[[0,1,2,3,4]].mean(axis=1).values","metadata":{"execution":{"iopub.status.busy":"2022-08-01T03:47:11.324684Z","iopub.execute_input":"2022-08-01T03:47:11.325091Z","iopub.status.idle":"2022-08-01T03:47:11.334402Z","shell.execute_reply.started":"2022-08-01T03:47:11.325059Z","shell.execute_reply":"2022-08-01T03:47:11.333175Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sub.to_csv(\"submission.csv\", index=False)\nsub","metadata":{"execution":{"iopub.status.busy":"2022-08-01T03:47:29.854359Z","iopub.execute_input":"2022-08-01T03:47:29.854743Z","iopub.status.idle":"2022-08-01T03:47:29.953552Z","shell.execute_reply.started":"2022-08-01T03:47:29.854713Z","shell.execute_reply":"2022-08-01T03:47:29.952499Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## このあとの流れ\n\n* 文字列を整数にするところをもっと工夫する。例えば、one_hot_encodingとか\n* 欠損値補完をもっと工夫する。TPS2206みたいに。\n* TabNetやPytorchのようなより高度なモデルを使ってみる\n* SHAPなどを使って、特徴量の相互関係を分析する\n* UMAPなどを使って、データの可視化などを取り入れる\n* すごい人のnotebookを見て、良いアイデアを取り入れていく","metadata":{}}]}