{"cells":[{"metadata":{"_uuid":"cf675c2dde7950e45a15230a1e7f321f9a3305fe"},"cell_type":"markdown","source":"## Kernels and discussions used in this kernel\n- [Oliver's kernel](https://www.kaggle.com/ogrellier/plasticc-in-a-kernel-meta-and-data)\n- [Alexander Firsov's kernel](https://www.kaggle.com/alexfir/fast-test-set-reading)\n- [Iprapas' kernel](https://www.kaggle.com/iprapas/ideas-from-kernels-and-discussion-lb-1-135)\n- [Chia-Ta Tsai's kernel](https://www.kaggle.com/cttsai/forked-lgbm-w-ideas-from-kernels-and-discuss)\n- [Lving's kernel](https://www.kaggle.com/qianchao/smote-with-imbalance-data)\n- [Scirpus' class 99 method](https://www.kaggle.com/c/PLAsTiCC-2018/discussion/72104)\n- [My something different kernel](https://www.kaggle.com/jimpsull/something-different)\n- [My Smote the training set kernel](https://www.kaggle.com/jimpsull/smote-the-training-sets)"},{"metadata":{"_uuid":"656982859d470a3dd6b4da04916ea1b3ec84fa3d"},"cell_type":"markdown","source":"## The purpose of this kernel is to bring together features\n- the first 69 are from our 1.080 kernel which came via Oliver, Iprapas, and Chia-ta Tsai\n- integrating smote brought that to 1.052\n- adding Scirpus' class 99 method brought it to 1.039\n- adding seven (7) features from my 'something different' kernel brought it to 1.030\n- increasing k from 5 --> 12 brought that to 1.029\n- An additional class 99 method brought it to 1.028\n- ongoing efforts and tried and failed lists at the end of the kernel\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir('../input'))\nprint(os.listdir(\"../input/writefeaturetablefromsmotedartset\"))\nprint(os.listdir('../input/normalizesomethingdifferentfeatures'))\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f9ccb3ee938e6680e6876a80db3ccaa5801ff3e1"},"cell_type":"markdown","source":"## From Chia-Ta Tsai's script"},{"metadata":{"trusted":true,"_uuid":"458bb7b057c7d453c7f5acc33c70ff2646aabba5"},"cell_type":"code","source":"\"\"\"\n\nThis script is forked from chia-ta tsai's kernel of which he said:\n\nThis script is forked from iprapas's notebook \nhttps://www.kaggle.com/iprapas/ideas-from-kernels-and-discussion-lb-1-135\n\n#    https://www.kaggle.com/ogrellier/plasticc-in-a-kernel-meta-and-data\n#    https://www.kaggle.com/c/PLAsTiCC-2018/discussion/70908\n#    https://www.kaggle.com/meaninglesslives/simple-neural-net-for-time-series-classification\n#\n\"\"\"\n\nimport sys, os\nimport argparse\nimport time\nfrom datetime import datetime as dt\nimport gc; gc.enable()\nfrom functools import partial, wraps\n\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport numpy as np # linear algebra\nnp.warnings.filterwarnings('ignore')\n\nfrom sklearn.model_selection import StratifiedKFold\nfrom tsfresh.feature_extraction import extract_features\nfrom xgboost import XGBClassifier\nfrom lightgbm import LGBMClassifier","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"96be97c7c35646a87560d4aec858b03d5089f850"},"cell_type":"markdown","source":"## Oliver's functions"},{"metadata":{"trusted":true,"_uuid":"52bba423f7570305957eb2b7236d20559ae27295"},"cell_type":"code","source":"\ndef multi_weighted_logloss(y_true, y_preds, classes, class_weights):\n    \"\"\"\n    refactor from\n    @author olivier https://www.kaggle.com/ogrellier\n    multi logloss for PLAsTiCC challenge\n    \"\"\"\n    y_p = y_preds.reshape(y_true.shape[0], len(classes), order='F')\n    # Trasform y_true in dummies\n    y_ohe = pd.get_dummies(y_true)\n    # Normalize rows and limit y_preds to 1e-15, 1-1e-15\n    y_p = np.clip(a=y_p, a_min=1e-15, a_max=1 - 1e-15)\n    # Transform to log\n    y_p_log = np.log(y_p)\n    # Get the log for ones, .values is used to drop the index of DataFrames\n    # Exclude class 99 for now, since there is no class99 in the training set\n    # we gave a special process for that class\n    y_log_ones = np.sum(y_ohe.values * y_p_log, axis=0)\n    # Get the number of positives for each class\n    nb_pos = y_ohe.sum(axis=0).values.astype(float)\n    # Weight average and divide by the number of positives\n    class_arr = np.array([class_weights[k] for k in sorted(class_weights.keys())])\n    y_w = y_log_ones * class_arr / nb_pos\n\n    loss = - np.sum(y_w) / np.sum(class_arr)\n    return loss","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2568f610eca18be4a298a5093bfe08ebddb844e9"},"cell_type":"code","source":"def lgbm_multi_weighted_logloss(y_true, y_preds):\n    \"\"\"\n    refactor from\n    @author olivier https://www.kaggle.com/ogrellier\n    multi logloss for PLAsTiCC challenge\n    \"\"\"  \n    # Taken from Giba's topic : https://www.kaggle.com/titericz\n    # https://www.kaggle.com/c/PLAsTiCC-2018/discussion/67194\n    # with Kyle Boone's post https://www.kaggle.com/kyleboone\n    classes = [6, 15, 16, 42, 52, 53, 62, 64, 65, 67, 88, 90, 92, 95]\n    class_weights = {6: 1, 15: 2, 16: 1, 42: 1, 52: 1, 53: 1, 62: 1, 64: 2, 65: 1, 67: 1, 88: 1, 90: 1, 92: 1, 95: 1}\n\n    loss = multi_weighted_logloss(y_true, y_preds, classes, class_weights)\n    return 'wloss', loss, False\n\n\ndef xgb_multi_weighted_logloss(y_predicted, y_true, classes, class_weights):\n    loss = multi_weighted_logloss(y_true.get_label(), y_predicted, \n                                  classes, class_weights)\n    return 'wloss', loss","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8ae52e3fbc737ba4f00855a9538f1ebc1eee91e7"},"cell_type":"markdown","source":"## Function to save feature importances (not sure who authored it)"},{"metadata":{"trusted":true,"_uuid":"e75adc9e2e0b62f8d155369fb7b26b1c496ea1af"},"cell_type":"code","source":"\ndef save_importances(importances_):\n    mean_gain = importances_[['gain', 'feature']].groupby('feature').mean()\n    importances_['mean_gain'] = importances_['feature'].map(mean_gain['gain'])\n    return importances_","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ac952e9d027773b3e86cac4e1302769b22d4b4fb"},"cell_type":"markdown","source":"## This method is our biggest contribution to our current score\n- This smote method improved iprapas kernel from 1.135 --> 1.110 and Chia-Ta Tsai's from 1.080 --> 1.052\n- The biggest challeng in integrating it was the data structures (pandas DataFrames vs Numpy arrays, mixed usage of data structures)"},{"metadata":{"trusted":true,"_uuid":"c38d90d50ea3b3a9d5f2b17c8a754670c7b42fe4"},"cell_type":"code","source":"\nfrom imblearn.over_sampling import SMOTE\nfrom sklearn.model_selection import train_test_split\nimport numpy as np # linear algebra\nimport pandas as pd\n\n#modify to work with kfold\n#def smoteAdataset(Xig, yig, test_size=0.2, random_state=0):\ndef smoteAdataset(Xig_train, yig_train, Xig_test, yig_test):\n    \n        \n    sm=SMOTE(random_state=2)\n    Xig_train_res, yig_train_res = sm.fit_sample(Xig_train, yig_train.ravel())\n\n        \n    return Xig_train_res, pd.Series(yig_train_res), Xig_test, pd.Series(yig_test)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b355e13e6709dbd1e9ab34512b6a44f637a1555b"},"cell_type":"markdown","source":"## This is Oliver and Iprapas method but we integrated our Smote method into it"},{"metadata":{"trusted":true,"_uuid":"5694ec57ca9e922509f137b2bd34a790943158c0"},"cell_type":"code","source":"\ndef lgbm_modeling_cross_validation(params,\n                                   full_train, \n                                   y, \n                                   classes, \n                                   class_weights, \n                                   nr_fold=12, \n                                   random_state=1):\n\n    # Compute weights\n    w = y.value_counts()\n    weights = {i : np.sum(w) / w[i] for i in w.index}\n   # print(weights)\n   # weights=class_weights\n    clfs = []\n    importances = pd.DataFrame()\n    folds = StratifiedKFold(n_splits=nr_fold, \n                            shuffle=True, \n                            random_state=random_state)\n    \n    oof_preds = np.zeros((len(full_train), np.unique(y).shape[0]))\n    for fold_, (trn_, val_) in enumerate(folds.split(y, y)):\n        trn_x, trn_y = full_train.iloc[trn_], y.iloc[trn_]\n        val_x, val_y = full_train.iloc[val_], y.iloc[val_]\n        \n                \n        trn_xa, trn_y, val_xa, val_y=smoteAdataset(trn_x.values, trn_y.values, val_x.values, val_y.values)\n        trn_x=pd.DataFrame(data=trn_xa, columns=trn_x.columns)\n    \n        val_x=pd.DataFrame(data=val_xa, columns=val_x.columns)\n        \n        clf = LGBMClassifier(**params)\n        clf.fit(\n            trn_x, trn_y,\n            eval_set=[(trn_x, trn_y), (val_x, val_y)],\n            eval_metric=lgbm_multi_weighted_logloss,\n            verbose=100,\n            early_stopping_rounds=50,\n            sample_weight=trn_y.map(weights)\n        )\n        clfs.append(clf)\n\n        oof_preds[val_, :] = clf.predict_proba(val_x, num_iteration=clf.best_iteration_)\n        print('no {}-fold loss: {}'.format(fold_ + 1, \n              multi_weighted_logloss(val_y, oof_preds[val_, :], \n                                     classes, class_weights)))\n    \n        imp_df = pd.DataFrame({\n                'feature': full_train.columns,\n                'gain': clf.feature_importances_,\n                'fold': [fold_ + 1] * len(full_train.columns),\n                })\n        importances = pd.concat([importances, imp_df], axis=0, sort=False)\n\n    score = multi_weighted_logloss(y_true=y, y_preds=oof_preds, \n                                   classes=classes, class_weights=class_weights)\n    \n    print('MULTI WEIGHTED LOG LOSS: {:.5f}'.format(score))\n    df_importances = save_importances(importances_=importances)\n    df_importances.to_csv('lgbm_importances.csv', index=False)\n    \n    return clfs, score","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8147e830d7ebdf8fa576355db83e9bb527e2a12d"},"cell_type":"markdown","source":"## Surprisingly making changes to these parameters didn't have a big impact on score\n- I thought adding Smote since they optimized would leave room for re-optimization\n- But couldn't get scores to come up "},{"metadata":{"trusted":true,"_uuid":"9d65a7023d08e79e2585e60cb05b7bcaf4ef64e9"},"cell_type":"code","source":"best_params = {\n            'device': 'cpu', \n            'objective': 'multiclass', \n            'num_class': 14, \n            'boosting_type': 'gbdt', \n            'n_jobs': -1, \n            'max_depth': 6, \n            'n_estimators': 1000, \n            'subsample_freq': 2, \n            'subsample_for_bin': 5000, \n            'min_data_per_group': 100, \n            'max_cat_to_onehot': 4, \n            'cat_l2': 1.0, \n            'cat_smooth': 59.5, \n            'max_cat_threshold': 32, \n            'metric_freq': 10, \n            'verbosity': -1, \n            'metric': 'multi_logloss', \n            'xgboost_dart_mode': False, \n            'uniform_drop': False, \n            'colsample_bytree': 0.5, \n            'drop_rate': 0.173, \n            'learning_rate': 0.025, \n            'max_drop': 5, \n            'min_child_samples': 10, \n            'min_child_weight': 200.0, \n            'min_split_gain': 0.01, \n            'num_leaves': 7, \n            'reg_alpha': 0.1, \n            'reg_lambda': 0.00023, \n            'skip_drop': 0.44, \n            'subsample': 0.75}\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1c81d19d8ede3f09f14db1ffac864e88bf78a9f1"},"cell_type":"markdown","source":"## Load and merge the training data\n- trainingDartDf is from Chai-Ta Tsai's kernel\n- trainingJimsDf is from my somethingDifferent kernel\n"},{"metadata":{"trusted":true,"_uuid":"925d35e22e6c762de3e87533c87b9a2e7db90613"},"cell_type":"code","source":"#Here is a change from the script\n#training features\ntrainingDartDf=pd.read_csv('../input/writefeaturetablefromsmotedartset/trainingFeatures1039.csv')\ntrainingJimsDf=pd.read_csv('../input/normalizesomethingdifferentfeatures/traindfNormal.csv')\nif 'Unnamed: 0' in trainingDartDf.columns:\n    trainingDartDf=trainingDartDf.drop('Unnamed: 0', axis=1)\nprint(trainingDartDf.shape)\n#trainingDartDf.head()\ncolumnsToAdd=['outlierScore', 'hipd', 'lipd', 'highEnergy_transitory_1.0_TF',\n          'highEnergy_transitory_1.5_TF', 'lowEnergy_transitory_1.0_TF', \n          'lowEnergy_transitory_1.5_TF']\n\nfor column in columnsToAdd:\n    trainingDartDf.loc[:,column]=trainingJimsDf.loc[:,column]\n\ntraindf=trainingDartDf\n\n#from the 1.052 kernel\ndel traindf['hostgal_specz']\ndel traindf['ra'], traindf['decl'], traindf['gal_l'], traindf['gal_b']\ndel traindf['ddf']\n\n\nprint(traindf.shape)\ntraindf.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a65a5893628573a017ec4517512742dfff1d4166"},"cell_type":"markdown","source":"## Prep training data for Oliver & company's cross validation methods"},{"metadata":{"trusted":true,"_uuid":"727a9cfef5c4f9fe34cb0290a5db3bb4c0ae1933"},"cell_type":"code","source":"full_train=traindf\nif 'target' in full_train:\n    y = full_train['target']\n    del full_train['target']\n\nclasses = sorted(y.unique())    \n# Taken from Giba's topic : https://www.kaggle.com/titericz\n# https://www.kaggle.com/c/PLAsTiCC-2018/discussion/67194\n# with Kyle Boone's post https://www.kaggle.com/kyleboone\nclass_weights = {c: 1 for c in classes}\nclass_weights.update({c:2 for c in [64, 15]})\nprint('Unique classes : {}, {}'.format(len(classes), classes))\nprint(class_weights)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"41d4356c98cb7c2038eb0ef0bc3b9a4427d3256d"},"cell_type":"markdown","source":"## Continue prepping traindf for cross validation, save object_ids"},{"metadata":{"trusted":true,"_uuid":"ed28379fd5769a75cd923f776cfad3b2c25dc519"},"cell_type":"code","source":"\nif 'object_id' in full_train:\n    oof_df = full_train[['object_id']]\n    del full_train['object_id'] \n    #del full_train['distmod'] \n\ntrain_mean = full_train.mean(axis=0)\n#train_mean.to_hdf('train_data.hdf5', 'data')\npd.set_option('display.max_rows', 500)\n#print(full_train.describe().T)\n#import pdb; pdb.set_trace()\nfull_train.fillna(0, inplace=True)\nprint(full_train.shape)\nfull_train.head()\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cceeff1b71b57711518450c09521679fa4bfada9"},"cell_type":"markdown","source":"## The first two lines (or lack thereof) have caused me more headache than I can count\n- it has to do with numpy data types when native data types are expected"},{"metadata":{"trusted":true,"_uuid":"829a1e13cd34498acbacd4101ea7a75c2d2cf9c9"},"cell_type":"code","source":"\ndef getCv(full_train, y, classes, class_weights, maxdepth, maxcat, colby, \n          regalpha, lrate, numleaves, splitgain, minchild, \n          minchildwt, subsample=.95, nfold=5):\n    \n    baseline_params = {\n            'device': 'cpu', \n            'objective': 'multiclass', \n            'num_class': 14, \n            'boosting_type': 'gbdt', \n            'n_jobs': -1, \n            'max_depth': maxdepth, \n            'n_estimators': 1000, #500, 2000\n            'subsample_freq': 2, #3, 5\n            'subsample_for_bin': 5000, \n            'min_data_per_group': 100, #50, #200\n            'max_cat_to_onehot': 4, \n            'cat_l2': 1.0, \n            'cat_smooth': 59.5, \n            'max_cat_threshold': maxcat, #16, 64\n            'metric_freq': 10, #5, 20\n            'verbosity': -1, \n            'metric': 'multi_logloss', \n            'xgboost_dart_mode': False, #true\n            'uniform_drop': False, \n            'colsample_bytree': colby, #.25, 1.0\n            'drop_rate': 0.173, #.0865, .346\n            'learning_rate': lrate, \n            'max_drop': 5, #2, 10\n            'min_child_samples': minchild, #5, 20\n            'min_child_weight': minchildwt, #100, 400\n            'min_split_gain': splitgain, #.1, .001\n            'num_leaves': numleaves, #7, 31\n            'reg_alpha': regalpha, #.01, .5\n            'reg_lambda': .000023,\n            'skip_drop': 0.44,\n            'subsample': subsample} #.5, .95\n\n    \n    for cindex in full_train.columns:\n        full_train.loc[:,cindex]=np.float64(full_train.loc[:,cindex])\n\n    eval_func = partial(lgbm_modeling_cross_validation, \n                            full_train=full_train, \n                            y=y, \n                            classes=classes, \n                            class_weights=class_weights, \n                            nr_fold=nfold, \n                            random_state=1)\n    \n    clfs, score = eval_func(baseline_params)\n    return score\n\n\nmaxdepths=[5]\nmaxcats=[32]\ncolbys=[.25, .35, .45]\nregalphas=[.05]\nlrates=[.02]\nnumleavess=[31]\nsplitgains=[.1]\nminchilds=[35]\nminchildwts=[350, 450]\n\nrdf=pd.DataFrame(columns=['maxdepth','maxcat','colby','regalpha', \n                          'lrate', 'numleaves','splitgain',\n                         'minchild','minchildwt','score'])\n\nfor maxdepth in maxdepths:\n    for maxcat in maxcats:\n        for colby in colbys:\n            for regalpha in regalphas:\n                for lrate in lrates:\n                    for numleaves in numleavess:\n                        for splitgain in splitgains:\n                            for minchild in minchilds:\n                                for minchildwt in minchildwts:\n                                    score=getCv(full_train, y, classes, class_weights, \n                                                maxdepth, maxcat, colby, \n                                                regalpha, lrate, numleaves, \n                                                splitgain, minchild, minchildwt)\n                                    rdf.loc[rdf.shape[0],:]=[maxdepth,maxcat,colby,regalpha, \n                                                             lrate, numleaves,splitgain,\n                                                             minchild,minchildwt,score]\n                                    rdf.to_csv('parameterGrid.csv')\n                                    \nrdf.to_csv('parameterGrid.csv')\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9f72e01648e6c7241484ff11b99bb5aa251d1c26"},"cell_type":"markdown","source":"## Save the results"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}