{"cells":[{"metadata":{"_uuid":"cf675c2dde7950e45a15230a1e7f321f9a3305fe"},"cell_type":"markdown","source":"## Kernels and discussions used in this kernel\n- [Oliver's kernel](https://www.kaggle.com/ogrellier/plasticc-in-a-kernel-meta-and-data)\n- [Alexander Firsov's kernel](https://www.kaggle.com/alexfir/fast-test-set-reading)\n- [Iprapas' kernel](https://www.kaggle.com/iprapas/ideas-from-kernels-and-discussion-lb-1-135)\n- [Chia-Ta Tsai's kernel](https://www.kaggle.com/cttsai/forked-lgbm-w-ideas-from-kernels-and-discuss)\n- [Lving's kernel](https://www.kaggle.com/qianchao/smote-with-imbalance-data)\n- [Scirpus' class 99 method](https://www.kaggle.com/c/PLAsTiCC-2018/discussion/72104)\n- [My something different kernel](https://www.kaggle.com/jimpsull/something-different)\n- [My Smote the training set kernel](https://www.kaggle.com/jimpsull/smote-the-training-sets)"},{"metadata":{"_uuid":"656982859d470a3dd6b4da04916ea1b3ec84fa3d"},"cell_type":"markdown","source":"## The purpose of this kernel is to bring together features\n- the first 69 are from our 1.080 kernel which came via Oliver, Iprapas, and Chia-ta Tsai\n- integrating smote brought that to 1.052\n- adding Scirpus' class 99 method brought it to 1.039\n- adding seven (7) features from my 'something different' kernel brought it to 1.030\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir('../input'))\nprint(os.listdir(\"../input/writefeaturetablefromsmotedartset\"))\nprint(os.listdir('../input/normalizesomethingdifferentfeatures'))\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f9ccb3ee938e6680e6876a80db3ccaa5801ff3e1"},"cell_type":"markdown","source":"## From Chia-Ta Tsai's script"},{"metadata":{"trusted":true,"_uuid":"458bb7b057c7d453c7f5acc33c70ff2646aabba5"},"cell_type":"code","source":"\"\"\"\n\nThis script is forked from chia-ta tsai's kernel of which he said:\n\nThis script is forked from iprapas's notebook \nhttps://www.kaggle.com/iprapas/ideas-from-kernels-and-discussion-lb-1-135\n\n#    https://www.kaggle.com/ogrellier/plasticc-in-a-kernel-meta-and-data\n#    https://www.kaggle.com/c/PLAsTiCC-2018/discussion/70908\n#    https://www.kaggle.com/meaninglesslives/simple-neural-net-for-time-series-classification\n#\n\"\"\"\n\nimport sys, os\nimport argparse\nimport time\nfrom datetime import datetime as dt\nimport gc; gc.enable()\nfrom functools import partial, wraps\n\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport numpy as np # linear algebra\nnp.warnings.filterwarnings('ignore')\n\nfrom sklearn.model_selection import StratifiedKFold\nfrom tsfresh.feature_extraction import extract_features\nfrom xgboost import XGBClassifier\nfrom lightgbm import LGBMClassifier\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"96be97c7c35646a87560d4aec858b03d5089f850"},"cell_type":"markdown","source":"## Oliver's functions"},{"metadata":{"trusted":true,"_uuid":"52bba423f7570305957eb2b7236d20559ae27295"},"cell_type":"code","source":"\ndef multi_weighted_logloss(y_true, y_preds, classes, class_weights):\n    \"\"\"\n    refactor from\n    @author olivier https://www.kaggle.com/ogrellier\n    multi logloss for PLAsTiCC challenge\n    \"\"\"\n    #print('in mwll')\n    #print(classes)\n    #print(class_weights)\n    \n    y_p = y_preds.reshape(y_true.shape[0], len(classes), order='F')\n    # Trasform y_true in dummies\n    y_ohe = pd.get_dummies(y_true%100)\n    # Normalize rows and limit y_preds to 1e-15, 1-1e-15\n    y_p = np.clip(a=y_p, a_min=1e-15, a_max=1 - 1e-15)\n    # Transform to log\n    y_p_log = np.log(y_p)\n    # Get the log for ones, .values is used to drop the index of DataFrames\n    # Exclude class 99 for now, since there is no class99 in the training set\n    # we gave a special process for that class\n    y_log_ones = np.sum(y_ohe.values * y_p_log, axis=0)\n    # Get the number of positives for each class\n    nb_pos = y_ohe.sum(axis=0).values.astype(float)\n    # Weight average and divide by the number of positives\n    class_arr = np.array([class_weights[k] for k in sorted(class_weights.keys())])\n    y_w = y_log_ones * class_arr / nb_pos\n\n    loss = - np.sum(y_w) / np.sum(class_arr)\n    return loss","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2568f610eca18be4a298a5093bfe08ebddb844e9"},"cell_type":"code","source":"def lgbm_multi_weighted_logloss(y_true, y_preds):\n    \"\"\"\n    refactor from\n    @author olivier https://www.kaggle.com/ogrellier\n    multi logloss for PLAsTiCC challenge\n    \"\"\"  \n    y_true=y_true%100\n    y_preds=y_preds%100\n    # Taken from Giba's topic : https://www.kaggle.com/titericz\n    # https://www.kaggle.com/c/PLAsTiCC-2018/discussion/67194\n    # with Kyle Boone's post https://www.kaggle.com/kyleboone\n    classes = [6, 15, 16, 42, 52, 53, 62, 64, 65, 67, 88, 90, 92, 95]\n    #classes=[6, 15, 16, 42, 52, 53, 62, 64, 65, 67, 88, 90, 92, 95, 115, 142, 152, 162, 164, 167, 188, 190, 195]\n    class_weights = {6: 1, 15: 2, 16: 1, 42: 1, 52: 1, 53: 1, 62: 1, 64: 2, 65: 1, 67: 1, 88: 1, 90: 1, 92: 1, 95: 1}\n    #class_weights={6: 1, 15: 2, 16: 1, 42: 1, 52: 1, 53: 1, 62: 1, 64: 2, 65: 1, 67: 1, 88: 1, \n    #               90: 1, 92: 1, 95: 1, 115: 2, 142: 1, 152: 1, 162: 1, 164: 2, 167: 1, 188: 1, 190: 1, 195: 1}\n    loss = multi_weighted_logloss(y_true, y_preds, classes, class_weights)\n    return 'wloss', loss, False\n\n\ndef xgb_multi_weighted_logloss(y_predicted, y_true, classes, class_weights):\n    loss = multi_weighted_logloss(y_true.get_label(), y_predicted, \n                                  classes, class_weights)\n    return 'wloss', loss","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8ae52e3fbc737ba4f00855a9538f1ebc1eee91e7"},"cell_type":"markdown","source":"## Function to save feature importances (not sure who authored it)"},{"metadata":{"trusted":true,"_uuid":"e75adc9e2e0b62f8d155369fb7b26b1c496ea1af"},"cell_type":"code","source":"\ndef save_importances(importances_):\n    mean_gain = importances_[['gain', 'feature']].groupby('feature').mean()\n    importances_['mean_gain'] = importances_['feature'].map(mean_gain['gain'])\n    return importances_","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ac952e9d027773b3e86cac4e1302769b22d4b4fb"},"cell_type":"markdown","source":"## This method is my main contribution\n- This smote method improved iprapas kernel from 1.135 --> 1.110 and Chia-Ta Tsai's from 1.080 --> 1.052\n- The biggest challeng in integrating it was the data structures (pandas DataFrames vs Numpy arrays, mixed usage of data structures)"},{"metadata":{"trusted":true,"_uuid":"c38d90d50ea3b3a9d5f2b17c8a754670c7b42fe4"},"cell_type":"code","source":"\nfrom imblearn.over_sampling import SMOTE\nfrom sklearn.model_selection import train_test_split\nimport numpy as np # linear algebra\nimport pandas as pd\n\n#modify to work with kfold\n#def smoteAdataset(Xig, yig, test_size=0.2, random_state=0):\ndef smoteAdataset(Xig_train, yig_train, Xig_test, yig_test):\n    \n        \n    sm=SMOTE(random_state=2)\n    Xig_train_res, yig_train_res = sm.fit_sample(Xig_train, yig_train.ravel())\n\n        \n    return Xig_train_res, pd.Series(yig_train_res), Xig_test, pd.Series(yig_test)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b355e13e6709dbd1e9ab34512b6a44f637a1555b"},"cell_type":"markdown","source":"## This is Oliver and Iprapas method but I integrated my Smote method into it"},{"metadata":{"trusted":true,"_uuid":"5694ec57ca9e922509f137b2bd34a790943158c0"},"cell_type":"code","source":"\ndef lgbm_modeling_cross_validation(params,\n                                   full_train, \n                                   y, \n                                   classes, \n                                   class_weights, \n                                   nr_fold=12, \n                                   random_state=1):\n    #print(classes)\n    #print(class_weights)\n    # Compute weights\n    yy=y%100\n    \n    w = y.value_counts()\n    weights = {i : np.sum(w) / w[i] for i in w.index}\n   # print(weights)\n   # weights=class_weights\n    clfs = []\n    importances = pd.DataFrame()\n    folds = StratifiedKFold(n_splits=nr_fold, \n                            shuffle=True, \n                            random_state=random_state)\n    \n    oof_preds = np.zeros((len(full_train), np.unique(yy).shape[0]))\n    for fold_, (trn_, val_) in enumerate(folds.split(y, y)):\n        trn_x, trn_y = full_train.iloc[trn_], y.iloc[trn_]\n        val_x, val_y = full_train.iloc[val_], y.iloc[val_]\n        \n                \n        trn_xa, trn_y, val_xa, val_y=smoteAdataset(trn_x.values, trn_y.values, val_x.values, val_y.values)\n        trn_y=trn_y%100\n        val_y=val_y%100\n        trn_x=pd.DataFrame(data=trn_xa, columns=trn_x.columns)\n    \n        val_x=pd.DataFrame(data=val_xa, columns=val_x.columns)\n        \n        clf = LGBMClassifier(**params)\n        clf.fit(\n            trn_x, trn_y,\n            eval_set=[(trn_x, trn_y), (val_x, val_y)],\n            eval_metric=lgbm_multi_weighted_logloss,\n            verbose=100,\n            early_stopping_rounds=100,\n            sample_weight=trn_y.map(weights)\n        )\n        clfs.append(clf)\n\n        oof_preds[val_, :] = clf.predict_proba(val_x, num_iteration=clf.best_iteration_)\n        print('no {}-fold loss: {}'.format(fold_ + 1, \n              multi_weighted_logloss(val_y, oof_preds[val_, :], \n                                     classes, class_weights)))\n    \n        imp_df = pd.DataFrame({\n                'feature': full_train.columns,\n                'gain': clf.feature_importances_,\n                'fold': [fold_ + 1] * len(full_train.columns),\n                })\n        importances = pd.concat([importances, imp_df], axis=0, sort=False)\n\n    score = multi_weighted_logloss(y_true=y, y_preds=oof_preds, \n                                   classes=classes, class_weights=class_weights)\n    print('MULTI WEIGHTED LOG LOSS: {:.5f}'.format(score))\n    df_importances = save_importances(importances_=importances)\n    df_importances.to_csv('lgbm_importances.csv', index=False)\n    \n    return clfs, score","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ae8c1fdc5411c40eab599dc6207774b3ccabd47b"},"cell_type":"markdown","source":"## These methods have several contributors\n- I'm not sure that they're still needed now that I've extracted the features from Chia-Ta Tsai's script\n- But when I tried to run the prediction on test all at once the kernel crashed\n- So I modified to read testdf in chunks and predict bit by bit"},{"metadata":{"trusted":true,"_uuid":"4c857770dd12b0c64c9bf9caab5576480e687df4"},"cell_type":"code","source":"\ndef predict_chunk(df_, clfs_, features, train_mean):\n    # Group by object id    \n    agg_ = df_\n    # Merge with meta data\n    full_test = agg_.reset_index()\n    print(full_test.head())\n\n    full_test = full_test.fillna(0)\n    # Make predictions\n    preds_ = None\n    for clf in clfs_:\n        if preds_ is None:\n            preds_ = clf.predict_proba(full_test[features]) / len(clfs_)\n        else:\n            preds_ += clf.predict_proba(full_test[features]) / len(clfs_)\n            \n    #going to recalc 99 below anyways\n    # Compute preds_99 as the proba of class not being any of the others\n    # preds_99 = 0.1 gives 1.769\n    preds_99 = np.ones(preds_.shape[0])\n    \n    \n    for i in range(preds_.shape[1]):\n        preds_99 *= (1 - preds_[:, i])\n\n    # Create DataFrame from predictions\n    preds_df_ = pd.DataFrame(preds_, columns=['class_' + str(s) for s in clfs_[0].classes_])\n    preds_df_['object_id'] = full_test['object_id']\n    preds_df_['class_99'] = 0.14 * preds_99 / np.mean(preds_99) \n\n    return preds_df_","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"63de6140c0f30202cf6ab1c53cb9c94778d7684c"},"cell_type":"markdown","source":"## Remove the end effect with good chunksize choice\n- testdf.shape[0]%40615=0 so there's no special 'end case'"},{"metadata":{"trusted":true,"_uuid":"0f32f3df59692561373a83fc08ef7768cc5f9ce3"},"cell_type":"code","source":"def process_test(clfs, \n                 testdf,\n                 full_train,\n                 train_mean,\n                 filename='submission.csv',\n                 chunks=40615):\n\n    import time\n\n    start = time.time()\n    chunks = 40615\n\n    testdf.to_csv(filename, index=False)\n    for i_c, df in enumerate(pd.read_csv(filename, chunksize=chunks, iterator=True)):\n\n        print(df.shape)\n        preds_df = predict_chunk(df_=df,\n                                 clfs_=clfs,\n                                 features=full_train.columns,\n                                 train_mean=train_mean)\n\n        if i_c == 0:\n            preds_df.to_csv('predictions.csv', header=True, mode='a', index=False)\n        else:\n            preds_df.to_csv('predictions.csv', header=False, mode='a', index=False)\n\n        del preds_df\n        gc.collect()\n\n        print('%15d done in %5.1f minutes' % (chunks * (i_c + 1), (time.time() - start) / 60), flush=True)\n\n    return","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"081a147eb35f3a4040ab589547790d04da22b1d4"},"cell_type":"markdown","source":"## This method helped change 1.052 lb to 1.039 lb\n- I had the same principle (classes with low max probability might be class 99)\n- but my implementation caused score to go up rather than down\n- I think this is because I thought class 99 would be rare"},{"metadata":{"trusted":true,"_uuid":"70a5ef4ba2d242a4470ebcb759547517632682f8"},"cell_type":"code","source":"#from Scirpus discussion:\n\ndef GenUnknown(data):\n    return ((((((data[\"mymedian\"]) + (((data[\"mymean\"]) / 2.0)))/2.0)) + (((((1.0) - (((data[\"mymax\"]) * (((data[\"mymax\"]) * (data[\"mymax\"]))))))) / 2.0)))/2.0)\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a5255bee1df4703419d3a1a1d00f6b071aa0daf8"},"cell_type":"markdown","source":"## This method is trying to find spots where test and train differ"},{"metadata":{"trusted":true,"_uuid":"af05c727b28c9fa4da5e9b310f3b820ebf1c30db"},"cell_type":"code","source":"import scipy.stats as ss\ndef compareMeans(traindf, testdf, excludeCols=['target', 'object_id'], sigHigh=.2, sigLow=.4, debug=False):\n    feats=[]\n    for col in traindf.columns:\n        if col not in excludeCols:\n            \n            \n            travg=np.average(traindf.loc[:,col])\n            trstd=np.std(traindf.loc[:,col])\n            \n            teavg=np.average(testdf.loc[:,col])\n            testd=np.std(testdf.loc[:,col])\n            \n            if debug:\n                tt=ss.ttest_ind(traindf.loc[:,col], testdf.loc[:,col], equal_var=False)\n                print(col)\n                print(tt)\n                print([travg, trstd, teavg, testd])\n                \n            if (teavg+testd*sigLow)<(travg-trstd*sigLow):\n                print('for ' + str(col) + ' test is smaller than train')\n                tt=ss.ttest_ind(traindf.loc[:,col], testdf.loc[:,col], equal_var=False)\n                print(col)\n                print(tt)\n                print([travg, trstd, teavg, testd])\n                feats.append(col)\n                \n            if (teavg-testd*sigHigh)>(travg+trstd*sigHigh):\n                print('for ' + str(col) + ' test is larger than train')\n                tt=ss.ttest_ind(traindf.loc[:,col], testdf.loc[:,col], equal_var=False)\n                print(col)\n                print(tt)\n                print([travg, trstd, teavg, testd])\n                feats.append(col)\n                \n    return feats\n            \n            ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8147e830d7ebdf8fa576355db83e9bb527e2a12d"},"cell_type":"markdown","source":"## Surprisingly making changes to these parameters didn't have a big impact on score\n- I thought adding Smote since they optimized would leave room for re-optimization\n- But couldn't get scores to come up "},{"metadata":{"trusted":true,"_uuid":"9d65a7023d08e79e2585e60cb05b7bcaf4ef64e9"},"cell_type":"code","source":"best_params = {\n            'device': 'cpu', \n            'objective': 'multiclass', \n            'num_class': 14, \n            'boosting_type': 'gbdt', \n            'n_jobs': -1, \n            'max_depth': 6, \n            'n_estimators': 1000, \n            'subsample_freq': 2, \n            'subsample_for_bin': 5000, \n            'min_data_per_group': 100, \n            'max_cat_to_onehot': 4, \n            'cat_l2': 1.0, \n            'cat_smooth': 59.5, \n            'max_cat_threshold': 32, \n            'metric_freq': 10, \n            'verbosity': -1, \n            'metric': 'multi_logloss', \n            'xgboost_dart_mode': False, \n            'uniform_drop': False, \n            'colsample_bytree': 0.5, \n            'drop_rate': 0.173, \n            'learning_rate': 0.025, \n            'max_drop': 5, \n            'min_child_samples': 10, \n            'min_child_weight': 200.0, \n            'min_split_gain': 0.01, \n            'num_leaves': 15, \n            'reg_alpha': 0.1, \n            'reg_lambda': 0.00023, \n            'skip_drop': 0.44, \n            'subsample': 0.75}\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ba1ac4d4175fc4edb175de331a6a25a8855ef222"},"cell_type":"markdown","source":"## Clean up methods for df"},{"metadata":{"trusted":true,"_uuid":"0be751c24df9edb2c264f0cdbc7ef63e4bd41041"},"cell_type":"code","source":"#prepTrain=rs().fit(full_train)\ndef convertTFToInt(df):\n    \n    for cindex in df.columns:\n        \n        if ('_TF_' in cindex):\n            #print(cindex)\n            df[cindex]=df[cindex].astype(int)\n        if ('_TF' in cindex):\n            df[cindex]=df[cindex].astype(int)\n    return df\n\ndef removeExtremeValues(df, maxSig=5.88):\n    \n    for cindex in df.columns:\n        if cindex not in ['object_id', 'target']:\n            med=np.median(df[cindex])\n            sig=np.std(df[cindex])\n            minVal=med-maxSig*sig\n            maxVal=med+maxSig*sig\n            highFilter=(df.loc[:,cindex]>maxVal) | (df.loc[:,cindex] == np.inf)\n            lowFilter=(df.loc[:,cindex]<minVal) | (df.loc[:,cindex] == -np.inf)\n            df.loc[lowFilter,cindex]=minVal\n            df.loc[highFilter,cindex]=maxVal\n    \n    return df\n\n\n#not sure why but na.fillna wasn't hacking it\ndef ensureNoNanOrInf(df):\n\n    df=df.round(5)\n    df.replace(np.inf, 9999, inplace=True)\n    for cindex in df.columns:\n        dropit=df[cindex].isna().sum()\n        #print(dropit)\n        finite=np.isfinite(df[cindex]).sum()\n        if finite != df.shape[0]:\n            print(finite)\n        if dropit>0:\n            #full_train=full_train.drop(cindex,axis=1)\n            print(cindex + ' has ' + str(dropit) + ' nans')\n            df[cindex].fillna(0,inplace=True)\n    print(df.shape)\n    return df\n\ndef cleanupDf(df):\n    \n    df=ensureNoNanOrInf(df)\n    df=convertTFToInt(df)\n    df=removeExtremeValues(df, maxSig=5.88)\n    return df\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1c81d19d8ede3f09f14db1ffac864e88bf78a9f1"},"cell_type":"markdown","source":"## Load and merge the training data\n- trainingDartDf is from Chai-Ta Tsai's kernel\n- trainingJimsDf is from my somethingDifferent kernel\n"},{"metadata":{"trusted":true,"_uuid":"925d35e22e6c762de3e87533c87b9a2e7db90613"},"cell_type":"code","source":"#Here is a change from the script\n#training features\ntrainingDartDf=pd.read_csv('../input/writefeaturetablefromsmotedartset/trainingFeatures1039.csv')\ntrainingJimsDf=pd.read_csv('../input/normalizesomethingdifferentfeatures/traindfNormal.csv')\nif 'Unnamed: 0' in trainingDartDf.columns:\n    trainingDartDf=trainingDartDf.drop('Unnamed: 0', axis=1)\nprint(trainingDartDf.shape)\n#trainingDartDf.head()\ncolumnsToAdd=['outlierScore', 'hipd', 'lipd', 'highEnergy_transitory_1.0_TF',\n          'highEnergy_transitory_1.5_TF', 'lowEnergy_transitory_1.0_TF', \n          'lowEnergy_transitory_1.5_TF']\n\nfor column in columnsToAdd:\n    trainingDartDf.loc[:,column]=trainingJimsDf.loc[:,column]\n\ntraindf=trainingDartDf\n\n#from the 1.052 kernel\ndel traindf['hostgal_specz']\ndel traindf['ra'], traindf['decl'], traindf['gal_l'], traindf['gal_b']\ndel traindf['ddf']\n\n\nprint(traindf.shape)\ntraindf.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1091baf1ea609c4f05ce7a587fc5117fefe55f78"},"cell_type":"markdown","source":"## Load the test data\n- be careful with memory"},{"metadata":{"trusted":true,"_uuid":"aa605f62c429d9f85b1b1a60da8bda7b3baf2ac4"},"cell_type":"code","source":"    #test features\n    testDartDf=pd.read_csv('../input/writefeaturetablefromsmotedartset/feat_0.648970_2018-11-23-09-00.csv')\n    testJimsDf=pd.read_csv('../input/normalizesomethingdifferentfeatures/testdfNormal.csv')\n\n    if 'Unnamed: 0' in testDartDf.columns:\n        testDartDf=testDartDf.drop('Unnamed: 0', axis=1)\n    print(testDartDf.shape)\n    testDartDf.head()\n\n    for column in columnsToAdd:\n        testDartDf.loc[:,column]=testJimsDf.loc[:,column]\n\n    testdf=testDartDf\n\n    #from the 1.052 kernel\n    del testdf['hostgal_specz']\n    del testdf['ra'], testdf['decl'], testdf['gal_l'], testdf['gal_b']\n    del testdf['ddf']\n\n    testdf.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"632ab3b8bd17bddd28c4ed6ea13acee8f2346538"},"cell_type":"code","source":"traindf=cleanupDf(traindf)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5fb5ebb3bba869ca3db433c6972be725bf1f1c97"},"cell_type":"code","source":"testdf=cleanupDf(testdf)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e2ba2e62ee57d62dc1e90a6d77de175405f25d85"},"cell_type":"code","source":"feats=compareMeans(traindf, testdf, debug=False, sigHigh=.15, sigLow=.3)\n#traindf.columns\nprint(feats)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7c86907d60dfe37c936f36d6433e2d50120f6cdc"},"cell_type":"code","source":"def medianMatcher(traindf, testdf, theClass, feat, debug=False):\n    \n    #purpose is to try to make train look more like test\n    #c = class, w = whole, m = median, tr=train, te=test\n    #assumption is that mcte = mctr * mwte / mwtr\n    \n    mwtr = np.median(traindf.loc[:,feat])\n    mwte = np.median(testdf.loc[:,feat])\n    \n    #not sure if the mod 100 is needed but it won't hurt\n    #I may be using a convention of psuedo-classes having multiples of 100 added to the true class\n    classFilter = (traindf.loc[:,'target']%100)==theClass\n    \n    mctr = np.median(traindf.loc[classFilter,feat])\n    mcte = mctr * mwte / mwtr\n    \n    aboveFilter = (traindf.loc[:,feat]>mcte) & classFilter\n    belowFilter = (traindf.loc[:,feat]<mcte) & classFilter\n    \n    if aboveFilter.sum() > belowFilter.sum():\n        traindf.loc[belowFilter,'target'] += 100\n    else:\n        traindf.loc[aboveFilter,'target'] += 100\n        \n    if debug:\n        print('whole train: ' + str(mwtr))\n        print('whole test: ' + str(mwte))\n        print('class train: ' + str(mctr))\n        print('class test projected: ' + str(mcte))\n        print('above test class median ' + str(aboveFilter.sum()))\n        print('below test class median ' + str(belowFilter.sum()))\n        print(traindf.loc[:,'target'].unique())\n        \n    #now we have a target median.  We want the median for the training set for this class and this feature\n    #to reach the value mcte\n    \n    return traindf\n    \n    \n\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0a2ff9d3e7d24bb4fb3f2aaedfc461c495d51543"},"cell_type":"code","source":"traindf.loc[(traindf.loc[:,'distmod']==0),'target'].unique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8cae8cda225b3b8e328404512b77cb6a426e2130"},"cell_type":"code","source":"def newTestizeTrain(traindf, testdf, feats, classes=[15, 42, 52, 62, 64, 67, 88, 90, 95]):\n    \n    for theClass in classes:\n        for theFeat in feats:\n            traindf=medianMatcher(traindf, testdf, theClass, theFeat, debug=False)\n            #print(len(traindf.loc[:,'target'].unique()))\n    \n    return traindf\n\ntraindf=newTestizeTrain(traindf, testdf, feats)\nprint(traindf.shape)\nprint(traindf.loc[:,'target'].unique())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"658b5e824d13a501db8a0799a4eb298f811d725c"},"cell_type":"code","source":"def consolidateClasses(traindf, minSize=7):\n    \n    for i in traindf.loc[:,'target'].unique():\n        theCount=(traindf.loc[:,'target']==i).sum()\n        if theCount < minSize:\n            traindf.loc[(traindf['target']==i),'target']=i%100\n            print('class ' + str(i) + ' consolidated due to low (' + str(theCount) + ') size')\n            \n    return traindf\nprint(len(traindf.loc[:,'target'].unique()))\ntraindf=consolidateClasses(traindf)\nprint(len(traindf.loc[:,'target'].unique()))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c413f55f63312afb6ef921f603f306b5f58ecb0b"},"cell_type":"code","source":"#traindf=testizeTrain(traindf)\n#print(traindf.shape)\n\n#traindf=testizeTrainSmoteNoRep(traindf,'distmod')\n#traindf=testizeTrainSmoteNoRep(traindf, 'flux__length', small=True)\n#print(traindf.shape)\n#traindf.loc[:,'target'].unique()\n\n#12 folds using testizeTrain multi weighted log loss .3245\n#5 folds using testizeTrain multi weighted log loss ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a65a5893628573a017ec4517512742dfff1d4166"},"cell_type":"markdown","source":"## Prep training data for Oliver & company's cross validation methods"},{"metadata":{"trusted":true,"_uuid":"727a9cfef5c4f9fe34cb0290a5db3bb4c0ae1933"},"cell_type":"code","source":"full_train=traindf\nif 'target' in full_train:\n    y = full_train['target']\n    del full_train['target']\nyy=y%100\nclasses = sorted(yy.unique())    \n# Taken from Giba's topic : https://www.kaggle.com/titericz\n# https://www.kaggle.com/c/PLAsTiCC-2018/discussion/67194\n# with Kyle Boone's post https://www.kaggle.com/kyleboone\nclass_weights = {c: 1 for c in classes}\nclass_weights.update({c:2 for c in [64, 15]})\nprint('Unique classes : {}, {}'.format(len(classes), classes))\nprint(class_weights)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"41d4356c98cb7c2038eb0ef0bc3b9a4427d3256d"},"cell_type":"markdown","source":"## Continue prepping traindf for cross validation, save object_ids"},{"metadata":{"trusted":true,"_uuid":"ed28379fd5769a75cd923f776cfad3b2c25dc519"},"cell_type":"code","source":"\nif 'object_id' in full_train:\n    oof_df = full_train[['object_id']]\n    del full_train['object_id'] \n    #del full_train['distmod'] \n\ntrain_mean = full_train.mean(axis=0)\n#train_mean.to_hdf('train_data.hdf5', 'data')\npd.set_option('display.max_rows', 500)\n#print(full_train.describe().T)\n#import pdb; pdb.set_trace()\nfull_train.fillna(0, inplace=True)\nprint(full_train.shape)\nfull_train.head()\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cceeff1b71b57711518450c09521679fa4bfada9"},"cell_type":"markdown","source":"## The first two lines (or lack thereof) have caused me more headache than I can count\n- it has to do with numpy data types when native data types are expected"},{"metadata":{"trusted":true,"_uuid":"829a1e13cd34498acbacd4101ea7a75c2d2cf9c9"},"cell_type":"code","source":"for cindex in full_train.columns:\n    full_train.loc[:,cindex]=np.float64(full_train.loc[:,cindex])\n\neval_func = partial(lgbm_modeling_cross_validation, \n                        full_train=full_train, \n                        y=y, \n                        classes=classes, \n                        class_weights=class_weights, \n                        nr_fold=12, \n                        random_state=1)\n\nbest_params.update({'n_estimators': 2000})\n    \n    # modeling from CV\nclfs, score = eval_func(best_params)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9f72e01648e6c7241484ff11b99bb5aa251d1c26"},"cell_type":"markdown","source":"## Chai-Ta Tsai's naming convention\n- stores the CV score and the timestamp in the filename"},{"metadata":{"trusted":true,"_uuid":"e10efd2330796c0bf1dad9f61ab61beeadd87f6f"},"cell_type":"code","source":"\nfilename = 'subm_{:.6f}_{}.csv'.format(score, \n                 dt.now().strftime('%Y-%m-%d-%H-%M'))\nprint('save to {}'.format(filename))\n# TEST\n\n\nprocess_test(clfs, \n             testdf,\n             full_train,\n             train_mean=train_mean, \n             filename=filename,\n             chunks=40615)\n\n\npdf = pd.read_csv('predictions.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"97ef3aee088521d55381e90d201ddfb452c3ad47"},"cell_type":"markdown","source":"## Reorder the columns and apply Scirpus' class 99 method\n- I later discovered the order of columns in the submission doesn't matter"},{"metadata":{"trusted":true,"_uuid":"a52038566d64b21ba8ec5fd954b95b5643cc3731"},"cell_type":"code","source":"# get a list of columns\ncols = list(pdf)\n# move the column to head of list using index, pop and insert\ncols.insert(0, cols.pop(cols.index('object_id')))\npdf = pdf.loc[:, cols]\n\n\n\nfeats = ['class_6', 'class_15', 'class_16', 'class_42', 'class_52', 'class_53',\n         'class_62', 'class_64', 'class_65', 'class_67', 'class_88', 'class_90',\n         'class_92', 'class_95']\n\ny = pd.DataFrame()\ny['mymean'] = pdf[feats].mean(axis=1)\ny['mymedian'] = pdf[feats].median(axis=1)\ny['mymax'] = pdf[feats].max(axis=1)\n\npdf['class_99'] = GenUnknown(y)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"683f8501c88cf41ecf1d685ebb680e5df02040ba"},"cell_type":"code","source":"import copy\ndef modUnknown(opdf, meta, ddfMult=1.0, mwMult=0.1, preserveMed=False):\n    pdf=copy.deepcopy(opdf)\n    mdf=pdf.merge(meta,on='object_id')\n    ddfilter=mdf.loc[:,'ddf']==1\n    mwfilter=mdf.loc[:,'hostgal_photoz']==0\n    print(ddfilter.sum())\n    print(mwfilter.sum())\n    \n    mdf.loc[mwfilter,'class_99']=mwMult*mdf.loc[mwfilter,'class_99']\n    mdf.loc[ddfilter,'class_99']=ddfMult*mdf.loc[ddfilter,'class_99']\n    pdf.loc[:,'class_99']=mdf.loc[:,'class_99']\n    \n    return pdf\n\n#npdf=modUnknown(pdf, meta)\n#npdf.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1b399ecb762eea98913b5654284ff52415f612bc"},"cell_type":"code","source":"\npdf.to_csv(filename, index=False)\n\n","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}