{"cells":[{"metadata":{"_uuid":"cf675c2dde7950e45a15230a1e7f321f9a3305fe"},"cell_type":"markdown","source":"## Kernels and discussions used in this kernel\n- [Oliver's kernel](https://www.kaggle.com/ogrellier/plasticc-in-a-kernel-meta-and-data)\n- [Alexander Firsov's kernel](https://www.kaggle.com/alexfir/fast-test-set-reading)\n- [Iprapas' kernel](https://www.kaggle.com/iprapas/ideas-from-kernels-and-discussion-lb-1-135)\n- [Chia-Ta Tsai's kernel](https://www.kaggle.com/cttsai/forked-lgbm-w-ideas-from-kernels-and-discuss)\n- [Lving's kernel](https://www.kaggle.com/qianchao/smote-with-imbalance-data)\n- [Scirpus' class 99 method](https://www.kaggle.com/c/PLAsTiCC-2018/discussion/72104)\n- [My something different kernel](https://www.kaggle.com/jimpsull/something-different)\n- [My Smote the training set kernel](https://www.kaggle.com/jimpsull/smote-the-training-sets)"},{"metadata":{"_uuid":"656982859d470a3dd6b4da04916ea1b3ec84fa3d"},"cell_type":"markdown","source":"## The purpose of this kernel is to bring together features\n- the first 69 are from our 1.080 kernel which came via Oliver, Iprapas, and Chia-ta Tsai\n- integrating smote brought that to 1.052\n- adding Scirpus' class 99 method brought it to 1.039\n- adding seven (7) features from my 'something different' kernel brought it to 1.030\n- increasing k from 5 --> 12 brought that to 1.029\n- An additional class 99 method brought it to 1.028\n- ongoing efforts and tried and failed lists at the end of the kernel\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir('../input'))\nprint(os.listdir(\"../input/writefeaturetablefromsmotedartset\"))\nprint(os.listdir('../input/normalizesomethingdifferentfeatures'))\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f9ccb3ee938e6680e6876a80db3ccaa5801ff3e1"},"cell_type":"markdown","source":"## From Chia-Ta Tsai's script"},{"metadata":{"trusted":true,"_uuid":"458bb7b057c7d453c7f5acc33c70ff2646aabba5"},"cell_type":"code","source":"\"\"\"\n\nThis script is forked from chia-ta tsai's kernel of which he said:\n\nThis script is forked from iprapas's notebook \nhttps://www.kaggle.com/iprapas/ideas-from-kernels-and-discussion-lb-1-135\n\n#    https://www.kaggle.com/ogrellier/plasticc-in-a-kernel-meta-and-data\n#    https://www.kaggle.com/c/PLAsTiCC-2018/discussion/70908\n#    https://www.kaggle.com/meaninglesslives/simple-neural-net-for-time-series-classification\n#\n\"\"\"\n\nimport sys, os\nimport argparse\nimport time\nfrom datetime import datetime as dt\nimport gc; gc.enable()\nfrom functools import partial, wraps\n\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport numpy as np # linear algebra\nnp.warnings.filterwarnings('ignore')\n\nfrom sklearn.model_selection import StratifiedKFold\nfrom tsfresh.feature_extraction import extract_features\nfrom xgboost import XGBClassifier\nfrom lightgbm import LGBMClassifier","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"96be97c7c35646a87560d4aec858b03d5089f850"},"cell_type":"markdown","source":"## Oliver's functions"},{"metadata":{"trusted":true,"_uuid":"52bba423f7570305957eb2b7236d20559ae27295"},"cell_type":"code","source":"\ndef multi_weighted_logloss(y_true, y_preds, classes, class_weights):\n    \"\"\"\n    refactor from\n    @author olivier https://www.kaggle.com/ogrellier\n    multi logloss for PLAsTiCC challenge\n    \"\"\"\n    y_p = y_preds.reshape(y_true.shape[0], len(classes), order='F')\n    # Trasform y_true in dummies\n    y_ohe = pd.get_dummies(y_true)\n    # Normalize rows and limit y_preds to 1e-15, 1-1e-15\n    y_p = np.clip(a=y_p, a_min=1e-15, a_max=1 - 1e-15)\n    # Transform to log\n    y_p_log = np.log(y_p)\n    # Get the log for ones, .values is used to drop the index of DataFrames\n    # Exclude class 99 for now, since there is no class99 in the training set\n    # we gave a special process for that class\n    y_log_ones = np.sum(y_ohe.values * y_p_log, axis=0)\n    # Get the number of positives for each class\n    nb_pos = y_ohe.sum(axis=0).values.astype(float)\n    # Weight average and divide by the number of positives\n    class_arr = np.array([class_weights[k] for k in sorted(class_weights.keys())])\n    y_w = y_log_ones * class_arr / nb_pos\n\n    loss = - np.sum(y_w) / np.sum(class_arr)\n    return loss","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2568f610eca18be4a298a5093bfe08ebddb844e9"},"cell_type":"code","source":"def lgbm_multi_weighted_logloss(y_true, y_preds):\n    \"\"\"\n    refactor from\n    @author olivier https://www.kaggle.com/ogrellier\n    multi logloss for PLAsTiCC challenge\n    \"\"\"  \n    # Taken from Giba's topic : https://www.kaggle.com/titericz\n    # https://www.kaggle.com/c/PLAsTiCC-2018/discussion/67194\n    # with Kyle Boone's post https://www.kaggle.com/kyleboone\n    classes = [6, 15, 16, 42, 52, 53, 62, 64, 65, 67, 88, 90, 92, 95]\n    class_weights = {6: 1, 15: 2, 16: 1, 42: 1, 52: 1, 53: 1, 62: 1, 64: 2, 65: 1, 67: 1, 88: 1, 90: 1, 92: 1, 95: 1}\n\n    loss = multi_weighted_logloss(y_true, y_preds, classes, class_weights)\n    return 'wloss', loss, False\n\n\ndef xgb_multi_weighted_logloss(y_predicted, y_true, classes, class_weights):\n    loss = multi_weighted_logloss(y_true.get_label(), y_predicted, \n                                  classes, class_weights)\n    return 'wloss', loss","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8ae52e3fbc737ba4f00855a9538f1ebc1eee91e7"},"cell_type":"markdown","source":"## Function to save feature importances (not sure who authored it)"},{"metadata":{"trusted":true,"_uuid":"e75adc9e2e0b62f8d155369fb7b26b1c496ea1af"},"cell_type":"code","source":"\ndef save_importances(importances_):\n    mean_gain = importances_[['gain', 'feature']].groupby('feature').mean()\n    importances_['mean_gain'] = importances_['feature'].map(mean_gain['gain'])\n    return importances_","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ac952e9d027773b3e86cac4e1302769b22d4b4fb"},"cell_type":"markdown","source":"## This method is our biggest contribution to our current score\n- This smote method improved iprapas kernel from 1.135 --> 1.110 and Chia-Ta Tsai's from 1.080 --> 1.052\n- The biggest challeng in integrating it was the data structures (pandas DataFrames vs Numpy arrays, mixed usage of data structures)"},{"metadata":{"trusted":true,"_uuid":"c38d90d50ea3b3a9d5f2b17c8a754670c7b42fe4"},"cell_type":"code","source":"\nfrom imblearn.over_sampling import SMOTE\nfrom sklearn.model_selection import train_test_split\nimport numpy as np # linear algebra\nimport pandas as pd\n\n#modify to work with kfold\n#def smoteAdataset(Xig, yig, test_size=0.2, random_state=0):\ndef smoteAdataset(Xig_train, yig_train, Xig_test, yig_test):\n    \n        \n    sm=SMOTE(random_state=2)\n    Xig_train_res, yig_train_res = sm.fit_sample(Xig_train, yig_train.ravel())\n\n        \n    return Xig_train_res, pd.Series(yig_train_res), Xig_test, pd.Series(yig_test)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b355e13e6709dbd1e9ab34512b6a44f637a1555b"},"cell_type":"markdown","source":"## This is Oliver and Iprapas method but we integrated our Smote method into it"},{"metadata":{"trusted":true,"_uuid":"5694ec57ca9e922509f137b2bd34a790943158c0"},"cell_type":"code","source":"\ndef lgbm_modeling_cross_validation(params,\n                                   full_train, \n                                   y, \n                                   classes, \n                                   class_weights, \n                                   nr_fold=12, \n                                   random_state=1):\n\n    # Compute weights\n    w = y.value_counts()\n    weights = {i : np.sum(w) / w[i] for i in w.index}\n   # print(weights)\n   # weights=class_weights\n    clfs = []\n    importances = pd.DataFrame()\n    folds = StratifiedKFold(n_splits=nr_fold, \n                            shuffle=True, \n                            random_state=random_state)\n    \n    oof_preds = np.zeros((len(full_train), np.unique(y).shape[0]))\n    for fold_, (trn_, val_) in enumerate(folds.split(y, y)):\n        trn_x, trn_y = full_train.iloc[trn_], y.iloc[trn_]\n        val_x, val_y = full_train.iloc[val_], y.iloc[val_]\n        \n                \n        trn_xa, trn_y, val_xa, val_y=smoteAdataset(trn_x.values, trn_y.values, val_x.values, val_y.values)\n        trn_x=pd.DataFrame(data=trn_xa, columns=trn_x.columns)\n    \n        val_x=pd.DataFrame(data=val_xa, columns=val_x.columns)\n        \n        clf = LGBMClassifier(**params)\n        clf.fit(\n            trn_x, trn_y,\n            eval_set=[(trn_x, trn_y), (val_x, val_y)],\n            eval_metric=lgbm_multi_weighted_logloss,\n            verbose=100,\n            early_stopping_rounds=50,\n            sample_weight=trn_y.map(weights)\n        )\n        clfs.append(clf)\n\n        oof_preds[val_, :] = clf.predict_proba(val_x, num_iteration=clf.best_iteration_)\n        print('no {}-fold loss: {}'.format(fold_ + 1, \n              multi_weighted_logloss(val_y, oof_preds[val_, :], \n                                     classes, class_weights)))\n    \n        imp_df = pd.DataFrame({\n                'feature': full_train.columns,\n                'gain': clf.feature_importances_,\n                'fold': [fold_ + 1] * len(full_train.columns),\n                })\n        importances = pd.concat([importances, imp_df], axis=0, sort=False)\n\n    score = multi_weighted_logloss(y_true=y, y_preds=oof_preds, \n                                   classes=classes, class_weights=class_weights)\n    print('MULTI WEIGHTED LOG LOSS: {:.5f}'.format(score))\n    df_importances = save_importances(importances_=importances)\n    df_importances.to_csv('lgbm_importances.csv', index=False)\n    \n    return clfs, score","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ae8c1fdc5411c40eab599dc6207774b3ccabd47b"},"cell_type":"markdown","source":"## These methods have several contributors\n- I'm not sure that they're still needed now that I've extracted the features from Chia-Ta Tsai's script\n- But when I tried to run the prediction on test all at once the kernel crashed\n- So I modified to read testdf in chunks and predict bit by bit"},{"metadata":{"trusted":true,"_uuid":"4c857770dd12b0c64c9bf9caab5576480e687df4"},"cell_type":"code","source":"\ndef predict_chunk(df_, clfs_, features, train_mean):\n    # Group by object id    \n    agg_ = df_\n    # Merge with meta data\n    full_test = agg_.reset_index()\n    #print(full_test.head())\n\n    full_test = full_test.fillna(0)\n    # Make predictions\n    preds_ = None\n    for clf in clfs_:\n        if preds_ is None:\n            preds_ = clf.predict_proba(full_test[features]) / len(clfs_)\n        else:\n            preds_ += clf.predict_proba(full_test[features]) / len(clfs_)\n            \n    #going to recalc 99 below anyways\n    # Compute preds_99 as the proba of class not being any of the others\n    # preds_99 = 0.1 gives 1.769\n    preds_99 = np.ones(preds_.shape[0])\n    \n    \n    for i in range(preds_.shape[1]):\n        preds_99 *= (1 - preds_[:, i])\n\n    # Create DataFrame from predictions\n    preds_df_ = pd.DataFrame(preds_, columns=['class_' + str(s) for s in clfs_[0].classes_])\n    preds_df_['object_id'] = full_test['object_id']\n    preds_df_['class_99'] = 0.14 * preds_99 / np.mean(preds_99) \n\n    return preds_df_","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"63de6140c0f30202cf6ab1c53cb9c94778d7684c"},"cell_type":"markdown","source":"## Remove the end effect with good chunksize choice\n- testdf.shape[0]%40615=0 so there's no special 'end case'"},{"metadata":{"trusted":true,"_uuid":"0f32f3df59692561373a83fc08ef7768cc5f9ce3"},"cell_type":"code","source":"def process_test(clfs, \n                 testdf,\n                 full_train,\n                 train_mean,\n                 filename='submission.csv',\n                 chunks=40615):\n\n    import time\n\n    start = time.time()\n    chunks = 40615\n    testdf=testdf.round(5)\n    \n    testdf.to_csv(filename, index=False)\n    for i_c, df in enumerate(pd.read_csv(filename, chunksize=chunks, iterator=True)):\n\n        print(df.shape)\n        preds_df = predict_chunk(df_=df,\n                                 clfs_=clfs,\n                                 features=full_train.columns,\n                                 train_mean=train_mean)\n\n        if i_c == 0:\n            preds_df.to_csv('predictions.csv', header=True, mode='a', index=False)\n        else:\n            preds_df.to_csv('predictions.csv', header=False, mode='a', index=False)\n\n        del preds_df\n        gc.collect()\n\n        print('%15d done in %5.1f minutes' % (chunks * (i_c + 1), (time.time() - start) / 60), flush=True)\n\n    return","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"081a147eb35f3a4040ab589547790d04da22b1d4"},"cell_type":"markdown","source":"## This method helped change 1.052 lb to 1.039 lb\n- We had the same principle (classes with low max probability might be class 99)\n- but our implementation caused score to go up rather than down\n- I think this is because I thought class 99 would be rare"},{"metadata":{"trusted":true,"_uuid":"70a5ef4ba2d242a4470ebcb759547517632682f8"},"cell_type":"code","source":"#from Scirpus discussion:\n\ndef GenUnknown(data):\n    return ((((((data[\"mymedian\"]) + (((data[\"mymean\"]) / 2.0)))/2.0)) + (((((1.0) - (((data[\"mymax\"]) * (((data[\"mymax\"]) * (data[\"mymax\"]))))))) / 2.0)))/2.0)\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8147e830d7ebdf8fa576355db83e9bb527e2a12d"},"cell_type":"markdown","source":"## Surprisingly making changes to these parameters didn't have a big impact on score\n- I thought adding Smote since they optimized would leave room for re-optimization\n- But couldn't get scores to come up "},{"metadata":{"trusted":true,"_uuid":"9d65a7023d08e79e2585e60cb05b7bcaf4ef64e9"},"cell_type":"code","source":"best_params = {\n            'device': 'cpu', \n            'objective': 'multiclass', \n            'num_class': 14, \n            'boosting_type': 'gbdt', \n            'n_jobs': -1, \n            'max_depth': 5, \n            'n_estimators': 1000, \n            'subsample_freq': 2, \n            'subsample_for_bin': 5000, \n            'min_data_per_group': 100, \n            'max_cat_to_onehot': 4, \n            'cat_l2': 1.0, \n            'cat_smooth': 59.5, \n            'max_cat_threshold': 32, \n            'metric_freq': 10, \n            'verbosity': -1, \n            'metric': 'multi_logloss', \n            'xgboost_dart_mode': False, \n            'uniform_drop': False, \n            'colsample_bytree': 0.25, \n            'drop_rate': 0.173, \n            'learning_rate': 0.015, \n            'max_drop': 5, \n            'min_child_samples': 35, \n            'min_child_weight': 350.0, \n            'min_split_gain': 0.01, \n            'num_leaves': 31, \n            'reg_alpha': 0.01, \n            'reg_lambda': 0.000023, \n            'skip_drop': 0.44, \n            'subsample': 0.95}\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1c81d19d8ede3f09f14db1ffac864e88bf78a9f1"},"cell_type":"markdown","source":"## Load and merge the training data\n- trainingDartDf is from Chai-Ta Tsai's kernel\n- trainingJimsDf is from my somethingDifferent kernel\n"},{"metadata":{"trusted":true,"_uuid":"925d35e22e6c762de3e87533c87b9a2e7db90613"},"cell_type":"code","source":"#Here is a change from the script\n#training features\ntrainingDartDf=pd.read_csv('../input/writefeaturetablefromsmotedartset/trainingFeatures1039.csv')\ntrainingJimsDf=pd.read_csv('../input/normalizesomethingdifferentfeatures/traindfNormal.csv')\nif 'Unnamed: 0' in trainingDartDf.columns:\n    trainingDartDf=trainingDartDf.drop('Unnamed: 0', axis=1)\nprint(trainingDartDf.shape)\n#trainingDartDf.head()\ncolumnsToAdd=['outlierScore', 'hipd', 'lipd', 'highEnergy_transitory_1.0_TF',\n          'highEnergy_transitory_1.5_TF', 'lowEnergy_transitory_1.0_TF', \n          'lowEnergy_transitory_1.5_TF']\n\nfor column in columnsToAdd:\n    trainingDartDf.loc[:,column]=trainingJimsDf.loc[:,column]\n\ntraindf=trainingDartDf\n\n#from the 1.052 kernel\ndel traindf['hostgal_specz']\ndel traindf['ra'], traindf['decl'], traindf['gal_l'], traindf['gal_b']\ndel traindf['ddf']\n\n\nprint(traindf.shape)\ntraindf.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1091baf1ea609c4f05ce7a587fc5117fefe55f78"},"cell_type":"markdown","source":"## Load the test data\n- be careful with memory\n- if you bring together features from multiple sources it is best to delete the dataframes once you have the features you need"},{"metadata":{"trusted":true,"_uuid":"aa605f62c429d9f85b1b1a60da8bda7b3baf2ac4"},"cell_type":"code","source":"    #test features\n    testDartDf=pd.read_csv('../input/writefeaturetablefromsmotedartset/feat_0.648970_2018-11-23-09-00.csv')\n    testJimsDf=pd.read_csv('../input/normalizesomethingdifferentfeatures/testdfNormal.csv')\n\n    if 'Unnamed: 0' in testDartDf.columns:\n        testDartDf=testDartDf.drop('Unnamed: 0', axis=1)\n    print(testDartDf.shape)\n    testDartDf.head()\n\n    for column in columnsToAdd:\n        testDartDf.loc[:,column]=testJimsDf.loc[:,column]\n\n    testdf=testDartDf\n\n    #from the 1.052 kernel\n    del testdf['hostgal_specz']\n    del testdf['ra'], testdf['decl'], testdf['gal_l'], testdf['gal_b']\n    del testdf['ddf']\n\n    testdf.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a65a5893628573a017ec4517512742dfff1d4166"},"cell_type":"markdown","source":"## Prep training data for Oliver & company's cross validation methods"},{"metadata":{"trusted":true,"_uuid":"727a9cfef5c4f9fe34cb0290a5db3bb4c0ae1933"},"cell_type":"code","source":"full_train=traindf\nif 'target' in full_train:\n    y = full_train['target']\n    del full_train['target']\n\nclasses = sorted(y.unique())    \n# Taken from Giba's topic : https://www.kaggle.com/titericz\n# https://www.kaggle.com/c/PLAsTiCC-2018/discussion/67194\n# with Kyle Boone's post https://www.kaggle.com/kyleboone\nclass_weights = {c: 1 for c in classes}\nclass_weights.update({c:2 for c in [64, 15]})\nprint('Unique classes : {}, {}'.format(len(classes), classes))\nprint(class_weights)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"41d4356c98cb7c2038eb0ef0bc3b9a4427d3256d"},"cell_type":"markdown","source":"## Continue prepping traindf for cross validation, save object_ids"},{"metadata":{"trusted":true,"_uuid":"ed28379fd5769a75cd923f776cfad3b2c25dc519"},"cell_type":"code","source":"\nif 'object_id' in full_train:\n    oof_df = full_train[['object_id']]\n    del full_train['object_id'] \n    #del full_train['distmod'] \n\ntrain_mean = full_train.mean(axis=0)\n#train_mean.to_hdf('train_data.hdf5', 'data')\npd.set_option('display.max_rows', 500)\n#print(full_train.describe().T)\n#import pdb; pdb.set_trace()\nfull_train.fillna(0, inplace=True)\nprint(full_train.shape)\nfull_train.head()\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cceeff1b71b57711518450c09521679fa4bfada9"},"cell_type":"markdown","source":"## The first two lines (or lack thereof) have caused me more headache than I can count\n- it has to do with numpy data types when native data types are expected"},{"metadata":{"trusted":true,"_uuid":"829a1e13cd34498acbacd4101ea7a75c2d2cf9c9"},"cell_type":"code","source":"for cindex in full_train.columns:\n    full_train.loc[:,cindex]=np.float64(full_train.loc[:,cindex])\n\neval_func = partial(lgbm_modeling_cross_validation, \n                        full_train=full_train, \n                        y=y, \n                        classes=classes, \n                        class_weights=class_weights, \n                        nr_fold=12, \n                        random_state=1)\n\nbest_params.update({'n_estimators': 1000})\n    \n    # modeling from CV\nclfs, score = eval_func(best_params)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9f72e01648e6c7241484ff11b99bb5aa251d1c26"},"cell_type":"markdown","source":"## Chai-Ta Tsai's naming convention\n- stores the CV score and the timestamp in the filename"},{"metadata":{"trusted":true,"_uuid":"e10efd2330796c0bf1dad9f61ab61beeadd87f6f"},"cell_type":"code","source":"\nfilename = 'subm_{:.6f}_{}.csv'.format(score, \n                 dt.now().strftime('%Y-%m-%d-%H-%M'))\nprint('save to {}'.format(filename))\n# TEST\n\n\nprocess_test(clfs, \n             testdf,\n             full_train,\n             train_mean=train_mean, \n             filename=filename,\n             chunks=40615)\n\n\npdf = pd.read_csv('predictions.csv')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"97ef3aee088521d55381e90d201ddfb452c3ad47"},"cell_type":"markdown","source":"## Reorder the columns and apply Scirpus' class 99 method\n- I later discovered the order of columns in the submission doesn't matter"},{"metadata":{"trusted":true,"_uuid":"a52038566d64b21ba8ec5fd954b95b5643cc3731"},"cell_type":"code","source":"\n\n\n# get a list of columns\ncols = list(pdf)\n# move the column to head of list using index, pop and insert\ncols.insert(0, cols.pop(cols.index('object_id')))\npdf = pdf.loc[:, cols]\n\n\n\nfeats = ['class_6', 'class_15', 'class_16', 'class_42', 'class_52', 'class_53',\n         'class_62', 'class_64', 'class_65', 'class_67', 'class_88', 'class_90',\n         'class_92', 'class_95']\n\ny = pd.DataFrame()\ny['mymean'] = pdf[feats].mean(axis=1)\ny['mymedian'] = pdf[feats].median(axis=1)\ny['mymax'] = pdf[feats].max(axis=1)\n\npdf['class_99'] = GenUnknown(y)\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6129a55cab9c409735803ebb795363e3d225b811"},"cell_type":"markdown","source":"## This last method boosted us from 1.029 to 1.028\n- not much but we've hit a wall"},{"metadata":{"trusted":true,"_uuid":"0f53e3cca6660b9e11b39ccff44ca83b16ad66f1"},"cell_type":"code","source":"meta=pd.read_csv('../input/PLAsTiCC-2018/test_set_metadata.csv')\nmeta.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"08b1e5d74672e24b917b272e83056eb039a3353c"},"cell_type":"code","source":"#1, 1 --> 1.029\n#0.5, 0.5 --> 1.028\n#1.0, 0.1 --> 1.030\n\nimport copy\ndef modUnknown(opdf, meta, ddfMult=0.5, mwMult=0.5, preserveMed=False):\n    pdf=copy.deepcopy(opdf)\n    mdf=pdf.merge(meta,on='object_id')\n    ddfilter=mdf.loc[:,'ddf']==1\n    mwfilter=mdf.loc[:,'hostgal_photoz']==0\n    print(ddfilter.sum())\n    print(mwfilter.sum())\n    \n    mdf.loc[mwfilter,'class_99']=mwMult*mdf.loc[mwfilter,'class_99']\n    mdf.loc[ddfilter,'class_99']=ddfMult*mdf.loc[ddfilter,'class_99']\n    pdf.loc[:,'class_99']=mdf.loc[:,'class_99']\n    \n    return pdf\n\nnpdf=modUnknown(pdf, meta)\nnpdf.head()\n    ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"26e394d8986946757fd69ea61811b7e5e26e148e"},"cell_type":"code","source":"#pdf.to_csv('mergedTest.csv', index=False)\nnpdf.to_csv(filename, index=False)\n#if you get an out of diskspace error try uncommenting and overwriting mergedTest\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"67ed4846899fb2fc07d80d89809d936ed4aa1ecc"},"cell_type":"markdown","source":"## In work with hope\n- Based on the idea that test was full of more distant and faint objects, we tried to emphasize things in train that were like test\n- Because the CV score was ridiculously good via the first testizeTrain method the second method was written\n- That method raised CV a bit but hopefully closes the gap between CV and LB"},{"metadata":{"trusted":true,"_uuid":"8bded577c4fca0f4842b4ee06d7f955b219b3552"},"cell_type":"code","source":"def testizeTrain(traindf, dupMult=5):\n    \n    for classval in traindf.loc[:,'target'].unique():\n        #print(classval)\n        classfilter=traindf.loc[:,'target']==classval\n        count=classfilter.sum()\n        #print(count)\n        if np.max(traindf.loc[classfilter,'distmod'])>0:\n            todup=np.ceil(count/4).astype(int)\n            #print(todup)\n            dups=traindf.loc[classfilter,:].nlargest(todup,'distmod')\n            #print(dups.head())\n            for i in range(dupMult):\n                traindf=traindf.append(dups,ignore_index=True)\n                #print(traindf.shape)\n            \n    return traindf\n\nimport copy\n\n###################################################\n\n\ndef testizeTrainSmoteNoRep(traindf):\n    \n    for classval in traindf.loc[:,'target'].unique():\n        #print(classval)\n        classfilter=traindf.loc[:,'target']==classval\n        count=classfilter.sum()\n        #print(count)\n        if np.max(traindf.loc[classfilter,'distmod'])>0:\n            todup=np.ceil(count/4).astype(int)\n            #print(todup)\n            dups=traindf.loc[classfilter,:].nlargest(todup,'distmod')\n            mindm=np.min(dups.loc[:,'distmod'])\n            flagfilter=(traindf.loc[:,'distmod']>=mindm) & classfilter\n            traindf.loc[flagfilter,'target']+=100\n            \n    return traindf","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b368fbd4b658c0d09663f78fd72a53003a657217"},"cell_type":"markdown","source":"## Tried and failed\n- building single class SVM classifiers and using them as probabilities on their own (1.9 lb)\n- using SVM classifiers as features in the LGBM model\n- using 3 class LGBM models (troubleA, troubleB, other) to compute troubleClassDeltas and using that in the main model\n- Again with 3 class LGBM but using features that were left out of the main model\n- trying to use PB0&3, PB1&4, various aggregations on PBs including an attempt at UV / IR ratios\n- Adding some of Mithrillion's Cesium features"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}