{"cells":[{"metadata":{"_uuid":"cf675c2dde7950e45a15230a1e7f321f9a3305fe"},"cell_type":"markdown","source":"## This kernel is now just bringing together our features and 'log normalizing' them\n- If there is a 'real' meaning of 'log normalization', this isn't it\n- This is my practical method for **compressing features with wide spreads across both negative and positive values**\n- I have no theoretical justification however empirically it has improved the performance of my models\n"},{"metadata":{"_uuid":"656982859d470a3dd6b4da04916ea1b3ec84fa3d"},"cell_type":"markdown","source":"## standardScaler and .fillna weren't working reliably for me\n- My cleanDf method has five stages\n- ensureNoNanOrInf\n- convertTFToInt (this method should be using the dataType of the input rather than feature naming conventions)\n- convertAllToLogBase\n- removeExtremeValues\n- featureScaleAllExcept\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir('../input'))\nprint(os.listdir(\"../input/writefeaturetablefromsmotedartset\"))\nprint(os.listdir('../input/normalizesomethingdifferentfeatures'))\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1c81d19d8ede3f09f14db1ffac864e88bf78a9f1"},"cell_type":"markdown","source":"## Load and merge the training data\n- trainingDartDf is from Chai-Ta Tsai's kernel\n- trainingJimsDf is from my somethingDifferent kernel\n"},{"metadata":{"trusted":true,"_uuid":"925d35e22e6c762de3e87533c87b9a2e7db90613"},"cell_type":"code","source":"#Here is a change from the script\n#training features\ntrainingDartDf=pd.read_csv('../input/writefeaturetablefromsmotedartset/trainingFeatures1039.csv')\ntrainingJimsDf=pd.read_csv('../input/normalizesomethingdifferentfeatures/traindfNormal.csv')\nif 'Unnamed: 0' in trainingDartDf.columns:\n    trainingDartDf=trainingDartDf.drop('Unnamed: 0', axis=1)\nprint(trainingDartDf.shape)\n#trainingDartDf.head()\ncolumnsToAdd=['outlierScore', 'hipd', 'lipd', 'highEnergy_transitory_1.0_TF',\n          'highEnergy_transitory_1.5_TF', 'lowEnergy_transitory_1.0_TF', \n          'lowEnergy_transitory_1.5_TF']\n\nfor column in columnsToAdd:\n    trainingDartDf.loc[:,column]=trainingJimsDf.loc[:,column]\n\ntraindf=trainingDartDf\n\n#from the 1.052 kernel\ndel traindf['hostgal_specz']\ndel traindf['ra'], traindf['decl'], traindf['gal_l'], traindf['gal_b']\ndel traindf['ddf']\n\n\nprint(traindf.shape)\ntraindf.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"eac74a53b9870bd08ef786526b5a19a4812474ec"},"cell_type":"markdown","source":"## My neural net model did better when I treated the intragalactic and extragalactic objects differently"},{"metadata":{"trusted":true,"_uuid":"8b0f405d8a3212cef38d4197d43bb671aa1d6293"},"cell_type":"code","source":"import copy\n\nigTrain=copy.deepcopy(traindf)[traindf['distmod'].isna()]\n\negTrain=copy.deepcopy(traindf)[~traindf['distmod'].isna()]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c10d1b8dabd16cdd65ad87920ff0c5c2fd8f76c0"},"cell_type":"code","source":"print(igTrain.shape)\nigTrain.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"67d73fbdbfae1785d595ccad34867c03b61a437b"},"cell_type":"code","source":"print(egTrain.shape)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"76ccdae4825b3f38c7a169538b304fe6f2f06ac8"},"cell_type":"markdown","source":"## I saw an improvement using the cleanupDf(traindf, testdf) method at the bottom here\n- Because train and test are quite different for this project it was important that they be processed the same way\n- Imagine if I trained on log(feat) but cleanUp(df) didn't transform test[feat].\n- This normalization scheme means that the test data will not necessarily be between 0 and 1"},{"metadata":{"trusted":true,"_uuid":"780f1afc3c26439f43903f735196627f7abf264d"},"cell_type":"code","source":"#you would really think this would be unnecessary.\n#however I kept getting nan errors even after df=df.fillna(0)\n\ndef ensureNoNanOrInf(df):\n\n    df=df.round(5)\n    df.replace(np.inf, 9999, inplace=True)\n    for cindex in df.columns:\n        dropit=df[cindex].isna().sum()\n        #print(dropit)\n        finite=np.isfinite(df[cindex]).sum()\n        if finite != df.shape[0]:\n            print(finite)\n        if dropit>0:\n            #full_train=full_train.drop(cindex,axis=1)\n            print(cindex + ' has ' + str(dropit) + ' nans')\n            df[cindex].fillna(0,inplace=True)\n    print(df.shape)\n    return df\n\ndef convertTFToInt(df):\n    \n    for cindex in df.columns:\n        \n        if '_TF' in cindex:\n            #print(cindex)\n            df[cindex]=df[cindex].astype(int)\n            \n        if '_TF_' in cindex:\n            #print(cindex)\n            df[cindex]=df[cindex].astype(int)\n            \n    return df\n\n#I used this because most of the data had a very wide spread\n#I have another method that compares kurtosis and skew of feat vs ln(feat) to decide whether to transform it\n#Values between -1 and 1 become 0 using this method\n#maxFrac is a threshold where if too much of the data is between -1 and 1 then it won't be transformed\n\ndef convertAllToLogBase(df, testdf, maxFrac=0.05, excludeLogCols=['ra', 'decl', 'gal_l', 'gal_b', 'hostgal_photoz', 'target',\n                                            'hostgal_photoz_err', 'distmod', 'outlierScore', 'object_id']):\n    \n    #the intent here is to find the magnitude and the directionality of the flux\n    #because of weird behavior between -1 and 1 we'll treat that as 0\n    #then the sign is separated from the magnitude using the filters below\n    #later we'll feature scale\n    dfSize=df.shape[0]\n    \n        \n    for cindex in df.columns:\n        \n        if (cindex not in excludeLogCols) & (len(df.loc[:,cindex].unique()) > 2):\n            \n            #zeroFilter=df.loc[:,cindex]==0 (stays zero)\n            zeroFilter=((df.loc[:,cindex]>-1) & (df.loc[:,cindex]<1))\n            zeroFilterTest=((testdf.loc[:,cindex]>-1) & (testdf.loc[:,cindex]<1))\n            if ((zeroFilter.sum() / dfSize) <= maxFrac):\n\n                negFilter=df.loc[:,cindex]<=-1\n                negFilterTest=testdf.loc[:,cindex]<=-1\n                \n                posFilter=df.loc[:,cindex]>=1\n                posFilterTest=testdf.loc[:,cindex]>=1\n                \n                df.loc[zeroFilter,cindex]=0\n                df.loc[negFilter,cindex]=-1.0*np.log(-1.0*df.loc[negFilter,cindex])\n                df.loc[posFilter,cindex]=np.log(df.loc[posFilter,cindex])\n                \n                testdf.loc[zeroFilterTest,cindex]=0\n                testdf.loc[negFilterTest,cindex]=-1.0*np.log(-1.0*testdf.loc[negFilterTest,cindex])\n                testdf.loc[posFilterTest,cindex]=np.log(testdf.loc[posFilterTest,cindex])\n                \n            else:\n                print('feature ' + str(cindex) + ' not log normalized because it has too many values near zero')\n                        \n            \n    return df, testdf\n\n#5.88 sigma was chosen based on 500,000,000 rows of data in the PLAsTiCC test set and the z-statistic\ndef removeExtremeValues(df, testdf, maxSig=5.88):\n    \n    for cindex in df.columns:\n        if cindex not in ['object_id', 'target']:\n            med=np.median(df[cindex])\n            sig=np.std(df[cindex])\n            minVal=med-maxSig*sig\n            maxVal=med+maxSig*sig\n            highFilter=(df.loc[:,cindex]>maxVal) | (df.loc[:,cindex] == np.inf)\n            highFilterTest=(testdf.loc[:,cindex]>maxVal) | (testdf.loc[:,cindex] == np.inf)\n            \n            lowFilter=(df.loc[:,cindex]<minVal) | (df.loc[:,cindex] == -np.inf)\n            lowFilterTest=(testdf.loc[:,cindex]<minVal) | (testdf.loc[:,cindex] == -np.inf)\n            \n            df.loc[lowFilter,cindex]=minVal\n            testdf.loc[lowFilterTest,cindex]=minVal\n            \n            df.loc[highFilter,cindex]=maxVal\n            testdf.loc[highFilterTest,cindex]=maxVal\n    \n    return df, testdf\n\ndef featureScaleAllExcept(df, testdf, targCol='target'):\n    \n    for cindex in df.columns:\n        if (cindex != targCol) & (cindex !='object_id'):\n            minval=np.min(df.loc[:,cindex])\n            maxval=np.max(df.loc[:,cindex])\n            theRange=(maxval-minval)\n            if theRange==0:\n                #this feature contains no information\n                df=df.drop(cindex,axis=1)\n                testdf=testdf.drop(cindex, axis=1)\n                print('dropped ' + str(cindex) + ' from dataFrame because training set has no useful information about this feature')\n                \n            elif type(df[cindex])==bool:\n                print(str(cindex) + ' is a boolean, doing nothing')\n                \n            elif theRange==1:\n                print('the range for ' + str(cindex) + ' is already 1, doing nothing')\n                \n            else:\n                #feature scale\n                df.loc[:,cindex]=(df.loc[:,cindex]-minval)/theRange\n                testdf.loc[:,cindex]=(testdf.loc[:,cindex]-minval)/theRange\n                \n    return df, testdf\n\ndef cleanupDf(traindf, testdf):\n    \n    traindf=ensureNoNanOrInf(traindf)\n    testdf=ensureNoNanOrInf(testdf)\n    \n    traindf=convertTFToInt(traindf)\n    testdf=convertTFToInt(testdf)\n    \n    traindf, testdf=convertAllToLogBase(traindf, testdf)\n    \n    traindf, testdf=removeExtremeValues(traindf, testdf, maxSig=5.88)\n    traindf, testdf=featureScaleAllExcept(traindf, testdf)\n    \n    traindf=traindf.round(5)\n    testdf=testdf.round(5)\n    \n    return traindf, testdf\n\n\n\n\n\n#traindf, forceCols=cleanupDf(traindf)\n#traindf.describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1091baf1ea609c4f05ce7a587fc5117fefe55f78"},"cell_type":"markdown","source":"## Load the test data\n- The purpose of forceCols is to ensure that the test set receives the same treatment as the training set\n- Imagine you set maxFrac at 0.05 and the training set had 4% near zero but the test set had 6% near zero.  The two sets would get a different treatment"},{"metadata":{"trusted":true,"_uuid":"aa605f62c429d9f85b1b1a60da8bda7b3baf2ac4"},"cell_type":"code","source":"#test features\ntestDartDf=pd.read_csv('../input/writefeaturetablefromsmotedartset/feat_0.648970_2018-11-23-09-00.csv')\ntestJimsDf=pd.read_csv('../input/normalizesomethingdifferentfeatures/testdfNormal.csv')\n\nif 'Unnamed: 0' in testDartDf.columns:\n    testDartDf=testDartDf.drop('Unnamed: 0', axis=1)\nprint(testDartDf.shape)\ntestDartDf.head()\n\nfor column in columnsToAdd:\n    testDartDf.loc[:,column]=testJimsDf.loc[:,column]\n\ntestdf=testDartDf\n\n#from the 1.052 kernel\ndel testdf['hostgal_specz']\ndel testdf['ra'], testdf['decl'], testdf['gal_l'], testdf['gal_b']\ndel testdf['ddf']\n\ntestdf.shape\nigTest=copy.deepcopy(testdf)[testdf['distmod'].isna() | testdf['distmod']==0]\nprint(igTest.shape)\negTest=copy.deepcopy(testdf)[~testdf['distmod'].isna() & testdf['distmod']!=0]\nprint(egTest.shape)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c8a8e82c6f2892ec83ee35a16588b50f813c0165"},"cell_type":"code","source":"igTrain, igTest = cleanupDf(igTrain, igTest)\negTrain, egTest = cleanupDf(egTrain, egTest)\n\n#testdf, forceCols=cleanupDf(testdf, forceCols)\n#print(testdf.shape)\n#testdf.describe()\n#testdf.to_csv('logNormalTest.csv', index=False)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"266760a18f13b6d052ae63b8e3a6d622e6be893a"},"cell_type":"code","source":"igTrain.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b4c589cb88742bbf3bd881dbf48315912890672c"},"cell_type":"code","source":"egTrain.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d5c385716a2dbc131b880512c669e7080401a271"},"cell_type":"code","source":"igTest.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1fa50c29c11c57a37cf7b803b7b24400912e9be4"},"cell_type":"code","source":"egTest.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0421353d8dd689d544df2fcab312202d69de6545"},"cell_type":"code","source":"igTrain.to_csv('igTrain.csv', index=False)\nigTest.to_csv('igTest.csv', index=False)\negTrain.to_csv('egTrain.csv', index=False)\negTest.to_csv('egTest.csv', index=False)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"51b0891cff1c2a9334e59a70bf9f13a54a205eae"},"cell_type":"markdown","source":"## My LGBM model didn't do any better with split data and it was easier to use unsplit data\n- So this kernel saves both split and unsplit data"},{"metadata":{"trusted":true,"_uuid":"fba636faa040ef418234d3486e3082cec4751cc9"},"cell_type":"code","source":"for cindex in egTest.columns:\n    if cindex not in igTest.columns:\n        igTest[cindex]=0\n        \nfor cindex in egTrain.columns:\n    if cindex not in igTrain.columns:\n        igTrain[cindex]=0\n        \ntestdf=pd.concat([igTest, egTest], sort=True)\n\ntraindf=pd.concat([igTrain, egTrain], sort=True)\n\nprint(testdf.shape)\nprint(traindf.shape)\n\ntraindf.describe()\ntraindf.to_csv('fullTrain.csv', index=False)\ntestdf.to_csv('fullTest.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}