{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Overview\nThis compared to the [99 accurate model](https://www.kaggle.com/nigelhenry/simple-99-accurate-floor-model) is a more of a brute force approach,added on with a bit of error analysis & post processing\n\nSo i studied Kouki's [LSTM](https://www.kaggle.com/kokitanisaka/lstm-by-keras-with-unified-wi-fi-feats) that utilizes [the unified Wi-Fi dataset](https://www.kaggle.com/kokitanisaka/indoorunifiedwifids).<br>\nand i found it rather intersting that it could score so well on the xy, but the floor prediction was never improving as it was pretty stable after a few epochs.\nHow could it be soo good for the xy and not for the floor?\n\nDidn't seem right so i set to work on this model:\n\n\nI know there is already a great floor predicting model out there that got already nearly 99% but seeing how competition is heating up, every decimal counts.\n\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport scipy.stats as stats\nfrom pathlib import Path\nimport glob\nimport pickle\nimport random\nimport os\n\nfrom sklearn.model_selection import StratifiedKFold\nfrom sklearn.preprocessing import StandardScaler, LabelEncoder\nfrom sklearn.metrics import accuracy_score\nfrom sklearn.model_selection import train_test_split\nimport tensorflow as tf\nimport tensorflow.keras.layers as L\nimport tensorflow.keras.models as M\nimport tensorflow.keras.backend as K\nimport tensorflow_addons as tfa\nfrom tensorflow_addons.layers import WeightNormalization\nfrom tensorflow.keras.callbacks import ReduceLROnPlateau, ModelCheckpoint, EarlyStopping","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Kouki's awsome code for preprocessing, is hidden below;"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# options\n\nN_SPLITS = 5\n\nSEED = 2021\n\nNUM_FEATS = 20 # number of features that we use. there are 100 feats but we don't need to use all of them\n\nbase_path = '/kaggle'\n\ndef set_seed(seed=42):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    tf.random.set_seed(seed)\n    session_conf = tf.compat.v1.ConfigProto(\n        intra_op_parallelism_threads=1,\n        inter_op_parallelism_threads=1\n    )\n    sess = tf.compat.v1.Session(graph=tf.compat.v1.get_default_graph(), config=session_conf)\n    tf.compat.v1.keras.backend.set_session(sess)\n    \ndef comp_metric(xhat, yhat, fhat, x, y, f):\n    intermediate = np.sqrt(np.power(xhat-x, 2) + np.power(yhat-y, 2)) + 15 * np.abs(fhat-f)\n    return intermediate.sum()/xhat.shape[0]\n\nfeature_dir = f\"{base_path}/input/indoorunifiedwifids\"\ntrain_files = sorted(glob.glob(os.path.join(feature_dir, '*_train.csv')))\ntest_files = sorted(glob.glob(os.path.join(feature_dir, '*_test.csv')))\nsubm = pd.read_csv(f'{base_path}/input/indoor-location-navigation/sample_submission.csv', index_col=0)\n\nwith open(f'{feature_dir}/train_all.pkl', 'rb') as f:\n  data = pickle.load( f)\n\nwith open(f'{feature_dir}/test_all.pkl', 'rb') as f:\n  test_data = pickle.load(f)\n\n\n# training target features\n\nBSSID_FEATS = [f'bssid_{i}' for i in range(NUM_FEATS)]\nRSSI_FEATS  = [f'rssi_{i}' for i in range(NUM_FEATS)]\n\n\n# get numbers of bssids to embed them in a layer\n\nwifi_bssids = []\nfor i in range(100):\n    wifi_bssids.extend(data.iloc[:,i].values.tolist())\nwifi_bssids = list(set(wifi_bssids))\n\nwifi_bssids_size = len(wifi_bssids)\nprint(f'BSSID TYPES: {wifi_bssids_size}')\n\nwifi_bssids_test = []\nfor i in range(100):\n    wifi_bssids_test.extend(test_data.iloc[:,i].values.tolist())\nwifi_bssids_test = list(set(wifi_bssids_test))\n\nwifi_bssids_size = len(wifi_bssids_test)\nprint(f'BSSID TYPES: {wifi_bssids_size}')\n\nwifi_bssids.extend(wifi_bssids_test)\nwifi_bssids_size = len(wifi_bssids)\n\n# preprocess\n\nle = LabelEncoder()\nle.fit(wifi_bssids)\nle_site = LabelEncoder()\nle_site.fit(data['site_id'])\n\nss = StandardScaler()\nss.fit(data.loc[:,RSSI_FEATS])\n\n\ndata.loc[:,RSSI_FEATS] = ss.transform(data.loc[:,RSSI_FEATS])\nfor i in BSSID_FEATS:\n    data.loc[:,i] = le.transform(data.loc[:,i])\n    data.loc[:,i] = data.loc[:,i] + 1\n    \ndata.loc[:, 'site_id'] = le_site.transform(data.loc[:, 'site_id'])\n\ndata.loc[:,RSSI_FEATS] = ss.transform(data.loc[:,RSSI_FEATS])\n\ntest_data.loc[:,RSSI_FEATS] = ss.transform(test_data.loc[:,RSSI_FEATS])\nfor i in BSSID_FEATS:\n    test_data.loc[:,i] = le.transform(test_data.loc[:,i])\n    test_data.loc[:,i] = test_data.loc[:,i] + 1\n    \ntest_data.loc[:, 'site_id'] = le_site.transform(test_data.loc[:, 'site_id'])\n\ntest_data.loc[:,RSSI_FEATS] = ss.transform(test_data.loc[:,RSSI_FEATS])\n\n\nsite_count = len(data['site_id'].unique())\ndata.reset_index(drop=True, inplace=True)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## The Reajusted model\nThe floor predictions wee being made by a softmax layer with just one dense unit, a pretty easy error to make and a difficult one to spot since there are two exits, and therefor the unit was prediciting between floors 1-0 (almost always one). So i set out to, test knowledge and fix it for the floor predictions and make a modol for them. \n\nI used one hot encoding( there are a total of 11 cats)with categorical loss and Sigmoid activation unit for the last layer &  added a bit of bilateral firing power for an over kill and came out with the accurate following result:\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"#FLOOR\ndef create_fmodel(input_data):\n\n    # bssid feats\n    input_dim = input_data[0].shape[1]\n\n    input_embd_layer = L.Input(shape=(input_dim,))\n    x1 = L.Embedding(wifi_bssids_size, 64)(input_embd_layer)\n    x1 = L.Flatten()(x1)\n\n    # rssi feats\n    input_dim = input_data[1].shape[1]\n\n    input_layer = L.Input(input_dim, )\n    x2 = L.BatchNormalization()(input_layer)\n    x2 = L.Dense(NUM_FEATS * 64, activation='relu')(x2)\n\n    # site\n    input_site_layer = L.Input(shape=(1,))\n    x3 = L.Embedding(site_count, 2)(input_site_layer)\n    x3 = L.Flatten()(x3)\n\n\n    # main stream\n    x = L.Concatenate(axis=1)([x1, x3, x2])\n\n\n    x = L.Reshape((1, -1))(x)\n    x = L.BatchNormalization()(x)\n    mod1=L.LSTM(256, dropout=0.4, recurrent_dropout=0.3, return_sequences=True, activation='tanh')\n    x = L.Bidirectional(mod1)(x)\n    x = L.Bidirectional(L.LSTM(32, dropout=0.4, return_sequences=False, activation='relu'))(x)\n    x = L.BatchNormalization()(x)\n    x = L.Dense(16, activation='tanh')(x) \n    \n    output_layer_1 = L.Dense(11, activation='softmax', name='floor')(x) \n\n    model = M.Model([input_embd_layer, input_layer, input_site_layer], \n                    [output_layer_1])\n\n    model.compile(optimizer=tf.optimizers.Adam(lr=0.001),\n                  loss=tf.keras.losses.CategoricalCrossentropy(), metrics=['mse','accuracy'])\n\n    return model\n\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.index=data['path']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#OneHot The floor\none_hot=pd.get_dummies(data['floor'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#500 Random, totally unseen paths\nval_p_ind=pd.DataFrame(data.path.unique()).sample(n=500,random_state=1).values.reshape((-1)) #100%/500samples so accuracy of preicision estimate should be around 0.2 % so +/- 0,1 % \nt_idx = data.path.unique().tolist() \nt_idx=[ a for a in t_idx if a not in val_p_ind.tolist()]\n\ntrain_data=data.loc[t_idx]\nX_ass_val= data.loc[val_p_ind]\nlen(t_idx),len(val_p_ind)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It is important that the paths are unseen for reasons shown in the error analysis\notherwise in the post processing i would have data leakage."},{"metadata":{"trusted":true},"cell_type":"code","source":"#check there is no cross contamination of the validation data\ntrain_data[train_data['path']==val_p_ind[5]]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"y_trainf = one_hot.loc[t_idx, :]\ny_validf = one_hot.loc[val_p_ind, :]\nX_train = train_data.loc[:, BSSID_FEATS + RSSI_FEATS + ['site_id']]\nX_valid = X_ass_val.loc[:, BSSID_FEATS + RSSI_FEATS + ['site_id']]\nfmodel = create_fmodel([X_train.loc[:,BSSID_FEATS], X_train.loc[:,RSSI_FEATS], X_train.loc[:,'site_id']])\n#     model = multi_gpu_model(model, 1)\nfmodel.fit([X_train.loc[:,BSSID_FEATS], X_train.loc[:,RSSI_FEATS], X_train.loc[:,'site_id']], y_trainf, \n            validation_data=([X_valid.loc[:,BSSID_FEATS], X_valid.loc[:,RSSI_FEATS], X_valid.loc[:,'site_id']], y_validf), \n            batch_size=128, epochs=100\n             ,shuffle=True\n            ,callbacks=[\n            ReduceLROnPlateau(monitor='val_loss', factor=0.1, patience=3, verbose=1, min_delta=1e-4, mode='min')\n            , ModelCheckpoint(f'{base_path}/RNN_{SEED}_.hdf5', monitor='val_loss', verbose=0, save_best_only=True, save_weights_only=True, mode='min')\n            , EarlyStopping(monitor='val_loss', min_delta=1e-4, patience=5, mode='min', baseline=None, restore_best_weights=True)\n        ]\n   )\n\nfmodel.load_weights(f'{base_path}/RNN_{SEED}_.hdf5')\nfvalid = fmodel.predict([X_ass_val.loc[:,BSSID_FEATS], X_ass_val.loc[:,RSSI_FEATS], X_ass_val.loc[:,'site_id']])#minus two is make the interval [-2:8] again\nfvalid = np.argmax(fvalid, axis=1)-2\n# ass_val_arr[:, fold] = fvalid\n\npred = fmodel.predict([test_data.loc[:,BSSID_FEATS], test_data.loc[:,RSSI_FEATS], test_data.loc[:,'site_id']]) # test_data.iloc[:, :-1])\npred =np.argmax(pred, axis=1)-2#minus two is make the interval [-2:8] again\n# preds_f_arr[:, fold] = pred\n\nass_val_floors=fvalid\nfloors=pred\n                                                                \naccuracy_score(X_ass_val['floor'], ass_val_floors)#second validation, checks the argmax and shifting","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Error Analysis "},{"metadata":{"trusted":true},"cell_type":"code","source":"#Error Analysis - how many paths i got wrong and how many times \nX_ass_val['wrong']=(X_ass_val['floor']- ass_val_floors)!=0\nwrongs= X_ass_val[X_ass_val['wrong']==True]\nrights= X_ass_val[X_ass_val['wrong']==False]\nwrongs.shape, wrongs['path'].unique().shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Only one floor per path right** ? \nwell i think this is a given as the original data is presented as in paths within the floors file;\nI double checked this assumptionto be true."},{"metadata":{},"cell_type":"markdown","source":"So i check if the ones i got wrong i ever got right..\nI check out the number of times i got that path right"},{"metadata":{"trusted":true,"_kg_hide-input":false,"_kg_hide-output":true},"cell_type":"code","source":"#create tuple\n#(Number of times predicted correctly left  vs numebr of times corrected incorecctly right)\n[(rights[rights['path']==p].shape[0],wrongs[wrongs['path']==p].shape[0]) for p in  wrongs['path'].unique() if p in rights['path'].unique()]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As you can see if you unhide the above result, in taking the most frequent column i would avoid many erorrs"},{"metadata":{"trusted":true},"cell_type":"code","source":"#re-elaboration taking the most frequent\nX_ass_val['p_floor']=ass_val_floors\nX_ass_val=X_ass_val.reset_index(drop=True)\nX_ass_val\n\ndef mode(a):\n    '''returns the mode of the group'''\n    return( a['p_floor'].value_counts().head(1).reset_index()['index'].values[0])\n\ndf = pd.DataFrame()    \n# df['path']=X_ass_val.groupby('path').apply(modee1)\ndf['blended_floor_pred']=X_ass_val.groupby('path').apply(mode)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Checking the post processing Bump"},{"metadata":{"trusted":true},"cell_type":"code","source":"X_ass_val=X_ass_val.merge(df, how='left', on='path')\naccuracy_score(X_ass_val['floor'], X_ass_val['blended_floor_pred'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I can round it up (defectivly) to 99,80%. This Rounding is because of the test size i have a scale unit of 0,2%. "},{"metadata":{},"cell_type":"markdown","source":"Satisfactory, so do it on the test data too and submit."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"test_data['path']=test_data['site_path_timestamp'].str.split(pat='_', n=- 1, expand=True)[1]\n(test_data['site_path_timestamp'].str.split(pat='_', n=- 1, expand=True)[0]+test_data['site_path_timestamp'].str.split(pat='_', n=- 1, expand=True)[1]).unique().shape\n\ntest_data['p_floor']=pred\ntest_data\n#re-elaboration taking the median\ndef modee1(a):\n    return (a['path'].unique())\ndef modee2(a):\n    return( a['p_floor'].value_counts().head(1).reset_index()['index'].values[0])\n\ndft = pd.DataFrame()    \n# df['path']=X_ass_val.groupby('path').apply(modee1)\ndft['my_b_floor_pred']=test_data.groupby('path').apply(modee2)\ntest_data=test_data.merge(dft, how='left', on='path')\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#fetching K' submissions to see if there is an improvement on the lb\nsub= pd.read_csv('../input/lstm-by-keras-with-unified-wi-fi-feats/submission.csv')\nsub['floor']=test_data['my_b_floor_pred']\nsub.index=sub['site_path_timestamp']\nsub.drop(columns=['site_path_timestamp'],inplace=True)\nsub.to_csv('submission.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"See if it gets and up grade on the score by substituting this on the floor prediction\n      unfortunately on the pubblic leaderboard score it doesn't improve on the visible decimal values, but on the private ? \n      \ncheck if there are differences that got noticed :"},{"metadata":{"trusted":true},"cell_type":"code","source":"sub= pd.read_csv('../input/lstm-by-keras-with-unified-wi-fi-feats/submission.csv')\nprint('the predictions differ on {} %'.format(((test_data['my_b_floor_pred']-sub['floor'])!=0).mean()*100 ))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"That's it Folks \n\nThank you for reading all of it, let me know your thoughts, insights or suggestions. \n\n"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}