{"cells":[{"metadata":{"_uuid":"d293d7c8878bcb7b5fe23cc232ea0fb695c8ccdf"},"cell_type":"markdown","source":"## Introduction"},{"metadata":{"_uuid":"abe31de84a078599e2c14e5b784f53872f9b860c"},"cell_type":"markdown","source":"In this kernel I **tried to use ANN** with the **data prepared** in [Let's get the party started](https://www.kaggle.com/ludovicoristori/vsb-data-prep-let-s-get-the-party-started) and the **lesson learned** with [In search of failures with a simple model](https://www.kaggle.com/ludovicoristori/in-search-of-failures-with-a-simple-model).\n\nI took the most sophisticated parts of the ANN code **from** the notebook of **Khnoi Nguyen** [5-fold LSTM with threshold tuning](https://www.kaggle.com/suicaokhoailang/5-fold-lstm-with-threshold-tuning-0-618-lb). Still out of reach for me, at present, but a **great source of inspiration**.\n\nTalking about the LB, things will go better **next times**: [Life is Life](https://www.youtube.com/watch?v=EGikhmjTSZI), just to finish with another old pop song."},{"metadata":{"_uuid":"e67b0691fbc2e0e525c8d96dd21fd4b0e6b6ad5b"},"cell_type":"markdown","source":"## Basic Imports"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport pyarrow.parquet as pq\nnp.random.seed(123456)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c956ac5363175abe7343996e134fa4e5993aa006"},"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\n%matplotlib inline\nplt.style.use('ggplot')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6e058f5779ff044b0bafb2d0eaa4e5e7daa33d3d"},"cell_type":"code","source":"from sklearn.model_selection import KFold\nfrom sklearn.model_selection import train_test_split\nfrom sklearn import metrics","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import warnings\nwarnings.filterwarnings('ignore')\nimport os\nprint(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"040cee29490d9cd89d5a3dba0536bcdd213002dd"},"cell_type":"code","source":"df_train = pd.read_csv('../input/vsb-data-prep-let-s-get-the-party-started/df_train.csv')\ndf_train.iloc[:,0:12].head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4669892a6e2605b2b87a2215ebfc560937f4238c"},"cell_type":"code","source":"df_test = pd.read_csv('../input/vsb-data-prep-let-s-get-the-party-started/df_test.csv')\ndf_test.iloc[:,0:12].head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2d7d0a319b1149dd05bb606287f22b2f98aa91ed"},"cell_type":"code","source":"df_subm = pd.read_csv('../input/vsb-power-line-fault-detection/sample_submission.csv')\ndf_subm.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"aca8c75f93ba1381ed5c0c66f4570a459132d2dd"},"cell_type":"code","source":"outname = 'target'\npredictors = list(df_train.columns)\npredictors.remove('signal_id')\npredictors.remove('id_measurement')\npredictors.remove(outname)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"71b5e854f0112cdc21408db04ec1408b078a8d80"},"cell_type":"markdown","source":"The phases have to stay united. We introduce this code:"},{"metadata":{"trusted":true,"_uuid":"d9fefa30f835d791ff7f955a4e610df01f6c26e5"},"cell_type":"code","source":"def remove_cols(df,col_to_delete):\n    df_0=df[df['phase']==0]\n    df_0.drop(col_to_delete,axis=1,inplace=True)\n    df_1=df[df['phase']==1]\n    df_1.drop(col_to_delete,axis=1,inplace=True)\n    df_2=df[df['phase']==2]\n    df_2.drop(col_to_delete,axis=1,inplace=True)\n    df_merge=df_0.merge(df_1, on='id_measurement')\n    df_merge=df_merge.merge(df_2, on='id_measurement')\n    return(df_merge)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"907e151d4851a6d1c5a9111ecd8e2d451425806f"},"cell_type":"code","source":"col_to_delete=['phase','signal_id','ErrFun','ErrGen','Amp0','Amp1','Pha0','Pha1','target']\ndf_train_r=remove_cols(df_train,col_to_delete)\ndf_test_r=remove_cols(df_test,col_to_delete)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a4fd683b176a029649d44bcc4c8adc7ae6855c25"},"cell_type":"code","source":"X_df=df_train_r\nXT_df=df_test_r\nX_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"873d252579538086226414525708ee2023183488"},"cell_type":"code","source":"y_df=df_train['target'].groupby(by=df_train['id_measurement']).first()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8bd45b0bad55180999956d78df76604d6016515e"},"cell_type":"code","source":"X_train_df, X_valid_df, Y_train_df, Y_valid_df = train_test_split(X_df, y_df, test_size=0.2, random_state=123)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"88a61a51de297b5ee37e0872db728cbf3eba6604"},"cell_type":"markdown","source":"Other considerations and (maybe) useful functions:"},{"metadata":{"trusted":true,"_uuid":"1fae5fb3ed853094ca4ef747864236a869fe955e"},"cell_type":"code","source":"absolute_max=max(max(df_train['max']),max(df_test['max']))\nabsolute_max","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"84e30d6cebbb238c0743bd8ff334cf314b7be70e"},"cell_type":"code","source":"absolute_min=min(min(df_train['min']),min(df_test['min']))\nabsolute_min","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f454ac26ee280af32a4a2a68307c532e177be09a"},"cell_type":"code","source":"absolute_std=np.mean(df_train['std'])\nabsolute_std","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e372760c9784c304811657a8204e9e83f85aa5fa"},"cell_type":"code","source":"def damaged_ratio(Y, thr):\n    dr = 100*sum(Y>=thr)/len(Y)\n    return (dr)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"705e33d46f6f3610322aad71e8eee78eb35a9a11"},"cell_type":"code","source":"def to_int_th(x,th,inverse):\n    y = np.zeros(len(x))\n    for i in range(0,len(x)):\n        if (x[i]>=th) :\n            y[i]=1\n        else:\n            y[i]=0\n        if (inverse==1):\n            y[i]=1-y[i]\n    y = y.astype(int)\n    return (y)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aea277404ff094c483c84c55482cf21451627e73"},"cell_type":"markdown","source":"## My First ANN"},{"metadata":{"_uuid":"9fa36eb15c5d03ba660244c5606d069528b74ed5"},"cell_type":"markdown","source":"Well, I would say \"my first ANN in the last 20 years\" as my masters thesis was about ANN and at the times, I wrote a VB program to implement a Feedforward Neural Network. But many things are happened since those years and the reality is that now, in 2019, I am an absolute beginner with ANN. OK, let's go."},{"metadata":{"_uuid":"da024f9f98225f3e6e01e11f0c7aef79f06de20f"},"cell_type":"markdown","source":"At first we have to scale the arrays of predictors. I prefer working on copies:"},{"metadata":{"trusted":true,"_uuid":"d49f8c515b93911eb34ea451ce12772d744ad87a"},"cell_type":"code","source":"X_train_base=np.array(X_train_df.values, copy=True)\nY_train_base=np.array(Y_train_df, copy=True)\nX_valid_base=np.array(X_valid_df.values, copy=True)\nY_valid_base=np.array(Y_valid_df, copy=True)\nX_test_base=np.array(XT_df.values, copy=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9c56c619729b752c0b97c77fcfc664669c72c717"},"cell_type":"code","source":"from sklearn.preprocessing import MinMaxScaler\nfrom sklearn.preprocessing import StandardScaler\nsc = StandardScaler()\nsc.fit(X_train_base)\nX_train = sc.transform(X_train_base) # label: Y_train_base (unscaled)\nX_valid = sc.transform(X_valid_base) # label: Y_valid_base (unscaled)\nX_test = sc.transform(X_test_base) # label: our goal ;-)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fd2e04c3010ac1a6983d7eda17d5cd265ef73955"},"cell_type":"markdown","source":"We are now ready to introduce Keras and the network."},{"metadata":{"trusted":true,"_uuid":"1074101c7e1aa080f41bf41ad5022da2192613ef"},"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow import keras\nfrom tensorflow.keras import layers","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8b167c7314330dde67450c360ab61b2369399ff1"},"cell_type":"code","source":"model = keras.Sequential([\n    layers.Dense(12, activation=\"relu\"),\n    layers.Dense(1, activation=\"relu\")\n])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"063451ff7d2b923bfad55a75cf0cab97a32664bb"},"cell_type":"markdown","source":"We defined a very simple network with two layes.  We respect the \"rule of the triangle\", which says that every layer has to have less nodes (neurons) than the previous one and we finished with 2 nodes, as we have two final classes: 0=healty and 1=damaged. The input layer implement an I/O transformation such as 18->12: each element of X_train (array 2D) is an array of 18 elements, which is transformed in a 12 elements array and finally in a 2-element one."},{"metadata":{"_uuid":"d92ced7793088d18f313541b0fe6ab42b9e099af"},"cell_type":"markdown","source":"ReLU and Sigmoid are activation functions, something that maps one interval (for example (-inf,+inf)) in another one (for example (0,1)). After the topology we have to define the optimizer (how the model try to minimize the error), the loss (how the previous error is measured) and the metric (something printed in the train log, useful only for the user)."},{"metadata":{"trusted":true,"_uuid":"bcdea4c8e567e5a292840d07579f2fe6507370c4"},"cell_type":"code","source":"NR_EPOCHS=5","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"075def1953c78798ebca05b296d9322ea11e342b"},"cell_type":"code","source":"model.compile(optimizer='adam', \n              loss='binary_crossentropy',\n              metrics=['accuracy'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dca84cda2bd3740c3393273a82cbcc91ccfc1d52"},"cell_type":"code","source":"history = model.fit(X_train, Y_train_base, \n                    validation_data=[X_valid, Y_valid_base], epochs=NR_EPOCHS)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"77887e663492f52a6d84d42dd4a71b3af1b5ee23"},"cell_type":"code","source":"Y_valid1 = model.predict(X_valid)\nY_valid1.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b62347060fd29c02bdd6583187accfd55a322b2c"},"cell_type":"code","source":"Y_valid1_df=pd.DataFrame(Y_valid1)\nY_valid1_df.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9d0c09a7eb7afc10ba89496a5f4da63791c2567f"},"cell_type":"code","source":"sns.distplot(Y_valid1, color='blue',bins=10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c74630aaacd1ad9a6bff63f74bb40196028b9d6a"},"cell_type":"code","source":"th1=(np.min(Y_valid1)+np.max(Y_valid1))/2 # damaged if > th1\nY_valid1_int=to_int_th(Y_valid1,th1,0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"296cf1cef359d05284785b7d9a5159891eaad909"},"cell_type":"code","source":"metrics.confusion_matrix(Y_valid_base,Y_valid1_int)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1d719e8a1a5bfc77eed8d1ba88c754f9d4607af2"},"cell_type":"markdown","source":"Here is the Matthews Correlation function I initially wrote. I continue using it in log messages, but I have to substitute it with a tensor version in learning, see further on."},{"metadata":{"trusted":true,"_uuid":"7a21d89da1ced79c4ff1a0540fe901b60a423c07"},"cell_type":"code","source":"def mmc(y_real_int, y_calc_int):\n    cm = metrics.confusion_matrix(y_real_int,y_calc_int)\n    tp = cm[0,0]\n    tn = cm[1,1]\n    fp = cm[0,1]\n    fn = cm[1,0]\n    num = tp*tn-fp*fn\n    den = np.sqrt((tp+fp)*(tp+fn)*(tn+fp)*(tn+fn))\n    if den==0:\n        mc=-1\n    else:\n        mc=num/den\n    return np.float64(mc)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c9a1cd8887dde01d1e92430674424d260f996749"},"cell_type":"code","source":"mmc(Y_valid_base,Y_valid1_int)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"42fc66054b4a0d608c98092eb711b1bd8af2929f"},"cell_type":"code","source":"metrics.accuracy_score(Y_valid_base,Y_valid1_int)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"65b2f81904896a9394d35d7fe3e2fc7a9c8915d7"},"cell_type":"code","source":"Y_pred = model.predict(X_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0daba7c700651ff8623b634f0e6a61c1bb766642"},"cell_type":"code","source":"Y_pred.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1badc9b07dafbd681b2ab5748477bbe93c00eab2"},"cell_type":"code","source":"sns.distplot(Y_pred,bins=10)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ea5431b4c363e9840a840cd167f8bd3a45f2bcd1"},"cell_type":"code","source":"Y_pred_int=to_int_th(Y_pred,th1,0)\nnp.unique(Y_pred_int,return_counts=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"08c674d6cee4db7b253eed07ebac3e5a242c10d9"},"cell_type":"markdown","source":"We have a lot of FN considering that in our train set results:"},{"metadata":{"trusted":true,"_uuid":"1d29c235fc11d919a67ee4ddc0591c8e725674cb"},"cell_type":"code","source":"damaged_ratio(Y_train_base,0.5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ba8568d03a584d6d8db46df9116fba5084d1e3a3"},"cell_type":"markdown","source":"Solution: better threshold search, better data, better model (;-))"},{"metadata":{"_uuid":"7b79b1e0e887f89d8cbfa870743fe764a45aad39"},"cell_type":"markdown","source":"## Doing Better (?)"},{"metadata":{"_uuid":"85254fc14d1a39488fd4f1c7c1480a83cb072acb"},"cell_type":"markdown","source":"Better threshold seach. Let's try with this function:"},{"metadata":{"trusted":true,"_uuid":"1389ba2059ffa1533e455fb0fac7529a038b06bc"},"cell_type":"code","source":"def find_thres(y_real, y_calc):\n    thr_ndiv=100\n    y_min=np.min(y_calc)\n    y_max=np.max(y_calc)\n    start_thres = (y_min+y_max)/2 # default, better than 0\n    stop_thres = y_max\n    opt_thres=start_thres\n    opt_mmc = -1\n    dthr=(stop_thres-start_thres)/thr_ndiv\n    if (dthr==0):\n        vec_thres = np.arange(start_thres, stop_thres+0.1,0.1)\n    else:\n        vec_thres = np.arange(start_thres,stop_thres, dthr)\n    for thres in vec_thres:\n        y_calc_int=to_int_th(y_calc,thres,0)\n        m = mmc(y_real,y_calc_int)\n        if (m > opt_mmc):\n            opt_mmc = m\n            opt_thres = thres\n    print('opt. thres={t:.5f} mmc={m:.5f}'.format(t=opt_thres,m=opt_mmc))\n    return opt_thres","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"87182bcb021fe2a765cb5430258a15fa798530ff"},"cell_type":"markdown","source":"Better Data. "},{"metadata":{"trusted":true,"_uuid":"a4d4324115898fa651ca7b29fc6fa589f4306008"},"cell_type":"code","source":"import gc\nimport pyarrow.parquet as pq","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7f84e35c18fd4449026c93ef3fcbcb1f0be7257f"},"cell_type":"code","source":"metadata_train = pd.read_csv(\"../input/vsb-power-line-fault-detection/metadata_train.csv\")\nmetadata_train.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3c262ce501bbf4d0733014c72e304139f8989bb3"},"cell_type":"code","source":"row_nr=800000\nrow_nr","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4e22b276091eb962397bda830b5eece9484e7863"},"cell_type":"code","source":"row_group_size=4000","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b692af02e59735c77a7653e7e71be89d577e4d20"},"cell_type":"code","source":"time_sample_idx=np.arange(0,row_nr,row_group_size)\ntime_sample_idx[0:8]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fadd7c7b39fe782f944fdb8afd0c8d108bfcfc67"},"cell_type":"code","source":"metadata_test = pd.read_csv(\"../input/vsb-power-line-fault-detection/metadata_test.csv\")\nmetadata_test.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"32a60d0f3d0996cd570971eb7c5dab0b0b23e9da"},"cell_type":"code","source":"sign_start=min(metadata_test['signal_id'])\nsign_start","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"64c1d5acd78320bd67bcedc15b1238c0374270f3"},"cell_type":"code","source":"sign_stop=max(metadata_test['signal_id'])+1\nsign_stop","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"dc78d1e9ceb349b839e3b313922383a7341cf9b2"},"cell_type":"code","source":"sign_group_size=2000","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"adcd1b34155ab639fa2d0bf43e84b099c8a2f36a"},"cell_type":"markdown","source":"Now, some functions to scale and interpolate data;"},{"metadata":{"trusted":true,"_uuid":"c74f5242be9f65814051f32e08268d504aad1aee"},"cell_type":"code","source":"def scale(val,orig_min,orig_max,des_min,des_max):\n    X_std = (val - orig_min) / (orig_max - orig_min)\n    X_scaled = X_std * (des_max - des_min) + des_min\n    return(X_scaled)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"85b58e071285d3a03f75e65ad288495fafd5756a"},"cell_type":"code","source":"def y_line(x,x1,y1,x2,y2):\n    if (x1==x2):\n        y=(y1+y2)/2\n    else:\n        m=(y1-y2)/(x1-x2)\n        q=y1-m*x1\n        y=m*x+q\n    return (y)    ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"dcad53bf992ec39b2ce6d0c24ed0e9a76c2b39aa"},"cell_type":"markdown","source":"Here the question is: **how to properly choose the features**? I started with something basic: taking a chunk (range) of the train/test set and finding its mean, max, min, std. Then, considering \"little\" the time interval and thus almost linear the signal y_line, I calculated the differences y_line-min/max. In this way I intended to find approximations of the upper and lower values of the error. Note: nr_ts=number of initial and final samples used to calculate initial and final points (just to avoid situations where one can have min/max in the initial/final time samples)."},{"metadata":{"trusted":true,"_uuid":"e339f9ceedb63d0a16c37c7d5fbd4be65376100b"},"cell_type":"code","source":"def extract_signal_features(signal_id,file_i,time_sample_idx,abs_max,abs_min):\n    feat_nr=6\n    signal_features=np.zeros((len(time_sample_idx),6))\n    for j in range(0,len(time_sample_idx)-1):\n        file_i_range_j = file_i.iloc[time_sample_idx[j]:time_sample_idx[j+1],signal_id]\n        nr_ts=5\n        x1=time_sample_idx[j]\n        y1=np.mean(file_i.iloc[x1:x1+nr_ts,signal_id])\n        x2=time_sample_idx[j+1]\n        y2=np.mean(file_i.iloc[x2-nr_ts:x2,signal_id])\n        x1=x1+nr_ts/2\n        x2=x2-nr_ts/2\n        range_mean = np.mean(file_i_range_j)\n        x_min=file_i_range_j.idxmin()\n        range_min = np.min(file_i_range_j)\n        err_min =range_min-y_line(x_min,x1,y1,x2,y2)\n        x_max=file_i_range_j.idxmax()\n        range_max = np.max(file_i_range_j)\n        err_max=range_max-y_line(x_max,x1,y1,x2,y2)\n        range_std = np.std(file_i_range_j)\n        if (range_std==0):\n            err_rel_rng=0\n            err_abs_rng=0\n        else:\n            err_rel_rng=(err_max-err_min)/range_std\n            err_abs_rng=err_max-err_min\n        prc_low=np.percentile(file_i_range_j,5)\n        prc_high=np.percentile(file_i_range_j,95)\n        sign_feat = np.array([range_mean,\n                        range_std,\n                        err_rel_rng,\n                        err_abs_rng,\n                        prc_low,\n                        prc_high])\n        signal_features[j]=sign_feat\n    return signal_features    ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e0b42e8dd73c988becec105ea071406ce458a0f4"},"cell_type":"markdown","source":"This is the basic idea behind the approximation:"},{"metadata":{"_uuid":"aa902d7950066161390223e4942f98bc5f6efed5"},"cell_type":"markdown","source":"![](https://i.imgur.com/q4VWwlj.png)"},{"metadata":{"trusted":true,"_uuid":"3c4517c9b3193c2f2eaa735a9a755b9d7d289cc6"},"cell_type":"code","source":"def fill_ar_samples(filepath,sign_start,sign_stop,sign_group_size,row_nr,time_sample_idx,\n                   abs_max,abs_min):\n    time_samples_str=[str(idx) for idx in time_sample_idx]\n    feat_nr=6\n    samples_ar=np.zeros((sign_stop-sign_start,len(time_sample_idx),feat_nr))\n    col_id_start=sign_start\n    n_groups = int(np.round((sign_stop-sign_start)/sign_group_size))+1\n    print('Steps = {}'.format(n_groups))\n    for i in range(0,n_groups):\n        col_id_stop = np.minimum(col_id_start+sign_group_size,sign_stop)\n        col_numbers = np.arange(col_id_start,col_id_stop)\n        print('Step {s} - cols = [{a},{b})'.format(s=i,a=col_id_start,b=col_id_stop))\n        col_names = [str(c_num) for c_num in col_numbers]\n        file_i = pq.read_pandas(filepath,columns=col_names).to_pandas()\n        for c in col_names:\n            if (int(c)%50==0):\n                print('.',end='')\n            col=int(c)-col_id_start\n            feat = extract_signal_features(col,file_i,time_sample_idx,abs_max,abs_min)\n            samples_ar[int(c)-sign_start] = feat\n        col_id_start=col_id_stop\n        print('')\n    return (samples_ar)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"838ae13ff07f67c530ac75880be7526a257df12a"},"cell_type":"markdown","source":"Let's use the loading function above defined:"},{"metadata":{"trusted":true,"_uuid":"d04479dd4b1342fe3d816d6f16497dcef801a263"},"cell_type":"code","source":"%%time\ntrain_samples=fill_ar_samples('../input/vsb-power-line-fault-detection/train.parquet',0,sign_start,sign_group_size,row_nr,time_sample_idx,absolute_max,absolute_min)\ntrain_samples.tofile('train.npy')\ntrain_samples.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7d2f1ef81f4ceedc61fd4b2f1b6afdeb2131953b"},"cell_type":"code","source":"def ar_compacted_phases(ar_samples,df_metadata,start_id_meas):\n    nr_id_meas=int(ar_samples.shape[0]/3)\n    nr_samples=ar_samples.shape[1]\n    nr_feats_per_phase=ar_samples.shape[2]\n    ar_measures=np.zeros((nr_id_meas,nr_samples,3*nr_feats_per_phase))\n    for idx_signal in range(0,len(ar_samples)):\n        idx_meas=df_metadata['id_measurement'].loc[idx_signal]-start_id_meas\n        idx_sample=int(idx_signal%nr_samples)\n        for phase in range(0,3):\n            f_start=int(phase*nr_feats_per_phase)\n            f_stop=int(f_start+nr_feats_per_phase)\n            ar_measures[idx_meas,\n                idx_sample,\n                f_start:f_stop]=ar_samples[idx_signal,idx_sample,0:nr_feats_per_phase]\n    return ar_measures","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ce2ae9e92414c9d27c1a9f93a43b99202e9a0be5"},"cell_type":"code","source":"train_cf=ar_compacted_phases(train_samples,metadata_train,0)\ntrain_cf.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"333c408e2155598135fb52f2ccc0208df49cd0ce"},"cell_type":"code","source":"y_cf=y_df.values\ny_cf","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f9bec3521d0c5cd39dcd0d68c94e39d0bd20be11"},"cell_type":"code","source":"%%time\ntest_samples=fill_ar_samples('../input/vsb-power-line-fault-detection/test.parquet',\n                             sign_start,sign_stop,sign_group_size,row_nr,time_sample_idx,\n                             absolute_max,absolute_min)\ntest_samples.tofile('test.npy')\ntest_samples.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"196961474f5b07334ef939ea319406f49552822e"},"cell_type":"code","source":"id_meas_start=min(metadata_test['id_measurement'])\ntest_cf=ar_compacted_phases(test_samples,metadata_test,id_meas_start)\ntest_cf.shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"25429688f7f0de2473e394730b96805eff063371"},"cell_type":"markdown","source":"Better previsional model: let's see at some good public kernel; for example this work of Khnoi Nguyen: [5-fold LSTM with threshold tuning](https://www.kaggle.com/suicaokhoailang/5-fold-lstm-with-threshold-tuning-0-618-lb). "},{"metadata":{"trusted":true,"_uuid":"db26e5de1249f2641c4cb5c825a0471e9602d858"},"cell_type":"code","source":"from keras.layers import *\nfrom keras.models import Model\nfrom keras.callbacks import ModelCheckpoint\nfrom keras.callbacks import *","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"40e39d07a7ce9b1c5f9174ad48f4bdb3a3806d5f"},"cell_type":"code","source":"class Attention(Layer):\n    def __init__(self, step_dim,\n                 W_regularizer=None, b_regularizer=None,\n                 W_constraint=None, b_constraint=None,\n                 bias=True, **kwargs):\n        self.supports_masking = True\n        self.init = initializers.get('glorot_uniform')\n\n        self.W_regularizer = regularizers.get(W_regularizer)\n        self.b_regularizer = regularizers.get(b_regularizer)\n\n        self.W_constraint = constraints.get(W_constraint)\n        self.b_constraint = constraints.get(b_constraint)\n\n        self.bias = bias\n        self.step_dim = step_dim\n        self.features_dim = 0\n        super(Attention, self).__init__(**kwargs)\n\n    def build(self, input_shape):\n        assert len(input_shape) == 3\n\n        self.W = self.add_weight((input_shape[-1],),\n                                 initializer=self.init,\n                                 name='{}_W'.format(self.name),\n                                 regularizer=self.W_regularizer,\n                                 constraint=self.W_constraint)\n        self.features_dim = input_shape[-1]\n\n        if self.bias:\n            self.b = self.add_weight((input_shape[1],),\n                                     initializer='zero',\n                                     name='{}_b'.format(self.name),\n                                     regularizer=self.b_regularizer,\n                                     constraint=self.b_constraint)\n        else:\n            self.b = None\n\n        self.built = True\n\n    def compute_mask(self, input, input_mask=None):\n        return None\n\n    def call(self, x, mask=None):\n        features_dim = self.features_dim\n        step_dim = self.step_dim\n\n        eij = K.reshape(K.dot(K.reshape(x, (-1, features_dim)),\n                        K.reshape(self.W, (features_dim, 1))), (-1, step_dim))\n\n        if self.bias:\n            eij += self.b\n\n        eij = K.tanh(eij)\n\n        a = K.exp(eij)\n\n        if mask is not None:\n            a *= K.cast(mask, K.floatx())\n\n        a /= K.cast(K.sum(a, axis=1, keepdims=True) + K.epsilon(), K.floatx())\n\n        a = K.expand_dims(a)\n        weighted_input = x * a\n        return K.sum(weighted_input, axis=1)\n\n    def compute_output_shape(self, input_shape):\n        return input_shape[0],  self.features_dim","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"69d41d3e3f8c1e8c8b4aab01595d6febd067c775"},"cell_type":"markdown","source":"The following is a tensor implementation of the Mattews Correlation. I discovered I can't use my function mmc in a Keras model, based on Tensors."},{"metadata":{"trusted":true,"_uuid":"8d58b941fcf6f2858e30f7faf9ea5404a56c1b21"},"cell_type":"code","source":"def matthews_correlation(y_true, y_pred):\n    '''Calculates the Matthews correlation coefficient measure for quality\n    of binary classification problems.\n    '''\n    y_pred = tf.convert_to_tensor(y_pred, np.float32)\n    y_true = tf.convert_to_tensor(y_true, np.float32)\n    \n    y_pred_pos = K.round(K.clip(y_pred, 0, 1))\n    y_pred_neg = 1 - y_pred_pos\n\n    y_pos = K.round(K.clip(y_true, 0, 1))\n    y_neg = 1 - y_pos\n\n    tp = K.sum(y_pos * y_pred_pos)\n    tn = K.sum(y_neg * y_pred_neg)\n\n    fp = K.sum(y_neg * y_pred_pos)\n    fn = K.sum(y_pos * y_pred_neg)\n\n    numerator = (tp * tn - fp * fn)\n    denominator = K.sqrt((tp + fp) * (tp + fn) * (tn + fp) * (tn + fn))\n\n    return numerator / (denominator + K.epsilon())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8704f64d0f94801b9d1963b9431fdbbdb4c0e5bf"},"cell_type":"code","source":"def model_lstm(input_shape):\n    inp = Input(shape=(input_shape[1], input_shape[2]))\n    x = Bidirectional(CuDNNLSTM(128, return_sequences=True))(inp)\n    x = Bidirectional(CuDNNLSTM(64, return_sequences=True))(x)\n    x = Attention(input_shape[1])(x)\n    x = Dense(64, activation=\"relu\")(x)\n    x = Dense(1, activation=\"sigmoid\")(x)\n    model = Model(inputs=inp, outputs=x)\n    model.compile(loss='binary_crossentropy', optimizer='adam', metrics=[matthews_correlation])\n    return model","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b2a5078fc9633776411d308e65120182f4d9b503"},"cell_type":"markdown","source":"To perform the fit you have to **Enable GPU**.  The dataframe defined below is used only to store the training scores:"},{"metadata":{"trusted":true,"_uuid":"8cb82c058aeada0c700f87c401047af8d9bee686"},"cell_type":"code","source":"CV_STEPS=8\nNR_EPOCHS=40","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"054d82fcd3ed57f6fb56be554175223980c1860b"},"cell_type":"code","source":"ar_cv=np.arange(0,CV_STEPS)\nep_cv=np.arange(0,NR_EPOCHS)\nmi=pd.MultiIndex.from_product([ar_cv,ep_cv], names=['cv','epoch'])\ndf=pd.DataFrame(index=mi,columns=['loss','val_loss','matthews_correlation','val_matthews_correlation'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"75e528ad2703b0585bf41ef4a2c6649056bd2787"},"cell_type":"code","source":"KF = KFold(n_splits=CV_STEPS, shuffle=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9a0a61c1fa00c7a85a896a1f74740fa2054fb821"},"cell_type":"code","source":"opt_thr=np.zeros(CV_STEPS)\nfor k in range(0,CV_STEPS):\n    print('Step {}'.format(k))\n    train_y=df_train[outname].values\n    X_train, X_valid, y_train, y_valid = train_test_split(train_cf, y_cf, test_size=1/CV_STEPS)\n    w_file_name='weights_best_{}.hdf5'.format(k)\n    model = model_lstm(X_train.shape)\n    ckpt = ModelCheckpoint(w_file_name, save_best_only=True, verbose=1,\n                           save_weights_only=True, monitor='val_matthews_correlation', \n                           mode='max')\n    history=model.fit(X_train, y_train, epochs=NR_EPOCHS, batch_size=128, shuffle=True,\n          validation_data=[X_valid, y_valid],callbacks=[ckpt])\n    if (os.path.exists(w_file_name)):\n        print('weight file loaded...')\n        model.load_weights(w_file_name)\n    y_valid1=model.predict(X_valid)\n    opt_thr[k]=find_thres(y_valid, y_valid1)\n    df['loss'].loc[k].iloc[0:NR_EPOCHS]=history.history['loss']\n    df['val_loss'].loc[k].iloc[0:NR_EPOCHS]=history.history['val_loss']\n    df['matthews_correlation'].loc[k].iloc[0:NR_EPOCHS]=history.history['matthews_correlation']\n    df['val_matthews_correlation'].loc[k].iloc[0:NR_EPOCHS]=history.history['val_matthews_correlation']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"598a800695056321fdb88e2b9e54a0dcdff76be2"},"cell_type":"code","source":"h=history.history\nprint(h.keys())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f6cadb117f1c2d9dd69612bb698c8acb0aea991e"},"cell_type":"code","source":"fig,ax = plt.subplots(1,2, figsize=(10,5))\nfor k in range(0,CV_STEPS):\n    loss=df['loss'].loc[k]\n    val_loss=df['val_loss'].loc[k]\n    ax[0].plot(loss, color='red')\n    ax[1].plot(val_loss, color='blue')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"16dbc903a6309e9f2d217475063c76fe57492626"},"cell_type":"code","source":"fig,ax = plt.subplots(1,2, figsize=(10,5))\nmmc_train_last=np.zeros(CV_STEPS)\nmmc_valid_last=np.zeros(CV_STEPS)\nfor k in range(0,CV_STEPS):\n    mc=df['matthews_correlation'].loc[k]\n    val_mc=df['val_matthews_correlation'].loc[k]\n    ax[0].plot(mc, color='red')\n    ax[1].plot(val_mc, color='blue')\n    mmc_train_last[k]=df['matthews_correlation'].loc[k].loc[NR_EPOCHS-1]\n    mmc_valid_last[k]=df['val_matthews_correlation'].loc[k].loc[NR_EPOCHS-1]\nm=np.mean(mmc_train_last)\ns=np.std(mmc_train_last) \nprint('matthews_correlation mean={m} std={s}'.format(k=k,m=m,s=s))\nm=np.mean(mmc_valid_last)\ns=np.std(mmc_valid_last) \nprint('val_matthews_correlation mean={m} std={s}'.format(k=k,m=m,s=s))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5f10c03e13537ff93b4bc7c65345f521943a4b88"},"cell_type":"code","source":"sns.distplot(y_valid1,color='green')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"thr_avg=np.mean(opt_thr)\nthr_std=np.std(opt_thr)\nprint('threshold={av} std={st}'.format(av=thr_avg,st=thr_std))\nfin_thr=thr_avg","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2260126d939bb8b32ecc961b8c8b1cf2f1689e06"},"cell_type":"code","source":"y_valid1_int=to_int_th(y_valid1,fin_thr,0)\nmetrics.confusion_matrix(y_valid,y_valid1_int)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4c4ece240d1ccc46688728053b35573d3c4a01b6"},"cell_type":"markdown","source":"We don't perform a final train, calculating instead predictions basing on the average of the folds:"},{"metadata":{"trusted":true,"_uuid":"52e4b9acb70c99419d018f4c0359b4826d43a938"},"cell_type":"code","source":"%%time\nar_pred = np.zeros((len(test_cf),CV_STEPS))\nfor k in range(0,CV_STEPS):\n    w_file_name='weights_best_{}.hdf5'.format(k)\n    model1 = model_lstm(X_train.shape)\n    model1.load_weights(w_file_name)\n    y_pred1=model1.predict(test_cf)\n    ar_pred[:,k]=np.squeeze(y_pred1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9654169f5d8dc186619082924aa56100e63e654c"},"cell_type":"code","source":"y_pred=ar_pred.mean(axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d5523469830b6e5383d4ac788e11e434d1a21b18"},"cell_type":"code","source":"sns.distplot(y_pred,color='green')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"aaf423737e54f49e14e0e507033bde80c5d48415"},"cell_type":"code","source":"y_pred_int=to_int_th(y_pred,fin_thr,0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"019d723fc0e942a35d690083f800fe1566717406"},"cell_type":"code","source":"np.unique(y_pred_int,return_counts=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6ffe79c06813218b2de5ee1189b3c00558003069"},"cell_type":"markdown","source":"We have to return to the prevision per signal:"},{"metadata":{"trusted":true,"_uuid":"c84273af51bed918d141554ebd7883c10c10478b"},"cell_type":"code","source":"XT_df['max']=y_pred_int\ndf_pred=XT_df[['id_measurement','max']]\ndf_pred.columns=['id_measurement','target']\ndf_pred.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"aad7776b359319512c853ba683087b7a8b315696"},"cell_type":"code","source":"df_subm=df_test[['signal_id','id_measurement']].merge(df_pred, on='id_measurement')\ndf_subm.drop('id_measurement',axis=1,inplace=True)\ndf_subm.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0667d925731b9046156204a66219225d2bffafe4"},"cell_type":"code","source":"sum(df_subm['target'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f78bcb918dade3e83d94680a80fc10450ab96210"},"cell_type":"code","source":"df_subm.to_csv('submission.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}