{"cells":[{"metadata":{},"cell_type":"markdown","source":"**Training and Validation set generator for LANL Earthquake Prediction**\n\nGenerates list of start rows in train.csv which can be used for creating training and validation data\n\n* Sets can be created in a way that they are approximate of a specific histogram of y ('time_to_failure') \n* Samples overlap adjustable from 0 to 150,000 rows\n* Samples with \"Earthquakes\" are excluded\n* Optional exclusion list of samples (good for example to exclude validation data from training set)\n* 150,000 rows samples assumed"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\nfrom matplotlib import pyplot as plt\n%matplotlib inline\nimport numpy as np\nfrom tqdm import tqdm","execution_count":16,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Seed set for testing only"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"np.random.seed(3)","execution_count":17,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To load full train.csv, comment first and uncomment second line"},{"metadata":{"trusted":true},"cell_type":"code","source":"raw_train_data = pd.read_csv ('../input/train.csv', nrows=100000000,dtype={'acoustic_data': np.int8, 'time_to_failure': np.float32})\n#raw_train_data = pd.read_csv ('../input/train.csv',dtype={'acoustic_data': np.int8, 'time_to_failure': np.float32})","execution_count":18,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**The generator itself**\n\n* **input_list** is numpy array from the train.csv (raw_train_data.values)\n\n* **set_size** is number of samples in the set\n\n* **max_overlap** is the highest allowed overlap in rows, set to zero if no overlap is requested. My humble opinion is that train data can overlap, validation data should not. **Note:** The max_overlap = 0 setting restrics dramatically number of possible samples, thus some data sets can't be created! Currently there is no protection for endless loop due to shortage of samples with specific criteria combination (overlap, no earthquake, specific histogram and set size) **Start with small sets!**\n\n* **histogram** is the required distribution of y. Please note that the histogram is required to be in whole second intervals. I think it's sufficient \"detail\" for the moment\n\n* **exclusions** is list of start rows which shall be excluded from the set - I use it to exclude validation data from train set, but it can be used to exclude specific sections of train.csv if required. Exclusions are always 150,000 rows long."},{"metadata":{"trusted":true},"cell_type":"code","source":"def generate_data_set (input_list,set_size,max_overlap=0,histogram=[],exclusions=[]):\n# input_list - numpy array(?,2) of train data\n# set_size - number of samples in the data set\n# overlap - define max overlap in rows (150000=100%)\n# histogram - target (y-)histogram for the data set (in 1 second steps). If [], random distribution\n# exclusions - array of index of samples to be excluded (eg. to generate more mutually exclusive validation sets) \n#              Use with care! There is a limited number of possibilites to create validation sets with specific \n#              histogram. There is no protection against endless loop built in!\n\n\n    i=0\n    m=input_list.shape[0]\n    index=[]\n\n    if histogram!=[]:\n        data_size=np.sum(histogram[0])\n        set_hist=np.round(histogram[0]/data_size*set_size)\n    else:\n        data_size=17        \n        set_hist=np.zeros((17,))\n        set_hist[:]=set_size\n\n#    print (\"Set buckets shape \"+str(set_hist.shape))\n#    print (\"Expected buckets\\n\"+str(set_hist))\n    if np.sum(set_hist)!=set_size:             # Correction for rounding errors\n        set_hist[0]=set_hist[0]-(np.sum(set_hist)-set_size)\n    set_bucket=np.zeros(set_hist.shape[0])\n    set_full=np.zeros(set_hist.shape[0])         \n    set_full[set_hist==0]=1.                    # When histogram bucket is empty, it is actually already full\n    for j in tqdm(range (0,set_size)):\n        while True:\n            start = np.int(np.random.rand()*(m-150000)) #reduced length by 150000 to avoid to hit prematurelly end\n#            print (\"Start \"+str(start))\n            overlap=False                               #test for overlap > max_overlap between samples\n            for i in index:\n                if (150000-np.abs(start-i)) > max_overlap:\n#                    print (\"overlap of \"+str(150000-np.abs(start-i))+\" detected at \"+str(i)+\" with \"+str(start))\n                    overlap=True\n                    break\n            if overlap:\n                continue \n            exclude=False\n            for i in exclusions:\n                if (150000-np.abs(start-i))>0:\n#                    print (\"In exclusion list \"+str(i)+\" with \"+str(start))\n                    overlap=True\n                    break\n            if overlap:\n                continue \n            x=input_list[start:start+150000]\n            if np.max(x[1:,1]-x[:-1,1]) > 1:           #testing for earthquake within the sample\n#                print (\"earthquake within sample\")\n                continue  \n            y=np.average(x[:,1])   \n            y_int=int(np.floor(y))\n            \n            if set_full[y_int]:                 #testing if we need this y - does it fit to histogram?\n#                print (\"Bucket \"+str(y_int)+\" full\")\n                continue\n            # If we reached it here, we should have a valid sample\n#            print (\"valid sample \"+str(start)+\" with y \"+str(y))\n            set_bucket[y_int]=set_bucket[y_int]+1\n#            print (set_bucket)\n            if set_bucket[y_int]==set_hist[y_int]:\n#                print (\"Bucket \"+str(y_int)+\" full\")\n                set_full[y_int]=1\n            break\n        index.append(start)\n#    print (\"Final buckets\\n\"+str(set_bucket))\n    return index","execution_count":19,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Here couple of examples:**"},{"metadata":{},"cell_type":"markdown","source":"First we create histogram of the train.csv"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data_hist=np.histogram(raw_train_data['time_to_failure'].values,\n                             bins=range(int(np.floor(raw_train_data['time_to_failure'].values.min())),\n                                        int(np.ceil(raw_train_data['time_to_failure'].values.max()))+1))\nprint (train_data_hist)","execution_count":20,"outputs":[{"output_type":"stream","text":"(array([7700478, 8679653, 7692331, 7700479, 7700478, 7700479, 7700478,\n       7700479, 7700053, 7692712, 7700478, 5935104, 3850239, 3850239,\n        696320]), array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15]))\n","name":"stdout"}]},{"metadata":{},"cell_type":"markdown","source":"3 validation sets with different parameters"},{"metadata":{"trusted":true},"cell_type":"code","source":"validation_set=np.zeros((3,64),dtype=int)","execution_count":21,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Random validation set of 64 samples"},{"metadata":{"trusted":true},"cell_type":"code","source":"validation_set[0,:] = generate_data_set(raw_train_data.values,64)","execution_count":22,"outputs":[{"output_type":"stream","text":"100%|██████████| 64/64 [00:00<00:00, 2942.31it/s]\n","name":"stderr"}]},{"metadata":{},"cell_type":"markdown","source":"Validation set according the train.csv distribution of 'time_to_failure'. Note: the example is on reduced set of 100,000,000 rows "},{"metadata":{"trusted":true},"cell_type":"code","source":"validation_set[1,:] = generate_data_set(raw_train_data.values,64,histogram=train_data_hist)","execution_count":23,"outputs":[{"output_type":"stream","text":"100%|██████████| 64/64 [00:00<00:00, 1542.49it/s]\n","name":"stderr"}]},{"metadata":{},"cell_type":"markdown","source":"Same as before, but all samples from the previous set are excluded"},{"metadata":{"trusted":true},"cell_type":"code","source":"validation_set[2,:] = generate_data_set(raw_train_data.values,64,histogram=train_data_hist,\n                                              exclusions=validation_set[1,:])","execution_count":24,"outputs":[{"output_type":"stream","text":"100%|██████████| 64/64 [00:00<00:00, 1016.71it/s]\n","name":"stderr"}]},{"metadata":{},"cell_type":"markdown","source":"Train data set, with validation set 1 excluded. Overlap of max 50% (75,000 rows) between samples is allowed."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data = generate_data_set(raw_train_data.values,500,histogram=train_data_hist,exclusions=validation_set[1,:],\n                              max_overlap=75000)","execution_count":25,"outputs":[{"output_type":"stream","text":"100%|██████████| 500/500 [00:00<00:00, 538.34it/s] \n","name":"stderr"}]},{"metadata":{},"cell_type":"markdown","source":"**Histograms to show the distbution of 'time_to_failure'**"},{"metadata":{},"cell_type":"raw","source":"This is the base histogram of the train.csv (first 100M rows in this case)"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.hist(raw_train_data['time_to_failure'].values,\n         bins=range(int(np.floor(raw_train_data['time_to_failure'].values.min())),\n                    int(np.ceil(raw_train_data['time_to_failure'].values.max()))+1))","execution_count":26,"outputs":[{"output_type":"execute_result","execution_count":26,"data":{"text/plain":"(array([7700478., 8679653., 7692331., 7700479., 7700478., 7700479.,\n        7700478., 7700479., 7700053., 7692712., 7700478., 5935104.,\n        3850239., 3850239.,  696320.]),\n array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15]),\n <a list of 15 Patch objects>)"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"<Figure size 432x288 with 1 Axes>","image/png":"iVBORw0KGgoAAAANSUhEUgAAAXcAAAEDCAYAAADOc0QpAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAADpJJREFUeJzt3X+s3Xddx/Hni9aBYzjQFtS2222wDJsJbLkZyBIkDEy3kdbEH+nCDISFxoTNKYumiFnMTEwRA5o41AbmEHFzTNTGFQaBmRmzLb37ydo61mx1u2W48mMgEh3Vt3+c0+Vwd2/v97an/Z77yfORND3f7/n0e9/pj2e//Z5zvk1VIUlqywv6HkCSNH7GXZIaZNwlqUHGXZIaZNwlqUHGXZIa1Gvck9yQ5OkkD3dY+5EkDwy/fSXJM6diRklajtLn+9yTvAn4LvBXVXXuEn7cVcB5VfXukzacJC1jvZ65V9WdwDdH9yV5ZZLPJbk3yb8kefU8P/Qy4KZTMqQkLUMr+x5gHjuBX6uqR5O8Hvgo8JajTyY5G1gPfKmn+SRp4k1U3JOcAbwR+HSSo7tfOGfZVuDWqvrfUzmbJC0nExV3BpeJnqmq1x1jzVbgvadoHklalibqrZBV9R3g8SS/DJCB1x59fnj9/WXAXT2NKEnLQt9vhbyJQajPSTKb5ArgHcAVSR4E9gJbRn7IVuDm8laWknRMvb4VUpJ0ckzUZRlJ0nj09oLqqlWrampqqq8vL0nL0r333vv1qlq92Lre4j41NcXMzExfX16SlqUk/95lnZdlJKlBxl2SGmTcJalBxl2SGmTcJalBxl2SGmTcJalBxl2SGmTcJalBk3Y/995Mbb9trMc7uOPSsR5PkpbCM3dJapBxl6QGGXdJapBxl6QGGXdJapBxl6QGGXdJapBxl6QGGXdJapBxl6QGGXdJapBxl6QGdYp7kk1JHklyIMn2eZ4/K8kdSe5P8lCSS8Y/qiSpq0XjnmQFcD1wMbARuCzJxjnLfhe4parOA7YCHx33oJKk7rqcuV8AHKiqx6rqWeBmYMucNQX8yPDxmcBXxzeiJGmpusR9DfDkyPbscN+o3wMuTzIL7Aaumu9ASbYlmUkyc/jw4eMYV5LUxbheUL0MuLGq1gKXAJ9M8rxjV9XOqpququnVq1eP6UtLkubqEvdDwLqR7bXDfaOuAG4BqKq7gBcBq8YxoCRp6brEfQ+wIcn6JKcxeMF015w1TwAXAST5aQZx97qLJPVk0bhX1RHgSuB2YD+Dd8XsTXJdks3DZdcA70nyIHAT8K6qqpM1tCTp2Dr9B9lVtZvBC6Wj+64debwPuHC8o0mSjpefUJWkBnU6c580U9tv63uERS2HGaWlOrjj0r5HUEeeuUtSg4y7JDXIuEtSg4y7JDXIuEtSg4y7JDXIuEtSg4y7JDXIuEtSg4y7JDXIuEtSg4y7JDXIuEtSg4y7JDXIuEtSg4y7JDXIuEtSg4y7JDXIuEtSg4y7JDXIuEtSg4y7JDXIuEtSg4y7JDXIuEtSg4y7JDXIuEtSg4y7JDXIuEtSg4y7JDXIuEtSg4y7JDXIuEtSg4y7JDXIuEtSg4y7JDWoU9yTbErySJIDSbYvsOZXkuxLsjfJ34x3TEnSUqxcbEGSFcD1wNuAWWBPkl1VtW9kzQbg/cCFVfWtJC8/WQNLkhbX5cz9AuBAVT1WVc8CNwNb5qx5D3B9VX0LoKqeHu+YkqSl6BL3NcCTI9uzw32jXgW8Ksm/Jrk7yaZxDShJWrpFL8ss4TgbgDcDa4E7k/xMVT0zuijJNmAbwFlnnTWmLy1JmqtL3A8B60a21w73jZoF7qmq7wOPJ/kKg9jvGV1UVTuBnQDT09N1vENL6sfU9tvGfsyDOy4d+zHV7bLMHmBDkvVJTgO2ArvmrPkHBmftJFnF4DLNY2OcU5K0BIvGvaqOAFcCtwP7gVuqam+S65JsHi67HfhGkn3AHcBvVdU3TtbQkqRj63TNvap2A7vn7Lt25HEB7xt+kyT1zE+oSlKDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDjLskNci4S1KDOsU9yaYkjyQ5kGT7Mdb9YpJKMj2+ESVJS7Vo3JOsAK4HLgY2Apcl2TjPupcAVwP3jHtISdLSdDlzvwA4UFWPVdWzwM3AlnnW/T7wQeC/xzifJOk4dIn7GuDJke3Z4b7nJDkfWFdVtx3rQEm2JZlJMnP48OElDytJ6uaEX1BN8gLgw8A1i62tqp1VNV1V06tXrz7RLy1JWkCXuB8C1o1srx3uO+olwLnAPyc5CLwB2OWLqpLUny5x3wNsSLI+yWnAVmDX0Ser6ttVtaqqpqpqCrgb2FxVMydlYknSolYutqCqjiS5ErgdWAHcUFV7k1wHzFTVrmMfQZIWNrX9mC/V9e7gjkv7HuG4LBp3gKraDeyes+/aBda++cTHkiSdCD+hKkkNMu6S1CDjLkkNMu6S1CDjLkkNMu6S1CDjLkkNMu6S1CDjLkkNMu6S1CDjLkkNMu6S1CDjLkkNMu6S1CDjLkkNMu6S1CDjLkkNMu6S1CDjLkkNMu6S1CDjLkkNMu6S1CDjLkkNMu6S1CDjLkkNMu6S1CDjLkkNMu6S1CDjLkkNMu6S1CDjLkkNMu6S1CDjLkkNMu6S1CDjLkkNMu6S1CDjLkkNMu6S1CDjLkkN6hT3JJuSPJLkQJLt8zz/viT7kjyU5ItJzh7/qJKkrhaNe5IVwPXAxcBG4LIkG+csux+YrqrXALcCfzjuQSVJ3XU5c78AOFBVj1XVs8DNwJbRBVV1R1V9b7h5N7B2vGNKkpaiS9zXAE+ObM8O9y3kCuCz8z2RZFuSmSQzhw8f7j6lJGlJxvqCapLLgWngQ/M9X1U7q2q6qqZXr149zi8tSRqxssOaQ8C6ke21w30/IMlbgQ8AP1dV/zOe8SRJx6PLmfseYEOS9UlOA7YCu0YXJDkP+Atgc1U9Pf4xJUlLsWjcq+oIcCVwO7AfuKWq9ia5Lsnm4bIPAWcAn07yQJJdCxxOknQKdLksQ1XtBnbP2XftyOO3jnkuSdIJ8BOqktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktSglX0PIEmTbGr7bWM/5sEdl479mHN55i5JDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDTLuktQg4y5JDeoU9ySbkjyS5ECS7fM8/8Ikfzt8/p4kU+MeVJLU3aJxT7ICuB64GNgIXJZk45xlVwDfqqqfAj4CfHDcg0qSuuty5n4BcKCqHquqZ4GbgS1z1mwBPjF8fCtwUZKMb0xJ0lJ0+c861gBPjmzPAq9faE1VHUnybeDHgK+PLkqyDdg23PxukkeOZ2hg1dxjT6BJn3HS5wNnHIdJnw8mf8axz5cTu7ZxdpdFp/R/YqqqncDOEz1Okpmqmh7DSCfNpM846fOBM47DpM8Hkz/jpM+3kC6XZQ4B60a21w73zbsmyUrgTOAb4xhQkrR0XeK+B9iQZH2S04CtwK45a3YB7xw+/iXgS1VV4xtTkrQUi16WGV5DvxK4HVgB3FBVe5NcB8xU1S7g48AnkxwAvsngL4CT6YQv7ZwCkz7jpM8HzjgOkz4fTP6Mkz7fvOIJtiS1x0+oSlKDjLskNWjZxX2xWyH0Kcm6JHck2Zdkb5Kr+55pIUlWJLk/yT/1Pct8krw0ya1J/i3J/iQ/2/dMo5L85vDX+OEkNyV50QTMdEOSp5M8PLLvR5N8Icmjw+9fNoEzfmj46/xQkr9P8tJJmm/kuWuSVJJVfcy2VMsq7h1vhdCnI8A1VbUReAPw3gmbb9TVwP6+hziGPwE+V1WvBl7LBM2aZA3w68B0VZ3L4I0GJ/tNBF3cCGyas2878MWq2gB8cbjdpxt5/oxfAM6tqtcAXwHef6qHGnEjz5+PJOuAnweeONUDHa9lFXe63QqhN1X1VFXdN3z8nwyCtKbfqZ4vyVrgUuBjfc8ynyRnAm9i8C4squrZqnqm36meZyXww8PPdZwOfLXneaiqOxm8W23U6K1BPgH8wikdao75Zqyqz1fVkeHm3Qw+S9OLBX4OYXDPrN8Gls07UJZb3Oe7FcLExRNgeGfM84B7+p1kXn/M4Dfq//U9yALWA4eBvxxeOvpYkhf3PdRRVXUI+CMGZ3FPAd+uqs/3O9WCXlFVTw0ffw14RZ/DdPBu4LN9DzEqyRbgUFU92PcsS7Hc4r4sJDkD+DvgN6rqO33PMyrJ24Gnq+revmc5hpXA+cCfVdV5wH/R/+WE5wyvW29h8JfQTwIvTnJ5v1MtbvjBwok980zyAQaXNj/V9yxHJTkd+B3g2r5nWarlFvcut0LoVZIfYhD2T1XVZ/qeZx4XApuTHGRwWestSf6635GeZxaYraqj/+q5lUHsJ8Vbgcer6nBVfR/4DPDGnmdayH8k+QmA4fdP9zzPvJK8C3g78I4J+3T7Kxn8Jf7g8M/MWuC+JD/e61QdLLe4d7kVQm+Gtzn+OLC/qj7c9zzzqar3V9Xaqppi8PP3paqaqLPOqvoa8GSSc4a7LgL29TjSXE8Ab0hy+vDX/CIm6AXfOUZvDfJO4B97nGVeSTYxuEy4uaq+1/c8o6rqy1X18qqaGv6ZmQXOH/4enWjLKu7DF12O3gphP3BLVe3td6ofcCHwqwzOhh8Yfruk76GWqauATyV5CHgd8Ac9z/Oc4b8obgXuA77M4M9R7x9RT3ITcBdwTpLZJFcAO4C3JXmUwb84dkzgjH8KvAT4wvDPzJ9P2HzLkrcfkKQGLaszd0lSN8Zdkhpk3CWpQcZdkhpk3CWpQcZdkhpk3CWpQf8P0tR7Ws4h27UAAAAASUVORK5CYII=\n"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"This is our 500 samples trainig set"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.hist(raw_train_data['time_to_failure'].values[train_data],\n         bins=range(int(np.floor(raw_train_data['time_to_failure'].values.min())),\n                    int(np.ceil(raw_train_data['time_to_failure'].values.max()))+1))","execution_count":27,"outputs":[{"output_type":"execute_result","execution_count":27,"data":{"text/plain":"(array([35., 44., 39., 38., 39., 40., 38., 40., 38., 39., 37., 31., 20.,\n        19.,  3.]),\n array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15]),\n <a list of 15 Patch objects>)"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"<Figure size 432x288 with 1 Axes>","image/png":"iVBORw0KGgoAAAANSUhEUgAAAXQAAAD8CAYAAABn919SAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAADNNJREFUeJzt3X+MZYVZh/Hn6y6kpTUCMq7Isg6xhIY0FpoNoZIYA8WgS7r80Ria2qyRZP+pSpWkLjUxMTFmiaa0iUazAWQTCZRQDASqdrOlaUwquvz+sVYQt+3iwm7TYltNrGtf/7gHMywznTs7d/beeXk+yWTuOffMve/uzDxz5tx77qSqkCStfz8y7QEkSZNh0CWpCYMuSU0YdElqwqBLUhMGXZKaMOiS1IRBl6QmDLokNbHxVN7ZOeecU/Pz86fyLiVp3Xvssce+WVVzy213SoM+Pz/PgQMHTuVdStK6l+Rr42znIRdJasKgS1ITBl2SmjDoktSEQZekJgy6JDVh0CWpCYMuSU0YdElq4pSeKTpL5nc9PPHbPLR728RvU5LG5R66JDVh0CWpCYMuSU0YdElqwqBLUhMGXZKaMOiS1IRBl6QmDLokNWHQJakJgy5JTRh0SWrCoEtSEwZdkpoYO+hJNiR5IslDw/IFSR5N8mKSzyY5fe3GlCQtZyV76DcCBxcs3wLcWlXvAr4N3DDJwSRJKzNW0JNsBrYBtw3LAa4E7hs22QtctxYDSpLGM+4e+qeBTwA/GJZ/HHitqo4Py4eB8yY8myRpBZb9E3RJrgWOVtVjSX5hpXeQZCewE2DLli0rHlCzaz38Gb9Jz+ifGdQsG2cP/Qrgg0kOAfcwOtTyGeDMJK//QNgMvLzYB1fVnqraWlVb5+bmJjCyJGkxywa9qm6uqs1VNQ9cD3yxqj4CPAJ8aNhsB/DAmk0pSVrWap6H/rvA7yR5kdEx9dsnM5Ik6WQsewx9oar6EvCl4fJLwGWTH2n9mvXjtWtxzFurN+tfN1o/PFNUkpow6JLUhEGXpCYMuiQ1saIHRXVqvRUfxJz1f/Osz6e3NvfQJakJgy5JTRh0SWrCoEtSEz4oKjXjmadvXe6hS1ITBl2SmjDoktSEQZekJgy6JDVh0CWpCYMuSU0YdElqYt2cWOSr3EnSD+ceuiQ1YdAlqQmDLklNGHRJasKgS1ITBl2SmjDoktSEQZekJgy6JDVh0CWpCYMuSU0YdElqwqBLUhMGXZKaMOiS1IRBl6QmDLokNWHQJamJdfMn6CRNx1r8+cdDu7dN/DY1xh56krcl+cckTyV5LskfDOsvSPJokheTfDbJ6Ws/riRpKeMccvlv4Mqqei9wCXBNksuBW4Bbq+pdwLeBG9ZuTEnScpYNeo18b1g8bXgr4ErgvmH9XuC6NZlQkjSWsR4UTbIhyZPAUWAf8K/Aa1V1fNjkMHDe2owoSRrHWEGvqv+tqkuAzcBlwLvHvYMkO5McSHLg2LFjJzmmJGk5K3raYlW9BjwCvB84M8nrz5LZDLy8xMfsqaqtVbV1bm5uVcNKkpY2zrNc5pKcOVx+O3A1cJBR2D80bLYDeGCthpQkLW+c56GfC+xNsoHRD4B7q+qhJM8D9yT5Q+AJ4PY1nFOStIxlg15VTwOXLrL+JUbH0yVJM8BT/yWpCYMuSU0YdElqwqBLUhMGXZKaMOiS1IRBl6QmDLokNWHQJakJgy5JTRh0SWrCoEtSEwZdkpow6JLUhEGXpCYMuiQ1YdAlqQmDLklNGHRJasKgS1ITBl2SmjDoktSEQZekJgy6JDVh0CWpCYMuSU0YdElqwqBLUhMGXZKaMOiS1IRBl6QmDLokNWHQJakJgy5JTRh0SWrCoEtSEwZdkpow6JLUxLJBT3J+kkeSPJ/kuSQ3DuvPTrIvyQvD+7PWflxJ0lLG2UM/DtxUVRcDlwMfS3IxsAvYX1UXAvuHZUnSlCwb9Ko6UlWPD5e/CxwEzgO2A3uHzfYC163VkJKk5a3oGHqSeeBS4FFgU1UdGa56Bdg00ckkSSuycdwNk7wT+Bzw8ar6TpL/v66qKkkt8XE7gZ0AW7ZsWd20klqY3/XwRG/v0O5tE7299WqsPfQkpzGK+V1Vdf+w+tUk5w7XnwscXexjq2pPVW2tqq1zc3OTmFmStIhxnuUS4HbgYFV9asFVDwI7hss7gAcmP54kaVzjHHK5Avgo8EySJ4d1nwR2A/cmuQH4GvArazOiJGkcywa9qv4eyBJXXzXZcSRp5TwmP+KZopLUhEGXpCYMuiQ1YdAlqQmDLklNGHRJasKgS1ITBl2SmjDoktSEQZekJgy6JDVh0CWpCYMuSU0YdElqwqBLUhMGXZKaMOiS1IRBl6QmDLokNWHQJakJgy5JTRh0SWrCoEtSEwZdkpow6JLUhEGXpCYMuiQ1YdAlqQmDLklNGHRJasKgS1ITBl2SmjDoktSEQZekJgy6JDVh0CWpCYMuSU0YdElqYtmgJ7kjydEkzy5Yd3aSfUleGN6ftbZjSpKWM84e+p3ANSes2wXsr6oLgf3DsiRpipYNelV9GfjWCau3A3uHy3uB6yY8lyRphU72GPqmqjoyXH4F2DSheSRJJ2nVD4pWVQG11PVJdiY5kOTAsWPHVnt3kqQlnGzQX01yLsDw/uhSG1bVnqraWlVb5+bmTvLuJEnLOdmgPwjsGC7vAB6YzDiSpJM1ztMW7wa+AlyU5HCSG4DdwNVJXgA+MCxLkqZo43IbVNWHl7jqqgnPIklaBc8UlaQmDLokNWHQJakJgy5JTRh0SWrCoEtSEwZdkpow6JLUhEGXpCYMuiQ1YdAlqQmDLklNGHRJasKgS1ITBl2SmjDoktSEQZekJgy6JDVh0CWpCYMuSU0YdElqwqBLUhMGXZKaMOiS1IRBl6QmDLokNWHQJakJgy5JTWyc9gCSNGvmdz080ds7tHvbRG9vKe6hS1ITBl2SmjDoktSEQZekJgy6JDVh0CWpCYMuSU0YdElqwqBLUhOrCnqSa5J8NcmLSXZNaihJ0sqddNCTbAD+DPgl4GLgw0kuntRgkqSVWc0e+mXAi1X1UlV9H7gH2D6ZsSRJK7WaoJ8HfGPB8uFhnSRpCtb81RaT7AR2DovfS/LVk7ypc4BvTmaqNTPrM876fDD7M876fOCMkzDR+XLLqm/ip8fZaDVBfxk4f8Hy5mHdG1TVHmDPKu4HgCQHqmrram9nLc36jLM+H8z+jLM+HzjjJMz6fEtZzSGXfwIuTHJBktOB64EHJzOWJGmlTnoPvaqOJ/kN4O+ADcAdVfXcxCaTJK3Iqo6hV9Xngc9PaJblrPqwzSkw6zPO+nww+zPO+nzgjJMw6/MtKlU17RkkSRPgqf+S1MS6CPosv8RAkvOTPJLk+STPJblx2jMtJcmGJE8keWjas5woyZlJ7kvyz0kOJnn/tGc6UZLfHj7Hzya5O8nbZmCmO5IcTfLsgnVnJ9mX5IXh/VkzNt8fD5/np5P8dZIzpzXfUjMuuO6mJJXknGnMtlIzH/R18BIDx4Gbqupi4HLgYzM230I3AgenPcQSPgP8bVW9G3gvMzZnkvOA3wK2VtV7GD0R4PrpTgXAncA1J6zbBeyvqguB/cPytNzJm+fbB7ynqn4W+Bfg5lM91Anu5M0zkuR84BeBr5/qgU7WzAedGX+Jgao6UlWPD5e/yyhEM3fGbJLNwDbgtmnPcqIkPwb8PHA7QFV9v6pem+5Ui9oIvD3JRuAM4N+nPA9V9WXgWyes3g7sHS7vBa47pUMtsNh8VfWFqjo+LP4Do3NYpmaJ/0OAW4FPAOvmgcb1EPR18xIDSeaBS4FHpzvJoj7N6IvzB9MeZBEXAMeAvxwOCd2W5B3THmqhqnoZ+BNGe2tHgP+oqi9Md6olbaqqI8PlV4BN0xxmGb8O/M20hzhRku3Ay1X11LRnWYn1EPR1Ick7gc8BH6+q70x7noWSXAscrarHpj3LEjYC7wP+vKouBf6T6R4meJPhOPR2Rj98fgp4R5Jfne5Uy6vR09hmcg8zye8xOmR517RnWSjJGcAngd+f9iwrtR6CPtZLDExTktMYxfyuqrp/2vMs4grgg0kOMTpkdWWSv5ruSG9wGDhcVa//ZnMfo8DPkg8A/1ZVx6rqf4D7gZ+b8kxLeTXJuQDD+6NTnudNkvwacC3wkZq9507/DKMf3E8N3zObgceT/ORUpxrDegj6TL/EQJIwOvZ7sKo+Ne15FlNVN1fV5qqaZ/T/98Wqmpm9y6p6BfhGkouGVVcBz09xpMV8Hbg8yRnD5/wqZuyB2wUeBHYMl3cAD0xxljdJcg2jw38frKr/mvY8J6qqZ6rqJ6pqfvieOQy8b/g6nWkzH/ThwZPXX2LgIHDvjL3EwBXARxnt9T45vP3ytIdah34TuCvJ08AlwB9NeZ43GH57uA94HHiG0ffO1M8mTHI38BXgoiSHk9wA7AauTvICo98sds/YfH8K/Ciwb/h++YtpzfdDZlyXPFNUkpqY+T10SdJ4DLokNWHQJakJgy5JTRh0SWrCoEtSEwZdkpow6JLUxP8BS6vk4ctwBFMAAAAASUVORK5CYII=\n"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"Validation set 0 - fully random, so no big match with the training set"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.hist(raw_train_data['time_to_failure'].values[validation_set[0,:]],\n         bins=range(int(np.floor(raw_train_data['time_to_failure'].values.min())),\n                    int(np.ceil(raw_train_data['time_to_failure'].values.max()))+1))","execution_count":28,"outputs":[{"output_type":"execute_result","execution_count":28,"data":{"text/plain":"(array([6., 6., 4., 5., 4., 5., 4., 7., 3., 7., 4., 3., 3., 3., 0.]),\n array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15]),\n <a list of 15 Patch objects>)"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"<Figure size 432x288 with 1 Axes>","image/png":"iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAADPtJREFUeJzt3X+onYV9x/HPZ7lKq5W6LaeuM95dGSVDZBq5lHaOwnSWtBHdH/tDWUu7Fe4/WxeHUOIGg/0zMja6FjY6grUKzSwlNayY1hlaixRa2ySNNia6di6rybSJlM4fg7l0n/1xTsb15px7nhvPc5/nq+8XXHJ/PDl+THLfee5zz0OcRACAOn6u6wEAgLUh3ABQDOEGgGIINwAUQ7gBoBjCDQDFEG4AKIZwA0AxhBsAiplr40E3btyYhYWFNh4aAN6QDh48+EKSQZNjWwn3wsKCDhw40MZDA8Abku1/b3osl0oAoBjCDQDFEG4AKIZwA0AxhBsAipkabtubbR9e9vKi7TvWYxwA4FxTnw6Y5GlJ10qS7Q2STkra2/IuAMAEa71UcqOkf03S+PmGAIDZWmu4b5N0fxtDAADNNL5z0vaFkm6RdNeEjy9JWpKk+fn5mYwDmlrYsW+mj3d857aZPt6s90mz34g61nLG/QFJh5L8eNwHk+xKsphkcTBodLs9AOA8rCXct4vLJADQuUbhtn2xpJskPdDuHADANI2ucSd5RdIvtrwFANAAd04CQDGEGwCKIdwAUAzhBoBiCDcAFEO4AaAYwg0AxRBuACiGcANAMYQbAIoh3ABQDOEGgGIINwAUQ7gBoBjCDQDFEG4AKIZwA0AxhBsAiiHcAFAM4QaAYgg3ABTTKNy2L7W9x/ZTto/Zfm/bwwAA4801PO7Tkh5K8ru2L5R0UYubAACrmBpu22+X9D5JH5WkJK9KerXdWQCASZqccV8p6bSkz9m+RtJBSduTvLL8INtLkpYkaX5+/rwHLezYd94/dz0c37mt6wkA3uSaXOOek3SdpM8k2SLpFUk7Vh6UZFeSxSSLg8FgxjMBAGc1CfcJSSeSPDZ6e4+GIQcAdGBquJM8L+lZ25tH77pR0tFWVwEAJmr6rJKPS9o9ekbJM5J+v71JAIDVNAp3ksOSFlveAgBogDsnAaAYwg0AxRBuACiGcANAMYQbAIoh3ABQDOEGgGIINwAUQ7gBoBjCDQDFEG4AKIZwA0AxhBsAiiHcAFAM4QaAYgg3ABRDuAGgGMINAMUQbgAohnADQDGN/rFg28clvSTpZ5LOJOEfDgaAjjQK98hvJXmhtSUAgEa4VAIAxTQNdyQ9bPug7aU2BwEAVtf0UslvJjlp+x2S9tt+Ksmjyw8YBX1Jkubn52c8841tYce+mT7e8Z3bZvp4s94nzX4j8GbS6Iw7ycnRj6ck7ZX07jHH7EqymGRxMBjMdiUA4P9NDbfti21fcvZ1Se+XdKTtYQCA8ZpcKrlM0l7bZ4//xyQPtboKADDR1HAneUbSNeuwBQDQAE8HBIBiCDcAFEO4AaAYwg0AxRBuACiGcANAMYQbAIoh3ABQDOEGgGIINwAUQ7gBoBjCDQDFEG4AKIZwA0AxhBsAiiHcAFAM4QaAYgg3ABRDuAGgGMINAMUQbgAopnG4bW+w/T3bD7Y5CACwurWccW+XdKytIQCAZhqF2/YmSdsk3d3uHADANHMNj/uUpE9IumTSAbaXJC1J0vz8/Otf1lMLO/Z1PWEqNr45zPrX8PjObTN9PLRn6hm37ZslnUpycLXjkuxKsphkcTAYzGwgAOC1mlwquV7SLbaPS/qCpBtsf77VVQCAiaaGO8ldSTYlWZB0m6SvJ/lQ68sAAGPxPG4AKKbpNyclSUm+IekbrSwBADTCGTcAFEO4AaAYwg0AxRBuACiGcANAMYQbAIoh3ABQDOEGgGIINwAUQ7gBoBjCDQDFEG4AKIZwA0AxhBsAiiHcAFAM4QaAYgg3ABRDuAGgGMINAMUQbgAoZmq4bb/F9ndsP277Sdt/sR7DAADjNflX3v9b0g1JXrZ9gaRv2v5qkm+3vA0AMMbUcCeJpJdHb14wekmbowAAkzW6xm17g+3Dkk5J2p/ksXZnAQAmaXKpREl+Jula25dK2mv76iRHlh9je0nSkiTNz8/PfCiwnhZ27Ot6wrp7M/4/H9+5resJ52VNzypJ8lNJj0jaOuZju5IsJlkcDAaz2gcAWKHJs0oGozNt2X6rpJskPdX2MADAeE0ulbxT0n22N2gY+i8mebDdWQCASZo8q+QJSVvWYQsAoAHunASAYgg3ABRDuAGgGMINAMUQbgAohnADQDGEGwCKIdwAUAzhBoBiCDcAFEO4AaAYwg0AxRBuACiGcANAMYQbAIoh3ABQDOEGgGIINwAUQ7gBoBjCDQDFEG4AKGZquG1fYfsR20dtP2l7+3oMAwCMN9fgmDOS7kxyyPYlkg7a3p/kaMvbAABjTD3jTvJckkOj11+SdEzS5W0PAwCMt6Zr3LYXJG2R9FgbYwAA0zUOt+23SfqSpDuSvDjm40u2D9g+cPr06VluBAAs0yjcti/QMNq7kzww7pgku5IsJlkcDAaz3AgAWKbJs0os6bOSjiX5ZPuTAACraXLGfb2kD0u6wfbh0csHW94FAJhg6tMBk3xTktdhCwCgAe6cBIBiCDcAFEO4AaAYwg0AxRBuACiGcANAMYQbAIoh3ABQDOEGgGIINwAUQ7gBoBjCDQDFEG4AKIZwA0AxhBsAiiHcAFAM4QaAYgg3ABRDuAGgGMINAMUQbgAoZmq4bd9j+5TtI+sxCACwuiZn3PdK2tryDgBAQ1PDneRRST9Zhy0AgAZmdo3b9pLtA7YPnD59elYPCwBYYWbhTrIryWKSxcFgMKuHBQCswLNKAKAYwg0AxTR5OuD9kr4labPtE7Y/1v4sAMAkc9MOSHL7egwBADTDpRIAKIZwA0AxhBsAiiHcAFAM4QaAYgg3ABRDuAGgGMINAMUQbgAohnADQDGEGwCKIdwAUAzhBoBiCDcAFEO4AaAYwg0AxRBuACiGcANAMYQbAIoh3ABQTKNw295q+2nbP7S9o+1RAIDJpobb9gZJfy/pA5KuknS77avaHgYAGK/JGfe7Jf0wyTNJXpX0BUm3tjsLADBJk3BfLunZZW+fGL0PANCBuVk9kO0lSUujN1+2/fR5PtRGSS/MZlUr+r5PYuMs9H2f1P+Nfd8n/1WvNv5K0wObhPukpCuWvb1p9L7XSLJL0q6m/+FJbB9Isvh6H6ctfd8nsXEW+r5P6v/Gvu+Tamwcp8mlku9KepftK21fKOk2SV9udxYAYJKpZ9xJztj+I0n/LGmDpHuSPNn6MgDAWI2ucSf5iqSvtLzlrNd9uaVlfd8nsXEW+r5P6v/Gvu+Tamw8h5N0vQEAsAbc8g4AxfQm3H2/rd72FbYfsX3U9pO2t3e9aRzbG2x/z/aDXW8Zx/altvfYfsr2Mdvv7XrTSrb/ZPR7fMT2/bbf0oNN99g+ZfvIsvf9gu39tn8w+vHne7bvr0e/z0/Y3mv70q72Tdq47GN32o7tjV1sW6tehLvIbfVnJN2Z5CpJ75H0hz3cKEnbJR3resQqPi3poSS/Juka9Wyr7csl/bGkxSRXa/gN+du6XSVJulfS1hXv2yHpa0neJelro7e7cq/O3bdf0tVJfl3Sv0i6a71HrXCvzt0o21dIer+kH633oPPVi3CrwG31SZ5Lcmj0+ksaBqdXd5Da3iRpm6S7u94yju23S3qfpM9KUpJXk/y021VjzUl6q+05SRdJ+o+O9yjJo5J+suLdt0q6b/T6fZJ+Z11HLTNuX5KHk5wZvfltDe8B6cyEX0NJ+ltJn5BU5ht+fQl3qdvqbS9I2iLpsW6XnONTGv4B/N+uh0xwpaTTkj43upxzt+2Lux61XJKTkv5Gw7Ov5yT9Z5KHu1010WVJnhu9/ryky7ocM8UfSPpq1yNWsn2rpJNJHu96y1r0Jdxl2H6bpC9JuiPJi13vOcv2zZJOJTnY9ZZVzEm6TtJnkmyR9Iq6/fL+HKPrxLdq+JfML0u62PaHul01XYZPD+vlGaPtP9PwUuPurrcsZ/siSX8q6c+73rJWfQl3o9vqu2b7Ag2jvTvJA13vWeF6SbfYPq7hpaYbbH++20nnOCHpRJKzX6ns0TDkffLbkv4tyekk/yPpAUm/0fGmSX5s+52SNPrxVMd7zmH7o5JulvR76d9zj39Vw7+gHx993mySdMj2L3W6qoG+hLv3t9XbtobXZo8l+WTXe1ZKcleSTUkWNPz1+3qSXp0pJnle0rO2N4/edaOkox1OGudHkt5j+6LR7/mN6tk3UJf5sqSPjF7/iKR/6nDLOWxv1fDS3S1J/qvrPSsl+X6SdyRZGH3enJB03ejPaa/1Ityjb2Ccva3+mKQv9vC2+uslfVjDM9nDo5cPdj2qoI9L2m37CUnXSvrLjve8xuirgT2SDkn6voafI53fXWf7fknfkrTZ9gnbH5O0U9JNtn+g4VcKO3u27+8kXSJp/+jz5R+62rfKxpK4cxIAiunFGTcAoDnCDQDFEG4AKIZwA0AxhBsAiiHcAFAM4QaAYgg3ABTzf/2gEAq/lN3UAAAAAElFTkSuQmCC\n"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"Validation set 1 - approximation of the training set distribution with 64 samples"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.hist(raw_train_data['time_to_failure'].values[validation_set[1,:]],\n         bins=range(int(np.floor(raw_train_data['time_to_failure'].values.min())),\n                    int(np.ceil(raw_train_data['time_to_failure'].values.max()))+1))","execution_count":29,"outputs":[{"output_type":"execute_result","execution_count":29,"data":{"text/plain":"(array([5., 6., 5., 5., 5., 5., 5., 5., 5., 5., 5., 4., 2., 2., 0.]),\n array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15]),\n <a list of 15 Patch objects>)"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"<Figure size 432x288 with 1 Axes>","image/png":"iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAADGNJREFUeJzt22+IZYV9xvHn6Y4h0Uhs2Vubuk5HSrCINFEGSWqRVpuwyYr2RV8YmpC0gXmTWlOEsLa00DdlS0saoaXtYIxCtoawURo0sS6JIQSSbXfXP9Fd0wS7jWu1OxLSaAu1mzx9cc+ScfbeuWc298w5v/D9wLL3zh6vD7s73z1z5h4nEQCgjp/qewAAYGsINwAUQ7gBoBjCDQDFEG4AKIZwA0AxhBsAiiHcAFAM4QaAYha6eNGdO3dmaWmpi5cGgJ9IR44ceSnJqM2xnYR7aWlJhw8f7uKlAeAnku1/b3ssl0oAoBjCDQDFEG4AKIZwA0AxhBsAimkVbtsX2T5g+xnbx22/o+thAIDJ2r4d8E5JDyf5Lduvk3R+h5sAAJuYGW7bb5J0naQPSlKSVyW92u0sAMA0bS6VXCZpTdInbT9m+y7bF3S8CwAwRZtLJQuSrpZ0a5JDtu+UtFfSH68/yPaKpBVJWlxcnPfOc7a096G5vt6JfXvm+noAsFVtzrhPSjqZ5FDz/IDGIX+NJKtJlpMsj0atbrcHAJyDmeFO8qKk52xf3nzoBknHOl0FAJiq7btKbpW0v3lHybOSfqe7SQCAzbQKd5LHJS13vAUA0AJ3TgJAMYQbAIoh3ABQDOEGgGIINwAUQ7gBoBjCDQDFEG4AKIZwA0AxhBsAiiHcAFAM4QaAYgg3ABRDuAGgGMINAMUQbgAohnADQDGEGwCKIdwAUAzhBoBiCDcAFEO4AaAYwg0AxRBuACiGcANAMYQbAIpZaHOQ7ROSXpb0A0mnkyx3OQoAMF2rcDd+PclLnS0BALTCpRIAKKbtGXckPWI7kv4+yerGA2yvSFqRpMXFxXMetLT3oXP+b7fD0PcB5+rEvj19T0BLbc+4fzXJ1ZLeLenDtq/beECS1STLSZZHo9FcRwIAfqRVuJM83/x8StIDkq7pchQAYLqZ4bZ9ge0LzzyW9C5JT3U9DAAwWZtr3BdLesD2meP/IcnDna4CAEw1M9xJnpX01m3YAgBogbcDAkAxhBsAiiHcAFAM4QaAYgg3ABRDuAGgGMINAMUQbgAohnADQDGEGwCKIdwAUAzhBoBiCDcAFEO4AaAYwg0AxRBuACiGcANAMYQbAIoh3ABQDOEGgGIINwAUQ7gBoBjCDQDFEG4AKIZwA0AxrcNte4ftx2w/2OUgAMDmtnLGfZuk410NAQC00yrctndJ2iPprm7nAABmaXvG/XFJH5X0ww63AABaWJh1gO0bJZ1KcsT2r21y3IqkFUlaXFyc20AA22Np70Nzfb0T+/bM9fXwI23OuK+VdJPtE5I+Lel625/aeFCS1STLSZZHo9GcZwIAzpgZ7iR3JNmVZEnSLZK+lOR9nS8DAEzE+7gBoJiZ17jXS/JlSV/uZAkAoBXOuAGgGMINAMUQbgAohnADQDGEGwCKIdwAUAzhBoBiCDcAFEO4AaAYwg0AxRBuACiGcANAMYQbAIoh3ABQDOEGgGIINwAUQ7gBoBjCDQDFEG4AKIZwA0AxhBsAiiHcAFAM4QaAYgg3ABRDuAGgGMINAMXMDLft19v+Z9tP2H7a9p9uxzAAwGQLLY75X0nXJ3nF9nmSvmr7C0m+3vE2AMAEM8OdJJJeaZ6e1/xIl6MAANO1usZte4ftxyWdknQwyaEJx6zYPmz78Nra2rx3AgAarcKd5AdJ3iZpl6RrbF854ZjVJMtJlkej0bx3AgAaW3pXSZLvSXpU0u5u5gAAZmnzrpKR7Yuax2+Q9E5Jz3Q9DAAwWZt3lbxZ0r22d2gc+s8kebDbWQCAadq8q+RJSVdtwxYAQAvcOQkAxRBuACiGcANAMYQbAIoh3ABQDOEGgGIINwAUQ7gBoBjCDQDFEG4AKIZwA0AxhBsAiiHcAFAM4QaAYgg3ABRDuAGgGMINAMUQbgAohnADQDGEGwCKIdwAUAzhBoBiCDcAFEO4AaAYwg0AxcwMt+1LbT9q+5jtp23fth3DAACTLbQ45rSk25MctX2hpCO2DyY51vE2AMAEM8+4k7yQ5Gjz+GVJxyVd0vUwAMBkW7rGbXtJ0lWSDnUxBgAwW5tLJZIk22+U9FlJH0ny/Qm/viJpRZIWFxfnNhBATUt7H+p7wkwn9u3pe8I5aXXGbfs8jaO9P8n9k45JsppkOcnyaDSa50YAwDpt3lViSZ+QdDzJx7qfBADYTJsz7mslvV/S9bYfb368p+NdAIApZl7jTvJVSd6GLQCAFrhzEgCKIdwAUAzhBoBiCDcAFEO4AaAYwg0AxRBuACiGcANAMYQbAIoh3ABQDOEGgGIINwAUQ7gBoBjCDQDFEG4AKIZwA0AxhBsAiiHcAFAM4QaAYgg3ABRDuAGgGMINAMUQbgAohnADQDGEGwCKIdwAUMzMcNu+2/Yp209txyAAwObanHHfI2l3xzsAAC3NDHeSr0j67jZsAQC0MLdr3LZXbB+2fXhtbW1eLwsA2GBu4U6ymmQ5yfJoNJrXywIANuBdJQBQDOEGgGLavB3wPklfk3S57ZO2P9T9LADANAuzDkjy3u0YAgBoh0slAFAM4QaAYgg3ABRDuAGgGMINAMUQbgAohnADQDGEGwCKIdwAUAzhBoBiCDcAFEO4AaAYwg0AxRBuACiGcANAMYQbAIoh3ABQDOEGgGIINwAUQ7gBoBjCDQDFEG4AKIZwA0AxhBsAiiHcAFBMq3Db3m37m7a/bXtv16MAANPNDLftHZL+RtK7JV0h6b22r+h6GABgsjZn3NdI+naSZ5O8KunTkm7udhYAYJo24b5E0nPrnp9sPgYA6MHCvF7I9oqklebpK7a/eY4vtVPSS/NZ1Ymh75PYOA9D3ycNf+PQ98l/PqiNv9D2wDbhfl7Speue72o+9hpJViWttv0fT2P7cJLlH/d1ujL0fRIb52Ho+6Thbxz6PqnGxknaXCr5F0lvsX2Z7ddJukXS57qdBQCYZuYZd5LTtn9P0j9J2iHp7iRPd74MADBRq2vcST4v6fMdbznjx77c0rGh75PYOA9D3ycNf+PQ90k1Np7FSfreAADYAm55B4BiBhPuod9Wb/tS24/aPmb7adu39b1pEts7bD9m+8G+t0xi+yLbB2w/Y/u47Xf0vWkj23/Q/Bk/Zfs+268fwKa7bZ+y/dS6j/2M7YO2v9X8/NMD2/cXzZ/zk7YfsH1RX/umbVz3a7fbju2dfWzbqkGEu8ht9acl3Z7kCklvl/ThAW6UpNskHe97xCbulPRwkl+S9FYNbKvtSyT9vqTlJFdq/A35W/pdJUm6R9LuDR/bK+mLSd4i6YvN877co7P3HZR0ZZJflvSvku7Y7lEb3KOzN8r2pZLeJek72z3oXA0i3CpwW32SF5IcbR6/rHFwBnUHqe1dkvZIuqvvLZPYfpOk6yR9QpKSvJrke/2ummhB0htsL0g6X9J/9LxHSb4i6bsbPnyzpHubx/dK+s1tHbXOpH1JHklyunn6dY3vAenNlN9DSforSR+VVOYbfkMJd6nb6m0vSbpK0qF+l5zl4xr/Bfxh30OmuEzSmqRPNpdz7rJ9Qd+j1kvyvKS/1Pjs6wVJ/5XkkX5XTXVxkheaxy9KurjPMTP8rqQv9D1iI9s3S3o+yRN9b9mKoYS7DNtvlPRZSR9J8v2+95xh+0ZJp5Ic6XvLJhYkXS3pb5NcJem/1e+X92dprhPfrPE/Mj8v6QLb7+t31WwZvz1skGeMtv9I40uN+/vesp7t8yX9oaQ/6XvLVg0l3K1uq++b7fM0jvb+JPf3vWeDayXdZPuExpearrf9qX4nneWkpJNJznylckDjkA/Jb0j6tyRrSf5P0v2SfqXnTdP8p+03S1Lz86me95zF9gcl3SjptzO89x7/osb/QD/RfN7sknTU9s/1uqqFoYR78LfV27bG12aPJ/lY33s2SnJHkl1JljT+/ftSkkGdKSZ5UdJzti9vPnSDpGM9TprkO5Lebvv85s/8Bg3sG6jrfE7SB5rHH5D0jz1uOYvt3Rpfurspyf/0vWejJN9I8rNJlprPm5OSrm7+ng7aIMLdfAPjzG31xyV9ZoC31V8r6f0an8k+3vx4T9+jCrpV0n7bT0p6m6Q/63nPazRfDRyQdFTSNzT+HOn97jrb90n6mqTLbZ+0/SFJ+yS90/a3NP5KYd/A9v21pAslHWw+X/6ur32bbCyJOycBoJhBnHEDANoj3ABQDOEGgGIINwAUQ7gBoBjCDQDFEG4AKIZwA0Ax/w8j7NeY4ZizpAAAAABJRU5ErkJggg==\n"},"metadata":{}}]},{"metadata":{},"cell_type":"markdown","source":"Validation set 2 - Another approximation of the training set distribution with 64 samples, independent from Validation set 1"},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.hist(raw_train_data['time_to_failure'].values[validation_set[2,:]],\n         bins=range(int(np.floor(raw_train_data['time_to_failure'].values.min())),\n                    int(np.ceil(raw_train_data['time_to_failure'].values.max()))+1))","execution_count":30,"outputs":[{"output_type":"execute_result","execution_count":30,"data":{"text/plain":"(array([5., 6., 5., 5., 5., 5., 5., 5., 5., 5., 5., 4., 2., 1., 1.]),\n array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15]),\n <a list of 15 Patch objects>)"},"metadata":{}},{"output_type":"display_data","data":{"text/plain":"<Figure size 432x288 with 1 Axes>","image/png":"iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAADHRJREFUeJzt23+oXgd9x/HPZ7kVbS12I8861/TuliEdpUxbLkXXUbZ2SjSl9Y/9UZmim3D/cV0dBUk3Ntg/I2PDWdjYdqm1BbOKxJZJo12DVkTQbEn6wzapU7rMpmuXFHG2G6yLfvbHczJvb57nPuemz7nnfOX9gpB7b05OPiS575yc+xwnEQCgjp/qewAAYHMINwAUQ7gBoBjCDQDFEG4AKIZwA0AxhBsAiiHcAFAM4QaAYha6OOn27duztLTUxakB4CfS4cOHX0wyanNsJ+FeWlrSoUOHujg1APxEsv1vbY/lVgkAFEO4AaAYwg0AxRBuACiGcANAMa3Cbfsi2/tsP237mO13dD0MADBZ25cD3inpoSS/aft1ks7vcBMAYAMzw237TZKuk/QhSUryiqRXup0FAJimza2SyySdkvQp24/avsv2BR3vAgBM0eZWyYKkqyXdmuSg7Tsl7Zb0R2sPsr0iaUWSFhcX573znC3t3j/X8x3fs2uu5wOAzWpzxX1C0okkB5v392kc8ldJsppkOcnyaNTqcXsAwDmYGe4kL0h61vblzYdukHS001UAgKnavqrkVkl7m1eUPCPpt7ubBADYSKtwJ3lM0nLHWwAALfDkJAAUQ7gBoBjCDQDFEG4AKIZwA0AxhBsAiiHcAFAM4QaAYgg3ABRDuAGgGMINAMUQbgAohnADQDGEGwCKIdwAUAzhBoBiCDcAFEO4AaAYwg0AxRBuACiGcANAMYQbAIoh3ABQDOEGgGIINwAUQ7gBoJiFNgfZPi7pJUk/lHQ6yXKXowAA07UKd+PXk7zY2RIAQCvcKgGAYtpecUfSw7Yj6e+SrK4/wPaKpBVJWlxcPOdBS7v3n/PP3QpD3wecq+N7dvU9AS21veL+1SRXS3q3pI/Yvm79AUlWkywnWR6NRnMdCQD4sVbhTvJc8/1JSQ9IuqbLUQCA6WaG2/YFti8887akd0l6suthAIDJ2tzjvljSA7bPHP/3SR7qdBUAYKqZ4U7yjKS3bsEWAEALvBwQAIoh3ABQDOEGgGIINwAUQ7gBoBjCDQDFEG4AKIZwA0AxhBsAiiHcAFAM4QaAYgg3ABRDuAGgGMINAMUQbgAohnADQDGEGwCKIdwAUAzhBoBiCDcAFEO4AaAYwg0AxRBuACiGcANAMYQbAIppHW7b22w/avvBLgcBADa2mSvu2yQd62oIAKCdVuG2vUPSLkl3dTsHADBL2yvuT0j6mKQfdbgFANDCwqwDbN8o6WSSw7Z/bYPjViStSNLi4uLcBgLYGku798/1fMf37Jrr+fBjba64r5V0k+3jkj4j6Xrbn15/UJLVJMtJlkej0ZxnAgDOmBnuJHck2ZFkSdItkr6c5P2dLwMATMTruAGgmJn3uNdK8hVJX+lkCQCgFa64AaAYwg0AxRBuACiGcANAMYQbAIoh3ABQDOEGgGIINwAUQ7gBoBjCDQDFEG4AKIZwA0AxhBsAiiHcAFAM4QaAYgg3ABRDuAGgGMINAMUQbgAohnADQDGEGwCKIdwAUAzhBoBiCDcAFEO4AaAYwg0AxcwMt+3X2/4n24/bfsr2n2zFMADAZAstjvkfSdcnedn2eZK+ZvuLSb7R8TYAwAQzw50kkl5u3j2v+ZYuRwEApmt1j9v2NtuPSTop6UCSgxOOWbF9yPahU6dOzXsnAKDRKtxJfpjkbZJ2SLrG9pUTjllNspxkeTQazXsnAKCxqVeVJPm+pEck7exmDgBgljavKhnZvqh5+w2S3inp6a6HAQAma/OqkjdLutf2No1D/9kkD3Y7CwAwTZtXlTwh6aot2AIAaIEnJwGgGMINAMUQbgAohnADQDGEGwCKIdwAUAzhBoBiCDcAFEO4AaAYwg0AxRBuACiGcANAMYQbAIoh3ABQDOEGgGIINwAUQ7gBoBjCDQDFEG4AKIZwA0AxhBsAiiHcAFAM4QaAYgg3ABRDuAGgmJnhtn2p7UdsH7X9lO3btmIYAGCyhRbHnJZ0e5Ijti+UdNj2gSRHO94GAJhg5hV3kueTHGnefknSMUmXdD0MADDZpu5x216SdJWkg12MAQDM1uZWiSTJ9hslfU7SR5P8YMKPr0hakaTFxcW5DQRQ09Lu/XM/5/E9u+Z+zopaXXHbPk/jaO9Ncv+kY5KsJllOsjwajea5EQCwRptXlVjSJyUdS/Lx7icBADbS5or7WkkfkHS97ceab+/peBcAYIqZ97iTfE2St2ALAKAFnpwEgGIINwAUQ7gBoBjCDQDFEG4AKIZwA0AxhBsAiiHcAFAM4QaAYgg3ABRDuAGgGMINAMUQbgAohnADQDGEGwCKIdwAUAzhBoBiCDcAFEO4AaAYwg0AxRBuACiGcANAMYQbAIoh3ABQDOEGgGIINwAUMzPctu+2fdL2k1sxCACwsTZX3PdI2tnxDgBASzPDneSrkr63BVsAAC0szOtEtlckrUjS4uLivE4LAP9vaff+vids6PieXVvy68zti5NJVpMsJ1kejUbzOi0AYB1eVQIAxRBuACimzcsB75P0dUmX2z5h+8PdzwIATDPzi5NJ3rcVQwAA7XCrBACKIdwAUAzhBoBiCDcAFEO4AaAYwg0AxRBuACiGcANAMYQbAIoh3ABQDOEGgGIINwAUQ7gBoBjCDQDFEG4AKIZwA0AxhBsAiiHcAFAM4QaAYgg3ABRDuAGgGMINAMUQbgAohnADQDGEGwCKaRVu2zttf8v2d2zv7noUAGC6meG2vU3SX0t6t6QrJL3P9hVdDwMATNbmivsaSd9J8kySVyR9RtLN3c4CAEzTJtyXSHp2zfsnmo8BAHqwMK8T2V6RtNK8+7Ltb53jqbZLenE+qzox9H0SG+dh6Puk4W8c+j5pzhv9Z6/pp/9C2wPbhPs5SZeueX9H87FXSbIqabXtLzyN7UNJll/reboy9H0SG+dh6Puk4W8c+j6pxsZJ2twq+WdJb7F9me3XSbpF0ue7nQUAmGbmFXeS07Z/V9I/Stom6e4kT3W+DAAwUat73Em+IOkLHW854zXfbunY0PdJbJyHoe+Thr9x6PukGhvP4iR9bwAAbAKPvANAMYMJ99Afq7d9qe1HbB+1/ZTt2/reNIntbbYftf1g31smsX2R7X22n7Z9zPY7+t60nu3fb/6Mn7R9n+3XD2DT3bZP2n5yzcd+xvYB299uvv/pge378+bP+QnbD9i+qK990zau+bHbbcf29j62bdYgwl3ksfrTkm5PcoWkt0v6yAA3StJtko71PWIDd0p6KMkvSXqrBrbV9iWSfk/ScpIrNf6C/C39rpIk3SNp57qP7Zb0pSRvkfSl5v2+3KOz9x2QdGWSX5b0L5Lu2OpR69yjszfK9qWS3iXpu1s96FwNItwq8Fh9kueTHGnefknj4AzqCVLbOyTtknRX31smsf0mSddJ+qQkJXklyff7XTXRgqQ32F6QdL6kf+95j5J8VdL31n34Zkn3Nm/fK+m9WzpqjUn7kjyc5HTz7jc0fgakN1N+DyXpLyV9TFKZL/gNJdylHqu3vSTpKkkH+11ylk9o/BfwR30PmeIySackfaq5nXOX7Qv6HrVWkuck/YXGV1/PS/rPJA/3u2qqi5M837z9gqSL+xwzw+9I+mLfI9azfbOk55I83veWzRhKuMuw/UZJn5P00SQ/6HvPGbZvlHQyyeG+t2xgQdLVkv4myVWS/kv9/vf+LM194ps1/kfm5yVdYPv9/a6aLeOXhw3yitH2H2p8q3Fv31vWsn2+pD+Q9Md9b9msoYS71WP1fbN9nsbR3pvk/r73rHOtpJtsH9f4VtP1tj/d76SznJB0IsmZ/6ns0zjkQ/Ibkv41yakk/yvpfkm/0vOmaf7D9pslqfn+ZM97zmL7Q5JulPRbGd5rj39R43+gH28+b3ZIOmL753pd1cJQwj34x+ptW+N7s8eSfLzvPesluSPJjiRLGv/+fTnJoK4Uk7wg6VnblzcfukHS0R4nTfJdSW+3fX7zZ36DBvYF1DU+L+mDzdsflPQPPW45i+2dGt+6uynJf/e9Z70k30zys0mWms+bE5Kubv6eDtogwt18AePMY/XHJH12gI/VXyvpAxpfyT7WfHtP36MKulXSXttPSHqbpD/tec+rNP8b2CfpiKRvavw50vvTdbbvk/R1SZfbPmH7w5L2SHqn7W9r/D+FPQPb91eSLpR0oPl8+du+9m2wsSSenASAYgZxxQ0AaI9wA0AxhBsAiiHcAFAM4QaAYgg3ABRDuAGgGMINAMX8H3A81vuXTy9VAAAAAElFTkSuQmCC\n"},"metadata":{}}]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}