{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"749312244db242e56a29c4724d1b3c71ebf45a2f"},"cell_type":"code","source":"import matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"%%time\ntrain = pd.read_csv('../input/train.csv', dtype={'acoustic_data': np.int16, 'time_to_failure': np.float32})","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e578092882ed7dacb19ee1e4a5a77cbfe018d75d"},"cell_type":"code","source":"#this one is high def data\ndf = pd.read_csv(\"../input/train.csv\", nrows=10000000)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"427cc7ef4c7f64c5e328ba60b938043dbd7d1f03"},"cell_type":"code","source":"train.rename({\"acoustic_data\": \"acd\", \"time_to_failure\": \"ttf\"}, axis=\"columns\", inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"54f57979e3f01c105c1586cf7bf0c8a621081a9b"},"cell_type":"markdown","source":"Let's take a look at the general distribution of the data points, dropping 49 out of every 50 data points, \nwe can take a more general look (since my kernel keeps crashing on the complete dataset :') ). \n\nThe quake takes place when the ttf (time to failure, green line) reaches 0. We can see these points correspond closely with the higher spikes in the acoustic data."},{"metadata":{"trusted":true,"_uuid":"e3d521e2b70f7ec04cd3027270bf542b25b81c7c"},"cell_type":"code","source":"acd_small = train['acd'].values[::50]\nttf_small = train['ttf'].values[::50]\n\nfig, ax1 = plt.subplots(figsize=(20, 8))\n\nplt.plot(acd_small, color='pink')\nax2 = ax1.twinx()\nplt.plot(ttf_small, color='g')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7ca82fa18adfd4927b7e0688ee39c6f25f2272ac"},"cell_type":"markdown","source":"Let's take a closer look and see how the ttf reaches zero and how this event corresponds with the acoustic data."},{"metadata":{"trusted":true,"_uuid":"5f6489069158ddc6ed663d076649586d7c1622cd"},"cell_type":"code","source":"acd_small = train['acd'].values[::50]\nttf_small = train['ttf'].values[::50]\n\nfig, ax1 = plt.subplots(figsize=(20, 8))\n\nplt.plot(acd_small[:300000], color='pink')\nax2 = ax1.twinx()\nplt.plot(ttf_small[:300000], color='g')\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"55fcc37cbaf54666f0225f9a8048286b6a43ac8e"},"cell_type":"markdown","source":"As we can see the acoustic data peak is not where the earthquake occurs. The reason is that the acoustic** data of the moment of the earthquake is (apparently) erased from the data**, as I understand.\nTo check if this holds, lets see if there are 0 ttfs in the dataset:\n\n(Just to make sure let's check our df, which is the 1/60th of the data but ttf in float64. This range covers the first quake, but there is still no 0 val in ttf of the earthquake region. Meaning the data for the earthquake is deleted)"},{"metadata":{"trusted":true,"_uuid":"56ff2d2546772743f51b49e68e73f32b5840880c"},"cell_type":"code","source":"smallest_tvals=train[train[\"ttf\"]==0]\nprint('t: ', (smallest_tvals.shape))\n\nsmallest_dvals=df[df[\"time_to_failure\"]==0]\nprint('d: ', (smallest_dvals.shape))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e6ad900ed91fc2a2bbfab0c63757e74ee8557c77"},"cell_type":"markdown","source":"Number of tffs equal to 0 is 0. There is no such data. \n\nLet's look at the linearly decreasing ttf value closely:"},{"metadata":{"trusted":true,"_uuid":"968c4dbad9833d9a773438fa97169b8fa60e0198"},"cell_type":"code","source":"acd_small = train['acd'].values[::50]\nttf_small = train['ttf'].values[::50]\n\nfig, ax1 = plt.subplots(figsize=(20, 8))\n\nplt.plot(acd_small[:3000], color='pink')\nax2 = ax1.twinx()\nplt.plot(ttf_small[:3000], color='g')\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a96c86f90c8d15e10079315964ca8e4a84f989f3"},"cell_type":"markdown","source":"Closer look:"},{"metadata":{"trusted":true,"_uuid":"4c551919dfcdfdde30dea3fbcaf2d846fbe28abc"},"cell_type":"code","source":"acd_small = train['acd'].values[::50]\nttf_small = train['ttf'].values[::50]\n\nfig, ax1 = plt.subplots(figsize=(20, 8))\n\nplt.plot(acd_small[:300], color='pink')\nax2 = ax1.twinx()\nplt.plot(ttf_small[:300], color='g')\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fa4f2535d055c81f251f76cefd0c26eb64a41eab"},"cell_type":"markdown","source":"We can see it is not linear but actually taking steps, this is caused by the recording device.\n[At the additional info given by Bertrand LD](https://www.kaggle.com/c/LANL-Earthquake-Prediction/discussion/77526) it is said that between every recorded sequence there is a gap much longer than the recording. These gaps correspond to the jumps we observe in the data. If it weren't for these gaps, the ttf data would be linear. See fig: (the jump is on i=4096)"},{"metadata":{"trusted":true,"_uuid":"a0c376d529fbe50df2240fe15728d78a64978905"},"cell_type":"code","source":"df.head(4095).plot(y=\"time_to_failure\",title=\"BEFORE JUMP SEGMENT VALS, 4095 DATA PTS\", color = \"green\")\ndf.head(4096).tail(2).plot(y=\"time_to_failure\",title=\"AFTER JUMP SEGMENT VALS, 2 DATA PTS\")\nprint(df.head(1)[\"time_to_failure\"])\nprint(df.head(4096).tail(3)[\"time_to_failure\"])\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"235461ab590d2482cb117876183ed78adfabce60"},"cell_type":"markdown","source":"From the first figure we see that in one segment before jump (on the first 4095 indices [0-4094] (noting that this is not always the case since the segment length may also be 4096)), the rate at which the ttf decreases is approximately 0.000005/4095 = , while the jump is 0.000905, approximately 75420 times the normal rate. **The \"silent\" period is thus 75420/4095 == 18,x segments. So we do not truly have a continuous data.**"},{"metadata":{"trusted":true,"_uuid":"25e42a97aeb84d32b563432b3fb6c38dd7f98bef"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"989cfe9c908f4f63c72cccd11c27a85d28ccfffb"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"088caf3afde8e5f6abcc7231f9370112f9ddbd79"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3dbdda8145863c038794d4dd15d669c1a86d7529"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"22fa60d8934a20a839a637a710e05b9e4c639e74"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}