{"cells":[{"metadata":{"_uuid":"63108ed7f840eba7d84f1fb5e6774a66316c1b4c"},"cell_type":"markdown","source":"# Fast Fourier Transform & Denoising\nIn this kernel, I briefly introduces two ways to perform denoising :\n- Using an averaging  smoothing technique\n- Using the FFT\n\n##### Enjoy!"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport seaborn as sns\nfrom numpy.fft import *\nimport pyarrow.parquet as pq\nimport matplotlib.pyplot as plt\n\nsns.set_style(\"whitegrid\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2791aa2e427c195b5434f37ed425bc8be5775767"},"cell_type":"markdown","source":"## 1 - Loading Data"},{"metadata":{"_uuid":"f9db7c1298259905c3a6276d2b09a76625049835"},"cell_type":"markdown","source":"### Signals"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"signals = pq.read_table('../input/train.parquet', columns=[str(i) for i in range(999)]).to_pandas()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1255c51404246d23465d50575049db937fd7c6f8"},"cell_type":"code","source":"signals = np.array(signals).T.reshape((999//3, 3, 800000))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"832d8fb176ea537856fd6eae5bfb3425874a1fbd"},"cell_type":"code","source":"plt.figure(figsize=(15, 10))\nplt.plot(signals[0, 0, :], label='Phase 0')\nplt.plot(signals[0, 1, :], label='Phase 1')\nplt.plot(signals[0, 2, :], label='Phase 2')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6fb88dc514013b35e5915a5e520aab7ef07babaa"},"cell_type":"markdown","source":"### Target"},{"metadata":{"trusted":true,"_uuid":"2221c7c5dd351386cbaa8162e622e535f64c3a1b"},"cell_type":"code","source":"train_df = pd.read_csv('../input/metadata_train.csv')\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"013f3ab108163f9272295192892ed44e7ec71e11"},"cell_type":"code","source":"target = train_df['target'][::3]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"226e535bbaf27f2f663c09fedddd555689ff11a8","scrolled":false},"cell_type":"code","source":"plt.figure(figsize=(15, 10))\nsns.countplot(target)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b4f641105de1800b26081e6b0fc0dd93d912e452"},"cell_type":"markdown","source":"## 2 - Smoothing by mean\nThe idea is to reduce the length and the noise of the signal by merging $k$ neighbour values into their average."},{"metadata":{"trusted":true,"_uuid":"1f48858b8e21bba91768295c1382c1717601a6b2"},"cell_type":"code","source":"def sample(signal, kernel_size):\n    sampled = np.zeros((signal.shape[0], signal.shape[1], signal.shape[2]//kernel_size))\n    for i in range(signal.shape[2]//kernel_size):\n        begin = kernel_size * i\n        end = min(kernel_size * (i + 1), signal.shape[2])\n        sampled[:, :, i] = np.mean(signal[:, :, begin:end], axis=2)\n    return sampled","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ca3b12709e3603dcc89625f0ff4adc948a9795e5"},"cell_type":"code","source":"sampled = sample(signals, 100)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"728175d3acac3e7779e486f6df5a54139711212b"},"cell_type":"code","source":"plt.figure(figsize=(15, 10))\nplt.plot(sampled[0, 0, :], label='Phase 0')\nplt.plot(sampled[0, 1, :], label='Phase 1')\nplt.plot(sampled[0, 2, :], label='Phase 2')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8594f0c680ec7ec26b214be18d0e5e5b72a19ae4"},"cell_type":"markdown","source":"# 3 - Fast Fourier Transform denoising\n\n#### A little bit of maths ...\nThe Fourier Transform of an 1D signal $x$ of length $n$ is the following : \n\n> ### $\\mathscr{f}_j = \\sum_{k=0}^{n-1} x_k e^{\\frac{2\\pi i}{n} jk} , ~~\\forall j=0, ... , n-1$ \n\nThe idea is to represent the signal in the complex space, It is roughly a sum of sinusoïdal functions. And there is one coefficient per frequency present in the signal.\n\nThe frequency takes the following values : \n- $f = \\frac{1}{dn} [0, 1, \\ldots ,   \\frac{n}{2}-1,  -\\frac{n}{2}, \\ldots , -1] $  if $n$ is even\n- $f =\\frac{1}{dn}  [0, 1, \\ldots,  \\frac{n-1}{2}, -\\frac{n-1}{2}, \\ldots, -1] $   if $n$ is odd\n\n#### Denoising algorithm\nThe denoising steps are the following :\n- Apply the fft to the signal\n- Compute the frequencies associated with each coefficient\n- Keep only the coefficients which have a low enough frequency (in absolute)\n- Compute the inverse fft\n"},{"metadata":{"trusted":true,"_uuid":"bf7d1a88706a1dfebfe70675806ed799d529536f"},"cell_type":"code","source":"def filter_signal(signal, threshold=1e8):\n    fourier = rfft(signal)\n    frequencies = rfftfreq(signal.size, d=20e-3/signal.size)\n    fourier[frequencies > threshold] = 0\n    return irfft(fourier)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"481f1062cadb64df8776fc2b92fe8202c8e34d8a"},"cell_type":"markdown","source":"### Testing some thresholds"},{"metadata":{"trusted":true,"_uuid":"a22aad3bdbf3369a2a180e4692ba29a99d4be078"},"cell_type":"code","source":"filtered = filter_signal(signals[0, 0, :], threshold=1e3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ac58d384afa698f801f4a4f49d4550b48835dae3"},"cell_type":"code","source":"plt.figure(figsize=(15, 10))\nplt.plot(signals[0, 0, :], label='Raw')\nplt.plot(filtered, label='Filtered')\nplt.legend()\nplt.title(\"FFT Denoising with threshold = 1e3\", size=15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"92e910fc06f227234c97d61a10409dcb8df13025"},"cell_type":"code","source":"filtered = filter_signal(signals[0, 0, :], threshold=1e5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1139e9e41ca5914d1c5857c1bd615c4a8c593349"},"cell_type":"code","source":"plt.figure(figsize=(15, 10))\nplt.plot(signals[0, 0, :], label='Raw')\nplt.plot(filtered, label='Filtered')\nplt.legend()\nplt.title(\"FFT Denoising with threshold = 1e5\", size=15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0ba9d1babc5176696201591b8192e22a51409422"},"cell_type":"code","source":"filtered = filter_signal(signals[0, 0, :], threshold=1e7)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6de0fa39dbbead16f9207681867b984433b6275d"},"cell_type":"code","source":"plt.figure(figsize=(15, 10))\nplt.plot(signals[0, 0, :], label='Raw')\nplt.plot(filtered, label='Filtered')\nplt.legend()\nplt.title(\"FFT Denoising with threshold = 1e7\", size=15)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6e0fc79ce57e97e66cdc036207bb34ea0dc9de82"},"cell_type":"markdown","source":"### Other uses of the fft ...\nThe fft coefficients can be used as features to represent the signal. I'll try that in a future kernel.\n\nHowever, there are too many of them so some more processing has to be done before feeding them into a classifier. Denoising being a solution."},{"metadata":{"_uuid":"e745528af3801510dce81ea056e1d2137142d578"},"cell_type":"markdown","source":"Thats all for now,\n#### *Thanks for reading !*\nAny feedback is appreciated"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}