{"cells":[{"metadata":{"_uuid":"77ab4d0ecb170f66f03dee50153d152761b2f0e0"},"cell_type":"markdown","source":"# **Synopsis of this EDA**\nSo, this seems to be a pretty difficult data set. I'm just getting started with Kaggle kernels and competitions and still getting used to dealing with big data sets. Most of the work that went into this kernel was figuring out how to manage a large data set and create useful features while not exceeding memory limits. Slowly but surely, I'm learning how to do this.\n\nAs for the difficulty of this data set, it has another aspect. If the data are useful and effective in distinguishing positive from negative samples, it seems we should, with good visualization, be able to see some basic hallmarks that help distinguish the positive examples. Sure, machine learning models are far more powerful than our eyes to discern differences, but I thought I'd at least be able to see something.\n\nAfter all the effort, I haven't been able to see differences. I did some rolling means on the data to smooth the samples, and thought I might be able to see some different low-frequency components between negative and positive examples. As this kernel shows, the smoothed samples look quite identical when viewing several negative and positive examples.\n\nI then thought I would look at higher frequency components (i.e. components that were eliminated by the mean-based smoothing) by using FFT. These are shown in the second set of visualizations at the end of the kernel. Here again, though, I can't pick out differences. I thought there might be some common peaks among positive examples that set them apart from the negative examples. But, no luck. Anyway, has anyone else had more luck with good feature engineering and visualization that reveals differences in the positive examples?"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport pyarrow.parquet as pq\nimport matplotlib.pyplot as plt\nfrom tqdm import tqdm\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\nprint(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aa4b0a45649847a6dff6714c725c41d6b1d83bec"},"cell_type":"markdown","source":"# **Function Definitions**\n## *Get column ids based on specified filter criteria*\nIf fiter_criteria is specified, it should be a dictionary where the keys are one or more metadata column names and the values are single integer values or lists of integers. For example, the following would select rows for 'id_measurement' values 0, 1, 2, 3, and 4 (for a total of 15 rows, since each id_measurement has three phases):\n\n    filter_criteria = {'id_measurement' : list(range(5))}\n\nThe following would select rows for positive examples only:\n\n    filter_criteria = {'target' : 1}"},{"metadata":{"trusted":true,"_uuid":"152ef989f4150d8901a39987442a8b46a9cd0ee1"},"cell_type":"code","source":"# Function to select column ids using filter criteria\n# Return the column ids and the corresponding metadata\n# If filter_criteria is None, get all column ids\ndef load_metadata_by_filter(dataset_type = 'train', filter_criteria=None):\n    # Set paths according to dataset type\n    if(dataset_type == 'test'):\n        metadata_filepath = '../input/metadata_test.csv'\n        data_filepath = '../input/test.parquet'\n    else:\n        metadata_filepath = '../input/metadata_train.csv'\n        data_filepath = '../input/train.parquet'\n    # Load metadata\n    metadata = pd.read_csv(metadata_filepath)\n    # Initialize a filter mask\n    filter_mask = [True]*metadata.shape[0]\n    # If filter criteria are specified, loop over criteria and generate filter mask\n    if(filter_criteria != None):\n        for k, v in filter_criteria.items():\n            # Initialize temporary filter mask\n            temp_filter_mask = [False]*metadata.shape[0]\n            # Make sure that v is a list\n            if((type(v)==int) | (type(v)==str)): v = [v]\n            # Temp filter mask should use OR operation\n            for value in v:\n                temp_filter_mask = temp_filter_mask | (metadata[k] == int(value))\n            # Final filter mask should use AND operation\n            filter_mask = filter_mask & temp_filter_mask\n    # Mask creation is done; Now, get indexes for data to select\n    subset_metadata = metadata.loc[filter_mask]\n    metadata_row_idxs = list(subset_metadata.index)\n    # Get the data subset for which to calculate statistics\n    subset_idxs = [str(idx) for idx in metadata_row_idxs]\n    # Return column indexes and corresponding metadata\n    return (subset_idxs, subset_metadata)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"822801fedadd65995355d6a7db25a71ddbebbb60"},"cell_type":"markdown","source":"## *Load train or test data for specified column ids*\nAfter running 'load_metadata_by_filter' to get a set of column ids, 'load_selected_data' can be called to load the data only for those column ids. This will be useful, because loading the entire training set at once uses too much memory (and the test set is even bigger)"},{"metadata":{"trusted":true,"_uuid":"e2975716fcdec53af512c8b11d133aa9f5433e16"},"cell_type":"code","source":"# Load data according to specified column ids\ndef load_selected_data(column_ids, dataset_type = 'train'):\n    # Set dataset path according to specified type\n    if(dataset_type == 'test'):\n        data_filepath = '../input/test.parquet'\n    else:\n        data_filepath = '../input/train.parquet'\n    # Import the corresponding data\n    data = pq.read_pandas(data_filepath, columns=column_ids).to_pandas()\n    # Return the data\n    return data","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"41a878d335c5efe915a5fb87f2c83d6498f8ff4b"},"cell_type":"markdown","source":"## *Compute statistics for selected data over shifted windows*\nBecause the training and test datasets are so large, and no attempt will be made to input raw data into a model, this function allows data specified by column ids to be loaded and statistics calculated on it. This function can be used in a loop to import data in chunks and compute statistics on it. Supported statistics are 'mean', 'std', 'min', and 'max'. Statistics are calculated within a window of size 'window_size' shifted by 'offset' over all the data."},{"metadata":{"trusted":true,"_uuid":"afde576dbcd920b1e0833ce427ef412dd42bd0e2"},"cell_type":"code","source":"# Function to compute summary statistics on a selection of data\n# 'window_size' is the size of the window for calculating rolling statistics\n# 'offset' is how far the window is shifted for each statistics value saved/returned\ndef compute_summary_statistics(column_ids, window_size, offset, statistic='mean', dataset_type = 'train'):\n    # Make sure window_size and offset are integers\n    window_size = int(window_size); offset = int(offset)\n    # Dictionary mapping stat string to numpy function\n    stat_mapping = {'mean':np.mean, 'std':np.std, 'min':np.min, 'max':np.max}\n    # Get the stat function\n    stat_function = stat_mapping[statistic]\n    # Load data in chunks to minimize memory usage, calculate FFT, and store results in a list\n    stat_results = []; max_set_size = 300\n    for idx in tqdm(range(int(np.ceil(len(column_ids)/max_set_size)))):\n        # Get column ids for the current data chunk\n        start_idx = max_set_size*idx\n        current_column_ids = column_ids[start_idx:start_idx+max_set_size]\n        # Load data for the current column ids\n        data = load_selected_data(current_column_ids, dataset_type)\n        # Take only every other data point to reduce memory usage\n        data = data[::2]\n        # Get the number of rows in data\n        data_len = data.shape[0]\n        # Assert that window_size <= data length (i.e. # of rows in the data)\n        assert window_size <= data_len, 'Error: window_size must be <= number of rows ({0}) in data!'.format(data_len)\n        # List comprehension to calculate statistics\n        num_loops = 1 + int((data_len - window_size)/offset)\n        summary_data = [stat_function(data.iloc[offset*i:offset*i+window_size].values, axis=0) for i in range(num_loops)]\n        # Convert to a dataframe and append to stat_results\n        summary_data_df = pd.DataFrame(summary_data, columns=current_column_ids)\n        stat_results.append(summary_data_df)\n        # Delete unneeded variables to manage memory usage\n        del data; del summary_data\n    # Concatenate the results (by column) into a single dataframe\n    stat_results_df = pd.concat(stat_results, axis=1)\n    # Delete stat_results for memory management\n    del stat_results\n    # Return the dataframe\n    return stat_results_df","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"17351df2e93418216efbf02675defad79e71dece"},"cell_type":"markdown","source":"## *Compute rolling mean on FFT results*\nTo get information about the frequency components of the data, calculate the FFT (Fast Fourier Transform) of the entire data (800000 data points) for a specified set of column ids. The magnitudes returned by the FFT vary over several orders of magnitude and the results are noisy. Therefore, they should be normalized across samples before calculating summary statistics."},{"metadata":{"trusted":true,"_uuid":"5cad2f14012edcf2e932d85348356d2e5844e97a"},"cell_type":"code","source":"# Function to get the FFT (Fast Fourier Transform) on a selection of data and compute rolling means\n# 'window_size' is the size of the window for calculating rolling means\n# 'offset' is how far the window is shifted for each mean value saved/returned\ndef compute_fft_statistics(column_ids, window_size, offset, dataset_type = 'train'):\n    # Make sure window_size and offset are integers\n    window_size = int(window_size); offset = int(offset)\n    # Load data in chunks to minimize memory usage, calculate FFT, and store results in a list\n    fft_results = []; max_set_size = 300\n    for idx in tqdm(range(int(np.ceil(len(column_ids)/max_set_size)))):\n        # Get column ids for the current data chunk\n        start_idx = max_set_size*idx\n        current_column_ids = column_ids[start_idx:start_idx+max_set_size]\n        # Load data for the current column ids\n        data = load_selected_data(current_column_ids, dataset_type)\n        # Take only every other data point to reduce memory usage\n        data = data[::2]\n        # Calculate the FFT of the data and select only the first half of the result\n        fft_data = np.abs(np.fft.fft(data, axis=0))[:int(len(data)/2)]\n        # Delete data to manage memory usage\n        del data\n        # Normalize FFT data by mean across columns (i.e. across samples)\n        #fft_data = fft_data / fft_data.mean(axis=1).reshape(-1,1)\n        # Get the number of rows in data\n        data_len = fft_data.shape[0]\n        # Assert that window_size <= data length (i.e. # of rows in the data)\n        assert window_size <= data_len, 'Error: window_size must be <= number of rows in data!'\n        # Use a list comprehension to calculate rolling means (down columns)\n        num_loops = 1 + int((data_len - window_size)/offset)\n        summary_data = [np.mean(fft_data[offset*i:offset*i+window_size,:], axis=0) for i in range(num_loops)]\n        # Take the log of the data and append results to fft_results\n        summary_data = np.log(summary_data)\n        fft_results.append(pd.DataFrame(summary_data, columns=current_column_ids))\n        # Delete unneeded variables to manage memory usage (hopefully Python will do the garbage collection)\n        del fft_data; del summary_data\n    # Concatenate the results (by column) into a single dataframe\n    all_fft_results = pd.concat(fft_results, axis=1)\n    # Delete fft_results for memory management\n    del fft_results\n    # Return normalized and smoothed FFT results\n    return all_fft_results","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"66cbadc1a50eaad66f7b0f66d13a25c87989c93b"},"cell_type":"markdown","source":"# **Data Exploration**\n## *Check the relative number of negative and positive examples*\nPositive examples (i.e. target == 1) are only 6% of the training set"},{"metadata":{"trusted":true,"_uuid":"0c7a12f81fb9c74b672beb7d07d164b1e23ccd4d"},"cell_type":"code","source":"# Import metadata for the training set\n_, metadata_train = load_metadata_by_filter('train')\n\n# Get the number of negative and positive examples\nnum_negative = np.sum(metadata_train.target == 0)\nnum_positive = np.sum(metadata_train.target == 1)\npercent_positive = int(1e3*num_positive/(num_positive + num_negative))/10\n\n# Print information about negative and positive examples\nprint('Number of training examples:', metadata_train.shape[0])\nprint('Negative examples: {0}'.format(num_negative))\nprint('Positive examples: {0} ({1}% of total)'.format(num_positive, percent_positive))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7c5149290965f5f56964c84645b9f02a744bb5df"},"cell_type":"markdown","source":"## *Get the range of measurement ids in the training set*\nThe measurement ids range from 0 to 2903. With three phases for each, this comes to a total of 8712 (i.e. the total size of the training set)"},{"metadata":{"trusted":true,"_uuid":"bec7fd54b5f277924638260a370c5fe79ecee6c3"},"cell_type":"code","source":"# Import metadata for the training set\n_, metadata_train = load_metadata_by_filter('train')\n# Get the minimum and maximum id_measurement values\nmin_id_measurement = metadata_train.id_measurement.min()\nmax_id_measurement = metadata_train.id_measurement.max()\nprint('Train measurement ids: {0} to {1}'.format(min_id_measurement, max_id_measurement))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5d6525f771d8787ce6638d6d5bd28bf7504a84ff"},"cell_type":"markdown","source":"## *Get column ids for which to plot features*\nGenerate two lists of id_measurement values: One containing the first 10 id_measurement values for which all three target values (corresponding to the three phases) are 0 (i.e. negative examples) and the other containing the first 10 id_measurement values for which all three target values are 1 (i.e. positive examples). These will be used to plot features (rolling mean and FFT results) for visual comparison of negative and positive examples."},{"metadata":{"trusted":true,"_uuid":"045de4f4ecd08254d98925ad00e611c3ce2c3db1"},"cell_type":"code","source":"# Import metadata for the training set\n_, metadata_train = load_metadata_by_filter('train')\n# Loop over id_measurements until 10 all-negative and 10 all-positive target value\n# ids (i.e. id_measurement ids for which target values for the three phases are\n# all 0 or all 1) have been identified.\nnegative_ids = []; positive_ids = []; max_ids = 10\nfor id_measurement in range(metadata_train.id_measurement.max()):\n    # Get metadata for id_measurement\n    current_metadata = metadata_train[metadata_train.id_measurement == id_measurement]\n    # If all target values are 0, add id to the negative list\n    if((current_metadata.target == 0).all()):\n        if(len(negative_ids) < max_ids): negative_ids.append(id_measurement)\n    # If all target values are 1, add id to the positive list\n    elif((current_metadata.target == 1).all()):\n        if(len(positive_ids) < max_ids): positive_ids.append(id_measurement)\n    # If 10 negative and 10 positive ids have been found, break from loop\n    if((len(negative_ids) >= max_ids) & (len(positive_ids) >= max_ids)):\n        break\n# Print the negative and positive id_measurement values\nprint('Negative ids:', negative_ids)\nprint('Positive ids:', positive_ids)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"812e8fe1829edb046c4d68ccb087c31136fb215c"},"cell_type":"markdown","source":"## *Load and preprocess data for negative and positive examples*\nNow, calculate rolling means and FFT features for the negative and positive examples. Note that window_size for rolling means is double that for FFT results, because the result of FFT analysis is half of the original data.\n\nNote that, for rolling means, a sliding window of 8000 is used to calculate the means over the data set. The window is moved by 800 on each shift. On the entire dataset, this would reduce the size of the data from 800000 (8e5) rows to only 1 + (800000 - 8000) / 800 = 991 rows. Since the dataset is immediately cut to half the size (i.e. 4e5 rows), the final number of rows after calculating rolling means is 1 + (400000 - 8000) / 800 = 491.\n\nFor the FFT results, the raw data is also cut to half the original size (i.e. 4e5 rows). Since the FFT results are the same dimension, but they are symmetric. So, only the first half of the data is used, reducing the size to 2e5 rows. Finally, a rolling mean is calculated down the columns with a sliding window of 4000 with a shift of 400. The final number of rows is 1 + (200000 - 4000) / 400 = 491 (the same dimension as the rolling mean results)."},{"metadata":{"trusted":true,"_uuid":"ad59e9f7fe226f594283dac366b9d2e18b13883e"},"cell_type":"code","source":"# Get column ids for the negative id_measurement values\ncolumn_ids_0, metadata_0 = load_metadata_by_filter('train', \n                            filter_criteria={'id_measurement':negative_ids})\n# Set window_size for rolling means and FFT preprocessing\nwindow_size_means = 8000; window_size_fft = int(window_size_means/2)\n# For the negative ids, get rolling means and FFT results\nrolling_means_0 = compute_summary_statistics(column_ids_0,\n    window_size_means, int(window_size_means/10), statistic='mean', dataset_type = 'train')\nfft_results_0 = compute_fft_statistics(column_ids_0,\n    window_size_fft, int(window_size_fft/10), dataset_type = 'train')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8604a0531aa6c2a5acb1cfdb7eed12587ed4018a"},"cell_type":"code","source":"# Get column ids for the positive id_measurement values\ncolumn_ids_1, metadata_1 = load_metadata_by_filter('train',\n                            filter_criteria={'id_measurement':positive_ids})\n# Set window_size for rolling means and FFT preprocessing\nwindow_size_means = 8000; window_size_fft = int(window_size_means/2)\n# For the positive ids, get rolling means and FFT results\nrolling_means_1 = compute_summary_statistics(column_ids_1,\n    window_size_means, int(window_size_means/10), statistic='mean', dataset_type = 'train')\nfft_results_1 = compute_fft_statistics(column_ids_1,\n    window_size_fft, int(window_size_fft/10), dataset_type = 'train')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a47aae31a7b27dbaaa4bd933b393f23296547c12"},"cell_type":"markdown","source":"## *Generate plots of rolling means and FFT results for negative and positive examples*\nTo visually compare rolling means and FFT results for negative and positive examples, generate plots of all preprocessed data. Ideally, if the data are useful for detecting faults in power lines, we would like to see some visually striking differences between the negative and positive examples. If we don't see this, we hope that machine learning models can effectively distinguish the groups based on non-obvious feature differences."},{"metadata":{"_uuid":"58c1d5699cf5fe6ae23e1d731b633870935a63fe"},"cell_type":"markdown","source":"### First define the plotting routine to use for all data sets"},{"metadata":{"trusted":true,"_uuid":"044a8ff80648f7770e9022050da5aea8c08d7ebd"},"cell_type":"code","source":"# Function to generate subplots for rolling means and FFT results for negative and postive examples\ndef generate_subplots(data, metadata, ylim=[0,10]):\n    # Get the number of plots to generate\n    num_plots = data.shape[1]\n    # Create the subplot object and get handles\n    fig, axs = plt.subplots(int(num_plots/3), 3, figsize=(16, int(1.3*num_plots)))\n    for idx in range(num_plots):\n        # Get metadata for the current example\n        signal_id, id_measurement, phase, target = metadata.iloc[idx,:]\n        # Get the current data to plot\n        current_data = data[str(signal_id)].values\n        # Get the current ax handle\n        ax = axs[int(idx/3), idx%3]\n        # Set the title and turn on the grid for the current plot\n        ax.set_title('sig id {0}; id meas {1}; phase {2}; target {3}'\n                     .format(signal_id, id_measurement, phase, target))\n        ax.grid(True)\n        # Set the yrange\n        ax.set_ylim(ylim)\n        # Plot the data\n        ax.plot(current_data)\n    # Show the plots\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"eaa0ae82800a09447a7e161b499e637876aa8eee"},"cell_type":"markdown","source":"### Now plot rolling means and FFT results for negative and positive examples"},{"metadata":{"trusted":true,"_uuid":"dab001460f166c8d8a68362666c9acf50c8144b5"},"cell_type":"code","source":"# --- Plot rolling means for negative examples --- #\ngenerate_subplots(rolling_means_0, metadata_0, ylim=[-25,25])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ba89059975bdd8a3c74acca5b3e1a270a5d6e25f"},"cell_type":"code","source":"# --- Plot rolling means for positive examples --- #\ngenerate_subplots(rolling_means_1, metadata_1, ylim=[-25,25])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b1937ad5a6d06b346ad8751f12d215fc1deed71a"},"cell_type":"code","source":"# --- Plot FFT results for negative examples --- #\ngenerate_subplots(fft_results_0, metadata_0, ylim=[5,8])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"9a949842fcb33a6cf757a27f5d975cb135155a56"},"cell_type":"code","source":"# --- Plot FFT results for positive examples --- #\ngenerate_subplots(fft_results_1, metadata_1, ylim=[5, 8])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d73fd13c4c54aeaca18f248dfb94e9e97c601d5d"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d5c436e5770772d4f94059154ad02a9a35164c2f"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c75e1c2684ca2de71bd845f884718ef16f5f7433"},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c5799681ec39f5e1cefbee98675adb4b3276f92e"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}