{"cells":[{"metadata":{},"cell_type":"markdown","source":"In this notebook, the dataset will be manipulated using Exploratory Data Analysis (EDA) together with Light Gradient Boosting Machine (LGBM)."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport matplotlib as mp\nimport seaborn as sb\nimport glob","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We imported *pandas* and below we look at *train* and *sample_submission* files."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train = pd.read_csv(\"../input/predict-volcanic-eruptions-ingv-oe/train.csv\")\nsample_submission = pd.read_csv(\"../input/predict-volcanic-eruptions-ingv-oe/sample_submission.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Train file has 2 columns and 4431 rows.\n* **segment_id** indicates the file name from the train set\n* **time_to_eruption** is the time from the end of the file until the first erruption.\n\n=> we have 4431 files in the train set."},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_submission","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Interesting enough, sample_submission.csv has 4520 rows, the exact number of files in the test set.\nIt is posible for this file to be the file with results that we have to complete with the time_to_eruption numbers for each test file."},{"metadata":{},"cell_type":"markdown","source":"Our dataset comprises of the following:\n* a test set with 4520 test files\n* a train set with 4431 train files\n* a train.csv file that contains the metadata for train set (time_to_eruption for each train file)\n* a sample_submission.csv file"},{"metadata":{},"cell_type":"markdown","source":"Verify number of files for train and test sets."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_set = glob.glob(\"../input/predict-volcanic-eruptions-ingv-oe/train/*\")\nlen(train_set)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_set = glob.glob(\"../input/predict-volcanic-eruptions-ingv-oe/test/*\")\nlen(test_set)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The lengths of the two sets are confirmed."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_set[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_set[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_file1 = pd.read_csv(\"../input/predict-volcanic-eruptions-ingv-oe/train/800654756.csv\")\ntrain_file1","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"In the train file above, we have 10 columns with measurements coming from 10 sensors installed on an active volcano.\nEach file contains readings from the 10 sensors equal in time with 10 minutes of recording."},{"metadata":{},"cell_type":"markdown","source":"The data is numerical but we have also NaN values (not a number)."},{"metadata":{"trusted":true},"cell_type":"code","source":"sensors = set()\nobservations = set()\n\nfor file in train_set:\n    f = pd.read_csv(file)\n    \n    sensors.add(len(f.columns))\n    observations.add(len(f))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Unique number of sensors: \", sensors)\nprint(\"Unique number of observations: \", observations)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"From the two lines above we conclude all files from the train set are identical as structure: all files have 10 columns and 60001 rows."},{"metadata":{"trusted":true},"cell_type":"code","source":"sensors = set()\nobservations = set()\n\nfor file in test_set:\n    f = pd.read_csv(file)\n    \n    sensors.add(len(f.columns))\n    observations.add(len(f))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"Number of sensors: \", sensors)\nprint(\"Number of observations: \", observations)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We have done the same thing for the test set. In this set we have also 10 sensors and 60001 observations for each file, so the dataset is consistent from this point of view.\n\nAll the files from the test and train sets have the same structure: 10 columns and 60001 rows each."},{"metadata":{"trusted":true},"cell_type":"code","source":"time2eruption = train[\"time_to_eruption\"]\ntime2eruption.plot(kind = \"hist\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train[\"time_to_eruption\"].describe()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}