{"cells":[{"metadata":{},"cell_type":"markdown","source":"<h1>Riid AIEd Challenge 2020 - Part I</h1>\n\nDue to memory/time restrictions in this competition, work is divided into several parts (kernels):\n<ul>\n    <li>Part I - Memory optimization</li>\n    <li>Part II - Splitting data</li>\n    <li>Part III - Feature engineering</li>\n    <li>Part IV - Training and validation</li>\n    <li>Part V - Prediction and submission</li>\n</ul>\n\nThis is Part I. In this part I'll \n<ul>\n    <li>Read the competition data with the <code>datatable</code> package, perform some optimizations in data types to reduce memory footprint</li>\n    <li>Divide the competition data into two parts. The first part, which I'll call <code>past_data</code> will be used to create features for training. Also a small part of it will be used as training and validation dataset. The second part will be the test dataset, and will be designed to be similar to the competition test set.</li>\n    <li>Save data in pickle format to be used by the next phase.</li>\n</ul>"},{"metadata":{"trusted":true},"cell_type":"code","source":"!python3.7 -m pip install --upgrade pip\n!pip install /kaggle/input/python-datatable/datatable-0.11.0-cp37-cp37m-manylinux2010_x86_64.whl","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Imports\n\nimport os\nimport datatable as dt\nimport pandas as pd\nimport numpy as np\nimport pickle\nimport gc","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Define directories used\n\nDATA_DIR = '/kaggle/input/riiid-test-answer-prediction'\nWORKING_DIR = '/kaggle/working'","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The train data is huge (over 101 million rows). Trying to load it into memory with a plain <code>pd.read_csv</code> leads to kernel crashing. For that reason, I use the <code>datatable</code> package, which is more efficient for loading CSV files."},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n# Load the train data set \ncompetition_data = dt.fread(os.path.join(DATA_DIR, 'train.csv' )).to_pandas()\ncompetition_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's check what we have\n\ncompetition_data.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's see if we have nulls\n\ncompetition_data.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's see statistics about our data. We're especially insterested in value ranges for every column\n\ncompetition_data.max()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's see the maximum value representable by each of the common numeric types "},{"metadata":{"trusted":true},"cell_type":"code","source":"types = pd.Series(\n    data = [np.iinfo(np.int8).max, np.iinfo(np.int16).max, np.iinfo(np.int32).max, np.iinfo(np.int64).max, \n            np.finfo(np.float16).max, np.finfo(np.float32).max, np.finfo(np.float64).max],\n    index = ['np.int8', 'np.int16', 'np.int32', 'np.int64', 'np.float16', 'np.float32', 'np.float64'],\n    name = 'max value'\n)\ntypes","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"With <code>prior_question_elapsed_time</code>, though its values would fit nicely in a <code>np.float32</code> type, calculating the mean (adding all values) requires a bigger type, so we set appart this mean before changing the type."},{"metadata":{"trusted":true},"cell_type":"code","source":"print('mean calculated with original type (np.float64): ', competition_data.prior_question_elapsed_time.mean())\nprint('mean calculated with np.float32 type: ', competition_data.prior_question_elapsed_time.astype(np.float32).mean())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's free a bit of memory by fitting some columns into smaller types\n\ncompetition_data['content_id'] = competition_data['content_id'].astype(np.int16)\ncompetition_data['content_type_id'] = competition_data['content_type_id'].astype(np.int8)    # code as in example_test\ncompetition_data['task_container_id'] = competition_data['task_container_id'].astype(np.int16)\ncompetition_data['answered_correctly'] = competition_data['answered_correctly'].astype(np.int8)\ncompetition_data['user_answer'] = competition_data['user_answer'].astype(np.int8)\ncompetition_data['prior_question_elapsed_time'] = competition_data['prior_question_elapsed_time'].astype(np.float32)\ncompetition_data['prior_question_had_explanation'] = competition_data['prior_question_had_explanation'].astype('bool')\n\n# We don't need row_id\ncompetition_data = competition_data.drop(columns='row_id')\n\n_ = gc.collect()\n\ncompetition_data.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Wow, 2.6 GB instead of 4.6 GB."},{"metadata":{},"cell_type":"markdown","source":"<h2>Save data</h2>"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Let's save data into pickle format for next kernels\n\ncompetition_data.to_pickle(os.path.join(WORKING_DIR, 'competition_data.pkl'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"That's all folks"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}