{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Let's firstly import the libraries"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"#loading data\n\nimport numpy as np \nimport pandas as pd \nimport riiideducation \nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport gc\nimport os\nimport warnings \nwarnings.filterwarnings('ignore')\n\nfor dirname, _, filenames in os.walk('/kaggle/input/riiid-test-answer-prediction'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train_df = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/train.csv', low_memory=False, nrows=10**6, \n                       dtype={'row_id': 'int64', 'timestamp': 'int64', 'user_id': 'int32', 'content_id': 'int16', 'content_type_id': 'int8',\n                              'task_container_id': 'int16', 'user_answer': 'int8', 'answered_correctly': 'int8', 'prior_question_elapsed_time': 'float32', \n                             'prior_question_had_explanation': 'boolean',\n                             }\n                      )\ntrain_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"question = pd.read_csv('../input/riiid-test-answer-prediction/questions.csv')\nquestion.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lecture = pd.read_csv('../input/riiid-test-answer-prediction/lectures.csv')\nlecture.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* The question csv and lecture csv files hold information about the questions and the lectures present in the training dataset as the types of the interactions in the training data sets are either an interaction with lectures or questions."},{"metadata":{},"cell_type":"markdown","source":"  * The interactions type is represented in the content_type_id column (0 for question interaction and 1 for question interaction)"},{"metadata":{"trusted":true},"cell_type":"code","source":"\nquestions_interactions = train_df.merge(question, left_on = 'content_id', right_on = 'question_id', how = 'left')\nquestions_interactions = questions_interactions[questions_interactions.content_type_id == 0]\nquestions_interactions.rename(columns = {'part': 'test_part'}, inplace = True)\n\nlectures_interactions = train_df.merge(lecture, left_on = 'content_id', right_on = 'lecture_id', how = 'left') \nlectures_interactions.rename(columns = {'part': 'category'}, inplace = True)\nlectures_interactions = lectures_interactions[lectures_interactions.content_type_id == 1]\n\nquestions_interactions.shape, lectures_interactions.shape","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> Most of the interactions are with questions."},{"metadata":{},"cell_type":"markdown","source":"### Let's deal with nulls"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(questions_interactions.isnull().sum())\nlectures_interactions.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"> The columns that has nulls in the question interactions are the columns that puts nulls in the place where ni previous records are present."},{"metadata":{"trusted":true},"cell_type":"code","source":"# let's give the prior_question_elapsed_time nulls -1 as the user didn't didn't have previous bundle yet\n# and -1 for the prior_question_had_explanation after converting booleans into integers\n#as they didn't have any questions before\nindeces = questions_interactions[questions_interactions.prior_question_had_explanation.isnull()].index\nprint(indeces)\nvalues = {'prior_question_elapsed_time': -1, 'prior_question_had_explanation': False}\nquestions_interactions.fillna(value=values,inplace=True)\nquestions_interactions.prior_question_had_explanation = questions_interactions.prior_question_had_explanation.astype('int8')\nquestions_interactions.loc[indeces,'prior_question_had_explanation'] = -1\nquestions_interactions.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions_interactions.prior_question_had_explanation.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Now let's deal with the lectures interactions nulls"},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures_interactions.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* content_type_id\n* user_answer\n* answered_correctly \n* prior_question_elapsed_time \n* prior_question_had_explanation\n\ndoesn't have any meaning in this data frame anymore so we will drop them."},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures_interactions.drop(columns=['content_type_id','user_answer','answered_correctly','prior_question_elapsed_time','prior_question_had_explanation'],inplace =True)\nlectures_interactions.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Let's try improving the memory usage for faster analysis"},{"metadata":{"trusted":true},"cell_type":"code","source":"start_mem_usg1 = questions_interactions.memory_usage().sum() / 1024**2 \nstart_mem_usg2 = lectures_interactions.memory_usage().sum() / 1024**2 \n\nprint(\"Memory usage of questions_interactions dataframe is :\",start_mem_usg1,\" MB\")\nprint(\"Memory usage of lectures_interactions dataframe is :\",start_mem_usg2,\" MB\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions_interactions.dtypes","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# as the content id is the same as the question id and the content_type_id has only the zero values\n# we will drop them\nquestions_interactions.drop(columns=['content_id','content_type_id'],inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def reduce_mem_usage(props):\n    start_mem_usg = props.memory_usage().sum() / 1024**2 \n    print(\"Memory usage of properties dataframe is :\",start_mem_usg,\" MB\")\n    NAlist = [] # Keeps track of columns that have missing values filled in. \n    for col in props.columns:\n        if props[col].dtype != object:  # Exclude strings\n            # make variables for Int, max and min\n            IsInt = False\n            mx = props[col].max()\n            mn = props[col].min()\n            \n            # Integer does not support NA, therefore, NA needs to be filled\n            if not np.isfinite(props[col]).all(): \n                NAlist.append(col)\n                props[col].fillna(mn-1,inplace=True)  \n                   \n            # test if column can be converted to an integer\n            asint = props[col].fillna(0).astype(np.int64)\n            result = (props[col] - asint)\n            result = result.sum()\n            if result > -0.01 and result < 0.01:\n                IsInt = True\n\n            \n\n            \n            # Make Integer/unsigned Integer datatypes\n            if IsInt:\n                if mn >= 0:\n                    if mx < 255:\n                        props[col] = props[col].astype(np.uint8)\n                    elif mx < 65535:\n                        props[col] = props[col].astype(np.uint16)\n                    elif mx < 4294967295:\n                        props[col] = props[col].astype(np.uint32)\n                    else:\n                        props[col] = props[col].astype(np.uint64)\n                else:\n                    if mn > np.iinfo(np.int8).min and mx < np.iinfo(np.int8).max:\n                        props[col] = props[col].astype(np.int8)\n                    elif mn > np.iinfo(np.int16).min and mx < np.iinfo(np.int16).max:\n                        props[col] = props[col].astype(np.int16)\n                    elif mn > np.iinfo(np.int32).min and mx < np.iinfo(np.int32).max:\n                        props[col] = props[col].astype(np.int32)\n                    elif mn > np.iinfo(np.int64).min and mx < np.iinfo(np.int64).max:\n                        props[col] = props[col].astype(np.int64)    \n            \n            # Make float datatypes 32 bit\n            else:\n                props[col] = props[col].astype(np.float32)\n            \n            # Print new column type\n           # print(\"dtype after: \",props[col].dtype)\n           # print(\"******************************\")\n    \n    # Print final result\n    print(\"___MEMORY USAGE AFTER COMPLETION:___\")\n    mem_usg = props.memory_usage().sum() / 1024**2 \n    print(\"Memory usage is: \",mem_usg,\" MB\")\n    print(\"This is \",100*mem_usg/start_mem_usg,\"% of the initial size\")\n    return props, NAlist\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions_interactions,_ = reduce_mem_usage(questions_interactions)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions_interactions.dtypes","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Let's explore the questions interaction distributions"},{"metadata":{"trusted":true},"cell_type":"code","source":"continous_columns = ['timestamp','user_id','task_container_id','prior_question_elapsed_time','question_id','bundle_id']\nquestions_interactions.hist(column=continous_columns, grid=False,figsize=(20,15));","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* The bundle_id and the question_id have the same exact distribution which may indicate a duplicate column.\n* user_id has uniform distributioin which indicates random user ids are used in the application\n* The time stamp has a left skewed distributions which may indicate the relative small times the users use the application in before stopping using it and \n* prior_question_elapsed_time is left skewed too which may indicate that most of the students don't take a lot of time before answering a question\n* task_container_id is left skewed which is strange and may indicate a meaning in these ids which makes thier distribution affected by the student actions"},{"metadata":{"trusted":true},"cell_type":"code","source":"color = sns.color_palette()[0]\ndiscrete_columns = ['answered_correctly','user_answer','correct_answer','prior_question_had_explanation','test_part']\nfor col in discrete_columns:\n    plt.figure(figsize=(5,5))\n    sns.countplot(data=questions_interactions,x=col,color=color)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* The number of answered correctly questions is double the wrong answered questions\n* Most of the questions had explanation\n* Users answers and correct answers are very similar (however there are a lot of wrong answers) which may indicates an interesting relation between them\n* The 5th test part has a lot of records followed by the second part"},{"metadata":{},"cell_type":"markdown","source":"## Let's preprocess the tags feature and explore it too"},{"metadata":{"trusted":true},"cell_type":"code","source":"df = questions_interactions.copy()\ndf = df.assign(tags2=df['tags'].str.split(' ')).explode('tags2')\ndf['tags2'] = df['tags2'].astype('int32') \ndf['tags2'].nunique()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"df['tags2'].hist(grid=False,figsize=(10,5));","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The distribution is more like normal distribution which may give the tags ids meaning."},{"metadata":{},"cell_type":"markdown","source":"## Let's take a look at the lectures_interactions data frame"},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures_interactions.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"continous_cols = ['timestamp','user_id','content_id','task_container_id','lecture_id','tag']\nlectures_interactions.hist(column=continous_cols, grid=False,figsize=(20,15));","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* The time stamp has a left skewed distributions which may indicate the relative small times the users use the application in before stopping using it and \n* user_id, content_id, lecture_id looks to have uniform distributions which means they are just random ids\n* task_container_id is left skewed which is strange and may indicate a meaning in these ids which makes thier distribution affected by the student actions\n* The tags here have uniform distribution."},{"metadata":{"trusted":true},"cell_type":"code","source":"color = sns.color_palette()[0]\ndiscrete_columns = ['type_of','category']\nfor col in discrete_columns:\n    plt.figure(figsize=(5,5))\n    sns.countplot(data=lectures_interactions,x=col,color=color)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* The category has the same shape like the test_part in the question_interactions which suggests close relationship between them\n* Most of the lectures types are concept based."},{"metadata":{},"cell_type":"markdown","source":"## Refrences \n1- memory used reduction: https://www.kaggle.com/cdeotte/dae-book3c from the cool grand master: Chris Deotte\n\n2- https://stackoverflow.com/questions/12680754/split-explode-pandas-dataframe-string-entry-to-separate-rows"},{"metadata":{},"cell_type":"markdown","source":"## Wait for part 2 with more insights and modular data wrangling :)"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}