{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport random\nimport matplotlib.pyplot as plt\nimport seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"chunk = 100000\ndf = pd.read_csv('../input/riiid-test-answer-prediction/train.csv', chunksize=chunk, iterator=True)\ntrain = pd.concat(df, ignore_index=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"questions = pd.read_csv(\"../input/riiid-test-answer-prediction/questions.csv\")\nlectures = pd.read_csv(\"../input/riiid-test-answer-prediction/lectures.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f\"Train Shape: {train.shape}\\nQuestions Shape: {questions.shape}\\nLectures Shape: {lectures.shape}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h2>Since the data is huge, we'll take a random sample (size: 10% of total data set) that fairly represents the data set"},{"metadata":{"trusted":true},"cell_type":"code","source":"random.seed(11)\nsamp = random.sample(range(len(train)),int(0.1*len(train)))\nprint(f\"No of Samples: {len(samp)}\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_samp = train.iloc[samp,:].copy()\ndel(train)\nprint(f\"Shape of Train Sample: {train_samp.shape}\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h2>Training Sample EDA</h2>\n\n<h3>Feature description of train.csv</h3>\n\nrow_id: (int64) ID code for the row.\n\ntimestamp: (int64) the time between this user interaction and the first event completion from that user.\n\nuser_id: (int32) ID code for the user.\n\ncontent_id: (int16) ID code for the user interaction\n\ncontent_type_id: (int8) 0 if the event was a question being posed to the user, 1 if the event was the user watching a lecture.\n\ntask_container_id: (int16) Id code for the batch of questions or lectures. For example, a user might see three questions in a row before seeing the explanations for any of them. Those three would all share a task_container_id.\n\nuser_answer: (int8) the user's answer to the question, if any. Read -1 as null, for lectures.\n\nanswered_correctly: (int8) if the user responded correctly. Read -1 as null, for lectures.\n\nprior_question_elapsed_time: (float32) The average time it took a user to answer each question in the previous question bundle, ignoring any lectures in between. Is null for a user's first question bundle or lecture. Note that the time is the average time a user took to solve each question in the previous bundle.\n\nprior_question_had_explanation: (bool) Whether or not the user saw an explanation and the correct response(s) after answering the previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Typically the first several questions a user sees were part of an onboarding diagnostic test where they did not get any feedback."},{"metadata":{"trusted":true},"cell_type":"code","source":"train_samp.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_samp.describe().T","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_samp.dtypes","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_samp.nunique()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h2>% of missing values per feature"},{"metadata":{"trusted":true},"cell_type":"code","source":"np.round(train_samp.isnull().mean()*100,2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"categorical_features = [\"content_type_id\",\"user_answer\",\"answered_correctly\",\"prior_question_had_explanation\"]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h3>Though the features above are of int data type, they can be considered as categorical nominal variables"},{"metadata":{"trusted":true},"cell_type":"code","source":"for col in categorical_features:\n    print(f\"{col} | dtype: {train_samp[col].dtypes} | nunique: {train_samp[col].nunique()}\\n{train_samp[col].value_counts()}\\n\\n\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for col in categorical_features:\n    sns.countplot(train_samp[col])\n    plt.title(col)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h3>Inferences From The Above Distribution Analysis</h3>\n\ncontent_type_id: In majority of the cases the event was a question being posed to the user.\n\nuser_answer: Analyzing this variable without additional information is irrelevant. Hence skipping it for now.\n\nanswered_correctly: Most of the students have correctly answered the questions\n\nprior_question_had_explanation: Majority of the prior questions had an explanation\n\n\n\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"prior_correct = train_samp.loc[train_samp[\"answered_correctly\"]>=0,[\"prior_question_had_explanation\",\"answered_correctly\"]].copy()\nct = pd.crosstab(index=prior_correct[\"prior_question_had_explanation\"],columns=prior_correct[\"answered_correctly\"],normalize=\"index\")\nct.plot(kind=\"bar\",stacked=True,figsize=(10,5))\nplt.ylabel(\"% of students\");","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<h3>When prior questions had an explanation most  of the students answered correctly. When there was no explanation to the prior question, around 50% of them answered correctly"},{"metadata":{},"cell_type":"markdown","source":"<h1>WORK IN PROGRESS"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}