{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport os\n\nimport matplotlib.pyplot as plt  \nimport seaborn as sns  ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# EDA"},{"metadata":{},"cell_type":"markdown","source":"## 数据说明 Features in Files\n\n\n\n\n### train.csv\n\n|Name|UniqueOrNot|Notes|ProcessAdvice|\n|:--:|:--:|:--:|:--:|\n|row_id|Unique|Suppose it is the index|ignore|\n|timestamp|Not|the time in **$\\color{red}{milliseconds}$** between this user interaction and the first event completion from that user. 对于一个用户来说第一个完成的题目为0，从第一个题目开始计算到当前题目用时|sort value & maybe also indicate how hard is this problem|\n|user_id|Not|user id||\n|content_id|Not|[i'm not really understanding it.] ID code for the user interaction||\n|content_type_id|[0, 1]| 0 if the event was a question being posed to the user, 1 if the event was the user watching a lecture. 讲座0-做题1||\n|task_container_id|Not|Id code for the batch of questions or lectures. For example, a user might see three questions in a row before seeing the explanations for any of them. Those three would all share a task_container_id.|\n|user_answer|[-1,0,1,2,3]|the user's answer to the question, if any. Read -1 as null, for lectures. |Indicates whether the answer was guessed|\n|answered_correctly|[-1, 0, 1]|-1 refers lectures|\n|prior_question_elapsed_time||The average time in **$\\color{red}{milliseconds}$** it took a user to answer each question in the previous **$\\color{red}{question bundle}$** , ignoring any lectures in between. Is null for a user's first question bundle or lecture. Note that the time is the average time a user took to solve each question in the previous bundle.时间是多个题目(在这个题目前面的题目(同一个bundle内的)的平均时间，如果是第一个题目，时间为空，用户回答前一个问题包中的每个问题所花费的平均时间(以毫秒为单位)|\n|prior_question_had_explanation|bool| Whether or not the user saw an explanation and the correct response(s) after answering the previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Typically the first several questions a user sees were part of an onboarding diagnostic test where they did not get any feedback. 对一个用户来说一般前几个为空因为是测试|\n\n\n\n\n\n\n### questions.csv\n\n|Name|UniqueOrNot|Notes|ProcessAdvice|Na_value|\n|:--:|:--:|:--:|:--:|:--:|\n|question_id|Unqiue|foreign key for the train/test **$\\color{red}{content\\_id}$** column, when the content type is question (0).与数据集中的 **$\\color{red}{content\\_id}$** 关联（content_id=0的时候）||0|\n|bundle_id||code for which questions are served together. May connect with questions(like a passage has 3 sub-problems) 可能和题型相关|Clues about the question type (people may have different perference to different types of questions): [pao/This is TOEIC task](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/190191)|0|\n|correct_answer|[0, 1, 2, 3]|the answer to the question. Can be compared with the train **$\\color{red}{user\\_answer}$** column to check if the user was right.||0|\n|part|[1, 2, 3, 4, 5, 6, 7]|the relevant [section of the TOEIC test.](https://www.iibc-global.org/english/toeic/test/lr/about/format.html)|students' average levels of different part(also related with the format of the TOEIC task)|0|\n|tags|188 unique values|one or more detailed tag codes for the question. The meaning of the tags will not be provided, but these codes are sufficient for clustering the questions together.||**$\\color{red}{1}$**|\n\n\n\n\n\n### lectures.csv\n**Could you link the part to the specific task?**\n\n\n|Name|UniqueOrNot|Notes|ProcessAdvice|Na_value|\n|:--:|:--:|:--:|:--:|:--:|\n|lecture_id|unique|foreign key for the train/test **$\\color{red}{content\\_id}$** column, when the content type is lecture (1).||0|\n|part|[1, 2, 3, 4, 5, 6, 7]|top level category code for the lecture.|TOEIC has 7 parts. |0|\n|tag|151 unique|one tag codes for the lecture. The meaning of the tags will not be provided, but these codes are sufficient for clustering the lectures together.|It seems that the tag is independent to the part|0|\n|type_of|['concept', 'intention', 'solving question', 'starter']| brief description of the core purpose of the lecture||0|\n\n\n\n\n\n\n\n### example_test_rows.csv\n\n\nprior_group_responses (string) provides all of the entries for previous group in a string representation of a list in the first row of the group. All other rows in each group are null. If you are using Python, you will likely want to call on the non-null rows. Some rows may be null, or empty lists.user_answereval\n\nprior_group_answers_correct (string) provides all the field for previous group, with the same format and caveats as . Some rows may be null, or empty lists.answered_correctlyprior_group_responses\n\n\n\nIt hard to understand, and I will try my best (T_T)\n\n"},{"metadata":{},"cell_type":"markdown","source":"# read file\nThere are many ways to read a big csv file in a short time, since read a big table is a difficult thing for pandas.\n\nexamples could be seen here [rohanrao/tutorial-on-reading-large-datasets/](https://www.kaggle.com/rohanrao/tutorial-on-reading-large-datasets/)"},{"metadata":{"trusted":true},"cell_type":"code","source":"BASE_DIR = '/kaggle/input/riiid-test-answer-prediction/'\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## train.csv"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n\n\ntrain_dtypes_dict = {\n    \"row_id\": \"int64\",\n    \"timestamp\": \"int64\",\n    \"user_id\": \"int32\",\n    \"content_id\": \"int16\",\n    \"content_type_id\": \"int8\",\n    \"task_container_id\": \"int16\",\n    \"user_answer\": \"int8\",\n    \"answered_correctly\": \"int8\",\n    \"prior_question_elapsed_time\": \"float32\", \n    \"prior_question_had_explanation\": \"boolean\"\n}\n\ndata = pd.read_csv(os.path.join(BASE_DIR, \"train.csv\"),\n                         usecols = train_dtypes_dict.keys(),\n                         dtype=train_dtypes_dict,\n                        )\n\nprint(\"Train size:\", data.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.sample(5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## question.csv"},{"metadata":{"trusted":true},"cell_type":"code","source":"questions = pd.read_csv(os.path.join(BASE_DIR, \"questions.csv\"))\nprint(\"Questions size:\", questions.shape)\nquestions.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# how many questions in this bundle_id\nquestions.groupby(['bundle_id']).agg(\"count\")['question_id'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## lectures.csv"},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures = pd.read_csv(os.path.join(BASE_DIR, \"lectures.csv\"))\nprint(\"Lectures size:\", lectures.shape)\nlectures","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# different part is seperate \ntmp = lectures[['part','tag']].groupby(['part']).groups\n# union number between different part?\nfrom itertools import combinations\nfor a, b in combinations(tmp.keys(), 2):\n    print(\"left key: {}, right key: {}, union number: {}\".format(a,b,len([x for x in tmp[a].tolist() if x in tmp[b].tolist()])) )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Could you link the part to the specific task?\nplt.figure(figsize=(10,6))\nax = sns.countplot(x=\"part\", hue=\"type_of\", data=lectures)\nfor p in ax.patches:\n        ax.annotate('{:.0f}'.format(p.get_height()), (p.get_x()+0.02, p.get_height()+1))\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## example_test_rows.csv"},{"metadata":{"trusted":true},"cell_type":"code","source":"test = pd.read_csv(os.path.join(BASE_DIR, \"example_test.csv\"))\nprint(\"Questions size:\", test.shape)\ntest.head(10)                   ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}