{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Riiid! Answer Correctness Prediction"},{"metadata":{},"cell_type":"markdown","source":"## DATA"},{"metadata":{},"cell_type":"markdown","source":"1. **Train.csv** \n\n`row_id`: (int64) ID code for the row.\n\n`timestamp`: (int64) the time in milliseconds between this user interaction and the first event completion from that user.\n\n`user_id`: (int32) ID code for the user\n\n`content_id`: (int16) ID code for the user interaction\n\n`content_type_id`: (int8) 0 if the event was a question being posed to the user, 1 if the event was the user watching a lecture.\n\n`task_container_id`: (int16) Id code for the batch of questions or lectures. For example, a user might see three questions in a row before seeing the explanations for any of them. Those three would all share a task_container_id.\n\n`user_answer`: (int8) the user's answer to the question, if any. Read -1 as null, for lectures.\n\n`answered_correctly`: (int8) if the user responded correctly. Read -1 as null, for lectures.\n\n`prior_question_elapsed_time`: (float32) The average time in milliseconds it took a user to answer each question in the previous question bundle, ignoring any lectures in between. Is null for a user's first question bundle or lecture. Note that the time is the average time a user took to solve each question in the previous bundle.\n\n`prior_question_had_explanation`: (bool) Whether or not the user saw an explanation and the correct response(s) after answering the previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Typically the first several questions a user sees were part of an onboarding diagnostic test where they did not get any feedback."},{"metadata":{},"cell_type":"markdown","source":"# Importing Libraries"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"\nimport numpy as np\nimport pandas as pd\nimport dask.dataframe as dd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport plotly.express as px\n%pylab inline","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Load data\ntrain_data=pd.read_csv('/kaggle/input/riiid-test-answer-prediction/train.csv',low_memory=False, nrows = 1000000)\ntest_data=pd.read_csv('/kaggle/input/riiid-test-answer-prediction/example_test.csv',low_memory=False)\nquestion=pd.read_csv('/kaggle/input/riiid-test-answer-prediction/questions.csv',low_memory=False)\nlecture=pd.read_csv('/kaggle/input/riiid-test-answer-prediction/lectures.csv',low_memory=False)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Explore train data\nprint(\"No of rows in the training set:\", len(train_data))\nprint(\"--------------------------\")\nprint(\"No of attributes:\",len(train_data.columns))\nprint(\"--------------------------\")\nprint(train_data.dtypes)\nprint(\"--------------------------\")\nprint(train_data.head())\nprint(\"--------------------------\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Finding missing values in each column.\nprint('Part of missing values for every column')\nprint(train_data.isnull().sum() / len(train_data))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#The number of unique users in the training set.\nunique_users=train_data.user_id.nunique()\nprint(\"We have\",unique_users,\"number of unique users in the training set\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**1. Trying to find the most active users on the platform**"},{"metadata":{"trusted":true},"cell_type":"code","source":"user = train_data.groupby(train_data.user_id).user_id.count()\nuser_10 = user.nlargest(10)\nuser_10 = user_10.reset_index(name = 'counts')\nuser_10.user_id = user_10.user_id.astype('str')\nrow = user_10.user_id.tolist()\ncol = user_10.counts.tolist()\nfig = plt.figure(figsize=(10,5))\nax = fig.add_axes([0,0,1,1])\nax = sns.barplot(x=\"user_id\", y=\"counts\", data=user_10)\nax.set_xticklabels(ax.get_xticklabels(), rotation=40, ha=\"right\")\nax.set_title('Top 10 users')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**2. Trying to plot the most frequently encountered contents on the platform between questions and lectures**"},{"metadata":{"trusted":true},"cell_type":"code","source":"content_type = train_data.groupby(train_data.content_type_id).content_type_id.count()\ncontent_type = content_type.reset_index(name = 'count')\nfig = px.pie(content_type, values='count', names='content_type_id', title='Content Type')\nfig.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"`Only 2% of Users watching the Lecture videos. 98% Users exposed to the questions.`"},{"metadata":{},"cell_type":"markdown","source":"**3. Finding the most recurring content**"},{"metadata":{"trusted":true},"cell_type":"code","source":"\ncids = train_data.content_id.value_counts()[:30]\nfig = plt.figure(figsize=(12,6))\ncids.plot.bar()\nplt.title(\"Thirty most used content id's\")\nplt.xticks(rotation=90)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**4. Trying to find the percentsge of users choosing different options while answering a question**"},{"metadata":{"trusted":true},"cell_type":"code","source":"user_answer = train_data.groupby(train_data.user_answer).user_answer.count()\nuser_answer = user_answer.reset_index(name = 'count')\nfig = px.pie(user_answer, values='count', names='user_answer', title='User Answer Distribution')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**5. Trying to see how many users correctly answered and how many didn't**"},{"metadata":{"trusted":true},"cell_type":"code","source":"answered_correctly = train_data.groupby(train_data.answered_correctly).answered_correctly.count()\nanswered_correctly = answered_correctly.reset_index(name = 'count')\nfig = px.pie(answered_correctly, values='count', names='answered_correctly', title='Answered correctly Distribution')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Questions.csv"},{"metadata":{},"cell_type":"markdown","source":" **questions.csv**: metadata for the questions posed to users.\n\n`question_id`: foreign key for the train/test content_id column, when the content type is question (0).\n\n`bundle_id`: code for which questions are served together.\n\n`correct_answer`: the answer to the question. Can be compared with the train user_answer column to check if the user was right.\n\n`part`: the relevant section of the TOEIC test.\n\n`tags`: one or more detailed tag codes for the question. The meaning of the tags will not be provided, but these codes are sufficient for clustering the questions together."},{"metadata":{"trusted":true},"cell_type":"code","source":"#Explore the questions.csv file\nprint(\"No of rows in questions.csv:\",len(question))\nprint(\"--------------------------\")\nprint(\"No of attributes:\",len(question.columns))\nprint(\"--------------------------\")\nprint(question.head())\nprint(\"--------------------------\")\nprint(question.dtypes)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Finding missing values in each column.\nprint('Part of missing values for every column')\nprint(question.isnull().sum() / len(question))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Distribution for correct answers**"},{"metadata":{"trusted":true},"cell_type":"code","source":"question = question\nfig = plt.figure(figsize=(8,6))\ncr = question.groupby(\"correct_answer\")['question_id'].count().reset_index(name = 'counts')\ncad = sns.barplot(x=\"correct_answer\", y=\"counts\", data=cr).set_title(\"Correct Answers Distribution\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Most recurring tags**"},{"metadata":{"trusted":true},"cell_type":"code","source":"check = question['tags'].str.split(' ').explode('tags').reset_index()\ncheck = check['tags'].value_counts().reset_index()\n\ncheck.columns = ['tag', 'count']\ncheck['tag'] = check['tag'].astype(str) + '-'\ncheck = check.sort_values(['count'])\n\nfig = px.bar(\n    check.tail(10), \n    x='count', \n    y='tag', \n    title='Top 10 most useful tags'\n)\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## lectures.csv"},{"metadata":{},"cell_type":"markdown","source":" **lectures.csv**: metadata for the lectures watched by users as they progress in their education.\n\n`lecture_id`: foreign key for the train/test content_id column, when the content type is lecture (1).\n\n`part`: top level category code for the lecture.\n\n`tag`: one tag codes for the lecture. The meaning of the tags will not be provided, but these codes are sufficient for clustering the lectures together.\n\n`type_of`: brief description of the core purpose of the lecture"},{"metadata":{},"cell_type":"markdown","source":"**Exploring how many parts are there in this data set**"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Explore Lecture data\nprint(\"No of rows in the lectures dataset:\", len(lecture))\nprint(\"--------------------------\")\nprint(\"No of attributes:\",len(lecture.columns))\nprint(\"--------------------------\")\nprint(lecture.dtypes)\nprint(\"--------------------------\")\nprint(lecture.head())\nprint(\"--------------------------\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lecture = lecture\npart_count = lecture.groupby(\"part\")['lecture_id'].count().reset_index(name = 'counts')\nchart = px.pie(part_count, values='counts', names='part', title='Part Type ')\nchart.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Different types of lectures**"},{"metadata":{"trusted":true},"cell_type":"code","source":"type_count = lecture.groupby(\"type_of\")['lecture_id'].count().reset_index(name = 'counts')\nchart2 = px.pie(type_count, values='counts', names='type_of', title='Lecture Type')\nchart2.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#The number of unique tags of lectures.\nunique_tags=lecture.tag.nunique()\nprint(\"We have\",unique_tags,\"number of unique tags.\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Trying to find which is the most import lecture"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Exploring example_test csv file\nprint(\"No of rows in test file:\",len(test_data))\nprint(\"--------------------------\")\nprint(\"No of attributes:\",len(test_data.columns))\nprint(\"--------------------------\")\nprint(test_data.dtypes)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Exploring the submission csv file \nsubmission=dd.read_csv('/kaggle/input/riiid-test-answer-prediction/example_sample_submission.csv',low_memory=False).compute()\nprint(\"Number of rows in submission file:\",len(submission))\nprint(\"--------------------------\")\nprint(\"No of attributes:\",len(submission.columns))\nprint(\"--------------------------\")\nprint(submission.head())\nprint(\"--------------------------\")\nprint(submission.dtypes)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# **Preprocessing Part**"},{"metadata":{"trusted":true},"cell_type":"code","source":"#Dropping unwanted attribute.\ndel train_data['row_id']","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Convert string to integer.\n\nfrom sklearn.preprocessing import LabelEncoder \nle = LabelEncoder() \ntrain_data['prior_question_had_explanation']=le.fit_transform(train_data['prior_question_had_explanation'])\nlecture['type_of']=le.fit_transform(lecture['type_of'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"\nlecture.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Selecting only numerical values\ntrain_corr =train_data._get_numeric_data()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Co-relation plot\ncorr=train_corr.corr()\nplt.subplots(figsize=(10,8))\nsns.heatmap(corr,annot=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Selecting only numerical values\nlectures_corr =lecture._get_numeric_data()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#Co-relation plot\ncorr=lectures_corr.corr()\nplt.subplots(figsize=(10,8))\nsns.heatmap(corr,annot=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import kmeans1d","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"x = lecture['tag']\nK = 7\n\nclusters, centroids = kmeans1d.cluster(x, K)\n\nprint(clusters)   \nprint(centroids)  ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.scatter(lecture['tag'],clusters)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}