{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport seaborn as sns\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 5GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Reading data"},{"metadata":{},"cell_type":"markdown","source":"We will read half of the train data but you can do all the shown preprocessing with another half."},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"%%time\n\ndtypes = {\n    \"row_id\": \"int64\",\n    \"timestamp\": \"int64\",\n    \"user_id\": \"int32\",\n    \"content_id\": \"int16\",\n    \"content_type_id\": \"boolean\",\n    \"task_container_id\": \"int16\",\n    \"user_answer\": \"int8\",\n    \"answered_correctly\": \"int8\",\n    \"prior_question_elapsed_time\": \"float32\", \n    \"prior_question_had_explanation\": \"boolean\"\n}\n\ndata = pd.read_csv(\"../input/riiid-test-answer-prediction/train.csv\", dtype=dtypes, nrows=50000000)\n\nprint(\"Train size:\", data.shape)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"qdata = pd.read_csv('../input/riiid-test-answer-prediction/questions.csv')\nldata = pd.read_csv('../input/riiid-test-answer-prediction/lectures.csv')\ntest_data = pd.read_csv('../input/riiid-test-answer-prediction/example_test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"qdata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ldata.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Feature description"},{"metadata":{},"cell_type":"markdown","source":"Let's talk more detalized about every single feature in train."},{"metadata":{},"cell_type":"markdown","source":"row_id: (int64) ID code for the row."},{"metadata":{"trusted":true},"cell_type":"code","source":"data.row_id.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that ID is unique."},{"metadata":{},"cell_type":"markdown","source":"timestamp: (int64) the time in milliseconds between this user interaction and the first event completion from that user."},{"metadata":{"trusted":true},"cell_type":"code","source":"data.timestamp.hist()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's look at some users timestamp distribution:"},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.distplot(data[data.user_id==data.user_id.unique()[0]].timestamp)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.distplot(data[data.user_id==data.user_id.unique()[1]].timestamp)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.distplot(data[data.user_id==data.user_id.unique()[2]].timestamp)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Or let's have a look at the smartest users timestamp distribution:"},{"metadata":{"trusted":true},"cell_type":"code","source":"data[data.user_id==data.groupby('user_id').answered_correctly.sum().sort_values(ascending=False).index[0]].timestamp.hist()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data[data.user_id==data.groupby('user_id').answered_correctly.sum().sort_values(ascending=False).index[1]].timestamp.hist()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Smart guys spend a lot of time on the platform"},{"metadata":{},"cell_type":"markdown","source":"user_id: (int32) ID code for the user."},{"metadata":{"trusted":true},"cell_type":"code","source":"data.user_id.hist()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"content_id: (int16) ID code for the user interaction"},{"metadata":{"trusted":true},"cell_type":"code","source":"data.content_id.hist()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"content_type_id: (int8) 0 if the event was a question being posed to the user, 1 if the event was the user watching a lecture."},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.countplot(data.content_type_id)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We can see that questions are much more than lectures."},{"metadata":{},"cell_type":"markdown","source":"task_container_id: (int16) Id code for the batch of questions or lectures. For example, a user might see three questions in a row before seeing the explanations for any of them. Those three would all share a task_container_id."},{"metadata":{},"cell_type":"markdown","source":"user_answer: (int8) the user's answer to the question, if any. Read -1 as null, for lectures."},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.countplot(data.user_answer)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.countplot(qdata.correct_answer)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It is strange but users prefer don't choose the second answer."},{"metadata":{},"cell_type":"markdown","source":"answered_correctly: (int8) if the user responded correctly. Read -1 as null, for lectures."},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.countplot(data.answered_correctly)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The right answers are about 2/3 and the wrong ones - 1/3. Let's have a look at the top of the smartest users:"},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.Series(data.groupby('user_id').answered_correctly.sum().sort_values(ascending=False).iloc[:30], index=data.groupby('user_id').answered_correctly.sum().sort_values(ascending=False).iloc[:30].index).sort_values().plot(kind='barh')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The median value of answered_correctly:"},{"metadata":{"trusted":true},"cell_type":"code","source":"data.groupby('user_id').answered_correctly.sum().median()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"And the mean:"},{"metadata":{"trusted":true},"cell_type":"code","source":"data.groupby('user_id').answered_correctly.sum().mean()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's create a new feature is_smart. We will consider a person to be smart if he answered right more questions than 75% quantile:"},{"metadata":{"trusted":true},"cell_type":"code","source":"sums = data.groupby('user_id').answered_correctly.sum()\nsmart_users = sums[sums > sums.quantile(0.75)].index\ndata['is_smart'] = 0\ndata.loc[data.user_id.isin(smart_users), 'is_smart'] = 1\ndel sums\ndel smart_users","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data.is_smart.value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Also let's count the summary time on the platform of each user in relation to the whole time of all the users.|"},{"metadata":{"trusted":true},"cell_type":"code","source":"sum_time = data.groupby('user_id').timestamp.sum()\nall_sum = sum_time.sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data['sum_timestamp'] = data['user_id'].apply(lambda x: sum_time.loc[x]/all_sum)\ndel sum_time\ndel all_sum","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mean0 = data[data.is_smart==0].sum_timestamp.mean()\nmean1 = data[data.is_smart==1].sum_timestamp.mean()\nmean0 / (mean0 + mean1), mean1 / (mean0 + mean1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As we can see, smart users spend a lot of time on the platform."},{"metadata":{},"cell_type":"markdown","source":"prior_question_elapsed_time: (float32) The average time in milliseconds it took a user to answer each question in the previous question bundle, ignoring any lectures in between. Is null for a user's first question bundle or lecture. Note that the time is the average time a user took to solve each question in the previous bundle."},{"metadata":{"trusted":true},"cell_type":"code","source":"data.prior_question_elapsed_time.hist()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"prior_question_had_explanation: (bool) Whether or not the user saw an explanation and the correct response(s) after answering the previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Typically the first several questions a user sees were part of an onboarding diagnostic test where they did not get any feedback."},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.countplot(data.prior_question_had_explanation)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's have a look if there are any users from train which are in test too:"},{"metadata":{"trusted":true},"cell_type":"code","source":"traintest_users = list(set(test_data.user_id) & set(data.user_id))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"traintest_users","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(traintest_users)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Yes! They are. And in larger dataset they will be more in count. So now we will make some preprocessing in test data"},{"metadata":{"trusted":true},"cell_type":"code","source":"test_data['is_smart'] = 0\nsum_time = test_data.groupby('user_id').timestamp.sum()\nall_sum = sum_time.sum()\ntest_data['sum_timestamp'] = test_data['user_id'].apply(lambda x: sum_time.loc[x]/all_sum)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"smart_guys = data[data.user_id.isin(traintest_users)].groupby('user_id')['is_smart'].max()\ntest_data.loc[test_data.user_id.isin(traintest_users), 'is_smart'] = test_data[test_data.user_id.isin(traintest_users)]['user_id'].apply(lambda x: smart_guys.loc[x])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_data","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"So here we have two new features named is_smart and sum_timestamp. They are correlating somehow:"},{"metadata":{"trusted":true},"cell_type":"code","source":"data[['is_smart','sum_timestamp']].corr()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"What I suggest is to count the traintest_users from another part of data and to make all the preprocessing deals. Then to take the users from traintest_users and transform the feaure is_smart from train to test. The least guys which are in test but aren't in traintest_users and their feauture is_smart we will predict."},{"metadata":{},"cell_type":"markdown","source":"## To be continued..."}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}