{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Note:\n**i noticed there is lag in loading the entire note book. Please scroll down to the very bottom and scroll up. The enitre notebook with figures will load after that**\n<br>\n**Please let me know if there are any issues in comment. It will help me fixing the issue**"},{"metadata":{},"cell_type":"markdown","source":"# INTRODUCTION\nThis notebook is for people who doesn't have a fancy workstation or a compute budget in cloud. In this notebook i am demonstrating how to do the data preprocessing, aggregation, merging etc. on the **complete data** without ever running into memory issues or having to wait long time for results. \nI am using gpu accelerated **cudf** package by Rapids which has a very similar API to pandas(almost mirrored) for data preprocessing and **Plotly** for visualization. \n\n\n<br>Content:\n1. [About Rapids](#1)\n    1. [Installation on Kaggle](#2)\n1. [Load Data](#3)\n1. [Explore Individual Features](#4)\n    1. [train.csv](#5)\n    2. [questions.csv](#6)\n    3. [lectures.csv](#7)\n1. [Explore Feature Groups](#8)\n    * In Progress..\n1. Reference\n    1. Ednet Paper: https://arxiv.org/pdf/1912.03072.pdf\n    1. Rapids ai: https://rapids.ai/about.html\n    1. Plotly Express: https://plotly.com/python/plotly-express/"},{"metadata":{},"cell_type":"markdown","source":"## About Rapids\nThe RAPIDS suite of open source software libraries and APIs gives you the ability to execute end-to-end data science and analytics pipelines entirely on GPUs.RAPIDS utilizes NVIDIA CUDA® primitives for low-level compute optimization, and exposes GPU parallelism and high-bandwidth memory speed through user-friendly Python interfaces.<br><br>\nRAPIDS also focuses on common data preparation tasks for analytics and data science. This includes a familiar dataframe API that integrates with a variety of machine learning algorithms for end-to-end pipeline accelerations without paying typical serialization costs. RAPIDS also includes support for multi-node, multi-GPU deployments, enabling vastly accelerated processing and training on much larger dataset sizes.<br><br>\nSome RAPIDS projects include cuDF, a pandas-like dataframe manipulation library; cuML, a collection of machine learning libraries that will provide GPU versions of algorithms available in scikit-learn; cuGraph, a NetworkX-like accelerated graph analytics library.\n"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"1\"></a> <br>\n### Rapids Installation on Kaggle\n* Add the [rapids dataset](https://www.kaggle.com/cdeotte/rapids) to you notebook\n* Turn on the GPU\n* Run the following code snippet which will unzip the data and add the neccessary packages to the sys path"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import gc\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import sys\n!cp ../input/rapids/rapids.0.15.0 /opt/conda/envs/rapids.tar.gz\n!cd /opt/conda/envs/ && tar -xzvf rapids.tar.gz > /dev/null\nsys.path = [\"/opt/conda/envs/rapids/lib/python3.7/site-packages\"] + sys.path\nsys.path = [\"/opt/conda/envs/rapids/lib/python3.7\"] + sys.path\nsys.path = [\"/opt/conda/envs/rapids/lib\"] + sys.path \n!cp /opt/conda/envs/rapids/lib/libxgboost.so /opt/conda/lib/","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Load Data"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import numpy as np\nimport dask.dataframe as dd\nimport pandas as pd\nfrom time import time\nfrom contextlib import contextmanager\n\n# rapids libsJJ\nimport cuml\nimport cudf\nimport cupy\n\n# ploting\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport plotly\nimport plotly.express as px","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"try:\n    del train\n    del question\n    del lectures\nexcept:\n    pass","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"@contextmanager\ndef timer(name):\n    t0 = time()\n    yield\n    print(f'[{name}] done in {time() - t0:.2f} s')\n\n\n\nwith timer(\"Data Loading Time\"):\n    train = cudf.read_csv(\"/kaggle/input/riiid-test-answer-prediction/train.csv\",)\n    question = cudf.read_csv(\"/kaggle/input/riiid-test-answer-prediction/questions.csv\",)\n    lectures = cudf.read_csv(\"/kaggle/input/riiid-test-answer-prediction/lectures.csv\",)\n    ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"4\"></a>\n## Explore Individual Features"},{"metadata":{},"cell_type":"markdown","source":"<a id=\"5\"></a>\n## train.csv"},{"metadata":{},"cell_type":"markdown","source":"* ```row_id```: (int64) ID code for the row.\n\n* ```timestamp```: (int64) the time in milliseconds between this user interaction and the first event completion from that user.\n\n* ```user_id```: (int32) ID code for the user.\n\n* ```content_id```: (int16) ID code for the user interaction\n\n* ```content_type_id```: (int8) 0 if the event was a question being posed to the user, 1 if the event was the user watching a lecture.\n\n* ```task_container_id```: (int16) Id code for the batch of questions or lectures. For example, a user might see three questions in a row before seeing the explanations for any of them. Those three would all share a task_container_id.\n\n* ```user_answer```: (int8) the user's answer to the question, if any. Read -1 as null, for lectures.\n\n* ```answered_correctly```: (int8) if the user responded correctly. Read -1 as null, for lectures.\n\n* ```prior_question_elapsed_time```: (float32) the average time a user took to solve each question in the previous bundle. [refer](https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/189768)\n\n* ```prior_question_had_explanation```: (bool) Whether or not the user saw an explanation and the correct response(s) after answering the previous question bundle, ignoring any lectures in between. The value is shared across a single question bundle, and is null for a user's first question bundle or lecture. Typically the first several questions a user sees were part of an onboarding diagnostic test where they did not get any feedback."},{"metadata":{},"cell_type":"markdown","source":"### timestamp"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"print(\"summary statistics of the feature\")\ntrain.timestamp.describe().to_frame()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The timestamp values are in milliseconds and is hard to intrepret.So converting it into days and looking at the statistics will be more intutive"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"to_days_factor = 24*60*60*1000.\n(train.timestamp/to_days_factor).describe().round(3).to_frame().rename({'timestamp': 'timestamp_days'}, axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"On an Average a user spend 89.16 days or nearly three months for the TOEIC exam preperation. The value looks bit too high!"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df = train.sample(frac=0.01).to_pandas()\n\nimport plotly.express as px\nfig = px.histogram(df, x=\"timestamp\", title=\"Histogram of timestamp\")\nfig.show()\n\ndf['timestamp_in_days'] = np.log10(((df['timestamp']+1)/to_days_factor))\n\nfig = px.histogram(df,x='timestamp_in_days', range_x=[-5, 3], \n                  title=\"Histogram of timestamp in days (x axis log scale)\")\nfig.update_layout(\n    xaxis = dict(\n        tickmode = 'array',\n        tickvals = [i for i in range(-5,4)],\n        ticktext = [str(10**i) + \" days\" for i in range(-5,4)]\n    )\n)\n\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"But the above timestamp feature is recorded whenever there is a user interaction. So in the following analysis i will find the maximum timestamp at user level and compute the summary statistics and histogram again"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df = train.groupby([\"user_id\"]).timestamp.max().to_frame().to_pandas()\ndf['user_max_timestamp_in_days'] = (df['timestamp']+1)/(to_days_factor)\ndf['user_max_timestamp_in_days_log'] = np.log10(df['user_max_timestamp_in_days'])\n\ndf['user_max_timestamp_in_days'].describe().astype(float).round(3).to_frame()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The avereage duration a an arbitary user uses the app is 60 days or nearly two months. Which is lesser than the mean value of the timestamp feature."},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.histogram(df, x='user_max_timestamp_in_days_log', range_x=[-5, 3], \n                  title=\"Histogram of maximum user timestamp in days (x axis log scale)\",)\nfig.update_layout(\n    xaxis = dict(\n        tickmode = 'array',\n        tickvals = [i for i in range(-5,4)],\n        ticktext = [str(10**i) + \" days\" for i in range(-5,4)]\n    )\n)\n\nfig.update_layout(\n    showlegend=False,\n    annotations=[\n        dict(\n            x=-2,\n            y=8000,\n            xref=\"x\",\n            yref=\"y\",\n            text=\"Could be Dropouts\",\n            showarrow=True,\n            arrowhead=7,\n            ax=-200,\n            ay=-50\n        ),\n        dict(\n            x=1.6,\n            y=4000,\n            xref=\"x\",\n            yref=\"y\",\n            text=\"Serious Students\",\n            showarrow=True,\n            arrowhead=7,\n            ax=-200,\n            ay=-100\n        ),\n    ]\n)\n\n\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The bimodality of the above histogram is interesting.This figure says that there are broadly two kinds of users in Santa app. The first category is just come and checkout the app churn away. I call them dropouts. Their distribution peaks in the left. The other category of students who spends significant time in the App. They could be Serious student. Their distribution peaks in the right side of the histogram.<br><br>\nBut draw back of the above categorization are the following\n* Users who just joined the app when this dataset was sampled who are very serious about cracking TOEIC could be miinterpreted as dropouts because we dont have the details about their future interactions in the training dataset.\n* train data may be  sampled in such a way that some users end up getting very few interactions to artificially indroduce some imbalance in order for the model to be robust enough to predict the behavious of the newly joined users. In that case of also my hypothesis will categorize them as dropouts. which is actually not true"},{"metadata":{},"cell_type":"markdown","source":"### user_id"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df = train.groupby(\"user_id\")['row_id'].count(). \\\nto_frame().sort_values(\"row_id\", ascending=False).rename({'row_id': 'number_of_user_interactions'}, axis=1).reset_index().to_pandas()\ndf['user_id'] =  df.user_id.astype(str) + \"-\" \n\n\nfig = px.bar(df.head(50), \n             y='user_id', \n             x='number_of_user_interactions', \n             title=\"Bar Plot of top 50 users\",\n            orientation='h',\n            )\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.bar(df.tail(50), \n             y='user_id', \n             x='number_of_user_interactions', \n             title=\"Bar Plot of bottom 50 users\",\n            orientation='h')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### content_id and content_type_id"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df = train.groupby([\"content_id\", 'content_type_id'])['row_id'].count(). \\\nto_frame().sort_values(\"row_id\", ascending=False).rename({'row_id': 'number_of_interactions_for_content_id'}, axis=1).reset_index().to_pandas()\n\n\ndf['content_type_id'] = df['content_type_id'].map({0:'questions(0)', 1: 'lectures(1)'})\n\nfig = px.scatter(df, \n             x='content_id', \n             y='number_of_interactions_for_content_id', \n             title=\"content ids vs number of interactions in log scale\",\n             log_x = True,\n             log_y=True,\n                color='content_type_id',\n            )\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.box(df, x=\"content_type_id\", y=\"number_of_interactions_for_content_id\",\n             log_y=True,\n            labels={\n                     \"number_of_interactions_for_content_id\": \"number_of_interactions_per_content_id\"\n                 },\n                title=\"Box Plot of number of interaction per content id\",\n            )\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### task_container_id and container_type_id"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df = train.groupby([\"task_container_id\", \"content_type_id\"]).agg({\"row_id\": \"count\", \"content_id\": 'nunique'})\ndf.shape\n\ntrain.groupby(['content_type_id']).content_id.nunique().to_frame().rename({\"content_id\": \"number_of_unique_content_ids\"}, axis=1)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### task_container_id"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df = train.groupby(\"task_container_id\")['row_id'].count(). \\\nto_frame().sort_values(\"row_id\", ascending=False).rename({\"row_id\": \"interactions_per_task_container_id\"}, axis=1).reset_index().to_pandas()\n\n\nfig = px.bar(df.head(100), x=\"task_container_id\", y=\"interactions_per_task_container_id\",\n             log_y=True,\n                title=\"Bar Plot of number of interaction per task container id:Top 100\",\n            )\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.scatter(df, x=\"task_container_id\", y=\"interactions_per_task_container_id\",\n             log_y=True,\n                title=\"Scatter Plot of number of interaction per task container id\",\n            )\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### user_answer"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df = train.groupby(\"user_answer\")['row_id'].count(). \\\nto_frame().sort_values(\"row_id\", ascending=False).rename({\"row_id\": \"count\"}, axis=1).reset_index().to_pandas()\n\n\ndf['user_answer'] = df['user_answer'].map({-1: \"lectures(-1)\", 0: \"choice-0\", 1: \"choice-1\", 2: \"choice-2\", 3: \"choice-3\", })\nfig = px.bar(df, x=\"user_answer\", y=\"count\",\n             log_y=True,\n                title=\"Bar Plot of user answer categoty wise count\",\n             color = \"user_answer\",\n            )\nfig.update_xaxes(categoryorder='array', categoryarray= [\"choice-0\",\"choice-1\",\"choice-2\",\"choice-3\", \"lectures(-1)\"])\n\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df['percentage'] = ((df['count'])/df['count'].sum()).round(4)*100\n\nfig = px.pie(df, values='percentage', names='user_answer', title='Pie chart of User answer')\nfig.update_layout(legend_title_text='user_answer')\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### prior_question_elapsed_time"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train.prior_question_elapsed_time.describe().to_frame().astype(int)\n\ntrain.prior_question_elapsed_time.describe().to_frame().astype(int)/1000.\n\nprint(\"percentage of missing values=\",((train.prior_question_elapsed_time.isna().sum()*100)/train.shape[0]).round(2))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df = train.sample(frac=0.02).to_pandas()\ndf['prior_question_elapsed_time(seconds)'] = df['prior_question_elapsed_time']/1000.\ndf = df.dropna()\n\nfig = px.histogram(df, x='prior_question_elapsed_time(seconds)', \n                  title=\"Histogram of prior_question_elapsed_time in seconds\",\n                   log_y=False, histnorm='probability',\n                   color='prior_question_had_explanation',\n                   range_x = [0, 75]\n                  )\n\n\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.box(df, x=\"prior_question_had_explanation\", y=\"prior_question_elapsed_time(seconds)\",\n             log_y=True,\n            \n                title=\"Box Plot of number of prior_question_had_explanation\",\n            )\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### prior_question_had_explanation"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df = train.groupby(['prior_question_had_explanation', 'content_type_id']).agg({\"row_id\": \"count\"}).reset_index().rename({\"row_id\": \"count\"}, axis=1).to_pandas()\ndf['content_type_id'] = df['content_type_id'].map({0: \"question\", 1: \"lecture\"})\n\nfig = px.histogram(df, x=\"prior_question_had_explanation\", y=\"count\",\n             log_y=False,\n                title=\"Bar Plot of user answer categoty wise count\",\n             color = \"content_type_id\",\n                   # histnorm=\"percent\",\n            )\n\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"del df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"6\"></a>\n## question.csv"},{"metadata":{},"cell_type":"markdown","source":"metadata for the questions posed to users.\n\n* ```question_id```: foreign key for the train/test content_id column, when the content type is question (0).\n\n* ```bundle_id```: code for which questions are served together.\n\n* ```correct_answer```: the answer to the question. Can be compared with the train user_answer column to check if the user was right.\n\n* ```part```: the relevant section of the TOEIC test.\n\n* ```tags```: one or more detailed tag codes for the question. The meaning of the tags will not be provided, but these codes are sufficient for clustering the questions together."},{"metadata":{"trusted":true},"cell_type":"code","source":"question.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"question.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### question_id"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.scatter(question.to_pandas(), y = \"question_id\",\n                title=\"Scatter Plot of question_id vs index\",\n            )\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### bundle_id"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df = question.groupby(['bundle_id'], as_index=False).agg({'question_id': 'nunique'}). \\\nsort_values('question_id', ascending=False).rename({'question_id': \"questions_per_bundle\"}, axis=1).to_pandas()\n\nfig = px.scatter(df, x = \"bundle_id\", y = \"questions_per_bundle\",\n                title=\"Scatter Plot of number of questions per bundle vs bundle_id\",\n            )\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"### correct_answer"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\n\ntemp = question.to_pandas().replace({\"correct_answer\":{0:\"choice-0\", 1:\"choice-1\",2:\"choice-2\",3:\"choice-3\"}},)\n\n# Create figure with secondary y-axis\nfig = make_subplots(specs=[[{\"secondary_y\": True}]])\n\n# Add traces\nfig.add_trace(\n    go.Histogram(x= temp.correct_answer, name=\"count\"),\n    secondary_y=False,\n)\n\nfig.add_trace(\n    go.Histogram(x=temp.correct_answer, name=\"percentage\", histnorm='percent'),\n    secondary_y=True,\n)\n\n# Add figure title\nfig.update_layout(\n    title_text=\"correct answer histogram\"\n)\n\n# Set x-axis title\nfig.update_xaxes(title_text=\"correct answer\")\n\n# Set y-axes titles\nfig.update_yaxes(title_text=\"count\", secondary_y=False)\nfig.update_yaxes(title_text=\"percentage\", secondary_y=True)\nfig.update_layout(showlegend=False)\nfig.update_traces(marker_color='green',)\n\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### part"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\n\ntemp = question.to_pandas()\ntemp = temp.replace({\"part\": {i: \"part_\" +str(i) for i in temp.part.unique()}},)\n\n# Create figure with secondary y-axis\nfig = make_subplots(specs=[[{\"secondary_y\": True}]])\n\n# Add traces\nfig.add_trace(\n    go.Histogram(x= temp.part, name=\"count\"),\n    secondary_y=False,\n)\n\nfig.add_trace(\n    go.Histogram(x=temp.part, name=\"percentage\", histnorm='percent'),\n    secondary_y=True,\n)\n\n# Add figure title\nfig.update_layout(\n    title_text=\"histogram of question part\"\n)\n\n# Set x-axis title\nfig.update_xaxes(title_text=\"part\")\n\n# Set y-axes titles\nfig.update_yaxes(title_text=\"count\", secondary_y=False)\nfig.update_yaxes(title_text=\"percentage\", secondary_y=True)\nfig.update_layout(showlegend=False)\nfig.update_traces(marker_color='green',)\n\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"question.groupby(\"bundle_id\").question_id.count().to_frame().sort_values(\"question_id\", ascending=False).head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"question[question.bundle_id==6940]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### tags"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"def encode_tags(x):\n    try:\n        tags = x['tags'].split(\" \")\n        return {i:1 for i in tags}\n    except:\n        return {}\n\nquestion_pd = question.to_pandas()\nquestion_pd['tag_encoded_dict'] = question_pd.apply(lambda x: encode_tags(x), axis=1)\n\ntmp = pd.DataFrame(question_pd.tag_encoded_dict.tolist()).fillna(0)\n\ntmp = tmp[tmp.columns.astype(int).sort_values().astype(str)]\nfrom sklearn.preprocessing import StandardScaler\n\nstd = StandardScaler()\ntmp = std.fit_transform(tmp)\n\n#tags_encoded_df = cudf.DataFrame.from_pandas(tmp)\n\n\ntsne = cuml.TSNE(n_components=2, perplexity=30, random_state=0, learning_rate=400)\nret = pd.DataFrame(tsne.fit_transform(tmp), index=question_pd.index, columns=[\"component_1\", \"component_2\"])\n\nret[\"part\"] = question_pd[\"part\"]\n\nret['part'] = \"part_\" + ret[\"part\"].astype(str)\n\nfig = px.scatter(ret, x=\"component_1\", y=\"component_2\",\n             log_y=False, log_x=False,\n                title=\"TSNE Plot of tags clustering\",\n                 color=\"part\",\n            )\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"<a id=\"7\"></a>\n## lectures.csv"},{"metadata":{},"cell_type":"markdown","source":"metadata for the lectures watched by users as they progress in their education.\n\n* ```lecture_id```: foreign key for the train/test content_id column, when the content type is lecture (1).\n\n* ```part```: top level category code for the lecture.\n\n* ```tag```: one tag codes for the lecture. The meaning of the tags will not be provided, but these codes are sufficient for clustering the lectures together.\n\n* ```type_of```: brief description of the core purpose of the lecture"},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"lectures.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### lecture_id"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig = px.scatter(lectures.to_pandas(), y = \"lecture_id\",\n                title=\"Scatter Plot of lecture_id vs index\", log_y=True,\n            )\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### part"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\n\ntemp = lectures.to_pandas()\ntemp = temp.replace({\"part\": {i: \"part_\" +str(i) for i in temp.part.unique()}},)\n\n# Create figure with secondary y-axis\nfig = make_subplots(specs=[[{\"secondary_y\": True}]])\n\n# Add traces\nfig.add_trace(\n    go.Histogram(x= temp.part, name=\"count\"),\n    secondary_y=False,\n)\n\nfig.add_trace(\n    go.Histogram(x=temp.part, name=\"percentage\", histnorm='percent'),\n    secondary_y=True,\n)\n\n# Add figure title\nfig.update_layout(\n    title_text=\"lectures part answer histogram\"\n)\n\n# Set x-axis title\nfig.update_xaxes(title_text=\"part\")\n\nfig.update_xaxes(categoryorder='array', categoryarray= [\"part_\"+str(i) for i in range(1,7)])\n\n# Set y-axes titles\nfig.update_yaxes(title_text=\"count\", secondary_y=False)\nfig.update_yaxes(title_text=\"percentage\", secondary_y=True)\nfig.update_layout(showlegend=False)\nfig.update_traces(marker_color='green',)\n\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### type_of"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import plotly.graph_objects as go\nfrom plotly.subplots import make_subplots\n\ntemp = lectures.to_pandas()\n# temp = temp.replace({\"part\": {i: \"part_\" +str(i) for i in temp.part.unique()}},)\n\n# Create figure with secondary y-axis\nfig = make_subplots(specs=[[{\"secondary_y\": True}]])\n\n# Add traces\nfig.add_trace(\n    go.Histogram(x= temp.type_of, name=\"count\"),\n    secondary_y=False,\n)\n\nfig.add_trace(\n    go.Histogram(x=temp.type_of, name=\"percentage\", histnorm='percent'),\n    secondary_y=True,\n)\n\n# Add figure title\nfig.update_layout(\n    title_text=\"lectures type_of answer histogram\"\n)\n\n# Set x-axis title\nfig.update_xaxes(title_text=\"part\")\n\nfig.update_xaxes(categoryorder='array', categoryarray= [\"part_\"+str(i) for i in range(1,7)])\n\n# Set y-axes titles\nfig.update_yaxes(title_text=\"count\", secondary_y=False)\nfig.update_yaxes(title_text=\"percentage\", secondary_y=True)\nfig.update_layout(showlegend=False)\nfig.update_traces(marker_color='green',)\n\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"### tags"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"df = lectures.to_pandas().groupby(\"tag\").lecture_id.count(). \\\nto_frame().sort_values(\"lecture_id\").rename({\"lecture_id\":\"count\"}, axis=1).reset_index()\n\nfig = px.bar(df, x = \"tag\",y=\"count\",\n                title=\"Bar Plot of tag vs count\", log_y=False,\n            )\nfig.show()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}