{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Intro\n오늘도 한가로운 심심한 나🫢\n\n인터넷에서 방황하다 [EdNet](https://github.com/riiid/ednet) 데이터 발견🤩\n\n우와! Education 데이터다\n\n(호기심 가득 가득) 오늘 미팅도 없는데 딴짓이나 해볼까?\n\n요즘 유행(?)한다는 *데이터 드리이이이븐 프로덕트*를 만들어봐야지!! \n\n이름부터 정해보자 음... [산타(Santa)](https://www.youtube.com/c/SANTA990)을 넘어설 **루돌프(Rudolph)**로 하겠어\n\n일단, 데이터부터 불러와야지","metadata":{}},{"cell_type":"markdown","source":"# 1편 데이터 불러오기","metadata":{}},{"cell_type":"markdown","source":"### DataSet\n- example_sample_submission.csv(971 B)\n- example_test.csv(6.14 KiB)\n- lectures.csv(9.7 KiB)\n- questions.csv(296.16 KiB)\n- train.csv(5.85 GiB)","metadata":{}},{"cell_type":"markdown","source":"데이터를 불러오자!!! 슝슝슝~\n\n어라~ 왜 이렇게 커?!?! 커도 너무 큰데\n\n전처리 하나 하는 순간 Ram이 터질 것만 같은 느낌?!\n\n오기가(5GiB)를 보니 오기가 생긴다 🧐\n\n용량 줄이는 법을 어디선가 본 기억이 (뒤적뒤적) 찾았다!! [여기 링크](https://www.kaggle.com/code/rohanrao/tutorial-on-reading-large-datasets) \n\n역시 캐글에는 너무 친절한 분들이 많아 🙏🙏🙏\n\n대충 읽어보니 4가지나 있네. 난 판다스 밖에 모르니깐 무조건 판다스 Pick","metadata":{}},{"cell_type":"code","source":"# %%time\n# import pandas as pd\n\n# dtypes = {\n#     \"row_id\": \"int64\",\n#     \"timestamp\": \"int64\",\n#     \"user_id\": \"int32\",\n#     \"content_id\": \"int16\",\n#     \"content_type_id\": \"boolean\",\n#     \"task_container_id\": \"int16\",\n#     \"user_answer\": \"int8\",\n#     \"answered_correctly\": \"int8\",\n#     \"prior_question_elapsed_time\": \"float32\", \n#     \"prior_question_had_explanation\": \"boolean\"\n# }\n\n# train = pd.read_csv(\"../input/riiid-test-answer-prediction/train.csv\", dtype=dtypes)\n# print(\"Train size:\", train.shape)","metadata":{"execution":{"iopub.status.busy":"2022-06-16T09:18:50.632350Z","iopub.execute_input":"2022-06-16T09:18:50.634015Z","iopub.status.idle":"2022-06-16T09:18:50.657689Z","shell.execute_reply.started":"2022-06-16T09:18:50.633841Z","shell.execute_reply":"2022-06-16T09:18:50.656794Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"데이터는 5분 뒤 공개합니다 ㅠㅠ\n\n이럴 수가!!! 데이터 로드에 5분?? 이건 아니야 ~\n\n더 좋은 방법이 있다고? \n\n**멋진 동료**: 데이터 셋을 (피클 형태로 저장해서 그 피클을 불러오면) 20초 만에 불러올 수 있습니다.\n\n20초 만에?? 대에박~ 빨리 알려주세요. 커피 쏠게요🧋","metadata":{}},{"cell_type":"code","source":"%%time\n# train 데이터 불러오기\nimport pandas as pd\ntrain = pd.read_feather(\"../input/riiid-train-data-multiple-formats/riiid_train.feather\")\nprint(\"Train size:\", train.shape)","metadata":{"execution":{"iopub.status.busy":"2022-06-16T09:18:50.659303Z","iopub.execute_input":"2022-06-16T09:18:50.659894Z","iopub.status.idle":"2022-06-16T09:19:17.721354Z","shell.execute_reply.started":"2022-06-16T09:18:50.659851Z","shell.execute_reply":"2022-06-16T09:19:17.720166Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"제로백 20초 붕붕카를 탄 날아갈 것만 같은 이 가벼운 기분 😍\n\n멋진 동료야~ 너무 고마워 👏👏👏","metadata":{}},{"cell_type":"code","source":"%%time\n# 남은 4개 데이터 불러오기\nquestions = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/questions.csv')\nlectures = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/lectures.csv')\nexample_test = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/example_test.csv')\nexample_sample_submission = pd.read_csv('/kaggle/input/riiid-test-answer-prediction/example_sample_submission.csv')","metadata":{"execution":{"iopub.status.busy":"2022-06-16T09:19:17.722653Z","iopub.execute_input":"2022-06-16T09:19:17.723243Z","iopub.status.idle":"2022-06-16T09:19:17.766601Z","shell.execute_reply.started":"2022-06-16T09:19:17.723205Z","shell.execute_reply":"2022-06-16T09:19:17.765482Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 2편 데이터 살펴보기","metadata":{}},{"cell_type":"markdown","source":"AI 교육 시간에 배웠던걸 써먹어 봐야지!\n\n탐색적 데이터 분석이라고 했던가?? \n\n데이터를 처음 마주한 기분!! 너무 좋아 😃","metadata":{}},{"cell_type":"code","source":"# 데이터 샘플\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-16T09:19:17.768926Z","iopub.execute_input":"2022-06-16T09:19:17.769288Z","iopub.status.idle":"2022-06-16T09:19:17.793102Z","shell.execute_reply.started":"2022-06-16T09:19:17.769255Z","shell.execute_reply":"2022-06-16T09:19:17.792108Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 데이터 타입 및 메모리 확인\ntrain.info()","metadata":{"execution":{"iopub.status.busy":"2022-06-16T09:19:17.794213Z","iopub.execute_input":"2022-06-16T09:19:17.794539Z","iopub.status.idle":"2022-06-16T09:19:17.812914Z","shell.execute_reply.started":"2022-06-16T09:19:17.794510Z","shell.execute_reply":"2022-06-16T09:19:17.811812Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"캐글에서 주어지는 RAM용량은 12.4GB인데 train만 불러왔을 뿐인데 4Gb ㅠㅠ \n\n1/3을 벌써 써버리다니...\n\n**멋진 동료**: 'object타입' 확인해보고 타입을 변경해보세요!! 그럼 용량이 줄어들걸요?\n\n오~~ 오브젝트 타입??","metadata":{}},{"cell_type":"code","source":"# prior_question_had_explanation 컬럼의 카테고리별 데이터 수\ntrain['prior_question_had_explanation'].value_counts()","metadata":{"execution":{"iopub.status.busy":"2022-06-16T09:19:17.814389Z","iopub.execute_input":"2022-06-16T09:19:17.814892Z","iopub.status.idle":"2022-06-16T09:19:30.656537Z","shell.execute_reply.started":"2022-06-16T09:19:17.814845Z","shell.execute_reply":"2022-06-16T09:19:30.655003Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"아 뭐야! True/False 밖에 없네!! 그럼 이걸 용량 작은 boolean 타입으로 변경해야지","metadata":{}},{"cell_type":"code","source":"# prior_question_had_explanation 컬럼 타입 변경\ntrain['prior_question_had_explanation'] = train['prior_question_had_explanation'].astype('boolean')\ntrain.info()","metadata":{"execution":{"iopub.status.busy":"2022-06-16T09:19:30.659081Z","iopub.execute_input":"2022-06-16T09:19:30.659622Z","iopub.status.idle":"2022-06-16T09:20:05.126062Z","shell.execute_reply.started":"2022-06-16T09:19:30.659549Z","shell.execute_reply":"2022-06-16T09:20:05.125014Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"좋아 좋아!! memory usage: 3.7+ GB 에서 memory usage: 3.1 GB 축소되었다\n\n나머지 데이터 들도 대충 살펴보고 데이터 전처리 시작","metadata":{}},{"cell_type":"markdown","source":"# 3편 데이터 전처리","metadata":{}},{"cell_type":"markdown","source":"멋진 루돌프 프로덕트를 생각하며, 단디, 매매, 꼭꼭 데이터 전처리를 해보려고 하는데 \n\n으악!!!계속해서\"램 초과\"메세지가 ㅠㅠ \n\n~~~\nYour notebook tried to allocate more memory than is available. It has restarted\n~~~\n\n일이나 할 걸 그랬나 ㅎㅎ\n\n어디선가 본 애자일 방식에 따르면 프로토타입이 중요하니 과감히 샘플링을!! \n\n이 모델 성공하면 대표님이 새로 나온 M2가 달린 메모리 24GB 정도 되는 빵빵한 장비 사주시겠지? (김칫국 벌컥) 기대기대 🤩","metadata":{}},{"cell_type":"code","source":"# 앞뒤 돌아보지 않는 과감한 샘플링\nprint(train.shape)\ntrain = train[:int(train.shape[0]/5)]\nprint(train.shape)","metadata":{"execution":{"iopub.status.busy":"2022-06-16T09:20:05.127453Z","iopub.execute_input":"2022-06-16T09:20:05.127826Z","iopub.status.idle":"2022-06-16T09:20:05.134995Z","shell.execute_reply.started":"2022-06-16T09:20:05.127791Z","shell.execute_reply":"2022-06-16T09:20:05.133493Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"전처리 코드는 일단 고수님 코드 필사를... 공유해주셔서 감사합니다 :) \n\n\nhttps://www.kaggle.com/code/erikbruin/riiid-comprehensive-eda-baseline","metadata":{}},{"cell_type":"code","source":"%%time\n\nimport numpy as np\n#adding user features\nuser_df = train[train.answered_correctly != -1].groupby('user_id').agg({'answered_correctly': ['count', 'mean']}).reset_index()\nuser_df.columns = ['user_id', 'user_questions', 'user_mean']\n\nuser_lect = train.groupby([\"user_id\", \"answered_correctly\"]).size().unstack()\nuser_lect.columns = ['Lecture', 'Wrong', 'Right']\nuser_lect = user_lect[['Lecture']].fillna(0).astype('int8')\n#user_lect = user_lect.astype('int8')\nuser_lect['watches_lecture'] = np.where(user_lect.Lecture > 0, 1, 0)\nuser_lect = user_lect.reset_index()\n\nuser_df = user_df.merge(user_lect, on = \"user_id\", how = \"left\")\ndel user_lect\nuser_df.head()","metadata":{"execution":{"iopub.status.busy":"2022-06-16T09:20:05.137000Z","iopub.execute_input":"2022-06-16T09:20:05.138065Z","iopub.status.idle":"2022-06-16T09:20:08.805419Z","shell.execute_reply.started":"2022-06-16T09:20:05.138012Z","shell.execute_reply":"2022-06-16T09:20:08.804238Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"cols = ['row_id', 'user_id', 'answered_correctly', 'content_id', 'prior_question_had_explanation', 'prior_question_elapsed_time']\ntrain = train[cols]","metadata":{"execution":{"iopub.status.busy":"2022-06-16T09:20:08.809235Z","iopub.execute_input":"2022-06-16T09:20:08.811385Z","iopub.status.idle":"2022-06-16T09:20:09.062450Z","shell.execute_reply.started":"2022-06-16T09:20:08.811313Z","shell.execute_reply":"2022-06-16T09:20:09.061400Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n#adding content features\ncontent_df = train[train.answered_correctly != -1].groupby('content_id').agg({'answered_correctly': ['count', 'mean']}).reset_index()\ncontent_df.columns = ['content_id', 'content_questions', 'content_mean']\ncontent_df.head(1)","metadata":{"execution":{"iopub.status.busy":"2022-06-16T09:20:09.063465Z","iopub.execute_input":"2022-06-16T09:20:09.063809Z","iopub.status.idle":"2022-06-16T09:20:10.785538Z","shell.execute_reply.started":"2022-06-16T09:20:09.063767Z","shell.execute_reply":"2022-06-16T09:20:10.784520Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# 이렇게 나눠도 되려나??\nfrom sklearn.model_selection import train_test_split\n\ntrain = train.drop(columns=\"row_id\")\ntrain, validation = train_test_split(train, test_size=0.2, random_state=2022)\n\nmean_prior = train.prior_question_elapsed_time.astype(\"float64\").mean()","metadata":{"execution":{"iopub.status.busy":"2022-06-16T09:20:10.786996Z","iopub.execute_input":"2022-06-16T09:20:10.788067Z","iopub.status.idle":"2022-06-16T09:20:16.034590Z","shell.execute_reply.started":"2022-06-16T09:20:10.788021Z","shell.execute_reply":"2022-06-16T09:20:16.033407Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\n\nlabel_enc = LabelEncoder()\n\ntrain = train.merge(user_df, on = \"user_id\", how = \"left\")\ntrain = train.merge(content_df, on = \"content_id\", how = \"left\")\ntrain['content_questions'].fillna(0, inplace = True)\ntrain['content_mean'].fillna(0.5, inplace = True)\ntrain['watches_lecture'].fillna(0, inplace = True)\ntrain['user_questions'].fillna(0, inplace = True)\ntrain['user_mean'].fillna(0.5, inplace = True)\ntrain['prior_question_elapsed_time'].fillna(mean_prior, inplace = True)\ntrain['prior_question_had_explanation'].fillna(False, inplace = True)\nlabel_enc.fit(train['prior_question_had_explanation'])\ntrain['prior_question_had_explanation'] = label_enc.transform(train['prior_question_had_explanation'])\ntrain[['content_questions', 'user_questions']] = train[['content_questions', 'user_questions']].astype(int)\ntrain.sample(5)","metadata":{"execution":{"iopub.status.busy":"2022-06-16T09:20:16.035957Z","iopub.execute_input":"2022-06-16T09:20:16.036441Z","iopub.status.idle":"2022-06-16T09:20:31.626709Z","shell.execute_reply.started":"2022-06-16T09:20:16.036402Z","shell.execute_reply":"2022-06-16T09:20:31.625625Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"validation = validation.merge(user_df, on = \"user_id\", how = \"left\")\nvalidation = validation.merge(content_df, on = \"content_id\", how = \"left\")\nvalidation['content_questions'].fillna(0, inplace = True)\nvalidation['content_mean'].fillna(0.5, inplace = True)\nvalidation['watches_lecture'].fillna(0, inplace = True)\nvalidation['user_questions'].fillna(0, inplace = True)\nvalidation['user_mean'].fillna(0.5, inplace = True)\nvalidation['prior_question_elapsed_time'].fillna(mean_prior, inplace = True)\nvalidation['prior_question_had_explanation'].fillna(False, inplace = True)\nvalidation['prior_question_had_explanation'] = label_enc.transform(validation['prior_question_had_explanation'])\nvalidation[['content_questions', 'user_questions']] = validation[['content_questions', 'user_questions']].astype(int)\nvalidation.sample(5)","metadata":{"execution":{"iopub.status.busy":"2022-06-16T09:20:31.628208Z","iopub.execute_input":"2022-06-16T09:20:31.628679Z","iopub.status.idle":"2022-06-16T09:20:34.734598Z","shell.execute_reply.started":"2022-06-16T09:20:31.628630Z","shell.execute_reply":"2022-06-16T09:20:34.733457Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"features = ['user_questions', 'user_mean', 'content_questions', 'content_mean', 'prior_question_elapsed_time']\n\ny_train = train['answered_correctly']\ntrain = train[features]\n\ny_val = validation['answered_correctly']\nvalidation = validation[features]","metadata":{"execution":{"iopub.status.busy":"2022-06-16T09:20:34.735749Z","iopub.execute_input":"2022-06-16T09:20:34.736078Z","iopub.status.idle":"2022-06-16T09:20:35.038861Z","shell.execute_reply.started":"2022-06-16T09:20:34.736050Z","shell.execute_reply":"2022-06-16T09:20:35.037925Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"복잡한 전처리는 끝냈고,\n\n머신러닝 모델을 써야 하는데\n\n루돌프는 200마력 정도면 썰매를 끌 수 있을 거야\n\n기본이 100마력이니 200마력으로 업그레이드 (num_boost_round=200) \n\n부스터~ 빠라바라바라밤⚡️\n","metadata":{}},{"cell_type":"markdown","source":"### 머신러닝 모델","metadata":{}},{"cell_type":"markdown","source":"머신러닝 모델 중 성능 좋고 빠르다던 라잇찌비엠(lightGBM)으로 돌려 돌려!!","metadata":{}},{"cell_type":"code","source":"params = {'objective': 'binary',\n          'metric': 'auc',\n          'seed': 2022,\n         }","metadata":{"execution":{"iopub.status.busy":"2022-06-16T09:20:35.040287Z","iopub.execute_input":"2022-06-16T09:20:35.040748Z","iopub.status.idle":"2022-06-16T09:20:35.046022Z","shell.execute_reply.started":"2022-06-16T09:20:35.040703Z","shell.execute_reply":"2022-06-16T09:20:35.045044Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import lightgbm as lgb\nimport gc\n\nlgb_train = lgb.Dataset(train, y_train, categorical_feature = None)\nlgb_eval = lgb.Dataset(validation, y_val, categorical_feature = None)\ndel train, y_train, validation, y_val\ngc.collect()","metadata":{"execution":{"iopub.status.busy":"2022-06-16T09:20:35.047640Z","iopub.execute_input":"2022-06-16T09:20:35.048434Z","iopub.status.idle":"2022-06-16T09:20:36.207544Z","shell.execute_reply.started":"2022-06-16T09:20:35.048394Z","shell.execute_reply":"2022-06-16T09:20:36.206449Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\nmodel = lgb.train(\n    params, lgb_train,\n    valid_sets=[lgb_train, lgb_eval],\n    verbose_eval=10,\n    num_boost_round=200,\n    early_stopping_rounds=5\n)","metadata":{"execution":{"iopub.status.busy":"2022-06-16T09:20:36.209303Z","iopub.execute_input":"2022-06-16T09:20:36.209678Z","iopub.status.idle":"2022-06-16T09:34:05.386571Z","shell.execute_reply.started":"2022-06-16T09:20:36.209645Z","shell.execute_reply":"2022-06-16T09:34:05.384730Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"음하하하~ 돌아간다. (워닝은 일단 무시) \n\n이거면 투자도 받고 ai 교육을 혁신할 수 있을 것 같아 역사에 길이길이 남을 패러담임 변화를 도전해 보겠어!! 한 산업을 변화시킬거야! 하하하🔥\n\n(멋진 동료에게 피드백 받는 중)\n\n뭐라고? 서비스하기에는 성능이 부족하다고? 데이터에 대한 인사이트가 더 필요하다고? \n\n산타는 Knowledge Tracing 모델을 활용해 예측하고 이를 발전하기 위해...\n\n놀리지 트래싱?","metadata":{}},{"cell_type":"markdown","source":"좋아! 그럼 난 놀리지 트래싱에 요즘 뜬다는 트랜스포머를 더해서 적용을??\n\n여기서 말하는 트랜스포머는 변신 로봇 아님!! 머신러닝 모델(인코더와 디코더로 구성되고, 셀프 어텐선과... 블라블라~)\n\n그 유명하다던  [Knowledge Tracing](https://medium.com/riiid-teamblog-kr/%EA%B5%90%EC%9C%A1ai%EC%9D%98-%EA%B8%B0%EB%B3%B8%EC%9D%B4%EC%9E%90-%EC%8B%9C%EC%9E%91-deep-knowledge-tracing-dkt-8bc132eda9ec)과 [Attention is All You Need](https://arxiv.org/abs/1706.03762) 논문을 읽는 중\n\n좋은 거구나 그런데 무슨 말인지 모르겠다\n\n외부에서 뛰어난 사람을 영입 해야지!\n\n루돌프 데이터 전처리, 모델 파이프라인 설계, 모델 서빙해 줄 [머신러닝 엔지니어](https://riiid.com/en/career/machine-learning-engineer)를 모십니다.🍓","metadata":{}},{"cell_type":"markdown","source":"영입 될 때까진 내가 더 개선해봐야지!","metadata":{}},{"cell_type":"markdown","source":"오늘 하루 딴짓한 것을 무마하려고 채용링크라도 땋!!! 열 일하는 척하는 데브렐 매니저 입니다.\n\n조금만 기다려줘요. 후편을 이어갈게요\n\n*참고: 본 코드와 내용은 뤼이드 ai 조직과 전혀 관련 없는 코드 입니다.*","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}