{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Introduction\n\nThis notebook aims at presenting my final solution for the Riid competition.\n\nAs I was limited in time, I decided to stay simple and go with single LGBM + a lot of feature engineering.\n\nCredit to Steve Majou for its notebook [Simple Elo Rating](https://www.kaggle.com/stevemju/riiid-simple-elo-rating) that gave me a very nice boost\n\n# Plan\n\n## 1. Model and feature importance\n\n## 2. Feature exploration\n\n## 3. Prediction pipeline"},{"metadata":{"trusted":true},"cell_type":"code","source":"import warnings\nwarnings.filterwarnings('ignore')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import pickle\nimport riiideducation\nfrom tqdm import tqdm \nimport glob\nimport pandas as pd\nimport numpy as np\n\nimport plotly.graph_objects as go","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Model and feature importance\n\nI wanted to get over the competition with a single model. \n\nThe model is a simple lightgbm classifier trained on 22 millions rows and tested on 3 millions rows.\nMy final model is using 38 features, but there is really few gain through the last features. If I had to make a trade off I could have probably stayed with 30 features only without loosing much accuracy"},{"metadata":{"trusted":true},"cell_type":"code","source":"metrics = [\n    'questions_part',\n    'questions_difficulty',\n    'questions_count',\n    'questions_good_answer_reaction_time',\n    'questions_bad_answer_reaction_time',\n    'questions_batch_size',\n    'questions_variance',\n    'questions_difficulty_given_prev_explanation',\n    'questions_difficulty_given_prev_no_explanation',\n    'questions_conditionnal_proba_cluster',\n    'lectures_count',\n    'student_general_elo_theta',\n    'student_general_elo_proba',\n    'student_part_elo_proba',\n    'student_tags_elo_theta',\n    'student_tags_elo_proba',\n    'student_part_score',\n    'timestamp_student_general_last_batch_average',\n    'timestamp_student_general_reactivity_rolling_19',\n    'timestamp_student_general_part_reactivity_average',\n    'timestamp_student_general_reactivity_good_answer_average',\n    'timestamp_last_delta_between_questions',\n    'timestamp_delta_between_questions_rolling_10',\n    'timestamp_delta_between_questions_rolling_5',\n    'timestamp_last_time_between_answers',\n    'student_general_good_answers_average',\n    'student_general_good_answers_rolling_2',\n    'student_general_good_answers_rolling_6',\n    'student_part_good_answers_rolling_6',\n    'questions_seen_with_explanation',\n    'questions_conditionnal_count',\n    'questions_conditionnal_proba',\n    'lectures_conditionnal_proba',\n]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"with open('../input/lgb-training/model.pkl', 'rb') as f:\n    lgb = pickle.load(f)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Feature importance, lgb\n\nBelow is a plot of the feature importance as per lgb classification"},{"metadata":{"trusted":true},"cell_type":"code","source":"lgb_importance = pd.Series(lgb.feature_importances_, index = metrics).sort_values()\n\nfig = go.Figure(\n    go.Bar(\n        y = lgb_importance.index,\n        x = lgb_importance.values, \n        orientation='h'\n    )\n)\nfig.update_layout(\n    title = 'LGB feature importance',\n    template = 'presentation', \n    height =1000,\n    margin=dict(\n        l=400,\n        r=50,\n        b=100,\n        t=100,\n        pad=4\n    ),\n)\nfig.update_yaxes(tickfont = dict(size=13))\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Average contribution by thematic"},{"metadata":{"trusted":true},"cell_type":"code","source":"grouper = [elmt.split('_')[0] for elmt in lgb_importance.index]\ncategory_importance = lgb_importance.groupby(grouper).mean().sort_values(ascending = False)\n\nfig = go.Figure(\n    go.Bar(\n        x = category_importance.index,\n        y = category_importance.values, \n    )\n)\nfig.update_layout(\n    title = 'LGB feature importance, grouped by category',\n    template = 'presentation', \n    height =400,\n    margin=dict(\n        l=50,\n        r=50,\n        b=100,\n        t=100,\n        pad=4\n    ),\n)\nfig.update_yaxes(tickfont = dict(size=13))\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Average contribution by \"Tag, Part, General\""},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_cat = np.array([1 if 'general' in elmt else 0 for elmt in metrics ]) + \\\nnp.array([2 if 'part' in elmt else 0 for elmt in metrics]) + \\\nnp.array([3 if 'tag' in elmt else 0 for elmt in metrics])\n\ncategory_importance_2 = lgb_importance.groupby(sub_cat).mean().loc[[1,2,3]]\ncategory_importance_2.index = ['general','part','tags']\ncategory_importance_2 = category_importance_2.sort_values(ascending = False)\n\nfig = go.Figure(\n    go.Bar(\n        x = category_importance_2.index,\n        y = category_importance_2.values, \n    )\n)\nfig.update_layout(\n    title = 'LGB feature importance, grouped by sub category',\n    template = 'presentation', \n)\nfig.update_yaxes(tickfont = dict(size=13))\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2 - Feature Creation\n\nIn this part I will detail a bit more the way I calculated some of the features.\nI am using a sample of 10 million data here to make it light, but during my own inference I used the full dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('../input/riiid-test-answer-prediction/train.csv', nrows = 1000000, index_col = 0)\nquestions = pd.read_csv('../input/riiid-test-answer-prediction/questions.csv')\nlectures = pd.read_csv('../input/riiid-test-answer-prediction/lectures.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### General features about questions\n\nIn this part, nothing fancy, I used the generic pandas functions to compute some general statistics about questions (difficulty, avg_reaction_time, etc...)\n\nI then saved each statistic in each own dictionnary, that I could call when needed.\n\nWhat could have been done better: including all the statistic in a general dictionnary rather than having one dictionnary for each"},{"metadata":{},"cell_type":"markdown","source":"#### Question difficulty"},{"metadata":{},"cell_type":"markdown","source":"Question to difficulty is only the average answer good answers of users, reversed so that difficult questions have a high score and easy questions a low one. \nI made this inversion because I am later using an asymetrical score function that attribute points to users answer hard questions and failing easy quesytions.\n\nPossible improvement: a threshold system for questions that are too close to 0 or to 1. Also remove score for questions with not enough statistical significance (too few people answering those questions)"},{"metadata":{"trusted":true},"cell_type":"code","source":"questions_to_difficulty = (1-train[train.content_type_id == False].groupby('content_id')['answered_correctly'].mean())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import seaborn as sns\nimport matplotlib.pyplot as plt\n\nplt.figure(figsize = (12,5))\nplt.title('Question difficulty')\nsns.distplot(questions_to_difficulty.values)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To compute the average reaction time of user when they answer correctly or not, we first have to shift the prior_question_reaction time as it refere to the previous row"},{"metadata":{},"cell_type":"markdown","source":"#### Questions reaction times"},{"metadata":{"trusted":true},"cell_type":"code","source":"subtrain = train[train.content_type_id ==0]\nsubtrain['shift_elapse_time'] = subtrain['prior_question_elapsed_time'].shift(-1)\ngood_answer_reaction_time = subtrain[subtrain.answered_correctly==True].dropna().groupby('content_id')['shift_elapse_time'].mean()\nbad_answer_reaction_time = subtrain[subtrain.answered_correctly==False].dropna().groupby('content_id')['shift_elapse_time'].mean()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize = (12,5))\nplt.title('Question difficulty')\nsns.distplot(good_answer_reaction_time.values, label = 'good answer reaction time')\nsns.distplot(bad_answer_reaction_time.values, label = 'bad answer reaction time')\nplt.legend()\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Conditionnal probabilities\n\nThis part is very interesting as it includes some of the features that gave me the nicest boost.\nThe objectif was to infere Bayesian probability of user answering correctly given their previous answers.\n\nI applyied the same approach for the lectures"},{"metadata":{},"cell_type":"markdown","source":"#### Selection of questions to use to compute the Bayesian probabilities\n\nUsing all questions would have been way to heavy both in computationnal time and in RAM. I decided to limit this part to the top 500 questions the most recurrent. An optimal value would have probably be around 1000 questions but I didn't have really time to try it."},{"metadata":{"trusted":true},"cell_type":"code","source":"(unique, counts) = np.unique(train.values[:,1], return_counts=True)\nqids = pd.Series(counts, index = unique).sort_values(ascending=False)[:500]\nqids.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#We just keep the questions id\nqids = qids.index","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"#### Challenge\n\nNow the main challenge here is to deal all the different possibilities.\n\nWe want to build a dictionnary of the shape {A1: {B1: P_A1B1_1}, {B2: P_A1B2_1}, ...}, where P_A1B2_1 is the bayesian probability P(A=1|B=1) for each question An in the questions table and for each question B in the top 500 questions most recurrent\n\nAlso, I wanted to include a notion of memory: when a user is seeing too many questions, he shall start to forget the impact of the previous ones. \n\nFinally we want the feature creation not to be time consuming nor ram consuming, so we have to use as few loop as possible and try to optimize the RAM usage. \n\n#### Solution\n\nI came up with a solution that requiere only a single loop:\n- Every new user, I generate a new user_meta dictionnary to store the history of a given user\n- I also create 4 global dictionnary to calculate general occurences of even A (question seen) and the conditionnal probability count. Note: I don't do the division in this part, this is done later based on those inputs\n- I then loop one time through the training set. Every time a user answer a question, if the question belong to the frequen questions, I add the answer to the \"memory\" of this user. For every row, I check if the user already saw the question, and in that case, I add the values to my conditionnal dictionnaries.\n- After the update of the dictionnary, I do an update of the memory of the user based on the current question. \n- Finally, I increment the time of each questions by 1, once a question has reach the maximum memory, it is reset to 0\n\nNote:\n- You can see I am iterating through the array rather than the pandas dataframe: it reduce a lot time processing\n- We transform the answered values to -1 or 1, it is then easier to make the filter for the previous questions seen\n\nThe code makes about 7000 iterations per second, so it takes about 5 hours to run on the full dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"from tqdm import tqdm\n\nmemory = 100\ntrain_arr = train[['user_id', 'content_id', 'content_type_id', 'answered_correctly']].values\n# The different metadata dictionnaries used in this part\nuser_meta = {}\nuser_meta_count = {}\nmeta_proba_pos = {}\nmeta_count_pos = {}\nmeta_proba_neg = {}\nmeta_count_neg = {}\n\n#I iterate through\nfor i in tqdm(range(len(train))):\n    \n    user_id, content_id, content_type_id, answered_correctly = train_arr[i]\n    \n    #If a new user is detected, we reset the user cache to save RAM\n    if user_id not in user_meta.keys():\n        user_meta = {}\n        user_meta_count = {}\n        user_meta[user_id] = {qid:0 for qid in qids}\n        user_meta_count[user_id] = {}\n        \n    #Then: update the questions dictionnaries based on the user cache\n    if content_type_id ==0:\n        A = content_id\n        X = np.array(list(user_meta[user_id].values()))\n        #If a new question is detected, we create a slot in our dictionnaries\n        if A not in meta_proba_pos.keys():\n            meta_proba_pos[A] = {i:0 for i in range(len(qids))}\n            meta_count_pos[A] = {i:0 for i in range(len(qids))}\n            meta_proba_neg[A] = {i:0 for i in range(len(qids))}\n            meta_count_neg[A] = {i:0 for i in range(len(qids))}\n         \n        #We check all previous answer of a user and update the respective dictionnary P(A=1|B=1) and P(A=0|B=0)\n        for B in np.where(X!=0)[0]:\n            #P(A=1|B=1)\n            if (X[B]==1):\n                if answered_correctly == 1:\n                    meta_proba_pos[A][B] += 1\n                meta_count_pos[A][B] += 1\n            #P(A=1|B=0)\n            if (X[B]==-1):\n                if answered_correctly == 1:\n                    meta_proba_neg[A][B] += 1\n                meta_count_neg[A][B] += 1             \n    \n    #update\n    if A in qids:\n        \n        for k,v in user_meta_count[user_id].copy().items():\n            #Reduce the memory clock, when it reach 0, reset the value to 0 for the given question\n            user_meta_count[user_id][k] = v-1\n            if v == 0:\n                del user_meta_count[user_id][k]\n                user_meta[user_id][k] = 0\n            \n        user_meta[user_id][content_id] = 2*answered_correctly-1\n        #Add the question to the memory of a user if not seen yet\n        if not A in user_meta_count[user_id].keys():\n            user_meta_count[user_id][A] = memory\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's load the full dictionnaries !"},{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\nimport pickle\n\nwith open('../input/riid-raw-samples/bad_answer_reaction_time', 'rb') as f:\n    bad_answer_reaction_time = pickle.load(f)\n    \nwith open('../input/riid-raw-samples/good_answer_reaction_time', 'rb') as f:\n    good_answer_reaction_time = pickle.load(f)\n    \nwith open('../input/riid-raw-samples/question_user_count', 'rb') as f:\n    question_user_count = pickle.load(f)\n\nwith open('../input/riid-raw-samples/questions_to_difficulty', 'rb') as f:\n    questions_to_difficulty = pickle.load(f)\n\nwith open('../input/riid-raw-samples/questions_to_difficulty', 'rb') as f:\n    questions_to_difficulty = pickle.load(f)\n    \nwith open('../input/riid-raw-samples/questions_to_difficulty_variance', 'rb') as f:\n    questions_to_difficulty_variance = pickle.load(f)\n    \nwith open('../input/riid-raw-samples/questions_to_difficulty_with_explanation', 'rb') as f:\n    questions_to_difficulty_with_explanation = pickle.load(f)\n    \nwith open('../input/riid-raw-samples/questions_to_difficulty_without_explanation', 'rb') as f:\n    questions_to_difficulty_without_explanation = pickle.load(f)\n    \nwith open('../input/riid-probabilities-2/meta_count.pkl', 'rb') as f:\n    meta_count = pickle.load(f)\n    \nwith open('../input/riid-probabilities-2/meta_count_neg.pkl', 'rb') as f:\n    meta_count_neg = pickle.load(f)\n    \nwith open('../input/riid-probabilities-2/meta_proba.pkl', 'rb') as f:\n    meta_proba = pickle.load(f)\n    \nwith open('../input/riid-probabilities-2/meta_proba_neg.pkl', 'rb') as f:\n    meta_proba_neg = pickle.load(f)\n    \nwith open('../input/riid-probabilities/question_position.pkl', 'rb') as f:\n    question_position = pickle.load(f)   \n    \nwith open('../input/riid-lectures/lecture_position.pkl', 'rb') as f:\n    lecture_position = pickle.load(f)\n    \nwith open('../input/riid-lectures/meta_count_lect.pkl', 'rb') as f:\n    meta_count_lect = pickle.load(f)\n    \nwith open('../input/riid-lectures/meta_proba_lect.pkl', 'rb') as f:\n    meta_proba_lect = pickle.load(f)\n    \nclust = pd.read_csv('../input/riid-cluster/test_cluster_4',index_col=0)['0']\n\nwith open('../input/riid-cache-questions/cache_questions.pck', 'rb') as f:\n    question_cache = pickle.load(f)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Given the global counts and the global counts for the probabilities, we can now calculate the bayesian probability.\nI am also setting a threshold of occurence as well as a treshold of significancy. Those values are hyperparameters of the problem"},{"metadata":{"trusted":true},"cell_type":"code","source":"proba_pos = {}\nproba_neg = {}\nTHRESH_COUNT = 50\nTHRESH_PROBA = 0.01\nfor A,v in tqdm(meta_proba.items()):\n    proba_pos[A] = {}\n    proba_neg[A] = {}\n    for B in v.keys():\n        if meta_count[A][B]>THRESH_COUNT:\n            proba = meta_proba[A][B]/meta_count[A][B]\n            improve = proba - (1-questions_to_difficulty[A])\n            if improve > THRESH_PROBA:\n                proba_pos[A][question_position[B]] = improve\n        if meta_count_neg[A][B]>THRESH_COUNT:\n            proba = meta_proba_neg[A][B]/meta_count_neg[A][B]\n            decrease = proba - (1-questions_to_difficulty[A])\n            if decrease < -THRESH_PROBA:\n                proba_neg[A][question_position[B]] = decrease\n                \n                \n#Same job for the lectures !\nproba_lec = {}\nthresh = 20\nfor k,v in tqdm(meta_proba_lect.items()):\n    proba_lec[k] = {}\n    for kk,vv in v.items():\n        if meta_count_lect[k][kk]>thresh:\n            proba = meta_proba_lect[k][kk]/meta_count_lect[k][kk]\n            improve = proba - (1-questions_to_difficulty[k])\n            if improve > 0.10:\n                proba_lec[k][lecture_position[kk]] = improve","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"example = pd.Series(proba_pos[7684]).sort_values(ascending = False)\nexample.index = ['q'+str(elmt) for elmt in example.index]\n\nfig = go.Figure(\n    go.Bar(\n        x = example.index,\n        y = example.values, \n    )\n)\nfig.update_layout(\n    title = 'Example of bayesian effet - Gain over difficulty metric',\n    template = 'presentation', \n)\nfig.update_yaxes(tickfont = dict(size=13))\nfig.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Clustering\n\nThose Bayesian values can be also used to find similarity between questions."},{"metadata":{"trusted":true},"cell_type":"code","source":"all_prob = {}\nfor k,v in proba_pos.items():\n    all_prob[k] = {}\n    for kk,vv in v.items():\n        all_prob[k][(kk,1)] = vv\nfor k,v in proba_neg.items():\n    if k not in all_prob.keys():\n        all_prob[k] = {}\n    for kk,vv in v.items():\n        all_prob[k][(kk,0)] = vv\n\nall_q = set()\nfor k,v in all_prob.items():\n    for kk in v.keys():\n        all_q.add(kk[0])\n        \nmprob1 = {}\nfor k,v in tqdm(all_prob.items()):\n    mprob1[k] = {}\n    kk = [e[0] for e in v.keys()]\n    for elmt in all_q:\n        mprob1[k][elmt]=0\n        if elmt in kk:\n            if (elmt,1) in v.keys():\n                mprob1[k][elmt]=v[(elmt,1)]\n            else:\n                mprob1[k][elmt]=v[(elmt,0)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mprob = pd.DataFrame(mprob1).T\nmprob.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We use a simple Kmean to find group of questions related"},{"metadata":{"trusted":true},"cell_type":"code","source":"import umap\nfrom sklearn.cluster import KMeans\n\nkmean = KMeans(n_clusters=20)\nc = kmean.fit_predict(mprob)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We use UMAP to project the data and check the clustering made.\n\nThe cluster obtained is used in the final features"},{"metadata":{"trusted":true},"cell_type":"code","source":"proj = umap.UMAP(n_components=2)\nx = proj.fit_transform(mprob)\n\nplt.figure(figsize = (20,10))\nplt.title('Projection of questions similarity by Bayesian Method', size = 20)\nplt.scatter(x[:,0],x[:,1], c=c, alpha = 0.1, cmap = 'tab20b')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Same type of cluster could have been done based on lectures probability, but I didn't have time to try..."},{"metadata":{},"cell_type":"markdown","source":"### Other features\n\nThe features presented above could be computed offline. The other features needs a continuous update to avoid obvious leakage.\nMore details are given in the next part"},{"metadata":{},"cell_type":"markdown","source":"# 3. Production pipeline"},{"metadata":{},"cell_type":"markdown","source":"## History and memory limitation\n\nAll ids in the test set are users already present in the train set.\nIn order to be efficient, our algorithm needs to keep tracks of the parameter of each student.\n\nUnfortunately, due to the high amount of students, my dictionnary was too big to be loaded at one.\nIn order to solve that issue, I decided to create a cache for each id. During inference, I was checking for new ids and adding them to my general cache if there were here already\n\nIt was taking about 5 hours to construct an up to date dictionnary for all users."},{"metadata":{"trusted":true},"cell_type":"code","source":"# This is the location of all the cache, I create a list of ids to keep track of what I alreadu have\nids = glob.glob('../input/riid-cache-6/content/drive/My Drive/riid/*')\nids = [int(elmt.split('/')[-1]) for elmt in ids]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"len(ids)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Some more dictionnaries !\n\nWe create references of parts and tags that will be later user to associate content_id to subcategories to create relationnal features"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"questions_to_tag = questions[['question_id','tags']].set_index('question_id')[['tags']]['tags'].fillna('-1').apply(lambda x: [np.int64(elmt) for elmt in x.split(' ')]).to_dict()\nquestions_to_parts = questions[['question_id','part']].set_index('question_id')['part'].to_dict()\nlecture_to_type = lectures[['lecture_id','type_of']].set_index('lecture_id')['type_of'].to_dict()\nlecture_to_part = lectures[['lecture_id','part']].set_index('lecture_id')['part'].to_dict()\nlecture_to_tag = lectures[['lecture_id','tag']].set_index('lecture_id')['tag'].to_dict()\n\ntags = []\nfor k,v in questions_to_tag.items():\n    tags+= v\ntags = {elmt:np.int32(0) for elmt in set(tags)}\n\nparts = np.unique(list(questions_to_parts.values()))\nquestion_list = list(questions_to_difficulty.keys())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Some useful functions\n\nOne of the big challenge for me was to try to maintain a clean code.\nHave some utility functions to make simple operations was very usefull has it allowed me to avoid a lot of unecessary lines later"},{"metadata":{"trusted":true},"cell_type":"code","source":"def qscore(answer, difficulty):\n    \n    '''This function calculate an asymetric score that is high when a user answer a \n    difficult question and penalise users that answers wrong in simple questions'''\n    if answer>0:\n        return (2*answer - 1)*difficulty\n    else:\n        return (2*answer - 1)*(1-difficulty)\n\ndef list_feature_average(n, vals):\n    '''a simple rolling average over last n elements'''\n    if len(vals)>n:\n        return np.mean(vals[-n:])\n    else:\n        return -1\n    \ndef make_feature_average(count, vals):\n    '''a simple average over all elements given a parameter and its count'''\n    if count>0:\n        return vals/count\n    else:\n        return -1\n    \ndef calculate_time_deltas(cache, timestamp, prior_question_elapsed_time,content_type_id, previous_part, user_id, batch_size):\n    \n    if timestamp != cache[user_id]['ts-1']:\n        if prior_question_elapsed_time != -1:\n            cache[user_id]['average_reactivities'].append(prior_question_elapsed_time)\n            cache[user_id]['last_ts_list'].append(timestamp)\n            if len(cache[user_id]['average_reactivities'])>20:\n                cache[user_id]['average_reactivities'].pop(0)\n            if len(cache[user_id]['last_ts_list'])>20:\n                cache[user_id]['last_ts_list'].pop(0)\n\n            if previous_part in cache[user_id]['part_average_reactivity_tot'].keys():\n                cache[user_id]['part_average_reactivity_tot'][previous_part]+=prior_question_elapsed_time\n\n            else:\n                if previous_part != -1:\n                    cache[user_id]['part_average_reactivity_tot'][previous_part] = prior_question_elapsed_time\n\n            if len(cache[user_id]['last_ts_list'])>3:\n                cache[user_id]['time_between_questions'].append(cache[user_id]['last_ts_list'][-2]-cache[user_id]['last_ts_list'][-3]-cache[user_id]['average_reactivities'][-1]*cache[user_id]['batch_size_list'])\n                if len(cache[user_id]['time_between_questions'])>20:\n                    cache[user_id]['time_between_questions'].pop(0)\n\n\n        if (cache[user_id]['last_delta_between_questions'] != -1) & (prior_question_elapsed_time != -1):\n            if cache[user_id]['average_delta_between_questions'] > 3600:\n                cache[user_id]['hour_count'] +=1\n\n            if cache[user_id]['average_delta_between_questions'] > 10000:\n                cache[user_id]['day_count'] +=1\n\n            if cache[user_id]['average_delta_between_questions'] < 3600:\n                cache[user_id]['average_delta_between_questions'] += cache[user_id]['last_delta_between_questions'] - prior_question_elapsed_time \n\n                \n        if cache[user_id]['ts-1'] != -1:\n            if content_type_id == 0:\n                cache[user_id]['last_delta_between_questions'] = (timestamp - cache[user_id]['ts-1'])/batch_size\n            else:\n                cache[user_id]['average_time_on_lecture'] += (timestamp - cache[user_id]['ts-1'])/batch_size\n            \n    return cache","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Elo functions\n\nCredit to Steve Majou for its notebook [Simple Elo Rating](https://www.kaggle.com/stevemju/riiid-simple-elo-rating). This function works very well to quantify the level of a student.\n\nI later applyied the method both at the global level, part level and tag level"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_new_theta(is_good_answer, beta, left_asymptote, theta, nb_previous_answers):\n    return theta + learning_rate_theta(nb_previous_answers) * (\n        is_good_answer - probability_of_good_answer(theta, beta, left_asymptote)\n    )\n\ndef get_new_beta(is_good_answer, beta, left_asymptote, theta, nb_previous_answers):\n    return beta - learning_rate_beta(nb_previous_answers) * (\n        is_good_answer - probability_of_good_answer(theta, beta, left_asymptote)\n    )\n\ndef learning_rate_theta(nb_answers):\n    return max(0.3 / (1 + 0.01 * nb_answers), 0.04)\n\ndef learning_rate_beta(nb_answers):\n    return 1 / (1 + 0.05 * nb_answers)\n\ndef probability_of_good_answer(theta, beta, left_asymptote):\n    return left_asymptote + (1 - left_asymptote) * sigmoid(theta - beta)\n\ndef sigmoid(x):\n    return 1 / (1 + np.exp(-x))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Cache template\n\nThis one is a very important one:\n\nEverytime a new user is seen, I need to create a new blank page from where I record all the necessary caracteristics \"on the fly\". \nIn order to be memory efficient, I took some decision such as not storing the full history but rather using troncated lists when needed. \n\nFor global indicator, I iteratively calculate sums, and produce simply the averages when needed.\n\nFor the final inference, my create_cache function also check first if an id has not already in an history in my database. If this is the case, I load this history directly in the current cache and keep it in RAM (so I don't need to reload it later if needed)"},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_cache(cache, user_id, ts, ids):\n        \n    if user_id in ids:\n        with open(f'../input/riid-cache-6/content/drive/My Drive/riid/{user_id}', 'rb') as f:\n            id_cache = pickle.load(f)[user_id]\n            cache[user_id] = id_cache\n    else:\n        #generate the cach for a new id\n        cache[user_id] = {}\n\n        #general\n        cache[user_id]['previous_content_type_id'] = -1\n        cache[user_id]['previous_part'] = -1\n        cache[user_id]['previous_question'] = -1\n        cache[user_id]['questions_count'] = 0\n        cache[user_id]['part_lectures_count'] = {i:0 for i in range(1,8)}\n        cache[user_id]['lectures_part'] = {i:0 for i in range(1,8)}\n        cache[user_id]['part_count'] = {}\n        cache[user_id]['question_with_explanation'] = set()\n        cache[user_id]['question_seen'] = set()\n        cache[user_id]['first_bundle'] = -1\n        cache[user_id]['batch_size_list'] = -1\n\n        #Timestamps analysis\n        cache[user_id]['ts-1'] = -1\n        cache[user_id]['last_ts_part'] = {i:0 for i in range(1,8)}\n        cache[user_id]['last_delta_between_questions'] = -1\n        cache[user_id]['day_count'] = 0\n        cache[user_id]['hour_count'] = 0\n        cache[user_id]['last_ts_list'] = []\n        cache[user_id]['time_between_questions'] = []\n        cache[user_id]['average_reactivity_tot'] = 0\n        cache[user_id]['average_reaction_good_answer_tot'] = 0\n        cache[user_id]['average_reaction_bad_answer_tot'] = 0\n        cache[user_id]['average_delta_between_good_answer'] = 0\n        cache[user_id]['part_average_delta_between_good_answer'] = {}\n        cache[user_id]['average_reactivities'] = []\n        cache[user_id]['average_delta_between_questions'] = 0\n        cache[user_id]['average_time_on_lecture'] = -1\n        cache[user_id]['part_average_reactivity_tot'] = {}\n\n        #QUESTION RELATED\n        cache[user_id]['last_questions'] = []\n        cache[user_id]['last_answers_correctly'] = []\n        cache[user_id]['last_answers_correctly_part'] = {i:[] for i in range(1,8)}\n        cache[user_id]['last_lectures'] = []\n        cache[user_id]['lectures_count'] = 0\n\n\n        #GOOD ANSWERS AND SCORE\n        cache[user_id]['general_good_answer'] = 0\n        cache[user_id]['part_good_answer'] = {}\n        cache[user_id]['part_score'] = {i:0 for i in range(1,8)}\n\n        #ELO\n        cache[user_id]['student_parameters'] = {\"theta\": 0, \"nb_answers\": 0}\n\n        #ELO PART\n        cache[user_id]['part_student_parameters'] = {i:{\"theta\": 0, \"nb_answers\": 0} for i in range(1,8)}\n\n        #ELO TAGS\n        cache[user_id]['tag_student_parameters'] = {i:{\"theta\": 0, \"nb_answers\": 0} for i in tags.keys()}\n\n        #TAG RELATED\n        cache[user_id]['tag_count'] = {}\n        cache[user_id]['tag_good_answers'] = {}\n\n    return cache","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Update the cache\n\nUpdating the cache his handled by the function below using the labeled data once available.\nThe only purpose of the function is to update the full history for a given user"},{"metadata":{"trusted":true},"cell_type":"code","source":"def update_cache(cache, arr, batch_size, question_cache):\n    \n    left_asymptote = 1/4\n    #unpack array\n    row_id, timestamp, user_id, content_id, content_type_id, task_container_id, prior_question_elapsed_time, prior_question_had_explanation, _, _, answered_correctly,user_answer  = arr\n    \n    if not prior_question_elapsed_time:\n        prior_question_elapsed_time=-1\n    if not prior_question_had_explanation:\n        prior_question_had_explanation =-1\n        \n    timestamp = int(float(timestamp)/1000)\n    user_id = np.uint32(user_id)\n    content_id = np.uint16(content_id)\n    content_type_id = np.int8(content_type_id)\n    task_container_id = np.uint32(task_container_id)\n    prior_question_elapsed_time = int(float(prior_question_elapsed_time)/1000)\n    prior_question_had_explanation = np.bool(prior_question_had_explanation)\n    user_answer = np.int8(user_answer)\n    answered_correctly = np.int8(answered_correctly)\n    previous_content_type_id = cache[user_id]['previous_content_type_id']\n    \n    part = -1\n    #General\n    if content_type_id == 0:\n\n        if content_id not in question_cache.keys():\n            question_cache[content_id] = {\"beta\": 0, \"nb_answers\": 0}\n\n        part = questions_to_parts[content_id]\n        tags = questions_to_tag[content_id]\n\n        difficulty = questions_to_difficulty[content_id]\n        count = question_user_count[content_id]\n        bad_answer_reaction = bad_answer_reaction_time[content_id]\n        good_answer_reaction = good_answer_reaction_time[content_id]\n        beta = question_cache[content_id][\"beta\"]\n        score = qscore(answered_correctly, difficulty)\n        theta = cache[user_id]['student_parameters'][\"theta\"]\n        theta_part = cache[user_id]['part_student_parameters'][part][\"theta\"]\n\n        for tag in tags:\n            theta_tag = cache[user_id]['tag_student_parameters'][tag][\"theta\"]\n            cache[user_id]['tag_student_parameters'][tag][\"theta\"] = get_new_theta(\n                answered_correctly, beta, left_asymptote, theta_tag, cache[user_id]['tag_student_parameters'][tag][\"nb_answers\"],\n            )\n            cache[user_id]['tag_student_parameters'][tag][\"nb_answers\"] += 1\n        \n\n        cache[user_id]['student_parameters'][\"theta\"] = get_new_theta(\n            answered_correctly, beta, left_asymptote, theta, cache[user_id]['student_parameters'][\"nb_answers\"],\n        )\n\n        cache[user_id]['part_student_parameters'][part][\"theta\"] = get_new_theta(\n            answered_correctly, beta, left_asymptote, theta_part, cache[user_id]['part_student_parameters'][part][\"nb_answers\"],\n        )\n\n        cache[user_id]['student_parameters'][\"nb_answers\"] += 1\n        cache[user_id]['part_student_parameters'][part][\"nb_answers\"] += 1\n        \n        cache[user_id]['questions_count'] +=1\n        if part in cache[user_id]['part_count'].keys():\n            cache[user_id]['part_count'][part] += 1\n        else:\n            cache[user_id]['part_count'][part] = 1\n            \n        cache[user_id]['last_answers_correctly'].append(answered_correctly)\n        cache[user_id]['last_answers_correctly_part'][part].append(answered_correctly)\n        if len(cache[user_id]['last_answers_correctly'])>20:\n            cache[user_id]['last_answers_correctly'].pop(0)\n        if len(cache[user_id]['last_answers_correctly_part'][part])>20:\n            cache[user_id]['last_answers_correctly_part'][part].pop(0)  \n\n        #GENERAL GOOD ANSWERS\n        if answered_correctly == 1:\n            cache[user_id]['average_reaction_good_answer_tot'] += prior_question_elapsed_time\n            if part in cache[user_id]['part_good_answer'].keys():\n                cache[user_id]['part_good_answer'][part]+=1\n            else:\n                cache[user_id]['part_good_answer'][part]=1\n                \n            cache[user_id]['general_good_answer'] += 1\n            score = qscore(answered_correctly, difficulty)\n    \n        if answered_correctly == 0:\n            cache[user_id]['average_reaction_bad_answer_tot'] += prior_question_elapsed_time\n            \n        cache[user_id]['part_score'][part]+=score\n\n        #TAGS\n        for tag in tags:\n            if tag in cache[user_id]['tag_count'].keys():\n                cache[user_id]['tag_count'][tag] +=1\n                cache[user_id]['tag_good_answers'][tag] += answered_correctly\n            else:\n                cache[user_id]['tag_count'][tag] = 1\n                cache[user_id]['tag_good_answers'][tag] = answered_correctly\n    \n    if (cache[user_id]['ts-1'] != -1) & (timestamp != cache[user_id]['ts-1']):\n        cache[user_id]['last_delta_between_questions'] = (timestamp - cache[user_id]['ts-1'])\n        cache[user_id]['average_delta_between_questions'] += (timestamp - cache[user_id]['ts-1'])\n                \n    cache = calculate_time_deltas(cache, timestamp, prior_question_elapsed_time,content_type_id, cache[user_id]['previous_part'], user_id, batch_size)\n    \n    cache[user_id]['ts-1'] = timestamp\n    cache[user_id]['last_ts_part'][part] = timestamp\n    cache[user_id]['previous_content_type_id'] = content_type_id\n    if part != -1:\n        cache[user_id]['previous_part'] = part\n        \n    if content_type_id == 0:\n        cache[user_id]['previous_question'] = content_id\n        \n    if content_type_id == 0:\n        cache[user_id]['last_questions'].append((content_id,answered_correctly))\n        if len(cache[user_id]['last_questions'])>100:\n            cache[user_id]['last_questions'].pop(0)\n            \n    if content_type_id == 1:\n        part = lecture_to_part[content_id]\n        cache[user_id]['lectures_count']+=1\n        cache[user_id]['lectures_part'][part]+=1\n        cache[user_id]['last_lectures'].append(content_id)\n        if len(cache[user_id]['last_lectures'])>100:\n            cache[user_id]['last_lectures'].pop(0)\n            \n    if (prior_question_had_explanation != -1) & (cache[user_id]['previous_question'] != -1):\n        cache[user_id]['question_with_explanation'].add(content_id)\n    \n    cache[user_id]['question_seen'].add(content_id)\n    if cache[user_id]['first_bundle'] == -1:\n        cache[user_id]['first_bundle'] = content_id    \n\n    cache[user_id]['batch_size_list'] = batch_size\n    \n    return cache","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Feature update\n\nThe feature update create the rows use in the model to make the prediction. It is based on the current information of the questions (content_id, features generated offline) as well as on the content from the user history\n\nnote: to build the training set and maximase RAM usage, I prefered to use a simple csv iterator rather than charging the whole dataset. \nnote: if the current row is a lecture, I return an empty list that will be handled in the main loop"},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_feature(cache, arr, batch_size, question_cache):\n    \n    features = []\n    \n    #unpack array\n    row_id, timestamp, user_id, content_id, content_type_id, task_container_id, prior_question_elapsed_time, prior_question_had_explanation, _, _ = arr\n    \n    if not prior_question_elapsed_time:\n        prior_question_elapsed_time=-1\n    if not prior_question_had_explanation:\n        prior_question_had_explanation =-1\n        \n       \n    ##CAST##\n    timestamp = float(timestamp)/1000\n    prior_question_elapsed_time = float(prior_question_elapsed_time)/1000\n\n    if content_type_id == 0:\n        \n        part = questions_to_parts[content_id]\n        tags = questions_to_tag[content_id]\n        difficulty = questions_to_difficulty[content_id]\n        count = question_user_count[content_id]\n        bad_answer_reaction = bad_answer_reaction_time[content_id]\n        good_answer_reaction = good_answer_reaction_time[content_id]\n        var = questions_to_difficulty_variance[content_id]\n\n        if content_id in questions_to_difficulty_with_explanation.keys():\n            difficulty1bis = questions_to_difficulty_with_explanation[content_id]\n        else:\n            difficulty1bis = -1\n        if content_id in questions_to_difficulty_without_explanation.keys():  \n            difficulty2bis = questions_to_difficulty_without_explanation[content_id]\n        else:\n            difficulty2bis = -1\n\n        kk = 0\n\n        features.append(part)\n        features.append(difficulty)\n        features.append(count)\n        features.append(good_answer_reaction)\n        features.append(bad_answer_reaction)\n        features.append(batch_size)\n        features.append(var)\n        features.append(difficulty1bis)\n        features.append(difficulty2bis)\n        features.append(clust[content_id])\n\n        #lecture\n        features.append(cache[user_id]['lectures_count'])\n            \n        #elo\n        a=0\n        if user_id in cache.keys():\n            features.append(cache[user_id]['student_parameters']['theta'])\n            a+=1\n        else:\n            features.append(-1) # 12\n        if content_id in question_cache.keys():\n\n            a+=1\n        else:\n            pass\n\n        if a ==2:\n            features.append(probability_of_good_answer(cache[user_id]['student_parameters']['theta'], question_cache[content_id]['beta'], 1/4))\n        else:\n            features.append(-1)\n\n        #PART ELO\n        if part in cache[user_id]['part_student_parameters'].keys():\n            features.append(probability_of_good_answer(cache[user_id]['part_student_parameters'][part]['theta'], question_cache[content_id]['beta'], 1/4))\n        else:\n            features.append(-1)\n\n        c = 0\n        p = 0 \n        s = len(tags)\n        for tag in tags:\n            c+=cache[user_id]['tag_student_parameters'][tag]['theta']\n            p+=probability_of_good_answer(cache[user_id]['tag_student_parameters'][tag]['theta'], question_cache[content_id]['beta'], 1/4)\n        features.append(c/s)\n        features.append(p/s)\n\n        features.append(cache[user_id]['part_score'][part])\n        #Timestamp features\n\n        features.append((timestamp - cache[user_id]['ts-1'])/batch_size)\n        features.append(list_feature_average(19, cache[user_id]['average_reactivities']))\n\n        if part in cache[user_id]['part_average_reactivity_tot'].keys():\n            features.append(make_feature_average(cache[user_id]['part_count'][part], cache[user_id]['part_average_reactivity_tot'][part]))\n        else:\n            features.append(-1)\n    \n        features.append(make_feature_average(cache[user_id]['general_good_answer'], cache[user_id]['average_reaction_good_answer_tot']))\n        features.append(cache[user_id]['last_delta_between_questions'])\n\n        features.append(list_feature_average(10, (cache[user_id]['time_between_questions'])))\n        features.append(list_feature_average(5, (cache[user_id]['time_between_questions'])))\n        if len(cache[user_id]['time_between_questions'])>1:\n            features.append(cache[user_id]['time_between_questions'][-1])\n        else:\n            features.append(-1)\n\n        #Score features\n        features.append(make_feature_average(cache[user_id]['questions_count'], cache[user_id]['general_good_answer']))\n        features.append(list_feature_average(2, cache[user_id]['last_answers_correctly']))\n         \n        features.append(list_feature_average(6, cache[user_id]['last_answers_correctly']))\n\n\n        features.append(list_feature_average(6, cache[user_id]['last_answers_correctly_part'][part]))\n\n        #Explanation features\n        if content_id in cache[user_id]['question_with_explanation']:\n            features.append(1)\n        else:\n            features.append(0)\n            \n        c = 0\n\n        ##Conditionnal Proba part\n        c1 = 0\n        c0 = 0\n        p=0\n        if content_id in all_prob.keys():\n            to_check = set(cache[user_id]['last_questions']).intersection(set(all_prob[content_id].keys()))\n            for content, answer in to_check:\n                p+=all_prob[content_id][(content,answer)]\n                if answer == 1:\n                    c1+=1\n                else:\n                    c0+=1\n\n        features.append(c1)\n\n        if c1+c0:\n            features.append(p/(c1+c0))\n        else:\n            features.append(0)\n        \n        ##Conditionnal Proba part\n        c = 0\n        p=0\n        if content_id in proba_lec.keys():\n            to_check = set(cache[user_id]['last_lectures']).intersection(set(proba_lec[content_id].keys()))\n            for content in to_check:\n                p+=proba_lec[content_id][content]\n                c+=1\n        if c:\n            features.append(p/c)\n        else:\n            features.append(0)\n\n    return features","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Update cache in batch\n\nAs the data are given by batch, I need another loop to update the cache row by row"},{"metadata":{"trusted":true},"cell_type":"code","source":"def update_cache_batch(cache, df_arr, question_cache):\n    \n    batch_array = []\n    task_init = -1\n    user_id0 = -1 \n    \n    for arr in df_arr:\n        \n        user_id = arr[2]\n        timestamp = arr[1]\n        task_container_id = arr[5]\n        if user_id not in cache.keys():\n            cache = create_cache(cache, user_id, timestamp, ids)\n            \n        if (task_container_id == task_init) & (user_id == user_id0):\n            batch_array.append(arr) \n        else:\n            #When we meet a new task_container_id, we calculate metrics, and update cache.\n            batch_size = len(batch_array)\n            for arr_ in batch_array:\n                cache = update_cache(cache, arr_, batch_size, question_cache)\n                \n            #Finally, create new batch_array and update task_init\n            task_init = task_container_id\n            if user_id != user_id0:\n                user_id0 = user_id\n                \n            batch_array = []\n            batch_array.append(arr)\n\n    #Eventually, compute the metrics for the last elements in batch_array\n    batch_size = len(batch_array)\n    \n    #Update cache\n    for arr_ in batch_array:\n        cache = update_cache(cache, arr_, batch_size, question_cache)\n        \n    return cache","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Calculate features\n\nThis is the main part of the loop\n\nIt detects new users, create the cache for a new user if needed, create the features and return an array to be directly plugged to the model"},{"metadata":{"trusted":true},"cell_type":"code","source":"def calculate_features(cache, df_arr, question_cache):\n\n    X = []\n\n    batch_array = []\n    task_init = -1\n    user_id0 = -1 \n\n    for arr in df_arr:\n\n        #First we check if the id has been seen before. If not, we create a new cache\n        user_id = arr[2]\n        timestamp = arr[1]\n        task_container_id = arr[5]\n        if user_id not in cache.keys():\n            cache = create_cache(cache, user_id, timestamp, ids)\n\n        #This is the tricky part: we check if the question comes in batch. If this is the case, we store the array in a \n        #temporary list the time to retrieve all the questions from the array in order to compute batch_size.\n        if (task_container_id == task_init) & (user_id == user_id0):\n            batch_array.append(arr)\n\n        else:\n            #When we meet a new task_container_id, we calculate metrics, and update cache.\n            batch_size = len(batch_array)\n\n            #Update metrics\n            for arr_ in batch_array:\n                feats = create_feature(cache, arr_, batch_size, question_cache)\n                if len(feats)>0:\n                    X.append(feats)\n\n            #Finally, create new batch_array and update task_init\n            task_init = task_container_id\n            if user_id != user_id0:\n                user_id0 = user_id\n                \n            batch_array = []\n            batch_array.append(arr)\n\n    #Eventually, compute the metrics for the last elements in batch_array\n    batch_size = len(batch_array)\n\n    #Update metrics\n    for arr_ in batch_array:\n        feats = create_feature(cache, arr_, batch_size, question_cache)\n        if len(feats)>0:\n            X.append(feats)\n\n    X = np.array(X)\n        \n    return X","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Final inference"},{"metadata":{"trusted":true},"cell_type":"code","source":"env = riiideducation.make_env()\niter_test = env.iter_test()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"p_test_df = pd.DataFrame()\ncache = {}\n\nfor idx, (test_df, _) in enumerate(iter_test):\n    \n    test_df['prior_question_had_explanation'] = test_df['prior_question_had_explanation'].astype(float)\n    test_df = test_df.fillna(-1)\n\n\n    submit_df = test_df.loc[test_df['content_type_id'] == False, ['row_id']].copy()\n    if not p_test_df.empty:\n\n        answered_correctly = [int(elmt) for elmt in test_df.iloc[0]['prior_group_answers_correct'][1:-1].replace(' ','').split(',')]\n        prior_group_responses = [int(elmt) for elmt in test_df.iloc[0]['prior_group_responses'][1:-1].replace(' ','').split(',')]\n        p_test_df['answered_correctly'] = answered_correctly\n        p_test_df['user_response'] = answered_correctly\n        test_arr = p_test_df.values\n        update_cache_batch(cache, test_arr, question_cache)\n\n\n\n    X = calculate_features(cache, test_df.values, question_cache)\n    if len(X):\n        preds = 1-lgb.predict_proba(X)[:,0]\n    else:\n        preds = None\n\n    submit_df['answered_correctly'] = preds\n    env.predict(submit_df)    \n    p_test_df = test_df","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}