TEST_PREDICTION must be set to False, otherwise it will use a smaller train dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"TEST_PREDICTION = True","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Read the data\nI read the data in feather format. Depending on the flag TEST_PREDICTION I will either read the full train dataset in feather format from this notebook or I will read two datasets for train/test from this notebook. You can check the notebook if you want more details on how the test set was created
When we create the classifier we must provide a dictionary of pandas DataFrames. Each dataframe must contain an index which is the id of the feature and three columns with the total number of occurences of the feature in the train dataset, the total positive cases and the total negative cases.
\nWe define a threshold as well (by default 10). If any instance of the feature in the prediction dataset has less elements than the threshold in the training dataset, it will be ignored.
\npredict receives a dictionary (the keys must match with the dictionary provided on the creation step) of lists. All list must be the same size. Each value of the list is an instance that we want to predict.\n
We can incrementally update the model by using the function update_model."},{"metadata":{"trusted":true},"cell_type":"code","source":"class NaiveBayes:\n def __init__(self, features, threshold=10):\n assert type(features)==dict, 'parameter features is not a dictionary!'\n for f in features.keys():\n assert type(features[f])==pd.core.frame.DataFrame, 'Wrong datatype for {0}. Each entry of the dictionary must contain a pandas DataFrame'.format(f)\n assert list(features[f].columns)==['total', 'positive', 'negative'], 'wrong columns in {0} DataFrame'.format(f)\n self.THRESHOLD = threshold\n self.features = features\n self.prior_probability = {}\n self.one_feature = list(features.keys())[0]\n self.total_answers = features[self.one_feature]['total'].sum()\n self.positive_answers = features[self.one_feature]['positive'].sum()\n self.negative_answers = features[self.one_feature]['negative'].sum()\n self.prior_probability['negative'] = self.negative_answers/self.total_answers\n self.prior_probability['positive'] = self.positive_answers/self.total_answers\n \n def update_model(self, data):\n assert data.keys()==self.features.keys(), \"Keys doesn't match!\"\n for f in self.features.keys():\n self.features[f].add(data[f], fill_value=0).astype('uint64')\n self.total_answers += data[self.one_feature]['total'].sum()\n self.positive_answers += data[self.one_feature]['positive'].sum()\n self.negative_answers += data[self.one_feature]['negative'].sum()\n self.prior_probability['negative'] = self.negative_answers/self.total_answers\n self.prior_probability['positive'] = self.positive_answers/self.total_answers\n \n \n def predict(self, data):\n assert data.keys()==self.features.keys(), \"Keys doesn't match!\"\n data_len = len(data[list(data.keys())[0]])\n # pos and neg are the priors for positive and negative classes\n pos = np.array([self.prior_probability['positive'] for _ in range(data_len)])\n neg = np.array([self.prior_probability['negative'] for _ in range(data_len)])\n # multiply the prior probability by the likelihood of each feature\n for d in data.keys():\n feature = pd.DataFrame({'id':data[d]})\n counts=pd.merge(feature,self.features[d],left_on='id',right_index=True,how='left').fillna(0).astype('uint64').values\n # counts.shape == (sample_len,4)\n # counts[:,0]==id ; counts[:,1]==total ; counts[:,2]==positive ; counts[:,3]==negative\n # e.g.: counts == array([[115,46,32,14],[124,10,7,3],[115,46,32,14]],dtype=uint64)\n updatable = np.where(counts[:,1]>self.THRESHOLD)[0]\n # e.g.: updatable == array([True,False,True])\n pos[updatable] *= counts[updatable,2]/counts[updatable,1]\n neg[updatable] *= counts[updatable,3]/counts[updatable,1]\n return pos/(pos+neg)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Prepare simple features
\nGiven a column name, this function creates a dataframe that we can use for the class NaiveBayes."},{"metadata":{"trusted":true},"cell_type":"code","source":"def prepare_features(dataset, col_name):\n df = dataset.loc[dataset.content_type_id==0,[col_name,'answered_correctly']].groupby(col_name).agg(['count','sum'])\n df.columns=['total', 'positive']\n df = df.astype('uint64')\n df['negative'] = df['total']-df['positive']\n return df","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Questions
\nWe group by content_id to get the number of times that a question has been asked and how many times it has been answered correctly or incorrectly."},{"metadata":{"trusted":true},"cell_type":"code","source":"question_df = prepare_features(train,'content_id')\nquestion_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.hist((question_df['positive']/question_df['total']).values, bins =30)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Users
\nGrouping by user_id, we will get information about how good or bad is the student."},{"metadata":{"trusted":true},"cell_type":"code","source":"user_df = prepare_features(train,'user_id')\nuser_df.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.hist((user_df['positive']/user_df['total']).values, bins =30)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is just a simple baseline, but more features could be added in a similar way. Naive Bayes is making the \"naive\" assumption that all the features are independent. If we want it to work well, we should add feature that are not correlated."},{"metadata":{},"cell_type":"markdown","source":"## Predict on a test dataset\n
In this case we don't want to submit the solution but just evaluate it on a test dataset (as explained here)
"},{"metadata":{"trusted":true},"cell_type":"code","source":"class TestGenerator:\n def __init__(self, df, grp_size=[50,10]):\n self.df = df\n self.answered_correctly = self.df.answered_correctly[self.df.content_type_id==0].values\n self.predictions = np.zeros(len(self.answered_correctly))\n self.grp_size = grp_size\n self.start_idx=0\n self.last_prediction_idx = 0\n self.prediction_called = True\n self.test_cols = [c for c in df.columns if c not in ['answered_correctly','user_answer']]\n self.current_batch = {'prior_group_answers_correct':[], 'prior_group_responses':[]}\n\n def iter_test(self):\n while self.start_idxchrono can be useful to detect bottlenecks in the submission."},{"metadata":{},"cell_type":"markdown","source":"## Submission\nHere I create the real submission for the competition."},{"metadata":{"trusted":true},"cell_type":"code","source":"env = riiideducation.make_env()\niter_test = env.iter_test()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For each batch in test_df, we will predict the probability of answering correctly (nb.predict) and then we will send the resulting data back to the environment. This last part is done in env.predict. Notice that we must not create any submission.csv file, this is done automatically by env.predict."},{"metadata":{"trusted":true},"cell_type":"code","source":"for (test_df, sample_prediction_df) in iter_test:\n test_questions = test_df['content_id']\n test_users = test_df['user_id']\n answered_correctly = nb.predict({'question':test_questions, 'user':test_users})\n test_df['answered_correctly'] = answered_correctly\n env.predict(test_df.loc[test_df['content_type_id']==0,['row_id','answered_correctly']])","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}