{"cells":[{"metadata":{},"cell_type":"markdown","source":"### Description\n\n* `theta` represents the global level of the student\n* `beta` represents the difficulty of the item (items are represented by `content_id` in that case but you could change it by modifying the `granularity_feature_name` parameters)\n* `left_asymptote` represents the probability of correctly answering a question by chance. The `correct_answer` from the questions dataset has 4 possible values, so this probability is at least 1/4 \n\n### Remarks\n\n* I used a different learning rate for thetas and for betas. The learning rate for thetas has a floor value to keep it dynamic even after many answers as the student level is always evolving contrary to the difficulty of questions.\n* The learning rate for theta update is absolutely arbitrary (I picked one that looked good, nothing more). The learning rate for betas is one that was recommended [this paper](https://www.fi.muni.cz/~xpelanek/publications/CAE-elo.pdf) but I'm sure there's a better one.\n* I tried to compute a theta by `part` but performance is slightly worse (0.764). \n* I tried both a theta by `part` and a global theta, which I combined with proportions based on the nb_answers of the part, and performance is slightly better (0.767).\n\n### Possible improvements:\n\n* Fine tune learning rates (simply changing the LR for betas made the score go from 0.749 to 0.766)\n* Learning rate function of time: if the student hasn't played for a long time, we might want to increase the LR to reestimate his/her level quickly\n* Add a coefficient on `theta` computation based on the time to answer (a good and quick answer is probably more indicative of mastery than a good and slow answer)\n* Investigate on extreme `theta` and `beta` values\n* Using betas & thetas as model features could be interesting, as well as estimating betas by tags"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import riiideducation\nimport numpy as np \nimport pandas as pd \nfrom tqdm import tqdm","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## ELO functions"},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_new_theta(is_good_answer, beta, left_asymptote, theta, nb_previous_answers):\n    return theta + learning_rate_theta(nb_previous_answers) * (\n        is_good_answer - probability_of_good_answer(theta, beta, left_asymptote)\n    )\n\ndef get_new_beta(is_good_answer, beta, left_asymptote, theta, nb_previous_answers):\n    return beta - learning_rate_beta(nb_previous_answers) * (\n        is_good_answer - probability_of_good_answer(theta, beta, left_asymptote)\n    )\n\ndef learning_rate_theta(nb_answers):\n    return max(0.3 / (1 + 0.01 * nb_answers), 0.04)\n\ndef learning_rate_beta(nb_answers):\n    return 1 / (1 + 0.05 * nb_answers)\n\ndef probability_of_good_answer(theta, beta, left_asymptote):\n    return left_asymptote + (1 - left_asymptote) * sigmoid(theta - beta)\n\ndef sigmoid(x):\n    return 1 / (1 + np.exp(-x))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Parameters Estimation"},{"metadata":{"trusted":true},"cell_type":"code","source":"def estimate_parameters(answers_df, granularity_feature_name='content_id'):\n    item_parameters = {\n        granularity_feature_value: {\"beta\": 0, \"nb_answers\": 0}\n        for granularity_feature_value in np.unique(answers_df[granularity_feature_name])\n    }\n    student_parameters = {\n        student_id: {\"theta\": 0, \"nb_answers\": 0}\n        for student_id in np.unique(answers_df.student_id)\n    }\n\n    print(\"Parameter estimation is starting...\")\n\n    for student_id, item_id, left_asymptote, answered_correctly in tqdm(\n        zip(answers_df.student_id.values, answers_df[granularity_feature_name].values, answers_df.left_asymptote.values, answers_df.answered_correctly.values)\n    ):\n        theta = student_parameters[student_id][\"theta\"]\n        beta = item_parameters[item_id][\"beta\"]\n\n        item_parameters[item_id][\"beta\"] = get_new_beta(\n            answered_correctly, beta, left_asymptote, theta, item_parameters[item_id][\"nb_answers\"],\n        )\n        student_parameters[student_id][\"theta\"] = get_new_theta(\n            answered_correctly, beta, left_asymptote, theta, student_parameters[student_id][\"nb_answers\"],\n        )\n        \n        item_parameters[item_id][\"nb_answers\"] += 1\n        student_parameters[student_id][\"nb_answers\"] += 1\n\n    print(f\"Theta & beta estimations on {granularity_feature_name} are completed.\")\n    return student_parameters, item_parameters","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Update Parameters"},{"metadata":{"trusted":true},"cell_type":"code","source":"def update_parameters(answers_df, student_parameters, item_parameters, granularity_feature_name='content_id'):\n    for student_id, item_id, left_asymptote, answered_correctly in tqdm(zip(\n        answers_df.student_id.values, \n        answers_df[granularity_feature_name].values, \n        answers_df.left_asymptote.values, \n        answers_df.answered_correctly.values)\n    ):\n        if student_id not in student_parameters:\n            student_parameters[student_id] = {'theta': 0, 'nb_answers': 0}\n        if item_id not in item_parameters:\n            item_parameters[item_id] = {'beta': 0, 'nb_answers': 0}\n            \n        theta = student_parameters[student_id]['theta']\n        beta = item_parameters[item_id]['beta']\n\n        student_parameters[student_id]['theta'] = get_new_theta(\n            answered_correctly, beta, left_asymptote, theta, student_parameters[student_id]['nb_answers']\n        )\n        item_parameters[item_id]['beta'] = get_new_beta(\n            answered_correctly, beta, left_asymptote, theta, item_parameters[item_id]['nb_answers']\n        )\n        \n        student_parameters[student_id]['nb_answers'] += 1\n        item_parameters[item_id]['nb_answers'] += 1\n        \n    return student_parameters, item_parameters","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Probability Estimation"},{"metadata":{"trusted":true},"cell_type":"code","source":"def estimate_probas(test_df, student_parameters, item_parameters, granularity_feature_name='content_id'):\n    probability_of_success_list = []\n    \n    for student_id, item_id, left_asymptote in tqdm(\n        zip(test_df.student_id.values, test_df[granularity_feature_name].values, test_df.left_asymptote.values)\n    ):\n        theta = student_parameters[student_id]['theta'] if student_id in student_parameters else 0\n        beta = item_parameters[item_id]['beta'] if item_id in item_parameters else 0\n\n        probability_of_success_list.append(probability_of_good_answer(theta, beta, left_asymptote))\n\n    return probability_of_success_list","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Main"},{"metadata":{"trusted":true},"cell_type":"code","source":"compute_estimations = False\nnb_rows_training = None","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if compute_estimations:\n    training = pd.read_csv(\n        filepath_or_buffer=\"/kaggle/input/riiid-test-answer-prediction/train.csv\", \n        usecols=[\"content_id\", \"user_id\", \"answered_correctly\"], \n        dtype={'answered_correctly': \"int8\"},\n        nrows=nb_rows_training\n    )\n\n    training.rename(columns={'user_id': 'student_id'}, inplace=True)\n    training = training[training.answered_correctly != -1]\n    training['left_asymptote'] = 1/4\n\n    print(f\"Dataset of shape {training.shape}\")\n    print(f\"Columns are {list(training.columns)}\")\n    \n    student_parameters, item_parameters = estimate_parameters(training)\nelse:\n    student_data = pd.read_csv('../input/thetas-20201217/thetas_20201217.csv', index_col='student_id')\n    student_parameters = student_data.to_dict('index')\n    print(f\"Successfully read student parameter file and converted to dict.\")\n    \n    content_data = pd.read_csv('../input/betas-content-id-20201217/betas_content_id_20201217.csv', index_col='content_id')\n    item_parameters = content_data.to_dict('index')\n    print(f\"Successfully read item parameter file and converted to dict.\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Submission"},{"metadata":{"trusted":true},"cell_type":"code","source":"def format_test_df(test_df):\n    test_copy = test_df.copy()\n    test_copy = test_copy[test_copy['content_type_id'] == 0]\n    test_copy['left_asymptote'] = 1/4\n    test_copy = test_copy.rename(columns={'user_id': 'student_id'})\n    return test_copy","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"env = riiideducation.make_env()\niter_test = env.iter_test()\n\nprevious_test_df = None\n\nfor (test_df, sample_prediction_df) in iter_test:\n    if previous_test_df is not None:\n        previous_test_df['answered_correctly'] = eval(test_df[\"prior_group_answers_correct\"].iloc[0])\n        previous_test_df = format_test_df(previous_test_df)\n        student_parameters, item_parameters = update_parameters(previous_test_df, student_parameters, item_parameters)\n\n    previous_test_df = test_df.copy()\n    test_df = format_test_df(test_df)\n    test_df['answered_correctly'] = estimate_probas(test_df, student_parameters, item_parameters)\n    env.predict(test_df[['row_id', 'answered_correctly']])","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}