{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":59093,"databundleVersionId":7469972,"sourceType":"competition"},{"sourceId":154204277,"sourceType":"kernelVersion"},{"sourceId":155730565,"sourceType":"kernelVersion"}],"dockerImageVersionId":30635,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# How is this competition scored?\nTo do well in this competition it is very important to understand the algorithm by which the competition is scored - The Kullback Leibler Divergence.\n\nHere I have borrowed the code from Kaggle's example scoring script (https://www.kaggle.com/code/metric/kullback-leibler-divergence/notebook), and used it to create a notebook where you can play around with the probability scoring to improve your understanding.\n\nThere's a useful discussion posted here: https://www.kaggle.com/competitions/hms-harmful-brain-activity-classification/discussion/466731 by MARÍLIA PRATA.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport warnings","metadata":{"execution":{"iopub.status.busy":"2024-01-11T01:26:37.946673Z","iopub.execute_input":"2024-01-11T01:26:37.947066Z","iopub.status.idle":"2024-01-11T01:26:37.951572Z","shell.execute_reply.started":"2024-01-11T01:26:37.947031Z","shell.execute_reply":"2024-01-11T01:26:37.950778Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import metric","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-01-11T01:26:37.957845Z","iopub.execute_input":"2024-01-11T01:26:37.958436Z","iopub.status.idle":"2024-01-11T01:26:37.964093Z","shell.execute_reply.started":"2024-01-11T01:26:37.958402Z","shell.execute_reply":"2024-01-11T01:26:37.963274Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"^ For this import code to run make sure you've gone to the far right of the notebook and have kullback-leibler-divergence in your Input. Otherwise use the \"+ Add Data\" button to add it. You may also need kaggle_metric_utilities.\n\nI struggled to get this to work in my other notebook, probably because the Internet setting is required to be turned off for entry submission.","metadata":{}},{"cell_type":"code","source":"# Filter out SettingWithCopyWarning\nwarnings.filterwarnings(\"ignore\")\n# Setup useful functions\nvote_list = ['seizure_vote', 'lpd_vote', 'gpd_vote', 'lrda_vote', 'grda_vote', 'other_vote']\ndef valid_submission(submission_df):\n    for i, eeg_id in enumerate(submission_df['eeg_id']):\n        vote_total = 0\n        for vote in vote_list:\n            vote_total += submission_df[vote].iloc[i]\n        if 0.99 > vote_total > 1.01:\n            return False\n    return True","metadata":{"execution":{"iopub.status.busy":"2024-01-11T01:32:23.956878Z","iopub.execute_input":"2024-01-11T01:32:23.957257Z","iopub.status.idle":"2024-01-11T01:32:23.963682Z","shell.execute_reply.started":"2024-01-11T01:32:23.957217Z","shell.execute_reply":"2024-01-11T01:32:23.962562Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's define 3 'types' of probability distributions:\n* 1 certainty\n* 2 main posibilities\n* 6 evenly weighted probabilities\n\nThen we will compare each combination of these for both the submission and solution.","metadata":{}},{"cell_type":"code","source":"submission_dict = {\n    'eeg_id': [0, 1, 2, 3, 4, 5, 6, 7, 8],\n    'seizure_vote': [1, 1, 1, 1/2, 1/2, 1/2, 1/6, 1/6, 1/6],\n    'lpd_vote': [0, 0, 0, 1/2, 1/2, 1/2, 1/6, 1/6, 1/6],\n    'gpd_vote': [0, 0, 0, 0, 0, 0, 1/6, 1/6, 1/6],\n    'lrda_vote': [0, 0, 0, 0, 0, 0, 1/6, 1/6, 1/6],\n    'grda_vote': [0, 0, 0, 0, 0, 0, 1/6, 1/6, 1/6],\n    'other_vote': [0, 0, 0, 0, 0, 0, 1/6, 1/6, 1/6],\n}\nsolution_dict = {\n    'eeg_id': [0, 1, 2, 3, 4, 5, 6, 7, 8],\n    'seizure_vote': [1, 1/2, 1/6, 1, 1/2, 1/6, 1, 1/2, 1/6],\n    'lpd_vote': [0, 1/2, 1/6, 0, 1/2, 1/6, 0, 1/2, 1/6],\n    'gpd_vote': [0, 0, 1/6, 0, 0, 1/6, 0, 0, 1/6],\n    'lrda_vote': [0, 0, 1/6, 0, 0, 1/6, 0, 0, 1/6],\n    'grda_vote': [0, 0, 1/6, 0, 0, 1/6, 0, 0, 1/6],\n    'other_vote': [0, 0, 1/6, 0, 0, 1/6, 0, 0, 1/6],\n}\nsubmission_df = pd.DataFrame(submission_dict)\nsolution_df = pd.DataFrame(solution_dict)\nprint(f\"Submission df is valid: {valid_submission(submission_df)}\")\nprint(f\"Solution df is valid: {valid_submission(solution_df)}\")\n\n# Check the values are ok. Fill remaining probability with other.\nprint(\"\\nIndividual row scores:\")\nfor i, eeg_id in enumerate(solution_dict['eeg_id']):\n    print(f\"Score {i+1}: {metric.score(solution_df[solution_df['eeg_id'] == eeg_id], submission_df[submission_df['eeg_id'] == eeg_id], 'eeg_id')}\")\n\nprint(f\"\\nOverall score: {metric.score(solution_df, submission_df, 'eeg_id')}\")","metadata":{"execution":{"iopub.status.busy":"2024-01-11T01:33:13.888072Z","iopub.execute_input":"2024-01-11T01:33:13.889196Z","iopub.status.idle":"2024-01-11T01:33:14.153851Z","shell.execute_reply.started":"2024-01-11T01:33:13.889147Z","shell.execute_reply":"2024-01-11T01:33:14.152681Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It seems to me that it's nearly always best to play it safe with an even spread of probabilities in your submission (Scores 7-9), since they're guaranteed to be below 1.8.\n\nSetting a high probability is very risky if you're wrong!","metadata":{}}]}