parquet files.![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F83267%2F83ae2309640454544a6377e657c9b325%2FScreenshot%202022-11-07%20171835.png?generation=1667805551208861&alt=media\")
Hey!
\nI created and uploaded onto Kaggle a dataset for local validation using code in the organizer's repo.
\nPlease find the dataset here.
\nI created the dataset passing the competition train data to the script and used the last 7 days to create the test set.
\nGenerally, the interesting bit here is that the test set created on the last week of the train set seems considerably easier to predict on than what we have in the competition.
\nAnother interesting observation s regarding the calculation of the metric. It is not a mean of per-row recall, rather it is recall calculated on hits across the entire dataset! (might be an important piece of information for anyone implementing the competition metric themselves!)
\nHere is the relevant piece of code from evaluate.py:
def recall_by_event_type(evalutated_events: dict, total_number_events: dict):\n clicks = 0\n carts = 0\n orders = 0\n for event in evalutated_events.values():\n if 'clicks' in event and event['clicks']:\n clicks += event['clicks']\n if 'carts' in event and event['carts']:\n carts += event['carts']\n if 'orders' in event and event['orders']:\n orders += event['orders']\n\n return {\n 'clicks': clicks / total_number_events['clicks'],\n 'carts': carts / total_number_events['carts'],\n 'orders': orders / total_number_events['orders']\n }\nAnd here is how I am calculating the metric based on the files I uploaded (and a submission).
\nThis code outputs the same result as the code in the organizer's repository! 🥳
\nsubmission['session'] = submission.session_type.apply(lambda x: int(x.split('_')[0]))\nsubmission['type'] = submission.session_type.apply(lambda x: x.split('_')[1])\nsubmission.labels = submission.labels.apply(lambda x: [int(i) for i in x.split(' ')[:20]])\n\ntest_labels = pd.read_parquet('out_processed/test_labels.parquet')\ntest_labels = test_labels.merge(submission, how='left', on=['session', 'type'])\ntest_labels['hits'] = test_labels.apply(lambda df: len(set(df.ground_truth).intersection(set(df.labels))), axis=1)\ntest_labels['gt_count'] = test_labels.ground_truth.str.len().clip(0,20)\n\nrecall_per_type = test_labels.groupby(['type'])['hits'].sum() / test_labels.groupby(['type'])['gt_count'].sum() \n\nscore = (recall_per_type * pd.Series({'clicks': 0.10, 'carts': 0.30, 'orders': 0.60})).sum()\nHope this can be of help 🙂
\nHey!
\nI created and uploaded onto Kaggle a dataset for local validation using code in the organizer's repo.
\nPlease find the dataset here.
\nI created the dataset passing the competition train data to the script and used the last 7 days to create the test set.
\nGenerally, the interesting bit here is that the test set created on the last week of the train set seems considerably easier to predict on than what we have in the competition.
\nAnother interesting observation s regarding the calculation of the metric. It is not a mean of per-row recall, rather it is recall calculated on hits across the entire dataset! (might be an important piece of information for anyone implementing the competition metric themselves!)
\nHere is the relevant piece of code from evaluate.py:
def recall_by_event_type(evalutated_events: dict, total_number_events: dict):\n clicks = 0\n carts = 0\n orders = 0\n for event in evalutated_events.values():\n if 'clicks' in event and event['clicks']:\n clicks += event['clicks']\n if 'carts' in event and event['carts']:\n carts += event['carts']\n if 'orders' in event and event['orders']:\n orders += event['orders']\n\n return {\n 'clicks': clicks / total_number_events['clicks'],\n 'carts': carts / total_number_events['carts'],\n 'orders': orders / total_number_events['orders']\n }\nAnd here is how I am calculating the metric based on the files I uploaded (and a submission).
\nThis code outputs the same result as the code in the organizer's repository! 🥳
\nsubmission['session'] = submission.session_type.apply(lambda x: int(x.split('_')[0]))\nsubmission['type'] = submission.session_type.apply(lambda x: x.split('_')[1])\nsubmission.labels = submission.labels.apply(lambda x: [int(i) for i in x.split(' ')[:20]])\n\ntest_labels = pd.read_parquet('out_processed/test_labels.parquet')\ntest_labels = test_labels.merge(submission, how='left', on=['session', 'type'])\ntest_labels['hits'] = test_labels.apply(lambda df: len(set(df.ground_truth).intersection(set(df.labels))), axis=1)\ntest_labels['gt_count'] = test_labels.ground_truth.str.len().clip(0,20)\n\nrecall_per_type = test_labels.groupby(['type'])['hits'].sum() / test_labels.groupby(['type'])['gt_count'].sum() \n\nscore = (recall_per_type * pd.Series({'clicks': 0.10, 'carts': 0.30, 'orders': 0.60})).sum()\nHope this can be of help 🙂
\nThis is the code I used to convert the data to parquet files.
This is the code I used to convert the data to parquet files.