{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Evaluation Metric\n\nThere has been some discussion in a number of thread forums. This is not a definitive kernel as I am just as unclear as many participants. However, it does seem to achieve a similar score on the `tiny-dev` set to what the same model achieves on the public test set on the leaderboard. So it could be something close to the metric being used for the LB."},{"metadata":{"trusted":true},"cell_type":"code","source":"import json\nimport pandas as pd","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The `tiny-dev` set fron NQ is quite useful and can be downloaded from google storage as follows:"},{"metadata":{"trusted":true},"cell_type":"code","source":"!gsutil cp -r gs://bert-nq/tiny-dev .\n!gunzip tiny-dev/*\n!ls tiny-dev -hl","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Please excuse the naming confusion, but theres also a `tinydev` dataset which is a submission csv based on my model's predictions from `nq-dev-sample.jsonl` downloaded above. To evaluate your own model, just make a private dataset with your own csv predictions from this json."},{"metadata":{"trusted":true},"cell_type":"code","source":"predictions = pd.read_csv('../input/tinydev/ken_predictions.csv', na_filter=False).set_index('example_id')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def long_annotations(example):\n    longs = [('%s:%s' % (l['start_token'],l['end_token']))\n                for l in [a['long_answer'] for a in example['annotations']]\n                if not l['candidate_index'] == -1\n            ]\n    return longs #list of long annotations","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def short_annotations(example):\n    shorts = [('%s:%s' % (s['start_token'],s['end_token']))\n              for s in \n              # sum(list_of_lists, []) is not very efficient gives an easy flat map for short lists\n              sum([a['short_answers'] for a in example['annotations']], [])\n             ]\n    return shorts #list of short annotations","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def yes_nos(example):\n    return [\n        yesno for yesno in [a['yes_no_answer'] for a in example['annotations']]\n        if not yesno == 'NONE'\n    ]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# This is the critical method where I guess at the competition metric.\nclass Score():\n    def __init__(self):\n        self.TP = 0\n        self.FP = 0\n        self.FN = 0\n        self.TN = 0\n    def F1(self):\n        return 2 * self.TP / (2 * self.TP + self.FP + self.FN)\n    def increment(self, prediction, annotations, yes_nos):\n        if prediction in yes_nos:\n            print(prediction, yes_nos)\n            self.TP += 1\n        elif len(prediction) > 0:\n            if prediction in annotations:\n                self.TP += 1\n            else:\n                self.FP += 1\n        elif len(annotations) == 0:\n            self.TN += 1\n        else:\n            self.FN +=1\n    def scores(self):\n        return 'TP = {}   FP = {}   FN = {}   TN = {}   F1 = {:.2f}'.format(\n            self.TP, self.FP, self.FN, self.TN, self.F1())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"long_score = Score()\nshort_score = Score()\ntotal_score = Score()\nfor example in map(json.loads, open('tiny-dev/nq-dev-sample.jsonl', 'r')):\n    long_pred = predictions.loc[str(example['example_id']) + '_long', 'PredictionString']\n    long_score.increment(long_pred, long_annotations(example), [])\n    total_score.increment(long_pred, long_annotations(example), [])\n    short_pred = predictions.loc[str(example['example_id']) + '_short', 'PredictionString']\n    short_score.increment(short_pred, short_annotations(example), yes_nos(example))\n    total_score.increment(short_pred, short_annotations(example), [])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(short_score.scores())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(long_score.scores())","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(total_score.scores() + ' (LB score)')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"kaggle","language":"python","name":"kaggle"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.5.3"}},"nbformat":4,"nbformat_minor":1}