{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Kendall Tau with `evaluate`\n\nThe Kendall Tau correlation used for this challenge is available as a community metric in 🤗 Evaluate. ","metadata":{}},{"cell_type":"markdown","source":"## Install","metadata":{}},{"cell_type":"code","source":"%%capture\n!pip install evaluate","metadata":{"execution":{"iopub.status.busy":"2022-06-14T13:07:37.260008Z","iopub.execute_input":"2022-06-14T13:07:37.260498Z","iopub.status.idle":"2022-06-14T13:07:52.726002Z","shell.execute_reply.started":"2022-06-14T13:07:37.260460Z","shell.execute_reply":"2022-06-14T13:07:52.724537Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Load metric\n\nWith `evaluate` anybody can add community metrics (see the [guide](https://huggingface.co/docs/evaluate/creating_and_sharing) in the documentation). We added the AI4Code evaluation metric as a community metric so anybody can easily use it which comes with an interactive widget and a metric card:\n\nhttps://huggingface.co/spaces/kaggle/ai4code\n\nUsing a community metric with evaluate is just a few lines of code:","metadata":{}},{"cell_type":"code","source":"import evaluate\n\nai4code_metric = evaluate.load(\"kaggle/ai4code\")","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-06-14T13:07:52.728207Z","iopub.execute_input":"2022-06-14T13:07:52.729583Z","iopub.status.idle":"2022-06-14T13:08:07.727928Z","shell.execute_reply.started":"2022-06-14T13:07:52.729532Z","shell.execute_reply":"2022-06-14T13:08:07.726615Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Compute result","metadata":{}},{"cell_type":"markdown","source":"With `evaluate` you can either pass all predictions/references directly to `compute`:","metadata":{}},{"cell_type":"code","source":"result = ai4code_metric.compute(\n    references=[[\"a\", \"b\", \"c\"], [\"x\", \"y\"]],\n    predictions=[[\"b\", \"a\", \"c\"], [\"x\", \"y\"]]\n)\n\nprint(result)","metadata":{"execution":{"iopub.status.busy":"2022-06-14T13:09:23.797346Z","iopub.execute_input":"2022-06-14T13:09:23.798274Z","iopub.status.idle":"2022-06-14T13:09:23.885162Z","shell.execute_reply.started":"2022-06-14T13:09:23.798224Z","shell.execute_reply":"2022-06-14T13:09:23.884256Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Or you can add single samples with `add` (or batches with `add_batch`):","metadata":{}},{"cell_type":"code","source":"ai4code_metric.add(references=[\"a\", \"b\", \"c\"],  predictions=[\"b\", \"a\", \"c\"])\nai4code_metric.add(references=[\"x\", \"y\"],  predictions=[\"x\", \"y\"])\n\nresult = ai4code_metric.compute()\nprint(result)","metadata":{"execution":{"iopub.status.busy":"2022-06-14T13:10:15.828170Z","iopub.execute_input":"2022-06-14T13:10:15.828642Z","iopub.status.idle":"2022-06-14T13:10:15.844173Z","shell.execute_reply.started":"2022-06-14T13:10:15.828605Z","shell.execute_reply":"2022-06-14T13:10:15.843275Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Save results\n\nThe `evaluate` library comes with a save function so you can also save the results locally as a JSON file with additional useful information for reproducibility:","metadata":{}},{"cell_type":"code","source":"path = evaluate.save(\"./result/\", **result, **{\"model_name\": \"my_model\"})\npath","metadata":{"execution":{"iopub.status.busy":"2022-06-14T13:19:22.407218Z","iopub.execute_input":"2022-06-14T13:19:22.407649Z","iopub.status.idle":"2022-06-14T13:19:22.440431Z","shell.execute_reply.started":"2022-06-14T13:19:22.407617Z","shell.execute_reply":"2022-06-14T13:19:22.439063Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import json\n\nwith open(path, \"r\") as f:\n    print(json.dumps(json.load(f), indent=4))","metadata":{"execution":{"iopub.status.busy":"2022-06-14T13:19:26.866940Z","iopub.execute_input":"2022-06-14T13:19:26.867653Z","iopub.status.idle":"2022-06-14T13:19:26.874873Z","shell.execute_reply.started":"2022-06-14T13:19:26.867606Z","shell.execute_reply":"2022-06-14T13:19:26.873734Z"},"trusted":true},"execution_count":null,"outputs":[]}]}