{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":89850,"databundleVersionId":11256103,"sourceType":"competition"}],"dockerImageVersionId":30918,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"execution":{"iopub.status.busy":"2025-03-22T03:24:49.886282Z","iopub.execute_input":"2025-03-22T03:24:49.886445Z","iopub.status.idle":"2025-03-22T03:24:52.059801Z","shell.execute_reply.started":"2025-03-22T03:24:49.886429Z","shell.execute_reply":"2025-03-22T03:24:52.059014Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Understanding the Evaluation Metric\n\nMacro-Averaged F1 Score per Sample:\n- F1 Score: Balances precision and recall.\n- Macro-Averaged: Averages the F1 scores across all samples (quadrats).\n\nKey Components of the F1 Score\n- Precision: [ \\text{Precision}_j = \\frac{\\text{TP}_j}{\\text{TP}_j + \\text{FP}_j} ]\n    - TP_j (True Positives): Correctly predicted species.\n    - FP_j (False Positives): Incorrectly predicted species.\n- Recall: [ \\text{Recall}_j = \\frac{\\text{TP}_j}{\\text{TP}_j + \\text{FN}_j} ]\n    - FN_j (False Negatives): Missed species.\n- F1 Score: [ \\text{F1}_j = \\frac{2 \\cdot \\text{Precision}_j \\cdot \\text{Recall}_j}{\\text{Precision}_j + \\text{Recall}_j} ]\n\nEvaluation:\nCompute F1 Score:\n- For each quadrat image, calculate TP, FP, and FN.\n- Compute the F1 score for each image.\n- Average the F1 scores across all images in each transect.\n- Compute the final score by averaging the F1 scores across all transects.","metadata":{}},{"cell_type":"markdown","source":"## Quick Evaluation","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nfrom sklearn.metrics import f1_score\n\n# Example ground truth and predictions\nground_truth = pd.DataFrame({\n    'quadrat_id': ['CBN-Pla-B1-20130724', 'CBN-PdlC-A1-20130807'],\n    'species_ids': [[1395806], [1351284, 1494911, 1381367, 1396535, 1412857, 1295807]]\n})\n\npredictions = pd.DataFrame({\n    'quadrat_id': ['CBN-Pla-B1-20130724', 'CBN-PdlC-A1-20130807'],\n    'species_ids': [[1395806], [1351284, 1494911, 1381367, 1396535, 1412857]]\n})\n\n# Function to compute F1 score for a single quadrat\ndef compute_f1_score(ground_truth_species, predicted_species):\n    tp = len(set(ground_truth_species) & set(predicted_species))\n    fp = len(set(predicted_species) - set(ground_truth_species))\n    fn = len(set(ground_truth_species) - set(predicted_species))\n    \n    precision = tp / (tp + fp) if (tp + fp) > 0 else 0\n    recall = tp / (tp + fn) if (tp + fn) > 0 else 0\n    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0\n    return f1\n\n# Compute F1 scores for each quadrat\nf1_scores = []\nfor i in range(len(ground_truth)):\n    ground_truth_species = ground_truth.iloc[i]['species_ids']\n    predicted_species = predictions.iloc[i]['species_ids']\n    f1 = compute_f1_score(ground_truth_species, predicted_species)\n    f1_scores.append(f1)\n\n# Compute the macro-averaged F1 score per sample\nmacro_averaged_f1_per_sample = sum(f1_scores) / len(f1_scores)\n\nprint(f\"Macro-averaged F1 score per sample: {macro_averaged_f1_per_sample}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-22T03:24:52.060455Z","iopub.execute_input":"2025-03-22T03:24:52.060730Z","iopub.status.idle":"2025-03-22T03:24:52.571476Z","shell.execute_reply.started":"2025-03-22T03:24:52.060714Z","shell.execute_reply":"2025-03-22T03:24:52.570590Z"}},"outputs":[],"execution_count":null}]}