{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **<span style='color:#A80808'>🎯 Goal</span>**\n\nIdentify plant species of the Americas from herbarium specimens.\n\n# **<span style='color:#A80808'>🔑 Metric</span>**\nSubmissions are evaluated using the macro F1 score. For each image Id, you should predict the corresponding image label (category_id) in the Predicted column. ","metadata":{"_uuid":"9231df95-1c3d-462a-87e5-636abd931aee","_cell_guid":"89b6cdac-2393-40a9-ab77-18f30e61df96","papermill":{"duration":0.021857,"end_time":"2022-02-15T21:24:34.348842","exception":false,"start_time":"2022-02-15T21:24:34.326985","status":"completed"},"tags":[],"trusted":true}},{"cell_type":"markdown","source":"# **<span style='color:#A80808'>💾 Data</span>**\n\nThis dataset uses the COCO dataset format with additional annotation fields. In addition to the species category labels, we also provide supercategory information.\n\nThe training set metadata (train_metadata.json) and test set metadata (test_metadata.json) are JSON files in the format below. Naturally, the test set metadata file omits the \"annotations\", \"categories,\" and \"regions\" elements.\n\n```\n\n{ \"annotations\" : [annotation], \"categories\" : [category], \"genera\" : [genus] \"images\" : [image], \"distances\" : [distance], \"licenses\" : [license], \"institutions\" : [institution] }\n\nannotation { \"image_id\" : int, \"category_id\" : int, \"genus_id\" : int, \"institution_id\" : int\n}\n\nimage { \"image_id\" : int, \"file_name\" : str, \"license\" : int }\n\ncategory { \"category_id\" : int, \"scientificName\" : str, # We also provide a super-category for each species. \"authors\" : str, # correspond to 'authors' field in the wcvp \"family\" : str, # correspond to 'family' field in the wcvp \"genus\" : str, # correspond to 'genus' field in the wcvp \"species\" : str, # correspond to 'species' field in the wcvp }\n\ngenera { \"genus_id\" : int, \"genus\" : str }\n\ndistance { # We provide the pairwise evolutionary distance between categories (genus_id0 < genus_id1). \"genus_id0\" : int,\n\"genus_id1\" : int,\n\"distance\" : float }\n\ninstitution { \"institution_id\" : int \"collectionCode\" : str }\n\nlicense { \"id\" : int, \"name\" : str, \"url\" : str }\n\n```\n\nThe training set images are organized in subfolders h22-train/images/subfolder1/subfolder2/image_id.jpg, where subfolder1 and subfolder2 comes from the first three and the last two digits of the image_id. Image_id is a result of combination between category_id and unique numbers that differentiates images within plant taxa.\n\nThe test set images are organized in subfolders test/images/subfolder/image id.jpg, where subfolder corresponds to the integer division of the image_id by 1000. For example, a test image with and image_id of 8005, can be found at h22-test/images/008/test-008005.jpg.","metadata":{"_uuid":"79616505-83b0-4ccd-b778-4d0f36f84c90","_cell_guid":"890dcec3-968b-4e20-8901-eb04eb1f1442","papermill":{"duration":0.020138,"end_time":"2022-02-15T21:24:34.390118","exception":false,"start_time":"2022-02-15T21:24:34.36998","status":"completed"},"tags":[],"trusted":true}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport plotly.express as px\nimport json","metadata":{"_uuid":"6e9e797c-c500-4212-a8df-7b7c1cd4d4df","_cell_guid":"fa1bdf26-3132-4a3f-b248-aa2e8fa51cdf","collapsed":false,"_kg_hide-input":true,"papermill":{"duration":1.64771,"end_time":"2022-02-15T21:24:36.058438","exception":false,"start_time":"2022-02-15T21:24:34.410728","status":"completed"},"tags":[],"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-02-23T20:43:41.047605Z","iopub.execute_input":"2022-02-23T20:43:41.048305Z","iopub.status.idle":"2022-02-23T20:43:43.160764Z","shell.execute_reply.started":"2022-02-23T20:43:41.048268Z","shell.execute_reply":"2022-02-23T20:43:43.159766Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **<span style='color:#A80808'>Train metadata</span>**","metadata":{"_uuid":"db5bf281-a959-4270-af55-aa5b6a90fbed","_cell_guid":"14da2a92-f65c-48d7-b523-81e1a571f1e1","papermill":{"duration":0.020625,"end_time":"2022-02-15T21:24:36.100013","exception":false,"start_time":"2022-02-15T21:24:36.079388","status":"completed"},"tags":[],"trusted":true}},{"cell_type":"code","source":"with open(\"../input/herbarium-2022-fgvc9/train_metadata.json\") as json_file:\n    train_metadata = json.load(json_file)\n    \nprint('The keys in train_metadata.json:')\nfor key in train_metadata.keys(): \n    print(key)\n    print(len(train_metadata[key]), 'rows')\n    print(train_metadata[key][0])","metadata":{"_uuid":"96aa830a-51e5-4961-bbbc-59e1a2728faf","_cell_guid":"d0982ed1-1f73-4cb3-be70-c6496f2af1e1","collapsed":false,"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-02-23T20:11:36.498055Z","iopub.execute_input":"2022-02-23T20:11:36.498322Z","iopub.status.idle":"2022-02-23T20:11:47.228517Z","shell.execute_reply.started":"2022-02-23T20:11:36.498294Z","shell.execute_reply":"2022-02-23T20:11:47.227812Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **<span style='color:#A80808'>Annotations</span>**","metadata":{"_uuid":"b2aa647c-1da6-429a-a83a-ccd4b5b2d6f3","_cell_guid":"04f4e51e-d88c-4514-93c1-63afd320b254","papermill":{"duration":0.021574,"end_time":"2022-02-15T21:24:36.322407","exception":false,"start_time":"2022-02-15T21:24:36.300833","status":"completed"},"tags":[],"trusted":true}},{"cell_type":"code","source":"annotations = pd.DataFrame()\nannotations['image_id'] = [annotation[\"image_id\"] for annotation in train_metadata[\"annotations\"]]\nannotations['category_id'] = [annotation[\"category_id\"] for annotation in train_metadata[\"annotations\"]]\nannotations['genus_id'] = [annotation[\"genus_id\"] for annotation in train_metadata[\"annotations\"]]\nannotations['institution_id'] = [annotation[\"institution_id\"] for annotation in train_metadata[\"annotations\"]]\nannotations.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-23T20:42:42.336151Z","iopub.execute_input":"2022-02-23T20:42:42.337147Z","iopub.status.idle":"2022-02-23T20:42:43.870969Z","shell.execute_reply.started":"2022-02-23T20:42:42.337104Z","shell.execute_reply":"2022-02-23T20:42:43.870387Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.histogram(annotations['category_id'],  marginal=\"violin\", nbins = 100, template=\"plotly_white\", color_discrete_sequence=['red'])\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-23T20:44:54.814321Z","iopub.execute_input":"2022-02-23T20:44:54.81468Z","iopub.status.idle":"2022-02-23T20:44:59.544999Z","shell.execute_reply.started":"2022-02-23T20:44:54.814646Z","shell.execute_reply":"2022-02-23T20:44:59.543738Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.histogram(annotations['genus_id'],  marginal=\"violin\", nbins = 100, template=\"plotly_white\", color_discrete_sequence=['blue'])\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-23T20:45:28.475226Z","iopub.execute_input":"2022-02-23T20:45:28.475973Z","iopub.status.idle":"2022-02-23T20:45:32.852686Z","shell.execute_reply.started":"2022-02-23T20:45:28.475936Z","shell.execute_reply":"2022-02-23T20:45:32.849261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.histogram(annotations['institution_id'],  marginal=\"violin\", nbins = 100, template=\"plotly_white\", color_discrete_sequence=['green'])\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-23T20:45:59.627388Z","iopub.execute_input":"2022-02-23T20:45:59.628024Z","iopub.status.idle":"2022-02-23T20:46:04.568294Z","shell.execute_reply.started":"2022-02-23T20:45:59.627985Z","shell.execute_reply":"2022-02-23T20:46:04.56715Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **<span style='color:#A80808'>Images</span>**","metadata":{}},{"cell_type":"code","source":"images = pd.DataFrame()\nimages['image_id'] = [image[\"image_id\"] for image in train_metadata[\"images\"]]\nimages['file_name'] = [f'../input/herbarium-2022-fgvc9/train_images/{image[\"file_name\"]}' for image in train_metadata[\"images\"]]\nimages['license'] = [image[\"license\"] for image in train_metadata[\"images\"]]\n\nprint(f'Shape of images: {images.shape}')\nimages.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-23T20:48:24.837419Z","iopub.execute_input":"2022-02-23T20:48:24.838058Z","iopub.status.idle":"2022-02-23T20:48:25.805773Z","shell.execute_reply.started":"2022-02-23T20:48:24.838016Z","shell.execute_reply":"2022-02-23T20:48:25.805208Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# There is only one license\nimages.license.unique()","metadata":{"execution":{"iopub.status.busy":"2022-02-23T20:49:26.096946Z","iopub.execute_input":"2022-02-23T20:49:26.097225Z","iopub.status.idle":"2022-02-23T20:49:26.108057Z","shell.execute_reply.started":"2022-02-23T20:49:26.097198Z","shell.execute_reply":"2022-02-23T20:49:26.107455Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **<span style='color:#A80808'>Categories</span>**","metadata":{}},{"cell_type":"code","source":"categories = pd.DataFrame()\ncategories['category_id'] = [category[\"category_id\"] for category in train_metadata[\"categories\"]]\ncategories['scientificName'] = [category[\"scientificName\"] for category in train_metadata[\"categories\"]]\ncategories['family'] = [category[\"family\"] for category in train_metadata[\"categories\"]]\ncategories['genus'] = [category[\"genus\"] for category in train_metadata[\"categories\"]]\ncategories['species'] = [category[\"species\"] for category in train_metadata[\"categories\"]]\ncategories['authors'] = [category[\"authors\"] for category in train_metadata[\"categories\"]]\n\nprint(f'Shape of categories: {categories.shape}')\ncategories.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-23T20:52:45.970015Z","iopub.execute_input":"2022-02-23T20:52:45.970385Z","iopub.status.idle":"2022-02-23T20:52:46.03689Z","shell.execute_reply.started":"2022-02-23T20:52:45.970352Z","shell.execute_reply":"2022-02-23T20:52:46.036003Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.histogram(categories['family'],  marginal=\"violin\", nbins = 100, template=\"plotly_white\", color_discrete_sequence=['red'])\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-23T20:53:59.677737Z","iopub.execute_input":"2022-02-23T20:53:59.678186Z","iopub.status.idle":"2022-02-23T20:54:00.269798Z","shell.execute_reply.started":"2022-02-23T20:53:59.67814Z","shell.execute_reply":"2022-02-23T20:54:00.268941Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.histogram(categories['genus'],  marginal=\"violin\", nbins = 100, template=\"plotly_white\", color_discrete_sequence=['blue'])\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-23T20:54:18.873733Z","iopub.execute_input":"2022-02-23T20:54:18.873987Z","iopub.status.idle":"2022-02-23T20:54:19.158093Z","shell.execute_reply.started":"2022-02-23T20:54:18.87396Z","shell.execute_reply":"2022-02-23T20:54:19.157138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.histogram(categories['species'],  marginal=\"violin\", nbins = 100, template=\"plotly_white\", color_discrete_sequence=['green'])\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-23T20:54:41.818636Z","iopub.execute_input":"2022-02-23T20:54:41.818917Z","iopub.status.idle":"2022-02-23T20:54:42.114509Z","shell.execute_reply.started":"2022-02-23T20:54:41.818889Z","shell.execute_reply":"2022-02-23T20:54:42.113631Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.histogram(categories['authors'],  marginal=\"violin\", nbins = 100, template=\"plotly_white\", color_discrete_sequence=['orange'])\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-23T20:55:17.759183Z","iopub.execute_input":"2022-02-23T20:55:17.759911Z","iopub.status.idle":"2022-02-23T20:55:18.065969Z","shell.execute_reply.started":"2022-02-23T20:55:17.759854Z","shell.execute_reply":"2022-02-23T20:55:18.065369Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **<span style='color:#A80808'>Genera</span>**","metadata":{}},{"cell_type":"code","source":"genera = pd.DataFrame()\ngenera['genus_id'] = [genus[\"genus_id\"] for genus in train_metadata[\"genera\"]]\ngenera['genus'] = [genus[\"genus\"] for genus in train_metadata[\"genera\"]]\n\nprint(f'Shape of genera: {genera.shape}')\ngenera.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-23T20:57:48.421526Z","iopub.execute_input":"2022-02-23T20:57:48.422238Z","iopub.status.idle":"2022-02-23T20:57:48.43887Z","shell.execute_reply.started":"2022-02-23T20:57:48.422198Z","shell.execute_reply":"2022-02-23T20:57:48.438116Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## **<span style='color:#A80808'>Institutions</span>**","metadata":{}},{"cell_type":"code","source":"institutions = pd.DataFrame()\ninstitutions['institution_id'] = [institution[\"institution_id\"] for institution in train_metadata[\"institutions\"]]\ninstitutions['collectionCode'] = [institution[\"collectionCode\"] for institution in train_metadata[\"institutions\"]]\n\nprint(f'Shape of institutions: {institutions.shape}')\ninstitutions.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-23T21:00:37.574153Z","iopub.execute_input":"2022-02-23T21:00:37.574471Z","iopub.status.idle":"2022-02-23T21:00:37.58848Z","shell.execute_reply.started":"2022-02-23T21:00:37.574437Z","shell.execute_reply":"2022-02-23T21:00:37.587825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"print(f'There are {institutions.collectionCode.nunique()} unique collection codes:\\n{institutions.collectionCode.unique()}')","metadata":{"execution":{"iopub.status.busy":"2022-02-23T21:03:20.471731Z","iopub.execute_input":"2022-02-23T21:03:20.472147Z","iopub.status.idle":"2022-02-23T21:03:20.478478Z","shell.execute_reply.started":"2022-02-23T21:03:20.472117Z","shell.execute_reply":"2022-02-23T21:03:20.4776Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **<span style='color:#A80808'>Distances</span>**","metadata":{}},{"cell_type":"code","source":"distances = pd.DataFrame()\ndistances['genus_id_x'] = [distance[\"genus_id_x\"] for distance in train_metadata[\"distances\"]]\ndistances['genus_id_y'] = [distance[\"genus_id_y\"] for distance in train_metadata[\"distances\"]]\ndistances['distance'] = [distance[\"distance\"] for distance in train_metadata[\"distances\"]]\n\nprint(f'Shape of distances: {distances.shape}')\ndistances.head()","metadata":{"execution":{"iopub.status.busy":"2022-02-23T21:05:42.922065Z","iopub.execute_input":"2022-02-23T21:05:42.923074Z","iopub.status.idle":"2022-02-23T21:05:49.732656Z","shell.execute_reply.started":"2022-02-23T21:05:42.923029Z","shell.execute_reply":"2022-02-23T21:05:49.731785Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = px.histogram(distances['distance'],  marginal=\"violin\", nbins = 100, template=\"plotly_white\", color_discrete_sequence=['blue'])\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-02-23T21:06:27.61521Z","iopub.execute_input":"2022-02-23T21:06:27.616014Z","iopub.status.idle":"2022-02-23T21:06:44.746063Z","shell.execute_reply.started":"2022-02-23T21:06:27.615953Z","shell.execute_reply":"2022-02-23T21:06:44.744719Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# **<span style='color:#A80808'>🏆 Submission</span>**","metadata":{"_uuid":"0bae0019-82fa-4cc1-bc2f-6db6568b052d","_cell_guid":"04253909-5eda-489a-9b34-c5f7c11bf2c8","papermill":{"duration":0.036761,"end_time":"2022-02-15T21:24:43.788468","exception":false,"start_time":"2022-02-15T21:24:43.751707","status":"completed"},"tags":[],"trusted":true}},{"cell_type":"code","source":"submission = pd.read_csv('../input/herbarium-2022-fgvc9/sample_submission.csv')\nsubmission['Predicted'] = np.random.randint(0,6932)\nsubmission.to_csv('submission.csv', index=False)\nsubmission.head()","metadata":{"_uuid":"8f1db095-0c60-48ad-a71b-6d3d9b76416e","_cell_guid":"402ce0da-465d-4f45-a5f3-ba898b195e25","collapsed":false,"papermill":{"duration":0.060765,"end_time":"2022-02-15T21:24:43.886523","exception":false,"start_time":"2022-02-15T21:24:43.825758","status":"completed"},"tags":[],"jupyter":{"outputs_hidden":false},"execution":{"iopub.status.busy":"2022-02-16T08:01:17.501269Z","iopub.execute_input":"2022-02-16T08:01:17.501547Z","iopub.status.idle":"2022-02-16T08:01:17.574951Z","shell.execute_reply.started":"2022-02-16T08:01:17.501519Z","shell.execute_reply":"2022-02-16T08:01:17.574143Z"},"trusted":true},"execution_count":null,"outputs":[]}]}