{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Making a sample submission\n\nIn this notebook, we will introduce the very basics of how to run inference on a hidden test set and submit detection results to the competition.\n\nFirst thing we need to do is to look for test soundscapes. **The hidden test set will only appear if you submit the notebook.** Yet, you can use one test soundscapes to validate your workflow.","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"code","source":"import os\nimport json\nimport numpy as np\nimport pandas as pd\nimport librosa\n\n# First, load list of audio files. We could use 'test.csv' as well,\n# but for now, let's stick with parsing the test_soundscape folder.\ntest_audio_dir = '../input/birdclef-2022/test_soundscapes/'\nfile_list = [f.split('.')[0] for f in sorted(os.listdir(test_audio_dir))]\n\n# At the moment, there should only be a single soundscape visible.\n# During the submission re-run, all other hidden soundscapes\n# will be visible too and can be processed by your notebook.\nprint('Number of test soundscapes:', len(file_list))","metadata":{"execution":{"iopub.status.busy":"2022-02-15T13:29:19.697349Z","iopub.execute_input":"2022-02-15T13:29:19.697814Z","iopub.status.idle":"2022-02-15T13:29:19.705799Z","shell.execute_reply.started":"2022-02-15T13:29:19.697764Z","shell.execute_reply":"2022-02-15T13:29:19.705008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In this competition, not all training classes are equally important. During evaluation, only birds from the 'scored_birds.json' file need to be in the submission file.","metadata":{}},{"cell_type":"code","source":"# Load scored birds\nwith open('../input/birdclef-2022/scored_birds.json') as sbfile:\n    scored_birds = json.load(sbfile)","metadata":{"execution":{"iopub.status.busy":"2022-02-15T13:29:19.707286Z","iopub.execute_input":"2022-02-15T13:29:19.707975Z","iopub.status.idle":"2022-02-15T13:29:19.716894Z","shell.execute_reply.started":"2022-02-15T13:29:19.707922Z","shell.execute_reply":"2022-02-15T13:29:19.716006Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we need to open each test soundscape, process it, make predictions with our model, and store the results for all the birds evaluated. In this notebook, we will not use a trained model. Instead, we will give dummy scores to each target species and randomly assign \"True\" or \"False\" to indicate absence or presence.","metadata":{}},{"cell_type":"code","source":"# This is where we will store our results\npred = {'row_id': [], 'target': []}\n\n# Process audio files and make predictions\nfor afile in file_list:\n    \n    # Complete file path\n    path = test_audio_dir + afile + '.ogg'\n    \n    # Open file with librosa and split signal into 5-second chunks\n    # sig, rate = librosa.load(path)\n    # ...\n    \n    # Let's assume we have a list of 12 audio chunks (1min / 5s == 12 segments)\n    chunks = [[] for i in range(12)]\n    \n    # Make prediction for each chunk\n    # Each scored bird gets a random value in our case\n    # since we don't actually have a model\n    for i in range(len(chunks)):        \n        chunk_end_time = (i + 1) * 5\n        for bird in scored_birds:\n            \n            # This is our random prediction score for this bird\n            score = np.random.uniform()\n            \n            # Assemble the row_id which we need to do for each scored bird\n            row_id = afile + '_' + bird + '_' + str(chunk_end_time)\n            \n            # Put the result into our prediction dict and\n            # apply a \"confidence\" threshold of 0.5\n            pred['row_id'].append(row_id)\n            pred['target'].append(True if score > 0.5 else False)","metadata":{"execution":{"iopub.status.busy":"2022-02-15T13:29:19.718084Z","iopub.execute_input":"2022-02-15T13:29:19.718454Z","iopub.status.idle":"2022-02-15T13:29:19.728984Z","shell.execute_reply.started":"2022-02-15T13:29:19.718419Z","shell.execute_reply":"2022-02-15T13:29:19.728031Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, we need to save our results to a csv-file named 'submission.csv'.\n\n**Important:** Make sure to include 'True' or 'False' for *all* scored birds for *every* 5-second segment of *every* file. If the number of rows in your 'submission.csv' doesn't match the ground truth, submission will fail.","metadata":{}},{"cell_type":"code","source":"# Make a new data frame and look at some results        \nresults = pd.DataFrame(pred, columns = ['row_id', 'target'])\n\n# Quick sanity check\nprint(results.head()) \n    \n# Convert our results to csv\nresults.to_csv(\"submission.csv\", index=False)    ","metadata":{"execution":{"iopub.status.busy":"2022-02-15T13:29:19.730150Z","iopub.execute_input":"2022-02-15T13:29:19.730383Z","iopub.status.idle":"2022-02-15T13:29:19.751325Z","shell.execute_reply.started":"2022-02-15T13:29:19.730344Z","shell.execute_reply":"2022-02-15T13:29:19.750426Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we are ready to subnmit, and these are the steps we need to take:\n\n1. Go to notebook settings (on the left, below the \"Data\" explorer) and disable \"Internet\".\n2. Click \"Save Version\" (top right).\n3. Open notebook under the \"Code\" tab of the competition. It will show up under \"Your work\".\n4. Now click on the three dots in the upper right corner and select \"Submit to Competition\" (see screenshot below).\n5. Follow the on-screen instructions.\n6. Wait for the notebook to finish, results will show up under \"My Submissions\".\n\n![How to submit to the competition](https://tuc.cloud/index.php/s/z9eWEA8ZtbHki3i/preview)\n\nThat's it. Leave a comment if you have any remarks and please don't hesitate to start a new forum thread if you have any questions.","metadata":{}}]}