{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Summary of this notebook\n\nI'm affraid that this notebook has become so long.\nHere I summarize what we are gonna do.\n\n1. Import & Configration\n\nHere we import libraries and set some configurations as usual.\n\n2. Make Candidate\n\nHere we define functions which extract bird candidates for lightgbm stage training. \nThe output of melspectrogram multiclass classifier is used for that.\n\n3. Add features\n\nHere we define functions which make features for lightgbm stage training.\n\n4. Calculate birdcall probabilities (397dims per clip) from melspectrograms\n\nHere we define functions which calculate birdcall probabilities from melspectrograms.\nThey are used for training_soundscapes audios in this notebook.\nSpeaking of train_short_audio, we have already prepare birdcall probabilities data in csv format.\n(Check '../input/metadata-probability-v0525-2100')\n\n5. Training (Lightgbm)\n\nHere we define functions which train lightgbm models.\n\n6. Optimize Threshold\n\nHere we define functions which optimize the thresholds.\n\n7. Make Submission\n\nHere we define functions which make submission for Kaggle BirdCLEF 2021 Competition.\n\n8. Main\n\nHere we run the functions we have defined.\nTo take a quick look this notebook, this part is good place to start.\n\n# input & output of this notebook\n\n[input]\n\nbirdclef-2021 (original data)\n\nmelspectrogram multiclassifier models (Ⅰ)\n\nhttps://www.kaggle.com/namakemono/birdclef-groupby-author-05221040-728258\n\nhttps://www.kaggle.com/kami634/clefmodel\n\ntrain_short_audio birdcall probabilities calculated by melspectrogram multiclassifier models (Ⅰ)\n\nhttps://www.kaggle.com/namakemono/metadata-probability-v0525-2100\n\nresnest library\n\nhttps://www.kaggle.com/ttahara/resnest50-fast-package\n\nsklearn library (To use StratifiedGroupKfold, we have to install scikit-learn 1.0.dev0)\n\nhttps://www.kaggle.com/namakemono/scikit-learn-10dev0\n\n[output]\n\nsubmission.csv","metadata":{"papermill":{"duration":0.02392,"end_time":"2021-06-03T14:28:53.64268","exception":false,"start_time":"2021-06-03T14:28:53.61876","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Import & Configuration","metadata":{"papermill":{"duration":0.022238,"end_time":"2021-06-03T14:28:53.6874","exception":false,"start_time":"2021-06-03T14:28:53.665162","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import pandas as pd\n# Read the CSV file into a dataframe\ndf = pd.read_csv('/kaggle/input/birdclef-2021/test.csv')\n\n# Remove the \"site\" column and the last three columns\ndf = df.iloc[:, :-3]\n\n# Add a new column filled with \"nocall\"\ndf['birds'] = 'rudpig'\n\ndf.to_csv('submission.csv', encoding='utf-8', index=False)","metadata":{"execution":{"iopub.status.busy":"2023-06-02T17:27:33.210118Z","iopub.execute_input":"2023-06-02T17:27:33.210804Z","iopub.status.idle":"2023-06-02T17:27:33.248406Z","shell.execute_reply.started":"2023-06-02T17:27:33.210766Z","shell.execute_reply":"2023-06-02T17:27:33.247527Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Make Candidate\nhelper functions for making candidates","metadata":{"papermill":{"duration":0.023979,"end_time":"2021-06-03T14:29:32.886126","exception":false,"start_time":"2021-06-03T14:29:32.862147","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"making candidates main func","metadata":{"papermill":{"duration":0.024216,"end_time":"2021-06-03T14:29:32.996454","exception":false,"start_time":"2021-06-03T14:29:32.972238","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Add Features\nhelper functions for adding features","metadata":{"papermill":{"duration":0.024856,"end_time":"2021-06-03T14:29:33.120328","exception":false,"start_time":"2021-06-03T14:29:33.095472","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"making candidates main func","metadata":{}},{"cell_type":"markdown","source":"# Calculate birdcall probabilities (397dims per clip) from melspectrograms\nhelper functions for calculating birdcall probabilities","metadata":{"papermill":{"duration":0.024251,"end_time":"2021-06-03T14:29:33.296327","exception":false,"start_time":"2021-06-03T14:29:33.272076","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"calculating birdcall probabilities main func","metadata":{"papermill":{"duration":0.037678,"end_time":"2021-06-03T14:29:33.521001","exception":false,"start_time":"2021-06-03T14:29:33.483323","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Training (LightGBM)\nhelper function for training","metadata":{"papermill":{"duration":0.024599,"end_time":"2021-06-03T14:29:33.640248","exception":false,"start_time":"2021-06-03T14:29:33.615649","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"training main func","metadata":{"papermill":{"duration":0.024879,"end_time":"2021-06-03T14:29:33.748128","exception":false,"start_time":"2021-06-03T14:29:33.723249","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Optimize Threshold","metadata":{"papermill":{"duration":0.024325,"end_time":"2021-06-03T14:29:33.86862","exception":false,"start_time":"2021-06-03T14:29:33.844295","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"helper functions for optimizing threshold","metadata":{"papermill":{"duration":0.024112,"end_time":"2021-06-03T14:29:33.917545","exception":false,"start_time":"2021-06-03T14:29:33.893433","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"optimizing threshold main func","metadata":{"papermill":{"duration":0.02412,"end_time":"2021-06-03T14:29:34.026936","exception":false,"start_time":"2021-06-03T14:29:34.002816","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Make Submission\nhelper functions for making submission","metadata":{"papermill":{"duration":0.024158,"end_time":"2021-06-03T14:29:34.14762","exception":false,"start_time":"2021-06-03T14:29:34.123462","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"make_submission main func","metadata":{"papermill":{"duration":0.024182,"end_time":"2021-06-03T14:29:34.256677","exception":false,"start_time":"2021-06-03T14:29:34.232495","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"# Main","metadata":{"papermill":{"duration":0.024431,"end_time":"2021-06-03T14:29:34.372753","exception":false,"start_time":"2021-06-03T14:29:34.348322","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# prob_df","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# prob_df","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# candidate_df","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# best_th, best_nocall_th","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# submission_df = make_submission(\n#     candidate_df,\n#     prob_df,\n#     num_kfolds=config.num_kfolds,\n#     th=best_th,\n#     nocall_th=best_nocall_th,\n#     weights_filepath_dict=config.weights_filepath_dict,\n#     max_distance=config.max_distance\n# )","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# submission_df.to_csv(\"submission.csv\", index=False)","metadata":{"papermill":{"duration":0.056387,"end_time":"2021-06-03T14:33:49.310563","exception":false,"start_time":"2021-06-03T14:33:49.254176","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# !cp '/kaggle/input/birdclef-2021/sample_submission.csv' ./","metadata":{"papermill":{"duration":0.042051,"end_time":"2021-06-03T14:33:49.394309","exception":false,"start_time":"2021-06-03T14:33:49.352258","status":"completed"},"tags":[],"trusted":true},"execution_count":null,"outputs":[]}]}