![\"image\"width=\"40%\"](\"https://www.kaggle.com/competitions/91844/images/header\")
\n\nIntroduction
\n\n*Biodiversity monitoring is essential for conservation, but traditional surveys are expensive and time-consuming. This competition leverages passive acoustic monitoring (PAM) and machine learning to identify species based on their sounds, enabling large-scale, cost-effective biodiversity assessments. Participants will develop models to classify under-studied species with limited labeled data, contributing to conservation efforts in Colombia’s Magdalena Valley. The region, a biodiversity hotspot, faces deforestation threats, making effective monitoring crucial. Fundación Biodiversa Colombia leads efforts to protect and restore these ecosystems, with El Silencio Natural Reserve serving as a key conservation site.*\n\n***Overall Goal:** The code aims to build a simple bird sound classifier for the BirdCLEF competition. It involves data loading, exploration, feature extraction, model training, evaluation, and submission file creation.*","metadata":{}},{"cell_type":"markdown","source":"#Import Libraries
","metadata":{}},{"cell_type":"code","source":"import os\nimport re\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport librosa\nimport librosa.display\nimport IPython.display as ipd\nimport soundfile as sf\nfrom sklearn.metrics import roc_auc_score, roc_curve\nfrom sklearn.preprocessing import MultiLabelBinarizer\nfrom sklearn.model_selection import train_test_split\nimport xgboost as xgb \nimport lightgbm as lgb \n\nimport warnings\nwarnings.filterwarnings(\"ignore\")\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-13T03:17:30.794350Z","iopub.execute_input":"2025-03-13T03:17:30.794673Z","iopub.status.idle":"2025-03-13T03:17:34.274466Z","shell.execute_reply.started":"2025-03-13T03:17:30.794648Z","shell.execute_reply":"2025-03-13T03:17:34.273738Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#Load Data
\n\n- Load Data: Read CSV files (metadata, taxonomy, submission).","metadata":{}},{"cell_type":"code","source":"# Define paths\nINPUT_PATH = '/kaggle/input/birdclef-2025/'\nTRAIN_AUDIO_PATH = os.path.join(INPUT_PATH, 'train_audio')\nTEST_SOUNDSCAPES_PATH = os.path.join(INPUT_PATH, 'test_soundscapes')\nTRAIN_SOUNDSCAPES_PATH = os.path.join(INPUT_PATH, 'train_soundscapes')\n\n# Load data\ntaxonomy = pd.read_csv(os.path.join(INPUT_PATH, 'taxonomy.csv'))\ntrain_meta = pd.read_csv(os.path.join(INPUT_PATH, 'train.csv'))\nsample_submission = pd.read_csv(os.path.join(INPUT_PATH, 'sample_submission.csv'))\nrecording_locations = pd.read_csv(os.path.join(INPUT_PATH, 'recording_location.txt'))\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-13T03:17:45.314598Z","iopub.execute_input":"2025-03-13T03:17:45.315172Z","iopub.status.idle":"2025-03-13T03:17:45.453193Z","shell.execute_reply.started":"2025-03-13T03:17:45.315144Z","shell.execute_reply":"2025-03-13T03:17:45.452501Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#Data Preprocessing
\n\n- Cleans and prepares the training metadata. Specifically, it extracts secondary labels and constructs file paths for audio files.\n- The `preprocess_train_meta` function uses regular expressions (`re.findall`) to extract secondary labels from strings and `os.path.join` to create full file paths.","metadata":{}},{"cell_type":"code","source":"# Data Preprocessing\ndef preprocess_train_meta(df):\n \"\"\"Preprocesses the training metadata.\"\"\"\n df['secondary_labels'] = df['secondary_labels'].apply(lambda x: re.findall(r\"'(\\w+)'\", x))\n df['len_sec_labels'] = df['secondary_labels'].map(len)\n df['file_path'] = df.apply(lambda row: os.path.join(TRAIN_AUDIO_PATH, row['filename']), axis=1)\n return df\n\ntrain_meta = preprocess_train_meta(train_meta)\n\nprint(\"Train Meta Shape:\", train_meta.shape)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-13T03:17:49.019106Z","iopub.execute_input":"2025-03-13T03:17:49.019442Z","iopub.status.idle":"2025-03-13T03:17:49.211693Z","shell.execute_reply.started":"2025-03-13T03:17:49.019414Z","shell.execute_reply":"2025-03-13T03:17:49.210735Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(\"Train Meta Head:\")\ntrain_meta.head().style.background_gradient(cmap='YlOrBr')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-13T03:17:54.834238Z","iopub.execute_input":"2025-03-13T03:17:54.834511Z","iopub.status.idle":"2025-03-13T03:17:54.904728Z","shell.execute_reply.started":"2025-03-13T03:17:54.834489Z","shell.execute_reply":"2025-03-13T03:17:54.904017Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(\"Taxonomy Head:\")\ntaxonomy.head().style.background_gradient(cmap='plasma')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-13T03:17:59.001695Z","iopub.execute_input":"2025-03-13T03:17:59.002024Z","iopub.status.idle":"2025-03-13T03:17:59.012169Z","shell.execute_reply.started":"2025-03-13T03:17:59.002002Z","shell.execute_reply":"2025-03-13T03:17:59.011308Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"print(\"Recording Locations Head:\")\nrecording_locations.head().style.background_gradient(cmap='plasma')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-13T03:18:02.562919Z","iopub.execute_input":"2025-03-13T03:18:02.563207Z","iopub.status.idle":"2025-03-13T03:18:02.570596Z","shell.execute_reply.started":"2025-03-13T03:18:02.563186Z","shell.execute_reply":"2025-03-13T03:18:02.569842Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#Visualization
\n\n**Audio Visualization:**\n- Visualizes audio waveforms and spectrograms to get a sense of the data.\n- The `visualize_audio` function uses `librosa.load` to load audio, `librosa.display.waveshow` to plot waveforms, and `librosa.display.specshow` to plot spectrograms. It displays these plots using `matplotlib.pyplot`.\n\n**Soundscapes Visualization:**\n- Visualizes soundscape audio files similarly to individual training examples.\n- The `visualize_soundscape` function mirrors `visualize_audio` but operates on soundscape files. `get_available_soundscapes` is used to locate soundscape files.\n\n**Recording Location Map:**\n- Creates an interactive map showing the geographic locations of the audio recordings.\n- Uses the `folium` library to create a map centered on the average latitude and longitude of the recordings. It adds markers to the map for each recording location.\n\n**Taxonomy Visualization:**\n- Visualizes the distribution of different categories within the taxonomy (e.g., class names, common names).\n- The `plot_taxonomy_distribution` function uses `pandas.value_counts` to count the occurrences of each category and `seaborn.barplot` to create a bar plot of the top N categories.","metadata":{}},{"cell_type":"code","source":"# Audio Visualization (Train Data Examples)\ndef visualize_audio(file_paths, titles):\n \"\"\"Visualizes audio waveforms and spectrograms.\"\"\"\n plt.figure(figsize=(15, 4 * len(file_paths)))\n for i, file_path in enumerate(file_paths):\n try:\n y, sr = librosa.load(file_path)\n plt.subplot(len(file_paths), 2, 2 * i + 1)\n librosa.display.waveshow(y, sr=sr)\n plt.title(f'Waveform: {titles[i]}')\n\n plt.subplot(len(file_paths), 2, 2 * i + 2)\n D = librosa.stft(y)\n S_db = librosa.amplitude_to_db(np.abs(D), ref=np.max)\n librosa.display.specshow(S_db, sr=sr, x_axis='time', y_axis='log')\n plt.title(f'Spectrogram: {titles[i]}')\n except Exception as e:\n print(f\"Error processing {file_path}: {e}\")\n plt.tight_layout()\n plt.show()\n\n# Select 4 audio examples\naudio_examples = train_meta.sample(4)\nfile_paths = audio_examples['file_path'].tolist()\ntitles = audio_examples['primary_label'].tolist()\n\nprint(\"\\nVisualizing Train Audio Examples:\")\nvisualize_audio(file_paths, titles)\n\n# Soundscapes Visualization\ndef visualize_soundscape(file_paths, titles):\n \"\"\"Visualizes soundscape audio waveforms and spectrograms.\"\"\"\n plt.figure(figsize=(15, 4 * len(file_paths)))\n for i, file_path in enumerate(file_paths):\n try:\n y, sr = librosa.load(file_path)\n plt.subplot(len(file_paths), 2, 2 * i + 1)\n librosa.display.waveshow(y, sr=sr)\n plt.title(f'Soundscape Waveform: {titles[i]}')\n\n plt.subplot(len(file_paths), 2, 2 * i + 2)\n D = librosa.stft(y)\n S_db = librosa.amplitude_to_db(np.abs(D), ref=np.max)\n librosa.display.specshow(S_db, sr=sr, x_axis='time', y_axis='log')\n plt.title(f'Soundscape Spectrogram: {titles[i]}')\n except Exception as e:\n print(f\"Error processing {file_path}: {e}\")\n plt.tight_layout()\n plt.show()\n\n# Select 4 diverse soundscape examples\ndef get_available_soundscapes(soundscapes_path):\n \"\"\"Returns a list of available soundscape files.\"\"\"\n try:\n soundscape_files = [f for f in os.listdir(soundscapes_path) if f.endswith('.ogg')]\n soundscape_files = [os.path.join(soundscapes_path, f) for f in soundscape_files]\n return soundscape_files\n except FileNotFoundError:\n print(f\"Error: The directory {soundscapes_path} was not found.\")\n return []\n except Exception as e:\n print(f\"Error listing soundscape files: {e}\")\n return []\n\navailable_soundscapes = get_available_soundscapes(TRAIN_SOUNDSCAPES_PATH)\n\nif len(available_soundscapes) >= 4:\n soundscape_files = available_soundscapes[:4] # Select the first 4\n soundscape_titles = [f'Soundscape {i+1}' for i in range(4)]\n\n print(\"\\nVisualizing Soundscapes:\")\n visualize_soundscape(soundscape_files, soundscape_titles)\nelse:\n print(\"\\nNot enough soundscapes available to visualize 4 examples.\")\n\n# Recording Location Map\nimport folium\n\ndef create_location_map(train_meta, n_samples=200):\n \"\"\"Creates an interactive map of recording locations using Folium.\"\"\"\n location_df = train_meta[['latitude', 'longitude']].dropna().sample(n_samples)\n latitudes = location_df['latitude'].values.tolist()\n longitudes = location_df['longitude'].values.tolist()\n\n # Calculate the average location to center the map\n avg_lat = sum(latitudes) / len(latitudes)\n avg_lon = sum(longitudes) / len(longitudes)\n\n # Create the map\n m = folium.Map(location=[avg_lat, avg_lon], zoom_start=6)\n\n # Add markers for each location\n for lat, lon in zip(latitudes, longitudes):\n folium.CircleMarker(location=[lat, lon], radius=5, color='blue', fill=True, fill_color='blue').add_to(m)\n\n return m\n\nprint(\"\\nCreating Recording Location Map:\")\nmap_obj = create_location_map(train_meta)\nmap_obj # Display the map\n\n# Taxonomy Visualization\ndef plot_taxonomy_distribution(taxonomy, column_name, top_n=10):\n \"\"\"Plots the distribution of taxonomy categories.\"\"\"\n plt.figure(figsize=(12, 6))\n counts = taxonomy[column_name].value_counts().nlargest(top_n)\n sns.barplot(x=counts.index, y=counts.values, palette='viridis')\n plt.title(f'Top {top_n} {column_name} Distribution')\n plt.xlabel(column_name)\n plt.ylabel('Frequency')\n plt.xticks(rotation=45, ha='right')\n plt.tight_layout()\n plt.show()\n\nprint(\"\\nVisualizing Taxonomy Distributions:\")\nplot_taxonomy_distribution(taxonomy, 'class_name')\nplot_taxonomy_distribution(taxonomy, 'common_name')","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-13T03:18:07.132462Z","iopub.execute_input":"2025-03-13T03:18:07.132739Z","iopub.status.idle":"2025-03-13T03:18:49.345199Z","shell.execute_reply.started":"2025-03-13T03:18:07.132717Z","shell.execute_reply":"2025-03-13T03:18:49.344405Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#Feature Extraction
\n\n- Extracts Mel-Frequency Cepstral Coefficients (MFCCs) from audio files, which are a common audio feature used in machine learning.\n\n- The `extract_mfcc` function uses `librosa.load` to load audio, `librosa.feature.mfcc` to compute MFCCs, and `numpy.mean` to average the MFCCs over time.\n","metadata":{}},{"cell_type":"code","source":"# Feature Extraction (Simple MFCC)\ndef extract_mfcc(file_path, sr=22050, n_mfcc=20):\n \"\"\"Extracts MFCC features from an audio file.\"\"\"\n try:\n y, sr = librosa.load(file_path, sr=sr)\n mfccs = librosa.feature.mfcc(y=y, sr=sr, n_mfcc=n_mfcc)\n mfccs_processed = np.mean(mfccs.T, axis=0) # Average across time\n except Exception as e:\n # Comment out this line to suppress error messages\n # print(f\"Error processing {file_path}: {e}\")\n return None\n return mfccs_processed\n\n# Example MFCC extraction\nexample_file = train_meta['file_path'].iloc[0]\nmfccs = extract_mfcc(example_file)\nprint(\"\\nMFCC Features Example:\", mfccs)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-13T03:18:55.234711Z","iopub.execute_input":"2025-03-13T03:18:55.235261Z","iopub.status.idle":"2025-03-13T03:18:56.420364Z","shell.execute_reply.started":"2025-03-13T03:18:55.235237Z","shell.execute_reply":"2025-03-13T03:18:56.419578Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#Model Training
\n\n- `train_model` **Function Modification:**\n\n - Added a `model_type` parameter to the `train_model` function. This allows you to specify whether you want to train an XGBoost or LightGBM model. The default can be set as needed.\n - Implemented conditional logic to train either an XGBoost model (using `xgb.XGBClassifier`) or a LightGBM model (using `lgb.LGBMClassifier`) based on the `model_type` parameter. Crucially, `use_label_encoder=False` and `eval_metric='logloss'` were added to the `XGBClassifier` constructor. This is essential to prevent warnings and ensure the model trains correctly with string labels. `eval_metric` is required. I've also added a random state for reproducibility. A similar `random_state` was added to the `LGBMClassifier`.\n - Added a `ValueError` exception if an invalid `model_type` is specified.\n\n","metadata":{}},{"cell_type":"code","source":"# Model Training (XGBoost or LightGBM)\ndef train_model(train_meta, model_type='xgboost', n_samples=500, n_mfcc=20):\n \"\"\"Trains a XGBoost or LightGBM model.\"\"\"\n # Sample a subset of data for faster training\n train_subset = train_meta.sample(n_samples, random_state=42)\n\n # Extract MFCC features\n features = []\n labels = []\n for index, row in train_subset.iterrows():\n mfccs = extract_mfcc(row['file_path'], n_mfcc=n_mfcc)\n if mfccs is not None:\n features.append(mfccs)\n labels.append(row['primary_label'])\n\n X = np.array(features)\n y = np.array(labels)\n\n # Train/Test split\n X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n\n\n if model_type == 'xgboost':\n model = xgb.XGBClassifier(use_label_encoder=False, eval_metric='logloss', random_state=42) #added eval_metric as it is needed\n model.fit(X_train, y_train)\n\n elif model_type == 'lightgbm':\n model = lgb.LGBMClassifier(random_state=42)\n model.fit(X_train, y_train)\n\n else:\n raise ValueError(\"Invalid model_type. Choose 'xgboost' or 'lightgbm'.\")\n\n return model, X_test, y_test, y_train\n\n#Choose between 'xgboost' or 'lightgbm'\nmodel, X_test, y_test, y_train = train_model(train_meta, model_type='lightgbm') #or 'xgboost'\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-13T03:19:03.588997Z","iopub.execute_input":"2025-03-13T03:19:03.589293Z","iopub.status.idle":"2025-03-13T03:19:50.205429Z","shell.execute_reply.started":"2025-03-13T03:19:03.589272Z","shell.execute_reply":"2025-03-13T03:19:50.204682Z"},"_kg_hide-output":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#Macro-averaged ROC-AUC Evaluation
\n\n- Evaluates the model using macro-averaged ROC-AUC, which is suitable for multi-class classification problems.\n - The `evaluate_model` function predicts probabilities for each class using `model.predict_proba`.\n - Binarizes the true labels using `MultiLabelBinarizer` to convert them into a one-vs-all format.\n - Calculates the ROC-AUC score for each class using `roc_auc_score` and plots the ROC curves.\n - Computes the macro-averaged ROC-AUC score by averaging the ROC-AUC scores for all classes, handling potential `ValueError` and `IndexError` exceptions.\n","metadata":{}},{"cell_type":"code","source":"# Macro-averaged ROC-AUC Evaluation\ndef evaluate_model(model, X_test, y_test, labels, y_train):\n \"\"\"Evaluates the model using macro-averaged ROC-AUC.\"\"\"\n\n # Get predictions\n y_pred_proba = model.predict_proba(X_test)\n\n # Binarize the true labels (fitting on both train and test labels to ensure all classes are present)\n mlb = MultiLabelBinarizer(classes=labels)\n mlb.fit([[label] for label in np.concatenate([y_train, y_test])])\n y_test_bin = mlb.transform([[label] for label in y_test])\n\n # Calculate ROC AUC for each class\n roc_auc_scores = []\n fprs, tprs, thresholds = [], [], [] # Store ROC curve data\n\n for i, label in enumerate(labels):\n try:\n # Get the index of the label in the MultiLabelBinarizer's classes_\n label_index = list(mlb.classes_).index(label)\n\n # Check if there's only one class present in the true labels\n if len(np.unique(y_test_bin[:, label_index])) <= 1:\n raise ValueError(f\"Only one class present in y_true for label {label}. ROC AUC score is not defined in that case.\")\n\n # Calculate ROC AUC\n roc_auc = roc_auc_score(y_test_bin[:, label_index], y_pred_proba[:, list(mlb.classes_).index(label)])\n roc_auc_scores.append(roc_auc)\n\n # Calculate ROC curve\n fpr, tpr, threshold = roc_curve(y_test_bin[:, label_index], y_pred_proba[:, list(mlb.classes_).index(label)])\n fprs.append(fpr)\n tprs.append(tpr)\n thresholds.append(threshold)\n\n except ValueError as e:\n #Remove this line if you want to see the error messages\n #print(f\"ValueError for label {label}: {e}\") # Added error message\n roc_auc_scores.append(np.nan)\n fprs.append(None)\n tprs.append(None)\n thresholds.append(None)\n\n except IndexError as e:\n #Remove this line if you want to see the error messages\n #print(f\"IndexError for label {label}: {e}\")\n roc_auc_scores.append(np.nan)\n fprs.append(None)\n tprs.append(None)\n thresholds.append(None)\n\n # Calculate the macro-averaged ROC AUC, ignoring NaN values\n macro_roc_auc = np.nanmean(roc_auc_scores)\n print(f\"Macro-Averaged ROC AUC: {macro_roc_auc:.4f}\")\n\n # Plot ROC curves\n plt.figure(figsize=(8, 6))\n for i, label in enumerate(labels):\n if fprs[i] is not None and tprs[i] is not None:\n plt.plot(fprs[i], tprs[i], label=f'{label} (AUC = {roc_auc_scores[i]:.2f})')\n plt.plot([0, 1], [0, 1], 'k--', label='Random')\n plt.xlabel('False Positive Rate')\n plt.ylabel('True Positive Rate')\n plt.title('ROC Curves for Each Class')\n plt.legend(loc='lower right')\n plt.show()\n\n return macro_roc_auc\n\n# Prepare labels for MultiLabelBinarizer\nlabels = train_meta['primary_label'].unique()\n\n# Evaluate the model\nmacro_roc_auc = evaluate_model(model, X_test, y_test, labels, y_train)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-13T03:20:06.457656Z","iopub.execute_input":"2025-03-13T03:20:06.457974Z","iopub.status.idle":"2025-03-13T03:20:06.874816Z","shell.execute_reply.started":"2025-03-13T03:20:06.457949Z","shell.execute_reply":"2025-03-13T03:20:06.873728Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"#Prediction and Submission
\n\n- Predicts bird presence in the test soundscapes and creates a submission file in the required format.\n- The `predict_and_submit` function iterates through the sample submission file.\n- For each soundscape, it extracts MFCC features.\n- Uses the trained model to predict the probabilities of each bird species being present.\n- Creates a Pandas DataFrame in the correct format for submission and saves it to a CSV file. It also handles `KeyError` exceptions that might arise if certain `row_id` values are missing.","metadata":{}},{"cell_type":"code","source":"# Prediction and Submission\ndef predict_and_submit(model, sample_submission, train_meta, labels, n_mfcc=20):\n \"\"\"Predicts bird presence and creates a submission file.\"\"\"\n predictions = {} # Store predictions\n\n # Create a mapping from file_id to audio file path in train_meta\n file_id_to_path = {row['filename'].replace('.ogg', ''): row['file_path'] for _, row in train_meta.iterrows()}\n\n for index, row in sample_submission.iterrows():\n try:\n # Extract file_id from row_id (required format is soundscape_{file_id}_{time})\n file_id = row['row_id'].split('_')[1]\n\n # Construct the audio path using file_id and the mapping\n audio_path = os.path.join(TEST_SOUNDSCAPES_PATH, file_id + '.ogg')\n\n # Or if the audio file is available in the train_meta data\n # audio_path = file_id_to_path.get(file_id, None) # Try to fetch from train_meta\n # if audio_path is None:\n # audio_path = os.path.join(TEST_SOUNDSCAPES_PATH, file_id + '.ogg') # Fallback to TEST_SOUNDSCAPES\n\n mfccs = extract_mfcc(audio_path, n_mfcc=n_mfcc)\n\n if mfccs is None:\n # Handle missing or corrupted audio files\n prediction_values = [0.01] * len(labels) # Give small default probability\n else:\n # Get the predicted probabilities for each class\n prediction_values = model.predict_proba(mfccs.reshape(1, -1))[0] # Predict probabilities\n\n # Create a dictionary mapping labels to prediction probabilities\n label_predictions = dict(zip(labels, prediction_values))\n predictions[row['row_id']] = label_predictions\n\n except KeyError as e:\n print(f\"KeyError in predict_and_submit: {e}. Skipping row {row['row_id']}\")\n # Create a dictionary with default probabilities for all labels\n label_predictions = {label: 0.01 for label in labels} # small probability to all labels\n predictions[row['row_id']] = label_predictions\n except FileNotFoundError as e:\n print(f\"FileNotFoundError in predict_and_submit: {e}. Skipping row {row['row_id']}\")\n # Create a dictionary with default probabilities for all labels\n label_predictions = {label: 0.01 for label in labels} # small probability to all labels\n predictions[row['row_id']] = label_predictions\n\n\n # Create submission DataFrame\n submission_data = []\n for row_id, label_predictions in predictions.items():\n row_data = {'row_id': row_id}\n row_data.update(label_predictions)\n submission_data.append(row_data)\n\n submission_df = pd.DataFrame(submission_data)\n submission_df = submission_df.set_index('row_id')\n\n # Ensure that the order of columns in the submission matches the sample_submission\n cols = sample_submission.columns[1:]\n submission_df = submission_df[cols]\n\n submission_df.to_csv('submission.csv')\n print(\"Submission file created successfully!\")\n return submission_df\n\n# Predict and submit\nsubmission_df = predict_and_submit(model, sample_submission, train_meta, labels)\n\n# Print the head of the created submission file\nsubmission = pd.read_csv('submission.csv')\nprint(\"Submission File Head:\")\nsubmission.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-13T03:20:26.110222Z","iopub.execute_input":"2025-03-13T03:20:26.110519Z","iopub.status.idle":"2025-03-13T03:20:27.193239Z","shell.execute_reply.started":"2025-03-13T03:20:26.110494Z","shell.execute_reply":"2025-03-13T03:20:27.191860Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import os\nimport re\nimport pandas as pd\nimport plotly.graph_objects as go\nfrom scipy.interpolate import interp1d\nimport plotly.io as pio\nimport plotly.graph_objects as go\n\npio.renderers.default = 'iframe'\n\n# Define Paths \nINPUT_PATH = './' \nTRAIN_AUDIO_PATH = './' \n\n# Load the training metadata\ntrain_meta = pd.read_csv(os.path.join(INPUT_PATH, '/kaggle/input/birdclef-2025/train.csv'))\n\n\n# Data Preprocessing\ndef preprocess_train_meta(df):\n \"\"\"Preprocesses the training metadata.\"\"\"\n df['secondary_labels'] = df['secondary_labels'].apply(lambda x: re.findall(r\"'(\\w+)'\", x))\n df['len_sec_labels'] = df['secondary_labels'].map(len)\n df['file_path'] = df.apply(lambda row: os.path.join(TRAIN_AUDIO_PATH, row['filename']), axis=1)\n return df\n\n\ntrain_meta = preprocess_train_meta(train_meta)\n\n# Data aggregation and preparation\ntrain_meta_grouped = train_meta.groupby(['primary_label', 'latitude', 'longitude']).count().reset_index()[\n ['primary_label', 'scientific_name', 'latitude', 'longitude']].rename(columns={'scientific_name': 'count'})\n\n\ndf = train_meta.merge(train_meta_grouped, on=['primary_label', 'latitude', 'longitude'], how='left').dropna(\n subset=['count'])\ndf['count'] = df['count'].astype('int')\n\nvalues_list = df['count'].values.tolist()\n\n# Radius scaling using interpolation\ninterpolation = interp1d([1, max(values_list)], [3, 20]) # Adjust range [min_radius, max_radius] as needed\nradius = interpolation(values_list)\n\n# Color scale selection (using Plotly's built-in options for better aesthetics)\ncolor_scale = \"Rainbow\" # \"Hot\", \"Viridis\", \"Plasma\", \"Cividis\", \"Rainbow\"\n\n# Densitymapbox plot with enhanced aesthetics\nfig = go.Figure(go.Densitymapbox(\n lat=df['latitude'],\n lon=df['longitude'],\n z=df['count'],\n radius=radius,\n colorscale=color_scale, # Use chosen colorscale\n zmin=min(df['count']), # Explicitly set zmin and zmax for consistent color mapping\n zmax=max(df['count']),\n opacity=0.7, # Adjust opacity for better visualization\n colorbar=dict(title=\"Observation Count\") # Add a title to the colorbar\n))\n\n# Map layout customization\nfig.update_layout(\n title=\"Geographic Distribution of Primary Labels\", # Adding a descriptive title\n title_x=0.5, # Center the title\n mapbox_style=\"carto-positron\", # Choose a suitable basemap style: 'open-street-map', 'carto-positron', 'stamen-terrain', 'white-bg'\n height=800,\n mapbox={\n 'center': {'lat': df['latitude'].mean(), 'lon': df['longitude'].mean()}, # Center on the data\n 'zoom': 2, # Adjust initial zoom level\n 'accesstoken': \"pk.eyJ1IjoiZXJpY21mYXJsaW5nIiwiYSI6ImNqOHB2eTMxOTAza2EzMm1xeDFiaG9zNnoifQ.xLWC_nwyxG2WbNlWr33oIA\"\n },\n margin={\"r\": 0, \"t\": 50, \"l\": 0, \"b\": 0}, # Adjust margins for better appearance\n template=\"plotly_white\" # Use a clean template\n)\n\nfig.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-13T03:20:38.625942Z","iopub.execute_input":"2025-03-13T03:20:38.626298Z","iopub.status.idle":"2025-03-13T03:20:40.411775Z","shell.execute_reply.started":"2025-03-13T03:20:38.626269Z","shell.execute_reply":"2025-03-13T03:20:40.411017Z"},"_kg_hide-input":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}