{
  "id": 412410,
  "title": "Find the perfect ensemble weights",
  "url": "/competitions/birdclef-2023/discussion/412410",
  "author_name": "",
  "post_date": "2023-05-23T16:22:13.798087800Z",
  "votes": null,
  "comment_count": 1,
  "views": 0,
  "content": "<p><strong>Here is a suggestion, if you haven't made your ensemble pipeline yet, of a way to find weights that might be a bit more optimal than a simple [1/num_models for model in models]:</strong></p>\n<pre><code> ():\n     model  models:\n        model.()\n\n    valid_running_loss = \n    valid_running_correct = \n    labels_stacked = torch.empty().to(device).long()\n    preds_stacked = torch.empty().to(device).long()\n    counter = \n     torch.no_grad():\n         i, data  tqdm((valid_loader), total=(valid_loader)):\n            counter += \n\n            image, labels = data\n\n            labels = labels.to(device)\n\n            labels_encoded = torch.tensor(np.eye(num_classes)[labels.cpu().numpy().reshape(-)]).to(device)\n            outputs = [model(image)  model  models]\n\n            \n            outputs = torch.stack(outputs, dim=)\n            outputs = torch.(outputs * torch.tensor(weights).view(-, , ).to(device), dim=)\n\n\n             (criterion, nn.CrossEntropyLoss):\n                loss = criterion(outputs, labels)\n\n            valid_running_loss += loss.item()\n\n             torch.no_grad():\n                preds_encoded = torch.softmax(outputs, dim = )\n                preds_stacked = torch.cat([preds_stacked,preds_encoded], dim =)\n                labels_stacked = torch.cat([labels_stacked,labels_encoded], dim =)\n\n    epoch_loss = valid_running_loss / counter\n    epoch_cmap = padded_cmap(pd.DataFrame(labels_stacked.cpu().detach().numpy()),pd.DataFrame(preds_stacked.cpu().detach().numpy()))\n    (epoch_cmap)\n     epoch_loss, epoch_cmap\n\n\n ():\n    num_models = (models)\n    weights_values = [, , ]  \n    weights_combinations = (product(weights_values, repeat=num_models))\n\n    \n    \n    models_names = [  i  (num_models)]\n    weights_df = pd.DataFrame(columns=models_names + [])\n\n     weights  weights_combinations:\n        \n        weights = np.array(weights)\n        weights = weights / np.(weights)\n\n        \n        _, cmap = validate_ensemble(models, valid_loader, criterion, device, weights)\n\n        \n        weights_df.loc[(weights_df)] = (weights) + [cmap]\n\n    \n    best_row = weights_df.loc[weights_df[].idxmax()]\n    best_weights = best_row[models_names].values\n\n     best_weights, weights_df\n</code></pre>\n<p>You will have to set a number of weights to try and see what works best with your models</p>",
  "messages": [
    {
      "id": "2271111",
      "postDate": "05/23/2023 16:22:13",
      "content": "<p><strong>Here is a suggestion, if you haven't made your ensemble pipeline yet, of a way to find weights that might be a bit more optimal than a simple [1/num_models for model in models]:</strong></p>\n<pre><code> ():\n     model  models:\n        model.()\n\n    valid_running_loss = \n    valid_running_correct = \n    labels_stacked = torch.empty().to(device).long()\n    preds_stacked = torch.empty().to(device).long()\n    counter = \n     torch.no_grad():\n         i, data  tqdm((valid_loader), total=(valid_loader)):\n            counter += \n\n            image, labels = data\n\n            labels = labels.to(device)\n\n            labels_encoded = torch.tensor(np.eye(num_classes)[labels.cpu().numpy().reshape(-)]).to(device)\n            outputs = [model(image)  model  models]\n\n            \n            outputs = torch.stack(outputs, dim=)\n            outputs = torch.(outputs * torch.tensor(weights).view(-, , ).to(device), dim=)\n\n\n             (criterion, nn.CrossEntropyLoss):\n                loss = criterion(outputs, labels)\n\n            valid_running_loss += loss.item()\n\n             torch.no_grad():\n                preds_encoded = torch.softmax(outputs, dim = )\n                preds_stacked = torch.cat([preds_stacked,preds_encoded], dim =)\n                labels_stacked = torch.cat([labels_stacked,labels_encoded], dim =)\n\n    epoch_loss = valid_running_loss / counter\n    epoch_cmap = padded_cmap(pd.DataFrame(labels_stacked.cpu().detach().numpy()),pd.DataFrame(preds_stacked.cpu().detach().numpy()))\n    (epoch_cmap)\n     epoch_loss, epoch_cmap\n\n\n ():\n    num_models = (models)\n    weights_values = [, , ]  \n    weights_combinations = (product(weights_values, repeat=num_models))\n\n    \n    \n    models_names = [  i  (num_models)]\n    weights_df = pd.DataFrame(columns=models_names + [])\n\n     weights  weights_combinations:\n        \n        weights = np.array(weights)\n        weights = weights / np.(weights)\n\n        \n        _, cmap = validate_ensemble(models, valid_loader, criterion, device, weights)\n\n        \n        weights_df.loc[(weights_df)] = (weights) + [cmap]\n\n    \n    best_row = weights_df.loc[weights_df[].idxmax()]\n    best_weights = best_row[models_names].values\n\n     best_weights, weights_df\n</code></pre>\n<p>You will have to set a number of weights to try and see what works best with your models</p>",
      "rawMarkdown": "**Here is a suggestion, if you haven't made your ensemble pipeline yet, of a way to find weights that might be a bit more optimal than a simple [1/num_models for model in models]:**\n\n```python\ndef validate_ensemble(models, valid_loader, criterion, device,weights):\n    for model in models:\n        model.eval()\n\n    valid_running_loss = 0.0\n    valid_running_correct = 0\n    labels_stacked = torch.empty(0).to(device).long()\n    preds_stacked = torch.empty(0).to(device).long()\n    counter = 0\n    with torch.no_grad():\n        for i, data in tqdm(enumerate(valid_loader), total=len(valid_loader)):\n            counter += 1\n\n            image, labels = data\n\n            labels = labels.to(device)\n\n            labels_encoded = torch.tensor(np.eye(num_classes)[labels.cpu().numpy().reshape(-1)]).to(device)\n            outputs = [model(image) for model in models]\n            \n            # Ensemble prediction is weighted average of individual model predictions\n            outputs = torch.stack(outputs, dim=0)\n            outputs = torch.sum(outputs * torch.tensor(weights).view(-1, 1, 1).to(device), dim=0)\n\n\n            if isinstance(criterion, nn.CrossEntropyLoss):\n                loss = criterion(outputs, labels)\n                \n            valid_running_loss += loss.item()\n\n            with torch.no_grad():\n                preds_encoded = torch.softmax(outputs, dim = 1)\n                preds_stacked = torch.cat([preds_stacked,preds_encoded], dim =0)\n                labels_stacked = torch.cat([labels_stacked,labels_encoded], dim =0)\n\n    epoch_loss = valid_running_loss / counter\n    epoch_cmap = padded_cmap(pd.DataFrame(labels_stacked.cpu().detach().numpy()),pd.DataFrame(preds_stacked.cpu().detach().numpy()))\n    print(epoch_cmap)\n    return epoch_loss, epoch_cmap\n\n\ndef find_optimal_weights(models, valid_loader, criterion, device):\n    num_models = len(models)\n    weights_values = [0.00001, 0.5, 1.0]  # Customize this list as needed\n    weights_combinations = list(product(weights_values, repeat=num_models))\n\n    # Initialize a DataFrame to store the weights and cmap values.\n    # The columns of the DataFrame will be the model names and 'cmap'.\n    models_names = [f'model_{i+1}' for i in range(num_models)]\n    weights_df = pd.DataFrame(columns=models_names + ['cmap'])\n\n    for weights in weights_combinations:\n        # Normalize the weights so they sum to 1\n        weights = np.array(weights)\n        weights = weights / np.sum(weights)\n\n        # Calculate the cmap for this set of weights\n        _, cmap = validate_ensemble(models, valid_loader, criterion, device, weights)\n        \n        # Add the weights and cmap value to the DataFrame.\n        weights_df.loc[len(weights_df)] = list(weights) + [cmap]\n\n    # Find the weights with the best cmap\n    best_row = weights_df.loc[weights_df['cmap'].idxmax()]\n    best_weights = best_row[models_names].values\n\n    return best_weights, weights_df\n\n```\n\nYou will have to set a number of weights to try and see what works best with your models",
      "votes": null
    },
    {
      "id": "2271280",
      "postDate": "05/23/2023 18:28:30",
      "content": "<p>Very interesting - how was your experience? Did this get you up the LB? </p>\n<p>Best,<br>\nJan</p>",
      "rawMarkdown": "Very interesting - how was your experience? Did this get you up the LB? \n\nBest,\nJan",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2271280,
      "author_name": "janbrederecke",
      "author_url": "",
      "post_date": "05/23/2023 18:28:30",
      "content": "<p>Very interesting - how was your experience? Did this get you up the LB? </p>\n<p>Best,<br>\nJan</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2271111": "**Here is a suggestion, if you haven't made your ensemble pipeline yet, of a way to find weights that might be a bit more optimal than a simple [1/num_models for model in models]:**\n\n```python\ndef validate_ensemble(models, valid_loader, criterion, device,weights):\n    for model in models:\n        model.eval()\n\n    valid_running_loss = 0.0\n    valid_running_correct = 0\n    labels_stacked = torch.empty(0).to(device).long()\n    preds_stacked = torch.empty(0).to(device).long()\n    counter = 0\n    with torch.no_grad():\n        for i, data in tqdm(enumerate(valid_loader), total=len(valid_loader)):\n            counter += 1\n\n            image, labels = data\n\n            labels = labels.to(device)\n\n            labels_encoded = torch.tensor(np.eye(num_classes)[labels.cpu().numpy().reshape(-1)]).to(device)\n            outputs = [model(image) for model in models]\n            \n            # Ensemble prediction is weighted average of individual model predictions\n            outputs = torch.stack(outputs, dim=0)\n            outputs = torch.sum(outputs * torch.tensor(weights).view(-1, 1, 1).to(device), dim=0)\n\n\n            if isinstance(criterion, nn.CrossEntropyLoss):\n                loss = criterion(outputs, labels)\n                \n            valid_running_loss += loss.item()\n\n            with torch.no_grad():\n                preds_encoded = torch.softmax(outputs, dim = 1)\n                preds_stacked = torch.cat([preds_stacked,preds_encoded], dim =0)\n                labels_stacked = torch.cat([labels_stacked,labels_encoded], dim =0)\n\n    epoch_loss = valid_running_loss / counter\n    epoch_cmap = padded_cmap(pd.DataFrame(labels_stacked.cpu().detach().numpy()),pd.DataFrame(preds_stacked.cpu().detach().numpy()))\n    print(epoch_cmap)\n    return epoch_loss, epoch_cmap\n\n\ndef find_optimal_weights(models, valid_loader, criterion, device):\n    num_models = len(models)\n    weights_values = [0.00001, 0.5, 1.0]  # Customize this list as needed\n    weights_combinations = list(product(weights_values, repeat=num_models))\n\n    # Initialize a DataFrame to store the weights and cmap values.\n    # The columns of the DataFrame will be the model names and 'cmap'.\n    models_names = [f'model_{i+1}' for i in range(num_models)]\n    weights_df = pd.DataFrame(columns=models_names + ['cmap'])\n\n    for weights in weights_combinations:\n        # Normalize the weights so they sum to 1\n        weights = np.array(weights)\n        weights = weights / np.sum(weights)\n\n        # Calculate the cmap for this set of weights\n        _, cmap = validate_ensemble(models, valid_loader, criterion, device, weights)\n        \n        # Add the weights and cmap value to the DataFrame.\n        weights_df.loc[len(weights_df)] = list(weights) + [cmap]\n\n    # Find the weights with the best cmap\n    best_row = weights_df.loc[weights_df['cmap'].idxmax()]\n    best_weights = best_row[models_names].values\n\n    return best_weights, weights_df\n\n```\n\nYou will have to set a number of weights to try and see what works best with your models",
    "2271280": "Very interesting - how was your experience? Did this get you up the LB? \n\nBest,\nJan"
  },
  "source": "meta"
}