{
  "id": 550255,
  "title": "sharing autoencoder with NaN values handling",
  "url": "/competitions/child-mind-institute-problematic-internet-use/discussion/550255",
  "author_name": "MJeremy",
  "post_date": "2024-12-06T08:43:28.021000",
  "votes": 7,
  "comment_count": 0,
  "views": 0,
  "content": "<p>using autoencoder to reduce overall dimissions to reduce overfitting</p>\n<pre><code> ():\n    device = torch.device(  torch.cuda.is_available()  )\n    ()\n\n    \n    scaler = StandardScaler()\n    df_scaled = scaler.fit_transform(df.fillna())  \n    nan_mask = ~np.isnan(df.values)  \n\n    \n    data_tensor = torch.FloatTensor(df_scaled).to(device)\n    mask_tensor = torch.FloatTensor(nan_mask).to(device)\n\n    input_dim = data_tensor.shape[]\n    autoencoder = AutoEncoder(input_dim, encoding_dim).to(device)  \n\n    criterion = nn.MSELoss(reduction=)  \n    optimizer = optim.Adam(autoencoder.parameters())\n\n     epoch  (epochs):\n         i  (, (data_tensor), batch_size):\n            batch = data_tensor[i : i + batch_size]\n            batch_mask = mask_tensor[i : i + batch_size]\n\n            optimizer.zero_grad()\n            reconstructed = autoencoder(batch)\n\n            \n            loss = criterion(reconstructed, batch)\n            masked_loss = (loss * batch_mask).() / batch_mask.()  \n\n            masked_loss.backward()\n            optimizer.step()\n\n         (epoch + ) %  == :\n            ()\n\n     torch.no_grad():\n        encoded_data = autoencoder.encoder(data_tensor).cpu().numpy()  \n\n    df_encoded = pd.DataFrame(encoded_data, columns=[  i  (encoded_data.shape[])])\n     df_encoded\n</code></pre>",
  "messages": [
    {
      "id": 3065014,
      "postDate": "2024-12-06T08:43:28.020Z",
      "content": "<p>using autoencoder to reduce overall dimissions to reduce overfitting</p>\n<pre><code> ():\n    device = torch.device(  torch.cuda.is_available()  )\n    ()\n\n    \n    scaler = StandardScaler()\n    df_scaled = scaler.fit_transform(df.fillna())  \n    nan_mask = ~np.isnan(df.values)  \n\n    \n    data_tensor = torch.FloatTensor(df_scaled).to(device)\n    mask_tensor = torch.FloatTensor(nan_mask).to(device)\n\n    input_dim = data_tensor.shape[]\n    autoencoder = AutoEncoder(input_dim, encoding_dim).to(device)  \n\n    criterion = nn.MSELoss(reduction=)  \n    optimizer = optim.Adam(autoencoder.parameters())\n\n     epoch  (epochs):\n         i  (, (data_tensor), batch_size):\n            batch = data_tensor[i : i + batch_size]\n            batch_mask = mask_tensor[i : i + batch_size]\n\n            optimizer.zero_grad()\n            reconstructed = autoencoder(batch)\n\n            \n            loss = criterion(reconstructed, batch)\n            masked_loss = (loss * batch_mask).() / batch_mask.()  \n\n            masked_loss.backward()\n            optimizer.step()\n\n         (epoch + ) %  == :\n            ()\n\n     torch.no_grad():\n        encoded_data = autoencoder.encoder(data_tensor).cpu().numpy()  \n\n    df_encoded = pd.DataFrame(encoded_data, columns=[  i  (encoded_data.shape[])])\n     df_encoded\n</code></pre>",
      "rawMarkdown": "using autoencoder to reduce overall dimissions to reduce overfitting\n\n```\ndef perform_autoencoder_with_nan_handling(df, encoding_dim=50, epochs=50, batch_size=32):\n    device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n    print(f\"Using device: {device}\")\n    \n    # Scale the data and preserve NaNs\n    scaler = StandardScaler()\n    df_scaled = scaler.fit_transform(df.fillna(0))  # Temporarily fill NaNs for scaling\n    nan_mask = ~np.isnan(df.values)  # Create a mask for valid entries (non-NaN)\n    \n    # Convert data and mask to tensors and move to GPU\n    data_tensor = torch.FloatTensor(df_scaled).to(device)\n    mask_tensor = torch.FloatTensor(nan_mask).to(device)\n    \n    input_dim = data_tensor.shape[1]\n    autoencoder = AutoEncoder(input_dim, encoding_dim).to(device)  # Move model to GPU\n    \n    criterion = nn.MSELoss(reduction='none')  # Use element-wise loss for masking\n    optimizer = optim.Adam(autoencoder.parameters())\n    \n    for epoch in range(epochs):\n        for i in range(0, len(data_tensor), batch_size):\n            batch = data_tensor[i : i + batch_size]\n            batch_mask = mask_tensor[i : i + batch_size]\n            \n            optimizer.zero_grad()\n            reconstructed = autoencoder(batch)\n            \n            # Compute loss only for valid values\n            loss = criterion(reconstructed, batch)\n            masked_loss = (loss * batch_mask).sum() / batch_mask.sum()  # Masked average loss\n            \n            masked_loss.backward()\n            optimizer.step()\n        \n        if (epoch + 1) % 10 == 0:\n            print(f'Epoch [{epoch + 1}/{epochs}], Loss: {masked_loss.item():.4f}')\n    \n    with torch.no_grad():\n        encoded_data = autoencoder.encoder(data_tensor).cpu().numpy()  # Move back to CPU for output\n    \n    df_encoded = pd.DataFrame(encoded_data, columns=[f'Enc_{i + 1}' for i in range(encoded_data.shape[1])])\n    return df_encoded\n```",
      "votes": 7
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "3065014": "using autoencoder to reduce overall dimissions to reduce overfitting\n\n```\ndef perform_autoencoder_with_nan_handling(df, encoding_dim=50, epochs=50, batch_size=32):\n    device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n    print(f\"Using device: {device}\")\n    \n    # Scale the data and preserve NaNs\n    scaler = StandardScaler()\n    df_scaled = scaler.fit_transform(df.fillna(0))  # Temporarily fill NaNs for scaling\n    nan_mask = ~np.isnan(df.values)  # Create a mask for valid entries (non-NaN)\n    \n    # Convert data and mask to tensors and move to GPU\n    data_tensor = torch.FloatTensor(df_scaled).to(device)\n    mask_tensor = torch.FloatTensor(nan_mask).to(device)\n    \n    input_dim = data_tensor.shape[1]\n    autoencoder = AutoEncoder(input_dim, encoding_dim).to(device)  # Move model to GPU\n    \n    criterion = nn.MSELoss(reduction='none')  # Use element-wise loss for masking\n    optimizer = optim.Adam(autoencoder.parameters())\n    \n    for epoch in range(epochs):\n        for i in range(0, len(data_tensor), batch_size):\n            batch = data_tensor[i : i + batch_size]\n            batch_mask = mask_tensor[i : i + batch_size]\n            \n            optimizer.zero_grad()\n            reconstructed = autoencoder(batch)\n            \n            # Compute loss only for valid values\n            loss = criterion(reconstructed, batch)\n            masked_loss = (loss * batch_mask).sum() / batch_mask.sum()  # Masked average loss\n            \n            masked_loss.backward()\n            optimizer.step()\n        \n        if (epoch + 1) % 10 == 0:\n            print(f'Epoch [{epoch + 1}/{epochs}], Loss: {masked_loss.item():.4f}')\n    \n    with torch.no_grad():\n        encoded_data = autoencoder.encoder(data_tensor).cpu().numpy()  # Move back to CPU for output\n    \n    df_encoded = pd.DataFrame(encoded_data, columns=[f'Enc_{i + 1}' for i in range(encoded_data.shape[1])])\n    return df_encoded\n```"
  }
}