{
  "id": 381078,
  "title": "Right approach for using sensor and pulse data: embedding ?",
  "url": "/competitions/icecube-neutrinos-in-deep-ice/discussion/381078",
  "author_name": "",
  "post_date": "2023-01-25T03:52:23.535425400Z",
  "votes": 2,
  "comment_count": 2,
  "views": 0,
  "content": "<ul>\n<li>My assumption: for predict direction, sensor data may be very important for suggest direction. As example, if a neutrino beam is come from west north to north, it will never activate a sensor in south pole. So by filtering sensor data, we can suggest some directions for neutrino beams.</li>\n<li>My experiment: I tried to <strong>embed a batch sensor data by PCA,</strong> then predict with validation data in same batch. But in this notebook: <a href=\"https://www.kaggle.com/astrung/pcaembed-gpu-catboost-optuna-approach\" target=\"_blank\">https://www.kaggle.com/astrung/pcaembed-gpu-catboost-optuna-approach</a> , my tuning process doesn't increase its accuracy, so <strong>it suggest that may be our embedding data isn't good enough to suggest differences between direction of beams.</strong> If my notebook is helpful for you, please upvote too.</li>\n<li>I don't think one hot encoding will be good, because we have so many sensor (~5500 sensors), and it make our data is so sparse. <br>\nSo can we have another way for better using sensor data ? Please sharing other ideal with us. May be I need to use better embedding (deep learning, BERT, ..) ?</li>\n</ul>",
  "messages": [
    {
      "id": "2114519",
      "postDate": "01/25/2023 03:52:23",
      "content": "<ul>\n<li>My assumption: for predict direction, sensor data may be very important for suggest direction. As example, if a neutrino beam is come from west north to north, it will never activate a sensor in south pole. So by filtering sensor data, we can suggest some directions for neutrino beams.</li>\n<li>My experiment: I tried to <strong>embed a batch sensor data by PCA,</strong> then predict with validation data in same batch. But in this notebook: <a href=\"https://www.kaggle.com/astrung/pcaembed-gpu-catboost-optuna-approach\" target=\"_blank\">https://www.kaggle.com/astrung/pcaembed-gpu-catboost-optuna-approach</a> , my tuning process doesn't increase its accuracy, so <strong>it suggest that may be our embedding data isn't good enough to suggest differences between direction of beams.</strong> If my notebook is helpful for you, please upvote too.</li>\n<li>I don't think one hot encoding will be good, because we have so many sensor (~5500 sensors), and it make our data is so sparse. <br>\nSo can we have another way for better using sensor data ? Please sharing other ideal with us. May be I need to use better embedding (deep learning, BERT, ..) ?</li>\n</ul>",
      "rawMarkdown": "* My assumption: for predict direction, sensor data may be very important for suggest direction. As example, if a neutrino beam is come from west north to north, it will never activate a sensor in south pole. So by filtering sensor data, we can suggest some directions for neutrino beams.\n* My experiment: I tried to **embed a batch sensor data by PCA,** then predict with validation data in same batch. But in this notebook: https://www.kaggle.com/astrung/pcaembed-gpu-catboost-optuna-approach , my tuning process doesn't increase its accuracy, so **it suggest that may be our embedding data isn't good enough to suggest differences between direction of beams.** If my notebook is helpful for you, please upvote too.\n* I don't think one hot encoding will be good, because we have so many sensor (~5500 sensors), and it make our data is so sparse. \n So can we have another way for better using sensor data ? Please sharing other ideal with us. May be I need to use better embedding (deep learning, BERT, ..) ?",
      "votes": null
    },
    {
      "id": "2221308",
      "postDate": "04/14/2023 06:13:12",
      "content": "<p>I also think that using PCA to embed the sensor data might not be the best approach, because PCA is a linear dimensionality reduction technique that might not capture the nonlinear relationships between the sensors and the direction. If you cluster the sensor data based on their charge, time, or position, and assign each cluster a label or a color. it might help you to visualize or identify the regions of the sensor data that are more likely to be activated by different directions of beams.</p>",
      "rawMarkdown": "I also think that using PCA to embed the sensor data might not be the best approach, because PCA is a linear dimensionality reduction technique that might not capture the nonlinear relationships between the sensors and the direction. If you cluster the sensor data based on their charge, time, or position, and assign each cluster a label or a color. it might help you to visualize or identify the regions of the sensor data that are more likely to be activated by different directions of beams.",
      "votes": null
    },
    {
      "id": "2224259",
      "postDate": "04/17/2023 06:39:29",
      "content": "<p>Put the memory issue aside I'd say that one-hot encoding is not that absurd, since a non-hit is also a valuable information. </p>",
      "rawMarkdown": "Put the memory issue aside I'd say that one-hot encoding is not that absurd, since a non-hit is also a valuable information.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2221308,
      "author_name": "yus002",
      "author_url": "",
      "post_date": "04/14/2023 06:13:12",
      "content": "<p>I also think that using PCA to embed the sensor data might not be the best approach, because PCA is a linear dimensionality reduction technique that might not capture the nonlinear relationships between the sensors and the direction. If you cluster the sensor data based on their charge, time, or position, and assign each cluster a label or a color. it might help you to visualize or identify the regions of the sensor data that are more likely to be activated by different directions of beams.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2224259,
      "author_name": "taqseorangpun",
      "author_url": "",
      "post_date": "04/17/2023 06:39:29",
      "content": "<p>Put the memory issue aside I'd say that one-hot encoding is not that absurd, since a non-hit is also a valuable information. </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2114519": "* My assumption: for predict direction, sensor data may be very important for suggest direction. As example, if a neutrino beam is come from west north to north, it will never activate a sensor in south pole. So by filtering sensor data, we can suggest some directions for neutrino beams.\n* My experiment: I tried to **embed a batch sensor data by PCA,** then predict with validation data in same batch. But in this notebook: https://www.kaggle.com/astrung/pcaembed-gpu-catboost-optuna-approach , my tuning process doesn't increase its accuracy, so **it suggest that may be our embedding data isn't good enough to suggest differences between direction of beams.** If my notebook is helpful for you, please upvote too.\n* I don't think one hot encoding will be good, because we have so many sensor (~5500 sensors), and it make our data is so sparse. \n So can we have another way for better using sensor data ? Please sharing other ideal with us. May be I need to use better embedding (deep learning, BERT, ..) ?",
    "2221308": "I also think that using PCA to embed the sensor data might not be the best approach, because PCA is a linear dimensionality reduction technique that might not capture the nonlinear relationships between the sensors and the direction. If you cluster the sensor data based on their charge, time, or position, and assign each cluster a label or a color. it might help you to visualize or identify the regions of the sensor data that are more likely to be activated by different directions of beams.",
    "2224259": "Put the memory issue aside I'd say that one-hot encoding is not that absurd, since a non-hit is also a valuable information."
  },
  "source": "meta"
}