![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F5517090%2Fa5fd25af7c5dbf46f7b732c1ab815194%2Foverall_pipeline.png?generation=1775626772760115&alt=media\")
I would like to thank the organizers for putting together the competition, and for addressing the early dataset issues so quickly.
\nMost of the public leaderboard eventually converged around 0.9115 (public) / 0.9119 (private), so I was happy to finish a bit above that score.
It was a fun competition, and I learned a lot from it. Thank you!
\nTLDR: This notebook reproduces my final submission, and the full implementation is available in my personal repository.
\nThe dataset is highly structured, and in the original CSV files it is dominated by two exact identity tuples.
\nMore concretely, I build a 5-field fingerprint from common[0].Mn, common[0].Md, common[0].Opt, common[0].Vr, and common[0].SN. The two dominant tuples correspond to the 10 kW and 100 kW simulator rows, which I refer to as canon10 and canon100, and I use that fingerprint to split the data into canon10, canon100, and a small noncanonical bucket.
This distinction is very important for the rest of the pipeline. The two canonical families are modeled separately, while the noncanonical rows are handled outside the main learned-model path because they already form a very strong anomaly signal in training.
\nFrom 665 selected source columns, I build semantic features that make the DER structure explicit: missingness and schema-integrity signals, measurement-vs-rating/setting consistency, control-compliance features, protection and curve features, and AC/DC consistency checks.
For each canonical family, the final predictor is a small ensemble of XGBoost on the semantic numeric table and CatBoost on a narrower raw + categorical table, with the blend and threshold tuned directly for F2.
\n
High level view of the pipeline. Rows are first split by device fingerprint; the two canonical families go through family-specific feature engineering and models, while the small noncanonical bucket is handled separately.
\nThe raw dataset has 723 features, but many are fully empty or not especially useful for the final model. In particular, 182 features are completely null in both train and test, while there are no fully empty rows. In practice, the useful signal is not in dropping rows, but in being selective about columns and preserving informative missingness.
A second key observation is that the data is overwhelmingly concentrated around two exact fingerprints from the original CSV identity columns:
\n('DERSec', 'DER Simulator', '10 kW DER', '1.2.3', 'SN-Three-Phase') -> canon10('DERSec', 'DER Simulator 100 kW', '1.2.3.1', '1.0.0', '1100058974') -> canon100| Bucket | \nDefinition | \nTrain rows | \nTest rows | \nHow I use it | \n
|---|---|---|---|---|
canon10 | \nExact match to the canonical 10 kW simulator fingerprint | \n1,181,604 | \n505,993 | \nFamily-specific learned models + overrides | \n
canon100 | \nExact match to the canonical 100 kW simulator fingerprint | \n1,168,218 | \n501,106 | \nFamily-specific learned models + overrides | \n
other / noncanonical | \nAnything else | \n13,341 | \n5,686 | \nIdentity-based anomaly bucket | \n
This matters because all 13,341 noncanonical training rows are anomalous. So I did not treat other as a third modeling family. Instead, I used it as a strong identity-based anomaly signal.
The main feature groups are:
\nI also used a small set of high-confidence hard overrides. They are simply anomaly signatures that were clean enough in training to justify overriding the learned model.
\nThe most important case is the noncanonical bucket itself. If the 5-field fingerprint does not exactly match one of the two canonical simulator identities, the row is assigned to other, which acts as an identity-based anomaly bucket rather than a learned family.
Inside the canonical families, the overrides are limited to a few high-precision patterns such as:
\nAn important implementation detail is that candidate overrides are audited on the training data. Only the most precise ones remain true hard overrides; the rest can still help as features without forcing the final prediction.
\nFor canon10 and canon100, I train separate models. This worked better than a single global classifier because the two canonical devices operate in different regimes and have different local patterns.
The main model is XGBoost on the full semantic numeric feature set. The companion model is CatBoost on a narrower raw + categorical table containing selected raw numeric columns, raw identity fields, the full device fingerprint, and additional identity / missingness-engineered features.
\nI also add family-specific regressors trained on normal rows to predict quantities such as active power, apparent power, reactive power, power factor, and current. Their residuals become new features for the final classifiers. For validation, I use both a primary 5-fold split based on Id and an audit split based on a hashed representation of the operating scenario.
Since the competition metric is F2, I did not use the default 0.5 threshold. For each canonical family, I search directly over out-of-fold probabilities to find the threshold that maximizes F2. I do the same for the XGBoost / CatBoost blend weight, and keep the family-specific operating point that performs well on both the primary and audit splits.
To develop the solution, I used uv, ruff, Docker, and Codex CLI. Most of the work was done on a MacBook Pro, without a GPU.
\nTo keep the results reproducible, I seeded the full pipeline and used a Kaggle official Docker image for local development. In the GitHub repository, the solution can be reproduced by building the pinned Docker image and running the run_docker.sh script, which executes the code inside the container and writes the final submission.csv.
I am an applied mathematician turned into software developer, currently based in Madrid. My interests are AI, machine learning, numerical simulation, and optimization.
\nIf you would like to reach out, you can find me on LinkedIn, or via Gmail as alvaroborrasf.
Thank you again and looking forward to seeing the approaches from other participants!
", "messages": [ { "id": "3437808", "postDate": "04/08/2026 06:03:48", "content": "I would like to thank the organizers for putting together the competition, and for addressing the early dataset issues so quickly.
\nMost of the public leaderboard eventually converged around 0.9115 (public) / 0.9119 (private), so I was happy to finish a bit above that score.
It was a fun competition, and I learned a lot from it. Thank you!
\nTLDR: This notebook reproduces my final submission, and the full implementation is available in my personal repository.
\nThe dataset is highly structured, and in the original CSV files it is dominated by two exact identity tuples.
\nMore concretely, I build a 5-field fingerprint from common[0].Mn, common[0].Md, common[0].Opt, common[0].Vr, and common[0].SN. The two dominant tuples correspond to the 10 kW and 100 kW simulator rows, which I refer to as canon10 and canon100, and I use that fingerprint to split the data into canon10, canon100, and a small noncanonical bucket.
This distinction is very important for the rest of the pipeline. The two canonical families are modeled separately, while the noncanonical rows are handled outside the main learned-model path because they already form a very strong anomaly signal in training.
\nFrom 665 selected source columns, I build semantic features that make the DER structure explicit: missingness and schema-integrity signals, measurement-vs-rating/setting consistency, control-compliance features, protection and curve features, and AC/DC consistency checks.
For each canonical family, the final predictor is a small ensemble of XGBoost on the semantic numeric table and CatBoost on a narrower raw + categorical table, with the blend and threshold tuned directly for F2.
\n
High level view of the pipeline. Rows are first split by device fingerprint; the two canonical families go through family-specific feature engineering and models, while the small noncanonical bucket is handled separately.
\nThe raw dataset has 723 features, but many are fully empty or not especially useful for the final model. In particular, 182 features are completely null in both train and test, while there are no fully empty rows. In practice, the useful signal is not in dropping rows, but in being selective about columns and preserving informative missingness.
A second key observation is that the data is overwhelmingly concentrated around two exact fingerprints from the original CSV identity columns:
\n('DERSec', 'DER Simulator', '10 kW DER', '1.2.3', 'SN-Three-Phase') -> canon10('DERSec', 'DER Simulator 100 kW', '1.2.3.1', '1.0.0', '1100058974') -> canon100| Bucket | \nDefinition | \nTrain rows | \nTest rows | \nHow I use it | \n
|---|---|---|---|---|
canon10 | \nExact match to the canonical 10 kW simulator fingerprint | \n1,181,604 | \n505,993 | \nFamily-specific learned models + overrides | \n
canon100 | \nExact match to the canonical 100 kW simulator fingerprint | \n1,168,218 | \n501,106 | \nFamily-specific learned models + overrides | \n
other / noncanonical | \nAnything else | \n13,341 | \n5,686 | \nIdentity-based anomaly bucket | \n
This matters because all 13,341 noncanonical training rows are anomalous. So I did not treat other as a third modeling family. Instead, I used it as a strong identity-based anomaly signal.
The main feature groups are:
\nI also used a small set of high-confidence hard overrides. They are simply anomaly signatures that were clean enough in training to justify overriding the learned model.
\nThe most important case is the noncanonical bucket itself. If the 5-field fingerprint does not exactly match one of the two canonical simulator identities, the row is assigned to other, which acts as an identity-based anomaly bucket rather than a learned family.
Inside the canonical families, the overrides are limited to a few high-precision patterns such as:
\nAn important implementation detail is that candidate overrides are audited on the training data. Only the most precise ones remain true hard overrides; the rest can still help as features without forcing the final prediction.
\nFor canon10 and canon100, I train separate models. This worked better than a single global classifier because the two canonical devices operate in different regimes and have different local patterns.
The main model is XGBoost on the full semantic numeric feature set. The companion model is CatBoost on a narrower raw + categorical table containing selected raw numeric columns, raw identity fields, the full device fingerprint, and additional identity / missingness-engineered features.
\nI also add family-specific regressors trained on normal rows to predict quantities such as active power, apparent power, reactive power, power factor, and current. Their residuals become new features for the final classifiers. For validation, I use both a primary 5-fold split based on Id and an audit split based on a hashed representation of the operating scenario.
Since the competition metric is F2, I did not use the default 0.5 threshold. For each canonical family, I search directly over out-of-fold probabilities to find the threshold that maximizes F2. I do the same for the XGBoost / CatBoost blend weight, and keep the family-specific operating point that performs well on both the primary and audit splits.
To develop the solution, I used uv, ruff, Docker, and Codex CLI. Most of the work was done on a MacBook Pro, without a GPU.
\nTo keep the results reproducible, I seeded the full pipeline and used a Kaggle official Docker image for local development. In the GitHub repository, the solution can be reproduced by building the pinned Docker image and running the run_docker.sh script, which executes the code inside the container and writes the final submission.csv.
I am an applied mathematician turned into software developer, currently based in Madrid. My interests are AI, machine learning, numerical simulation, and optimization.
\nIf you would like to reach out, you can find me on LinkedIn, or via Gmail as alvaroborrasf.
Thank you again and looking forward to seeing the approaches from other participants!
", "rawMarkdown": "I would like to thank the organizers for putting together the competition, and for addressing the early dataset issues so quickly.\n\nMost of the public leaderboard eventually converged around `0.9115` (public) / `0.9119` (private), so I was happy to finish a bit above that score.\n\nIt was a fun competition, and I learned a lot from it. Thank you!\n\n**TLDR**: [This notebook](https://www.kaggle.com/code/alvaroborras/anomaly-detection-cyber-der-final-submission) reproduces my final submission, and the full implementation is available in my [personal repository](https://github.com/alvaroborras/Anomaly-Detection-DER-Systems).\n\n# Overview of the solution\n\nThe dataset is highly structured, and in the original CSV files it is dominated by two exact identity tuples.\n\nMore concretely, I build a 5-field fingerprint from `common[0].Mn`, `common[0].Md`, `common[0].Opt`, `common[0].Vr`, and `common[0].SN`. The two dominant tuples correspond to the 10 kW and 100 kW simulator rows, which I refer to as `canon10` and `canon100`, and I use that fingerprint to split the data into `canon10`, `canon100`, and a small noncanonical bucket.\n\nThis distinction is very important for the rest of the pipeline. The two canonical families are modeled separately, while the noncanonical rows are handled outside the main learned-model path because they already form a very strong anomaly signal in training.\n\nFrom `665` selected source columns, I build semantic features that make the DER structure explicit: missingness and schema-integrity signals, measurement-vs-rating/setting consistency, control-compliance features, protection and curve features, and AC/DC consistency checks.\n\nFor each canonical family, the final predictor is a small ensemble of XGBoost on the semantic numeric table and CatBoost on a narrower raw + categorical table, with the blend and threshold tuned directly for F2.\n\n\n\n*High level view of the pipeline. Rows are first split by device fingerprint; the two canonical families go through family-specific feature engineering and models, while the small noncanonical bucket is handled separately.*\n\n## Observations about the dataset\n\nThe raw dataset has `723` features, but many are fully empty or not especially useful for the final model. In particular, `182` features are completely null in both train and test, while there are no fully empty rows. In practice, the useful signal is not in dropping rows, but in being selective about columns and preserving informative missingness.\n\nA second key observation is that the data is overwhelmingly concentrated around two exact fingerprints from the original CSV identity columns:\n\n- `('DERSec', 'DER Simulator', '10 kW DER', '1.2.3', 'SN-Three-Phase')` -> `canon10`\n- `('DERSec', 'DER Simulator 100 kW', '1.2.3.1', '1.0.0', '1100058974')` -> `canon100`\n\n| Bucket | Definition | Train rows | Test rows | How I use it |\n|---|---|---:|---:|---|\n| `canon10` | Exact match to the canonical 10 kW simulator fingerprint | 1,181,604 | 505,993 | Family-specific learned models + overrides |\n| `canon100` | Exact match to the canonical 100 kW simulator fingerprint | 1,168,218 | 501,106 | Family-specific learned models + overrides |\n| `other` / noncanonical | Anything else | 13,341 | 5,686 | Identity-based anomaly bucket |\n\nThis matters because all `13,341` noncanonical training rows are anomalous. So I did not treat `other` as a third modeling family. Instead, I used it as a strong identity-based anomaly signal.\n\n## Feature engineering\n\nThe main feature groups are:\n\n1. **Identity and missingness**: device fingerprint, missing identity fields, per-block missingness counts, and missingness patterns.\n2. **Schema integrity**: expected model IDs and lengths, used to detect malformed or structurally inconsistent rows.\n3. **Physical consistency**: measurements compared against ratings and limits, plus phase-sum and power-factor consistency checks.\n4. **Control compliance**: whether the reported active or reactive control state matches the observed output.\n5. **Protection and curves**: enter-service logic, trip blocks, Volt-Var / Volt-Watt / Watt-Var behavior, and frequency-droop features.\n6. **AC/DC consistency**: agreement between AC and DC measurements, per-port values, and sign patterns.\n7. **Residual and scenario features**: surrogate-model residuals and smoothed anomaly rates for recurring operating scenarios.\n\n## Hard overrides\n\nI also used a small set of high-confidence hard overrides. They are simply anomaly signatures that were clean enough in training to justify overriding the learned model.\n\nThe most important case is the noncanonical bucket itself. If the 5-field fingerprint does not exactly match one of the two canonical simulator identities, the row is assigned to `other`, which acts as an identity-based anomaly bucket rather than a learned family.\n\nInside the canonical families, the overrides are limited to a few high-precision patterns such as:\n\n- clear electrical envelope violations,\n- large mismatches between enabled controls and measured output,\n- rare metadata or AC/DC patterns,\n- enter-service contradictions,\n- producing power while already outside a must-trip region.\n\nAn important implementation detail is that candidate overrides are audited on the training data. Only the most precise ones remain true hard overrides; the rest can still help as features without forcing the final prediction.\n\n## Family-specific models and ensembling\n\nFor `canon10` and `canon100`, I train separate models. This worked better than a single global classifier because the two canonical devices operate in different regimes and have different local patterns.\n\nThe main model is **XGBoost** on the full semantic numeric feature set. The companion model is **CatBoost** on a narrower raw + categorical table containing selected raw numeric columns, raw identity fields, the full device fingerprint, and additional identity / missingness-engineered features.\n\nI also add family-specific regressors trained on normal rows to predict quantities such as active power, apparent power, reactive power, power factor, and current. Their residuals become new features for the final classifiers. For validation, I use both a primary 5-fold split based on `Id` and an audit split based on a hashed representation of the operating scenario.\n\n## Threshold optimization\n\nSince the competition metric is F2, I did not use the default `0.5` threshold. For each canonical family, I search directly over out-of-fold probabilities to find the threshold that maximizes F2. I do the same for the XGBoost / CatBoost blend weight, and keep the family-specific operating point that performs well on both the primary and audit splits.\n\n## Tools used\n\nTo develop the solution, I used [uv](https://docs.astral.sh/uv/), [ruff](https://github.com/astral-sh/ruff), [Docker](https://www.docker.com/), and [Codex CLI](https://developers.openai.com/codex/cli). Most of the work was done on a MacBook Pro, without a GPU.\n\n## Reproducibility\n\nTo keep the results reproducible, I seeded the full pipeline and used a [Kaggle official Docker image](https://console.cloud.google.com/artifacts/docker/kaggle-images/us/gcr.io/python/sha256:02c72a7c98e5e0895056901d9c715d181cd30eae392491235dfea93e6d0de3ed) for local development. In the GitHub repository, the solution can be reproduced by building the pinned Docker image and running the `run_docker.sh` script, which executes the code inside the container and writes the final `submission.csv`.\n\n## About me\n\nI am an applied mathematician turned into software developer, currently based in Madrid. My interests are AI, machine learning, numerical simulation, and optimization.\n\nIf you would like to reach out, you can find me on [LinkedIn](https://www.linkedin.com/in/alvaro-borras/), or via Gmail as `alvaroborrasf`.\n\nThank you again and looking forward to seeing the approaches from other participants!", "votes": null }, { "id": "3438095", "postDate": "04/08/2026 16:55:32", "content": "Great work!! Also congrats on the win
", "rawMarkdown": "Great work!! Also congrats on the win", "votes": null } ], "comments": [ { "id": 3438095, "author_name": "adwaittagalpallewar", "author_url": "", "post_date": "04/08/2026 16:55:32", "content": "Great work!! Also congrats on the win
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "3437808": "I would like to thank the organizers for putting together the competition, and for addressing the early dataset issues so quickly.\n\nMost of the public leaderboard eventually converged around `0.9115` (public) / `0.9119` (private), so I was happy to finish a bit above that score.\n\nIt was a fun competition, and I learned a lot from it. Thank you!\n\n**TLDR**: [This notebook](https://www.kaggle.com/code/alvaroborras/anomaly-detection-cyber-der-final-submission) reproduces my final submission, and the full implementation is available in my [personal repository](https://github.com/alvaroborras/Anomaly-Detection-DER-Systems).\n\n# Overview of the solution\n\nThe dataset is highly structured, and in the original CSV files it is dominated by two exact identity tuples.\n\nMore concretely, I build a 5-field fingerprint from `common[0].Mn`, `common[0].Md`, `common[0].Opt`, `common[0].Vr`, and `common[0].SN`. The two dominant tuples correspond to the 10 kW and 100 kW simulator rows, which I refer to as `canon10` and `canon100`, and I use that fingerprint to split the data into `canon10`, `canon100`, and a small noncanonical bucket.\n\nThis distinction is very important for the rest of the pipeline. The two canonical families are modeled separately, while the noncanonical rows are handled outside the main learned-model path because they already form a very strong anomaly signal in training.\n\nFrom `665` selected source columns, I build semantic features that make the DER structure explicit: missingness and schema-integrity signals, measurement-vs-rating/setting consistency, control-compliance features, protection and curve features, and AC/DC consistency checks.\n\nFor each canonical family, the final predictor is a small ensemble of XGBoost on the semantic numeric table and CatBoost on a narrower raw + categorical table, with the blend and threshold tuned directly for F2.\n\n\n\n*High level view of the pipeline. Rows are first split by device fingerprint; the two canonical families go through family-specific feature engineering and models, while the small noncanonical bucket is handled separately.*\n\n## Observations about the dataset\n\nThe raw dataset has `723` features, but many are fully empty or not especially useful for the final model. In particular, `182` features are completely null in both train and test, while there are no fully empty rows. In practice, the useful signal is not in dropping rows, but in being selective about columns and preserving informative missingness.\n\nA second key observation is that the data is overwhelmingly concentrated around two exact fingerprints from the original CSV identity columns:\n\n- `('DERSec', 'DER Simulator', '10 kW DER', '1.2.3', 'SN-Three-Phase')` -> `canon10`\n- `('DERSec', 'DER Simulator 100 kW', '1.2.3.1', '1.0.0', '1100058974')` -> `canon100`\n\n| Bucket | Definition | Train rows | Test rows | How I use it |\n|---|---|---:|---:|---|\n| `canon10` | Exact match to the canonical 10 kW simulator fingerprint | 1,181,604 | 505,993 | Family-specific learned models + overrides |\n| `canon100` | Exact match to the canonical 100 kW simulator fingerprint | 1,168,218 | 501,106 | Family-specific learned models + overrides |\n| `other` / noncanonical | Anything else | 13,341 | 5,686 | Identity-based anomaly bucket |\n\nThis matters because all `13,341` noncanonical training rows are anomalous. So I did not treat `other` as a third modeling family. Instead, I used it as a strong identity-based anomaly signal.\n\n## Feature engineering\n\nThe main feature groups are:\n\n1. **Identity and missingness**: device fingerprint, missing identity fields, per-block missingness counts, and missingness patterns.\n2. **Schema integrity**: expected model IDs and lengths, used to detect malformed or structurally inconsistent rows.\n3. **Physical consistency**: measurements compared against ratings and limits, plus phase-sum and power-factor consistency checks.\n4. **Control compliance**: whether the reported active or reactive control state matches the observed output.\n5. **Protection and curves**: enter-service logic, trip blocks, Volt-Var / Volt-Watt / Watt-Var behavior, and frequency-droop features.\n6. **AC/DC consistency**: agreement between AC and DC measurements, per-port values, and sign patterns.\n7. **Residual and scenario features**: surrogate-model residuals and smoothed anomaly rates for recurring operating scenarios.\n\n## Hard overrides\n\nI also used a small set of high-confidence hard overrides. They are simply anomaly signatures that were clean enough in training to justify overriding the learned model.\n\nThe most important case is the noncanonical bucket itself. If the 5-field fingerprint does not exactly match one of the two canonical simulator identities, the row is assigned to `other`, which acts as an identity-based anomaly bucket rather than a learned family.\n\nInside the canonical families, the overrides are limited to a few high-precision patterns such as:\n\n- clear electrical envelope violations,\n- large mismatches between enabled controls and measured output,\n- rare metadata or AC/DC patterns,\n- enter-service contradictions,\n- producing power while already outside a must-trip region.\n\nAn important implementation detail is that candidate overrides are audited on the training data. Only the most precise ones remain true hard overrides; the rest can still help as features without forcing the final prediction.\n\n## Family-specific models and ensembling\n\nFor `canon10` and `canon100`, I train separate models. This worked better than a single global classifier because the two canonical devices operate in different regimes and have different local patterns.\n\nThe main model is **XGBoost** on the full semantic numeric feature set. The companion model is **CatBoost** on a narrower raw + categorical table containing selected raw numeric columns, raw identity fields, the full device fingerprint, and additional identity / missingness-engineered features.\n\nI also add family-specific regressors trained on normal rows to predict quantities such as active power, apparent power, reactive power, power factor, and current. Their residuals become new features for the final classifiers. For validation, I use both a primary 5-fold split based on `Id` and an audit split based on a hashed representation of the operating scenario.\n\n## Threshold optimization\n\nSince the competition metric is F2, I did not use the default `0.5` threshold. For each canonical family, I search directly over out-of-fold probabilities to find the threshold that maximizes F2. I do the same for the XGBoost / CatBoost blend weight, and keep the family-specific operating point that performs well on both the primary and audit splits.\n\n## Tools used\n\nTo develop the solution, I used [uv](https://docs.astral.sh/uv/), [ruff](https://github.com/astral-sh/ruff), [Docker](https://www.docker.com/), and [Codex CLI](https://developers.openai.com/codex/cli). Most of the work was done on a MacBook Pro, without a GPU.\n\n## Reproducibility\n\nTo keep the results reproducible, I seeded the full pipeline and used a [Kaggle official Docker image](https://console.cloud.google.com/artifacts/docker/kaggle-images/us/gcr.io/python/sha256:02c72a7c98e5e0895056901d9c715d181cd30eae392491235dfea93e6d0de3ed) for local development. In the GitHub repository, the solution can be reproduced by building the pinned Docker image and running the `run_docker.sh` script, which executes the code inside the container and writes the final `submission.csv`.\n\n## About me\n\nI am an applied mathematician turned into software developer, currently based in Madrid. My interests are AI, machine learning, numerical simulation, and optimization.\n\nIf you would like to reach out, you can find me on [LinkedIn](https://www.linkedin.com/in/alvaro-borras/), or via Gmail as `alvaroborrasf`.\n\nThank you again and looking forward to seeing the approaches from other participants!", "3438095": "Great work!! Also congrats on the win" }, "source": "meta" }