Hi there,
\nI started working on the challenge starting from the single-cell count matrix, but there are still some differences between the DE data created from the adata_train.parquet file and the de_train.parquet file.
I tried to reproduce the DE train file on the SaturnCloud instance (to avoid version issues that would cause problems). One obvious thing is the mismatch of the number of genes, which were already noted by others (e.g. https://github.com/openproblems-bio/neurips-2023-scripts/issues/1).
\nI can live with subsetting to the number of genes in the de_train.parquet file, however, that does not explain the differences between the DE data created from adata_train.parquet and the de_train.parquet. I attached one of the direct comparison files created on SaturnCloud with the notebook compute_de.ipynb.
So examined the pseudobulk data in more detail. The most striking differences between the reference pseudobulk on the SaturnCloud (train_or_control_bulk_by_cell_type_adata.h5ad) and the pseudobulk created from adata_train.parquet are the compounds Sgc-cbp30, YK 4-279, where the metadata for the plate_id and the donor_id do not match, and the pseudobulk counts seem to be entirely different.
I hope that you understand my frustration with the provided data. To my mind, it is a strong disadvantage for anyone who is attempting to work on a single-cell count data approach for the challenge. It would be really great if the organizers would provide the updated version of the adata_train.parquet file in due time as promised already at the beginning of the month. In case they have done so, I would be grateful for a pointer to the corrected data files.
\nThanks.
Hi there,
\nI started working on the challenge starting from the single-cell count matrix, but there are still some differences between the DE data created from the adata_train.parquet file and the de_train.parquet file.
I tried to reproduce the DE train file on the SaturnCloud instance (to avoid version issues that would cause problems). One obvious thing is the mismatch of the number of genes, which were already noted by others (e.g. https://github.com/openproblems-bio/neurips-2023-scripts/issues/1).
\nI can live with subsetting to the number of genes in the de_train.parquet file, however, that does not explain the differences between the DE data created from adata_train.parquet and the de_train.parquet. I attached one of the direct comparison files created on SaturnCloud with the notebook compute_de.ipynb.
So examined the pseudobulk data in more detail. The most striking differences between the reference pseudobulk on the SaturnCloud (train_or_control_bulk_by_cell_type_adata.h5ad) and the pseudobulk created from adata_train.parquet are the compounds Sgc-cbp30, YK 4-279, where the metadata for the plate_id and the donor_id do not match, and the pseudobulk counts seem to be entirely different.
I hope that you understand my frustration with the provided data. To my mind, it is a strong disadvantage for anyone who is attempting to work on a single-cell count data approach for the challenge. It would be really great if the organizers would provide the updated version of the adata_train.parquet file in due time as promised already at the beginning of the month. In case they have done so, I would be grateful for a pointer to the corrected data files.
\nThanks.
Yes, I totally understand your frustration. When I found this, I gave up on the idea of using the single-cell data.
\nHere is anthor post and two of my notebooks (op2-02-pseudobulk-v3 and op2-04-merge-limma-results) with the result and analysis about this problem.
\nI think the organizers should fix this ASAP to avoid more people wasting their time and effort to \"debug\" this inconsistency.
\nTechnically, I guess the organizers forget to remove (or add) some filters at gene level when preprocessing the data, (even worse, the filters are inconsistent among samples, so they need more time to fix it, this is my conspiracy).
\nUsually, this kind of inconsistency can be ignored (only negligible effect about the biological findings), but for this competition, the limma's p-value calculation amplified this difference in the value, and the MRRMSE scoring function made this difference has huge impact on the score. That made my very frustrated.
Thank you for sharing your insights. It's pretty discouraging to use the single-cell data, then.
", "rawMarkdown": "Thank you for sharing your insights. It's pretty discouraging to use the single-cell data, then.", "votes": null } ], "comments": [ { "id": 2497932, "author_name": "awater1223", "author_url": "", "post_date": "10/25/2023 02:42:59", "content": "Yes, I totally understand your frustration. When I found this, I gave up on the idea of using the single-cell data.
\nHere is anthor post and two of my notebooks (op2-02-pseudobulk-v3 and op2-04-merge-limma-results) with the result and analysis about this problem.
\nI think the organizers should fix this ASAP to avoid more people wasting their time and effort to \"debug\" this inconsistency.
\nTechnically, I guess the organizers forget to remove (or add) some filters at gene level when preprocessing the data, (even worse, the filters are inconsistent among samples, so they need more time to fix it, this is my conspiracy).
\nUsually, this kind of inconsistency can be ignored (only negligible effect about the biological findings), but for this competition, the limma's p-value calculation amplified this difference in the value, and the MRRMSE scoring function made this difference has huge impact on the score. That made my very frustrated.
Thank you for sharing your insights. It's pretty discouraging to use the single-cell data, then.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2497620": "Hi there,\nI started working on the challenge starting from the single-cell count matrix, but there are still some differences between the DE data created from the `adata_train.parquet` file and the `de_train.parquet` file. \n\nI tried to reproduce the DE train file on the SaturnCloud instance (to avoid version issues that would cause problems). One obvious thing is the mismatch of the number of genes, which were already noted by others (e.g. https://github.com/openproblems-bio/neurips-2023-scripts/issues/1). \nI can live with subsetting to the number of genes in the `de_train.parquet` file, however, that does not explain the differences between the DE data created from `adata_train.parquet` and the `de_train.parquet`. I attached one of the direct comparison files created on SaturnCloud with the notebook `compute_de.ipynb`. \n\nSo examined the pseudobulk data in more detail. The most striking differences between the reference pseudobulk on the SaturnCloud (`train_or_control_bulk_by_cell_type_adata.h5ad`) and the pseudobulk created from `adata_train.parquet` are the compounds Sgc-cbp30, YK 4-279, where the metadata for the plate_id and the donor_id do not match, and the pseudobulk counts seem to be entirely different. \n\nI hope that you understand my frustration with the provided data. To my mind, it is a strong disadvantage for anyone who is attempting to work on a single-cell count data approach for the challenge. It would be really great if the organizers would provide the updated version of the `adata_train.parquet` file in due time as promised already at the beginning of the month. In case they have done so, I would be grateful for a pointer to the corrected data files. \nThanks.", "2497932": "Yes, I totally understand your frustration. When I found this, I gave up on the idea of using the single-cell data.\n\nHere is anthor [post](https://www.kaggle.com/competitions/open-problems-single-cell-perturbations/discussion/448033) and two of my notebooks ([op2-02-pseudobulk-v3](https://www.kaggle.com/code/awater1223/op2-02-pseudobulk-v3) and [op2-04-merge-limma-results](https://www.kaggle.com/code/awater1223/op2-04-merge-limma-results)) with the result and analysis about this problem.\n\nI think the organizers should fix this ASAP to avoid more people wasting their time and effort to \"debug\" this inconsistency.\n\n---\n\nTechnically, I guess the organizers forget to remove (or add) some filters at gene level when preprocessing the data, (even worse, the filters are inconsistent among samples, so they need more time to fix it, this is my conspiracy).\nUsually, this kind of inconsistency can be ignored (only negligible effect about the biological findings), but for this competition, the limma's p-value calculation amplified this difference in the value, and the MRRMSE scoring function made this difference has huge impact on the score. That made my very frustrated.", "2498914": "Thank you for sharing your insights. It's pretty discouraging to use the single-cell data, then." }, "source": "meta" }