![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7642360%2Fc8afef4f94c3d7e72d9a91a393a13d7a%2Fvaedgd_withC_nontransparent.png?generation=1702466649193114&alt=media\"){kind=spacer}
Disregarding good MRRMSE scores for this particular dataset, I focused on generative models for the single-cell transcript data. These generative models can be used in a larger variety of research tasks than the regression models that dominated the competition. Few teams attempted this route, so I believe one writeup is warranted.
\n\nTo handle the count data, I use a negative binomial (NB) posterior. Like the Poisson, this takes sequencing depths into account. The NB further accounts for overdispersion, and is empirically found to fit scRNA-seq data well — also outperforming zero-inflated versions (Grønbech, 2020). The NB density for observing $k$ gene counts is parameterized by mean $m$ and dispersion $r$:
\n$$ {NB}(k;m,r) = \\frac{\\Gamma(k+r)}{k!\\,\\Gamma(r)} \\left( \\frac{m}{r+m} \\right)^k
\n\\left( \\frac{r}{r+m} \\right)^r $$
The generative model outputs the mean scaled by the max count for the sample, while one $r$ is learned for each gene (sample independent).
\nSince the ATAC data does not have cell ids matching the scRNA-seq data, I did not integrate it into the model.
\n\nIn principle, generating cells in the 'requested' cell_type/compound pair and feeding them to Limma seems reasonable. However, running the provided compute_de.ipynb notebook gives a mismatch with the p-values of de_train (also after taking the excluded_ids file into account and fixing metadata slightly). This means that even if the generative model learns a perfect reconstruction of the training data, it will still have an MRRMSE offset of 0.0325. Further, limma is sensitive to the amount of cells in each covariate group, making the sampling of realistic cell counts with corresponding metadata non-trivial. These aspects favored directly regressing the DE results.
From simple statistical measures, NK cells were found to overlap best with the cell types in the private test set. This was further noted by others during the competition.
\nThere is a large proportion of cells from the positive control compounds compared to the rest of the data; I found it better to remove these cells from the dataset. It might further be useful to remove outlier compounds with few cell measurements.
\n\nI experimented with the Deep Generative Decoder (DGD) of Schuster & Krogh, 2023 and found it to achieve better scores (PB 1.39) than an NB-VAE variant (PB 1.81). Both types outperformed a traditional VAE on normalized data subsequently rescaled back. In each case, I maximize the ELBO using Pytorch with, e.g., NB negative log likelihood as reconstruction error and with early stopping to prevent overfitting. Annealing the KL term from 0 to 1 over 50 warmup epochs avoided some cases with exploding loss. The encoder/decoder architectures are fully connected networks with ReLU activations and three layers transformings the dimensions to [5; 256; 1024; 18211]. Models are visualized on the diagram below.
\n
When predicting compounds as a task, the latent space separates into meaningful clusters, but the model does not generalize well for compounds in the test set (perhaps due to the large imbalance of cell type classes). For the below image, a latent space of 2 dimensions was used with the DGD model — the clusters highly correspond to compounds, cell types and donors.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7642360%2F5e40babd13a2a0ee7033e2bb453b7362%2Fclusters.png?generation=1702466667622762&alt=media\")
Conditioning the generative model on the covariates necessary for the limma analysis (cell type; plate name; well) gave best results on the PB. These conditional variables correspond to the c in the earlier model overview. In practice, the models are thus a conditional-DGD and a conditional-NB-VAE, where the one-hot-encoded covariates are appended to the data before encoder and decoder forward passes. After conditioning, the cells do not form distinct clusters based on the metadata — a natural effect of the conditional variables (Kristiadl, 2016). Instead, they appear to be completely mixed together in a N(0,1) distribution (see github notebook). To generate a realistic number of cells and library sizes, two random forests were trained using the covariates as one-hot features; this step is important to get better p-values from limma.
Random forest regression directly on the pseudo-bulk counts yielded a better score (PB 1.01), while bleak compared to just the zero-matrix (PB 0.9) or an out-of-the-box XGBoost on the p-values (PB 0.62) with one-hot-encoded sm_name/cell_type features. Interestingly, when combining the regression XGB submission with the DGD submission, performance improved slightly (0.002 improvement). p-values from the generative AI -> limma pipeline had much larger variance than XGB p-values (see figure below), which could be reduced by averaging over multiple runs, but time did not allow for this. In general, these results provide a large degree of skepticism towards either 1) the generalization ability of generative models for this data, or 2) the metric used.
\n![](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7642360%2F3081e6596cde902115fea777e4d397d2%2Fhistogram.png?generation=1702466720038500&alt=media\")
\nThe performance of models already in the perturbation literature such as scGen (Lotfollahi, 2019), scTenifoldKnk (Osorio, 2022), and CellOT (Bunne, 2023) would require more engineering to generate cells feasible for good limma p-values -- I did not have luck in initial experiments, which is why I sticked to the more direct approach of cell generation conditioned on the covariates.
\n\nA validation set is always needed during the stochastic training of the model — we cannot be sure that by running it again with a full dataset that the training will go in the same way. Here, I used a small subset of randomly sampled points (3%) but manually including cells relevant to the test set in the training set. These relevant cells have either cell type or compound occurring in the test set. Randomly sampled cells remove some complexity regarding choice of validation scheme, and was feasible since the location of reconstruction minima was generally the same across cell types. I used the smallest set size where the validation loss was still stable during training. During development and to experiment with hyperparameters, the validation set was instead made by excluding some compounds with cell type NK from training, which were found to best overlap with the test cells.
\n \nInstead of counts or p-values, a potential ideal task for estimating perturbations could be to compute the likelihood of private test counts under the density of parameterized distributions per gene per cell. The task would then be to give parameters which maximize the likelihood of the held-out data. In this setup, the metric would be the log-likelihood summed over all genes over all cells. Further, instead of removing the variability from technical sequencing effects, these could be used as just additional features. This gives some more leg room for model choice as it does not force teams to try predicting the distribution of the metadata in order to get good p-values from limma.
\n\nThe code is documented on the Github page.
\n\nCode is available on GitHub: https://github.com/yhsure/kaggle-perturbations.
\nInstructions for running the project are available in the README.md file.
Grønbech, Christopher Heje, et al. \"scVAE: variational auto-encoders for single-cell gene expression data.\" Bioinformatics 36.16 (2020): 4415-4422.
\nSchuster, Viktoria, and Anders Krogh. \"The Deep Generative Decoder: MAP estimation of representations improves modeling of single-cell RNA data.\" arXiv preprint arXiv:2110.06672 (2021).
\nKristiadi, Agustinus. \"Conditional Variational Autoencoder: Intuition and Implementation.\" (2016).
\nLotfollahi, Mohammad, F. Alexander Wolf, and Fabian J. Theis. \"scGen predicts single-cell perturbation responses.\" Nature methods 16.8 (2019): 715-721.
\nOsorio, Daniel, et al. \"scTenifoldKnk: An efficient virtual knockout tool for gene function predictions via single-cell gene regulatory network perturbation.\" Patterns 3.3 (2022).
\nBunne, Charlotte, et al. \"Learning single-cell perturbation responses using neural optimal transport.\" Nature methods (2023): 1-10.
", "messages": [ { "id": 2560222, "postDate": "2023-12-13T11:32:26.583Z", "content": "Disregarding good MRRMSE scores for this particular dataset, I focused on generative models for the single-cell transcript data. These generative models can be used in a larger variety of research tasks than the regression models that dominated the competition. Few teams attempted this route, so I believe one writeup is warranted.
\n\nTo handle the count data, I use a negative binomial (NB) posterior. Like the Poisson, this takes sequencing depths into account. The NB further accounts for overdispersion, and is empirically found to fit scRNA-seq data well — also outperforming zero-inflated versions (Grønbech, 2020). The NB density for observing $k$ gene counts is parameterized by mean $m$ and dispersion $r$:
\n$$ {NB}(k;m,r) = \\frac{\\Gamma(k+r)}{k!\\,\\Gamma(r)} \\left( \\frac{m}{r+m} \\right)^k
\n\\left( \\frac{r}{r+m} \\right)^r $$
The generative model outputs the mean scaled by the max count for the sample, while one $r$ is learned for each gene (sample independent).
\nSince the ATAC data does not have cell ids matching the scRNA-seq data, I did not integrate it into the model.
\n\nIn principle, generating cells in the 'requested' cell_type/compound pair and feeding them to Limma seems reasonable. However, running the provided compute_de.ipynb notebook gives a mismatch with the p-values of de_train (also after taking the excluded_ids file into account and fixing metadata slightly). This means that even if the generative model learns a perfect reconstruction of the training data, it will still have an MRRMSE offset of 0.0325. Further, limma is sensitive to the amount of cells in each covariate group, making the sampling of realistic cell counts with corresponding metadata non-trivial. These aspects favored directly regressing the DE results.
From simple statistical measures, NK cells were found to overlap best with the cell types in the private test set. This was further noted by others during the competition.
\nThere is a large proportion of cells from the positive control compounds compared to the rest of the data; I found it better to remove these cells from the dataset. It might further be useful to remove outlier compounds with few cell measurements.
\n\nI experimented with the Deep Generative Decoder (DGD) of Schuster & Krogh, 2023 and found it to achieve better scores (PB 1.39) than an NB-VAE variant (PB 1.81). Both types outperformed a traditional VAE on normalized data subsequently rescaled back. In each case, I maximize the ELBO using Pytorch with, e.g., NB negative log likelihood as reconstruction error and with early stopping to prevent overfitting. Annealing the KL term from 0 to 1 over 50 warmup epochs avoided some cases with exploding loss. The encoder/decoder architectures are fully connected networks with ReLU activations and three layers transformings the dimensions to [5; 256; 1024; 18211]. Models are visualized on the diagram below.
\n
When predicting compounds as a task, the latent space separates into meaningful clusters, but the model does not generalize well for compounds in the test set (perhaps due to the large imbalance of cell type classes). For the below image, a latent space of 2 dimensions was used with the DGD model — the clusters highly correspond to compounds, cell types and donors.
\n
Conditioning the generative model on the covariates necessary for the limma analysis (cell type; plate name; well) gave best results on the PB. These conditional variables correspond to the c in the earlier model overview. In practice, the models are thus a conditional-DGD and a conditional-NB-VAE, where the one-hot-encoded covariates are appended to the data before encoder and decoder forward passes. After conditioning, the cells do not form distinct clusters based on the metadata — a natural effect of the conditional variables (Kristiadl, 2016). Instead, they appear to be completely mixed together in a N(0,1) distribution (see github notebook). To generate a realistic number of cells and library sizes, two random forests were trained using the covariates as one-hot features; this step is important to get better p-values from limma.
Random forest regression directly on the pseudo-bulk counts yielded a better score (PB 1.01), while bleak compared to just the zero-matrix (PB 0.9) or an out-of-the-box XGBoost on the p-values (PB 0.62) with one-hot-encoded sm_name/cell_type features. Interestingly, when combining the regression XGB submission with the DGD submission, performance improved slightly (0.002 improvement). p-values from the generative AI -> limma pipeline had much larger variance than XGB p-values (see figure below), which could be reduced by averaging over multiple runs, but time did not allow for this. In general, these results provide a large degree of skepticism towards either 1) the generalization ability of generative models for this data, or 2) the metric used.
\n\nThe performance of models already in the perturbation literature such as scGen (Lotfollahi, 2019), scTenifoldKnk (Osorio, 2022), and CellOT (Bunne, 2023) would require more engineering to generate cells feasible for good limma p-values -- I did not have luck in initial experiments, which is why I sticked to the more direct approach of cell generation conditioned on the covariates.
\n\nA validation set is always needed during the stochastic training of the model — we cannot be sure that by running it again with a full dataset that the training will go in the same way. Here, I used a small subset of randomly sampled points (3%) but manually including cells relevant to the test set in the training set. These relevant cells have either cell type or compound occurring in the test set. Randomly sampled cells remove some complexity regarding choice of validation scheme, and was feasible since the location of reconstruction minima was generally the same across cell types. I used the smallest set size where the validation loss was still stable during training. During development and to experiment with hyperparameters, the validation set was instead made by excluding some compounds with cell type NK from training, which were found to best overlap with the test cells.
\n \nInstead of counts or p-values, a potential ideal task for estimating perturbations could be to compute the likelihood of private test counts under the density of parameterized distributions per gene per cell. The task would then be to give parameters which maximize the likelihood of the held-out data. In this setup, the metric would be the log-likelihood summed over all genes over all cells. Further, instead of removing the variability from technical sequencing effects, these could be used as just additional features. This gives some more leg room for model choice as it does not force teams to try predicting the distribution of the metadata in order to get good p-values from limma.
\n\nThe code is documented on the Github page.
\n\nCode is available on GitHub: https://github.com/yhsure/kaggle-perturbations.
\nInstructions for running the project are available in the README.md file.
Grønbech, Christopher Heje, et al. \"scVAE: variational auto-encoders for single-cell gene expression data.\" Bioinformatics 36.16 (2020): 4415-4422.
\nSchuster, Viktoria, and Anders Krogh. \"The Deep Generative Decoder: MAP estimation of representations improves modeling of single-cell RNA data.\" arXiv preprint arXiv:2110.06672 (2021).
\nKristiadi, Agustinus. \"Conditional Variational Autoencoder: Intuition and Implementation.\" (2016).
\nLotfollahi, Mohammad, F. Alexander Wolf, and Fabian J. Theis. \"scGen predicts single-cell perturbation responses.\" Nature methods 16.8 (2019): 715-721.
\nOsorio, Daniel, et al. \"scTenifoldKnk: An efficient virtual knockout tool for gene function predictions via single-cell gene regulatory network perturbation.\" Patterns 3.3 (2022).
\nBunne, Charlotte, et al. \"Learning single-cell perturbation responses using neural optimal transport.\" Nature methods (2023): 1-10.
", "rawMarkdown": "# Learnings from generative models\nDisregarding good MRRMSE scores for this particular dataset, I focused on generative models for the single-cell transcript data. These generative models can be used in a larger variety of research tasks than the regression models that dominated the competition. Few teams attempted this route, so I believe one writeup is warranted.\n\n\n \n## 1. Integration of biological knowledge\nTo handle the count data, I use a negative binomial (NB) posterior. Like the Poisson, this takes sequencing depths into account. The NB further accounts for overdispersion, and is empirically found to fit scRNA-seq data well — also outperforming zero-inflated versions ([Grønbech, 2020][scVAE]). The NB density for observing $k$ gene counts is parameterized by mean $m$ and dispersion $r$:\n\n$$ {NB}(k;m,r) = \\frac{\\Gamma(k+r)}{k!\\,\\Gamma(r)} \\left( \\frac{m}{r+m} \\right)^k\n\\left( \\frac{r}{r+m} \\right)^r $$\n\nThe generative model outputs the mean scaled by the max count for the sample, while one $r$ is learned for each gene (sample independent). \n\nSince the ATAC data does not have cell ids matching the scRNA-seq data, I did not integrate it into the model. \n\n\n\n\n\n\n## 2. Exploration of the problem\nIn principle, generating cells in the 'requested' cell_type/compound pair and feeding them to Limma seems reasonable. However, running the provided `compute_de.ipynb` notebook gives a mismatch with the p-values of `de_train` (also after taking the `excluded_ids` file into account and fixing metadata slightly). This means that even if the generative model learns a perfect reconstruction of the training data, it will still have an MRRMSE offset of 0.0325. Further, limma is sensitive to the amount of cells in each covariate group, making the sampling of realistic cell counts with corresponding metadata non-trivial. These aspects favored directly regressing the DE results.\n\nFrom simple statistical measures, NK cells were found to overlap best with the cell types in the private test set. This was further noted by [others during the competition][nk_cells]. \n\nThere is a large proportion of cells from the positive control compounds compared to the rest of the data; I found it better to remove these cells from the dataset. It might further be useful to remove outlier compounds with few cell measurements. \n\n\n\n\n\n\n## 3. Model design\nI experimented with the Deep Generative Decoder (DGD) of [Schuster & Krogh, 2023][DGD] and found it to achieve better scores (PB 1.39) than an NB-VAE variant (PB 1.81). Both types outperformed a traditional VAE on normalized data subsequently rescaled back. In each case, I maximize the ELBO using Pytorch with, e.g., NB negative log likelihood as reconstruction error and with early stopping to prevent overfitting. Annealing the KL term from 0 to 1 over 50 warmup epochs avoided some cases with exploding loss. The encoder/decoder architectures are fully connected networks with ReLU activations and three layers transformings the dimensions to [5; 256; 1024; 18211]. Models are visualized on the diagram below. \n\n\n\nWhen predicting compounds as a task, the latent space separates into meaningful clusters, but the model does not generalize well for compounds in the test set (perhaps due to the large imbalance of cell type classes). For the below image, a latent space of 2 dimensions was used with the DGD model — the clusters highly correspond to compounds, cell types and donors. \n\n\n\nConditioning the generative model on the covariates necessary for the limma analysis (cell type; plate name; well) gave best results on the PB. These conditional variables correspond to the `c` in the earlier model overview. In practice, the models are thus a conditional-DGD and a conditional-NB-VAE, where the one-hot-encoded covariates are appended to the data before encoder and decoder forward passes. After conditioning, the cells do not form distinct clusters based on the metadata — a natural effect of the conditional variables ([Kristiadl, 2016][cvae]). Instead, they appear to be completely mixed together in a N(0,1) distribution (see github notebook). To generate a realistic number of cells and library sizes, two random forests were trained using the covariates as one-hot features; this step is important to get better p-values from limma.\n\nRandom forest regression directly on the pseudo-bulk counts yielded a better score (PB 1.01), while bleak compared to just the zero-matrix (PB 0.9) or an out-of-the-box XGBoost on the p-values (PB 0.62) with one-hot-encoded sm_name/cell_type features. Interestingly, when combining the regression XGB submission with the DGD submission, performance improved slightly (0.002 improvement). p-values from the generative AI -> limma pipeline had much larger variance than XGB p-values (see figure below), which could be reduced by averaging over multiple runs, but time did not allow for this. In general, these results provide a large degree of skepticism towards either 1) the generalization ability of generative models for this data, or 2) the metric used. \n\n\n\n\n