{ "id": 440956, "title": "Batch effects between donors", "url": "/competitions/open-problems-single-cell-perturbations/discussion/440956", "author_name": "", "post_date": "2023-09-17T00:27:59.971718500Z", "votes": 5, "comment_count": 6, "views": 0, "content": "

Thanks for organizing this competition.

\n

I am trying to visualize the cells by using the adata_train.parquet. When I plotted the UMAP using the normalized count, the cell types were not clustered together,

\n

\"\"

\n

It seems that there are some batch effects between donors. So I used BBKNN to correct the batch effect, however, the results didn't look much better:
\n\"\"

\n

May I ask how you corrected the batch effect when clustering and annotating the cells?

\n

Also, would it be possible to share the codes for pre-processing?

", "messages": [ { "id": "2442362", "postDate": "09/17/2023 00:27:59", "content": "

Thanks for organizing this competition.

\n

I am trying to visualize the cells by using the adata_train.parquet. When I plotted the UMAP using the normalized count, the cell types were not clustered together,

\n

\"\"

\n

It seems that there are some batch effects between donors. So I used BBKNN to correct the batch effect, however, the results didn't look much better:
\n\"\"

\n

May I ask how you corrected the batch effect when clustering and annotating the cells?

\n

Also, would it be possible to share the codes for pre-processing?

", "rawMarkdown": "Thanks for organizing this competition.\n\nI am trying to visualize the cells by using the adata_train.parquet. When I plotted the UMAP using the normalized count, the cell types were not clustered together,\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F642483%2Fbfd6395de77237255c4bfb29b3946d00%2Fnormalized.png?generation=1694910012053915&alt=media)\n\nIt seems that there are some batch effects between donors. So I used BBKNN to correct the batch effect, however, the results didn't look much better:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F642483%2F37b5e9431417cd7a995e24e156af879d%2Fbbknn.png?generation=1694911139213681&alt=media)\n\nMay I ask how you corrected the batch effect when clustering and annotating the cells?\n\nAlso, would it be possible to share the codes for pre-processing?", "votes": null }, { "id": "2443674", "postDate": "09/17/2023 21:04:43", "content": "

it is a bit more complex than just cell type. Keep in mind that the same cell type has been treated with different compounds. So it is hard to predict the outcome, but there could (should?) be subclusters…
\nSo if you apply any batch correction you might want to take the drug treatment as confounder into account.

", "rawMarkdown": "it is a bit more complex than just cell type. Keep in mind that the same cell type has been treated with different compounds. So it is hard to predict the outcome, but there could (should?) be subclusters...\nSo if you apply any batch correction you might want to take the drug treatment as confounder into account.", "votes": null }, { "id": "2444955", "postDate": "09/18/2023 14:54:58", "content": "

We annotated cell types on each plate individually, so no donor variation affected the analysis.

\n

To get nice UMAPs, all I've had to do is run sc.pp.highly_variable_genes(adata, batch_key=\"donor\") and then make sure to go through adata.var to make sure that I keep genes with n_batches_variable is greater than 2.

", "rawMarkdown": "We annotated cell types on each plate individually, so no donor variation affected the analysis. \n\nTo get nice UMAPs, all I've had to do is run `sc.pp.highly_variable_genes(adata, batch_key=\"donor\")` and then make sure to go through `adata.var` to make sure that I keep genes with `n_batches_variable` is greater than `2`.", "votes": null }, { "id": "2445721", "postDate": "09/19/2023 03:42:35", "content": "

Thanks, this makes sense.

\n

One more question. Did you correct the perturbation effect by data integration when annotating the cells for each plate?

", "rawMarkdown": "Thanks, this makes sense.\n\nOne more question. Did you correct the perturbation effect by data integration when annotating the cells for each plate?", "votes": null }, { "id": "2449067", "postDate": "09/21/2023 01:57:00", "content": "

Hi, @zhijianli and @danielburkhardt, sorry to bother

\n

I have tried getting the UMAP on Kaggle notebook.
\nAm I pre-processing the data correctly? Why the UMAP seem to be \"not clearly\"?

\n

BTW, I used the raw counts instead of the normalized counts

", "rawMarkdown": "Hi, @zhijianli and @danielburkhardt, sorry to bother\n\nI have tried getting the UMAP on Kaggle [notebook](https://www.kaggle.com/code/awater1223/op2-01-singlecell-eda).\nAm I pre-processing the data correctly? Why the UMAP seem to be \"not clearly\"?\n\nBTW, I used the raw counts instead of the normalized counts", "votes": null }, { "id": "2449975", "postDate": "09/21/2023 14:53:16", "content": "

Hi! A couple points:

\n
    \n
  1. Use normalized counts
    2. \n
  2. After running highly_variable_genes look in adata.var for a col names n_batches. This is the number of batches (provided in batch_key) that the gene is variable in. By default, scanpy keeps all genes variable in at least 1 batch. You want to increase that to at least 2 or 3 batches. You need to manually overwrite the is_highly_variable column to apply this filter.
    3. \n
\n

Hope this helps!

\n

P.S. the column names here might not be exactly right, I'm writing this from memory

", "rawMarkdown": "Hi! A couple points:\n1. Use normalized counts\n2. After running `highly_variable_genes` look in `adata.var` for a col names `n_batches`. This is the number of batches (provided in `batch_key`) that the gene is variable in. By default, scanpy keeps all genes variable in at least 1 batch. You want to increase that to at least 2 or 3 batches. You need to manually overwrite the `is_highly_variable` column to apply this filter.\n\nHope this helps!\n\nP.S. the column names here might not be exactly right, I'm writing this from memory", "votes": null }, { "id": "2450562", "postDate": "09/22/2023 02:25:24", "content": "

Thanks for your reply!

\n
    \n
  1. Sorry, I didn't make it clear. I used the raw counts and ran sc.pp.normalize_total thensc.pp.log1p to make \"normalized counts\".
    2. \n
  2. That really helpful! The column name is highly_variable_nbatches. I will try it later.
    3. \n
", "rawMarkdown": "Thanks for your reply!\n\n1. Sorry, I didn't make it clear. I used the raw counts and ran `sc.pp.normalize_total` then`sc.pp.log1p` to make \"normalized counts\".\n2. That really helpful! The column name is `highly_variable_nbatches`. I will try it later.", "votes": null } ], "comments": [ { "id": 2443674, "author_name": "rahi37", "author_url": "", "post_date": "09/17/2023 21:04:43", "content": "

it is a bit more complex than just cell type. Keep in mind that the same cell type has been treated with different compounds. So it is hard to predict the outcome, but there could (should?) be subclusters…
\nSo if you apply any batch correction you might want to take the drug treatment as confounder into account.

", "votes": null, "replies": [] }, { "id": 2444955, "author_name": "danielburkhardt", "author_url": "", "post_date": "09/18/2023 14:54:58", "content": "

We annotated cell types on each plate individually, so no donor variation affected the analysis.

\n

To get nice UMAPs, all I've had to do is run sc.pp.highly_variable_genes(adata, batch_key=\"donor\") and then make sure to go through adata.var to make sure that I keep genes with n_batches_variable is greater than 2.

", "votes": null, "replies": [ { "id": 2445721, "author_name": "zhijianli", "author_url": "", "post_date": "09/19/2023 03:42:35", "content": "

Thanks, this makes sense.

\n

One more question. Did you correct the perturbation effect by data integration when annotating the cells for each plate?

", "votes": null, "replies": [] } ] }, { "id": 2449067, "author_name": "awater1223", "author_url": "", "post_date": "09/21/2023 01:57:00", "content": "

Hi, @zhijianli and @danielburkhardt, sorry to bother

\n

I have tried getting the UMAP on Kaggle notebook.
\nAm I pre-processing the data correctly? Why the UMAP seem to be \"not clearly\"?

\n

BTW, I used the raw counts instead of the normalized counts

", "votes": null, "replies": [ { "id": 2449975, "author_name": "danielburkhardt", "author_url": "", "post_date": "09/21/2023 14:53:16", "content": "

Hi! A couple points:

\n
    \n
  1. Use normalized counts
    2. \n
  2. After running highly_variable_genes look in adata.var for a col names n_batches. This is the number of batches (provided in batch_key) that the gene is variable in. By default, scanpy keeps all genes variable in at least 1 batch. You want to increase that to at least 2 or 3 batches. You need to manually overwrite the is_highly_variable column to apply this filter.
    3. \n
\n

Hope this helps!

\n

P.S. the column names here might not be exactly right, I'm writing this from memory

", "votes": null, "replies": [ { "id": 2450562, "author_name": "awater1223", "author_url": "", "post_date": "09/22/2023 02:25:24", "content": "

Thanks for your reply!

\n
    \n
  1. Sorry, I didn't make it clear. I used the raw counts and ran sc.pp.normalize_total thensc.pp.log1p to make \"normalized counts\".
    2. \n
  2. That really helpful! The column name is highly_variable_nbatches. I will try it later.
    3. \n
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2442362": "Thanks for organizing this competition.\n\nI am trying to visualize the cells by using the adata_train.parquet. When I plotted the UMAP using the normalized count, the cell types were not clustered together,\n\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F642483%2Fbfd6395de77237255c4bfb29b3946d00%2Fnormalized.png?generation=1694910012053915&alt=media)\n\nIt seems that there are some batch effects between donors. So I used BBKNN to correct the batch effect, however, the results didn't look much better:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F642483%2F37b5e9431417cd7a995e24e156af879d%2Fbbknn.png?generation=1694911139213681&alt=media)\n\nMay I ask how you corrected the batch effect when clustering and annotating the cells?\n\nAlso, would it be possible to share the codes for pre-processing?", "2443674": "it is a bit more complex than just cell type. Keep in mind that the same cell type has been treated with different compounds. So it is hard to predict the outcome, but there could (should?) be subclusters...\nSo if you apply any batch correction you might want to take the drug treatment as confounder into account.", "2444955": "We annotated cell types on each plate individually, so no donor variation affected the analysis. \n\nTo get nice UMAPs, all I've had to do is run `sc.pp.highly_variable_genes(adata, batch_key=\"donor\")` and then make sure to go through `adata.var` to make sure that I keep genes with `n_batches_variable` is greater than `2`.", "2445721": "Thanks, this makes sense.\n\nOne more question. Did you correct the perturbation effect by data integration when annotating the cells for each plate?", "2449067": "Hi, @zhijianli and @danielburkhardt, sorry to bother\n\nI have tried getting the UMAP on Kaggle [notebook](https://www.kaggle.com/code/awater1223/op2-01-singlecell-eda).\nAm I pre-processing the data correctly? Why the UMAP seem to be \"not clearly\"?\n\nBTW, I used the raw counts instead of the normalized counts", "2449975": "Hi! A couple points:\n1. Use normalized counts\n2. After running `highly_variable_genes` look in `adata.var` for a col names `n_batches`. This is the number of batches (provided in `batch_key`) that the gene is variable in. By default, scanpy keeps all genes variable in at least 1 batch. You want to increase that to at least 2 or 3 batches. You need to manually overwrite the `is_highly_variable` column to apply this filter.\n\nHope this helps!\n\nP.S. the column names here might not be exactly right, I'm writing this from memory", "2450562": "Thanks for your reply!\n\n1. Sorry, I didn't make it clear. I used the raw counts and ran `sc.pp.normalize_total` then`sc.pp.log1p` to make \"normalized counts\".\n2. That really helpful! The column name is `highly_variable_nbatches`. I will try it later." }, "source": "meta" }