As we have raw count data now, could we reproduce provided competition normalized data(library-size normalized and log1p transformed, dsb normalized ) from raw count data ?
\nI tried to use scanpy to transform raw count to library-size normalized and log1p transformed data,but very different from provided data.And I can't find easy way to transform to dsb normalized,any suggestions?
As we have raw count data now, could we reproduce provided competition normalized data(library-size normalized and log1p transformed, dsb normalized ) from raw count data ?
\nI tried to use scanpy to transform raw count to library-size normalized and log1p transformed data,but very different from provided data.And I can't find easy way to transform to dsb normalized,any suggestions?
for RNA-seq part of data
\nit should be log(1+ v/sum(v)*1e6), modula details like adding 35 new genes.
\nAt least normalized data (initial input) satisfy:
\nsum( exp(v)-1 ) = 1e6. See checks here: https://www.kaggle.com/competitions/open-problems-multimodal/discussion/349132
For ATAC-seq, and CD-Proteins - I do not know.
\nPS
\nIt seems NUMBER of raw counts for CITE seq is differnt from initial input - what can it mean ? (just different number of samples by about 460)
", "rawMarkdown": "for RNA-seq part of data \nit should be log(1+ v/sum(v)*1e6), modula details like adding 35 new genes.\nAt least normalized data (initial input) satisfy:\nsum( exp(v)-1 ) = 1e6. See checks here: https://www.kaggle.com/competitions/open-problems-multimodal/discussion/349132\n\nFor ATAC-seq, and CD-Proteins - I do not know. \n\nPS\n\nIt seems NUMBER of raw counts for CITE seq is differnt from initial input - what can it mean ? (just different number of samples by about 460)\n\n ", "votes": 2, "replies": [ { "id": 1991434, "postDate": "2022-10-17T06:45:31.660Z", "content": "Thanks your insight,it's useful.I hope organizer can tell us the method of transformation in case of some teams know and others don't know.
", "rawMarkdown": "Thanks your insight,it's useful.I hope organizer can tell us the method of transformation in case of some teams know and others don't know.", "votes": 1 }, { "id": 1991461, "postDate": "2022-10-17T07:04:26.437Z", "content": "@danielburkhardt
\nCan you please share preprocessing algo and code, please ?
PS
\nAlso there is problem with number of test samples for CITEseq. It is different from initial one. How to match?
Hi! A few items here:
\nDiscrepancies in cell counts or total cells between raw and train / test data
\nSome of the filtering steps we use are stochastic, and unfortunately we did not save the raw output for the competition data before preparing the raw counts data. As a result, there are differences in terms of the cells present or the total counts. They should be minor.
Processing
\nWe will release our cell filtering, preprocessing, normalization, and transformation notebooks after competition close, but the main code you want to go from raw to normalized counts is:
ATAC
\nfrom muon import atac as ac\n\nac.pp.tfidf(atac, scale_factor=1e4)\nac.tl.lsi(atac)\nRNA
\nimport scanpy as sc\nsc.pp.normalize_per_cell(rna, counts_per_cell_after = 1e6)\nsc.pp.log1p(rna\nPROT
\nfrom muon import prot as pt\nDSB_LOWER=1.5 #These are set manually for each sample\nDSB_UPPER=2.8\n\nmdata['prot'].layers['counts'] = mdata['prot'].X # keep count data\npt.pp.dsb(mdata, \n data_raw = mdata_raw, \n empty_counts_range=(DSB_LOWER, DSB_UPPER), \n isotype_controls=isotypes, \n random_state=1)\n\nprot = mdata['prot']\n@danielburkhardt thanks for your information, I am not familiar with muon, could you share more details about PROT, how to define mdata,mdata_raw,isotypes?
", "rawMarkdown": "@danielburkhardt thanks for your information, I am not familiar with muon, could you share more details about PROT, how to define mdata,mdata_raw,isotypes?", "votes": 1 }, { "id": 1992338, "postDate": "2022-10-17T16:38:36.263Z", "content": "@senkin13 happy to help! Can you please help me understand more how you want to use this?
\nThe DSB normalization uses both the raw filtered (released) and raw unfiltered (not released) data. Because the unfiltered data is very large, you won't be able to recreate mdata_raw with the files we released. I'm worried we won't have time to prepare that data for release before the competition closes. The challenge is there are many different stages of data processing, and as much as I wish it were easy, it takes some time to prep the intermediate data for public release.
However, if there's another way we can help you here, I would like to do that!
", "rawMarkdown": "@senkin13 happy to help! Can you please help me understand more how you want to use this? \n\nThe DSB normalization uses both the raw filtered (released) and raw unfiltered (not released) data. Because the unfiltered data is very large, you won't be able to recreate `mdata_raw` with the files we released. I'm worried we won't have time to prepare that data for release before the competition closes. The challenge is there are many different stages of data processing, and as much as I wish it were easy, it takes some time to prep the intermediate data for public release.\n\nHowever, if there's another way we can help you here, I would like to do that!", "votes": 1 }, { "id": 1992375, "postDate": "2022-10-17T16:56:27.060Z", "content": "Totally understood, I think you have provided enough information for me, thanks!
", "rawMarkdown": "Totally understood, I think you have provided enough information for me, thanks!", "votes": 1 } ] } ], "comments": [ { "id": 1990317, "author_name": "Alexander Chervov", "author_url": "", "post_date": "2022-10-16T13:07:57.590000", "content": "for RNA-seq part of data
\nit should be log(1+ v/sum(v)*1e6), modula details like adding 35 new genes.
\nAt least normalized data (initial input) satisfy:
\nsum( exp(v)-1 ) = 1e6. See checks here: https://www.kaggle.com/competitions/open-problems-multimodal/discussion/349132
For ATAC-seq, and CD-Proteins - I do not know.
\nPS
\nIt seems NUMBER of raw counts for CITE seq is differnt from initial input - what can it mean ? (just different number of samples by about 460)
", "votes": 2, "replies": [ { "id": 1991434, "author_name": "senkin13", "author_url": "", "post_date": "2022-10-17T06:45:31.660000", "content": "Thanks your insight,it's useful.I hope organizer can tell us the method of transformation in case of some teams know and others don't know.
", "votes": 1, "replies": [] }, { "id": 1991461, "author_name": "Alexander Chervov", "author_url": "", "post_date": "2022-10-17T07:04:26.437000", "content": "@danielburkhardt
\nCan you please share preprocessing algo and code, please ?
PS
\nAlso there is problem with number of test samples for CITEseq. It is different from initial one. How to match?
Hi! A few items here:
\nDiscrepancies in cell counts or total cells between raw and train / test data
\nSome of the filtering steps we use are stochastic, and unfortunately we did not save the raw output for the competition data before preparing the raw counts data. As a result, there are differences in terms of the cells present or the total counts. They should be minor.
Processing
\nWe will release our cell filtering, preprocessing, normalization, and transformation notebooks after competition close, but the main code you want to go from raw to normalized counts is:
ATAC
\nfrom muon import atac as ac\n\nac.pp.tfidf(atac, scale_factor=1e4)\nac.tl.lsi(atac)\nRNA
\nimport scanpy as sc\nsc.pp.normalize_per_cell(rna, counts_per_cell_after = 1e6)\nsc.pp.log1p(rna\nPROT
\nfrom muon import prot as pt\nDSB_LOWER=1.5 #These are set manually for each sample\nDSB_UPPER=2.8\n\nmdata['prot'].layers['counts'] = mdata['prot'].X # keep count data\npt.pp.dsb(mdata, \n data_raw = mdata_raw, \n empty_counts_range=(DSB_LOWER, DSB_UPPER), \n isotype_controls=isotypes, \n random_state=1)\n\nprot = mdata['prot']\n@danielburkhardt thanks for your information, I am not familiar with muon, could you share more details about PROT, how to define mdata,mdata_raw,isotypes?
", "votes": 1, "replies": [] }, { "id": 1992338, "author_name": "Daniel Burkhardt", "author_url": "", "post_date": "2022-10-17T16:38:36.263000", "content": "@senkin13 happy to help! Can you please help me understand more how you want to use this?
\nThe DSB normalization uses both the raw filtered (released) and raw unfiltered (not released) data. Because the unfiltered data is very large, you won't be able to recreate mdata_raw with the files we released. I'm worried we won't have time to prepare that data for release before the competition closes. The challenge is there are many different stages of data processing, and as much as I wish it were easy, it takes some time to prep the intermediate data for public release.
However, if there's another way we can help you here, I would like to do that!
", "votes": 1, "replies": [] }, { "id": 1992375, "author_name": "senkin13", "author_url": "", "post_date": "2022-10-17T16:56:27.060000", "content": "Totally understood, I think you have provided enough information for me, thanks!
", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "1990310": "As we have raw count data now, could we reproduce provided competition normalized data(library-size normalized and log1p transformed, dsb normalized ) from raw count data ?\nI tried to use scanpy to transform raw count to library-size normalized and log1p transformed data,but very different from provided data.And I can't find easy way to transform to dsb normalized,any suggestions?", "1990317": "for RNA-seq part of data \nit should be log(1+ v/sum(v)*1e6), modula details like adding 35 new genes.\nAt least normalized data (initial input) satisfy:\nsum( exp(v)-1 ) = 1e6. See checks here: https://www.kaggle.com/competitions/open-problems-multimodal/discussion/349132\n\nFor ATAC-seq, and CD-Proteins - I do not know. \n\nPS\n\nIt seems NUMBER of raw counts for CITE seq is differnt from initial input - what can it mean ? (just different number of samples by about 460)\n\n " } }