{ "id": 443748, "title": "Having trouble locating LINCS data", "url": "/competitions/open-problems-single-cell-perturbations/discussion/443748", "author_name": "", "post_date": "2023-09-28T15:22:33.240050100Z", "votes": 4, "comment_count": 2, "views": 0, "content": "

Hello,
\nIn the main discussion post the organizers stated they uploaded LINCS data to s3://saturn-kaggle-datasets/open-problems-single-cell-perturbations-optional/
\nI'd like to browse the files in Chrome, so I navigated to https://s3.amazonaws.com/saturn-kaggle-datasets/open-problems-single-cell-perturbations-optional and was redirected to https://saturn-kaggle-datasets.s3.amazonaws.com/

\n

I don't seen the open-problems-single-cell-perturbations-optional folder or any related data. I think I'm accessing this wrong, but I'm not sure how to proceed. Using boto3 in Python I had no success either.

", "messages": [ { "id": "2460110", "postDate": "09/28/2023 15:22:33", "content": "

Hello,
\nIn the main discussion post the organizers stated they uploaded LINCS data to s3://saturn-kaggle-datasets/open-problems-single-cell-perturbations-optional/
\nI'd like to browse the files in Chrome, so I navigated to https://s3.amazonaws.com/saturn-kaggle-datasets/open-problems-single-cell-perturbations-optional and was redirected to https://saturn-kaggle-datasets.s3.amazonaws.com/

\n

I don't seen the open-problems-single-cell-perturbations-optional folder or any related data. I think I'm accessing this wrong, but I'm not sure how to proceed. Using boto3 in Python I had no success either.

", "rawMarkdown": "Hello,\n[In the main discussion post](https://www.kaggle.com/competitions/open-problems-single-cell-perturbations/discussion/440680) the organizers stated they uploaded LINCS data to `s3://saturn-kaggle-datasets/open-problems-single-cell-perturbations-optional/`\nI'd like to browse the files in Chrome, so I navigated to [https://s3.amazonaws.com/saturn-kaggle-datasets/open-problems-single-cell-perturbations-optional](https://s3.amazonaws.com/saturn-kaggle-datasets/open-problems-single-cell-perturbations-optional) and was redirected to [https://saturn-kaggle-datasets.s3.amazonaws.com/](https://saturn-kaggle-datasets.s3.amazonaws.com/)\n\nI don't seen the `open-problems-single-cell-perturbations-optional` folder or any related data. I think I'm accessing this wrong, but I'm not sure how to proceed. Using boto3 in Python I had no success either.", "votes": null }, { "id": "2460122", "postDate": "09/28/2023 15:29:37", "content": "

Hi! I don't think you can't easily view AWS buckets in Chrome, but you can browse via CLI. e.g. https://docs.aws.amazon.com/cli/latest/reference/s3/ls.html

\n

You can also download the data from the LINCS Data Portal: https://lincsportal.ccs.miami.edu/datasets/view/LDS-1611

\n

For more information on LINCS data format, I recommend: https://www.cell.com/cell/fulltext/S0092-8674(17)31309-0

", "rawMarkdown": "Hi! I don't think you can't easily view AWS buckets in Chrome, but you can browse via CLI. e.g. https://docs.aws.amazon.com/cli/latest/reference/s3/ls.html\n\nYou can also download the data from the LINCS Data Portal: https://lincsportal.ccs.miami.edu/datasets/view/LDS-1611\n\nFor more information on LINCS data format, I recommend: https://www.cell.com/cell/fulltext/S0092-8674(17)31309-0", "votes": null }, { "id": "2460162", "postDate": "09/28/2023 15:52:10", "content": "

Thank you!
\nAccording the NIH LINCS page
\nThe five levels of data are

\n
 : Raw unprocessed flow cytometry data  Luminex (LXB)\n : Gene expression  per  genes  deconvolution (GEX)\n : Quantile-normalized gene expression profiles  landmark genes  imputed transcripts (Q2NORM  INF)\n : Gene signatures computed  z-scores relative  the plate population  control (ZSPCINF)  relative  the plate vehicle control (ZSVCINF)\n : Differential gene expression signatures\n
\n

In my understanding for this competition, the pseudobulked data produced something most similar to the Level 4 values relative to the plate population. The LINCS data and the competition data diverge when the trained LIMMA model produces the p-values.

\n

In your opinion, are the Level 4 z-scores a good proxy for the predictions values in the training/test set? If we want to leverage the LINCS data, should we attempt to train our own LIMMA models? Or is the Level 5 data, with a smaller memory footprint, also appropriate? I understand if some of these questions would reveal too much, but any advice would be helpful.

", "rawMarkdown": "Thank you!\nAccording the [NIH LINCS page](https://lincsproject.org/LINCS/tools/workflows/find-the-best-place-to-obtain-the-lincs-l1000-data)\nThe five levels of data are\n```\nLevel 1: Raw unprocessed flow cytometry data from Luminex (LXB)\nLevel 2: Gene expression values per 1000 genes after deconvolution (GEX)\nLevel 3: Quantile-normalized gene expression profiles of landmark genes and imputed transcripts (Q2NORM or INF)\nLevel 4: Gene signatures computed using z-scores relative to the plate population as control (ZSPCINF) or relative to the plate vehicle control (ZSVCINF)\nLevel 5: Differential gene expression signatures\n```\nIn my understanding for this competition, the pseudobulked data produced something most similar to the Level 4 values relative to the plate population. The LINCS data and the competition data diverge when the trained LIMMA model produces the p-values.\n\nIn your opinion, are the Level 4 z-scores a good proxy for the predictions values in the training/test set? If we want to leverage the LINCS data, should we attempt to train our own LIMMA models? Or is the Level 5 data, with a smaller memory footprint, also appropriate? I understand if some of these questions would reveal too much, but any advice would be helpful.", "votes": null } ], "comments": [ { "id": 2460122, "author_name": "danielburkhardt", "author_url": "", "post_date": "09/28/2023 15:29:37", "content": "

Hi! I don't think you can't easily view AWS buckets in Chrome, but you can browse via CLI. e.g. https://docs.aws.amazon.com/cli/latest/reference/s3/ls.html

\n

You can also download the data from the LINCS Data Portal: https://lincsportal.ccs.miami.edu/datasets/view/LDS-1611

\n

For more information on LINCS data format, I recommend: https://www.cell.com/cell/fulltext/S0092-8674(17)31309-0

", "votes": null, "replies": [ { "id": 2460162, "author_name": "laurasisson", "author_url": "", "post_date": "09/28/2023 15:52:10", "content": "

Thank you!
\nAccording the NIH LINCS page
\nThe five levels of data are

\n
 : Raw unprocessed flow cytometry data  Luminex (LXB)\n : Gene expression  per  genes  deconvolution (GEX)\n : Quantile-normalized gene expression profiles  landmark genes  imputed transcripts (Q2NORM  INF)\n : Gene signatures computed  z-scores relative  the plate population  control (ZSPCINF)  relative  the plate vehicle control (ZSVCINF)\n : Differential gene expression signatures\n
\n

In my understanding for this competition, the pseudobulked data produced something most similar to the Level 4 values relative to the plate population. The LINCS data and the competition data diverge when the trained LIMMA model produces the p-values.

\n

In your opinion, are the Level 4 z-scores a good proxy for the predictions values in the training/test set? If we want to leverage the LINCS data, should we attempt to train our own LIMMA models? Or is the Level 5 data, with a smaller memory footprint, also appropriate? I understand if some of these questions would reveal too much, but any advice would be helpful.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "2460110": "Hello,\n[In the main discussion post](https://www.kaggle.com/competitions/open-problems-single-cell-perturbations/discussion/440680) the organizers stated they uploaded LINCS data to `s3://saturn-kaggle-datasets/open-problems-single-cell-perturbations-optional/`\nI'd like to browse the files in Chrome, so I navigated to [https://s3.amazonaws.com/saturn-kaggle-datasets/open-problems-single-cell-perturbations-optional](https://s3.amazonaws.com/saturn-kaggle-datasets/open-problems-single-cell-perturbations-optional) and was redirected to [https://saturn-kaggle-datasets.s3.amazonaws.com/](https://saturn-kaggle-datasets.s3.amazonaws.com/)\n\nI don't seen the `open-problems-single-cell-perturbations-optional` folder or any related data. I think I'm accessing this wrong, but I'm not sure how to proceed. Using boto3 in Python I had no success either.", "2460122": "Hi! I don't think you can't easily view AWS buckets in Chrome, but you can browse via CLI. e.g. https://docs.aws.amazon.com/cli/latest/reference/s3/ls.html\n\nYou can also download the data from the LINCS Data Portal: https://lincsportal.ccs.miami.edu/datasets/view/LDS-1611\n\nFor more information on LINCS data format, I recommend: https://www.cell.com/cell/fulltext/S0092-8674(17)31309-0", "2460162": "Thank you!\nAccording the [NIH LINCS page](https://lincsproject.org/LINCS/tools/workflows/find-the-best-place-to-obtain-the-lincs-l1000-data)\nThe five levels of data are\n```\nLevel 1: Raw unprocessed flow cytometry data from Luminex (LXB)\nLevel 2: Gene expression values per 1000 genes after deconvolution (GEX)\nLevel 3: Quantile-normalized gene expression profiles of landmark genes and imputed transcripts (Q2NORM or INF)\nLevel 4: Gene signatures computed using z-scores relative to the plate population as control (ZSPCINF) or relative to the plate vehicle control (ZSVCINF)\nLevel 5: Differential gene expression signatures\n```\nIn my understanding for this competition, the pseudobulked data produced something most similar to the Level 4 values relative to the plate population. The LINCS data and the competition data diverge when the trained LIMMA model produces the p-values.\n\nIn your opinion, are the Level 4 z-scores a good proxy for the predictions values in the training/test set? If we want to leverage the LINCS data, should we attempt to train our own LIMMA models? Or is the Level 5 data, with a smaller memory footprint, also appropriate? I understand if some of these questions would reveal too much, but any advice would be helpful." }, "source": "meta" }