Q1. Most reactivity values in data vary from (-3 to 3). What is the significance of values being negative for both profiles (DMS & 2A3)? While making predictions, should the model only give values between (0 to 1) or make predictions in negative value also?
", "messages": [ { "id": "2503147", "postDate": "10/28/2023 19:33:27", "content": "Q1. Most reactivity values in data vary from (-3 to 3). What is the significance of values being negative for both profiles (DMS & 2A3)? While making predictions, should the model only give values between (0 to 1) or make predictions in negative value also?
", "rawMarkdown": "Q1. Most reactivity values in data vary from (-3 to 3). What is the significance of values being negative for both profiles (DMS & 2A3)? While making predictions, should the model only give values between (0 to 1) or make predictions in negative value also?", "votes": null }, { "id": "2503311", "postDate": "10/29/2023 00:14:11", "content": "Thanks for the question -- other folks have wondered too. I've added to the data description the explanation: \"The values should be greater than or equal to zero, but due to experimental errors can become negative. The values are normalized so that the 90th percentile value within each dataset is 1.0.\"
", "rawMarkdown": "Thanks for the question -- other folks have wondered too. I've added to the data description the explanation: \"The values should be greater than or equal to zero, but due to experimental errors can become negative. The values are normalized so that the 90th percentile value within each dataset is 1.0.\"", "votes": null }, { "id": "2503369", "postDate": "10/29/2023 03:09:13", "content": "Interesting, that seems to slightly conflict with this comment where @shujun717 says that the values should lie within [0, 1]. Can you clarify? https://www.kaggle.com/competitions/stanford-ribonanza-rna-folding/discussion/439094#2436956
", "rawMarkdown": "Interesting, that seems to slightly conflict with this comment where @shujun717 says that the values should lie within [0, 1]. Can you clarify? https://www.kaggle.com/competitions/stanford-ribonanza-rna-folding/discussion/439094#2436956", "votes": null }, { "id": "2503395", "postDate": "10/29/2023 04:01:23", "content": "what is the contradiction?
\narray = np.linspace(, , )\narray[] = \n()\nThank you for the answer. One more question:
\nIs there any relation between reactivity values and reactivity errors (i.e. value ± errors)? If not, then it would be hard to incorporate the error values in the model training.
So if it helps the model perform better, we can set all negative reactivity values to 0 while data pre-processing?
", "rawMarkdown": "So if it helps the model perform better, we can set all negative reactivity values to **0** while data pre-processing?", "votes": null }, { "id": "2503971", "postDate": "10/29/2023 15:09:09", "content": "@shujun717 is right -- the clamping between [0,1] occurs with both your models and the data before MAE computation for scoring.
\nWe've chosen however to give you the raw data values in case that provides any kind of additional useful signal during training.
", "rawMarkdown": "shujun717 is right -- the clamping between [0,1] occurs with both your models and the data before MAE computation for scoring. \n\nWe've chosen however to give you the raw data values in case that provides any kind of additional useful signal during training.", "votes": null }, { "id": "2503988", "postDate": "10/29/2023 15:14:25", "content": "Yes, you are welcome to clamp the data values between 0 and 1 during training.
", "rawMarkdown": "Yes, you are welcome to clamp the data values between 0 and 1 during training.", "votes": null }, { "id": "2503991", "postDate": "10/29/2023 15:15:14", "content": "That makes sense, thank you!
", "rawMarkdown": "That makes sense, thank you!", "votes": null }, { "id": "2503996", "postDate": "10/29/2023 15:18:37", "content": "You are welcome to ignore the reactivity errors.
\nWe do not know if there is any useful signal in the experimental errors in this competition for auxiliary learning tasks, for data augmentation, or for definition of your training loss function, though we would be curious to find out!
\nYou may be interested to look at how the error values were used in the previous Kaggle OpenVaccine competition, e.g., by first place solution @nullrecurrent, as described in her summary post.
", "rawMarkdown": "You are welcome to ignore the reactivity errors. \n\nWe do not know if there is any useful signal in the experimental errors in this competition for auxiliary learning tasks, for data augmentation, or for definition of your training loss function, though we would be curious to find out! \n\nYou may be interested to look at how the error values were used in the previous Kaggle OpenVaccine competition, e.g., by first place solution @nullrecurrent, as described in her [summary post](https://www.kaggle.com/c/stanford-covid-vaccine/discussion/189620).", "votes": null }, { "id": "2504599", "postDate": "10/30/2023 03:22:55", "content": "That maybe true but I was working on the hypothesis that the negative values came from the Shannon entropy aggregation. The values of multiple reads are reduced to a probability function 0.5 being equal probability and 1 being perfect information. And it reads that values in the top percentiles are taken and the max of the lower quartile or r = 0.001. Is there any credibility to my assumption?
", "rawMarkdown": "That maybe true but I was working on the hypothesis that the negative values came from the Shannon entropy aggregation. The values of multiple reads are reduced to a probability function 0.5 being equal probability and 1 being perfect information. And it reads that values in the top percentiles are taken and the max of the lower quartile or r = 0.001. Is there any credibility to my assumption?", "votes": null }, { "id": "2505296", "postDate": "10/30/2023 14:02:16", "content": "Good question.
\nThe reactivity_error_* values are not Shannon entropy. These values are our best estimate of what the standard deviation of measurements would be if we repeated the measurements (and we have actually checked this expectation in some pilot experiments!).
For the Ribonanza data, the error values are computed based on:
\nThe error is assumed to be Poisson and is estimated as sqrt(counts), where counts is an integer that literally counts the number of mutational events seen in sequencers. These events are the signature stored in the sequenced DNA that there was a chemical modification in the RNA during the chemical mapping. The mutational signature is placed in the DNA by the enzymatic machine, called reverse transcriptase, which we use to convert the RNA to DNA for measurements on Illumina DNA sequencers.
The errors (along with the raw counts) are divided by the total number of sequences captured by the sequencer ('coverage'). So the reactivity estimates are, to start with, numbers between 0 and 1.
Our profiles rely on background subtraction of measurements made on the RNA with the chemical mapping and measurements made without chemical mapping. The errors of the two measurements are summed in quadrature, along with a 'pseudocount' of 1/coverage which ensures that errors do not go below zero. Note that this background subtraction step can lead to negative values for the reactivity, although chemically such values should not go below zero -- when they do, that's a sign of statistical or systematic error. The values of the estimated error on the reactivity, however, are positive.
The reactivity values are normalized to the 90th percentile value seen across all sequence positions across all sequences probed in a given experiment. The same normalization constant used to normalize the reactivity is also applied to scale the reactivity error.
If you want to get into the details, the data processing code and examples are publicly available, with the error estimation largely happening in the get_reactivity MATLAB function. (Note that there are some additional steps that classify different mutation types --deletions, insertions, mutations-- and correct for some ambiguities that occur when a deletion occurs in a stretch of the same nucleotides.)
", "rawMarkdown": "Good question. \n\nThe `reactivity_error_*` values are not Shannon entropy. These values are our best estimate of what the standard deviation of measurements would be if we repeated the measurements (and we have actually checked this expectation in some pilot experiments!).\n\nFor the Ribonanza data, the error values are computed based on:\n\n1. The error is assumed to be Poisson and is estimated as `sqrt(counts)`, where `counts` is an integer that literally counts the number of mutational events seen in sequencers. These events are the signature stored in the sequenced DNA that there was a chemical modification in the RNA during the chemical mapping. The mutational signature is placed in the DNA by the enzymatic machine, called reverse transcriptase, which we use to convert the RNA to DNA for measurements on Illumina DNA sequencers. \n\n2. The errors (along with the raw counts) are divided by the total number of sequences captured by the sequencer ('coverage'). So the reactivity estimates are, to start with, numbers between 0 and 1.\n\n3. Our profiles rely on background subtraction of measurements made on the RNA with the chemical mapping and measurements made without chemical mapping. The errors of the two measurements are summed in quadrature, along with a 'pseudocount' of 1/coverage which ensures that errors do not go below zero. Note that this background subtraction step can lead to negative values for the reactivity, although chemically such values should not go below zero -- when they do, that's a sign of statistical or systematic error. The values of the estimated error on the reactivity, however, are positive.\n\n4. The reactivity values are normalized to the 90th percentile value seen across all sequence positions across all sequences probed in a given experiment. The same normalization constant used to normalize the reactivity is also applied to scale the reactivity error.\n\nIf you want to get into the details, the data processing code and examples are [publicly available](https://github.com/DasLab/ubr), with the error estimation largely happening in the [get_reactivity](https://github.com/DasLab/ubr/blob/main/matlab/data/get_reactivity.m) MATLAB function. (Note that there are some additional steps that classify different mutation types --deletions, insertions, mutations-- and correct for some ambiguities that occur when a deletion occurs in a stretch of the same nucleotides.)", "votes": null }, { "id": "2505951", "postDate": "10/31/2023 02:15:02", "content": "We were discussing on discord. Turns out Ubr is the culprit for this data generation. I am still working out how it actually ends up negative.
", "rawMarkdown": "We were discussing on discord. Turns out [Ubr](https://github.com/DasLab/ubr) is the culprit for this data generation. I am still working out how it actually ends up negative.", "votes": null }, { "id": "2510436", "postDate": "11/03/2023 02:02:45", "content": "Great! Nicely explained.
", "rawMarkdown": "Great! Nicely explained.", "votes": null } ], "comments": [ { "id": 2503311, "author_name": "rhijudas", "author_url": "", "post_date": "10/29/2023 00:14:11", "content": "Thanks for the question -- other folks have wondered too. I've added to the data description the explanation: \"The values should be greater than or equal to zero, but due to experimental errors can become negative. The values are normalized so that the 90th percentile value within each dataset is 1.0.\"
", "votes": null, "replies": [ { "id": 2503369, "author_name": "marktenenholtz", "author_url": "", "post_date": "10/29/2023 03:09:13", "content": "Interesting, that seems to slightly conflict with this comment where @shujun717 says that the values should lie within [0, 1]. Can you clarify? https://www.kaggle.com/competitions/stanford-ribonanza-rna-folding/discussion/439094#2436956
", "votes": null, "replies": [ { "id": 2503395, "author_name": "sapr3s", "author_url": "", "post_date": "10/29/2023 04:01:23", "content": "what is the contradiction?
\narray = np.linspace(, , )\narray[] = \n()\n@shujun717 is right -- the clamping between [0,1] occurs with both your models and the data before MAE computation for scoring.
\nWe've chosen however to give you the raw data values in case that provides any kind of additional useful signal during training.
", "votes": null, "replies": [ { "id": 2503991, "author_name": "marktenenholtz", "author_url": "", "post_date": "10/29/2023 15:15:14", "content": "That makes sense, thank you!
", "votes": null, "replies": [] } ] } ] }, { "id": 2503591, "author_name": "satishgaurav", "author_url": "", "post_date": "10/29/2023 08:29:34", "content": "So if it helps the model perform better, we can set all negative reactivity values to 0 while data pre-processing?
", "votes": null, "replies": [ { "id": 2503988, "author_name": "rhijudas", "author_url": "", "post_date": "10/29/2023 15:14:25", "content": "Yes, you are welcome to clamp the data values between 0 and 1 during training.
", "votes": null, "replies": [] } ] }, { "id": 2504599, "author_name": "tuttlen", "author_url": "", "post_date": "10/30/2023 03:22:55", "content": "That maybe true but I was working on the hypothesis that the negative values came from the Shannon entropy aggregation. The values of multiple reads are reduced to a probability function 0.5 being equal probability and 1 being perfect information. And it reads that values in the top percentiles are taken and the max of the lower quartile or r = 0.001. Is there any credibility to my assumption?
", "votes": null, "replies": [ { "id": 2505296, "author_name": "rhijudas", "author_url": "", "post_date": "10/30/2023 14:02:16", "content": "Good question.
\nThe reactivity_error_* values are not Shannon entropy. These values are our best estimate of what the standard deviation of measurements would be if we repeated the measurements (and we have actually checked this expectation in some pilot experiments!).
For the Ribonanza data, the error values are computed based on:
\nThe error is assumed to be Poisson and is estimated as sqrt(counts), where counts is an integer that literally counts the number of mutational events seen in sequencers. These events are the signature stored in the sequenced DNA that there was a chemical modification in the RNA during the chemical mapping. The mutational signature is placed in the DNA by the enzymatic machine, called reverse transcriptase, which we use to convert the RNA to DNA for measurements on Illumina DNA sequencers.
The errors (along with the raw counts) are divided by the total number of sequences captured by the sequencer ('coverage'). So the reactivity estimates are, to start with, numbers between 0 and 1.
Our profiles rely on background subtraction of measurements made on the RNA with the chemical mapping and measurements made without chemical mapping. The errors of the two measurements are summed in quadrature, along with a 'pseudocount' of 1/coverage which ensures that errors do not go below zero. Note that this background subtraction step can lead to negative values for the reactivity, although chemically such values should not go below zero -- when they do, that's a sign of statistical or systematic error. The values of the estimated error on the reactivity, however, are positive.
The reactivity values are normalized to the 90th percentile value seen across all sequence positions across all sequences probed in a given experiment. The same normalization constant used to normalize the reactivity is also applied to scale the reactivity error.
If you want to get into the details, the data processing code and examples are publicly available, with the error estimation largely happening in the get_reactivity MATLAB function. (Note that there are some additional steps that classify different mutation types --deletions, insertions, mutations-- and correct for some ambiguities that occur when a deletion occurs in a stretch of the same nucleotides.)
", "votes": null, "replies": [ { "id": 2510436, "author_name": "mdrakibtrofder", "author_url": "", "post_date": "11/03/2023 02:02:45", "content": "Great! Nicely explained.
", "votes": null, "replies": [] } ] } ] } ] }, { "id": 2503586, "author_name": "satishgaurav", "author_url": "", "post_date": "10/29/2023 08:27:08", "content": "Thank you for the answer. One more question:
\nIs there any relation between reactivity values and reactivity errors (i.e. value ± errors)? If not, then it would be hard to incorporate the error values in the model training.
You are welcome to ignore the reactivity errors.
\nWe do not know if there is any useful signal in the experimental errors in this competition for auxiliary learning tasks, for data augmentation, or for definition of your training loss function, though we would be curious to find out!
\nYou may be interested to look at how the error values were used in the previous Kaggle OpenVaccine competition, e.g., by first place solution @nullrecurrent, as described in her summary post.
", "votes": null, "replies": [] } ] }, { "id": 2505951, "author_name": "tuttlen", "author_url": "", "post_date": "10/31/2023 02:15:02", "content": "We were discussing on discord. Turns out Ubr is the culprit for this data generation. I am still working out how it actually ends up negative.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "2503147": "Q1. Most reactivity values in data vary from (-3 to 3). What is the significance of values being negative for both profiles (DMS & 2A3)? While making predictions, should the model only give values between (0 to 1) or make predictions in negative value also?", "2503311": "Thanks for the question -- other folks have wondered too. I've added to the data description the explanation: \"The values should be greater than or equal to zero, but due to experimental errors can become negative. The values are normalized so that the 90th percentile value within each dataset is 1.0.\"", "2503369": "Interesting, that seems to slightly conflict with this comment where @shujun717 says that the values should lie within [0, 1]. Can you clarify? https://www.kaggle.com/competitions/stanford-ribonanza-rna-folding/discussion/439094#2436956", "2503395": "what is the contradiction? \n```Python\narray = np.linspace(0, 1, 10)\narray[8] = 1\nprint(f\"the 90th percentile: {np.percentile(array, 90)}\")\n```", "2503586": "Thank you for the answer. One more question: \nIs there any relation between reactivity values and reactivity errors (i.e. value ± errors)? If not, then it would be hard to incorporate the error values in the model training.", "2503591": "So if it helps the model perform better, we can set all negative reactivity values to **0** while data pre-processing?", "2503971": "shujun717 is right -- the clamping between [0,1] occurs with both your models and the data before MAE computation for scoring. \n\nWe've chosen however to give you the raw data values in case that provides any kind of additional useful signal during training.", "2503988": "Yes, you are welcome to clamp the data values between 0 and 1 during training.", "2503991": "That makes sense, thank you!", "2503996": "You are welcome to ignore the reactivity errors. \n\nWe do not know if there is any useful signal in the experimental errors in this competition for auxiliary learning tasks, for data augmentation, or for definition of your training loss function, though we would be curious to find out! \n\nYou may be interested to look at how the error values were used in the previous Kaggle OpenVaccine competition, e.g., by first place solution @nullrecurrent, as described in her [summary post](https://www.kaggle.com/c/stanford-covid-vaccine/discussion/189620).", "2504599": "That maybe true but I was working on the hypothesis that the negative values came from the Shannon entropy aggregation. The values of multiple reads are reduced to a probability function 0.5 being equal probability and 1 being perfect information. And it reads that values in the top percentiles are taken and the max of the lower quartile or r = 0.001. Is there any credibility to my assumption?", "2505296": "Good question. \n\nThe `reactivity_error_*` values are not Shannon entropy. These values are our best estimate of what the standard deviation of measurements would be if we repeated the measurements (and we have actually checked this expectation in some pilot experiments!).\n\nFor the Ribonanza data, the error values are computed based on:\n\n1. The error is assumed to be Poisson and is estimated as `sqrt(counts)`, where `counts` is an integer that literally counts the number of mutational events seen in sequencers. These events are the signature stored in the sequenced DNA that there was a chemical modification in the RNA during the chemical mapping. The mutational signature is placed in the DNA by the enzymatic machine, called reverse transcriptase, which we use to convert the RNA to DNA for measurements on Illumina DNA sequencers. \n\n2. The errors (along with the raw counts) are divided by the total number of sequences captured by the sequencer ('coverage'). So the reactivity estimates are, to start with, numbers between 0 and 1.\n\n3. Our profiles rely on background subtraction of measurements made on the RNA with the chemical mapping and measurements made without chemical mapping. The errors of the two measurements are summed in quadrature, along with a 'pseudocount' of 1/coverage which ensures that errors do not go below zero. Note that this background subtraction step can lead to negative values for the reactivity, although chemically such values should not go below zero -- when they do, that's a sign of statistical or systematic error. The values of the estimated error on the reactivity, however, are positive.\n\n4. The reactivity values are normalized to the 90th percentile value seen across all sequence positions across all sequences probed in a given experiment. The same normalization constant used to normalize the reactivity is also applied to scale the reactivity error.\n\nIf you want to get into the details, the data processing code and examples are [publicly available](https://github.com/DasLab/ubr), with the error estimation largely happening in the [get_reactivity](https://github.com/DasLab/ubr/blob/main/matlab/data/get_reactivity.m) MATLAB function. (Note that there are some additional steps that classify different mutation types --deletions, insertions, mutations-- and correct for some ambiguities that occur when a deletion occurs in a stretch of the same nucleotides.)", "2505951": "We were discussing on discord. Turns out [Ubr](https://github.com/DasLab/ubr) is the culprit for this data generation. I am still working out how it actually ends up negative.", "2510436": "Great! Nicely explained." }, "source": "meta" }