Hi everyone! I am not a biologist and I am not sure if I could show some result in this competition, but I would love to try. I can't understand how do I obtain the prediction target from the dataset. Do I need some specific biology knowledge for doing this?
", "messages": [ { "id": 2429118, "postDate": "2023-09-08T11:42:07.743Z", "content": "Hi everyone! I am not a biologist and I am not sure if I could show some result in this competition, but I would love to try. I can't understand how do I obtain the prediction target from the dataset. Do I need some specific biology knowledge for doing this?
", "rawMarkdown": "Hi everyone! I am not a biologist and I am not sure if I could show some result in this competition, but I would love to try. I can't understand how do I obtain the prediction target from the dataset. Do I need some specific biology knowledge for doing this?", "votes": 2 }, { "id": 2434422, "postDate": "2023-09-12T09:42:58.103Z", "content": "Am i right?
\n[GGGAA….]
\n[reactivity_0001, reactivity_0002, reactivity_0003, reactivity_0004, reactivity_0005….]
and thats what we should to predict?
", "rawMarkdown": "Am i right?\n\n[GGGAA....]\n[reactivity_0001, reactivity_0002,\treactivity_0003, reactivity_0004, reactivity_0005....]\n\nand thats what we should to predict?", "votes": 1, "replies": [ { "id": 2434698, "postDate": "2023-09-12T13:14:15.140Z", "content": "Correct. You are predicting the reactivity at each position.
", "rawMarkdown": "Correct. You are predicting the reactivity at each position.", "votes": 1 }, { "id": 2434873, "postDate": "2023-09-12T15:15:57.950Z", "content": "And in the submission file you have to have a running id from 0 to the end, increment for each reactivity. Inclusive id_min and id_max (index the RNA with index - id_min). So, keep the same order as the test_sequences, else you'll garble everything.
", "rawMarkdown": "And in the submission file you have to have a running id from 0 to the end, increment for each reactivity. Inclusive id_min and id_max (index the RNA with index - id_min). So, keep the same order as the test_sequences, else you'll garble everything.", "votes": 1 } ] }, { "id": 2429420, "postDate": "2023-09-08T15:34:52.213Z", "content": "While you don't need to be a full-fledge molecular biologist to compete, it does help to understand some of the basic concepts of bioinformatics and where it differs from general Data Science.
\nWe're trying to predict the \"structural properties\" of RNA, essentially. As such, it does help to have a modest understanding of thermodynamic relations in regards to RNA.
", "rawMarkdown": "While you don't need to be a full-fledge molecular biologist to compete, it does help to understand some of the basic concepts of bioinformatics and where it differs from general Data Science.\n\nWe're trying to predict the \"structural properties\" of RNA, essentially. As such, it does help to have a modest understanding of thermodynamic relations in regards to RNA.\n\n", "votes": 2, "replies": [ { "id": 2429746, "postDate": "2023-09-08T19:12:14.227Z", "content": "Thanks for the context! My education is not related to biology in any way, so I would probably skip this competition by now. Anyway it is not for an ML beginner for sure 😅
", "rawMarkdown": "Thanks for the context! My education is not related to biology in any way, so I would probably skip this competition by now. Anyway it is not for an ML beginner for sure 😅", "replies": [ { "id": 2429749, "postDate": "2023-09-08T19:17:55.620Z", "content": "If you're a beginner, there are several starter competitions available. I'd also highly recommend the Playground Series competitions. They are more challenging, but with the aim of learning new techniques to help you become a better data scientist!
", "rawMarkdown": "If you're a beginner, there are several starter competitions available. I'd also highly recommend the Playground Series competitions. They are more challenging, but with the aim of learning new techniques to help you become a better data scientist!", "votes": 2 } ] } ] }, { "id": 2429647, "postDate": "2023-09-08T17:46:16.227Z", "content": "According to the description:
\n\n\nFrom the Data page: At each position of each RNA sequence, there will be two ground truth values, corresponding to reactivity determined from two kinds of chemical mapping experiments, DMS_MaP and 2A3_MaP.
\n
The reactivity values can be seen in the training_data.csv
\nhowever what is not clear to me, is why they are clamping the predicted outputs to [0,1) when the reported reactivities in the csv can be negative
", "rawMarkdown": "According to the description:\n> From the Data page: At each position of each RNA sequence, there will be two ground truth values, corresponding to reactivity determined from two kinds of chemical mapping experiments, DMS_MaP and 2A3_MaP.\n\nThe reactivity values can be seen in the training_data.csv\n\n**however** what is not clear to me, is why they are clamping the predicted outputs to [0,1) when the reported reactivities in the csv can be **negative**", "replies": [ { "id": 2429772, "postDate": "2023-09-08T19:40:56.423Z", "content": "Any negative reported reactivities are experimental artifacts, not real values. Generally speaking, the experimental procedure used to generate these data allow us to chemically modify RNA nucleotides with dimethyl sulfate (DMS) signal molecules (see this paper if you're interested in more detail). Unpaired bases can accept the DMS modification, and paired bases cannot. The modifications can be quantified with genetic sequencing techniques, and we can normalize the data to a range from 1 (indicating a base is highly reactive) to 0 (indicating a base is not reactive). Values within this range can indicate that sometimes this base is paired, and sometimes it's not. RNA can form a wide range of structures for a given sequence, with different likelihoods of forming each structure in certain conditions.
", "rawMarkdown": "Any negative reported reactivities are experimental artifacts, not real values. Generally speaking, the experimental procedure used to generate these data allow us to chemically modify RNA nucleotides with dimethyl sulfate (DMS) signal molecules (see [this paper](https://www.nature.com/articles/nmeth.4057) if you're interested in more detail). Unpaired bases can accept the DMS modification, and paired bases cannot. The modifications can be quantified with genetic sequencing techniques, and we can normalize the data to a range from 1 (indicating a base is highly reactive) to 0 (indicating a base is not reactive). Values within this range can indicate that sometimes this base is paired, and sometimes it's not. RNA can form a wide range of structures for a given sequence, with different likelihoods of forming each structure in certain conditions.", "votes": 6, "replies": [ { "id": 2429874, "postDate": "2023-09-08T21:32:29.603Z", "content": "Thanks for the in-depth answer! Makes sense… 👍
", "rawMarkdown": "Thanks for the in-depth answer! Makes sense... 👍" }, { "id": 2429894, "postDate": "2023-09-08T21:59:45.927Z", "content": "Could we have more insight into the data preprocessing and normalization?
\nMy main question is: Are negative number or numbers larger than one meaningful? Or can they be clipped to be between 0 and 1 without any loss of information?
", "rawMarkdown": "Could we have more insight into the data preprocessing and normalization?\n\nMy main question is: Are negative number or numbers larger than one meaningful? Or can they be clipped to be between 0 and 1 without any loss of information?", "votes": 5 }, { "id": 2430288, "postDate": "2023-09-09T07:24:39.270Z", "content": "So during evaluation, all values should be capped between [0,1], right?
", "rawMarkdown": "So during evaluation, all values should be capped between [0,1], right?" }, { "id": 2469616, "postDate": "2023-10-06T13:30:46.707Z", "content": "I have the same question.
", "rawMarkdown": "I have the same question.", "isDeleted": true } ] }, { "id": 2430604, "postDate": "2023-09-09T12:05:33.937Z", "content": "\"there will be two ground truth values, corresponding to reactivity determined from two kinds of chemical mapping experiments, DMS_MaP and 2A3_MaP\"
\nThey cannot be seen in the downloadable file. Nor \"future\".
\nEdit: id, id_min and id_max are missing, too
The measured values are found in the reactivity_0001 through reactivity_0170 columns in the training data (as well as reactivity error columns which provide experimental error). Each row of training data will either be providing results for either DMS or 2A3, as specified by the experiment_type column. \"future\" is not relevant to training - it indicates whether the data will only be available in the future (all the training data is already available!). id/id_min/id_max are basically just bookkeeping that allow you to connect rows in your submission to rows in the test data (eg, the second sequence is id_min 177 through id_max 353, which means id 177 in your submission corresponds to the predicted value at the first position in the second sequence, and 353 corresponds to the predicted value at the last position of the second sequence).
", "rawMarkdown": "The measured values are found in the reactivity_0001 through reactivity_0170 columns in the training data (as well as reactivity error columns which provide experimental error). Each row of training data will either be providing results for either DMS or 2A3, as specified by the experiment_type column. \"future\" is not relevant to training - it indicates whether the data will only be available in the future (all the training data is already available!). id/id_min/id_max are basically just bookkeeping that allow you to connect rows in your submission to rows in the test data (eg, the second sequence is id_min 177 through id_max 353, which means id 177 in your submission corresponds to the predicted value at the first position in the second sequence, and 353 corresponds to the predicted value at the last position of the second sequence).", "votes": 5 }, { "id": 2434992, "postDate": "2023-09-12T16:24:35.847Z", "content": "CORRECTION: reactivity_0001 through reactivity_0206 - I misread the available columns
", "rawMarkdown": "CORRECTION: reactivity_0001 through reactivity_0206 - I misread the available columns", "votes": 1 } ] } ] } ], "comments": [ { "id": 2434422, "author_name": "Andrey Nemerov", "author_url": "", "post_date": "2023-09-12T09:42:58.103000", "content": "Am i right?
\n[GGGAA….]
\n[reactivity_0001, reactivity_0002, reactivity_0003, reactivity_0004, reactivity_0005….]
and thats what we should to predict?
", "votes": 1, "replies": [ { "id": 2434698, "author_name": "DigitalEmbrace", "author_url": "", "post_date": "2023-09-12T13:14:15.140000", "content": "Correct. You are predicting the reactivity at each position.
", "votes": 1, "replies": [] }, { "id": 2434873, "author_name": "Tord Malmgren", "author_url": "", "post_date": "2023-09-12T15:15:57.950000", "content": "And in the submission file you have to have a running id from 0 to the end, increment for each reactivity. Inclusive id_min and id_max (index the RNA with index - id_min). So, keep the same order as the test_sequences, else you'll garble everything.
", "votes": 1, "replies": [] } ] }, { "id": 2429420, "author_name": "Kent RB", "author_url": "", "post_date": "2023-09-08T15:34:52.213000", "content": "While you don't need to be a full-fledge molecular biologist to compete, it does help to understand some of the basic concepts of bioinformatics and where it differs from general Data Science.
\nWe're trying to predict the \"structural properties\" of RNA, essentially. As such, it does help to have a modest understanding of thermodynamic relations in regards to RNA.
", "votes": 2, "replies": [ { "id": 2429746, "author_name": "Danil Nizamov", "author_url": "", "post_date": "2023-09-08T19:12:14.227000", "content": "Thanks for the context! My education is not related to biology in any way, so I would probably skip this competition by now. Anyway it is not for an ML beginner for sure 😅
", "votes": 0, "replies": [ { "id": 2429749, "author_name": "Kent RB", "author_url": "", "post_date": "2023-09-08T19:17:55.620000", "content": "If you're a beginner, there are several starter competitions available. I'd also highly recommend the Playground Series competitions. They are more challenging, but with the aim of learning new techniques to help you become a better data scientist!
", "votes": 2, "replies": [] } ] } ] }, { "id": 2429647, "author_name": "E/S Pronk", "author_url": "", "post_date": "2023-09-08T17:46:16.227000", "content": "According to the description:
\n\n\nFrom the Data page: At each position of each RNA sequence, there will be two ground truth values, corresponding to reactivity determined from two kinds of chemical mapping experiments, DMS_MaP and 2A3_MaP.
\n
The reactivity values can be seen in the training_data.csv
\nhowever what is not clear to me, is why they are clamping the predicted outputs to [0,1) when the reported reactivities in the csv can be negative
", "votes": 0, "replies": [ { "id": 2429772, "author_name": "Thomas", "author_url": "", "post_date": "2023-09-08T19:40:56.423000", "content": "Any negative reported reactivities are experimental artifacts, not real values. Generally speaking, the experimental procedure used to generate these data allow us to chemically modify RNA nucleotides with dimethyl sulfate (DMS) signal molecules (see this paper if you're interested in more detail). Unpaired bases can accept the DMS modification, and paired bases cannot. The modifications can be quantified with genetic sequencing techniques, and we can normalize the data to a range from 1 (indicating a base is highly reactive) to 0 (indicating a base is not reactive). Values within this range can indicate that sometimes this base is paired, and sometimes it's not. RNA can form a wide range of structures for a given sequence, with different likelihoods of forming each structure in certain conditions.
", "votes": 6, "replies": [ { "id": 2429874, "author_name": "E/S Pronk", "author_url": "", "post_date": "2023-09-08T21:32:29.603000", "content": "Thanks for the in-depth answer! Makes sense… 👍
", "votes": 0, "replies": [] }, { "id": 2429894, "author_name": "Matt", "author_url": "", "post_date": "2023-09-08T21:59:45.927000", "content": "Could we have more insight into the data preprocessing and normalization?
\nMy main question is: Are negative number or numbers larger than one meaningful? Or can they be clipped to be between 0 and 1 without any loss of information?
", "votes": 5, "replies": [] }, { "id": 2430288, "author_name": "Ai Vu Hong", "author_url": "", "post_date": "2023-09-09T07:24:39.270000", "content": "So during evaluation, all values should be capped between [0,1], right?
", "votes": 0, "replies": [] }, { "id": 2469616, "author_name": "", "author_url": "", "post_date": "2023-10-06T13:30:46.707000", "content": "I have the same question.
", "votes": 0, "replies": [] } ] }, { "id": 2430604, "author_name": "Tord Malmgren", "author_url": "", "post_date": "2023-09-09T12:05:33.937000", "content": "\"there will be two ground truth values, corresponding to reactivity determined from two kinds of chemical mapping experiments, DMS_MaP and 2A3_MaP\"
\nThey cannot be seen in the downloadable file. Nor \"future\".
\nEdit: id, id_min and id_max are missing, too
The measured values are found in the reactivity_0001 through reactivity_0170 columns in the training data (as well as reactivity error columns which provide experimental error). Each row of training data will either be providing results for either DMS or 2A3, as specified by the experiment_type column. \"future\" is not relevant to training - it indicates whether the data will only be available in the future (all the training data is already available!). id/id_min/id_max are basically just bookkeeping that allow you to connect rows in your submission to rows in the test data (eg, the second sequence is id_min 177 through id_max 353, which means id 177 in your submission corresponds to the predicted value at the first position in the second sequence, and 353 corresponds to the predicted value at the last position of the second sequence).
", "votes": 5, "replies": [] }, { "id": 2434992, "author_name": "Jonathan Romano", "author_url": "", "post_date": "2023-09-12T16:24:35.847000", "content": "CORRECTION: reactivity_0001 through reactivity_0206 - I misread the available columns
", "votes": 1, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "2429118": "Hi everyone! I am not a biologist and I am not sure if I could show some result in this competition, but I would love to try. I can't understand how do I obtain the prediction target from the dataset. Do I need some specific biology knowledge for doing this?", "2434422": "Am i right?\n\n[GGGAA....]\n[reactivity_0001, reactivity_0002,\treactivity_0003, reactivity_0004, reactivity_0005....]\n\nand thats what we should to predict?", "2429420": "While you don't need to be a full-fledge molecular biologist to compete, it does help to understand some of the basic concepts of bioinformatics and where it differs from general Data Science.\n\nWe're trying to predict the \"structural properties\" of RNA, essentially. As such, it does help to have a modest understanding of thermodynamic relations in regards to RNA.\n\n", "2429647": "According to the description:\n> From the Data page: At each position of each RNA sequence, there will be two ground truth values, corresponding to reactivity determined from two kinds of chemical mapping experiments, DMS_MaP and 2A3_MaP.\n\nThe reactivity values can be seen in the training_data.csv\n\n**however** what is not clear to me, is why they are clamping the predicted outputs to [0,1) when the reported reactivities in the csv can be **negative**" } }