{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"\n\n

🧬 Stanford Ribonanza RNA Folding 🧬

","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","id":"BO0Qzc3gX9Wu"}},{"cell_type":"markdown","source":"# 1. Introduction\n\nI'm *excited* about this new competition, which combines biologic knowhow of RNA with Engineering and Machine learning skills 👟❀\n\n\nThis notebook is aimed as a first introduction to the world of RNA and the biology behind it. 🔬
\nWe will go through some *theory* first, explain the concepts and provide illustrations when needed!

\nI tried to make it simple, so that more people can follow. The theory in science and biology is constantly evolving, so you can use the information below as a form of basics to get started and tackle more advanced topics in this competition.","metadata":{"id":"jk5JyoRVX9Wx"}},{"cell_type":"markdown","source":"# 2. RNA theory basic 🧬 \n","metadata":{"id":"f8c6gyc2X9Wx"}},{"cell_type":"markdown","source":"## 2.1 What is RNA?","metadata":{"id":"3mNBTMMPX9Wy"}},{"cell_type":"markdown","source":"RNA, or ribonucleic acid, is a molecule in your body that plays a crucial role in carrying out instructions for building and maintaining your cells. Think of it as a messenger that uses genetic information from your DNA (deoxyribonucleic acid) to the parts of your cells that make proteins. Proteins are like the workers in your body, performing various tasks to keep you alive and functioning.\n\nSo, in simple terms, RNA is a molecule that helps transfer genetic information from DNA to make sure cells can do their jobs properly.","metadata":{"id":"ML0pQ_1-X9Wy"}},{"cell_type":"markdown","source":"## 2.2 What is the structure of RNA?\n\"screen2021-08-26-14-50-37\"\n\n[Image source](https://www.science.org/doi/10.1126/science.abk1971)\n\n### Primary structure\nThe primary structure of RNA refers to the linear sequence of nucleotides that make up the RNA molecule. Nucleotides are the building blocks of RNA, and each nucleotide consists of three components:\n\n* Sugar : In RNA, the sugar component is called ribose. It's a five-carbon sugar molecule that forms the backbone of the RNA strand. This ribose sugar is connected to the other components of the nucleotide.\n\n* Phosphate Group : Each nucleotide also contains a phosphate group. The phosphate group links the ribose sugars together to form the sugar-phosphate backbone of the RNA molecule.\n\n* Nucleotide Base : The third component of a nucleotide is a nitrogenous base. In RNA, there are four possible bases:\n\n\n

Adenine (A)

\n

Cyosine (C)

\n

Guanine (G)

\n

Uracil (U)

\n \nThe primary structure of RNA is the specific order in which these four bases are arranged along the sugar-phosphate backbone. This sequence of bases carries the genetic information in the RNA molecule, much like the letters in a sentence convey meaning in written language.\n\nFor example, a simple RNA sequence might look like this:\nAGCUAGUCAUCGGU\n\nIn this sequence, the order of bases (A, G, C, U, A, etc.) represents the primary structure of the RNA molecule. This sequence encodes the instructions for building a specific protein or performing other cellular functions, depending on the type of RNA and its role in the cell.\n\n### Secondary structure\n\n\nThe secondary structure of RNA refers to the specific way in which the RNA molecule folds and forms various structural elements based on the interactions between its constituent nucleotides. These interactions primarily involve the complementary base pairing between adenine (A) and uracil (U), as well as cytosine (C) and guanine (G). Here are some key aspects of RNA secondary structure:\n\n* Base Pairing: In Watson-Crick base pairing, adenine (A) pairs with uracil (U), and cytosine (C) pairs with guanine (G). The pairing happens through hydrogen bonds, A and U pair with two hydrogen bonds and G and C with three hydrogen bonds. This complementary base pairing forms one of the basics for the secondary structure.\n\n* Stacking: Stacking refers to the phenomenon where adjacent nucleotide bases along the RNA strand form close, stable interactions with each other. This interaction is sometimes called \"base stacking\" or \"π-π stacking\". Imagine you have a stack of coins, and each coin represents a nucleotide base (A, U, C, or G) in the RNA molecule. When these bases come together, they can arrange themselves in a way that allows the flat surfaces of the bases to stack on top of each other, much like how coins stack in a neat pile.
Unlike in base pairing, here each nucleotide (A, U, C or G) can stack on top of the other.\n\n* Stem-Loop Structures or hairpin loops : A very common secondary structure element in RNA is the formation of stem-loop structures, also known as hairpin loops. This occurs when a sequence of nucleotides in the RNA molecule folds back on itself, creating a double-stranded region (the stem) connected to a loop. The stem is formed by base pairing, while the loop is a single-stranded region with no pairing. These stem-loop structures can have functional roles within the cell.\n\n* Bulges and Internal Loops: Sometimes, in the process of forming a double-stranded structure, there may be unpaired nucleotides that create bulges or internal loops. Bulges have unpaired nucleotides on one side while internal loops have unpaired nucleotides on both sides of the double strand. These regions can also have specific functions or influence the overall shape of the RNA molecule.\n\n\n* Pseudoknots and multi-loops: In more complex RNA structures, pseudoknots can form. These are regions where the RNA strand folds back on itself and creates additional levels of base pairing, resulting in a knot-like structure. Pseudoknots can take on many various forms like Kissing Hairpin, H-Type and more. They play important roles in the function of certain RNA molecules.\n\nOverall, the secondary structure of RNA is essential for its function. It determines how the molecule folds into specific shapes, which can be critical for various cellular processes, including protein synthesis, RNA processing, and regulatory functions. Think of it as the way the RNA molecule folds itself, creating patterns like loops and stems, to carry out its biological tasks within the cell.\n\n### Tertiary structure\n\nThe tertiary structure of RNA is the overall three-dimensional shape that the RNA molecule adopts. It results from the secondary structure and more complex interactions between distant parts of the molecule. The 3D shape determines how an RNA molecule interacts with other molecules, such as proteins, metal ions, or other RNAs. \n\nThe tertiary structure is critical for the RNA's specific function. It's like the final, folded origami form of the RNA molecule, determining its function within the cel.\n\nHere's an overview of RNA tertiary structure:\n\n* Interaction of Distant Regions: The tertiary structure arises from interactions between distant parts of the RNA molecule, which can also be far apart in the primary structure. These interactions are formed by many complex interactions in secondary structure like base stacking, base stacking, formation of loops or stems and pseudoknots.\n\n* RNA Motifs: Certain structural regions structural involve specific arrangements of nucleotides, called motifs, that stabilize the RNA's 3D shape and enable interactions with other molecules. Examples are the GNRA tetraloop, kink-turn, and riboswitches, which play important roles in RNA tertiary structure.\n\n* Metal Ions: Metal ions, such as magnesium (Mg²⁺), can bind to RNA molecules and play a crucial role in stabilizing their tertiary structure. These ions often coordinate with phosphate groups in the RNA backbone.\n\n* Protein Interactions: RNA molecules can interact with proteins, and these interactions can influence RNA folding. Some RNA-binding proteins help RNA molecules adopt their proper 3D shapes for specific functions.\n\n* Function: The tertiary structure of RNA is intimately linked to its function. Different RNA molecules, such as ribosomal RNA (rRNA), transfer RNA (tRNA), and ribozymes, have distinct tertiary structures that enable them to perform essential cellular functions, such as protein synthesis and catalyzing chemical reactions.\n\nIn summary, RNA tertiary structure is the complex 3D arrangement of an RNA molecule that arises from interactions between distant parts of the molecule. This structure is crucial for the RNA's function, whether it's guiding protein synthesis, participating in cellular processes, or carrying out catalytic activities. Understanding RNA's tertiary structure is essential for advancing our knowledge of cellular biology and designing RNA-based therapies and technologies.","metadata":{}},{"cell_type":"markdown","source":"## 2.3 RNA folding\n\nRNA folding is the process in which the RNA folds into itself involving the primary, secondary and tertiary structure.
RNA folding involves molecule bending, twisting, and arranging itself into the specific 3D pattern. This process is not static; it can change in response to environmental conditions or interactions with other molecules.\n\nPredicting RNA folding and the tertiary structure of RNA computationally is a challenging task. Researchers use various algorithms and experimental data to model and understand the 3D shapes of RNA molecules. The study of RNA and its structure is expanding, and reasearch constantly exposes new findings.\n\nIn this competition, understanding the biological aspects of RNA folding is key to create accurate models!\n\nHere's a beautiful animation of the process of RNA folding in real-time action:","metadata":{}},{"cell_type":"code","source":"from IPython.display import YouTubeVideo\nYouTubeVideo('2XTi9LG9NnU', width=800, height=300)","metadata":{"execution":{"iopub.status.busy":"2023-09-15T12:13:58.729595Z","iopub.execute_input":"2023-09-15T12:13:58.732064Z","iopub.status.idle":"2023-09-15T12:13:58.854433Z","shell.execute_reply.started":"2023-09-15T12:13:58.731988Z","shell.execute_reply":"2023-09-15T12:13:58.853087Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2.4 What are the different types of RNA?\n\n### Coding RNA (mRNA) vs. Non-coding RNA\nRNA's can be divided into those that are coding (≈2%) and those that are non-coding (≈98%).
\n\n Coding RNA (= mRNA) , acts like a chef's recipe book, providing the step-by-step instructions to assemble proteins in the cell. It carries the instructions (a copy from DNA) in the cell's nucleus to the ribosomes (protein-making machinery) in the cytoplasm. These instructions tell the ribosome how to build a specific protein. Without mRNA, the cell wouldn't know how to make the proteins it needs for growth, repair, and various cellular processes.\n\n Non-coding RNA , on the other hand, doesn't provide instructions for making proteins. Instead, it takes on various behind-the-scenes roles within the cell. Some types help manage when and how genes are used, while others assist in the processing and modification of different RNA molecules. Essentially, non-coding RNA plays more of a supportive and regulatory role in the cell's overall operations.\n\n> 📌**Note**: Describing all types of non-coding RNA in more detail is currently beyond the scope of the notebook. If you are interested to know more, I recommend checking out following Youtube videos about [Non-coding RNAs](https://www.youtube.com/watch?v=hKU8JQOgYmU) and [Structue, function, types of RNAs](https://www.youtube.com/watch?v=FThA4Vxs3v4)\n\n### Involved in protein synthesis: mRNA, tRNA and rRNA\n\n\n[Image source](https://www.yourgenome.org/wp-content/uploads/2022/04/rna-types_yourgenome.png)\n\nThe three main types in protein synthesis: mRNA, tRNA, and rRNA play critical roles in the process of making proteins, which are fundamental to life and the functioning of cells. Here's why each of them is important, using the analogy of preparing a recipe:
\n\n* mRNA (Messenger RNA) : mRNA is the coding RNA and acts like a recipe card. It contains the instructions for making a specific dish (protein) that is made of ingredients (amino acids). Just as a recipe tells you what ingredients to use and how to prepare them, mRNA carries the recipe for a protein and tells the cell which amino acids to use and in what order.\n\n* tRNA (Transfer RNA) : tRNA is like the chef's assistant who brings the ingredients (amino acids) to the kitchen. Imagine you're cooking, and you need specific ingredients (amino acids) for your dish. Each tRNA molecule is like a little delivery person that brings the right ingredient (amino acid) to the chef (ribosome) according to the recipe's instructions (mRNA).\n\n* rRNA (Ribosomal RNA) : rRNA is like the kitchen and cooking utensils. It forms the structure of the kitchen and helps the chef (ribosome) assemble the ingredients (amino acids) and cook them into the final dish (protein). It's the essential framework for the entire cooking process.\n\nIn summary, mRNA, tRNA, and rRNA are essential components of the cellular machinery that translates genetic information into functional proteins. Proteins are involved in virtually every aspect of cellular function, making these RNA molecules indispensable for the survival and proper functioning of cells and organisms.\n\n> 📌**Note**: For more detailed knowledge about mRNA, tRNA and rRNA please refer to Youtube videos about [mRNA, tRNA and rRNA](https://www.youtube.com/watch?v=NQ4is2HeoHI) and [mRNA, tRNA, and rRNA function](http://https://www.youtube.com/watch?v=1THyMOk3WU0)","metadata":{}},{"cell_type":"markdown","source":"## 2.5 How is RNA different from DNA?\n\n\n[Image source](https://assets.technologynetworks.com/production/dynamic/images/content/296719/what-are-the-key-differences-between-dna-and-rna-296719-1280x720.webp?cb=12456568)\n\nRNA and DNA are both nucleic acids.
\n\nThink of nucleic acid as a ladder, where the sides (backbone) of the ladder is made from a structure called sugar phosphate and the rungs of the ladder are made from four different alternating nucleobases - for `DNA it's (A, C, G and T)`, for `RNA it's (A, C, G and U)`. .
\n\nEach step in the ladder - one side and rung of ladder - is called a nucleotide.
\nThus, nucleotides are the main building blocks of DNA and RNA.\n\nThe main differences of DNA (left side of the picture) and RNA (right side of picture):\n\n* Structure : DNA has a double-stranded helical structure, like a twisted ladder, while RNA is typically single-stranded.\n\n* Sugar (side of ladder) : DNA contains deoxyribose sugar in its sugar phosphate structure, while RNA contains ribose sugar. The sugar ribose has an extra oxygen atom compared to sugar deoxyribose.\n\n* Nucleobases (rungs of ladder) : Both DNA and RNA use four chemical bases, but RNA uses uracil (U) instead of thymine (T), which is found in DNA. So, RNA has adenine (A), cytosine (C), guanine (G), and uracil (U), while DNA has adenine (A), cytosine (C), guanine (G), and thymine (T).\n\n\n","metadata":{}},{"cell_type":"markdown","source":"# 3. Competition short EDA 📈","metadata":{"id":"hn6S9MMGX9W4"}},{"cell_type":"markdown","source":"## 3.1 Imports 📚⬇","metadata":{"id":"JRRhNIRKX9W4"}},{"cell_type":"code","source":"# General\nimport glob\nimport os\nimport pandas as pd\nimport numpy as np\n# import pandas_profiling as pp\n\n# Plotting\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport plotly.express as px\nimport missingno as msno\n\n# Options\n# pd.set_option('display.max_columns', 1000)","metadata":{"id":"aWJsAbWSX9W4","execution":{"iopub.status.busy":"2023-09-15T11:21:08.291641Z","iopub.execute_input":"2023-09-15T11:21:08.292053Z","iopub.status.idle":"2023-09-15T11:21:10.722795Z","shell.execute_reply.started":"2023-09-15T11:21:08.292001Z","shell.execute_reply":"2023-09-15T11:21:10.721949Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3.2 The train and test .csv files 📁\n\nYou can have a look at the [competition description](https://www.kaggle.com/competitions/stanford-ribonanza-rna-folding/data) to understand more what kind of data and features are available to us.
Here are the main findings:\n\n* `train_data.csv` contains RNA sequences of the train set. It has 1,643,680 rows and 419 columns.\n * There are `806,573 unique` RNA sequences\n * Dataset divided equally into `two experiment types`: `2A3_MaP` and `DMS_MaP`\n * There are `23 same dataset sources` for both experiments\n * The sequence length varies can be `115, bases`, with vast majority of `177 bases (95.4%)`\n * Some `reactivity` and `reactivity_error` columns have `nulls`, and `134 columns` are `always null` especially early and late in the sequence because technically they couldn't be probed\n * About 26.6% only pass the `SN_filter` which is a filter for when `signal_to_noise>1.0` and `reads>100`\n \n* `test_sequences.csv` contains RNA sequences of the test set. It has 1,343,823 rows and 5 columns. Main findings:\n * There are `1,343,823 unique` RNA sequences\n * Of these, `335,823 sequences` are associated with `future=0` and `1,000,800 sequences` are associated with `future=1`\n * `future=0` will be used for public test LB, `future=1` will be used for private test LB\n * The sequence length for `future=0` is always `177 bases`, and for `future=1` can be `207 to 457 bases` with vast majority of `207 bases (99.21%)`\n * Test set has `no nulls`\n * Test set doesn't have a `SN_filter`, but behind the scenes only `SN_filter=1` will be used for scoring","metadata":{"id":"n6VMt5GmX9W4"}},{"cell_type":"markdown","source":"### Helper functions\n","metadata":{"id":"enLgYgMdX9W5"}},{"cell_type":"code","source":"def describe_num_col(col, percentiles = [0.01, 0.05, 0.1, 0.25, 0.5, 0.75, 0.9, 0.95, 0.99]):\n d = pd.DataFrame(col.describe(percentiles = [0.01, 0.05, 0.1, 0.25, 0.5, 0.75, 0.9, 0.95, 0.99]).round(2)).T\n display(d)\n \ndef my_fmt(x):\n return '{:.1f}%\\n({:.0f})'.format(x, total*x/100)\n","metadata":{"id":"DtwIBnIWX9W5","execution":{"iopub.status.busy":"2023-09-15T11:21:10.724470Z","iopub.execute_input":"2023-09-15T11:21:10.725572Z","iopub.status.idle":"2023-09-15T11:21:10.731452Z","shell.execute_reply.started":"2023-09-15T11:21:10.725539Z","shell.execute_reply":"2023-09-15T11:21:10.730522Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Datasets description","metadata":{"id":"BSW3FyQTX9W5"}},{"cell_type":"code","source":"train = pd.read_csv('../input/stanford-ribonanza-rna-folding/train_data.csv')\nprint(train.shape)\ntrain","metadata":{"id":"MwX1TguoX9W5","outputId":"9651ee92-1eec-4b44-f66a-aa7f042cc03c","execution":{"iopub.status.busy":"2023-09-15T11:21:10.732963Z","iopub.execute_input":"2023-09-15T11:21:10.733541Z","iopub.status.idle":"2023-09-15T11:23:03.311533Z","shell.execute_reply.started":"2023-09-15T11:21:10.733483Z","shell.execute_reply":"2023-09-15T11:23:03.310417Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"test = pd.read_csv('../input/stanford-ribonanza-rna-folding/test_sequences.csv')\nprint(test.shape)\ntest","metadata":{"id":"QGZdnMLLX9W6","outputId":"3d9e794a-e593-4bb9-f945-ab1468b56364","execution":{"iopub.status.busy":"2023-09-15T11:23:03.314220Z","iopub.execute_input":"2023-09-15T11:23:03.314523Z","iopub.status.idle":"2023-09-15T11:23:11.051421Z","shell.execute_reply.started":"2023-09-15T11:23:03.314497Z","shell.execute_reply":"2023-09-15T11:23:11.050177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Null values\n> 📌**Note**:
Train set: 134 columns always contain null. Others have null in varying percentage
\nTest set: there are no null values!","metadata":{"id":"pnnUJDGNX9W6"}},{"cell_type":"code","source":"# Percentage of nulls for columns of train set\ntrain_nulls = ((pd.DataFrame(train.isnull().sum())/len(train))*100).round(2)\nnull_cols = list(train_nulls[(train_nulls==100).values].index)\nprint(f\"There are {len(null_cols)} reactivity and reactivity_error columns with all NaN values\")\ntrain_nulls.T","metadata":{"id":"kUCz6YtOX9W7","outputId":"4012c27c-9a20-4e75-b7c3-ba81506d2d91","execution":{"iopub.status.busy":"2023-09-15T11:23:11.052606Z","iopub.execute_input":"2023-09-15T11:23:11.052956Z","iopub.status.idle":"2023-09-15T11:23:12.969832Z","shell.execute_reply.started":"2023-09-15T11:23:11.052927Z","shell.execute_reply":"2023-09-15T11:23:12.968427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Percentage of nulls for columns of test set\ntest_nulls = ((pd.DataFrame(test.isnull().sum())/len(test))*100).round(2).T\nprint(f\"There are no columns with NaN values\")\n\ntest_nulls","metadata":{"id":"AiN_U-i-X9W7","outputId":"fff8ea84-6b16-4367-82d3-f3d35f9791ca","execution":{"iopub.status.busy":"2023-09-15T11:23:12.971124Z","iopub.execute_input":"2023-09-15T11:23:12.971450Z","iopub.status.idle":"2023-09-15T11:23:13.277110Z","shell.execute_reply.started":"2023-09-15T11:23:12.971421Z","shell.execute_reply":"2023-09-15T11:23:13.275951Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Experiment types\n> 📌**Note**: Either `DMS_MaP` or `2A3_MaP` to describe the type of chemical mapping experiment that was used to generate the reactivity of the sequence","metadata":{"id":"0vLOTTI5X9W7"}},{"cell_type":"code","source":"values = train['experiment_type'] \nv_counts = values.value_counts()\ntotal = len(values)\nfig = plt.figure()\nplt.title('Experiment types in train set')\nplt.pie(v_counts, labels=v_counts.index, autopct=my_fmt, shadow=True)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-09-15T11:23:13.278698Z","iopub.execute_input":"2023-09-15T11:23:13.279070Z","iopub.status.idle":"2023-09-15T11:23:13.763449Z","shell.execute_reply.started":"2023-09-15T11:23:13.279040Z","shell.execute_reply":"2023-09-15T11:23:13.760164Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Dataset names\n> 📌**Note**: Some dataset sources are more abundant than others. The dataset sources come from the same origin for both DMS_MaP or 2A3_MaP","metadata":{"id":"H7jKLUC0X9W8"}},{"cell_type":"code","source":"train['data_source'] = train['dataset_name'].apply(lambda x: x.replace('DMS', '').replace('2A3', ''))\ntrain['data_source_experiment'] = train['dataset_name'].apply(lambda x: 'DMS' if 'DMS' in x else '2A3')","metadata":{"id":"Mptx6cX_X9W8","execution":{"iopub.status.busy":"2023-09-15T11:23:13.765204Z","iopub.execute_input":"2023-09-15T11:23:13.765645Z","iopub.status.idle":"2023-09-15T11:23:15.365569Z","shell.execute_reply.started":"2023-09-15T11:23:13.765601Z","shell.execute_reply":"2023-09-15T11:23:15.364596Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plot = sns.countplot(data = train, y = 'data_source', order = train['data_source'].value_counts(ascending=False).index, hue = 'data_source_experiment')\n# plot.set_xticklabels(plot.get_xticklabels(), rotation=90)\nplt.title('Dataset sources for both 2A3 and DMS')\nplt.show()","metadata":{"id":"EUCCi8gAX9W8","outputId":"eb522e75-a5bc-4f21-fbdd-2b6c136a13e0","execution":{"iopub.status.busy":"2023-09-15T11:23:15.366770Z","iopub.execute_input":"2023-09-15T11:23:15.367120Z","iopub.status.idle":"2023-09-15T11:23:19.088023Z","shell.execute_reply.started":"2023-09-15T11:23:15.367092Z","shell.execute_reply":"2023-09-15T11:23:19.086863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Sequence length\n> 📌**Note**:
\nFor train set: sequence length varies between `115 to 206 bases`, with vast majority of `177 bases (95.4%)`
\nFor test set:`for ","metadata":{"id":"i3SMoTd5X9W8"}},{"cell_type":"code","source":"train['seq_len'] = train['sequence'].apply(lambda x: len(x))\ntest['seq_len'] = test['sequence'].apply(lambda x: len(x))","metadata":{"id":"5igeOHDmX9W9","execution":{"iopub.status.busy":"2023-09-15T11:23:19.091210Z","iopub.execute_input":"2023-09-15T11:23:19.091637Z","iopub.status.idle":"2023-09-15T11:23:21.340137Z","shell.execute_reply.started":"2023-09-15T11:23:19.091607Z","shell.execute_reply":"2023-09-15T11:23:21.339219Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Train\nvalues = train['seq_len'].value_counts(True)\nvalues_abs = train['seq_len'].value_counts()\nv_counts = values*100\nlabels = values.index\nfig = plt.figure()\npatches, text = plt.pie(v_counts);\nlabels = ['{0} - {1:1.2f}% ({2})'.format(i,j, k) for i,j,k in zip(labels, v_counts, values_abs)]\nplt.title('Distribution of sequence length in train set')\nplt.legend(patches, labels, loc='best', \n bbox_to_anchor=(-0.1, 1.),\n fontsize=10)\nplt.show()\n\n# Future = 0\nvalues = test[test['future']==0]['seq_len'].value_counts(True)\nvalues_abs = test[test['future']==0]['seq_len'].value_counts()\nv_counts = values*100\nlabels = values.index\nfig = plt.figure()\npatches, text = plt.pie(v_counts);\nlabels = ['{0} - {1:1.2f}% ({2})'.format(i,j, k) for i,j,k in zip(labels, v_counts, values_abs)]\nplt.title('Distribution of sequence length in public test set (future=0)')\nplt.legend(patches, labels, loc='best', bbox_to_anchor=(-0.1, 1.),\n fontsize=10)\nplt.show()\n\n# Future = 1\nvalues = test[test['future']==1]['seq_len'].value_counts(True)\nvalues_abs = test[test['future']==1]['seq_len'].value_counts()\nv_counts = values*100\nlabels = values.index\nfig = plt.figure()\npatches, text = plt.pie(v_counts);\nlabels = ['{0} - {1:1.2f}% ({2})'.format(i,j, k) for i,j,k in zip(labels, v_counts, values_abs)]\nplt.title('Distribution of sequence length in private test set (future=1)')\nplt.legend(patches, labels, loc='best', bbox_to_anchor=(-0.1, 1.),\n fontsize=10)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-09-15T11:23:21.341490Z","iopub.execute_input":"2023-09-15T11:23:21.342150Z","iopub.status.idle":"2023-09-15T11:23:22.253411Z","shell.execute_reply.started":"2023-09-15T11:23:21.342100Z","shell.execute_reply":"2023-09-15T11:23:22.251866Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"reactivity_cols = [i for i in train.head(1).columns if 'reactivity' in i and 'error' not in i]\nreactivity_error_cols = [i for i in train.head(1).columns if 'reactivity' in i and 'error' in i]","metadata":{"execution":{"iopub.status.busy":"2023-09-15T11:23:22.255828Z","iopub.execute_input":"2023-09-15T11:23:22.256839Z","iopub.status.idle":"2023-09-15T11:23:22.265949Z","shell.execute_reply.started":"2023-09-15T11:23:22.256777Z","shell.execute_reply":"2023-09-15T11:23:22.264553Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### reads, signal_to_noise and SN_filter\n> ℹ **Background**:
The `reads` are like tiny pieces of a jigsaw puzzle that scientists use to understand genetic information. These reads are short snippets of the RNA (or DNA), and they are generated by special machines during a process called sequencing. Imagine you have a book, but instead of reading the whole book, you cut it into small pieces, like individual words or sentences. Each of these small pieces is a \"read.\" Scientists collect many of these reads and then use computer programs to put them back together, like solving a puzzle.\n\n> `signal_to_noise` is an indicator to how clear and reliable the measurements are. First, we look at the average (mean) of all measurement readings. Then, we look at the average (mean) of the errors or mistakes in those readings. We divide the first average (signal) by the second average (noise). This division gives you a number that shows how much your measurements (signal) compare to the errors (noise). A higher number means your measurements are more reliable because they are much bigger than the errors.\n\n>`SN_filter` - (Boolean) `0 or 1` depending on whether the sequence has `signal_to_noise>1.0` and `reads>100`.
For `scoring`, only experiments with `SN_filter = 1` will be used.","metadata":{}},{"cell_type":"code","source":"describe_num_col(train['reads'])\nsns.histplot(train[train['reads']<2500].reads)#.hist(bins=100)\nplt.title('Distribution of number of reads')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-09-15T11:23:22.268180Z","iopub.execute_input":"2023-09-15T11:23:22.269007Z","iopub.status.idle":"2023-09-15T11:23:27.388104Z","shell.execute_reply.started":"2023-09-15T11:23:22.268978Z","shell.execute_reply":"2023-09-15T11:23:27.386987Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"describe_num_col(train['signal_to_noise'])\nsns.histplot(train[train['signal_to_noise']<13].signal_to_noise)#.hist(bins=100)\nplt.title('Distribution of signal to noise ratio')\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-09-15T11:24:31.631008Z","iopub.execute_input":"2023-09-15T11:24:31.631448Z","iopub.status.idle":"2023-09-15T11:24:37.783409Z","shell.execute_reply.started":"2023-09-15T11:24:31.631420Z","shell.execute_reply":"2023-09-15T11:24:37.782299Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"values = train['SN_filter'] \nv_counts = values.value_counts()\ntotal = len(values)\nfig = plt.figure()\nplt.title('SN filter distribution in train set')\nplt.pie(v_counts, labels=v_counts.index, autopct=my_fmt, shadow=True)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-09-15T11:27:36.099186Z","iopub.execute_input":"2023-09-15T11:27:36.099615Z","iopub.status.idle":"2023-09-15T11:27:36.286261Z","shell.execute_reply.started":"2023-09-15T11:27:36.099581Z","shell.execute_reply":"2023-09-15T11:27:36.284669Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 4. Conclusion\nThis notebook provides a simple introduction to the world of RNA, that forms the basics of this competition. A short EDA was given into the .cvs files of the competition.
\nI hope it can help you get started! Feedback always appreciated :)
\nStay tuned for more...","metadata":{}}]}