{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"
\n\n
\"Competition
\n\n
\n\n

\n \n LET'S \n LEARN \n TOGETHER !\n


\n\n EXPLAIN IT \n LIKE \n I'M\n 5 \n   (ELI5)\n
+
\n EXPLORATORY\n DATA\n ANALYSIS \n   (EDA)\n
+
\n BASELINE\n
\n

\n\n

CREATED BY: DARIEN SCHETTLER

\n\n
\n\n🎨 Font colour choices were inspired by the Barbie movie. (#FD3792 & #1bd19d) 🎨
If the colours cause issues when viewing, please let me know and I can edit it to make it more accessible. Thanks!\n
\n\n---\n\n
\n\n
\n ⚠️   NOTE   ⚠️

This notebook's primary purpose is to educate through exploration of this competition's data, the background information, and other notebooks and discussion posts that help in building a complete understanding.

Kaggle user's Notebooks and Discussions that I include will be listed below!
Please upvote the original authors/contributors.

\n \n\n\n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n \n \n\n
         Contributor                   Notebook          
Hiram CoriaRibonanza: RNA Visualization
iafossRNA starter [0.186 LB]
placeholder 3notebook 3
\n \n
\n \n
\n\n\n\n
\n 🛑   WARNING:

THIS IS A WORK IN PROGRESS
\n
\n\n\n
\n 👏   IF YOU FORK THIS OR FIND THIS HELPFUL   👏

PLEASE UPVOTE!

This was a lot of work for me and it makes me feel appreciated when others like my work. 😅\n
\n\n\n","metadata":{}},{"cell_type":"markdown","source":"

\n\n

\n\n

TABLE OF CONTENTS

\n\n---\n\n

1    INTRODUCTION & JUSTIFICATION

\n\n---\n\n

2    HOST PROVIDED COMPETITION INFORMATION

\n\n---\n\n

3    MY UNDERSTANDING OF THE COMPETITION

\n\n---\n\n\n

4    BACKGROUND INFORMATION GLOSSARY

\n\n---\n\n

5    IMPORTS

\n\n---\n\n

6    SETUP & HELPER FUNCTIONS

\n\n---\n\n

7    DATA EXPLORATION

\n\n---\n\n

8    TBD

\n\n---\n\n

9    NEXT STEPS

\n\n---\n\n

10    BASELINE

\n\n---","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n

1  INTRODUCTION & JUSTIFICATION    ⤒

\n\n
\n\nThis notebook aims to provide an educational walkthrough following my personal journey of understanding. This will hopefully bring me from completely ignorant of the background and science for this competition to a place where I feel ready to tackle the challenge in earnest!
\n\nI hope you can join me and we can learn together!\n\n> NOTE: Unlike my previous notebooks that were primarily Exploratory Data Analyses (EDAs) or Educational (ELI5) or Baselines... this notebook will merge those two educational pursuits into a single unified resource.","metadata":{}},{"cell_type":"markdown","source":"
\n\n

1.1 WHAT IS THIS?

\n\n---\n\nThis notebook will attempt to do two (maybe three) things:\n1. \"Explain It Like I'm Five\". \n * Where \"IT\" refers to everything you need to know to understand this competition (data, background information, etc)\n * Note that while I aim to honour the five-year old part of this statement, it mostly used because it is a catchy term, and I will just be explaining things as simply as possible\n2. \"Exploratory Data Analysis\"\n * I will perform a traditional Exploratory Data Analysis (EDA), which is a critical early step in tackling any data science problem!\n * I see this as a secondary and complimentary piece to the ELI5 piece, and as such, this part of the notebook comes AFTER the ELI5 part.\n3. Baseline (optional)\n * If I think I have time and if it's appropriate, I will finish this notebook out with a final section that performs a baseline submission resulting in a valid leaderboard submission.\n \nREMINDER: If I have cited other authors, please don't forget to upvote them. I'll reiterate their works below but it can be found above too!\n\n\n\n \n \n \n \n\n\n \n \n \n \n \n \n \n \n \n \n \n \n\n
         Contributor                   Notebook          
Hiram CoriaRibonanza: RNA Visualization
iafossRNA starter [0.186 LB]
placeholder 3notebook 3
\n ","metadata":{}},{"cell_type":"markdown","source":"
\n\n

1.2 WHY IS THIS?

\n\n---\n\nI wanted to share my learning journey with others as this type of work has resonated well in the past. \n* Note that I make these type of notebooks primarily for myself, however, I am finding it brings value when I share them with others. As such I plan to continue this format in the future.\n* By upvoting/sharing this work you are supporting me and encouraging me to spend more time on similar endeavours in the future!","metadata":{}},{"cell_type":"markdown","source":"
\n\n

1.3 WHO IS THIS FOR?

\n\n---\n\nThe primary purpose of this notebook is to educate MYSELF, however, my review/learning might be beneficial to others. Hence this notebook.\n\nPlease understand that not everyone needs nor wants to read a notebook like this. That being said the following are some good reasons why you may wish to read on:\n* If you want to learn with me!\n* If you want to learn more about the background information (ELI5)\n* If you want to learn more about the competition specifics (ELI5)\n* If you want to learn more about the competition data (EDA)\n* If you have a good understanding of the competition specifics but want to doublecheck against my understanding or reinforce your own understanding (ELI5)\n* If you have a good understanding of the background information but want to doublecheck against my understanding or reinforce your own understanding (ELI5)\n* If you have a good understanding of the competition data but want to doublecheck against my understanding or reinforce your own understanding (EDA)\n* If you want to see some baseline approaches implemented (baseline)","metadata":{}},{"cell_type":"markdown","source":"
\n\n

1.4 HOW WILL THIS WORK?

\n\n---\n\n1. I'm going to assemble some markdown cells (like this one) at the beginning of the notebook to go over the competition details\n2. General background information mostly expressed as a glossary with lots of visuals (this will include a table to track my learning).\n3. I will perform a reasonably in-depth Exploratory Data Analysis\n4. I will create one or more baseline submissions (including)","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n

2  HOST PROVIDED COMPETITION INFORMATION    ⤒

\n\n
\n\n**NOTE***\n* The first sections are dedicated to exploring the basic resources provided by the host.\n* The final sections are dedicated to my understanding of the competition","metadata":{}},{"cell_type":"markdown","source":"
\n\n

2.1 PRIMARY TASK DESCRIPTION

\n\n---\n\nThe goal of this competition is to predict the reactivity (DMS, and 2A3) at every position for a particular RNA sequence.\n* To do this we need to create a model that understands the structure of RNA and as such can infer the reactivity at a particular nucleotide\n* The model will leverage millions of RNA sequences along with the labelled reactivity data (and additional supplemental data).\n* The model will be required to output two float values (***regression***) for each **`id`** in the test set.\n * **reactivity_DMS_MaP** – A value from **0-1** measuring reactivity to the **DMS** chemical, with 1 being the most reactive and 0 being the least reactive. *[A float]*\n * **reactivity_2A3_MaP** – A value from **0-1** measuring reactivity to the **2A3** chemical, with 1 being the most reactive and 0 being the least reactive. *[A float]*\n \nNote: MaP refers to Mutational Profiling not Mean average Precision. Mutational Profiling experiments are used in the determination of the reactivity at a particular nucleotide location.","metadata":{}},{"cell_type":"markdown","source":"
\n\n

2.2 BASIC HOST PROVIDED BACKGROUND INFO AND CONTEXT

\n\n---\n\n
This section is kept intentionally brief and only reflects the host provided information. The concepts that help better define the context will be explored in the GLOSSARY BACKGROUND INFORMATION section of this notebook. Some terms are linked to their glossary entries for convenience.

\n\n
DATA DESCRIPTION\n\nWithout being able to understand how RNA molecules fold, we are missing a deeper understanding of how nature works, how life began, and how we can design better medicines and biological approaches to grand problems like climate change.\n\nIn this competition, your goal is to predict how an RNA sequence reacts to two specific chemical modifiers: DMS and 2A3. You can obtain this reactivity data through a MutAtional Profiling (MaP) experiment which is then analyzed via high-throughput sequencing. Positions in the RNA sequence that remain unmodified (protected from chemical modification) are more likely involved in forming base pairs or other structural elements.\n\n
CONTEXT DESCRIPTION\n\nRiboNucleic Acid (RNA) is essential for most biological functions. A better understanding of how to manipulate RNA could help usher in an age of programmable medicine, including first cures for pancreatic cancer and Alzheimer’s disease as well as much-needed antibiotics and new biotechnology approaches for climate change. But first, researchers must better understand each RNA molecule's structure, an ideal problem for data science.\n\nRecent efforts to predict RNA structure have run into a number of challenges:\n1. Paucity (small or insufficient quantities or amounts) of training data (lack of data)\n2. Lack of intellectual and computational power (lack of compute)\n3. Difficulties in rigorously splitting training and test data. (lack of cross validation rigor)\n\nCan a Kaggle competition close these gaps?\n\nTowards sourcing a large and diverse collection of RNA molecules for data acquisition, the host Das laboratory (Stanford) brings together scientists and gamers to solve puzzles and invent medicine in the Eterna project. The Eterna community has previously unlocked new scientific principles, designed thermodynamically-optimized riboswitches, and revealed RNA degradation patterns for improving the shelf life of mRNA vaccines, which formed the basis of the Kaggle OpenVaccine challenge. Now the Eterna project is creating diverse sequences that are predicted to form complex structures.\n\nThe data for this new Ribonanza Competition are experimental measurements of the chemical reactivity at each position of an RNA molecule. These data are exquisitely sensitive to the structure – or multiple structures – that each RNA forms in the test tube. An algorithm that could perfectly predict these chemical reactivities would need to have an implicit ‘understanding’ of RNA structure to do so. Such an oracle could be then utilized to predictively model structures of novel RNA molecules. As in the OpenVaccine competition, the majority of the private leaderboard data will be collected in parallel with your prediction efforts -- no one will know the answers until the competition closes!\n\n
IMPACT DECLARATION\n\nTo reiterate the potential impact of your participation, an accurate model that solves the RNA structure prediction problem could be a game changer for medical researchers who are trying to identify unique RNA-based drug targets in the many bacterial, viral, neurological, and cancer genes that remain undruggable at the protein level. In addition, accurate RNA structure prediction is needed to predictively design RNA-based medicines such as mRNA vaccines and CRISPR gene therapeutics that promise to treat nearly all human disease. More than its medical implications, RNA molecules underlie and can even dominate core biological processes for all of life, including the very first forms of life on Earth and the plants and marine organisms that fix most of the carbon on our planet now. A full understanding of life requires a full, predictive understanding of RNA.\n\nIn addition to being made available as open source code for the entire science and medicine communities, winners’ innovations will be featured in a keynote talk by Rhiju Das at the flagship conference in machine learning in structure biology at NeurIPS in December 2023.\n\nIt’s possible that every biologist and biotechnologist in the world could one day use your algorithmic solution.\n \n","metadata":{}},{"cell_type":"markdown","source":"
\n\n

2.3 EVALUATION INFORMATION

\n\n---\n\n
GENERAL EVALUATION INFORMATION\n\nSubmissions are scored using Mean Absolute Error (MAE):\n\n$$\n\\text{MAE} = \\frac{1}{N}\\sum_{i=1}^{N} \\left| y_{i} - \\widehat{y}_{i} \\right|\n$$\n\n**WHERE:** \n* $N$ is the number of scored ground truth values\n* $y$ and $\\widehat{y}$ are the actual and predicted values, respectively. \n* The $y$ values will be clipped between 0 and 1 before calculating MAE\n * $y_{i}=max(min(y^{RAW}_{i},1.0),0.0)$\n * where $y^{RAW}$ are the raw data values\n\n\nEach RNA sequence has some number of nucleotides (positions). For each position there will be two ground truth values, corresponding to reactivity determined from two kinds of chemical mapping experiments, **DMS_MaP and 2A3_MaP**.\n\n
SUBMISSION FILE INFORMATION\n\nFor each **sequence** in the test set, you must predict **target reactivities** for each sequence position **id**, **two per row**. \n* i.e. If the sum of the lengths of all 1,343,824 sequences in the test set is 269,796,671, then you should make 2 × 269,796,670 = 539,593,340 predictions.\n\nThe submission file should contain a header and have the following format:\n\n```\nid,reactivity_DMS_MaP,reactivity_2A3_MaP\n0,0.1,0.3\n1,0.3,0.2\n2,0.5,0.4\netc.\n```\n\n
IS THIS A CODE COMPETITION ??\n\nNO\n\n> This competition provides the entirety of the test set. You will be submitting your final predictions for all the test data and some portion of that will be used for public LB evaluation while the rest will be held out for private LB evaluation.","metadata":{}},{"cell_type":"markdown","source":"
\n\n

2.4 DATASET INFORMATION

\n\n---\n\n
HIGH LEVEL DATA SUMMARY\n\nAt the beginning of the competition, Stanford scientists have data on 1,118,513 RNA sequences of lengths ranging from 115 to 206.\n\nWe have split out 311,935 of these 1,118,513 sequences for a public test set to allow for continuous evaluation through the competition, on the Public Leaderboard. This set has been additionally filtered to ensure high signal-to-noise and read coverage. The remaining 806,578 sequences for which we have data are in train_data.csv. We note that 37,828 of the test sequences, derived from the RFAM database, are identical or near-identical to the train set; for these cases the leaderboard test set contains higher signal-to-noise measurements than the train_data and serve as a test of model ability to 'denoise' chemical mapping data.\n\nOur final and most important scoring (the Private Leaderboard) involves 1,031,888 sequences. Within this set, the majority (1,008,000 RNAs) will be experimentally synthesized and profiled after the Kaggle competition begins, to help ensure rigor. Furthermore, these test sequences will have lengths ranging from 207 to 457 bases -- we intentionally chose a different length distribution compared to the train data and Public Leaderboard, to help test the generality of your models. As those profiles are collected, they will again be filtered for acceptable signal to noise before computing your final Private Leaderboard scores.\n\n
DATA FILE DESCRIPTIONS\n\ntrain_data.csv - the training data\ntest_sequences.csv - the test set sequences, without any columns associated with the ground truth.\nsample_submission.csv - a sample submission file in the correct format\n\n[train_data|test_sequences].csv columns:\n* **`id` (integer)**\n * Integer (0,1,…) that identifies each sequence position in the sample submission.\n* **`id_min, id_max` (integer)**\n * Minimum and maximum id values for each test sequence.\n* **`sequence_id` (string)**\n * An arbitrary identifier like `8cdfeef00` for each sequence.\n* **`sequence` (string)**\n * Describes the RNA sequence, a combination of A, G, U, and C for each sample. \n * Should be **115 to 457 characters long**.\n* **`experiment_type` (string)**\n * Either `DMS_MaP` or `2A3_MaP` to describe the type of chemical mapping experiment that was used to generate each profile. \n * References: DMS, 2A3.\n* **`dataset_name` (string)**\n * Name of high throughput sequencing dataset from which the reactivity profile was extracted.\n* **`reads` (integer)**\n * Number of reads in the high throughput sequencing experiment that were assigned to the RNA sequence, and whose mutations were tabulated to compile the reactivity profile.\n* **`signal_to_noise` (float)**\n * Signal/noise value for the profile\n * Defined as: **`mean(measurement value over probed nts) / mean(statistical error in measurement value over probed nts)`**.\n* **`SN_filter` (Boolean)**\n * 0 **or** 1 depending on whether the profile has `signal_to_noise > 1.0` and `reads > 100`. \n * For evaluation, only sequences whose DMS_MaP and 2A3_MaP profiles both pass this filter will be used to score submissions.\n* **`reactivity_0001, reactivity_0002,…` (float)**\n * An array of floating-point numbers of the train data, should have the same length as the RNA sequence, which defines the reactivity profile for the RNA. \n * Notes on null values\n * For sequences shorter than the maximum RNA length, positions that go beyond the sequence length have null. \n * Several positions early and late in the sequence also cannot be probed due to technical reasons, and their reactivity values are null.\n* **`reactivity_error_0001, reactivity_error_0002,…` (float)**\n * An array of floating-point numbers, should have the same length as the corresponding `reactivity_*` columns, calculated errors in experimental values obtained in reactivity derived from counting statistics in the high-throughput sequencing experiment.\n* **`reactivity_DMS_MaP, reactivity_2A3_MaP` (float)**\n * Sample submission values.\n* **`future` (Boolean)**\n * Sequences whose data will be collected after the start of the competition (but before final scoring) are labeled as 1.\n\n
\n\nsample_submission.csv description\n* An example submission with the correct columns and properly ordered event IDs. \n\n
\n\nAdditional Folders\n* **`sequence_libraries`**\n * all 2,150,401 sequences, with titles, in FASTA format, grouped into the actual collections that were synthesized together. \n * Note: some files list DNA not RNA sequences (T instead of U).\n* **`supplementary_silico_predictions`**\n * CSV files of predicted RNA secondary structures in dot-parentheses notations. \n * Note: not all packages were run for all sequences due to computational expense of prediction packages that are able to predict pseudoknots (non-nested pairings).\n* **`eterna_openknot_metadata`**\n * Contains TSV files with the following columns:\n * id\n * name (author)\n * title (author-provided design title)\n * body (description)\n * sequence for Eterna OpenKnot sequences\n * Note: some files have additional tabs in title or description that may affect reading.\n* **`Ribonanza_bpp_files`**\n * TXT files of base pair probabilities from LinearPartition-EternaFold package for train and test sequences. \n * Indexed by `sequence_id`. \n * Note: this package simulates RNA secondary structure ensembles without pseudoknots or other tertiary structure features.","metadata":{}},{"cell_type":"markdown","source":"
\n\n

2.5 SUPPLEMENTARY INFORMATION

\n\n---\n\n\n","metadata":{}},{"cell_type":"markdown","source":"
\n\n

2.6 COMPETITION HOST INFORMATION

\n\n---\n\n\n\n\n\n\n\n\n\n \n\n\n\n
EteRNA teaches about RNA and the different shapes it takes. You are are given control over RNA bonds (ACGU), which directly affect the RNA molecule's shape. Changing proteins in a chain impacts the natural shape that the chain would take with the goal of matching the shape of the puzzle.
The DAS Lab seeks an agile and predictive understanding of how RNAs code for information processing and replication in living systems. They are creating new computational and chemical tools to enable the precise modeling and design of these RNAs.
","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n

3  MY UNDERSTANDING OF THE COMPETITION    ⤒

","metadata":{}},{"cell_type":"markdown","source":"
\n\n

3.1 BASICS - ELI5+

\n\n---\n\nChemical reactivity in the context of DMS (dimethyl sulfate) and 2A3 (2-aminopyridine-3-carboxylic acid imidazolide) mapping experiments provides a way to probe the structure of RNA molecules. Here's how it works:\n\nIn an RNA molecule, some regions are more \"exposed\" and some are more \"hidden\" due to the way the molecule folds into its unique 3D structure. The exposed regions are more likely to react with chemicals like DMS or 2A3, while the hidden regions are generally less reactive.\n\nDMS and 2A3 are chemicals that can bind to certain positions on the RNA molecule. The reactivity measured using these chemicals serves as an indirect indicator of the molecule's structure. If a certain position on the RNA is highly reactive to DMS or 2A3, it's likely that this position is more exposed in the structure. Conversely, if it's less reactive, that position is probably more hidden or protected within the folded structure.\n\nBy measuring the chemical reactivity of each position within the RNA molecule using DMS and 2A3, scientists can gather data that is sensitive to the RNA's 3D structure. A machine learning model that can accurately predict these reactivity values essentially \"understands\" the likely 3D structure of the RNA molecule, which is why this reactivity data is so important for the competition.","metadata":{}},{"cell_type":"markdown","source":"
\n\n

3.2 MY INTERNAL Q&A

\n\n---\n\nThis will be a list where there is a question/thought followed by my learnings about that thing...\n\n
\n\n---\n\nWhat are the similarities between DMS and 2A3?\n\n\n---\n\nWhat are the differences between DMS and 2A3?\n\n\n---\n\nWhat is 'SHAPE'?\n\n\n \n---\n\nCan DMS be used as a reagent for SHAPE?\n\n\n---\n\nWhat are 'reads' and why are they important (a column in our dataset)\n\n\n---\n\nWhat is a pseudoknot and why do I care?\n\n\n","metadata":{}},{"cell_type":"markdown","source":"TABLE TBD\n\n","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n

4  BACKGROUND INFORMATION GLOSSARY    ⤒

\n\nWIP ... STAY TUNED\n

\n\n---\n\n
\n\nDMS\n\n\n2A3\n\n\nRNA STUFF SOON!\n\n\n
","metadata":{}},{"cell_type":"markdown","source":"

\n\n

4.1 DIMETHYL SULFATE (DMS)

\n\n---\n\n\n

\n\nPOINTFORM DEFINITION\n* WIP – More coming\n\n**REFERENCES**\n* TBD\n\n

\nELI5 COMPETITION DEFINITION\n* TBD\n\n

\n\nEXPLAIN IT WITH PICTURES\n* TBD\n\n
","metadata":{}},{"cell_type":"markdown","source":"

\n\n

4.2 2-AMINOPYRIDINE-3-CARBOXYLIC ACID IMIDAZOLIDE (2A3)

\n\n---\n\n\n

\n\nPOINTFORM DEFINITION\n* WIP – More coming\n\n**REFERENCES**\n* TBD\n\n

\nELI5 COMPETITION DEFINITION\n* TBD\n\n

\n\nEXPLAIN IT WITH PICTURES\n* TBD","metadata":{}},{"cell_type":"markdown","source":"
\n\n\n\n

5  IMPORTS    ⤒

\n\n
COMPETITION SPECIFIC LIBRARIES\n\n* **`arnie`**\n * a helpful utility library that simplifies interacting with various secondary structure prediction packages.\n * **Arnie** needs at least one secondary structure predictor. Below are some examples:\n * **EternaFold**: Specialized algorithm developed for RNA secondary structure prediction. EternaFold aims to integrate human intuition and computational power to predict RNA secondary structures more accurately.\n * **ViennaRNA**: Provides various algorithms and utilities for predicting RNA secondary structures.\n * **Mfold**: Another popular tool used for RNA secondary structure prediction.\n * **RNAstructure**: A software package that predicts RNA secondary structure.\n * **NUPACK**: Focuses on the analysis and prediction of nucleic acid secondary structures.\n * **RNAfold**: Part of the ViennaRNA package, specifically used for RNA secondary structure prediction.\n* **`eternafold`**\n * Eternafold is a leading prediction package that was trained using sequences collected via the citizen science game Eterna. \n * In fact, Eterna players provided many of the sequences in the data for this competition.\n* **`forgi` (stands for Finding of RNA Graph Information.)**\n * Provides various utilities for representing, manipulating, and visualizing RNA structures.\n * In forgi, RNA secondary structures are represented as graphs, making it easier to perform operations that would be complicated in sequence or matrix representations. \n * These graphs can be straightforward, representing just the base pairs, or more complex, representing multi-loop or pseudoknot structures.\n* **`viennarna`**\n * ViennaRNA is a package that includes a collection of algorithms and utilities for RNA secondary structure prediction.\n * Part of the package, **RNAfold**, is specifically used for RNA secondary structure prediction, and there are other tools for things like comparison and visualization of RNA structures. \n * This is installed via bioconda...\n * **`bioconda`**\n * A distribution of bioinformatics software for the Conda package manager.\n * **Bioconda** makes it easy to install bioinformatics software and their dependencies, including many of the RNA secondary structure prediction packages mentioned above.\n","metadata":{}},{"cell_type":"code","source":"print(\"\\n... PIP INSTALLS STARTING ...\\n\")\n# Install Arnie\n!pip install -q --upgrade arnie \n\n# Install Eternafold; not available via conda-forge quite yet, so we install from the pipeline build artifacts directly\n# !conda install eternafold\n!wget -q https://artprodeus21.artifacts.visualstudio.com/A910fa339-c7c2-46e8-a579-7ea247548706/84710dde-1620-425b-80d0-4cf5baca359d/_apis/artifact/cGlwZWxpbmVhcnRpZmFjdDovL2NvbmRhLWZvcmdlL3Byb2plY3RJZC84NDcxMGRkZS0xNjIwLTQyNWItODBkMC00Y2Y1YmFjYTM1OWQvYnVpbGRJZC83NzI5MTAvYXJ0aWZhY3ROYW1lL2NvbmRhX3BrZ3NfbGludXg1/content?format=zip\n!unzip -q content\\?format\\=zip\n!conda install -y -q conda_pkgs_linux/eternafold-1.3.1-h00ab1b0_0.conda\n\n# Ordinarily, the EternaFold conda package will automatically set necessary environment variables, \n# but Kaggle's conda install works a little differently. \n#\n# Let's set them manually here using %env.\n%env ETERNAFOLD_PATH=/opt/conda/bin/eternafold-bin\n%env ETERNAFOLD_PARAMETERS=/opt/conda/lib/eternafold-lib/parameters/EternaFoldParams.v1\n\n# Install forgi - a library for working with and visualizing RNA structures\n!pip install -q --upgrade forgi\n\n# Install viennarna\n!conda install -y -q -c bioconda viennarna\n\n# Install draw_rna - a library from DaS for visualizing RNA structures (works with Arnie)\n!pip install -q git+https://github.com/DasLab/draw_rna.git\n\nprint(\"\\n... PIP INSTALLS COMPLETE ...\\n\")\n\nprint(\"\\n... IMPORTS STARTING ...\\n\")\nprint(\"\\n\\tVERSION INFORMATION\")\n\n# Competition Specific Imports\nfrom arnie.mfe import mfe # mfe(sequence, package=\"eternafold\") - or the package of your choice ()\nfrom arnie.bpps import bpps # bpps(sequence, package=\"eternafold\") - or the package of your choice\nimport forgi.visual.mplotlib as fvm\nimport forgi.graph.bulge_graph as fgb\nimport draw_rna\n\n# Machine Learning and Data Science Imports (basics)\nimport tensorflow as tf; print(f\"\\t\\t– TENSORFLOW VERSION: {tf.__version__}\");\nimport pandas as pd; pd.options.mode.chained_assignment = None; pd.set_option('display.max_columns', None);\nimport numpy as np; print(f\"\\t\\t– NUMPY VERSION: {np.__version__}\");\nimport sklearn; print(f\"\\t\\t– SKLEARN VERSION: {sklearn.__version__}\");\n\n# Built-In Imports (mostly don't worry about these)\nfrom kaggle_datasets import KaggleDatasets\nfrom collections import Counter\nfrom datetime import datetime\nfrom zipfile import ZipFile\nfrom glob import glob\nimport Levenshtein\nimport subprocess\nimport warnings\nimport requests\nimport hashlib\nimport imageio\nimport IPython\nimport sklearn\nimport urllib\nimport zipfile\nimport pickle\nimport random\nimport shutil\nimport string\nimport json\nimport math\nimport time\nimport gzip\nimport ast\nimport sys\nimport io\nimport os\nimport gc\nimport re\n\n# Visualization Imports (overkill)\nfrom matplotlib.colors import ListedColormap, LinearSegmentedColormap\nfrom matplotlib.patches import Rectangle\nimport matplotlib.patches as patches\nimport plotly.graph_objects as go\nimport matplotlib.pyplot as plt\nfrom tqdm.notebook import tqdm; tqdm.pandas();\nimport plotly.express as px\nimport seaborn as sns\nfrom PIL import Image, ImageEnhance; Image.MAX_IMAGE_PIXELS = 5_000_000_000;\nimport matplotlib; print(f\"\\t\\t– MATPLOTLIB VERSION: {matplotlib.__version__}\");\nfrom matplotlib import animation, rc; rc('animation', html='jshtml')\nimport plotly\nimport PIL\nimport cv2\n\nimport plotly.io as pio\nprint(pio.renderers)\n\ndef check_nvidia_gpu():\n try:\n subprocess.check_output('nvidia-smi')\n print('\\n... NVIDIA GPU DETECTED ...\\n')\n return True\n except Exception: # this command not being found can raise quite a few different errors depending on the configuration\n print('\\n... NO NVIDIA GPU DETECTED ...\\n')\n return False\n\ndef seed_it_all(seed=7):\n \"\"\" Attempt to be Reproducible \"\"\"\n os.environ['PYTHONHASHSEED'] = str(seed)\n random.seed(seed)\n np.random.seed(seed)\n tf.random.set_seed(seed)\n\nseed_it_all()\n\n# Check if GPU attached and imports RAPIDS if so.\nHAS_GPU = check_nvidia_gpu()\nif HAS_GPU: import cudf, cuml, cupy\n\nprint(\"\\n\\n... IMPORTS COMPLETE ...\\n\")","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-09-12T13:54:14.716757Z","iopub.execute_input":"2023-09-12T13:54:14.717235Z","iopub.status.idle":"2023-09-12T13:57:24.999474Z","shell.execute_reply.started":"2023-09-12T13:54:14.717183Z","shell.execute_reply":"2023-09-12T13:57:24.998453Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n

6  SETUP AND HELPER FUNCTIONS    ⤒

","metadata":{}},{"cell_type":"markdown","source":"
\n\n

6.1 HELPER FUNCTIONS

\n\n---\n\n
","metadata":{}},{"cell_type":"code","source":"def flatten_l_o_l(nested_list):\n \"\"\" Flatten a list of lists \"\"\"\n return [item for sublist in nested_list for item in sublist]\n\ndef print_ln(symbol=\"-\", line_len=110):\n \"\"\" Prints a line for asthetics \"\"\"\n print(symbol*line_len)\n \ndef downcast_df(df):\n \"\"\" Simple function that takes any column represented as 64 bits and casts to 32 \"\"\"\n for col in df.columns:\n df[col] = df[col].astype(df[col].dtype.name.replace('64', '32'))\n return df\n\ndef get_helper_data_files(_train_df):\n \"\"\" \n This will downcast the training data (use all of it not a subsample).\n We then split the downcast data based on their respective experimental methods (2A3-SHAPE and DMS).\n It will then split out the reactivity and reactivity_error columns as numpy arrays\n It will then save everything (csv for the dataframes and npz for the reactivity values)\n \n This data is available as a public dataset \n --> https://www.kaggle.com/datasets/dschettler8845/sr-rna-folding-helpers\n \"\"\"\n _train_df = downcast_df(_train_df)\n \n train_dms_df = _train_df[_train_df.experiment_type==\"DMS_MaP\"].reset_index(drop=True)\n train_2a3_df = _train_df[_train_df.experiment_type==\"2A3_MaP\"].reset_index(drop=True)\n \n train_dms_reactivity_np = train_dms_df[[_c for _c in train_dms_df.columns if _c.startswith(\"reactivity_0\")]].to_numpy()\n train_dms_reactivity_error_np = train_dms_df[[_c for _c in train_dms_df.columns if _c.startswith(\"reactivity_error_0\")]].to_numpy()\n np.save(\"train_dms_reactivity_fp32\", train_dms_reactivity_np.astype(np.float32))\n np.save(\"train_dms_reactivity_error_fp32\", train_dms_reactivity_error_np.astype(np.float32))\n\n train_2a3_reactivity_np = train_2a3_df[[_c for _c in train_2a3_df.columns if _c.startswith(\"reactivity_0\")]].to_numpy()\n train_2a3_reactivity_error_np = train_2a3_df[[_c for _c in train_2a3_df.columns if _c.startswith(\"reactivity_error_0\")]].to_numpy()\n np.save(\"train_2a3_reactivity_fp32\", train_2a3_reactivity_np.astype(np.float32))\n np.save(\"train_2a3_reactivity_error_fp32\", train_2a3_reactivity_error_np.astype(np.float32))\n\n train_dms_df[[_c for _c in train_dms_df.columns if not _c.startswith(\"reactivity\")]].to_csv(\"train_dms_wo_reactivity.csv\", index=False)\n train_2a3_df[[_c for _c in train_2a3_df.columns if not _c.startswith(\"reactivity\")]].to_csv(\"train_2a3_wo_reactivity.csv\", index=False)","metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"execution":{"iopub.status.busy":"2023-09-12T13:57:25.001704Z","iopub.execute_input":"2023-09-12T13:57:25.002700Z","iopub.status.idle":"2023-09-12T13:57:25.014075Z","shell.execute_reply.started":"2023-09-12T13:57:25.002656Z","shell.execute_reply":"2023-09-12T13:57:25.013053Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

6.2 LOAD THE DATA

\n\n---\n\nThe full data is huge and we will be loading the metadata separately later anyway... so let's just load a portion (10K)","metadata":{}},{"cell_type":"code","source":"# Define the path to theflags root data directory\nUSE_CUDF = False\nDATA_DIR = \"/kaggle/input/stanford-ribonanza-rna-folding\"\n\n# DEBUG to only open a fraction\nDEBUG = True\n\n# Define the paths to the batches of train and test data respectively\nTRAIN_CSV_PATH = os.path.join(DATA_DIR, \"train_data.csv\")\nTEST_CSV_PATH = os.path.join(DATA_DIR, \"test_sequences.csv\")\nSS_CSV_PATH = os.path.join(DATA_DIR, \"sample_submission.csv\")\n\n# These columns are NaN for the entire train dataset\ntrain_nan_cols = ['reactivity_0001', 'reactivity_0002', 'reactivity_0003', 'reactivity_0004', 'reactivity_0005', 'reactivity_0006', 'reactivity_0007', 'reactivity_0008', 'reactivity_0009', 'reactivity_0010', 'reactivity_0011', 'reactivity_0012', 'reactivity_0013', 'reactivity_0014', 'reactivity_0015', 'reactivity_0016', 'reactivity_0017', 'reactivity_0018', 'reactivity_0019', 'reactivity_0020', 'reactivity_0021', 'reactivity_0022', 'reactivity_0023', 'reactivity_0024', 'reactivity_0025', 'reactivity_0026', 'reactivity_0166', 'reactivity_0167', 'reactivity_0168', 'reactivity_0169', 'reactivity_0170', 'reactivity_error_0001', 'reactivity_error_0002', 'reactivity_error_0003', 'reactivity_error_0004', 'reactivity_error_0005', 'reactivity_error_0006', 'reactivity_error_0007', 'reactivity_error_0008', 'reactivity_error_0009', 'reactivity_error_0010', 'reactivity_error_0011', 'reactivity_error_0012', 'reactivity_error_0013', 'reactivity_error_0014', 'reactivity_error_0015', 'reactivity_error_0016', 'reactivity_error_0017', 'reactivity_error_0018', 'reactivity_error_0019', 'reactivity_error_0020', 'reactivity_error_0021', 'reactivity_error_0022', 'reactivity_error_0023', 'reactivity_error_0024', 'reactivity_error_0025', 'reactivity_error_0026', 'reactivity_error_0166', 'reactivity_error_0167', 'reactivity_error_0168', 'reactivity_error_0169', 'reactivity_error_0170', 'reactivity_0171', 'reactivity_0172', 'reactivity_0173', 'reactivity_0174', 'reactivity_0175', 'reactivity_0176', 'reactivity_0177', 'reactivity_error_0171', 'reactivity_error_0172', 'reactivity_error_0173', 'reactivity_error_0174', 'reactivity_error_0175', 'reactivity_error_0176', 'reactivity_error_0177', 'reactivity_0178', 'reactivity_0179', 'reactivity_0180', 'reactivity_0181', 'reactivity_0182', 'reactivity_0183', 'reactivity_0184', 'reactivity_0185', 'reactivity_0186', 'reactivity_0187', 'reactivity_0188', 'reactivity_0189', 'reactivity_0190', 'reactivity_0191', 'reactivity_0192', 'reactivity_0193', 'reactivity_0194', 'reactivity_0195', 'reactivity_0196', 'reactivity_0197', 'reactivity_0198', 'reactivity_0199', 'reactivity_0200', 'reactivity_0201', 'reactivity_0202', 'reactivity_0203', 'reactivity_0204', 'reactivity_0205', 'reactivity_0206', 'reactivity_error_0178', 'reactivity_error_0179', 'reactivity_error_0180', 'reactivity_error_0181', 'reactivity_error_0182', 'reactivity_error_0183', 'reactivity_error_0184', 'reactivity_error_0185', 'reactivity_error_0186', 'reactivity_error_0187', 'reactivity_error_0188', 'reactivity_error_0189', 'reactivity_error_0190', 'reactivity_error_0191', 'reactivity_error_0192', 'reactivity_error_0193', 'reactivity_error_0194', 'reactivity_error_0195', 'reactivity_error_0196', 'reactivity_error_0197', 'reactivity_error_0198', 'reactivity_error_0199', 'reactivity_error_0200', 'reactivity_error_0201', 'reactivity_error_0202', 'reactivity_error_0203', 'reactivity_error_0204', 'reactivity_error_0205', 'reactivity_error_0206']\n\nprint(\"\\n... BASIC DATA SETUP STARTING ...\\n\")\nprint(\"\\n\\n... LOAD TRAIN DATAFRAME FROM CSV FILE ...\\n\")\nif (USE_CUDF and HAS_GPU):\n raise NotImplementedError()\n # train_df = cudf.read_csv(TRAIN_CSV_PATH)\nelse:\n train_df = pd.read_csv(TRAIN_CSV_PATH, nrows=10_000 if DEBUG else None)\ntrain_df = downcast_df(train_df)\n\nprint(\"\\n\\n... LOAD TEST META DATAFRAME FROM CSV FILE ...\\n\")\nif (USE_CUDF and HAS_GPU):\n raise NotImplementedError()\n # test_df = cudf.read_csv(TEST_CSV_PATH)\nelse:\n test_df = pd.read_csv(TEST_CSV_PATH, nrows=10_000 if DEBUG else None)\ndisplay(test_df)\n\nprint(\"\\n\\n... LOAD SAMPLE SUBMISSION DATAFRAME FROM CSV FILE ...\\n\")\nss_df = pd.read_csv(SS_CSV_PATH, nrows=10 if DEBUG else None)\ndisplay(ss_df)\n\n# Cleanup just in case....\ngc.collect(); gc.collect(); gc.collect();\n\nprint(\"\\n\\n\\n... BASIC DATA SETUP FINISHED ...\\n\\n\")","metadata":{"execution":{"iopub.status.busy":"2023-09-12T13:57:25.015629Z","iopub.execute_input":"2023-09-12T13:57:25.016037Z","iopub.status.idle":"2023-09-12T13:57:26.569734Z","shell.execute_reply.started":"2023-09-12T13:57:25.015982Z","shell.execute_reply":"2023-09-12T13:57:26.568928Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

6.3 SUPERFICIAL EXAMINATION OF THE DATA

\n\n---\n\nWe will use **`.describe`** and **`.info`** to quickly see information about NaN frequency, datatype, etc.\n\n
TRAIN DATA OBSERVATIONS\n* Many of the columns appear to be completely composed of NaN values\n * This is confirmed when viewing the full data as well (although to a lesser extent) and I have added the columns that are always missing as a list of strings above title `train_nan_cols` (see full list below)\n* While negative reactivity values are possible they are abnormal. All **predicted** reactivity should be between 0-1.\n* Reactivity values by column appear to have different means and standard deviations... we will examine in more detail later.\n* There are non missing values for **`sequence`**, **`experiment_type`**, **`dataset_name`**, **`reads`**, **`signal_to_noise`** or **`SN_filter`**\n* **`experiment_type`** is either **`2A3_MaP`** or **`DMS_MaP`** for the respective approaches. (I think we should approach these separately and make two models)\n* **`dataset_name`** looks to not be missing any data and contains approximately 40-50 unique strings. (to be confirmed below)\n \n```python\ntrain_nan_cols = [\n 'reactivity_0001', 'reactivity_0002', 'reactivity_0003', 'reactivity_0004', 'reactivity_0005', 'reactivity_0006', 'reactivity_0007',\n 'reactivity_0008', 'reactivity_0009', 'reactivity_0010', 'reactivity_0011', 'reactivity_0012', 'reactivity_0013', 'reactivity_0014',\n 'reactivity_0015', 'reactivity_0016', 'reactivity_0017', 'reactivity_0018', 'reactivity_0019', 'reactivity_0020', 'reactivity_0021',\n 'reactivity_0022', 'reactivity_0023', 'reactivity_0024', 'reactivity_0025', 'reactivity_0026', 'reactivity_0166', 'reactivity_0167',\n 'reactivity_0168', 'reactivity_0169', 'reactivity_0170', 'reactivity_0171', 'reactivity_0172', 'reactivity_0173', 'reactivity_0174',\n 'reactivity_0175', 'reactivity_0176', 'reactivity_0177', 'reactivity_0178', 'reactivity_0179', 'reactivity_0180', 'reactivity_0181',\n 'reactivity_0182', 'reactivity_0183', 'reactivity_0184', 'reactivity_0185', 'reactivity_0186', 'reactivity_0187', 'reactivity_0188',\n 'reactivity_0189', 'reactivity_0190', 'reactivity_0191', 'reactivity_0192', 'reactivity_0193', 'reactivity_0194', 'reactivity_0195',\n 'reactivity_0196', 'reactivity_0197', 'reactivity_0198', 'reactivity_0199', 'reactivity_0200', 'reactivity_0201', 'reactivity_0202',\n 'reactivity_0203', 'reactivity_0204', 'reactivity_0205', 'reactivity_0206', 'reactivity_error_0001', 'reactivity_error_0002',\n 'reactivity_error_0003', 'reactivity_error_0004', 'reactivity_error_0005', 'reactivity_error_0006', 'reactivity_error_0007',\n 'reactivity_error_0008', 'reactivity_error_0009', 'reactivity_error_0010', 'reactivity_error_0011', 'reactivity_error_0012',\n 'reactivity_error_0013', 'reactivity_error_0014', 'reactivity_error_0015', 'reactivity_error_0016', 'reactivity_error_0017',\n 'reactivity_error_0018', 'reactivity_error_0019', 'reactivity_error_0020', 'reactivity_error_0021', 'reactivity_error_0022',\n 'reactivity_error_0023', 'reactivity_error_0024', 'reactivity_error_0025', 'reactivity_error_0026', 'reactivity_error_0166',\n 'reactivity_error_0167', 'reactivity_error_0168', 'reactivity_error_0169', 'reactivity_error_0170', 'reactivity_error_0171',\n 'reactivity_error_0172', 'reactivity_error_0173', 'reactivity_error_0174', 'reactivity_error_0175', 'reactivity_error_0176',\n 'reactivity_error_0177', 'reactivity_error_0178', 'reactivity_error_0179', 'reactivity_error_0180', 'reactivity_error_0181',\n 'reactivity_error_0182', 'reactivity_error_0183', 'reactivity_error_0184', 'reactivity_error_0185', 'reactivity_error_0186',\n 'reactivity_error_0187', 'reactivity_error_0188', 'reactivity_error_0189', 'reactivity_error_0190', 'reactivity_error_0191',\n 'reactivity_error_0192', 'reactivity_error_0193', 'reactivity_error_0194', 'reactivity_error_0195', 'reactivity_error_0196',\n 'reactivity_error_0197', 'reactivity_error_0198', 'reactivity_error_0199', 'reactivity_error_0200', 'reactivity_error_0201',\n 'reactivity_error_0202', 'reactivity_error_0203', 'reactivity_error_0204', 'reactivity_error_0205', 'reactivity_error_0206'\n]\n```\n\n
\n\nTEST DATA OBSERVATIONS\n* No missing data\n* Nothing else of note at this level","metadata":{}},{"cell_type":"code","source":"print(\"\\n... TRAIN DATA DESCRIPTION ...\\n\")\ndisplay(train_df.describe())\n\nprint(\"\\n... TRAIN DATA COLUMN INFO ...\\n\")\ntrain_df.info()\n\nprint(\"\\n... TEST DATA DESCRIPTION ...\\n\")\ndisplay(test_df.describe())\n\nprint(\"\\n... TEST DATA COLUMN INFO ...\\n\")\ntest_df.info()","metadata":{"execution":{"iopub.status.busy":"2023-09-12T13:57:26.571818Z","iopub.execute_input":"2023-09-12T13:57:26.572751Z","iopub.status.idle":"2023-09-12T13:57:27.844816Z","shell.execute_reply.started":"2023-09-12T13:57:26.572721Z","shell.execute_reply":"2023-09-12T13:57:27.843985Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

6.4 LOAD SUPPLEMENTARY METADATA PART IN FULL

\n\n---\n\nWe previously split the metadata (non-reactivity) part of the dataframe into it's own dataset. Let's load this so we can examine the columns one at a time in their entirety before moving on to examining the reactivity data.","metadata":{}},{"cell_type":"code","source":"HELPER_DATASET_DIR = \"/kaggle/input/sr-rna-folding-helpers\"\ntrain_meta_dms = downcast_df(pd.read_csv(os.path.join(HELPER_DATASET_DIR, \"train_dms_wo_reactivity.csv\"))).copy()\ntrain_meta_2a3 = downcast_df(pd.read_csv(os.path.join(HELPER_DATASET_DIR, \"train_2a3_wo_reactivity.csv\"))).copy()\n\nprint(\"\\n... TRAIN METADATA FOR DMS ...\\n\")\ndisplay(train_meta_dms.head())\n\nprint(\"\\n... TRAIN METADATA FOR 2A3 ...\\n\")\ndisplay(train_meta_2a3.head())","metadata":{"execution":{"iopub.status.busy":"2023-09-12T13:57:27.845946Z","iopub.execute_input":"2023-09-12T13:57:27.846341Z","iopub.status.idle":"2023-09-12T13:57:37.678518Z","shell.execute_reply.started":"2023-09-12T13:57:27.846315Z","shell.execute_reply":"2023-09-12T13:57:37.677789Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

6.5 ADD HELPFUL COLUMNS

\n\n---\n\nLet's add some helpful columnar information to support our current datasets\n\n
NEW COLUMNS\n1. **`seq_len`** - Sequence Length\n * Useful for ordering/filtering based on sequence length and comparing distributions","metadata":{}},{"cell_type":"code","source":"# 1. Add the sequence length column\ntrain_df['seq_len'] = train_df['sequence'].apply(len)\ntrain_meta_dms['seq_len'] = train_meta_dms['sequence'].apply(len)\ntrain_meta_2a3['seq_len'] = train_meta_2a3['sequence'].apply(len)\ntest_df['seq_len'] = test_df['sequence'].apply(len)\n\n\n##### See the updated dataframes ...\nprint(\"\\n... TRAIN DATAFRAME - UPDATED ...\\n\")\ndisplay(train_df.head(3))\n\nprint(\"\\n... TEST DATAFRAME - UPDATED ...\\n\")\ndisplay(test_df.head(3))\n\nprint(\"\\n... TRAIN METADATA FOR DMS - UPDATED ...\\n\")\ndisplay(train_meta_dms.head(3))\n\nprint(\"\\n... TRAIN METADATA FOR 2A3 - UPDATED ...\\n\")\ndisplay(train_meta_2a3.head(3))","metadata":{"execution":{"iopub.status.busy":"2023-09-12T13:57:37.679840Z","iopub.execute_input":"2023-09-12T13:57:37.680332Z","iopub.status.idle":"2023-09-12T13:57:38.417730Z","shell.execute_reply.started":"2023-09-12T13:57:37.680301Z","shell.execute_reply":"2023-09-12T13:57:38.416744Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n\n\n

7  DATA EXPLORATION    ⤒

\n\nWe will do columnar exploration of the respective DMS and 2A3 metadata dataframes prior to investigating the reactivity values themselves. We will also use this opportunity to plot some examples (when examining sequence).\n\nNote we grab a few helpful constants here before we start too...","metadata":{}},{"cell_type":"code","source":"N_2A3_TRAIN = len(train_meta_2a3)\nN_DMS_TRAIN = len(train_meta_dms)\nN_TRAIN = N_2A3_TRAIN+N_DMS_TRAIN\nN_TEST = len(test_df) # not really though... cause DEBUG...\n\nprint(\"\\n... Training Data Statistics ...\")\nprint(f\"\\tNumber of 2A3 Training Examples : {N_2A3_TRAIN:,}\")\nprint(f\"\\tNumber of DMS Training Examples : {N_DMS_TRAIN:,}\")\nprint(f\"\\tTotal Number of Training Examples : {N_TRAIN:,}\")\n\nprint(\"\\n... Test Data Statistics (Will be wrong if debug is on) ...\")\nprint(f\"\\tNumber of Test Examples : {N_TEST:,}\")","metadata":{"execution":{"iopub.status.busy":"2023-09-12T13:57:38.419080Z","iopub.execute_input":"2023-09-12T13:57:38.419398Z","iopub.status.idle":"2023-09-12T13:57:38.426271Z","shell.execute_reply.started":"2023-09-12T13:57:38.419371Z","shell.execute_reply":"2023-09-12T13:57:38.425261Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

7.1 SEQUENCE ID

\n\n---\n\n
HOST DESCRIPTION\n\n* **`sequence_id`** - (string) An arbitrary identifier like `8cdfeef00` for each sequence.\n\n
MY OBSERVATIONS\n* I initially thought each sequence would be unique in our dataset. This is untrue.\n * There are over 20,000 repeats per testing type (40,000 total) in the training dataset\n * This corresponds to the following distribution of repeats by sequence (each testing type is identical so 2x for total training data)\n * 2729 sequences are repeated 5 times for each testing method (2x) --> impacts 13645 rows per test (27290 total)\n * 2173 sequences are repeated 3 times for each testing method (2x) --> impacts 6519 rows per test (13038 total)\n * 5 sequences are repeated 2 times for each testing method (2x) --> impacts 10 rows per test (20 total)","metadata":{}},{"cell_type":"code","source":"repeated_dms_sequence_ids = {k:v for k,v in train_meta_dms.sequence_id.value_counts().items() if v!=1}\nrepeated_2a3_sequence_ids = {k:v for k,v in train_meta_2a3.sequence_id.value_counts().items() if v!=1}\n\nprint(\"\\n... REPEATED DMS SEQUENCE IDS ...\")\nprint(f\"\\tNumber of repeated ids : {len(repeated_dms_sequence_ids):,}\")\nprint(f\"\\tTotal repeated sequences : {sum(list(repeated_dms_sequence_ids.values())):,}\")\n\nprint(\"\\n... REPEATED 2A3 SEQUENCE IDS ...\")\nprint(f\"\\t- Number of repeated ids : {len(repeated_2a3_sequence_ids):,}\")\nprint(f\"\\t- Total repeated sequences : {sum(list(repeated_2a3_sequence_ids.values())):,}\")\n\nprint(\"\\n\\n... EXAMPLE WITH SEQUENCE ID '1728e0d67d7c' BELOW ...\\n\")\nprint(\"\\n... TRAIN METADATA FOR DMS ('1728e0d67d7c')...\")\ndisplay(train_meta_dms[train_meta_dms.sequence_id==\"1728e0d67d7c\"])\n\nprint(\"\\n... TRAIN METADATA FOR 2A3 ('1728e0d67d7c')...\")\ndisplay(train_meta_2a3[train_meta_2a3.sequence_id==\"1728e0d67d7c\"])","metadata":{"execution":{"iopub.status.busy":"2023-09-12T13:57:38.427429Z","iopub.execute_input":"2023-09-12T13:57:38.427722Z","iopub.status.idle":"2023-09-12T13:57:40.619587Z","shell.execute_reply.started":"2023-09-12T13:57:38.427697Z","shell.execute_reply":"2023-09-12T13:57:40.618589Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

7.2 SEQUENCE

\n\n---\n\n
HOST DESCRIPTION\n\n* **`sequence`** - (string) \n * Describes the RNA sequence, a combination of A, G, U, and C for each sample. \n * Should be 115 to 457 characters long.\n\n
MY OBSERVATIONS\n* There will be the same number of repeats per test as observed in the sequence ID.\n* Just some viz below... I am taking a break before investigating this further as I want to write a separate viz notebook first...\n * Much of the initial inspiration is taken from This Visualization Starter Notebook.\n * I plan to create my own detailed visualization notebook to teach myself more about all of this and will incorporate that at a later time...","metadata":{}},{"cell_type":"code","source":"def plot_rna(sequence, structure=None, visualizer=\"forgi\", viz_kwargs=None):\n from ipynb.draw import draw_struct\n if structure is None:\n structure=mfe(sequence, package=\"eternafold\")\n \n if viz_kwargs is None:\n viz_kwargs = {}\n \n if visualizer==\"das\":\n draw_struct(sequence, structure, **viz_kwargs)\n \n elif visualizer==\"forgi\":\n plt.figure(figsize=(12,10))\n bg = fgb.BulgeGraph.from_fasta_text(f'>rna1\\n{structure}\\n{sequence}')[0]\n fvm.plot_rna(bg, lighten=0.25, text_kwargs={\"fontweight\":None})\n plt.show()\n \ndemo_sequence = train_meta_2a3[train_meta_2a3.sequence_id==\"8cdfeef009ea\"].sequence.values[0]\n\n# Normal forgi version\nplot_rna(demo_sequence, visualizer=\"forgi\")\n\n# Normal DaS version\nplot_rna(demo_sequence, visualizer=\"das\")\n\n# Line version\nplot_rna(demo_sequence, visualizer=\"das\", viz_kwargs={\"line\":True})\n\n# Large version\nplot_rna(demo_sequence, visualizer=\"das\", viz_kwargs={\"large_mode\":True})","metadata":{"execution":{"iopub.status.busy":"2023-09-12T14:03:03.015282Z","iopub.execute_input":"2023-09-12T14:03:03.015764Z","iopub.status.idle":"2023-09-12T14:03:10.719335Z","shell.execute_reply.started":"2023-09-12T14:03:03.015725Z","shell.execute_reply":"2023-09-12T14:03:10.718108Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"bp_matrix = bpps(demo_sequence, package=\"eternafold\")\nplt.figure(figsize=(10,10))\nplt.title('Base-Pairing Probability NxN Array', fontweight=\"bold\")\nplt.xlabel(\"Sequence Position\", fontweight=\"bold\")\nplt.ylabel(\"Sequence Position\", fontweight=\"bold\")\nplt.imshow(bp_matrix, origin='lower', cmap='magma')\nplt.colorbar(shrink=0.8)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-09-12T14:03:32.049315Z","iopub.execute_input":"2023-09-12T14:03:32.050352Z","iopub.status.idle":"2023-09-12T14:03:32.555962Z","shell.execute_reply.started":"2023-09-12T14:03:32.050312Z","shell.execute_reply":"2023-09-12T14:03:32.555212Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"p_unp_vec = 1 - np.sum(bp_matrix, axis=0)\n\nplt.figure(figsize=(13,5))\nplt.plot(p_unp_vec)\nplt.title('Probability that Nucleotide is Unpaired', fontweight=\"bold\")\nplt.xlabel(\"Sequence Position\", fontweight=\"bold\")\nplt.ylabel('p(unpaired)', fontweight=\"bold\")\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2023-09-12T14:03:35.655863Z","iopub.execute_input":"2023-09-12T14:03:35.656988Z","iopub.status.idle":"2023-09-12T14:03:35.934194Z","shell.execute_reply.started":"2023-09-12T14:03:35.656946Z","shell.execute_reply":"2023-09-12T14:03:35.933330Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

7.3 SEQUENCE LENGTH

\n\n---\n\n
HOST DESCRIPTION\n\nSequences should be 115 to 457 characters long.\n\n
MY OBSERVATIONS\n* There are only 5 different sequence lengths in the training dataset and the vast majority are of length 177 (over 95%)\n * Sequence length of **177** occurs **784,177** per method (DMS, 2A3)\n * Sequence length of **170** occurs **15,000** per method (DMS, 2A3)\n * Sequence length of **115** occurs **13,645** per method (DMS, 2A3)\n * Sequence length of **155** occurs **6,519** per method (DMS, 2A3)\n * Sequence length of **206** occurs **2,499** per method (DMS, 2A3)","metadata":{}},{"cell_type":"code","source":"print(train_meta_2a3.seq_len.value_counts())\nfig = px.pie(train_meta_2a3, \"seq_len\")\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2023-09-12T14:03:41.927026Z","iopub.execute_input":"2023-09-12T14:03:41.927710Z","iopub.status.idle":"2023-09-12T14:03:43.303542Z","shell.execute_reply.started":"2023-09-12T14:03:41.927674Z","shell.execute_reply":"2023-09-12T14:03:43.300501Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n\n

7.4 DATASET NAME

\n\n---\n\n\n
HOST DESCRIPTION\n\n**`dataset_name`** (string):\n * name of high throughput sequencing dataset from which the reactivity profile was extracted\n\n
MY OBSERVATIONS\n* Unequal distribution across datasets (to be expected)\n* More investigation into how the dataset informs on other properties will be required","metadata":{}},{"cell_type":"code","source":"fig = px.bar(train_meta_2a3.dataset_name.value_counts().to_frame().reset_index(), \"dataset_name\", \"count\", color=\"dataset_name\", height=1000, title=\"Distribution of Samples by Dataset\")\nfig.update_layout(showlegend=False)\nfig.update_xaxes(tickangle=45, showgrid=True, title='')\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2023-09-12T14:17:38.006213Z","iopub.execute_input":"2023-09-12T14:17:38.006641Z","iopub.status.idle":"2023-09-12T14:17:39.010207Z","shell.execute_reply.started":"2023-09-12T14:17:38.006609Z","shell.execute_reply":"2023-09-12T14:17:39.009063Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
WIP
More work coming over the next week ...
releasing as a placeholder for now
WIP
","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}],"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}}