{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.12","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":51294,"databundleVersionId":6923401,"sourceType":"competition"}],"dockerImageVersionId":30558,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"[WIP] **This is still a WIP**.\n\nWelcome to an EDA notebook where we will explore the data using the \ngreat [polars](https://www.pola.rs/) library.\n\nAs described by GPT-4:\n\n> ### Polars: A Fast DataFrame Library in Rust\n>\n> - **Purpose**: Polars provides data manipulation tools similar to libraries like `pandas` in Python but aims to be faster by leveraging Rust's performance.\n> - **Lazy Evaluation**: Polars features lazy evaluation, allowing operations to be not immediately executed. Instead, a computation plan is built and evaluated at once for optimized performance.\n> - **Key Modules**:\n>   - `pl`: Houses the DataFrame and Series data structures.\n>   - `lazy`: Contains tools for lazy evaluation.\n> - **Operations**: Polars offers a spectrum of operations such as filtering, aggregation, joins, group-by, and more.\n> - **Interoperability**: Facilitates interoperability with Arrow, promoting seamless data exchange with other Arrow-supporting systems.\n> - **Python Binding**: Beyond its Rust roots, Polars offers a Python binding named `polars`, allowing Python users to harness its speed within a familiar ecosystem.\n> - **Performance**: Due to Rust's efficiency and zero-cost abstractions, Polars frequently showcases performance advantages over similar DataFrame libraries, especially in multi-threaded contexts.\n>\n> _Polars presents a compelling choice for data professionals seeking swift data manipulation tools, both in the Rust landscape and through its Python bindings._\n\n\n\nThe submission part is inspired by the following [notebook](https://www.kaggle.com/code/martynoveduard/save-csv-faster-using-polars) by [slime](https://www.kaggle.com/martynoveduard).\n\nAs you have noticed above, I am also using GPT-4 to get some insights and descriptions of different \nkey concepts that I am not an expert in.","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport polars as pl\nimport numpy as np\n\nsub_path = 'sub.csv'","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2024-01-02T16:43:36.137872Z","iopub.execute_input":"2024-01-02T16:43:36.138341Z","iopub.status.idle":"2024-01-02T16:43:36.849080Z","shell.execute_reply.started":"2024-01-02T16:43:36.138304Z","shell.execute_reply":"2024-01-02T16:43:36.847611Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"num_preds = 100_000_000\nids = np.arange(num_preds, dtype=np.int64)\npreds_dms = np.random.rand(num_preds)\npreds_2a3 = np.random.rand(num_preds)\nids.shape, preds_dms.shape, preds_2a3.shape","metadata":{"execution":{"iopub.status.busy":"2024-01-02T16:43:36.851882Z","iopub.execute_input":"2024-01-02T16:43:36.852945Z","iopub.status.idle":"2024-01-02T16:43:39.695543Z","shell.execute_reply.started":"2024-01-02T16:43:36.852887Z","shell.execute_reply":"2024-01-02T16:43:39.694095Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\nschema={k:pl.Float32 for k in ['id', 'reactivity_DMS_MaP', 'reactivity_2A3_MaP']}\nschema['id'] = pl.Int64\n\ndf = pl.DataFrame(\n    data=[ids, preds_dms, preds_2a3],\n    schema=schema\n)\ndf.write_csv(sub_path, float_precision=4) \nprint(df.shape)\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2024-01-02T16:43:39.697083Z","iopub.execute_input":"2024-01-02T16:43:39.697411Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\ndf = pd.DataFrame(\n    {\n        'id': ids,\n        'reativity_DMS_MaP' : preds_dms,\n        'reactivity_2A3_MaP' : preds_2a3\n    }\n)\ndf.to_csv(sub_path, float_format='%.4f', index=False)\nprint(df.shape)\ndf.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Quick EDA","metadata":{}},{"cell_type":"markdown","source":"Let's explore the datasets.","metadata":{}},{"cell_type":"code","source":"path = \"/kaggle/input/stanford-ribonanza-rna-folding/train_data.csv\"","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = pl.read_csv(path)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.columns","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.head().transpose()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.select(\"sequence_id\").unique()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The sequence column is an important one. It contains\nthe RNA structure as a combination of the \n4 ","metadata":{}},{"cell_type":"markdown","source":"# Key concetps","metadata":{"execution":{"iopub.status.busy":"2023-10-20T08:11:52.577161Z","iopub.execute_input":"2023-10-20T08:11:52.579266Z","iopub.status.idle":"2023-10-20T08:11:52.583650Z","shell.execute_reply.started":"2023-10-20T08:11:52.579221Z","shell.execute_reply":"2023-10-20T08:11:52.582923Z"}}},{"cell_type":"markdown","source":"Here is a list of some key concepts to get started with\nthis competition. A lot of definitions come from Wikipedia pages.","metadata":{}},{"cell_type":"markdown","source":"* **RNA**: [Ribonucleic acid](https://ecn.wikipedia.org/wiki/RNA) is a polymeric molecule that is essential for most biological functions by either performing the function or forming a template for it.\n* Chemical reactivity: in chemistry [reactivity](https://en.wikipedia.org/wiki/Reactivity_(chemistry)) measures whether or not a substance reacts and how fast it reacts.\n* Synthetic data: ","metadata":{"execution":{"iopub.status.busy":"2023-10-20T08:12:07.233687Z","iopub.execute_input":"2023-10-20T08:12:07.234516Z","iopub.status.idle":"2023-10-20T08:12:07.241297Z","shell.execute_reply.started":"2023-10-20T08:12:07.234478Z","shell.execute_reply":"2023-10-20T08:12:07.239990Z"}}},{"cell_type":"markdown","source":"In addition, here is how RNA is explained by GPT-4:\n    \n> RNA, or Ribonucleic Acid, is like a messenger in our cells. Imagine a kitchen. DNA is like the recipe book that stays in the drawer, while RNA is like a photocopy of a specific recipe that you can carry to the counter to cook. It helps the cell make proteins using instructions from DNA. So, RNA carries the \"cooking\" instructions from the \"recipe book\" (DNA) to the cell's \"kitchen\" to make things like muscles, enzymes, and more!\n\n\nand here is the main differences between RNA and DNA:\n\n\n\n1. **Structure**: DNA is double-stranded, like a twisted ladder, while RNA is single-stranded.\n2. **Sugar**: The sugar in the backbone of DNA is deoxyribose; in RNA, it's ribose.\n3. **Bases**: Both have four bases each, but they differ slightly. DNA has Adenine (A), Thymine (T), Cytosine (C), and Guanine (G). RNA has Adenine (A), Uracil (U), Cytosine (C), and Guanine (G). Notice RNA has U instead of T.\n4. **Function**: DNA stores genetic information, like a blueprint. RNA acts to carry out these instructions, helping to produce proteins and perform other cellular activities.\n5. **Location**: Typically, DNA is found in the cell's nucleus, while RNA can be found throughout the cell.\n\nImagine DNA as the master blueprint stored safely away, and RNA as the working copy used to build and maintain the house (your body).\n\n","metadata":{}},{"cell_type":"markdown","source":"Adenine (A): A purine base with a double ring structure. Its complementary base is Uracil in RNA.\n\nUracil (U): A pyrimidine base with a single ring structure. It pairs with Adenine in RNA.\n\nCytosine (C): Another pyrimidine base with a single ring structure. Its complementary base is Guanine.\n\nGuanine (G): A purine base with a double ring structure. It pairs with Cytosine.\n\nIn summary:\n\nPurines (A and G) have a double ring structure.\nPyrimidines (U and C) have a single ring structure.\nA pairs with U, and C pairs with G in RNA.","metadata":{}},{"cell_type":"markdown","source":"More from GPT-4 about how it would tackle the problem:\n\n> **Predicting Chemical Reactivity in RNA Structure**\n>\n> 1. **Secondary and Tertiary Structures**: RNA molecules can fold into various structures, including hairpin loops, bulges, and internal loops. The accessibility of a base or a region in RNA often determines its reactivity. Bases buried inside a double-stranded region will generally be less reactive than those in single-stranded or loop regions.\n>\n> 2. **pH**: RNA contains functional groups that can be protonated or deprotonated. The state of these groups can influence reactivity. For example, the 2'-hydroxyl group on ribose is more nucleophilic at higher pH.\n>\n> 3. **Metal Ions**: Metal ions, especially divalent cations like Mg²⁺, play a critical role in stabilizing RNA structures and can influence reactivity. They can participate directly in catalysis, bridge phosphate groups, or stabilize negative charges.\n>\n> 4. **Base Stacking**: In RNA, bases can stack on top of each other, leading to stabilization and reduced reactivity. Bases that are not involved in stacking interactions or base pairing are more exposed and, hence, more reactive.\n>\n> 5. **Functional Groups**: The reactivity of specific bases can be attributed to their functional groups. For instance, the amino group of cytosine or the carbonyl group of uracil can engage in specific reactions.\n>\n> 6. **Base Modifications**: Naturally occurring RNA can have modified bases, and these modifications can influence reactivity. For example, pseudouridine or methylated bases can have different reactivity profiles compared to their unmodified counterparts.\n>\n> 7. **Surrounding Environment**: The solvent, ionic strength, and presence of other molecules (like proteins) can influence RNA structure and, consequently, its reactivity.\n>\n> Experimental techniques like **SHAPE** or **DMS probing** and computational methods, such as molecular dynamics simulations, can provide insights into RNA's reactivity based on its structure.\n\n\n","metadata":{}},{"cell_type":"markdown","source":"There are two methods to measure the reactivity:\n    \n* 2A3_MaP\n* DMS_MaP\n\n\nThere are 206 reactivity columns.\n\nFor the evaluation part, we need to filter using `SN_filter = 1` for both measurement methods.","metadata":{}},{"cell_type":"markdown","source":"# From sequence to 3D structure","metadata":{}},{"cell_type":"code","source":"with pl.Config(fmt_str_lengths=1000):\n    print(df.head(1).select(\"sequence\"))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_sequence = df.head(1).select(\"sequence\")","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sample_sequence","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Model ideas","metadata":{}},{"cell_type":"markdown","source":"Here are some models to get started:\n    \n\n* \n* SRRF\n\n","metadata":{}}]}