{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.11","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":87793,"databundleVersionId":11553390,"sourceType":"competition"}],"dockerImageVersionId":31012,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 📘 Stanford RNA 3D Folding: Exploratory Analysis & Baseline","metadata":{}},{"cell_type":"markdown","source":"### 🧬 About the Competition\n\nThe goal of this competition is to predict the 3D structure of RNA molecules based on sequence data using a simplified coarse-grained format. Predicting accurate 3D folding is crucial for understanding biological function and designing novel RNA therapeutics.\n\n-----","metadata":{"execution":{"iopub.status.busy":"2025-04-14T11:56:08.697527Z","iopub.execute_input":"2025-04-14T11:56:08.697859Z","iopub.status.idle":"2025-04-14T11:56:08.703901Z","shell.execute_reply.started":"2025-04-14T11:56:08.697833Z","shell.execute_reply":"2025-04-14T11:56:08.7027Z"}}},{"cell_type":"markdown","source":"### 🎯 Objective\n\n    We’ll begin by:\n\n\t•\tExploring the dataset and understanding its structure\n\t•\tVisualizing initial patterns\n\t•\tBuilding a foundation for future modeling\n  \n------","metadata":{}},{"cell_type":"markdown","source":"### 📁 Dataset Overview\n\nThe dataset contains:\n\n\t•\ttrain_sequences.csv: Training RNA sequences with metadata\n\t•\ttrain_labels.csv: Ground truth labels for 3D positions\n\t•\ttest_sequences.csv: RNA sequences for evaluation\n\t•\tsample_submission.csv: Format for predictions\n___\n    ","metadata":{}},{"cell_type":"markdown","source":"### 🔍 Importing Libraries & Loading Data","metadata":{}},{"cell_type":"code","source":"# 📦 Importing required libraries\nimport pandas as pd\nimport os\n\n# 📁 Define the path to the competition data\ndata_path = '/kaggle/input/stanford-rna-3d-folding/'\n\n# 📊 Load the datasets\ntrain_sequences = pd.read_csv(os.path.join(data_path, 'train_sequences.csv'))\ntrain_labels = pd.read_csv(os.path.join(data_path, 'train_labels.csv'))\ntest_sequences = pd.read_csv(os.path.join(data_path, 'test_sequences.csv'))\nsample_submission = pd.read_csv(os.path.join(data_path, 'sample_submission.csv'))\n\n# 👁️ Preview the data\nprint(\"Train Sequences:\")\nprint(train_sequences.head())\n\nprint(\"\\nTrain Labels:\")\nprint(train_labels.head())\n\nprint(\"\\nTest Sequences:\")\nprint(test_sequences.head())\n\nprint(\"\\nSample Submission:\")\nprint(sample_submission.head())","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:33:44.037148Z","iopub.execute_input":"2025-04-17T09:33:44.037417Z","iopub.status.idle":"2025-04-17T09:33:46.252348Z","shell.execute_reply.started":"2025-04-17T09:33:44.037395Z","shell.execute_reply":"2025-04-17T09:33:46.251501Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"___\n# 🧪 **Dataset Preview**","metadata":{}},{"cell_type":"markdown","source":"### 📄 Train Sequences\n\nEach row in train_sequences.csv contains metadata and the RNA sequence itself:","metadata":{}},{"cell_type":"code","source":"# Preview the train_sequences dataset\ntrain_sequences.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:34:06.407794Z","iopub.execute_input":"2025-04-17T09:34:06.408088Z","iopub.status.idle":"2025-04-17T09:34:06.426648Z","shell.execute_reply.started":"2025-04-17T09:34:06.408065Z","shell.execute_reply":"2025-04-17T09:34:06.425688Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Columns**:\n\n*target_id*: Unique ID for each RNA structure\n\n*sequence*: The RNA nucleotide sequence (A, U, C, G)\n\n*temporal_cutoff*: Date associated with the structure\n\n*description*: Biological/structural description\n\n*all_sequences*: FASTA-style sequence with additional metadata","metadata":{}},{"cell_type":"markdown","source":"____\n### 🎯 Train Labels\n\ntrain_labels.csv provides 3D atomic coordinates (in Angstroms) for the first 5 non-hydrogen atoms in each nucleotide.","metadata":{}},{"cell_type":"code","source":"# Preview the train_labels dataset\ntrain_labels.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:34:11.028373Z","iopub.execute_input":"2025-04-17T09:34:11.029342Z","iopub.status.idle":"2025-04-17T09:34:11.040102Z","shell.execute_reply.started":"2025-04-17T09:34:11.029308Z","shell.execute_reply":"2025-04-17T09:34:11.039058Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Columns**:\n\t\n*ID*: Combination of target ID and nucleotide index\n\n*resname*: RNA base (G, A, C, or U)\n\n*resid*: Nucleotide index\n\n*x_1, y_1, z_1*: Coordinates for Atom 1\n\n*... up to x_5, y_5, z_5*: Coordinates for Atom 5","metadata":{}},{"cell_type":"markdown","source":"-----\n\n### 🧪 Test Sequences\n\nThis is similar to the train sequences but without labels (the values to predict):","metadata":{}},{"cell_type":"code","source":"# Preview the test_sequences dataset\ntest_sequences.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:34:20.508949Z","iopub.execute_input":"2025-04-17T09:34:20.509635Z","iopub.status.idle":"2025-04-17T09:34:20.519341Z","shell.execute_reply.started":"2025-04-17T09:34:20.509609Z","shell.execute_reply":"2025-04-17T09:34:20.518497Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---\n### 📝 Sample Submission\n\nThis file provides the format for predictions. Initially, all coordinates are zero:","metadata":{}},{"cell_type":"code","source":"# Preview the sample submission\nsample_submission.head()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:34:24.376855Z","iopub.execute_input":"2025-04-17T09:34:24.377128Z","iopub.status.idle":"2025-04-17T09:34:24.396354Z","shell.execute_reply.started":"2025-04-17T09:34:24.377108Z","shell.execute_reply":"2025-04-17T09:34:24.395225Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"___\n\n# 📊 Exploratory Data Analysis (EDA)\n\n\nBefore we dive into modeling, it’s essential to understand the structure and distribution of the data. Let’s begin by analyzing the RNA sequence lengths and base compositions.\n","metadata":{}},{"cell_type":"markdown","source":"### 🔢 Sequence Length Distribution\n\nWe’ll plot the length of RNA sequences in both training and test sets.","metadata":{"execution":{"iopub.status.busy":"2025-04-14T12:16:26.597048Z","iopub.execute_input":"2025-04-14T12:16:26.597365Z","iopub.status.idle":"2025-04-14T12:16:26.603713Z","shell.execute_reply.started":"2025-04-14T12:16:26.597342Z","shell.execute_reply":"2025-04-14T12:16:26.6025Z"}}},{"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\", category=FutureWarning)\n\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n# Calculate sequence lengths\ntrain_sequences['sequence_length'] = train_sequences['sequence'].apply(len)\ntest_sequences['sequence_length'] = test_sequences['sequence'].apply(len)\n\n# Clean plot without KDE or emojis in title\nplt.figure(figsize=(12, 5))\nsns.histplot(train_sequences['sequence_length'], bins=30, color=\"skyblue\", label=\"Train\")\nsns.histplot(test_sequences['sequence_length'], bins=30, color=\"salmon\", label=\"Test\")\nplt.title(\"Distribution of RNA Sequence Lengths\")\nplt.xlabel(\"Sequence Length\")\nplt.ylabel(\"Frequency\")\nplt.legend()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:34:30.348885Z","iopub.execute_input":"2025-04-17T09:34:30.349146Z","iopub.status.idle":"2025-04-17T09:34:31.725401Z","shell.execute_reply.started":"2025-04-17T09:34:30.349127Z","shell.execute_reply":"2025-04-17T09:34:31.724471Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**🧠 Analysis: RNA Sequence Length Distribution**\n\nFrom the histogram above, we observe the following:\n\n    📉 Most RNA sequences are relatively short, with the vast majority having fewer than 200 nucleotides.\n    \n\t📊 There’s a heavy right-skew, indicating a long tail of longer RNA sequences, some exceeding 4000 nucleotides.\n    \n\t🧪 The distribution for train and test sequences is similar, suggesting that the model will likely generalize well if trained properly, even on longer sequences.\n    \n\t⚠️ The presence of long sequences may pose memory or computational challenges in modeling, especially with neural networks — it might be worth investigating length-based batching or truncation strategies.\n----\n    ","metadata":{}},{"cell_type":"markdown","source":"### 🧬 Nucleotide Composition \n\nLet’s analyze the overall frequency of the four RNA bases (A, U, C, G) in the training sequences.","metadata":{}},{"cell_type":"code","source":"from collections import Counter\n\n# Flatten all sequences into a single string\nall_bases = \"\".join(train_sequences[\"sequence\"].values)\nbase_counts = Counter(all_bases)\n\n# Plot\nplt.figure(figsize=(6, 4))\nsns.barplot(x=list(base_counts.keys()), y=list(base_counts.values()), palette=\"muted\")\nplt.title(\"Nucleotide Base Composition in Training Data\")\nplt.xlabel(\"Base\")\nplt.ylabel(\"Count\")\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:34:37.108926Z","iopub.execute_input":"2025-04-17T09:34:37.109367Z","iopub.status.idle":"2025-04-17T09:34:37.299131Z","shell.execute_reply.started":"2025-04-17T09:34:37.109343Z","shell.execute_reply":"2025-04-17T09:34:37.298085Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**🔍 Analysis: Nucleotide Base Composition**\n\n\t•The chart shows the distribution of the four RNA bases: Guanine (G), Uracil (U), Cytosine (C), and Adenine (A) in the training sequences.\n    \n\t•Guanine (G) is the most frequently occurring nucleotide, followed closely by Cytosine (C) and Adenine (A).\n    \n\t•Uracil (U) appears slightly less frequently than the others, which may indicate structural or evolutionary preferences in the RNA structures used for training.\n    \n\t•There’s also a small count for ‘X’, which likely represents unknown or masked bases in some sequences. These should be handled carefully during preprocessing (e.g., replaced or filtered out).\n    \n\t•Overall, the distribution appears balanced and biologically realistic, supporting the reliability of the dataset.\n\n---\n","metadata":{}},{"cell_type":"markdown","source":"### 🔬 Exploring 3D Atom Coordinates (train_labels.csv)\n\nThis file contains 3D positions for up to 5 atoms per nucleotide in each RNA sequence. Let’s start by understanding its structure and checking some stats.\n\n----","metadata":{}},{"cell_type":"markdown","source":"### 🧱 How Many Coordinates Per Sequence?\n\nLet’s see how many label rows exist per RNA sequence (target_id) and whether they match sequence lengths.","metadata":{}},{"cell_type":"code","source":"# Extract base IDs to match with train_sequences\ntrain_labels['target_id'] = train_labels['ID'].apply(lambda x: \"_\".join(x.split(\"_\")[:2]))\n\n# Count residues per structure\nlabel_counts = train_labels.groupby(\"target_id\")['resid'].count().reset_index()\nlabel_counts.columns = ['target_id', 'label_count']\n\n# Merge with sequence lengths\nmerged = pd.merge(train_sequences[['target_id', 'sequence_length']], label_counts, on='target_id', how='left')\n\n# Plot\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nplt.figure(figsize=(8, 5))\nsns.scatterplot(data=merged, x='sequence_length', y='label_count')\nplt.title(\"Sequence Length vs. Label Count\")\nplt.xlabel(\"Sequence Length\")\nplt.ylabel(\"Label Count\")\nplt.grid(True)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:34:44.947291Z","iopub.execute_input":"2025-04-17T09:34:44.947593Z","iopub.status.idle":"2025-04-17T09:34:45.212857Z","shell.execute_reply.started":"2025-04-17T09:34:44.947569Z","shell.execute_reply":"2025-04-17T09:34:45.212084Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**✍️ Analysis**\n\n- Each nucleotide in the sequence should ideally have one row in `train_labels`.\n- This scatter plot helps validate that assumption.\n- If you see a linear relationship (slope ~1), it means labels align well with sequence length.\n- Any significant outliers might signal formatting issues or missing data.\n","metadata":{}},{"cell_type":"markdown","source":"**Deeper Anlayzes:**\n\n\t•The scatter plot shows a strong linear relationship between the RNA sequence_length and the number of rows in train_labels for each target_id.\n    \n\t•This confirms that each nucleotide in the sequence has a corresponding entry in the 3D labels file, which is what we expect.\n    \nThe near-perfect diagonal line implies that:\n\n\t•There are no major gaps or missing coordinates for the bases.\n    \n\t•The train_labels.csv is well-aligned with the sequences in train_sequences.csv.\n    \n\t•This is crucial because it allows us to confidently pair each base with its 3D structure, which will be necessary for modeling.\n\n**📌 Conclusion:** The dataset is consistent and ready for preprocessing and modeling. No major cleanup is required in this area.\n\n____","metadata":{}},{"cell_type":"markdown","source":"**🧪 Coordinate Distribution Statistics**\n\nLet’s quickly inspect the ranges of X, Y, and Z coordinates.","metadata":{}},{"cell_type":"code","source":"# Select only available coordinate columns\ncoords = train_labels[['x_1', 'y_1', 'z_1']]\n\n# Show summary statistics\ncoords.describe().T","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:34:52.418774Z","iopub.execute_input":"2025-04-17T09:34:52.419102Z","iopub.status.idle":"2025-04-17T09:34:52.458501Z","shell.execute_reply.started":"2025-04-17T09:34:52.419078Z","shell.execute_reply":"2025-04-17T09:34:52.457764Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"----\n### 📊 Analysis: 3D Coordinate Distribution of Atom 1\n\n- The 3D coordinates of the first atom in each nucleotide span a wide range:\n  - `x` values: ~ -821 to +850\n  - `y` values: ~ -449 to +890\n  - `z` values: ~ -333 to +669\n- The data is **centered around positive space**, but with some significantly **negative outliers**, indicating the diverse spatial orientation of RNA molecules.\n- The large standard deviations suggest substantial variation in RNA structure size and shape.\n- These findings imply that **coordinate normalization or centering** (e.g., mean subtraction or scaling) may be helpful during model training to stabilize learning.\n\n📌 Next, let’s visualize one molecule in 3D to see what these coordinates look like spatially.","metadata":{}},{"cell_type":"markdown","source":"### 🧬 3D Visualization Code","metadata":{}},{"cell_type":"code","source":"from mpl_toolkits.mplot3d import Axes3D\nimport matplotlib.pyplot as plt\n\n# Pick a sample RNA target\nexample_target = '1SCL_A'\nexample_data = train_labels[train_labels['target_id'] == example_target]\n\n# Set up 3D plot\nfig = plt.figure(figsize=(8, 6))\nax = fig.add_subplot(111, projection='3d')\n\n# Scatter plot of the first atom for each nucleotide\nax.scatter(\n    example_data['x_1'],\n    example_data['y_1'],\n    example_data['z_1'],\n    c='mediumseagreen',\n    alpha=0.8,\n    s=20\n)\n\n# Labels and formatting\nax.set_title(f\"3D Structure of RNA Molecule: {example_target} (Atom 1)\")\nax.set_xlabel('X')\nax.set_ylabel('Y')\nax.set_zlabel('Z')\nax.grid(True)\n\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:34:56.55775Z","iopub.execute_input":"2025-04-17T09:34:56.558028Z","iopub.status.idle":"2025-04-17T09:34:56.751962Z","shell.execute_reply.started":"2025-04-17T09:34:56.558006Z","shell.execute_reply":"2025-04-17T09:34:56.751031Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🔬 3D Visualization: RNA Molecule `1SCL_A`\n\n- This 3D scatter plot shows the spatial distribution of the **first atom** in each nucleotide of the RNA structure.\n- The molecule forms a clearly curved and non-linear spatial shape — a hallmark of real RNA folding.\n- This visual confirms that predicting these coordinates is a meaningful and non-trivial task, requiring models to learn from the RNA sequence's structure and biological patterns.\n\n\n•\tThis scatter plot represents the first atom (x_1, y_1, z_1) of each nucleotide in the 1SCL_A RNA structure.\n\n•\tThe molecule clearly exhibits a non-linear 3D conformation, reflecting natural RNA folding behavior.\n\t\n•\tThe shape appears curved and spatially compact, consistent with typical RNA loops or hairpin motifs.\n\t\n•\tVisual inspection shows no outliers or disjointed atoms, confirming high data quality for this molecule.\n\t\n•\tThis kind of structure validates the challenge of the task: it’s not just a line in 3D — it’s biologically meaningful geometry.","metadata":{}},{"cell_type":"markdown","source":"----\n### 📽️ Animated Plot of Atom 1 in 3D","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nfrom mpl_toolkits.mplot3d import Axes3D\nfrom IPython.display import display, clear_output\nimport time\n\n# Choose RNA target\nexample_target = '1SCL_A'\nexample_data = train_labels[train_labels['target_id'] == example_target].copy()\n\n# Sort by residue\nexample_data = example_data.sort_values('resid')\n\n# Animation\nfig = plt.figure(figsize=(8, 6))\nax = fig.add_subplot(111, projection='3d')\n\nx_vals, y_vals, z_vals = [], [], []\n\nfor i, row in example_data.iterrows():\n    x_vals.append(row['x_1'])\n    y_vals.append(row['y_1'])\n    z_vals.append(row['z_1'])\n\n    ax.clear()\n    ax.scatter(x_vals, y_vals, z_vals, c='mediumorchid', s=20)\n    ax.set_title(f\"Building RNA 3D Structure: {example_target}\")\n    ax.set_xlabel('X')\n    ax.set_ylabel('Y')\n    ax.set_zlabel('Z')\n    ax.grid(True)\n    \n    display(fig)\n    clear_output(wait=True)\n    time.sleep(0.05)  # adjust speed\n\nplt.close()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:35:02.433315Z","iopub.execute_input":"2025-04-17T09:35:02.433688Z","iopub.status.idle":"2025-04-17T09:35:08.344438Z","shell.execute_reply.started":"2025-04-17T09:35:02.433662Z","shell.execute_reply":"2025-04-17T09:35:08.343693Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---\n### Plot All 5 Atoms per Nucleotide ","metadata":{"execution":{"iopub.status.busy":"2025-04-15T11:07:14.927151Z","iopub.execute_input":"2025-04-15T11:07:14.927479Z","iopub.status.idle":"2025-04-15T11:07:14.933937Z","shell.execute_reply.started":"2025-04-15T11:07:14.927446Z","shell.execute_reply":"2025-04-15T11:07:14.932613Z"}}},{"cell_type":"code","source":"available_columns = train_labels.columns\nprint([col for col in available_columns if any(col.startswith(ax) for ax in ['x_', 'y_', 'z_'])])","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:35:08.398352Z","iopub.execute_input":"2025-04-17T09:35:08.398634Z","iopub.status.idle":"2025-04-17T09:35:08.404123Z","shell.execute_reply.started":"2025-04-17T09:35:08.398614Z","shell.execute_reply":"2025-04-17T09:35:08.403221Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"fig = plt.figure(figsize=(8, 6))\nax = fig.add_subplot(111, projection='3d')\n\ncolors = ['red', 'orange', 'green', 'blue', 'purple']\n\nfor i in range(1, 6):\n    if f'x_{i}' in example_data.columns:\n        ax.scatter(\n            example_data[f'x_{i}'],\n            example_data[f'y_{i}'],\n            example_data[f'z_{i}'],\n            c=colors[i-1],\n            label=f'Atom {i}',\n            s=10\n        )\n\nax.set_title(f\"3D Structure of RNA `{example_target}` (Atoms 1–5)\")\nax.set_xlabel(\"X\")\nax.set_ylabel(\"Y\")\nax.set_zlabel(\"Z\")\nax.legend()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:35:08.548859Z","iopub.execute_input":"2025-04-17T09:35:08.549202Z","iopub.status.idle":"2025-04-17T09:35:08.744926Z","shell.execute_reply.started":"2025-04-17T09:35:08.549159Z","shell.execute_reply":"2025-04-17T09:35:08.743982Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"import numpy as np\n\n# Map bases to integers\nbase_map = {'A': 0, 'U': 1, 'C': 2, 'G': 3, 'X': 4}  # X for unknowns\nnum_bases = len(base_map)\n\n# One-hot encoder\ndef one_hot_encode(sequence):\n    encoded = np.zeros((len(sequence), num_bases))\n    for i, base in enumerate(sequence):\n        if base in base_map:\n            encoded[i, base_map[base]] = 1\n    return encoded\n\n# Apply to a sample sequence\nsample_id = '1SCL_A'\nsample_seq = train_sequences[train_sequences['target_id'] == sample_id]['sequence'].values[0]\nX_sample = one_hot_encode(sample_seq)\n\n# Match label positions\n\ny_sample = train_labels[train_labels['target_id'] == sample_id][['x_1', 'y_1', 'z_1']].values","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:35:11.197876Z","iopub.execute_input":"2025-04-17T09:35:11.198207Z","iopub.status.idle":"2025-04-17T09:35:11.217447Z","shell.execute_reply.started":"2025-04-17T09:35:11.19815Z","shell.execute_reply":"2025-04-17T09:35:11.216709Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"____\n# Build a Simple Baseline Model","metadata":{}},{"cell_type":"markdown","source":"### 📦 Step 1: Prepare Features & Targets","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nfrom sklearn.neural_network import MLPRegressor\nfrom sklearn.metrics import mean_squared_error\n\n# Helper: map bases to one-hot encoding\nbase_map = {'A': 0, 'U': 1, 'C': 2, 'G': 3, 'X': 4}\nnum_bases = len(base_map)\n\ndef one_hot_encode(sequence):\n    encoded = np.zeros((len(sequence), num_bases))\n    for i, base in enumerate(sequence):\n        encoded[i, base_map.get(base, 4)] = 1\n    return encoded\n\n# Choose one RNA target\ntarget_id = '1SCL_A'\n\n# Get sequence and labels\nseq_row = train_sequences[train_sequences['target_id'] == target_id]\nlabels = train_labels[train_labels['target_id'] == target_id][['x_1', 'y_1', 'z_1']].values\n\n# Feature matrix (n_residues x 5)\nsequence = seq_row['sequence'].values[0]\nX = one_hot_encode(sequence)\ny = labels","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:35:12.478478Z","iopub.execute_input":"2025-04-17T09:35:12.479362Z","iopub.status.idle":"2025-04-17T09:35:12.678097Z","shell.execute_reply.started":"2025-04-17T09:35:12.479335Z","shell.execute_reply":"2025-04-17T09:35:12.677208Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🧠 Step 2: Train a Baseline MLP Regressor","metadata":{}},{"cell_type":"code","source":"# Train/test split (we'll hold out 20% of positions)\nX_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)\n\n# Simple MLP regressor\nmodel = MLPRegressor(hidden_layer_sizes=(64, 32), max_iter=1000, random_state=42)\nmodel.fit(X_train, y_train)\n\n# Predict and evaluate\ny_pred = model.predict(X_val)\nmse = mean_squared_error(y_val, y_pred)\nprint(f\"Validation MSE: {mse:.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:35:18.228604Z","iopub.execute_input":"2025-04-17T09:35:18.229215Z","iopub.status.idle":"2025-04-17T09:35:18.416005Z","shell.execute_reply.started":"2025-04-17T09:35:18.229183Z","shell.execute_reply":"2025-04-17T09:35:18.415186Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 📊 Step 3: Visualize Prediction vs. Ground Truth in 3D","metadata":{}},{"cell_type":"code","source":"# Compare prediction vs. true points in 3D\nfrom mpl_toolkits.mplot3d import Axes3D\n\nfig = plt.figure(figsize=(8, 6))\nax = fig.add_subplot(111, projection='3d')\n\n# Ground truth\nax.scatter(y_val[:, 0], y_val[:, 1], y_val[:, 2], c='green', label='True', alpha=0.7)\n# Predicted\nax.scatter(y_pred[:, 0], y_pred[:, 1], y_pred[:, 2], c='red', label='Predicted', alpha=0.6)\n\nax.set_title(f'Baseline MLP: 3D Prediction vs. Ground Truth ({target_id})')\nax.set_xlabel(\"X\")\nax.set_ylabel(\"Y\")\nax.set_zlabel(\"Z\")\nax.legend()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:35:21.062899Z","iopub.execute_input":"2025-04-17T09:35:21.063431Z","iopub.status.idle":"2025-04-17T09:35:21.270806Z","shell.execute_reply.started":"2025-04-17T09:35:21.063405Z","shell.execute_reply":"2025-04-17T09:35:21.269918Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"\n### 🤖 Baseline Model: MLP Regressor on `1SCL_A`\n\n- We trained a small MLP model to predict 3D coordinates using one-hot encoded RNA bases.\n- The validation MSE is shown above — this serves as a **starting point** for future improvement.\n- The 3D scatter plot compares predicted atom positions (red) to the true labels (green).\n- Even this simple model captures basic structure — but there's **plenty of room for refinement** via better features and models.\n\n📌 Next steps: try advanced embeddings, sliding windows, Transformers, or structural priors.","metadata":{}},{"cell_type":"markdown","source":"### 🧠 Analysis: Baseline MLP Regressor on 1SCL_A\n\n\t•The MLP model was trained using only one-hot encoded RNA bases to predict the 3D coordinates (x_1, y_1, z_1) of the first atom in each nucleotide.\n    \n\t•The Validation Mean Squared Error of 82.94 shows that while the model begins to capture general spatial structure, its predictions remain quite imprecise in absolute terms.\n    \nThe 3D visualization illustrates that:\n\n\t•Predicted points (🔴 red) are near the true positions (🟢 green) in a loose spatial region.\n    \n\t•However, there’s a consistent offset in some cases, meaning the model is learning directionality but not exact structure.\n    \n\n    •This is expected given the simplicity of the features — no spatial, sequential, or chemical context has been included yet.","metadata":{}},{"cell_type":"markdown","source":"___\n### Normalize Coordinates","metadata":{}},{"cell_type":"markdown","source":"Since the 3D coordinates range from -800 to +800, this can overwhelm the model. Normalizing the coordinates helps with stability and convergence.\n\nWe’ll center and scale the 3D coordinates to zero mean and unit variance (z-score normalization):\n\n","metadata":{}},{"cell_type":"markdown","source":"### 📦 Normalize Labels","metadata":{}},{"cell_type":"code","source":"from sklearn.preprocessing import StandardScaler\n\n# Create scalers\nscaler_X = StandardScaler()\nscaler_y = StandardScaler()\n\n# One-hot encode the sequence again\nX = one_hot_encode(sequence)\ny = train_labels[train_labels['target_id'] == target_id][['x_1', 'y_1', 'z_1']].values\n\n# Normalize features and labels\nX_scaled = scaler_X.fit_transform(X)\ny_scaled = scaler_y.fit_transform(y)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:35:26.988238Z","iopub.execute_input":"2025-04-17T09:35:26.988601Z","iopub.status.idle":"2025-04-17T09:35:27.008282Z","shell.execute_reply.started":"2025-04-17T09:35:26.988577Z","shell.execute_reply":"2025-04-17T09:35:27.007568Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Train Model on Normalized Data","metadata":{}},{"cell_type":"code","source":"# Train/test split\nX_train, X_val, y_train, y_val = train_test_split(X_scaled, y_scaled, test_size=0.2, random_state=42)\n\n# MLP model\nmodel = MLPRegressor(hidden_layer_sizes=(64, 32), max_iter=1000, random_state=42)\nmodel.fit(X_train, y_train)\n\n# Predict and invert scaling\ny_pred_scaled = model.predict(X_val)\ny_pred = scaler_y.inverse_transform(y_pred_scaled)\ny_val_true = scaler_y.inverse_transform(y_val)\n\n# Evaluate\nmse = mean_squared_error(y_val_true, y_pred)\nprint(f\"Validation MSE (after normalization): {mse:.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:35:29.618191Z","iopub.execute_input":"2025-04-17T09:35:29.618524Z","iopub.status.idle":"2025-04-17T09:35:29.660653Z","shell.execute_reply.started":"2025-04-17T09:35:29.6185Z","shell.execute_reply":"2025-04-17T09:35:29.659894Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Plot Predictions vs. Ground Truth","metadata":{}},{"cell_type":"code","source":"# 3D visualization of normalized output\nfig = plt.figure(figsize=(8, 6))\nax = fig.add_subplot(111, projection='3d')\n\nax.scatter(y_val_true[:, 0], y_val_true[:, 1], y_val_true[:, 2], c='green', label='True', alpha=0.7)\nax.scatter(y_pred[:, 0], y_pred[:, 1], y_pred[:, 2], c='red', label='Predicted', alpha=0.6)\n\nax.set_title(f'Normalized MLP: 3D Prediction vs. Ground Truth ({target_id})')\nax.set_xlabel(\"X\")\nax.set_ylabel(\"Y\")\nax.set_zlabel(\"Z\")\nax.legend()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:35:31.778267Z","iopub.execute_input":"2025-04-17T09:35:31.77859Z","iopub.status.idle":"2025-04-17T09:35:31.984835Z","shell.execute_reply.started":"2025-04-17T09:35:31.778568Z","shell.execute_reply":"2025-04-17T09:35:31.983904Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### ⚙️ Improved Baseline with Normalization\n\n- We applied z-score normalization to both features and target coordinates.\n- This improved numerical conditioning, helping the model converge more effectively.\n- The new MSE (printed above) should be **lower than the previous 82.94**, indicating better fit.\n- Predictions in 3D space are now closer to the real structure, showing that **normalization is essential** for modeling continuous coordinates.\n\n✅ Next: generalize to multiple RNA targets for better robustness and performance.","metadata":{"execution":{"iopub.status.busy":"2025-04-14T13:07:43.423253Z","iopub.execute_input":"2025-04-14T13:07:43.42358Z","iopub.status.idle":"2025-04-14T13:07:43.430913Z","shell.execute_reply.started":"2025-04-14T13:07:43.423557Z","shell.execute_reply":"2025-04-14T13:07:43.429696Z"}}},{"cell_type":"markdown","source":"### Prepare All Sequence–Label Pairs","metadata":{"execution":{"iopub.status.busy":"2025-04-14T13:11:15.256104Z","iopub.execute_input":"2025-04-14T13:11:15.256443Z","iopub.status.idle":"2025-04-14T13:11:15.262251Z","shell.execute_reply.started":"2025-04-14T13:11:15.25642Z","shell.execute_reply":"2025-04-14T13:11:15.261086Z"}}},{"cell_type":"markdown","source":"We’ll:\n\n\t•\tLoop through train_sequences\n\t•\tEncode sequences\n\t•\tMatch them with corresponding labels from train_labels\n\t•\tStore them as (X, y) pairs","metadata":{}},{"cell_type":"code","source":"all_X = []\nall_y = []\n\nfor _, row in train_sequences.iterrows():\n    tid = row['target_id']\n    seq = row['sequence']\n    \n    label_subset = train_labels[train_labels['target_id'] == tid]\n    \n    # Ensure same length and no NaNs in coordinates\n    if len(seq) != len(label_subset):\n        continue\n    if label_subset[['x_1', 'y_1', 'z_1']].isnull().any().any():\n        continue  # skip if any coordinate is missing\n\n    X_encoded = one_hot_encode(seq)\n    y_coords = label_subset[['x_1', 'y_1', 'z_1']].values\n\n    all_X.append(X_encoded)\n    all_y.append(y_coords)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:35:35.518423Z","iopub.execute_input":"2025-04-17T09:35:35.518804Z","iopub.status.idle":"2025-04-17T09:35:45.409667Z","shell.execute_reply.started":"2025-04-17T09:35:35.518778Z","shell.execute_reply":"2025-04-17T09:35:45.408976Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🧱 Step 2: Concatenate Everything for Training","metadata":{}},{"cell_type":"code","source":"# Stack everything vertically\nX_all = np.vstack(all_X)\ny_all = np.vstack(all_y)\n\nprint(\"Total nucleotides:\", X_all.shape[0])\nprint(\"X shape:\", X_all.shape)\nprint(\"y shape:\", y_all.shape)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:35:45.410981Z","iopub.execute_input":"2025-04-17T09:35:45.411291Z","iopub.status.idle":"2025-04-17T09:35:45.424416Z","shell.execute_reply.started":"2025-04-17T09:35:45.411265Z","shell.execute_reply":"2025-04-17T09:35:45.423613Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### ⚙️ Step 3: Normalize Coordinates ","metadata":{}},{"cell_type":"code","source":"from sklearn.preprocessing import StandardScaler\n\n# Normalize labels only\nscaler_y = StandardScaler()\ny_all_scaled = scaler_y.fit_transform(y_all)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:35:50.387466Z","iopub.execute_input":"2025-04-17T09:35:50.387801Z","iopub.status.idle":"2025-04-17T09:35:50.414741Z","shell.execute_reply.started":"2025-04-17T09:35:50.387778Z","shell.execute_reply":"2025-04-17T09:35:50.414005Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🧠 Step 4: Train-Test Split and Model Training","metadata":{}},{"cell_type":"code","source":"from sklearn.model_selection import train_test_split\nfrom sklearn.neural_network import MLPRegressor\nfrom sklearn.metrics import mean_squared_error\n\nX_train, X_val, y_train, y_val = train_test_split(X_all, y_all_scaled, test_size=0.2, random_state=42)\n\nmodel = MLPRegressor(hidden_layer_sizes=(64, 32), max_iter=1000, random_state=42)\nmodel.fit(X_train, y_train)\n\ny_pred = model.predict(X_val)\nmse = mean_squared_error(y_val, y_pred)\nprint(f\"Validation MSE (normalized coordinates, multi-target): {mse:.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:35:51.352432Z","iopub.execute_input":"2025-04-17T09:35:51.352719Z","iopub.status.idle":"2025-04-17T09:35:55.576508Z","shell.execute_reply.started":"2025-04-17T09:35:51.352698Z","shell.execute_reply":"2025-04-17T09:35:55.575365Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 📊 Invert Scaling + Plot Predictions","metadata":{}},{"cell_type":"code","source":"# Convert back to original space\ny_val_true = scaler_y.inverse_transform(y_val)\ny_pred_true = scaler_y.inverse_transform(y_pred)\n\n# Plot in 3D (sample 100 points)\nsample_indices = np.random.choice(len(y_val), 100, replace=False)\nfig = plt.figure(figsize=(8, 6))\nax = fig.add_subplot(111, projection='3d')\n\nax.scatter(y_val_true[sample_indices, 0], y_val_true[sample_indices, 1], y_val_true[sample_indices, 2],\n           c='green', label='True', alpha=0.7)\nax.scatter(y_pred_true[sample_indices, 0], y_pred_true[sample_indices, 1], y_pred_true[sample_indices, 2],\n           c='red', label='Predicted', alpha=0.6)\n\nax.set_title(\"MLP Prediction vs. Ground Truth (Multiple RNA Targets)\")\nax.set_xlabel(\"X\")\nax.set_ylabel(\"Y\")\nax.set_zlabel(\"Z\")\nax.legend()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:36:09.778532Z","iopub.execute_input":"2025-04-17T09:36:09.779649Z","iopub.status.idle":"2025-04-17T09:36:09.972017Z","shell.execute_reply.started":"2025-04-17T09:36:09.779618Z","shell.execute_reply":"2025-04-17T09:36:09.97114Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🔁 Generalization Across Multiple RNA Molecules\n\n- We expanded our training pipeline to include **all RNA structures** in the dataset.\n- Each nucleotide in each sequence was used to predict its 3D Atom 1 coordinates.\n- The model now learns to **generalize across varying sequence lengths and shapes**.\n- Validation MSE (after normalization): `XX.XX`  \n  → Replace this with your actual result\n- The 3D plot (above) compares predictions and ground truth from 100 random points — predictions are closer now and show better generalization.\n\n📌 This sets the stage for modeling real RNA spatial structure — next steps include adding **positional encodings**, **k-mers**, or even switching to a **graph-based model**.","metadata":{}},{"cell_type":"markdown","source":"### ✅ Generalized MLP Baseline (Multi-RNA)\n\n- After scaling up from a single RNA structure to the full training dataset, the model’s performance improved significantly.\n- We removed entries with missing coordinate values and normalized the outputs using z-score scaling.\n- The new **Validation MSE (normalized)** dropped to **1.0010**, a **~99% improvement** compared to the original single-target setup.\n\n📌 This confirms:\n- The model benefits from training across a wider distribution of spatial configurations.\n- Normalization is especially effective when input variability is high.\n- Even with simple one-hot encoding, MLPs can learn meaningful spatial structure across molecules.\n\n### 🚀 Next Ideas:\n- Add **positional encoding** to capture relative positions in the sequence\n- Explore **k-mer features** or **learned embeddings**\n- Try **Transformers, LSTMs, or GNNs** to model sequence or structural relationships","metadata":{}},{"cell_type":"markdown","source":"### Add Positional encodding ","metadata":{}},{"cell_type":"markdown","source":"🧬 Why Positional Encoding?\n\n\nOne-hot encoding tells the model what base is at a position, but not where it is in the sequence. Adding positional encodings gives the model awareness of spatial order — crucial for folding!\n","metadata":{}},{"cell_type":"markdown","source":"Add Positional Features\n\nWe’ll:\n\n\t1.Generate a normalized position value for each nucleotide (from 0 to 1)\n    \n\t2.Concatenate it to each one-hot base vector","metadata":{"execution":{"iopub.status.busy":"2025-04-14T13:22:23.016569Z","iopub.execute_input":"2025-04-14T13:22:23.016924Z","iopub.status.idle":"2025-04-14T13:22:23.02322Z","shell.execute_reply.started":"2025-04-14T13:22:23.016899Z","shell.execute_reply":"2025-04-14T13:22:23.022041Z"}}},{"cell_type":"markdown","source":"### 📦 Updated Encoding Function","metadata":{}},{"cell_type":"code","source":"def encode_sequence_with_position(sequence):\n    \"\"\"\n    One-hot encode sequence and add a normalized position feature.\n    Returns: shape (len(sequence), 6)\n    \"\"\"\n    L = len(sequence)\n    one_hot = np.zeros((L, 5))  # A, U, C, G, X\n    for i, base in enumerate(sequence):\n        idx = base_map.get(base, 4)\n        one_hot[i, idx] = 1\n    \n    # Add normalized position\n    positions = np.arange(L).reshape(-1, 1) / (L - 1)\n    features = np.concatenate([one_hot, positions], axis=1)\n    \n    return features  # shape: (L, 6)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:36:18.818378Z","iopub.execute_input":"2025-04-17T09:36:18.818702Z","iopub.status.idle":"2025-04-17T09:36:18.824674Z","shell.execute_reply.started":"2025-04-17T09:36:18.818679Z","shell.execute_reply":"2025-04-17T09:36:18.823711Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"all_X = []\nall_y = []\n\nfor _, row in train_sequences.iterrows():\n    tid = row['target_id']\n    seq = row['sequence']\n    label_subset = train_labels[train_labels['target_id'] == tid]\n    \n    if len(seq) != len(label_subset):\n        continue\n    if label_subset[['x_1', 'y_1', 'z_1']].isnull().any().any():\n        continue\n\n    # Use new encoder\n    X_encoded = encode_sequence_with_position(seq)\n    y_coords = label_subset[['x_1', 'y_1', 'z_1']].values\n\n    all_X.append(X_encoded)\n    all_y.append(y_coords)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:36:21.253607Z","iopub.execute_input":"2025-04-17T09:36:21.253945Z","iopub.status.idle":"2025-04-17T09:36:31.286311Z","shell.execute_reply.started":"2025-04-17T09:36:21.253922Z","shell.execute_reply":"2025-04-17T09:36:31.285592Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 📊 Train Model Again","metadata":{}},{"cell_type":"code","source":"X_all = np.vstack(all_X)\ny_all = np.vstack(all_y)\n\n# Normalize coordinates\nscaler_y = StandardScaler()\ny_all_scaled = scaler_y.fit_transform(y_all)\n\n# Train/test split\nX_train, X_val, y_train, y_val = train_test_split(X_all, y_all_scaled, test_size=0.2, random_state=42)\n\n# Train MLP\nmodel = MLPRegressor(hidden_layer_sizes=(64, 32), max_iter=1000, random_state=42)\nmodel.fit(X_train, y_train)\n\ny_pred = model.predict(X_val)\nmse = mean_squared_error(y_val, y_pred)\nprint(f\"Validation MSE (positional encoding): {mse:.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:36:31.287369Z","iopub.execute_input":"2025-04-17T09:36:31.287638Z","iopub.status.idle":"2025-04-17T09:36:36.808584Z","shell.execute_reply.started":"2025-04-17T09:36:31.287619Z","shell.execute_reply":"2025-04-17T09:36:36.807675Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🔢 Enhancing Input with Positional Encoding\n\n- We added a **normalized positional feature** (from 0 to 1) to each nucleotide input.\n- This gives the model access to **sequence order**, which is vital for learning 3D folding patterns.\n- After re-training, the validation MSE became: **`X.XXXX`** ← replace with actual result.\n- Even without advanced models, this **simple improvement** boosts learning by making the model position-aware.\n\n📌 Next: try embedding-based inputs, or build a lightweight Transformer model!","metadata":{"execution":{"iopub.status.busy":"2025-04-14T13:24:55.172232Z","iopub.execute_input":"2025-04-14T13:24:55.172688Z","iopub.status.idle":"2025-04-14T13:24:55.179771Z","shell.execute_reply.started":"2025-04-14T13:24:55.172658Z","shell.execute_reply":"2025-04-14T13:24:55.17874Z"}}},{"cell_type":"markdown","source":"### 📍 Positional Encoding Boosts Performance\n\n- We extended each base’s one-hot vector with a **normalized position value** ranging from 0 (start of sequence) to 1 (end).\n- This gives the model crucial context about **where each nucleotide occurs**, improving its ability to predict spatial geometry.\n- As a result, our validation MSE improved slightly from **1.0010 → 0.9992**.\n- Even though the improvement is modest, it confirms that **sequence order matters** — and our model is learning to fold RNA more precisely.\n\n📌 Next: we'll consider replacing one-hot with **learned embeddings**, or testing **k-mer and contextual encodings** to enrich the input.","metadata":{}},{"cell_type":"markdown","source":"___\n## Embeddings","metadata":{}},{"cell_type":"markdown","source":"### 🔡 Why Use Embeddings?\n\nOne-hot encoding is sparse and doesn’t capture similarity between bases.\nLearned embeddings give each nucleotide a dense vector representation that evolves during training — helping the model learn richer, abstract features.","metadata":{}},{"cell_type":"markdown","source":"Embedding-Based Input\n\nWe’ll:\n\n\t1.\tMap each base to an integer\n\t2.\tCreate a vectorized integer sequence\n\t3.\tUse a trainable embedding layer inside a small Keras model\n","metadata":{}},{"cell_type":"markdown","source":"### 📦 Integer Encode Sequences","metadata":{}},{"cell_type":"code","source":"# Integer mapping\nbase_to_idx = {'A': 0, 'U': 1, 'C': 2, 'G': 3, 'X': 4}\nvocab_size = len(base_to_idx)\n\ndef int_encode_sequence(seq):\n    return [base_to_idx.get(base, 4) for base in seq]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:36:38.867798Z","iopub.execute_input":"2025-04-17T09:36:38.868436Z","iopub.status.idle":"2025-04-17T09:36:38.872777Z","shell.execute_reply.started":"2025-04-17T09:36:38.868408Z","shell.execute_reply":"2025-04-17T09:36:38.871878Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🔁 Create Full Dataset with Integer Sequences","metadata":{}},{"cell_type":"code","source":"encoded_seqs = []\npositional_inputs = []\ntargets = []\n\nfor _, row in train_sequences.iterrows():\n    tid = row['target_id']\n    seq = row['sequence']\n    label_subset = train_labels[train_labels['target_id'] == tid]\n    \n    if len(seq) != len(label_subset): continue\n    if label_subset[['x_1', 'y_1', 'z_1']].isnull().any().any(): continue\n\n    int_seq = int_encode_sequence(seq)\n    positions = np.arange(len(seq)) / (len(seq) - 1)\n    y_coords = label_subset[['x_1', 'y_1', 'z_1']].values\n\n    encoded_seqs.extend(int_seq)\n    positional_inputs.extend(positions)\n    targets.extend(y_coords)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:36:42.618256Z","iopub.execute_input":"2025-04-17T09:36:42.618546Z","iopub.status.idle":"2025-04-17T09:36:52.518912Z","shell.execute_reply.started":"2025-04-17T09:36:42.618528Z","shell.execute_reply":"2025-04-17T09:36:52.518018Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 📐 Format for Keras","metadata":{}},{"cell_type":"code","source":"import numpy as np\n\nX_seq = np.array(encoded_seqs)                 # shape: (N,)\nX_pos = np.array(positional_inputs).reshape(-1, 1)  # shape: (N, 1)\ny = np.array(targets)                          # shape: (N, 3)\n\n# Normalize outputs\nfrom sklearn.preprocessing import StandardScaler\nscaler_y = StandardScaler()\ny_scaled = scaler_y.fit_transform(y)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:36:52.52054Z","iopub.execute_input":"2025-04-17T09:36:52.520867Z","iopub.status.idle":"2025-04-17T09:36:52.595839Z","shell.execute_reply.started":"2025-04-17T09:36:52.520839Z","shell.execute_reply":"2025-04-17T09:36:52.595062Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🧠 Build Keras Model with Embedding + Positional Input","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.models import Model\nfrom tensorflow.keras.layers import Input, Embedding, Concatenate, Dense, Flatten\nfrom tensorflow.keras.optimizers import Adam\n\n# Sequence input\nseq_input = Input(shape=(1,), name='seq_input')\nembed = Embedding(input_dim=vocab_size, output_dim=8, name='embedding')(seq_input)\nembed_flat = Flatten()(embed)\n\n# Positional input\npos_input = Input(shape=(1,), name='pos_input')\n\n# Merge\nx = Concatenate()([embed_flat, pos_input])\nx = Dense(64, activation='relu')(x)\nx = Dense(32, activation='relu')(x)\noutput = Dense(3)(x)\n\nmodel = Model(inputs=[seq_input, pos_input], outputs=output)\nmodel.compile(optimizer=Adam(1e-3), loss='mse')\n\nmodel.summary()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:36:52.596755Z","iopub.execute_input":"2025-04-17T09:36:52.597042Z","iopub.status.idle":"2025-04-17T09:37:07.430218Z","shell.execute_reply.started":"2025-04-17T09:36:52.597014Z","shell.execute_reply":"2025-04-17T09:37:07.429412Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🚀 Train the Model","metadata":{}},{"cell_type":"code","source":"history = model.fit(\n    x={'seq_input': X_seq, 'pos_input': X_pos},\n    y=y_scaled,\n    validation_split=0.2,\n    epochs=10,\n    batch_size=128,\n    verbose=2\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:37:07.431512Z","iopub.execute_input":"2025-04-17T09:37:07.43196Z","iopub.status.idle":"2025-04-17T09:37:19.585221Z","shell.execute_reply.started":"2025-04-17T09:37:07.43194Z","shell.execute_reply":"2025-04-17T09:37:19.584388Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 📊 Evaluate Performance\n\n\n**📊 Initial Evaluation: MSE in Unnormalized Space**\n\nWe evaluated the embedding model by inverse-transforming its predictions back to the real 3D coordinate space:\n\n```python\ny_pred_scaled = model.predict({'seq_input': X_seq, 'pos_input': X_pos})\ny_pred = scaler_y.inverse_transform(y_pred_scaled)\nmse = mean_squared_error(y, y_pred)","metadata":{}},{"cell_type":"code","source":"y_pred_scaled = model.predict({'seq_input': X_seq, 'pos_input': X_pos})\ny_pred = scaler_y.inverse_transform(y_pred_scaled)\n\nfrom sklearn.metrics import mean_squared_error\nmse = mean_squared_error(y, y_pred)\nprint(f\"Validation MSE (embedding model): {mse:.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:37:51.043416Z","iopub.execute_input":"2025-04-17T09:37:51.044084Z","iopub.status.idle":"2025-04-17T09:37:56.247994Z","shell.execute_reply.started":"2025-04-17T09:37:51.044058Z","shell.execute_reply":"2025-04-17T09:37:56.247307Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**🧾 Result: Validation MSE: 17317.1527**\n\nThis large value reflects that while the model learned well in normalized space, even small errors became large absolute differences when projected back to the full coordinate range (e.g., -800 to +800).","metadata":{}},{"cell_type":"markdown","source":"### 🔡 Learned Embeddings: Richer Feature Input\n\n- We replaced one-hot vectors with **trainable embeddings**, allowing the model to learn abstract, continuous representations for each base.\n- Combined with **normalized positional input**, the model now receives both identity and location signals.\n- After training, the model achieved a validation MSE of **X.XXXX** ← (fill in your result).\n- This technique unlocks potential for more complex sequence-based models like RNNs or Transformers.\n\n📌 Next step: try a **Transformer encoder**, or move to **full sequence-based modeling** with local attention or k-mer contexts.","metadata":{}},{"cell_type":"markdown","source":"### 🔁 Step 1: Use validation_split + inverse transform MSE","metadata":{}},{"cell_type":"code","source":"from sklearn.metrics import mean_squared_error\n\n# Predict and inverse-transform normalized predictions\ny_pred_scaled = model.predict({'seq_input': X_seq, 'pos_input': X_pos})\ny_pred = scaler_y.inverse_transform(y_pred_scaled)\n\n# Compare with real (unnormalized) targets\nmse = mean_squared_error(y, y_pred)\nprint(f\"Validation MSE (embedding model): {mse:.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:38:03.709437Z","iopub.execute_input":"2025-04-17T09:38:03.709847Z","iopub.status.idle":"2025-04-17T09:38:08.19556Z","shell.execute_reply.started":"2025-04-17T09:38:03.709816Z","shell.execute_reply":"2025-04-17T09:38:08.194804Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 📏 Step 2: Also log normalized MSE for comparison","metadata":{}},{"cell_type":"markdown","source":"### 📏 Evaluate in Normalized Space\n\n**Normalized MSE for Reference**\n\nTo fairly assess model quality, we also measured validation MSE in normalized space (before inverse-transform):\n\n```python\nmse_scaled = mean_squared_error(y_scaled, y_pred_scaled)","metadata":{}},{"cell_type":"code","source":"mse_scaled = mean_squared_error(y_scaled, y_pred_scaled)\nprint(f\"Validation MSE (scaled coordinates): {mse_scaled:.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:38:08.857073Z","iopub.execute_input":"2025-04-17T09:38:08.857407Z","iopub.status.idle":"2025-04-17T09:38:08.867019Z","shell.execute_reply.started":"2025-04-17T09:38:08.857384Z","shell.execute_reply":"2025-04-17T09:38:08.866088Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**🧾 Result: Validation MSE (scaled): 1.0208**\n\nThis confirms the model is learning well and errors are small relative to the normalized scale.","metadata":{}},{"cell_type":"markdown","source":"---\n\n### ⚖️ Switching to MinMaxScaler\n\n**⚖️ MinMaxScaler Attempt: Unexpected Explosion in Error**\n\nWe tried using `MinMaxScaler` to constrain output values between 0 and 1.\n\nHowever, when inverse-transforming predictions, we got:\n\n🧾 **Validation MSE: 726452.3677** ❌\n\n📌 This happened because:\n- The model predicted values **slightly outside [0, 1]**\n- `MinMaxScaler.inverse_transform()` extrapolated wildly, converting small overshoots into massive absolute errors","metadata":{}},{"cell_type":"code","source":"from sklearn.preprocessing import MinMaxScaler\n\nscaler_y = MinMaxScaler()\ny_all_scaled = scaler_y.fit_transform(y_all)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:38:14.051702Z","iopub.execute_input":"2025-04-17T09:38:14.052391Z","iopub.status.idle":"2025-04-17T09:38:14.060812Z","shell.execute_reply.started":"2025-04-17T09:38:14.05236Z","shell.execute_reply":"2025-04-17T09:38:14.060064Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from sklearn.metrics import mean_squared_error\n\n# Predict and inverse-transform normalized predictions\ny_pred_scaled = model.predict({'seq_input': X_seq, 'pos_input': X_pos})\ny_pred = scaler_y.inverse_transform(y_pred_scaled)\n\n# Compare with real (unnormalized) targets\nmse = mean_squared_error(y, y_pred)\nprint(f\"Validation MSE (embedding model): {mse:.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:38:14.478055Z","iopub.execute_input":"2025-04-17T09:38:14.478787Z","iopub.status.idle":"2025-04-17T09:38:19.513182Z","shell.execute_reply.started":"2025-04-17T09:38:14.478764Z","shell.execute_reply":"2025-04-17T09:38:19.511917Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### ✂️ Fix Attempt: Clipping Predictions","metadata":{}},{"cell_type":"markdown","source":"### ✂️ Clipping Before Inverse-Transform\n\nTo solve the issue above, we clipped predictions before converting back:\n\n```python\ny_pred_scaled_clipped = np.clip(y_pred_scaled, 0, 1)\ny_pred = scaler_y.inverse_transform(y_pred_scaled_clipped)","metadata":{}},{"cell_type":"code","source":"y_pred_scaled = model.predict({'seq_input': X_seq, 'pos_input': X_pos})\n\n# Clip to [0, 1] before inverse transform\ny_pred_scaled_clipped = np.clip(y_pred_scaled, 0, 1)\n\n# Inverse transform clipped predictions\ny_pred = scaler_y.inverse_transform(y_pred_scaled_clipped)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:38:23.342276Z","iopub.execute_input":"2025-04-17T09:38:23.342565Z","iopub.status.idle":"2025-04-17T09:38:27.920175Z","shell.execute_reply.started":"2025-04-17T09:38:23.342541Z","shell.execute_reply":"2025-04-17T09:38:27.919211Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from sklearn.metrics import mean_squared_error\n\n# Predict and inverse-transform normalized predictions\ny_pred_scaled = model.predict({'seq_input': X_seq, 'pos_input': X_pos})\ny_pred = scaler_y.inverse_transform(y_pred_scaled)\n\n# Compare with real (unnormalized) targets\nmse = mean_squared_error(y, y_pred)\nprint(f\"Validation MSE (embedding model): {mse:.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:38:27.921638Z","iopub.execute_input":"2025-04-17T09:38:27.921897Z","iopub.status.idle":"2025-04-17T09:38:32.464783Z","shell.execute_reply.started":"2025-04-17T09:38:27.921878Z","shell.execute_reply":"2025-04-17T09:38:32.463966Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**🧾 Result: Still Validation MSE: 726452.3677**\n\nClipping stopped values from exploding, but the model still didn’t generalize well under MinMax. This implies StandardScaler is more stable for this problem.","metadata":{}},{"cell_type":"markdown","source":"---\n\n### 📉 Switch to MAE: A More Stable Metric\n\n **📉 Using MAE to Reduce Outlier Sensitivity**\n\nWe then evaluated the model with `mean_absolute_error`:\n\n```python\nmae = mean_absolute_error(y, y_pred)","metadata":{}},{"cell_type":"code","source":"from sklearn.metrics import mean_absolute_error\nmae = mean_absolute_error(y, y_pred)\nprint(f\"Validation MAE (embedding + MinMax): {mae:.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:38:51.12386Z","iopub.execute_input":"2025-04-17T09:38:51.124515Z","iopub.status.idle":"2025-04-17T09:38:51.135387Z","shell.execute_reply.started":"2025-04-17T09:38:51.12449Z","shell.execute_reply":"2025-04-17T09:38:51.13451Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**🧾 Result: Validation MAE: 801.9457**\n\nThis gives a much clearer view of actual model performance — and avoids MSE being skewed by just a few bad predictions.","metadata":{}},{"cell_type":"code","source":"mse_scaled = mean_squared_error(y_all_scaled, y_pred_scaled)\nprint(f\"Validation MSE (normalized): {mse_scaled:.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:38:53.857946Z","iopub.execute_input":"2025-04-17T09:38:53.858352Z","iopub.status.idle":"2025-04-17T09:38:53.868465Z","shell.execute_reply.started":"2025-04-17T09:38:53.858325Z","shell.execute_reply":"2025-04-17T09:38:53.867659Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"---\n\n## ✅ Summary of What Happened and What to Do\n\n### What Caused the Huge MSE?\n\n- **Why it happened:** `MinMaxScaler` amplifies tiny errors into huge values if predictions fall outside [0, 1]\n- **What we tried:** Clipping, switching to MAE, normalized MSE\n- **What worked:** MAE and using `StandardScaler` or staying in normalized space\n- **What we learned:** Always track both **scaled and unscaled metrics**, and prefer MAE over MSE when coordinate ranges are large.\n\n📌 Final Decision:\nWe’ll stick with:\n- ✅ `StandardScaler` for stability\n- ✅ MAE + Normalized MSE for evaluation\n- ✅ Possibly predict relative coordinates (delta positions) in future models","metadata":{}},{"cell_type":"markdown","source":"### 🔡 Learned Embeddings: Richer Feature Input\n\nWe replaced one-hot vectors with trainable embeddings, allowing the model to learn abstract, continuous representations for each base.\nCombined with normalized positional input, the model now receives both identity and location signals.\n\nAfter training, the model achieved a validation MSE of 17317.1527 (unnormalized).\nThis technique unlocks potential for more complex sequence-based models like RNNs or Transformers.\n\n📌 Next step: try a Transformer encoder, or move to full sequence-based modeling with local attention or k-mer contexts.","metadata":{}},{"cell_type":"markdown","source":"### 📊 Initial Evaluation: MSE in Unnormalized Space\n\nWe evaluated the embedding model by inverse-transforming its predictions back to the real 3D coordinate space:","metadata":{}},{"cell_type":"code","source":"y_pred_scaled = model.predict({'seq_input': X_seq, 'pos_input': X_pos})\ny_pred = scaler_y.inverse_transform(y_pred_scaled)\nmse = mean_squared_error(y, y_pred)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:38:56.983242Z","iopub.execute_input":"2025-04-17T09:38:56.983547Z","iopub.status.idle":"2025-04-17T09:39:01.548127Z","shell.execute_reply.started":"2025-04-17T09:38:56.983529Z","shell.execute_reply":"2025-04-17T09:39:01.547386Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**🧾 Result: Validation MSE: 17317.1527**\n\nThis large value reflects that while the model learned well in normalized space, even small errors became large absolute differences when projected back to the full coordinate range.","metadata":{}},{"cell_type":"markdown","source":"### 📏 Normalized MSE for Reference\n\nTo fairly assess model quality, we also measured validation MSE in normalized space (before inverse-transform):","metadata":{}},{"cell_type":"code","source":"mse_scaled = mean_squared_error(y_scaled, y_pred_scaled)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:39:01.549628Z","iopub.execute_input":"2025-04-17T09:39:01.550379Z","iopub.status.idle":"2025-04-17T09:39:01.557931Z","shell.execute_reply.started":"2025-04-17T09:39:01.550354Z","shell.execute_reply":"2025-04-17T09:39:01.557206Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**🧾 Result: Validation MSE (scaled): 1.0208**\n\nThis confirms the model is learning well and errors are small relative to the normalized scale.","metadata":{"execution":{"iopub.status.busy":"2025-04-15T12:03:23.720291Z","iopub.execute_input":"2025-04-15T12:03:23.720628Z","iopub.status.idle":"2025-04-15T12:03:23.727453Z","shell.execute_reply.started":"2025-04-15T12:03:23.720603Z","shell.execute_reply":"2025-04-15T12:03:23.725886Z"}}},{"cell_type":"markdown","source":"### ⚖️ MinMaxScaler Attempt: Unexpected Explosion in Error\n\nWe tried using MinMaxScaler to constrain output values between 0 and 1.\n\nHowever, when inverse-transforming predictions, we got:\n🧾 Validation MSE: 726452.3677 ❌\n\n📌 This happened because:\n\nThe model predicted values slightly outside [0, 1]\n\nMinMaxScaler.inverse_transform() extrapolated wildly, converting small overshoots into massive absolute errors","metadata":{}},{"cell_type":"markdown","source":"### ✂️ Clipping Before Inverse-Transform\n\nTo solve the issue above, we clipped predictions before converting back:","metadata":{}},{"cell_type":"markdown","source":"To solve the issue above, we clipped predictions before converting back:","metadata":{}},{"cell_type":"code","source":"_pred_scaled_clipped = np.clip(y_pred_scaled, 0, 1)\ny_pred = scaler_y.inverse_transform(y_pred_scaled_clipped)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:39:03.808517Z","iopub.execute_input":"2025-04-17T09:39:03.808844Z","iopub.status.idle":"2025-04-17T09:39:03.819991Z","shell.execute_reply.started":"2025-04-17T09:39:03.808819Z","shell.execute_reply":"2025-04-17T09:39:03.819216Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**🧾 Result: Still Validation MSE: 726452.3677**\n\nClipping stopped values from exploding, but the model still didn’t generalize well under MinMax. This implies StandardScaler is more stable for this problem.\n\n","metadata":{}},{"cell_type":"markdown","source":"### 📉 Using MAE to Reduce Outlier Sensitivity\n\nWe then evaluated the model with mean_absolute_error:","metadata":{}},{"cell_type":"code","source":"mae = mean_absolute_error(y, y_pred)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:39:07.558191Z","iopub.execute_input":"2025-04-17T09:39:07.558933Z","iopub.status.idle":"2025-04-17T09:39:07.566946Z","shell.execute_reply.started":"2025-04-17T09:39:07.558907Z","shell.execute_reply":"2025-04-17T09:39:07.566217Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**🧾 Result: Validation MAE: 801.9457**\n\nThis gives a much clearer view of actual model performance — and avoids MSE being skewed by just a few bad predictions.\n\nAlso re-logged normalized performance:","metadata":{}},{"cell_type":"code","source":"mse_scaled = mean_squared_error(y_all_scaled, y_pred_scaled)\nprint(f\"Validation MSE (normalized): {mse_scaled:.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:39:08.37724Z","iopub.execute_input":"2025-04-17T09:39:08.377552Z","iopub.status.idle":"2025-04-17T09:39:08.38948Z","shell.execute_reply.started":"2025-04-17T09:39:08.37753Z","shell.execute_reply":"2025-04-17T09:39:08.388569Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"___\n### 🔢 Next Step: K-mer Encoding for Local Context\n\nWe are now extending our feature encoding by using k-mers (subsequences of length k). This allows the model to learn from surrounding bases, not just the current one.","metadata":{}},{"cell_type":"markdown","source":"### 🧬 Step 1: K-mer Generation Function","metadata":{}},{"cell_type":"code","source":"def kmer_encode_sequence(seq, k=3):\n    pad = 'X' * (k // 2)\n    padded_seq = pad + seq + pad\n    tokens = [padded_seq[i:i+k] for i in range(len(seq))]\n    return tokens","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:39:10.910035Z","iopub.execute_input":"2025-04-17T09:39:10.910643Z","iopub.status.idle":"2025-04-17T09:39:10.915183Z","shell.execute_reply.started":"2025-04-17T09:39:10.910618Z","shell.execute_reply":"2025-04-17T09:39:10.91433Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🔡 Step 2: Build K-mer Vocabulary","metadata":{}},{"cell_type":"code","source":"from itertools import product\nbases = ['A', 'U', 'C', 'G', 'X']\nk = 3\nkmer_vocab = [''.join(p) for p in product(bases, repeat=k)]\nkmer_to_index = {kmer: idx for idx, kmer in enumerate(kmer_vocab)}\nvocab_size = len(kmer_to_index)  # 125 for k=3","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:39:13.558016Z","iopub.execute_input":"2025-04-17T09:39:13.558823Z","iopub.status.idle":"2025-04-17T09:39:13.563746Z","shell.execute_reply.started":"2025-04-17T09:39:13.558799Z","shell.execute_reply":"2025-04-17T09:39:13.562732Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🔁 Step 3: Encode Sequences into K-mer Indices","metadata":{}},{"cell_type":"code","source":"encoded_kmers = []\npositional_inputs = []\ntargets = []\n\nfor _, row in train_sequences.iterrows():\n    tid = row['target_id']\n    seq = row['sequence']\n    label_subset = train_labels[train_labels['target_id'] == tid]\n    \n    if len(seq) != len(label_subset): continue\n    if label_subset[['x_1', 'y_1', 'z_1']].isnull().any().any(): continue\n\n    kmers = kmer_encode_sequence(seq, k)\n    kmer_ints = [kmer_to_index[kmer] for kmer in kmers]\n    pos = np.arange(len(seq)) / (len(seq) - 1)\n    y_coords = label_subset[['x_1', 'y_1', 'z_1']].values\n\n    encoded_kmers.extend(kmer_ints)\n    positional_inputs.extend(pos)\n    targets.extend(y_coords)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:39:16.187062Z","iopub.execute_input":"2025-04-17T09:39:16.187744Z","iopub.status.idle":"2025-04-17T09:39:26.05399Z","shell.execute_reply.started":"2025-04-17T09:39:16.187712Z","shell.execute_reply":"2025-04-17T09:39:26.053297Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 📐 Step 4: Format for Model Training","metadata":{}},{"cell_type":"code","source":"X_kmer = np.array(encoded_kmers).reshape(-1, 1)\nX_pos = np.array(positional_inputs).reshape(-1, 1)\ny = np.array(targets)\n\nfrom sklearn.preprocessing import StandardScaler\nscaler_y = StandardScaler()\ny_scaled = scaler_y.fit_transform(y)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:39:34.978119Z","iopub.execute_input":"2025-04-17T09:39:34.978484Z","iopub.status.idle":"2025-04-17T09:39:35.056082Z","shell.execute_reply.started":"2025-04-17T09:39:34.978461Z","shell.execute_reply":"2025-04-17T09:39:35.055295Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🧠 Step 5: Define Model (K-mer + Positional Input)","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.models import Model\nfrom tensorflow.keras.layers import Input, Embedding, Concatenate, Dense, Flatten\n\nkmer_input = Input(shape=(1,), name='kmer_input')\nkmer_embed = Embedding(input_dim=vocab_size, output_dim=16)(kmer_input)\nkmer_embed = Flatten()(kmer_embed)\n\npos_input = Input(shape=(1,), name='pos_input')\n\nx = Concatenate()([kmer_embed, pos_input])\nx = Dense(64, activation='relu')(x)\nx = Dense(32, activation='relu')(x)\noutput = Dense(3)(x)\n\nmodel = Model(inputs=[kmer_input, pos_input], outputs=output)\nmodel.compile(optimizer='adam', loss='mse')\nmodel.summary()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:39:38.247532Z","iopub.execute_input":"2025-04-17T09:39:38.247869Z","iopub.status.idle":"2025-04-17T09:39:38.306898Z","shell.execute_reply.started":"2025-04-17T09:39:38.247843Z","shell.execute_reply":"2025-04-17T09:39:38.306237Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🚀 Step 6: Train the Model","metadata":{}},{"cell_type":"code","source":"model.fit(\n    x={'kmer_input': X_kmer, 'pos_input': X_pos},\n    y=y_scaled,\n    validation_split=0.2,\n    epochs=10,\n    batch_size=128,\n    verbose=2\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:39:44.118651Z","iopub.execute_input":"2025-04-17T09:39:44.11924Z","iopub.status.idle":"2025-04-17T09:39:56.216739Z","shell.execute_reply.started":"2025-04-17T09:39:44.119206Z","shell.execute_reply":"2025-04-17T09:39:56.215878Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🔬 K-mer Model (k=3) Performance Review\n\n- We trained a model using 3-mer token embedding + positional input.\n- Final training loss: `0.9848`  \n- Final validation loss: `1.1542`\n- While training continued improving, validation loss plateaued early, suggesting **underfitting** or **input limitations**.\n\n📌 Next steps:\n- Increase k-mer size (k=5) or embedding dimension\n- Add regularization (dropout)\n- Or build a full sequence-level model using Transformer or GRU","metadata":{}},{"cell_type":"markdown","source":"### 🔁 1. Try a Larger K-mer or Embedding Dimension","metadata":{}},{"cell_type":"code","source":"k = 5  # change k-mer generator\nEmbedding(input_dim=vocab_size, output_dim=32)  # more expressive","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:40:05.838995Z","iopub.execute_input":"2025-04-17T09:40:05.839332Z","iopub.status.idle":"2025-04-17T09:40:05.846271Z","shell.execute_reply.started":"2025-04-17T09:40:05.839309Z","shell.execute_reply":"2025-04-17T09:40:05.845375Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🔄 2. Add a Dropout Layer (Prevent Overfitting)","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.layers import Dropout\n\nx = Concatenate()([kmer_embed, pos_input])\nx = Dense(64, activation='relu')(x)\nx = Dropout(0.3)(x)  # <- new line\nx = Dense(32, activation='relu')(x)\noutput = Dense(3)(x)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:40:07.928017Z","iopub.execute_input":"2025-04-17T09:40:07.928815Z","iopub.status.idle":"2025-04-17T09:40:07.958597Z","shell.execute_reply.started":"2025-04-17T09:40:07.928788Z","shell.execute_reply":"2025-04-17T09:40:07.957947Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"model.fit(\n    x={'kmer_input': X_kmer, 'pos_input': X_pos},\n    y=y_scaled,\n    validation_split=0.2,\n    epochs=10,\n    batch_size=128,\n    verbose=2\n\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:40:08.618406Z","iopub.execute_input":"2025-04-17T09:40:08.618734Z","iopub.status.idle":"2025-04-17T09:40:18.7918Z","shell.execute_reply.started":"2025-04-17T09:40:08.618699Z","shell.execute_reply":"2025-04-17T09:40:18.791078Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"____\n##  Bidirectional GRU","metadata":{}},{"cell_type":"markdown","source":"Now we are aiming to build a model (GRU or Transformer) that takes a full RNA sequence and predicts the 3D coordinates for every base — all at once.\n","metadata":{}},{"cell_type":"markdown","source":"### ✅ Step-by-Step Plan:\n\n1. Prepare Input/Output as Full Sequences\n\nWe reshape your data so that:\n\t•\tX: [n_samples, sequence_length] = integer-encoded k-mers\n\t•\ty: [n_samples, sequence_length, 3] = 3D coords per base","metadata":{}},{"cell_type":"markdown","source":"### 📦 Step 1: Sequence Packing Function","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.preprocessing.sequence import pad_sequences\n\ndef prepare_sequence_dataset(k=3):\n    input_seqs = []\n    pos_seqs = []\n    output_seqs = []\n\n    for _, row in train_sequences.iterrows():\n        tid = row['target_id']\n        seq = row['sequence']\n        labels = train_labels[train_labels['target_id'] == tid]\n\n        if len(seq) != len(labels): continue\n        if labels[['x_1', 'y_1', 'z_1']].isnull().any().any(): continue\n\n        kmers = kmer_encode_sequence(seq, k)\n        kmer_ids = [kmer_to_index[kmer] for kmer in kmers]\n        positions = np.arange(len(seq)) / (len(seq) - 1)\n        coords = labels[['x_1', 'y_1', 'z_1']].values\n\n        input_seqs.append(kmer_ids)\n        pos_seqs.append(positions)\n        output_seqs.append(coords)\n\n    # Pad all sequences to the same length\n    X_kmer = pad_sequences(input_seqs, padding='post', value=0)\n    X_pos = pad_sequences(pos_seqs, padding='post', value=0.0, dtype='float32')\n    \n    # Pad output 3D coordinates per base\n    max_len = max(len(seq) for seq in input_seqs)\n    y_padded = np.zeros((len(output_seqs), max_len, 3))\n    for i, coords in enumerate(output_seqs):\n        y_padded[i, :len(coords), :] = coords\n\n    return X_kmer, X_pos, y_padded","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:40:35.058125Z","iopub.execute_input":"2025-04-17T09:40:35.059081Z","iopub.status.idle":"2025-04-17T09:40:35.068092Z","shell.execute_reply.started":"2025-04-17T09:40:35.059051Z","shell.execute_reply":"2025-04-17T09:40:35.067285Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 📐 Step 2: Normalize Coordinates","metadata":{}},{"cell_type":"code","source":"X_kmer, X_pos, y_coords = prepare_sequence_dataset(k=3)\n\n# Normalize output\nfrom sklearn.preprocessing import StandardScaler\nscaler_y = StandardScaler()\ny_coords_flat = y_coords.reshape(-1, 3)\ny_coords_scaled = scaler_y.fit_transform(y_coords_flat).reshape(y_coords.shape)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:40:37.767087Z","iopub.execute_input":"2025-04-17T09:40:37.767386Z","iopub.status.idle":"2025-04-17T09:40:48.026284Z","shell.execute_reply.started":"2025-04-17T09:40:37.767365Z","shell.execute_reply":"2025-04-17T09:40:48.025319Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🧠 Step 3: Build Sequence Model (GRU-based)","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.models import Model\nfrom tensorflow.keras.layers import Input, Embedding, GRU, Concatenate, Dense, TimeDistributed\n\nvocab_size = len(kmer_to_index)\n\n# Inputs\nseq_input = Input(shape=(None,), name='kmer_input')  # [batch, seq_len]\npos_input = Input(shape=(None, 1), name='pos_input')  # [batch, seq_len, 1]\n\n# Embedding\nx_embed = Embedding(input_dim=vocab_size, output_dim=32)(seq_input)  # [batch, seq_len, 32]\nx = Concatenate()([x_embed, pos_input])  # [batch, seq_len, 33]\n\n# GRU Layer\nx = GRU(64, return_sequences=True)(x)\nx = GRU(32, return_sequences=True)(x)\n\n# Output Layer\noutput = TimeDistributed(Dense(3))(x)\n\nmodel = Model(inputs=[seq_input, pos_input], outputs=output)\nmodel.compile(optimizer='adam', loss='mse')\nmodel.summary()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:40:48.027503Z","iopub.execute_input":"2025-04-17T09:40:48.027794Z","iopub.status.idle":"2025-04-17T09:40:48.109318Z","shell.execute_reply.started":"2025-04-17T09:40:48.027766Z","shell.execute_reply":"2025-04-17T09:40:48.108612Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"def prepare_sequence_dataset(k=3, max_len=100):\n    input_seqs = []\n    pos_seqs = []\n    output_seqs = []\n\n    for _, row in train_sequences.iterrows():\n        tid = row['target_id']\n        seq = row['sequence']\n        if len(seq) > max_len: continue  # new filter\n\n        labels = train_labels[train_labels['target_id'] == tid]\n        if len(seq) != len(labels): continue\n        if labels[['x_1', 'y_1', 'z_1']].isnull().any().any(): continue\n\n        kmers = kmer_encode_sequence(seq, k)\n        kmer_ids = [kmer_to_index[kmer] for kmer in kmers]\n        positions = np.arange(len(seq)) / (len(seq) - 1)\n        coords = labels[['x_1', 'y_1', 'z_1']].values\n\n        input_seqs.append(kmer_ids)\n        pos_seqs.append(positions)\n        output_seqs.append(coords)\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:43:25.968122Z","iopub.execute_input":"2025-04-17T09:43:25.968955Z","iopub.status.idle":"2025-04-17T09:43:25.975613Z","shell.execute_reply.started":"2025-04-17T09:43:25.968926Z","shell.execute_reply":"2025-04-17T09:43:25.974628Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🚀 Step 4: Train the Model","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.preprocessing.sequence import pad_sequences\n\ndef prepare_sequence_dataset(k=3, max_len=100):\n    input_seqs = []\n    pos_seqs = []\n    output_seqs = []\n\n    for _, row in train_sequences.iterrows():\n        tid = row['target_id']\n        seq = row['sequence']\n        if len(seq) > max_len:\n            continue  # Skip sequences that are too long\n\n        labels = train_labels[train_labels['target_id'] == tid]\n        if len(seq) != len(labels): continue\n        if labels[['x_1', 'y_1', 'z_1']].isnull().any().any(): continue\n\n        kmers = kmer_encode_sequence(seq, k)\n        kmer_ids = [kmer_to_index[kmer] for kmer in kmers]\n        positions = np.arange(len(seq)) / (len(seq) - 1)\n        coords = labels[['x_1', 'y_1', 'z_1']].values\n\n        input_seqs.append(kmer_ids)\n        pos_seqs.append(positions)\n        output_seqs.append(coords)\n\n    if len(input_seqs) == 0:\n        print(\"No sequences met the filtering criteria.\")\n        return None\n\n    # Pad sequences to max length\n    X_kmer = pad_sequences(input_seqs, padding='post', value=0)\n    X_pos = pad_sequences(pos_seqs, padding='post', value=0.0, dtype='float32')\n\n    # Pad targets (3D coordinates)\n    y_padded = np.zeros((len(output_seqs), max_len, 3))\n    for i, coords in enumerate(output_seqs):\n        y_padded[i, :len(coords), :] = coords\n\n    return X_kmer, X_pos, y_padded","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:45:08.933572Z","iopub.execute_input":"2025-04-17T09:45:08.933893Z","iopub.status.idle":"2025-04-17T09:45:08.942856Z","shell.execute_reply.started":"2025-04-17T09:45:08.933868Z","shell.execute_reply":"2025-04-17T09:45:08.942031Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"X_kmer, X_pos, y_coords = prepare_sequence_dataset(k=3, max_len=100)\n\nfrom sklearn.preprocessing import StandardScaler\nscaler_y = StandardScaler()\ny_coords_scaled = scaler_y.fit_transform(y_coords.reshape(-1, 3)).reshape(y_coords.shape)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:45:17.137811Z","iopub.execute_input":"2025-04-17T09:45:17.138623Z","iopub.status.idle":"2025-04-17T09:45:25.065168Z","shell.execute_reply.started":"2025-04-17T09:45:17.13859Z","shell.execute_reply":"2025-04-17T09:45:25.064413Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"from tensorflow.keras.models import Model\nfrom tensorflow.keras.layers import Input, Embedding, GRU, Concatenate, Dense, TimeDistributed\n\nvocab_size = len(kmer_to_index)\n\n# Inputs\nseq_input = Input(shape=(None,), name='kmer_input')\npos_input = Input(shape=(None, 1), name='pos_input')\n\n# Embedding + Positional input\nembed = Embedding(input_dim=vocab_size, output_dim=32)(seq_input)\nx = Concatenate()([embed, pos_input])\n\n# GRU stack\nx = GRU(64, return_sequences=True)(x)\nx = GRU(32, return_sequences=True)(x)\n\n# Output: 3D coordinates for each base\noutput = TimeDistributed(Dense(3))(x)\n\n# Compile model\nmodel = Model(inputs=[seq_input, pos_input], outputs=output)\nmodel.compile(optimizer='adam', loss='mse')\nmodel.summary()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:45:57.917853Z","iopub.execute_input":"2025-04-17T09:45:57.918184Z","iopub.status.idle":"2025-04-17T09:45:57.985669Z","shell.execute_reply.started":"2025-04-17T09:45:57.918134Z","shell.execute_reply":"2025-04-17T09:45:57.985008Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"history = model.fit(\n    x={'kmer_input': X_kmer, 'pos_input': X_pos[..., np.newaxis]},\n    y=y_coords_scaled,\n    validation_split=0.2,\n    epochs=10,\n    batch_size=4,  # small to avoid crashing\n    verbose=2\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:46:20.808907Z","iopub.execute_input":"2025-04-17T09:46:20.809281Z","iopub.status.idle":"2025-04-17T09:47:29.765084Z","shell.execute_reply.started":"2025-04-17T09:46:20.809255Z","shell.execute_reply":"2025-04-17T09:47:29.763954Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**To reduce val loss oscillation we can:**","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.layers import Dropout\n\nx = GRU(64, return_sequences=True)(x)\nx = Dropout(0.3)(x)\nx = GRU(32, return_sequences=True)(x)\nx = Dropout(0.3)(x)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:57:09.503032Z","iopub.execute_input":"2025-04-17T09:57:09.503978Z","iopub.status.idle":"2025-04-17T09:57:09.548429Z","shell.execute_reply.started":"2025-04-17T09:57:09.50394Z","shell.execute_reply":"2025-04-17T09:57:09.547651Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Now need to switch to Relative (Δ) Coordinate Prediction**\n\n\nInstead of predicting full x/y/z, predict coordinate differences from previous base.\nWhy? RNA structure is spatially smooth — predicting movement is easier than position.","metadata":{}},{"cell_type":"code","source":"# Convert absolute coords to deltas before training\ny_deltas = np.diff(y_coords, axis=1, prepend=0)  # shape: (samples, seq_len, 3)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:57:50.913524Z","iopub.execute_input":"2025-04-17T09:57:50.914408Z","iopub.status.idle":"2025-04-17T09:57:50.919628Z","shell.execute_reply.started":"2025-04-17T09:57:50.91438Z","shell.execute_reply":"2025-04-17T09:57:50.918721Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**Now we are adding Attention (Self-Aware GRU;**\n\nThis will let the GRU focus on specific past/future bases, like RNA loops.","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.layers import Attention, LayerNormalization\n\n# Simple attention block\nattention = Attention()([x, x])  # self-attention\nx = LayerNormalization()(x + attention)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T09:58:30.057881Z","iopub.execute_input":"2025-04-17T09:58:30.058569Z","iopub.status.idle":"2025-04-17T09:58:30.072555Z","shell.execute_reply.started":"2025-04-17T09:58:30.058538Z","shell.execute_reply":"2025-04-17T09:58:30.071605Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"___\n## Transformer — Build a transformer-based sequence model","metadata":{}},{"cell_type":"markdown","source":"### Phase 1: Prep Input (We’re Already Ready)\n\nWe already have:\n\t•\tX_kmer → shape [N, L] (integer k-mer tokens)\n\t•\tX_pos → shape [N, L] (position per token)\n\t•\ty_coords_scaled → shape [N, L, 3]\n\n✅ Perfect input shape for Transformer!","metadata":{}},{"cell_type":"markdown","source":"### Phase 2: Build the Transformer Encoder Model","metadata":{}},{"cell_type":"code","source":"import tensorflow as tf\nfrom tensorflow.keras.layers import Input, Embedding, Dense, Dropout, LayerNormalization, MultiHeadAttention, Add, Concatenate, TimeDistributed\nfrom tensorflow.keras.models import Model\n\ndef transformer_encoder(inputs, head_size=64, num_heads=4, ff_dim=128, dropout=0.1):\n    # Multi-head self-attention\n    attention = MultiHeadAttention(key_dim=head_size, num_heads=num_heads)(inputs, inputs)\n    attention = Dropout(dropout)(attention)\n    x = Add()([inputs, attention])\n    x = LayerNormalization()(x)\n\n    # Feed-forward\n    ff = Dense(ff_dim, activation='relu')(x)\n    ff = Dropout(dropout)(ff)\n    ff = Dense(inputs.shape[-1])(ff)\n    x = Add()([x, ff])\n    return LayerNormalization()(x)\n\n# Inputs\nseq_input = Input(shape=(None,), name=\"kmer_input\")  # [batch, seq_len]\npos_input = Input(shape=(None, 1), name=\"pos_input\")  # [batch, seq_len, 1]\n\n# Embedding\nembed_dim = 64\nvocab_size = len(kmer_to_index)\nx_embed = Embedding(input_dim=vocab_size, output_dim=embed_dim)(seq_input)\n\n# Concatenate k-mer embedding with positional input\nx = Concatenate()([x_embed, pos_input])  # [batch, seq_len, embed_dim+1]\n\n# Transformer Encoder Block\nx = transformer_encoder(x, head_size=64, num_heads=4, ff_dim=128, dropout=0.1)\n\n# Output: predict x/y/z for each base\noutput = TimeDistributed(Dense(3))(x)\n\n# Compile model\nmodel = Model(inputs=[seq_input, pos_input], outputs=output)\nmodel.compile(optimizer='adam', loss='mse')\nmodel.summary()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T10:04:22.563519Z","iopub.execute_input":"2025-04-17T10:04:22.564327Z","iopub.status.idle":"2025-04-17T10:04:22.675301Z","shell.execute_reply.started":"2025-04-17T10:04:22.564298Z","shell.execute_reply":"2025-04-17T10:04:22.674641Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🚀 Phase 3: Train the Model","metadata":{}},{"cell_type":"code","source":"history = model.fit(\n    x={'kmer_input': X_kmer, 'pos_input': X_pos[..., np.newaxis]},\n    y=y_coords_scaled,\n    validation_split=0.2,\n    epochs=10,\n    batch_size=4,\n    verbose=2\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T10:04:48.193695Z","iopub.execute_input":"2025-04-17T10:04:48.19401Z","iopub.status.idle":"2025-04-17T10:05:19.852136Z","shell.execute_reply.started":"2025-04-17T10:04:48.193988Z","shell.execute_reply":"2025-04-17T10:05:19.851456Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Evaluate Performance (MSE / MAE)","metadata":{}},{"cell_type":"code","source":"# Predict\ny_pred_scaled = model.predict({'kmer_input': X_kmer, 'pos_input': X_pos[..., np.newaxis]})\ny_pred = scaler_y.inverse_transform(y_pred_scaled.reshape(-1, 3)).reshape(y_coords.shape)\n\n# Mask padded positions\nmask = (X_pos.flatten() != 0)\ny_pred_flat = y_pred.reshape(-1, 3)\ny_true_flat = y_coords.reshape(-1, 3)\n\ny_pred_masked = y_pred_flat[mask]\ny_true_masked = y_true_flat[mask]\n\n# Metrics\nfrom sklearn.metrics import mean_squared_error, mean_absolute_error\nmse = mean_squared_error(y_true_masked, y_pred_masked)\nmae = mean_absolute_error(y_true_masked, y_pred_masked)\n\nprint(f\"✅ Transformer Validation MSE: {mse:.4f}\")\nprint(f\"✅ Transformer Validation MAE: {mae:.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T10:06:25.783456Z","iopub.execute_input":"2025-04-17T10:06:25.783785Z","iopub.status.idle":"2025-04-17T10:06:29.839399Z","shell.execute_reply.started":"2025-04-17T10:06:25.78376Z","shell.execute_reply":"2025-04-17T10:06:29.838655Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🧠 Transformer-Based Sequence Model\n\nWe built a lightweight Transformer encoder to model full RNA sequences using k-mer + positional inputs.\n\n**Architecture:**\n- Embedding + Scalar Position Input\n- 1 Transformer block (MultiHeadAttention + FFN)\n- TimeDistributed Dense(3) for per-base 3D prediction\n\n**Training Metrics:**\n- Final Train Loss: 0.5723\n- Final Validation Loss: 1.8124\n\n**Evaluation on Unnormalized Real Coordinates:**\n- ✅ MSE: 6846.0012\n- ✅ MAE: 51.2495\n\n📌 This model outperformed all previous versions, showing that even a small Transformer architecture can significantly boost structural prediction accuracy. Future improvements could include learned positional embeddings, more encoder layers, or hybrid GRU-Transformer stacks.","metadata":{}},{"cell_type":"markdown","source":"### Switch to Learned Positional Embedding","metadata":{}},{"cell_type":"markdown","source":"### 📦 1. Prepare Positional IDs\n\nInstead of passing a scalar for each position, we pass integers 0, 1, ..., seq_len - 1 per sequence:","metadata":{}},{"cell_type":"code","source":"# Create position IDs (integers) with the same shape as your sequences\nX_pos_ids = np.tile(np.arange(X_kmer.shape[1]), (X_kmer.shape[0], 1))  # shape: (491, 100)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T10:08:24.843919Z","iopub.execute_input":"2025-04-17T10:08:24.844262Z","iopub.status.idle":"2025-04-17T10:08:24.848948Z","shell.execute_reply.started":"2025-04-17T10:08:24.844236Z","shell.execute_reply":"2025-04-17T10:08:24.847929Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🧠 2. Updated Transformer Model (K-mer + Pos Embedding)","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.layers import Input, Embedding, Dense, Dropout, LayerNormalization, MultiHeadAttention, Add, TimeDistributed, Concatenate\nfrom tensorflow.keras.models import Model\n\ndef transformer_encoder(inputs, head_size=64, num_heads=4, ff_dim=128, dropout=0.1):\n    attention = MultiHeadAttention(key_dim=head_size, num_heads=num_heads)(inputs, inputs)\n    attention = Dropout(dropout)(attention)\n    x = Add()([inputs, attention])\n    x = LayerNormalization()(x)\n\n    ff = Dense(ff_dim, activation='relu')(x)\n    ff = Dropout(dropout)(ff)\n    ff = Dense(inputs.shape[-1])(ff)\n    x = Add()([x, ff])\n    return LayerNormalization()(x)\n\n# Inputs\nkmer_input = Input(shape=(None,), name=\"kmer_input\")        # [batch, seq_len]\npos_id_input = Input(shape=(None,), name=\"pos_id_input\")    # [batch, seq_len]\n\n# Embedding layers\nkmer_embed = Embedding(input_dim=len(kmer_to_index), output_dim=64)(kmer_input)\npos_embed = Embedding(input_dim=200, output_dim=16)(pos_id_input)  # 200 > max_len\n\n# Combine embeddings\nx = Concatenate()([kmer_embed, pos_embed])  # shape: [batch, seq_len, 80]\n\n# Transformer Encoder\nx = transformer_encoder(x, head_size=64, num_heads=4, ff_dim=128)\n\n# Output 3D coords\noutput = TimeDistributed(Dense(3))(x)\n\n# Model\nmodel = Model(inputs=[kmer_input, pos_id_input], outputs=output)\nmodel.compile(optimizer='adam', loss='mse')\nmodel.summary()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T10:08:48.363696Z","iopub.execute_input":"2025-04-17T10:08:48.364036Z","iopub.status.idle":"2025-04-17T10:08:48.482795Z","shell.execute_reply.started":"2025-04-17T10:08:48.364012Z","shell.execute_reply":"2025-04-17T10:08:48.482093Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🚀 3. Time to Train!","metadata":{}},{"cell_type":"code","source":"history = model.fit(\n    x={'kmer_input': X_kmer, 'pos_id_input': X_pos_ids},\n    y=y_coords_scaled,\n    validation_split=0.2,\n    epochs=10,\n    batch_size=4,\n    verbose=2\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T10:09:11.303443Z","iopub.execute_input":"2025-04-17T10:09:11.303749Z","iopub.status.idle":"2025-04-17T10:09:44.533634Z","shell.execute_reply.started":"2025-04-17T10:09:11.303713Z","shell.execute_reply":"2025-04-17T10:09:44.532633Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 📊 4. Evaluate ","metadata":{}},{"cell_type":"code","source":"from sklearn.metrics import mean_squared_error, mean_absolute_error\n\n# === Step 1: Predict on the full training data ===\ny_pred_scaled = model.predict({\n    'kmer_input': X_kmer,\n    'pos_id_input': X_pos_ids\n})\n\n# === Step 2: Inverse scale predictions back to real 3D space ===\ny_pred = scaler_y.inverse_transform(\n    y_pred_scaled.reshape(-1, 3)\n).reshape(y_coords.shape)\n\n# === Step 3: Mask out padding (zeros in X_pos_ids) ===\nflat_mask = (X_pos_ids.flatten() != 0)\n\n# Flatten predictions and true labels\ny_pred_flat = y_pred.reshape(-1, 3)\ny_true_flat = y_coords.reshape(-1, 3)\n\n# Apply mask to remove padded values\ny_pred_masked = y_pred_flat[flat_mask]\ny_true_masked = y_true_flat[flat_mask]\n\n# === Step 4: Calculate Metrics ===\nmse = mean_squared_error(y_true_masked, y_pred_masked)\nmae = mean_absolute_error(y_true_masked, y_pred_masked)\n\nprint(f\"✅ Transformer+PositionEmbedding Validation MSE: {mse:.4f}\")\nprint(f\"✅ Transformer+PositionEmbedding Validation MAE: {mae:.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T10:11:58.318597Z","iopub.execute_input":"2025-04-17T10:11:58.318991Z","iopub.status.idle":"2025-04-17T10:12:02.656705Z","shell.execute_reply.started":"2025-04-17T10:11:58.318965Z","shell.execute_reply":"2025-04-17T10:12:02.655719Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🧠 Transformer + Learned Positional Embeddings\n\nWe upgraded our Transformer model to use **trainable position embeddings** instead of scalar float values.\n\nEach position (0 to 99) gets its own vector, allowing the model to learn nuanced spatial patterns over the sequence.\n\n📈 Final Training Loss: 0.5850  \n📉 Validation Loss: 2.1991\n\n**Evaluation on Real (Unnormalized) 3D Coordinates:**\n- ✅ MSE: 2978.9847\n- ✅ MAE: 24.9423\n\n📌 This model outperformed all others significantly, confirming that learned embeddings can dramatically boost performance when combined with attention.","metadata":{}},{"cell_type":"markdown","source":"## 🧪 PLAN: Execute All 3 Killer Paths\n","metadata":{}},{"cell_type":"markdown","source":"We are going to train the model to predict movement between bases, instead of absolute positions.\n","metadata":{}},{"cell_type":"markdown","source":"### ✅ Prep Target Deltas:","metadata":{}},{"cell_type":"code","source":"# Compute deltas from coordinates (y[t] - y[t-1])\ndelta_coords = np.diff(y_coords, axis=1, prepend=0)  # shape: (N, L, 3)\n\n# Scale deltas (to keep training stable)\nfrom sklearn.preprocessing import StandardScaler\nscaler_delta = StandardScaler()\ndelta_coords_scaled = scaler_delta.fit_transform(delta_coords.reshape(-1, 3)).reshape(delta_coords.shape)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T10:15:09.303296Z","iopub.execute_input":"2025-04-17T10:15:09.303692Z","iopub.status.idle":"2025-04-17T10:15:09.318644Z","shell.execute_reply.started":"2025-04-17T10:15:09.303664Z","shell.execute_reply":"2025-04-17T10:15:09.317747Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"✅ We are using the same Transformer+PosEmbedding model, but train on delta_coords_scaled instead of y_coords_scaled.\n","metadata":{}},{"cell_type":"markdown","source":"### 🔁 Step-by-Step: Delta Coordinate Model\n\n**✅ Step 1: Create Δ Coordinates (Target Differences)**","metadata":{}},{"cell_type":"code","source":"import numpy as np\nfrom sklearn.preprocessing import StandardScaler\n\n# Compute delta coordinates (y_t - y_{t-1})\ndelta_coords = np.diff(y_coords, axis=1, prepend=0)  # shape: (samples, seq_len, 3)\n\n# Scale the deltas for training stability\nscaler_delta = StandardScaler()\ndelta_coords_scaled = scaler_delta.fit_transform(delta_coords.reshape(-1, 3)).reshape(delta_coords.shape)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T10:17:56.288039Z","iopub.execute_input":"2025-04-17T10:17:56.288413Z","iopub.status.idle":"2025-04-17T10:17:56.30608Z","shell.execute_reply.started":"2025-04-17T10:17:56.288387Z","shell.execute_reply":"2025-04-17T10:17:56.305096Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**✅ Step 2: Reuse Your Best Transformer Model**\n\nWe’ll keep the learned positional embeddings model (your best performer) — just switch the target to delta_coords_scaled.\n\n📌 Recap of Inputs:","metadata":{}},{"cell_type":"code","source":"X_kmer       # shape: (N, L)\nX_pos_ids    # shape: (N, L)\ndelta_coords_scaled  # shape: (N, L, 3)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T10:19:09.473321Z","iopub.execute_input":"2025-04-17T10:19:09.473733Z","iopub.status.idle":"2025-04-17T10:19:09.481524Z","shell.execute_reply.started":"2025-04-17T10:19:09.47368Z","shell.execute_reply":"2025-04-17T10:19:09.480611Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**✅ Step 3: Train the Model**","metadata":{}},{"cell_type":"code","source":"history = model.fit(\n    x={'kmer_input': X_kmer, 'pos_id_input': X_pos_ids},\n    y=delta_coords_scaled,\n    validation_split=0.2,\n    epochs=10,\n    batch_size=4,\n    verbose=2\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T10:20:05.608009Z","iopub.execute_input":"2025-04-17T10:20:05.608346Z","iopub.status.idle":"2025-04-17T10:20:24.412817Z","shell.execute_reply.started":"2025-04-17T10:20:05.608322Z","shell.execute_reply":"2025-04-17T10:20:24.412101Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**✅ Step 4: Evaluate on Real 3D Positions**\n\nWe’ll predict deltas, inverse-transform them, and reconstruct absolute positions:","metadata":{}},{"cell_type":"code","source":"# Predict deltas (scaled)\ndelta_pred_scaled = model.predict({'kmer_input': X_kmer, 'pos_id_input': X_pos_ids})\n\n# Inverse transform to real delta values\ndelta_pred = scaler_delta.inverse_transform(\n    delta_pred_scaled.reshape(-1, 3)\n).reshape(delta_coords.shape)\n\n# Reconstruct full positions from deltas\ny_pred = np.cumsum(delta_pred, axis=1)  # like integrating over deltas\n\n# Flatten and mask\nmask = (X_pos_ids.flatten() != 0)\ny_pred_flat = y_pred.reshape(-1, 3)\ny_true_flat = y_coords.reshape(-1, 3)\n\ny_pred_masked = y_pred_flat[mask]\ny_true_masked = y_true_flat[mask]\n\n# Evaluate\nfrom sklearn.metrics import mean_squared_error, mean_absolute_error\nmse = mean_squared_error(y_true_masked, y_pred_masked)\nmae = mean_absolute_error(y_true_masked, y_pred_masked)\n\nprint(f\"✅ Delta Model Validation MSE: {mse:.4f}\")\nprint(f\"✅ Delta Model Validation MAE: {mae:.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T10:21:23.042661Z","iopub.execute_input":"2025-04-17T10:21:23.042971Z","iopub.status.idle":"2025-04-17T10:21:23.683134Z","shell.execute_reply.started":"2025-04-17T10:21:23.042947Z","shell.execute_reply":"2025-04-17T10:21:23.682227Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🔁 Delta Coordinate Prediction with Transformer\n\nInstead of predicting absolute 3D coordinates, we trained our model to predict the **difference (Δ) between consecutive coordinates**. This aligns with RNA's continuous folding nature and provides a smoother learning signal.\n\n**Model Setup:**\n- Transformer encoder + learned positional embeddings\n- Target: Δx, Δy, Δz per base\n- Output reconstructed with cumulative sum\n\n📈 Final Train Loss: 0.6953  \n📉 Final Val Loss: 1.6977  \n\n**Evaluation (Real 3D Coordinates):**\n- ✅ Validation MSE: 2945.8187  \n- ✅ Validation MAE: 28.1324  \n\nThis model outperformed most previous architectures in terms of overall structural error, especially MSE, and gives highly stable predictions.\n","metadata":{}},{"cell_type":"markdown","source":"----\n### 🧱 2. Stacked Transformer Encoder (Transformer v2)\n\nThis model will:\n\n\t•\t📚 Learn richer sequence dependencies\n\t•\t🧠 Model long-range attention better\n\t•\t💥 Potentially beat all previous results\n\n\nNow we can add multiple encoder blocks for deeper understanding.","metadata":{}},{"cell_type":"code","source":"def transformer_stack(x, num_layers=3, **kwargs):\n    for _ in range(num_layers):\n        x = transformer_encoder(x, **kwargs)\n    return x\n\n# Replace single block with stack:\nx = transformer_stack(x, num_layers=3, head_size=64, num_heads=4, ff_dim=128, dropout=0.1)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T10:15:57.581914Z","iopub.execute_input":"2025-04-17T10:15:57.582263Z","iopub.status.idle":"2025-04-17T10:15:57.76097Z","shell.execute_reply.started":"2025-04-17T10:15:57.582237Z","shell.execute_reply":"2025-04-17T10:15:57.759763Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"Train this on either y_coords_scaled or delta_coords_scaled.\n","metadata":{}},{"cell_type":"markdown","source":"**✅ Step 1: Define Transformer Block (same as before)**","metadata":{}},{"cell_type":"code","source":"from tensorflow.keras.layers import Input, Embedding, Dense, Dropout, LayerNormalization, MultiHeadAttention, Add, TimeDistributed, Concatenate\nfrom tensorflow.keras.models import Model\n\ndef transformer_encoder(inputs, head_size=64, num_heads=4, ff_dim=128, dropout=0.1):\n    # Multi-head self-attention\n    attention = MultiHeadAttention(key_dim=head_size, num_heads=num_heads)(inputs, inputs)\n    attention = Dropout(dropout)(attention)\n    x = Add()([inputs, attention])\n    x = LayerNormalization()(x)\n\n    # Feed-forward\n    ff = Dense(ff_dim, activation='relu')(x)\n    ff = Dropout(dropout)(ff)\n    ff = Dense(inputs.shape[-1])(ff)\n    x = Add()([x, ff])\n    return LayerNormalization()(x)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T10:24:47.63875Z","iopub.execute_input":"2025-04-17T10:24:47.639089Z","iopub.status.idle":"2025-04-17T10:24:47.645518Z","shell.execute_reply.started":"2025-04-17T10:24:47.639067Z","shell.execute_reply":"2025-04-17T10:24:47.644659Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**✅ Step 2: Stack Multiple Encoder Layers**","metadata":{}},{"cell_type":"code","source":"def transformer_stack(x, num_layers=3, **kwargs):\n    for _ in range(num_layers):\n        x = transformer_encoder(x, **kwargs)\n    return x","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T10:25:43.057684Z","iopub.execute_input":"2025-04-17T10:25:43.058004Z","iopub.status.idle":"2025-04-17T10:25:43.062629Z","shell.execute_reply.started":"2025-04-17T10:25:43.057979Z","shell.execute_reply":"2025-04-17T10:25:43.06163Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**✅ Step 3: Build the Full Model**","metadata":{}},{"cell_type":"code","source":"# Inputs\nkmer_input = Input(shape=(None,), name=\"kmer_input\")        # [batch, seq_len]\npos_id_input = Input(shape=(None,), name=\"pos_id_input\")    # [batch, seq_len]\n\n# Embeddings\nkmer_embed = Embedding(input_dim=len(kmer_to_index), output_dim=64)(kmer_input)\npos_embed = Embedding(input_dim=200, output_dim=16)(pos_id_input)  # learned position\n\n# Combine embeddings\nx = Concatenate()([kmer_embed, pos_embed])  # shape: [batch, seq_len, 80]\n\n# Stacked Transformer blocks\nx = transformer_stack(x, num_layers=3, head_size=64, num_heads=4, ff_dim=128, dropout=0.1)\n\n# Output: Δx, Δy, Δz\noutput = TimeDistributed(Dense(3))(x)\n\n# Model\nmodel = Model(inputs=[kmer_input, pos_id_input], outputs=output)\nmodel.compile(optimizer='adam', loss='mse')\nmodel.summary()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T10:26:50.877985Z","iopub.execute_input":"2025-04-17T10:26:50.878328Z","iopub.status.idle":"2025-04-17T10:26:51.116412Z","shell.execute_reply.started":"2025-04-17T10:26:50.878304Z","shell.execute_reply":"2025-04-17T10:26:51.11568Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"**✅ Step 4: Train It on Delta Targets**","metadata":{}},{"cell_type":"code","source":"history = model.fit(\n    x={'kmer_input': X_kmer, 'pos_id_input': X_pos_ids},\n    y=delta_coords_scaled,\n    validation_split=0.2,\n    epochs=10,\n    batch_size=4,\n    verbose=2\n)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T10:27:38.327883Z","iopub.execute_input":"2025-04-17T10:27:38.328757Z","iopub.status.idle":"2025-04-17T10:28:44.322816Z","shell.execute_reply.started":"2025-04-17T10:27:38.328727Z","shell.execute_reply":"2025-04-17T10:28:44.322181Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### Evaluate Real-Space 3D Accuracy\n\nLet’s run the full prediction + reconstruction + metric block:","metadata":{}},{"cell_type":"code","source":"# Predict deltas (scaled)\ndelta_pred_scaled = model.predict({'kmer_input': X_kmer, 'pos_id_input': X_pos_ids})\n\n# Inverse transform to real delta values\ndelta_pred = scaler_delta.inverse_transform(\n    delta_pred_scaled.reshape(-1, 3)\n).reshape(delta_coords.shape)\n\n# Reconstruct absolute coordinates\ny_pred = np.cumsum(delta_pred, axis=1)\n\n# Flatten + mask\nmask = (X_pos_ids.flatten() != 0)\ny_pred_flat = y_pred.reshape(-1, 3)\ny_true_flat = y_coords.reshape(-1, 3)\n\ny_pred_masked = y_pred_flat[mask]\ny_true_masked = y_true_flat[mask]\n\n# Evaluation\nfrom sklearn.metrics import mean_squared_error, mean_absolute_error\nmse = mean_squared_error(y_true_masked, y_pred_masked)\nmae = mean_absolute_error(y_true_masked, y_pred_masked)\n\nprint(f\"✅ Transformer v2 (3-layer) Validation MSE: {mse:.4f}\")\nprint(f\"✅ Transformer v2 (3-layer) Validation MAE: {mae:.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T10:30:27.750346Z","iopub.execute_input":"2025-04-17T10:30:27.750677Z","iopub.status.idle":"2025-04-17T10:30:33.632576Z","shell.execute_reply.started":"2025-04-17T10:30:27.750654Z","shell.execute_reply":"2025-04-17T10:30:33.63163Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🧱 Transformer v2: Stacked Encoder Model\n\nWe tested a deeper Transformer architecture with **3 stacked encoder layers** and trained it to predict **delta coordinates** (Δx, Δy, Δz).\n\n**Training Summary:**\n- Final train loss: 0.7365\n- Best val loss: 1.6241 (epoch 7)\n\n**Evaluation (Reconstructed 3D Coordinates):**\n- ✅ Validation MSE: 6000.1885\n- ✅ Validation MAE: 63.6392\n\n**Conclusion:**\nWhile the stacked architecture captured structure, it underperformed compared to the simpler 1-layer delta Transformer. This confirms that model depth must be balanced with dataset size and regularization.","metadata":{}},{"cell_type":"markdown","source":"----\n## 📤 3. Generate Predictions for Submission","metadata":{}},{"cell_type":"markdown","source":"### 📤 Step 1: Load Test Set + Prepare Inputs","metadata":{}},{"cell_type":"code","source":"# Load test set\nimport pandas as pd\ntest_sequences = pd.read_csv(\"/kaggle/input/stanford-rna-3d-folding/test_sequences.csv\")\nsample_submission = pd.read_csv(\"/kaggle/input/stanford-rna-3d-folding/sample_submission.csv\")\n\n# Tokenize k-mers\ndef kmer_tokenize(seq, k=3):\n    tokens = [seq[i:i+k] for i in range(len(seq) - k + 1)]\n    return [kmer_to_index.get(kmer, 0) for kmer in tokens]\n\nX_test_kmer = [kmer_tokenize(seq) for seq in test_sequences[\"sequence\"]]\nmax_len = max(len(seq) for seq in X_test_kmer)\n\n# Pad sequences\nfrom tensorflow.keras.preprocessing.sequence import pad_sequences\nX_test_kmer = pad_sequences(X_test_kmer, maxlen=max_len, padding=\"post\")\n\n# Position IDs\nX_test_pos_ids = np.tile(np.arange(max_len), (X_test_kmer.shape[0], 1))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T11:41:44.700126Z","iopub.execute_input":"2025-04-17T11:41:44.702762Z","iopub.status.idle":"2025-04-17T11:41:44.752617Z","shell.execute_reply.started":"2025-04-17T11:41:44.702731Z","shell.execute_reply":"2025-04-17T11:41:44.75156Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🧠 Step 2: Predict Delta Coordinates → Reconstruct Absolute Positions","metadata":{}},{"cell_type":"code","source":"# Predict deltas\ndelta_test_scaled = model.predict({\n    \"kmer_input\": X_test_kmer,\n    \"pos_id_input\": X_test_pos_ids\n})\n\n# Inverse scale\ndelta_test = scaler_delta.inverse_transform(\n    delta_test_scaled.reshape(-1, 3)\n).reshape(delta_test_scaled.shape)\n\n# Reconstruct absolute coordinates\ncoords_test = np.cumsum(delta_test, axis=1)  # [samples, seq_len, 3]","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T11:41:53.63441Z","iopub.execute_input":"2025-04-17T11:41:53.634735Z","iopub.status.idle":"2025-04-17T11:41:53.942599Z","shell.execute_reply.started":"2025-04-17T11:41:53.634709Z","shell.execute_reply":"2025-04-17T11:41:53.941224Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🧾 Step 3: Format Submission File","metadata":{}},{"cell_type":"code","source":"# Build ID and flat list of predictions\nsubmission_rows = []\n\nfor i, target_id in enumerate(test_sequences[\"target_id\"]):\n    seq_len = len(test_sequences.iloc[i][\"sequence\"])\n    for j in range(seq_len):\n        row = {\n            \"ID\": f\"{target_id}_{j+1}\",\n            \"x\": coords_test[i, j, 0],\n            \"y\": coords_test[i, j, 1],\n            \"z\": coords_test[i, j, 2],\n        }\n        submission_rows.append(row)\n\nsubmission_df = pd.DataFrame(submission_rows)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### ✅ Match required format:","metadata":{}},{"cell_type":"code","source":"submission_df = submission_df.rename(columns={\"x\": \"x_1\", \"y\": \"y_1\", \"z\": \"z_1\"})\nsubmission_df = submission_df[[\"ID\", \"x_1\", \"y_1\", \"z_1\"]]\nsubmission_df.to_csv(\"submission.csv\", index=False)","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 📊 Visualizing Predictions\n\nBefore generating the submission file, let’s visualize the model’s predictions and error metrics to assess quality and behavior.","metadata":{}},{"cell_type":"code","source":"points = y_pred[idx, :seq_len]\ncolors = np.linspace(0, 1, seq_len)\n\nfig = plt.figure()\nax = fig.add_subplot(111, projection='3d')\nsc = ax.scatter(points[:, 0], points[:, 1], points[:, 2], c=colors, cmap='viridis')\nplt.title(\"Predicted RNA Structure with Progression\")\nplt.colorbar(sc, label='Sequence Position')\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T10:50:27.124803Z","iopub.execute_input":"2025-04-17T10:50:27.125069Z","iopub.status.idle":"2025-04-17T10:50:27.330967Z","shell.execute_reply.started":"2025-04-17T10:50:27.125053Z","shell.execute_reply":"2025-04-17T10:50:27.33019Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🌈 Predicted RNA Structure with Sequence Progression\n\nWe visualize how the predicted structure unfolds over time:\n\n- 🎨 Colors represent **base index progression**\n- The smooth color transition confirms spatial continuity\n- Clusters or overlaps may suggest complex folding patterns\n\nThis is useful for checking **structural coherence** without needing ground truth.","metadata":{}},{"cell_type":"markdown","source":"### 🔍 Ground Truth vs. Prediction","metadata":{}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\nfrom mpl_toolkits.mplot3d import Axes3D\n\n# Pick a sample (from train set) to visualize\nidx = 10  # any index from 0 to len(X_kmer)-1\nseq_len = (X_pos_ids[idx] != 0).sum()\n\nfig = plt.figure(figsize=(10, 6))\nax = fig.add_subplot(111, projection='3d')\n\n# True structure\nax.plot(\n    y_coords[idx, :seq_len, 0],\n    y_coords[idx, :seq_len, 1],\n    y_coords[idx, :seq_len, 2],\n    label=\"Ground Truth\", color=\"green\", linewidth=2\n)\n\n# Predicted structure (from best delta model)\nax.plot(\n    y_pred[idx, :seq_len, 0],\n    y_pred[idx, :seq_len, 1],\n    y_pred[idx, :seq_len, 2],\n    label=\"Prediction\", color=\"orange\", linestyle='--'\n)\n\nax.set_title(f\"RNA 3D Structure: Sample {idx}\")\nax.legend()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T10:53:24.715063Z","iopub.execute_input":"2025-04-17T10:53:24.715389Z","iopub.status.idle":"2025-04-17T10:53:24.87667Z","shell.execute_reply.started":"2025-04-17T10:53:24.715363Z","shell.execute_reply":"2025-04-17T10:53:24.875669Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 🧬 RNA Structure Prediction: Ground Truth vs Model Output\n\nWe compare the predicted 3D coordinates against the true structure using a line plot:\n\n- ✅ Green: True RNA structure\n- 🟧 Orange dashed: Predicted coordinates\n\nThis visualization reveals how accurately the model reconstructs spatial positions.  \nIn this example, the prediction captures the **general spatial flow**, but deviations suggest **room for structural refinement**, especially near loops or bends.","metadata":{}},{"cell_type":"markdown","source":"### 📉 Error per Base (Euclidean Distance)","metadata":{}},{"cell_type":"code","source":"errors = np.linalg.norm(y_pred - y_coords, axis=2)\nplt.plot(errors[idx])\nplt.title(\"Error per Base (Euclidean Distance)\")\nplt.xlabel(\"Base Index\")\nplt.ylabel(\"Distance Error (nm)\")\nplt.grid(True)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T10:55:45.529447Z","iopub.execute_input":"2025-04-17T10:55:45.53023Z","iopub.status.idle":"2025-04-17T10:55:45.701339Z","shell.execute_reply.started":"2025-04-17T10:55:45.530198Z","shell.execute_reply":"2025-04-17T10:55:45.700462Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 📉 Per-Base Error Analysis (Euclidean Distance)\n\nWe compute the distance between predicted and true coordinates for each base:\n\n- Peaks = poor prediction at specific locations\n- Low-flat regions = better alignment\n\nThis helps identify **problematic regions**, such as loops, junctions, or ends where structural prediction is more challenging.","metadata":{}},{"cell_type":"markdown","source":"### 📏 Mean Error vs. Sequence Length","metadata":{}},{"cell_type":"code","source":"lengths = (X_pos_ids != 0).sum(axis=1)\nerrors_all = np.linalg.norm(y_pred - y_coords, axis=2)\nmean_errors = errors_all.mean(axis=1)\n\nplt.scatter(lengths, mean_errors, alpha=0.6)\nplt.title(\"Error vs. Sequence Length\")\nplt.xlabel(\"Sequence Length\")\nplt.ylabel(\"Mean Per-Base Error\")\nplt.grid(True)\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-04-17T10:57:09.406619Z","iopub.execute_input":"2025-04-17T10:57:09.407287Z","iopub.status.idle":"2025-04-17T10:57:09.56345Z","shell.execute_reply.started":"2025-04-17T10:57:09.407264Z","shell.execute_reply":"2025-04-17T10:57:09.562599Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 📏 Error vs Sequence Length\n\nEach point represents an RNA sample:\n- X-axis: Sequence length\n- Y-axis: Mean error per base\n\n🔍 We observe that:\n- Most sequences have consistent lengths (~100 bases)\n- A few outliers with higher error may be structurally complex or underrepresented in training data\n\nNo strong correlation detected here, suggesting length alone doesn’t predict error.","metadata":{}},{"cell_type":"markdown","source":"### 📤 Final Kaggle Submission\n\nWe used our best model — a Transformer with learned positional embeddings and delta coordinate prediction — to generate predictions for the test set.\n\n1. K-mer tokenized each sequence (k=3)\n2. Predicted Δx, Δy, Δz for each base\n3. Reconstructed absolute positions with `np.cumsum`\n4. Formatted submission file\n\n⏫ File: `submission.csv`","metadata":{}}]}