{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.11.11","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":91249,"databundleVersionId":11294684,"sourceType":"competition"}],"dockerImageVersionId":31012,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **We will cover :-**\n\n\n1. Goal of Comeptition\n2. AboutData Set given\n3. Approach to DataSet and Guide to Train ML ( including import topics you must known )\n4. Data Visulization including 3D\n5. Discriptive and Infrential Statistiscs\n\n# **A. Goal of the Competition ------------------>**","metadata":{}},{"cell_type":"markdown","source":"The goal is to develop machine learning and computer vision models to accurately locate bacterial flagellar motors (BFMs) in microscopy images.\n**This competition bridges AI, biology, and medicine, with potential breakthroughs in infectious disease control and nanotechnology**\n\nThis Competition is all about Understanding where Bacteria lies in the Image","metadata":{}},{"cell_type":"markdown","source":"**Real-Life Applications**\n\n\nMedical & Antibiotic Research\n\n1. Understanding how bacterial flagella contribute to motility and infection can help in designing new antibiotics that disrupt bacterial movement.\n2. Some pathogens (like Salmonella or E. coli) use flagella to invade hosts—blocking their motors could prevent infections.\n\n\nSynthetic Biology & Bioengineering\n\n\n1. Scientists are engineering bacterial microrobots for drug delivery; precise motor control is crucial.\n2. Studying BFMs can lead to bio-inspired nanomachines.\n\n\nMicroscopy & Automation\n\n\n1. Reduces the need for manual annotation, speeding up microbiological research.\n2. Helps in high-throughput screening of bacterial behaviors.\n","metadata":{}},{"cell_type":"markdown","source":"## Some Introduction ","metadata":{}},{"cell_type":"markdown","source":"![](http://i0.wp.com/research.csiro.au/synthetic-biology-fsp/wp-content/uploads/sites/140/2020/02/11635966-3x2-large.jpg?fit=700%2C467&ssl=1)","metadata":{}},{"cell_type":"markdown","source":"**What is Bacteria Flagella**\nFlagellar bacteria are bacteria that possess flagella, whip-like appendages that enable them to move through liquid environments. These flagella are essential for bacterial motility, allowing them to swim and navigate their environment, often in search of nutrients or favorable conditions","metadata":{}},{"cell_type":"markdown","source":"# B. About the DataSet------------------>","metadata":{}},{"cell_type":"markdown","source":"The Folder contains 2 Folder and 2 files. ","metadata":{}},{"cell_type":"markdown","source":"## 1. test Folder\n\nThe Test Folder contains 3 Directories with jpg files for Testing Data. This is the folder to be Tested","metadata":{}},{"cell_type":"markdown","source":"## 2. train Folder\n\nConatians 683 Directories containing File to Train Data for jpg images. There may be max 500-1000 jpg files. The path is referenced to train_label.csv file","metadata":{}},{"cell_type":"markdown","source":"## 3. sample_submission.csv\n\nThis is the csv file showing Columns for submission as Demo. It contains 4 column : \"tomo_id\" for id number, and \"Motor axis 0\", \"Motor axis 1\". and \"Motor axis 2\" which contains 3 predicted for motor.","metadata":{}},{"cell_type":"markdown","source":"## 4. train_label.csv\n\nContains \"row_id\" as showing no of data, \"tomo_id\" where Each tomo_id represents a 3D volume (tomogram) of bacterial samples. \"Motor axis 0, 1, 2 \"contains in the axis where Training Data is Located or in sense indicate coordinates of flagellar motors in 3D space. \"Array Shape axis 1\" contains Array shape (dimensions in voxels), same for Voxel spacing (resolution in nm/pixel) and finally no. of Motors present.","metadata":{}},{"cell_type":"markdown","source":"# **C. Approach to DataSet ---------------------->**","metadata":{}},{"cell_type":"markdown","source":"## 1. Demo Training Model","metadata":{}},{"cell_type":"markdown","source":"### 1.1. Load the data and Import Important Files","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom skimage import io\nfrom sklearn.model_selection import train_test_split\nimport torch\nimport torch.nn as nn\nimport torch.optim as optim\nfrom torch.utils.data import Dataset, DataLoader\n\n\ndf = pd.read_csv('/kaggle/input/byu-locating-bacterial-flagellar-motors-2025/train_labels.csv')\nprint(df.head())\n\n\nsample = df.iloc[0]\nprint(f\"\\nSample tomogram: {sample['tomo_id']}\")\nprint(f\"Motor coordinates: {sample['Motor axis 0']}, {sample['Motor axis 1']}, {sample['Motor axis 2']}\")\nprint(f\"Volume shape: {sample['Array shape (axis 0)']}x{sample['Array shape (axis 1)']}x{sample['Array shape (axis 2)']}\")\nprint(f\"Voxel spacing: {sample['Voxel spacing']} nm\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-11T05:12:18.883239Z","iopub.execute_input":"2025-05-11T05:12:18.883508Z","iopub.status.idle":"2025-05-11T05:12:27.257819Z","shell.execute_reply.started":"2025-05-11T05:12:18.883480Z","shell.execute_reply":"2025-05-11T05:12:27.256834Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 1.2. Create Synthetic Data for Demonstration","metadata":{}},{"cell_type":"code","source":"class SyntheticTomogramDataset(Dataset):\n    def __init__(self, num_samples=100):\n        self.num_samples = num_samples\n        self.volume_shape = (64, 64, 64) \n    def __len__(self):\n        return self.num_samples\n    \n    def __getitem__(self, idx):\n      \n        volume = np.zeros(self.volume_shape, dtype=np.float32)\n        \n       \n        num_motors = np.random.randint(1, 4)\n        coordinates = []\n        \n        for _ in range(num_motors):\n            \n            x, y, z = np.random.randint(10, 54, size=3)\n            coordinates.append([x, y, z])\n            \n            \n            xx, yy, zz = np.mgrid[:64, :64, :64]\n            blob = np.exp(-((xx-x)**2 + (yy-y)**2 + (zz-z)**2) / 8.0)\n            volume += blob\n        \n      \n        volume = (volume - volume.min()) / (volume.max() - volume.min())\n        \n      \n        volume = torch.from_numpy(volume).unsqueeze(0)  # Add channel dim\n        \n        \n        coordinates = torch.FloatTensor(coordinates[0]) if coordinates else torch.zeros(3)\n        \n        return volume, coordinates\n\ndataset = SyntheticTomogramDataset()\nsample_vol, sample_coords = dataset[0]\n\nprint(f\"\\nSample volume shape: {sample_vol.shape}\")\nprint(f\"Sample motor coordinates: {sample_coords}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-11T05:12:27.259900Z","iopub.execute_input":"2025-05-11T05:12:27.260216Z","iopub.status.idle":"2025-05-11T05:12:27.414206Z","shell.execute_reply.started":"2025-05-11T05:12:27.260189Z","shell.execute_reply":"2025-05-11T05:12:27.413243Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 1.3. Visualize slices","metadata":{}},{"cell_type":"code","source":"\nfig, axes = plt.subplots(1, 3, figsize=(15, 5))\nfor i, (ax, slice_idx) in enumerate(zip(axes, [32, 32, 32])):\n    if i == 0:\n        ax.imshow(sample_vol[0, slice_idx, :, :], cmap='gray')\n    elif i == 1:\n        ax.imshow(sample_vol[0, :, slice_idx, :], cmap='gray')\n    else:\n        ax.imshow(sample_vol[0, :, :, slice_idx], cmap='gray')\n    ax.set_title(f\"Slice {slice_idx}\")\nplt.show()\n\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-11T05:12:27.414955Z","iopub.execute_input":"2025-05-11T05:12:27.415229Z","iopub.status.idle":"2025-05-11T05:12:27.961696Z","shell.execute_reply.started":"2025-05-11T05:12:27.415206Z","shell.execute_reply":"2025-05-11T05:12:27.960892Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 1.4. Simple 3D CNN Model","metadata":{}},{"cell_type":"code","source":"class MotorLocalizer(nn.Module):\n    def __init__(self):\n        super(MotorLocalizer, self).__init__()\n        \n        self.features = nn.Sequential(\n            nn.Conv3d(1, 8, kernel_size=3, padding=1),\n            nn.ReLU(),\n            nn.MaxPool3d(2),\n            \n            nn.Conv3d(8, 16, kernel_size=3, padding=1),\n            nn.ReLU(),\n            nn.MaxPool3d(2),\n            \n            nn.Conv3d(16, 32, kernel_size=3, padding=1),\n            nn.ReLU(),\n            nn.MaxPool3d(2)\n        )\n        \n        self.regressor = nn.Sequential(\n            nn.Flatten(),\n            nn.Linear(32*8*8*8, 128),\n            nn.ReLU(),\n            nn.Linear(128, 3)  \n        )\n        \n    def forward(self, x):\n        x = self.features(x)\n        return self.regressor(x)\n\nmodel = MotorLocalizer()\nprint(model)\n\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-11T05:12:27.962537Z","iopub.execute_input":"2025-05-11T05:12:27.962901Z","iopub.status.idle":"2025-05-11T05:12:27.997728Z","shell.execute_reply.started":"2025-05-11T05:12:27.962742Z","shell.execute_reply":"2025-05-11T05:12:27.996849Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 1.5. Training Setup","metadata":{}},{"cell_type":"code","source":"\ntrain_data, val_data = train_test_split(dataset, test_size=0.2)\n\n\ntrain_loader = DataLoader(train_data, batch_size=4, shuffle=True)\nval_loader = DataLoader(val_data, batch_size=4)\n\n\ncriterion = nn.MSELoss()\noptimizer = optim.Adam(model.parameters(), lr=0.001)\n\n","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-11T05:12:27.998503Z","iopub.execute_input":"2025-05-11T05:12:27.998793Z","iopub.status.idle":"2025-05-11T05:12:33.306215Z","shell.execute_reply.started":"2025-05-11T05:12:27.998755Z","shell.execute_reply":"2025-05-11T05:12:33.304739Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 1.6. Training Loop","metadata":{}},{"cell_type":"code","source":"def train_model(num_epochs=10):\n    for epoch in range(num_epochs):\n        model.train()\n        train_loss = 0.0\n        \n        for volumes, coords in train_loader:\n            optimizer.zero_grad()\n            \n            outputs = model(volumes)\n            loss = criterion(outputs, coords)\n            \n            loss.backward()\n            optimizer.step()\n            \n            train_loss += loss.item()\n        \n        # Validation\n        model.eval()\n        val_loss = 0.0\n        with torch.no_grad():\n            for volumes, coords in val_loader:\n                outputs = model(volumes)\n                val_loss += criterion(outputs, coords).item()\n        \n        print(f\"Epoch {epoch+1}/{num_epochs} - Train Loss: {train_loss/len(train_loader):.4f} - Val Loss: {val_loss/len(val_loader):.4f}\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-11T05:12:33.307705Z","iopub.execute_input":"2025-05-11T05:12:33.308351Z","iopub.status.idle":"2025-05-11T05:12:33.316624Z","shell.execute_reply.started":"2025-05-11T05:12:33.308316Z","shell.execute_reply":"2025-05-11T05:12:33.315016Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### 1.7. Inference Example","metadata":{}},{"cell_type":"code","source":"def predict_motor_location(volume):\n    model.eval()\n    with torch.no_grad():\n        volume = volume.unsqueeze(0)  \n        pred_coords = model(volume)\n    return pred_coords.squeeze(0)\n\ntest_vol, true_coords = dataset[5]\npred_coords = predict_motor_location(test_vol)\n\nprint(f\"\\nTrue coordinates: {true_coords}\")\nprint(f\"Predicted coordinates: {pred_coords}\")\nprint(f\"Euclidean distance error: {torch.norm(pred_coords - true_coords):.2f} voxels\")","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-11T05:12:33.319644Z","iopub.execute_input":"2025-05-11T05:12:33.319937Z","iopub.status.idle":"2025-05-11T05:12:33.540662Z","shell.execute_reply.started":"2025-05-11T05:12:33.319913Z","shell.execute_reply":"2025-05-11T05:12:33.539813Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 2. Important Topics to known","metadata":{}},{"cell_type":"markdown","source":"3D Computer Vision Fundamentals\n\n1. Volumetric data processing: Understanding 3D convolutions, pooling operation\n2. Tomographic reconstruction: Basics of cryo-ET image formation\n3. 3D feature extraction: Techniques for identifying structures in volumetric data\n\n\n3D CNN architectures:\n\n\n1. 3D U-Net variants (VoxelMorph, nnU-Net)\n2. ResNet3D, DenseNet\n3. 3DAttention mechanisms for 3D data\n\n\nObject detection in 3D:\n\n\n1. Adapting Faster R-CNN/YOLO for volumetric dataPoint\n2. Net++ for sparse detection\n\n\nTomogram normalization:\n\n1. Denoising techniques (Cryo-CARE, IsoNet)\n2. Contrast enhancement\n","metadata":{}},{"cell_type":"markdown","source":"## 3. Resources ","metadata":{}},{"cell_type":"markdown","source":"\n* \"Deep learning improves macromolecule identification in 3D cellular cryo-electron tomograms\" (Nature Methods 2021)\n* \"Cryo-electron tomography and deep learning - a comprehensive review\" (Current Opinion in Structural Biology 2022)\n* \"Locating macromolecules in cryo-electron tomograms using 3D convolutional neural networks\" (Journal of Structural Biology 2020)\n","metadata":{}},{"cell_type":"markdown","source":"# E. Visualization","metadata":{}},{"cell_type":"markdown","source":"## 1. Import Dependency and Data","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nfrom mpl_toolkits.mplot3d import Axes3D\nimport os\nfrom skimage import io, exposure\nimport seaborn as sns\n\n\ndf = pd.read_csv('/kaggle/input/byu-locating-bacterial-flagellar-motors-2025/train_labels.csv')\n\n\nIMAGE_DIR = '/kaggle/input/byu-locating-bacterial-flagellar-motors-2025/train'\nos.makedirs('visualizations', exist_ok=True)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-11T05:12:33.541413Z","iopub.execute_input":"2025-05-11T05:12:33.541684Z","iopub.status.idle":"2025-05-11T05:12:33.968360Z","shell.execute_reply.started":"2025-05-11T05:12:33.541655Z","shell.execute_reply":"2025-05-11T05:12:33.967282Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 2. Plot Metadata Distribution","metadata":{}},{"cell_type":"code","source":"def plot_metadata_distributions(df):\n    \"\"\"Visualize distributions of key metadata features\"\"\"\n    plt.figure(figsize=(15, 10))\n    \n    plt.subplot(2, 2, 1)\n    sns.histplot(df['Number of motors'], bins=20, kde=False)\n    plt.title('Distribution of Motors per Tomogram')\n    \n    plt.subplot(2, 2, 2)\n    sns.histplot(df['Voxel spacing'], bins=20, kde=False)\n    plt.title('Voxel Spacing Distribution')\n    \n    plt.subplot(2, 2, 3)\n    sizes = df['Array shape (axis 0)'] * df['Array shape (axis 1)'] * df['Array shape (axis 2)']\n    sns.histplot(sizes, bins=20, kde=False)\n    plt.title('Tomogram Volume Sizes (voxels)')\n    \n    plt.subplot(2, 2, 4)\n    sns.scatterplot(x='Array shape (axis 0)', y='Voxel spacing', data=df)\n    plt.title('Resolution vs Volume Size')\n    \n    plt.tight_layout()\n    plt.savefig('visualizations/metadata_distributions.png')\n    plt.show()\n\nplot_metadata_distributions(df)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-11T05:12:33.969367Z","iopub.execute_input":"2025-05-11T05:12:33.969642Z","iopub.status.idle":"2025-05-11T05:12:35.345266Z","shell.execute_reply.started":"2025-05-11T05:12:33.969621Z","shell.execute_reply":"2025-05-11T05:12:35.343940Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 3. Tomogram Visualization with Motor Locations","metadata":{}},{"cell_type":"code","source":"def visualize_tomogram(tomo_id, df, slice_frac=0.5):\n    \"\"\"Visualize a tomogram with motor locations marked\"\"\"\n    # Get tomogram info\n    tomo_info = df[df['tomo_id'] == tomo_id].iloc[0]\n    motor_locs = df[df['tomo_id'] == tomo_id][['Motor axis 0', 'Motor axis 1', 'Motor axis 2']].values\n    \n    try:\n        print(f\"Warning: Using synthetic data - replace with actual tomogram loading\")\n        volume = np.random.rand(tomo_info['Array shape (axis 0)'], \n                              tomo_info['Array shape (axis 1)'], \n                              tomo_info['Array shape (axis 2)'])\n    except Exception as e:\n        print(f\"Error loading {tomo_id}: {e}\")\n        return\n    \n    \n    slice_z = int(volume.shape[0] * slice_frac)\n    slice_y = int(volume.shape[1] * slice_frac)\n    slice_x = int(volume.shape[2] * slice_frac)\n    \n   \n    fig = plt.figure(figsize=(15, 5))\n    \n   \n    ax1 = fig.add_subplot(131)\n    ax1.imshow(volume[slice_z,:,:], cmap='gray')\n    \n    for motor in motor_locs:\n        if motor[0] >= 0: \n            ax1.plot(motor[2], motor[1], 'r+', markersize=10)\n    ax1.set_title(f'XY Slice (Z={slice_z})')\n    \n   \n    ax2 = fig.add_subplot(132)\n    ax2.imshow(volume[:,slice_y,:], cmap='gray')\n    for motor in motor_locs:\n        if motor[0] >= 0:\n            ax2.plot(motor[2], motor[0], 'r+', markersize=10)\n    ax2.set_title(f'XZ Slice (Y={slice_y})')\n    \n   \n    ax3 = fig.add_subplot(133)\n    ax3.imshow(volume[:,:,slice_x], cmap='gray')\n    for motor in motor_locs:\n        if motor[0] >= 0:\n            ax3.plot(motor[1], motor[0], 'r+', markersize=10)\n    ax3.set_title(f'YZ Slice (X={slice_x})')\n    \n    plt.suptitle(f\"Tomogram {tomo_id}\\nVoxel size: {tomo_info['Voxel spacing']}nm | Motors: {len(motor_locs)}\")\n    plt.tight_layout()\n    plt.savefig(f'visualizations/{tomo_id}_slices.png')\n    plt.show()\n\nsample_tomos = df['tomo_id'].unique()[:3]  \nfor tomo in sample_tomos:\n    visualize_tomogram(tomo, df)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-11T05:12:35.346401Z","iopub.execute_input":"2025-05-11T05:12:35.346803Z","iopub.status.idle":"2025-05-11T05:13:07.403552Z","shell.execute_reply.started":"2025-05-11T05:12:35.346752Z","shell.execute_reply":"2025-05-11T05:13:07.402607Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 4. 3D Motor Location Visualization","metadata":{}},{"cell_type":"code","source":"def plot_3d_motor_locations(tomo_id, df):\n    \"\"\"Create 3D plot of motor locations within a tomogram\"\"\"\n    motor_locs = df[df['tomo_id'] == tomo_id][['Motor axis 0', 'Motor axis 1', 'Motor axis 2']].values\n    tomo_info = df[df['tomo_id'] == tomo_id].iloc[0]\n    \n    \n    valid_locs = motor_locs[(motor_locs >= 0).all(axis=1)]\n    \n    if len(valid_locs) == 0:\n        print(f\"No valid motor coordinates for {tomo_id}\")\n        return\n    \n    fig = plt.figure(figsize=(10, 8))\n    ax = fig.add_subplot(111, projection='3d')\n    \n    \n    ax.scatter(valid_locs[:,0], valid_locs[:,1], valid_locs[:,2], \n               c='r', s=50, marker='o', label='Flagellar Motors')\n    \n   \n    ax.set_xlabel('Axis 0')\n    ax.set_ylabel('Axis 1')\n    ax.set_zlabel('Axis 2')\n    ax.set_title(f'3D Motor Locations in {tomo_id}\\n{len(valid_locs)} motors | Voxel size: {tomo_info[\"Voxel spacing\"]}nm')\n    \n   \n    ax.set_xlim(0, tomo_info['Array shape (axis 0)'])\n    ax.set_ylim(0, tomo_info['Array shape (axis 1)'])\n    ax.set_zlim(0, tomo_info['Array shape (axis 2)'])\n    \n    plt.legend()\n    plt.tight_layout()\n    plt.savefig(f'visualizations/{tomo_id}_3d.png')\n    plt.show()\n    \nfor tomo in sample_tomos:\n    plot_3d_motor_locations(tomo, df)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-11T05:13:07.404494Z","iopub.execute_input":"2025-05-11T05:13:07.404722Z","iopub.status.idle":"2025-05-11T05:13:08.197327Z","shell.execute_reply.started":"2025-05-11T05:13:07.404706Z","shell.execute_reply":"2025-05-11T05:13:08.196372Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 5. Motor Density Analysis","metadata":{}},{"cell_type":"code","source":"def analyze_motor_density(df):\n    \"\"\"Analyze spatial distribution patterns of motors\"\"\"\n    plt.figure(figsize=(12, 8))\n    \n    all_motors = []\n    for tomo_id in df['tomo_id'].unique():\n        motor_locs = df[df['tomo_id'] == tomo_id][['Motor axis 0', 'Motor axis 1', 'Motor axis 2']].values\n        valid_locs = motor_locs[(motor_locs >= 0).all(axis=1)]\n        if len(valid_locs) > 0:\n            all_motors.extend(valid_locs)\n    \n    if not all_motors:\n        print(\"No valid motor coordinates found\")\n        return\n    \n    all_motors = np.array(all_motors)\n    \n    # Create 2D histograms for each axis pair\n    plt.subplot(2, 2, 1)\n    plt.hist2d(all_motors[:,0], all_motors[:,1], bins=50, cmap='viridis')\n    plt.colorbar()\n    plt.xlabel('Axis 0')\n    plt.ylabel('Axis 1')\n    plt.title('Motor Density (Axis 0 vs 1)')\n    \n    plt.subplot(2, 2, 2)\n    plt.hist2d(all_motors[:,0], all_motors[:,2], bins=50, cmap='viridis')\n    plt.colorbar()\n    plt.xlabel('Axis 0')\n    plt.ylabel('Axis 2')\n    plt.title('Motor Density (Axis 0 vs 2)')\n    \n    plt.subplot(2, 2, 3)\n    plt.hist2d(all_motors[:,1], all_motors[:,2], bins=50, cmap='viridis')\n    plt.colorbar()\n    plt.xlabel('Axis 1')\n    plt.ylabel('Axis 2')\n    plt.title('Motor Density (Axis 1 vs 2)')\n    \n    plt.tight_layout()\n    plt.savefig('visualizations/motor_density_analysis.png')\n    plt.show()\n\nanalyze_motor_density(df)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-11T05:13:08.198294Z","iopub.execute_input":"2025-05-11T05:13:08.198582Z","iopub.status.idle":"2025-05-11T05:13:09.963399Z","shell.execute_reply.started":"2025-05-11T05:13:08.198559Z","shell.execute_reply":"2025-05-11T05:13:09.962570Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"# D. Statistics","metadata":{}},{"cell_type":"markdown","source":"## 1. Statistical Analysis Framework","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nfrom scipy import stats\nimport statsmodels.api as sm\nfrom statsmodels.formula.api import ols\n\n\n\ndf = pd.read_csv('/kaggle/input/byu-locating-bacterial-flagellar-motors-2025/train_labels.csv')\n\ndf['volume_size'] = df['Array shape (axis 0)'] * df['Array shape (axis 1)'] * df['Array shape (axis 2)']\ndf['has_motors'] = df['Number of motors'] > 0\nvalid_motors = df[df['Motor axis 0'] >= 0]  # Filter valid motor coordinates","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-11T05:13:09.964368Z","iopub.execute_input":"2025-05-11T05:13:09.965146Z","iopub.status.idle":"2025-05-11T05:13:12.051464Z","shell.execute_reply.started":"2025-05-11T05:13:09.965118Z","shell.execute_reply":"2025-05-11T05:13:12.050613Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 2. Descriptive Statistics","metadata":{}},{"cell_type":"code","source":"def compute_descriptive_stats(df):\n    \"\"\"Calculate comprehensive descriptive statistics\"\"\"\n    print(\"=== Global Descriptive Statistics ===\")\n    print(f\"Total tomograms: {len(df)}\")\n    print(f\"Tomograms with motors: {df['has_motors'].sum()} ({df['has_motors'].mean()*100:.1f}%)\")\n    \n    print(\"\\n=== Motor Characteristics ===\")\n    motor_stats = valid_motors[['Motor axis 0', 'Motor axis 1', 'Motor axis 2']].describe()\n    print(motor_stats)\n    \n    print(\"\\n=== Tomogram Characteristics ===\")\n    print(df[['volume_size', 'Voxel spacing', 'Number of motors']].describe())\n    \n    \n    plt.figure(figsize=(12, 5))\n    sns.boxplot(x='Number of motors', y='Voxel spacing', data=df)\n    plt.title('Voxel Spacing Distribution by Motor Count')\n    plt.tight_layout()\n    plt.show()\n\ncompute_descriptive_stats(df)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-11T05:13:12.052421Z","iopub.execute_input":"2025-05-11T05:13:12.052724Z","iopub.status.idle":"2025-05-11T05:13:12.347320Z","shell.execute_reply.started":"2025-05-11T05:13:12.052701Z","shell.execute_reply":"2025-05-11T05:13:12.346121Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 3. Spatial Distribution Analysis","metadata":{}},{"cell_type":"code","source":"def analyze_spatial_distribution(valid_motors):\n    \"\"\"Analyze spatial distribution patterns of motors\"\"\"\n    print(\"\\n=== Spatial Distribution Analysis ===\")\n    \n    \n    for axis in ['Motor axis 0', 'Motor axis 1', 'Motor axis 2']:\n        valid_motors[f'{axis}_norm'] = valid_motors[axis] / valid_motors[f'Array shape (axis {axis[-1]})']\n    \n    \n    print(\"\\nUniform Distribution Tests (per axis):\")\n    for axis in ['0', '1', '2']:\n        stat, p = stats.kstest(valid_motors[f'Motor axis {axis}_norm'], 'uniform')\n        print(f\"Axis {axis}: KS stat = {stat:.3f}, p = {p:.4f}\")\n    \n\n    try:\n        from libpysal.weights import DistanceBand\n        from esda.moran import Moran\n        \n        # Create spatial weights matrix (using first 500 motors for demo)\n        sample = valid_motors.iloc[:500]\n        coords = sample[['Motor axis 0', 'Motor axis 1', 'Motor axis 2']].values\n        w = DistanceBand(coords, threshold=50, binary=False)\n        moran = Moran(sample['Motor axis 0'], w)\n        print(f\"\\nSpatial Autocorrelation (Moran's I): {moran.I:.3f}, p = {moran.p_norm:.4f}\")\n    except ImportError:\n        print(\"\\nSpatial autocorrelation analysis requires libpysal and esda packages\")\n\nanalyze_spatial_distribution(valid_motors)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-11T05:13:12.348420Z","iopub.execute_input":"2025-05-11T05:13:12.348840Z","iopub.status.idle":"2025-05-11T05:13:16.441026Z","shell.execute_reply.started":"2025-05-11T05:13:12.348809Z","shell.execute_reply":"2025-05-11T05:13:16.440095Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## 4. Regression Analysis","metadata":{}},{"cell_type":"code","source":"def motor_location_regression(valid_motors):\n    \"\"\"Analyze spatial relationships between motor coordinates\"\"\"\n    print(\"\\n=== Motor Coordinate Regression ===\")\n    \n    \n    for axis in ['0', '1', '2']:\n        valid_motors[f'motor_{axis}_z'] = stats.zscore(valid_motors[f'Motor axis {axis}'])\n    \n    \n    model = ols('motor_2_z ~ motor_0_z + motor_1_z', data=valid_motors).fit()\n    print(model.summary())\n    \n    \n    fig = plt.figure(figsize=(12, 5))\n    ax1 = fig.add_subplot(121, projection='3d')\n    sample = valid_motors.sample(500) if len(valid_motors) > 500 else valid_motors\n    ax1.scatter(sample['motor_0_z'], sample['motor_1_z'], sample['motor_2_z'], alpha=0.5)\n    ax1.set_xlabel('X (Axis 0)')\n    ax1.set_ylabel('Y (Axis 1)')\n    ax1.set_zlabel('Z (Axis 2)')\n    ax1.set_title('Standardized Motor Coordinates')\n    \n    ax2 = fig.add_subplot(122)\n    sm.graphics.plot_partregress_grid(model, fig=fig, exog_idx=['motor_0_z', 'motor_1_z'])\n    plt.tight_layout()\n    plt.show()\n\nmotor_location_regression(valid_motors)","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-05-11T05:13:16.442162Z","iopub.execute_input":"2025-05-11T05:13:16.443054Z","iopub.status.idle":"2025-05-11T05:13:17.167855Z","shell.execute_reply.started":"2025-05-11T05:13:16.443029Z","shell.execute_reply":"2025-05-11T05:13:17.166860Z"}},"outputs":[],"execution_count":null}]}