{"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.12"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":91196,"databundleVersionId":11432986,"sourceType":"competition"}],"dockerImageVersionId":30918,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 🌍 Sentinel-2 Data Processing and Normalization\n\nUnlike last year, we decided to change the representation of Sentinel-2 data, and rather than use the \"compressed\" JPEG data, we provided the raw values in TIFF files, which provided some benefits.\nTo make it easier for you, we provide this tutorial on how to work with the data. In this notebook, we cover: \n- **Understanding TIFF vs. JPG**: Why TIFF is better for geospatial analysis.\n- **Exploring Sentinel-2's 4 Bands**: What information they provide.\n- **Loading and Visualizing TIFF Data**: Using rasterio, matplotlib and folium.\n- **Normalization Techniques**: Different approaches to scale raw Sentinel-2 values, including Log Normalization (LOD).","metadata":{"execution":{"iopub.status.busy":"2025-03-25T10:01:06.847579Z","iopub.execute_input":"2025-03-25T10:01:06.848030Z","iopub.status.idle":"2025-03-25T10:01:06.856526Z","shell.execute_reply.started":"2025-03-25T10:01:06.848001Z","shell.execute_reply":"2025-03-25T10:01:06.854857Z"}}},{"cell_type":"markdown","source":"## ✅ Step1: Installing Rasterio and Folium\n- `rasterio` is a powerful library for reading and writing raster datasets, essential for handling Sentinel-2 imagery.\n- `folium` is a lightweight library for interactive map visualization, useful for geospatial data overlay and exploration.\n\n**If you haven't installed them yet, run the command below.**\n\n**Alternative Approach:** Consider using `GDAL` for more extensive geospatial data support.","metadata":{}},{"cell_type":"code","source":"!pip install rasterio\n!pip install folium","metadata":{"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## ✅ Step 2: Loading Essential Libraries\nFor further processing, we nned to import some Python libraries for data processing:\n- `numpy`: For numerical operations.\n- `pandas`: For handling tabular data, especially CSV files.\n- `rasterio`: For handling geospatial raster data.\n- `matplotlib.pyplot`: For visualizing data.\n- `numpy`: For numerical computations.","metadata":{}},{"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\nimport folium\nimport rasterio\nimport matplotlib.pyplot as plt\nfrom folium.raster_layers import ImageOverlay\nfrom folium.plugins import MousePosition\nfrom rasterio.warp import calculate_default_transform, reproject, Resampling","metadata":{"execution":{"iopub.status.busy":"2025-03-25T15:33:22.484878Z","iopub.execute_input":"2025-03-25T15:33:22.485282Z","iopub.status.idle":"2025-03-25T15:33:24.403215Z","shell.execute_reply.started":"2025-03-25T15:33:22.485246Z","shell.execute_reply":"2025-03-25T15:33:24.402242Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## ✅ Step 3: Understanding TIFF vs. JPG\n\n**Why Use TIFF over JPG?**\n- **Multi-band Support**: TIFF stores multiple spectral bands (e.g., Sentinel-2 has 4+ bands).\n- **Higher Precision**: TIFF supports floating-point values, while JPG is limited to 8-bit integers.\n- **No Compression Loss**: TIFF retains original sensor data, while JPG introduces artifacts.\n- **Better for Analysis**: Geospatial calculations (e.g., NDVI) require accurate reflectance values.","metadata":{"execution":{"iopub.status.busy":"2025-03-25T10:18:25.175870Z","iopub.execute_input":"2025-03-25T10:18:25.176197Z","iopub.status.idle":"2025-03-25T10:18:25.181052Z","shell.execute_reply.started":"2025-03-25T10:18:25.176173Z","shell.execute_reply":"2025-03-25T10:18:25.179611Z"}}},{"cell_type":"code","source":"# Open the file\nwith rasterio.open(\"/kaggle/input/geolifeclef-2025/SatelitePatches/PA-test/00/00/5000000.tiff\") as src:\n    print('Metadata:')\n    print(src.meta)\n    \n    print('\\nBand count:', src.count)\n\n    # Loop over each band to see what info is there\n    for idx in range(1, src.count + 1):\n        desc = src.descriptions[idx - 1]\n        print(f'Band {idx}: {desc}')","metadata":{"execution":{"iopub.status.busy":"2025-03-25T15:33:24.405047Z","iopub.execute_input":"2025-03-25T15:33:24.405690Z","iopub.status.idle":"2025-03-25T15:33:24.470049Z","shell.execute_reply.started":"2025-03-25T15:33:24.405652Z","shell.execute_reply":"2025-03-25T15:33:24.469050Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## ✅ Step 4: Load Sentinel-2 TIFF File and Visualize the Bands","metadata":{}},{"cell_type":"code","source":"file_path = '/kaggle/input/geolifeclef-2025/SatelitePatches/PA-test/00/00/5000000.tiff'\nwith rasterio.open(file_path) as dataset:\n    image = dataset.read()  # Reads all bands\n\n\nfig, axes = plt.subplots(1, 4, figsize=(12, 4))\nband_names = ['Blue', 'Green', 'Red', 'NIR']\n\n# Display the individual bands\nfor i in range(4):\n    axes[i].imshow(image[i-1], cmap='gray')\n    axes[i].set_title(f\"{band_names[i]} Band\")\n    axes[i].axis('off')\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-25T15:33:33.340878Z","iopub.execute_input":"2025-03-25T15:33:33.341254Z","iopub.status.idle":"2025-03-25T15:33:33.826605Z","shell.execute_reply.started":"2025-03-25T15:33:33.341223Z","shell.execute_reply":"2025-03-25T15:33:33.825353Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## ✅ Step 5: Normalizing Sentinel-2 Data\n\n📌 **Why Normalize?**\n- Sentinel-2 values range from 0 to 10,000 (scaled reflectance).\n- Normalization helps standardize data for ML models and visualization.\n\n🔄 **So, what are the different normalization approaches?**\n- 1️⃣ **Min-Max Scaling** (scales values to [0,1])\n- 2️⃣ **Standardization (Z-Score)** (centers around mean with unit variance)\n- 3️⃣ **Histogram Equalization** (enhances contrast)\n- 4️⃣ **Log Scaling (LOD)** (useful for highly skewed distributions)","metadata":{"execution":{"iopub.status.busy":"2025-03-25T13:05:45.910626Z","iopub.status.idle":"2025-03-25T13:05:45.911014Z","shell.execute_reply":"2025-03-25T13:05:45.910866Z"}}},{"cell_type":"markdown","source":"### ✅ Step 5.1: Plotting Histograms of Raw Bands Before Normalization","metadata":{}},{"cell_type":"code","source":"fig, axes = plt.subplots(1, 4, figsize=(12, 4))\n\n# Plot histograms for each band\nfor i in range(4):\n    axes[i].hist(image[i].flatten(), bins=100, color='gray', edgecolor='black')\n    axes[i].set_title(f\"{band_names[i]} Band Histogram\")\n    axes[i].set_xlabel(\"Pixel Value\")\n    axes[i].set_ylabel(\"Frequency\")\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-25T15:33:37.363276Z","iopub.execute_input":"2025-03-25T15:33:37.363606Z","iopub.status.idle":"2025-03-25T15:33:38.740924Z","shell.execute_reply.started":"2025-03-25T15:33:37.363580Z","shell.execute_reply":"2025-03-25T15:33:38.739151Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### ✅ Step 5.2: Functions for Bands Normalization","metadata":{}},{"cell_type":"code","source":"def safe_rescale_to_255(band):\n    \"\"\"Rescale the band to [0, 255] and clip values.\"\"\"\n    return np.clip(band * 255, 0, 255).astype(np.uint8)\n\ndef min_max_normalize(band):\n    \"\"\"Normalize the band to [0, 1] and rescale to [0, 255].\"\"\"\n    normalized = (band - band.min()) / (band.max() - band.min())\n    return safe_rescale_to_255(normalized)\n\ndef standardize(band):\n    \"\"\"Standardize the band and rescale to [0, 255].\"\"\"\n    standardized = (band - band.mean()) / band.std()\n    return safe_rescale_to_255(np.clip(standardized, 0, 1))\n\ndef log_normalize(band):\n    \"\"\"Log normalize the band and rescale to [0, 255].\"\"\"\n    normalized = np.log1p(band - band.min()) / np.log1p(band.max() - band.min())\n    return safe_rescale_to_255(normalized)\n\ndef quantile_normalize(band, low=2, high=98):\n    \"\"\"Normalize the band based on quantiles and rescale to [0, 255].\"\"\"\n    sorted_band = np.sort(band.flatten())\n    quantiles = np.percentile(sorted_band, np.linspace(low, high, len(sorted_band)))\n    normalized_band = np.interp(band.flatten(), sorted_band, quantiles).reshape(band.shape)\n    min_val = np.min(normalized_band)\n    max_val = np.max(normalized_band)\n    return safe_rescale_to_255((normalized_band - min_val) / (max_val - min_val))","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-25T15:33:40.197045Z","iopub.execute_input":"2025-03-25T15:33:40.197455Z","iopub.status.idle":"2025-03-25T15:33:40.206919Z","shell.execute_reply.started":"2025-03-25T15:33:40.197423Z","shell.execute_reply":"2025-03-25T15:33:40.205605Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"data_min_max = np.array([min_max_normalize(band) for band in image])\ndata_log = np.array([log_normalize(band) for band in image])\ndata_quantile = np.array([quantile_normalize(band) for band in image])","metadata":{"execution":{"iopub.status.busy":"2025-03-25T15:33:42.166631Z","iopub.execute_input":"2025-03-25T15:33:42.167016Z","iopub.status.idle":"2025-03-25T15:33:42.197837Z","shell.execute_reply.started":"2025-03-25T15:33:42.166982Z","shell.execute_reply":"2025-03-25T15:33:42.196765Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### ✅ Step 5.3: Visualizing All Bands and All Normalizations in One Figure","metadata":{}},{"cell_type":"code","source":"fig, axes = plt.subplots(4, 5, figsize=(20, 16))\n\n# Original Bands (Raw data)\nfor i in range(4):\n    axes[i, 0].imshow(image[i], cmap='gray')\n    axes[i, 0].set_title(f\"Original {band_names[i]} Band\")\n    axes[i, 0].axis('off')\n\n# Min-Max Normalized Bands\nfor i in range(4):\n    axes[i, 1].imshow(data_min_max[i], cmap='gray')\n    axes[i, 1].set_title(f\"Min-Max {band_names[i]} Band\")\n    axes[i, 1].axis('off')\n\n# Log Normalized Bands\nfor i in range(4):\n    axes[i, 2].imshow(data_log[i], cmap='gray')\n    axes[i, 2].set_title(f\"Log Normalized {band_names[i]} Band\")\n    axes[i, 2].axis('off')\n\n# Quantile Normalized Bands\nfor i in range(4):\n    axes[i, 3].imshow(data_quantile[i], cmap='gray')\n    axes[i, 3].set_title(f\"Quantile Normalized {band_names[i]} Band\")\n    axes[i, 3].axis('off')\n\n# Histograms of Original Bands\nfor i in range(4):\n    axes[i, 4].hist(image[i].flatten(), bins=100, color='gray', edgecolor='black')\n    axes[i, 4].set_title(f\"Histogram {band_names[i]} Band\")\n    axes[i, 4].set_xlabel(\"Pixel Value\")\n    axes[i, 4].set_ylabel(\"Frequency\")\n\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-25T15:33:43.569895Z","iopub.execute_input":"2025-03-25T15:33:43.570275Z","iopub.status.idle":"2025-03-25T15:33:46.644378Z","shell.execute_reply.started":"2025-03-25T15:33:43.570241Z","shell.execute_reply":"2025-03-25T15:33:46.643025Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"### ✅ Step 5.4: Visualizing RGB Image with No and All Normalizations\n","metadata":{}},{"cell_type":"code","source":"def create_rgb_image(red, green, blue):\n    \"\"\"\n    Combine the Red, Green, and Blue bands into a single RGB image.\n    \"\"\"\n    # Stack the bands along the third axis to create an RGB image\n    return np.stack((red, green, blue), axis=-1)\n\n# Create RGB composites for raw and normalized bands\nrgb_original = create_rgb_image(image[0], image[1], image[2])  # Red, Green, Blue bands (original)\nrgb_min_max = create_rgb_image(data_min_max[0], data_min_max[1], data_min_max[2])  # Min-Max normalized\nrgb_log = create_rgb_image(data_log[0], data_log[1], data_log[2])  # Log normalized\nrgb_quantile = create_rgb_image(data_quantile[0], data_quantile[1], data_quantile[2])  # Quantile normalized\n\n# Visualize all RGB images in one figure\nfig, axes = plt.subplots(1, 4, figsize=(12, 12))\n\n# RGB - No normalization (Raw)\naxes[0].imshow(rgb_original)\naxes[0].set_title(\"No normalization (Raw)\")\naxes[0].axis('off')\n\n# Min-Max Normalized RGB Image\naxes[1].imshow(rgb_min_max)\naxes[1].set_title(\"Min-Max Normalized RGB\")\naxes[1].axis('off')\n\n# Log Normalized RGB Image\naxes[2].imshow(rgb_log)\naxes[2].set_title(\"Log Normalized RGB\")\naxes[2].axis('off')\n\n# Quantile Normalized RGB Image\naxes[3].imshow(rgb_quantile)\naxes[3].set_title(\"Quantile Normalized RGB\")\naxes[3].axis('off')\n\n# Make sure the layout is tight\nplt.tight_layout()\nplt.show()","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-25T15:33:46.645953Z","iopub.execute_input":"2025-03-25T15:33:46.646271Z","iopub.status.idle":"2025-03-25T15:33:47.028586Z","shell.execute_reply.started":"2025-03-25T15:33:46.646242Z","shell.execute_reply":"2025-03-25T15:33:47.027521Z"}},"outputs":[],"execution_count":null},{"cell_type":"markdown","source":"## ✅ Step 6: Reproject the patch to WGS84 and Visualize using Rasterio.","metadata":{"execution":{"iopub.status.busy":"2025-03-25T14:42:00.214437Z","iopub.execute_input":"2025-03-25T14:42:00.214815Z","iopub.status.idle":"2025-03-25T14:42:00.219367Z","shell.execute_reply.started":"2025-03-25T14:42:00.214785Z","shell.execute_reply":"2025-03-25T14:42:00.217956Z"}}},{"cell_type":"code","source":"import rasterio\nfrom rasterio.warp import calculate_default_transform, reproject\nfrom rasterio.enums import Resampling\n\ndef reproject_to_wgs84(src_path, dst_path):\n    with rasterio.open(src_path) as src:\n        dst_crs = 'EPSG:4326'\n        transform, width, height = calculate_default_transform(\n            src.crs, dst_crs, src.width, src.height, *src.bounds)\n        \n        kwargs = src.meta.copy()\n        kwargs.update({\n            'crs': dst_crs,\n            'transform': transform,\n            'width': width,\n            'height': height,\n            'nodata': None  # Set nodata to None to avoid adding black/zero values\n        })\n        \n        with rasterio.open(dst_path, 'w', **kwargs) as dst:\n            for i in range(1, src.count + 1):\n                reproject(\n                    source=rasterio.band(src, i),\n                    destination=rasterio.band(dst, i),\n                    src_transform=src.transform,\n                    src_crs=src.crs,\n                    dst_transform=transform,\n                    dst_crs=dst_crs,\n                    resampling=Resampling.nearest)\n\nsrc_tiff = \"/kaggle/input/geolifeclef-2025/SatelitePatches/PA-test/00/00/5000000.tiff\"\nreprojected_tiff = \"reprojected_patch_wgs84.tif\"\nreproject_to_wgs84(src_tiff, reprojected_tiff)","metadata":{"execution":{"iopub.status.busy":"2025-03-25T15:33:50.185827Z","iopub.execute_input":"2025-03-25T15:33:50.186281Z","iopub.status.idle":"2025-03-25T15:33:50.277791Z","shell.execute_reply.started":"2025-03-25T15:33:50.186232Z","shell.execute_reply":"2025-03-25T15:33:50.276216Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"with rasterio.open(reprojected_tiff) as tiff_file:\n    image = tiff_file.read()\n    bounds = tiff_file.bounds\ndata_quantile = np.array([quantile_normalize(band) for band in image])\nrgb_quantile = create_rgb_image(data_quantile[0], data_quantile[1], data_quantile[2])  # Quantile normalized","metadata":{"execution":{"iopub.status.busy":"2025-03-25T15:34:59.839693Z","iopub.execute_input":"2025-03-25T15:34:59.840048Z","iopub.status.idle":"2025-03-25T15:34:59.874877Z","shell.execute_reply.started":"2025-03-25T15:34:59.840019Z","shell.execute_reply":"2025-03-25T15:34:59.873898Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"# Center point of the patch\ncenter_lat = (bounds.top + bounds.bottom) / 2\ncenter_lon = (bounds.left + bounds.right) / 2\n\n# Create folium map with Google Satellite background\nm = folium.Map(location=[center_lat, center_lon], zoom_start=16, tiles=None)\n\n# Add Google Satellite basemap\nfolium.TileLayer(\n    tiles='https://mt1.google.com/vt/lyrs=s&x={x}&y={y}&z={z}',\n    attr='Google Satellite',\n    name='Google Satellite',\n    overlay=False,\n    control=True\n).add_to(m)\n\n# Define the bounds for overlays\nimg_bounds = [[bounds.bottom, bounds.left], [bounds.top, bounds.right]]\n\n# Add RGB composite overlay\nrgb_overlay = ImageOverlay(\n    name='RGB Composite',\n    image=rgb_quantile,\n    bounds=img_bounds,\n    opacity=0.4\n)\nrgb_overlay.add_to(m)\n\n# Add controls\nfolium.LayerControl().add_to(m)\nMousePosition().add_to(m)","metadata":{"execution":{"iopub.status.busy":"2025-03-25T15:35:21.336862Z","iopub.execute_input":"2025-03-25T15:35:21.337275Z","iopub.status.idle":"2025-03-25T15:35:21.354436Z","shell.execute_reply.started":"2025-03-25T15:35:21.337245Z","shell.execute_reply":"2025-03-25T15:35:21.352783Z"},"trusted":true},"outputs":[],"execution_count":null},{"cell_type":"code","source":"m","metadata":{"trusted":true,"execution":{"iopub.status.busy":"2025-03-25T15:35:23.024891Z","iopub.execute_input":"2025-03-25T15:35:23.025272Z","iopub.status.idle":"2025-03-25T15:35:23.041337Z","shell.execute_reply.started":"2025-03-25T15:35:23.025242Z","shell.execute_reply":"2025-03-25T15:35:23.039998Z"}},"outputs":[],"execution_count":null},{"cell_type":"code","source":"","metadata":{"trusted":true},"outputs":[],"execution_count":null}]}