{"metadata":{"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":75176,"databundleVersionId":8252256,"sourceType":"competition"}],"dockerImageVersionId":30761,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true},"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.10.14"},"papermill":{"default_parameters":{},"duration":4645.554184,"end_time":"2024-08-27T17:31:50.209885","environment_variables":{},"exception":null,"input_path":"__notebook__.ipynb","output_path":"__notebook__.ipynb","parameters":{},"start_time":"2024-08-27T16:14:24.655701","version":"2.6.0"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"!pip install ultralytics\n!pip install albumentations  # Thêm thư viện augment dữ liệu\n!pip install ensemble-boxes  # Thêm thư viện cho WBF","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":16.473218,"end_time":"2024-08-27T16:14:43.925897","exception":false,"start_time":"2024-08-27T16:14:27.452679","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-08-29T12:45:50.660745Z","iopub.execute_input":"2024-08-29T12:45:50.661280Z","iopub.status.idle":"2024-08-29T12:46:32.658868Z","shell.execute_reply.started":"2024-08-29T12:45:50.661240Z","shell.execute_reply":"2024-08-29T12:46:32.657753Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!pip install wandb","metadata":{"papermill":{"duration":13.728732,"end_time":"2024-08-27T16:14:57.659734","exception":false,"start_time":"2024-08-27T16:14:43.931002","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2024-08-29T12:46:32.661343Z","iopub.execute_input":"2024-08-29T12:46:32.662104Z","iopub.status.idle":"2024-08-29T12:46:45.761815Z","shell.execute_reply.started":"2024-08-29T12:46:32.662053Z","shell.execute_reply":"2024-08-29T12:46:45.760617Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import wandb\nimport pandas as pd\nfrom pathlib import Path\nfrom sklearn.utils import resample\nfrom sklearn.model_selection import train_test_split\nimport os\nfrom shutil import copyfile\nimport yaml\nfrom albumentations import Compose, RandomRotate90, Flip, Transpose, ShiftScaleRotate, RandomBrightnessContrast\nfrom ensemble_boxes import weighted_boxes_fusion\nfrom ultralytics import YOLO","metadata":{"execution":{"iopub.status.busy":"2024-08-29T12:46:45.763287Z","iopub.execute_input":"2024-08-29T12:46:45.763615Z","iopub.status.idle":"2024-08-29T12:46:53.252073Z","shell.execute_reply.started":"2024-08-29T12:46:45.763578Z","shell.execute_reply":"2024-08-29T12:46:53.250929Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Thiết lập API key cho wandb\nwandb.login(key='4f88ff0bbc6e3485258bca7079d7da4c47798ccd')\n\n# Khởi tạo một run mới với tên dự án và tên lượt chạy cụ thể\nwandb.init(project='xuan2261_yolov8_n29th8', name='hopnhatbox10epoch2')\n\n# Paths to data directories\nROOT = Path(\"/kaggle/input/amia-public-challenge-2024\")","metadata":{"execution":{"iopub.status.busy":"2024-08-29T12:46:53.254513Z","iopub.execute_input":"2024-08-29T12:46:53.255066Z","iopub.status.idle":"2024-08-29T12:47:13.390817Z","shell.execute_reply.started":"2024-08-29T12:46:53.255028Z","shell.execute_reply":"2024-08-29T12:47:13.389878Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Read CSV files\ntrain_df = pd.read_csv(ROOT / \"train.csv\")\nimg_size_df = pd.read_csv(ROOT / \"img_size.csv\")\n\n# Merge the original image size information with the training data\ntrain_df = train_df.merge(img_size_df, on='image_id', how='left')\n\n# Map class IDs to new categories\ndef map_class(row):\n    if row['class_id'] == 14:  # \"No finding\"\n        row['new_class_id'] = 0\n        row['x_min'] = 0.0\n        row['y_min'] = 0.0\n        row['x_max'] = 1.0\n        row['y_max'] = 1.0\n    elif row['class_id'] == 0:  # \"Aortic enlargement\"\n        row['new_class_id'] = 1\n    elif row['class_id'] in [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]:  # Assuming these are other abnormalities\n        row['new_class_id'] = 2\n    else:\n        return None  # Ignore any data points that don't match the desired classes\n    return row\n\n# Apply the class mapping function and remove any None values\ntrain_df = train_df.apply(map_class, axis=1)\ntrain_df = train_df.dropna()  # Remove rows where the class is None\n\n# Ensure only 3 classes\nassert train_df['new_class_id'].nunique() == 3, \"There are more than 3 classes in the dataset!\"\n\n# Balance dataset by downsampling and upsampling\ndf_class_0 = train_df[train_df['new_class_id'] == 0]\ndf_class_1 = train_df[train_df['new_class_id'] == 1]\ndf_class_2 = train_df[train_df['new_class_id'] == 2]\n\n# Downsample class 0 (Normal)\ndf_class_0_downsampled = resample(df_class_0, replace=False, n_samples=len(df_class_1), random_state=42)\n\n# Upsample class 2 to match class 1\ndf_class_2_upsampled = resample(df_class_2, replace=True, n_samples=len(df_class_1), random_state=42)\n\n# Combine the balanced data\nbalanced_df = pd.concat([df_class_0_downsampled, df_class_1, df_class_2])\n\n# Debug: Kiểm tra lại phân bổ dữ liệu sau khi cân bằng\nprint(\"Data distribution after balancing:\")\nprint(balanced_df['new_class_id'].value_counts())","metadata":{"execution":{"iopub.status.busy":"2024-08-29T12:47:13.392956Z","iopub.execute_input":"2024-08-29T12:47:13.393388Z","iopub.status.idle":"2024-08-29T12:47:43.626335Z","shell.execute_reply.started":"2024-08-29T12:47:13.393342Z","shell.execute_reply":"2024-08-29T12:47:43.625408Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Chỉ hợp nhất box cho lớp \"Other abnormality\"\ndef handle_overlapping_boxes(df):\n    merged_df = pd.DataFrame(columns=df.columns)\n    \n    for image_id, group in df.groupby('image_id'):\n        group_other_abnormality = group[group['new_class_id'] == 2]\n        if len(group_other_abnormality) > 1:  # Nếu có nhiều box cho lớp \"Other abnormality\"\n            boxes = group_other_abnormality[['x_min', 'y_min', 'x_max', 'y_max']].values.tolist()\n            scores = [1.0] * len(boxes)  # Giả định tất cả box có cùng điểm tin cậy là 1.0\n            labels = group_other_abnormality['new_class_id'].tolist()\n            \n            # Chuẩn hóa box về khoảng [0, 1]\n            orig_width = group_other_abnormality['dim1'].values[0]\n            orig_height = group_other_abnormality['dim0'].values[0]\n            normalized_boxes = [[x_min/orig_width, y_min/orig_height, x_max/orig_width, y_max/orig_height] for x_min, y_min, x_max, y_max in boxes]\n            \n            # Sử dụng WBF để hợp nhất box\n            boxes, scores, labels = weighted_boxes_fusion([normalized_boxes], [scores], [labels], iou_thr=0.4)\n\n            # Đảo ngược chuẩn hóa\n            boxes = [[x_min * orig_width, y_min * orig_height, x_max * orig_width, y_max * orig_height] for x_min, y_min, x_max, y_max in boxes]\n\n            # Thêm các box hợp nhất vào dataframe mới\n            for box, score, label in zip(boxes, scores, labels):\n                x_min, y_min, x_max, y_max = box\n                merged_df = pd.concat([merged_df, pd.DataFrame({\n                    'image_id': [image_id],\n                    'x_min': [x_min],\n                    'y_min': [y_min],\n                    'x_max': [x_max],\n                    'y_max': [y_max],\n                    'new_class_id': [int(label)],\n                    'dim0': group['dim0'].iloc[0],  # Giữ lại kích thước gốc của ảnh\n                    'dim1': group['dim1'].iloc[0]\n                })])\n        else:\n            merged_df = pd.concat([merged_df, group])\n\n    return merged_df.reset_index(drop=True)\n\n# Hợp nhất các box chồng chéo cho lớp \"Other abnormality\"\nbalanced_df = handle_overlapping_boxes(balanced_df)","metadata":{"execution":{"iopub.status.busy":"2024-08-29T12:47:43.628039Z","iopub.execute_input":"2024-08-29T12:47:43.628357Z","iopub.status.idle":"2024-08-29T12:47:58.939719Z","shell.execute_reply.started":"2024-08-29T12:47:43.628324Z","shell.execute_reply":"2024-08-29T12:47:58.938568Z"}}},{"cell_type":"code","source":"# Split dataset into train, validation, and test sets\ntrain_data, temp_data = train_test_split(balanced_df, test_size=0.3, random_state=42, stratify=balanced_df['new_class_id'])\nvalid_data, test_data = train_test_split(temp_data, test_size=2/3, random_state=42, stratify=temp_data['new_class_id'])\n\n# YOLO directory paths\nYOLO_DIR = Path('/kaggle/working/yolov8_data/')\nYOLO_DIR.mkdir(parents=True, exist_ok=True)\n\n# Create necessary directories for YOLO\nfor folder in ['train/images', 'train/labels', 'val/images', 'val/labels', 'test/images', 'test/labels']:\n    (YOLO_DIR / folder).mkdir(parents=True, exist_ok=True)\n\n# Convert dataset to YOLO format\ndef convert_to_yolo_format(df, data_type):\n    \"\"\"\n    Chuyển đổi dataset sang định dạng YOLO và lưu vào thư mục tương ứng.\n\n    Args:\n        df (DataFrame): DataFrame chứa thông tin ảnh và bounding box.\n        data_type (str): Loại dữ liệu ('train', 'val', hoặc 'test').\n\n    Returns:\n        None\n    \"\"\"\n    for i, row in df.iterrows():\n        # Đường dẫn ảnh gốc và đích\n        img_path = ROOT / f\"train/train/{row['image_id']}.png\"\n        img_dest = YOLO_DIR / f\"{data_type}/images/{row['image_id']}.png\"\n        copyfile(img_path, img_dest)\n        \n        # Đường dẫn file nhãn YOLO\n        label_path = YOLO_DIR / f\"{data_type}/labels/{row['image_id']}.txt\"\n        \n        # Tính toán bounding box chuẩn hóa theo kích thước gốc\n        if row['new_class_id'] == 0:  # Ảnh bình thường\n            label_content = \"0 0.5 0.5 1.0 1.0\\n\"\n        else:\n            try:\n                # Kích thước gốc của ảnh\n                orig_width, orig_height = row['dim1'], row['dim0']\n                \n                # Chuẩn hóa bounding box\n                x_center = (row['x_min'] + row['x_max']) / 2 / orig_width\n                y_center = (row['y_min'] + row['y_max']) / 2 / orig_height\n                width = (row['x_max'] - row['x_min']) / orig_width\n                height = (row['y_max'] - row['y_min']) / orig_height\n\n                # Đảm bảo các tọa độ trong phạm vi [0, 1]\n                if not (0 <= x_center <= 1 and 0 <= y_center <= 1 and 0 <= width <= 1 and 0 <= height <= 1):\n                    print(f\"Skipping image {row['image_id']} due to out of bounds coordinates after merging.\")\n                    continue\n\n                # Nội dung nhãn YOLO\n                label_content = f\"{row['new_class_id']} {x_center} {y_center} {width} {height}\\n\"\n            except Exception as e:\n                print(f\"Error processing image {row['image_id']}: {e}\")\n                continue\n\n        # Ghi nội dung nhãn vào file\n        with open(label_path, 'w') as f:\n            f.write(label_content)\n\n\n# Convert all datasets to YOLO format\nconvert_to_yolo_format(train_data, 'train')\nconvert_to_yolo_format(valid_data, 'val')\nconvert_to_yolo_format(test_data, 'test')","metadata":{"execution":{"iopub.status.busy":"2024-08-29T12:47:58.941068Z","iopub.execute_input":"2024-08-29T12:47:58.941390Z","iopub.status.idle":"2024-08-29T12:50:21.745759Z","shell.execute_reply.started":"2024-08-29T12:47:58.941357Z","shell.execute_reply":"2024-08-29T12:50:21.744634Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Remove empty label files if any exist\nlabel_path = YOLO_DIR / 'train/labels'\nfor label_file in os.listdir(label_path):\n    with open(label_path / label_file, 'r') as f:\n        labels = f.readlines()\n        if not labels:  # Nếu tệp nhãn rỗng\n            print(f\"Removing empty label file: {label_file}\")\n            os.remove(label_path / label_file)\n            os.remove(YOLO_DIR / 'train/images' / label_file.replace('.txt', '.png'))  # Xóa hình ảnh tương ứng\n\n# Generate YOLO data config file\ndata_yaml = dict(\n    train=str(YOLO_DIR / 'train/images'),\n    val=str(YOLO_DIR / 'val/images'),\n    nc=3,  # Chỉ định số lượng lớp là 3\n    names=['Normal', 'Aortic enlargement', 'Other abnormality']\n)\n\nwith open(YOLO_DIR / 'data.yaml', 'w') as outfile:\n    yaml.dump(data_yaml, outfile, default_flow_style=False)","metadata":{"execution":{"iopub.status.busy":"2024-08-29T12:50:21.747305Z","iopub.execute_input":"2024-08-29T12:50:21.747701Z","iopub.status.idle":"2024-08-29T12:50:22.025758Z","shell.execute_reply.started":"2024-08-29T12:50:21.747653Z","shell.execute_reply":"2024-08-29T12:50:22.024696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Load model YOLOv8\nmodel = YOLO('yolov8m.pt')  # Bạn có thể thử nghiệm với các phiên bản khác như 'yolov8s.pt', 'yolov8l.pt', hoặc 'yolov8x.pt'\n\n# Huấn luyện mô hình với các tham số đã điều chỉnh\nmodel.train(\n    data='/kaggle/working/yolov8_data/data.yaml', \n    epochs=10,  # Tăng số lượng epoch nếu cần thiết\n    batch=32, \n    imgsz=640,\n    lr0=0.001,  # Giảm learning rate ban đầu\n    augment=True  # Bật data augmentation để tăng tính đa dạng của dữ liệu\n)","metadata":{"execution":{"iopub.status.busy":"2024-08-29T12:50:22.027091Z","iopub.execute_input":"2024-08-29T12:50:22.027418Z","iopub.status.idle":"2024-08-29T12:56:08.628983Z","shell.execute_reply.started":"2024-08-29T12:50:22.027383Z","shell.execute_reply":"2024-08-29T12:56:08.627876Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Đánh giá mô hình\nmetrics = model.val()\nprint(metrics)","metadata":{"execution":{"iopub.status.busy":"2024-08-29T12:56:08.631976Z","iopub.execute_input":"2024-08-29T12:56:08.632326Z","iopub.status.idle":"2024-08-29T12:56:51.158280Z","shell.execute_reply.started":"2024-08-29T12:56:08.632286Z","shell.execute_reply":"2024-08-29T12:56:51.157129Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Dự đoán với mô hình YOLOv8\nresults = model.predict(source='/kaggle/working/yolov8_data/test/images', save=False)","metadata":{"execution":{"iopub.status.busy":"2024-08-29T12:56:51.160214Z","iopub.execute_input":"2024-08-29T12:56:51.160584Z","iopub.status.idle":"2024-08-29T12:59:21.736367Z","shell.execute_reply.started":"2024-08-29T12:56:51.160539Z","shell.execute_reply":"2024-08-29T12:59:21.735443Z"},"trusted":true},"execution_count":null,"outputs":[]}]}