{"cells":[{"metadata":{},"cell_type":"markdown","source":"## 💥 Includes\n\n* Creates RGB Image dataset of size 512x512. I am creating this for quick prototyping. The RGB image is created by stacking red(microtubules), green(protein of interest) and blue(nucleoplasm) stain images. \n* Weights and Biases [Artifacts](https://docs.wandb.ai/artifacts) for Dataset versioning. I am splitting the `train.csv` file into train and validation splits. They are logged as Artifacts. \n    * Random Train-Validation split\n    * Stratified Train-Validation split.\n    \n    \n### Datasets\n\n* [HPA: 256x256 dataset](https://www.kaggle.com/ayuraj/HPA256x256DATASET)\n* [HPA: 512x512 dataset](https://www.kaggle.com/ayuraj/HPA512X512DATASET)\n\n### To use Artifacts\n\n* For Random Split\n\n```Python\nimport wandb\nrun = wandb.init()\nartifact = run.use_artifact('ayush-thakur/hpa/split:v0', type='dataset')\nartifact_dir = artifact.download()\n```\n\n* For Stratified Split\n\n```Python\nimport wandb\nrun = wandb.init()\nartifact = run.use_artifact('ayush-thakur/hpa/stratified_split:v0', type='dataset')\nartifact_dir = artifact.download()\n```\n\n![](https://i.imgur.com/xO31ZUL.png)"},{"metadata":{},"cell_type":"markdown","source":"## ❄️ Imports and Setups"},{"metadata":{"trusted":true},"cell_type":"code","source":"%%capture\n!pip install wandb --upgrade","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import os\nimport re\nimport cv2\nimport glob\nimport imageio\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nfrom PIL import Image\nfrom tqdm.notebook import tqdm\nimport matplotlib.pyplot as plt\nfrom skimage.transform import resize\nfrom sklearn.model_selection import train_test_split\n\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import wandb\nfrom kaggle_secrets import UserSecretsClient\n\nuser_secrets = UserSecretsClient()\nwandb_api = user_secrets.get_secret(\"wandb_api\")\n\nwandb.login(key=wandb_api)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"WORKING_DIR_PATH = '../input/hpa-single-cell-image-classification/'\nIMAGE_HEIGHT = 512\nIMAGE_WIDTH = 512","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Ref: https://www.kaggle.com/divyanshuusingh/eda-image-segmentation\nlabel_names= {\n0: \"Nucleoplasm\",\n1: \"Nuclear membrane\",\n2: \"Nucleoli\",\n3: \"Nucleoli fibrillar center\",\n4: \"Nuclear speckles\",\n5: \"Nuclear bodies\",\n6: \"Endoplasmic reticulum\",\n7: \"Golgi apparatus\",\n8: \"Intermediate filaments\",\n9: \"Actin filaments\",\n10: \"Microtubules\",\n11: \"Mitotic spindle\",\n12: \"Centrosome\",\n13: \"Plasma membrane\",\n14: \"Mitochondria\",\n15: \"Aggresome\",\n16: \"Cytosol\",\n17: \"Vesicles and punctate cytosolic patterns\",\n18: \"Negative\"\n}","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Path to image channels(stain)"},{"metadata":{},"cell_type":"markdown","source":"# 🚅 Train Dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"red_images = sorted(glob.glob(WORKING_DIR_PATH+'train/*_red.png'))\ngreen_images = sorted(glob.glob(WORKING_DIR_PATH+'train/*_green.png'))\nblue_images = sorted(glob.glob(WORKING_DIR_PATH+'train/*_blue.png'))\nyellow_images = sorted(glob.glob(WORKING_DIR_PATH+'train/*_yellow.png'))\n\nprint(len(red_images), len(green_images), len(blue_images), len(yellow_images))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Test if the image ids are aligned properly\nfor r, g, b, y in zip(red_images, green_images, blue_images, yellow_images):\n    if re.findall(r'[^\\/]+(?=\\_.)', r)[0] == re.findall(r'[^\\/]+(?=\\_.)', g)[0] == re.findall(r'[^\\/]+(?=\\_.)', b)[0] == re.findall(r'[^\\/]+(?=\\_.)', y)[0]:\n        pass\n    else:\n        print(r)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"TRAIN_SAVE_DIR = '/kaggle/tmp/hpa_512x512_dataset/train/'\n\nos.makedirs(TRAIN_SAVE_DIR+'rgb', exist_ok=True)\n\n!ls /kaggle/tmp/hpa_512x512_dataset/train/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in tqdm(range(len(red_images))):\n    # Image ID \n    image_id = re.findall(r'[^\\/]+(?=\\_.)', red_images[i])[0]\n    \n    # Get red, blue and green channel images. \n    red = np.array(Image.open(red_images[i]))\n    green = np.array(Image.open(green_images[i]))\n    blue = np.array(Image.open(blue_images[i]))\n    \n    # Stack the channels to form RGB image.\n    image_rgb = np.dstack((red, green, blue))\n    # Resize\n    image_rgb = cv2.resize(image_rgb, (IMAGE_HEIGHT, IMAGE_WIDTH), interpolation=cv2.INTER_AREA)\n    # Save image\n    cv2.imwrite(TRAIN_SAVE_DIR+'rgb/'+image_id+'.png', image_rgb)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(len(os.listdir(TRAIN_SAVE_DIR+'rgb/')))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# ⛽ Test Dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"red_images = sorted(glob.glob(WORKING_DIR_PATH+'test/*_red.png'))\ngreen_images = sorted(glob.glob(WORKING_DIR_PATH+'test/*_green.png'))\nblue_images = sorted(glob.glob(WORKING_DIR_PATH+'test/*_blue.png'))\nyellow_images = sorted(glob.glob(WORKING_DIR_PATH+'test/*_yellow.png'))\n\nprint(len(red_images), len(green_images), len(blue_images), len(yellow_images))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Test if the image ids are aligned properly\nfor r, g, b, y in zip(red_images, green_images, blue_images, yellow_images):\n    if re.findall(r'[^\\/]+(?=\\_.)', r)[0] == re.findall(r'[^\\/]+(?=\\_.)', g)[0] == re.findall(r'[^\\/]+(?=\\_.)', b)[0] == re.findall(r'[^\\/]+(?=\\_.)', y)[0]:\n        pass\n    else:\n        print(r)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"TEST_SAVE_DIR = '/kaggle/tmp/hpa_512x512_dataset/test/'\n\nos.makedirs(TEST_SAVE_DIR+'rgb', exist_ok=True)\n\n!ls /kaggle/tmp/hpa_512x512_dataset/test/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in tqdm(range(len(red_images))):\n    # Image ID \n    image_id = re.findall(r'[^\\/]+(?=\\_.)', red_images[i])[0]\n    \n    # Get red, blue and green channel images. \n    red = np.array(Image.open(red_images[i]))\n    green = np.array(Image.open(green_images[i]))\n    blue = np.array(Image.open(blue_images[i]))\n    \n    # Stack the channels to form RGB image.\n    image_rgb = np.dstack((red, green, blue))\n    # Resize\n    image_rgb = cv2.resize(image_rgb, (IMAGE_HEIGHT, IMAGE_WIDTH), interpolation=cv2.INTER_NEAREST)\n    # Save image\n    cv2.imwrite(TEST_SAVE_DIR+'rgb/'+image_id+'.png', image_rgb)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(len(os.listdir(TEST_SAVE_DIR+'rgb/')))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 🎪 Create Kaggle Dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Copy Kaggle API token to ~/.kaggle\n! mkdir -p /root/.kaggle/\n! cp ../input/apitoken/kaggle.json /root/.kaggle/kaggle.json\n# Initialize dataset creation\n! kaggle datasets init -p /kaggle/tmp/hpa_512x512_dataset","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!ls /kaggle/tmp/hpa_512x512_dataset/","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%bash\necho \"{\n  \\\"title\\\": \\\"HPA: 512x512 dataset\\\",\n  \\\"id\\\": \\\"ayuraj/HPA512X512DATASET\\\",\n  \\\"licenses\\\": [\n    {\n      \\\"name\\\": \\\"CC0-1.0\\\"\n    }\n  ]\n}\" > /kaggle/tmp/hpa_512x512_dataset/dataset-metadata.json","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!kaggle datasets create -p /kaggle/tmp/hpa_512x512_dataset/ -u --dir-mode tar\n# ! kaggle datasets version -p /kaggle/tmp/hpa_512x512_dataset -m \"add rgb images\"  --dir-mode tar","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"!rm -rf /root/.kaggle/kaggle.json","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 🎳 Dataset Versioning with W&B\n\nIn this section we will create train and validation dataset using `train.csv`. We will use Weights and Biases Artifacts for dataset versioning. \n\n🐤 Quick introduction on Weights and Biases Artifacts\n\nYou can use W&B Artifacts to store and keep track of datasets, models, and evaluation results across machine learning pipelines. Think of an artifact as a versioned folder of data. You can store entire datasets directly in artifacts, or use artifact references to point to data in other systems.\n\nLearn more about W&B artifacts [here](https://docs.wandb.ai/artifacts). Check out this [YouTube tutorial](https://www.youtube.com/watch?v=Hd94gatGMic&list=PLD80i8An1OEGajeVo15ohAQYF1Ttle0lk&index=3) as well."},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train = pd.read_csv(WORKING_DIR_PATH+'train.csv')\ndf_train.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Ref: https://www.kaggle.com/thedrcat/hpa-single-cell-classification-eda\ndef plot_data_distribution(df):\n    labels = [str(i) for i in range(19)]\n\n    # The number of times a label appears alone.\n    unique_counts = {}\n    for lbl in labels:\n        unique_counts[lbl] = len(df[df.Label == lbl])\n\n    # The total number of times a label appears.\n    full_counts = {}\n    for lbl in labels:\n        count = 0\n        for row_label in df['Label']:\n            if lbl in row_label.split('|'): count += 1\n        full_counts[lbl] = count\n\n    counts = list(zip(map(int,full_counts.keys()), full_counts.values(), unique_counts.values()))\n    counts = np.array(sorted(counts, key=lambda x:-x[1]))\n    counts = pd.DataFrame(counts, columns=['label', 'full_count', 'unique_count'])\n\n    sns.set(style=\"whitegrid\")\n    f, ax = plt.subplots(figsize=(16, 12))\n\n    sns.set_color_codes(\"pastel\")\n    sns.barplot(x=\"full_count\", y=\"label\", data=counts, order=counts.label.values,\n                label=\"full count\", color=\"b\", orient = 'h')\n\n    # Plot the crashes where alcohol was involved\n    sns.set_color_codes(\"muted\")\n    sns.barplot(x=\"unique_count\", y=\"label\", data=counts, order=counts.label.values,\n                label=\"unique count\", color=\"b\", orient = 'h')\n\n    # Add a legend and informative axis label\n    ax.legend(ncol=2, loc=\"lower right\", frameon=True)\n    ax.set(ylabel=\"\",\n           xlabel=\"Counts\")\n    sns.despine(left=True, bottom=True)\n    \n    return unique_counts, full_counts","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"_, full_count = plot_data_distribution(df_train)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Log as Artifact\n\n* Log `train.csv` as artifacts since this is the raw dataset. \n* It will be followed by different splits of this raw dataset. We want to train and validate our model on meaningful split of the dataset. "},{"metadata":{"trusted":true},"cell_type":"code","source":"run = wandb.init(entity='ayush-thakur', project='hpa', job_type='dataset_creation')\nartifact = wandb.Artifact('raw', type='dataset')\nartifact.add_file(WORKING_DIR_PATH+'train.csv')\nrun.log_artifact(artifact)\nrun.join()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 🎱 Random Train-Validation Split"},{"metadata":{"trusted":true},"cell_type":"code","source":"df_train_shuffled = df_train.sample(frac=1)\ndf_train_shuffled.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_split, val_split = train_test_split(df_train_shuffled, test_size=0.2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(f'Training split got {len(train_split.values)} and valdiation split got {len(val_split.values)}')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Plot Train Split distribution"},{"metadata":{"trusted":true},"cell_type":"code","source":"_, train_full_count = plot_data_distribution(train_split)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Plot Validation Split distribution"},{"metadata":{"trusted":true},"cell_type":"code","source":"_, val_full_count = plot_data_distribution(val_split)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Log the splits as Artifact"},{"metadata":{"trusted":true},"cell_type":"code","source":"train_split.to_csv('train_split.csv', index=False)\nval_split.to_csv('val_split.csv', index=False)\n\nrun = wandb.init(entity='ayush-thakur', project='hpa', job_type='dataset_split')\n\nartifact_raw = run.use_artifact('ayush-thakur/hpa/raw:v0', type='dataset')\n\nartifact = wandb.Artifact('split', type='dataset')\nartifact.add_file('train_split.csv')\nartifact.add_file('val_split.csv')\nrun.log_artifact(artifact)\nrun.join()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## ⚽ Stratified Train-Validation Split\n\nStratify based on combination of labels. The unique combinations will be put into train.\nAnother similar stratification idea can be found in this [Stack Overflow thread](https://stackoverflow.com/questions/54890899/not-able-to-use-stratified-k-fold-on-multi-label-classifier)."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Ref: https://www.kaggle.com/samusram/hpa-classifier-explainability-segmentation/comments#Plan\n\nlabel_combinations = df_train['Label'].map(lambda x: str(sorted(list(x))))\nf'There are {sum(label_combinations.value_counts() == 1)} images with unique label combinations out of {len(label_combinations)}.'\n\nlabel_combinations_counts = label_combinations.value_counts()\nunique_label_combs = label_combinations_counts.index[(label_combinations_counts == 1).values]\n\ntrain_ids_unique_combs = df_train['ID'].loc[label_combinations.map(lambda x: x in unique_label_combs)]\n\nnon_unique_combo_bool_idx = label_combinations.map(lambda x: x not in unique_label_combs)\ntrain_ids, val_ids = train_test_split(df_train['ID'].loc[non_unique_combo_bool_idx].values, \n                                        test_size=0.2, \n                                        stratify=label_combinations.loc[non_unique_combo_bool_idx], # sorting present classes in lexicographical order, just to be sure\n                                        random_state=42)\n\ntrain_ids = np.concatenate((train_ids, train_ids_unique_combs))\n\nprint(f'Number of training samples: {len(train_ids)} and validation samples: {len(val_ids)}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"stratified_train_split = df_train.loc[df_train['ID'].isin(train_ids)]\nstratified_val_split = df_train.loc[df_train['ID'].isin(val_ids)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"_, stratified_train_full_count = plot_data_distribution(stratified_train_split)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"_, stratified_val_full_count = plot_data_distribution(stratified_val_split)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Log the Stratified splits as Artifact"},{"metadata":{"trusted":true},"cell_type":"code","source":"stratified_train_split.to_csv('stratified_train_split.csv', index=False)\nstratified_val_split.to_csv('stratified_val_split.csv', index=False)\n\nrun = wandb.init(entity='ayush-thakur', project='hpa', job_type='dataset_stratified_split')\n\nartifact_raw = run.use_artifact('ayush-thakur/hpa/raw:v0', type='dataset')\n\nartifact = wandb.Artifact('stratified_split', type='dataset')\nartifact.add_file('stratified_train_split.csv')\nartifact.add_file('stratified_val_split.csv')\nrun.log_artifact(artifact)\nrun.join()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}