{"cells":[{"metadata":{},"cell_type":"markdown","source":"# About this Notebook\n\nIn my previous notebook [here](https://www.kaggle.com/tanulsingh077/how-to-become-leaf-doctor-with-deep-learning) I showed that there are similarity in Images across different Labels and also some mislabels present . This idea is the outcome of that same fact\n\nIn a normal case where the values/images across different labels might not be related to each other we could have gone with simple StratifiedKfold CV strategy without even a doubt but here where there is a lot of similarity in leaves across different labels and also with the mislabels present would it be a good idea to go with StratifiedKfold?\n\nIf not StratifiedKfold then what do you suggest?\nIn the same notebook I also show that clustering the images in our dataset gives very interesting results and the clusters are also very well formed . In this notebook I cluster the images and form image groups and based on those groups I suggest GroupStratifiedKfold as cv strategy and also compare it with StratifiedKfold Cv strategy\n\n<font color='red'> Note : It would be great if the community also puts forward their views on what is a better CV strategy and why the following dicussion thread </font> :\n\nhttps://www.kaggle.com/c/cassava-leaf-disease-classification/discussion/201699"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"# Essentials\nfrom pathlib import Path\nimport json\nfrom tqdm import tqdm\ntqdm.pandas()\nimport random\n\n# Visuals and CV2\nimport seaborn as sns\nimport matplotlib.pyplot as plt\nimport cv2\nfrom PIL import Image\n%matplotlib inline\n\n# Prelims\nimport pandas as pd\nimport numpy as np\nfrom collections import Counter, defaultdict\n\n# Clustering\nfrom sklearn.cluster import KMeans\nfrom sklearn.decomposition import PCA\nfrom sklearn.cluster import DBSCAN\n\n#keras\nfrom keras.preprocessing.image import load_img\nfrom keras.preprocessing.image import img_to_array \nfrom keras.applications.resnet50 import preprocess_input \n\n# models \nfrom keras.applications.resnet50 import ResNet50\nfrom keras.models import Model","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"BASE_DIR = Path('../input/cassava-leaf-disease-classification')\n\n## Reading DataFrame having Labels\ntrain = pd.read_csv(BASE_DIR/'train.csv')\n\n## Label Mappings\nwith open(BASE_DIR/'label_num_to_disease_map.json') as f:\n    mapping = json.loads(f.read())\n    mapping = {int(k): v for k,v in mapping.items()}\n\nprint(mapping)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train['label_names'] = train['label'].map(mapping)\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Extracting Features for Clustering"},{"metadata":{"trusted":true},"cell_type":"code","source":"def extract_features(image_id, model):\n    file = BASE_DIR/'train_images'/image_id\n    # load the image as a 224x224 array\n    img = load_img(file, target_size=(224,224))\n    # convert from 'PIL.Image.Image' to numpy array\n    img = np.array(img) \n    # reshape the data for the model reshape(num_of_samples, dim 1, dim 2, channels)\n    reshaped_img = img.reshape(1,224,224,3) \n    # prepare image for model\n    imgx = preprocess_input(reshaped_img)\n    # get the feature vector\n    features = model.predict(imgx, use_multiprocessing=True)\n    \n    return features","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model = ResNet50()\nmodel = Model(inputs = model.inputs, outputs = model.layers[-2].output)\n\n''' You can uncommnet the below to get the features but I have already done that and saved the features as a numpy \nfile which you can load directly'''\n\n#train['features'] = train['image_id'].progress_apply(lambda x:extract_features(x,model))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Clustering"},{"metadata":{"trusted":true},"cell_type":"code","source":"###################### use this when extracting features instead of loading from numpy array #########################################\n'''\nfeatures = np.array(train['features'].values.tolist()).reshape(-1,2048)\n'''\n################### else use the following ######################\nfeatures = np.load('../input/cassava/features.npy')\nimage_ids = np.array(train['image_id'].values.tolist())\nlabels = train['label'].values.tolist()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Clustering\nkmeans = KMeans(n_clusters=10,random_state=22)\nkmeans.fit(features)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train['cluster'] = kmeans.labels_","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Visualizing Clusters"},{"metadata":{"trusted":true},"cell_type":"code","source":"#PCA with three principal components\npca_3d = PCA(n_components=3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#And this DataFrame contains three principal components that will aid us\n#in visualizing our clusters in 3-D\nPCs_3d = pd.DataFrame(pca_3d.fit_transform(features))\nPCs_3d.columns = [\"PC1_3d\", \"PC2_3d\", \"PC3_3d\"]\n\ntrain = pd.concat([train,PCs_3d], axis=1, join='inner')\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.set(style = \"darkgrid\")\n\nfig = plt.figure(figsize=(16,11))\nax = fig.add_subplot(111, projection = '3d')\n\nx = train['PC1_3d']\ny = train['PC2_3d']\nz = train['PC3_3d']\n\nax.set_xlabel(\"PC1\")\nax.set_ylabel(\"PC2\")\nax.set_zlabel(\"PC3\")\n\nax.scatter(x, y, z,c=train['cluster'].values)\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* We can see some clusters are very nicely separated and some are intertwined , but all in all it looks like the image grouping is done alright"},{"metadata":{"trusted":true},"cell_type":"code","source":"train['cluster'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* Let's now also visualize the images in respective clusters starting with the cluster having lowest number of images to clusters having highest number of Images"},{"metadata":{},"cell_type":"markdown","source":"# Visualizing Images in Clusters"},{"metadata":{"trusted":true},"cell_type":"code","source":"groups = {}\nfor file,label,cluster in zip(image_ids,labels,kmeans.labels_):\n    if cluster not in groups.keys():\n        groups[cluster] = []\n        groups[cluster].append((file,label))\n    else:\n        groups[cluster].append((file,label))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def view_cluster(cluster):\n    plt.figure(figsize = (25,25));\n    # gets the list of filenames for a cluster\n    files = [ids for ids,_ in groups[cluster]]\n    labels = [lab for _,lab in groups[5]]    \n    # only allow up to 30 images to be shown at a time\n    if len(files) > 30:\n        print(f\"Clipping cluster size from {len(files)} to 25\")\n        start = np.random.randint(0,len(files))\n        print(start)\n        files = files[start:start+25]\n        labels = labels[start:start+25]\n    # plot each image in the cluster\n    for index,(label,file) in enumerate(zip(labels,files)):\n        plt.subplot(5,5,index+1);\n        img = load_img(BASE_DIR/'train_images'/file)\n        img = np.array(img)\n        plt.imshow(img)\n        plt.title(file+' '+\"label: \"+str(label))\n        plt.axis('off')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"view_cluster(3)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"view_cluster(4)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"view_cluster(9)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"view_cluster(6)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* The clusters seem to be reasonable now let's see if the clusters are well represented in every label or not"},{"metadata":{"trusted":true},"cell_type":"code","source":"temp = train.groupby(['label','cluster']).count()['image_id'].reset_index()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(temp[temp['label']==0])\nprint(temp[temp['label']==1])\nprint(temp[temp['label']==2])\nprint(temp[temp['label']==3])\nprint(temp[temp['label']==4])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"* The representations also seem to be fine , lets now create folds and analyze them"},{"metadata":{},"cell_type":"markdown","source":"# GroupStratifiedKFold\n\nThe code for  creation of GroupStratified-Kfold is taken from [here](https://www.kaggle.com/jakubwasikowski/stratified-group-k-fold-cross-validation)"},{"metadata":{"trusted":true},"cell_type":"code","source":"def stratified_group_k_fold(X, y, groups, k, seed=None):\n    labels_num = np.max(y) + 1\n    y_counts_per_group = defaultdict(lambda: np.zeros(labels_num))\n    y_distr = Counter()\n    for label, g in zip(y, groups):\n        y_counts_per_group[g][label] += 1\n        y_distr[label] += 1\n\n    y_counts_per_fold = defaultdict(lambda: np.zeros(labels_num))\n    groups_per_fold = defaultdict(set)\n\n    def eval_y_counts_per_fold(y_counts, fold):\n        y_counts_per_fold[fold] += y_counts\n        std_per_label = []\n        for label in range(labels_num):\n            label_std = np.std([y_counts_per_fold[i][label] / y_distr[label] for i in range(k)])\n            std_per_label.append(label_std)\n        y_counts_per_fold[fold] -= y_counts\n        return np.mean(std_per_label)\n    \n    groups_and_y_counts = list(y_counts_per_group.items())\n    random.Random(seed).shuffle(groups_and_y_counts)\n    for g, y_counts in sorted(groups_and_y_counts, key=lambda x: -np.std(x[1])):\n        best_fold = None\n        min_eval = None\n        for i in range(k):\n            fold_eval = eval_y_counts_per_fold(y_counts, i)\n            if min_eval is None or fold_eval < min_eval:\n                min_eval = fold_eval\n                best_fold = i\n        y_counts_per_fold[best_fold] += y_counts\n        groups_per_fold[best_fold].add(g)\n\n    all_groups = set(groups)\n    for i in range(k):\n        train_groups = all_groups - groups_per_fold[i]\n        test_groups = groups_per_fold[i]\n\n        train_indices = [i for i, g in enumerate(groups) if g in train_groups]\n        test_indices = [i for i, g in enumerate(groups) if g in test_groups]\n\n        yield train_indices, test_indices","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_x = train['image_id'].values\ntrain_y = train.label.values\ngroups = np.array(train.cluster.values)\n\ndef get_distribution(y_vals):\n        y_distr = Counter(y_vals)\n        y_vals_sum = sum(y_distr.values())\n        return [f'{y_distr[i] / y_vals_sum:.2%}' for i in range(np.max(y_vals) + 1)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train['kfold'] = -1\ndistrs = [get_distribution(train_y)]\nindex = ['training set']\n\nfor fold_ind, (dev_ind, val_ind) in enumerate(stratified_group_k_fold(train_x, train_y, groups, k=5)):\n    dev_y, val_y = train_y[dev_ind], train_y[val_ind]\n    dev_groups, val_groups = groups[dev_ind], groups[val_ind]\n    train.loc[val_ind, 'kfold'] = fold_ind\n    \n    assert len(set(dev_groups) & set(val_groups)) == 0\n    \n    distrs.append(get_distribution(dev_y))\n    index.append(f'development set - fold {fold_ind}')\n    distrs.append(get_distribution(val_y))\n    index.append(f'validation set - fold {fold_ind}')\n\ndisplay('Distribution per class:')\npd.DataFrame(distrs, index=index, columns=[f'Label {l}' for l in range(np.max(train_y) + 1)])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train.groupby('kfold')['label'].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train[['image_id','label','label_names','cluster','kfold']].to_csv('cassava_folds.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Conclusion\n\nHere I have analyzed and created StratitifedGroup - 5Fold based on clustering of images. According to me the overall distribution and the folds looks better with this . It would be great to hear community's thoughts on this"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}