{"cells":[{"metadata":{},"cell_type":"markdown","source":"### Purpose\nI have created this notebook as a culmination of the great amount of work done by other kaggler's in this competition. As I was developing this I realized so much is already done but there is so much we can explore. If you find this remotely useful, please upvote the original authors before even thinking about upvoting this notebook"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"#### Before Starting I want to point out the resources that helped me get fammilliar with the data and this problem\n
Ekhtiar Syed's kernel : https://www.kaggle.com/ekhtiar/eda-find-me-in-the-clouds\n
Aleksandra Deis's kernel: https://www.kaggle.com/aleksandradeis/understanding-clouds-eda\n
Andrew Lukeyanenko's kernel: https://www.kaggle.com/artgor/segmentation-in-pytorch-using-convenient-tools\n
Research Paper: https://arxiv.org/pdf/1906.01906.pdf"},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"Before diving into the data i believe its always important ( at times underrated ) to understand how the data was recorded or generated. Lets look at some important aspects from the research paper shared by the competition organizers
\n>Based on visual inspection four subjective patterns or organization were defined: Sugar, Flower, Fish and Gravel. On cloud labeling days at two institutes, 67 participants classified more than 30,000 satellite images on\na crowd-sourcing platform\n"},{"metadata":{},"cell_type":"markdown","source":"### Descriptions of Cloud Types\nAlthough the classes are subjective but they do exhibit certain characteristics according to the research paper:\n
Sugar: Fine, Random | Flower: Clustered, Well Seperated | Fish: Netwrok like, Skeletal | Gravel: Arcs, intermediate granularity\n\n\n### Labeling\nIts interesting to note the labelling is not definititve but crowdsourced
\n- > Researchers downloaded roughly 10,000 21◦ longitude by 14◦ latitude Terra and Aqua MODIS visible images from NASA Worldview \n- > On the web interface, participants are served an image randomly drawn from our library of 10,000 images. \n- > Users were then asked to draw rectangles around regions where one of the four cloud patterns dominates (Fig. 2a). \n- > Participants had the possibility to draw any number of boxes, including none, with the caveat that the box would _cover at least 10% of the image_. \n- > When an image was classified by _four_ different users, it was retired, i.e. removed from the image library. No user was shown the same image twice.\n\n\n\n### Agreement and Accuracy\nSo agreeing most on Flower, it seems Fish and Gravel are most controversial. This to me is surprising as to my noobie eyes, Sugar and Gravel seem like close cousins. Its important to keep in mind that these were researchers so they must have not gone by an instinctive approach and were used to looking at clouds on a daily basis. Cognitive biases will be inherent though.\nAlso, authors provide transperancy on these metrics:
\n- > the agreement score, used to compare the inter-human agreement, defined as follows: “In which percentage of cases, if one user drew a box of a certain class, did another user also draw a box of the same class, under the condition that the boxes overlap.” The overlap is measured using the Intersection-over-Union (IoU) metric. For above metric an IoU of larger than 0.1 is required\n- > The second metric is the pixel accuracy used to compare the machine learning models to the human predictions. Here, for each pixel, the accuracy of one user (or a machine learning prediction) compared to another user is computed for each pattern. Pixels where both users predict no pattern are omitted for this score.\n\n\n\n#### Deep Learning models used by authors\n1. RetinaNet Object detection with images downscaled to 1050 by 700 pixels\n\n
Source: https://arxiv.org/pdf/1708.02002.pdf\n\n2. Semantic segmentation using U-net structure with a Resnet50 backbone, images downscaled to 700 by 466 pixels\n\n
Source: https://arxiv.org/pdf/1505.04597.pdf\n"},{"metadata":{},"cell_type":"markdown","source":"\n#### Now that we have read the major portions of the paper and have aclear understanding of the background methodology and what all has been tried, lets delve into the data\n\nPS: I blatantly copy code sets from [ekhtiar's](https://www.kaggle.com/ekhtiar/eda-find-me-in-the-clouds) and [aleksandradeis's](https://www.kaggle.com/aleksandradeis/understanding-clouds-eda) kernels so please upvote their work if you find this useful"},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nimport pandas as pd\nimport random\nimport numpy as np\nfrom matplotlib import pyplot as plt\nfrom glob import glob\nfrom PIL import Image\nimport imageio\nimport cv2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"data_path = '/kaggle/input/understanding_cloud_organization'\ntrain_csv_path = os.path.join('/kaggle/input/understanding_cloud_organization','train.csv')\ntrain_image_path = os.path.join('/kaggle/input/understanding_cloud_organization','train_images')\n\n# set paths to train and test image datasets\nTRAIN_PATH = '../input/understanding_cloud_organization/train_images/'\nTEST_PATH = '../input/understanding_cloud_organization/test_images/'\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def load_processdata(loc,**kwargs):\n nomaskvalue = kwargs.get('nomaskvalue',-1)\n \n train_df = pd.read_csv(loc).fillna(nomaskvalue)\n \n # split column\n split_df = train_df[\"Image_Label\"].str.split(\"_\", n = 1, expand = True)\n # add new columns to train_df\n train_df['img'] = split_df[0]\n train_df['lbl'] = split_df[1]\n \n del split_df\n \n # Create labeled cloud type dummies ( but why? idk)\n train_df['fish'] = np.where((train_df['lbl'].str.lower()=='fish') & (train_df['EncodedPixels']!=-1),1,0)\n train_df['sugar'] = np.where((train_df['lbl'].str.lower()=='sugar') & (train_df['EncodedPixels']!=-1),1,0)\n train_df['gravel'] = np.where((train_df['lbl'].str.lower()=='gravel') & (train_df['EncodedPixels']!=-1),1,0)\n train_df['flower'] = np.where((train_df['lbl'].str.lower()=='flower') & (train_df['EncodedPixels']!=-1),1,0)\n \n train_df['Label_EncodedPixels'] = train_df.apply(lambda row: (row['lbl'], row['EncodedPixels']), axis = 1)\n\n return train_df\n\ndef get_image_sizes(train = True):\n '''\n Function to get sizes of images from test and train sets.\n INPUT:\n train - indicates whether we are getting sizes of images from train or test set\n '''\n if train:\n path = TRAIN_PATH\n else:\n path = TEST_PATH\n \n widths = []\n heights = []\n \n images = sorted(glob(path + '*.jpg'))\n \n max_im = Image.open(images[0])\n min_im = Image.open(images[0])\n \n for im in range(0, len(images)):\n image = Image.open(images[im])\n width, height = image.size\n \n if len(widths) > 0:\n if width > max(widths):\n max_im = image\n\n if width < min(widths):\n min_im = image\n\n widths.append(width)\n heights.append(height)\n \n return widths, heights, max_im, min_im\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"trdf = load_processdata(train_csv_path)\ntrdf.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Lets look at some data on cloud type occurances\ntypecols = ['fish','sugar','gravel','flower']\n\nco_occ = trdf.groupby('img')[typecols].sum().T.dot(trdf.groupby('img')[typecols].sum())\nimport seaborn as sns\nsns.heatmap(co_occ, cmap = 'YlGnBu', annot=True, fmt=\"d\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is interesting, gravel-sugar is most popular combo ( sounds like ice-cream tbh ) but its not that far away from sugar-fish which is just 14% lower at 1782. Flower is the least couccoring. Do we have patterns that look alike occuring together? It does make cognitive sense but let's leave that for now ( already hungry ). This is just a two way pairing, Let's use a mutivariate approac to see if absence or presence of certain c;loud types tells us about other cloud types"},{"metadata":{"trusted":true},"cell_type":"code","source":"trdf.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn import svm\nfrom sklearn.model_selection import cross_val_score\nfor c in typecols:\n print(f'{c}: ')\n yvar=c\n xvars = [i for i in typecols if i!=yvar][0:2]\n X = trdf[xvars]\n y = trdf[yvar]\n clf = svm.SVC(kernel='linear', C=1.0)\n\n scores = cross_val_score(clf, X, y, cv=5)\n print(\" Accuracy: %0.2f (+/- %0.2f)\" % (scores.mean(), scores.std() * 2))\n clf.fit(X, y)\n print(f' {xvars},{list(clf.coef_)}')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"result_list = []\nimport statsmodels.api as sm\ntrdf['const'] = 0\nfor c in typecols:\n print(f'-------{c}-------')\n for i in [1]:\n print(f' Iteration: {i}')\n trdf_s = trdf.sample(frac=0.8, replace=False, random_state=i)\n yvar = c\n xvars = [i for i in typecols if i!=yvar][0:2]\n X = trdf_s[xvars]\n y = trdf_s[yvar]\n\n logit = sm.Logit(y, X)\n result = logit.fit()\n result_list.extend([result])\n print(result.summary())\n\ndel trdf['const']","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Given the quasi complete and low coefficients we dont get much from this exercise, the data does not seem to sufficiently indicate any relationship between the clouds presence"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Label count freq\ntrdf.groupby('img')[typecols].sum().sum(axis=1).plot.hist(title='Freq of # labels per image')\nplt.show()\n\ntrdf.groupby('img')[typecols].sum().sum(axis=0).plot(kind='bar',color='green',title='Occuarnce of the Cloud Types')\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's look at them clouds now"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Function to decode the run length mask\ndef rle_to_mask(rle_string, height, width):\n '''\n convert RLE(run length encoding) string to numpy array\n\n Parameters: \n rle_string (str): string of rle encoded mask\n height (int): height of the mask\n width (int): width of the mask \n\n Returns: \n numpy.array: numpy array of the mask\n '''\n \n rows, cols = height, width\n \n if rle_string == -1:\n return np.zeros((height, width))\n else:\n rle_numbers = [int(num_string) for num_string in rle_string.split(' ')]\n #print(rle_numbers)\n rle_pairs = np.array(rle_numbers).reshape(-1,2)\n #print(rle_pairs)\n img = np.zeros(rows*cols, dtype=np.uint8)\n #print(img)\n for index, length in rle_pairs:\n index -= 1\n img[index:index+length] = 255\n img = img.reshape(cols,rows)\n img = img.T\n return img\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"Surface area of masks: Given that the researchers were asked to mark a box with atleast 10% area of the image, its going to be inetersting to see what the distribution looks like ( blatant copy from [ekhtiar's kernel](https://www.kaggle.com/ekhtiar/eda-find-me-in-the-clouds) )"},{"metadata":{"trusted":true},"cell_type":"code","source":"# we will use the following function to decode our mask to binary and count the sum of the pixels for our mask.\ndef get_binary_mask_sum(encoded_mask):\n mask_decoded = rle_to_mask(encoded_mask, width=2100, height=1400)\n binary_mask = (mask_decoded > 0.0).astype(int)\n return binary_mask.sum()\n\n# calculate sum of the pixels for the mask per cloud formation\ntrdf['mask_pixel_sum'] = trdf.apply(lambda x: get_binary_mask_sum(x['EncodedPixels']), axis=1)\n\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Hope I'm doing this right\ntrdf['mask_pixel_perc'] = trdf['mask_pixel_sum']/(2100*1400)\ntrdf.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"trdf.groupby('lbl')['mask_pixel_perc'].describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"trdf.loc[trdf.mask_pixel_perc>0,:].groupby('lbl')['mask_pixel_perc'].describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"g = sns.FacetGrid(trdf, col=\"lbl\")\ng.map(plt.hist, \"mask_pixel_perc\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"g = sns.FacetGrid(trdf.loc[trdf.mask_pixel_perc>0,:], col=\"lbl\")\ng.map(plt.hist, \"mask_pixel_perc\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This makes sense, peaks are around 20% area and standard deviations are comparable accross classes"},{"metadata":{"trusted":true},"cell_type":"code","source":"for i in range(0,5):\n img = cv2.imread(os.path.join(train_image_path, trdf['img'][i]))\n mask_decoded = rle_to_mask(trdf['Label_EncodedPixels'][i][1], img.shape[0], img.shape[1])\n fig, ax = plt.subplots(nrows=1, ncols=2, sharey=True, figsize=(20,10))\n ax[0].imshow(img)\n ax[1].imshow(mask_decoded)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looks about right\n\nSo far we have just tried to see if
\n1. There is any realationship between presence and/or absence of cloud types in an image\n2. How is the bounding box for these cloud types distributed\n3. Commonly occuring pairs of cloud types"},{"metadata":{},"cell_type":"markdown","source":"## *WIP*\nI was always looking at clouds and finding shapes, am curious what a pre trained ResNet50 finds in these clouds"},{"metadata":{"trusted":true},"cell_type":"code","source":"from keras.applications.resnet50 import ResNet50\nfrom keras.preprocessing import image\nfrom keras.applications.resnet50 import preprocess_input, decode_predictions\nimport numpy as np\n\nmodel = ResNet50(weights='imagenet')\ntop_preds=[]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def get_mask_cloud(img_path, img_id, label, mask):\n img = cv2.imread(os.path.join(img_path, img_id), 0)\n mask_decoded = rle_to_mask(mask, img.shape[0], img.shape[1])\n mask_decoded = (mask_decoded > 0.0).astype(int)\n img = np.multiply(img, mask_decoded)\n return img","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# top_preds = []\n# for i in range(0,1):#trdf.shape[0]):\n# img_path = os.path.join(train_image_path, trdf['img'][i])\n# img = image.load_img(img_path, target_size=(224, 224))\n \n# img_print = cv2.imread(os.path.join(train_image_path, trdf['img'][i]))\n# mask_decoded = get_mask_cloud(img_print, trdf['lbl'][i], trdf['EncodedPixels'][i])\n# #img = get_mask_cloud(train_image_path, sample['ImageId'], sample['Label'],sample['EncodedPixels'])\n# print(type(mask_decoded),type(image.img_to_array(img)))\n# print(mask_decoded)\n# print(\"+\"*50)\n# print(image.img_to_array(img))\n \n# x = image.img_to_array(img)\n# x = np.expand_dims(x, axis=0)\n# x = preprocess_input(x)\n\n# preds = model.predict(x)\n# # decode the results into a list of tuples (class, description, probability)\n# # (one such list for each sample in the batch)\n# #print('Predicted:', decode_predictions(preds, top=2)[0])\n# top_preds.extend([decode_predictions(preds, top=2)[0]])\n\n# #trdf['ResNet_toppreds'] = top_preds\n\n# #trdf.head(50)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}