{"cells":[{"metadata":{"_uuid":"d30127c5abec33501bff60055ab83187a546a5c6"},"cell_type":"markdown","source":"As already noted in this competition  we are going to give a struggle *in  the open sea*. About `75000` images contain no ship at all and succesfully sampling from those may benefit any model. However, stratification using no-ship images would be difficult. In the case of images with ships we can either stratify using\n* the number of ships in the image\n* the area of ship pixels in the mask\n\nIn this notebook, I attempt to group no ship images, based on *color information* using bits and pieces from the following two kernels.\n* [https://www.kaggle.com/mpware/stage1-eda-microscope-image-types-clustering](http://)  \n* [https://www.kaggle.com/meaninglesslives/airbus-ship-detection-data-visualization](http://)\n\nFor  every image the dominant color is extracted by applying kmeans on pixel intesities. Then all `75000` are partitioned using this dominant HSV color information into `NUM_CLASSES` classes.  As a result of this analysis we could group no-ship images in a arbitrary number of classes, and sample from those  at random. "},{"metadata":{"trusted":true,"_uuid":"f926376d1380568f248346353c42fcf599e6c0e6","_kg_hide-input":true,"collapsed":true},"cell_type":"code","source":"import os\nimport sys\nimport random\nimport warnings\n\nimport pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nfrom tqdm import tqdm_notebook, tnrange\n\nfrom scipy import signal\n\nimport cv2\nfrom PIL import Image\nimport pdb\nfrom tqdm import tqdm\nimport seaborn as sns\nimport os \nfrom glob import glob\n\nimport warnings\nwarnings.filterwarnings(\"ignore\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d8d19639fac647b9933ba36071a63ea26a638416"},"cell_type":"markdown","source":"<h2> Setting paths"},{"metadata":{"trusted":true,"_uuid":"f1a78c7e479a65719964c9536298f82afb969ebf","_kg_hide-input":true,"collapsed":true},"cell_type":"code","source":"INPUT_PATH = '../input'\nDATA_PATH = INPUT_PATH\nTRAIN_DATA = os.path.join(DATA_PATH, \"train\")\nTRAIN_MASKS_DATA = os.path.join(DATA_PATH, \"train/masks\")\nTEST_DATA = os.path.join(DATA_PATH, \"test\")\ndf = pd.read_csv(DATA_PATH+'/train_ship_segmentations.csv')\npath_train = '../input/train/'\npath_test = '../input/test/'\ntrain_ids = df.ImageId.values\ndf = df.set_index('ImageId')","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8e37b935e5e12c8684e6841ef4bc1578372fcdfa"},"cell_type":"markdown","source":"## Find images containing no ships "},{"metadata":{"trusted":true,"_uuid":"700a11fa65c17917bb60264c9113be54f26d44b1","collapsed":true},"cell_type":"code","source":"images_with_no_ship = df.index[df.EncodedPixels.isnull()==True]\nprint ('Found ' + str(len(images_with_no_ship)) + ' no-ship images') ","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"07b659c908cc30731a71baecef393f36f95e9a01"},"cell_type":"markdown","source":"## Some utility definitions and functions"},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"ec35b771f844066187d0d184f8dad7e3e4a8b61d"},"cell_type":"code","source":"# \n# Number of distinct classes \nNUM_CLASSES = 50\n#\n# In order to reduce computation time, downsample train images. \n# Sure we loose some pixel information this way.....\nIMG_SIZE = 32","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"e73156fc1fca8f80e4f8ff8743660fd9744103ac"},"cell_type":"code","source":"from sklearn.preprocessing import StandardScaler\nfrom sklearn.cluster import KMeans\n\ndef get_filename(image_id, image_type):\n    check_dir = False\n    if \"Train\" == image_type:\n        data_path = TRAIN_DATA\n    elif \"mask\" in image_type:\n        data_path = TRAIN_MASKS_DATA\n    elif \"Test\" in image_type:\n        data_path = TEST_DATA\n    else:\n        raise Exception(\"Image type '%s' is not recognized\" % image_type)\n\n    if check_dir and not os.path.exists(data_path):\n        os.makedirs(data_path)\n\n    return os.path.join(data_path, \"{}\".format(image_id))\n\n\ndef get_image_data_opencv(image_id, image_type, **kwargs):\n    fname = get_filename(image_id, image_type)\n    img = cv2.imread(fname)\n    assert img is not None, \"Failed to read image : %s, %s\" % (image_id, image_type)\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2HSV)\n    img = cv2.resize(img, (IMG_SIZE, IMG_SIZE))\n    return img\n\ndef get_domimant_colors(img, top_colors=2):\n    img_l = img.reshape((img.shape[0] * img.shape[1], img.shape[2]))\n    clt = KMeans(n_clusters = top_colors)\n    clt.fit(img_l)\n    # grab the number of different clusters and create a histogram\n    # based on the number of pixels assigned to each cluster\n    numLabels = np.arange(0, len(np.unique(clt.labels_)) + 1)\n    (hist, _) = np.histogram(clt.labels_, bins = numLabels)\n    # normalize the histogram, such that it sums to one\n    hist = hist.astype(\"float\")\n    hist /= hist.sum()\n    return clt.cluster_centers_, hist","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e70140983b0cda9bd56e72bd624025b02f018cf4","collapsed":true},"cell_type":"code","source":"details = []\nfor imfile in tqdm(images_with_no_ship):\n    image_hsv = get_image_data_opencv(imfile, \"Train\")\n    height, width, l = image_hsv.shape\n    dominant_colors_hsv, dominant_rates_hsv = get_domimant_colors(image_hsv, top_colors=1)\n    dominant_colors_hsv = dominant_colors_hsv.reshape(1, dominant_colors_hsv.shape[0] * dominant_colors_hsv.shape[1])\n    info = (imfile, width, height, dominant_colors_hsv.squeeze())\n    details.append(info)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"c25b2aa1872d2c7bd2a30944c03290c609d213b7"},"cell_type":"markdown","source":" ## Apply kmeans on HSV dominant color"},{"metadata":{"trusted":true,"_uuid":"5bc9fe98c9943b9d7e7379e24c8258b69c89b7da","collapsed":true},"cell_type":"code","source":"cols  = ['image_id', 'image_width', 'image_height', 'hsv_dominant']\ntrainPD = pd.DataFrame(details, columns=cols)\nX = (pd.DataFrame(trainPD['hsv_dominant'].values.tolist())).as_matrix()\nkmeans = KMeans(n_clusters=50).fit(X)\nclusters = kmeans.predict(X)\ntrainPD['hsv_cluster'] = clusters\ntrainPD.head()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"021560f37aa298476c9aaaaf59c238ea67140cdf"},"cell_type":"code","source":"## View partitioning counts","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c944bcd8b6ce6a5b84e83314f681cf490a7eb726","collapsed":true},"cell_type":"code","source":"hist = trainPD.groupby('hsv_cluster')['image_id'].count()\nhist","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e65b0192cd3db15b0e98a2dd7774ee30e78ebff1","collapsed":true},"cell_type":"code","source":"plt.figure(figsize=(12, 6))\nplt.title('#images per partition')\nplt.bar(np.arange(50), hist.values)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"336ce6c4d318752fb4876f2964b8c02ba52113f3"},"cell_type":"code","source":"def plot_images(images, images_rows, images_cols):\n    f, axarr = plt.subplots(images_rows,images_cols,figsize=(16,images_rows*2))\n    for row in range(images_rows):\n        for col in range(images_cols):\n            image_id = images[row*images_cols + col]\n            image = cv2.imread(get_filename(image_id, 'Train'))\n            height, width, l = image.shape\n            ax = axarr[row,col]\n            ax.axis('off')\n            ax.set_title(\"%dx%d\"%(width, height))\n            ax.imshow(image)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f8cd08a734cb2b6b1070601d8b703d198308d9bf"},"cell_type":"markdown","source":"## Some arbitrary plots for clusters, 0, 1 and 30...."},{"metadata":{"trusted":true,"_uuid":"2154b1c5e56d5459e8d52c5a4d46e874faeab315","collapsed":true},"cell_type":"code","source":"plot_images(trainPD[trainPD['hsv_cluster'] == 0]['image_id'].values, 4, 4)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1401fd8395baa9ed3bde2cfe23c2b3869c50bee5","collapsed":true},"cell_type":"code","source":"plot_images(trainPD[trainPD['hsv_cluster'] == 1]['image_id'].values, 4, 4)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"144d5eeece64f4fc17ae5ea2e2abf72783413ff8","collapsed":true},"cell_type":"code","source":"plot_images(trainPD[trainPD['hsv_cluster'] == 30]['image_id'].values, 4, 4)\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1e424345ca6da4f7113b9ec4139080409232d1c2"},"cell_type":"markdown","source":"## Save cluster information "},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"cfcd8dc91e33d20a7f3faa1a75f17b4a4bad56b0"},"cell_type":"code","source":"trainPD.to_csv('noship_clusters.csv', index = False)\n","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}