{"cells":[{"metadata":{"trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport cv2\nimport matplotlib.pyplot as plt\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Possible augmentations : 8 isometries of the square\n\nIn this competition, resize or deform images is a bad idea, because of loss of information about steganography. So the ways to do image augmentation are limited.\n\nHowever it's possible to do rotations and flips, by randomly transform images with one of the 8 isometries of the square. Those isometries are elements of dihedral group D4 (https://en.wikipedia.org/wiki/Examples_of_groups#dihedral_group_of_order_8).\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"img = cv2.imread('/kaggle/input/alaska2-image-steganalysis/Cover/00002.jpg')\nimg = img[:,:,::-1]\n\nfig, axs = plt.subplots(2,4, figsize=(12,6))\n\nfor i in range(8):\n    ax = axs[i//4, i%4]\n    im = img.transpose((i//4,1-i//4,2)) # optionnal rotation\n    if i%2 == 1:\n        im = im[::-1,:,:] # optionnal vertical flip\n    if (i//2)%2 == 1:\n        im = im[:,::-1,:] # optionnal horizontal flip\n    \n    ax.set_title(f'isometry_{i}')\n    ax.imshow(im)\n    ax.axis('off')\n\nplt.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Loss and competition metric results by isometry\n\nThe aim of this little kernel is to check if TTA is usefull.\n\nTo perform TTA with one image, we take the mean of probabilities (given by the CNN) of the 8 isometries of this image.\n\nOnce the model is trained, we realise TTA on a validation set of 15k images. Each isometry of each image is present once in the validation set, so we have 15k x 8 = 120k in our validation set.\n\nIn the following tab we present the results for each isometry and for the mean of the 8 isometries.\n\n\n","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df_results_by_iso = pd.read_csv('/kaggle/input/resultsbyiso/results_by_iso.csv')\ndf_results_iso_0 = pd.read_csv('/kaggle/input/resultsiso0/results_iso_0.csv').rename(columns={'compression JPEG':'compression_JPEG'})\ndf_results_iso_2 = pd.read_csv('/kaggle/input/results-iso-2/results_iso_2.csv')\n\n\n# errors in dataframe names :\n\nmask = df_results_by_iso.isometry == 'mean of 8 Iso_0'\ndf_results_by_iso.loc[mask, 'isometry'] = 'mean of 8 isometries'\n\nfor i in range(8):\n    mask = df_results_by_iso.isometry == str(i)\n    df_results_by_iso.loc[mask, 'isometry'] = f'iso_{i}'\n    \ndf_plot = pd.concat([df_results_by_iso.iloc[-9:,:], df_results_iso_0.iloc[-1:,:], df_results_iso_2.iloc[-1:,:]])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"mask = df_results_by_iso.compression_JPEG == 'all'\ndf_results_by_iso[mask].iloc[:,1:]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Comparing the results with 8 repetitions of the same isometry\n\nIt seems that TTA is very usefull, indeed the competition metric is jumping from approximately 0.912 to 0.923 !\n\nHowever, since my pytorch CNN is evaluating images not with option \"model.eval()\" but \"model.train()\", the predictions are not deterministic. So the improvement with TTA can be due to the variability of predictions. In this case repeating the validation several times and take the mean of probabilities is a good way to improve the score.\n\nTo ensure that TTA is still usefull, a good way is to repeat 8 predictions of the same isometry, take the mean of probabilities and compare the results with TTA. Here we are doing this for both \"iso 0\" and \"iso 2\", always on the 15k validation set :","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"df = pd.concat([df_results_iso_0, df_results_iso_2])\nmask = df.compression_JPEG == 'all'\ndf[mask].iloc[:,1:]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"By repeating 8 times the evaluation of each image and taking the mean we reach 0.919, against 0.923 with TTA.\nWe can conclude than TTA is usefull, gaining a 0.004 competition metric in our case.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(12,6))\n\nax.bar(range(11), df_plot.weighted_AUC, width=0.7)\n\nax.set_xticks(range(11))\nax.set_xticklabels(df_plot.isometry, rotation=50)\n\nplt.title('Competition metric scores : one isometry vs TTA')\nplt.ylim([0.91, 0.924])\nplt.ylabel('weighted AUC')\n\nplt.show()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}