{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Disclaimer: I have **NO** knowledge in art in any way.\n## Partially Inspired By [iMet Data Analysis, Plus!](https://www.kaggle.com/hsakizero/imet-data-analysis-plus)"},{"metadata":{},"cell_type":"markdown","source":"# Prerequsites\nImport necessary libraries and configure custom plot style "},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import os\nimport cv2\nimport torch\nimport random\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\n# custom styling\nsns.set_style('dark')\nsns.set(font_scale=1.75)\n\n# For reproducibility purpose\ndef seed_everything(seed=2019):\n    random.seed(seed)\n    os.environ['PYTHONHASHSEED'] = str(seed)\n    np.random.seed(seed)\n    torch.manual_seed(seed)\n    torch.cuda.manual_seed(seed)\n    torch.backends.cudnn.deterministic = True\n\nseed_everything()\n\n# Read Data\ntraining_data = pd.read_csv('../input/train.csv')\nattribute_name = pd.read_csv('../input/labels.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Most Frequent Attrbutes\nDisplay top 30 most frequent attributes in training data  "},{"metadata":{"trusted":true},"cell_type":"code","source":"# put all training labels to a list\nattributes = training_data.attribute_ids.values.tolist()\n# count frequency of each unique label\ncounter = pd.Series(' '.join(attributes).split()).value_counts().to_frame().reset_index()\n# reset column name\ncounter.columns = ['attribute_id', 'frequency']\n# cast id into type int\ncounter['attribute_id'] = counter['attribute_id'].astype(int)\n# merge with attribute names\ncounter = counter.merge(attribute_name)\n# plot top 30\nplt.figure(figsize=(18, 12))\nsns.barplot(data=counter.head(30), x='frequency', y='attribute_name');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# How Much Do They Weight?\nFor any label $k$, define its frequency as $f_k$ and weight as $w_k$:\n\\begin{align} w_k = {f_k \\over \\sum_{i=1}^n f_i}\\text{ , n = total number of classes} \\end{align}\n\nLet's take a look at how much do top 30 most frequent attributes weight in total"},{"metadata":{"trusted":true},"cell_type":"code","source":"# sum of frequency of all labels\nsum_of_frequency = counter['frequency'].values.sum()\n# calculate weight of each label\ncounter['weight'] = counter['frequency'] / sum_of_frequency\n\ndef show_top_n(n):\n    assert isinstance(n, int) and n > 0\n    top_n = counter['weight'].head(n).values.sum()\n    plt.figure(figsize=(7, 7))\n    plt.pie(x=[top_n, 1.0 - top_n], \n            explode=[0.1, 0.0], \n            labels=['Top %s Most Frequent Attributes' %n, 'Other Attributes'], \n            autopct='%.2f%%', \n            textprops={'size':'larger'})\n\nshow_top_n(30)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Observation 1\n**30 Attributes Account For Over 50% of Total Attribute Counts!!**\n\nRemember we have in total 1103 classes, this implies serious class imbalance in the training data."},{"metadata":{},"cell_type":"markdown","source":"# What Do Arts About French Men Look Like?\nLet's look at some potentially interesting combination of frequent attributes.\n\nWe will start off wth `culture::french` and `tag::men`."},{"metadata":{"trusted":true},"cell_type":"code","source":"labels = pd.read_csv('../input/labels.csv')\nfrench = labels.loc[labels['attribute_name'] == 'culture::french']['attribute_id'].item()\nmen = labels.loc[labels['attribute_name'] == 'tag::men']['attribute_id'].item()\n\ndef is_french_men(row):\n    attr = row['attribute_ids']\n    return len(attr.split()) == 2 and str(french) in attr and str(men) in attr\n\n# Source: https://stackoverflow.com/questions/11159436/multiple-figures-in-a-single-window\ndef plot_figures(figures, nrows = 1, ncols=1):\n    \"\"\"Plot a dictionary of figures.\n\n    Parameters\n    ----------\n    figures : <title, figure> dictionary\n    ncols : number of columns of subplots wanted in the display\n    nrows : number of rows of subplots wanted in the figure\n    \"\"\"\n\n    fig, axeslist = plt.subplots(ncols=ncols, nrows=nrows)\n    fig.set_size_inches(20, 18)\n    for ind,title in enumerate(figures):\n        axeslist.ravel()[ind].imshow(figures[title], cmap=plt.gray())\n        axeslist.ravel()[ind].set_title(title)\n        axeslist.ravel()[ind].set_axis_off()\n    plt.tight_layout() # optional\n    \ndef sample(data, n):\n    result = {}\n    samples = data.sample(n, random_state=2019)\n    for i, info in enumerate(samples.values):\n        filename, _ = info\n        title = 'sample_%s' % str(i+1)\n        img_path = ''.join(('../input/train/', filename,'.png'))\n        result[title] = cv2.imread(img_path)\n    return result\n\nsamples = sample(training_data[training_data.apply(is_french_men, axis=1)], 20)\nplot_figures(samples, 4, 5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# What About British Women?\nLet's see if there are obvious visual differences between the two"},{"metadata":{"trusted":true},"cell_type":"code","source":"british = labels.loc[labels['attribute_name'] == 'culture::british']['attribute_id'].item()\nwomen = labels.loc[labels['attribute_name'] == 'tag::women']['attribute_id'].item()\ndef is_british_women(row):\n    attr = row['attribute_ids']\n    return len(attr.split()) == 2 and str(british) in attr and str(women) in attr\n\nsamples = sample(training_data[training_data.apply(is_british_women, axis=1)], 20)\nplot_figures(samples, 4, 5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Observation 2\nNotice how **sample_19** in both groups are very noisy. I suspect that even though these two images have completely different attributes, there is still a certain level of unnecessary similarity between the two due to all those noisy black and white pixels. **This certainly makes it harder for models to capture distinctive features.**  "},{"metadata":{},"cell_type":"markdown","source":"# Observation 3\nNotice many samples drawn look very sketchy, i.e. **sample_14** in group 2. I suspect that **slightly increasing contrast or strengthen edges during training may help our model understand these images better.**"},{"metadata":{},"cell_type":"markdown","source":"## To Be Continued..."}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}