{"cells":[{"metadata":{},"cell_type":"markdown","source":"## In this step by step EDA, you will find: \n\n* **0. Data dummification before further preprocessing**\n    A way to dummify data in order to get them useable \n    \n* **I. Number and type of labels**\n    A brief look at the number of labels by image and their type\n    \n* **II. What are the most frequent labels?** \n    A brief look at the labels repartition\n    \n* **III. Label coexistence**  \n    A brief study on labels correlation\n    \n* **IV. Random image displayer**\n    A simple function randomly displaying an image of a given label together with its shape and all its tags  "},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport datetime\nimport re\nfrom collections import Counter\n\n# Ploting\nimport matplotlib.pyplot as plt\nimport seaborn as sns \n\nplt.rcParams['figure.figsize'] = (30,30)\n%matplotlib inline\n\nimport os\nprint(os.listdir(\"../input\"))\n\ndef append_ext(fn):\n    return fn+\".png\"\n\ndef remove_ext(fn):\n    return fn[:-4]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"labels_list = pd.read_csv('../input/labels.csv')\nlabels = pd.read_csv(\"../input/train.csv\")\ntest_submission = pd.read_csv('../input/sample_submission.csv')\nlabels['attribute_ids'] = labels['attribute_ids'].str.split(\" \")\nlabels['id'] = labels['id'].apply(append_ext)\ntest_submission['id'] =test_submission['id'].apply(append_ext)\nlabels_list.head()\nlabels = labels","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### 0. Data dummification before further preprocessing"},{"metadata":{"trusted":true},"cell_type":"code","source":"start = datetime.datetime.now()\n\nlabels_dummified = pd.DataFrame(columns=labels_list['attribute_id'])\nd_list = []\nfor index, row in labels.iterrows():\n    for value in row['attribute_ids']:\n        d_list.append({'name':row['id'], \n                       'value':value})\nlabels_dummified = labels_dummified.append(d_list, ignore_index=True)\nlabels_dummified = labels_dummified.groupby('name')['value'].value_counts()\nlabels_dummified = labels_dummified.unstack(level=-1).fillna(0)\nlabels_dummified = labels_dummified[[str(y) for y in sorted([int(x) for x in labels_dummified.columns])]]\nlabels_dummified.columns = labels_list['attribute_name']\nend = datetime.datetime.now()\nprint(\"Elapsed time:\",end-start)\nlabels_dummified.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### I. Number and type of labels\n\nHow many labels do the images have?  "},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize=(20,10))\nax.hist(labels_dummified.sum(axis=1),bins=10)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"All the object have at least one label, and usually two and more. Having more than 7 labels is pretty rare. "},{"metadata":{},"cell_type":"markdown","source":"What are the types of labels and how frequent are they?"},{"metadata":{"trusted":true},"cell_type":"code","source":"n_labels = labels_dummified.sum()\nCounter([re.match('([a-z]+)::\\w+',x)[1] for x in n_labels.index])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There are only 2 type of labels: \n- the culture of the object\n- and the tag, ie its content\n\n2/3 of the labels are related to tags, and 398 different cultures are present. Now it's interesting to see how often images do have a tag and/or a culture label. "},{"metadata":{},"cell_type":"markdown","source":"How frequently images have culture & tag labels? "},{"metadata":{"trusted":true},"cell_type":"code","source":"culture_columns = [x for x in labels_dummified.columns if x.startswith('culture')]\ntag_columns = [x for x in labels_dummified.columns if x.startswith('tag')]\n\nn_culture_labels = labels_dummified[culture_columns].sum(axis=1)\nn_tag_labels = labels_dummified[tag_columns].sum(axis=1)\n\n\nfig, ax = plt.subplots(figsize=(20,10))\nax.hist(n_culture_labels,bins=4,alpha=0.7)\nax.axvline(n_culture_labels.mean())\n\nax.hist(n_tag_labels,bins=9,alpha=0.7)\nax.axvline(n_tag_labels.mean(),color='orange')\n\nprint(\"Number of images with 0 culture label:\",(n_culture_labels == 0 ).sum())\n\nprint(\"Number of images with 0 tag label:\",(n_tag_labels == 0 ).sum())\n\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Tag and culture labels have very different behaviour: \n- There is almost always at least one tag labels. They are also largely non-exclusive, as the average number of tag labels per image is more than 2. \n- On the contrary, the culture label are often exclusive, with about 90k images having exactly one culture. It's also pretty common to have an object without identified culture in the dataset: more than 10k image have not culture label. \n\nThose differences in behaviour will probably allows specific strategies to take them into account. "},{"metadata":{},"cell_type":"markdown","source":"### II. What are the most frequent labels?"},{"metadata":{"trusted":true},"cell_type":"code","source":"labels_count = labels_dummified.sum().reset_index().sort_values(ascending=False,by=0)\nlabels_count","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"labels_count.loc[labels_count['attribute_name'].str.startswith('culture')]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"labels_count.loc[labels_count['attribute_name'].str.startswith('tag')]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The most frequent labels are either very common contents (men, women, flowers etc.)/type of artwork (portrait, inscription etc.) or cultures famous in art history and cultural production (French, Italian, British, American, Japanese, Chinese, Egyptian etc.). \n\nInterestingly, almost all of the less frequent labels are culture ones (subculture, less famous culture, combination of cultures etc.). It may be interesting to see if some culture should not basically be merged, especially when the description seems so accurate that I am not sure that we will find them in the dataset. "},{"metadata":{},"cell_type":"markdown","source":"### III. Label coexistence "},{"metadata":{},"cell_type":"markdown","source":"First, let's have a look at the coexistence between the 100 most frequent labels"},{"metadata":{"trusted":true},"cell_type":"code","source":"most_frequent_labels = labels_dummified[labels_dummified.sum().sort_values(ascending=False)[:100].index]\nlabels_corr = most_frequent_labels.corr()\n\nfig_dims = (30, 16)\nfig, ax = plt.subplots(figsize=fig_dims)\nsns.heatmap(labels_corr,ax=ax)\nplt.plot()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"To have a better look at the correlation, let's now take the 200 most frequent tags, and let's see what are the 40 most important correlation between them. "},{"metadata":{"trusted":true},"cell_type":"code","source":"most_frequent_200_labels = labels_dummified[labels_dummified.sum().sort_values(ascending=False)[:200].index] # We take the most frequent labels\nlabels_corr_200 = most_frequent_200_labels.corr() # We then look at their correlations\nlargest_corr_200 = pd.DataFrame(np.sort(abs(labels_corr_200).values)[:,-2:-1], columns=['2nd-largest'],index=labels_corr_200.index) # And order by the second largest (absolute) correlation \nlargest_corr_40 = largest_corr_200.sort_values(by='2nd-largest',ascending=False).iloc[:40] # And then just take the 40 first.\n\nfig_dims = (30, 16)\nfig, ax = plt.subplots(figsize=fig_dims)\nsns.heatmap(most_frequent_200_labels.loc[:,largest_corr_40.index].corr(),ax=ax, vmax=0.7)\nplt.plot()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looking at the most frequent labels, we can see some main topics:\n- Plants ( tag::flowers, tag::leaves) \n- American (holywoodian/star?) modern culture (culture::american, tag::actress, tag::portrait, tag::women)\n- Ancient Greece (culture::greek, culture::attic, culture::south_italia)\n- Ancient Egypt (culture::egyptian, tag::scarabs,tag::hyeroglyph)\n- Ancient Mesopotamia (tag::cuneiform, tag::tablets, culture::babylonian) \n- Water scene (tag::bodies of water, tag::boat)\n- British culture (culture::british, tag::london) \n- Christian religion (tag:: virgin mary, tag::christ, tag::christian imagery)\n- Landscape (tag:: landscape, tag::tree, tag::houses, tag::mountain)\n- French culture (culture::french, culture::paris) \n- etc."},{"metadata":{"trusted":true},"cell_type":"code","source":"labels_count.loc[labels_count['attribute_name'].str.startswith('culture')]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's now look at the correlation between the culture labels only"},{"metadata":{"trusted":true},"cell_type":"code","source":"culture_labels_dummified = labels_dummified[labels_count.loc[labels_count['attribute_name'].str.startswith('culture')]['attribute_name']]\nmost_frequent_400_culture_labels = culture_labels_dummified[culture_labels_dummified.sum().sort_values(ascending=False)[:400].index] # We take the 400 most frequent labels\nculture_labels_corr = most_frequent_400_culture_labels.corr() #\nculture_corr = pd.DataFrame(np.sort(abs(culture_labels_corr).values)[:,-2:-1], columns=['2nd-largest'],index=culture_labels_corr.index)\nculture_corr = culture_corr.sort_values(by='2nd-largest',ascending=False).iloc[:40]\n\nfig_dims = (30, 16)\nfig, ax = plt.subplots(figsize=fig_dims)\nsns.heatmap(culture_labels_dummified.loc[:,culture_corr.index].corr(),ax=ax, vmax=0.7)\nplt.plot()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Some culture labels are very similar, almost synonyms (london original/after british, augsburg original/after german). Notice that those strong correlations are also probably caused by a relative scarcity of the labels, as we see below."},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.merge(culture_corr,labels_count,left_index=True,right_on='attribute_name') # To see the number of images for those tags","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Let's now look at the correlation between the tag labels only"},{"metadata":{"trusted":true},"cell_type":"code","source":"tag_labels_dummified = labels_dummified[labels_count.loc[labels_count['attribute_name'].str.startswith('tag')]['attribute_name']]\nmost_frequent_400_tag_labels = tag_labels_dummified[tag_labels_dummified.sum().sort_values(ascending=False)[:400].index] # We take the 300 most frequent labels\ntag_labels_corr = most_frequent_400_tag_labels.corr() #\ntag_corr = pd.DataFrame(np.sort(abs(tag_labels_corr).values)[:,-2:-1], columns=['2nd-largest'],index=tag_labels_corr.index)\ntag_corr = tag_corr.sort_values(by='2nd-largest',ascending=False).iloc[:40]\n\nfig_dims = (30, 16)\nfig, ax = plt.subplots(figsize=fig_dims)\nsns.heatmap(tag_labels_dummified.loc[:,tag_corr.index].corr(),ax=ax, vmax=0.7)\nplt.plot()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"raw","source":"There is less strong correlations for tags than for culture. However, the number of images here are more important - and thuse more useful. "},{"metadata":{"trusted":true},"cell_type":"code","source":"pd.merge(tag_corr,labels_count,left_index=True,right_on='attribute_name') # To see the number ofimages for those tags","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There is somehow less strong tags coexistence than for culture labels. However, some thema are clearly delineated: adam & eve, sport, bouddhism etc. "},{"metadata":{},"cell_type":"markdown","source":"### IV. Random image displayer"},{"metadata":{"trusted":true},"cell_type":"code","source":"def print_next_image(attribute_name=\"culture::abruzzi\",dataset='train'):\n    ''' This function generate images having \"attribute_name\" as label'''\n    file = next(att_gen[attribute_name])\n    img=plt.imread('../input/'+dataset+'/'+next(att_gen[attribute_name]))\n    plt.imshow(img,aspect='auto')\n    print(\"File:\",file)\n    print(\"Image shape:\",img.shape)\n    if dataset == 'train':\n        idx_to_name = labels_list.set_index('attribute_id').to_dict()['attribute_name']\n        labs = labels.loc[labels['id'] == file,'attribute_ids'].iloc[0]\n        print('labels:', labs)\n        print('labels names:', [idx_to_name[int(x)] for x in labs])\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"att_gen= {}\nfor att in labels_list['attribute_name'][:10]:\n    att_gen[att] = (x for x in labels_dummified.loc[labels_dummified[att] > 0].index)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print_next_image('culture::akkadian')","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}