{"cells":[{"metadata":{"_uuid":"171fbcb2ae32b4875e5b95bcc13848a5ae549f2f"},"cell_type":"markdown","source":"# Data Exploration\nDutch AI team\n\n## Version\n\n25/10/18\nSetting up notebook and team. Doing first data exploration."},{"metadata":{"_uuid":"655e90232728d6d8761745cd1c1ad4e372aba059"},"cell_type":"markdown","source":"# Introduction\nStarting a new Kaggle competition is always an exciting moment! You will start to explore most often a new domain with a never seen before dataset. Over the coming few week we will not only learn a lot about proteins, but also learn new AI techniques. In the best case, we might actually improve upon existing algorithms. How nice would that be? ;)\n\nIn this notebook we explore the dataset to get a better understanding of the domain and how to handle the data. Machine learning models will we build in a seperate notebook, once we feel confortable with the data.\n\n\n## File descriptions\nA full description of the files can be read here https://www.kaggle.com/c/human-protein-atlas-image-classification/data. The dataset consist of the following files:\n\n- *train.csv* - filenames and labels for the training set.\n- *sample_submission.csv* - filenames for the test set, and a guide to constructing a working submission.\n- *train.zip* - All images for the training set.\n- *test.zip* - All images for the test set.\n\nLet's first start with exploring the train.csv file."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# load some default libraries\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\n# define a PATH variables so you easily use this notebook in a different computing environment\nPATH = \"../input/\"\n\n# read csv ad Pandas dataframe\ndf_train = pd.read_csv(PATH + \"train.csv\")\ndf_train.info()","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"# print head of dataframe\ndf_train.head(5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1c0ccf96f1d67f333c955deeecccb5e5f4d8f7ca"},"cell_type":"markdown","source":"From the full description we learn that the labels are protein organelle localization labels for each sample, defined by\n\n0.  Nucleoplasm  \n1.  Nuclear membrane   \n2.  Nucleoli   \n3.  Nucleoli fibrillar center   \n4.  Nuclear speckles   \n5.  Nuclear bodies   \n6.  Endoplasmic reticulum   \n7.  Golgi apparatus   \n8.  Peroxisomes   \n9.  Endosomes   \n10.  Lysosomes   \n11.  Intermediate filaments   \n12.  Actin filaments   \n13.  Focal adhesion sites   \n14.  Microtubules   \n15.  Microtubule ends    \n16.  Cytokinetic bridge   \n17.  Mitotic spindle   \n18.  Microtubule organizing center   \n19.  Centrosome   \n20.  Lipid droplets   \n21.  Plasma membrane   \n22.  Cell junctions   \n23.  Mitochondria   \n24.  Aggresome   \n25.  Cytosol   \n26.  Cytoplasmic bodies   \n27.  Rods & rings\n\n## Plotting images\n\nTo get a better intuition for the data, let's plot the images for the first data sample (first row in dataframe)."},{"metadata":{"trusted":true,"_uuid":"81eeb1255ceb70670c9d6cde3d92f3ca70ae6dfd"},"cell_type":"code","source":"# put labels in list for easy printing\nprotein_labels = [\n    'Nucleoplasm', \n    'Nuclear membrane', \n    'Nucleoli', \n    'Nucleoli fibrillar center', \n    'Nuclear speckles', \n    'Nuclear bodies', \n    'Endoplasmic reticulum', \n    'Golgi apparatus', \n    'Peroxisomes', \n    'Endosomes', \n    'Lysosomes', \n    'Intermediate filaments', \n    'Actin filaments', \n    'Focal adhesion sites', \n    'Microtubules',\n    'Microtubule ends', \n    'Cytokinetic bridge', \n    'Mitotic spindle', \n    'Microtubule organizing center',\n    'Centrosome', \n    'Lipid droplets',\n    'Plasma membrane', \n    'Cell junctions',\n    'Mitochondria', \n    'Aggresome', \n    'Cytosol', \n    'Cytoplasmic bodies',\n    'Rods & rings']\n\n# define function to print labels\ndef print_labels(target):\n    label_ints = [int(l) for l in target.split()]\n    for i in label_ints:\n        print(\"{} - {}\".format(i, protein_labels[i]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5e72020084a94e741bac2ee6720731aa78892597"},"cell_type":"code","source":"# each datapoint consists of four images with different color \n# define function to plot those four images\ndef plot_protein_images(id):\n    fig, axs = plt.subplots(1, 4, figsize=(16,4))\n\n    for i, color in enumerate(['red', 'green', 'yellow', 'blue']):\n        filename = \"train/{}_{}.png\".format(id, color)\n        im = plt.imread(PATH + filename)\n        axs[i].imshow(im, cmap='binary')\n        axs[i].set_title(color)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f5bd20355caccdf8b1eeb546ba0a10d79c81de3c"},"cell_type":"code","source":"plot_protein_images(df_train.Id[0])\nprint_labels(df_train.Target[0])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f38c775678966b615248bcf215ed6474509aede4"},"cell_type":"code","source":"plot_protein_images(df_train.Id[1])\nprint_labels(df_train.Target[1])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"69584d701e2fae4386cc3f0ae636e4ca04b105d7"},"cell_type":"markdown","source":"From these images we get a better intuition of the data. Sometimes, you should spent some time on making a 'nice' representation of your data. It's not only fun to do, but representing data in different formats might emphasize different aspects of data. In this case, we could try to merge all four images ('channels') into a RGB images (3 channels). For this we define a 4x3 transformation matrix that turns 4 channels into 3 rgb channels."},{"metadata":{"trusted":true,"_uuid":"bfa8eb28fed73224b94f54dc08ca6085be9d5caf"},"cell_type":"code","source":"def plot_color_protein_images(id, ax=None, figsize=(10,10)):\n    # use ax argument so this function can be using to plot in a grid using axes\n    if ax==None:\n        fig, ax = plt.subplots(1, 1, figsize=figsize)\n\n    # read all color images\n    all_images = np.empty((512,512,4))\n    for i, color in enumerate(['red', 'green', 'yellow', 'blue']):\n        all_images[:,:,i] = plt.imread(PATH + \"train/{}_{}.png\".format(id, color))\n\n    # define transformation matrix\n    # note that yellow is made usign red and green\n    # but you can tune this color conversion yourself\n    T = np.array([[1,0,1,0],[0,1,1,0],[0,0,0,1]])\n    \n    # convert to rgb\n    rgb_image = np.matmul(all_images.reshape(-1, 4), np.transpose(T))\n    rgb_image = rgb_image.reshape(all_images.shape[0], all_images.shape[0], 3)\n    rgb_image = np.clip(rgb_image, 0, 1)\n    \n    # plot\n    ax.imshow(rgb_image)\n    ax.set(xticks=[], yticks=[])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b59bc7b76b355aaee982cd89901ecd08098da597"},"cell_type":"code","source":"plot_color_protein_images(df_train.Id[1])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d6e0aab785d4363f1fab0d24137ff611789397fb"},"cell_type":"markdown","source":"Finally, let's print several nice plots of the proteins."},{"metadata":{"trusted":true,"_uuid":"4155966b6b6ce375f83d2fe1dcf2e1044df2ddb4"},"cell_type":"code","source":"# plot color protein images with target as title\nn_rows, n_cols = 4, 8\nfig, axs = plt.subplots(n_rows, n_cols, figsize=(18, 10))\naxs = axs.ravel()\nN = n_rows * n_cols\nfor i in range(N):\n    plot_color_protein_images(df_train.Id[i], axs[i])\n    axs[i].set_title(df_train.Target[i])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f4add8213a67db2a63197780a69966bb03437408"},"cell_type":"markdown","source":"## Statistics\n\nIt is important to know how many samples there are per target. Sometimes a dataset is 'imbalanced', e.g. there are many more samples of target A than target B. This could impact the performance of the model and requires special training preparation."},{"metadata":{"trusted":true,"_uuid":"4c4fb002dbd4d40986c3ec6cfdb414c85bec7f62"},"cell_type":"code","source":"# number of samples\nn_datapoints = df_train.shape[0]\nn_datapoints","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"918a5b7c415785812c688f6487399acb011ec127"},"cell_type":"code","source":"# count protein targets\ncount = np.zeros(len(protein_labels))\nfor target in df_train.Target:\n    label_ints = [int(l) for l in target.split()]\n    count[label_ints] = count[label_ints] + 1\n\nplt.figure(figsize=(14,6))\nplt.bar(range(len(protein_labels)), count)\nplt.ylabel('count')\nplt.xticks(range(len(protein_labels)), protein_labels, rotation=-90);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2164927911f04080c3970da1d2e67cdb0b964eeb"},"cell_type":"markdown","source":"As we can see from this bar diagram, the dataset is imbalanced. This we should take into account when training a model.\n\nNext, let's see is some targets are correlated."},{"metadata":{"trusted":true,"_uuid":"57d6e3413f2a53a3e046847a484b2a14dc33f238"},"cell_type":"code","source":"# create array with target encoding\nn_labels = len(protein_labels)\na_targets = np.zeros((n_datapoints, n_labels))\nfor i, target in enumerate(df_train.Target):\n    label_ints = [int(l) for l in target.split()]\n    a_targets[i, label_ints] = 1 ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"121afe4c06a72588b4a353ae1786e9345da4f76b"},"cell_type":"code","source":"# calculate correlation matrix\nC = np.corrcoef(a_targets, rowvar=False)\nC.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d8c1216819d5d16daadefcee6782bec66f2219f0"},"cell_type":"code","source":"plt.figure(figsize=(10,8))\nsns.heatmap(C, xticklabels=protein_labels, yticklabels=protein_labels)\nplt.title('Correlation matrix')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"19c6fdfcd6bb4f40240846edbe8927b80f10cb83"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}