{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nimport os\n#print(os.listdir(\"../input\"))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ef0eeaf0f247441eb06a9162229e2c8e6537f744"},"cell_type":"markdown","source":"### Aim of the Compition\n\n\nAnalyse the Protein cell from the biomedical image and find the pattern to accelerate the understanding of human cells behaviour and optimise disease [such as breast cancer, prostate cancer, colon cancer, diabetes, autoimmune diseases, ovarian cancer and renal failure]."},{"metadata":{"_uuid":"15003c1aea0d83f530336843091c9355759a7b2b"},"cell_type":"markdown","source":"### Company Information \n\n#### Human Protein Atlas\n\n![](https://www.ebi.ac.uk/gxa/resources/images/experiment-list-latest/human_protein_atlas.png)\n\nThe Human Protein Atlas (HPA) is a Swedish-based program started in 2003 with the aim to map of all the human proteins in cells, tissues and organs using integration of various omics technologies, including antibody-based imaging, mass spectrometry-based proteomics, transcriptomics and systems biology. All the data in the knowledge resource is open access to allow scientists both in academia and industry to freely access the data for exploration of the human proteome. [ More Information ](https://en.wikipedia.org/wiki/Human_Protein_Atlas)\n\nCompany Mojor Working with Three Project :\n* [Tissue Atlas ](https://www.proteinatlas.org/tissue)\n* [Cell Atlas](https://www.proteinatlas.org/cell)\n* [Pathology Atlas](https://www.proteinatlas.org/pathology)\n\n"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"### Definition of Protein Structure\n\nProtein structure is the three-dimensional arrangement of atoms in an amino acid-chain molecule. Proteins are polymers – specifically polypeptides – formed from sequences of amino acids, the monomers of the polymer. A single amino acid monomer may also be called a residue indicating a repeating unit of a polymer [Reference](https://en.wikipedia.org/wiki/Protein_structure)\n\n![](http://paulbrinson.weebly.com/uploads/5/9/8/1/59812627/1628628_orig.gif)"},{"metadata":{"_uuid":"8d79d4273a0d5965199648d45230d86ad4a5b5d6"},"cell_type":"markdown","source":"### Cell Structure \n\nWhat do all cells have in common?\n\nSame chemical makeup\n\n* Proteins (made up of amino acids; many are enzymes)\n* Nucleic acids (DNA, RNA)\n* Lipids (fatty or oily molecules)\n* Carbohydrates (sugars and starches)\n\n![](https://s3.studylib.net/store/data/008655064_1-388dce9b3c81ae4c6ed884d95c10f722-260x520.png)"},{"metadata":{"_uuid":"587dde44d0a5a4eee32115603c844d78b5895ca0"},"cell_type":"markdown","source":"![](https://biologydictionary.net/wp-content/uploads/2017/03/Cell-membrane-diagram.jpg)\n### Important Functions of Protein in Your Body\n\n* Growth and Maintenance\n* Causes Biochemical Reactions\n\t* Digestion\n\t* Energy production\n\t* Blood clotting\n\t* Muscle contraction\n* Acts as a Messenger\n* Provides Structure\n* Maintains Proper pH\n\t* The balance between acids and bases is measured using the pH scale. It ranges from 0 to 14, with 0 being the most acidic, 7 neutral and 14 the most alkaline.\n* Balances Fluids\n* Bolsters Immune Health\n* Transports and Stores Nutrients\n* Provides Energy"},{"metadata":{"_uuid":"157815c11847ee2bacfa98b74b79b1b543d8764d"},"cell_type":"markdown","source":"### Protein Interactions with Disease\n\n\nProteins do not function in isolation; it is their interactions with one another and also with other molecules (e.g. DNA, RNA) that mediate metabolic and signaling pathways, cellular processes, and organismal systems. Due to their central role in biological function, protein interactions also control the mechanisms leading to healthy and diseased states in organisms. Diseases are often caused by mutations affecting the binding interface or leading to biochemically dysfunctional allosteric changes in proteins. Therefore, protein interaction networks can elucidate the molecular basis of disease, which in turn can inform methods for prevention, diagnosis, and treatment. In this chapter, we will describe the computational approaches to predict and map networks of protein interactions and briefly review the experimental methods to detect protein interactions. We will describe the application of protein interaction networks as a translational approach to the study of human disease and evaluate the challenges faced by these approaches. [More Information](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3531279/)\n![](https://slideplayer.com/slide/5698688/18/images/32/The+role+of+protein+interaction+in+disease.jpg)"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"4c7f915bd0c441c886d94e02ca9c75e84e407c91"},"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport seaborn as sns\nimport cv2\nfrom PIL import Image\nfrom collections import Counter\n\nimport os\nprint(os.listdir(\"../input\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"db83e00645cb9c22df3ccda39eb7537cbc4b7d9a"},"cell_type":"code","source":"#import training data\ntrain = pd.read_csv(\"../input/train.csv\")\nprint(train.head())\n\n#map of targets in a dictionary\nsubcell_locs = {\n0:  \"Nucleoplasm\", \n1:  \"Nuclear membrane\",   \n2:  \"Nucleoli\",   \n3:  \"Nucleoli fibrillar center\" ,  \n4:  \"Nuclear speckles\",\n5:  \"Nuclear bodies\",\n6:  \"Endoplasmic reticulum\",   \n7:  \"Golgi apparatus\",\n8:  \"Peroxisomes\",\n9:  \"Endosomes\",\n10:  \"Lysosomes\",\n11:  \"Intermediate filaments\",   \n12:  \"Actin filaments\",\n13:  \"Focal adhesion sites\",   \n14:  \"Microtubules\",\n15:  \"Microtubule ends\",   \n16:  \"Cytokinetic bridge\",   \n17:  \"Mitotic spindle\",\n18:  \"Microtubule organizing center\",  \n19:  \"Centrosome\",\n20:  \"Lipid droplets\",   \n21:  \"Plasma membrane\",   \n22:  \"Cell junctions\", \n23:  \"Mitochondria\",\n24:  \"Aggresome\",\n25:  \"Cytosol\",\n26:  \"Cytoplasmic bodies\",   \n27:  \"Rods & rings\" \n}","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a57af30f7ff64d02a4fc311234237954ae650e03"},"cell_type":"markdown","source":"### Important Information \n\nYou are predicting protein organelle localization labels for each sample. There are in total 28 different labels present in the dataset. The dataset is acquired in a highly standardized way using one imaging modality (confocal microscopy). However, the dataset comprises 27 different cell types of highly different morphology, which affect the protein patterns of the different organelles. All image samples are represented by four filters (stored as individual files).\n* the protein of interest (green) \n* nucleus (blue), \n* microtubules (red), \n* endoplasmic reticulum (yellow). \n\nThe green filter should hence be used to predict the label, and the other filters are used as references."},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"867d58d09ee43af168b3f905c9d53e2bc996f81e"},"cell_type":"code","source":"print(\"The image with ID == 1 has the following labels:\", train.loc[1, \"Target\"])\nprint(\"These labels correspond to:\")\nfor location in train.loc[1, \"Target\"].split():\n    print(\"-\", subcell_locs[int(location)])\n\n#reset seaborn style\nsns.reset_orig()\n\n#get image id\nim_id = train.loc[1, \"Id\"]\n\n#create custom color maps\ncdict1 = {'red':   ((0.0,  0.0, 0.0),\n                   (1.0,  0.0, 0.0)),\n\n         'green': ((0.0,  0.0, 0.0),\n                   (0.75, 1.0, 1.0),\n                   (1.0,  1.0, 1.0)),\n\n         'blue':  ((0.0,  0.0, 0.0),\n                   (1.0,  0.0, 0.0))}\n\ncdict2 = {'red':   ((0.0,  0.0, 0.0),\n                   (0.75, 1.0, 1.0),\n                   (1.0,  1.0, 1.0)),\n\n         'green': ((0.0,  0.0, 0.0),\n                   (1.0,  0.0, 0.0)),\n\n         'blue':  ((0.0,  0.0, 0.0),\n                   (1.0,  0.0, 0.0))}\n\ncdict3 = {'red':   ((0.0,  0.0, 0.0),\n                   (1.0,  0.0, 0.0)),\n\n         'green': ((0.0,  0.0, 0.0),\n                   (1.0,  0.0, 0.0)),\n\n         'blue':  ((0.0,  0.0, 0.0),\n                   (0.75, 1.0, 1.0),\n                   (1.0,  1.0, 1.0))}\n\ncdict4 = {'red': ((0.0,  0.0, 0.0),\n                   (0.75, 1.0, 1.0),\n                   (1.0,  1.0, 1.0)),\n\n         'green': ((0.0,  0.0, 0.0),\n                   (0.75, 1.0, 1.0),\n                   (1.0,  1.0, 1.0)),\n\n         'blue':  ((0.0,  0.0, 0.0),\n                   (1.0,  0.0, 0.0))}\n\nplt.register_cmap(name='greens', data=cdict1)\nplt.register_cmap(name='reds', data=cdict2)\nplt.register_cmap(name='blues', data=cdict3)\nplt.register_cmap(name='yellows', data=cdict4)\n\n#get each image channel as a greyscale image (second argument 0 in imread)\ngreen = cv2.imread('../input/train/{}_green.png'.format(im_id), 0)\nred = cv2.imread('../input/train/{}_red.png'.format(im_id), 0)\nblue = cv2.imread('../input/train/{}_blue.png'.format(im_id), 0)\nyellow = cv2.imread('../input/train/{}_yellow.png'.format(im_id), 0)\n\n#display each channel separately\nfig, ax = plt.subplots(nrows = 2, ncols=2, figsize=(15, 15))\nax[0, 0].imshow(green, cmap=\"greens\")\nax[0, 0].set_title(\"Protein of interest\", fontsize=18)\nax[0, 1].imshow(red, cmap=\"reds\")\nax[0, 1].set_title(\"Microtubules\", fontsize=18)\nax[1, 0].imshow(blue, cmap=\"blues\")\nax[1, 0].set_title(\"Nucleus\", fontsize=18)\nax[1, 1].imshow(yellow, cmap=\"yellows\")\nax[1, 1].set_title(\"Endoplasmic reticulum\", fontsize=18)\nfor i in range(2):\n    for j in range(2):\n        ax[i, j].set_xticklabels([])\n        ax[i, j].set_yticklabels([])\n        ax[i, j].tick_params(left=False, bottom=False)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"05b5a8083b0992d47342a714119e632b62782f5b"},"cell_type":"code","source":"labels_num = [value.split() for value in train['Target']]\nlabels_num_flat = list(map(int, [item for sublist in labels_num for item in sublist]))\nlabels = [\"\" for _ in range(len(labels_num_flat))]\nfor i in range(len(labels_num_flat)):\n    labels[i] = subcell_locs[labels_num_flat[i]]\n\nfig, ax = plt.subplots(figsize=(15, 5))\npd.Series(labels).value_counts().plot('bar', fontsize=14)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"92ef861816e2466cb59fa30b738053067ee7b06b"},"cell_type":"code","source":"train_img = os.listdir(\"../input/train/\")\ntest_img = os.listdir(\"../input/test/\")\n\ntrain_path = \"../input/train/\"\ntest_path = \"../input/test/\"\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4f28958e0805035ec9b2a64c4e5a952e1a090d2a"},"cell_type":"markdown","source":"### Total Number of  Train and Test Images"},{"metadata":{"trusted":true,"_uuid":"2d8ecab557a016f96fe52256be356f9636a70db0"},"cell_type":"code","source":"train_df = pd.DataFrame(train_img,columns=['image_id'])\ntest_df = pd.DataFrame(test_img,columns=['image_id'])\nprint(\"Number of Total Train Image : \",len(train_df))\nprint(\"Number of Test Train Image : \",len(test_df))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0913822d58f833baa4e6f4217beb00f55bd918e8"},"cell_type":"markdown","source":"### Get the Color Name From Train and Test Images"},{"metadata":{"trusted":true,"_uuid":"a71af2d16faaef7c2a525f8ce9bfad7d02d9a8ac"},"cell_type":"code","source":"color = []\nfor n in train_img:\n    if \"red\" in n:\n       color.append('red')\n    elif \"blue\" in n:\n       color.append('blue')\n    elif \"yellow\" in n:\n       color.append('yellow')\n    elif \"green\" in n:\n       color.append('green')\ntrain_df['c_name'] = pd.DataFrame(color)\ncolor = []\nfor n in test_img:\n    if \"red\" in n:\n       color.append('red')\n    elif \"blue\" in n:\n       color.append('blue')\n    elif \"yellow\" in n:\n       color.append('yellow')\n    elif \"green\" in n:\n       color.append('green')  \ntest_df['c_name'] = pd.DataFrame(color)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"559308fc0b8136e4076846f9628df15a23799420","_kg_hide-input":true},"cell_type":"code","source":"plt.figure(figsize=(15,12))\ntrain_df[\"c_name\"].value_counts().plot(kind=\"bar\")\nplt.xlabel(\"Counts\")\nplt.ylabel(\"Colors\")\nplt.legend(\"Colors\")\nplt.title(\"Color Image Counts\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"073816e2ad8b9448a0bd71587f2aedc1e745f2e9"},"cell_type":"code","source":"train_df.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"e58d6efdb163243f71420c7ca21a90d8ffb37811"},"cell_type":"code","source":"test_df.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"34213d621b03aef64468d59e1429b72bf273c39d"},"cell_type":"code","source":"train_df['id'] = train_df['image_id'].str.split('_').str[0]\ntest_df['id'] = test_df['image_id'].str.split('_').str[0]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"734cafae86ee6661249fd5af8a25902054213442"},"cell_type":"markdown","source":"### Total Number of Unique Image on Train and Test Data\n\n\nwhen we consider as single unique id and combine four color channel at that time we have got only 31072 in train data and 11702 in test data\n"},{"metadata":{"trusted":true,"_uuid":"9cff495c7577b7893617166ac5f27ccdc7ca461d"},"cell_type":"code","source":"print(\"Total Number of Unique Image on Train Data \",len(train_df['id'].value_counts()))\nprint(\"Total Number of Unique Image on Test Data \",len(test_df['id'].value_counts()))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7d9c7604ce55c373029c9a2f7251e4b2ca1391a2"},"cell_type":"markdown","source":"### Now Sort the value id and color name wise\n\nBelow DataFame you will Better Understand\n\n**For Each unique id you will get the four color **"},{"metadata":{"trusted":true,"_uuid":"e94adc5db3d9c625044a37fdb1582d39dc7b0d3c"},"cell_type":"code","source":"train_df = train_df.sort_values(by=['id', 'c_name']).reset_index(drop=True)\ntest_df = test_df.sort_values(by=['id', 'c_name']).reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b4b6d7d97305cee885dce2ff074eadaf7f42f804"},"cell_type":"code","source":"train_df.head(12)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3d70e5a189425ec634186ba89dc9946f4fb4ad64"},"cell_type":"code","source":"test_df.head(12)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c4007708b0248e705ece692a79650bf86f09846f"},"cell_type":"markdown","source":"### we have only Read One Image and take some insight \nID is ** 00070df0-bbc3-11e8-b2bc-ac1f6b6435d0** from train data"},{"metadata":{"trusted":true,"_uuid":"08616a42ae066cb322b82ddf405c31b16cd5572b"},"cell_type":"code","source":"import cv2\nimport gc\ngc.collect()\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6bc4f72568022efc68845269127c305b512957f6"},"cell_type":"markdown","source":"### Display Four color chanel image on single id"},{"metadata":{"trusted":true,"_uuid":"af737d20f102605db39fe4246f23766d3807dc81","_kg_hide-input":true},"cell_type":"code","source":"img_1 = cv2.imread('../input/train/00070df0-bbc3-11e8-b2bc-ac1f6b6435d0_blue.png',0)\nimg_2 = cv2.imread('../input/train/00070df0-bbc3-11e8-b2bc-ac1f6b6435d0_green.png',0)\nimg_3 = cv2.imread('../input/train/00070df0-bbc3-11e8-b2bc-ac1f6b6435d0_red.png',0)\nimg_4 = cv2.imread('../input/train/00070df0-bbc3-11e8-b2bc-ac1f6b6435d0_yellow.png',0)\nfig, ax = plt.subplots(nrows = 2, ncols=2, figsize=(20, 20))\nax[0, 0].imshow(img_1)\nax[0, 0].set_title(\"Protein of interest\", fontsize=18)\nax[0, 1].imshow(img_2)\nax[0, 1].set_title(\"Microtubules\", fontsize=18)\nax[1, 0].imshow(img_3)\nax[1, 0].set_title(\"Nucleus\", fontsize=18)\nax[1, 1].imshow(img_4)\nax[1, 1].set_title(\"Endoplasmic reticulum\", fontsize=18)\nfor i in range(2):\n    for j in range(2):\n        ax[i, j].set_xticklabels([])\n        ax[i, j].set_yticklabels([])\n        ax[i, j].tick_params(left=False, bottom=False)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4f06fc20a137edf06e9e873b683ba76f61ee1cc3","_kg_hide-input":true},"cell_type":"code","source":"img_1 = cv2.imread('../input/train/00070df0-bbc3-11e8-b2bc-ac1f6b6435d0_blue.png',0)\nimg_2 = cv2.imread('../input/train/00070df0-bbc3-11e8-b2bc-ac1f6b6435d0_green.png',0)\nimg_3 = cv2.imread('../input/train/00070df0-bbc3-11e8-b2bc-ac1f6b6435d0_red.png',0)\nimg_4 = cv2.imread('../input/train/00070df0-bbc3-11e8-b2bc-ac1f6b6435d0_yellow.png',0)\nfig, ax = plt.subplots(nrows = 2, ncols=2, figsize=(20, 20))\nax[0, 0].imshow(img_1, cmap=\"blues\")\nax[0, 0].set_title(\"Protein of interest\", fontsize=18)\nax[0, 1].imshow(img_2, cmap=\"greens\")\nax[0, 1].set_title(\"Microtubules\", fontsize=18)\nax[1, 0].imshow(img_3, cmap=\"reds\")\nax[1, 0].set_title(\"Nucleus\", fontsize=18)\nax[1, 1].imshow(img_4, cmap=\"yellows\")\nax[1, 1].set_title(\"Endoplasmic reticulum\", fontsize=18)\nfor i in range(2):\n    for j in range(2):\n        ax[i, j].set_xticklabels([])\n        ax[i, j].set_yticklabels([])\n        ax[i, j].tick_params(left=False, bottom=False)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e48af102448d9a3cfa36b478686b958a07ed5859"},"cell_type":"markdown","source":"\n## [opencv python merge different channel images into one](https://stackoverflow.com/questions/44112358/opencv-python-merge-different-channel-images-into-one)\n\nAs OpenCV 3.x stored image as numpy array, we can simply average each image and add them together, provided that the height and width of the images are exactly the same.\n"},{"metadata":{"trusted":true,"_uuid":"0503f139297432b332f4369c6afcbf2639ec283a"},"cell_type":"code","source":"img_1 = cv2.imread('../input/train/00070df0-bbc3-11e8-b2bc-ac1f6b6435d0_blue.png',0)\nimg_2 = cv2.imread('../input/train/00070df0-bbc3-11e8-b2bc-ac1f6b6435d0_green.png',0)\nimg_3 = cv2.imread('../input/train/00070df0-bbc3-11e8-b2bc-ac1f6b6435d0_red.png',0)\nimg_4 = cv2.imread('../input/train/00070df0-bbc3-11e8-b2bc-ac1f6b6435d0_yellow.png',0)\n\nno_img = 4\nimg = img_1/no_img + img_2/no_img + img_3/no_img + img_4/no_img\nimg.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"15dc0a59b60c251ee0132b2d1eb3af47dacccc8c"},"cell_type":"code","source":"plt.figure(figsize=(10, 10))\nplt.imshow(img)\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"31edbee4479bd76171ed67573f7c6012f973780a"},"cell_type":"markdown","source":"### Combine all Four Images and Here is the view of few Images"},{"metadata":{"trusted":true,"_uuid":"19e01697d6d9c95f85fd140b04639973af198493"},"cell_type":"code","source":"train_df1 = train_df[0:12]\n\nfor name, group in train_df1.groupby(['id'])['image_id']:\n    img_1 = cv2.imread('../input/train/'+group.values[0],0)\n    img_2 = cv2.imread('../input/train/'+group.values[1],0)\n    img_3 = cv2.imread('../input/train/'+group.values[2],0)\n    img_4 = cv2.imread('../input/train/'+group.values[3],0)\n\n    no_img = 4\n    img = img_1/no_img + img_2/no_img + img_3/no_img + img_4/no_img\n    print(img.shape)\n    plt.figure(figsize=(10, 10))\n    plt.imshow(img)\n    plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2c385b2b1093b2045a870aae2cf1766f38b49889"},"cell_type":"code","source":"train_image = []\ntrain = train_df[0:6000]\nno_img = 4\nfor name, group in train.groupby(['id'])['image_id']:\n    img_1 = cv2.imread('../input/train/'+group.values[0],0)\n    img_2 = cv2.imread('../input/train/'+group.values[1],0)\n    img_3 = cv2.imread('../input/train/'+group.values[2],0)\n    img_4 = cv2.imread('../input/train/'+group.values[3],0)\n    img = []\n    img = img_1/no_img + img_2/no_img + img_3/no_img + img_4/no_img\n    train_image.append(img)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"af24450d16aa0cc199006867df243dc0653933ab"},"cell_type":"markdown","source":"### Now Set the Traget variable according id wise\nFrist you have to doing shorting on target variable "},{"metadata":{"trusted":true,"_uuid":"c5cc15ab22cfb2a61105e677562c2c9ead2cc66d"},"cell_type":"code","source":"train = pd.read_csv(\"../input/train.csv\")\ntrain = train.sort_values(['Id']).reset_index(drop=True)\ntrain.head(5)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"813ca599a655f36a9935075f6da297bde8cdf35e"},"cell_type":"code","source":"labels = []\nfor i in train['Target'][0:1500]:\n    li = list(i.split(\" \")) \n    labels.append(li)\nprint(\"length of Traget Variable :\",len(labels))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6b6fa50bbfead8c4648aec69eaf2b051fba461cd"},"cell_type":"code","source":"image = np.array(train_image)\nlabels = np.array(labels)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"402738cbbc12e2a04d1dfde675a1942d0f0a0ef8"},"cell_type":"code","source":"gc.collect()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5daf497eb7f43dfbcc4f5ba88c40524c3c80294c"},"cell_type":"code","source":"from sklearn.preprocessing import MultiLabelBinarizer\nfrom sklearn.model_selection import train_test_split","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c17610de9a4472523e7a6a5eeb705063ef83bb1f","_kg_hide-input":true},"cell_type":"code","source":"# binarize the labels using scikit-learn's special multi-label\n# binarizer implementation\nprint(\"[INFO] class labels:\")\nmlb = MultiLabelBinarizer()\nlabels = mlb.fit_transform(labels)\n \n#loop over each of the possible class labels and show them\nfor (i, label) in enumerate(mlb.classes_):\n    print(\"{}. {}\".format(i + 1, label))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"021e4bc539ab4e35dcef3137ffd6ace731e5573d","_kg_hide-input":true},"cell_type":"code","source":"from keras.preprocessing.image import ImageDataGenerator\n\n#================================\n# import the necessary packages\nfrom keras.models import Sequential\nfrom keras.layers.normalization import BatchNormalization\nfrom keras.layers.convolutional import Conv2D\nfrom keras.layers.convolutional import MaxPooling2D\nfrom keras.layers.core import Activation\nfrom keras.layers.core import Flatten\nfrom keras.layers.core import Dropout\nfrom keras.layers.core import Dense\nfrom keras import backend as K\n\n#================================\n\nimport matplotlib\n#matplotlib.use(\"Agg\")\n \n# import the necessary packages\nfrom keras.preprocessing.image import ImageDataGenerator\nfrom keras.optimizers import Adam\nfrom keras.preprocessing.image import img_to_array","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4990e3127f56efc007d7fc836ffd2983e33e100b"},"cell_type":"code","source":"img_width = 512\nimg_height = 512\n(trainX, testX, trainY, testY) = train_test_split(image,labels, test_size=0.3, random_state=42)\n\ntrainX = trainX.reshape(trainX.shape[0], img_width, img_height,1) \ntestX = testX.reshape(testX.shape[0], img_width, img_height,1) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"45ba0ca636eb7f31add11a2dccf87bbb3537c911"},"cell_type":"code","source":"aug = ImageDataGenerator()\nEPOCHS = 20\nINIT_LR = 1e-3\nBS = 32","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f2aa9053e5879a32ced20c280317bc6db4190ae7"},"cell_type":"markdown","source":"### Make keras deep learning model for image classification - VGG"},{"metadata":{"trusted":true,"_uuid":"7f1e3c85fde1e888115ea35cbea29b68c5df01de"},"cell_type":"code","source":"depth=1\nchanDim = -1\nclasses=28, \nfinalAct=\"sigmoid\"\n\n\ninputShape = (img_width, img_height, depth)\n\nmodel = Sequential()\n# CONV => RELU => POOL\nmodel.add(Conv2D(32, (3, 3), padding=\"same\",\ninput_shape=inputShape))\nmodel.add(Activation(\"relu\"))\nmodel.add(BatchNormalization(axis=chanDim))\nmodel.add(MaxPooling2D(pool_size=(3, 3)))\nmodel.add(Dropout(0.25))\n# (CONV => RELU) * 2 => POOL\nmodel.add(Conv2D(64, (3, 3), padding=\"same\"))\nmodel.add(Activation(\"relu\"))\nmodel.add(BatchNormalization(axis=chanDim))\nmodel.add(Conv2D(64, (3, 3), padding=\"same\"))\nmodel.add(Activation(\"relu\"))\nmodel.add(BatchNormalization(axis=chanDim))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\nmodel.add(Dropout(0.25))\n# (CONV => RELU) * 2 => POOL\nmodel.add(Conv2D(128, (3, 3), padding=\"same\"))\nmodel.add(Activation(\"relu\"))\nmodel.add(BatchNormalization(axis=chanDim))\nmodel.add(Conv2D(128, (3, 3), padding=\"same\"))\nmodel.add(Activation(\"relu\"))\nmodel.add(BatchNormalization(axis=chanDim))\nmodel.add(MaxPooling2D(pool_size=(2, 2)))\nmodel.add(Dropout(0.25))\n\n# first (and only) set of FC => RELU layers\nmodel.add(Flatten())\nmodel.add(Dense(1024))\nmodel.add(Activation(\"relu\"))\nmodel.add(BatchNormalization())\nmodel.add(Dropout(0.5))\n\n# use a *softmax* activation for single-label classification\n# and *sigmoid* activation for multi-label classification\nmodel.add(Dense(27))\nmodel.add(Activation(finalAct))\n \nopt = Adam(lr=INIT_LR, decay=INIT_LR / EPOCHS)\n\nmodel.compile(loss=\"binary_crossentropy\", optimizer=opt,metrics=[\"accuracy\"])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0811d41ec07f3da2edd7d3a6f6d83ed7b1139ae4"},"cell_type":"code","source":"model_vgg = model.fit_generator(aug.flow(trainX, trainY, batch_size=1),validation_data=(testX, testY),steps_per_epoch=len(trainX) // BS,epochs=EPOCHS, verbose=1)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7aaee9c115f14aa9599834faff12c4ab13978439"},"cell_type":"code","source":"fig, ax = plt.subplots(1, 2, figsize=(15,5))\nax[0].set_title('loss')\nax[0].plot(model_vgg.epoch, model_vgg.history[\"loss\"], label=\"Train loss\")\nax[0].plot(model_vgg.epoch, model_vgg.history[\"val_loss\"], label=\"Validation loss\")\nax[1].set_title('acc')\nax[1].plot(model_vgg.epoch, model_vgg.history[\"acc\"], label=\"Train acc\")\nax[1].plot(model_vgg.epoch, model_vgg.history[\"val_acc\"], label=\"Validation acc\")\nax[0].legend()\nax[1].legend()\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f6a8beb413375d2bc72443b95dec6bd02b0d8157"},"cell_type":"code","source":"#sub = pd.read_csv(\"../input/sample_submission.csv\")\n# test_image = []\n# for name, group in test_df.groupby(['id'])['image_id']:\n#     img_1 = cv2.imread('../input/test/'+group.values[0],0)\n#     img_2 = cv2.imread('../input/test/'+group.values[1],0)\n#     img_3 = cv2.imread('../input/test/'+group.values[2],0)\n#     img_4 = cv2.imread('../input/test/'+group.values[3],0)\n#     img = []\n#     img = img_1/no_img + img_2/no_img + img_3/no_img + img_4/no_img\n#     i = i + 1\n#     print(i)\n#     test_image.append(img)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5c3680c4226855a7809ec01b9e863dcde3d2dc99"},"cell_type":"markdown","source":"# TO-DO\n\n* optimise the ram/release the ram\n* read all test dataset and prediction value on our prepared model\n\n# If you know any technique which read all dataset images without consuming whole kaggle RAM please provide the details for same it will be great help for me and others to.\n\n### You like this kernal please give the *upvote*.\n"},{"metadata":{"trusted":true,"_uuid":"e565ff070862600d439a715f3748a70d667be84a"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}