{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Preparation before creating predictive model, EDA","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## In order to construct a classification model for melanoma, I confirm the contents of the training and test data sets and EDA the data.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Libraries","execution_count":null},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# data processing\nimport numpy as np \nimport pandas as pd \n\n# basic\nimport glob\nimport os\nimport time\nimport gc\nimport sys\n\n# visualization\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nplt.style.use(\"fivethirtyeight\")\n\n# Open CV\nimport cv2\n\n# dicom data\nimport pydicom\n\n#for dirname, _, filenames in os.walk('/kaggle/input'):\n#    for filename in filenames:\n#        print(os.path.join(dirname, filename))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"np.random.seed(100)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Data loading","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Table data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"path = '/kaggle/input/siim-isic-melanoma-classification/'\n\ntrain = pd.read_csv(os.path.join(path, \"train.csv\"))\ntest = pd.read_csv(os.path.join(path, \"test.csv\"))\nsample = pd.read_csv(os.path.join(path, \"sample_submission.csv\"))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Check of table data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"train data size:{}\".format(train.shape))\ntrain.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"test data size:{}\".format(test.shape))\ntest.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Submission data format\nsample.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Unique data\nprint(\"*\"*10,\"train data\",\"*\"*10)\nprint(\"anatom_site_general_challenge\\n{}\".format(train.anatom_site_general_challenge.unique()))\nprint(\"diagnosis\\n{}\".format(train.diagnosis.unique()))\nprint(\"/\"*100)\nprint(\"*\"*10,\"test data\",\"*\"*10)\nprint(\"anatom_site_general_challenge\\n{}\".format(test.anatom_site_general_challenge.unique()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Unique values\nanatom_site_general_challenge : Including null data, 7 kind of category values.<br>\n<br>\ndiagnosis : 9 kind of category values.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## patient_id Unique values","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train[\"patient_id\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,6))\nsns.distplot(train[\"patient_id\"].value_counts())\nplt.ylabel(\"frequency\")\nplt.title(\"train data\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test[\"patient_id\"].value_counts()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"plt.figure(figsize=(10,6))\nsns.distplot(test[\"patient_id\"].value_counts())\nplt.ylabel(\"frequency\")\nplt.title(\"test data\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Most of the images are not from another patient, but from a single patient in large numbers. Also, it can be seen that the deviation of the numbers is very large.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train[train[\"patient_id\"]==\"IP_0656529\"]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"When I output the data of patients with multiple images, we can see that there are various parts of the images. In other words, multiple test results are given as data for one patient.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Check patient overlap between training and test data\npd.Series(list(train[\"patient_id\"].unique()) + list(test[\"patient_id\"].unique())).value_counts()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Patients do not overlap in training and test data.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Null check\nprint(\"*\"*10,\"train data\",\"*\"*10)\nprint(\"train data null count\\n{}\".format(train.isnull().sum()))\nprint(\"/\"*100)\nprint(\"*\"*10,\"test data\",\"*\"*10)\nprint(\"test data null count\\n{}\".format(test.isnull().sum()))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"sex, age_approx, anatom_site_general_challenge have null data. Especially, anatom.. data have many null count about 1.5% in train and 3% in test data.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create dcm dataframe\nimg_list = glob.glob(os.path.join(path, \"train\", \"*\"))\nimg_list = [str(img_id.replace(\"/kaggle/input/siim-isic-melanoma-classification/train/\", \"\")) for img_id in img_list]\nimg_id = [str(img_id.split(\".\")[0]) for img_id in img_list]\nextension_list = [str(img_id.split(\".\")[1]) for img_id in img_list]\n\nimg_train_dcm = pd.DataFrame({\"img_id\":img_id, \"extension\":extension_list})\nprint(\"train data size of dcm data:{}\".format(len(img_train_dcm)))\nimg_train_dcm.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create jpeg dataframe\nimg_list = glob.glob(os.path.join(path, \"jpeg\",\"train\",\"*\"))\nimg_list = [str(img_id.replace(\"/kaggle/input/siim-isic-melanoma-classification/jpeg/train/\", \"\")) for img_id in img_list]\nimg_id = [str(img_id.split(\".\")[0]) for img_id in img_list]\nextension_list = [str(img_id.split(\".\")[1]) for img_id in img_list]\n\nimg_train_jpg = pd.DataFrame({\"img_id\":img_id, \"extension\":extension_list})\nprint(\"train data size of jpeg data:{}\".format(len(img_train_jpg)))\nimg_train_jpg.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create dcm dataframe\nimg_list = glob.glob(os.path.join(path, \"test\", \"*\"))\nimg_list = [str(img_id.replace(\"/kaggle/input/siim-isic-melanoma-classification/test/\", \"\")) for img_id in img_list]\nimg_id = [str(img_id.split(\".\")[0]) for img_id in img_list]\nextension_list = [str(img_id.split(\".\")[1]) for img_id in img_list]\n\nimg_test_dcm = pd.DataFrame({\"img_id\":img_id, \"extension\":extension_list})\nprint(\"test data size of dcm data:{}\".format(len(img_test_dcm)))\nimg_test_dcm.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Create jpeg dataframe\nimg_list = glob.glob(os.path.join(path, \"jpeg\",\"test\",\"*\"))\nimg_list = [str(img_id.replace(\"/kaggle/input/siim-isic-melanoma-classification/jpeg/test/\", \"\")) for img_id in img_list]\nimg_id = [str(img_id.split(\".\")[0]) for img_id in img_list]\nextension_list = [str(img_id.split(\".\")[1]) for img_id in img_list]\n\nimg_test_jpg = pd.DataFrame({\"img_id\":img_id, \"extension\":extension_list})\nprint(\"test data size of jpeg data:{}\".format(len(img_test_jpg)))\nimg_test_jpg.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It was confirmed that similar data was prepared for dcm and jpg data.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Table data information","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Organize category data for train data\n# pivot table\npd.pivot_table(data=train, index =\"anatom_site_general_challenge\" , columns=[\"sex\", \"benign_malignant\"], values=\"image_name\", aggfunc=\"count\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It can be confirmed that the number of malignant data is very small compared to the benign data. Especially, oral/genital and palms/soles have very few positive numbers.<br>\nThe head/neck and palms/soles and torso seem to have a large gender difference of benign/malignant.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# test data \npd.pivot_table(test, index=\"anatom_site_general_challenge\", columns=\"sex\", values=\"image_name\", aggfunc=\"count\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"About test dataset, female and male difference is similar to traindata.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# benign_malignant data count\ncnt = train[\"benign_malignant\"].value_counts()\nplt.figure(figsize=(10,6))\nplt.bar(cnt.index, cnt)\nplt.ylabel(\"count\")\nplt.title(\"benign_count : {}\\nmalignant_count : {}\".format(cnt.values[0], cnt.values[1]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"It can be seen that there is a large bias between positive and negative in the training data set.\nPredictive models will need to be accommodated because they will need to be trained on impalanced datasets.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# benign vs malignant, age band\nbenign_df = train[train[\"benign_malignant\"]==\"benign\"]\nmalignant_df = train[train[\"benign_malignant\"]==\"malignant\"]\n\n# Visualization\nfig, ax = plt.subplots(1,2, figsize=(20,6))\nplt.rcParams[\"font.size\"] = 15\n\nsns.violinplot(x=\"sex\", y=\"age_approx\", data = benign_df, ax=ax[0])\nax[0].set_title(\"benign group\")\nax[0].set_ylim([0,100])\nsns.violinplot(x=\"sex\", y=\"age_approx\", data = malignant_df, ax=ax[1])\nax[1].set_title(\"malignant group\")\nax[1].set_ylim([0,100])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"About age, malignant group is higher than benign group. Especially, the tendency is strong especially for female groups","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# test data\nplt.figure(figsize=(10,6))\nplt.rcParams[\"font.size\"] = 15\n\nsns.violinplot(x=\"sex\", y=\"age_approx\", data=test)\nplt.title(\"test data set age data\")\nplt.ylim([0,100])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The age distribution of the test data was confirmed.<br>\nFor male, the data bias characteristic of 70 years old can be confirmed. In addition, there are many data for women aged 40 to 50.<br>\nIn the training data, in particular, there were many positive cases of women around the age of 70, but there were few test data for that age group in test data. It may be necessary to pay attention to the difference in the parameter space of the learning model.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Visualization\nfig, ax = plt.subplots(1,2, figsize=(20,6))\nplt.rcParams[\"font.size\"] = 15\n\nsns.distplot(benign_df[benign_df[\"sex\"]==\"male\"][\"age_approx\"], ax=ax[0], label=\"male\")\nsns.distplot(benign_df[benign_df[\"sex\"]==\"female\"][\"age_approx\"], ax=ax[0], label=\"female\")\nax[0].set_title(\"Train data benign group\")\nax[0].set_xlabel(\"age\")\nax[0].set_ylabel(\"frequency\")\nax[0].legend()\n\nsns.distplot(malignant_df[malignant_df[\"sex\"]==\"male\"][\"age_approx\"], ax=ax[1], label=\"male\")\nsns.distplot(malignant_df[malignant_df[\"sex\"]==\"female\"][\"age_approx\"], ax=ax[1], label=\"female\")\nax[1].set_title(\"Train data malignant group\")\nax[1].set_xlabel(\"age\")\nax[1].set_ylabel(\"frequency\")\nax[1].legend()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Although age may be the difference in the acquisition method, the exact age is recorded for the person with malignant judgment, and the person for benign is recorded in the age zone, which is discrete data.<br>\n<br>\nThe distribution of age is relatively regular. The benign group is particularly uniform and the age of the health-care examinees is normal. It can be inferred that there are various backgrounds such as strong interest around 40 years old, young people not interested.<Br>\nOn the other hand, in the positive (malignant) group, females are normal, but males are biased toward older age.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Understand the characteristics of the image data set to be handled. Especially, we checked dicom data and jpeg data.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"markdown","source":"## Image data check","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The dicom data is a data format used in the medical system and includes not only image data but also various information as metadata.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# sample data\nsamp_img_name = str(img_train_dcm[\"img_id\"][0] + '.dcm')\n\nsamp_dcm = pydicom.dcmread(os.path.join(path, \"train\", samp_img_name))\n\n# dicom data\nprint(samp_dcm)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# image data , comparison with jpeg\nsamp_img = samp_dcm.pixel_array\n\n# jpeg image loading\nsamp_img_name = str(img_train_dcm[\"img_id\"][0] + '.jpg')\nsamp_img_jpg = cv2.imread(os.path.join(path, \"jpeg\", \"train\", samp_img_name))\nsamp_img_jpg = cv2.cvtColor(samp_img_jpg, cv2.COLOR_BGR2RGB)\n\n# visualization\nfig, ax = plt.subplots(1,2,figsize=(20,6))\nax[0].imshow(samp_img)\nax[0].grid()\nax[0].set_title(\"dcm image\")\n\nax[1].imshow(samp_img_jpg)\nax[1].grid()\nax[1].set_title(\"jpeg image\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"print(\"dicom data shape:{}\".format(samp_img.shape))\nprint(\"jpeg data shape:{}\".format(samp_img_jpg.shape))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For some of the sample data, images were acquired and compared with jpeg data.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Image data are different, btw dcm and jpg image.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# Hist gram\nfig, ax = plt.subplots(1, 2, figsize=(20,6))\n\n# dcm data\ncolor = [\"red\", \"green\", \"blue\"]\nfor i in range(0,3):\n    hist_dcm = cv2.calcHist([samp_img], [i], None, [256], [0,256])\n    ax[0].plot(hist_dcm, color=color[i], label=color[i])\n    ax[0].set_title(\"dcm data, YBR space\")\n    ax[0].legend()\n    \n# jpeg data\nfor i in range(0,3):\n    hist_jpg = cv2.calcHist([samp_img_jpg], [i], None, [256], [0,256])\n    ax[1].plot(hist_jpg, color=color[i], label=color[i])\n    ax[1].set_title(\"jpg data, RGB space\")\n    ax[1].legend()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Looking histgram, difference is more obvious.<br>\ndcm data has more sharpe distribution, and about blue color it has 2 characteristic peaks can be confirmed, and it can be confirmed that they are not in the jpeg data.<br>\nThe data range of jpeg is wider, and the contrast is visually emphasized for easier viewing.<br>\n\nI'm not a medical expert, so I don't know what kind of information this difference in characteristics can give from the data, but in constructing a predictive model from images, an important point that may make a difference in the results.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"markdown","source":"## image size distribution","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"I want to unify the shapes of images when learning and building models. The image size varies, so I checked the data size.<br>\nAll data does not fit in the memory, so 1000 samples were acquired and verified. This would be good information to some extent in estimating the population from the sample.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# sampling 1000 data\n# Check with jpeg data\npath = '/kaggle/input/siim-isic-melanoma-classification/jpeg/train'\n\nimage = []\nimage_h = []\nimage_w = []\nfor name in train[\"image_name\"].sample(1000):\n    img = cv2.imread(os.path.join(path, name+'.jpg')).astype(\"uint8\")\n    img_h = img.shape[0]\n    img_w = img.shape[1]\n    image_h.append(img_h)\n    image_w.append(img_w)\n    \n# visualization\nplt.figure(figsize=(6,6))\nplt.scatter(image_w, image_h)\nplt.xlabel(\"image width\")\nplt.ylabel(\"image height\")\nplt.xlim([0,7000])\nplt.ylim([0,7000])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"The size of the image data varies from small to large. It can be seen that the aspect ratios are not constant, and those that vary greatly are included.\nHowever, as a tendency, it seems that width:height = 6:4 can be unified.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"del image_h, image_w\ngc.collect()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Time of image data processing","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The larger the data size and the number of training data, the better.<br>\nHowever, due to the limitations of the computing environment, we took samples and confirmed them in order to estimate the factors that affect memory and computation time.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# loading and reshape to (600,400)\n# sampling 100 data\npath = '/kaggle/input/siim-isic-melanoma-classification'\nsample_name = train[\"image_name\"].sample(100)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# dicom data\nimage = []\n\nfor name in sample_name:\n    img = pydicom.dcmread(os.path.join(path, 'train', name+'.dcm')).pixel_array.astype(\"uint8\")\n    img = cv2.resize(img, (600,400), interpolation=cv2.INTER_AREA)\n    image.append(img)\n    \nprint(\"dicom data size : {} byte\".format(sys.getsizeof(image)))\nprint(\"dicom data loading and resize time :\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\n# jpeg data\nimage = []\n\nfor name in sample_name:\n    img = cv2.imread(os.path.join(path, 'jpeg', 'train', name+'.jpg')).astype(\"uint8\")\n    img = cv2.resize(img, (600,400), interpolation=cv2.INTER_AREA)\n    image.append(img)\n    \nprint(\"jpeg data size : {} byte\".format(sys.getsizeof(image)))\nprint(\"jpeg data loading and resize time :\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We took 100 samples and compared the load time and the calculation time of the resizing process only.<br>\nIt was found that the dicom data requires more than twice the calculation time.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# data shape vs time and size\n# jpeg data\ntimes = [50,100,200,400,800]\npro_time = []\ndata_size = []\n\nfor t in times:\n    start = time.time()\n    img = cv2.imread(os.path.join(path, 'jpeg', 'train', name+'.jpg')).astype(\"uint8\")\n    img = cv2.resize(img, (6*t, 4*t), interpolation=cv2.INTER_AREA)\n    s = sys.getsizeof(img)\n    t = time.time() - start\n    pro_time.append(t)\n    data_size.append(s)\n\n# visualization\n# For 33,126 data of train, multiply and assume\nimage_shape = [\"(300,200)\", \"(600,400)\", \"(1200,800)\", \"(2400,1600)\", \"(4800,2100)\"]\n\nfig, ax = plt.subplots(1,2,figsize=(20,6))\n\n# Time\nax[0].bar(image_shape, np.array(pro_time)*33126/60)\nax[0].set_xlabel(\"image shape\")\nax[0].set_ylabel(\"Time(min)\")\nax[0].set_title(\"image shape vs data processing time(load and resize)\\n all train data 33,216\")\n\n# data size\nax[1].bar(image_shape, np.array(data_size)*33126/1000000000)\nax[1].set_xlabel(\"image shape\")\nax[1].set_ylabel(\"Data size(Gbyte)\")\nax[1].set_yscale(\"log\")\nax[1].set_ylim([0,10000])\nax[1].set_title(\"image shape vs data size\\n all train data 33,216\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I compared the required processing time and data size for each image size. Based on the above example, rough estimation was performed by directly multiplying the total number of training samples. Assuming jpeg data processing <br>\nAs a result, it can be seen that the time is the shortest (400,600) size. On the other hand, the data size increases as the image size increases. Since its size is giga size, it must be processed well in memory.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# data size vs time and size\n# image shape condition : (600,400)\n\n# jpeg data\nbase_time = pro_time[1]\nbase_size = data_size[1]\n\nsize = np.array([5000,10000,15000,20000,30000,33216])\n\npro_time_size = base_time*size\ndata_size_size = base_size*size\n\nfig, ax = plt.subplots(1,2,figsize=(20,6))\n\n# Time\nax[0].plot(size, pro_time_size/60)\nax[0].set_xlabel(\"data size\")\nax[0].set_xlim([0,35000])\nax[0].set_ylabel(\"Time(min)\")\nax[0].set_title(\"data size vs data processing time(load and resize)\\n image shape (600,400)\")\n\n# data size\nax[1].plot(size, data_size_size/1000000000)\nax[1].set_xlabel(\"data size\")\nax[1].set_xlim([0,35000])\nax[1].set_ylabel(\"Data size(Gbyte)\")\nax[1].set_ylim([0,30])\nax[1].set_title(\"image shape vs data size\\n image shape (600,400)\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"When the image size is (600,400), the processing time of the number of training data and the data size when using the jpeg image were plotted. Assuming that the processing time increases according to the number of data, the data size is 100 giga in the case of all data. The premise of this calculation is the list format, but if you convert it to a numpy array for model training, the size will increase further.<br>\nIt is also necessary to optimize the prediction model within the memory limit based on the image size and the number of data.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Image data check","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"Image confirmation is performed to construct a prediction model from image images. Consider optimizing training data by removing noise that is not only in the tumor of interest but also in other areas.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# loading and reshape to (120,80)\n# sampling 20 data from each pos and neg target.\npath = '/kaggle/input/siim-isic-melanoma-classification'\nsample_neg = train[train[\"target\"]==0].sample(20)\nsample_pos = train[train[\"target\"]==1].sample(20)\n\nsample_neg_name = sample_neg[\"image_name\"]\nsample_neg_target = sample_neg[\"target\"]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Sample image from dcm data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nsample_neg_img = []\nfor name in sample_neg_name:\n    img = pydicom.dcmread(os.path.join(path, \"train\", name+'.dcm')).pixel_array.astype(\"uint8\")\n#    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB) # dicom image data is RGB\n    img = cv2.resize(img, (120,80), interpolation=cv2.INTER_AREA)\n    sample_neg_img.append(img)\n    \nsample_pos_name = sample_pos[\"image_name\"]\nsample_pos_target = sample_pos[\"target\"]\n\nsample_pos_img = []\nfor name in sample_pos_name:\n    img = pydicom.dcmread(os.path.join(path, \"train\", name+'.dcm')).pixel_array.astype(\"uint8\")\n#    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    img = cv2.resize(img, (120,80), interpolation=cv2.INTER_AREA)\n    sample_pos_img.append(img)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Visualization\n# negative sample\nfig, ax = plt.subplots(4,5, figsize=(30,24))\n\nfor i in range(0,4):\n    for j in range(0,5):\n        ax[i,j].imshow(sample_neg_img[5*i+j])\n        ax[i,j].grid()\n        ax[i,j].set_title(\"benign sample : \\n{}\".format(sample_neg_name.values[5*i+j]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Visualization\n# positive sample\nfig, ax = plt.subplots(4,5, figsize=(30,24))\n\nfor i in range(0,4):\n    for j in range(0,5):\n        ax[i,j].imshow(sample_pos_img[5*i+j])\n        ax[i,j].grid()\n        ax[i,j].set_title(\"malignant sample : \\n{}\".format(sample_pos_name.values[5*i+j]))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"## Sample image from jpeg data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nsample_neg_img = []\nfor name in sample_neg_name:\n    img = cv2.imread(os.path.join(path,'jpeg', 'train', name+'.jpg')).astype(\"uint8\")\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    img = cv2.resize(img, (120,80), interpolation=cv2.INTER_AREA)\n    sample_neg_img.append(img)\n    \nsample_pos_name = sample_pos[\"image_name\"]\nsample_pos_target = sample_pos[\"target\"]\n\nsample_pos_img = []\nfor name in sample_pos_name:\n    img = cv2.imread(os.path.join(path, \"jpeg\", \"train\", name+'.jpg')).astype(\"uint8\")\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    img = cv2.resize(img, (120,80), interpolation=cv2.INTER_AREA)\n    sample_pos_img.append(img)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Visualization\n# negative sample\nfig, ax = plt.subplots(4,5, figsize=(30,24))\n\nfor i in range(0,4):\n    for j in range(0,5):\n        ax[i,j].imshow(sample_neg_img[5*i+j])\n        ax[i,j].grid()\n        ax[i,j].set_title(\"benign sample : \\n{}\".format(sample_neg_name.values[5*i+j]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Visualization\n# positive sample\nfig, ax = plt.subplots(4,5, figsize=(30,24))\n\nfor i in range(0,4):\n    for j in range(0,5):\n        ax[i,j].imshow(sample_pos_img[5*i+j])\n        ax[i,j].grid()\n        ax[i,j].set_title(\"malignant sample : \\n{}\".format(sample_pos_name.values[5*i+j]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"I confirmed the image based on dcm and jpeg data. <br>\nFrom this result, the malignant group often has a large dark area and a large area. However, benign contains similar data, and it is impossible for an unseen person to judge. You can also see different types, such as different shapes and distributions. Foreseeing such things may require pre-classification, just like real people do.<br>\nAlso, it can be seen that the image contains not only tumors but also image noise such as hairs and pores.\n\nNext, I confirm images of each part.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"# Comparison each point (anatom_site_general_challenge point)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"# image_name from each part\nneg_hn_name = train[(train[\"anatom_site_general_challenge\"]==\"head/neck\") & (train[\"benign_malignant\"]==\"benign\")][\"image_name\"]\npos_hn_name = train[(train[\"anatom_site_general_challenge\"]==\"head/neck\") & (train[\"benign_malignant\"]==\"malignant\")][\"image_name\"]\n\nneg_ue_name = train[(train[\"anatom_site_general_challenge\"]==\"upper extremity\") & (train[\"benign_malignant\"]==\"benign\")][\"image_name\"]\npos_ue_name = train[(train[\"anatom_site_general_challenge\"]==\"upper extremity\") & (train[\"benign_malignant\"]==\"malignant\")][\"image_name\"]\n\nneg_le_name = train[(train[\"anatom_site_general_challenge\"]==\"lower extremity\") & (train[\"benign_malignant\"]==\"benign\")][\"image_name\"]\npos_le_name = train[(train[\"anatom_site_general_challenge\"]==\"lower extremity\") & (train[\"benign_malignant\"]==\"malignant\")][\"image_name\"]\n\nneg_tr_name = train[(train[\"anatom_site_general_challenge\"]==\"torso\") & (train[\"benign_malignant\"]==\"benign\")][\"image_name\"]\npos_tr_name = train[(train[\"anatom_site_general_challenge\"]==\"torso\") & (train[\"benign_malignant\"]==\"malignant\")][\"image_name\"]\n\nneg_ps_name = train[(train[\"anatom_site_general_challenge\"]==\"palms/soles\") & (train[\"benign_malignant\"]==\"benign\")][\"image_name\"]\npos_ps_name = train[(train[\"anatom_site_general_challenge\"]==\"palms/soles\") & (train[\"benign_malignant\"]==\"malignant\")][\"image_name\"]\n\nneg_og_name = train[(train[\"anatom_site_general_challenge\"]==\"oral/genital\") & (train[\"benign_malignant\"]==\"benign\")][\"image_name\"]\npos_og_name = train[(train[\"anatom_site_general_challenge\"]==\"oral/genital\") & (train[\"benign_malignant\"]==\"malignant\")][\"image_name\"]\n\ndata_list = [neg_hn_name, pos_hn_name, neg_ue_name, pos_ue_name, \n             neg_le_name, pos_le_name, neg_tr_name, pos_tr_name, \n             neg_ps_name, pos_ps_name, neg_og_name, pos_og_name]\npos_neg = [\"benign\", \"malignant\"]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# head/neck\nfig, ax = plt.subplots(1, 4, figsize=(24, 6))\n\nfor i in range(0,2):\n    for j in range(0,2):\n        img = cv2.imread(os.path.join(path, \"jpeg\", \"train\", data_list[0+i].values[j]+\".jpg\"))\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        img = cv2.resize(img, (120,80), interpolation=cv2.INTER_AREA)\n        \n        ax[i*2+j].imshow(img)\n        ax[i*2+j].grid()\n        ax[i*2+j].set_title(\"head/neck : {}\".format(pos_neg[i]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# upper extremity\nfig, ax = plt.subplots(1, 4, figsize=(24, 6))\n\nfor i in range(0,2):\n    for j in range(0,2):\n        img = cv2.imread(os.path.join(path, \"jpeg\", \"train\", data_list[2+i].values[j]+\".jpg\"))\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        img = cv2.resize(img, (120,80), interpolation=cv2.INTER_AREA)\n        \n        ax[i*2+j].imshow(img)\n        ax[i*2+j].grid()\n        ax[i*2+j].set_title(\"upper extremity : {}\".format(pos_neg[i]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# lower extremity\nfig, ax = plt.subplots(1, 4, figsize=(24, 6))\n\nfor i in range(0,2):\n    for j in range(0,2):\n        img = cv2.imread(os.path.join(path, \"jpeg\", \"train\", data_list[4+i].values[j]+\".jpg\"))\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        img = cv2.resize(img, (120,80), interpolation=cv2.INTER_AREA)\n        \n        ax[i*2+j].imshow(img)\n        ax[i*2+j].grid()\n        ax[i*2+j].set_title(\"lower extremity : {}\".format(pos_neg[i]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# torso\nfig, ax = plt.subplots(1, 4, figsize=(24, 6))\n\nfor i in range(0,2):\n    for j in range(0,2):\n        img = cv2.imread(os.path.join(path, \"jpeg\", \"train\", data_list[6+i].values[j]+\".jpg\"))\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        img = cv2.resize(img, (120,80), interpolation=cv2.INTER_AREA)\n        \n        ax[i*2+j].imshow(img)\n        ax[i*2+j].grid()\n        ax[i*2+j].set_title(\"torso : {}\".format(pos_neg[i]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# palms/soles\nfig, ax = plt.subplots(1, 4, figsize=(24, 6))\n\nfor i in range(0,2):\n    for j in range(0,2):\n        img = cv2.imread(os.path.join(path, \"jpeg\", \"train\", data_list[8+i].values[j]+\".jpg\"))\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        img = cv2.resize(img, (120,80), interpolation=cv2.INTER_AREA)\n        \n        ax[i*2+j].imshow(img)\n        ax[i*2+j].grid()\n        ax[i*2+j].set_title(\"palms/soles : {}\".format(pos_neg[i]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# oral/genital\nfig, ax = plt.subplots(1, 4, figsize=(24, 6))\n\nfor i in range(0,2):\n    for j in range(0,2):\n        img = cv2.imread(os.path.join(path, \"jpeg\", \"train\", data_list[10+i].values[j]+\".jpg\"))\n        img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n        img = cv2.resize(img, (120,80), interpolation=cv2.INTER_AREA)\n        \n        ax[i*2+j].imshow(img)\n        ax[i*2+j].grid()\n        ax[i*2+j].set_title(\"oral/genital : {}\".format(pos_neg[i]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"When the images were checked for each part, it was found that the head/neck sample and the oral/genital sample had a lot of noise such as hair.\nIf the image classification for each part is possible in this way, it may be possible to examine the improvement of accuracy by composing a program such as filter processing according to each.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"markdown","source":"### Here, try to reduction of image noise(like hair), the result of some image processing is confirmed.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"markdown","source":"### Gaussian weighted averaging (GaussianBlur)","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_neg_img_gb = []\n\nfor i in sample_neg_img:\n    img = cv2.GaussianBlur(i, (5,5), 1, 1)\n    sample_neg_img_gb.append(img)\n\n# Visualization\n# negative sample\nfig, ax = plt.subplots(4,5, figsize=(30,24))\n\nfor i in range(0,4):\n    for j in range(0,5):\n        ax[i,j].imshow(sample_neg_img_gb[5*i+j])\n        ax[i,j].grid()\n        ax[i,j].set_title(\"benign sample : \\n{}\".format(sample_neg_name.values[5*i+j]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_pos_img_gb = []\n\nfor i in sample_pos_img:\n    img = cv2.GaussianBlur(i, (5,5), 1, 1)\n    sample_pos_img_gb.append(img)\n\n# Visualization\n# negative sample\nfig, ax = plt.subplots(4,5, figsize=(30,24))\n\nfor i in range(0,4):\n    for j in range(0,5):\n        ax[i,j].imshow(sample_pos_img_gb[5*i+j])\n        ax[i,j].grid()\n        ax[i,j].set_title(\"benign sample : \\n{}\".format(sample_pos_name.values[5*i+j]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"By adding filtering, in some cases noise such as hair and pores can be reduced. This may be effective in suppressing overfitting in the learning model. However, verification is required.\nAnd, small tumors or those with many hairs may be difficult to identify. Again, it is necessary to actually create a model and verify it.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Gray scale and Canny edge detection","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_neg_img_gc = []\n\nfor i in sample_neg_img:\n    img = cv2.cvtColor(i, cv2.COLOR_RGB2GRAY)\n    img = cv2.Canny(img, 50, 150)\n    sample_neg_img_gc.append(img)\n\n# Visualization\n# negative sample\nfig, ax = plt.subplots(4,5, figsize=(30,24))\n\nfor i in range(0,4):\n    for j in range(0,5):\n        ax[i,j].imshow(sample_neg_img_gc[5*i+j], cmap=plt.get_cmap(\"gray\"), vmin=0, vmax=255)\n        ax[i,j].grid()\n        ax[i,j].set_title(\"benign sample : \\n{}\".format(sample_neg_name.values[5*i+j]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_pos_img_gc = []\n\nfor i in sample_pos_img:\n    img = cv2.cvtColor(i, cv2.COLOR_RGB2GRAY)\n    img = cv2.Canny(img, 50, 150)\n    sample_pos_img_gc.append(img)\n\n# Visualization\n# negative sample\nfig, ax = plt.subplots(4,5, figsize=(30,24))\n\nfor i in range(0,4):\n    for j in range(0,5):\n        ax[i,j].imshow(sample_pos_img_gc[5*i+j], cmap=plt.get_cmap(\"gray\"), vmin=0, vmax=255)\n        ax[i,j].grid()\n        ax[i,j].set_title(\"benign sample : \\n{}\".format(sample_pos_name.values[5*i+j]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Both negative and positive data show the shape of the tumor. However, the difference between them is not visible. <br>\nIn particular, there is a lot of noise such as hair and pores, and there are many data whose shapes are not captured, which is a problem.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"The Canny process itself includes a filter process as a pre-process, but the result of applying the above-mentioned Blur process to the pre-process and strengthening the filter is confirmed.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_neg_img_gb_gc = []\n\nfor i in sample_neg_img_gb:\n    img = cv2.cvtColor(i, cv2.COLOR_RGB2GRAY)\n    img = cv2.Canny(img, 50, 150)\n    sample_neg_img_gb_gc.append(img)\n\n# Visualization\n# negative sample\nfig, ax = plt.subplots(4,5, figsize=(30,24))\n\nfor i in range(0,4):\n    for j in range(0,5):\n        ax[i,j].imshow(sample_neg_img_gb_gc[5*i+j], cmap=plt.get_cmap(\"gray\"), vmin=0, vmax=255)\n        ax[i,j].grid()\n        ax[i,j].set_title(\"benign sample : \\n{}\".format(sample_neg_name.values[5*i+j]))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sample_pos_img_gb_gc = []\n\nfor i in sample_pos_img_gb:\n    img = cv2.cvtColor(i, cv2.COLOR_RGB2GRAY)\n    img = cv2.Canny(img, 50, 150)\n    sample_pos_img_gb_gc.append(img)\n\n# Visualization\n# negative sample\nfig, ax = plt.subplots(4,5, figsize=(30,24))\n\nfor i in range(0,4):\n    for j in range(0,5):\n        ax[i,j].imshow(sample_pos_img_gb_gc[5*i+j], cmap=plt.get_cmap(\"gray\"), vmin=0, vmax=255)\n        ax[i,j].grid()\n        ax[i,j].set_title(\"benign sample : \\n{}\".format(sample_pos_name.values[5*i+j]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"For some samples, the negative data disappears but the positive data retains shape. This may suggest that the slope of the data is often stronger for positive data. But not all are true.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## One of patient images","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### sample_patient IP_4938382","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nsample_pat_img = []\nsample_pat_img_name = train[train[\"patient_id\"]==\"IP_4938382\"][\"image_name\"]\nsample_pat_img_posneg = train[train[\"patient_id\"]==\"IP_4938382\"][\"benign_malignant\"]\n\nfor name in sample_pat_img_name:\n    img = cv2.imread(os.path.join(path,'jpeg', 'train', name+'.jpg')).astype(\"uint8\")\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    img = cv2.resize(img, (48,32), interpolation=cv2.INTER_AREA)\n    sample_pat_img.append(img)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Visualization\n# positive sample\nfig, ax = plt.subplots(10,10, figsize=(40,40))\n\nfor i in range(0,10):\n    for j in range(0,10):\n        ax[i,j].imshow(sample_pat_img[10*i+j])\n        ax[i,j].grid()\n        ax[i,j].set_title(\"sample_patient\\nIP_4938382\\n{}\".format(sample_pat_img_posneg.values[10*i+j]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### sample_patient IP_0656529","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"%%time\nsample_pat_img = []\nsample_pat_img_name = train[train[\"patient_id\"]==\"IP_0656529\"][\"image_name\"]\nsample_pat_img_posneg = train[train[\"patient_id\"]==\"IP_0656529\"][\"benign_malignant\"]\n\nfor name in sample_pat_img_name:\n    img = cv2.imread(os.path.join(path,'jpeg', 'train', name+'.jpg')).astype(\"uint8\")\n    img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)\n    img = cv2.resize(img, (48,32), interpolation=cv2.INTER_AREA)\n    sample_pat_img.append(img)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# Visualization\n# positive sample\nfig, ax = plt.subplots(10,10, figsize=(40,40))\n\nfor i in range(0,10):\n    for j in range(0,10):\n        ax[i,j].imshow(sample_pat_img[10*i+j])\n        ax[i,j].grid()\n        ax[i,j].set_title(\"sample_patient\\nIP_0656529\\n{}\".format(sample_pat_img_posneg.values[10*i+j]))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"These are examples of 2 patients, but it can be seen that there are many very similar images. In particular, the former is so similar that the images are indistinguishable. Moreover, both examples are benign.<Br>\nThis can bias the training data and increase the tendency for overfitting, so caution is required.<br>\nThis must be taken into consideration when selecting training data.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Next step","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"From the results so far, it may be possible to improve accuracy in classification by images by combining not only one data-processed model but also a plurality of models.<br>\nIt is necessary to consider not only the image data but also other medical information.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"## Based on the above results, for create the prediction model.\n- Decide how to deal with imbalanced data. Oversampling and undersampling, etc.\n- Organize tabular data, digitize missing or categorical data, and create other models to build models.\n- Pre-processing of image data and determination of data format. Image data size, number of training data, etc. according to the computing environment.\n- Build a predictive model based on deep learning.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}