{"cells":[{"metadata":{"trusted":true},"cell_type":"code","source":"import tensorflow as tf\nimport pandas as pd\nimport time\nimport os\nimport pydicom\nfrom pydicom.pixel_data_handlers.util import convert_color_space\nimport matplotlib.pyplot as plt\nimport cv2\nimport numpy as np\nfrom sklearn.model_selection import train_test_split\nfrom multiprocessing import Pool\nimport pickle","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"base_dir = \"/kaggle/input/siim-isic-melanoma-classification/\"","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### References\n* [Dataset and TFRecord tutorial](https://codelabs.developers.google.com/codelabs/keras-flowers-data/#3)\n* [TFRecord Notebook](https://www.kaggle.com/cdeotte/how-to-create-tfrecords) by [@Chris Deotte](https://www.kaggle.com/cdeotte)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### TF Record & Cropping\nIn order to formulate a good model, we need to try a lot of experiments. This is possible if we have an efficient data pipeline coupled with hardware like TPU that can perform matrix computations at bazing fast speed. \n\n#### MXU and VPU\n\nA TPU v2 core is made of a Matrix Multiply Unit (MXU) which runs matrix multiplications and a Vector Processing Unit (VPU) for all other tasks such as activations, softmax, etc. The VPU handles float32 and int32 computations. The MXU on the other hand operates in a mixed precision 16-32 bit floating point format.\n\n![](https://lh3.googleusercontent.com/pacQdCJFoCq5ME7h2FfKCTmd6HwoEnq38PzZZFpAIfuSs5kvL05luyNJo4BWQxHXBy2ij006yo_JPk2UGiZhuskcQDxX7xIqzEAZt0lLC9Kb6QQfR0_8aajJLRffpST4fPWGhsag)","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### tf.data.dataset\nIn tensorflow the way to feed data to the model is via tf.data.dataset. \nTalking about images we can create a dataset for files as follows:","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def decode_jpeg(filename):\n  bits = tf.io.read_file(filename)\n  image = tf.image.decode_jpeg(bits)\n  #label = train_data[train_data[\"image_name\"] == filename[-16:-4]][\"target\"].values\n  start = tf.strings.length(filename) - 16\n  filename = tf.strings.substr(filename, start, 12)\n  label = target_list[tf.where(tf.equal(image_name_list, filename))[0,0]]\n  return image, label\n  \ntrain_data = pd.read_csv(base_dir + \"train.csv\")\nimage_name_list = tf.constant(train_data[\"image_name\"].values)\ntarget_list = tf.constant(train_data[\"target\"].values)\n\nfilenames_dataset = tf.data.Dataset.list_files(base_dir + \"jpeg/train/*.jpg\")\n\nimage_dataset = filenames_dataset.map(decode_jpeg)\nt0 = time.process_time()\npos_label_count = 0\nfor image, label in image_dataset.take(50):\n    pos_label_count = label + pos_label_count\nt1 = time.process_time()\nprint(\"Elapsed time:\", t1-t0)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### TFRecord\nWhile we are able to create dataset successfully, it looks pretty slow to me. And this is where TFRecord comes into picture. \nTFRecord is a simple format for storing a sequence of binary data. It boosts up the performance as multiple records can be stored in same file along with associated meta data. \nLet us create dataset for tfrecords and see the improvement in performance. ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def parse_rec(data):           \n    feature_set = {\n        'image': tf.io.FixedLenFeature([], tf.string),\n        'target': tf.io.FixedLenFeature([], tf.int64)\n    }\n    features = tf.io.parse_single_example(data, features= feature_set )\n    image = tf.image.decode_image(features[\"image\"])\n    target = features[\"target\"]\n    return image, target\n\ndef parse_rec_target(data):           \n    feature_set = {\n        'target': tf.io.FixedLenFeature([], tf.int64)\n    }\n    features = tf.io.parse_single_example(data, features= feature_set )\n    target = features[\"target\"]\n    return target\n\ndef parse_test_rec(data):           \n    feature_set = {\n        'image': tf.io.FixedLenFeature([], tf.string),\n    }\n    features = tf.io.parse_single_example(data, features= feature_set )\n    image = tf.image.decode_image(features[\"image\"])\n    return image\n\ntfrec_files_train = [base_dir + \"/tfrecords/\" + file for file in os.listdir(base_dir + \"tfrecords/\") if \"test\" not in file]\n\n\ntfrec_dataset = tf.data.TFRecordDataset(tfrec_files_train)\n\nimage_dataset = tfrec_dataset.map(parse_rec)\nt2 = time.process_time()\npos_label_count = 0\nfor image, label in image_dataset.take(50):\n    pos_label_count = label + pos_label_count\nt3 = time.process_time()\nprint(\"Elapsed time:\", t3-t2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"import seaborn as sns\nsns.set_palette(\"hls\")\n\nsns.barplot(x=[\"File Dataset\",\"TFRec Dataset\"], y=[t1-t0, t3-t2])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### TFRec Shines\n* As we can see above even with such a small datasize, perf improvement with tfrec is phenominal.\nSo let us now see how we can create TFRec ","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Create TF Record for some sample data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def _bytestring_feature(list_of_bytestrings):\n  return tf.train.Feature(bytes_list=tf.train.BytesList(value=list_of_bytestrings))\n\ndef _int_feature(list_of_ints): # int64\n  return tf.train.Feature(int64_list=tf.train.Int64List(value=list_of_ints))\n\ndef _float_feature(list_of_floats): # float32\n  return tf.train.Feature(float_list=tf.train.FloatList(value=list_of_floats))\n\ndef read_dcm_image(file_path):\n    dataset = pydicom.dcmread(file_path)\n    image = dataset.pixel_array\n    image = convert_color_space(image, \"YBR_FULL_422\", \"RGB\")\n    return image, dataset\n\ndef crop_image(image):\n    lower = [0,0,0]\n    upper = [int(image[:,:,0].mean()), int(image[:,:,1].mean()), int(image[:,:,2].mean())]\n    lower = np.array(lower, dtype=\"uint8\")\n    upper = np.array(1.01*np.array(upper, dtype=\"uint8\"), dtype=\"uint8\")\n    mask = cv2.inRange(image, lower, upper)\n\n    output = cv2.bitwise_and(image, image, mask=mask)\n\n    ret,thresh = cv2.threshold(mask, 40, 255, 0)\n\n    contours, hierarchy = cv2.findContours(thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)\n\n    if len(contours) != 0:\n        c = max(contours, key = cv2.contourArea)\n        x,y,w,h = cv2.boundingRect(c)\n\n        image = image[y:y+h, x:x+w]\n        image = cv2.resize(image,(224,224))\n    return image\n\ndef is_tf_file_valid(tfrec_path):\n    if tfrec_path[-1] == \"_\":\n        return False\n    print(tfrec_path)\n    i = int(tfrec_path.split(\"_\")[-1])\n    filename = \"train_\" + str(i)\n    try:\n        num_tfrec = len(list(tf.data.TFRecordDataset(\"../input/croppedskincancerimagestrain/\" + filename).map(parse_rec_target).as_numpy_iterator()))\n        num_df = arr_dic[i][\"df\"].shape[0]\n        print(num_tfrec, num_df)\n        return num_tfrec == num_df\n    except:\n        return False\n         \n    \ndef write_all_tfrec(tfrec_path, train_data, bln_crop=False):\n    t0 = time.process_time()\n    tfrec_full_path = \"../input/croppedskincancerimagestrain/\" + tfrec_path\n    is_valid_file = False\n    if os.path.exists(tfrec_full_path):\n        print(\"File exists!\")\n        if is_tf_file_valid(tfrec_path):\n            print(\"File is valid!\")\n            os.popen('cp ' + tfrec_full_path + ' ' + tfrec_path)\n            is_valid_file = True\n        \n    if is_valid_file == False:\n        print(tfrec_path, \": Valid file not found!\")\n        with tf.io.TFRecordWriter(tfrec_path) as out_file:\n            for idx,row in train_data.iterrows():\n                if \"train\" in tfrec_path:\n                    image_path = base_dir + \"train/\" + row[\"image_name\"] + \".dcm\"\n                else:\n                    image_path = base_dir + \"test/\" + row[\"image_name\"] + \".dcm\"\n                img, dataset = read_dcm_image(image_path)\n                if bln_crop:\n                    img = crop_image(img)\n                img = cv2.imencode('.jpg', img, (cv2.IMWRITE_JPEG_QUALITY, 94))[1].tostring()\n                feature = {\n                    \"image\": _bytestring_feature([img]),\n                }\n                if \"train\" in tfrec_path:\n                    feature[\"target\"] = _int_feature([row[\"target\"]])\n                tf_record = tf.train.Example(features=tf.train.Features(feature=feature))\n                out_file.write(tf_record.SerializeToString())\n    else:\n        print(\"File exists!\")\n    t1 = time.process_time()\n    print(\"Process time:\", t1-t0)\n        \n        \n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data = pd.read_csv(base_dir + \"train.csv\").head(24)\ntfrec_path = \"train_\"\nwrite_all_tfrec(tfrec_path, train_data)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### View some TFRecord data","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"def show_img(img_list1, img_list2):\n    row=3; col=8;\n    plt.figure(figsize=(20,row*12/col))\n    x = 1\n    \n    for img1, img2 in zip(img_list1, img_list2):\n        plt.subplot(row,col,x)\n        plt.imshow(img1)\n        x = x + 1\n        plt.subplot(row,col,x)\n        plt.imshow(img2)\n        x = x + 1\n        \ndef peek_dataset(filename):\n    tfrec_dataset = tf.data.TFRecordDataset(filename)\n    image_dataset = tfrec_dataset.map(parse_rec)\n    arr_img1 = []\n    arr_img2 = []\n    for img, label in image_dataset.take(12):\n        arr_img1.append(img)\n    for img, label in image_dataset.skip(12).take(12):\n        arr_img2.append(img)\n    show_img(arr_img1, arr_img2)\n    \ndef peek_test_dataset(filename):\n    tfrec_dataset = tf.data.TFRecordDataset(filename)\n    image_dataset = tfrec_dataset.map(parse_test_rec)\n    arr_img1 = []\n    arr_img2 = []\n    for img in image_dataset.take(12):\n        arr_img1.append(img)\n    for img in image_dataset.skip(12).take(12):\n        arr_img2.append(img)\n    show_img(arr_img1, arr_img2)\n    \npeek_dataset(\"train_\")","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"trusted":true},"cell_type":"markdown","source":"### Handling Hair\nIdeally I wanted to remove hair. but could not do so. Will come back to this in some time.","execution_count":null},{"metadata":{"_kg_hide-output":true,"_kg_hide-input":true,"trusted":true},"cell_type":"code","source":"tfrec_dataset = tf.data.TFRecordDataset(\"train_\")\nimage_dataset = tfrec_dataset.map(parse_rec)\n   \nfor image, label in image_dataset.take(1):\n    image = image.numpy()\n    gray = cv2.cvtColor(image,cv2.COLOR_BGR2GRAY) \n    edges = cv2.Canny(gray,50,150,apertureSize = 3) \n  \n    lines = cv2.HoughLines(edges,1,np.pi/180, 200) \n  \n\n    for r,theta in lines[0]: \n        a = np.cos(theta) \n        b = np.sin(theta) \n        x0 = a*r \n        y0 = b*r \n        x1 = int(x0 + 1000*(-b)) \n        y1 = int(y0 + 1000*(a)) \n        x2 = int(x0 - 1000*(-b)) \n        y2 = int(y0 - 1000*(a))  \n        line_color = (int(image[:,:,0].mean()), int(image[:,:,1].mean()), int(image[:,:,2].mean()))\n        cv2.line(image,(x1,y1), (x2,y2), line_color,50) \n\n    plt.imshow(image)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Reading & writing tfrec files is pretty simple\nSo now that it is all working, reading an writing tfrec files looks pretty simple. 2 weeks back, it looked like so much jargon. But life is never static...","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Crop Image\nIn my earlier notebook, I trained a model and achieved auc of 0.886, it is pretty low considering there are pulblic notebooks with auc of 0.94. And so there is lots to do currently. So one idea that I wanted to try is that I wanted to crop the affected area and then try to train the model. Also I really want to get rid of hair and marker lines.","execution_count":null},{"metadata":{},"cell_type":"markdown","source":"### Strategy to split data into multiple tfrec files\nNow what we can do is split these files so that number of malignant cases are evenly distributed. Another concern is  male/female and body part. So let us split so that data is uniformly distributed.","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.head(1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def create_stratify_col():\n    train_data = pd.read_csv(base_dir + \"train.csv\")\n    train_data[[\"target\",\"sex\",\"anatom_site_general_challenge\", \"image_name\"]].groupby([\"target\",\"sex\",\"anatom_site_general_challenge\"]).count().unstack(-1).unstack(-1)\n\n    train_data[\"anatom_site_general_challenge\"].fillna(\"\", inplace=True)\n    train_data[\"bodypart\"] = train_data[\"anatom_site_general_challenge\"].map(lambda x: x if x not in [\"head/neck\",\"palms/soles\",\"oral/genital\",\"\"] else \"other\")\n    train_data[\"stratify\"] = train_data.apply(lambda row: str(row[\"target\"]) + \"_\" + str(row[\"sex\"]) + \"_\" + str(row[\"bodypart\"]), axis=1)\n\n    low_freq_val = train_data[\"stratify\"].value_counts().tail(2).index.values\n    train_data[\"stratify\"] = train_data[\"stratify\"].map(lambda x: \"other\" if x in low_freq_val else x)\n    train_data.fillna(0, inplace=True)\n    return train_data\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data = create_stratify_col()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def split_train_data():\n    pickle_filename = \"../input/data-pipeline/train_dic.pkl\"\n    if not os.path.exists(pickle_filename):\n        print(\"Pickle file not found!\")\n        pickle_filename = \"train_dic.pkl\"\n    if not os.path.exists(pickle_filename):\n        print(\"Pickle file not found!\")\n        size = train_data.shape[0]//16\n        arr_data = []\n        split_data = train_data\n        for i in range(15):\n            split_data, test_data = train_test_split(split_data, test_size=size, stratify= split_data[\"stratify\"])\n            arr_data.append(test_data)\n        arr_data.append(split_data)\n        for df in arr_data[-3:]:\n            print(df.shape)\n\n        i  = 0\n        arr_dic = []\n        for df in arr_data:\n            tfrec_path = \"train_\" + str(i)\n            arr_dic.append({\"tfrec_path\":tfrec_path, \"df\":df, \"bln_crop\":True})\n            i = i + 1\n        with open(pickle_filename, 'wb') as file:\n            pickle.dump(arr_dic, file)\n        \n    else:\n        print(\"Pickle file found!\")\n        with open(pickle_filename, 'rb') as file:\n            arr_dic = pickle.load(file)\n    return arr_dic","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"arr_dic = split_train_data()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def mp_write_all_tfrec(param):\n    write_all_tfrec(param[\"tfrec_path\"], param[\"df\"], param[\"bln_crop\"])\n\nfor elem in arr_dic:\n    mp_write_all_tfrec(elem)\nif 1==2:\n    p = Pool(1)\n    p.map(mp_write_all_tfrec, arr_dic)\n    p.close()\n    p.join() ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Validate tfrecprd","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"peek_dataset(\"train_15\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if 1==2:\n    pickle_filename = \"../input/data-pipeline/test_dic.pkl\"\n    if not os.path.exists(pickle_filename):\n        print(\"Pickle file not found!\")\n        pickle_filename = \"test_dic.pkl\"\n    if not os.path.exists(pickle_filename):\n        print(\"Pickle file not found!\")\n        test_data = pd.read_csv(base_dir + \"test.csv\")\n        size = test_data.shape[0]//16\n        arr_data = []\n        split_data = test_data\n        for i in range(15):\n            split_data, data = train_test_split(split_data, test_size=size)\n            arr_data.append(data)\n        arr_data.append(split_data)\n        for df in arr_data[-3:]:\n            print(df.shape)\n        i  = 0\n        arr_dic = []\n        for df in arr_data:\n            tfrec_path = \"test_\" + str(i)\n            arr_dic.append({\"tfrec_path\":tfrec_path, \"df\":df, \"bln_crop\":True})\n            i = i + 1\n        with open(pickle_filename, 'wb') as file:\n            pickle.dump(arr_dic, file)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if 1==2:\n    p = Pool(4)\n    p.map(mp_write_all_tfrec, arr_dic)\n    p.close()\n    p.join() ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"if 1==2:\n    peek_test_dataset(\"test_9\")\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"### Thank You!\nThank you all for being part of this wonderful cmmunity. It has been the most powerful source of learning for me! ","execution_count":null},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}