{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Exercise 3 / part 1"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport cv2\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\n#for dirname, _, filenames in os.walk('/kaggle/input'):\n#    for filename in filenames:\n#        print(os.path.join(dirname, filename))\n\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"train_img_path = \"/kaggle/input/understanding_cloud_organization/train_images/\"\ntest_img_path = \"/kaggle/input/understanding_cloud_organization/test_images/\"\n## 1.a\nprint(\"1.a How many images does the dataset consist of?\")\n\nprint(f\"\\tNumber of images in the train set is {len(os.listdir(train_img_path))}\")\nprint(f\"\\tNumber of images in the test set is {len(os.listdir(test_img_path))}\")\n\n## 1.b\nprint(\"\\n1.b How many classes? How many images per class?\")\ntrain = pd.read_csv(\"/kaggle/input/understanding_cloud_organization/train.csv\")\nlabels_times = {\"Fish\":0,\"Flower\":0,\"Gravel\":0,\"Sugar\":0}\nimg_per_label = [0,0,0,0]\nfor name in labels_times:\n    for p in range(train.shape[0]):\n        if name in train[\"Image_Label\"][p] and pd.isna(train[\"EncodedPixels\"][p]) == False:\n            labels_times[name] +=1\n    print(f\"\\t{name}:\\t{labels_times[name]}\")\n\n## 1.c\nprint(\"1.c Show 4 samples from each class.\")\nprint(\"\\t**The pictures can have more than one class\")\nprint()\nprint()\nfor name in labels_times:\n    i = 0\n    print(f\"\\tClass: {name}\")\n    plt.figure(1)\n    for p in range(train.shape[0]):\n        if name in train[\"Image_Label\"][p] and pd.isna(train[\"EncodedPixels\"][p]) == False and i <4:\n            i += 1\n            plt.subplot(140+i)\n            plt.imshow(cv2.imread(train_img_path +train[\"Image_Label\"][p].split(\"_\")[0]))\n    plt.show()\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 1.b Consider if/how the data distribution will affect training of a clasifier.\n\nYes, it affects.\n\nFirst, having different amount of data for the different classes is a crucial element that affect the training process. Some classes are easier to learn than others, we can compensate this by adding more data of the difficult to learn classes.\n\nSecond, how the data is distributed is also important in the training process since mini-batches are used. When the dataset is properly suffled then every mini-batch will contain a balance number of the different classes. If the dataset is not suffle, then every minibatch will form likely by data from the same class and the training won't generalize, leading to jumps in the training loss and bad results.\n\n"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}