{"cells":[{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"# project description"},{"metadata":{"_uuid":"955a475f92c45f86324b0a8bda10106037bf0b21"},"cell_type":"markdown","source":"经过几个世纪的强烈捕鲸，恢复鲸鱼种群仍然很难适应变暖的海洋，并且每天都在努力与工业捕鱼业争夺食物。\n\n为了帮助鲸鱼保护工作，科学家们使用照片监控系统来监测海洋活动。他们使用鲸鱼尾巴的形状和镜头中的独特标记来识别他们正在分析的鲸鱼种类，并精心记录鲸鱼荚的动态和运动。在过去的40年里，大部分工作都由个别科学家手工完成，留下了大量未开发和未充分利用的数据。\n\n在本次比赛中，您面临的挑战是建立一种算法来识别图像中的个体鲸鱼。您将分析Happywhale的数据库，其中包含来自研究机构和公共贡献者的超过25,000张图像。通过贡献，您将有助于为全球海洋哺乳动物种群动态开辟丰富的理解领域。\n\n请注意，此竞赛在本质上与[此竞赛](https://www.kaggle.com/c/whale-categorization-playground)类似，具有扩展和更新的数据集。\n\n我们要感谢[Happywhale](https://happywhale.com/)提供这些数据和问题。 Happywhale是一个平台，它使用图像处理算法让任何人提交他们的鲸鱼照片并自动识别。"},{"metadata":{"_uuid":"a89ed0cc50b9251aadc9a9ef68b24ebd39e8acac"},"cell_type":"markdown","source":"# evaluation"},{"metadata":{"_uuid":"6bd08e685aebeecc3731a295d2b8aca1efecf82e"},"cell_type":"markdown","source":"Submissions are evaluated according to the Mean Average Precision $@ 5 (MAP@5)$:\n\n$$MAP@5=\\frac{1}{U}\\sum_{u=1}^{U}\\sum_{k=1}^{min(n,5)}P(k)$$\n\nwhere `U` is the number of images, `P(k)` is the precision at cutoff `k`, and `n` is the number predictions per image.\n"},{"metadata":{"_uuid":"836ffec15c796f5c3df7b174a28aa3616e92b936"},"cell_type":"markdown","source":"# data description"},{"metadata":{"trusted":true,"_uuid":"6d5bc7381bee6948ee26c25a374a7d9727e531e6"},"cell_type":"code","source":"import matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\n\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7f2fd2033ebc1d33067dfb37a528813e658dd466"},"cell_type":"markdown","source":"看一下有哪些文件撒"},{"metadata":{"trusted":true,"_uuid":"3703b86df9fd0b721e44e1cd3f5e3131d20a7395"},"cell_type":"code","source":"! ls ../input/","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"515f9607e5533cec05dc336404d47064cef8df92"},"cell_type":"markdown","source":"train.csv: 训练数据的一些数据\n\ntrain: 训练图片\n\ntest: 测试图片\n\nsample_submission.csv: 提交文件的一个样例，可以从这里拿id然后构造要提交的文件"},{"metadata":{"_uuid":"fe24de4add0294aad3e83aa49c58b9bf5a988bf1"},"cell_type":"markdown","source":"## 首先看一看 `train.csv`"},{"metadata":{"trusted":true,"_uuid":"bd14e480d2b7c603f53806d56e031d0c48d14b08"},"cell_type":"code","source":"train_org = pd.read_csv(\"../input/train.csv\")\nprint(\"train shape:\", train_org.shape)\ntrain_org.head(5)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e088657b5192f0f3b788c49eede836aa5c113e9d"},"cell_type":"markdown","source":"其实就是一个图片和id对应的表，我们在 `test` 中需要预测的数据就是这里的 `Id`"},{"metadata":{"trusted":true,"_uuid":"5204a30f53ea4d4ba80b7656b68e2c20457597da"},"cell_type":"code","source":"pd.DataFrame(train_org.Id.value_counts().describe([0.25, 0.40, 0.75, 0.8, 0.9, 0.95, 0.99, 0.999])).T","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f6fbc5a2944c1383384c589a9eb83660e7bd96fd"},"cell_type":"markdown","source":"1. 可以看到，有近 **`40%`** 的鲸鱼，都只有一张照片，那么我们肯定需要数据增强了；\n2. 最多的鲸鱼有 **`9664`** 张图片，我们是否需要减少一点这样的图片？\n3. 数据的整体分布，我们可以看看下图。"},{"metadata":{"trusted":true,"_uuid":"c0a7f5e7170888cc88528f5724a370750b2a2634"},"cell_type":"code","source":"_id_counts = train_org.Id.value_counts()\n_id_counts[_id_counts <= 20].hist(bins=21)\n_ = plt.xlim(0, 20)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cc4430b9a0375832921495f81458177eeda0ff0e"},"cell_type":"markdown","source":"过滤掉了大于20张图片的样例，因为这些都太少了。\n\n可以发现，大多数的图片都少于5张。接下来怎么办再商量撒。"},{"metadata":{"_uuid":"1038a1226a4d09db77132b1748111a6785d7bb1e"},"cell_type":"markdown","source":"## 再来看一看 `train`"},{"metadata":{"trusted":true,"_uuid":"d5b06a53934b92816f2e9387ad22a6805accf0ae"},"cell_type":"code","source":"import os\n\nfrom PIL import Image","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"91a0a8dc29e7d6dc12e6093ed242c273ab30505f"},"cell_type":"code","source":"path_train = \"../input/train/\"\nfilenames = os.listdir(path_train)\nfilenames[0]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"56f7a5595926affd670966505f9f737dbfc7a8fe"},"cell_type":"code","source":"with Image.open(os.path.join(path_train, filenames[0]), 'r') as img:\n    print(\"filename:\", img.filename)\n    print(\"image shape:\", np.array(img).shape)\n    plt.imshow(img)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1d86b8538c3cfd585d1dbc680c9147189bd42122"},"cell_type":"code","source":"with Image.open(os.path.join(path_train, \"00d641885.jpg\"), 'r') as img:\n    print(\"filename:\", img.filename)\n    print(\"image shape:\", np.array(img).shape)\n    plt.imshow(img)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"24a7b174ea13e040b8cce9994b27ae56deb464d0"},"cell_type":"markdown","source":"都是3通道图片，但是不都一样大，有些还不是彩色的。"},{"metadata":{"_uuid":"b7202d20b6d0f6e3aed55c0bd4cebd0dbc22caed"},"cell_type":"markdown","source":"- 是不是需要预处理一下？怎么预处理？"},{"metadata":{"_uuid":"11b56cdc476419e946274713761593f5fdc9c423"},"cell_type":"markdown","source":"# submission"},{"metadata":{"trusted":true,"_uuid":"da94011bccd6502a12c4b48a3d231495e999f82f"},"cell_type":"code","source":"submission = pd.read_csv(\"../input/sample_submission.csv\")\nprint(\"test shape:\", submission.shape)\nsubmission.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"fcdfe5ed7d40a74a710be306802881ce73a582be"},"cell_type":"markdown","source":"测试数据中有 **`7960`** 个样本，需要我们每个样本最多预测5个类别，每个类别id用**空格**隔开。metric是啥，我们之后再说。"},{"metadata":{"_uuid":"0346916e85d52cc5fcc01360d29ec9aa84779939"},"cell_type":"markdown","source":"好了，接下来的，之后讨论。"},{"metadata":{"trusted":true,"_uuid":"d56740086c3b6305f7430de3c43bcfe028d57a4b"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}