{"metadata": {"language_info": {"codemirror_mode": {"version": 3, "name": "ipython"}, "nbconvert_exporter": "python", "file_extension": ".py", "version": "3.6.1", "name": "python", "pygments_lexer": "ipython3", "mimetype": "text/x-python"}, "kernelspec": {"language": "python", "name": "python3", "display_name": "Python 3"}}, "nbformat_minor": 1, "cells": [{"metadata": {}, "cell_type": "markdown", "source": ["I was interested to see if any features / data leaks could be gleaned by inspecting the low-level details of the JPEG files. This could be things as obvious as EXIF data, or minor encoding details that difffered between classes (maybe image quality is different, some are greyscale, etc).\n", "\n", "Spoiler: the dataset seems clean :)\n", "\n", "I started with imagemagick. This didn't reveal anything useful for the few files I tried it on.\n", "Note that much of the data below could be used as simple features, although it's simply computed from the image data itself, and thus would also be picked up by training a DNN on the decoded image data.\n", "\n", "```\n", "magick identify -verbose build/images/1000001859/26.jpg \n", "Image: build/images/1000001859/26.jpg\n", "  Format: JPEG (Joint Photographic Experts Group JFIF format)\n", "  Mime type: image/jpeg\n", "  Class: DirectClass\n", "  Geometry: 180x180+0+0\n", "  Units: Undefined\n", "  Type: Grayscale\n", "  Base type: TrueColor\n", "  Endianess: Undefined\n", "  Colorspace: sRGB\n", "  Depth: 8-bit\n", "  Channel depth:\n", "    Gray: 8-bit\n", "  Channel statistics:\n", "    Pixels: 32400\n", "    Gray:\n", "      min: 24  (0.0941176)\n", "      max: 255 (1)\n", "      mean: 246.61 (0.967099)\n", "      standard deviation: 17.9087 (0.0702302)\n", "      kurtosis: 53.0955\n", "      skewness: -6.11374\n", "      entropy: 0.483356\n", "  Colors: 204\n", "  Histogram:\n", "         1: ( 24, 24, 24) #181818 srgb(24,24,24)\n", "         1: ( 26, 26, 26) #1A1A1A grey10\n", "         2: ( 27, 27, 27) #1B1B1B srgb(27,27,27)\n", "         2: ( 28, 28, 28) #1C1C1C grey11\n", "         5: ( 29, 29, 29) #1D1D1D srgb(29,29,29)\n", "         3: ( 30, 30, 30) #1E1E1E srgb(30,30,30)\n", "         6: ( 31, 31, 31) #1F1F1F grey12\n", "         3: ( 33, 33, 33) #212121 grey13\n", "         2: ( 34, 34, 34) #222222 srgb(34,34,34)\n", "         2: ( 35, 35, 35) #232323 srgb(35,35,35)\n", "         2: ( 36, 36, 36) #242424 grey14\n", "         3: ( 37, 37, 37) #252525 srgb(37,37,37)\n", "         3: ( 38, 38, 38) #262626 grey15\n", "         1: ( 39, 39, 39) #272727 srgb(39,39,39)\n", "         1: ( 40, 40, 40) #282828 srgb(40,40,40)\n", "         1: ( 41, 41, 41) #292929 grey16\n", "         1: ( 42, 42, 42) #2A2A2A srgb(42,42,42)\n", "         1: ( 43, 43, 43) #2B2B2B grey17\n", "         3: ( 44, 44, 44) #2C2C2C srgb(44,44,44)\n", "         3: ( 45, 45, 45) #2D2D2D srgb(45,45,45)\n", "         1: ( 47, 47, 47) #2F2F2F srgb(47,47,47)\n", "         1: ( 48, 48, 48) #303030 grey19\n", "         1: ( 50, 50, 50) #323232 srgb(50,50,50)\n", "         2: ( 51, 51, 51) #333333 grey20\n", "         1: ( 52, 52, 52) #343434 srgb(52,52,52)\n", "         3: ( 53, 53, 53) #353535 srgb(53,53,53)\n", "         1: ( 54, 54, 54) #363636 grey21\n", "         2: ( 55, 55, 55) #373737 srgb(55,55,55)\n", "         1: ( 56, 56, 56) #383838 grey22\n", "         1: ( 57, 57, 57) #393939 srgb(57,57,57)\n", "         1: ( 58, 58, 58) #3A3A3A srgb(58,58,58)\n", "         1: ( 59, 59, 59) #3B3B3B grey23\n", "         1: ( 60, 60, 60) #3C3C3C srgb(60,60,60)\n", "         2: ( 61, 61, 61) #3D3D3D grey24\n", "         1: ( 62, 62, 62) #3E3E3E srgb(62,62,62)\n", "         2: ( 63, 63, 63) #3F3F3F srgb(63,63,63)\n", "         1: ( 64, 64, 64) #404040 grey25\n", "         2: ( 66, 66, 66) #424242 grey26\n", "         3: ( 67, 67, 67) #434343 srgb(67,67,67)\n", "         1: ( 68, 68, 68) #444444 srgb(68,68,68)\n", "         1: ( 70, 70, 70) #464646 srgb(70,70,70)\n", "         2: ( 71, 71, 71) #474747 grey28\n", "         2: ( 73, 73, 73) #494949 srgb(73,73,73)\n", "         3: ( 74, 74, 74) #4A4A4A grey29\n", "         1: ( 75, 75, 75) #4B4B4B srgb(75,75,75)\n", "         2: ( 77, 77, 77) #4D4D4D grey30\n", "         4: ( 81, 81, 81) #515151 srgb(81,81,81)\n", "         2: ( 82, 82, 82) #525252 grey32\n", "         3: ( 83, 83, 83) #535353 srgb(83,83,83)\n", "         2: ( 84, 84, 84) #545454 grey33\n", "         1: ( 86, 86, 86) #565656 srgb(86,86,86)\n", "         1: ( 90, 90, 90) #5A5A5A srgb(90,90,90)\n", "         1: ( 93, 93, 93) #5D5D5D srgb(93,93,93)\n", "         3: ( 98, 98, 98) #626262 srgb(98,98,98)\n", "         1: ( 99, 99, 99) #636363 grey39\n", "         2: (100,100,100) #646464 srgb(100,100,100)\n", "         1: (101,101,101) #656565 srgb(101,101,101)\n", "         1: (103,103,103) #676767 srgb(103,103,103)\n", "         1: (104,104,104) #686868 srgb(104,104,104)\n", "         2: (106,106,106) #6A6A6A srgb(106,106,106)\n", "         1: (107,107,107) #6B6B6B grey42\n", "         1: (109,109,109) #6D6D6D srgb(109,109,109)\n", "         1: (110,110,110) #6E6E6E grey43\n", "         3: (111,111,111) #6F6F6F srgb(111,111,111)\n", "         1: (112,112,112) #707070 grey44\n", "         3: (113,113,113) #717171 srgb(113,113,113)\n", "         1: (114,114,114) #727272 srgb(114,114,114)\n", "         1: (116,116,116) #747474 srgb(116,116,116)\n", "         4: (117,117,117) #757575 grey46\n", "         2: (118,118,118) #767676 srgb(118,118,118)\n", "         2: (119,119,119) #777777 srgb(119,119,119)\n", "         2: (120,120,120) #787878 grey47\n", "         1: (121,121,121) #797979 srgb(121,121,121)\n", "         1: (122,122,122) #7A7A7A grey48\n", "         2: (123,123,123) #7B7B7B srgb(123,123,123)\n", "         1: (124,124,124) #7C7C7C srgb(124,124,124)\n", "         1: (125,125,125) #7D7D7D grey49\n", "         5: (126,126,126) #7E7E7E gray\n", "         2: (127,127,127) #7F7F7F grey50\n", "         2: (128,128,128) #808080 fractal\n", "         1: (130,130,130) #828282 grey51\n", "         2: (131,131,131) #838383 srgb(131,131,131)\n", "         2: (132,132,132) #848484 srgb(132,132,132)\n", "         3: (133,133,133) #858585 grey52\n", "         3: (134,134,134) #868686 srgb(134,134,134)\n", "         2: (135,135,135) #878787 grey53\n", "         2: (136,136,136) #888888 srgb(136,136,136)\n", "         1: (137,137,137) #898989 srgb(137,137,137)\n", "         4: (139,139,139) #8B8B8B srgb(139,139,139)\n", "         1: (140,140,140) #8C8C8C grey55\n", "         1: (141,141,141) #8D8D8D srgb(141,141,141)\n", "         1: (142,142,142) #8E8E8E srgb(142,142,142)\n", "         2: (144,144,144) #909090 srgb(144,144,144)\n", "         3: (145,145,145) #919191 grey57\n", "         1: (146,146,146) #929292 srgb(146,146,146)\n", "         3: (147,147,147) #939393 srgb(147,147,147)\n", "         4: (148,148,148) #949494 grey58\n", "         2: (149,149,149) #959595 srgb(149,149,149)\n", "         3: (150,150,150) #969696 grey59\n", "        13: (151,151,151) #979797 srgb(151,151,151)\n", "        17: (152,152,152) #989898 srgb(152,152,152)\n", "        14: (153,153,153) #999999 grey60\n", "        11: (154,154,154) #9A9A9A srgb(154,154,154)\n", "         6: (155,155,155) #9B9B9B srgb(155,155,155)\n", "        10: (156,156,156) #9C9C9C grey61\n", "        12: (157,157,157) #9D9D9D srgb(157,157,157)\n", "         3: (158,158,158) #9E9E9E grey62\n", "         3: (159,159,159) #9F9F9F srgb(159,159,159)\n", "         4: (160,160,160) #A0A0A0 srgb(160,160,160)\n", "         5: (161,161,161) #A1A1A1 grey63\n", "         7: (162,162,162) #A2A2A2 srgb(162,162,162)\n", "        14: (163,163,163) #A3A3A3 grey64\n", "         8: (164,164,164) #A4A4A4 srgb(164,164,164)\n", "        13: (165,165,165) #A5A5A5 srgb(165,165,165)\n", "        23: (166,166,166) #A6A6A6 grey65\n", "        10: (167,167,167) #A7A7A7 srgb(167,167,167)\n", "        13: (168,168,168) #A8A8A8 grey66\n", "        10: (169,169,169) #A9A9A9 DarkGray\n", "         5: (170,170,170) #AAAAAA srgb(170,170,170)\n", "         4: (171,171,171) #ABABAB grey67\n", "         5: (172,172,172) #ACACAC srgb(172,172,172)\n", "         7: (173,173,173) #ADADAD grey68\n", "        17: (174,174,174) #AEAEAE srgb(174,174,174)\n", "         3: (175,175,175) #AFAFAF srgb(175,175,175)\n", "         1: (176,176,176) #B0B0B0 grey69\n", "         5: (177,177,177) #B1B1B1 srgb(177,177,177)\n", "         4: (178,178,178) #B2B2B2 srgb(178,178,178)\n", "         5: (179,179,179) #B3B3B3 grey70\n", "         7: (180,180,180) #B4B4B4 srgb(180,180,180)\n", "         4: (181,181,181) #B5B5B5 grey71\n", "         7: (182,182,182) #B6B6B6 srgb(182,182,182)\n", "         7: (183,183,183) #B7B7B7 srgb(183,183,183)\n", "        20: (184,184,184) #B8B8B8 grey72\n", "        11: (185,185,185) #B9B9B9 srgb(185,185,185)\n", "         9: (186,186,186) #BABABA grey73\n", "         6: (187,187,187) #BBBBBB srgb(187,187,187)\n", "         9: (188,188,188) #BCBCBC srgb(188,188,188)\n", "        11: (189,189,189) #BDBDBD grey74\n", "        13: (190,190,190) #BEBEBE grey\n", "        10: (191,191,191) #BFBFBF grey75\n", "        12: (192,192,192) #C0C0C0 silver\n", "        12: (193,193,193) #C1C1C1 srgb(193,193,193)\n", "        13: (194,194,194) #C2C2C2 grey76\n", "         9: (195,195,195) #C3C3C3 srgb(195,195,195)\n", "        16: (196,196,196) #C4C4C4 grey77\n", "        33: (197,197,197) #C5C5C5 srgb(197,197,197)\n", "        19: (198,198,198) #C6C6C6 srgb(198,198,198)\n", "        25: (199,199,199) #C7C7C7 grey78\n", "        16: (200,200,200) #C8C8C8 srgb(200,200,200)\n", "        29: (201,201,201) #C9C9C9 grey79\n", "        20: (202,202,202) #CACACA srgb(202,202,202)\n", "        21: (203,203,203) #CBCBCB srgb(203,203,203)\n", "        31: (204,204,204) #CCCCCC grey80\n", "        16: (205,205,205) #CDCDCD srgb(205,205,205)\n", "        17: (206,206,206) #CECECE srgb(206,206,206)\n", "        28: (207,207,207) #CFCFCF grey81\n", "        27: (208,208,208) #D0D0D0 srgb(208,208,208)\n", "        27: (209,209,209) #D1D1D1 grey82\n", "        22: (210,210,210) #D2D2D2 srgb(210,210,210)\n", "        22: (211,211,211) #D3D3D3 LightGray\n", "        23: (212,212,212) #D4D4D4 grey83\n", "        30: (213,213,213) #D5D5D5 srgb(213,213,213)\n", "        27: (214,214,214) #D6D6D6 grey84\n", "        21: (215,215,215) #D7D7D7 srgb(215,215,215)\n", "        40: (216,216,216) #D8D8D8 srgb(216,216,216)\n", "        28: (217,217,217) #D9D9D9 grey85\n", "        47: (218,218,218) #DADADA srgb(218,218,218)\n", "        44: (219,219,219) #DBDBDB grey86\n", "        47: (220,220,220) #DCDCDC gainsboro\n", "        68: (221,221,221) #DDDDDD srgb(221,221,221)\n", "        60: (222,222,222) #DEDEDE grey87\n", "        65: (223,223,223) #DFDFDF srgb(223,223,223)\n", "        49: (224,224,224) #E0E0E0 grey88\n", "        76: (225,225,225) #E1E1E1 srgb(225,225,225)\n", "       106: (226,226,226) #E2E2E2 srgb(226,226,226)\n", "       109: (227,227,227) #E3E3E3 grey89\n", "       113: (228,228,228) #E4E4E4 srgb(228,228,228)\n", "       125: (229,229,229) #E5E5E5 grey90\n", "       130: (230,230,230) #E6E6E6 srgb(230,230,230)\n", "       183: (231,231,231) #E7E7E7 srgb(231,231,231)\n", "       263: (232,232,232) #E8E8E8 grey91\n", "       345: (233,233,233) #E9E9E9 srgb(233,233,233)\n", "       412: (234,234,234) #EAEAEA srgb(234,234,234)\n", "       357: (235,235,235) #EBEBEB grey92\n", "       320: (236,236,236) #ECECEC srgb(236,236,236)\n", "       219: (237,237,237) #EDEDED grey93\n", "       284: (238,238,238) #EEEEEE srgb(238,238,238)\n", "       378: (239,239,239) #EFEFEF srgb(239,239,239)\n", "       483: (240,240,240) #F0F0F0 grey94\n", "       596: (241,241,241) #F1F1F1 srgb(241,241,241)\n", "       631: (242,242,242) #F2F2F2 grey95\n", "       570: (243,243,243) #F3F3F3 srgb(243,243,243)\n", "       799: (244,244,244) #F4F4F4 srgb(244,244,244)\n", "       833: (245,245,245) #F5F5F5 grey96\n", "       827: (246,246,246) #F6F6F6 srgb(246,246,246)\n", "      1526: (247,247,247) #F7F7F7 grey97\n", "      1244: (248,248,248) #F8F8F8 srgb(248,248,248)\n", "       974: (249,249,249) #F9F9F9 srgb(249,249,249)\n", "      1369: (250,250,250) #FAFAFA grey98\n", "       568: (251,251,251) #FBFBFB srgb(251,251,251)\n", "       907: (252,252,252) #FCFCFC grey99\n", "       396: (253,253,253) #FDFDFD srgb(253,253,253)\n", "       439: (254,254,254) #FEFEFE srgb(254,254,254)\n", "     15299: (255,255,255) #FFFFFF white\n", "  Rendering intent: Perceptual\n", "  Gamma: 0.454545\n", "  Chromaticity:\n", "    red primary: (0.64,0.33)\n", "    green primary: (0.3,0.6)\n", "    blue primary: (0.15,0.06)\n", "    white point: (0.3127,0.329)\n", "  Matte color: grey74\n", "  Background color: white\n", "  Border color: srgb(223,223,223)\n", "  Transparent color: none\n", "  Interlace: None\n", "  Intensity: Undefined\n", "  Compose: Over\n", "  Page geometry: 180x180+0+0\n", "  Dispose: Undefined\n", "  Iterations: 0\n", "  Compression: JPEG\n", "  Quality: 75\n", "  Orientation: Undefined\n", "  Properties:\n", "    date:create: 2017-09-19T16:51:07-07:00\n", "    date:modify: 2017-09-19T16:51:07-07:00\n", "    jpeg:colorspace: 2\n", "    jpeg:sampling-factor: 2x2,1x1,1x1\n", "    signature: 085266d922dfd20f4d5de095323ac25c1a29911081b3a66f9d7f29ad5d094847\n", "  Artifacts:\n", "    verbose: true\n", "  Tainted: False\n", "  Filesize: 2527B\n", "  Number pixels: 32400\n", "  Pixels per second: 3.24MB\n", "  User time: 0.000u\n", "  Elapsed time: 0:01.009\n", "  Version: ImageMagick 7.0.7-3 Q16 x86_64 2017-09-18 http://www.imagemagick.org\n", "```\n", "\n"]}, {"metadata": {}, "cell_type": "markdown", "source": ["Next, I wanted to ensure that each image in the training and test datasets was clean of unique features in the JPEG File Interchange Format (JFIF).\n", "\n", "Based on the info on [this excellent webpage](http://vip.sugovica.hu/Sardi/kepnezo/JPEG%20File%20Layout%20and%20Format.htm), I wrote a simple parser for the file, and verified byte-by-byte that the parameters matched for all files."]}, {"metadata": {"_uuid": "3d5256fcdf0221617829438927659aaffa57ca38", "collapsed": true, "_cell_guid": "f7bf61e3-4af3-4a99-8c5a-f3d8c6f1db4f"}, "outputs": [], "cell_type": "code", "source": ["import bson\n", "import struct\n", "\n", "\n", "def parse_jfif_header(jfif_header):\n", "    # APP0 Marker.\n", "    assert jfif_header[:2] == b'\\xff\\xe0'\n", "\n", "    app0_len = struct.unpack('>H', jfif_header[2:4])[0]\n", "    assert app0_len == 16\n", "\n", "    # \"JFIF\" (zero terminated) Id String\n", "    assert jfif_header[4:9] == b'JFIF\\x00'\n", "\n", "    # JFIF Format Revision\n", "    assert jfif_header[9:11] == b'\\x01\\x01'\n", "\n", "    # Units used for Resolution\n", "    assert jfif_header[11] == 0\n", "\n", "    horizontal_resolution = struct.unpack('>H', jfif_header[12:14])[0]\n", "    assert horizontal_resolution == 1\n", "\n", "    vertical_resolution = struct.unpack('>H', jfif_header[14:16])[0]\n", "    assert vertical_resolution == 1\n", "\n", "    x_thumbnail = jfif_header[16]\n", "    assert x_thumbnail == 0\n", "    y_thumbnail = jfif_header[17]\n", "    assert y_thumbnail == 0\n", "\n", "    return jfif_header[app0_len+2:]\n", "\n", "\n", "def parse_quantization_table(quantization_table, expected_table_num):\n", "    # SOI / Start of image marker.\n", "    assert quantization_table[:2] == b'\\xff\\xdb'\n", "\n", "    quantization_table_length = struct.unpack('>H', quantization_table[2:4])[0]\n", "    assert quantization_table_length == 67\n", "\n", "    # Quantization table information (1 table, 8-bit precision).\n", "    info = quantization_table[4]\n", "    table_num = info & 0b00000111\n", "    assert(table_num == expected_table_num)\n", "    precision = info & 0b11111000\n", "    assert(precision == 0)\n", "\n", "    return quantization_table[quantization_table_length+2:]\n", "\n", "\n", "def parse_frame(frame):\n", "    # SOF0 Marker (Start Of Frame)\n", "    assert frame[:2] == b'\\xff\\xc0'\n", "\n", "    sof0_len = struct.unpack('>H', frame[2:4])[0]\n", "    assert sof0_len == 17\n", "\n", "    # Data precision (8 bits)\n", "    assert frame[4] == 8\n", "\n", "    image_height = struct.unpack('>H', frame[5:7])[0]\n", "    assert image_height == 180\n", "    image_width = struct.unpack('>H', frame[7:9])[0]\n", "    assert image_width == 180\n", "\n", "    number_of_components = frame[10]\n", "    assert number_of_components == 1  # Grey scale\n", "\n", "    component_id = frame[11]\n", "    assert component_id == 34\n", "    sampling_factors = frame[12]\n", "    assert sampling_factors == 0\n", "    quant_table_num = frame[13]\n", "    assert quant_table_num == 2\n", "\n", "    return frame[sof0_len+2:]\n", "\n", "\n", "def parse_huffman_table(huffman_table, expected_table_num, expected_table_type):\n", "    \"\"\"\n", "    See http://www.impulseadventure.com/photo/jpeg-huffman-coding.html for details.\n", "    \"\"\"\n", "\n", "    # DHT( Define Huffman Table) marker\n", "    assert huffman_table[:2] == b'\\xff\\xc4'\n", "\n", "    dht_len = struct.unpack('>H', huffman_table[2:4])[0]\n", "    if expected_table_type == 0:\n", "        assert dht_len == 31\n", "    else:\n", "        assert dht_len == 181\n", "\n", "    info = huffman_table[4]\n", "    table_num = info & 0b00000111\n", "    assert table_num == expected_table_num\n", "    table_type = (info & 0b00010000) >> 4\n", "    assert table_type == expected_table_type\n", "    # Bits 5-7 must be 0.\n", "    assert info & 0b11100000 == 0\n", "\n", "    return huffman_table[dht_len + 2:]\n", "\n", "\n", "def parse_scan(scan):\n", "\n", "    # SOS (Start Of Scan) marker\n", "    assert scan[:2] == b'\\xff\\xda'\n", "\n", "    scan_len = struct.unpack('>H', scan[2:4])[0]\n", "    assert scan_len == 12\n", "\n", "    num_components = scan[4]\n", "    assert num_components == 3\n", "\n", "    for i in range(num_components):\n", "        component_id = scan[5 + 2 * i]\n", "        assert component_id == i + 1\n", "        huffman_table = scan[5 + 2 * i + 1]\n", "        if i == 0:\n", "            assert huffman_table == 0\n", "        else:\n", "            assert huffman_table == 17\n", "\n", "\n", "def inspect_jpeg_file_layout(bytestream):\n", "\n", "    # SOI (Start Of Image) marker.\n", "    assert bytestream[:2] == b'\\xff\\xd8'\n", "    remaining = bytestream[2:]\n", "\n", "    remaining = parse_jfif_header(remaining)\n", "    remaining = parse_quantization_table(remaining, 0)\n", "    remaining = parse_quantization_table(remaining, 1)\n", "    remaining = parse_frame(remaining)\n", "\n", "    # Used for DC component of Luminance\n", "    remaining = parse_huffman_table(remaining, 0, 0)\n", "\n", "    # Used for AC component of Luminance\n", "    remaining = parse_huffman_table(remaining, 0, 1)\n", "\n", "    # Used for DC component of Chrominance\n", "    remaining = parse_huffman_table(remaining, 1, 0)\n", "\n", "    # Used for AC component of Chrominance\n", "    remaining = parse_huffman_table(remaining, 1, 1)\n", "    parse_scan(remaining)\n", "\n", "    # End of scan\n", "    assert bytestream[-2:] == b'\\xff\\xd9'\n"], "execution_count": 1}, {"metadata": {}, "outputs": [], "cell_type": "code", "source": ["from tqdm import tqdm_notebook\n", "\n", "with open('../input/train_example.bson', 'rb') as f:\n", "    data = bson.decode_file_iter(f)\n", "\n", "    for d in data:\n", "        for img in d['imgs']:\n", "            inspect_jpeg_file_layout(img['picture'])\n", "    \n", "    print(\"All files are clean!\")"], "execution_count": 2}, {"metadata": {}, "outputs": [], "cell_type": "code", "source": ["with open('../input/test.bson', 'rb') as f:\n", "    data = bson.decode_file_iter(f)\n", "    \n", "    bar = tqdm_notebook(total=1768182)\n", "    for d in data:\n", "        bar.update()\n", "        \n", "        for img in d['imgs']:\n", "            inspect_jpeg_file_layout(img['picture'])\n", "    \n", "    print(\"All files are clean!\")"], "execution_count": null}, {"metadata": {}, "outputs": [], "cell_type": "code", "source": ["with open('../input/train.bson', 'rb') as f:\n", "    data = bson.decode_file_iter(f)\n", "    \n", "    bar = tqdm_notebook(total=7069896)\n", "    for d in data:\n", "        bar.update()\n", "        \n", "        for img in d['imgs']:\n", "            inspect_jpeg_file_layout(img['picture'])\n", "    \n", "    print(\"All files are clean!\")"], "execution_count": null}, {"metadata": {}, "cell_type": "markdown", "source": ["Everything looks to be in order. Back to deep learning :)"]}], "nbformat": 4}