{"cells":[{"metadata":{},"cell_type":"markdown","source":"# First let's examine our data:\n\n* train/test folder: list of files where each file contains values of (up to) 10 sensors containing the read value from that sensor of 10 minutes sampling.\n    * Numbers are normalized to int16. \n* train.csv: List of pairs of:\n     * segment id: train file id number\n     * eruption time: The target value, the time until the next eruption.\n* sample_submission.csv: same as train.csv but for the test dataset\n    * This file is empty -> we are responsible to fill it up using the system we create.\n"},{"metadata":{},"cell_type":"markdown","source":"# Describing the data\n\nEach data item can be interpreted as such:\n![image.png](attachment:image.png)\n* The input is [$N_i \\times 10$] matrix of sensors readings.\n* The output is the eruption time.\n    * $ x_{i,j} $ Represents the i’th reading of sensor j.\n* There are M items as such in the dataset.\n* $ M = 10 (min) * 60 (secs) * 100 (csecs) = 60,000 samples $\n","attachments":{"image.png":{"image/png":"iVBORw0KGgoAAAANSUhEUgAAAgQAAACcCAYAAAAAqLORAAAgAElEQVR4Ae2dZ9AlRdXHt8qvUIJVVklR+JoQEWU/IIUCwqIgFBhAkCAsOSgZJEdBQIIgWaKSJOeoCCI5SI5LzmlhDRhgTf3Wr5/b9/btp89Md997Z2fvnq66z8x0z8x/zukzff5zOjxTTEX63//+Z/72t79VnDG56P333zf/+te/JhcIOSUY//znP1uH8d///jdbV8jx73//W9DM5OymMP7+979PBq/IKZEjF+Mf//hHtq5GjfGf//zHlGBwXWpqCgP95iTOz5UjFwPdtg2D97VEDt7d1NQUBu9tTqI+cuVoIwY+LUcO/FmuHE1h4D9TE3Lgn6vSlKrCEmethKBKo/1lJY40l6CVYOQ6uSYwaIRzyVOuHLkYTTnrEjlG7UjR1agxkLttGE056yZIR66Toz5yHCm6aiNGU846R1elpEMJgTHWyNoWhaDym3DWTWDkOiAlBP1Er+qoxJHm1kcJRq4DagJDCUGVJfWXjYuzbkIOJQT9tuMfaYTA14a3j5PLIR1KCPK6PkqcnEYIPAOt2G3CWTeBoYSgopKDoiYc6bhgKCEIjMc7nPLBBx8Y6Uf4f9asWWJ57Lq//vWvtl81VhbLA+NPf/pT6zD+8pe/jFyOXAxISq6umsLAQcTqN5ZXKkcuxp///OfkZ+I50dWoMXimtmHwPLm6ypWjFANbidlQLK8pDOwkhi/loatcOXIxaBfahgGBGLUcpRj4Ham+wvymMPCdIXbVMXWeK0cdxpTZs2cb6cfDACqVx/IdIYiVxfIcIYiVSXlNYdDASM8Q5pfK0UYMGrBQvqpj50irzvHL0NWoMWgc24rBs/n6qNovlaONGNhJlaxhmXOkYb50jMyjxuB9bSsG75WkmzC/VI42YuAPQvmqjp0jrTrHL0NXo8aAdLQBQ7sMvHCJv0vjol0GvkbkfXTVtnA+XTi8ZDmJFz9HDh1UmDdbAv3mJOpPBxWmaQxd5Qxia+uAvybk0C4D2aaUEAi6UUIgKCaSrYQgohQhC6eY6+RKiE0uRq6zLpEjF0MJgWBEkewmHOm4YCghiBhQJ0sJgaAbJQSCYiLZSggiShGyShypEgJBmUE2JGjUpIMv6xKM3K/3JjB4b3OSEoJ0bTVFOnTaoU47TLfKjq5ywuAloXYlBOlVooQgXVc4oFFHOnIxlBDkLbDURtLRlLPOIYG6DkGnXShZ/Agjy+nfbwKDysfQclKJI20CI/eLtESOXAwc6aiJTS4GzqpEjlwn1wRG7hdpCbHJxUDuXF2NGkMJgRKClDa+KdKhEQKNEKTYY/ecJpx1Exg09EoIutVaudOEs24CQwlBZTX3FaKrnC9S3iXe25w0LhhNOeuc+tAIQccSm/h6bwJDIwS6MFFK41riSGmIc1IJBtfkpCYwlBCk18i4OOsm5FBCINuVDioUdAN7zumWUEKghEAwpb7sEkeqhKBPheIB3Qu5xCaXdGiXgXYZiAboFTRFOrTLQLsMPLOr320inN8EBg29dhnU1zdnlJCOXEfaBEaus1ZCMPc7a40QpL3jnAXpUEKghCDdYnSWQZauckkHDqiJr/cmMJQQpJmKRgjmftLR1Ne7jiEwxq6tnBNqb6J/vwkM7TLQLoMUl1LyZa2EIEWzxs5IyCU26DZnJoMSAiUEKdbYFOkYeoSA9ZOlH2t2v/XWW2J57Lp33nnH/v+DWFksrymMt99+e+RyzKsYM2fOzK7zXF2Bwdr2MRuK5XHuuGAge0xGKa9EV4oht4O+nrGrXF1hh7m2W4JBW+o/a9U+6/k3gYE/qHqOsAxd5cqhGGm2S53X6UoHFQp0TwcVCoqJZOsYgohShCyNEAiKiWTnfr3rGIK5/+udOs8JtROxof3JSU19vefIodMOOzXYRDi/CQztMtAug5RGSQlBipYmzlFCkKerHAdU4kibcNZNYCghkO1KIwSCbjRCICgmkq0RgohShCwlBIJiItlKCCJKEbKacKTjgqGEQDAiY4wSAkE3SggExUSylRBElCJkKSEQFBPJVkIQUYqQNS7Ougk5lBAIRqSEQFaMEgJZN2GJEoJQI/KxEgJZN2GJEoJQI/JxE450XDCUEMh2pBECQTdKCATFRLKVEESUImQpIRAUE8lWQhBRipA1Ls66CTmUEAhGpBGCnmKmTJliwt+WW27ZO6FmTwcV6qDCGhOxxfMqIVhnnXUmvV/Tpk2rVJkSgkr19BU24UjHBUMJQZ/p9B1ohKCjDsjAj3/84+5v3333Nddff32fsqoOlBAoIaiyD1c2rxKCCy+80Oyzzz7d94t3TQmBs4r+bVOLHxHZy0lKCNK11RTpGPrCRFUilkzXe//997P+KVAJRm44PwUDQnDLLbd01ZGLoYRACUHXeCp25lVCEK4RoIRANhIlBKNdT6EpZ50zDVTXIei8DynOOnx1cp11CoYSggktY8R8CeQk6oNGLDWVYOBI24ah/8vgP6lVPmlZYSUEsuqUECghkK2jVwKxwbelJkgHH+xVSbsMOtpRQjChiBJnrYSg6hXrL9MIwYQ+lBD024V/pIRACYFvD9L+SAjBBx98YKQfbGLWrFlieew61qbmCzNWFssDgzWWY2VS3igwIAQ33nhj9zlYT3vUcuRi4HhzddUUBo5Oqq8wv1SOXAzWjw+xq47R1agxeKa2YfA8ubrKlSPE2G+//cwKK6xQWT9gYCtVdeaXhRh+mbRfgoGdSPeL5TeBQbuQq6tcOXIxaD/bioHfidVVLK9UjlwM/FoMX8qjPoaNMWX27NlG+vEggErlsXznrGNlsTxHCGJlUt4oMCAEv/vd77qygkEDIz1DmF8qRxsxaMBC+aqOnSOtOscvQ1ejxqBxbCsGz+bro2q/VI42YmAnTtb999/fEgJ3HNs6Rxori+Uhs48ROyfMy8XgfW0rBu9VKJ90XCpHGzFoqyU5Y/nOkcbKYnnoatQYkI42YGiXQSceo10GE4rQLoP0sRA6hkDHEEjhXD+fxp73KjVpl0GeriCCOUkHFcraUkLQ0Y0SgglFKCFQQiA3F70SvpogRKlJZxnkOTn0m5NKSEeuIx0XDCUEsmUpIejoRgnBhCKUECghkJuLXokSgp4u6vZKHKkSgjqtTpQTTcklNkoIZN0qIejoZliEgEGYhx12mP2xOtsDDzxgnn32WXPccceZnXfe2ey+++7daX0YMgadmnDWGHNOKsGgAatLvpxrrbWWue+++0Q5w3uVkA4ayFxdpcjhP1suRlu6DPy6wObuvPNOM2PGjKjN+fK6/fDr3eVXbZUQVGmnv0wJQb8+qo5KdKWEoEqjvTKddtjRRVPrELz99tt2Nbb777/fIp933nlm6tSpdnU2BiKdfPLJZv755zd33HGHLS9x1m0gBLy0rDrn5PzlL39pllxySVHOnklO7CkhyAu1VxGbsC6wuS9+8YvmwAMPtIPfQpsL64JjJQTp9dFU/75GCGKWOjlPIwS6DoENEcF2UlNThIDlWXGONLAkpjFCAH7729/aEaTTp083q666qnn++edt+dxKCK688so+Oa+55ppKOcN6UkKQ7oDqohBhXTibu+GGG6I2F9YFx0oI0utDCUHeWIjcr3eNEMTe0HgeH4e6MJExrSQETNE8+OCDzYsvvtitvVNOOcVGCAjfxhJfATTgP/rRj8y1115rdtppp77lk8NrmBJz1lln2S/xyy+/3J7/2GOPhaf1HZeQjqovUojY4Ycf3ifnCSecYCMEkpwY7c0339yVc8cddzQ4rKpEw0s3xOqrr27uvfdeOwWUvNTUBOmoc9axZy0JtUv1EasLbI4IwRNPPBGDtw2IXxfYHMfSFyk6v+CCC8x2221nzj77bLPZZptZQlsih4+hCxNFq8dmNkU62uislRDIdhGWKCHoaARDbmOEwA/n47x32WUXs95665l33303rEt7zPgCHJ4jES+88IL59re/bfviYxewTsLaa69t14agHEf5ve99z7zzzjux023esAlBCIScO+ywg1l33XVFOR9//HGzxhprdOV87rnn7DFjK2IJefi/EjgiIix33323EoKYooI8Z3OMI6D7KpbCusDmvvWtb5lHH300drohArHyyiub1157zZZfdtll1qbfeuutbiQsemGQGUYhlBAECvIOlRCMNgpBO80HQ2rC1+SSp6YwNELQ0ggBBuYTAha/wAnuscce4mA4vqxxpCxIQXrvvfcM3QpnnHHGJFulQd1rr73swEQaftLrr79u/2PcTTfdNOl8lzFqQoCcq622mtltt91EOflqhRg5ORlPscEGG0TldM/NlrEWSgj+0x2E6usmtu9sDiJKxCqWwrrA5jbaaCM7viU839njtttu210D/ZFHHjGLL764JWzYZGpSQpDugJQQ5Omqrc66CdKhhGAuIQRPPfWUDd3GnDuNKA0kMw5+8IMfdBtbDHubbbYx/Ovl0Jhcw82XlUtEHhiTIGFw3qgJAXIuscQS5rTTTnOP1bdFzr333tv4ToWw4BZbbBGV079YCcGEnUhdBr6u2Hc2x0DCmLOO1QX2sfXWW1viGtqcsy+/7ugWYqCsP1YmfI7YMdjaZRDTzOQ8JQRKCCZbxeQcPkCVEMwlhOCqq64yiy66qA13u6rkS/7888+3hzTEOMUYIfAbYHeta5xjhIBpjlIaNSFAzs985jPdmRM8RygnJMeXyRECPy/2/HOaEKA7P+rjnpE8ynByzlnzYrolbd15sW1J37vDiN3Pz3M2R3eLIwR1dYEcEAJ+LEvrJ8rCumNmyUILLWQOOuigLoZ/jbSvhCDPyfnkSdKpn4+NhITOLw/3IR3Ub04aFwze3xxdzVNdBoRxpR8NHH2FUnksn/5fQpexslheUxj0qcbwXR7rEFx33XXdc3LlYIbBAgssYBvKmTNnWke/zDLL2MFdYNBfvuuuu9otx+h1k002saTAPRt5DNracsstbR+weza2jDP4+te/bqf7uXyXxxQzlxduc+WgPtzzhPfiGDkXXHDBPjmXXnpp2wdNeUzOUCbk3HjjjaNy+pjMzphvvvkswUCnrDvvl1ftc26VHLFrfYynn37adscsvPDC5rbbbuvi3nPPPWaxxRYz1C0DOh3G1VdfbZ91zTXXNK+++mr3/BDHxwjLYsfIwTWxslhd8FwPPvig1VVqXVA//JwsPhbvBGMIiAxgG4ceeqjhXSGKlVsfvhx0fy2//PJRuRz+MHXl7hlum8BAr4PoKnzm2DEY1E+sLJZHG+3XR+ycMK8Eg/YnvE/VsWLI/jjU2yh0pQsTdSjyoAsTsRgMAwSPPvpoc+yxx1pyQaN5wAEHmIsvvticfvrpts/fMXK+mKQuA8YduHEC7vyqLgPGIkiJrwC+BlITzLnqi5QBfgx89OXcc889Df+9TpJT6jKIyek/55yMENBgbrjhhma55ZazztA910svvWSJAoM5GcPhdMWXM0SBUfsuz13jb4cZIYjVBTaHviELMZsL68JFCBh3ENocz03k4+GHH7brGhxxxBF2rAGzGJhx4KIQvnzSvkYINEIg2Yaf30QUQiMEvsb795UQdPQxKCHAkcbCy/3q7j8KBxXC8BkhHnPwNKhuUKGbYYFzwmExZVFKwyYEMZw6jHAgG86WL+mYnP795yQh8J9D2qdOqpx/7LphEoLY/cmrwgjrAptj5goELyXdeuutdgwBM2SUEKRozFgbyQlR6xiCPPJE+5OTlBDI2lJC0NHNnCAENKr8T/gnn3zSPgVTwlZaaaXuHHK+Ok899VQbbuQEph2ussoqtruBY/qHiUoQgpdSnbMOr6uLEITnc1yHgVwrrrhiV06muCG3mysfyukwlBDkDSp0eqsiBGFdcDxt2jS7xDbXh3UBaaOcbhAcFRGII4880oaBlRA4jVdvS756qcOcVIKR60jHBUMJgWxZSgg6upkThICXnoWJCDPzlc8CRSwS40aO0hjzXC4CQEj3V7/6leF/yTM/nDEJDz30kFy7Cc46vHgUhAB5kMvJyXPTNy3JSffI7bffbhd6Qn4WfIL88DWbmkrkoD5yulfmxghBWBfYHETTRTpCm6NbYquttrL1dcghh9i1IZjSWEU6YnWErnwnp+sQxLQ0kacRAo0QyNbRK4HYuDa0lyvvEVkOBw6HZysh6GhkThCCui9rHo0BbY4Q4ORyuyVSMHyjKHGkw8Dw5fSfx+3nOusSOXIx5kZC4PTpb0NnXVcXXKuEwNdg9X7Jl7VPnqrvPlFagqERghTNGrsIXq6umopCKCGYS6YdpphanSOloWYglwutjyshCOWM6S7XWSshKFs0KKUuqB8lBDErjeeVOGslBHFdhrlEU9rqrGmDUlPp1EYlBPMQIWCMwRVXXNENC40rIQjljL1ESghiWonnDeKsU+oC1EEwuF67DOJ1R652GaQ7UiUE+t8OLSN0I+3l16pXAouqC7W3tcugJ4Wxi2nUyeGfz35dFCI8v+TLugkMJQRhTcnHgzpr+c69kkExlBD0dBnuKSFQQhDaROwYX6ARgnkoQhAawbhGCEI5Y8dKCGJaiecN6qzjd+3PHRRDCUG/Pv0jJQRKCHx7kPaVEHQ0wxepRggkM+nPb+LrvQkMJQT99Vp1NKizrrq3KxsUQwmB0+TkrRICJQSTrWJyjhKCjk6UEEw2DimnCWfdBIYSAqmGJ+cP6qwn33FyzqAYSggm69TlKCFQQuBsoWo7EkLAnGLpx5zFWbNmieWx61hvmVG1sbJYHhisXBcrk/JGgcEYAtYEcJjMeR+1HLkYON5cXTWFgYNwuqvblsqRi8H68XXP4pejq1Fj8Extw+B5cnWVK0eIwVLXLE7l6z/cBwNbCfOl4xBDOs/PL8HATvx71O03gUG7kKurXDlyMWg/24qB36mrN1deKkcuBn7NYaZsqY9hY0xhsRvpx0MBKpXH8p2zjpXF8hwhiJVJeaPAgBCwQIvDBIMGxh3XbUvlaCMGDVidvH65c6R+XtU+uho1Bo1jWzF4tir9+GWlcrQRAztxsrG4FoTAHce2zpHGymJ5yOxjxM4J83IxeF/bisF7FconHZfK0UYM2mpJzli+c6Sxslgeuho1BqSjDRi6MFEnJqOzDCYUobMM0v8RFHP2eZFzEo0L16WmpjB4rpxUIoePoV0Gsra1y0C7DGTr6JWMpMugd/vJeynT9cKrYJDDHvAXYvAlMGwMJQQTWlZCoIQgfN9ix0oIYlqJ50Eaea9SkxKCPF3hD3ISjjSnPvA1bcXQaYc67TDH9nUdggxt4eRojFNTU1/vTUQh/K/3FPmVEKRoaeIcJQSj1VVbnXUTpEMJgRKC9LdLFybK0pUSgnR1KSFI15USgtHqSglBmn6JdBDBr0o6hqCjHe0ymFCEdhlohKCqwXBlSgicJuq3SgjqdeTOKNGVEgKnveqtEoKOflLGQighmFCWEgIlBNXNykSpEoIULU2cU+Lk0G9OKsHIdaTjgqFjCGTL0ghBRzfDIgSs23DYYYfZ3zrrrGP4ZzHPPvusOe6448zOO+9sdt999+7IdF7InD5rnDXGnJNKMHjx65Iv51prrWXuu+8+Uc7wXiWkgwYyV1cpcvjPlovRljEEfl1gc3feeaeZMWNG1OZ8ed0+cuQ6IM7nutQUYugsA1lzOqhQBxXK1tEr0VkGHV3g5No4y+Dtt982++yzj7n//vvtk5533nlm6tSp9j+7MXf55JNPNvPPP7+54447bHmJs24DIcDR+nL+8pe/NEsuuaQoZ8+EJ/aUEOQ50ipiE9YFNvfFL37RHHjggXa+fGhzYV1wHDrr2DlhnhKCUCPyMXWEzacmJQR5uqIdzUkaIZC1pRGCjm6GESG48MILDc7RfTmx8iEE4Le//a1ddGL69Olm1VVXNc8//7xFnVsJwZVXXtkn5zXXXFMpZ2h+SgiGRwjCunA2d8MNN0RtLqwLjpUQpNdHU846N2JTQjpyHem4YCghiLUCE3lKCDq6GZQQsKrjwQcfbF588cWutk855RQbISB8G0u8kKyM9cwzz5gNNtjAXHrppbHTunmce9ZZZ9kv8csvv9zstNNO5rHHHuuWx3ZKSAcvvpSIzBx++OF9cp5wwgk2QiDJyb0gAU7OSy65pNttIuHQ8NINsfrqq5t7773XhrTJS01NkA4caZWuYs9a8mUtYcTqApsjQvDEE0/E4G2eXxfYXBUhQOcXXHCB2W677czZZ59tNttsM0toS+TwnZx2GYjVY7vGfF3JZ/ZKxsVZNyGHEoKe3YR7Sgg6GhmUENDI+uF8nPcuu+xi1ltvPfPuu++GerfHdDH84Q9/MFdffbVtxIkwVCWWVl577bXtctKch6P83ve+Z9555x3xsmETghAIOXfYYQez7rrrinK+99575vbbb+/KiYORnBz3R55bbrnFOiIiLHfffbcSglDxkWNnc4wjwLZiKawLbK6KEBCBWHnllc1rr71mb3fZZZdZm37rrbe6kbAYTpgXYighCDXUO24qCqERgp7Oq/Yg3rm6aop0DH0dAtZPln70e/PiS+WxfBpz1oqOlcXymsKggYzhuzwIwXXXXdc9p0QOH+Pll1+23QMMJGTQl8Pxtw6DMQdf+MIXzK9+9avoeVyDTnfddVez/fbbW4dJHl/kX/3qV62j9e/r7zsMP69qn/rw5ag6lzLkXGWVVcyOO+4oyunu4eSkWyUFg66W+eabz9x0001m5syZ9n8TuHvVbVmjPgXDv09bMXgu/zmlfWdzvo1I57q6wObQVQzj9ddfN+uvv77Zcsstu7q86667zGKLLWbokuA66f5hfoix9957m+WXX77y+pL6iMkRPot/3AQGdjiIrvznlfbB4N2VysN82pNcXZVg0P6E2FXHiiH741Bvo9CVRgg61HDYEYKnnnrKfvWfccYZIvl0X+84dgYfVkUI+LLbaKONbHeBuyGRB8YkpGC4a+q2uaF25FxiiSXMaaedVndrS2CQsy5C4G7E4EuNEKR3SzibYyAhX+RVybe58OvdXefsa9ttt+0uaOKu88fKuPOrtiGGRghkbWmEQAcVytbRKyEKMfQIQe/2k/dS5u+HV+n/MpjQyFVXXWUWXXRRG+52OuJL9/zzz3eH3WWFXSNbRQhc40xD6pLLY5qjlBzpkMrD/FxCgJyf+cxnujMnuF8op8NwciohqHbWTl9scaRV3Sv+uc7m6G5xhKCuLqq6DLCdbbbZxviEgMjCQgstZA466KAuhv8M0r4Sgjwnp2MIJEvqz4c8tTWcT1uamkq7JZQQtHTpYhrjBRdc0BxxxBH2f1oTQl922WVtSB2joOuFMClbl5yzdo5ybiAEjHf4yEc+0ifnl7/85e7MiZicTl4npxKC4RCCWF1gc+gZB5xSF1WEgHpjjMtqq61m6D6g8Tn66KMN0TQlBM6q67clA+WUENTrlTOUEPwvTVHG2Kn6jS5dzCjyAw44wHzpS1+yjQYNxze/+U1z7bXX2lHmMKZzzjnH9kFS9n//93/mhz/8oWHaWk7CkcKoUlNKpGPQLgMWg2FEPA3msccea8cj7LvvvlYfF198sTn99NNto+o/cw4hqOoyYJS/lByGVB7m10UIGOD37W9/u0/OPffc0+y3335GktNhKCEwdnCk+3p3eqnacq4UIYjVBTYH8cTRx2zOYbm6qCMEvDsPP/ywXdcAskt3BLMYmHGQK4fv5LTLwNXE5K12GaR/WSshaDEhwLRhIDhHworLLLOMeeGFF/osngrcY489DA0XIe8UZ913g5ZGCHCk/iyD8Jljx85Z+41z7DzyaHz32msvu9qhI0MvvfSSWW655Szhkq5zGFJ5mF9HCMLzOU7FcHJqhGA4EYJYXZCH461z1q4u6ghBiHHrrbfa8S6swFmH4V/LuUoIfI3I+0oIlBDI1tErmWvGEDCidtNNN7VRAr4kXML54wx+8pOfdPt9lBD8uzvYjsbZT/TXnnrqqXaEMvlMO2REv+t2oH+YqIQ79q91+6nO2p0/JwhBKKd7Fh1UmDeGwOltEEIQ1gXRp2nTpplXX33Vhmch8kceeaQdOa6EwGm8eqtdBtX68UtLdEUbl5OamhJIW5qaxn4MAfOVCcMzT57pMCT6I5kSxTQ8l+ZlQgBxwumdeOKJNqKCbn7/+9939UNjjA7pciExz5xpYvvvv79hfjjTEB966CGnyui2DYSARZsIbzs5WeTm+uuvF+V0c+VZ6An52UJ+nB1FBQ0yS4gNjpSvs9SEQ5TC+dI9Upy1f+2wMcK6wObQrVtnILQ56m2rrbayXWCHHHKIJfTco0QOrnFJuwycJiZvNUKQ7kjRlRKCyTYUy4F0NDqGgIdwswzeeOMNu6AJDTphxieffNLOZybM7ad5mRCkOOvbbrutSwhwcqXdEr7Oq/ZLHGmKHD5mDMOX0z/X7ec66xiGu5e0zcUYtrOOPVdTGL6zrqsLnpPzebbUxLk+hhICWXNKCJQQyNbRK5lrugxgIjh6NyJ5zTXXtFOXQjKAaEoI5C9SGlEGcrllaMeVEIRy9ky+t5frrJUQlDnrlLqgVpQQ9Gyzbq8kDO6Tp7r7Ux5ibLLJJnZws7+Uun+fki/rEMO/X2y/rRjaZRCrrYm8kS5M5OYsMx+fpWtjqYoQQCwOPfRQ841vfKPvx1Kq9KWH+f6xC7ODWYXhnolIBvO3XeKrF/zUNApnzcCtK664wj4/zzEKjFC+Ekc6aIQglDN8Jo6VEMS0Es8bxFmn1IWrD8hDauJc38lphEDW3DAiBD//+c/tLC7aNT7I6GL0U1uddROko82E4M0337Rj7JitR5cw+nAJcscgcjfWbK6KECAEYwW+853vTBpc6ARkm+Ks/fPZz3XWKRhtJASh3ONKCEI5Y8dKCGJaiecNQgjid5ycOyiGEoLJOnU5wyAE7l585BAtoH37xCc+YVc6xbEoIUjvluDDEJ+Tk0pIB8s882/ln3vuOTu2jDpjfI9L7j+ZsqQ7aSSEgLcaPmsAAB1lSURBVAFC0o/xADh1qTyWz3rLsBp+jEbeYostrDHyT3lYezm8BgzW1Q7zq44dRtU5flkKBspH4e46BrAhgzuu26ZghPfIxcAoc3XVFAYOIpRPOi6VIxeDQZvSM8Ty0dWoMXimtmHwPLm6ypUjxGDdihVWWKGyfsDAVmJ1FcsLMWLnhHklGNhJeJ+q4yYwaBeqdMVMpKOOOsp8/OMft+3xhz/8Yfu/JBigzH8fffrpp2tlqsMIdUD7maurpjBor8PnlY5L5cjFYKGx448/3rYPDMD+3Oc+Z/74xz92n5PB1UsuuaR59NFHbR66ysXAd0pykj+FkevSjxMAlcpj+QBiBPR9s779888/b77+9a9bI7z55psn3cs50ti9pDxHCKTyMD8FA0LAtD53LRg0MO64bpuCEd6jrRg0YOGzVh07R1p1jl+GrkaNQePYVgyezddH1X6pHG3EwE6crDgiCIE7jm2dI42VxfKQ2ceInePysEEWRsNBsuXr2f998pOfNFW/T33qU8b9Pv3pT5vwx3Le7kcZ+3Sdhr/PfvazJvbjPP6JlPvhHGK/xRdf3PDjPLf/+c9/3oQ//t+I+/E8Cy+8sF1x9EMf+pBtm2n/Flhggdr6QG9Oh3Vb2s/U+nD3os5zMWhH3fUpW+dIU87lHOQYNQakg26dRx55xK7Ps+GGG9qPaed/0cvmm29uNthgA/tPqXguyh588EG7ECD/mK9OHjDq5Bj6GAJeSlYjhM2wT7iedQcwOFZQC/sdq8L58/oYAhcqclvtMpAHYDoduW3JWAhefEKpqQlb5iXLSWCE70DV9U1h8Fw5qUQOH2NOdxnQ7jCF95RTTjFnnnmm3eeYf9oU/ih3Pz5yWK2RfygW/lgZMvxxPl97rCfCvvtxHP54Fn7cny9F9n/xi19M+lEe/hgzAM5JJ53U/XEc/lhSfaWVVrIEhTaZH+SArt0DDzyw0gSwdd6r1NTWbomScD6+LCcNggEpgBCee+653fFjDMhnWXKItGs/wOCdIoqQ0g7hTyFbVWnohICwOyvq+WsN3HvvvXaevb+2v3uoKkLgzgm3VE7OgL8UDF6Mtg0qDOVWQpDurJUQlA/4C+0udjy3EwInEw2pa2BdXtV2mP37Es6wMfi6ZDl1IiC0czh/HMtvfvOb7qJn0rP4+UoIfG1U7w9CCM477zwb7YEYuOQWabv88std1mjGEHTvHtlJcaT+ZcwqYGRkuHIehhRbuZBrczG4RgmBr/XqfXSV89Vb4kibwMABjVqOXIymvt5T2L9vBYM6a/9e0v6gGHM6QuDkGmdCwAwlZhhAAugWIYrgpiHyLvHe5iQlBOnaKiUEtClE0plJx1o+LhEtokvIkQTqghl4buXQFFLbWIQApw6DWXHFFc0999wT/Xq/6KKLrGH6KxcirBKCvK9eDC0nNeGsm8DIddYlxCYXQwlBeRRCCYH8Fg8jQjB16lTb3jLDwI98OlQlBOldHzjSXPJUSgjA4V+OT58+3bBiK4mxfHT18E/lGJhPOv/88+2Yg0svvdRGfBhPUJdGTgh4iGOOOaY7aBAmyqAHtwwqD4hiGN3Kf0ijnB+kgL40WI4SAiUEdYZMea6zVkJQ7qxT6yPlq8Tdi3OpQ5eUEDhNTN4OgxBAAugqkJISgnYSAvwhC/qttdZa3fpjyX8iPDvttJMlB7RtdPewYB2R98MPP7w71kCqb/JHTghi4AxaADg1KSFQQpBiK0oIUrQ0cQ66GsRZpyANiqGEQNbyMAiBfPeJEiUE7SQE1M7MmTPNbrvtZliciEGiRAv4kGYQq58gCswcYVBhSlJC0NFSCulA4X5ojdBNDrGBtZWG83k5mZ7DParSIBhV9/XLwCByk5O0yyBdWyWONLc+SjC4JicNiqGEQNa2EoLqdtDXXAmxKQ3n+7h1+8PEcOMH7rvvvi4sPo2BoUTbn332WfuPA7uFwo4Sgo5i2kwI6B+iYiEkjASuSkoI8qIpJY6UBiY16RiC8m4JJQSylSkhmDcJAWMGcP5bb721jRJgIXwosiYBg/X9cQx016+++up2aXvWL+C/5NYlJQQdDbWZEFBJ/MvZhRZayP5rWZ5VSkoI0p11SaSDr96QEFAfbnGVsF6UECghCG0idgwxxR5TkxKCPF35jjJFx8P8epfwSjAee+wxs9RSS9mvfmbq0fb8+te/toTglVde6YPCRljr57DDDjPHHXdcUlRXCUFHhW0mBKED6qv14EAJQfOEgLXE559/fvuSQgz8pIRACYFvD9K+EgJJM5PzS3Q1LoSACMFZZ51lHTzLGLOgH7MJpEgnpKPqAzLUrhKCjkaUEIzWkY7zGALW1mBZWEb4hi+mEgIlBGGjGzsucXK5YzpKMHId6bhglHy95+qqKYyhEwLWNpZ+fBERupDKY/n8xyb6N2JlsbymMJi/GcN3efThsx60Oy6Row7D3dttxwWDUbG5dZ6rKzCYRuV0V7fl3HHBQPY6ef3yEl01jcHiK8svv3ylXHODHL7epX3sMNd2c+sDDNpS6RnCfN7XJjBo40LsquMSORRD9uG+rqnzOl0NfelinXYY+0aI57kva7oNeJnr+hm1y2C0kY7YGIJYzcH+qTuNEGiEIGYfYV7Jl7VGCEItxo9pO9v69V7XnvsSEc4vkWPoEQL/ocL9lFB7eI0SglAj8jEGoLMMZP34JanO2l3Dy0hDnJNSMB5//HG7hCj/l+OFF14owoBIpKamSEeuA+L8XDl8DJ1lIFsATs7XlXxmr6SEdOQ6oHHBaCqcr4TAGPvflGA7qamEdGDIw8aYU+sQIIfOMqi3FhrI3AGYoyAE/NexadOm2UGGr7/+uhKC+qqzZ0AefCenhEBWnBKCeXOWQS5Bg9hohED/uZHckkRKMLJRO9ImMNpCCHwVN/X1XkJsBvl692WU9qmPQTCUEEiaNfZ99cmTfGavZFy+3puQQyMEPbsJ93QMQUcjcypCkOusMeac1ISzbgKjzYQA+VPrpcSRKiFIs/gwCpFyFbrNITZNfb0rIUipvQnyxPuXk5QQyNpSQtDRjRKCCUWU9L3Py4Tgueees/+qdJFFFjEPPPCA/KZ1SpQQTChCIwSyqTRFOnIdaRNf701gKCGQbU8JQUc3c5IQ0ADoLAPZSClpa4SA/+zJ0qLLLbecmTFjRrUQHTlyvkg5VyMEtWq1J2iEYLR970046yYwlBDI75MSgo5u5hQh0FkGsnH6JW0lBE046yYwckPUJZEOH0MjBL519+9rhGC0xEYJQb+9+UdKCDramFOEQGcZ+OYo7yshkHUTlgzqrMP7xY4HxVBCENPqRJ4SAiUEsnX0SiA2OstAZxn0LCJhr4n+/SYwlBAkVHbnlEGddQrSoBhKCGQtKyFQQiBbR69ECUFHFzigcVmHgJc/NTHgDyPISU046yYwlBCk1/qgzjoFaVAMJQSylpUQKCGQraNXMhJCQB+29GPVwVmzZonlsetYO5k+z1hZLA8M1liOlUl5o8Cgy+DGG2/sPgeD/EYtRy4GjjdXV01h4CCk+grzS+XIxWD9+BC76hhdjRqDZ2obBs+Tq6tcOUKM/fbbz6ywwgqV9QMGtlJVZ35ZiOGXSfslGNiJdL9YfhMYtAu5usqVIxeD9rOtGPidWF3F8krlyMXAr8XwpTzqY9gYU2bPnm2kHw8CqFQey3fOOlYWy3OEIFYm5Y0CA0Lwu9/9risrGDQw0jOE+aVytBGDBiyUr+rYOdKqc/wydDVqDBrHtmLwbL4+qvZL5WgjBnbiZN1///0tIXDHsa1zpLGyWB4y+xixc8K8XAze17Zi8F6F8knHpXK0EYO2WpIzlu8caawsloeuRo0B6WgDhg4q7ERg5tSgQu0y6IXAqvZ4KXN1xUuWk3IxmpoSWCJH7tRGZM9JnD8IhnYZyNrWLgPtMpCto1cyki6D3u0n75X8nwEY5LD798Mn40tg2BhKCCa0rAsTpY/pUEKQ90+afNKhhCBs1XrHSgiUEPSsQd5TQtDRjRIC2UjCEnQ16i/rJjByv95LiE0uhhICJQTh+xY7JsKDPaYmJQR5uqL9yUk40pz64OOzrRg67VCnHebYvjVkJQRpKlNCkKYnzkJX2mWQpi8lBGl64qwSXbXVWTdBOpQQKCFIf7s6ulJCkKYyJQRpeuIsJQTpuipxcug3J5Vg5DrSccHQCIFsWTqosKMbHUMwoYiSUDsNy6hJR66zLpEjF0O7DLTLQG5aeyUljlQJQU9/VXu0O7nERgmBrFElBB3dKCGYUESJI1VCIL9gYQkNfW6oHYeSk0owch3QoBg6qFCuUZxcbn2UkI5cRzouGEoIZNtTQtDRjRKCCUUoIdBZBnJz0StRQtDTRd1eiSNVQlCn1YlyjRD8L01RxtiZecwCrEpKCDraUUIwoQglBEoIqhoMV6aEwGmifquEoF5H7owSXeVGOjRC4LQ9eauEoKMTJQQTilBCoIRgcjMxOUcJwWSdSDklTk4jBJI2+/M1QqARAjuIZE4vTMR0j5tvvtnss88+ZvvttzeHHnqoefPNN815551nTjzxRLP66qub2267rd96g6OSvnfYbU4qwaABcymU87DDDjPvvfden5ws+cyLmZpKSAcNZNswmh5UmFIX2NygzjqlHgfF0DEEspaxcyUEsn78EiUEQyYErJ8s/Viz+6233hLLY9e988479v8fxMpieU1hvP3225VyECG47rrruufUycG5Bx54oNXPCy+8YFZeeWWz5pprmuuvv9488sgj5mtf+5rZbLPNKvVXhxHqC13VyRFeMyhGTM7vfve7fXJusskm5o033ujqLnyG8LhEjpkzZ9r/TRDeSzpmjfpcXbUVg+dCzpS6wOZeeeWVbF05DEmfYX6JrnyMvffe2yy//PKVNjMoRvjMseMmMLBD7DGGH8vjXF9XsXPCPDB4r8J86Zj1/JvAoP2RniGWXyKHYsg+3NcxdV6nK+0y6NDNnC4DFPujH/3INrxczhfzRhttZL7//e9b46fhXmSRRcwvfvGLvq9aohr33Xef2WmnnQx4p512Wrec/yp5yimnmMUWW8wcdNBB5rHHHvOJsN3ny7rJCEFMzunTp5uNN964T87jjz/e/tct98ChnGeddZYrsv89E70suuiiopzdk72deT1CkFoX6JZGwM1kqKsLZ3P898GYzXlV0LerEYI+dVQeaJdBpXr6Ckt0pWMI+lQoHtAW6KBCYwyh1jpHmkMIHnroIevMXaP7+uuvm2nTphnCoC5PqhXK9913X0sY1lprLfPuu+92T+U+e+65p12tq5vp7UiEgC/CXXfd1Vx88cXmyCOPNCeccIL9N5pcOkiXgSQn3Qa+nDEMX8511lnHfr04UV599VX7vH7XhCuTto4Q4BiPOuooS6r4b5xSaqJbAhlzZOBZSxwpGKl1EcOoqgtsbo899rBfi5IuY/lPP/202XbbbS3pxWYht75NhNdQhuwuaZeB08TkrXYZ6NLFk61icg4+Dd+WmpQQdDQ1bEIQVsD9999vFlpoIXP++eeHRZOOcWg777yzOfXUU83HPvYxc++993bP4T7HHHNM9zjciRECnMXmm29urrzySns6TnLHHXc0F154oT2OOevwvv5xlSN1cl5++eX+JVHS4cuJbrjWJaIkhx9+uDtM2hKF4R633nqrIUqBM6piu1VySICOdEjlYX6ThCDEluqC80LSUVUX3OdnP/tZn7MOscJj7ke3EdEF3i2OIX3XXntteGr3WAlBnpPzyVNXiRU7tAPYfGoq6XsfFwwcaY6ucKRtjUIoIWjh0sU4X0Lgd9xxR+37+NRTT1ln+Mwzz5ill17aHHDAAV3jPPfcc81NN90k3gMjDiMdjFdYaqml7Beku5CGetNNN7Vfr8MkBE7OBx980EHZbQzDl3OZZZax0RP3Ep5zzjmVzqPv5p0D56zB2mabbeZ5QiDVBeoKCUFVXWBzN954YxYh4Pz555/f3HnnnbZ2cPaMCagiaUoI8py1EoJYKzA5r4TYKCGYrEeXo2MIOprI6TJwymMLQ2NwIYMKn332Wb8oun/11VebCy64wIZXaUS/8pWvmJdeeske89UMUZBSjBDQOEMIfGxmOqy00kp2QGPMWUv3Jx8MvgTC5MvJAEI/xTB8Offff3+z7LLLduX86U9/agde+veo21dC0NNQVV1wVkgIquoCm4Mw5Dgg7MsnBGAefPDBZtVVV+3rAus9sbH27WNol4Gvnf597TLII0+0PzlJCYGsLSUEHd2UEgJG9hIuJUzvxgPgUGnwGDHrJxryI444ovs1zxQxwul87RF23WuvveyAMP8afz9GCGicY4TA5cWctX/PcF8iBE5OBkS6vnsn58svv9wdHMn9QjnpFvHlpM+ZKZo5SQlBT1tVdYHN+YSgri6wOWzPd9Y9pPgekbCPfvSj3QjB7NmzzS677DLJDv2rNUKQ5+Ry6gM9j0s4vwk5lBD4b2b/vhKCjj5SCQGN54YbbmgIgz///PN2DADOjpH29HPTABMBuOiii+y+r27KcYbuC5stUxW32GILc9ddd9m1DGg4pdQkIZDkZMAiMvpyOmftnjuUk3sxgNLJecghh1QSH3cff+swIDjzWpfBa6+9FrW5WF1QL+jK2VFdXbB+Bg49xwHRaG+99dbm5JNPtlXEtFveB0dC/Xpz+0oIlBA4W6jaKiGo0k5/GcSG9z016aDCjqZQGsqrSqmEAIPl65/53meccYY5/fTTDYPs1l9/fXP22WfbgVbSQj10DTCl0A2Gw7kxOhtCsdtuu5lLLrmk6hFtOD+UQ+oyWHHFFQ1f7qURAknODTbYwJIdxik4OUOMUE6EQldOTmZDcP+cNC8TAr76YzYXqwt06hOCurrA5kJnnVIvzGGHEBDxYiGuH/zgB2aNNdaw0YbY9SGGdhnEtDSRp10GeeSJ9icnaYRA1pZGCDq6SSUEkipjX+/huYyQ56vOJQyZvn++rhZffHHz+OOPu6Luli881jxgjngMIzaokJkK6623nv0KD51198bCjtRlIJxus0OMUE5Ocl+RyPnoo49OIgS+nDGseZkQlJAnFyGoqwtsLnTWdXVB/fikg/0tt9zSkl2HG9ZhiKGEINRQ71gJgRKCnjXIexoh6OgGB0T4IzUNM0IgYcacdXjuscce2zcTATlo7Bl/sNpqq9lVpMJr/GMw+Frka9vNaOB6ZhS4xX84ZhqiOw6dtX+/2P4wCEEoJzhuOiRyIkOdk6Pcl1MJQay24nm+s66rC1YuC511eNewLog8ffWrXzUnnXSSPRVSylLdM2bMCC/tHocYSgi6qpm0o4RACcEko4hkKCHoKGVuIwQ0um5FOAZfuZkEzlkz1ZDBXfTl+onowQ9/+ENLGGiUcdZPPvmkoTvARQA4n4WJGOzH4MSjjz7aRiG4N8lh+Pet2h+EEEhyOjwnJ10mPiEI5eR8nI4vJ06OKAnrLTCDgh/7LJATSyVyONIRu18sDyfnyxE7J8zznXVYFjsuxWDJ8ZjNOQxXF9ic76xT6gKZ+R8ejJuhy2H33Xc3L774ort1dOtjcIISgqiabKYSAiUEsnX0SkZCCPhyk3403CypK5XH8lk2lQYjVhbLA4NBZ7EyKW8UGHQZ0B/vMBnJPWo56jCYr89StFtttZVdCx3njq74MXWPflz3vNK2DiO8zmGE+VXHYODoqs7xy0KMUE53ri9nCQZrwrt7pWybwOCZcnWVK0cuBs/jMFLqAl0OgsH1LJW8wgorVNYPGNhKSt1xji9H6jUlGNhJ6v05rwkM3pVcXeXKkYtB+9lWDPxOah2WypGLgV9LfSbOoz6GjTGFLwTp50Cl8li+c9axslieIwSxMilvFBgQAgbJOUwwaGDccd22VI4qDPTPlxThf/AdBsvYQghoAOqeq0QOGrC6+/rlzpH6eVX7yOFjhHK6a305czHQjY/h7lm1bQojpd7cc5bKUYqRUhc8G7rKxeAaJxdrU0AI3HFs6xxprCyWx/P4GLFzwrxcDN7XtmLwXoXyScelcrQRgzZOkjOW7xxprCyWh65GjQHpaAOGDirsRGCaGFTYC/ZM7NGAER6UEiF4RpKfeeaZlqwQBmetA0LBhIRTUh1GeI+SUPugGKGcPBOOyZeTl7JKV8OQIxejNJzPdampKQxkJ6XUBedxfq4cDoPrtcvAqjv6R7sMtMsgahhB5ki6DAKMvsOUwXh9Fxhjv2KHPeAvxMABDRujjYQApw8hYEVDugdw1hhBThrUWadgDYoRyhnDzHXWJcQmF6MpZ83XQ04axFmn1AXPMggG1yshkGtUCYESAtk6eiVKCDq6GAUhmDp1ql1TwKk7F6MpZz2OhMDpvGqb66yVEORFIdBvThqUELBA19prr10JCREaJApRefNOYS5GU846tz6QA5tPTchBG5eTxgWDNjRHV3x85uqqKQxdmKiF/9yIl0oJgdz1ETY6TTjrJjDGMUIQ1pV0PCgh4P9/MIukKuU6a+qjxJHmkA4lBHM/6WjKWTdBOpQQjJAQMJ+fUCY/RkEzyDA1KSFQQpBiKyWOFMeYk0owch1pLsZll11m9t133+77NW3aNCUEQqU2RTpyv3o1QiBUWCS7KdKhhGCEhIBxBP5v+vTpkaqOZykhUEIQt4z+3FxH2lQUYtSEgBkF/rvFvkYI+m3DHSkhGG0UoilnrRGCuXhQoXsZ3VbHEDhN1G/RFY1YauJFKfnqbRtGU866RFc5YXDOHTUhKMFA7lHLkYvRlLPOrQ/kyHFAyKERgrQWS8cQdPQ0L80yCE1DCUGoEflYCYGsm7BkXo0QKCHI++pVQhC+OfHjEmKjEYK4LsnVdQgE3SghEBQTyVZCEFGKkKWEQFBMJDv3670J0qERgjxiQ9uQk5qIdCghkGtECYGgGyUEgmIi2UoIIkoRspQQCIqJZCshiChFyGrCkY4LhhICwYg0QiArRgmBrJuwRAlBqBH5WAmBrJuwRAlBqBH5eFycdRNyKCGQ7WgK6ydLP9bsZuUyqTyWz9KnrBUdK4vlNYXBv9yN4Ut5JXLMqxisophb57m6AoN156X6CvM5d1wwkD2Ur+q4RFeKIbeDvq6xq1xdYYe5tluCQVvqP2vVPu9rExi0o1XPEZahq1w5FCPNdqnzOl1pl4FAljRCICgmkq0RgohShCyNEAiKiWRrhCCiFCGriS/rccHQCIFgRNplICtGCYGsm7BECUGoEflYCYGsm7BECUGoEfl4XJx1E3IoIZDtSCMEgm6UEAiKiWQrIYgoRchSQiAoJpKthCCiFCGrCUc6LhhKCAQj0giBrBglBLJuwhIlBKFG5GMlBLJuwhIlBKFG5ONxcdZNyKGEQLYjjRAIulFCICgmkq2EIKIUIUsJgaCYSLYSgohShKwmHOm4YCghEIxIIwSyYpQQyLoJS5QQhBqRj5UQyLoJS5QQhBqRj8fFWTchhxIC2Y40QiDoRgmBoJhIthKCiFKELCUEgmIi2UoIIkoRsppwpOOCoYRAMCKNEMiKUUIg6yYsUUIQakQ+VkIg6yYsUUIQakQ+Hhdn3YQcSghkO9IIgaAbJQSCYiLZSggiShGylBAIiolkKyGIKEXIasKRjguGEgLBiDRCICtGCYGsm7BECUGoEflYCYGsm7BECUGoEfl4XJx1E3IoIZDtSCMEgm6UEAiKiWQrIYgoRchSQiAoJpKthCCiFCGrCUc6LhhKCAQjIkLwwQcfGOn3/vvvm1mzZonlsetYmxrDiZXF8sBgjeVYmZTXBAbraY9ajlwMHG+urprCwNFJ9RXml8qRi8H68SF21TG6GjUGz9Q2DJ4nV1e5cpRiYCtVdeaXNYWBnfi4dfvoKleOXAzahbZh0H6OWo5SDPxOXb258qYw8GsOM2VLnefKUYcxZfbs2Ub68VCASuWxfOesY2WxPEcIYmVSXlMYNDDSM4T5pXK0EYMGLJSv6tg50qpz/DJ0NWoMGse2YvBsvj6q9kvlaCMGdlIla1jmHGmYLx0j86gxeF/bisF7JekmzC+Vo40Y+INQvqpj50irzvHL0NWoMSAdbcD4f1Jbf+dltkrmAAAAAElFTkSuQmCC"}}},{"metadata":{},"cell_type":"markdown","source":"lets add some classes and methods for later usage:"},{"metadata":{},"cell_type":"markdown","source":"## imports"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport scipy.fftpack\n# from scipy import signal\nimport seaborn as sns","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## classes & methods"},{"metadata":{"trusted":true},"cell_type":"code","source":"# =============================== constants ===============================\n\n\nclass C:\n\n    data_path = '../input/predict-volcanic-eruptions-ingv-oe/'\n    col_name_index = \"segment_id\"\n    col_name_target = \"time_to_eruption\"\n    col_name_sensor = \"sensor_\"\n    N_sensors = 10\n    N_samples = 60000  # in hundredths of seconds\n    N_length_in_csec = 60000\n    N_length_in_sec = 600\n    N_length_in_min = 10\n\n\n# =============================== data methods ===============================\n\n\nclass Data_Methods:\n    @staticmethod\n    def fetch_matrix(dataset, i_data):\n        matrix = np.zeros([C.N_samples + 1, C.N_sensors])\n        data_i = dataset.get_x_i(i_data)\n        for i in range(0, C.N_sensors):\n            sens_col = \"%s%d\" % (C.col_name_sensor, i + 1)\n            matrix[:, i] = data_i[sens_col]\n\n        return matrix\n\n    @staticmethod\n    def create_time_vec(num_samples=C.N_samples, units=\"min\"):\n        if units == \"min\":\n            times = np.linspace(0, C.N_length_in_min, num_samples + 1)\n        elif units == \"sec\":\n            times = np.linspace(0, C.N_length_in_sec, num_samples + 1)\n        else:\n            exit(\"please choose units for time vec, either min or sec\")\n            times = []\n\n        return times\n\n    @staticmethod\n    def validate_set_type(set_type):\n        if set_type == 'train' or set_type == 'test':\n            return set_type\n        else:\n            exit('please choose appropriate data set type, either \"train\" or \"test\"')\n\n\nclass Dataset():\n    def __init__(self, path=C.data_path, set_type='train'):\n        self.base_path = path\n        self.set_type = Data_Methods.validate_set_type(set_type)\n        # self.X_ids, self.targets = self.fetch_targets()\n\n        # def fetch_targets(self):\n        # read train.csv returns panda format of [x->y] pairs\n\n        if self.set_type == 'train':\n            full_path = self.base_path + 'train.csv'\n        else:\n            exit(\"TODO: #0001\")  # TODO #0001\n            full_path = []\n        try:\n            data = pd.read_csv(full_path)\n        except Exception:\n            exit('train.csv is not located at: ' + self.base_path)\n            data = None\n        self.X_ids = data[C.col_name_index]\n        self.targets = data[C.col_name_target]\n\n    def get_x_i(self, i_data):\n        # read <names[i_data]>.csv from train/test folder\n        # returns panda format of matrix X[0:60,000,0:10]\n        try:\n            file_name = self.X_ids[i_data]\n            full_path = self.base_path + '%s/%d' % (self.set_type, file_name) + '.csv'\n            data = pd.read_csv(full_path)\n            return data\n        except Exception:\n            exit('could not find at %s folder the file #%d' % (self.set_type, i_data))\n\n    def fetch_vector(self, i_data, i_sensor):\n        X_matrix_i = self.get_x_i(i_data)\n        vec = self.x_matrix_to_vec(X_matrix_i,i_sensor)\n        return vec\n\n    def x_matrix_to_vec(self, X_matrix, i_sensor):\n        sens_col = \"%s%d\" % (C.col_name_sensor, (i_sensor + 1))\n        vec = X_matrix[sens_col]\n        return vec\n\n    def get_y(self):\n        return self.targets\n\n    def get_x_i_name(self, i):\n        # returns the name of the file for the i'th file\n        return self.X_ids[i]\n\n# =============================== Math methods ===============================\n    \n\nclass Math_Methods:\n\n    @staticmethod\n    def divide_to_square(N):\n        # splitting N into N1 and N2 such that:\n        # N1*N2 = N\n        # N1,N2 are integers\n        # N1,N2 are close to each other as possible\n        # designated mostly for square'ish subplot N vectors.\n\n        fs = Math_Methods.factors(N)\n        N1, N2 = fs[-1]\n\n        return int(N1), int(N2)\n\n    @staticmethod\n    def factors(N):\n        factors = []\n        for i in range(1, int(N ** 0.5) + 1):\n            if N % i == 0:\n                factors.append((i, N / i))\n        return factors\n\n    @staticmethod\n    def convert_cs_to_min(cs):\n        return cs/100/60\n\n    @staticmethod\n    def generate_spectrogram(signal_t):\n        fc = C.N_samples / C.N_length_in_sec # sampling frequency\n        freqs, times, spectrogram = signal.spectrogram(signal_t, fc)\n        return freqs, times, spectrogram    \n    \n# =============================== UI methods ===============================\n\nclass UI_Methods:\n\n    @staticmethod\n    def plot_vec(y, x=None, xlabel='', ylabel='', title=''):\n        plt.figure()\n        if x is None:\n            plt.plot(y)\n        else:\n            plt.plot(x, y)\n\n        plt.xlabel(xlabel)\n        plt.ylabel(ylabel)\n        plt.title(title)\n        plt.grid(True)\n        plt.show()\n\n    @staticmethod\n    def plot_vec_with_line(y, x, line_x, xlabel='', ylabel='', title=''):\n        plt.figure()\n\n        plt.plot(x, y)\n        plt.axvline(x=line_x, color='r')\n\n        plt.xlabel(xlabel)\n        plt.ylabel(ylabel)\n        plt.title(title)\n        plt.grid(True)\n        plt.show()\n\n    @staticmethod\n    def plot_sensor(sensor_values, sensor_index, data_index):\n        times = methods_data.Data_Methods.create_time_vec(units=\"min\")\n\n        UI_Methods.plot_vec(sensor_values, times,\n                            xlabel='t[min]',\n                            ylabel='sensor value',\n                            title='sensor[%d] of sample[%d] over time' % (sensor_index, data_index))\n\n    @staticmethod\n    def plot_sensor_with_timestamp(sensor_values, sensor_index, data_index, timestamp):\n        times = methods_data.Data_Methods.create_time_vec(units=\"min\")\n        x_line = times[int(timestamp)]\n\n        volcano_time = UI_Methods.timestamp_to_time(timestamp)\n        UI_Methods.plot_vec_with_line(sensor_values, times, x_line,\n                                      xlabel='t[min]',\n                                      ylabel='sensor value',\n                                      title='sensor[%d] of sample[%d], volcano@%s' % (sensor_index, data_index, volcano_time))\n\n    @staticmethod\n    def subplot_all_sensors(y_array, data_index, timestamp = None):\n        # plot N vectors in subplot form,\n        # make sure y's in y_array are columns\n        # also make sure y_array is numpy matrix\n\n        xlabel = 't[min]'\n        ylabel = 'sensor value'\n        super_title = 'sensors of sample[%d] over time' % data_index\n        subtitle = 'sensor_'\n\n        x = methods_data.Data_Methods.create_time_vec(units=\"min\")\n\n        [N, M] = y_array.shape\n        if len(x) != N:\n            exit('when using plot_multiple vecs, make sure y_array is column wise vectors')\n\n        fig, axs = plt.subplots(M, 1, figsize = (12, M))\n\n        for i in range(M):\n            index = i\n            y = y_array[:, index]\n            axs[i].plot(x, y)\n            if timestamp is not None:\n                axs[i].axvline(x=x[timestamp], color='r')\n\n            axs[i].grid(True)\n            # axs[i].set_title(subtitle + \"%d\" % index)\n            # axs[i].set_ylabel(ylabel)\n        axs[M - 1].set_xlabel(xlabel)\n        fig.suptitle(super_title)\n        fig.show()\n\n    @staticmethod\n    def plot_spectrogram(times, freqs, spectrogram):\n        plt.pcolormesh(times, freqs, spectrogram)\n        plt.imshow(spectrogram)\n        plt.ylabel('Frequency [Hz]')\n        plt.xlabel('Time [sec]')\n        plt.show()\n\n    @staticmethod\n    def timestamp_to_time(timestamp):\n        cs = timestamp % 100\n        sec = (timestamp // 100) % 60\n        min = (timestamp // (60 * 100)) % 60\n        hrs = timestamp // (60 * 60 * 100)\n\n        str = '%02d:%02d:%02d.%02d' % (hrs, min, sec, cs)\n        return str\n    \n    @staticmethod\n    def plot_hist(vec):\n        # calculating freedman-Diaconis's bin value:\n        q25, q75 = np.percentile(vec, [.25, .75])\n        bin_width = 2 * (q75 - q25) * len(vec) ** (-1 / 3)\n        bins = round((vec.max() - vec.min()) / bin_width)\n        bins= 2035\n\n        plt.hist(vec, density=True, bins=bins)  # density=False would make counts\n        plt.ylabel('Probability')\n        plt.xlabel('Data')\n        plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## translation from pycharm to notebook:"},{"metadata":{"trusted":true},"cell_type":"code","source":"class methods_data:\n    Data_Methods = Data_Methods\n    Dataset = Dataset\n    \nclass methods_UI:\n    UI_Methods = UI_Methods\nclass methods_math:\n    Math_Methods = Math_Methods\nC.data_path = '../input/predict-volcanic-eruptions-ingv-oe/'\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Examine the data\n\nLet’s take a single column of sensor over time (10 minutes) and see how it behaves:"},{"metadata":{},"cell_type":"markdown","source":"Lets first load the dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"m_data = methods_data.Dataset()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## picture of single sensor values\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"def examine_single_sensor(m_data, data_index, sensor_index):\n    sensor_values = m_data.fetch_vector(data_index, sensor_index)\n    methods_UI.UI_Methods.plot_sensor(sensor_values, sensor_index, data_index)\n\nexamine_single_sensor(m_data, 3,5)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"very noisy, not much of info here.."},{"metadata":{},"cell_type":"markdown","source":"## comparing sensor powers\nLet's put sensor value ranges next to each other to see their values:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_box_plots(data):\n    sns.set(rc={'figure.figsize':(10,3)})\n    ax = sns.boxplot(data=data)\n\ndef compare_sensor_values(m_data, data_index):\n    x = m_data.get_x_i(data_index)\n    plot_box_plots(x)\n    print(x.describe())\n\ncompare_sensor_values(m_data, 2)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Examine target values"},{"metadata":{"trusted":true},"cell_type":"code","source":"y = m_data.get_y()\ny_min = y/100/60\ny_min.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"ax = sns.boxplot(data=y_min)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"as we can see, most of the occurance times of the volcanos occure after more than 10 minutes, which means the occurance time isn't shown on the sensor data\n"},{"metadata":{},"cell_type":"markdown","source":"let's check only the data sets where the volcano erupted in the 10 minutes window and see if we can find some intreresting information there:\n\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"def find_volcano_in_data():\n    indices = []\n    m_data = methods_data.Dataset()\n    y = m_data.get_y()\n    plot_counter = 0\n    for i, y_i in enumerate(y):\n        if y_i < C.N_samples:\n            x_i_name = m_data.get_x_i_name(i)\n            time = methods_UI.UI_Methods.timestamp_to_time(y[i])\n            print('the file %s has volcano occurrence at %s' % (x_i_name, time))\n            indices.append(i)\n\n\n    return indices\n            \nids = find_volcano_in_data()\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Lets see those sensors:\nfirst lets create printing method:"},{"metadata":{"trusted":true},"cell_type":"code","source":"def print_sensors(indices):\n    m_data = methods_data.Dataset()\n    y = m_data.get_y()\n    for i, y_i in enumerate(y[indices]):\n        sensor_matrix = methods_data.Data_Methods.fetch_matrix(m_data, i)\n        methods_UI.UI_Methods.subplot_all_sensors(sensor_matrix, i, y_i)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## lets now check the first 2:"},{"metadata":{"trusted":true},"cell_type":"code","source":"print_sensors(ids[0:2])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"nothing really happens at the eruption time.. maybe another one:"},{"metadata":{"trusted":true},"cell_type":"code","source":"print_sensors([ids[6]])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"hard to tell.. :\\"},{"metadata":{},"cell_type":"markdown","source":"# Lets check spectrograms"},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_spectrogram2(vec_t):\n    Fs = C.N_samples / C.N_length_in_sec # sampling frequency\n    spectrum, freqs, times, im = plt.specgram(vec_t, Fs=Fs)\n    N_time_ticks = 10+1\n    times_min = ['%.0d'%i for i in np.linspace(0,C.N_length_in_min,N_time_ticks)]\n    times_indices = np.linspace(0,C.N_length_in_sec,N_time_ticks)\n    plt.xticks(times_indices, times_min)\n    plt.xlabel('time [min]')\n    plt.ylabel('freq[Hz]')\n    plt.set_cmap('jet')\n    plt.colorbar()\n    \n    plt.show()\n    \nx = m_data.fetch_vector(i_data=0, i_sensor=1)\nplot_spectrogram2(x)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# simple fft view"},{"metadata":{"trusted":true},"cell_type":"code","source":"def examine_fft(i_data, i_sensor):\n    m_data = methods_data.Dataset()\n    x = m_data.fetch_vector(i_data=0, i_sensor=1)\n    N = len(x)\n    \n    freqs = np.fft.fftfreq(N,C.N_length_in_sec/C.N_length_in_csec)\n    Y_jw = np.fft.fft(x)\n    Y_dBm = 20*np.log10(np.abs(Y_jw)/N/(1e-3))\n    methods_UI.UI_Methods.plot_vec(Y_dBm,freqs,\n                                   xlabel=\"freq[Hz]\",\n                                   ylabel='sensor power [dBm]',\n                                   title='sensor[%d] from data[%d] in freq domain'%(i_sensor,i_data))\n\n    \n#     N = C.N_length_in_sec\n#     T = N / C.N_length_in_min\n#     X_jw = scipy.fftpack.fft(x.to_numpy())\n#     freqs = np.linspace(0.0, 1.0 / (2.0 * T), int(N / 2))\n#     freqs2 = scipy.fftpack.fftfreq(N)\n#     Yf = 2.0 / N * np.abs(X_jw[:N // 2])\n#     methods_UI.UI_Methods.plot_vec(Y_dBm2,freqs,\n#                                    xlabel='freq[Hz]',\n#                                    ylabel='|sensor(f)|',\n#                                    title='sensor[%d] from data[%d] in freq domain'%(i_sensor,i_data))\n\n\nexamine_fft(i_data=0,i_sensor=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"# Lets devide the target values to binary sets using k-means clustering"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.cluster import KMeans\ny2 = pd.DataFrame(y)\nkmeans = KMeans(n_clusters = 10).fit(y2)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"centers = kmeans.cluster_centers_\ncenters_in_hrs = centers / 60/100/60\nnp.sort(centers_in_hrs, axis=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# now lets now use those centers and split [x,y] accordingly\n\n","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}