{"cells":[{"metadata":{"trusted":true,"_uuid":"3e3bdc8a9b07fc6953b739c8108f9512a74a7221"},"cell_type":"markdown","source":"The kernel is an attempt to overcome kernels limitation (RAM, disk space) and allow fast test_set reading from kernels. It also can be helpful for owners of low memory computers.\n\nThe idea is to store each column as numpy array, then access them using numpy.memmap and combine column into pandas when needed. Reading is available by object id or in chunks. A chunk may contain multiple object ids, object ids are not spread between chunks.\n\nData preparation is done in two kernels because of max disk space limitation. See [kernel 1](https://www.kaggle.com/alexfir/test-set-columns-part-1) and [kernel 2](https://www.kaggle.com/alexfir/test-set-columns-part-2). Output of the kernels is added as input to this kernel."},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\nimport os.path\nimport time\n\nCOLUMN_TO_TYPE = {\n    'object_id': np.int32,\n    'mjd': np.float32,\n    'passband': np.int8,\n    'flux': np.float32,\n    'flux_err': np.float32,\n    'detected': np.int8\n}\n\npart1_directory = r'../input/test-set-columns-part-1'\npart2_directory = r'../input/test-set-columns-part-2'\n\nCOLUMN_TO_FOLDER = {\n    'object_id': part2_directory,\n    'mjd': part2_directory,\n    'passband': part2_directory,\n    'flux': part1_directory,\n    'flux_err': part1_directory,\n    'detected': part1_directory\n}\n\n\ndef init_reading():\n    info = {}\n    object_range_file_path = os.path.join(COLUMN_TO_FOLDER['object_id'], 'object_id_range.h5')\n    print('reading {}'.format(object_range_file_path))\n    object_id_to_range = pd.read_hdf(object_range_file_path, 'data')\n    info['object_id_to_range'] = object_id_to_range\n    id_to_range = object_id_to_range.set_index('object_id')\n    info['object_id_start'] = id_to_range['start'].to_dict()\n    info['object_id_end'] = id_to_range['end'].to_dict()\n\n    records_number = object_id_to_range['end'].max()\n\n    mmaps = {}\n    for column, dtype in COLUMN_TO_TYPE.items():\n        directory = COLUMN_TO_FOLDER[column]\n        file_path = os.path.join(directory, 'test_set_{}.bin'.format(column))\n        mmap = np.memmap(file_path, dtype=COLUMN_TO_TYPE[column], mode='r', shape=(records_number,))\n        mmaps[column] = mmap\n\n    info['mmaps'] = mmaps\n\n    return info\n\n\ndef read_object_info(info, object_id, as_pandas=True, columns=None):\n    start = info['object_id_start'][object_id]\n    end = info['object_id_end'][object_id]\n\n    data = read_object_by_index_range(info, start, end, as_pandas, columns)\n    return data\n\n\ndef read_object_by_index_range(info, start, end, as_pandas=True, columns=None):\n    data = {}\n    for column, mmap in info['mmaps'].items():\n        if columns is None or column in columns:\n            data[column] = mmap[start: end]\n\n    if as_pandas:\n        data = pd.DataFrame(data)\n\n    return data\n\n\ndef get_chunks(info, chunk_size=1000):\n    object_id_to_range = info['object_id_to_range']\n    end_of_file_offset = object_id_to_range['end'].max()\n    start_offsets = object_id_to_range['start'].values[::chunk_size]\n    end_offsets = object_id_to_range['end'].values[(chunk_size - 1)::chunk_size]\n\n    end_offsets = list(end_offsets) + [end_of_file_offset]\n\n    chunks = pd.DataFrame({'start': start_offsets, 'end': end_offsets})\n    chunks = chunks.values.tolist()\n\n    return chunks","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1ffba8dd401bcdc68420cd963ecbaf80fc5039e3"},"cell_type":"markdown","source":"Before reading data, call init function and get info object that will be used later."},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"info = init_reading()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"e29d3dc61dcdd7d73dec31fb8c2c258f1757561c"},"cell_type":"markdown","source":"Reading can be done just by object id, like in code below."},{"metadata":{"trusted":true,"_uuid":"06baac27f130843fd6a9131ca6dd151aa60b35eb"},"cell_type":"code","source":"# single object read as pandas object, first object\nobject_info13 = read_object_info(info, 13)\nobject_info13.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1fcf2f1a66ec9fa1f3821408716288993a386dba"},"cell_type":"code","source":"# last object from test_set\nobject_info104853812 = read_object_info(info, 104853812)\nobject_info104853812.tail()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"13eb5429762fd51fbb0c83724a183e4d71dfed71"},"cell_type":"markdown","source":"Data can be returned as pandas object or as dict of numpy arrays (numpy memmap). "},{"metadata":{"trusted":true,"_uuid":"c5449f7efacf30381ea25f1bddda5cd4db06d653"},"cell_type":"code","source":"object_info104853812 = read_object_info(info, 104853812, as_pandas=False, columns=['flux', 'flux_err'])\nobject_info104853812['flux'][-5:]","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c95d418e637874e14e9dd5f53b8495cc4c07a25c"},"cell_type":"markdown","source":"Let's read all test set and perform some operation (we need doing something as data is not read until we use them) and measure required time."},{"metadata":{"trusted":true,"_uuid":"6fe270f4f00176c8037c1414dbd3a86bd15861e4"},"cell_type":"code","source":"    object_ids = info['object_id_to_range']['object_id'].values.tolist()\n    start_time = time.time()\n    records_read = 0\n    for object_id in object_ids:\n        object_info = read_object_info(info, object_id, columns=['flux'], as_pandas=False)\n        flux = object_info['flux']\n        records_read += flux.shape[0]\n        max = flux.max()\n\n    print(\"Single field reading took {:6.4f} secs, records = {}\".format((time.time() - start_time), records_read))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b336399748691048fa9c696bc7fbe5cf019bc680"},"cell_type":"markdown","source":"Not bad, but still only reading took a lot of time, even so single field was read and no pandas object was created.\nNow let's see what is performance of chunk reading. Each chunk contains 10000 objects and pandas object is created."},{"metadata":{"trusted":true,"_uuid":"c1df626563a8002a0d32e0df4d32780255106bfe"},"cell_type":"code","source":"    start_time = time.time()\n    records_read = 0\n    chunks = get_chunks(info, 10_000)\n    for index_start, index_end in chunks:\n        data = read_object_by_index_range(info, index_start, index_end)\n        flux = data['flux']\n        max = flux.max()\n        records_read += data.shape[0]\n\n    print(\"Chunks reading took {:6.4f} secs, records = {}\".format((time.time() - start_time), records_read))\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ef11d1f71dcccd2c4e819c9045663ed498508253"},"cell_type":"markdown","source":""}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}