{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"<h2 align=\"center\"> Converting TARGET data with many zeros into a frequency matrix </h2>\n\n<div style=\"font-family:verdana;\">Most of MULTIOME_TRAIN_TARGETS data are zeros.</div>\nOut of 23 thousand target columns, in average, 98% of data presented by zeros. \n\n<div class=\"alert alert-success\" role=\"alert\">\nWe will pick up only non-zero values and present data with index of related columns.\n</div>\nThe original data couldd be like that:\n<br>\n<table>\n    <tr>\n        <td style=\"border:1px solid green;\">0</td> \n        <td style=\"border:1px solid green;\">0</td>\n        <td style=\"border:1px solid green;\">0</td>\n        <td style=\"border:1px solid green;\">0</td>\n        <td style=\"border:1px solid green;\">3.1</td>\n        <td style=\"border:1px solid green;\">0</td>\n        <td>....</td>\n        <td style=\"border:1px solid green;\">0</td>\n        <td style=\"border:1px solid green;\">5.2</td>\n        <td style=\"border:1px solid green;\">0</td>\n        <td style=\"border:1px solid green;\">0</td>\n        <td style=\"border:1px solid green;\">0</td>\n        <td style=\"border:1px solid green;\">0</td>\n        <td>.. only 2% are <b>non-zero</b> values ..</td>\n    </tr>\n</table>\n\nWe will preset it in a format like that: <b>{'pos':4, 'val': 3.1}</b>\n<br>\nTo make the example simple, all values will be rounded to an average value. So the data will present only column indexes with non-zero (or any other threshold) values.\n","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19"}},{"cell_type":"markdown","source":"Here is a simple code to convert indexes of columns into the list:\n> train_y_df['index_lst'] = (train_y_df[train_y_df.columns[0:MAX_COLS]].gt(0) * np.arange(0,MAX_COLS)).values.tolist()","metadata":{}},{"cell_type":"code","source":"import os\nimport pandas as pd\nimport numpy as np\nimport datetime\nimport matplotlib.pyplot as plt\n\nDATA_DIR = \"/kaggle/input/open-problems-multimodal/\"\nFP_MULTIOME_TRAIN_TARGETS = os.path.join(DATA_DIR,\"train_multi_targets.h5\")\n\n## Find positions\ndef target_2_freq(train_y_df, max_cols = 1000):\n    \n    print(\"Columns to process: \", max_cols)\n    train_y_df['index_lst'] = (train_y_df[train_y_df.columns[0:max_cols]].gt(0) * np.arange(0,max_cols)).values.tolist()\n  \n    # Assign fake value to the zero elements \n    def g(row):  \n        res = np.where(np.array(row) < 1, 99999, row) \n        res.sort()\n        return (res)\n\n    train_y_df['index_lst'] = train_y_df['index_lst'].apply(g)\n    display(train_y_df['index_lst'])\n    \n    y_freq_df = pd.DataFrame(train_y_df['index_lst'].values.ravel().tolist())\n\n    FIXED_COLS = int(max_cols/50)\n    y_freq_df = y_freq_df.loc[:, 0:FIXED_COLS]\n\n    return y_freq_df\n\n\n## Read 500 test records\ntrain_y_df = pd.read_hdf(FP_MULTIOME_TRAIN_TARGETS, start=0, stop=500)\nprint(\"Original data (a lot of zeros)\")\ndisplay(train_y_df.head(5))\n\nprint(\"Finding datapoints across of first 2000 columns .. \")\nfreq_df = target_2_freq(train_y_df, 2000)\ndisplay(freq_df.head(5))\n","metadata":{"execution":{"iopub.status.busy":"2022-10-23T17:24:47.509714Z","iopub.execute_input":"2022-10-23T17:24:47.510109Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we have very compact data with data that reflects frequency of the original data.","metadata":{}}]}