{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Let's learn GNN with \"IceCube - Neutrinos in Deep Ice\" competition data\n\nAfter [EDA](https://www.kaggle.com/code/hechtjp/icecube-quick-eda-with-polars), I checked several notebooks which were shared by other participants and realized there was not much approach using Gradient Boosting algorizm in this competition. On the other hand, there are several noteboks approaching with Graph Neural Network(GNN) and showing better score on public LB.\n\nI do not have an experience to use GNN in the past. So first I try to learn basic of that in this notebook and think about how to apply it to this competition dataset.\n\nThis notebook is still under working. If you have any advice or sudgestion to this notebook, please feel free to leave it in comment field:)","metadata":{}},{"cell_type":"markdown","source":"# Import Library & set configuration\n\ntorch_geometric is commonly use for GNN and it should be installed in notebook additionally. First I tried it via internet but somehow it did not work. So I refer this notebook [Make your own GNN by kilogrand](https://www.kaggle.com/code/kilogrand/make-your-own-gnn) how to install it from input datase.","metadata":{}},{"cell_type":"code","source":"# Install packages\nimport subprocess\n\nwhls = [\n    \"/kaggle/input/pytorchgeometric/torch_cluster-1.6.0-cp37-cp37m-linux_x86_64.whl\",\n    \"/kaggle/input/pytorchgeometric/torch_scatter-2.1.0-cp37-cp37m-linux_x86_64.whl\",\n    \"/kaggle/input/pytorchgeometric/torch_sparse-0.6.16-cp37-cp37m-linux_x86_64.whl\",\n    \"/kaggle/input/pytorchgeometric/torch_spline_conv-1.2.1-cp37-cp37m-linux_x86_64.whl\",\n    \"/kaggle/input/pytorchgeometric/torch_geometric-2.2.0-py3-none-any.whl\",\n    \"/kaggle/input/pytorchgeometric/ruamel.yaml-0.17.21-py3-none-any.whl\",\n]\nfor w in whls:\n    print(\"Installing\", w)\n    subprocess.call([\"pip\", \"install\", w, \"--no-deps\", \"--upgrade\"])","metadata":{"_kg_hide-input":false,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-04-02T02:54:36.305692Z","iopub.execute_input":"2023-04-02T02:54:36.306253Z","iopub.status.idle":"2023-04-02T02:54:45.197156Z","shell.execute_reply.started":"2023-04-02T02:54:36.306212Z","shell.execute_reply":"2023-04-02T02:54:45.195702Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# !pip install polars\n!pip install -q ../input/polars01516/typing_extensions-4.4.0-py3-none-any.whl\n!pip install -q ../input/polars01516/polars-0.15.16-cp37-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl","metadata":{"_kg_hide-input":false,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2023-04-02T02:54:45.199417Z","iopub.execute_input":"2023-04-02T02:54:45.199818Z","iopub.status.idle":"2023-04-02T02:55:05.868375Z","shell.execute_reply.started":"2023-04-02T02:54:45.199780Z","shell.execute_reply":"2023-04-02T02:55:05.867073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# General\nimport sys, warnings, time, os, copy, gc, re, random, json\nwarnings.filterwarnings('ignore')\nimport pickle as pkl\nfrom IPython.display import display\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nimport polars as pl\nimport seaborn as sns\nsns.set()\nfrom pprint import pprint\nfrom pathlib import Path\nfrom tqdm import tqdm\ntqdm.pandas()\nfrom datetime import datetime, timedelta\nfrom scipy import sparse\nfrom multiprocessing import Pool\n\n# GNN\nimport torch\nfrom torch_geometric.data import Data, DataLoader, Batch\nimport torch.nn.functional as F \nfrom torch_geometric.nn import GCNConv\nfrom torch_geometric.datasets import KarateClub\nfrom torch_geometric.utils import to_networkx\nimport networkx as nx\nfrom sklearn.metrics import mean_squared_error\n","metadata":{"execution":{"iopub.status.busy":"2023-04-02T02:55:05.870064Z","iopub.execute_input":"2023-04-02T02:55:05.871183Z","iopub.status.idle":"2023-04-02T02:55:05.882263Z","shell.execute_reply.started":"2023-04-02T02:55:05.871149Z","shell.execute_reply":"2023-04-02T02:55:05.881198Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Configuration\nDEBUG = True\nPATH_INPUT = Path(\"/kaggle/input/icecube-neutrinos-in-deep-ice\")","metadata":{"execution":{"iopub.status.busy":"2023-04-02T02:55:05.885229Z","iopub.execute_input":"2023-04-02T02:55:05.886352Z","iopub.status.idle":"2023-04-02T02:55:05.893275Z","shell.execute_reply.started":"2023-04-02T02:55:05.886308Z","shell.execute_reply":"2023-04-02T02:55:05.892216Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Simple GNN with toy data\n\nBefore touching actual competition dataset, I do the same excercise with this blog post [PyTorch GeometricでGraph Neural Network（GNN）入門](https://cpp-learning.com/pytorch-geometric/) to understand of basic of GNN with torch_geometric. Let's use KarateClub dataset, which is builtin dataset in torch_geometric. ","metadata":{}},{"cell_type":"code","source":"# DL. KarateClub dataset\ndataset = KarateClub()\n \nprint(\"num of graph:\", len(dataset)) \nprint(\"num of classes:\",dataset.num_classes) \n \ndata = dataset[0]","metadata":{"execution":{"iopub.status.busy":"2023-04-02T02:55:05.895089Z","iopub.execute_input":"2023-04-02T02:55:05.895650Z","iopub.status.idle":"2023-04-02T02:55:05.907915Z","shell.execute_reply.started":"2023-04-02T02:55:05.895607Z","shell.execute_reply":"2023-04-02T02:55:05.906851Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def check_graph(data):\n    '''Display detail of graph'''\n    print(\"shape of graph:\", data)\n    print(\"Key of graph: \", data.keys)\n    print(\"Num of nodes:\", data.num_nodes)\n    print(\"Num of edges:\", data.num_edges)\n    print(\"Num of node's features:\", data.num_node_features)\n    print(\"Contain isolated nodes:\", data.contains_isolated_nodes())\n    print(\"Contain self loops:\", data.contains_self_loops())\n    print(\"====== Feature of node:x ======\")\n    print(data['x'])\n    print(\"====== Class of node:y ======\")\n    print(data['y'])\n    print(\"========= Shape of edges =========\")\n    print(data['edge_index'])\n    \ncheck_graph(data)","metadata":{"execution":{"iopub.status.busy":"2023-04-02T02:55:05.911245Z","iopub.execute_input":"2023-04-02T02:55:05.911555Z","iopub.status.idle":"2023-04-02T02:55:05.925719Z","shell.execute_reply.started":"2023-04-02T02:55:05.911528Z","shell.execute_reply":"2023-04-02T02:55:05.924476Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def show_graph(data, title, label=True):\n    # Convert data to graph of networkx\n    nxg = to_networkx(data)\n\n    # Pagerank of graph network\n    pr = nx.pagerank(nxg)\n    pr_max = np.array(list(pr.values())).max()\n\n    # Position of nodes\n    draw_pos = nx.spring_layout(nxg, seed=0) \n\n    # Colors of nodes\n    if label:\n        cmap = plt.get_cmap('tab10')\n        labels = data.y.numpy()\n        colors = [cmap(l) for l in labels]\n        # Draw the nodes of network\n        plt.figure(figsize=(10, 10))\n        nx.draw_networkx_nodes(nxg, \n                               draw_pos,\n                               node_size=[v / pr_max * 1000 for v in pr.values()],\n                               node_color=colors,\n                               alpha=0.5)\n        nx.draw_networkx_edges(nxg, draw_pos, arrowstyle='-', alpha=0.2)\n        nx.draw_networkx_labels(nxg, draw_pos, font_size=10)\n\n        plt.title(title)\n        plt.show()\n        \n    else:\n        cmap = plt.get_cmap('tab10')\n        # Draw the nodes of network\n        plt.figure(figsize=(10, 10))\n        nx.draw_networkx_nodes(nxg, \n                               draw_pos,\n                               node_size=[v / pr_max * 1000 for v in pr.values()],\n                               alpha=0.5)\n        nx.draw_networkx_edges(nxg, draw_pos, arrowstyle='-', alpha=0.2)\n        nx.draw_networkx_labels(nxg, draw_pos, font_size=10)\n\n        plt.title(title)\n        plt.show()","metadata":{"execution":{"iopub.status.busy":"2023-04-02T02:55:05.927472Z","iopub.execute_input":"2023-04-02T02:55:05.928160Z","iopub.status.idle":"2023-04-02T02:55:05.941278Z","shell.execute_reply.started":"2023-04-02T02:55:05.928120Z","shell.execute_reply":"2023-04-02T02:55:05.940250Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"show_graph(data, \"Karate club\", label=True)","metadata":{"execution":{"iopub.status.busy":"2023-04-02T02:55:05.943170Z","iopub.execute_input":"2023-04-02T02:55:05.943904Z","iopub.status.idle":"2023-04-02T02:55:07.256556Z","shell.execute_reply.started":"2023-04-02T02:55:05.943793Z","shell.execute_reply":"2023-04-02T02:55:07.255662Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Define model\nclass Net(torch.nn.Module):\n    def __init__(self):\n        super(Net, self).__init__()\n        hidden_size = 5\n        self.conv1 = GCNConv(dataset.num_node_features, hidden_size)\n        self.conv2 = GCNConv(hidden_size, dataset.num_classes)\n    \n    def forward(self, data):\n        x, edge_index = data.x, data.edge_index\n        x = self.conv1(x, edge_index)\n        x = F.relu(x)\n        x = self.conv2(x, edge_index)\n        \n        return F.log_softmax(x, dim=1)","metadata":{"execution":{"iopub.status.busy":"2023-04-02T02:55:07.258244Z","iopub.execute_input":"2023-04-02T02:55:07.259180Z","iopub.status.idle":"2023-04-02T02:55:07.266858Z","shell.execute_reply.started":"2023-04-02T02:55:07.259140Z","shell.execute_reply":"2023-04-02T02:55:07.265748Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Set device\ndevice = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\n \n# Instance of model\nmodel = Net()\nprint(model)\n \n# Set model for training\nmodel.train()","metadata":{"execution":{"iopub.status.busy":"2023-04-02T02:55:07.271220Z","iopub.execute_input":"2023-04-02T02:55:07.271481Z","iopub.status.idle":"2023-04-02T02:55:07.285250Z","shell.execute_reply.started":"2023-04-02T02:55:07.271457Z","shell.execute_reply":"2023-04-02T02:55:07.283858Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Optimizer\noptimizer = torch.optim.Adam(model.parameters(), lr=0.01)\n \n# Training loop\nfor epoch in range(100):\n    optimizer.zero_grad()\n    out = model(data)\n    loss = F.nll_loss(out, data.y)\n    loss.backward()\n    optimizer.step()\n    if epoch % 20 == 0:\n        print('Epoch %d | Loss: %.4f' % (epoch, loss.item()))","metadata":{"execution":{"iopub.status.busy":"2023-04-02T02:55:07.287059Z","iopub.execute_input":"2023-04-02T02:55:07.287469Z","iopub.status.idle":"2023-04-02T02:55:07.608291Z","shell.execute_reply.started":"2023-04-02T02:55:07.287431Z","shell.execute_reply":"2023-04-02T02:55:07.604777Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Set model for evaluation\nmodel.eval()\n \n# Prediction\n_, pred = model(data).max(dim=1)\n \nprint(\"Prediction：\", pred)\nprint(\"True：\", data[\"y\"])\nprint(\"Accuracy:\", sum(pred == data[\"y\"])/len(pred))","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2023-04-02T02:55:07.609918Z","iopub.execute_input":"2023-04-02T02:55:07.610548Z","iopub.status.idle":"2023-04-02T02:55:07.622306Z","shell.execute_reply.started":"2023-04-02T02:55:07.610508Z","shell.execute_reply":"2023-04-02T02:55:07.621229Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Load competition datasets\n\nNext, let's look into this competition dataset. First, load the dataset with Polars which is faster library than Pandas.","metadata":{}},{"cell_type":"code","source":"# Train meta data \ndfTrainMeta = pl.read_parquet(PATH_INPUT / \"train_meta.parquet\")\ndfTrainMeta = dfTrainMeta.filter(pl.col(\"batch_id\") == 1)\n\nif DEBUG:\n    dfTrainMeta = dfTrainMeta.sample(frac=0.1, seed=0)  # sampling for faster debuging \n\n# Sensor position data\ndfSensorGeo = pl.read_csv(PATH_INPUT / \"sensor_geometry.csv\")\n\n# Normalize position of each sensor\ndfSensorGeo = dfSensorGeo.with_columns((pl.col(\"x\") / pl.col(\"x\").max()).alias(\"x\"))\ndfSensorGeo = dfSensorGeo.with_columns((pl.col(\"y\") / pl.col(\"y\").max()).alias(\"y\"))\ndfSensorGeo = dfSensorGeo.with_columns(((pl.col(\"z\") - pl.col(\"z\").min()) / (pl.col(\"z\").max() - pl.col(\"z\").min())).alias(\"z\"))\n\n# Clip all values in the range btw. -1 & 1.\nx = dfSensorGeo.apply(lambda t: (np.clip(t[1], -1, 1), np.clip(t[2], -1, 1), np.clip(t[3], -1, 1)))\nx.columns = [\"x\", \"y\", \"z\"]\ndfSensorGeo = pl.concat([dfSensorGeo.select(\"sensor_id\"), x], how=\"horizontal\")\n\n# Train batch 1 data\ndfTrainBatch = pl.read_parquet(PATH_INPUT / \"train/batch_1.parquet\")\ndfTrainBatch = dfTrainBatch.with_column(pl.col(\"sensor_id\").cast(pl.Int64))\n\n# Merge all DF\ndfTrain = dfTrainMeta.join(dfTrainBatch, on=\"event_id\", how=\"inner\")\ndfTrain = dfTrain.join(dfSensorGeo, on=\"sensor_id\", how=\"inner\")\n\n# Add next sensor_id column\nnextSensor = pl.col(\"sensor_id\").shift(-1).over(\"event_id\").alias(\"next_sensor\")\ndfTrain = dfTrain.with_columns(nextSensor)\n\ndfTrain","metadata":{"_kg_hide-output":false,"execution":{"iopub.status.busy":"2023-04-02T02:55:07.624319Z","iopub.execute_input":"2023-04-02T02:55:07.625036Z","iopub.status.idle":"2023-04-02T02:55:30.324074Z","shell.execute_reply.started":"2023-04-02T02:55:07.625002Z","shell.execute_reply":"2023-04-02T02:55:30.322707Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Create graph samples of few events\n\nChoosing some events and drawing their graphs.","metadata":{}},{"cell_type":"code","source":"listEvent = list(set(dfTrain.get_column(\"event_id\").to_list()))\n\ndfSample1 = dfTrain.filter(pl.col(\"event_id\")==listEvent[0])\ndfSample2 = dfTrain.filter(pl.col(\"event_id\")==listEvent[1])\ndfSample3 = dfTrain.filter(pl.col(\"event_id\")==listEvent[2])","metadata":{"execution":{"iopub.status.busy":"2023-04-02T02:55:30.326488Z","iopub.execute_input":"2023-04-02T02:55:30.326986Z","iopub.status.idle":"2023-04-02T02:55:30.632035Z","shell.execute_reply.started":"2023-04-02T02:55:30.326945Z","shell.execute_reply":"2023-04-02T02:55:30.630940Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"n = 0\nfor df in [dfSample1, dfSample2, dfSample3]:\n    \n    # Create dictionary {sensor ID:n}\n    l = list(set(df.get_column(\"sensor_id\").to_list()))\n    dictSensors = {}\n    for k, v in enumerate(l):\n        dictSensors[v] = k\n    \n    # Add columns about new sensor ID & next new sensor ID\n    newSensorID = df.apply(lambda t:dictSensors[t[6]])\n    newSensorID.columns = [\"newSensorID\"]\n    df = pl.concat([df, newSensorID], how=\"horizontal\")\n    nextSensor = pl.col(\"newSensorID\").shift(-1).over(\"event_id\").alias(\"next_newSensorID\")\n    df = df.with_columns(nextSensor)\n    \n    # Prepare edge_index: new sensor ID -> next new sensor ID.\n    edgeID = [df.get_column(\"newSensorID\").to_list()[:-1], df.get_column(\"next_newSensorID\").to_list()[:-1]]\n    edge_index = torch.tensor(edgeID, dtype=torch.long)\n    \n    # Prepare x: [[x0,y0,z0], [x1,y1,z1],...]\n    dfNodeF = df.select([\"x\", \"y\", \"z\", \"newSensorID\"]).sort(\"newSensorID\").unique().select([\"x\", \"y\", \"z\"])\n    nodeID = [dfNodeF.get_column(\"x\").to_list(), dfNodeF.get_column(\"y\").to_list(), dfNodeF.get_column(\"z\").to_list()]\n    nodeID = np.array(nodeID).T\n    x = torch.tensor(nodeID, dtype=torch.float)\n    \n    # Prepare y: [azimuth, zenith]\n    y = [df.get_column(\"azimuth\").to_list()[0], df.get_column(\"zenith\").to_list()[0]]\n    \n    # Prepare data\n    data = Data(x=x, edge_index=edge_index, y=y)\n    \n    # Show the data\n    print(\"+\"*100)\n    show_graph(data, f'Event ID: {listEvent[n]}', label=False)\n    n += 1\n    check_graph(data)","metadata":{"execution":{"iopub.status.busy":"2023-04-02T02:55:30.633775Z","iopub.execute_input":"2023-04-02T02:55:30.634137Z","iopub.status.idle":"2023-04-02T02:55:43.591889Z","shell.execute_reply.started":"2023-04-02T02:55:30.634101Z","shell.execute_reply":"2023-04-02T02:55:43.590773Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Prepare input data for training\n\nPrepare 1,000 graphs for training of model using pytorch Batch and DataLoader.","metadata":{}},{"cell_type":"code","source":"# create list of sensor ID\nlistSenID = dfSensorGeo.get_column(\"sensor_id\").to_list()\nnodeID = np.array(dfSensorGeo.select([\"x\", \"y\" ,\"z\"])).T\n\nbatch_input = []\nfor eventID in listEvent[:1000]:\n    \n    df = dfTrain.filter(pl.col(\"event_id\")==eventID)\n    \n    # Prepare edge_index: sensor ID -> next sensor ID.\n    edgeID = [df.get_column(\"sensor_id\").to_list()[:-1], df.get_column(\"next_sensor\").to_list()[:-1]]\n    edge_index = torch.tensor(edgeID, dtype=torch.long)\n    \n    # Create list of sensor ID in scope\n    l = list(set(df.get_column(\"sensor_id\").to_list()))\n    \n    # Prepare x: the position of sensor in [[x0,y0,z0], [x1,y1,z1],...]\n    x = torch.tensor(nodeID, dtype=torch.float)\n    x[list(set(listSenID) - set(l))] = 0   # Mask on sensor not in scope.\n    \n    # Prepare y: [azimuth, zenith]\n    y = torch.tensor([df.get_column(\"azimuth\").to_list()[0], df.get_column(\"zenith\").to_list()[0]])\n    \n    # Prepare data\n    data = Data(x=x, edge_index=edge_index, y=y)\n    \n    # Append data to the list of batch\n    batch_input.append(data)\n    \nbatch = Batch.from_data_list(batch_input)","metadata":{"execution":{"iopub.status.busy":"2023-04-02T02:55:43.593953Z","iopub.execute_input":"2023-04-02T02:55:43.594406Z","iopub.status.idle":"2023-04-02T02:55:49.931074Z","shell.execute_reply.started":"2023-04-02T02:55:43.594365Z","shell.execute_reply":"2023-04-02T02:55:49.930032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# To show the detail of batch\n# batch.to_data_list()","metadata":{"execution":{"iopub.status.busy":"2023-04-02T02:55:49.932765Z","iopub.execute_input":"2023-04-02T02:55:49.933132Z","iopub.status.idle":"2023-04-02T02:55:49.938124Z","shell.execute_reply.started":"2023-04-02T02:55:49.933093Z","shell.execute_reply":"2023-04-02T02:55:49.936770Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# create DataLoader obj. from batch obj. in case of edge_index have different shape\nloader = DataLoader(batch, batch_size=1, shuffle=True)","metadata":{"execution":{"iopub.status.busy":"2023-04-02T02:55:49.939944Z","iopub.execute_input":"2023-04-02T02:55:49.940596Z","iopub.status.idle":"2023-04-02T02:55:49.948901Z","shell.execute_reply.started":"2023-04-02T02:55:49.940555Z","shell.execute_reply":"2023-04-02T02:55:49.947720Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Define GNN model\n\nUsing GCNConv from pytorch geometric and define simple three layers network.\nThe output of GCNConv has shape [number of nodes, 2] and take the mean of that along the first dimension to much to the target.\nI am not sure is this way OK or not. If someone has any advice, I appreciate your sharing that with me.","metadata":{}},{"cell_type":"code","source":"# Define the GCN model\nclass GCN(torch.nn.Module):\n    def __init__(self):\n        super(GCN, self).__init__()\n        self.conv1 = GCNConv(3, 16)\n        self.conv2 = GCNConv(16, 32)\n        self.conv3 = GCNConv(32, 2)\n\n    def forward(self, x, edge_index):\n        x = F.relu(self.conv1(x, edge_index))\n        x = F.relu(self.conv2(x, edge_index))\n        x = self.conv3(x, edge_index)\n        x = x.mean(dim=0, keepdim=True)  # take the mean along the first dimension\n        x = x.reshape(1, -1)  # reshape to [1, 2]\n        return x\n\n# Define the model, optimizer and loss function\ndevice = torch.device('cuda' if torch.cuda.is_available() else 'cpu')\nmodel = GCN().to(device)\noptimizer = torch.optim.Adam(model.parameters(), lr=0.01)\ncriterion = torch.nn.MSELoss()","metadata":{"execution":{"iopub.status.busy":"2023-04-02T02:55:49.950882Z","iopub.execute_input":"2023-04-02T02:55:49.951264Z","iopub.status.idle":"2023-04-02T02:55:49.969408Z","shell.execute_reply.started":"2023-04-02T02:55:49.951223Z","shell.execute_reply":"2023-04-02T02:55:49.967839Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Train the model\n\nLet's train the model with prepared 1,000 graphs.","metadata":{}},{"cell_type":"code","source":"# Train the model\nmodel.train()\nfor epoch in range(100):\n    running_loss = 0\n    for data in loader:\n        data = data.to(device)\n        optimizer.zero_grad()\n        out = model(data.x.float(), data.edge_index)\n        loss = criterion(out, data.y.float())\n        loss.backward()\n        optimizer.step()\n        running_loss += loss.item()\n    if (epoch+1) % 20 == 0:\n        print(\"Epoch {}: loss={}\".format(epoch+1, running_loss))","metadata":{"execution":{"iopub.status.busy":"2023-04-02T02:55:49.970685Z","iopub.execute_input":"2023-04-02T02:55:49.971326Z","iopub.status.idle":"2023-04-02T03:04:46.906057Z","shell.execute_reply.started":"2023-04-02T02:55:49.971290Z","shell.execute_reply":"2023-04-02T03:04:46.904936Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Validation\n\nPreparing 100 graphs additionaly and predict [azimuth, zenith] using the trained model. The results are validated by calculating Mean Squad Error and Angular Distance Score.","metadata":{}},{"cell_type":"code","source":"listY = []\naz_true = []\nzen_true = []\nbatch_input = []\nfor eventID in listEvent[-100:]:\n    \n    df = dfTrain.filter(pl.col(\"event_id\")==eventID)\n    \n    # Prepare edge_index: sensor ID -> next sensor ID.\n    edgeID = [df.get_column(\"sensor_id\").to_list()[:-1], df.get_column(\"next_sensor\").to_list()[:-1]]\n    edge_index = torch.tensor(edgeID, dtype=torch.long)\n    \n    # Create list of sensor ID in scope\n    l = list(set(df.get_column(\"sensor_id\").to_list()))\n    \n    # Prepare x: the position of sensor in [[x0,y0,z0], [x1,y1,z1],...]\n    x = torch.tensor(nodeID, dtype=torch.float)\n    x[list(set(listSenID) - set(l))] = 0   # Mask on sensor not in scope.\n    \n    # Prepare y: [azimuth, zenith]\n    az_true.append(df.get_column(\"azimuth\").to_list()[0])\n    zen_true.append(df.get_column(\"zenith\").to_list()[0])\n    listY.append([df.get_column(\"azimuth\").to_list()[0], df.get_column(\"zenith\").to_list()[0]])\n    \n    # Prepare data\n    data = Data(x=x, edge_index=edge_index)\n    \n    # Append data to the list of batch\n    batch_input.append(data)\n    \nbatch = Batch.from_data_list(batch_input)\nloader = DataLoader(batch, batch_size=1, shuffle=True)","metadata":{"execution":{"iopub.status.busy":"2023-04-02T03:04:46.907682Z","iopub.execute_input":"2023-04-02T03:04:46.908080Z","iopub.status.idle":"2023-04-02T03:04:47.537017Z","shell.execute_reply.started":"2023-04-02T03:04:46.908041Z","shell.execute_reply":"2023-04-02T03:04:47.535993Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"model.eval()\nwith torch.no_grad():\n    y_pred = []\n    for data in loader:\n        data = data.to(device)\n        out = model(data.x.float(), data.edge_index)\n        y_pred.append(out.cpu().numpy())\n    y_pred = np.concatenate(y_pred, axis=0)\n    print(\"Predicted values: \", y_pred)\n    print(y_pred.shape)\n    \naz_pred = y_pred[:, 0].tolist()\nzen_pred = y_pred[:, 1].tolist()","metadata":{"execution":{"iopub.status.busy":"2023-04-02T03:04:47.538697Z","iopub.execute_input":"2023-04-02T03:04:47.539123Z","iopub.status.idle":"2023-04-02T03:04:47.902051Z","shell.execute_reply.started":"2023-04-02T03:04:47.539076Z","shell.execute_reply":"2023-04-02T03:04:47.900768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Compute the mean squared error.\nmse = mean_squared_error(listY, y_pred)\nprint(\"MSE: \", mse)","metadata":{"execution":{"iopub.status.busy":"2023-04-02T03:04:47.903954Z","iopub.execute_input":"2023-04-02T03:04:47.904356Z","iopub.status.idle":"2023-04-02T03:04:47.911834Z","shell.execute_reply.started":"2023-04-02T03:04:47.904315Z","shell.execute_reply":"2023-04-02T03:04:47.910611Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# import function to culcurate mean angular error from utility scripts.\nfrom copy_mean_angular_error import angular_dist_score\n\nscoreAllMean = angular_dist_score(az_true, zen_true, az_pred, zen_pred)\nprint(f\"Angular dist score: {scoreAllMean}\")","metadata":{"execution":{"iopub.status.busy":"2023-04-02T03:04:47.913367Z","iopub.execute_input":"2023-04-02T03:04:47.914691Z","iopub.status.idle":"2023-04-02T03:04:47.922608Z","shell.execute_reply.started":"2023-04-02T03:04:47.914652Z","shell.execute_reply":"2023-04-02T03:04:47.921459Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Conclusion\n\n- pytorch geometric is easy to use to define GNN. Through the exercise with toy data, could understand how to approach classification task of each nodes.\n- The task in this competition is different from that exercise since this is not classification task, this is regression task for each graph, not for each nodes. It is still question for me what kind of model architecture could be appropriate in this task.\n- The validation result shows that the model could not learn from data correctly. Further investigation is necessary!","metadata":{}},{"cell_type":"markdown","source":"# Reference:\n\n[Dataset description](https://www.kaggle.com/competitions/icecube-neutrinos-in-deep-ice/data)\n\n[Mean Angular Error](https://www.kaggle.com/code/sohier/mean-angular-error)\n\n[pandasから移行する人向け polars使用ガイド](https://qiita.com/nkay/items/9cfb2776156dc7e054c8)\n","metadata":{"execution":{"iopub.status.busy":"2023-02-11T12:20:36.191262Z","iopub.execute_input":"2023-02-11T12:20:36.191719Z","iopub.status.idle":"2023-02-11T12:20:36.200643Z","shell.execute_reply.started":"2023-02-11T12:20:36.191686Z","shell.execute_reply":"2023-02-11T12:20:36.198825Z"}}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}