![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/341256/9590/Slide4.PNG\")
i haven't tried the below. but here is one possibility here:
\n\n
![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/341256/9591/Slide5.PNG\")
i haven't tried the below. but here is one possibility here:
\n\n
pytorch kdtree and sampling can be found at:
\n\nhttps://github.com/erikwijmans/Pointnet2_PyTorch
\n\n\"PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space\" -Charles R. Qi Li Yi Hao Su Leonidas J. Guibas, NIPS 2017
", "rawMarkdown": "pytorch kdtree and sampling can be found at:\n\nhttps://github.com/erikwijmans/Pointnet2_PyTorch\n\n\"PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space\" -Charles R. Qi Li Yi Hao Su Leonidas J. Guibas, NIPS 2017", "votes": null }, { "id": "342966", "postDate": "06/14/2018 13:03:18", "content": "Hi Heng,\nI have been working with PointNet (a keras version) and have some indication that the model is capable of recognizing the track pattern out of several thousand points. (see the discussion item: (https://www.kaggle.com/c/trackml-particle-identification/discussion/58782).\nI had to split the input into transverse momentum bins in order not to make too big a change in the model. As I'm just a user of models with little insight on their actual inner works, I have only made changes at the very last FC layers to be able to identify at most 256 tracks in one event.
\n\nI would rather have the model processing all tracks together because I'm not sure how to do the learning piece-wise. I can always change the number of filters in the 1D convolutions and see if the model learns something or even fits in the 10GB\nof a GPU, but I should consult with experts as well.
\n\nI sent and email to the PointNet authors at Stanford yesterday and I'm still hoping for a reply. \nThe model increases the 1D conv. number of filters to 1024 at the 5th convolution and then drops to 256 (my change). I understand that with more filters the model can identify more features. Using the Hough transform the only feature of relevance appears to me as \"points aligned along a line\".
\n\nIf I expect some 3K tracks in the events, should I change the number of filters at the fifth convolution to something like 4096 ?
\n\nIt would be great I you could give me some hints. Thanks
", "rawMarkdown": "Hi Heng,\nI have been working with PointNet (a keras version) and have some indication that the model is capable of recognizing the track pattern out of several thousand points. (see the discussion item: (https://www.kaggle.com/c/trackml-particle-identification/discussion/58782).\nI had to split the input into transverse momentum bins in order not to make too big a change in the model. As I'm just a user of models with little insight on their actual inner works, I have only made changes at the very last FC layers to be able to identify at most 256 tracks in one event. \n\nI would rather have the model processing all tracks together because I'm not sure how to do the learning piece-wise. I can always change the number of filters in the 1D convolutions and see if the model learns something or even fits in the 10GB\nof a GPU, but I should consult with experts as well.\n\nI sent and email to the PointNet authors at Stanford yesterday and I'm still hoping for a reply. \nThe model increases the 1D conv. number of filters to 1024 at the 5th convolution and then drops to 256 (my change). I understand that with more filters the model can identify more features. Using the Hough transform the only feature of relevance appears to me as \"points aligned along a line\". \n\nIf I expect some 3K tracks in the events, should I change the number of filters at the fifth convolution to something like 4096 ?\n\nIt would be great I you could give me some hints. Thanks", "votes": null }, { "id": "343861", "postDate": "06/16/2018 11:26:22", "content": "I'm thinking of training multiple very simple LTSMs to do different tasks. For example, one is only to predict the next hit using the first 3-5 predicted hits that belong to the same track. The problem is granularity of the output class. If the output class (one-hot encoding) presents all possible coordinates either in the polar system or Cartesian system, the size is going to blow up, says, x*y*z, so we may need to grid the 3D-output box to predict the possible \"cube\" of the next hit, but the prediction error would be pretty high, but on the other hand it allows more grey area too. In this sense it almost looks like a CNN problem to me again, I tend to avoid CNN in this competition. This thought has lots of flaws, but I'm hoping for more discussions in deep learning architecture.
@Nicole Finnie
\n\ni am trying to write the code below. The idea is simple:
\n\ngiven an affinity matrix (pairwise similarity), generate track candidates. e.g. choosing the strong links sequentially, or spectral clustering (but i have to implement some automatic way to determine the no. of clusters)
for a given current track candidate, classify each point of the track as valid or invalid and also the probability of the link. This is used to update the affinity matrix.
The process is repeated.
\n\nclass GraphNet(nn.Module):\ndef __init__(self, dim):\n super(GraphNet, self).__init__()\n\n self.conv1 = ConvBlock(track_size= 2, dim=dim)\n self.conv2 = ConvBlock(track_size= 4, dim=dim)\n self.conv3 = ConvBlock(track_size= 8, dim=dim)\n self.conv4 = ConvBlock(track_size=16, dim=dim)\n\ndef forward(self, point, radius, track):\n num_point= len(point)\n affinity = torch.zeros((num_point,num_point)).float().cuda() \n x = point.permute(1,0)\n\n logit = self.conv1(x, radius, track)\n affinity = logit_to_affinity(logit, track, affinity)\n\n track = affinity_to_track(affinity)\n logit = self.conv2(x, radius, track)\n affinity = logit_to_affinity(logit, track, affinity)\n\n track = affinity_to_track(affinity)\n logit = self.conv3(x, radius, track)\n affinity = logit_to_affinity(logit, track, affinity)\n\n ...\n\n return affinity\nthis is how learning of affinity matrix will look like.
\n\nleft: ground truth.
\n\nright : results on a training sample (a cone slice) over training iterations.
\n\nfor one training sample, input is about 100K pairs (consider only pair where radius(hit1)>radius(hit2)). For one scale, if threshold at recall = 100%, precision is about 1%, giving false positive of about 6k pairs. This effectively reduce the search space to 6/100 of input for linking the pair to triplet, etc ....
\n\n![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/344262/9629/animated.gif\")
@Nicole Finnie
\n\nDo it the combinatorial kalman filter way. you can call it combinatorial-LSTM. With reference to my track extension code:
\n\ngiven a track
use kdtree to find \"nearest\" neighbours
consider only neighbours that are in the same direction of the given track
feed the valid neighbours to LSTM
you can also read this: sketch-rnn. They are using LSTM to predict the path in hand-writing
\n\nhttps://distill.pub/2016/handwriting/
\n\nhttp://blog.otoro.net/2015/12/12/handwriting-generation-demo-in-tensorflow/
\n\n![\"enter](\"https://distill.pub/2016/handwriting/assets/strokes/jpg/2-65-8.jpg\")
\"A Neural Representation of Sketch Drawings\"\nhttps://arxiv.org/pdf/1704.03477.pdf
", "rawMarkdown": "you can also read this: sketch-rnn. They are using LSTM to predict the path in hand-writing\n\n\n\nhttps://distill.pub/2016/handwriting/\n\nhttp://blog.otoro.net/2015/12/12/handwriting-generation-demo-in-tensorflow/\n\n ![enter image description here][1]\n\n\n\"A Neural Representation of Sketch Drawings\"\nhttps://arxiv.org/pdf/1704.03477.pdf\n\n\n [1]: https://distill.pub/2016/handwriting/assets/strokes/jpg/2-65-8.jpg", "votes": null }, { "id": "344339", "postDate": "06/17/2018 17:39:04", "content": "@Heng, Thanks for sharing. Is there a reason to find the nearest neighbours to feed the LSTM for training when you have the ground truth? My original idea was a bit different, feeding the LSTM with the ground truth, says, the first 5 hits and it should learn to predict the next one, but there are a lot more issues to deal with than I thought.
\n\npi / -pi if I train it with cylindrical coordinates. For prediction I may miss those hits if the LSTM can only predict (mapped) continuity. However, I think it's an area CERN would like us to explore, so I'm hoping people will move on to a DL supervised learning approach, including myself. I'm still struggling with the clustering.
", "rawMarkdown": "Heng, Thanks for sharing. Is there a reason to find the nearest neighbours to feed the LSTM for training when you have the ground truth? My original idea was a bit different, feeding the LSTM with the ground truth, says, the first 5 hits and it should learn to predict the next one, but there are a lot more issues to deal with than I thought. \n\n 1. Some hits are across `pi` / `-pi` if I train it with cylindrical coordinates. For prediction I may miss those hits if the LSTM can only predict (mapped) continuity. \n 2. Some hits (very common) are at almost the same coordinates, so with either LSTM/CNN or dbscan, we have to face the same track fitting problem when it comes to those ambiguous hits. And that makes me wonder if training a DL model is the way to go since it may not outperform a traditional clustering approach. \n\nHowever, I think it's an area CERN would like us to explore, so I'm hoping people will move on to a DL supervised learning approach, including myself. I'm still struggling with the clustering.", "votes": null }, { "id": "344396", "postDate": "06/17/2018 21:31:19", "content": "there are 2 ways. At inference:
\n\ninput 5 hits --> LSTM --> predict the 6th location (and uncertainty, etc)
input 5 hits + possible 6th candidates (e.g. from kNN) -->LSTM -->predict if each of the candidates are valid or not
\"Some hits are across pi / -pi if I train it with cylindrical coordinates\" ...
\n\nfor direction, either:
\n\npredict the change, e.g. \"delta angle\"
predict unit vector (cos and sin), rather than the angle itself
you can modify the idea from the paper. Learn a set of \"probing direction vectors\" to detect lines?
\n\n![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/346849/9661/FNPP.png\")
Another possibility
\n\n![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/346851/9662/direct1.png\")
ideally, this is what the network should be encapsulating. You can force the network to do this by enforcing supervisory signal (and corresponding loss) at each of the sub-blocks:
\n\n![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/346851/9663/direct2.png\")
turns out that things are \"easier\" than expected:
\n\ntransform to a,r,z coordinates.
predict tracks on \"selected layers of points\" (rather than cone slices)
if the layers are few, tracks are quite straight. There will always be some projection to map stright 3d line onto single 2d point.
you can do a knn on the 2d projection and do a line fit on the clusters found.
\n\n![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/347565/9685/Slide5.png\")
![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/347565/9683/Slide4.png\")
![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/347565/9684/Slide3.png\")
For 1, you'd still hit the pi discontinuity problem using a = arctan(y/x), and these are your suggestions a few days ago:
predict the change, e.g. \"delta angle\"\n\npredict unit vector (cos and sin), rather than the angle itself\nDid you still throw a to your model directly?
as shown below:
\n\n![\"enter](\"https://storage.googleapis.com/kaggle-forum-message-attachments/347715/9686/update.png\")
@Heng Would you please explain the following:
\n\n1- The purpose of KNN clustering in the first figure.\n2- What do you mean by track fitting and how do you do it?
\n\nThanks
", "rawMarkdown": "Heng Would you please explain the following:\n\n1- The purpose of KNN clustering in the first figure.\n2- What do you mean by track fitting and how do you do it?\n\nThanks", "votes": null } ], "comments": [ { "id": 341258, "author_name": "hengck23", "author_url": "", "post_date": "06/11/2018 10:10:22", "content": "pytorch kdtree and sampling can be found at:
\n\nhttps://github.com/erikwijmans/Pointnet2_PyTorch
\n\n\"PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space\" -Charles R. Qi Li Yi Hao Su Leonidas J. Guibas, NIPS 2017
", "votes": null, "replies": [] }, { "id": 342966, "author_name": "rdebbe", "author_url": "", "post_date": "06/14/2018 13:03:18", "content": "Hi Heng,\nI have been working with PointNet (a keras version) and have some indication that the model is capable of recognizing the track pattern out of several thousand points. (see the discussion item: (https://www.kaggle.com/c/trackml-particle-identification/discussion/58782).\nI had to split the input into transverse momentum bins in order not to make too big a change in the model. As I'm just a user of models with little insight on their actual inner works, I have only made changes at the very last FC layers to be able to identify at most 256 tracks in one event.
\n\nI would rather have the model processing all tracks together because I'm not sure how to do the learning piece-wise. I can always change the number of filters in the 1D convolutions and see if the model learns something or even fits in the 10GB\nof a GPU, but I should consult with experts as well.
\n\nI sent and email to the PointNet authors at Stanford yesterday and I'm still hoping for a reply. \nThe model increases the 1D conv. number of filters to 1024 at the 5th convolution and then drops to 256 (my change). I understand that with more filters the model can identify more features. Using the Hough transform the only feature of relevance appears to me as \"points aligned along a line\".
\n\nIf I expect some 3K tracks in the events, should I change the number of filters at the fifth convolution to something like 4096 ?
\n\nIt would be great I you could give me some hints. Thanks
", "votes": null, "replies": [] }, { "id": 343861, "author_name": "nicolefinnie", "author_url": "", "post_date": "06/16/2018 11:26:22", "content": "I'm thinking of training multiple very simple LTSMs to do different tasks. For example, one is only to predict the next hit using the first 3-5 predicted hits that belong to the same track. The problem is granularity of the output class. If the output class (one-hot encoding) presents all possible coordinates either in the polar system or Cartesian system, the size is going to blow up, says, x*y*z, so we may need to grid the 3D-output box to predict the possible \"cube\" of the next hit, but the prediction error would be pretty high, but on the other hand it allows more grey area too. In this sense it almost looks like a CNN problem to me again, I tend to avoid CNN in this competition. This thought has lots of flaws, but I'm hoping for more discussions in deep learning architecture.
@Nicole Finnie
\n\nDo it the combinatorial kalman filter way. you can call it combinatorial-LSTM. With reference to my track extension code:
\n\ngiven a track
use kdtree to find \"nearest\" neighbours
consider only neighbours that are in the same direction of the given track
feed the valid neighbours to LSTM
you can also read this: sketch-rnn. They are using LSTM to predict the path in hand-writing
\n\nhttps://distill.pub/2016/handwriting/
\n\nhttp://blog.otoro.net/2015/12/12/handwriting-generation-demo-in-tensorflow/
\n\n
\"A Neural Representation of Sketch Drawings\"\nhttps://arxiv.org/pdf/1704.03477.pdf
", "votes": null, "replies": [] }, { "id": 344339, "author_name": "nicolefinnie", "author_url": "", "post_date": "06/17/2018 17:39:04", "content": "@Heng, Thanks for sharing. Is there a reason to find the nearest neighbours to feed the LSTM for training when you have the ground truth? My original idea was a bit different, feeding the LSTM with the ground truth, says, the first 5 hits and it should learn to predict the next one, but there are a lot more issues to deal with than I thought.
\n\npi / -pi if I train it with cylindrical coordinates. For prediction I may miss those hits if the LSTM can only predict (mapped) continuity. However, I think it's an area CERN would like us to explore, so I'm hoping people will move on to a DL supervised learning approach, including myself. I'm still struggling with the clustering.
", "votes": null, "replies": [] }, { "id": 344396, "author_name": "hengck23", "author_url": "", "post_date": "06/17/2018 21:31:19", "content": "there are 2 ways. At inference:
\n\ninput 5 hits --> LSTM --> predict the 6th location (and uncertainty, etc)
input 5 hits + possible 6th candidates (e.g. from kNN) -->LSTM -->predict if each of the candidates are valid or not
\"Some hits are across pi / -pi if I train it with cylindrical coordinates\" ...
\n\nfor direction, either:
\n\npredict the change, e.g. \"delta angle\"
predict unit vector (cos and sin), rather than the angle itself
@Nicole Finnie
\n\ni am trying to write the code below. The idea is simple:
\n\ngiven an affinity matrix (pairwise similarity), generate track candidates. e.g. choosing the strong links sequentially, or spectral clustering (but i have to implement some automatic way to determine the no. of clusters)
for a given current track candidate, classify each point of the track as valid or invalid and also the probability of the link. This is used to update the affinity matrix.
The process is repeated.
\n\nclass GraphNet(nn.Module):\ndef __init__(self, dim):\n super(GraphNet, self).__init__()\n\n self.conv1 = ConvBlock(track_size= 2, dim=dim)\n self.conv2 = ConvBlock(track_size= 4, dim=dim)\n self.conv3 = ConvBlock(track_size= 8, dim=dim)\n self.conv4 = ConvBlock(track_size=16, dim=dim)\n\ndef forward(self, point, radius, track):\n num_point= len(point)\n affinity = torch.zeros((num_point,num_point)).float().cuda() \n x = point.permute(1,0)\n\n logit = self.conv1(x, radius, track)\n affinity = logit_to_affinity(logit, track, affinity)\n\n track = affinity_to_track(affinity)\n logit = self.conv2(x, radius, track)\n affinity = logit_to_affinity(logit, track, affinity)\n\n track = affinity_to_track(affinity)\n logit = self.conv3(x, radius, track)\n affinity = logit_to_affinity(logit, track, affinity)\n\n ...\n\n return affinity\nthis is how learning of affinity matrix will look like.
\n\nleft: ground truth.
\n\nright : results on a training sample (a cone slice) over training iterations.
\n\nfor one training sample, input is about 100K pairs (consider only pair where radius(hit1)>radius(hit2)). For one scale, if threshold at recall = 100%, precision is about 1%, giving false positive of about 6k pairs. This effectively reduce the search space to 6/100 of input for linking the pair to triplet, etc ....
\n\n
you can modify the idea from the paper. Learn a set of \"probing direction vectors\" to detect lines?
\n\n
Another possibility
\n\n
ideally, this is what the network should be encapsulating. You can force the network to do this by enforcing supervisory signal (and corresponding loss) at each of the sub-blocks:
\n\n
@Heng Would you please explain the following:
\n\n1- The purpose of KNN clustering in the first figure.\n2- What do you mean by track fitting and how do you do it?
\n\nThanks
", "votes": null, "replies": [] } ] }, { "id": 347565, "author_name": "hengck23", "author_url": "", "post_date": "06/24/2018 17:46:21", "content": "turns out that things are \"easier\" than expected:
\n\ntransform to a,r,z coordinates.
predict tracks on \"selected layers of points\" (rather than cone slices)
if the layers are few, tracks are quite straight. There will always be some projection to map stright 3d line onto single 2d point.
you can do a knn on the 2d projection and do a line fit on the clusters found.
\n\n
For 1, you'd still hit the pi discontinuity problem using a = arctan(y/x), and these are your suggestions a few days ago:
predict the change, e.g. \"delta angle\"\n\npredict unit vector (cos and sin), rather than the angle itself\nDid you still throw a to your model directly?
as shown below:
\n\n