Like many others I started with the dbscan solutions published in the public kernels and later extended with many excellent suggestions from @heng, @grzegorz, etc.
\n\nI eventually ended up with an ensemble of 3 models reaching a score of just over 0.66. Each model, with 10 z-shifts took several hours to run per event.
\n\nThen came Yuval and CPMP on the forums.
\n\nI realized that I was using poor, unreliable features - and I needed to go back to basics to understand the mathematics of a helix!
\n\nI essentially started over and re-designed my features using the basic mathematics of a helix as my design principle
\n\nA helix can be described by the following parameters:
\n\n R - radius of curvature (radius of the circle projected on the x-y plane)\n xc,yc - Center of the projected circle on the x-y plane\n (These two can also be represented as R,theta0 in polar coordinates)\n\n Pitch - a measure of how much z changes when the helix makes a full 360 degree rotation\n\n Direction of rotation (i.e. does the helix turn to the right (clockwise) or to the left (counter-clockwise))\n\n A single point the helix passes through - which, we assume is (0,0,z0) (for nearly all the helices we care about)\nTo reconstruct the tracks (helices), I scanned over values of R and z0 and clustered based on derived values of theta0 and Pitch (expressed as an angle)
\n\nI used the Hough Equation for finding theta0, given a value for R, which lets me then find the location of the center of the helix in the x-y plane
\n\n The Hough equation is: r = 2R*cos(phi - theta0), where:\n r is a hit's distance from x,y = 0,0\n phi is arctan2(y,x) for a hit\n R = radius of curvature of helix\n theta0 = angle of inclination such that R,theta0 gives the center of the helix when projected on the x-y plane\nAfter finding the helix center I shifted the x,y values of the hits so that the helix center is at 0,0 (new coord's for a hit are x'=x-xc, y'=y-yc,z)
\n\nThen I calculated what should be a constant based on the pitch of the helix (length of the arc of the helix/delta-z)
\n\nThe length of the arc is R*angle of rotation from (0,0,z0) to current hit (x',y',z). You need to do some geometry to insure you get the right value for the angle of rotation. I know I didn't handle values >180 correctly...
\n\nFinally, use theta0 and (R*angle of rotation/delta-z) as features for dbscan.
\n\nWith this approach I exceeded the score for my previous 3 models with a few minutes of processing (vs many hours) and eventually reached a score just under 0.7 (0.695) with a few hours of processing time per event.
\n\nSee attached code (warning, it's ugly - but it worked)
\n\nCheers! I hope you all enjoyed the competition as much as I did!
", "messages": [ { "id": "370294", "postDate": "08/14/2018 15:22:58", "content": "Like many others I started with the dbscan solutions published in the public kernels and later extended with many excellent suggestions from @heng, @grzegorz, etc.
\n\nI eventually ended up with an ensemble of 3 models reaching a score of just over 0.66. Each model, with 10 z-shifts took several hours to run per event.
\n\nThen came Yuval and CPMP on the forums.
\n\nI realized that I was using poor, unreliable features - and I needed to go back to basics to understand the mathematics of a helix!
\n\nI essentially started over and re-designed my features using the basic mathematics of a helix as my design principle
\n\nA helix can be described by the following parameters:
\n\n R - radius of curvature (radius of the circle projected on the x-y plane)\n xc,yc - Center of the projected circle on the x-y plane\n (These two can also be represented as R,theta0 in polar coordinates)\n\n Pitch - a measure of how much z changes when the helix makes a full 360 degree rotation\n\n Direction of rotation (i.e. does the helix turn to the right (clockwise) or to the left (counter-clockwise))\n\n A single point the helix passes through - which, we assume is (0,0,z0) (for nearly all the helices we care about)\nTo reconstruct the tracks (helices), I scanned over values of R and z0 and clustered based on derived values of theta0 and Pitch (expressed as an angle)
\n\nI used the Hough Equation for finding theta0, given a value for R, which lets me then find the location of the center of the helix in the x-y plane
\n\n The Hough equation is: r = 2R*cos(phi - theta0), where:\n r is a hit's distance from x,y = 0,0\n phi is arctan2(y,x) for a hit\n R = radius of curvature of helix\n theta0 = angle of inclination such that R,theta0 gives the center of the helix when projected on the x-y plane\nAfter finding the helix center I shifted the x,y values of the hits so that the helix center is at 0,0 (new coord's for a hit are x'=x-xc, y'=y-yc,z)
\n\nThen I calculated what should be a constant based on the pitch of the helix (length of the arc of the helix/delta-z)
\n\nThe length of the arc is R*angle of rotation from (0,0,z0) to current hit (x',y',z). You need to do some geometry to insure you get the right value for the angle of rotation. I know I didn't handle values >180 correctly...
\n\nFinally, use theta0 and (R*angle of rotation/delta-z) as features for dbscan.
\n\nWith this approach I exceeded the score for my previous 3 models with a few minutes of processing (vs many hours) and eventually reached a score just under 0.7 (0.695) with a few hours of processing time per event.
\n\nSee attached code (warning, it's ugly - but it worked)
\n\nCheers! I hope you all enjoyed the competition as much as I did!
", "rawMarkdown": "Like many others I started with the dbscan solutions published in the public kernels and later extended with many excellent suggestions from @heng, @grzegorz, etc.\n\nI eventually ended up with an ensemble of 3 models reaching a score of just over 0.66. Each model, with 10 z-shifts took several hours to run per event.\n\nThen came Yuval and CPMP on the forums.\n\nI realized that I was using poor, unreliable features - and I needed to go back to basics to understand the mathematics of a helix!\n\n I essentially started over and re-designed my features using the basic mathematics of a helix as my design principle\n\n A helix can be described by the following parameters:\n\n R - radius of curvature (radius of the circle projected on the x-y plane)\n xc,yc - Center of the projected circle on the x-y plane\n (These two can also be represented as R,theta0 in polar coordinates)\n\n Pitch - a measure of how much z changes when the helix makes a full 360 degree rotation\n\n Direction of rotation (i.e. does the helix turn to the right (clockwise) or to the left (counter-clockwise))\n\n A single point the helix passes through - which, we assume is (0,0,z0) (for nearly all the helices we care about)\n\nTo reconstruct the tracks (helices), I scanned over values of R and z0 and clustered based on derived values of theta0 and Pitch (expressed as an angle)\n\nI used the Hough Equation for finding theta0, given a value for R, which lets me then find the location of the center of the helix in the x-y plane\n\n The Hough equation is: r = 2R*cos(phi - theta0), where:\n r is a hit's distance from x,y = 0,0\n phi is arctan2(y,x) for a hit\n R = radius of curvature of helix\n theta0 = angle of inclination such that R,theta0 gives the center of the helix when projected on the x-y plane\n\nAfter finding the helix center I shifted the x,y values of the hits so that the helix center is at 0,0 (new coord's for a hit are x'=x-xc, y'=y-yc,z)\n\nThen I calculated what should be a constant based on the pitch of the helix (length of the arc of the helix/delta-z)\n\nThe length of the arc is R*angle of rotation from (0,0,z0) to current hit (x',y',z). You need to do some geometry to insure you get the right value for the angle of rotation. I know I didn't handle values >180 correctly...\n\nFinally, use theta0 and (R*angle of rotation/delta-z) as features for dbscan.\n\nWith this approach I exceeded the score for my previous 3 models with a few minutes of processing (vs many hours) and eventually reached a score just under 0.7 (0.695) with a few hours of processing time per event.\n\nSee attached code (warning, it's ugly - but it worked)\n\nCheers! I hope you all enjoyed the competition as much as I did!", "votes": null }, { "id": "370305", "postDate": "08/14/2018 15:41:11", "content": "Thanks for sharing, and congrats on the result. I went same way as you, tuning the helix unrolling kernels and could not pass 0.7. Then I restarted from scratch with a parameters describing perfect helix.
", "rawMarkdown": "Thanks for sharing, and congrats on the result. I went same way as you, tuning the helix unrolling kernels and could not pass 0.7. Then I restarted from scratch with a parameters describing perfect helix.", "votes": null }, { "id": "370319", "postDate": "08/14/2018 15:58:46", "content": "Thanks! I read your post describing your solution, and found a lot of good ideas I need to spend time digesting to fully understand.
\n\nI especially liked your method to choose \"random\" values for r0 and z0 based on observed values from training files. I did similar EDA and threw away the real data for an imperfect mathematical representation!
\n\nI look forward to competing with you again in the future 😀
", "rawMarkdown": "Thanks! I read your post describing your solution, and found a lot of good ideas I need to spend time digesting to fully understand.\n\nI especially liked your method to choose \"random\" values for r0 and z0 based on observed values from training files. I did similar EDA and threw away the real data for an imperfect mathematical representation!\n\nI look forward to competing with you again in the future 😀", "votes": null }, { "id": "370321", "postDate": "08/14/2018 16:05:27", "content": "\n\n\nI especially liked your method to choose \"random\" values for r0 and z0 based on observed values from training files.
\n
Thanks. This may have a hidden benefit if the simulator does not generate tracks randomly ;) Not sure it is the case though...
", "rawMarkdown": "> I especially liked your method to choose \"random\" values for r0 and z0 based on observed values from training files.\n\nThanks. This may have a hidden benefit if the simulator does not generate tracks randomly ;) Not sure it is the case though...", "votes": null } ], "comments": [ { "id": 370305, "author_name": "cpmpml", "author_url": "", "post_date": "08/14/2018 15:41:11", "content": "Thanks for sharing, and congrats on the result. I went same way as you, tuning the helix unrolling kernels and could not pass 0.7. Then I restarted from scratch with a parameters describing perfect helix.
", "votes": null, "replies": [ { "id": 370319, "author_name": "johnhsweeney", "author_url": "", "post_date": "08/14/2018 15:58:46", "content": "Thanks! I read your post describing your solution, and found a lot of good ideas I need to spend time digesting to fully understand.
\n\nI especially liked your method to choose \"random\" values for r0 and z0 based on observed values from training files. I did similar EDA and threw away the real data for an imperfect mathematical representation!
\n\nI look forward to competing with you again in the future 😀
", "votes": null, "replies": [] }, { "id": 370321, "author_name": "cpmpml", "author_url": "", "post_date": "08/14/2018 16:05:27", "content": "\n\n\nI especially liked your method to choose \"random\" values for r0 and z0 based on observed values from training files.
\n
Thanks. This may have a hidden benefit if the simulator does not generate tracks randomly ;) Not sure it is the case though...
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "370294": "Like many others I started with the dbscan solutions published in the public kernels and later extended with many excellent suggestions from @heng, @grzegorz, etc.\n\nI eventually ended up with an ensemble of 3 models reaching a score of just over 0.66. Each model, with 10 z-shifts took several hours to run per event.\n\nThen came Yuval and CPMP on the forums.\n\nI realized that I was using poor, unreliable features - and I needed to go back to basics to understand the mathematics of a helix!\n\n I essentially started over and re-designed my features using the basic mathematics of a helix as my design principle\n\n A helix can be described by the following parameters:\n\n R - radius of curvature (radius of the circle projected on the x-y plane)\n xc,yc - Center of the projected circle on the x-y plane\n (These two can also be represented as R,theta0 in polar coordinates)\n\n Pitch - a measure of how much z changes when the helix makes a full 360 degree rotation\n\n Direction of rotation (i.e. does the helix turn to the right (clockwise) or to the left (counter-clockwise))\n\n A single point the helix passes through - which, we assume is (0,0,z0) (for nearly all the helices we care about)\n\nTo reconstruct the tracks (helices), I scanned over values of R and z0 and clustered based on derived values of theta0 and Pitch (expressed as an angle)\n\nI used the Hough Equation for finding theta0, given a value for R, which lets me then find the location of the center of the helix in the x-y plane\n\n The Hough equation is: r = 2R*cos(phi - theta0), where:\n r is a hit's distance from x,y = 0,0\n phi is arctan2(y,x) for a hit\n R = radius of curvature of helix\n theta0 = angle of inclination such that R,theta0 gives the center of the helix when projected on the x-y plane\n\nAfter finding the helix center I shifted the x,y values of the hits so that the helix center is at 0,0 (new coord's for a hit are x'=x-xc, y'=y-yc,z)\n\nThen I calculated what should be a constant based on the pitch of the helix (length of the arc of the helix/delta-z)\n\nThe length of the arc is R*angle of rotation from (0,0,z0) to current hit (x',y',z). You need to do some geometry to insure you get the right value for the angle of rotation. I know I didn't handle values >180 correctly...\n\nFinally, use theta0 and (R*angle of rotation/delta-z) as features for dbscan.\n\nWith this approach I exceeded the score for my previous 3 models with a few minutes of processing (vs many hours) and eventually reached a score just under 0.7 (0.695) with a few hours of processing time per event.\n\nSee attached code (warning, it's ugly - but it worked)\n\nCheers! I hope you all enjoyed the competition as much as I did!", "370305": "Thanks for sharing, and congrats on the result. I went same way as you, tuning the helix unrolling kernels and could not pass 0.7. Then I restarted from scratch with a parameters describing perfect helix.", "370319": "Thanks! I read your post describing your solution, and found a lot of good ideas I need to spend time digesting to fully understand.\n\nI especially liked your method to choose \"random\" values for r0 and z0 based on observed values from training files. I did similar EDA and threw away the real data for an imperfect mathematical representation!\n\nI look forward to competing with you again in the future 😀", "370321": "> I especially liked your method to choose \"random\" values for r0 and z0 based on observed values from training files.\n\nThanks. This may have a hidden benefit if the simulator does not generate tracks randomly ;) Not sure it is the case though..." }, "source": "meta" }