{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"**Track ML Challenge – Data Insights via Graph Analytics**\n\nTo explore what our universe is made of, scientists at CERN are colliding protons, essentially recreating mini big bangs, and meticulously observing these collisions with intricate silicon detectors.\nWhile orchestrating the collisions and observations is already a massive scientific accomplishment, analysing the enormous amounts of data produced from the experiments is becoming an overwhelming challenge.\nEvent rates have already reached hundreds of millions of collisions per second, meaning physicists must sift through tens of petabytes of data per year. \nAnd, as the resolution of detectors improve, ever better software is needed for real-time pre-processing and filtering of the most promising events, producing even more data.\n\nIn this post/paper, it has been researched and studied, if Graph Analytics or Graph Databases can help the physics scientists working at CERN to discover and characterize new particles?\n\n**Quick Problem Description**\n\n*Link every track to one hit.*"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"![image.png](https://datafreakankur.com/wp-content/uploads/2019/02/image-18.png)\n"},{"metadata":{"_uuid":"d5dbf57c41ec9811702f5f8c7f3998f86e4a4bbc"},"cell_type":"markdown","source":"Every particle leaves a track behind it, like a car leaving tire marks in the sand. We did not catch the particle in action. Now we want to link every track (tire mark) to one hit that the particle created.\nIn every event, a large number of particles are released. They move along a path leaving behind their tracks. They eventually hit a particle detector surface on the other end.\nIn the training data we have the following information on each event:\n•\tHits: x,y,zx,y,z coordinates of each hit on the particle detector\n•\tParticles: Each particle's initial position (vx,vy,vzvx,vy,vz), momentum (px,py,pzpx,py,pz), charge (qq) and number of hits\n•\tTruth: Mapping between hits and generating particles; the particle's trajectory, momentum and the hit weight\n•\tCells: Precise location of where each particle hit the detector and how much energy it deposited\n\nThe complete Dataset and all explanations related to the same can be directly loaded/seen from the Kaggle website - [https://www.kaggle.com/c/trackml-particle-identification/data]\n"},{"metadata":{"_uuid":"550bb2c40bf519ffc392f412189cf00d3276eb73"},"cell_type":"markdown","source":"**What is Graph Analytics**\n\n“A picture speaks a thousand words” is one of the most commonly used phrases. But a graph speaks so much more than that. \n"},{"metadata":{"_uuid":"846b7de7b5a5d143f2baabc1168016fe35f9d903"},"cell_type":"markdown","source":"![image.png](https://datafreakankur.com/wp-content/uploads/2019/02/image-19.png)"},{"metadata":{"_uuid":"0baaf4644fa09a7e8eba4efdd344b42ecab47f1c"},"cell_type":"markdown","source":"A visual representation of data, in the form of graphs, helps us gain actionable insights and make better data driven decisions based on them.\nThe science or the branch to analyse Graphs to make better data driven decisions is termed as Graph Analytics.\n\n**Graph Analytics and ML Challenge**\n\nThe studies & research on over the data and end result that is required for this ML challenge shows infers that - this will be the perfect Used Case for Graph Analytics and that the Hit ID’s can be easily linked to the Track ID and that the users could have a visual representation of the same.\nTo begin with – Data Exploration has been started and a rough image of the output Graph have been framed in the mind and also have been drawn on a piece of paper. The initial rough images define various relations and looks as follows:\n"},{"metadata":{"_uuid":"e28d75344cbc436aa307bf6aa76bd506a173b5c7"},"cell_type":"markdown","source":"![image.png](https://datafreakankur.com/wp-content/uploads/2019/02/image-20.png)"},{"metadata":{"_uuid":"2eb79eb1d416ed24f59cc54e30cc915902210839"},"cell_type":"markdown","source":"![image.png](https://datafreakankur.com/wp-content/uploads/2019/02/image-21.png)"},{"metadata":{"_uuid":"0a5c8ccb4e3abc0cd9dfbf36468565ddd54d6e87"},"cell_type":"markdown","source":"**Tool Selection**\n\nOnce the images are drawn and the data is explored, the 2nd important task is the Tool Selection. There are wide variety of Tools and all have various capabilities. Based on the expertise following tools have been listed:\n•\tGephi\n•\tNeo4J\n•\tIGraph (R Package)\n•\tGraphX with GraphStream and BreezeViz(Apache Spark)\n•\tNetworkX (Python)\nOut of these tools as mentioned above, GEPHI is the most sort out choice for the initial draft to have a look, if our purpose will be solved via Graph Analytics.\n\nAlso, for the initial Draft, Data selection has been done. The reason is that the Data is too large.\nTwo events i.e. Event event000001000 and event000001000 are utilized for this task.\n\nThe next important task is to create Nodes and Edges in the Gephi Format that requires csv files to be imported directly onto Gephi. \nFor this Data creation – SQL server Management Studio is utilized to retrieve the results in the desired format based on various Queries:\n"},{"metadata":{"_uuid":"433a54638735784d0c51d961bb263d1cde3f2de7"},"cell_type":"markdown","source":"![image.png](https://datafreakankur.com/wp-content/uploads/2019/02/image-22.png)"},{"metadata":{"_uuid":"b24cde336abacd9dc06b332a8afe93edcdd42ad2"},"cell_type":"markdown","source":"Using all sort of Queries and logics – following two CSV files are created as per the Gephi’s Format:\n•\tNodes.csv\n•\tEdges.csv\nThese files are then imported to Gephi and looks like as follows:\n"},{"metadata":{"_uuid":"042a68b152215cd99e0b047389a4e2ce93ac84d8"},"cell_type":"markdown","source":"**NODES**\n![image.png](https://datafreakankur.com/wp-content/uploads/2019/02/image-23.png)"},{"metadata":{"_uuid":"08a55b7e1b39775cea7ba301b21b0204cb99cc34"},"cell_type":"markdown","source":"**EDGES**\n![image.png](https://datafreakankur.com/wp-content/uploads/2019/02/image-24.png)"},{"metadata":{"_uuid":"43c16c03b2cb33e31c87880f6a7b8a74806c1f46"},"cell_type":"markdown","source":"Then the coordinate columns are re-casted to Graph Coordinates and nodes are partitioned to colours on the basis of particle_id.\n\n**The initial Graph looks like as follows:**\n"},{"metadata":{"_uuid":"8c23984377d7c2adcaf90b1cafb4425a5571b442"},"cell_type":"markdown","source":"![image.png](https://datafreakankur.com/wp-content/uploads/2019/02/image-25.png)"},{"metadata":{"_uuid":"10c410fb5805f8c4b5bb2875a90adfa6a3e045e8"},"cell_type":"markdown","source":"**The 2nd Graph shows the edges Labels and Tracks are clearly visible:**"},{"metadata":{"_uuid":"d7b25f9d4ee1a5abbdcf00652bb0d766ddd21d05"},"cell_type":"markdown","source":"![image.png](https://datafreakankur.com/wp-content/uploads/2019/02/image-26.png)"},{"metadata":{"_uuid":"62c7688b78135a8aedba685df87939c70928116c"},"cell_type":"markdown","source":"**A close look at the Graph with All Tracks defined:**"},{"metadata":{"_uuid":"ac156380963650e4f392d007fe6dc3da92c4a659"},"cell_type":"markdown","source":"![image.png](https://datafreakankur.com/wp-content/uploads/2019/02/image-27.png)"},{"metadata":{"_uuid":"44fe0c3e0ec69efc456c1b0c180fd1b45a988fee"},"cell_type":"markdown","source":"**Hit ID’s with Tracks**"},{"metadata":{"_uuid":"a4e279c4341edf7b83305e9d28245fe249d5e320"},"cell_type":"markdown","source":"![image.png](https://datafreakankur.com/wp-content/uploads/2019/02/image-28.png)"},{"metadata":{"_uuid":"1b8fedf293f18c79de4f2f0a4dc6b850cc6433fc"},"cell_type":"markdown","source":"**Does this solve our Purpose?**\n\nThe Track ML challenge requires a file to be submitted based on the Event Hit ID’s with their Specific Tracks defined.\nThe same role is achieved using Graph Analytics. We can easily link an event Hit ID with a Track and can also visualise the same.\nAlso, for the Test Data – these Tracks can be analysed based on the close proximity and connections.\n\n**Challenges faced:**\n•\tVery Large amount of Data\n•\tData cleaning and Data Processing.\n\n**Future Action Plan**\n•\tImplementing more Events and enhance Data capabilities\n•\tUsing R iGraph package for Graph Analytics\n\n**Readings & Data Understanding**\n•\t[https://www.kaggle.com/c/trackml-particle-identification]\n\n•\t[https://www.kaggle.com/wesamelshamy/trackml-problem-explanation-and-data-exploration/comments#323803]\n\n•\t[https://www.kaggle.com/makahana/quick-trajectory-plot]\n\n•\t[https://www.kaggle.com/jbonatt/trackml-eda-etc]\n"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}