{"cells":[{"metadata":{"trusted":true,"collapsed":true,"_uuid":"c6baea89c12af6c9a567bb27df9564ccaa1b4a35"},"cell_type":"code","source":"import pandas as pd\nimport featuretools as ft\nimport os","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ffc253b682f7616d8c96159a804c5c2895ea3117"},"cell_type":"markdown","source":"#### Exploring train_sample  files"},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"14e80385e24d6c784807d5e3f7f05d8ae22fc144"},"cell_type":"code","source":"# choosing event event000002387\ndf_particle = pd.read_csv(\"../input/train_1/event000002387-particles.csv\") ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"d63f6606127f357882e0aae7a79f4e75756c4c9a"},"cell_type":"code","source":"df_hits = pd.read_csv(\"../input/train_1/event000002387-hits.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"f4964da1e541c2abbf44020811b8847a09ad2c1a"},"cell_type":"code","source":"df_cells = pd.read_csv(\"../input/train_1/event000002387-cells.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"725176af1700396877777667cb9c9f75c3f4e49b"},"cell_type":"code","source":"df_truth = pd.read_csv(\"../input/train_1/event000002387-truth.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4ed00f6a1e469423aa2c5029c21ee749c75b70f3","collapsed":true},"cell_type":"code","source":"df_particle.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c34173569a4a695c38fbbaa20a9cd659cd6eaf4b","collapsed":true},"cell_type":"code","source":"df_hits.info()","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"57585fa01fbfcd4c9085914f7cbca9341d9ab3d6","collapsed":true},"cell_type":"code","source":"df_particle.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c59fd014bee9791a4d38e60ee8186078430ea397","collapsed":true},"cell_type":"code","source":"df_truth.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"661e465a6d909e4911649ed0bcbe7f946e6853c1","collapsed":true},"cell_type":"code","source":"df_cells.info()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f4112ccaa5423662db5bd6c02bf96b903d46e345"},"cell_type":"markdown","source":"Hits\n* hit_id: numerical identifier of the hit inside the event.\n* x, y, z: measured x, y, z position (in millimeter) of the hit in global coordinates.\n* volume_id: numerical identifier of the detector group.\n* layer_id: numerical identifier of the detector layer inside the group.\n* module_id: numerical identifier of the detector module inside the layer."},{"metadata":{"trusted":true,"_uuid":"b3da0b8a83ea4af5b8ac56ca4ce7b84fca66923e","collapsed":true},"cell_type":"code","source":"df_hits.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"b321e0726cbb622113f6a5ea7a0df38b05c8312b","collapsed":true},"cell_type":"code","source":"df_hits.tail(2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0a3afb1b8d1719dce497d296c1768f49659a31bd"},"cell_type":"markdown","source":"Particles-\n* particle_id: numerical identifier of the particle inside the event.\n* vx, vy, vz: initial position or vertex (in millimeters) in global coordinates.\n* px, py, pz: initial momentum (in GeV/c) along each global axis.\n* q: particle charge (as multiple of the absolute electron charge).\n* nhits: number of hits generated by this particle."},{"metadata":{"trusted":true,"_uuid":"d18f4f5d62771027a962df23550157ba571a601c","collapsed":true},"cell_type":"code","source":"df_particle.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"faa3793a00098b6a38983c4a3c3ac3b7b6fff61e","collapsed":true},"cell_type":"code","source":"df_particle.tail(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f66231bd7c640ab454a660709744d1149c675765","collapsed":true},"cell_type":"code","source":"df_particle.describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f901725d43dee347e4c6e902e11f0c0adf9f4aa1"},"cell_type":"markdown","source":"Truth-\nThe truth file contains the mapping between hits and generating particles and the true particle state at each measured hit. Each entry maps one hit to one particle.\n\n* hit_id: numerical identifier of the hit as defined in the hits file.\n* particle_id: numerical identifier of the generating particle as defined in the particles file. A value of 0 means that the hit did not originate from a reconstructible particle, but e.g. from detector noise.\n* tx, ty, tz true intersection point in global coordinates (in millimeters) between the particle trajectory and the sensitive surface.\n* tpx, tpy, tpz true particle momentum (in GeV/c) in the global coordinate system at the intersection point. The corresponding vector is tangent to the particle trajectory at the intersection point.\n* weight per-hit weight used for the scoring metric; total sum of weights within one event equals to one.\n\n(Note:- Multiple hits can belong to the same particle (at different cooridinates) . That is how we will get a track of a single particle with multiple hits (sites of sensing) along its path of travel."},{"metadata":{"trusted":true,"_uuid":"10f19c4d9cb3880acb2704fde23bfc0f29537dda","collapsed":true},"cell_type":"code","source":"df_truth.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8ca5d680795044d9e6ba19f6713a7ce2ae1ae107","collapsed":true},"cell_type":"code","source":"df_truth.tail(2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ef9b68b2388208b64d13cf005ecfb47449aeda34"},"cell_type":"markdown","source":"Cells\n\nThe cells file contains the constituent active detector cells that comprise each hit. The cells can be used to refine the hit to track association. A cell is the smallest granularity inside each detector module, much like a pixel on a screen, except that depending on the volume_id a cell can be a square or a long rectangle. It is identified by two channel identifiers that are unique within each detector module and encode the position, much like column/row numbers of a matrix. A cell can provide signal information that the detector module has recorded in addition to the position. Depending on the detector type only one of the channel identifiers is valid, e.g. for the strip detectors, and the value might have different resolution.\n\n* hit_id: numerical identifier of the hit as defined in the hits file.\n* ch0, ch1: channel identifier/coordinates unique within one module.\n* value: signal value information, e.g. how much charge a particle has deposited."},{"metadata":{"trusted":true,"_uuid":"02079e31ce6693f83dff4c42e219f5616ecdf4db","collapsed":true},"cell_type":"code","source":"df_cells.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"22992ed6fed348f6fc7d3c3d8f20ec8dfe40bd45","collapsed":true},"cell_type":"code","source":"df_cells.tail()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d9ed9ea2e35399d15164d64d8dd24b70724a7c14"},"cell_type":"markdown","source":"> At index 669470 and 669471 we have different cell coordinates for the same hit_id 121511"},{"metadata":{"_uuid":"2de10f907df2d2413466493f55d93961cc99dda1"},"cell_type":"markdown","source":"#### Exploring detectors file"},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"106224cd4dbf3afcd30bc092f9774c5cb2d7e6c6"},"cell_type":"code","source":"# This file contains additional detector geometry information.\n\ndf_detectors = pd.read_csv(\"../input/detectors.csv\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5fdbb76f6808c4db5b803010219d1dc52ad16059","collapsed":true},"cell_type":"code","source":"# Each module has a different position and orientation described in the detectors file.\n\ndf_detectors","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"93f4b96eb3cdef31e15e3e2576631d5e00afd756"},"cell_type":"markdown","source":"####  Linking df_hits, df_truth and df_particle for the same hit_id"},{"metadata":{"trusted":true,"_uuid":"9031cd6ce8477608d8be23ac626cf54f44596bf8","collapsed":true},"cell_type":"code","source":"print(df_hits.shape)\ndf_hits.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"094f2508fbd0f97e84397465d4c1d49cb78e8a0e","collapsed":true},"cell_type":"code","source":"print(df_particle.shape)\ndf_particle.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"61cd9ffa5de7d91d718f4ae386da92b3cf49f98f","collapsed":true},"cell_type":"code","source":"print(df_truth.shape)\ndf_truth.head(2)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"981795c30285688fc786993aa35eaf949c4d3f3c"},"cell_type":"markdown","source":"** tx, ty, tz ** : true intersection point in global coordinates (in millimeters) between the particle trajectory and the sensitive surface (df_truth).\n \n** x, y, z **: measured x, y, z position (in millimeter) of the hit in global coordinates (df_hits). \n\n--The above observations from df_hits and df_truth for the same hit_ids are quite close to each other but not identical."},{"metadata":{"_uuid":"dce4fb43ac8f5138ea13aa44855763424d6a4a43"},"cell_type":"markdown","source":"Mapping each row in df_hits containing hit information with its corresponding particle information provided as each row in df_truth"},{"metadata":{"trusted":true,"_uuid":"93e93660175e30d753a8b882ac96039218abced9","collapsed":true},"cell_type":"code","source":"df_hits.index","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"50388744276b2c15e85f7da450c03d5b72a54e46"},"cell_type":"code","source":"hits_truth = df_hits.set_index('hit_id').join(df_truth.set_index('hit_id'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d75e8ee5f6d2853170afb9112a9f37a71c45573e","collapsed":true},"cell_type":"code","source":"df_hits.head(1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"cf7172f29df784e80ea80131e2275d5c5b228fe9","collapsed":true},"cell_type":"code","source":"df_truth.head(1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"527e422750b058ef6f44b928b64b31f16b6df6ad","collapsed":true},"cell_type":"code","source":"hits_truth.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"6d62c3d9c7bd2636f3c9d459509fdf48e07158fa"},"cell_type":"code","source":"hits_truth.reset_index(inplace=True)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4b4d957d7205b6f66e91e6f6bfec2f3f887d3b16","collapsed":true},"cell_type":"code","source":"hits_truth.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1c6fcd3ef600846d3412adc695aecbdc7b58293d","collapsed":true},"cell_type":"code","source":"hits_truth.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"3e3ec9d84dbdfe2bc24557a03ed8ea3229d52c1e","collapsed":true},"cell_type":"code","source":"df_particle.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"506b581ccf66eda9ec7fea7c51c8887fa0e7a6df","collapsed":true},"cell_type":"code","source":"df_particle.head(2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"daa48f3070c0adb39651fe53031ea7621b65bddf"},"cell_type":"code","source":"## Creating Entity set\n\nes = ft.EntitySet(id=\"trackml\")","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"eb475a6d31ab5dc57b00302bd048573ce828f195"},"cell_type":"code","source":"es1 = es.entity_from_dataframe(entity_id='hits_truth', dataframe=hits_truth,\n                               index = 'hit_id',\n                               variable_types = { \"volume_id\":ft.variable_types.Categorical,\n                                                  \"layer_id\":ft.variable_types.Categorical,\n                                                  \"module_id\":ft.variable_types.Categorical })","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"834da43b6763285d7cf818403a691a6b5ba8ee17"},"cell_type":"code","source":"es2 = es1.entity_from_dataframe(entity_id='particle', dataframe=df_particle,\n                               index = 'particle_id' )","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"37eb0078c72be3303552a711a1688d8a65132d4e","collapsed":true},"cell_type":"code","source":"es2","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"collapsed":true,"_uuid":"d3cfc9164cbd8c996ccb4a5c0df5911b2fcb3deb"},"cell_type":"code","source":"# Defining one-to-many relationships among features of different entities\n\nrelation1 = ft.Relationship(es2['particle']['particle_id'],es2['hits_truth']['particle_id'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ff9200033a83b7e3dcb7fa7de9eb171737589b4a","collapsed":true},"cell_type":"code","source":"relation1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"c00bacc9e8b0975988d7483d95212b9cc2574217","collapsed":true},"cell_type":"code","source":"es2.add_relationships([relation1])","execution_count":null,"outputs":[]},{"metadata":{"scrolled":true,"trusted":true,"_uuid":"8c285d316ecfa757396d27f31b2ccb67c8bf32f8","collapsed":true},"cell_type":"code","source":"es2.entities","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a812e501c2c92202495830d2a184c2728104afb1"},"cell_type":"markdown","source":"Now, we want collective information for each particle_id. "},{"metadata":{"_uuid":"85020a243f000601c9641e0887642d8e596f168d"},"cell_type":"markdown","source":"This unsupervised learning approach will involve merging data of hits,cells as primary input features which will then be mapped with corresponding particle information with truth data to include as something which is similar to target just to guide the ML algo about the kind of input features to be clustered as belongingg to the same particle (specifying the particle_id is not in the scope of this problem )."},{"metadata":{"trusted":true,"_uuid":"d1eb35e78e79f60afcb3e4104ad7410986ed99e6","collapsed":true},"cell_type":"code","source":"df_hits.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ed747d71fb3298a334b8ed41df84c390b45e0cc0","collapsed":true},"cell_type":"code","source":"df_cells.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false,"collapsed":true,"_uuid":"e17142df815848c427cfe4f79d2c23f91f8a38d5"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}