{ "id": 215308, "title": "Sensor data dose not have line feed at the end of row sometimes.", "url": "/competitions/indoor-location-navigation/discussion/215308", "author_name": "", "post_date": "2021-01-29T11:36:10.664331Z", "votes": 6, "comment_count": 8, "views": 0, "content": "

\"/kaggle/input/indoor-location-navigation/train/5cd56b90e2acfd2d33b5e33f/F1/5d0868bdbb84450008f569ca.txt\"'s line 483 dose not have line feed at appropriate position.

\n
\n

sample_filepath3 = \"/kaggle/input/indoor-location-navigation/train/5cd56b90e2acfd2d33b5e33f/F1/5d0868bdbb84450008f569ca.txt\"
\n with open(sample_filepath3, 'r') as f:
\n sample_content3 = f.readlines()
\n f.close()
\n sample_content3[482:485] # show line 482, 483, 484, please watch 483 carefully!

\n
\n

Executing python code above on notebook will show following result, then we can know that line 483 have line feed (='\\n') at inappropriate position thus many row is contained in this line.

\n
\n

['1560830842488\\tTYPE_ACCELEROMETER_UNCALIBRATED\\t0.6177063\\t-0.3902588\\t9.904816\\n',
\n '1560830841553\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\tae38c9dac6a05831fe3016a1ed0519fb7f74feea\\t-59\\t-86\\t16.545591294123085\\t833d5531741cba213e7030266fb6d9e2ed4c2aca1560830841566\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\ta77ff5d7252bab87e1eed09b2f29d47622ab9bd0\\t-59\\t-92\\t27.752738449875483\\t6c4ab94d11c2dfb10b9dd8fe35b59ab49cc2c5d61560830841579\\tTYPE_BEACON\\t71f98f1f7b22bfa3f5995790c82431aed3cad143\\t9471a0c7c3b3b94bedcd2ee23186934e67b282ea\\te49643fa7a08253cc4675b041e302c7a4a6e134c\\t-50\\t-82\\t40.89270029241089\\t132b161e5815182e2e0cd96d7a172d76fb69f4381560830841610\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\t630afdd1beda6cf46d6453132ef1855488358042\\t-59\\t-92\\t27.752738449875483\\t815f779cc7325751c44667652830581560a2b3ec1560830841665\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\tb6c7e1a702b4ccfd0f53c4df113fbecc9ed369bc\\t-59\\t-72\\t4.287841349532929\\tad21473af0bb67eddfd6929334a1f428b62b60221560830841682\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\t6af2d433b42bf0b9c73b68bc8189d7dd140c94b6\\t-59\\t-91\\t25.519130565680634\\tce78c6d9e1b67fc3e6aafee45295f8d2e5f7181d1560830841685\\tTYPE_BEACON\\t07efd69e3167537492f0ead89fb2779633b04949\\tb6589fc6ab0dc82cf12099d1c2d40ab994e8410c\\tcd71309513115f935c21612cf52b35fef0ae3fd5\\t-57\\t-88\\t25.70856433929632\\tff6021e77bedf72f51037f108bb9ff1d9853d34e1560830841853\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\tcfc84f0752adc96b489f71195d91a946c5f6d3e8\\t8159618423dfa22f1ca0b62543e2f18eef630ce8\\t-65\\t-82\\t5.50634293288929\\tc1bfecc6701c2cf7c1c1313d057eecf3eb22cdbe1560830841899\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\tdd3ce4a6d6663cfbfd5d87c9de86ea469c4d011f\\t-59\\t-87\\t18.07763630020772\\tc0316935986dc35c1d11b71d6a38febf97ae4f111560830842006\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\t7b37b9ee3e5868308085f4b419c9715b57f6a288\\t-59\\t-78\\t7.852880042745979\\td83732095e320584637fe11dee2a5bfc9e1f833e1560830842013\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\tc8fe745b9bc106416c224e561fac9502b1337ce9\\t-59\\t-93\\t30.155320780411284\\t8e1d0991e5eb1527fa4b8f31ee0f9c3e08f565561560830842025\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\ta5b56798ede30bb10ad5e125321f2e5c396e6a94\\t-59\\t-81\\t10.467388920465797\\t16b617161a8a2d4c73193fcbdda686ecdbea21061560830842047\\tTYPE_BEACON\\t07efd69e3167537492f0ead89fb2779633b04949\\tb6589fc6ab0dc82cf12099d1c2d40ab994e8410c\\t54f6d21e62c7f28225452ca22145a2101ec5e023\\t-57\\t-85\\t19.702364739804867\\t508b247d74755a72cc06ac2a68c761efe933ce5e1560830842446\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\te031005e7f6f7c2d5f91eb742b12f6a4b2e00434\\t-59\\t-87\\t18.07763630020772\\t3573eb6d74efbc27bbe52ff20b411e6bdbb5afe01560830842509\\tTYPE_ACCELEROMETER\\t0.38067627\\t-0.21069336\\t9.275131\\n',
\n '1560830842509\\tTYPE_MAGNETIC_FIELD\\t23.039999\\t-9.42\\t-38.399998\\n']

\n
\n

I found such inappropriate line feed position at many lines in the same file. Probably it is just the tip of the iceberg.
\nNote: The reason why I say this inappropriate is that sensor data files are composed of header, footer, and body, and each body row has maximum 10 columns according to Official guide,

\n

I feel that this competition's dataset is not so clean (meaning that data format is not arranged enough) then kagglers should implement a lot of data-cleansing routine for efficient analysis.
\nHave you already solve this problem ? Would you please share your findings about data quality?

\n

Thank you for reading.

", "messages": [ { "id": "1175855", "postDate": "01/29/2021 11:36:10", "content": "

\"/kaggle/input/indoor-location-navigation/train/5cd56b90e2acfd2d33b5e33f/F1/5d0868bdbb84450008f569ca.txt\"'s line 483 dose not have line feed at appropriate position.

\n
\n

sample_filepath3 = \"/kaggle/input/indoor-location-navigation/train/5cd56b90e2acfd2d33b5e33f/F1/5d0868bdbb84450008f569ca.txt\"
\n with open(sample_filepath3, 'r') as f:
\n sample_content3 = f.readlines()
\n f.close()
\n sample_content3[482:485] # show line 482, 483, 484, please watch 483 carefully!

\n
\n

Executing python code above on notebook will show following result, then we can know that line 483 have line feed (='\\n') at inappropriate position thus many row is contained in this line.

\n
\n

['1560830842488\\tTYPE_ACCELEROMETER_UNCALIBRATED\\t0.6177063\\t-0.3902588\\t9.904816\\n',
\n '1560830841553\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\tae38c9dac6a05831fe3016a1ed0519fb7f74feea\\t-59\\t-86\\t16.545591294123085\\t833d5531741cba213e7030266fb6d9e2ed4c2aca1560830841566\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\ta77ff5d7252bab87e1eed09b2f29d47622ab9bd0\\t-59\\t-92\\t27.752738449875483\\t6c4ab94d11c2dfb10b9dd8fe35b59ab49cc2c5d61560830841579\\tTYPE_BEACON\\t71f98f1f7b22bfa3f5995790c82431aed3cad143\\t9471a0c7c3b3b94bedcd2ee23186934e67b282ea\\te49643fa7a08253cc4675b041e302c7a4a6e134c\\t-50\\t-82\\t40.89270029241089\\t132b161e5815182e2e0cd96d7a172d76fb69f4381560830841610\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\t630afdd1beda6cf46d6453132ef1855488358042\\t-59\\t-92\\t27.752738449875483\\t815f779cc7325751c44667652830581560a2b3ec1560830841665\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\tb6c7e1a702b4ccfd0f53c4df113fbecc9ed369bc\\t-59\\t-72\\t4.287841349532929\\tad21473af0bb67eddfd6929334a1f428b62b60221560830841682\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\t6af2d433b42bf0b9c73b68bc8189d7dd140c94b6\\t-59\\t-91\\t25.519130565680634\\tce78c6d9e1b67fc3e6aafee45295f8d2e5f7181d1560830841685\\tTYPE_BEACON\\t07efd69e3167537492f0ead89fb2779633b04949\\tb6589fc6ab0dc82cf12099d1c2d40ab994e8410c\\tcd71309513115f935c21612cf52b35fef0ae3fd5\\t-57\\t-88\\t25.70856433929632\\tff6021e77bedf72f51037f108bb9ff1d9853d34e1560830841853\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\tcfc84f0752adc96b489f71195d91a946c5f6d3e8\\t8159618423dfa22f1ca0b62543e2f18eef630ce8\\t-65\\t-82\\t5.50634293288929\\tc1bfecc6701c2cf7c1c1313d057eecf3eb22cdbe1560830841899\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\tdd3ce4a6d6663cfbfd5d87c9de86ea469c4d011f\\t-59\\t-87\\t18.07763630020772\\tc0316935986dc35c1d11b71d6a38febf97ae4f111560830842006\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\t7b37b9ee3e5868308085f4b419c9715b57f6a288\\t-59\\t-78\\t7.852880042745979\\td83732095e320584637fe11dee2a5bfc9e1f833e1560830842013\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\tc8fe745b9bc106416c224e561fac9502b1337ce9\\t-59\\t-93\\t30.155320780411284\\t8e1d0991e5eb1527fa4b8f31ee0f9c3e08f565561560830842025\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\ta5b56798ede30bb10ad5e125321f2e5c396e6a94\\t-59\\t-81\\t10.467388920465797\\t16b617161a8a2d4c73193fcbdda686ecdbea21061560830842047\\tTYPE_BEACON\\t07efd69e3167537492f0ead89fb2779633b04949\\tb6589fc6ab0dc82cf12099d1c2d40ab994e8410c\\t54f6d21e62c7f28225452ca22145a2101ec5e023\\t-57\\t-85\\t19.702364739804867\\t508b247d74755a72cc06ac2a68c761efe933ce5e1560830842446\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\te031005e7f6f7c2d5f91eb742b12f6a4b2e00434\\t-59\\t-87\\t18.07763630020772\\t3573eb6d74efbc27bbe52ff20b411e6bdbb5afe01560830842509\\tTYPE_ACCELEROMETER\\t0.38067627\\t-0.21069336\\t9.275131\\n',
\n '1560830842509\\tTYPE_MAGNETIC_FIELD\\t23.039999\\t-9.42\\t-38.399998\\n']

\n
\n

I found such inappropriate line feed position at many lines in the same file. Probably it is just the tip of the iceberg.
\nNote: The reason why I say this inappropriate is that sensor data files are composed of header, footer, and body, and each body row has maximum 10 columns according to Official guide,

\n

I feel that this competition's dataset is not so clean (meaning that data format is not arranged enough) then kagglers should implement a lot of data-cleansing routine for efficient analysis.
\nHave you already solve this problem ? Would you please share your findings about data quality?

\n

Thank you for reading.

", "rawMarkdown": "\"/kaggle/input/indoor-location-navigation/train/5cd56b90e2acfd2d33b5e33f/F1/5d0868bdbb84450008f569ca.txt\"'s line 483 dose not have line feed at appropriate position.\n\n> sample_filepath3 = \"/kaggle/input/indoor-location-navigation/train/5cd56b90e2acfd2d33b5e33f/F1/5d0868bdbb84450008f569ca.txt\"\nwith open(sample_filepath3, 'r') as f:\n sample_content3 = f.readlines()\n f.close()\nsample_content3[482:485] # show line 482, 483, 484, please watch 483 carefully!\n\nExecuting python code above on notebook will show following result, then we can know that line 483 have line feed (='\\n') at inappropriate position thus many row is contained in this line. \n\n> ['1560830842488\\tTYPE_ACCELEROMETER_UNCALIBRATED\\t0.6177063\\t-0.3902588\\t9.904816\\n',\n '1560830841553\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\tae38c9dac6a05831fe3016a1ed0519fb7f74feea\\t-59\\t-86\\t16.545591294123085\\t833d5531741cba213e7030266fb6d9e2ed4c2aca1560830841566\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\ta77ff5d7252bab87e1eed09b2f29d47622ab9bd0\\t-59\\t-92\\t27.752738449875483\\t6c4ab94d11c2dfb10b9dd8fe35b59ab49cc2c5d61560830841579\\tTYPE_BEACON\\t71f98f1f7b22bfa3f5995790c82431aed3cad143\\t9471a0c7c3b3b94bedcd2ee23186934e67b282ea\\te49643fa7a08253cc4675b041e302c7a4a6e134c\\t-50\\t-82\\t40.89270029241089\\t132b161e5815182e2e0cd96d7a172d76fb69f4381560830841610\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\t630afdd1beda6cf46d6453132ef1855488358042\\t-59\\t-92\\t27.752738449875483\\t815f779cc7325751c44667652830581560a2b3ec1560830841665\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\tb6c7e1a702b4ccfd0f53c4df113fbecc9ed369bc\\t-59\\t-72\\t4.287841349532929\\tad21473af0bb67eddfd6929334a1f428b62b60221560830841682\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\t6af2d433b42bf0b9c73b68bc8189d7dd140c94b6\\t-59\\t-91\\t25.519130565680634\\tce78c6d9e1b67fc3e6aafee45295f8d2e5f7181d1560830841685\\tTYPE_BEACON\\t07efd69e3167537492f0ead89fb2779633b04949\\tb6589fc6ab0dc82cf12099d1c2d40ab994e8410c\\tcd71309513115f935c21612cf52b35fef0ae3fd5\\t-57\\t-88\\t25.70856433929632\\tff6021e77bedf72f51037f108bb9ff1d9853d34e1560830841853\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\tcfc84f0752adc96b489f71195d91a946c5f6d3e8\\t8159618423dfa22f1ca0b62543e2f18eef630ce8\\t-65\\t-82\\t5.50634293288929\\tc1bfecc6701c2cf7c1c1313d057eecf3eb22cdbe1560830841899\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\tdd3ce4a6d6663cfbfd5d87c9de86ea469c4d011f\\t-59\\t-87\\t18.07763630020772\\tc0316935986dc35c1d11b71d6a38febf97ae4f111560830842006\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\t7b37b9ee3e5868308085f4b419c9715b57f6a288\\t-59\\t-78\\t7.852880042745979\\td83732095e320584637fe11dee2a5bfc9e1f833e1560830842013\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\tc8fe745b9bc106416c224e561fac9502b1337ce9\\t-59\\t-93\\t30.155320780411284\\t8e1d0991e5eb1527fa4b8f31ee0f9c3e08f565561560830842025\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\ta5b56798ede30bb10ad5e125321f2e5c396e6a94\\t-59\\t-81\\t10.467388920465797\\t16b617161a8a2d4c73193fcbdda686ecdbea21061560830842047\\tTYPE_BEACON\\t07efd69e3167537492f0ead89fb2779633b04949\\tb6589fc6ab0dc82cf12099d1c2d40ab994e8410c\\t54f6d21e62c7f28225452ca22145a2101ec5e023\\t-57\\t-85\\t19.702364739804867\\t508b247d74755a72cc06ac2a68c761efe933ce5e1560830842446\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\te031005e7f6f7c2d5f91eb742b12f6a4b2e00434\\t-59\\t-87\\t18.07763630020772\\t3573eb6d74efbc27bbe52ff20b411e6bdbb5afe01560830842509\\tTYPE_ACCELEROMETER\\t0.38067627\\t-0.21069336\\t9.275131\\n',\n '1560830842509\\tTYPE_MAGNETIC_FIELD\\t23.039999\\t-9.42\\t-38.399998\\n']\n\nI found such inappropriate line feed position at many lines in the same file. Probably it is just the tip of the iceberg.\nNote: The reason why I say this inappropriate is that sensor data files are composed of header, footer, and body, and each body row has maximum 10 columns according to [Official guide](https://github.com/location-competition/indoor-location-competition-20), \n\nI feel that this competition's dataset is not so clean (meaning that data format is not arranged enough) then kagglers should implement a lot of data-cleansing routine for efficient analysis. \nHave you already solve this problem ? Would you please share your findings about data quality? \n\nThank you for reading.", "votes": null }, { "id": "1176080", "postDate": "01/29/2021 13:49:06", "content": "

Hello there,

\n

in the Data Description of this competition https://www.kaggle.com/c/indoor-location-navigation/data it says:

\n
\n

A note on data quality: In the training files, you may find occasionally that a line is missing the ending newline character, causing it to run on to the next line. It is up to you how you want to handle this issue. This issue is not found in the test data.

\n
\n

So this is not a mistake or anything, they did it on purpose. But i dont know what this purpose is.

", "rawMarkdown": "Hello there,\n\nin the Data Description of this competition https://www.kaggle.com/c/indoor-location-navigation/data it says:\n\n> A note on data quality: In the training files, you may find occasionally that a line is missing the ending newline character, causing it to run on to the next line. It is up to you how you want to handle this issue. This issue is not found in the test data.\n\nSo this is not a mistake or anything, they did it on purpose. But i dont know what this purpose is.", "votes": null }, { "id": "1176203", "postDate": "01/29/2021 14:38:49", "content": "

@jonas0
\nI missed that note. Thank you !

", "rawMarkdown": "jonas0 \nI missed that note. Thank you !", "votes": null }, { "id": "1176281", "postDate": "01/29/2021 15:03:46", "content": "

Could you share with me how you would handle that problem? My first idea would be to simply check if the newline character is missing and then put it in, or is there more behind this problem?

", "rawMarkdown": "Could you share with me how you would handle that problem? My first idea would be to simply check if the newline character is missing and then put it in, or is there more behind this problem?", "votes": null }, { "id": "1176298", "postDate": "01/29/2021 15:12:15", "content": "

@jonas0
\nI am trying to develop the solution. I will publish my notebook if I can !

", "rawMarkdown": "jonas0 \nI am trying to develop the solution. I will publish my notebook if I can !", "votes": null }, { "id": "1177834", "postDate": "01/30/2021 14:38:52", "content": "

@jonas0
\nI opened the solution. Please check following discussion.
\nhttps://www.kaggle.com/c/indoor-location-navigation/discussion/215604

", "rawMarkdown": "jonas0 \nI opened the solution. Please check following discussion. \nhttps://www.kaggle.com/c/indoor-location-navigation/discussion/215604", "votes": null }, { "id": "1178624", "postDate": "01/31/2021 03:03:16", "content": "

After a visual inspection, I see that only the lines with TYPE_BEACON seems to be problematic in the file you mentioned. The final value of the line and the first value of the next line are merged to each other without any break.

", "rawMarkdown": "After a visual inspection, I see that only the lines with TYPE_BEACON seems to be problematic in the file you mentioned. The final value of the line and the first value of the next line are merged to each other without any break.", "votes": null }, { "id": "1179508", "postDate": "01/31/2021 15:45:48", "content": "

@tolgadincer
\nThank you for replying. I agree with you. Some of TYPE_BEACON line dose not have '\\n' at end of themselves, but now I cannot find other data type lines have same problem.
\nI cannot guess why it happened, but it was a good data cleansing practice for me.

", "rawMarkdown": "tolgadincer \nThank you for replying. I agree with you. Some of TYPE_BEACON line dose not have '\\n' at end of themselves, but now I cannot find other data type lines have same problem.\nI cannot guess why it happened, but it was a good data cleansing practice for me.", "votes": null }, { "id": "1227643", "postDate": "03/05/2021 17:52:33", "content": "

I guess they did not do it on purpose, they just did not clean it themselves and let us do the work (that's some data cleaning which is part of the competition :)))

", "rawMarkdown": "I guess they did not do it on purpose, they just did not clean it themselves and let us do the work (that's some data cleaning which is part of the competition :)))", "votes": null } ], "comments": [ { "id": 1176080, "author_name": "jonas0", "author_url": "", "post_date": "01/29/2021 13:49:06", "content": "

Hello there,

\n

in the Data Description of this competition https://www.kaggle.com/c/indoor-location-navigation/data it says:

\n
\n

A note on data quality: In the training files, you may find occasionally that a line is missing the ending newline character, causing it to run on to the next line. It is up to you how you want to handle this issue. This issue is not found in the test data.

\n
\n

So this is not a mistake or anything, they did it on purpose. But i dont know what this purpose is.

", "votes": null, "replies": [ { "id": 1176203, "author_name": "tomokikmogura", "author_url": "", "post_date": "01/29/2021 14:38:49", "content": "

@jonas0
\nI missed that note. Thank you !

", "votes": null, "replies": [] }, { "id": 1176281, "author_name": "jonas0", "author_url": "", "post_date": "01/29/2021 15:03:46", "content": "

Could you share with me how you would handle that problem? My first idea would be to simply check if the newline character is missing and then put it in, or is there more behind this problem?

", "votes": null, "replies": [] }, { "id": 1176298, "author_name": "tomokikmogura", "author_url": "", "post_date": "01/29/2021 15:12:15", "content": "

@jonas0
\nI am trying to develop the solution. I will publish my notebook if I can !

", "votes": null, "replies": [] }, { "id": 1177834, "author_name": "tomokikmogura", "author_url": "", "post_date": "01/30/2021 14:38:52", "content": "

@jonas0
\nI opened the solution. Please check following discussion.
\nhttps://www.kaggle.com/c/indoor-location-navigation/discussion/215604

", "votes": null, "replies": [] }, { "id": 1227643, "author_name": "franoisboyer", "author_url": "", "post_date": "03/05/2021 17:52:33", "content": "

I guess they did not do it on purpose, they just did not clean it themselves and let us do the work (that's some data cleaning which is part of the competition :)))

", "votes": null, "replies": [] } ] }, { "id": 1178624, "author_name": "tolgadincer", "author_url": "", "post_date": "01/31/2021 03:03:16", "content": "

After a visual inspection, I see that only the lines with TYPE_BEACON seems to be problematic in the file you mentioned. The final value of the line and the first value of the next line are merged to each other without any break.

", "votes": null, "replies": [ { "id": 1179508, "author_name": "tomokikmogura", "author_url": "", "post_date": "01/31/2021 15:45:48", "content": "

@tolgadincer
\nThank you for replying. I agree with you. Some of TYPE_BEACON line dose not have '\\n' at end of themselves, but now I cannot find other data type lines have same problem.
\nI cannot guess why it happened, but it was a good data cleansing practice for me.

", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1175855": "\"/kaggle/input/indoor-location-navigation/train/5cd56b90e2acfd2d33b5e33f/F1/5d0868bdbb84450008f569ca.txt\"'s line 483 dose not have line feed at appropriate position.\n\n> sample_filepath3 = \"/kaggle/input/indoor-location-navigation/train/5cd56b90e2acfd2d33b5e33f/F1/5d0868bdbb84450008f569ca.txt\"\nwith open(sample_filepath3, 'r') as f:\n sample_content3 = f.readlines()\n f.close()\nsample_content3[482:485] # show line 482, 483, 484, please watch 483 carefully!\n\nExecuting python code above on notebook will show following result, then we can know that line 483 have line feed (='\\n') at inappropriate position thus many row is contained in this line. \n\n> ['1560830842488\\tTYPE_ACCELEROMETER_UNCALIBRATED\\t0.6177063\\t-0.3902588\\t9.904816\\n',\n '1560830841553\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\tae38c9dac6a05831fe3016a1ed0519fb7f74feea\\t-59\\t-86\\t16.545591294123085\\t833d5531741cba213e7030266fb6d9e2ed4c2aca1560830841566\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\ta77ff5d7252bab87e1eed09b2f29d47622ab9bd0\\t-59\\t-92\\t27.752738449875483\\t6c4ab94d11c2dfb10b9dd8fe35b59ab49cc2c5d61560830841579\\tTYPE_BEACON\\t71f98f1f7b22bfa3f5995790c82431aed3cad143\\t9471a0c7c3b3b94bedcd2ee23186934e67b282ea\\te49643fa7a08253cc4675b041e302c7a4a6e134c\\t-50\\t-82\\t40.89270029241089\\t132b161e5815182e2e0cd96d7a172d76fb69f4381560830841610\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\t630afdd1beda6cf46d6453132ef1855488358042\\t-59\\t-92\\t27.752738449875483\\t815f779cc7325751c44667652830581560a2b3ec1560830841665\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\tb6c7e1a702b4ccfd0f53c4df113fbecc9ed369bc\\t-59\\t-72\\t4.287841349532929\\tad21473af0bb67eddfd6929334a1f428b62b60221560830841682\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\t6af2d433b42bf0b9c73b68bc8189d7dd140c94b6\\t-59\\t-91\\t25.519130565680634\\tce78c6d9e1b67fc3e6aafee45295f8d2e5f7181d1560830841685\\tTYPE_BEACON\\t07efd69e3167537492f0ead89fb2779633b04949\\tb6589fc6ab0dc82cf12099d1c2d40ab994e8410c\\tcd71309513115f935c21612cf52b35fef0ae3fd5\\t-57\\t-88\\t25.70856433929632\\tff6021e77bedf72f51037f108bb9ff1d9853d34e1560830841853\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\tcfc84f0752adc96b489f71195d91a946c5f6d3e8\\t8159618423dfa22f1ca0b62543e2f18eef630ce8\\t-65\\t-82\\t5.50634293288929\\tc1bfecc6701c2cf7c1c1313d057eecf3eb22cdbe1560830841899\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\tdd3ce4a6d6663cfbfd5d87c9de86ea469c4d011f\\t-59\\t-87\\t18.07763630020772\\tc0316935986dc35c1d11b71d6a38febf97ae4f111560830842006\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\t7b37b9ee3e5868308085f4b419c9715b57f6a288\\t-59\\t-78\\t7.852880042745979\\td83732095e320584637fe11dee2a5bfc9e1f833e1560830842013\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\tc8fe745b9bc106416c224e561fac9502b1337ce9\\t-59\\t-93\\t30.155320780411284\\t8e1d0991e5eb1527fa4b8f31ee0f9c3e08f565561560830842025\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\ta5b56798ede30bb10ad5e125321f2e5c396e6a94\\t-59\\t-81\\t10.467388920465797\\t16b617161a8a2d4c73193fcbdda686ecdbea21061560830842047\\tTYPE_BEACON\\t07efd69e3167537492f0ead89fb2779633b04949\\tb6589fc6ab0dc82cf12099d1c2d40ab994e8410c\\t54f6d21e62c7f28225452ca22145a2101ec5e023\\t-57\\t-85\\t19.702364739804867\\t508b247d74755a72cc06ac2a68c761efe933ce5e1560830842446\\tTYPE_BEACON\\t89cb11b04122cef23388b0da06bd426c1f48a9b5\\t4bd29af61db57eb8675b56f79c5ff4ae6f81c03a\\te031005e7f6f7c2d5f91eb742b12f6a4b2e00434\\t-59\\t-87\\t18.07763630020772\\t3573eb6d74efbc27bbe52ff20b411e6bdbb5afe01560830842509\\tTYPE_ACCELEROMETER\\t0.38067627\\t-0.21069336\\t9.275131\\n',\n '1560830842509\\tTYPE_MAGNETIC_FIELD\\t23.039999\\t-9.42\\t-38.399998\\n']\n\nI found such inappropriate line feed position at many lines in the same file. Probably it is just the tip of the iceberg.\nNote: The reason why I say this inappropriate is that sensor data files are composed of header, footer, and body, and each body row has maximum 10 columns according to [Official guide](https://github.com/location-competition/indoor-location-competition-20), \n\nI feel that this competition's dataset is not so clean (meaning that data format is not arranged enough) then kagglers should implement a lot of data-cleansing routine for efficient analysis. \nHave you already solve this problem ? Would you please share your findings about data quality? \n\nThank you for reading.", "1176080": "Hello there,\n\nin the Data Description of this competition https://www.kaggle.com/c/indoor-location-navigation/data it says:\n\n> A note on data quality: In the training files, you may find occasionally that a line is missing the ending newline character, causing it to run on to the next line. It is up to you how you want to handle this issue. This issue is not found in the test data.\n\nSo this is not a mistake or anything, they did it on purpose. But i dont know what this purpose is.", "1176203": "jonas0 \nI missed that note. Thank you !", "1176281": "Could you share with me how you would handle that problem? My first idea would be to simply check if the newline character is missing and then put it in, or is there more behind this problem?", "1176298": "jonas0 \nI am trying to develop the solution. I will publish my notebook if I can !", "1177834": "jonas0 \nI opened the solution. Please check following discussion. \nhttps://www.kaggle.com/c/indoor-location-navigation/discussion/215604", "1178624": "After a visual inspection, I see that only the lines with TYPE_BEACON seems to be problematic in the file you mentioned. The final value of the line and the first value of the next line are merged to each other without any break.", "1179508": "tolgadincer \nThank you for replying. I agree with you. Some of TYPE_BEACON line dose not have '\\n' at end of themselves, but now I cannot find other data type lines have same problem.\nI cannot guess why it happened, but it was a good data cleansing practice for me.", "1227643": "I guess they did not do it on purpose, they just did not clean it themselves and let us do the work (that's some data cleaning which is part of the competition :)))" }, "source": "meta" }