Lots of failed runs, lots of failed experiments and finally a great big shakeup, nothing amazing but making our kernel public at https://www.kaggle.com/dicksonchin93/kfold-tfidf-trial just to share , public CV before second stage was at 0.683 and private after second stage is 0.69721, something like 1k + position before second stage (with another kernel, 0.683 should be at the 4k position LOL) and shot to 27th after, a pleasant surprise, solution is an average blend of 5 models which includes all 4 embeddings with a local CV of 0.7028. Some notable difference is we did lemmatization and lowering and uppering the words to find oov words in each embeddings , made a keras sparse model with tfidf and some additional feature engineering from kernels and previous competitions , made a boosting RNN model on top of a TFIDF to Ridge model, this made the optimal threshold to be pushed to the 0.4-0.5 range in local CV, we used a little parallel computation to made all the processing to run in time, tested quite a few setups and find that its best to run pytorch gpu models in the main interface as is and run the other thread asynchronously, initializing pytorch gpu variables in a thread in kaggle kernels fail for some reason. Also, we had to opt for a 'weaker' set of RNN models( RNN architecture with less hidden cells, etc. ) so that the kernel could run in time although stronger RNN model gave us a higher local CV, we find that concatenation of the embedding's does help the NN models to converge faster and it uses all the embedding information's more optimally with regards to time and resource utilized than just running individual embedding models. Last but not least, we opt out of CNN models since it takes a lot more time to run the model. We made some mock test too in the end by sampling the test dataframe with a times 6 fraction just to make sure the kernel will run in the second stage. I once thought collaboration in a big team is hard, and should be even harder in a kernels competition! But now I know its easy, you just have to work independently from the other half of the team :) special thanks to @learnmower for great contributions to the kernel, and also thanks to @wrosinski for optimizing some parts of the code. Also thanks to @mchahhou for his kernel in mercari
", "messages": [ { "id": 471105, "postDate": "2019-02-14T03:15:05.440Z", "content": "Lots of failed runs, lots of failed experiments and finally a great big shakeup, nothing amazing but making our kernel public at https://www.kaggle.com/dicksonchin93/kfold-tfidf-trial just to share , public CV before second stage was at 0.683 and private after second stage is 0.69721, something like 1k + position before second stage (with another kernel, 0.683 should be at the 4k position LOL) and shot to 27th after, a pleasant surprise, solution is an average blend of 5 models which includes all 4 embeddings with a local CV of 0.7028. Some notable difference is we did lemmatization and lowering and uppering the words to find oov words in each embeddings , made a keras sparse model with tfidf and some additional feature engineering from kernels and previous competitions , made a boosting RNN model on top of a TFIDF to Ridge model, this made the optimal threshold to be pushed to the 0.4-0.5 range in local CV, we used a little parallel computation to made all the processing to run in time, tested quite a few setups and find that its best to run pytorch gpu models in the main interface as is and run the other thread asynchronously, initializing pytorch gpu variables in a thread in kaggle kernels fail for some reason. Also, we had to opt for a 'weaker' set of RNN models( RNN architecture with less hidden cells, etc. ) so that the kernel could run in time although stronger RNN model gave us a higher local CV, we find that concatenation of the embedding's does help the NN models to converge faster and it uses all the embedding information's more optimally with regards to time and resource utilized than just running individual embedding models. Last but not least, we opt out of CNN models since it takes a lot more time to run the model. We made some mock test too in the end by sampling the test dataframe with a times 6 fraction just to make sure the kernel will run in the second stage. I once thought collaboration in a big team is hard, and should be even harder in a kernels competition! But now I know its easy, you just have to work independently from the other half of the team :) special thanks to @learnmower for great contributions to the kernel, and also thanks to @wrosinski for optimizing some parts of the code. Also thanks to @mchahhou for his kernel in mercari
", "rawMarkdown": "Lots of failed runs, lots of failed experiments and finally a great big shakeup, nothing amazing but making our kernel public at https://www.kaggle.com/dicksonchin93/kfold-tfidf-trial just to share , public CV before second stage was at 0.683 and private after second stage is 0.69721, something like 1k + position before second stage (with another kernel, 0.683 should be at the 4k position LOL) and shot to 27th after, a pleasant surprise, solution is an average blend of 5 models which includes all 4 embeddings with a local CV of 0.7028. Some notable difference is we did lemmatization and lowering and uppering the words to find oov words in each embeddings , made a keras sparse model with tfidf and some additional feature engineering from kernels and previous competitions , made a boosting RNN model on top of a TFIDF to Ridge model, this made the optimal threshold to be pushed to the 0.4-0.5 range in local CV, we used a little parallel computation to made all the processing to run in time, tested quite a few setups and find that its best to run pytorch gpu models in the main interface as is and run the other thread asynchronously, initializing pytorch gpu variables in a thread in kaggle kernels fail for some reason. Also, we had to opt for a 'weaker' set of RNN models( RNN architecture with less hidden cells, etc. ) so that the kernel could run in time although stronger RNN model gave us a higher local CV, we find that concatenation of the embedding's does help the NN models to converge faster and it uses all the embedding information's more optimally with regards to time and resource utilized than just running individual embedding models. Last but not least, we opt out of CNN models since it takes a lot more time to run the model. We made some mock test too in the end by sampling the test dataframe with a times 6 fraction just to make sure the kernel will run in the second stage. I once thought collaboration in a big team is hard, and should be even harder in a kernels competition! But now I know its easy, you just have to work independently from the other half of the team :) special thanks to @learnmower for great contributions to the kernel, and also thanks to @wrosinski for optimizing some parts of the code. Also thanks to @mchahhou for his kernel in mercari ", "votes": 21 }, { "id": 471252, "postDate": "2019-02-14T07:41:48.457Z", "content": "Thank you for your work Chin and Thomas (learnmower) !\nI'm happy that I could have contributed at least the code optimizations. \nKernels competitions are an interesting type, with their own challenges and additional patience requirement, so it's great to see such a good final score.
", "rawMarkdown": "Thank you for your work Chin and Thomas (learnmower) !\nI'm happy that I could have contributed at least the code optimizations. \nKernels competitions are an interesting type, with their own challenges and additional patience requirement, so it's great to see such a good final score.", "votes": 2 }, { "id": 471389, "postDate": "2019-02-14T11:37:46.517Z", "content": "@dicksonchin93, congrats and thanks for sharing your solution.
", "rawMarkdown": "@dicksonchin93, congrats and thanks for sharing your solution.", "replies": [ { "id": 471897, "postDate": "2019-02-15T03:55:06.093Z", "content": "welcome!
", "rawMarkdown": "welcome!" } ] }, { "id": 471255, "postDate": "2019-02-14T07:51:18.060Z", "content": "I was in the same shoe. My 20th kernel sat 1000+ in public LB
", "rawMarkdown": "I was in the same shoe. My 20th kernel sat 1000+ in public LB", "replies": [ { "id": 471898, "postDate": "2019-02-15T03:55:17.227Z", "content": "Hi 5 :)
", "rawMarkdown": "Hi 5 :)" } ] }, { "id": 471147, "postDate": "2019-02-14T04:41:00.660Z", "content": "Congrats!
", "rawMarkdown": "Congrats!", "replies": [ { "id": 471900, "postDate": "2019-02-15T03:55:47.430Z", "content": "Thanks! congrats to you too
", "rawMarkdown": "Thanks! congrats to you too" } ] }, { "id": 471863, "postDate": "2019-02-15T02:11:06.497Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 471252, "author_name": "Wojtek Rosinski", "author_url": "", "post_date": "2019-02-14T07:41:48.457000", "content": "Thank you for your work Chin and Thomas (learnmower) !\nI'm happy that I could have contributed at least the code optimizations. \nKernels competitions are an interesting type, with their own challenges and additional patience requirement, so it's great to see such a good final score.
", "votes": 2, "replies": [] }, { "id": 471389, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-02-14T11:37:46.517000", "content": "@dicksonchin93, congrats and thanks for sharing your solution.
", "votes": 0, "replies": [ { "id": 471897, "author_name": "Ee Kin Chin", "author_url": "", "post_date": "2019-02-15T03:55:06.093000", "content": "welcome!
", "votes": 0, "replies": [] } ] }, { "id": 471255, "author_name": "Darkate", "author_url": "", "post_date": "2019-02-14T07:51:18.060000", "content": "I was in the same shoe. My 20th kernel sat 1000+ in public LB
", "votes": 0, "replies": [ { "id": 471898, "author_name": "Ee Kin Chin", "author_url": "", "post_date": "2019-02-15T03:55:17.227000", "content": "Hi 5 :)
", "votes": 0, "replies": [] } ] }, { "id": 471147, "author_name": "Laevatein", "author_url": "", "post_date": "2019-02-14T04:41:00.660000", "content": "Congrats!
", "votes": 0, "replies": [ { "id": 471900, "author_name": "Ee Kin Chin", "author_url": "", "post_date": "2019-02-15T03:55:47.430000", "content": "Thanks! congrats to you too
", "votes": 0, "replies": [] } ] }, { "id": 471863, "author_name": "", "author_url": "", "post_date": "2019-02-15T02:11:06.497000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "471105": "Lots of failed runs, lots of failed experiments and finally a great big shakeup, nothing amazing but making our kernel public at https://www.kaggle.com/dicksonchin93/kfold-tfidf-trial just to share , public CV before second stage was at 0.683 and private after second stage is 0.69721, something like 1k + position before second stage (with another kernel, 0.683 should be at the 4k position LOL) and shot to 27th after, a pleasant surprise, solution is an average blend of 5 models which includes all 4 embeddings with a local CV of 0.7028. Some notable difference is we did lemmatization and lowering and uppering the words to find oov words in each embeddings , made a keras sparse model with tfidf and some additional feature engineering from kernels and previous competitions , made a boosting RNN model on top of a TFIDF to Ridge model, this made the optimal threshold to be pushed to the 0.4-0.5 range in local CV, we used a little parallel computation to made all the processing to run in time, tested quite a few setups and find that its best to run pytorch gpu models in the main interface as is and run the other thread asynchronously, initializing pytorch gpu variables in a thread in kaggle kernels fail for some reason. Also, we had to opt for a 'weaker' set of RNN models( RNN architecture with less hidden cells, etc. ) so that the kernel could run in time although stronger RNN model gave us a higher local CV, we find that concatenation of the embedding's does help the NN models to converge faster and it uses all the embedding information's more optimally with regards to time and resource utilized than just running individual embedding models. Last but not least, we opt out of CNN models since it takes a lot more time to run the model. We made some mock test too in the end by sampling the test dataframe with a times 6 fraction just to make sure the kernel will run in the second stage. I once thought collaboration in a big team is hard, and should be even harder in a kernels competition! But now I know its easy, you just have to work independently from the other half of the team :) special thanks to @learnmower for great contributions to the kernel, and also thanks to @wrosinski for optimizing some parts of the code. Also thanks to @mchahhou for his kernel in mercari ", "471252": "Thank you for your work Chin and Thomas (learnmower) !\nI'm happy that I could have contributed at least the code optimizations. \nKernels competitions are an interesting type, with their own challenges and additional patience requirement, so it's great to see such a good final score.", "471389": "@dicksonchin93, congrats and thanks for sharing your solution.", "471255": "I was in the same shoe. My 20th kernel sat 1000+ in public LB", "471147": "Congrats!", "471863": "" } }