{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"## TODO : make a submission on kaggle\n\n\nHints to improve the score\n\n- We were using only 50% of the dataset to train our NN\n- decrease the batch size (we were using 2048 for prototyping)\n- The NN architecture can be improved\n- We didnt use any regularization (L1/L2, dropout, SpatialDropout)\n- improve the text preprocessing\n- try other embeddings\n- try other finetuning approaches :\n   1. Setting a different LR per layer, https://erikbrorson.github.io/2018/04/30/Adam-with-learning-rate-multipliers/\n   2. ReduceOnPlateau schedule\n- increase the size of bagging\n- ensembe different architectures\n- During text preprocessing, some information is lost. Make additional numerical features as inputs for the NN. For example : number of uppercase characters, punctuations, bad words, etc\n- We have removed rare words from the vocabularym maybe performance will improve if we keep them\n\n- read kernels from https://www.kaggle.com/c/quora-insincere-questions-classification/kernels and try other ideas"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","collapsed":true,"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":false},"cell_type":"markdown","source":"## To keep in mind\n- Learning rates may be quit different depending on the way you initialize your embeddings (random vs transfer learning)\n\n- Always use transfer learning (cant't be worse than random initialization)\n\n- when you find an optimal batch size, you will probably not need to tune it anymore\n\n- Always do bagging to check for improvements (not only for NN)\n\n- Sometimes simple things works better (Attention mechanism didnt work for me on this dataset)\n\n- Dont be misleaded by the sequential approach I used for this lecture. Looking for the best performance is an iterative approach\n\n\n\n\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.4","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}