Hi All, \nI want to know if you guys have a specific code structure that you guys follow? For me, it is getting increasingly unmanageable to plug in features or creating orthogonal models. Most of the kernels I see have a single python file with all the code dumped.
\n\nHi All, \nI want to know if you guys have a specific code structure that you guys follow? For me, it is getting increasingly unmanageable to plug in features or creating orthogonal models. Most of the kernels I see have a single python file with all the code dumped.
\n\nThis is also a problem for me, right now I basically separate codes into different structure, namely, train validation separate, feature generation, model tuning, test and prediction outcome, each in different folder and import as class object.however, I often need to rename all the files (transformed data, models and corresponding outputs) within a manageable style which is quite headache.\nwaiting for top players sharing.
", "rawMarkdown": "This is also a problem for me, right now I basically separate codes into different structure, namely, train validation separate, feature generation, model tuning, test and prediction outcome, each in different folder and import as class object.however, I often need to rename all the files (transformed data, models and corresponding outputs) within a manageable style which is quite headache.\nwaiting for top players sharing.", "votes": null }, { "id": "310092", "postDate": "04/06/2018 15:34:22", "content": "As far as one goes on refactoring stuff, I tired an approach of using IntelliJ as the IDE(this surprisingly supports python notebooks very well.), where refactoring variables with py files was easy, but used to get messy with imports overtime and changes not getting reflected in real time. Had to restart kernels.
", "rawMarkdown": "As far as one goes on refactoring stuff, I tired an approach of using IntelliJ as the IDE(this surprisingly supports python notebooks very well.), where refactoring variables with py files was easy, but used to get messy with imports overtime and changes not getting reflected in real time. Had to restart kernels.", "votes": null }, { "id": "310241", "postDate": "04/06/2018 22:05:37", "content": "You don't need anything too fancy, I separate my code like so:
\n\nI don't use notebooks/ides at all, but being able to load your finished df whenever you want will probably save you a lot of time.
", "rawMarkdown": "You don't need anything too fancy, I separate my code like so:\n\n 1. Separate all feature engineering code from your modeling code\n 2. Save your dataframe as pkl/hdf5/ etc in between the FE/ preprocessing steps, sharding if your df gets too big.\n 3. Load finished df when you want to start modeling\n 4. Repeat 2-3 as many times as you want.\n\nI don't use notebooks/ides at all, but being able to load your finished df whenever you want will probably save you a lot of time.", "votes": null } ], "comments": [ { "id": 310088, "author_name": "daqixu", "author_url": "", "post_date": "04/06/2018 15:25:26", "content": "This is also a problem for me, right now I basically separate codes into different structure, namely, train validation separate, feature generation, model tuning, test and prediction outcome, each in different folder and import as class object.however, I often need to rename all the files (transformed data, models and corresponding outputs) within a manageable style which is quite headache.\nwaiting for top players sharing.
", "votes": null, "replies": [ { "id": 310092, "author_name": "ronakagrawal", "author_url": "", "post_date": "04/06/2018 15:34:22", "content": "As far as one goes on refactoring stuff, I tired an approach of using IntelliJ as the IDE(this surprisingly supports python notebooks very well.), where refactoring variables with py files was easy, but used to get messy with imports overtime and changes not getting reflected in real time. Had to restart kernels.
", "votes": null, "replies": [] } ] }, { "id": 310241, "author_name": "stevenknguyen", "author_url": "", "post_date": "04/06/2018 22:05:37", "content": "You don't need anything too fancy, I separate my code like so:
\n\nI don't use notebooks/ides at all, but being able to load your finished df whenever you want will probably save you a lot of time.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "309976": "Hi All, \nI want to know if you guys have a specific code structure that you guys follow? For me, it is getting increasingly unmanageable to plug in features or creating orthogonal models. Most of the kernels I see have a single python file with all the code dumped. \n\n 1. Do you guys do EDA in a different Notebooks and then dump code to single notebook and then publish?\n 2. Do you use a notebook importer like nbimporter or something else?\n 3. Do you use notebooks in PyCharm or some other IDE?\n 4. Has anyone tried an object oriented approach to these solutions, to try and manage better? on the lines of helper classes, or like wise?\n 5. Do you have a CI pipeline or something with the EC2 Instance?\n 6. Do you have tests or use a test framework as a safety net?", "310088": "This is also a problem for me, right now I basically separate codes into different structure, namely, train validation separate, feature generation, model tuning, test and prediction outcome, each in different folder and import as class object.however, I often need to rename all the files (transformed data, models and corresponding outputs) within a manageable style which is quite headache.\nwaiting for top players sharing.", "310092": "As far as one goes on refactoring stuff, I tired an approach of using IntelliJ as the IDE(this surprisingly supports python notebooks very well.), where refactoring variables with py files was easy, but used to get messy with imports overtime and changes not getting reflected in real time. Had to restart kernels.", "310241": "You don't need anything too fancy, I separate my code like so:\n\n 1. Separate all feature engineering code from your modeling code\n 2. Save your dataframe as pkl/hdf5/ etc in between the FE/ preprocessing steps, sharding if your df gets too big.\n 3. Load finished df when you want to start modeling\n 4. Repeat 2-3 as many times as you want.\n\nI don't use notebooks/ides at all, but being able to load your finished df whenever you want will probably save you a lot of time." }, "source": "meta" }