\n
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F83267%2F6683c4e7ae54f9a4b2ba79c6c9f10abe%2Fvariable_names.png?generation=1669071906008304&alt=media\")
This competition is extremely complex.
\nWe are presented with this seemingly simple data, but THERE IS SO MUCH you can do with it. Just note the explosion of Kaggle kernels sharing ideas already (co-visitation matrices, matrix factorization, reranking, word2vec, training sequential DL modelsβ¦)
\nIn part 1 of this two-post series I talked about one aspect of combating complexity -- how to navigate the landscape of ideas to identify what to work on and how to structure your solution. The answer has been simple and could be summed up in a single word: @cdeotte π
\nIn this post let's look at the other type of complexity we need to deal with to be successful - the complexity of the code that we will write/have to work with.
\nThis type of complexity can crush you extremely quickly. In fact, I only feel I stand a chance in this competition to do something interesting because I got crushed by code complexity in ML projects so many times before!
\nWhat can the pipeline in this competition look like?
\ncreate datasets to train on and for evaluation -> train MF/word2vec -> create a covisitation matrix (separate for train with validation and for the full train set for submission) -> create features & diagnostic code (measure hit rate, recall) for ranking models -> train a ranking model -> create submission based on this output
And that is a simplification of what the final pipeline will look like! There are so many things you can do along the way (for instance, what could be the equivalent of training with CV in this competition? what interesting things can you do to the data? what about training ranking models with different seeds? what about training ranking models on subsets of columns? how about doing feature selection? what about combining outputs from multiple ranking models?)
\nThis is a humongous undertaking⦠one that the field of Machine Learning doesn't have a solution to!
\nWith infinite resources, you could use something like Metaflow. But a) most Kagglers don't have the money to spend thousands of dollars on infra b) most Kagglers don't know where to begin setting up stuff like this.
\nSo what can we do? We need to solve this problem with the right tool for the job given our circumstances.
\nAnd the answer, at its core, is just a single word. Refactoring.
\nYOU HAVE TO TIRELESSLY REFACTOR. It is not enough that your code runs and seems to output something useful.
\nYou need to make it readable. You (ideally) should make it composable.
\nBut let's stop with these big words. People create a lot of mystique about relatively straightforward things. Let's not fall into that trap here.
\nThe act of refactoring is very simple (though some absolutely kickass books have been written on it)
\nA couple of very powerful components of refactoring techniques are:
\nWhat is the idea behind naming? Use variable names that communicate what the variable holds. Trade being succinct for readability. Always.
\nHere are a couple of variable names from my code:
\n
I really couldn't care less how long my variable names are AS LONG AS I UNDERSTAND what they hold.
\nCreating Kaggle kernels is a great exercise in writing readable code. In doing so you learn how to express your ideas so that people with varying levels of experience will be able to follow along.
\nAnd as for \"creating functions to communicate intent\", this is taking the idea of naming to another level.
\nThis is a function from my code that I am using so far in a single place. Yes, you want to create functions so that you do not repeat yourself (DRY). But even if I had this bit of code in a single place in my notebook, it might still be valuable to wrap it up in a function.
\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F83267%2F55ef1fe5123c4dd0bede945f4a8fed60%2Ffunctions_to_communicate_intent.png?generation=1669072142790250&alt=media\")
How much easier it is to understand what is happening in your code when you encounter cast_to_df_dtypes vs if you were to encounter just the blob of code that is in the body of the method?
Code is meant to be read by humans, the fact that it runs on a computer is secondary. Code is meant to express (and communicate!) ideas to other human beings. Be that someone reading your code who might have just picked up Python a month ago, or you from the future.
\nI am now experimenting with execnb by fast.ai and am thinking this might be another way to reduce the complexity of my code. We'll see.
\nBut it all starts with refactoring π
\nAnd the wildest story is I have been looking for resources on my Twitter on refactoring to share with you and it turns out I created a repository on refactoring three years ago and completely forgot about π³ But some really good stuff there that I accumulated over months if not years.
\nAnyhow, hope this can be of help π All these ideas are experimental at best, but will try to put them into action in this competition (if I manage to find the time and not get pulled into some other direction π) and will let you know how it goes
\nThis competition is extremely complex.
\nWe are presented with this seemingly simple data, but THERE IS SO MUCH you can do with it. Just note the explosion of Kaggle kernels sharing ideas already (co-visitation matrices, matrix factorization, reranking, word2vec, training sequential DL modelsβ¦)
\nIn part 1 of this two-post series I talked about one aspect of combating complexity -- how to navigate the landscape of ideas to identify what to work on and how to structure your solution. The answer has been simple and could be summed up in a single word: @cdeotte π
\nIn this post let's look at the other type of complexity we need to deal with to be successful - the complexity of the code that we will write/have to work with.
\nThis type of complexity can crush you extremely quickly. In fact, I only feel I stand a chance in this competition to do something interesting because I got crushed by code complexity in ML projects so many times before!
\nWhat can the pipeline in this competition look like?
\ncreate datasets to train on and for evaluation -> train MF/word2vec -> create a covisitation matrix (separate for train with validation and for the full train set for submission) -> create features & diagnostic code (measure hit rate, recall) for ranking models -> train a ranking model -> create submission based on this output
And that is a simplification of what the final pipeline will look like! There are so many things you can do along the way (for instance, what could be the equivalent of training with CV in this competition? what interesting things can you do to the data? what about training ranking models with different seeds? what about training ranking models on subsets of columns? how about doing feature selection? what about combining outputs from multiple ranking models?)
\nThis is a humongous undertaking⦠one that the field of Machine Learning doesn't have a solution to!
\nWith infinite resources, you could use something like Metaflow. But a) most Kagglers don't have the money to spend thousands of dollars on infra b) most Kagglers don't know where to begin setting up stuff like this.
\nSo what can we do? We need to solve this problem with the right tool for the job given our circumstances.
\nAnd the answer, at its core, is just a single word. Refactoring.
\nYOU HAVE TO TIRELESSLY REFACTOR. It is not enough that your code runs and seems to output something useful.
\nYou need to make it readable. You (ideally) should make it composable.
\nBut let's stop with these big words. People create a lot of mystique about relatively straightforward things. Let's not fall into that trap here.
\nThe act of refactoring is very simple (though some absolutely kickass books have been written on it)
\nA couple of very powerful components of refactoring techniques are:
\nWhat is the idea behind naming? Use variable names that communicate what the variable holds. Trade being succinct for readability. Always.
\nHere are a couple of variable names from my code:
\n
I really couldn't care less how long my variable names are AS LONG AS I UNDERSTAND what they hold.
\nCreating Kaggle kernels is a great exercise in writing readable code. In doing so you learn how to express your ideas so that people with varying levels of experience will be able to follow along.
\nAnd as for \"creating functions to communicate intent\", this is taking the idea of naming to another level.
\nThis is a function from my code that I am using so far in a single place. Yes, you want to create functions so that you do not repeat yourself (DRY). But even if I had this bit of code in a single place in my notebook, it might still be valuable to wrap it up in a function.
\n
How much easier it is to understand what is happening in your code when you encounter cast_to_df_dtypes vs if you were to encounter just the blob of code that is in the body of the method?
Code is meant to be read by humans, the fact that it runs on a computer is secondary. Code is meant to express (and communicate!) ideas to other human beings. Be that someone reading your code who might have just picked up Python a month ago, or you from the future.
\nI am now experimenting with execnb by fast.ai and am thinking this might be another way to reduce the complexity of my code. We'll see.
\nBut it all starts with refactoring π
\nAnd the wildest story is I have been looking for resources on my Twitter on refactoring to share with you and it turns out I created a repository on refactoring three years ago and completely forgot about π³ But some really good stuff there that I accumulated over months if not years.
\nAnyhow, hope this can be of help π All these ideas are experimental at best, but will try to put them into action in this competition (if I manage to find the time and not get pulled into some other direction π) and will let you know how it goes
\nThanks for sharing! π Such posts and experience sharing definitely have a place in Kaggle.
\nGood suggestions:
\nDo share your experience with execnb when you have tried it out.
\n\n", "rawMarkdown": "Thanks for sharing! π Such posts and experience sharing definitely have a place in Kaggle.\n\nGood suggestions:\n- to keep naming meaningful and \n- functions as a means for explaining code blocks\n\nDo share your experience with execnb when you have tried it out.\n> \"Kaggle is a place to share and learn\"", "votes": null }, { "id": "2040218", "postDate": "11/22/2022 20:34:46", "content": "\"Kaggle is a place to share and learn\"
\n
The easiest thing to do is become crazy and get used to it π
\nAlso, your casting function for large-scale data would consume a lot of computational power!
The biggest issue with all kagglers including myself is a machine-learning strategy!
\nWe trial and test algorithms without exploring the business context and researching the right al algorithms to use to save time and computing power!
Thanks for your efforts in this competition. Many thanks for the experience you're sharing with us!
\nThis competition is really strict for optimization and computational power usage, love that π
Thanks for sharing! π Such posts and experience sharing definitely have a place in Kaggle.
\nGood suggestions:
\nDo share your experience with execnb when you have tried it out.
\n\n", "votes": null, "replies": [] }, { "id": 2040218, "author_name": "muhammadammarjamshed", "author_url": "", "post_date": "11/22/2022 20:34:46", "content": "\"Kaggle is a place to share and learn\"
\n
The easiest thing to do is become crazy and get used to it π
\nAlso, your casting function for large-scale data would consume a lot of computational power!
The biggest issue with all kagglers including myself is a machine-learning strategy!
\nWe trial and test algorithms without exploring the business context and researching the right al algorithms to use to save time and computing power!
Thanks for your efforts in this competition. Many thanks for the experience you're sharing with us!
\nThis competition is really strict for optimization and computational power usage, love that π