Hi, my name is Kyle. I am new to Kaggle and am interested in this specific competition to teach myself image processing. I searched through the forums and didn't really see any good beginner threads. I have some questions, if anyone could help me out.
\n\nI should also say that my overall goal here is to gain experience in image processing, with the specific purpose of being able to leverage this experience in a future job interview. So I ultimately would like to accomplish this project in a way that a software/research company would go about doing it. My specific career interests are in aerospace and astrophysics, so if anyone has an specific insights on how this project might relate to the aerospace industry, I'd love to hear it.
\n\nWhile these are the only questions I have right now, I will certainly have more once a conversation builds up. I hope to make this an ongoing thread about how to get started with this project.
\n\n\nThanks!
", "messages": [ { "id": "74955", "postDate": "04/27/2015 02:40:18", "content": "Hi, my name is Kyle. I am new to Kaggle and am interested in this specific competition to teach myself image processing. I searched through the forums and didn't really see any good beginner threads. I have some questions, if anyone could help me out.
\n\nI should also say that my overall goal here is to gain experience in image processing, with the specific purpose of being able to leverage this experience in a future job interview. So I ultimately would like to accomplish this project in a way that a software/research company would go about doing it. My specific career interests are in aerospace and astrophysics, so if anyone has an specific insights on how this project might relate to the aerospace industry, I'd love to hear it.
\n\nWhile these are the only questions I have right now, I will certainly have more once a conversation builds up. I hope to make this an ongoing thread about how to get started with this project.
\n\n\nThanks!
", "rawMarkdown": "", "votes": null }, { "id": "75212", "postDate": "04/28/2015 08:55:34", "content": "Welcome home!
\nFor me data volume is staggering, I havent started yet but I would start with aws. Also for speedy processing you may require GPU.
\nIf you never heard of CUDA, give it a shot. There is mooc on Udacity created by Nividia folks
\nhttps://www.udacity.com/course/intro-to-parallel-programming--cs344
", "rawMarkdown": "", "votes": null }, { "id": "75388", "postDate": "04/29/2015 10:04:29", "content": "Hi Kyle,
\nI guess it comes down to whether you want to learn a particular tool or language (it sounds like you do) or whether you want to just try out interesting approaches to the problem without getting awkward tools get in the way.
\n"Is this project doable in C++?" - Probably not. I think this is a difficult enough problem that there's no need to make it harder by picking difficult tools as well. I know C++ quite well, but let's just say my code would be ten times larger and I would be still debugging it sometime next year if I was using that :) The focus should be very much on the algorithms not the implementation.
\n"What packages lend themselves well to this project?" - You might learn scikit-learn, scikit-image, statsmodels, pandas, theano, numpy, scipy ... Even just implementing one of the published papers on diabetic retinopathy detection using that toolset and getting useful predictions from that would be a nice learning curve. I almost hesitate to mention convnets because they are so highly hyped these days, but yeah, there's also convnets.
\n"leverage this experience in a future job interview" - if I was interviewing someone I would be impressed by good results, and by choosing and learning tools well suited to the task you were working on (since I do care about results). Implementing something like (for example) Gabor filters in C++ from scratch could make me think you are a good C++ programmer, but I would also have a negative impression of you as a non-pragmatic programmer (since this should be one line of code with skimage, and take maybe five minutes), and I probably wouldn't want to hire you. Sorry, but that's just how it is. CUDA may be better suited here, but I'd hope you're using PyCUDA, and even better using a package like theano and just adding a few CUDA kernels where there is nothing off-the-shelf that you can use.
\n"Is computer performance a big deal with this project?" - Yes, it lets you iterate faster. There's many a time when I've wanted to try something "simple", and quickly realized that "simple" takes half a minute per image, or in other words something like a day to work through a fraction of the dataset which is just barely large enough that it allows some degree of statistical confidence. Ouch. This is where running the same work split among a dozen c4.8xlarge boxes on Amazon and getting results in 15 minutes suddenly starts helping a lot. Sure, you can leave it running on your laptop for a few days, but...
\n"any recommended resources to help learn about image processing algorithms" - check out winning code from other Kaggle competitions, particularly ones dealing with images. It's usually posted in the forums. Figuring out all the packages they used, and how to use them, is an excellent starting point.
\nGood luck!
\nAlex
\n[quote=Kyle Schluns;74955]
\nHi, my name is Kyle. I am new to Kaggle and am interested in this specific competition to teach myself image processing. I searched through the forums and didn't really see any good beginner threads. I have some questions, if anyone could help me out.
\nI should also say that my overall goal here is to gain experience in image processing, with the specific purpose of being able to leverage this experience in a future job interview. So I ultimately would like to accomplish this project in a way that a software/research company would go about doing it. My specific career interests are in aerospace and astrophysics, so if anyone has an specific insights on how this project might relate to the aerospace industry, I'd love to hear it.
\nWhile these are the only questions I have right now, I will certainly have more once a conversation builds up. I hope to make this an ongoing thread about how to get started with this project.
\nThanks!
\n[/quote]
", "rawMarkdown": "", "votes": null }, { "id": "75482", "postDate": "04/29/2015 19:26:10", "content": "Hi Alexander,
\n\nWow, your insight is exactly what I am looking for. I appreciate the response.
\n\nTo respond to your first question, "do you want to learn a particular tool/language or do you want to learn/try interesting approaches to solving the problem?" - I guess at first, I was pretty focused on learning how to do image processing in C++, because I had gotten the sense that all the image processing jobs out there want you to know how to do it in C++. However, you make some great points that a good programmer knows how to be pragmatic. This brings me to question whether advanced image processing algorithms are more often written in C++ or something like Python? It seems like Python makes it easier to code, which results in less bugs and less lines of code, but is it not as fast as C++? The reason I ask is because I want to do this project in a way that would be valuable for my career. Sounds like Python is the way to go right now, especially because I don't have much experience in image processing algorithms in general, and those seem to be more valuable than the language you know how to implement them with (at this point).
\nI see what you are saying about the computer performance. Half a minute per image isn't so bad when you're getting started, but when you get to the point where you need to fully test the statistical confidence of your algorithm, you don't want to wait days for the answer. I think I'll just try to get the basics down with a laptop and if I can get through a decent amount of images with good results, I'll consider splitting up the work among more computer boxes.
\nThanks again for the advice. I will check out all those packages! I'm very excited about your response... it is exactly the kind of guidance I needed. I really appreciate it.
", "rawMarkdown": "", "votes": null }, { "id": "75489", "postDate": "04/29/2015 20:02:00", "content": "Hi Kyle,
\nSince you don't seem to be aware of this I thought I'd let you know that Python is an object-oriented programming language. Also, Python is marketable and suitable for learning image processing techniques without unnecessary complications due the language you are using.
\nKeep in mind that this competition is about making predictions, so image processing is only a part of the puzzle. You will need to consider what machine learning strategy you want to employ and develop an image processing pipeline to meet the needs of your methodology. Your methods need to be flexible so you can rapidly explore, discover, and iterate without rewriting everything from scratch, or, in this competition, repeating an operation on all of your images. Python is great for this.
", "rawMarkdown": "", "votes": null }, { "id": "75617", "postDate": "04/30/2015 12:26:31", "content": "That's a good point BPerea. Are many people choosing to make their code OOP? Or are people more commonly scripting their pipeline? Can you give an example of how you would use OOP for this project? I took a class on Java a while back and that's about my only OOP experience, so I kinda forget why it's useful.
\n\nI'm pretty excited about jumping into Python, after the two recommendations. I understand why everyone is using it. Just curious though, let's say a company was developing software to detect diabetic retinopathy. Would they use Python? This goes back to my earlier question about whether Python is faster than C++? From Alexander's response, it seems like the sheer amount of Python code is much less than that of C++ and it is less complex, which are great reasons to use Python over C++ for a small team project like this. However, I also understand that when you are a company you can afford to hire QA and product managers, which is probably how you handle the complexity of the code, once you write it in C++. Is my understanding correct?
\n\nSorry for all the questions, I'm just not only trying to get a grasp for how to do the project best, but I am also curious how that may differ or be the same as the way they do it in a company setting or research setting.
", "rawMarkdown": "", "votes": null }, { "id": "75638", "postDate": "04/30/2015 13:54:41", "content": "Kyle
\nI will try to answer your questions.
\nOne way of scripting would be to use an OOP framework for the pipeline. see eg scikit-learns http://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html . The point is that you use OOP to define the standard interface for transformations, and then the pipeline is a chain of objects which you script)
\n\nEssentially the basic image processing/machine learning.. algorithms are written in C++/C/ fortran for speed. But the whole point of python is that it allows you to call c code. So python is giving you a quick way of building a pipeline of fast c functions. So for instance opencv is an image processing library in C++ but with a python wrapper (see eg http://opencv-python-tutroals.readthedocs.org/en/latest/py_tutorials/py_tutorials.html) .
\nWhat people are telling you is that for this project you will probably not be writing a completely new image processing algorithm [ some new form of edge detection etc] but rather concatenating standard parts ( eg Convolution/Filtering/ Edge Detection / PCA / Thresholding / etc and similarly for machine learning algos)
\nIn a company, it is quite typical to use python /matlab to develop a new algorithm (try out different things etc), only when you have got a fully functioning debugged algorithm you code it in c++.
", "rawMarkdown": "", "votes": null }, { "id": "75641", "postDate": "04/30/2015 14:02:00", "content": "Okay, that makes sense. Thanks Sean!
", "rawMarkdown": "", "votes": null }, { "id": "76057", "postDate": "05/02/2015 11:39:07", "content": "Thanks Kyle for asking this question, it will be really helpful to those who are just now starting with machine learning. I am also new to this community and I have been learning about these platforms from past few days. What I learned is first you should spend some time on the finished competitions so that you could get the feel of how things work in an orderly manner or what are the minimum basic requirements. Then after you feel comfortable with the resources in your hand you can anytime try for the current competition.
\nThanks Alexander for such an honest and insightful response. Hope we guys learn more from you.
", "rawMarkdown": "", "votes": null }, { "id": "80643", "postDate": "06/02/2015 08:39:45", "content": "Can anyone please advice on how to use amazon S3 for this data with python (sickit , numpy , pandas) ?
\nany blog or tutorial anything ? , complete noob here !
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 75212, "author_name": "pbkaran", "author_url": "", "post_date": "04/28/2015 08:55:34", "content": "Welcome home!
\nFor me data volume is staggering, I havent started yet but I would start with aws. Also for speedy processing you may require GPU.
\nIf you never heard of CUDA, give it a shot. There is mooc on Udacity created by Nividia folks
\nhttps://www.udacity.com/course/intro-to-parallel-programming--cs344
", "votes": null, "replies": [] }, { "id": 75388, "author_name": "aizvorski", "author_url": "", "post_date": "04/29/2015 10:04:29", "content": "Hi Kyle,
\nI guess it comes down to whether you want to learn a particular tool or language (it sounds like you do) or whether you want to just try out interesting approaches to the problem without getting awkward tools get in the way.
\n"Is this project doable in C++?" - Probably not. I think this is a difficult enough problem that there's no need to make it harder by picking difficult tools as well. I know C++ quite well, but let's just say my code would be ten times larger and I would be still debugging it sometime next year if I was using that :) The focus should be very much on the algorithms not the implementation.
\n"What packages lend themselves well to this project?" - You might learn scikit-learn, scikit-image, statsmodels, pandas, theano, numpy, scipy ... Even just implementing one of the published papers on diabetic retinopathy detection using that toolset and getting useful predictions from that would be a nice learning curve. I almost hesitate to mention convnets because they are so highly hyped these days, but yeah, there's also convnets.
\n"leverage this experience in a future job interview" - if I was interviewing someone I would be impressed by good results, and by choosing and learning tools well suited to the task you were working on (since I do care about results). Implementing something like (for example) Gabor filters in C++ from scratch could make me think you are a good C++ programmer, but I would also have a negative impression of you as a non-pragmatic programmer (since this should be one line of code with skimage, and take maybe five minutes), and I probably wouldn't want to hire you. Sorry, but that's just how it is. CUDA may be better suited here, but I'd hope you're using PyCUDA, and even better using a package like theano and just adding a few CUDA kernels where there is nothing off-the-shelf that you can use.
\n"Is computer performance a big deal with this project?" - Yes, it lets you iterate faster. There's many a time when I've wanted to try something "simple", and quickly realized that "simple" takes half a minute per image, or in other words something like a day to work through a fraction of the dataset which is just barely large enough that it allows some degree of statistical confidence. Ouch. This is where running the same work split among a dozen c4.8xlarge boxes on Amazon and getting results in 15 minutes suddenly starts helping a lot. Sure, you can leave it running on your laptop for a few days, but...
\n"any recommended resources to help learn about image processing algorithms" - check out winning code from other Kaggle competitions, particularly ones dealing with images. It's usually posted in the forums. Figuring out all the packages they used, and how to use them, is an excellent starting point.
\nGood luck!
\nAlex
\n[quote=Kyle Schluns;74955]
\nHi, my name is Kyle. I am new to Kaggle and am interested in this specific competition to teach myself image processing. I searched through the forums and didn't really see any good beginner threads. I have some questions, if anyone could help me out.
\nI should also say that my overall goal here is to gain experience in image processing, with the specific purpose of being able to leverage this experience in a future job interview. So I ultimately would like to accomplish this project in a way that a software/research company would go about doing it. My specific career interests are in aerospace and astrophysics, so if anyone has an specific insights on how this project might relate to the aerospace industry, I'd love to hear it.
\nWhile these are the only questions I have right now, I will certainly have more once a conversation builds up. I hope to make this an ongoing thread about how to get started with this project.
\nThanks!
\n[/quote]
", "votes": null, "replies": [] }, { "id": 75482, "author_name": "kschluns", "author_url": "", "post_date": "04/29/2015 19:26:10", "content": "Hi Alexander,
\n\nWow, your insight is exactly what I am looking for. I appreciate the response.
\n\nTo respond to your first question, "do you want to learn a particular tool/language or do you want to learn/try interesting approaches to solving the problem?" - I guess at first, I was pretty focused on learning how to do image processing in C++, because I had gotten the sense that all the image processing jobs out there want you to know how to do it in C++. However, you make some great points that a good programmer knows how to be pragmatic. This brings me to question whether advanced image processing algorithms are more often written in C++ or something like Python? It seems like Python makes it easier to code, which results in less bugs and less lines of code, but is it not as fast as C++? The reason I ask is because I want to do this project in a way that would be valuable for my career. Sounds like Python is the way to go right now, especially because I don't have much experience in image processing algorithms in general, and those seem to be more valuable than the language you know how to implement them with (at this point).
\nI see what you are saying about the computer performance. Half a minute per image isn't so bad when you're getting started, but when you get to the point where you need to fully test the statistical confidence of your algorithm, you don't want to wait days for the answer. I think I'll just try to get the basics down with a laptop and if I can get through a decent amount of images with good results, I'll consider splitting up the work among more computer boxes.
\nThanks again for the advice. I will check out all those packages! I'm very excited about your response... it is exactly the kind of guidance I needed. I really appreciate it.
", "votes": null, "replies": [] }, { "id": 75489, "author_name": "bcperea", "author_url": "", "post_date": "04/29/2015 20:02:00", "content": "Hi Kyle,
\nSince you don't seem to be aware of this I thought I'd let you know that Python is an object-oriented programming language. Also, Python is marketable and suitable for learning image processing techniques without unnecessary complications due the language you are using.
\nKeep in mind that this competition is about making predictions, so image processing is only a part of the puzzle. You will need to consider what machine learning strategy you want to employ and develop an image processing pipeline to meet the needs of your methodology. Your methods need to be flexible so you can rapidly explore, discover, and iterate without rewriting everything from scratch, or, in this competition, repeating an operation on all of your images. Python is great for this.
", "votes": null, "replies": [] }, { "id": 75617, "author_name": "kschluns", "author_url": "", "post_date": "04/30/2015 12:26:31", "content": "That's a good point BPerea. Are many people choosing to make their code OOP? Or are people more commonly scripting their pipeline? Can you give an example of how you would use OOP for this project? I took a class on Java a while back and that's about my only OOP experience, so I kinda forget why it's useful.
\n\nI'm pretty excited about jumping into Python, after the two recommendations. I understand why everyone is using it. Just curious though, let's say a company was developing software to detect diabetic retinopathy. Would they use Python? This goes back to my earlier question about whether Python is faster than C++? From Alexander's response, it seems like the sheer amount of Python code is much less than that of C++ and it is less complex, which are great reasons to use Python over C++ for a small team project like this. However, I also understand that when you are a company you can afford to hire QA and product managers, which is probably how you handle the complexity of the code, once you write it in C++. Is my understanding correct?
\n\nSorry for all the questions, I'm just not only trying to get a grasp for how to do the project best, but I am also curious how that may differ or be the same as the way they do it in a company setting or research setting.
", "votes": null, "replies": [] }, { "id": 75638, "author_name": "seanv507", "author_url": "", "post_date": "04/30/2015 13:54:41", "content": "Kyle
\nI will try to answer your questions.
\nOne way of scripting would be to use an OOP framework for the pipeline. see eg scikit-learns http://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html . The point is that you use OOP to define the standard interface for transformations, and then the pipeline is a chain of objects which you script)
\n\nEssentially the basic image processing/machine learning.. algorithms are written in C++/C/ fortran for speed. But the whole point of python is that it allows you to call c code. So python is giving you a quick way of building a pipeline of fast c functions. So for instance opencv is an image processing library in C++ but with a python wrapper (see eg http://opencv-python-tutroals.readthedocs.org/en/latest/py_tutorials/py_tutorials.html) .
\nWhat people are telling you is that for this project you will probably not be writing a completely new image processing algorithm [ some new form of edge detection etc] but rather concatenating standard parts ( eg Convolution/Filtering/ Edge Detection / PCA / Thresholding / etc and similarly for machine learning algos)
\nIn a company, it is quite typical to use python /matlab to develop a new algorithm (try out different things etc), only when you have got a fully functioning debugged algorithm you code it in c++.
", "votes": null, "replies": [] }, { "id": 75641, "author_name": "kschluns", "author_url": "", "post_date": "04/30/2015 14:02:00", "content": "Okay, that makes sense. Thanks Sean!
", "votes": null, "replies": [] }, { "id": 76057, "author_name": "transcending", "author_url": "", "post_date": "05/02/2015 11:39:07", "content": "Thanks Kyle for asking this question, it will be really helpful to those who are just now starting with machine learning. I am also new to this community and I have been learning about these platforms from past few days. What I learned is first you should spend some time on the finished competitions so that you could get the feel of how things work in an orderly manner or what are the minimum basic requirements. Then after you feel comfortable with the resources in your hand you can anytime try for the current competition.
\nThanks Alexander for such an honest and insightful response. Hope we guys learn more from you.
", "votes": null, "replies": [] }, { "id": 80643, "author_name": "singhanurag", "author_url": "", "post_date": "06/02/2015 08:39:45", "content": "Can anyone please advice on how to use amazon S3 for this data with python (sickit , numpy , pandas) ?
\nany blog or tutorial anything ? , complete noob here !
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "74955": "", "75212": "", "75388": "", "75482": "", "75489": "", "75617": "", "75638": "", "75641": "", "76057": "", "80643": "" }, "source": "meta" }