{ "id": 12842, "title": "Starter code for leaderboard score of ~0.46.", "url": "/competitions/diabetic-retinopathy-detection/discussion/12842", "author_name": "", "post_date": "2015-03-16T23:22:01.660Z", "votes": 47, "comment_count": 76, "views": 34494, "content": "

Hello, 

\n

I just put up code that I used to get a leaderboard score of ~0.46412, and I would like to hear feedback on it.  Feel free to use it, and I am open to feedback.  I am sure many parts of it can be improved. 

\n

https://github.com/hoytak/diabetic-retinopathy-code

\n

This code uses the ImageMagick convert tool to preprocess the images, then uses the neural net toolkit (basically cxxnet) and the boosted regression trees in Dato's Graphlab create toolkits to build the classifier.  

\n

Main Ideas:

\n
    \n
  1. Use ImageMagick's convert tool to trim off the blank space to the sides of the images, then pad them so that they are all 256x256. Thus the eye is always centered with edges against the edges of the image. I avoided scaling the images to improve the neural net performance.
    2. \n
  2. Create multiple versions of each image varying by hue and contrast and white balance.  This step can be simplified to get to a quick though less accurate solution.
    3. \n
  3. Duplicate each class so each class is represented equally, then shuffle the data.
    4. \n
  4. Train several neural nets, one trained to predict the level membership, and another 4 to distinguish 0 vs 1-4, 0-1 vs. 2-4, 0-2 vs. 3-4, and 0-3 vs. 4.
    5. \n
  5. For each image, extract both class predictions and the values of the final neural net layer. Pool all of these features across models and variations of the same images.
    6. \n
  6. Train boosted regression trees on these pooled features to predict the level.
    7. \n
  7. Round to the nearest integer as the class prediction.
    8. \n
\n

I know that each of these steps could definitely be tuned for greater accuracy, and I would love to hear your feedback. In particular, I don't think the neural net architecture is optimal for these images. 

", "messages": [ { "id": "66564", "postDate": "03/16/2015 23:22:01", "content": "

Hello, 

\n

I just put up code that I used to get a leaderboard score of ~0.46412, and I would like to hear feedback on it.  Feel free to use it, and I am open to feedback.  I am sure many parts of it can be improved. 

\n

https://github.com/hoytak/diabetic-retinopathy-code

\n

This code uses the ImageMagick convert tool to preprocess the images, then uses the neural net toolkit (basically cxxnet) and the boosted regression trees in Dato's Graphlab create toolkits to build the classifier.  

\n

Main Ideas:

\n
    \n
  1. Use ImageMagick's convert tool to trim off the blank space to the sides of the images, then pad them so that they are all 256x256. Thus the eye is always centered with edges against the edges of the image. I avoided scaling the images to improve the neural net performance.
    2. \n
  2. Create multiple versions of each image varying by hue and contrast and white balance.  This step can be simplified to get to a quick though less accurate solution.
    3. \n
  3. Duplicate each class so each class is represented equally, then shuffle the data.
    4. \n
  4. Train several neural nets, one trained to predict the level membership, and another 4 to distinguish 0 vs 1-4, 0-1 vs. 2-4, 0-2 vs. 3-4, and 0-3 vs. 4.
    5. \n
  5. For each image, extract both class predictions and the values of the final neural net layer. Pool all of these features across models and variations of the same images.
    6. \n
  6. Train boosted regression trees on these pooled features to predict the level.
    7. \n
  7. Round to the nearest integer as the class prediction.
    8. \n
\n

I know that each of these steps could definitely be tuned for greater accuracy, and I would love to hear your feedback. In particular, I don't think the neural net architecture is optimal for these images. 

", "rawMarkdown": "", "votes": null }, { "id": "66589", "postDate": "03/17/2015 00:43:55", "content": "

Thank you. Could you share the cross validation kappa score of this approach?

", "rawMarkdown": "", "votes": null }, { "id": "66601", "postDate": "03/17/2015 01:17:26", "content": "

Hi RCarson,

\n

I have not yet run full cross-validation on this model yet to get a final kappa score.  My goal was to get something done quickly, but running cross validation and tuning the intermediate parameters would be the next step and likely give a much better result.  

\n

My holdout validation RMSE for the regression problem at the end to predict the level was around 0.63 if that helps.

\n

-- Hoyt

", "rawMarkdown": "", "votes": null }, { "id": "66607", "postDate": "03/17/2015 01:36:13", "content": "

Great! my model gets rmse is 1.8 and LB 0.09. Thank you.

", "rawMarkdown": "", "votes": null }, { "id": "66626", "postDate": "03/17/2015 03:50:25", "content": "

Thanks for the great shared code!

\n

Just one thing which I hope you could clarify (thanks in advance): I noticed that your code used the Graphlab Create, which is not fully open source. Are you using any non-open source elements of the Graphlab Create in your code or not?

\n

Thanks!

\n

Best wishes,

\n

Shize

\n

[quote=Hoyt Koepke;66564]

\n

Hello, 

\n

I just put up code that I used to get a leaderboard score of ~0.46412, and I would like to hear feedback on it.  Feel free to use it, and I am open to feedback.  I am sure many parts of it can be improved. 

\n

https://github.com/hoytak/diabetic-retinopathy-code

\n

This code uses the ImageMagick convert tool to preprocess the images, then uses the neural net toolkit (basically cxxnet) and the boosted regression trees in Dato's Graphlab create toolkits to build the classifier.  

\n

Main Ideas:

\n
    \n
  1. Use ImageMagick's convert tool to trim off the blank space to the sides of the images, then pad them so that they are all 256x256. Thus the eye is always centered with edges against the edges of the image. I avoided scaling the images to improve the neural net performance.
    2. \n
  2. Create multiple versions of each image varying by hue and contrast and white balance.  This step can be simplified to get to a quick though less accurate solution.
    3. \n
  3. Duplicate each class so each class is represented equally, then shuffle the data.
    4. \n
  4. Train several neural nets, one trained to predict the level membership, and another 4 to distinguish 0 vs 1-4, 0-1 vs. 2-4, 0-2 vs. 3-4, and 0-3 vs. 4.
    5. \n
  5. For each image, extract both class predictions and the values of the final neural net layer. Pool all of these features across models and variations of the same images.
    6. \n
  6. Train boosted regression trees on these pooled features to predict the level.
    7. \n
  7. Round to the nearest integer as the class prediction.
    8. \n
\n

I know that each of these steps could definitely be tuned for greater accuracy, and I would love to hear your feedback. In particular, I don't think the neural net architecture is optimal for these images. 

\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "66654", "postDate": "03/17/2015 07:31:20", "content": "

Hello Shize,

\n

That's a fair question. I used Dato's Graphlab Create to prototype different models to quickly get at a good solution, which is what I've posted. However, everything that I ended up using is either open source or can be substituted with another open source package. The SFrames is open sourced in dato-core, the GLC deep learning toolkit generates config files that are compatible with CXXNet, and the boosted tree regression can be translated to use xgboost.

\n

Thanks!

\n

-- Hoyt

\n

[quote=Shize Su;66626]

\n

Thanks for the great shared code!

\n

Just one thing which I hope you could clarify (thanks in advance): I noticed that your code used the Graphlab Create, which is not fully open source. Are you using any non-open source elements of the Graphlab Create in your code or not?

\n

Thanks!

\n

Best wishes,

\n

Shize

\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "67948", "postDate": "03/24/2015 11:50:00", "content": "

Hello,

\n

Running the create_image_sframes.py script I get the following errors:

\n

PROGRESS: Read 163807 images in 745.984 secs speed: 0 file/sec
PROGRESS: Unknown error reading image file: /data2/DRD/processed/run-hue-2/test/18344_right.jpeg
PROGRESS: Unknown error reading image file: /data2/DRD/processed/run-stretch/train/4400_right.jpeg
PROGRESS: Unknown error reading image file: /data2/DRD/processed/run-contrast-2/train/6210_left.jpeg
PROGRESS: Unknown error reading image file: /data2/DRD/processed/run-hue-2/train/27213_left.jpeg
PROGRESS: Unknown error reading image file: /data2/DRD/processed/run-sat-1/test/8671_left.jpeg

\n\n\n

any idea where this might come from?

\n

thanks.

", "rawMarkdown": "", "votes": null }, { "id": "67987", "postDate": "03/24/2015 15:34:17", "content": "

[quote=knarfben;67948]

\n

Hello,

\n

Running the create_image_sframes.py script I get the following errors:

\n

PROGRESS: Read 163807 images in 745.984 secs speed: 0 file/sec
PROGRESS: Unknown error reading image file: /data2/DRD/processed/run-hue-2/test/18344_right.jpeg
PROGRESS: Unknown error reading image file: /data2/DRD/processed/run-stretch/train/4400_right.jpeg
PROGRESS: Unknown error reading image file: /data2/DRD/processed/run-contrast-2/train/6210_left.jpeg
PROGRESS: Unknown error reading image file: /data2/DRD/processed/run-hue-2/train/27213_left.jpeg
PROGRESS: Unknown error reading image file: /data2/DRD/processed/run-sat-1/test/8671_left.jpeg

\n

any idea where this might come from?

\n

thanks.

\n

[/quote]

\n\n

ok.... not enough disk space, had to change GRAPHLAB_CACHE_FILE_LOCATIONS

", "rawMarkdown": "", "votes": null }, { "id": "67989", "postDate": "03/24/2015 15:45:20", "content": "

Thank you Hoyt! it takes my computer a whole week to run this benchmark and it gets 0.44. In the following months I'm going to figure out what is going on and replace all the Dato tool with open source tool. It will be a great learning process!

\n

Again, Dato is really too good for kaggle. Hope you could publish more analysis, ipython notebook style tutorial, to help us learn your thought process, besides generating full submission. I think most kagglers will be more happy to use Dato as an analysis tool. It is indeed really powerful. Great work!

", "rawMarkdown": "", "votes": null }, { "id": "68033", "postDate": "03/24/2015 17:58:21", "content": "

@rcarson I ran this code for only the original dataset no hue, sat, ... processing and got 0.045. I was just curious what I could get in a day or so. Now, I am running the full set and expect me near you on the LB sometime this week :)

\n

As for the code, there is no magic to it. I actually think the network is so primitive (much more shallow (5 vs 2 conv layers) than the starter code based on cxxnet on the plankton competition though the fully connected layer has one more hidden layer).

\n

I think, the almost brute-force style pre-processing of color spaces, trying to balance the dataset, and extracting features from almost all kinds of class combinations are responsible for the higher score. This is evident in my score of only 0.045 on the original dataset (although balanced) vs. 0.44 or 0.46 for all this brute force. Overall, the data pre processing is not systematic rather brute-force and the network architecture is too shallow.

\n

Mind you although graphlab create net could be optimized, under the hood it is still cxxnet. 

\n

It is a good starting place regardless. 

", "rawMarkdown": "", "votes": null }, { "id": "68057", "postDate": "03/24/2015 19:33:59", "content": "

Thank you Hoyt. A kind reminder, multiple accounts are against Kaggle's rule. Luckily your new account doesn't enter this contest, so don't! and I suggest you delete the new account. You certainly don't need that. This happens to others who are new to kaggle.

", "rawMarkdown": "", "votes": null }, { "id": "68061", "postDate": "03/24/2015 19:41:37", "content": "

@rcarson -- Thank you for pointing this out.  I accidently logged in with the wrong google+ account and didn't even notice!  I closed that one and will just stick with this one. 

\n

In case the above post gets deleted, here is the text:

\n

---------------------------------------

\n

Hello @deep and @rcarson,

\n

I'm really glad you found my code helpful. I will try to make a notebook regarding my thought process here. I'm at work now, so I can't write much, but I'll try to fill in more when I get home.

\n

I agree with @deep -- this method is pretty brute force. The guiding principle was to create more observations that varied in the ways you want the NN to ignore. There are also a lot of different things that could be tuned and improved, especially with the NN architectures. I'm pretty new to neural nets, so I'll try playing around with it in the way you suggested.

\n

I think the other thing to realize here is that this is an ordinal regression problem rather than a classification problem. This motivated using the boosted regression trees over the activation levels in the final layers of the NN at the end to produce the result rather than using the straight NN to classify the images.

\n

@rcarson -- FYI, a lot of the Dato stuff is open source -- see https://github.com/dato-code/Dato-Core.

\n

@deep -- I have also found that sometimes what I'm doing here would overfit the training data, resulting in poorer predictions. E.g., it seemed like the NN overfit for the 0,1,2 vs 3,4 case based on the training / validation difference, and leaving that one out of the final regression problem resulted in a slightly better score. Just one thing I found.

", "rawMarkdown": "", "votes": null }, { "id": "68570", "postDate": "03/27/2015 13:25:48", "content": "

[quote=Hoyt Koepke;66564]

\n

Hello, 

\n

I just put up code that I used to get a leaderboard score of ~0.46412, and I would like to hear feedback on it.  Feel free to use it, and I am open to feedback.  I am sure many parts of it can be improved. 

\n

https://github.com/hoytak/diabetic-retinopathy-code

\n

This code uses the ImageMagick convert tool to preprocess the images, then uses the neural net toolkit (basically cxxnet) and the boosted regression trees in Dato's Graphlab create toolkits to build the classifier.  

\n

Main Ideas:

\n
    \n
  1. Use ImageMagick's convert tool to trim off the blank space to the sides of the images, then pad them so that they are all 256x256. Thus the eye is always centered with edges against the edges of the image. I avoided scaling the images to improve the neural net performance.
    2. \n
  2. Create multiple versions of each image varying by hue and contrast and white balance.  This step can be simplified to get to a quick though less accurate solution.
    3. \n
  3. Duplicate each class so each class is represented equally, then shuffle the data.
    4. \n
  4. Train several neural nets, one trained to predict the level membership, and another 4 to distinguish 0 vs 1-4, 0-1 vs. 2-4, 0-2 vs. 3-4, and 0-3 vs. 4.
    5. \n
  5. For each image, extract both class predictions and the values of the final neural net layer. Pool all of these features across models and variations of the same images.
    6. \n
  6. Train boosted regression trees on these pooled features to predict the level.
    7. \n
  7. Round to the nearest integer as the class prediction.
    8. \n
\n

I know that each of these steps could definitely be tuned for greater accuracy, and I would love to hear your feedback. In particular, I don't think the neural net architecture is optimal for these images. 

\n

[/quote]

\n\n

Hello Hoyt

\n\n

I ran into the following issue while running create_image_sframes.py:

\n

[INFO] Starting new HTTP connection (1): d3dou3jr4nway6.cloudfront.net
[INFO] Starting new HTTP connection (1): d3dou3jr4nway6.cloudfront.net
Get roughly equal class representation by duplicating the different levels.
Do a poor mans random shuffle
Unable to reach server for 3 consecutive pings. Server is considered dead. Please exit and restart.
Unable to reach server for 4 consecutive pings. Server is considered dead. Please exit and restart.
Traceback (most recent call last):
File "/home/ubuntu/drd/create_image_sframes.py", line 47, in <module>
X_train = X_train.sort("_random_")
File "/usr/local/lib/python2.7/dist-packages/graphlab/data_structures/sframe.py", line 4988, in sort
return SFrame(_proxy=self.__proxy__.sort(sort_column_names, sort_column_orders))
File "/usr/local/lib/python2.7/dist-packages/graphlab/cython/context.py", line 39, in __exit__
raise exc_type(exc_value)
RuntimeError: Communication Failure: 113.
Unable to reach server for 5 consecutive pings. Server is considered dead. Please exit and restart.
Unable to reach server for 6 consecutive pings. Server is considered dead. Please exit and restart.
Unable to reach server for 7 consecutive pings. Server is considered dead. Please exit and restart.
Unable to reach server for 8 consecutive pings. Server is considered dead. Please exit and restart.
Unable to reach server for 9 consecutive pings. Server is considered dead. Please exit and restart.
[INFO] Stopping the server connection.

\n

Any idea where this might come from? I ran the script many times and always end up with this error, although previous communication with the server were ok....

\n\n

thanks 

\n

Frank

", "rawMarkdown": "", "votes": null }, { "id": "68602", "postDate": "03/27/2015 18:02:14", "content": "

[quote=knarfben;68570]

\n

Hello Hoyt

\n

I ran into the following issue while running create_image_sframes.py:

\n

[INFO] Starting new HTTP connection (1): d3dou3jr4nway6.cloudfront.net
[INFO] Starting new HTTP connection (1): d3dou3jr4nway6.cloudfront.net
Get roughly equal class representation by duplicating the different levels.
Do a poor mans random shuffle
Unable to reach server for 3 consecutive pings. Server is considered dead. Please exit and restart.
Unable to reach server for 4 consecutive pings. Server is considered dead. Please exit and restart.
Traceback (most recent call last):
File "/home/ubuntu/drd/create_image_sframes.py", line 47, in <module>
X_train = X_train.sort("_random_")
File "/usr/local/lib/python2.7/dist-packages/graphlab/data_structures/sframe.py", line 4988, in sort
return SFrame(_proxy=self.__proxy__.sort(sort_column_names, sort_column_orders))
File "/usr/local/lib/python2.7/dist-packages/graphlab/cython/context.py", line 39, in __exit__
raise exc_type(exc_value)
RuntimeError: Communication Failure: 113.
Unable to reach server for 5 consecutive pings. Server is considered dead. Please exit and restart.
Unable to reach server for 6 consecutive pings. Server is considered dead. Please exit and restart.
Unable to reach server for 7 consecutive pings. Server is considered dead. Please exit and restart.
Unable to reach server for 8 consecutive pings. Server is considered dead. Please exit and restart.
Unable to reach server for 9 consecutive pings. Server is considered dead. Please exit and restart.
[INFO] Stopping the server connection.

\n

Any idea where this might come from? I ran the script many times and always end up with this error, although previous communication with the server were ok....

\n

thanks 

\n

Frank

\n

[/quote]

\n

Hello Frank, I also hit this error a couple of times -- hence the "if not os.path.exists(...)" line at the top, but running it a few times seems to get through it.  Sorting the SFrame with the images in it seems really expensive, and all the disk IO is causing something to time out. I filed a bug report on this, so hopefully it will get fixed in the next Graphlab create version.

\n

I'm also working on doing all of the operations up to this point with filenames instead of the actual images, which would be much faster.  

\n

Does it complete any of the rounds before crashing, or does it simply crash on the first one every time?  

", "rawMarkdown": "", "votes": null }, { "id": "68603", "postDate": "03/27/2015 18:07:58", "content": "", "rawMarkdown": "", "votes": null }, { "id": "68903", "postDate": "03/29/2015 21:32:30", "content": "

The g2.2xlarge instance in AWS comes with 60GB of hard disk space. Could you let me know how do you increase this space to accommodate the dataset we are dealing with here?

", "rawMarkdown": "", "votes": null }, { "id": "68904", "postDate": "03/29/2015 21:39:20", "content": "

Create and mount a volume of sufficient size. I think I used 300GB but that was more than I needed.

", "rawMarkdown": "", "votes": null }, { "id": "68914", "postDate": "03/29/2015 22:39:55", "content": "

Did you create a EBS volume? I am new to AWS. If you store something in the 60GB instance store that comes with g2.2xlarge, does that data get deleted when you stop your instance? **If so, why would anyone ever use that store**? Second question I have is that when I log into my machine, how do I know if I am accessing the instance store or the EBS? i.e., given a path like /home/ubuntu how to know is this directory is under instance store or EBS?

\n\n

References:

\n

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonEBS.html

\n

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-using-volumes.html

\n

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Storage.html

\n

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html

", "rawMarkdown": "", "votes": null }, { "id": "68915", "postDate": "03/29/2015 22:57:29", "content": "

Yes it's an ebs volume. The data in the root device will disappear by default unless you select an option to make it persist when you create the AMI. The idea is to use it for software rather than data. 

\n

To create a volume and attach it to a running instance, select create volume at the volumes screen, and then right click to and select attach volume. It will probably say it's attaching to /dev/sdf but in reality it will be on /dev/xvdf on ubuntu. From your instance, execute

\n\n

sudo mkfs -t ext4 /dev/xvdf  # (only the first time you use the volume).

\n

sudo mkdir /data #  (replace /data with wherever you want to mount the volume).

\n

sudo mount /dev/xvdf /data

", "rawMarkdown": "", "votes": null }, { "id": "68947", "postDate": "03/30/2015 00:23:41", "content": "

hey i have one more question. i created a ebs volume and mounted it to /data but am unable to save anything in it because i lack the permissions. 

\n

ubuntu:~$ ls -all /data
total 24
drwxr-xr-x 3 root root 4096 Mar 30 00:04 .
drwxr-xr-x 24 root root 4096 Mar 30 00:12 ..

\n

what can i do so that root becomes ubuntu? whoami gives ubuntu.

", "rawMarkdown": "", "votes": null }, { "id": "68963", "postDate": "03/30/2015 02:29:50", "content": "

sudo chown ubuntu /data

\n

chmod if necessary

", "rawMarkdown": "", "votes": null }, { "id": "68980", "postDate": "03/30/2015 05:28:02", "content": "

thanks. that worked.

", "rawMarkdown": "", "votes": null }, { "id": "68986", "postDate": "03/30/2015 07:05:40", "content": "

@Hoyt, just finished running your btb code. I like the intuition of pulling all the features and use GBT for classification instead of just straight softmax from the CNN.

\n

As you mentioned, the last two comparisons for 0-1-2 vs. 3-4 and 0-1-2-3 vs 4 overfit a lot (training: 0.91 and 0.95 vs. validation around 0.64 and 0.65) for this shallow network. This tells me that the network training params aren't quite optimal. For instance, adding regularization and adaptive learning rate could help.

\n

Great work overall and thanks for sharing!

", "rawMarkdown": "", "votes": null }, { "id": "69072", "postDate": "03/30/2015 20:00:08", "content": "

@Hoyt, can you say a few sentences about why you decided to go with just two convolutional layers and a stride of 4? I'm mostly trying to understand why two conv layers might be considered sufficient and whether a stride of 4 (>1) was mostly selected to speed things up.

\n

This thread has been really instructive to me - thanks to all of you nice folks!

", "rawMarkdown": "", "votes": null }, { "id": "69074", "postDate": "03/30/2015 20:16:46", "content": "

@small yellow duck --

\n

To be honest, I'm by no means the expert in this thread on deep learning -- others probably have better advice to share, but I would be happy to tell you what I was thinking. 

\n

I based the architecture off of the configuration for imagenet, but I cut down a number of the parameters and the number of convolution layers.  The main reason for only choosing 2 convolution layers was that I didn't feel this problem needed as much location invariance as the imagenet task, which had more variance in where the relevant features were located.  After the processing by the imagemagick convert tool, a lot of relevant features appear in similar places in the image, so I didn't think as many convolution layers would be needed to capture it.

\n

As for the stride, I found that smaller strides generated out-of-memory errors on my GPU (a GTX 780), so that's the only reason for a stride of 4. 

", "rawMarkdown": "", "votes": null }, { "id": "69703", "postDate": "04/05/2015 10:01:15", "content": "

Hello Hoyt

\n

Thanks for sharing ~~

\n

However, I ran into the following issue while running create_nn_model.py:

\n

File "create_nn_model.py", line 85, in <module>
mean_image = X_train["image"].mean()

\n

graphlab.toolkits._main.ToolkitError: Cannot perform sum or average over images of different sizes. Found images of total size (ie. width * height * channels) of both 196608 and 65536. Please use graplab.image_analysis.resize() to make images a uniform size.

\n

Any idea where this might come from? 

\n

thanks

", "rawMarkdown": "", "votes": null }, { "id": "69772", "postDate": "04/06/2015 06:29:35", "content": "

Since 196608 = 65536*3, it seems to me you are mixing 256x256 RGB (3 channels) and 256x256 grey level (1 channel) images, while the NN expects images with the same size AND number of channels.

\n

Just a guess...

", "rawMarkdown": "", "votes": null }, { "id": "69782", "postDate": "04/06/2015 08:58:59", "content": "

hi 

\n

how to start this project

", "rawMarkdown": "", "votes": null }, { "id": "70063", "postDate": "04/09/2015 00:30:36", "content": "

Another Dato "benchmark".  What is the purpose of posting code that puts people in the top ~7% with 3 months left in the competition (this is a sincere question)?

", "rawMarkdown": "", "votes": null }, { "id": "70065", "postDate": "04/09/2015 01:07:31", "content": "

Better than 3 days before contest end (like in Plankton).

\n

[quote=John Martinez;70063]

\n

Another Dato "benchmark".  What is the purpose of posting code that puts people in the top ~7% with 3 months left in the competition (this is a sincere question)?

\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "70066", "postDate": "04/09/2015 01:13:57", "content": "

[quote=John Martinez;70063]

\n

Another Dato "benchmark".  What is the purpose of posting code that puts people in the top ~7% with 3 months left in the competition (this is a sincere question)?

\n

[/quote]

\n

It may be the top ~7% currently but as others have mentioned it's not overly advanced and surely anyone who spent the length of the competition on their work would've bested it anyways.  Additionally, I think we can all agree that openly discussing models that detect diseases has benefits outside of the competition alone :-)

", "rawMarkdown": "", "votes": null }, { "id": "70071", "postDate": "04/09/2015 02:30:17", "content": "

[quote=J Kolb;70066]

\n

[quote=John Martinez;70063]

\n

Another Dato "benchmark".  What is the purpose of posting code that puts people in the top ~7% with 3 months left in the competition (this is a sincere question)?

\n

[/quote]

\n

It may be the top ~7% currently but as others have mentioned it's not overly advanced and surely anyone who spent the length of the competition on their work would've bested it anyways.  Additionally, I think we can all agree that openly discussing models that detect diseases has benefits outside of the competition alone :-)

\n

[/quote]

\n

Just saying, there is currently a user (not me) that is at 0.38 that has 59 submissions.  Assuming that "surely anyone who spent the length of the competition on their work would've bested it anyways" is an unfair assumption in my estimation.  Maybe that user did they best they possibly could and their work was original and may have been good enough to squeak into the top 25%.  Seeing a score posted in the forums that trumps your work has to be extremely disheartening.  

", "rawMarkdown": "", "votes": null }, { "id": "70072", "postDate": "04/09/2015 02:38:11", "content": "

It's like you're working on a puzzle, and you're really absorbed in it and making progress, and then someone comes along and tells you the answer.

", "rawMarkdown": "", "votes": null }, { "id": "70078", "postDate": "04/09/2015 03:49:40", "content": "

[quote=James King;70072]

\n

It's like you're working on a puzzle, and you're really absorbed in it and making progress, and then someone comes along and tells you the answer.

\n

[/quote]

\n

I can understand the debate over how it affects people's scores, but this analogy doesn't really apply because if it's solely about working on the puzzle, no one makes you open the thread, much less go to his github, set up Graphlab, and do a couple days of processing and machine learning.  He stated in the title what was in his thread.

", "rawMarkdown": "", "votes": null }, { "id": "70079", "postDate": "04/09/2015 03:57:56", "content": "

John,

\n

Assuming the converse is true, and the given person would not be able to beat the score of .46 after four months of work, it gives them the ability to peek into what other people are doing, if they wish, and not spend those four months without obtaining that result.  They don't have to, however, and can still try things on their own, but at least they have that option.  Also, it's well before the deadline so it's not like people are getting upended last minute.  I'm not going to go further into whether it is disheartening vs. enlightening because I think this has already been beaten like a dead horse and there is obviously cases where each is true.  I do want to also reiterate that this is an active research topic.  Would you ask a research group not to publish their results because another research group is still working on the same problem?

", "rawMarkdown": "", "votes": null }, { "id": "70083", "postDate": "04/09/2015 04:33:19", "content": "

[quote=J Kolb;70079]

\n

John,

\n

Assuming the converse is true, and the given person would not be able to beat the score of .46 after four months of work, it gives them the ability to peek into what other people are doing, if they wish, and not spend those four months without obtaining that result.  They don't have to, however, and can still try things on their own, but at least they have that option.  Also, it's well before the deadline so it's not like people are getting upended last minute.  I'm not going to go farther into whether it is disheartening vs. enlightening because I think this has already been beaten like a dead horse and there is obviously cases where each is true.  I do want to also reiterate that this is an active research topic.  Would you ask a research group not to publish their results because another research group is still working on the same problem?

\n

[/quote]

\n

My objections have nothing to do with publishing results. At the heart of scientific discovery is collaboration. It’s what makes this site and science in general, pretty awesome. My objections are with a company whoring out their products at the expense of others hard work. Users have posted successful models, plugging the same company’s tools, in this, the Otto challenge, and as inversion mentioned, one 3 days before the deadline in the Plankton competition, which easily put you in the top 25%. I am aware I have no results on this site to speak of and am not an influential member of this community, so if my objections are unwarranted, I will certainly acquiesce but I must remind those reading of the Law of Unintended Consequences; I had never heard of Dato until a few weeks ago. Maybe that was a good thing for them …

", "rawMarkdown": "", "votes": null }, { "id": "70084", "postDate": "04/09/2015 04:34:41", "content": "

[quote=John Martinez;70063]

\n

Another Dato "benchmark".  What is the purpose of posting code that puts people in the top ~7% with 3 months left in the competition (this is a sincere question)?

\n

[/quote]

\n

Hi John, thanks for your question, and hopefully I can give you a satisfactory answer.  

I posted the code and write-up of my insights into the problem in the hope that others would be able to build on my ideas, incorporate their own, and hopefully come out ahead of either approach individually.  I've been a researcher in machine learning for over 9 years -- first in my PhD program, and now at Dato -- and most of my research advances have benefitted greatly from open collaboration and an open exchange of ideas.  As a result, I'm always free to share what insights I have in the hopes someone can build on them.  I know this isn't necessarily academia, but this problem is still an open research problem, and there are several research papers posted elsewhere in the forum describing techniques that allegedly get a much better kappa score than my method.  

I simply don't have the free time needed to win this contest, so that's not my aim; rather, I see it as an interesting and very difficult research problem that I want to feel like I can contribute to.  And, from experience, I'll bet the winning idea is likely going to be a combination of openly shared ideas, published research, and original insights -- but it definitely won't be entirely original ideas. 

", "rawMarkdown": "", "votes": null }, { "id": "70085", "postDate": "04/09/2015 04:51:48", "content": "

[quote=Hoyt Koepke;68061]

\n

I think the other thing to realize here is that this is an ordinal regression problem rather than a classification problem. This motivated using the boosted regression trees over the activation levels in the final layers of the NN at the end to produce the result rather than using the straight NN to classify the images.

\n

[/quote]

\n

I'm not sure I understand your thought process here.  Wouldn't it be possible to use the Neural net for a regression problem and then round the result to the nearest value of 0, 1, 2, 3, or 4 instead of doing a one-vs-all classification? And then you wouldn't use the boosted trees at all.  What is the benefit of using the trees then?  Does using the NN for regression not work well in practice in this situation?

", "rawMarkdown": "", "votes": null }, { "id": "70086", "postDate": "04/09/2015 05:00:15", "content": "

[quote=John Martinez;70083]

\n

[quote=J Kolb;70079]

\n

John,

\n

Assuming the converse is true, and the given person would not be able to beat the score of .46 after four months of work, it gives them the ability to peek into what other people are doing, if they wish, and not spend those four months without obtaining that result.  They don't have to, however, and can still try things on their own, but at least they have that option.  Also, it's well before the deadline so it's not like people are getting upended last minute.  I'm not going to go farther into whether it is disheartening vs. enlightening because I think this has already been beaten like a dead horse and there is obviously cases where each is true.  I do want to also reiterate that this is an active research topic.  Would you ask a research group not to publish their results because another research group is still working on the same problem?

\n

[/quote]

\n

My objections have nothing to do with publishing results. At the heart of scientific discovery is collaboration. It’s what makes this site and science in general, pretty awesome. My objections are with a company whoring out their products at the expense of others hard work. Users have posted successful models, plugging the same company’s tools, in this, the Otto challenge, and as inversion mentioned, one 3 days before the deadline in the Plankton competition, which easily put you in the top 25%. I am aware I have no results on this site to speak of and am not an influential member of this community, so if my objections are unwarranted, I will certainly acquiesce but I must remind those reading of the Law of Unintended Consequences; I had never heard of Dato until a few weeks ago. Maybe that was a good thing for them …

\n

[/quote]

\n

Hi John, 

\n

I hope you don't think that poorly of Dato :-).  We're a fairly new startup company originally formed in part to support Graphlab, an open source academic project, and now we have expanded to provide other tools for machine learning research and general data science.  Because of our academic roots, we've made sure our stuff here is either open source or completely free for academic/non-commercial use to benefit research-oriented endeavors like this.  As mentioned earlier, all of my code can either use our open source offering or be translated into other open source tools.  I used it since it's (obviously) what I'm most familiar with, so I find it easier to quickly prototype ideas that way.   

", "rawMarkdown": "", "votes": null }, { "id": "70088", "postDate": "04/09/2015 05:06:51", "content": "

[quote=J Kolb;70085]

\n

[quote=Hoyt Koepke;68061]

\n

I think the other thing to realize here is that this is an ordinal regression problem rather than a classification problem. This motivated using the boosted regression trees over the activation levels in the final layers of the NN at the end to produce the result rather than using the straight NN to classify the images.

\n

[/quote]

\n

I'm not sure I understand your thought process here.  Wouldn't it be possible to use the Neural net for a regression problem and then round the result to the nearest value of 0, 1, 2, 3, or 4 instead of doing a one-vs-all classification? And then you wouldn't use the boosted trees at all.  What is the benefit of using the trees then?  Does using the NN for regression not work well in practice in this situation?

\n

[/quote]

\n

Hi J Kolb,

\n

It's likely that you could get NN regression to work in this context, but I haven't really tried.  The main reason is 2-fold.  First, I had several NN models at the end, and this presented a quick and dirty way to ensemble them together.  (I tried other classifiers and other tricks, but this was the only one that worked).  Second, it took me a day or so to train each of the neural nets, but only a minute or two to train the boosted tree model at the end.  By playing around with the boosted tree parameters, I was able to get a much better final score without spending 8 hours between each attempt.

", "rawMarkdown": "", "votes": null }, { "id": "70177", "postDate": "04/10/2015 06:53:49", "content": "

[quote=zyy1990;69703]

\n

Hello Hoyt

\n

Thanks for sharing ~~

\n

However, I ran into the following issue while running create_nn_model.py:

\n

File "create_nn_model.py", line 85, in <module>
mean_image = X_train["image"].mean()

\n

graphlab.toolkits._main.ToolkitError: Cannot perform sum or average over images of different sizes. Found images of total size (ie. width * height * channels) of both 196608 and 65536. Please use graplab.image_analysis.resize() to make images a uniform size.

\n

Any idea where this might come from? 

\n

thanks

\n

[/quote]

\n

zyy1990: Have you solved this problem?

", "rawMarkdown": "", "votes": null }, { "id": "70186", "postDate": "04/10/2015 07:29:29", "content": "

[quote=vnneo;70177]

\n

[quote=zyy1990;69703]

\n

Hello Hoyt

\n

Thanks for sharing ~~

\n

However, I ran into the following issue while running create_nn_model.py:

\n

File "create_nn_model.py", line 85, in <module>
mean_image = X_train["image"].mean()

\n

graphlab.toolkits._main.ToolkitError: Cannot perform sum or average over images of different sizes. Found images of total size (ie. width * height * channels) of both 196608 and 65536. Please use graplab.image_analysis.resize() to make images a uniform size.

\n

Any idea where this might come from? 

\n

thanks

\n

[/quote]

\n

zyy1990: Have you solved this problem?

\n

[/quote]

\n\n

Yes. For some unknown reason, the data i downloaded had many images which were all black, so i picked   those images out. Here are the id of those images:

\n

train/41176_left.jpeg
train/34689_left.jpeg
train/32253_right.jpeg
train/43457_left.jpeg
train/1986_left.jpeg
test/28544_left.jpeg
test/16519_left.jpeg
test/38549_left.jpeg
test/35433_left.jpeg
test/3517_left.jpeg
test/35762_left.jpeg

\n

train/15222_left.jpeg
test/16349_left.jpeg

", "rawMarkdown": "", "votes": null }, { "id": "71221", "postDate": "04/10/2015 15:44:27", "content": "

[quote=James King;70072]

\n

It's like you're working on a puzzle, and you're really absorbed in it and making progress, and then someone comes along and tells you the answer.

\n

[/quote]

\n\n

I agree.

\n

My opinion was that posting this solution was poor sportsmanship. If part of the motivation was to market a company's tools, then it leads me to have a negative opinion of that company as well. Seeing this solution decreased my motivation to participant in this contest.

\n

Obviously the original poster did not think feel that way, so what we have right now is a difference of opinion. However, we have an opportunity to resolve this question by means of (yes!) data.

\n

If lots of people agree with me, then as a community we should discourage this kind of posting, and the site administrators should note that it will be bad for Kaggle if people are turned off by the practice and stop participating. If people think this kind of posting is a good idea, then we should encourage people to post full solutions early and often.

\n

Please voice your opinion. What do you think?

", "rawMarkdown": "", "votes": null }, { "id": "71224", "postDate": "04/10/2015 15:56:11", "content": "

28 people upvoted the OP... so, we've already had a vote of sorts. Please don't conflate self-promotion with code sharing. Although code sharing is not entirely altruistic, I am fundamentally for it (both in general and specifically in Kaggle competitions). I myself was motivated (rather than discouraged) by this thread. I felt stimulated to brainstorm ways to improve upon the OP's score, yet also comforted that if I failed to get a good model during these next few months, that I could fall back on this shared code. A HUGE thank you to the OP.

\n\n

My strong opinion is that collaboration is beautiful. If you don't want to look at the shared code, don't look at it. Easy solution. Live and let live.

\n\n

Perhaps, another question we should ask (for another thread), is at what point does it become rude to hijack someone else's thread simply because you have a different opinion?

", "rawMarkdown": "", "votes": null }, { "id": "71227", "postDate": "04/10/2015 16:26:40", "content": "

[quote=Michael 7402;71221]Please voice your opinion. What do you think?[/quote]

\n

Idem datanewb: please stop this flood. (yeah.. self-defeating demand :-) )

", "rawMarkdown": "", "votes": null }, { "id": "71229", "postDate": "04/10/2015 16:57:53", "content": "

[quote=datanewb;71224]

\n

28 people upvoted the OP... so, we've already had a vote of sorts. Please don't conflate self-promotion with code sharing. Although code sharing is not entirely altruistic, I am fundamentally for it (both in general and specifically in Kaggle competitions). I myself was motivated (rather than discouraged) by this thread. I felt stimulated to brainstorm ways to improve upon the OP's score, yet also comforted that if I failed to get a good model during these next few months, that I could fall back on this shared code. A HUGE thank you to the OP.

\n

My strong opinion is that collaboration is beautiful. If you don't want to look at the shared code, don't look at it. Easy solution. Live and let live.

\n

Perhaps, another question we should ask (for another thread), is at what point does it become rude to hijack someone else's thread simply because you have a different opinion?

\n

[/quote]

\n

There was no conflation of self-promotion and code-sharing.  The evidence is in the forums of those 3 competitions (and maybe others), as I stated.  I completely agree, and would assume that Kaggle does as well, that collaboration is a beautiful thing, which is why we are allowed to form Teams. 

", "rawMarkdown": "", "votes": null }, { "id": "71248", "postDate": "04/10/2015 20:51:45", "content": "

[quote=datanewb;71224]

\n

if I failed to get a good model during these next few months, that I could fall back on this shared code.

\n

[/quote]

\n

At the detriment of everyone you passed over on the leader board. Just by copying and pasting code. Great. Now the LB is no longer reflective of what you are capable of.

\n

This issue could be completely resolved if the Dato folks posted models without optimized hyperparameters, and perhaps even stating what the model is capable of. And let their users work at it a bit.

\n

In fact, they'd be doing their users a favor.

", "rawMarkdown": "", "votes": null }, { "id": "71249", "postDate": "04/10/2015 21:08:44", "content": "

This is a hard one ----yes or no to.

\n

I follow a good many Kaggle Forums and to be open I have learned quite a few things by looking into everything everyone has posted and have learned new ways to compete in this environment.

\n

Being a Novice here (or anywhere for that matter) for this type of competition I initially had no idea how I was going to use what I know about ML to compete.

\n

I agree with all of you these type of posing are a type of self promotion.... by thanks to a few of them I am now feeling a bit more confident about being able to participate in these competitions and see nothing wrong as long as we are able to learn something from the post.... even if this particular one seem to be very self-promoting.... IMHO

\n

I do commend you all for coming out against it to keep self-promotion under control 

", "rawMarkdown": "", "votes": null }, { "id": "71382", "postDate": "04/13/2015 12:09:41", "content": "

Hi Hoyt,

\n

Thank you for sharing!

\n

Do you have any data, what is the best benchmark with this base solution so far?

", "rawMarkdown": "", "votes": null }, { "id": "71397", "postDate": "04/13/2015 17:55:08", "content": "

Hello Hoyt,

\n

I've just started the NN training process on my GTX 980 and since you wrote: "and about 320 images / second on a GTX 780" in the readme, I expected about 20% better performance on my configuration.

\n

I got 280 img/s.

\n

Do you have any idea what can be the reason?

", "rawMarkdown": "", "votes": null }, { "id": "71399", "postDate": "04/13/2015 18:16:20", "content": "

[quote=George Solymosi;71382]

\n

Hi Hoyt,

\n

Thank you for sharing!

\n

Do you have any data, what is the best benchmark with this base solution so far?

\n

[/quote]

\n

Hi George,

\n

No, I'm not sure at this point.  It's a pretty crude approach; there are a lot of parameters to tune, and I've since realized there are a lot of things holding back the accuracy that become obvious when you examine the output solutions.  In particular, as others have mentioned, the NN architecture here overfits to some parts of the the training data, it doesn't take into account the unbalanced class sizes at prediction time, etc.  I got somewhere close to 0.46 with this solution; some tweaks to boosted tree algorithm (unpublished) boosted the score somewhat.   I'm sure that more sophistication in this approach, a better NN architecture, and/or ensembling this model with other approaches is needed to get much better than this.

", "rawMarkdown": "", "votes": null }, { "id": "71415", "postDate": "04/13/2015 23:06:20", "content": "

@hoyt, just some comments on the image processing... I have yet to implement something myself

\n

a) equalisation is normally only done on the intensity component so that you don't distort the colour balance (ie you don't want to stretch each colour channel separately)

\n

http://www.imagemagick.org/script/command-line-options.php#equalize

\n\n

b) i would think that its better to convert to png rather than jpg (which might introduce new jpeg artifacts at 256x256 size?)

", "rawMarkdown": "", "votes": null }, { "id": "71417", "postDate": "04/13/2015 23:11:41", "content": "

[quote=George Solymosi;71397]

\n

Hello Hoyt,

\n

I've just started the NN training process on my GTX 980 and since you wrote: "and about 320 images / second on a GTX 780" in the readme, I expected about 20% better performance on my configuration.

\n

I got 280 img/s.

\n

Do you have any idea what can be the reason?

\n

[/quote]

\n

Hey George, 

\n

Sorry -- I have no idea.  I suspect it also has to do with the other configurations of your system. The only thing I can think of is that it's IO bound in moving data on and off the card -- maybe it's mounted in the x8 PCIe slot instead of the x16 PCIe slot (although that shouldn't make a difference).  

\n

The batch_size parameter controls how many images are loaded onto the graphics card at a time for training.  It's possible that adjusting this parameter will help with performance.

", "rawMarkdown": "", "votes": null }, { "id": "71418", "postDate": "04/13/2015 23:16:15", "content": "

[quote=Sean;71415]

\n

@hoyt, just some comments on the image processing... I have yet to implement something myself

\n

a) equalisation is normally only done on the intensity component so that you don't distort the colour balance (ie you don't want to stretch each colour channel separately)

\n

http://www.imagemagick.org/script/command-line-options.php#equalize

\n

b) i would think that its better to convert to png rather than jpg (which might introduce new jpeg artifacts at 256x256 size?)

\n

[/quote]

\n

Hey Sean, 

\n

Thanks for your comments!  I called equalize mainly so that the 3 different channels of the NN would be on the same scale, although I'm not sure that would have any significant effect.  And yes, png might be better.  I'll give that a shot for my latest iteration. 

", "rawMarkdown": "", "votes": null }, { "id": "72504", "postDate": "04/19/2015 19:26:31", "content": "

We can fight with each other, but if we can quicker get closer with a bit help of something like Hoyt's shortcut to a better solution for help to prevent and cure people's illness, then I don't think that, that is unfair...

\n

Consider it.

", "rawMarkdown": "", "votes": null }, { "id": "72719", "postDate": "04/20/2015 19:06:49", "content": "

[quote=Hoyt Koepke;66564]

\n

Use ImageMagick's convert tool to trim off the blank space to the sides of the images, then pad them so that they are all 256x256. Thus the eye is always centered with edges against the edges of the image. I avoided scaling the images to improve the neural net performance.

\n

[/quote]

\n

I'm a bit puzzled though, you say you convert images to 256x256 and at the same time you say you avoid scaling the images. Could you kindly clarify this part?

", "rawMarkdown": "", "votes": null }, { "id": "72986", "postDate": "04/21/2015 17:58:09", "content": "

[quote=K0stIa;72719]

\n

I'm a bit puzzled though, you say you convert images to 256x256 and at the same time you say you avoid scaling the images. Could you kindly clarify this part?

\n

[/quote]

\n

Sorry, I could have been clearer there.  What I meant is that I made sure that the aspect ratio of the images and the location of the eyes in the picture stayed the same so that the resulting NN did not have to handle scale and location invariance along with learning to classify the images.  

\n\n

Hope that helps!

", "rawMarkdown": "", "votes": null }, { "id": "73007", "postDate": "04/21/2015 18:50:03", "content": "

You do realize that the beauty of CNN's that scaling and translation does really cause any performance penalties of the CNN's..... However make sure you don't scale away the useful information ;)

", "rawMarkdown": "", "votes": null }, { "id": "73184", "postDate": "04/22/2015 15:24:24", "content": "

Thanks Dato Inc / Hoyt for trying to democratize Big Data / Deep Learning algorithms. I was curious what pretrained filters the GraphLab's deeplearning algorithm make use of ? Is it Gabor Filters or BSD Caffe or something else ?

", "rawMarkdown": "", "votes": null }, { "id": "74244", "postDate": "04/23/2015 04:23:32", "content": "

Hi Hoyt,

\n

I'm a little confused on the performance you mentioned. 300 images/s tells me each epoch takes around 2 min, and 5-10 epochs for each 2-class model comes out to be 20 min of training. How does this translate to about a day (24h) worth of training? Would you mind clarifying? I'm using theano so I'm curious how my performance compares to your cxxnet implementation.

", "rawMarkdown": "", "votes": null }, { "id": "74358", "postDate": "04/23/2015 18:12:36", "content": "

[quote=Michael George Hart;73007]

\n

You do realize that the beauty of CNN's that scaling and translation does really cause any performance penalties of the CNN's..... However make sure you don't scale away the useful information ;)

\n

[/quote]

\n

Hi Michael, 

\n

Yes, true, though in general, when you add more convolution/pooling layers in order to get closer to true scale invariance, you end up needing more data in order to get it to train effectively.  I opted for having much fewer convolution layers than is needed for true translation invariance, since it seemed like location in the image could actually be informative.  I think there may be something to this approach, as I've been trying to train with deeper architectures since then, but haven't really got significantly better results yet.

", "rawMarkdown": "", "votes": null }, { "id": "74360", "postDate": "04/23/2015 18:15:58", "content": "

[quote=saurk;73184]

\n

Thanks Dato Inc / Hoyt for trying to democratize Big Data / Deep Learning algorithms. I was curious what pretrained filters the GraphLab's deeplearning algorithm make use of ? Is it Gabor Filters or BSD Caffe or something else ?

\n

[/quote]

\n

Hello Saurk,

\n

Thanks!  Under the hood, graphlab create doesn't use any pretrained filters. However, the filters in the convolution layers will often end up resembling Gabor filters, but they are learned directly from the data during training.  

\n

I think Caffe will end up doing the same thing...

\n

-- Hoyt

", "rawMarkdown": "", "votes": null }, { "id": "74361", "postDate": "04/23/2015 18:16:22", "content": "

[quote=jstaker7;74244]

\n

Hi Hoyt,

\n

I'm a little confused on the performance you mentioned. 300 images/s tells me each epoch takes around 2 min, and 5-10 epochs for each 2-class model comes out to be 20 min of training. How does this translate to about a day (24h) worth of training? Would you mind clarifying? I'm using theano so I'm curious how my performance compares to your cxxnet implementation.

\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "74364", "postDate": "04/23/2015 18:30:44", "content": "

[quote=jstaker7;74244]

\n

Hi Hoyt,

\n

I'm a little confused on the performance you mentioned. 300 images/s tells me each epoch takes around 2 min, and 5-10 epochs for each 2-class model comes out to be 20 min of training. How does this translate to about a day (24h) worth of training? Would you mind clarifying? I'm using theano so I'm curious how my performance compares to your cxxnet implementation.

\n

[/quote]

\n

Hello jstaker,

\n

The code I use brute-force creates a bunch of different versions of each image, as well as duplicating some of them to balance the classes in the training set.  Thus there end up being hundreds of thousands of images trained in each epoch instead of the original 35,000 or so.  

", "rawMarkdown": "", "votes": null }, { "id": "74417", "postDate": "04/24/2015 00:44:22", "content": "

[quote=Hoyt Koepke;74358]

\n

... I opted for having much fewer convolution layers than is needed for true translation invariance, since it seemed like location in the image could actually be informative...

\n

[/quote]

\n\n

This challenges my understanding of CNNs. Isn't just feature detection translation invariant? I thought one of the great advantages of CNNs is that it can capture spacial relationships. Am I understanding your comment correctly?

", "rawMarkdown": "", "votes": null }, { "id": "74445", "postDate": "04/24/2015 06:36:51", "content": "

[quote=jstaker7;74417]

\n

[quote=Hoyt Koepke;74358]

\n

... I opted for having much fewer convolution layers than is needed for true translation invariance, since it seemed like location in the image could actually be informative...

\n

[/quote]

\n

This challenges my understanding of CNNs. Isn't just feature detection translation invariant? I thought one of the great advantages of CNNs is that it can capture spacial relationships. Am I understanding your comment correctly?

\n

[/quote]

\n

CNNs can learn not only translation invariance, but even scale and rotation invariances, but for this to happen, those characteristics need to be derived from the training set, i.e. the independence between (position, rotation, scale) of the features and class label needs to be present in the dataset. If, for some reason, some examples contain some features which are positioned constantly in the same general area of the image, even the translation invariance will go away, because the CNN would 'think' in this case that the position of those features should affect the decision in some way, i.e. the position of the feature and the class label would not be independent and identically distributed (i.i.d.) any more. In conclusion, CNNs CAN learn a lot of invariances, but in order for them to learn, those invariances need to be present in the training set, and this is why data augmentation is done.

", "rawMarkdown": "", "votes": null }, { "id": "75145", "postDate": "04/27/2015 23:47:21", "content": "

I'm not an expert on CNNs, but my understanding is that the translation invariance comes purely from the pooling layers.  Thus CNNs are only truly translation invariant if there are enough pooling layers that sufficiently significant features detected on one side of the image can affect the outcome of any of the nodes before the final fully connected layers.  In my original net, I didn't have enough of these layers to achieve this, so it then relied on the fully connected layers at the end to do more of the work.

\n

Similarly, scale and rotation invariances come from the number of layers and having enough channels in the convolution layers to adequately fit the range of features present in the data. 

\n

The problem, however, is that the more channels and layers you add, the more parameters need to be trained, and thus the more data you need to train the net without overfitting.  Ideally, you want a complicated enough structure to capture all relevant features, but a simple enough structure that it doesn't overfit (though dropout and regularization can help with that). 

", "rawMarkdown": "", "votes": null }, { "id": "75242", "postDate": "04/28/2015 13:38:09", "content": "

Yes pooling helps with the invariance ... A type of averaging mechanism.....

\n

however it is the Convolution Layer, a, more realistic, model of the visual system, that makes i variances .... Get the spacing right on the convolution will be more useful for this problem than the adding more pooling. ... Most of the work has already done for us by GoogLeNet, AlexNet etc... The trick here is to modify those convolution layers and perhaps tweeting the pooling layers to meet the needs of this particular problem....

\n

i imagine many people already see this as a solution... That is why I am only using it as a starting point to see how far I can get in the rankings, because the competition will come to who can tune the existing CNN's for a solution to this challenge.... 

\n

To win this challenge someone will have to step outside the CNN box since it is such an abvious solution... ;)

", "rawMarkdown": "", "votes": null }, { "id": "77297", "postDate": "05/06/2015 11:02:26", "content": "

Kaggle has made their position clear on this in the competition rules I think: "It's okay to share code if made available to all participants on the forums."

\n

Why would publishing any competition entry bad or discouraging?

\n

- it's absolutely *not* the same as someone coming and telling you the 'answer to a puzzle' as you say.

\n

Here, you are not forced to read the forums, let alone study/run the code on github.

\n

I think that *anyone* who reads these forums is doing so *exactly* because they want to find out what other people are doing.

\n

- how is seeing a solution that scores better than you discouraging? You already can see in the leaderboard people's scores. Obviously those who scored above you have a solution to go with it, there is no surprise there.

\n

- if there is a tool that can help, and it's allowed by the rules (perhaps), why not use it? You say "seeing this solution decreased by motivation to participant [sic] in this contest", others would say "seeing this solution increased my motivation" as they have something to get started with.

\n

[quote=Michael 7402;71221]

\n

[quote=James King;70072]

\n

It's like you're working on a puzzle, and you're really absorbed in it and making progress, and then someone comes along and tells you the answer.

\n

[/quote]

\n

I agree.

\n

My opinion was that posting this solution was poor sportsmanship. If part of the motivation was to market a company's tools, then it leads me to have a negative opinion of that company as well. Seeing this solution decreased my motivation to participant in this contest.

\n

Obviously the original poster did not think feel that way, so what we have right now is a difference of opinion. However, we have an opportunity to resolve this question by means of (yes!) data.

\n

If lots of people agree with me, then as a community we should discourage this kind of posting, and the site administrators should note that it will be bad for Kaggle if people are turned off by the practice and stop participating. If people think this kind of posting is a good idea, then we should encourage people to post full solutions early and often.

\n

Please voice your opinion. What do you think?

\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "80553", "postDate": "06/01/2015 06:46:22", "content": "

Hi Hoyt,

\n

Thanks for the code. During prediction, I am getting different prediction levels for the same image in different runs. Please help me to sort out this. The results are as follows for consecutive runs of same set of images.

\n

+-------------+-------+----------------+
| name | level | actual_level |
+-------------+-------+----------------+
| 10046_right | 1 | 0.751127591425 |
| 11730_right | 0 | 0.358918072124 |
+-------------+-------+----------------+

\n

+-------------+-------+----------------+
| name | level | actual_level |
+-------------+-------+----------------+
| 10046_right | 0 | 0.423049250692 |
| 11730_right | 0 | 0.326992818052 |
+-------------+-------+----------------+

\n

+-------------+-------+----------------+
| name | level | actual_level |
+-------------+-------+----------------+
| 10046_right | 1 | 0.70960257402 |
| 11730_right | 0 | 0.357797767757 |
+-------------+-------+----------------+

\n

+-------------+-------+----------------+
| name | level | actual_level |
+-------------+-------+----------------+
| 10046_right | 0 | 0.408557624363 |
| 11730_right | 0 | 0.355133798881 |
+-------------+-------+----------------+

", "rawMarkdown": "", "votes": null }, { "id": "81044", "postDate": "06/05/2015 19:35:40", "content": "

Hi Hoyt,

\n

Thanks for posting this started code. I am impressed by what Graphlab can do with just a few hundred lines of Python code! I tried to run your Github code but looks like the Graphlab API crashed in sframe sorting - see below and the log file attached. If I comment out the sorting code then it will run fine. Any idea?

\n

I'm running Ubuntu 14.04 on a desktop Intel Core i7-5820K CPU @ 3.30GHz × 12, with 8GB Memory plus GeForce GTX 970 GPUs.

\n

=========================

\n

Traceback (most recent call last):
File "create_image_sframes.py", line 47, in <module>
X_train = X_train.sort("_random_")
File "/home/ekang/anaconda/lib/python2.7/dist-packages/graphlab/data_structures/sframe.py", line 5077, in sort
return SFrame(_proxy=self.__proxy__.sort(sort_column_names, sort_column_orders))
File "/home/ekang/anaconda/lib/python2.7/dist-packages/graphlab/cython/context.py", line 31, in __exit__
raise exc_type(exc_value)
RuntimeError: Communication Failure: 113.

\n

[quote=Hoyt Koepke;66564]

\n

Hello, 

\n

I just put up code that I used to get a leaderboard score of ~0.46412, and I would like to hear feedback on it.  Feel free to use it, and I am open to feedback.  I am sure many parts of it can be improved. 

\n

https://github.com/hoytak/diabetic-retinopathy-code

\n

This code uses the ImageMagick convert tool to preprocess the images, then uses the neural net toolkit (basically cxxnet) and the boosted regression trees in Dato's Graphlab create toolkits to build the classifier.  

\n

Main Ideas:

\n
    \n
  1. Use ImageMagick's convert tool to trim off the blank space to the sides of the images, then pad them so that they are all 256x256. Thus the eye is always centered with edges against the edges of the image. I avoided scaling the images to improve the neural net performance.
    2. \n
  2. Create multiple versions of each image varying by hue and contrast and white balance.  This step can be simplified to get to a quick though less accurate solution.
    3. \n
  3. Duplicate each class so each class is represented equally, then shuffle the data.
    4. \n
  4. Train several neural nets, one trained to predict the level membership, and another 4 to distinguish 0 vs 1-4, 0-1 vs. 2-4, 0-2 vs. 3-4, and 0-3 vs. 4.
    5. \n
  5. For each image, extract both class predictions and the values of the final neural net layer. Pool all of these features across models and variations of the same images.
    6. \n
  6. Train boosted regression trees on these pooled features to predict the level.
    7. \n
  7. Round to the nearest integer as the class prediction.
    8. \n
\n

I know that each of these steps could definitely be tuned for greater accuracy, and I would love to hear your feedback. In particular, I don't think the neural net architecture is optimal for these images. 

\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "81046", "postDate": "06/05/2015 19:52:09", "content": "

Hi Rajkumar,

\n

It turns out that this is a known issue that the NN people here are working on.  I'm not sure there's a way around it now.  I think if you use the boosted tree regression as a post-processing step on the activation layers, it's a bit more robust. 

\n

-- Hoyt

\n\n

[quote=Rajkumar;80553]

\n

Hi Hoyt,

\n

Thanks for the code. During prediction, I am getting different prediction levels for the same image in different runs. Please help me to sort out this. The results are as follows for consecutive runs of same set of images.

\n

+-------------+-------+----------------+
| name | level | actual_level |
+-------------+-------+----------------+
| 10046_right | 1 | 0.751127591425 |
| 11730_right | 0 | 0.358918072124 |
+-------------+-------+----------------+

\n

+-------------+-------+----------------+
| name | level | actual_level |
+-------------+-------+----------------+
| 10046_right | 0 | 0.423049250692 |
| 11730_right | 0 | 0.326992818052 |
+-------------+-------+----------------+

\n

+-------------+-------+----------------+
| name | level | actual_level |
+-------------+-------+----------------+
| 10046_right | 1 | 0.70960257402 |
| 11730_right | 0 | 0.357797767757 |
+-------------+-------+----------------+

\n

+-------------+-------+----------------+
| name | level | actual_level |
+-------------+-------+----------------+
| 10046_right | 0 | 0.408557624363 |
| 11730_right | 0 | 0.355133798881 |
+-------------+-------+----------------+

\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "81048", "postDate": "06/05/2015 20:01:48", "content": "

Hello fisherman,

\n

In this case, I kept hitting issues on sort running out of memory, etc. as well.  I'm not sure of a workaround except to use a machine with more memory.  I did talk with the people here at Dato responsible for the sort stuff, and it's getting better.  

\n

The solution I ended up using was to modify the code so that it just used the file paths up until the final part when it generated the sframes, at which point I load them using a lambda function.  This makes it significantly faster and the sort is much easier.

\n

Something like: 

\n

X["path"] = list(chain(*[[abspath(join(root, f)) for f in files
                  if f.endswith('jpeg')]
                  for root, dir_list, files in os.walk(image_path)]))


\n

Then use something like

\n

X["image"] = X["path"].apply(lambda p: gl.Image(p))

\n

to load it at the end. 

\n

Hope something like that helps! 

\n

-- Hoyt

\n

[quote=fisherman;81044]

\n

Hi Hoyt,

\n

Thanks for posting this started code. I am impressed by what Graphlab can do with just a few hundred lines of Python code! I tried to run your Github code but looks like the Graphlab API crashed in sframe sorting - see below and the log file attached. If I comment out the sorting code then it will run fine. Any idea?

\n

I'm running Ubuntu 14.04 on a desktop Intel Core i7-5820K CPU @ 3.30GHz × 12, with 8GB Memory plus GeForce GTX 970 GPUs.

\n

=========================

\n

Traceback (most recent call last):
File "create_image_sframes.py", line 47, in <module>
X_train = X_train.sort("_random_")
File "/home/ekang/anaconda/lib/python2.7/dist-packages/graphlab/data_structures/sframe.py", line 5077, in sort
return SFrame(_proxy=self.__proxy__.sort(sort_column_names, sort_column_orders))
File "/home/ekang/anaconda/lib/python2.7/dist-packages/graphlab/cython/context.py", line 31, in __exit__
raise exc_type(exc_value)
RuntimeError: Communication Failure: 113.

\n

[quote=Hoyt Koepke;66564]

\n

Hello, 

\n

I just put up code that I used to get a leaderboard score of ~0.46412, and I would like to hear feedback on it.  Feel free to use it, and I am open to feedback.  I am sure many parts of it can be improved. 

\n

https://github.com/hoytak/diabetic-retinopathy-code

\n

This code uses the ImageMagick convert tool to preprocess the images, then uses the neural net toolkit (basically cxxnet) and the boosted regression trees in Dato's Graphlab create toolkits to build the classifier.  

\n

Main Ideas:

\n
    \n
  1. Use ImageMagick's convert tool to trim off the blank space to the sides of the images, then pad them so that they are all 256x256. Thus the eye is always centered with edges against the edges of the image. I avoided scaling the images to improve the neural net performance.
    2. \n
  2. Create multiple versions of each image varying by hue and contrast and white balance.  This step can be simplified to get to a quick though less accurate solution.
    3. \n
  3. Duplicate each class so each class is represented equally, then shuffle the data.
    4. \n
  4. Train several neural nets, one trained to predict the level membership, and another 4 to distinguish 0 vs 1-4, 0-1 vs. 2-4, 0-2 vs. 3-4, and 0-3 vs. 4.
    5. \n
  5. For each image, extract both class predictions and the values of the final neural net layer. Pool all of these features across models and variations of the same images.
    6. \n
  6. Train boosted regression trees on these pooled features to predict the level.
    7. \n
  7. Round to the nearest integer as the class prediction.
    8. \n
\n

I know that each of these steps could definitely be tuned for greater accuracy, and I would love to hear your feedback. In particular, I don't think the neural net architecture is optimal for these images. 

\n

[/quote]

\n

[/quote]

", "rawMarkdown": "", "votes": null }, { "id": "82722", "postDate": "06/25/2015 11:12:43", "content": "

Hi Hoyt, thanks for the startercode to begin with.

\n

I'm having an issue when I run create_image_sframes.py

\n

I get the message below repeatedly

\n

PROGRESS: Unexpected JPEG decode failure file: /mnt/my-data/kaggle-data/processed/run-normal/test/21697_left.jpeg

\n

Any ideas off-hand how to fix this?

", "rawMarkdown": "", "votes": null }, { "id": "82746", "postDate": "06/25/2015 17:50:51", "content": "

Hey Octave1,

\n

I think it's happening because somehow the shrinking part of the imagemagick script ended up shrinking one of those images to zero.  It should be fine if you just delete that image, or rerun the convert command for just that image. 

\n

-- Hoyt

", "rawMarkdown": "", "votes": null }, { "id": "84211", "postDate": "07/12/2015 15:02:54", "content": "

thank you Hoyt, I can run following your guides.

", "rawMarkdown": "thank you Hoyt, I can run following your guides.", "votes": null }, { "id": "86129", "postDate": "07/20/2015 06:58:23", "content": "

Hi Hoyt,

\n\n

I tried running your code. I did it only to train which_model = 4 i.e. predicting 4-vs-all retinopathy grade.\nMy training and validation accuracy figures were quite low so I checked the features that have been stored in Xty and Xtst in create_nn_model.py and it shows nan values in both scores_4 and features_4 columns. Any idea about what is it that I did wrong?

\n\n

Following are the details of the model that was stored:

\n\n
Class               : NeuralNetClassifier\n\nSchema\n------\nExamples            : 848387\nFeatures            : 1\nTarget column       : class\n\nTraining Summary\n----------------\nTraining accuracy   : 0.4992\nValidation accuracy : 0.5416\nTraining time (sec) : 85850.3034\n
\n\n

Following are the details of Xf_train from create_submission.py code.

\n\n
Columns:\n    name    str\n    scores_4    dict\n    level   int\n    features_4  dict\n\nRows: 34424\n\nData:\n+-------------+-------------------------------+-------+\n|     name    |            scores_4           | level |\n+-------------+-------------------------------+-------+\n|  2157_right | {'contrast-2.1': nan, 'con... |   0   |\n|  8649_right | {'contrast-2.1': nan, 'con... |   0   |\n|  497_right  | {'contrast-2.1': nan, 'con... |   0   |\n| 34024_right | {'contrast-2.1': nan, 'con... |   0   |\n|  2613_left  | {'contrast-2.1': nan, 'con... |   2   |\n|  21769_left | {'contrast-2.1': nan, 'con... |   0   |\n| 25197_right | {'contrast-2.1': nan, 'con... |   2   |\n|  21833_left | {'contrast-2.1': nan, 'con... |   1   |\n|  11417_left | {'contrast-2.1': nan, 'con... |   2   |\n|  5096_right | {'contrast-2.1': nan, 'con... |   0   |\n+-------------+-------------------------------+-------+\n+-------------------------------+\n|           features_4          |\n+-------------------------------+\n| {'sat-2.23': nan, 'sat-2.2... |\n| {'sat-2.23': nan, 'sat-2.2... |\n| {'sat-2.23': nan, 'sat-2.2... |\n| {'sat-2.23': nan, 'sat-2.2... |\n| {'sat-2.23': nan, 'sat-2.2... |\n| {'sat-2.23': nan, 'sat-2.2... |\n| {'sat-2.23': nan, 'sat-2.2... |\n| {'sat-2.23': nan, 'sat-2.2... |\n| {'sat-2.23': nan, 'sat-2.2... |\n| {'sat-2.23': nan, 'sat-2.2... |\n+-------------------------------+\n[34424 rows x 4 columns]\nNote: Only the head of the SFrame is printed.\nYou can use print_rows(num_rows=m, num_columns=n) to print more rows and columns.\n
", "rawMarkdown": "Hi Hoyt,\r\n\r\nI tried running your code. I did it only to train `which_model = 4` i.e. predicting 4-vs-all retinopathy grade.\r\nMy training and validation accuracy figures were quite low so I checked the features that have been stored in `Xty` and `Xtst` in `create_nn_model.py` and it shows `nan` values in both `scores_4` and `features_4` columns. Any idea about what is it that I did wrong?\r\n\r\nFollowing are the details of the model that was stored:\r\n\r\n \r\n Class : NeuralNetClassifier\r\n \r\n Schema\r\n ------\r\n Examples : 848387\r\n Features : 1\r\n Target column : class\r\n \r\n Training Summary\r\n ----------------\r\n Training accuracy : 0.4992\r\n Validation accuracy : 0.5416\r\n Training time (sec) : 85850.3034\r\n\r\n\r\nFollowing are the details of `Xf_train` from `create_submission.py` code.\r\n\r\n Columns:\r\n \tname\tstr\r\n \tscores_4\tdict\r\n \tlevel\tint\r\n \tfeatures_4\tdict\r\n \r\n Rows: 34424\r\n \r\n Data:\r\n +-------------+-------------------------------+-------+\r\n | name | scores_4 | level |\r\n +-------------+-------------------------------+-------+\r\n | 2157_right | {'contrast-2.1': nan, 'con... | 0 |\r\n | 8649_right | {'contrast-2.1': nan, 'con... | 0 |\r\n | 497_right | {'contrast-2.1': nan, 'con... | 0 |\r\n | 34024_right | {'contrast-2.1': nan, 'con... | 0 |\r\n | 2613_left | {'contrast-2.1': nan, 'con... | 2 |\r\n | 21769_left | {'contrast-2.1': nan, 'con... | 0 |\r\n | 25197_right | {'contrast-2.1': nan, 'con... | 2 |\r\n | 21833_left | {'contrast-2.1': nan, 'con... | 1 |\r\n | 11417_left | {'contrast-2.1': nan, 'con... | 2 |\r\n | 5096_right | {'contrast-2.1': nan, 'con... | 0 |\r\n +-------------+-------------------------------+-------+\r\n +-------------------------------+\r\n | features_4 |\r\n +-------------------------------+\r\n | {'sat-2.23': nan, 'sat-2.2... |\r\n | {'sat-2.23': nan, 'sat-2.2... |\r\n | {'sat-2.23': nan, 'sat-2.2... |\r\n | {'sat-2.23': nan, 'sat-2.2... |\r\n | {'sat-2.23': nan, 'sat-2.2... |\r\n | {'sat-2.23': nan, 'sat-2.2... |\r\n | {'sat-2.23': nan, 'sat-2.2... |\r\n | {'sat-2.23': nan, 'sat-2.2... |\r\n | {'sat-2.23': nan, 'sat-2.2... |\r\n | {'sat-2.23': nan, 'sat-2.2... |\r\n +-------------------------------+\r\n [34424 rows x 4 columns]\r\n Note: Only the head of the SFrame is printed.\r\n You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns.", "votes": null } ], "comments": [ { "id": 66589, "author_name": "jiweiliu", "author_url": "", "post_date": "03/17/2015 00:43:55", "content": "

Thank you. Could you share the cross validation kappa score of this approach?

", "votes": null, "replies": [] }, { "id": 66601, "author_name": "hoytkoepke1", "author_url": "", "post_date": "03/17/2015 01:17:26", "content": "

Hi RCarson,

\n

I have not yet run full cross-validation on this model yet to get a final kappa score.  My goal was to get something done quickly, but running cross validation and tuning the intermediate parameters would be the next step and likely give a much better result.  

\n

My holdout validation RMSE for the regression problem at the end to predict the level was around 0.63 if that helps.

\n

-- Hoyt

", "votes": null, "replies": [] }, { "id": 66607, "author_name": "jiweiliu", "author_url": "", "post_date": "03/17/2015 01:36:13", "content": "

Great! my model gets rmse is 1.8 and LB 0.09. Thank you.

", "votes": null, "replies": [] }, { "id": 66626, "author_name": "sushize", "author_url": "", "post_date": "03/17/2015 03:50:25", "content": "

Thanks for the great shared code!

\n

Just one thing which I hope you could clarify (thanks in advance): I noticed that your code used the Graphlab Create, which is not fully open source. Are you using any non-open source elements of the Graphlab Create in your code or not?

\n

Thanks!

\n

Best wishes,

\n

Shize

\n

[quote=Hoyt Koepke;66564]

\n

Hello, 

\n

I just put up code that I used to get a leaderboard score of ~0.46412, and I would like to hear feedback on it.  Feel free to use it, and I am open to feedback.  I am sure many parts of it can be improved. 

\n

https://github.com/hoytak/diabetic-retinopathy-code

\n

This code uses the ImageMagick convert tool to preprocess the images, then uses the neural net toolkit (basically cxxnet) and the boosted regression trees in Dato's Graphlab create toolkits to build the classifier.  

\n

Main Ideas:

\n
    \n
  1. Use ImageMagick's convert tool to trim off the blank space to the sides of the images, then pad them so that they are all 256x256. Thus the eye is always centered with edges against the edges of the image. I avoided scaling the images to improve the neural net performance.
    2. \n
  2. Create multiple versions of each image varying by hue and contrast and white balance.  This step can be simplified to get to a quick though less accurate solution.
    3. \n
  3. Duplicate each class so each class is represented equally, then shuffle the data.
    4. \n
  4. Train several neural nets, one trained to predict the level membership, and another 4 to distinguish 0 vs 1-4, 0-1 vs. 2-4, 0-2 vs. 3-4, and 0-3 vs. 4.
    5. \n
  5. For each image, extract both class predictions and the values of the final neural net layer. Pool all of these features across models and variations of the same images.
    6. \n
  6. Train boosted regression trees on these pooled features to predict the level.
    7. \n
  7. Round to the nearest integer as the class prediction.
    8. \n
\n

I know that each of these steps could definitely be tuned for greater accuracy, and I would love to hear your feedback. In particular, I don't think the neural net architecture is optimal for these images. 

\n

[/quote]

", "votes": null, "replies": [] }, { "id": 66654, "author_name": "hoytkoepke1", "author_url": "", "post_date": "03/17/2015 07:31:20", "content": "

Hello Shize,

\n

That's a fair question. I used Dato's Graphlab Create to prototype different models to quickly get at a good solution, which is what I've posted. However, everything that I ended up using is either open source or can be substituted with another open source package. The SFrames is open sourced in dato-core, the GLC deep learning toolkit generates config files that are compatible with CXXNet, and the boosted tree regression can be translated to use xgboost.

\n

Thanks!

\n

-- Hoyt

\n

[quote=Shize Su;66626]

\n

Thanks for the great shared code!

\n

Just one thing which I hope you could clarify (thanks in advance): I noticed that your code used the Graphlab Create, which is not fully open source. Are you using any non-open source elements of the Graphlab Create in your code or not?

\n

Thanks!

\n

Best wishes,

\n

Shize

\n

[/quote]

", "votes": null, "replies": [] }, { "id": 67948, "author_name": "knarfben", "author_url": "", "post_date": "03/24/2015 11:50:00", "content": "

Hello,

\n

Running the create_image_sframes.py script I get the following errors:

\n

PROGRESS: Read 163807 images in 745.984 secs speed: 0 file/sec
PROGRESS: Unknown error reading image file: /data2/DRD/processed/run-hue-2/test/18344_right.jpeg
PROGRESS: Unknown error reading image file: /data2/DRD/processed/run-stretch/train/4400_right.jpeg
PROGRESS: Unknown error reading image file: /data2/DRD/processed/run-contrast-2/train/6210_left.jpeg
PROGRESS: Unknown error reading image file: /data2/DRD/processed/run-hue-2/train/27213_left.jpeg
PROGRESS: Unknown error reading image file: /data2/DRD/processed/run-sat-1/test/8671_left.jpeg

\n\n\n

any idea where this might come from?

\n

thanks.

", "votes": null, "replies": [] }, { "id": 67987, "author_name": "knarfben", "author_url": "", "post_date": "03/24/2015 15:34:17", "content": "

[quote=knarfben;67948]

\n

Hello,

\n

Running the create_image_sframes.py script I get the following errors:

\n

PROGRESS: Read 163807 images in 745.984 secs speed: 0 file/sec
PROGRESS: Unknown error reading image file: /data2/DRD/processed/run-hue-2/test/18344_right.jpeg
PROGRESS: Unknown error reading image file: /data2/DRD/processed/run-stretch/train/4400_right.jpeg
PROGRESS: Unknown error reading image file: /data2/DRD/processed/run-contrast-2/train/6210_left.jpeg
PROGRESS: Unknown error reading image file: /data2/DRD/processed/run-hue-2/train/27213_left.jpeg
PROGRESS: Unknown error reading image file: /data2/DRD/processed/run-sat-1/test/8671_left.jpeg

\n

any idea where this might come from?

\n

thanks.

\n

[/quote]

\n\n

ok.... not enough disk space, had to change GRAPHLAB_CACHE_FILE_LOCATIONS

", "votes": null, "replies": [] }, { "id": 67989, "author_name": "jiweiliu", "author_url": "", "post_date": "03/24/2015 15:45:20", "content": "

Thank you Hoyt! it takes my computer a whole week to run this benchmark and it gets 0.44. In the following months I'm going to figure out what is going on and replace all the Dato tool with open source tool. It will be a great learning process!

\n

Again, Dato is really too good for kaggle. Hope you could publish more analysis, ipython notebook style tutorial, to help us learn your thought process, besides generating full submission. I think most kagglers will be more happy to use Dato as an analysis tool. It is indeed really powerful. Great work!

", "votes": null, "replies": [] }, { "id": 68033, "author_name": "deepcnn", "author_url": "", "post_date": "03/24/2015 17:58:21", "content": "

@rcarson I ran this code for only the original dataset no hue, sat, ... processing and got 0.045. I was just curious what I could get in a day or so. Now, I am running the full set and expect me near you on the LB sometime this week :)

\n

As for the code, there is no magic to it. I actually think the network is so primitive (much more shallow (5 vs 2 conv layers) than the starter code based on cxxnet on the plankton competition though the fully connected layer has one more hidden layer).

\n

I think, the almost brute-force style pre-processing of color spaces, trying to balance the dataset, and extracting features from almost all kinds of class combinations are responsible for the higher score. This is evident in my score of only 0.045 on the original dataset (although balanced) vs. 0.44 or 0.46 for all this brute force. Overall, the data pre processing is not systematic rather brute-force and the network architecture is too shallow.

\n

Mind you although graphlab create net could be optimized, under the hood it is still cxxnet. 

\n

It is a good starting place regardless. 

", "votes": null, "replies": [] }, { "id": 68057, "author_name": "jiweiliu", "author_url": "", "post_date": "03/24/2015 19:33:59", "content": "

Thank you Hoyt. A kind reminder, multiple accounts are against Kaggle's rule. Luckily your new account doesn't enter this contest, so don't! and I suggest you delete the new account. You certainly don't need that. This happens to others who are new to kaggle.

", "votes": null, "replies": [] }, { "id": 68061, "author_name": "hoytkoepke1", "author_url": "", "post_date": "03/24/2015 19:41:37", "content": "

@rcarson -- Thank you for pointing this out.  I accidently logged in with the wrong google+ account and didn't even notice!  I closed that one and will just stick with this one. 

\n

In case the above post gets deleted, here is the text:

\n

---------------------------------------

\n

Hello @deep and @rcarson,

\n

I'm really glad you found my code helpful. I will try to make a notebook regarding my thought process here. I'm at work now, so I can't write much, but I'll try to fill in more when I get home.

\n

I agree with @deep -- this method is pretty brute force. The guiding principle was to create more observations that varied in the ways you want the NN to ignore. There are also a lot of different things that could be tuned and improved, especially with the NN architectures. I'm pretty new to neural nets, so I'll try playing around with it in the way you suggested.

\n

I think the other thing to realize here is that this is an ordinal regression problem rather than a classification problem. This motivated using the boosted regression trees over the activation levels in the final layers of the NN at the end to produce the result rather than using the straight NN to classify the images.

\n

@rcarson -- FYI, a lot of the Dato stuff is open source -- see https://github.com/dato-code/Dato-Core.

\n

@deep -- I have also found that sometimes what I'm doing here would overfit the training data, resulting in poorer predictions. E.g., it seemed like the NN overfit for the 0,1,2 vs 3,4 case based on the training / validation difference, and leaving that one out of the final regression problem resulted in a slightly better score. Just one thing I found.

", "votes": null, "replies": [] }, { "id": 68570, "author_name": "knarfben", "author_url": "", "post_date": "03/27/2015 13:25:48", "content": "

[quote=Hoyt Koepke;66564]

\n

Hello, 

\n

I just put up code that I used to get a leaderboard score of ~0.46412, and I would like to hear feedback on it.  Feel free to use it, and I am open to feedback.  I am sure many parts of it can be improved. 

\n

https://github.com/hoytak/diabetic-retinopathy-code

\n

This code uses the ImageMagick convert tool to preprocess the images, then uses the neural net toolkit (basically cxxnet) and the boosted regression trees in Dato's Graphlab create toolkits to build the classifier.  

\n

Main Ideas:

\n
    \n
  1. Use ImageMagick's convert tool to trim off the blank space to the sides of the images, then pad them so that they are all 256x256. Thus the eye is always centered with edges against the edges of the image. I avoided scaling the images to improve the neural net performance.
    2. \n
  2. Create multiple versions of each image varying by hue and contrast and white balance.  This step can be simplified to get to a quick though less accurate solution.
    3. \n
  3. Duplicate each class so each class is represented equally, then shuffle the data.
    4. \n
  4. Train several neural nets, one trained to predict the level membership, and another 4 to distinguish 0 vs 1-4, 0-1 vs. 2-4, 0-2 vs. 3-4, and 0-3 vs. 4.
    5. \n
  5. For each image, extract both class predictions and the values of the final neural net layer. Pool all of these features across models and variations of the same images.
    6. \n
  6. Train boosted regression trees on these pooled features to predict the level.
    7. \n
  7. Round to the nearest integer as the class prediction.
    8. \n
\n

I know that each of these steps could definitely be tuned for greater accuracy, and I would love to hear your feedback. In particular, I don't think the neural net architecture is optimal for these images. 

\n

[/quote]

\n\n

Hello Hoyt

\n\n

I ran into the following issue while running create_image_sframes.py:

\n

[INFO] Starting new HTTP connection (1): d3dou3jr4nway6.cloudfront.net
[INFO] Starting new HTTP connection (1): d3dou3jr4nway6.cloudfront.net
Get roughly equal class representation by duplicating the different levels.
Do a poor mans random shuffle
Unable to reach server for 3 consecutive pings. Server is considered dead. Please exit and restart.
Unable to reach server for 4 consecutive pings. Server is considered dead. Please exit and restart.
Traceback (most recent call last):
File "/home/ubuntu/drd/create_image_sframes.py", line 47, in <module>
X_train = X_train.sort("_random_")
File "/usr/local/lib/python2.7/dist-packages/graphlab/data_structures/sframe.py", line 4988, in sort
return SFrame(_proxy=self.__proxy__.sort(sort_column_names, sort_column_orders))
File "/usr/local/lib/python2.7/dist-packages/graphlab/cython/context.py", line 39, in __exit__
raise exc_type(exc_value)
RuntimeError: Communication Failure: 113.
Unable to reach server for 5 consecutive pings. Server is considered dead. Please exit and restart.
Unable to reach server for 6 consecutive pings. Server is considered dead. Please exit and restart.
Unable to reach server for 7 consecutive pings. Server is considered dead. Please exit and restart.
Unable to reach server for 8 consecutive pings. Server is considered dead. Please exit and restart.
Unable to reach server for 9 consecutive pings. Server is considered dead. Please exit and restart.
[INFO] Stopping the server connection.

\n

Any idea where this might come from? I ran the script many times and always end up with this error, although previous communication with the server were ok....

\n\n

thanks 

\n

Frank

", "votes": null, "replies": [] }, { "id": 68602, "author_name": "hoytkoepke1", "author_url": "", "post_date": "03/27/2015 18:02:14", "content": "

[quote=knarfben;68570]

\n

Hello Hoyt

\n

I ran into the following issue while running create_image_sframes.py:

\n

[INFO] Starting new HTTP connection (1): d3dou3jr4nway6.cloudfront.net
[INFO] Starting new HTTP connection (1): d3dou3jr4nway6.cloudfront.net
Get roughly equal class representation by duplicating the different levels.
Do a poor mans random shuffle
Unable to reach server for 3 consecutive pings. Server is considered dead. Please exit and restart.
Unable to reach server for 4 consecutive pings. Server is considered dead. Please exit and restart.
Traceback (most recent call last):
File "/home/ubuntu/drd/create_image_sframes.py", line 47, in <module>
X_train = X_train.sort("_random_")
File "/usr/local/lib/python2.7/dist-packages/graphlab/data_structures/sframe.py", line 4988, in sort
return SFrame(_proxy=self.__proxy__.sort(sort_column_names, sort_column_orders))
File "/usr/local/lib/python2.7/dist-packages/graphlab/cython/context.py", line 39, in __exit__
raise exc_type(exc_value)
RuntimeError: Communication Failure: 113.
Unable to reach server for 5 consecutive pings. Server is considered dead. Please exit and restart.
Unable to reach server for 6 consecutive pings. Server is considered dead. Please exit and restart.
Unable to reach server for 7 consecutive pings. Server is considered dead. Please exit and restart.
Unable to reach server for 8 consecutive pings. Server is considered dead. Please exit and restart.
Unable to reach server for 9 consecutive pings. Server is considered dead. Please exit and restart.
[INFO] Stopping the server connection.

\n

Any idea where this might come from? I ran the script many times and always end up with this error, although previous communication with the server were ok....

\n

thanks 

\n

Frank

\n

[/quote]

\n

Hello Frank, I also hit this error a couple of times -- hence the "if not os.path.exists(...)" line at the top, but running it a few times seems to get through it.  Sorting the SFrame with the images in it seems really expensive, and all the disk IO is causing something to time out. I filed a bug report on this, so hopefully it will get fixed in the next Graphlab create version.

\n

I'm also working on doing all of the operations up to this point with filenames instead of the actual images, which would be much faster.  

\n

Does it complete any of the rounds before crashing, or does it simply crash on the first one every time?  

", "votes": null, "replies": [] }, { "id": 68603, "author_name": "hoytkoepke1", "author_url": "", "post_date": "03/27/2015 18:07:58", "content": "", "votes": null, "replies": [] }, { "id": 68903, "author_name": "siddjain", "author_url": "", "post_date": "03/29/2015 21:32:30", "content": "

The g2.2xlarge instance in AWS comes with 60GB of hard disk space. Could you let me know how do you increase this space to accommodate the dataset we are dealing with here?

", "votes": null, "replies": [] }, { "id": 68904, "author_name": "jfkingiii", "author_url": "", "post_date": "03/29/2015 21:39:20", "content": "

Create and mount a volume of sufficient size. I think I used 300GB but that was more than I needed.

", "votes": null, "replies": [] }, { "id": 68914, "author_name": "siddjain", "author_url": "", "post_date": "03/29/2015 22:39:55", "content": "

Did you create a EBS volume? I am new to AWS. If you store something in the 60GB instance store that comes with g2.2xlarge, does that data get deleted when you stop your instance? **If so, why would anyone ever use that store**? Second question I have is that when I log into my machine, how do I know if I am accessing the instance store or the EBS? i.e., given a path like /home/ubuntu how to know is this directory is under instance store or EBS?

\n\n

References:

\n

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/AmazonEBS.html

\n

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-using-volumes.html

\n

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Storage.html

\n

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/InstanceStorage.html

", "votes": null, "replies": [] }, { "id": 68915, "author_name": "jfkingiii", "author_url": "", "post_date": "03/29/2015 22:57:29", "content": "

Yes it's an ebs volume. The data in the root device will disappear by default unless you select an option to make it persist when you create the AMI. The idea is to use it for software rather than data. 

\n

To create a volume and attach it to a running instance, select create volume at the volumes screen, and then right click to and select attach volume. It will probably say it's attaching to /dev/sdf but in reality it will be on /dev/xvdf on ubuntu. From your instance, execute

\n\n

sudo mkfs -t ext4 /dev/xvdf  # (only the first time you use the volume).

\n

sudo mkdir /data #  (replace /data with wherever you want to mount the volume).

\n

sudo mount /dev/xvdf /data

", "votes": null, "replies": [] }, { "id": 68947, "author_name": "siddjain", "author_url": "", "post_date": "03/30/2015 00:23:41", "content": "

hey i have one more question. i created a ebs volume and mounted it to /data but am unable to save anything in it because i lack the permissions. 

\n

ubuntu:~$ ls -all /data
total 24
drwxr-xr-x 3 root root 4096 Mar 30 00:04 .
drwxr-xr-x 24 root root 4096 Mar 30 00:12 ..

\n

what can i do so that root becomes ubuntu? whoami gives ubuntu.

", "votes": null, "replies": [] }, { "id": 68963, "author_name": "jfkingiii", "author_url": "", "post_date": "03/30/2015 02:29:50", "content": "

sudo chown ubuntu /data

\n

chmod if necessary

", "votes": null, "replies": [] }, { "id": 68980, "author_name": "siddjain", "author_url": "", "post_date": "03/30/2015 05:28:02", "content": "

thanks. that worked.

", "votes": null, "replies": [] }, { "id": 68986, "author_name": "deepcnn", "author_url": "", "post_date": "03/30/2015 07:05:40", "content": "

@Hoyt, just finished running your btb code. I like the intuition of pulling all the features and use GBT for classification instead of just straight softmax from the CNN.

\n

As you mentioned, the last two comparisons for 0-1-2 vs. 3-4 and 0-1-2-3 vs 4 overfit a lot (training: 0.91 and 0.95 vs. validation around 0.64 and 0.65) for this shallow network. This tells me that the network training params aren't quite optimal. For instance, adding regularization and adaptive learning rate could help.

\n

Great work overall and thanks for sharing!

", "votes": null, "replies": [] }, { "id": 69072, "author_name": "smallyellowduck", "author_url": "", "post_date": "03/30/2015 20:00:08", "content": "

@Hoyt, can you say a few sentences about why you decided to go with just two convolutional layers and a stride of 4? I'm mostly trying to understand why two conv layers might be considered sufficient and whether a stride of 4 (>1) was mostly selected to speed things up.

\n

This thread has been really instructive to me - thanks to all of you nice folks!

", "votes": null, "replies": [] }, { "id": 69074, "author_name": "hoytkoepke1", "author_url": "", "post_date": "03/30/2015 20:16:46", "content": "

@small yellow duck --

\n

To be honest, I'm by no means the expert in this thread on deep learning -- others probably have better advice to share, but I would be happy to tell you what I was thinking. 

\n

I based the architecture off of the configuration for imagenet, but I cut down a number of the parameters and the number of convolution layers.  The main reason for only choosing 2 convolution layers was that I didn't feel this problem needed as much location invariance as the imagenet task, which had more variance in where the relevant features were located.  After the processing by the imagemagick convert tool, a lot of relevant features appear in similar places in the image, so I didn't think as many convolution layers would be needed to capture it.

\n

As for the stride, I found that smaller strides generated out-of-memory errors on my GPU (a GTX 780), so that's the only reason for a stride of 4. 

", "votes": null, "replies": [] }, { "id": 69703, "author_name": "danielzhang1990", "author_url": "", "post_date": "04/05/2015 10:01:15", "content": "

Hello Hoyt

\n

Thanks for sharing ~~

\n

However, I ran into the following issue while running create_nn_model.py:

\n

File "create_nn_model.py", line 85, in <module>
mean_image = X_train["image"].mean()

\n

graphlab.toolkits._main.ToolkitError: Cannot perform sum or average over images of different sizes. Found images of total size (ie. width * height * channels) of both 196608 and 65536. Please use graplab.image_analysis.resize() to make images a uniform size.

\n

Any idea where this might come from? 

\n

thanks

", "votes": null, "replies": [] }, { "id": 69772, "author_name": "ajulian", "author_url": "", "post_date": "04/06/2015 06:29:35", "content": "

Since 196608 = 65536*3, it seems to me you are mixing 256x256 RGB (3 channels) and 256x256 grey level (1 channel) images, while the NN expects images with the same size AND number of channels.

\n

Just a guess...

", "votes": null, "replies": [] }, { "id": 69782, "author_name": "pavanikrishna0", "author_url": "", "post_date": "04/06/2015 08:58:59", "content": "

hi 

\n

how to start this project

", "votes": null, "replies": [] }, { "id": 70063, "author_name": "gobrewers14", "author_url": "", "post_date": "04/09/2015 00:30:36", "content": "

Another Dato "benchmark".  What is the purpose of posting code that puts people in the top ~7% with 3 months left in the competition (this is a sincere question)?

", "votes": null, "replies": [] }, { "id": 70065, "author_name": "inversion", "author_url": "", "post_date": "04/09/2015 01:07:31", "content": "

Better than 3 days before contest end (like in Plankton).

\n

[quote=John Martinez;70063]

\n

Another Dato "benchmark".  What is the purpose of posting code that puts people in the top ~7% with 3 months left in the competition (this is a sincere question)?

\n

[/quote]

", "votes": null, "replies": [] }, { "id": 70066, "author_name": "jessekolb", "author_url": "", "post_date": "04/09/2015 01:13:57", "content": "

[quote=John Martinez;70063]

\n

Another Dato "benchmark".  What is the purpose of posting code that puts people in the top ~7% with 3 months left in the competition (this is a sincere question)?

\n

[/quote]

\n

It may be the top ~7% currently but as others have mentioned it's not overly advanced and surely anyone who spent the length of the competition on their work would've bested it anyways.  Additionally, I think we can all agree that openly discussing models that detect diseases has benefits outside of the competition alone :-)

", "votes": null, "replies": [] }, { "id": 70071, "author_name": "gobrewers14", "author_url": "", "post_date": "04/09/2015 02:30:17", "content": "

[quote=J Kolb;70066]

\n

[quote=John Martinez;70063]

\n

Another Dato "benchmark".  What is the purpose of posting code that puts people in the top ~7% with 3 months left in the competition (this is a sincere question)?

\n

[/quote]

\n

It may be the top ~7% currently but as others have mentioned it's not overly advanced and surely anyone who spent the length of the competition on their work would've bested it anyways.  Additionally, I think we can all agree that openly discussing models that detect diseases has benefits outside of the competition alone :-)

\n

[/quote]

\n

Just saying, there is currently a user (not me) that is at 0.38 that has 59 submissions.  Assuming that "surely anyone who spent the length of the competition on their work would've bested it anyways" is an unfair assumption in my estimation.  Maybe that user did they best they possibly could and their work was original and may have been good enough to squeak into the top 25%.  Seeing a score posted in the forums that trumps your work has to be extremely disheartening.  

", "votes": null, "replies": [] }, { "id": 70072, "author_name": "jfkingiii", "author_url": "", "post_date": "04/09/2015 02:38:11", "content": "

It's like you're working on a puzzle, and you're really absorbed in it and making progress, and then someone comes along and tells you the answer.

", "votes": null, "replies": [] }, { "id": 70078, "author_name": "jessekolb", "author_url": "", "post_date": "04/09/2015 03:49:40", "content": "

[quote=James King;70072]

\n

It's like you're working on a puzzle, and you're really absorbed in it and making progress, and then someone comes along and tells you the answer.

\n

[/quote]

\n

I can understand the debate over how it affects people's scores, but this analogy doesn't really apply because if it's solely about working on the puzzle, no one makes you open the thread, much less go to his github, set up Graphlab, and do a couple days of processing and machine learning.  He stated in the title what was in his thread.

", "votes": null, "replies": [] }, { "id": 70079, "author_name": "jessekolb", "author_url": "", "post_date": "04/09/2015 03:57:56", "content": "

John,

\n

Assuming the converse is true, and the given person would not be able to beat the score of .46 after four months of work, it gives them the ability to peek into what other people are doing, if they wish, and not spend those four months without obtaining that result.  They don't have to, however, and can still try things on their own, but at least they have that option.  Also, it's well before the deadline so it's not like people are getting upended last minute.  I'm not going to go further into whether it is disheartening vs. enlightening because I think this has already been beaten like a dead horse and there is obviously cases where each is true.  I do want to also reiterate that this is an active research topic.  Would you ask a research group not to publish their results because another research group is still working on the same problem?

", "votes": null, "replies": [] }, { "id": 70083, "author_name": "gobrewers14", "author_url": "", "post_date": "04/09/2015 04:33:19", "content": "

[quote=J Kolb;70079]

\n

John,

\n

Assuming the converse is true, and the given person would not be able to beat the score of .46 after four months of work, it gives them the ability to peek into what other people are doing, if they wish, and not spend those four months without obtaining that result.  They don't have to, however, and can still try things on their own, but at least they have that option.  Also, it's well before the deadline so it's not like people are getting upended last minute.  I'm not going to go farther into whether it is disheartening vs. enlightening because I think this has already been beaten like a dead horse and there is obviously cases where each is true.  I do want to also reiterate that this is an active research topic.  Would you ask a research group not to publish their results because another research group is still working on the same problem?

\n

[/quote]

\n

My objections have nothing to do with publishing results. At the heart of scientific discovery is collaboration. It’s what makes this site and science in general, pretty awesome. My objections are with a company whoring out their products at the expense of others hard work. Users have posted successful models, plugging the same company’s tools, in this, the Otto challenge, and as inversion mentioned, one 3 days before the deadline in the Plankton competition, which easily put you in the top 25%. I am aware I have no results on this site to speak of and am not an influential member of this community, so if my objections are unwarranted, I will certainly acquiesce but I must remind those reading of the Law of Unintended Consequences; I had never heard of Dato until a few weeks ago. Maybe that was a good thing for them …

", "votes": null, "replies": [] }, { "id": 70084, "author_name": "hoytkoepke1", "author_url": "", "post_date": "04/09/2015 04:34:41", "content": "

[quote=John Martinez;70063]

\n

Another Dato "benchmark".  What is the purpose of posting code that puts people in the top ~7% with 3 months left in the competition (this is a sincere question)?

\n

[/quote]

\n

Hi John, thanks for your question, and hopefully I can give you a satisfactory answer.  

I posted the code and write-up of my insights into the problem in the hope that others would be able to build on my ideas, incorporate their own, and hopefully come out ahead of either approach individually.  I've been a researcher in machine learning for over 9 years -- first in my PhD program, and now at Dato -- and most of my research advances have benefitted greatly from open collaboration and an open exchange of ideas.  As a result, I'm always free to share what insights I have in the hopes someone can build on them.  I know this isn't necessarily academia, but this problem is still an open research problem, and there are several research papers posted elsewhere in the forum describing techniques that allegedly get a much better kappa score than my method.  

I simply don't have the free time needed to win this contest, so that's not my aim; rather, I see it as an interesting and very difficult research problem that I want to feel like I can contribute to.  And, from experience, I'll bet the winning idea is likely going to be a combination of openly shared ideas, published research, and original insights -- but it definitely won't be entirely original ideas. 

", "votes": null, "replies": [] }, { "id": 70085, "author_name": "jessekolb", "author_url": "", "post_date": "04/09/2015 04:51:48", "content": "

[quote=Hoyt Koepke;68061]

\n

I think the other thing to realize here is that this is an ordinal regression problem rather than a classification problem. This motivated using the boosted regression trees over the activation levels in the final layers of the NN at the end to produce the result rather than using the straight NN to classify the images.

\n

[/quote]

\n

I'm not sure I understand your thought process here.  Wouldn't it be possible to use the Neural net for a regression problem and then round the result to the nearest value of 0, 1, 2, 3, or 4 instead of doing a one-vs-all classification? And then you wouldn't use the boosted trees at all.  What is the benefit of using the trees then?  Does using the NN for regression not work well in practice in this situation?

", "votes": null, "replies": [] }, { "id": 70086, "author_name": "hoytkoepke1", "author_url": "", "post_date": "04/09/2015 05:00:15", "content": "

[quote=John Martinez;70083]

\n

[quote=J Kolb;70079]

\n

John,

\n

Assuming the converse is true, and the given person would not be able to beat the score of .46 after four months of work, it gives them the ability to peek into what other people are doing, if they wish, and not spend those four months without obtaining that result.  They don't have to, however, and can still try things on their own, but at least they have that option.  Also, it's well before the deadline so it's not like people are getting upended last minute.  I'm not going to go farther into whether it is disheartening vs. enlightening because I think this has already been beaten like a dead horse and there is obviously cases where each is true.  I do want to also reiterate that this is an active research topic.  Would you ask a research group not to publish their results because another research group is still working on the same problem?

\n

[/quote]

\n

My objections have nothing to do with publishing results. At the heart of scientific discovery is collaboration. It’s what makes this site and science in general, pretty awesome. My objections are with a company whoring out their products at the expense of others hard work. Users have posted successful models, plugging the same company’s tools, in this, the Otto challenge, and as inversion mentioned, one 3 days before the deadline in the Plankton competition, which easily put you in the top 25%. I am aware I have no results on this site to speak of and am not an influential member of this community, so if my objections are unwarranted, I will certainly acquiesce but I must remind those reading of the Law of Unintended Consequences; I had never heard of Dato until a few weeks ago. Maybe that was a good thing for them …

\n

[/quote]

\n

Hi John, 

\n

I hope you don't think that poorly of Dato :-).  We're a fairly new startup company originally formed in part to support Graphlab, an open source academic project, and now we have expanded to provide other tools for machine learning research and general data science.  Because of our academic roots, we've made sure our stuff here is either open source or completely free for academic/non-commercial use to benefit research-oriented endeavors like this.  As mentioned earlier, all of my code can either use our open source offering or be translated into other open source tools.  I used it since it's (obviously) what I'm most familiar with, so I find it easier to quickly prototype ideas that way.   

", "votes": null, "replies": [] }, { "id": 70088, "author_name": "hoytkoepke1", "author_url": "", "post_date": "04/09/2015 05:06:51", "content": "

[quote=J Kolb;70085]

\n

[quote=Hoyt Koepke;68061]

\n

I think the other thing to realize here is that this is an ordinal regression problem rather than a classification problem. This motivated using the boosted regression trees over the activation levels in the final layers of the NN at the end to produce the result rather than using the straight NN to classify the images.

\n

[/quote]

\n

I'm not sure I understand your thought process here.  Wouldn't it be possible to use the Neural net for a regression problem and then round the result to the nearest value of 0, 1, 2, 3, or 4 instead of doing a one-vs-all classification? And then you wouldn't use the boosted trees at all.  What is the benefit of using the trees then?  Does using the NN for regression not work well in practice in this situation?

\n

[/quote]

\n

Hi J Kolb,

\n

It's likely that you could get NN regression to work in this context, but I haven't really tried.  The main reason is 2-fold.  First, I had several NN models at the end, and this presented a quick and dirty way to ensemble them together.  (I tried other classifiers and other tricks, but this was the only one that worked).  Second, it took me a day or so to train each of the neural nets, but only a minute or two to train the boosted tree model at the end.  By playing around with the boosted tree parameters, I was able to get a much better final score without spending 8 hours between each attempt.

", "votes": null, "replies": [] }, { "id": 70177, "author_name": "nthanhtam", "author_url": "", "post_date": "04/10/2015 06:53:49", "content": "

[quote=zyy1990;69703]

\n

Hello Hoyt

\n

Thanks for sharing ~~

\n

However, I ran into the following issue while running create_nn_model.py:

\n

File "create_nn_model.py", line 85, in <module>
mean_image = X_train["image"].mean()

\n

graphlab.toolkits._main.ToolkitError: Cannot perform sum or average over images of different sizes. Found images of total size (ie. width * height * channels) of both 196608 and 65536. Please use graplab.image_analysis.resize() to make images a uniform size.

\n

Any idea where this might come from? 

\n

thanks

\n

[/quote]

\n

zyy1990: Have you solved this problem?

", "votes": null, "replies": [] }, { "id": 70186, "author_name": "danielzhang1990", "author_url": "", "post_date": "04/10/2015 07:29:29", "content": "

[quote=vnneo;70177]

\n

[quote=zyy1990;69703]

\n

Hello Hoyt

\n

Thanks for sharing ~~

\n

However, I ran into the following issue while running create_nn_model.py:

\n

File "create_nn_model.py", line 85, in <module>
mean_image = X_train["image"].mean()

\n

graphlab.toolkits._main.ToolkitError: Cannot perform sum or average over images of different sizes. Found images of total size (ie. width * height * channels) of both 196608 and 65536. Please use graplab.image_analysis.resize() to make images a uniform size.

\n

Any idea where this might come from? 

\n

thanks

\n

[/quote]

\n

zyy1990: Have you solved this problem?

\n

[/quote]

\n\n

Yes. For some unknown reason, the data i downloaded had many images which were all black, so i picked   those images out. Here are the id of those images:

\n

train/41176_left.jpeg
train/34689_left.jpeg
train/32253_right.jpeg
train/43457_left.jpeg
train/1986_left.jpeg
test/28544_left.jpeg
test/16519_left.jpeg
test/38549_left.jpeg
test/35433_left.jpeg
test/3517_left.jpeg
test/35762_left.jpeg

\n

train/15222_left.jpeg
test/16349_left.jpeg

", "votes": null, "replies": [] }, { "id": 71221, "author_name": "budiansky", "author_url": "", "post_date": "04/10/2015 15:44:27", "content": "

[quote=James King;70072]

\n

It's like you're working on a puzzle, and you're really absorbed in it and making progress, and then someone comes along and tells you the answer.

\n

[/quote]

\n\n

I agree.

\n

My opinion was that posting this solution was poor sportsmanship. If part of the motivation was to market a company's tools, then it leads me to have a negative opinion of that company as well. Seeing this solution decreased my motivation to participant in this contest.

\n

Obviously the original poster did not think feel that way, so what we have right now is a difference of opinion. However, we have an opportunity to resolve this question by means of (yes!) data.

\n

If lots of people agree with me, then as a community we should discourage this kind of posting, and the site administrators should note that it will be bad for Kaggle if people are turned off by the practice and stop participating. If people think this kind of posting is a good idea, then we should encourage people to post full solutions early and often.

\n

Please voice your opinion. What do you think?

", "votes": null, "replies": [] }, { "id": 71224, "author_name": "jbools", "author_url": "", "post_date": "04/10/2015 15:56:11", "content": "

28 people upvoted the OP... so, we've already had a vote of sorts. Please don't conflate self-promotion with code sharing. Although code sharing is not entirely altruistic, I am fundamentally for it (both in general and specifically in Kaggle competitions). I myself was motivated (rather than discouraged) by this thread. I felt stimulated to brainstorm ways to improve upon the OP's score, yet also comforted that if I failed to get a good model during these next few months, that I could fall back on this shared code. A HUGE thank you to the OP.

\n\n

My strong opinion is that collaboration is beautiful. If you don't want to look at the shared code, don't look at it. Easy solution. Live and let live.

\n\n

Perhaps, another question we should ask (for another thread), is at what point does it become rude to hijack someone else's thread simply because you have a different opinion?

", "votes": null, "replies": [] }, { "id": 71227, "author_name": "woolsey", "author_url": "", "post_date": "04/10/2015 16:26:40", "content": "

[quote=Michael 7402;71221]Please voice your opinion. What do you think?[/quote]

\n

Idem datanewb: please stop this flood. (yeah.. self-defeating demand :-) )

", "votes": null, "replies": [] }, { "id": 71229, "author_name": "gobrewers14", "author_url": "", "post_date": "04/10/2015 16:57:53", "content": "

[quote=datanewb;71224]

\n

28 people upvoted the OP... so, we've already had a vote of sorts. Please don't conflate self-promotion with code sharing. Although code sharing is not entirely altruistic, I am fundamentally for it (both in general and specifically in Kaggle competitions). I myself was motivated (rather than discouraged) by this thread. I felt stimulated to brainstorm ways to improve upon the OP's score, yet also comforted that if I failed to get a good model during these next few months, that I could fall back on this shared code. A HUGE thank you to the OP.

\n

My strong opinion is that collaboration is beautiful. If you don't want to look at the shared code, don't look at it. Easy solution. Live and let live.

\n

Perhaps, another question we should ask (for another thread), is at what point does it become rude to hijack someone else's thread simply because you have a different opinion?

\n

[/quote]

\n

There was no conflation of self-promotion and code-sharing.  The evidence is in the forums of those 3 competitions (and maybe others), as I stated.  I completely agree, and would assume that Kaggle does as well, that collaboration is a beautiful thing, which is why we are allowed to form Teams. 

", "votes": null, "replies": [] }, { "id": 71248, "author_name": "inversion", "author_url": "", "post_date": "04/10/2015 20:51:45", "content": "

[quote=datanewb;71224]

\n

if I failed to get a good model during these next few months, that I could fall back on this shared code.

\n

[/quote]

\n

At the detriment of everyone you passed over on the leader board. Just by copying and pasting code. Great. Now the LB is no longer reflective of what you are capable of.

\n

This issue could be completely resolved if the Dato folks posted models without optimized hyperparameters, and perhaps even stating what the model is capable of. And let their users work at it a bit.

\n

In fact, they'd be doing their users a favor.

", "votes": null, "replies": [] }, { "id": 71249, "author_name": "spaceman", "author_url": "", "post_date": "04/10/2015 21:08:44", "content": "

This is a hard one ----yes or no to.

\n

I follow a good many Kaggle Forums and to be open I have learned quite a few things by looking into everything everyone has posted and have learned new ways to compete in this environment.

\n

Being a Novice here (or anywhere for that matter) for this type of competition I initially had no idea how I was going to use what I know about ML to compete.

\n

I agree with all of you these type of posing are a type of self promotion.... by thanks to a few of them I am now feeling a bit more confident about being able to participate in these competitions and see nothing wrong as long as we are able to learn something from the post.... even if this particular one seem to be very self-promoting.... IMHO

\n

I do commend you all for coming out against it to keep self-promotion under control 

", "votes": null, "replies": [] }, { "id": 71382, "author_name": "", "author_url": "", "post_date": "04/13/2015 12:09:41", "content": "

Hi Hoyt,

\n

Thank you for sharing!

\n

Do you have any data, what is the best benchmark with this base solution so far?

", "votes": null, "replies": [] }, { "id": 71397, "author_name": "", "author_url": "", "post_date": "04/13/2015 17:55:08", "content": "

Hello Hoyt,

\n

I've just started the NN training process on my GTX 980 and since you wrote: "and about 320 images / second on a GTX 780" in the readme, I expected about 20% better performance on my configuration.

\n

I got 280 img/s.

\n

Do you have any idea what can be the reason?

", "votes": null, "replies": [] }, { "id": 71399, "author_name": "hoytkoepke1", "author_url": "", "post_date": "04/13/2015 18:16:20", "content": "

[quote=George Solymosi;71382]

\n

Hi Hoyt,

\n

Thank you for sharing!

\n

Do you have any data, what is the best benchmark with this base solution so far?

\n

[/quote]

\n

Hi George,

\n

No, I'm not sure at this point.  It's a pretty crude approach; there are a lot of parameters to tune, and I've since realized there are a lot of things holding back the accuracy that become obvious when you examine the output solutions.  In particular, as others have mentioned, the NN architecture here overfits to some parts of the the training data, it doesn't take into account the unbalanced class sizes at prediction time, etc.  I got somewhere close to 0.46 with this solution; some tweaks to boosted tree algorithm (unpublished) boosted the score somewhat.   I'm sure that more sophistication in this approach, a better NN architecture, and/or ensembling this model with other approaches is needed to get much better than this.

", "votes": null, "replies": [] }, { "id": 71415, "author_name": "seanv507", "author_url": "", "post_date": "04/13/2015 23:06:20", "content": "

@hoyt, just some comments on the image processing... I have yet to implement something myself

\n

a) equalisation is normally only done on the intensity component so that you don't distort the colour balance (ie you don't want to stretch each colour channel separately)

\n

http://www.imagemagick.org/script/command-line-options.php#equalize

\n\n

b) i would think that its better to convert to png rather than jpg (which might introduce new jpeg artifacts at 256x256 size?)

", "votes": null, "replies": [] }, { "id": 71417, "author_name": "hoytkoepke1", "author_url": "", "post_date": "04/13/2015 23:11:41", "content": "

[quote=George Solymosi;71397]

\n

Hello Hoyt,

\n

I've just started the NN training process on my GTX 980 and since you wrote: "and about 320 images / second on a GTX 780" in the readme, I expected about 20% better performance on my configuration.

\n

I got 280 img/s.

\n

Do you have any idea what can be the reason?

\n

[/quote]

\n

Hey George, 

\n

Sorry -- I have no idea.  I suspect it also has to do with the other configurations of your system. The only thing I can think of is that it's IO bound in moving data on and off the card -- maybe it's mounted in the x8 PCIe slot instead of the x16 PCIe slot (although that shouldn't make a difference).  

\n

The batch_size parameter controls how many images are loaded onto the graphics card at a time for training.  It's possible that adjusting this parameter will help with performance.

", "votes": null, "replies": [] }, { "id": 71418, "author_name": "hoytkoepke1", "author_url": "", "post_date": "04/13/2015 23:16:15", "content": "

[quote=Sean;71415]

\n

@hoyt, just some comments on the image processing... I have yet to implement something myself

\n

a) equalisation is normally only done on the intensity component so that you don't distort the colour balance (ie you don't want to stretch each colour channel separately)

\n

http://www.imagemagick.org/script/command-line-options.php#equalize

\n

b) i would think that its better to convert to png rather than jpg (which might introduce new jpeg artifacts at 256x256 size?)

\n

[/quote]

\n

Hey Sean, 

\n

Thanks for your comments!  I called equalize mainly so that the 3 different channels of the NN would be on the same scale, although I'm not sure that would have any significant effect.  And yes, png might be better.  I'll give that a shot for my latest iteration. 

", "votes": null, "replies": [] }, { "id": 72504, "author_name": "", "author_url": "", "post_date": "04/19/2015 19:26:31", "content": "

We can fight with each other, but if we can quicker get closer with a bit help of something like Hoyt's shortcut to a better solution for help to prevent and cure people's illness, then I don't think that, that is unfair...

\n

Consider it.

", "votes": null, "replies": [] }, { "id": 72719, "author_name": "", "author_url": "", "post_date": "04/20/2015 19:06:49", "content": "

[quote=Hoyt Koepke;66564]

\n

Use ImageMagick's convert tool to trim off the blank space to the sides of the images, then pad them so that they are all 256x256. Thus the eye is always centered with edges against the edges of the image. I avoided scaling the images to improve the neural net performance.

\n

[/quote]

\n

I'm a bit puzzled though, you say you convert images to 256x256 and at the same time you say you avoid scaling the images. Could you kindly clarify this part?

", "votes": null, "replies": [] }, { "id": 72986, "author_name": "hoytkoepke1", "author_url": "", "post_date": "04/21/2015 17:58:09", "content": "

[quote=K0stIa;72719]

\n

I'm a bit puzzled though, you say you convert images to 256x256 and at the same time you say you avoid scaling the images. Could you kindly clarify this part?

\n

[/quote]

\n

Sorry, I could have been clearer there.  What I meant is that I made sure that the aspect ratio of the images and the location of the eyes in the picture stayed the same so that the resulting NN did not have to handle scale and location invariance along with learning to classify the images.  

\n\n

Hope that helps!

", "votes": null, "replies": [] }, { "id": 73007, "author_name": "spaceman", "author_url": "", "post_date": "04/21/2015 18:50:03", "content": "

You do realize that the beauty of CNN's that scaling and translation does really cause any performance penalties of the CNN's..... However make sure you don't scale away the useful information ;)

", "votes": null, "replies": [] }, { "id": 73184, "author_name": "zsaurk", "author_url": "", "post_date": "04/22/2015 15:24:24", "content": "

Thanks Dato Inc / Hoyt for trying to democratize Big Data / Deep Learning algorithms. I was curious what pretrained filters the GraphLab's deeplearning algorithm make use of ? Is it Gabor Filters or BSD Caffe or something else ?

", "votes": null, "replies": [] }, { "id": 74244, "author_name": "jstaker7", "author_url": "", "post_date": "04/23/2015 04:23:32", "content": "

Hi Hoyt,

\n

I'm a little confused on the performance you mentioned. 300 images/s tells me each epoch takes around 2 min, and 5-10 epochs for each 2-class model comes out to be 20 min of training. How does this translate to about a day (24h) worth of training? Would you mind clarifying? I'm using theano so I'm curious how my performance compares to your cxxnet implementation.

", "votes": null, "replies": [] }, { "id": 74358, "author_name": "hoytkoepke1", "author_url": "", "post_date": "04/23/2015 18:12:36", "content": "

[quote=Michael George Hart;73007]

\n

You do realize that the beauty of CNN's that scaling and translation does really cause any performance penalties of the CNN's..... However make sure you don't scale away the useful information ;)

\n

[/quote]

\n

Hi Michael, 

\n

Yes, true, though in general, when you add more convolution/pooling layers in order to get closer to true scale invariance, you end up needing more data in order to get it to train effectively.  I opted for having much fewer convolution layers than is needed for true translation invariance, since it seemed like location in the image could actually be informative.  I think there may be something to this approach, as I've been trying to train with deeper architectures since then, but haven't really got significantly better results yet.

", "votes": null, "replies": [] }, { "id": 74360, "author_name": "hoytkoepke1", "author_url": "", "post_date": "04/23/2015 18:15:58", "content": "

[quote=saurk;73184]

\n

Thanks Dato Inc / Hoyt for trying to democratize Big Data / Deep Learning algorithms. I was curious what pretrained filters the GraphLab's deeplearning algorithm make use of ? Is it Gabor Filters or BSD Caffe or something else ?

\n

[/quote]

\n

Hello Saurk,

\n

Thanks!  Under the hood, graphlab create doesn't use any pretrained filters. However, the filters in the convolution layers will often end up resembling Gabor filters, but they are learned directly from the data during training.  

\n

I think Caffe will end up doing the same thing...

\n

-- Hoyt

", "votes": null, "replies": [] }, { "id": 74361, "author_name": "hoytkoepke1", "author_url": "", "post_date": "04/23/2015 18:16:22", "content": "

[quote=jstaker7;74244]

\n

Hi Hoyt,

\n

I'm a little confused on the performance you mentioned. 300 images/s tells me each epoch takes around 2 min, and 5-10 epochs for each 2-class model comes out to be 20 min of training. How does this translate to about a day (24h) worth of training? Would you mind clarifying? I'm using theano so I'm curious how my performance compares to your cxxnet implementation.

\n

[/quote]

", "votes": null, "replies": [] }, { "id": 74364, "author_name": "hoytkoepke1", "author_url": "", "post_date": "04/23/2015 18:30:44", "content": "

[quote=jstaker7;74244]

\n

Hi Hoyt,

\n

I'm a little confused on the performance you mentioned. 300 images/s tells me each epoch takes around 2 min, and 5-10 epochs for each 2-class model comes out to be 20 min of training. How does this translate to about a day (24h) worth of training? Would you mind clarifying? I'm using theano so I'm curious how my performance compares to your cxxnet implementation.

\n

[/quote]

\n

Hello jstaker,

\n

The code I use brute-force creates a bunch of different versions of each image, as well as duplicating some of them to balance the classes in the training set.  Thus there end up being hundreds of thousands of images trained in each epoch instead of the original 35,000 or so.  

", "votes": null, "replies": [] }, { "id": 74417, "author_name": "jstaker7", "author_url": "", "post_date": "04/24/2015 00:44:22", "content": "

[quote=Hoyt Koepke;74358]

\n

... I opted for having much fewer convolution layers than is needed for true translation invariance, since it seemed like location in the image could actually be informative...

\n

[/quote]

\n\n

This challenges my understanding of CNNs. Isn't just feature detection translation invariant? I thought one of the great advantages of CNNs is that it can capture spacial relationships. Am I understanding your comment correctly?

", "votes": null, "replies": [] }, { "id": 74445, "author_name": "ionelhosu", "author_url": "", "post_date": "04/24/2015 06:36:51", "content": "

[quote=jstaker7;74417]

\n

[quote=Hoyt Koepke;74358]

\n

... I opted for having much fewer convolution layers than is needed for true translation invariance, since it seemed like location in the image could actually be informative...

\n

[/quote]

\n

This challenges my understanding of CNNs. Isn't just feature detection translation invariant? I thought one of the great advantages of CNNs is that it can capture spacial relationships. Am I understanding your comment correctly?

\n

[/quote]

\n

CNNs can learn not only translation invariance, but even scale and rotation invariances, but for this to happen, those characteristics need to be derived from the training set, i.e. the independence between (position, rotation, scale) of the features and class label needs to be present in the dataset. If, for some reason, some examples contain some features which are positioned constantly in the same general area of the image, even the translation invariance will go away, because the CNN would 'think' in this case that the position of those features should affect the decision in some way, i.e. the position of the feature and the class label would not be independent and identically distributed (i.i.d.) any more. In conclusion, CNNs CAN learn a lot of invariances, but in order for them to learn, those invariances need to be present in the training set, and this is why data augmentation is done.

", "votes": null, "replies": [] }, { "id": 75145, "author_name": "hoytkoepke1", "author_url": "", "post_date": "04/27/2015 23:47:21", "content": "

I'm not an expert on CNNs, but my understanding is that the translation invariance comes purely from the pooling layers.  Thus CNNs are only truly translation invariant if there are enough pooling layers that sufficiently significant features detected on one side of the image can affect the outcome of any of the nodes before the final fully connected layers.  In my original net, I didn't have enough of these layers to achieve this, so it then relied on the fully connected layers at the end to do more of the work.

\n

Similarly, scale and rotation invariances come from the number of layers and having enough channels in the convolution layers to adequately fit the range of features present in the data. 

\n

The problem, however, is that the more channels and layers you add, the more parameters need to be trained, and thus the more data you need to train the net without overfitting.  Ideally, you want a complicated enough structure to capture all relevant features, but a simple enough structure that it doesn't overfit (though dropout and regularization can help with that). 

", "votes": null, "replies": [] }, { "id": 75242, "author_name": "spaceman", "author_url": "", "post_date": "04/28/2015 13:38:09", "content": "

Yes pooling helps with the invariance ... A type of averaging mechanism.....

\n

however it is the Convolution Layer, a, more realistic, model of the visual system, that makes i variances .... Get the spacing right on the convolution will be more useful for this problem than the adding more pooling. ... Most of the work has already done for us by GoogLeNet, AlexNet etc... The trick here is to modify those convolution layers and perhaps tweeting the pooling layers to meet the needs of this particular problem....

\n

i imagine many people already see this as a solution... That is why I am only using it as a starting point to see how far I can get in the rankings, because the competition will come to who can tune the existing CNN's for a solution to this challenge.... 

\n

To win this challenge someone will have to step outside the CNN box since it is such an abvious solution... ;)

", "votes": null, "replies": [] }, { "id": 77297, "author_name": "donfulanito", "author_url": "", "post_date": "05/06/2015 11:02:26", "content": "

Kaggle has made their position clear on this in the competition rules I think: "It's okay to share code if made available to all participants on the forums."

\n

Why would publishing any competition entry bad or discouraging?

\n

- it's absolutely *not* the same as someone coming and telling you the 'answer to a puzzle' as you say.

\n

Here, you are not forced to read the forums, let alone study/run the code on github.

\n

I think that *anyone* who reads these forums is doing so *exactly* because they want to find out what other people are doing.

\n

- how is seeing a solution that scores better than you discouraging? You already can see in the leaderboard people's scores. Obviously those who scored above you have a solution to go with it, there is no surprise there.

\n

- if there is a tool that can help, and it's allowed by the rules (perhaps), why not use it? You say "seeing this solution decreased by motivation to participant [sic] in this contest", others would say "seeing this solution increased my motivation" as they have something to get started with.

\n

[quote=Michael 7402;71221]

\n

[quote=James King;70072]

\n

It's like you're working on a puzzle, and you're really absorbed in it and making progress, and then someone comes along and tells you the answer.

\n

[/quote]

\n

I agree.

\n

My opinion was that posting this solution was poor sportsmanship. If part of the motivation was to market a company's tools, then it leads me to have a negative opinion of that company as well. Seeing this solution decreased my motivation to participant in this contest.

\n

Obviously the original poster did not think feel that way, so what we have right now is a difference of opinion. However, we have an opportunity to resolve this question by means of (yes!) data.

\n

If lots of people agree with me, then as a community we should discourage this kind of posting, and the site administrators should note that it will be bad for Kaggle if people are turned off by the practice and stop participating. If people think this kind of posting is a good idea, then we should encourage people to post full solutions early and often.

\n

Please voice your opinion. What do you think?

\n

[/quote]

", "votes": null, "replies": [] }, { "id": 80553, "author_name": "rajkumarb", "author_url": "", "post_date": "06/01/2015 06:46:22", "content": "

Hi Hoyt,

\n

Thanks for the code. During prediction, I am getting different prediction levels for the same image in different runs. Please help me to sort out this. The results are as follows for consecutive runs of same set of images.

\n

+-------------+-------+----------------+
| name | level | actual_level |
+-------------+-------+----------------+
| 10046_right | 1 | 0.751127591425 |
| 11730_right | 0 | 0.358918072124 |
+-------------+-------+----------------+

\n

+-------------+-------+----------------+
| name | level | actual_level |
+-------------+-------+----------------+
| 10046_right | 0 | 0.423049250692 |
| 11730_right | 0 | 0.326992818052 |
+-------------+-------+----------------+

\n

+-------------+-------+----------------+
| name | level | actual_level |
+-------------+-------+----------------+
| 10046_right | 1 | 0.70960257402 |
| 11730_right | 0 | 0.357797767757 |
+-------------+-------+----------------+

\n

+-------------+-------+----------------+
| name | level | actual_level |
+-------------+-------+----------------+
| 10046_right | 0 | 0.408557624363 |
| 11730_right | 0 | 0.355133798881 |
+-------------+-------+----------------+

", "votes": null, "replies": [] }, { "id": 81044, "author_name": "fisherman", "author_url": "", "post_date": "06/05/2015 19:35:40", "content": "

Hi Hoyt,

\n

Thanks for posting this started code. I am impressed by what Graphlab can do with just a few hundred lines of Python code! I tried to run your Github code but looks like the Graphlab API crashed in sframe sorting - see below and the log file attached. If I comment out the sorting code then it will run fine. Any idea?

\n

I'm running Ubuntu 14.04 on a desktop Intel Core i7-5820K CPU @ 3.30GHz × 12, with 8GB Memory plus GeForce GTX 970 GPUs.

\n

=========================

\n

Traceback (most recent call last):
File "create_image_sframes.py", line 47, in <module>
X_train = X_train.sort("_random_")
File "/home/ekang/anaconda/lib/python2.7/dist-packages/graphlab/data_structures/sframe.py", line 5077, in sort
return SFrame(_proxy=self.__proxy__.sort(sort_column_names, sort_column_orders))
File "/home/ekang/anaconda/lib/python2.7/dist-packages/graphlab/cython/context.py", line 31, in __exit__
raise exc_type(exc_value)
RuntimeError: Communication Failure: 113.

\n

[quote=Hoyt Koepke;66564]

\n

Hello, 

\n

I just put up code that I used to get a leaderboard score of ~0.46412, and I would like to hear feedback on it.  Feel free to use it, and I am open to feedback.  I am sure many parts of it can be improved. 

\n

https://github.com/hoytak/diabetic-retinopathy-code

\n

This code uses the ImageMagick convert tool to preprocess the images, then uses the neural net toolkit (basically cxxnet) and the boosted regression trees in Dato's Graphlab create toolkits to build the classifier.  

\n

Main Ideas:

\n
    \n
  1. Use ImageMagick's convert tool to trim off the blank space to the sides of the images, then pad them so that they are all 256x256. Thus the eye is always centered with edges against the edges of the image. I avoided scaling the images to improve the neural net performance.
    2. \n
  2. Create multiple versions of each image varying by hue and contrast and white balance.  This step can be simplified to get to a quick though less accurate solution.
    3. \n
  3. Duplicate each class so each class is represented equally, then shuffle the data.
    4. \n
  4. Train several neural nets, one trained to predict the level membership, and another 4 to distinguish 0 vs 1-4, 0-1 vs. 2-4, 0-2 vs. 3-4, and 0-3 vs. 4.
    5. \n
  5. For each image, extract both class predictions and the values of the final neural net layer. Pool all of these features across models and variations of the same images.
    6. \n
  6. Train boosted regression trees on these pooled features to predict the level.
    7. \n
  7. Round to the nearest integer as the class prediction.
    8. \n
\n

I know that each of these steps could definitely be tuned for greater accuracy, and I would love to hear your feedback. In particular, I don't think the neural net architecture is optimal for these images. 

\n

[/quote]

", "votes": null, "replies": [] }, { "id": 81046, "author_name": "hoytkoepke1", "author_url": "", "post_date": "06/05/2015 19:52:09", "content": "

Hi Rajkumar,

\n

It turns out that this is a known issue that the NN people here are working on.  I'm not sure there's a way around it now.  I think if you use the boosted tree regression as a post-processing step on the activation layers, it's a bit more robust. 

\n

-- Hoyt

\n\n

[quote=Rajkumar;80553]

\n

Hi Hoyt,

\n

Thanks for the code. During prediction, I am getting different prediction levels for the same image in different runs. Please help me to sort out this. The results are as follows for consecutive runs of same set of images.

\n

+-------------+-------+----------------+
| name | level | actual_level |
+-------------+-------+----------------+
| 10046_right | 1 | 0.751127591425 |
| 11730_right | 0 | 0.358918072124 |
+-------------+-------+----------------+

\n

+-------------+-------+----------------+
| name | level | actual_level |
+-------------+-------+----------------+
| 10046_right | 0 | 0.423049250692 |
| 11730_right | 0 | 0.326992818052 |
+-------------+-------+----------------+

\n

+-------------+-------+----------------+
| name | level | actual_level |
+-------------+-------+----------------+
| 10046_right | 1 | 0.70960257402 |
| 11730_right | 0 | 0.357797767757 |
+-------------+-------+----------------+

\n

+-------------+-------+----------------+
| name | level | actual_level |
+-------------+-------+----------------+
| 10046_right | 0 | 0.408557624363 |
| 11730_right | 0 | 0.355133798881 |
+-------------+-------+----------------+

\n

[/quote]

", "votes": null, "replies": [] }, { "id": 81048, "author_name": "hoytkoepke1", "author_url": "", "post_date": "06/05/2015 20:01:48", "content": "

Hello fisherman,

\n

In this case, I kept hitting issues on sort running out of memory, etc. as well.  I'm not sure of a workaround except to use a machine with more memory.  I did talk with the people here at Dato responsible for the sort stuff, and it's getting better.  

\n

The solution I ended up using was to modify the code so that it just used the file paths up until the final part when it generated the sframes, at which point I load them using a lambda function.  This makes it significantly faster and the sort is much easier.

\n

Something like: 

\n

X["path"] = list(chain(*[[abspath(join(root, f)) for f in files
                  if f.endswith('jpeg')]
                  for root, dir_list, files in os.walk(image_path)]))


\n

Then use something like

\n

X["image"] = X["path"].apply(lambda p: gl.Image(p))

\n

to load it at the end. 

\n

Hope something like that helps! 

\n

-- Hoyt

\n

[quote=fisherman;81044]

\n

Hi Hoyt,

\n

Thanks for posting this started code. I am impressed by what Graphlab can do with just a few hundred lines of Python code! I tried to run your Github code but looks like the Graphlab API crashed in sframe sorting - see below and the log file attached. If I comment out the sorting code then it will run fine. Any idea?

\n

I'm running Ubuntu 14.04 on a desktop Intel Core i7-5820K CPU @ 3.30GHz × 12, with 8GB Memory plus GeForce GTX 970 GPUs.

\n

=========================

\n

Traceback (most recent call last):
File "create_image_sframes.py", line 47, in <module>
X_train = X_train.sort("_random_")
File "/home/ekang/anaconda/lib/python2.7/dist-packages/graphlab/data_structures/sframe.py", line 5077, in sort
return SFrame(_proxy=self.__proxy__.sort(sort_column_names, sort_column_orders))
File "/home/ekang/anaconda/lib/python2.7/dist-packages/graphlab/cython/context.py", line 31, in __exit__
raise exc_type(exc_value)
RuntimeError: Communication Failure: 113.

\n

[quote=Hoyt Koepke;66564]

\n

Hello, 

\n

I just put up code that I used to get a leaderboard score of ~0.46412, and I would like to hear feedback on it.  Feel free to use it, and I am open to feedback.  I am sure many parts of it can be improved. 

\n

https://github.com/hoytak/diabetic-retinopathy-code

\n

This code uses the ImageMagick convert tool to preprocess the images, then uses the neural net toolkit (basically cxxnet) and the boosted regression trees in Dato's Graphlab create toolkits to build the classifier.  

\n

Main Ideas:

\n
    \n
  1. Use ImageMagick's convert tool to trim off the blank space to the sides of the images, then pad them so that they are all 256x256. Thus the eye is always centered with edges against the edges of the image. I avoided scaling the images to improve the neural net performance.
    2. \n
  2. Create multiple versions of each image varying by hue and contrast and white balance.  This step can be simplified to get to a quick though less accurate solution.
    3. \n
  3. Duplicate each class so each class is represented equally, then shuffle the data.
    4. \n
  4. Train several neural nets, one trained to predict the level membership, and another 4 to distinguish 0 vs 1-4, 0-1 vs. 2-4, 0-2 vs. 3-4, and 0-3 vs. 4.
    5. \n
  5. For each image, extract both class predictions and the values of the final neural net layer. Pool all of these features across models and variations of the same images.
    6. \n
  6. Train boosted regression trees on these pooled features to predict the level.
    7. \n
  7. Round to the nearest integer as the class prediction.
    8. \n
\n

I know that each of these steps could definitely be tuned for greater accuracy, and I would love to hear your feedback. In particular, I don't think the neural net architecture is optimal for these images. 

\n

[/quote]

\n

[/quote]

", "votes": null, "replies": [] }, { "id": 82722, "author_name": "tommoran", "author_url": "", "post_date": "06/25/2015 11:12:43", "content": "

Hi Hoyt, thanks for the startercode to begin with.

\n

I'm having an issue when I run create_image_sframes.py

\n

I get the message below repeatedly

\n

PROGRESS: Unexpected JPEG decode failure file: /mnt/my-data/kaggle-data/processed/run-normal/test/21697_left.jpeg

\n

Any ideas off-hand how to fix this?

", "votes": null, "replies": [] }, { "id": 82746, "author_name": "hoytkoepke1", "author_url": "", "post_date": "06/25/2015 17:50:51", "content": "

Hey Octave1,

\n

I think it's happening because somehow the shrinking part of the imagemagick script ended up shrinking one of those images to zero.  It should be fine if you just delete that image, or rerun the convert command for just that image. 

\n

-- Hoyt

", "votes": null, "replies": [] }, { "id": 84211, "author_name": "killniu", "author_url": "", "post_date": "07/12/2015 15:02:54", "content": "

thank you Hoyt, I can run following your guides.

", "votes": null, "replies": [] }, { "id": 86129, "author_name": "sangramkapre", "author_url": "", "post_date": "07/20/2015 06:58:23", "content": "

Hi Hoyt,

\n\n

I tried running your code. I did it only to train which_model = 4 i.e. predicting 4-vs-all retinopathy grade.\nMy training and validation accuracy figures were quite low so I checked the features that have been stored in Xty and Xtst in create_nn_model.py and it shows nan values in both scores_4 and features_4 columns. Any idea about what is it that I did wrong?

\n\n

Following are the details of the model that was stored:

\n\n
Class               : NeuralNetClassifier\n\nSchema\n------\nExamples            : 848387\nFeatures            : 1\nTarget column       : class\n\nTraining Summary\n----------------\nTraining accuracy   : 0.4992\nValidation accuracy : 0.5416\nTraining time (sec) : 85850.3034\n
\n\n

Following are the details of Xf_train from create_submission.py code.

\n\n
Columns:\n    name    str\n    scores_4    dict\n    level   int\n    features_4  dict\n\nRows: 34424\n\nData:\n+-------------+-------------------------------+-------+\n|     name    |            scores_4           | level |\n+-------------+-------------------------------+-------+\n|  2157_right | {'contrast-2.1': nan, 'con... |   0   |\n|  8649_right | {'contrast-2.1': nan, 'con... |   0   |\n|  497_right  | {'contrast-2.1': nan, 'con... |   0   |\n| 34024_right | {'contrast-2.1': nan, 'con... |   0   |\n|  2613_left  | {'contrast-2.1': nan, 'con... |   2   |\n|  21769_left | {'contrast-2.1': nan, 'con... |   0   |\n| 25197_right | {'contrast-2.1': nan, 'con... |   2   |\n|  21833_left | {'contrast-2.1': nan, 'con... |   1   |\n|  11417_left | {'contrast-2.1': nan, 'con... |   2   |\n|  5096_right | {'contrast-2.1': nan, 'con... |   0   |\n+-------------+-------------------------------+-------+\n+-------------------------------+\n|           features_4          |\n+-------------------------------+\n| {'sat-2.23': nan, 'sat-2.2... |\n| {'sat-2.23': nan, 'sat-2.2... |\n| {'sat-2.23': nan, 'sat-2.2... |\n| {'sat-2.23': nan, 'sat-2.2... |\n| {'sat-2.23': nan, 'sat-2.2... |\n| {'sat-2.23': nan, 'sat-2.2... |\n| {'sat-2.23': nan, 'sat-2.2... |\n| {'sat-2.23': nan, 'sat-2.2... |\n| {'sat-2.23': nan, 'sat-2.2... |\n| {'sat-2.23': nan, 'sat-2.2... |\n+-------------------------------+\n[34424 rows x 4 columns]\nNote: Only the head of the SFrame is printed.\nYou can use print_rows(num_rows=m, num_columns=n) to print more rows and columns.\n
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "66564": "", "66589": "", "66601": "", "66607": "", "66626": "", "66654": "", "67948": "", "67987": "", "67989": "", "68033": "", "68057": "", "68061": "", "68570": "", "68602": "", "68603": "", "68903": "", "68904": "", "68914": "", "68915": "", "68947": "", "68963": "", "68980": "", "68986": "", "69072": "", "69074": "", "69703": "", "69772": "", "69782": "", "70063": "", "70065": "", "70066": "", "70071": "", "70072": "", "70078": "", "70079": "", "70083": "", "70084": "", "70085": "", "70086": "", "70088": "", "70177": "", "70186": "", "71221": "", "71224": "", "71227": "", "71229": "", "71248": "", "71249": "", "71382": "", "71397": "", "71399": "", "71415": "", "71417": "", "71418": "", "72504": "", "72719": "", "72986": "", "73007": "", "73184": "", "74244": "", "74358": "", "74360": "", "74361": "", "74364": "", "74417": "", "74445": "", "75145": "", "75242": "", "77297": "", "80553": "", "81044": "", "81046": "", "81048": "", "82722": "", "82746": "", "84211": "thank you Hoyt, I can run following your guides.", "86129": "Hi Hoyt,\r\n\r\nI tried running your code. I did it only to train `which_model = 4` i.e. predicting 4-vs-all retinopathy grade.\r\nMy training and validation accuracy figures were quite low so I checked the features that have been stored in `Xty` and `Xtst` in `create_nn_model.py` and it shows `nan` values in both `scores_4` and `features_4` columns. Any idea about what is it that I did wrong?\r\n\r\nFollowing are the details of the model that was stored:\r\n\r\n \r\n Class : NeuralNetClassifier\r\n \r\n Schema\r\n ------\r\n Examples : 848387\r\n Features : 1\r\n Target column : class\r\n \r\n Training Summary\r\n ----------------\r\n Training accuracy : 0.4992\r\n Validation accuracy : 0.5416\r\n Training time (sec) : 85850.3034\r\n\r\n\r\nFollowing are the details of `Xf_train` from `create_submission.py` code.\r\n\r\n Columns:\r\n \tname\tstr\r\n \tscores_4\tdict\r\n \tlevel\tint\r\n \tfeatures_4\tdict\r\n \r\n Rows: 34424\r\n \r\n Data:\r\n +-------------+-------------------------------+-------+\r\n | name | scores_4 | level |\r\n +-------------+-------------------------------+-------+\r\n | 2157_right | {'contrast-2.1': nan, 'con... | 0 |\r\n | 8649_right | {'contrast-2.1': nan, 'con... | 0 |\r\n | 497_right | {'contrast-2.1': nan, 'con... | 0 |\r\n | 34024_right | {'contrast-2.1': nan, 'con... | 0 |\r\n | 2613_left | {'contrast-2.1': nan, 'con... | 2 |\r\n | 21769_left | {'contrast-2.1': nan, 'con... | 0 |\r\n | 25197_right | {'contrast-2.1': nan, 'con... | 2 |\r\n | 21833_left | {'contrast-2.1': nan, 'con... | 1 |\r\n | 11417_left | {'contrast-2.1': nan, 'con... | 2 |\r\n | 5096_right | {'contrast-2.1': nan, 'con... | 0 |\r\n +-------------+-------------------------------+-------+\r\n +-------------------------------+\r\n | features_4 |\r\n +-------------------------------+\r\n | {'sat-2.23': nan, 'sat-2.2... |\r\n | {'sat-2.23': nan, 'sat-2.2... |\r\n | {'sat-2.23': nan, 'sat-2.2... |\r\n | {'sat-2.23': nan, 'sat-2.2... |\r\n | {'sat-2.23': nan, 'sat-2.2... |\r\n | {'sat-2.23': nan, 'sat-2.2... |\r\n | {'sat-2.23': nan, 'sat-2.2... |\r\n | {'sat-2.23': nan, 'sat-2.2... |\r\n | {'sat-2.23': nan, 'sat-2.2... |\r\n | {'sat-2.23': nan, 'sat-2.2... |\r\n +-------------------------------+\r\n [34424 rows x 4 columns]\r\n Note: Only the head of the SFrame is printed.\r\n You can use print_rows(num_rows=m, num_columns=n) to print more rows and columns." }, "source": "meta" }