Here's an InceptionV3 CNN (299x299 transfer learning) submission with simple product averaging:
\n\n\n\nMy current submission is without test time augmentation (69%) - I think it's possible to squeeze into the 70% range with some augmentation which is currently running. Also a better voting approach than the average may work - but I don't have time :D.
\n\nHard competition for a single GPU - but fun!
\n\nHere's an InceptionV3 CNN (299x299 transfer learning) submission with simple product averaging:
\n\n\n\nMy current submission is without test time augmentation (69%) - I think it's possible to squeeze into the 70% range with some augmentation which is currently running. Also a better voting approach than the average may work - but I don't have time :D.
\n\nHard competition for a single GPU - but fun!
\n\nHi As you mentioned you ran on single GPU , can you please tell what GPU you used and how long it ran ?
", "rawMarkdown": "Hi As you mentioned you ran on single GPU , can you please tell what GPU you used and how long it ran ?", "votes": null }, { "id": "256749", "postDate": "12/12/2017 16:48:44", "content": "I'm testing it on 2 gtx 1060's with about 11 GB of VRAM, but it looks like 1 epoch takes about 40 hours.
", "rawMarkdown": "I'm testing it on 2 gtx 1060's with about 11 GB of VRAM, but it looks like 1 epoch takes about 40 hours.", "votes": null }, { "id": "256914", "postDate": "12/12/2017 23:04:22", "content": "\n\n\nHard competition for a single GPU - but fun!
\n
Try no GPU but using paid service. I am busy at work hence decided, I will try out a couple of runs from last weekend and get at least 2 submissions before competition closes. Big mistake, by the time I got one of my scripts to run, I have just from today until the end to make a real submission. At this point I am hoping for one good submission.
\n\nI have one inception model that I used the neptune.ml to train that I can run predictions on but am not sure how long that would take. I gave up on Neptune because, it took forever to end up with just that model. Their system aborted my runs several times before this last one that ran until I ran out of credits after 6 epochs. So I do not want to put money and run anything with them again. I will just use floyd and see if I would have enough time to run a submission script with that model after my current script finish running. By the way it took FloydHub 3 days to unpack and make my uploaded data available.
\n\nAnyway, thanks for reading my rant.
\n\nGood luck and happy kaggling!
", "rawMarkdown": "> Hard competition for a single GPU - but fun!\n\nTry no GPU but using paid service. I am busy at work hence decided, I will try out a couple of runs from last weekend and get at least 2 submissions before competition closes. Big mistake, by the time I got one of my scripts to run, I have just from today until the end to make a real submission. At this point I am hoping for one good submission.\n\nI have one inception model that I used the neptune.ml to train that I can run predictions on but am not sure how long that would take. I gave up on Neptune because, it took forever to end up with just that model. Their system aborted my runs several times before this last one that ran until I ran out of credits after 6 epochs. So I do not want to put money and run anything with them again. I will just use floyd and see if I would have enough time to run a submission script with that model after my current script finish running. By the way it took FloydHub 3 days to unpack and make my uploaded data available.\n\nAnyway, thanks for reading my rant.\n\nGood luck and happy kaggling!", "votes": null }, { "id": "256916", "postDate": "12/12/2017 23:06:42", "content": "Your code is referring to a \"Part 1\" ('#Uncomment these to Load in the data from part 1 so we do not need to run again') for 'train_offsets_df = pd.read_csv('train_offsets.csv', index_col=0)' et al.\nIs it located in a different github ?
", "rawMarkdown": "Your code is referring to a \"Part 1\" ('#Uncomment these to Load in the data from part 1 so we do not need to run again') for 'train_offsets_df = pd.read_csv('train_offsets.csv', index_col=0)' et al.\nIs it located in a different github ?", "votes": null }, { "id": "256923", "postDate": "12/12/2017 23:30:48", "content": "train_offsets.csv is the output from his read_bson function and Train/val are the output from the make_val_set function.
Thank you @Steven Nguyen for letting us know about your setup and running time.
", "rawMarkdown": "Thank you @Steven Nguyen for letting us know about your setup and running time.", "votes": null }, { "id": "257069", "postDate": "12/13/2017 10:25:12", "content": "I did a test run on a GTX 1080TI with 11GB VRAM, stopped it after an hour as indeed one epoch takes 35+ hours, so for 15 epochs -> 525 hours or 22 days approx.
", "rawMarkdown": "I did a test run on a GTX 1080TI with 11GB VRAM, stopped it after an hour as indeed one epoch takes 35+ hours, so for 15 epochs -> 525 hours or 22 days approx.", "votes": null }, { "id": "257071", "postDate": "12/13/2017 10:41:55", "content": "Thank you @Eric Perbos-Brinck for your reply. That's indeed a lot of time .. I did guessed this competition to be GPU intensive but couldn't guess to be this level. .. :(
", "rawMarkdown": "Thank you @Eric Perbos-Brinck for your reply. That's indeed a lot of time .. I did guessed this competition to be GPU intensive but couldn't guess to be this level. .. :(", "votes": null }, { "id": "257187", "postDate": "12/13/2017 17:01:22", "content": "If you want it to be less GPU intensive, remove the data augmentation. Should more than halve the epoch time.
", "rawMarkdown": "If you want it to be less GPU intensive, remove the data augmentation. Should more than halve the epoch time.", "votes": null }, { "id": "257270", "postDate": "12/13/2017 21:13:52", "content": "I made a mistake in parallelizing my Tensorflow code so I won't be able to take advantage of this code release. To be honest this feels like a \"most expensive computer wins\" type of contest.
", "rawMarkdown": "I made a mistake in parallelizing my Tensorflow code so I won't be able to take advantage of this code release. To be honest this feels like a \"most expensive computer wins\" type of contest.", "votes": null }, { "id": "257296", "postDate": "12/13/2017 22:57:37", "content": "There's definitely a \"minimum compute requirement\" to be even remotely competitive, but it's definitely not just a compute race. I spent a bunch of time messing around with side-projects and using this as a bigger-than-imagenet testbed for some research, but I think that you could get top10 easily by just starting early and properly training standard models on 4 x 1080Ti. An ensemble of just three of my smaller models breaks top 8 on public leaderboard, each taking about a week to train.
", "rawMarkdown": "There's definitely a \"minimum compute requirement\" to be even remotely competitive, but it's definitely not just a compute race. I spent a bunch of time messing around with side-projects and using this as a bigger-than-imagenet testbed for some research, but I think that you could get top10 easily by just starting early and properly training standard models on 4 x 1080Ti. An ensemble of just three of my smaller models breaks top 8 on public leaderboard, each taking about a week to train.", "votes": null }, { "id": "257337", "postDate": "12/14/2017 03:33:02", "content": "The challenge itself is interesting and will probably take skill to win. I'm just wondering if it would be worth investing into those new titan V chips after seeing how painfully slow the 1080ti chips are here.
", "rawMarkdown": "The challenge itself is interesting and will probably take skill to win. I'm just wondering if it would be worth investing into those new titan V chips after seeing how painfully slow the 1080ti chips are here.", "votes": null }, { "id": "257472", "postDate": "12/14/2017 10:09:35", "content": "Titan X Pascal
", "rawMarkdown": "Titan X Pascal", "votes": null }, { "id": "257503", "postDate": "12/14/2017 11:21:12", "content": "I would and have in the past used AWS spot instances - It's possible to get 8xGPUs each with 16GB of RAM for < $0.50 an hour.
\n\n\n", "rawMarkdown": "I would and have in the past used AWS spot instances - It's possible to get 8xGPUs each with 16GB of RAM for < $0.50 an hour.\n> **YaGana Sheriff-Hussaini wrote**\n> \n> > > Hard competition for a single GPU - but fun!\n> \n> Try no GPU but using paid service. I am busy at work hence decided, I will try out a couple of runs from last weekend and get at least 2 submissions before competition closes. Big mistake, by the time I got one of my scripts to run, I have just from today until the end to make a real submission. At this point I am hoping for one good submission.\n> \n> I have one inception model that I used the neptune.ml to train that I can run predictions on but am not sure how long that would take. I gave up on Neptune because, it took forever to end up with just that model. Their system aborted my runs several times before this last one that ran until I ran out of credits after 6 epochs. So I do not want to put money and run anything with them again. I will just use floyd and see if I would have enough time to run a submission script with that model after my current script finish running. By the way it took FloydHub 3 days to unpack and make my uploaded data available.\n> \n> Anyway, thanks for reading my rant.\n> \n> Good luck and happy kaggling!", "votes": null }, { "id": "257512", "postDate": "12/14/2017 11:40:28", "content": "YaGana Sheriff-Hussaini wrote
\n \n\n\n \n> Hard competition for a single GPU - but fun!
\nTry no GPU but using paid service. I am busy at work hence decided, I will try out a couple of runs from last weekend and get at least 2 submissions before competition closes. Big mistake, by the time I got one of my scripts to run, I have just from today until the end to make a real submission. At this point I am hoping for one good submission.
\n \nI have one inception model that I used the neptune.ml to train that I can run predictions on but am not sure how long that would take. I gave up on Neptune because, it took forever to end up with just that model. Their system aborted my runs several times before this last one that ran until I ran out of credits after 6 epochs. So I do not want to put money and run anything with them again. I will just use floyd and see if I would have enough time to run a submission script with that model after my current script finish running. By the way it took FloydHub 3 days to unpack and make my uploaded data available.
\n \nAnyway, thanks for reading my rant.
\n \nGood luck and happy kaggling!
\n
I haven't really had time to compete in this competition - Hence I've had a single model running for a very long time - There will be several ways of speeding this up:
\n\nIf you're just having fun it's also worth noting that a MASSIVE percentage of the training examples are duplicates or near-exact duplicates. The training set is wayyy overinflated.
", "rawMarkdown": "If you're just having fun it's also worth noting that a MASSIVE percentage of the training examples are duplicates or near-exact duplicates. The training set is wayyy overinflated.", "votes": null }, { "id": "257728", "postDate": "12/14/2017 21:24:22", "content": "does the test set has duplicates too?
", "rawMarkdown": "does the test set has duplicates too?", "votes": null }, { "id": "257729", "postDate": "12/14/2017 21:24:51", "content": "Thanks @CraigGlastonbury. I did not know I can get AWS for $0.50 an hour. When I checked a few months ago, before trying out FloydHub, AWS was more expensive. I will try them next time.
", "rawMarkdown": "Thanks @CraigGlastonbury. I did not know I can get AWS for $0.50 an hour. When I checked a few months ago, before trying out FloydHub, AWS was more expensive. I will try them next time.", "votes": null } ], "comments": [ { "id": 256570, "author_name": "reachkishore", "author_url": "", "post_date": "12/12/2017 09:22:03", "content": "Hi As you mentioned you ran on single GPU , can you please tell what GPU you used and how long it ran ?
", "votes": null, "replies": [ { "id": 256749, "author_name": "stevenknguyen", "author_url": "", "post_date": "12/12/2017 16:48:44", "content": "I'm testing it on 2 gtx 1060's with about 11 GB of VRAM, but it looks like 1 epoch takes about 40 hours.
", "votes": null, "replies": [] }, { "id": 256942, "author_name": "reachkishore", "author_url": "", "post_date": "12/13/2017 00:37:41", "content": "Thank you @Steven Nguyen for letting us know about your setup and running time.
", "votes": null, "replies": [] }, { "id": 257069, "author_name": "ericperbos", "author_url": "", "post_date": "12/13/2017 10:25:12", "content": "I did a test run on a GTX 1080TI with 11GB VRAM, stopped it after an hour as indeed one epoch takes 35+ hours, so for 15 epochs -> 525 hours or 22 days approx.
", "votes": null, "replies": [] }, { "id": 257071, "author_name": "reachkishore", "author_url": "", "post_date": "12/13/2017 10:41:55", "content": "Thank you @Eric Perbos-Brinck for your reply. That's indeed a lot of time .. I did guessed this competition to be GPU intensive but couldn't guess to be this level. .. :(
", "votes": null, "replies": [] }, { "id": 257270, "author_name": "stevenknguyen", "author_url": "", "post_date": "12/13/2017 21:13:52", "content": "I made a mistake in parallelizing my Tensorflow code so I won't be able to take advantage of this code release. To be honest this feels like a \"most expensive computer wins\" type of contest.
", "votes": null, "replies": [] }, { "id": 257296, "author_name": "ajmooch", "author_url": "", "post_date": "12/13/2017 22:57:37", "content": "There's definitely a \"minimum compute requirement\" to be even remotely competitive, but it's definitely not just a compute race. I spent a bunch of time messing around with side-projects and using this as a bigger-than-imagenet testbed for some research, but I think that you could get top10 easily by just starting early and properly training standard models on 4 x 1080Ti. An ensemble of just three of my smaller models breaks top 8 on public leaderboard, each taking about a week to train.
", "votes": null, "replies": [] }, { "id": 257337, "author_name": "stevenknguyen", "author_url": "", "post_date": "12/14/2017 03:33:02", "content": "The challenge itself is interesting and will probably take skill to win. I'm just wondering if it would be worth investing into those new titan V chips after seeing how painfully slow the 1080ti chips are here.
", "votes": null, "replies": [] }, { "id": 257472, "author_name": "craigglastonbury", "author_url": "", "post_date": "12/14/2017 10:09:35", "content": "Titan X Pascal
", "votes": null, "replies": [] }, { "id": 257512, "author_name": "craigglastonbury", "author_url": "", "post_date": "12/14/2017 11:40:28", "content": "I haven't really had time to compete in this competition - Hence I've had a single model running for a very long time - There will be several ways of speeding this up:
\n\nIf you're just having fun it's also worth noting that a MASSIVE percentage of the training examples are duplicates or near-exact duplicates. The training set is wayyy overinflated.
", "votes": null, "replies": [] }, { "id": 257728, "author_name": "hengck23", "author_url": "", "post_date": "12/14/2017 21:24:22", "content": "does the test set has duplicates too?
", "votes": null, "replies": [] } ] }, { "id": 256914, "author_name": "sheriytm", "author_url": "", "post_date": "12/12/2017 23:04:22", "content": "\n\n\nHard competition for a single GPU - but fun!
\n
Try no GPU but using paid service. I am busy at work hence decided, I will try out a couple of runs from last weekend and get at least 2 submissions before competition closes. Big mistake, by the time I got one of my scripts to run, I have just from today until the end to make a real submission. At this point I am hoping for one good submission.
\n\nI have one inception model that I used the neptune.ml to train that I can run predictions on but am not sure how long that would take. I gave up on Neptune because, it took forever to end up with just that model. Their system aborted my runs several times before this last one that ran until I ran out of credits after 6 epochs. So I do not want to put money and run anything with them again. I will just use floyd and see if I would have enough time to run a submission script with that model after my current script finish running. By the way it took FloydHub 3 days to unpack and make my uploaded data available.
\n\nAnyway, thanks for reading my rant.
\n\nGood luck and happy kaggling!
", "votes": null, "replies": [ { "id": 257503, "author_name": "craigglastonbury", "author_url": "", "post_date": "12/14/2017 11:21:12", "content": "I would and have in the past used AWS spot instances - It's possible to get 8xGPUs each with 16GB of RAM for < $0.50 an hour.
\n\n\n", "votes": null, "replies": [] }, { "id": 257729, "author_name": "sheriytm", "author_url": "", "post_date": "12/14/2017 21:24:51", "content": "YaGana Sheriff-Hussaini wrote
\n \n\n\n \n> Hard competition for a single GPU - but fun!
\nTry no GPU but using paid service. I am busy at work hence decided, I will try out a couple of runs from last weekend and get at least 2 submissions before competition closes. Big mistake, by the time I got one of my scripts to run, I have just from today until the end to make a real submission. At this point I am hoping for one good submission.
\n \nI have one inception model that I used the neptune.ml to train that I can run predictions on but am not sure how long that would take. I gave up on Neptune because, it took forever to end up with just that model. Their system aborted my runs several times before this last one that ran until I ran out of credits after 6 epochs. So I do not want to put money and run anything with them again. I will just use floyd and see if I would have enough time to run a submission script with that model after my current script finish running. By the way it took FloydHub 3 days to unpack and make my uploaded data available.
\n \nAnyway, thanks for reading my rant.
\n \nGood luck and happy kaggling!
\n
Thanks @CraigGlastonbury. I did not know I can get AWS for $0.50 an hour. When I checked a few months ago, before trying out FloydHub, AWS was more expensive. I will try them next time.
", "votes": null, "replies": [] } ] }, { "id": 256916, "author_name": "ericperbos", "author_url": "", "post_date": "12/12/2017 23:06:42", "content": "Your code is referring to a \"Part 1\" ('#Uncomment these to Load in the data from part 1 so we do not need to run again') for 'train_offsets_df = pd.read_csv('train_offsets.csv', index_col=0)' et al.\nIs it located in a different github ?
", "votes": null, "replies": [ { "id": 256923, "author_name": "stevenknguyen", "author_url": "", "post_date": "12/12/2017 23:30:48", "content": "train_offsets.csv is the output from his read_bson function and Train/val are the output from the make_val_set function.
If you want it to be less GPU intensive, remove the data augmentation. Should more than halve the epoch time.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "256536": "Here's an InceptionV3 CNN (299x299 transfer learning) submission with simple product averaging:\n\n[My github][1]\n\nMy current submission is without test time augmentation (69%) - I think it's possible to squeeze into the 70% range with some augmentation which is currently running. Also a better voting approach than the average may work - but I don't have time :D.\n\nHard competition for a single GPU - but fun! \n\n [1]: https://github.com/GlastonburyC/Cdiscount-Kaggle-Competition\n\n* Heavy code reuse from other Kaggle Kernels - Please mention your name for attribution :)", "256570": "Hi As you mentioned you ran on single GPU , can you please tell what GPU you used and how long it ran ?", "256749": "I'm testing it on 2 gtx 1060's with about 11 GB of VRAM, but it looks like 1 epoch takes about 40 hours.", "256914": "> Hard competition for a single GPU - but fun!\n\nTry no GPU but using paid service. I am busy at work hence decided, I will try out a couple of runs from last weekend and get at least 2 submissions before competition closes. Big mistake, by the time I got one of my scripts to run, I have just from today until the end to make a real submission. At this point I am hoping for one good submission.\n\nI have one inception model that I used the neptune.ml to train that I can run predictions on but am not sure how long that would take. I gave up on Neptune because, it took forever to end up with just that model. Their system aborted my runs several times before this last one that ran until I ran out of credits after 6 epochs. So I do not want to put money and run anything with them again. I will just use floyd and see if I would have enough time to run a submission script with that model after my current script finish running. By the way it took FloydHub 3 days to unpack and make my uploaded data available.\n\nAnyway, thanks for reading my rant.\n\nGood luck and happy kaggling!", "256916": "Your code is referring to a \"Part 1\" ('#Uncomment these to Load in the data from part 1 so we do not need to run again') for 'train_offsets_df = pd.read_csv('train_offsets.csv', index_col=0)' et al.\nIs it located in a different github ?", "256923": "`train_offsets.csv` is the output from his `read_bson` function and Train/val are the output from the `make_val_set` function.", "256942": "Thank you @Steven Nguyen for letting us know about your setup and running time.", "257069": "I did a test run on a GTX 1080TI with 11GB VRAM, stopped it after an hour as indeed one epoch takes 35+ hours, so for 15 epochs -> 525 hours or 22 days approx.", "257071": "Thank you @Eric Perbos-Brinck for your reply. That's indeed a lot of time .. I did guessed this competition to be GPU intensive but couldn't guess to be this level. .. :(", "257187": "If you want it to be less GPU intensive, remove the data augmentation. Should more than halve the epoch time.", "257270": "I made a mistake in parallelizing my Tensorflow code so I won't be able to take advantage of this code release. To be honest this feels like a \"most expensive computer wins\" type of contest.", "257296": "There's definitely a \"minimum compute requirement\" to be even remotely competitive, but it's definitely not just a compute race. I spent a bunch of time messing around with side-projects and using this as a bigger-than-imagenet testbed for some research, but I think that you could get top10 easily by just starting early and properly training standard models on 4 x 1080Ti. An ensemble of just three of my smaller models breaks top 8 on public leaderboard, each taking about a week to train.", "257337": "The challenge itself is interesting and will probably take skill to win. I'm just wondering if it would be worth investing into those new titan V chips after seeing how painfully slow the 1080ti chips are here.", "257472": "Titan X Pascal", "257503": "I would and have in the past used AWS spot instances - It's possible to get 8xGPUs each with 16GB of RAM for < $0.50 an hour.\n> **YaGana Sheriff-Hussaini wrote**\n> \n> > > Hard competition for a single GPU - but fun!\n> \n> Try no GPU but using paid service. I am busy at work hence decided, I will try out a couple of runs from last weekend and get at least 2 submissions before competition closes. Big mistake, by the time I got one of my scripts to run, I have just from today until the end to make a real submission. At this point I am hoping for one good submission.\n> \n> I have one inception model that I used the neptune.ml to train that I can run predictions on but am not sure how long that would take. I gave up on Neptune because, it took forever to end up with just that model. Their system aborted my runs several times before this last one that ran until I ran out of credits after 6 epochs. So I do not want to put money and run anything with them again. I will just use floyd and see if I would have enough time to run a submission script with that model after my current script finish running. By the way it took FloydHub 3 days to unpack and make my uploaded data available.\n> \n> Anyway, thanks for reading my rant.\n> \n> Good luck and happy kaggling!", "257512": "I haven't really had time to compete in this competition - Hence I've had a single model running for a very long time - There will be several ways of speeding this up:\n\n* removal of data augmentation\n* Full utilisation of GPU\n* Batch size not linked to number of product images\n* gradient accumulation so you can run it for fewer epochs\n* multi-GPU parallelism \n* non-random sub-sampling of dataset.", "257668": "If you're just having fun it's also worth noting that a MASSIVE percentage of the training examples are duplicates or near-exact duplicates. The training set is wayyy overinflated.", "257728": "does the test set has duplicates too?", "257729": "Thanks @CraigGlastonbury. I did not know I can get AWS for $0.50 an hour. When I checked a few months ago, before trying out FloydHub, AWS was more expensive. I will try them next time." }, "source": "meta" }