{ "id": 6183, "title": "Handling 703,000,000 URLs", "url": "/competitions/yandex-personalized-web-search-challenge/discussion/6183", "author_name": "", "post_date": "2013-10-31T21:46:36.140Z", "votes": null, "comment_count": 4, "views": 1979, "content": "

I decided to use MongoDB for this effort.  But after an hour of running, I only have about 300,000 URLs in my database. 

\n

Either I am doing something wrong or it will take 97 days to load my DB.  To that I say,  Nyet!

\n

Does anyone know what the how the performance for MongoDB compares to Sqlite3 or PostGres ? 

\n

BTW I am using Pypy with pymongo, so all code is jitted and fast, It is clearly hard drive bound.

\n

 

\n

 

\n

 

\n

 

\n

 

\n

 

", "messages": [ { "id": "32973", "postDate": "10/31/2013 21:46:36", "content": "

I decided to use MongoDB for this effort.  But after an hour of running, I only have about 300,000 URLs in my database. 

\n

Either I am doing something wrong or it will take 97 days to load my DB.  To that I say,  Nyet!

\n

Does anyone know what the how the performance for MongoDB compares to Sqlite3 or PostGres ? 

\n

BTW I am using Pypy with pymongo, so all code is jitted and fast, It is clearly hard drive bound.

\n

 

\n

 

\n

 

\n

 

\n

 

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "33177", "postDate": "11/04/2013 15:16:27", "content": "

Whatever the form of database you're using for this, loading all URL's will take too much time. Even if you manage to get it you'll have large query execution times.

\n

I would strongly suggest that you sample the training set first and use that subset for building a model and only run your model calibration on the full data set once you're pretty confident that it produces reasonable results. Of course this comes at the expense of the extra work of coming up with a good sampling method to maintain the structure of the original dataset in the reduced set. Hope that helps.

", "rawMarkdown": "", "votes": null }, { "id": "33212", "postDate": "11/04/2013 21:14:47", "content": "

Hi Bary,

\n

Agreed.  I was running a test to see if my approach would scale.  Obviously it does not. 

\n

Interesting point about sampling the input file.   I was going to use a percentage (maybe 5 days worth) for faster iteration and working out ideas.  

\n

What advantages do you see in a more clever sampling method over a simplistic approach like taking the first 5 days?

\n

Thanks !

\n

 

\n

 

\n

 

\n

 

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "33256", "postDate": "11/05/2013 11:12:40", "content": "

Sampling a percentage of days in the training period is one way to do it. What it provides you with is a data subset that is rich in user information since you'll probably be preserving a large number of users but that reduced dataset will not be as rich in pointing out temporal variations (short period of time relative to the original training set). On the other hand, you could decide to take a sample of users instead of days, say 5% of the unique users in the dataset with all their associated queries over the entire period of time. This gives you the advantage of having the full view over the time period provided but with a reduced number of users.

\n

So it really depends on what your model is attempting to explore. If you're looking at temporal or time-dependent characteristics of querying habits for users, then you might be better off with a longer period of time to work with, hence sampling a set of users. If you're more interested in detailed user preference analysis and how that relates to query relevance, then you might want to have a large enough set of users and sampling a short period of time with a large number of users might be a better way of doing this.

\n

One more thing to bear in mind if you decide to sample is to watch out for your model's robustness. You want to make sure that your model performs on the full dataset close to how it performs on the reduced one. This is important because you're inevitably losing information in the sampling process. One way you could do that is to repeat the sampling process several times (say producing 5 reduced datasets) and comparing the performance of your models between these different subsets. You want the spread between performance across the different data samples to be as small as possible to ensure robustness.

\n

Hope that helps

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "33265", "postDate": "11/05/2013 13:40:54", "content": "

 

\n

In general I think users are learning as they search and explore.  However, I don't believe I can capture this aspect given the limitations of the data set and the length of competition.   So, my bias is to look at this month period as a single snapshot in time. 

\n

The bigger challenge is in developing an objective performance measure because you don't want to optimize for the public leaderboard score.  It is much easier to cross-validate under a supervised learning problem, where you know the answer beforehand.  On my first pass reading of the DCG measure, I was left with the impression that I  was missing some of the inputs (probably because they used the term "Ideal DCG").   I'll dig deeper into DCG to see if all the information is at my disposal. 

\n

Thanks Bary, this has been a helpful train of thought...

\n

 

\n

 

\n

 

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 33177, "author_name": "mohamedabdelbary", "author_url": "", "post_date": "11/04/2013 15:16:27", "content": "

Whatever the form of database you're using for this, loading all URL's will take too much time. Even if you manage to get it you'll have large query execution times.

\n

I would strongly suggest that you sample the training set first and use that subset for building a model and only run your model calibration on the full data set once you're pretty confident that it produces reasonable results. Of course this comes at the expense of the extra work of coming up with a good sampling method to maintain the structure of the original dataset in the reduced set. Hope that helps.

", "votes": null, "replies": [] }, { "id": 33212, "author_name": "geringer", "author_url": "", "post_date": "11/04/2013 21:14:47", "content": "

Hi Bary,

\n

Agreed.  I was running a test to see if my approach would scale.  Obviously it does not. 

\n

Interesting point about sampling the input file.   I was going to use a percentage (maybe 5 days worth) for faster iteration and working out ideas.  

\n

What advantages do you see in a more clever sampling method over a simplistic approach like taking the first 5 days?

\n

Thanks !

\n

 

\n

 

\n

 

\n

 

\n

 

", "votes": null, "replies": [] }, { "id": 33256, "author_name": "mohamedabdelbary", "author_url": "", "post_date": "11/05/2013 11:12:40", "content": "

Sampling a percentage of days in the training period is one way to do it. What it provides you with is a data subset that is rich in user information since you'll probably be preserving a large number of users but that reduced dataset will not be as rich in pointing out temporal variations (short period of time relative to the original training set). On the other hand, you could decide to take a sample of users instead of days, say 5% of the unique users in the dataset with all their associated queries over the entire period of time. This gives you the advantage of having the full view over the time period provided but with a reduced number of users.

\n

So it really depends on what your model is attempting to explore. If you're looking at temporal or time-dependent characteristics of querying habits for users, then you might be better off with a longer period of time to work with, hence sampling a set of users. If you're more interested in detailed user preference analysis and how that relates to query relevance, then you might want to have a large enough set of users and sampling a short period of time with a large number of users might be a better way of doing this.

\n

One more thing to bear in mind if you decide to sample is to watch out for your model's robustness. You want to make sure that your model performs on the full dataset close to how it performs on the reduced one. This is important because you're inevitably losing information in the sampling process. One way you could do that is to repeat the sampling process several times (say producing 5 reduced datasets) and comparing the performance of your models between these different subsets. You want the spread between performance across the different data samples to be as small as possible to ensure robustness.

\n

Hope that helps

\n

 

", "votes": null, "replies": [] }, { "id": 33265, "author_name": "geringer", "author_url": "", "post_date": "11/05/2013 13:40:54", "content": "

 

\n

In general I think users are learning as they search and explore.  However, I don't believe I can capture this aspect given the limitations of the data set and the length of competition.   So, my bias is to look at this month period as a single snapshot in time. 

\n

The bigger challenge is in developing an objective performance measure because you don't want to optimize for the public leaderboard score.  It is much easier to cross-validate under a supervised learning problem, where you know the answer beforehand.  On my first pass reading of the DCG measure, I was left with the impression that I  was missing some of the inputs (probably because they used the term "Ideal DCG").   I'll dig deeper into DCG to see if all the information is at my disposal. 

\n

Thanks Bary, this has been a helpful train of thought...

\n

 

\n

 

\n

 

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "32973": "", "33177": "", "33212": "", "33256": "", "33265": "" }, "source": "meta" }