{ "id": 25526, "title": "FFM on-disk", "url": "/competitions/outbrain-click-prediction/discussion/25526", "author_name": "", "post_date": "2016-11-16T23:09:07.857Z", "votes": null, "comment_count": 15, "views": 841, "content": "

Did anybody try using --on-disk flag for FFM?

\n\n

It does not seem to work for me. I wanted to use some bigger k value (bigger than default) and with on-disk flag it still throws "bad_alloc".

", "messages": [ { "id": "145141", "postDate": "11/16/2016 23:09:07", "content": "

Did anybody try using --on-disk flag for FFM?

\n\n

It does not seem to work for me. I wanted to use some bigger k value (bigger than default) and with on-disk flag it still throws "bad_alloc".

", "rawMarkdown": "Did anybody try using --on-disk flag for FFM?\r\n\r\nIt does not seem to work for me. I wanted to use some bigger k value (bigger than default) and with on-disk flag it still throws \"bad_alloc\".", "votes": null }, { "id": "145188", "postDate": "11/17/2016 05:54:05", "content": "

We got it working with --on-disk flag and could play around with non-default k values.

\n\n

Perhaps, your train (or validation files) are not in proper format?

", "rawMarkdown": "We got it working with --on-disk flag and could play around with non-default k values. \r\n\r\nPerhaps, your train (or validation files) are not in proper format?", "votes": null }, { "id": "145320", "postDate": "11/17/2016 19:58:41", "content": "

Unfortunately that is not an issue. All the values in the file are correct integers from 0 to 2^31-1, so they should fit in int.

\n\n

What I found is that memory requirements are higher than one might expect. In theory that should be m*n*k, but, the code makes k_aligned (aligned to the power of 2) so if you change from 4 to 5, k_aligned changes from 4 to 8. Basically, the memory requirement doubles instead of growing by 25%.

\n\n

Still that does not answer my question why it does not work with on_disk for me, but at least I know why it does fail after allocating 23GB of RAM.

", "rawMarkdown": "Unfortunately that is not an issue. All the values in the file are correct integers from 0 to 2^31-1, so they should fit in int. \r\n\r\nWhat I found is that memory requirements are higher than one might expect. In theory that should be m*n*k, but, the code makes k_aligned (aligned to the power of 2) so if you change from 4 to 5, k_aligned changes from 4 to 8. Basically, the memory requirement doubles instead of growing by 25%. \r\n\r\nStill that does not answer my question why it does not work with on_disk for me, but at least I know why it does fail after allocating 23GB of RAM.", "votes": null }, { "id": "145339", "postDate": "11/17/2016 21:21:08", "content": "

The --on-disk flag only influences whether the training and validation sets are read into memory. The model parameters, whose size increases with -k, are always stored in memory.

\n\n

You may be able to optimize your memory usage by condensing the field indices. By this I mean that if the indices for your field entries are 10, 100, 1000, 10000,... then recoding them as 1, 2, 3, 4,... should lead to a smaller model.

", "rawMarkdown": "The --on-disk flag only influences whether the training and validation sets are read into memory. The model parameters, whose size increases with -k, are always stored in memory.\r\n\r\nYou may be able to optimize your memory usage by condensing the field indices. By this I mean that if the indices for your field entries are 10, 100, 1000, 10000,... then recoding them as 1, 2, 3, 4,... should lead to a smaller model.", "votes": null }, { "id": "145341", "postDate": "11/17/2016 21:39:52", "content": "

Actually disabling auto-stop helped. My previous assumption was partially wrong. The problem was a couple of lines lower. When using auto-stop it tries to allocate memory for a dense vector of floats to store previous value of weights (vector W).

\n\n

Without auto-stopping I can use k=8, it still takes around 20GB and everything is fine.

\n\n

idle_speculation, regarding your hint, I my fields had numbers from 1 to 14, so it was ok from the beginning, but now I wonder if recoding the indexes themselves would make sense.

", "rawMarkdown": "Actually disabling auto-stop helped. My previous assumption was partially wrong. The problem was a couple of lines lower. When using auto-stop it tries to allocate memory for a dense vector of floats to store previous value of weights (vector W).\r\n\r\nWithout auto-stopping I can use k=8, it still takes around 20GB and everything is fine. \r\n\r\nidle_speculation, regarding your hint, I my fields had numbers from 1 to 14, so it was ok from the beginning, but now I wonder if recoding the indexes themselves would make sense.", "votes": null }, { "id": "145343", "postDate": "11/17/2016 21:51:29", "content": "

[quote=Marcin Pękalski;145341]

\n\n

I my fields had numbers from 1 to 14, so it was ok from the beginning, but now I wonder if recoding the indexes themselves would make sense.

\n\n

[/quote]

\n\n

Actually you could start numbering from 0 so one fewer field in total :P

", "rawMarkdown": "[quote=Marcin Pękalski;145341]\r\n\r\nI my fields had numbers from 1 to 14, so it was ok from the beginning, but now I wonder if recoding the indexes themselves would make sense.\r\n\r\n[/quote]\r\n\r\nActually you could start numbering from 0 so one fewer field in total :P", "votes": null }, { "id": "145344", "postDate": "11/17/2016 21:53:44", "content": "

[quote=Marcin Pękalski;145320]\nIn theory that should be m*n*k\n[/quote]

\n\n

Using hashing trick, n could be reduce to any small number that fits in the memory.

", "rawMarkdown": "[quote=Marcin Pękalski;145320]\r\nIn theory that should be m*n*k\r\n[/quote]\r\n\r\nUsing hashing trick, n could be reduce to any small number that fits in the memory.", "votes": null }, { "id": "145348", "postDate": "11/17/2016 22:21:35", "content": "

I guess I should read the code in more detail.

\n\n

But does it make sense to go beyond k=8? \nI guess everything now goes down to good features.

", "rawMarkdown": "I guess I should read the code in more detail.\r\n\r\nBut does it make sense to go beyond k=8? \r\nI guess everything now goes down to good features.", "votes": null }, { "id": "145350", "postDate": "11/17/2016 22:26:04", "content": "

Yes, we use k=32. Larger k doesn't improve performance for us.

", "rawMarkdown": "Yes, we use k=32. Larger k doesn't improve performance for us.", "votes": null }, { "id": "145351", "postDate": "11/17/2016 22:34:10", "content": "

That is a lot of RAM ....

", "rawMarkdown": "That is a lot of RAM ....", "votes": null }, { "id": "145352", "postDate": "11/17/2016 22:38:16", "content": "

[quote=idle_speculation;145339]

\n\n

You may be able to optimize your memory usage by condensing the field indices. By this I mean that if the indices for your field entries are 10, 100, 1000, 10000,... then recoding them as 1, 2, 3, 4,... should lead to a smaller model.

\n\n

[/quote]

\n\n
\n

label field1:index1:value1 field2:index2:value2

\n
\n\n

You can do it with fields and values but not with the indices, am I right? I'm not sure about that, but my understanding is that if the indices are the same for different fields the library's algo would think it's exactly the same thing and wouldn't work correctly?

", "rawMarkdown": "[quote=idle_speculation;145339]\r\n\r\nYou may be able to optimize your memory usage by condensing the field indices. By this I mean that if the indices for your field entries are 10, 100, 1000, 10000,... then recoding them as 1, 2, 3, 4,... should lead to a smaller model.\r\n\r\n[/quote]\r\n\r\n> label field1:index1:value1 field2:index2:value2\r\n\r\nYou can do it with fields and values but not with the indices, am I right? I'm not sure about that, but my understanding is that if the indices are the same for different fields the library's algo would think it's exactly the same thing and wouldn't work correctly?", "votes": null }, { "id": "145353", "postDate": "11/17/2016 22:43:41", "content": "

I looked at the source code and it seems that if you have index for one field like

\n\n
1 3 5 6 891 2 100\n
\n\n

memory wise it would be better to encode them as

\n\n
1 2 3 4 5 6 7\n
\n\n

Exctly the same goes for fields, as idle_speculation worte before.

", "rawMarkdown": "I looked at the source code and it seems that if you have index for one field like\r\n\r\n 1 3 5 6 891 2 100\r\n\r\nmemory wise it would be better to encode them as\r\n\r\n 1 2 3 4 5 6 7\r\n\r\nExctly the same goes for fields, as idle_speculation worte before.", "votes": null }, { "id": "145354", "postDate": "11/17/2016 22:44:29", "content": "

[quote=Marcin Pękalski;145351]

\n\n

That is a lot of RAM ....

\n\n

[/quote]

\n\n

True with original FFM. But we rewrote ffm extensively and with hashing trick we use 10 GB RAM most of the time.

", "rawMarkdown": "[quote=Marcin Pękalski;145351]\r\n\r\nThat is a lot of RAM ....\r\n\r\n[/quote]\r\n\r\nTrue with original FFM. But we rewrote ffm extensively and with hashing trick we use 10 GB RAM most of the time.", "votes": null }, { "id": "145355", "postDate": "11/17/2016 22:46:44", "content": "

@rcarson I'm waiting impatiently to see your pull request back to libffm :D

", "rawMarkdown": "rcarson I'm waiting impatiently to see your pull request back to libffm :D", "votes": null }, { "id": "145356", "postDate": "11/17/2016 22:49:57", "content": "

I think I will end up doing the same. Three more questions:

\n\n

1) What was your weapon of choice C, Python, Scala, or sth else?\n2) Have you considered changing optimizer to ALS? \n3) Is the key to making it use less memory is using sparse vectors/matrices?

", "rawMarkdown": "I think I will end up doing the same. Three more questions:\r\n\r\n1) What was your weapon of choice C, Python, Scala, or sth else?\r\n2) Have you considered changing optimizer to ALS? \r\n3) Is the key to making it use less memory is using sparse vectors/matrices?", "votes": null }, { "id": "145359", "postDate": "11/17/2016 22:58:16", "content": "

Or maybe it is possible to make it an online algorithm. \nEither way, it is time for me now.

", "rawMarkdown": "Or maybe it is possible to make it an online algorithm. \r\nEither way, it is time for me now.", "votes": null } ], "comments": [ { "id": 145188, "author_name": "sachintyagi", "author_url": "", "post_date": "11/17/2016 05:54:05", "content": "

We got it working with --on-disk flag and could play around with non-default k values.

\n\n

Perhaps, your train (or validation files) are not in proper format?

", "votes": null, "replies": [] }, { "id": 145320, "author_name": "mpekalski", "author_url": "", "post_date": "11/17/2016 19:58:41", "content": "

Unfortunately that is not an issue. All the values in the file are correct integers from 0 to 2^31-1, so they should fit in int.

\n\n

What I found is that memory requirements are higher than one might expect. In theory that should be m*n*k, but, the code makes k_aligned (aligned to the power of 2) so if you change from 4 to 5, k_aligned changes from 4 to 8. Basically, the memory requirement doubles instead of growing by 25%.

\n\n

Still that does not answer my question why it does not work with on_disk for me, but at least I know why it does fail after allocating 23GB of RAM.

", "votes": null, "replies": [] }, { "id": 145339, "author_name": "speculation", "author_url": "", "post_date": "11/17/2016 21:21:08", "content": "

The --on-disk flag only influences whether the training and validation sets are read into memory. The model parameters, whose size increases with -k, are always stored in memory.

\n\n

You may be able to optimize your memory usage by condensing the field indices. By this I mean that if the indices for your field entries are 10, 100, 1000, 10000,... then recoding them as 1, 2, 3, 4,... should lead to a smaller model.

", "votes": null, "replies": [] }, { "id": 145341, "author_name": "mpekalski", "author_url": "", "post_date": "11/17/2016 21:39:52", "content": "

Actually disabling auto-stop helped. My previous assumption was partially wrong. The problem was a couple of lines lower. When using auto-stop it tries to allocate memory for a dense vector of floats to store previous value of weights (vector W).

\n\n

Without auto-stopping I can use k=8, it still takes around 20GB and everything is fine.

\n\n

idle_speculation, regarding your hint, I my fields had numbers from 1 to 14, so it was ok from the beginning, but now I wonder if recoding the indexes themselves would make sense.

", "votes": null, "replies": [] }, { "id": 145343, "author_name": "jiweiliu", "author_url": "", "post_date": "11/17/2016 21:51:29", "content": "

[quote=Marcin Pękalski;145341]

\n\n

I my fields had numbers from 1 to 14, so it was ok from the beginning, but now I wonder if recoding the indexes themselves would make sense.

\n\n

[/quote]

\n\n

Actually you could start numbering from 0 so one fewer field in total :P

", "votes": null, "replies": [] }, { "id": 145344, "author_name": "jiweiliu", "author_url": "", "post_date": "11/17/2016 21:53:44", "content": "

[quote=Marcin Pękalski;145320]\nIn theory that should be m*n*k\n[/quote]

\n\n

Using hashing trick, n could be reduce to any small number that fits in the memory.

", "votes": null, "replies": [] }, { "id": 145348, "author_name": "mpekalski", "author_url": "", "post_date": "11/17/2016 22:21:35", "content": "

I guess I should read the code in more detail.

\n\n

But does it make sense to go beyond k=8? \nI guess everything now goes down to good features.

", "votes": null, "replies": [] }, { "id": 145350, "author_name": "jiweiliu", "author_url": "", "post_date": "11/17/2016 22:26:04", "content": "

Yes, we use k=32. Larger k doesn't improve performance for us.

", "votes": null, "replies": [] }, { "id": 145351, "author_name": "mpekalski", "author_url": "", "post_date": "11/17/2016 22:34:10", "content": "

That is a lot of RAM ....

", "votes": null, "replies": [] }, { "id": 145352, "author_name": "radek1st", "author_url": "", "post_date": "11/17/2016 22:38:16", "content": "

[quote=idle_speculation;145339]

\n\n

You may be able to optimize your memory usage by condensing the field indices. By this I mean that if the indices for your field entries are 10, 100, 1000, 10000,... then recoding them as 1, 2, 3, 4,... should lead to a smaller model.

\n\n

[/quote]

\n\n
\n

label field1:index1:value1 field2:index2:value2

\n
\n\n

You can do it with fields and values but not with the indices, am I right? I'm not sure about that, but my understanding is that if the indices are the same for different fields the library's algo would think it's exactly the same thing and wouldn't work correctly?

", "votes": null, "replies": [] }, { "id": 145353, "author_name": "mpekalski", "author_url": "", "post_date": "11/17/2016 22:43:41", "content": "

I looked at the source code and it seems that if you have index for one field like

\n\n
1 3 5 6 891 2 100\n
\n\n

memory wise it would be better to encode them as

\n\n
1 2 3 4 5 6 7\n
\n\n

Exctly the same goes for fields, as idle_speculation worte before.

", "votes": null, "replies": [] }, { "id": 145354, "author_name": "jiweiliu", "author_url": "", "post_date": "11/17/2016 22:44:29", "content": "

[quote=Marcin Pękalski;145351]

\n\n

That is a lot of RAM ....

\n\n

[/quote]

\n\n

True with original FFM. But we rewrote ffm extensively and with hashing trick we use 10 GB RAM most of the time.

", "votes": null, "replies": [] }, { "id": 145355, "author_name": "radek1st", "author_url": "", "post_date": "11/17/2016 22:46:44", "content": "

@rcarson I'm waiting impatiently to see your pull request back to libffm :D

", "votes": null, "replies": [] }, { "id": 145356, "author_name": "mpekalski", "author_url": "", "post_date": "11/17/2016 22:49:57", "content": "

I think I will end up doing the same. Three more questions:

\n\n

1) What was your weapon of choice C, Python, Scala, or sth else?\n2) Have you considered changing optimizer to ALS? \n3) Is the key to making it use less memory is using sparse vectors/matrices?

", "votes": null, "replies": [] }, { "id": 145359, "author_name": "mpekalski", "author_url": "", "post_date": "11/17/2016 22:58:16", "content": "

Or maybe it is possible to make it an online algorithm. \nEither way, it is time for me now.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "145141": "Did anybody try using --on-disk flag for FFM?\r\n\r\nIt does not seem to work for me. I wanted to use some bigger k value (bigger than default) and with on-disk flag it still throws \"bad_alloc\".", "145188": "We got it working with --on-disk flag and could play around with non-default k values. \r\n\r\nPerhaps, your train (or validation files) are not in proper format?", "145320": "Unfortunately that is not an issue. All the values in the file are correct integers from 0 to 2^31-1, so they should fit in int. \r\n\r\nWhat I found is that memory requirements are higher than one might expect. In theory that should be m*n*k, but, the code makes k_aligned (aligned to the power of 2) so if you change from 4 to 5, k_aligned changes from 4 to 8. Basically, the memory requirement doubles instead of growing by 25%. \r\n\r\nStill that does not answer my question why it does not work with on_disk for me, but at least I know why it does fail after allocating 23GB of RAM.", "145339": "The --on-disk flag only influences whether the training and validation sets are read into memory. The model parameters, whose size increases with -k, are always stored in memory.\r\n\r\nYou may be able to optimize your memory usage by condensing the field indices. By this I mean that if the indices for your field entries are 10, 100, 1000, 10000,... then recoding them as 1, 2, 3, 4,... should lead to a smaller model.", "145341": "Actually disabling auto-stop helped. My previous assumption was partially wrong. The problem was a couple of lines lower. When using auto-stop it tries to allocate memory for a dense vector of floats to store previous value of weights (vector W).\r\n\r\nWithout auto-stopping I can use k=8, it still takes around 20GB and everything is fine. \r\n\r\nidle_speculation, regarding your hint, I my fields had numbers from 1 to 14, so it was ok from the beginning, but now I wonder if recoding the indexes themselves would make sense.", "145343": "[quote=Marcin Pękalski;145341]\r\n\r\nI my fields had numbers from 1 to 14, so it was ok from the beginning, but now I wonder if recoding the indexes themselves would make sense.\r\n\r\n[/quote]\r\n\r\nActually you could start numbering from 0 so one fewer field in total :P", "145344": "[quote=Marcin Pękalski;145320]\r\nIn theory that should be m*n*k\r\n[/quote]\r\n\r\nUsing hashing trick, n could be reduce to any small number that fits in the memory.", "145348": "I guess I should read the code in more detail.\r\n\r\nBut does it make sense to go beyond k=8? \r\nI guess everything now goes down to good features.", "145350": "Yes, we use k=32. Larger k doesn't improve performance for us.", "145351": "That is a lot of RAM ....", "145352": "[quote=idle_speculation;145339]\r\n\r\nYou may be able to optimize your memory usage by condensing the field indices. By this I mean that if the indices for your field entries are 10, 100, 1000, 10000,... then recoding them as 1, 2, 3, 4,... should lead to a smaller model.\r\n\r\n[/quote]\r\n\r\n> label field1:index1:value1 field2:index2:value2\r\n\r\nYou can do it with fields and values but not with the indices, am I right? I'm not sure about that, but my understanding is that if the indices are the same for different fields the library's algo would think it's exactly the same thing and wouldn't work correctly?", "145353": "I looked at the source code and it seems that if you have index for one field like\r\n\r\n 1 3 5 6 891 2 100\r\n\r\nmemory wise it would be better to encode them as\r\n\r\n 1 2 3 4 5 6 7\r\n\r\nExctly the same goes for fields, as idle_speculation worte before.", "145354": "[quote=Marcin Pękalski;145351]\r\n\r\nThat is a lot of RAM ....\r\n\r\n[/quote]\r\n\r\nTrue with original FFM. But we rewrote ffm extensively and with hashing trick we use 10 GB RAM most of the time.", "145355": "rcarson I'm waiting impatiently to see your pull request back to libffm :D", "145356": "I think I will end up doing the same. Three more questions:\r\n\r\n1) What was your weapon of choice C, Python, Scala, or sth else?\r\n2) Have you considered changing optimizer to ALS? \r\n3) Is the key to making it use less memory is using sparse vectors/matrices?", "145359": "Or maybe it is possible to make it an online algorithm. \r\nEither way, it is time for me now." }, "source": "meta" }