Hereby, I have gathered ideas on how to manage the huge amount of data without dropping (or not much) the predictive power. I will start with a very good observation from Konrad Banachewicz:
\n\nPlease, share more ideas/observations if you can think of smt.
", "messages": [ { "id": 292758, "postDate": "2018-03-08T15:21:01.487Z", "content": "Hereby, I have gathered ideas on how to manage the huge amount of data without dropping (or not much) the predictive power. I will start with a very good observation from Konrad Banachewicz:
\n\nPlease, share more ideas/observations if you can think of smt.
", "rawMarkdown": "Hereby, I have gathered ideas on how to manage the huge amount of data without dropping (or not much) the predictive power. I will start with a very good observation from Konrad Banachewicz:\n\n 1. https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/51347\nNamely, get rid of the year and month as they are all the same\n 2. I will propose to get rid of the minutes and seconds as well. One can still detect clicks coming from the same IP within an \"unreasonable\" amount of time by just comparing the date + hour. Even in the beginning simply using only the hour will be more than good enough.\n 3. use dask instead of pandas (way faster!!!!). I have also just read about \"pandas on ray\", have never used it myself but looks extremely promising.\n 4. in my opinion the column \"attributed time\" is totally useless - don't even read it when loading the files (the train data). What does it matter when it was attributed if you have to predict whether it is attributed or not?\n 5. Count the clicks from each IP and store this information (as a person with experience in online advertisement I can guarantee you that this will be one of the most influential features). Afterwards delete IP.\n 6. Actually before deleting it you can also create a column indicating if the user made more than x (say, 5) clicks within an hour.\n 7. Use the correct datatypes when loading the data\n 8. Use gc (garbage collector). Extremely easy to use and very helpful. Constantly delete anything that is not necessary anymore.\n\nPlease, share more ideas/observations if you can think of smt.", "votes": 69 }, { "id": 294940, "postDate": "2018-03-12T21:24:19.960Z", "content": "I created a tutorial kernel with some tips on how to manage data. It includes subsampling, loading in chunks, creative processing and basic dask tutorial.
\nhttps://www.kaggle.com/yuliagm/how-to-work-with-big-datasets-on-16g-ram-dask.
\nHope you guys find it useful!
Step 2 i.e. removal of attribute_time column and keeping click_time hr and day. I have done this using python and sqlite3 (inbuilt package of python). Code of the same is here :- https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/51823
\n\nFeel free to use this. I think after implementing the above stage, data size can be reduced to around 5 Gb which i think is quite manageable.
", "rawMarkdown": "Step 2 i.e. removal of attribute_time column and keeping click_time hr and day. I have done this using python and sqlite3 (inbuilt package of python). Code of the same is here :- https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/51823\n\nFeel free to use this. I think after implementing the above stage, data size can be reduced to around 5 Gb which i think is quite manageable. ", "votes": 4 }, { "id": 303115, "postDate": "2018-03-25T14:55:55.867Z", "content": "'the column \"attributed time\" is totally useless'\nI do not agree with this statement. How to know a priori whether the IPs in which that variable is a date one day after the last click are always reliable?\nThis is only a hypothetical example but I want to point out that it is algorithms and construction techniques that must decide whether a variable is important or not.\nObviously the intuition of the data scientist is important but I think it is good practice to verify the importance of having a variable within the model before eliminating it.
", "rawMarkdown": "**'the column \"attributed time\" is totally useless'**\nI do not agree with this statement. How to know a priori whether the IPs in which that variable is a date one day after the last click are always reliable?\nThis is only a hypothetical example but I want to point out that it is algorithms and construction techniques that must decide whether a variable is important or not.\nObviously the intuition of the data scientist is important but I think it is good practice to verify the importance of having a variable within the model before eliminating it.", "votes": 1 }, { "id": 296131, "postDate": "2018-03-14T19:01:03.620Z", "content": "I do not have enough computing resources to participate in this competition, 16 GB Ram, 16 GB Swap(SSD), even after using data size reduction techniques discussed in discussions I am not able to complete a full training pass on all data set.
I am planing to find a scheme to select a subset of train and validation data from whole train set and use that for model building through out the competition. Any suggestions?
You should take a look at this excellent kernel of @Yulia :
\n\nhttps://www.kaggle.com/yuliagm/how-to-work-with-big-datasets-on-16g-ram-dask.
", "rawMarkdown": "You should take a look at this excellent kernel of @Yulia :\n\nhttps://www.kaggle.com/yuliagm/how-to-work-with-big-datasets-on-16g-ram-dask. ", "votes": 1 } ] }, { "id": 293391, "postDate": "2018-03-09T19:17:39.630Z", "content": "Thanks for the shout-out - glad you liked the idea :-)
", "rawMarkdown": "Thanks for the shout-out - glad you liked the idea :-)", "votes": 1 }, { "id": 293073, "postDate": "2018-03-09T05:46:44.870Z", "content": "I Tried loading this via pyspark ... it does a decent job, Mllib does most of weightlifting of Random forest...
", "rawMarkdown": "I Tried loading this via pyspark ... it does a decent job, Mllib does most of weightlifting of Random forest... ", "votes": 1 }, { "id": 293009, "postDate": "2018-03-09T02:54:05.380Z", "content": "good insights. Are u ruling out the possibility of shared IP? I mean, multiple users with same IP can click on the ad without fraudulent intent, ryt?
", "rawMarkdown": "good insights. Are u ruling out the possibility of shared IP? I mean, multiple users with same IP can click on the ad without fraudulent intent, ryt?", "votes": 1, "replies": [ { "id": 293143, "postDate": "2018-03-09T08:55:43.403Z", "content": "Indeed, one should think of those things, my idea was just to share general guidelines. At further stage this can be tackled in not so complicated ways - for example instead of aggregating solely on IP level , group by the (IP, app, device/os) tuple.
", "rawMarkdown": "Indeed, one should think of those things, my idea was just to share general guidelines. At further stage this can be tackled in not so complicated ways - for example instead of aggregating solely on IP level , group by the (IP, app, device/os) tuple. ", "votes": 1 } ] }, { "id": 292783, "postDate": "2018-03-08T15:55:49.623Z", "content": "Negative downsampling (i.e downsample the majority class). We're looking for the behavior of a small subset, not the general pop (especially with data anonymized). \nhttps://www.kaggle.com/danofer/downsampling-for-fun-speed
\n\nI've had excellent results with this approach in RW problems involving more data, albeit non anonymized problems (and with corporate resources/servers handy).
", "rawMarkdown": "Negative downsampling (i.e downsample the majority class). We're looking for the behavior of a small subset, not the general pop (especially with data anonymized). \nhttps://www.kaggle.com/danofer/downsampling-for-fun-speed\n\nI've had excellent results with this approach in RW problems involving more data, albeit non anonymized problems (and with corporate resources/servers handy).", "votes": 2, "replies": [ { "id": 292788, "postDate": "2018-03-08T16:00:10.443Z", "content": "Agree. I believe a good downsampling approach would be to get rid of x% of the IP's that had only a single click and not conversion (is_attributed == 0). Those samples carry pretty much the least useful information...
", "rawMarkdown": "Agree. I believe a good downsampling approach would be to get rid of x% of the IP's that had only a single click and not conversion (is_attributed == 0). Those samples carry pretty much the least useful information...", "votes": 4 }, { "id": 292794, "postDate": "2018-03-08T16:17:11.957Z", "content": "Interesting idea. !\nI'd considered more \"directed\" downsampling (e.g. keep only IPs in the test set or is_attributed == 1 , but then there's a strong risk of bias/shift, e.g. in the other features)
", "rawMarkdown": "Interesting idea. !\nI'd considered more \"directed\" downsampling (e.g. keep only IPs in the test set or is_attributed == 1 , but then there's a strong risk of bias/shift, e.g. in the other features) ", "votes": 1 } ] }, { "id": 304699, "postDate": "2018-03-27T22:35:56.957Z", "content": "Are you guys training with all the data? If not, how will you choose the subsample?
\n\nMoreover, is it worth splitting the dataset in a way that the validation is closer time-wise to the test set? Or is it better to do a stratified separation?
", "rawMarkdown": "Are you guys training with all the data? If not, how will you choose the subsample? \n\nMoreover, is it worth splitting the dataset in a way that the validation is closer time-wise to the test set? Or is it better to do a stratified separation?" }, { "id": 303078, "postDate": "2018-03-25T13:03:33.377Z", "content": "Thank you for your wonderful ideas but I have yet to manage this dataset.\nOnce I load the dataset for xgboost (DMatrix function), that volume get bigger.\nI think it is useful to save memory to use libsvm-txt format because it can be directly loaded into xgboost but it's difficult to convert.
", "rawMarkdown": "Thank you for your wonderful ideas but I have yet to manage this dataset.\nOnce I load the dataset for xgboost (DMatrix function), that volume get bigger.\nI think it is useful to save memory to use libsvm-txt format because it can be directly loaded into xgboost but it's difficult to convert." }, { "id": 299940, "postDate": "2018-03-21T06:15:56.710Z", "content": "I am just an undergraduate student at iit (ism) , dhanbad , india. How long will it take me to learn everything?
", "rawMarkdown": "I am just an undergraduate student at iit (ism) , dhanbad , india. How long will it take me to learn everything?", "replies": [ { "id": 299946, "postDate": "2018-03-21T06:20:53.997Z", "content": "You can never learn everything in data science... Just do it regualary and enjoy
", "rawMarkdown": "You can never learn everything in data science... Just do it regualary and enjoy", "votes": 3 } ] }, { "id": 297458, "postDate": "2018-03-17T04:35:20.330Z", "content": "Here is another one, we can use out-of-core learning algos(vowpal wabbit), we can use algos that use stochastic gradient descent(logistic regression, neural nets)
", "rawMarkdown": "Here is another one, we can use out-of-core learning algos(vowpal wabbit), we can use algos that use stochastic gradient descent(logistic regression, neural nets) " }, { "id": 296499, "postDate": "2018-03-15T10:16:51.537Z", "content": "is there any packages/approach suggested using R?
", "rawMarkdown": "is there any packages/approach suggested using R?", "replies": [ { "id": 296502, "postDate": "2018-03-15T10:18:12.800Z", "content": "As a starting point, I would say data.table for aggregation / cleanup / processing, after that lightgbm for modeling.
", "rawMarkdown": "As a starting point, I would say data.table for aggregation / cleanup / processing, after that lightgbm for modeling.", "votes": 5 } ] }, { "id": 295970, "postDate": "2018-03-14T13:57:50.323Z", "content": "Good Insights.. I'm new to the field of Data Science, I searched a few datasets those were many hundreds GB (e.g. 800Gb, 900GB etc). How can we work on such datasets when we didn't have that much storage space or comparatively slower Internet connections. What are cloud resources we can utilize with cheaper costs.?
", "rawMarkdown": "Good Insights.. I'm new to the field of Data Science, I searched a few datasets those were many hundreds GB (e.g. 800Gb, 900GB etc). How can we work on such datasets when we didn't have that much storage space or comparatively slower Internet connections. What are cloud resources we can utilize with cheaper costs.?\n\n\n\n" }, { "id": 294650, "postDate": "2018-03-12T11:10:36.890Z", "content": "\"Count the clicks from each IP and store this information (as a person with experience in online advertisement I can guarantee you that this will be one of the most influential features). Afterwards delete IP.\"
\n\nWhich will be more useful information:- \n1. calculating this information using both train and test.\n2. calculating separately for test and train
\n\nIf I do it like the first option, what type of biases it will create?
", "rawMarkdown": "\"Count the clicks from each IP and store this information (as a person with experience in online advertisement I can guarantee you that this will be one of the most influential features). Afterwards delete IP.\"\n\nWhich will be more useful information:- \n1. calculating this information using both train and test.\n2. calculating separately for test and train\n\nIf I do it like the first option, what type of biases it will create?", "replies": [ { "id": 294877, "postDate": "2018-03-12T18:54:57.180Z", "content": "I would say number of clicks until conversion
", "rawMarkdown": "I would say number of clicks until conversion" } ] }, { "id": 294118, "postDate": "2018-03-11T10:17:27.680Z", "content": "Thanks for this tips !
\n\nI tried to use dask but when if we use sklearn or XGBoost algorithms, we have to use pandas so it's usefull just for data processing !
", "rawMarkdown": "Thanks for this tips ! \n\nI tried to use dask but when if we use sklearn or XGBoost algorithms, we have to use pandas so it's usefull just for data processing !" }, { "id": 293575, "postDate": "2018-03-10T06:55:14.317Z", "content": "hello, i'm new to kaggle and datascience, i wonder how i could use this huge volume of data, when i have a poor pc and internet connection. is there a way to run my script in cloud.
", "rawMarkdown": "hello, i'm new to kaggle and datascience, i wonder how i could use this huge volume of data, when i have a poor pc and internet connection. is there a way to run my script in cloud.", "replies": [ { "id": 293598, "postDate": "2018-03-10T07:47:46.907Z", "content": "Hey Zoubair, welcome to kaggle, There are kernel that kaggle offers which you could use, it burns cpus held by kaggle & wont put a load on your pc
", "rawMarkdown": "Hey Zoubair, welcome to kaggle, There are kernel that kaggle offers which you could use, it burns cpus held by kaggle & wont put a load on your pc", "votes": 1 }, { "id": 294221, "postDate": "2018-03-11T15:36:14.563Z", "content": "thank you.
", "rawMarkdown": "thank you." } ] }, { "id": 293308, "postDate": "2018-03-09T16:03:51.323Z", "content": "Thanks for sharing all this information, this could be very helpful to everyone.
", "rawMarkdown": "Thanks for sharing all this information, this could be very helpful to everyone." }, { "id": 300558, "postDate": "2018-03-21T18:03:23.157Z", "content": "Thanks for the insights
", "rawMarkdown": "Thanks for the insights" }, { "id": 296651, "postDate": "2018-03-15T15:08:37.663Z", "content": "wonderful idea, thanks!
", "rawMarkdown": "wonderful idea, thanks!" }, { "id": 295595, "postDate": "2018-03-13T22:03:45.533Z", "content": "Thanks! This is useful.
", "rawMarkdown": "Thanks! This is useful.", "isDeleted": true } ], "comments": [ { "id": 294940, "author_name": "yulia", "author_url": "", "post_date": "2018-03-12T21:24:19.960000", "content": "I created a tutorial kernel with some tips on how to manage data. It includes subsampling, loading in chunks, creative processing and basic dask tutorial.
\nhttps://www.kaggle.com/yuliagm/how-to-work-with-big-datasets-on-16g-ram-dask.
\nHope you guys find it useful!
Step 2 i.e. removal of attribute_time column and keeping click_time hr and day. I have done this using python and sqlite3 (inbuilt package of python). Code of the same is here :- https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/51823
\n\nFeel free to use this. I think after implementing the above stage, data size can be reduced to around 5 Gb which i think is quite manageable.
", "votes": 4, "replies": [] }, { "id": 303115, "author_name": "ddrbcn", "author_url": "", "post_date": "2018-03-25T14:55:55.867000", "content": "'the column \"attributed time\" is totally useless'\nI do not agree with this statement. How to know a priori whether the IPs in which that variable is a date one day after the last click are always reliable?\nThis is only a hypothetical example but I want to point out that it is algorithms and construction techniques that must decide whether a variable is important or not.\nObviously the intuition of the data scientist is important but I think it is good practice to verify the importance of having a variable within the model before eliminating it.
", "votes": 1, "replies": [] }, { "id": 296131, "author_name": "Sohaib Omar", "author_url": "", "post_date": "2018-03-14T19:01:03.620000", "content": "I do not have enough computing resources to participate in this competition, 16 GB Ram, 16 GB Swap(SSD), even after using data size reduction techniques discussed in discussions I am not able to complete a full training pass on all data set.
I am planing to find a scheme to select a subset of train and validation data from whole train set and use that for model building through out the competition. Any suggestions?
You should take a look at this excellent kernel of @Yulia :
\n\nhttps://www.kaggle.com/yuliagm/how-to-work-with-big-datasets-on-16g-ram-dask.
", "votes": 1, "replies": [] } ] }, { "id": 293391, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2018-03-09T19:17:39.630000", "content": "Thanks for the shout-out - glad you liked the idea :-)
", "votes": 1, "replies": [] }, { "id": 293073, "author_name": "tarun", "author_url": "", "post_date": "2018-03-09T05:46:44.870000", "content": "I Tried loading this via pyspark ... it does a decent job, Mllib does most of weightlifting of Random forest...
", "votes": 1, "replies": [] }, { "id": 293009, "author_name": "Nagaraju Oruganti", "author_url": "", "post_date": "2018-03-09T02:54:05.380000", "content": "good insights. Are u ruling out the possibility of shared IP? I mean, multiple users with same IP can click on the ad without fraudulent intent, ryt?
", "votes": 1, "replies": [ { "id": 293143, "author_name": "Asparuh Hristov", "author_url": "", "post_date": "2018-03-09T08:55:43.403000", "content": "Indeed, one should think of those things, my idea was just to share general guidelines. At further stage this can be tackled in not so complicated ways - for example instead of aggregating solely on IP level , group by the (IP, app, device/os) tuple.
", "votes": 1, "replies": [] } ] }, { "id": 292783, "author_name": "Dan Ofer", "author_url": "", "post_date": "2018-03-08T15:55:49.623000", "content": "Negative downsampling (i.e downsample the majority class). We're looking for the behavior of a small subset, not the general pop (especially with data anonymized). \nhttps://www.kaggle.com/danofer/downsampling-for-fun-speed
\n\nI've had excellent results with this approach in RW problems involving more data, albeit non anonymized problems (and with corporate resources/servers handy).
", "votes": 2, "replies": [ { "id": 292788, "author_name": "Asparuh Hristov", "author_url": "", "post_date": "2018-03-08T16:00:10.443000", "content": "Agree. I believe a good downsampling approach would be to get rid of x% of the IP's that had only a single click and not conversion (is_attributed == 0). Those samples carry pretty much the least useful information...
", "votes": 4, "replies": [] }, { "id": 292794, "author_name": "Dan Ofer", "author_url": "", "post_date": "2018-03-08T16:17:11.957000", "content": "Interesting idea. !\nI'd considered more \"directed\" downsampling (e.g. keep only IPs in the test set or is_attributed == 1 , but then there's a strong risk of bias/shift, e.g. in the other features)
", "votes": 1, "replies": [] } ] }, { "id": 304699, "author_name": "Skinish", "author_url": "", "post_date": "2018-03-27T22:35:56.957000", "content": "Are you guys training with all the data? If not, how will you choose the subsample?
\n\nMoreover, is it worth splitting the dataset in a way that the validation is closer time-wise to the test set? Or is it better to do a stratified separation?
", "votes": 0, "replies": [] }, { "id": 303078, "author_name": "NaN", "author_url": "", "post_date": "2018-03-25T13:03:33.377000", "content": "Thank you for your wonderful ideas but I have yet to manage this dataset.\nOnce I load the dataset for xgboost (DMatrix function), that volume get bigger.\nI think it is useful to save memory to use libsvm-txt format because it can be directly loaded into xgboost but it's difficult to convert.
", "votes": 0, "replies": [] }, { "id": 299940, "author_name": "abhishek kumar", "author_url": "", "post_date": "2018-03-21T06:15:56.710000", "content": "I am just an undergraduate student at iit (ism) , dhanbad , india. How long will it take me to learn everything?
", "votes": 0, "replies": [ { "id": 299946, "author_name": "Asparuh Hristov", "author_url": "", "post_date": "2018-03-21T06:20:53.997000", "content": "You can never learn everything in data science... Just do it regualary and enjoy
", "votes": 3, "replies": [] } ] }, { "id": 297458, "author_name": "Ravi Teja Gutta", "author_url": "", "post_date": "2018-03-17T04:35:20.330000", "content": "Here is another one, we can use out-of-core learning algos(vowpal wabbit), we can use algos that use stochastic gradient descent(logistic regression, neural nets)
", "votes": 0, "replies": [] }, { "id": 296499, "author_name": "Seymour", "author_url": "", "post_date": "2018-03-15T10:16:51.537000", "content": "is there any packages/approach suggested using R?
", "votes": 0, "replies": [ { "id": 296502, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2018-03-15T10:18:12.800000", "content": "As a starting point, I would say data.table for aggregation / cleanup / processing, after that lightgbm for modeling.
", "votes": 5, "replies": [] } ] }, { "id": 295970, "author_name": "Mutafaf Wahhaj", "author_url": "", "post_date": "2018-03-14T13:57:50.323000", "content": "Good Insights.. I'm new to the field of Data Science, I searched a few datasets those were many hundreds GB (e.g. 800Gb, 900GB etc). How can we work on such datasets when we didn't have that much storage space or comparatively slower Internet connections. What are cloud resources we can utilize with cheaper costs.?
", "votes": 0, "replies": [] }, { "id": 294650, "author_name": "", "author_url": "", "post_date": "2018-03-12T11:10:36.890000", "content": "\"Count the clicks from each IP and store this information (as a person with experience in online advertisement I can guarantee you that this will be one of the most influential features). Afterwards delete IP.\"
\n\nWhich will be more useful information:- \n1. calculating this information using both train and test.\n2. calculating separately for test and train
\n\nIf I do it like the first option, what type of biases it will create?
", "votes": 0, "replies": [ { "id": 294877, "author_name": "Asparuh Hristov", "author_url": "", "post_date": "2018-03-12T18:54:57.180000", "content": "I would say number of clicks until conversion
", "votes": 0, "replies": [] } ] }, { "id": 294118, "author_name": "Nathan Lauga", "author_url": "", "post_date": "2018-03-11T10:17:27.680000", "content": "Thanks for this tips !
\n\nI tried to use dask but when if we use sklearn or XGBoost algorithms, we have to use pandas so it's usefull just for data processing !
", "votes": 0, "replies": [] }, { "id": 293575, "author_name": "zoubairOmar", "author_url": "", "post_date": "2018-03-10T06:55:14.317000", "content": "hello, i'm new to kaggle and datascience, i wonder how i could use this huge volume of data, when i have a poor pc and internet connection. is there a way to run my script in cloud.
", "votes": 0, "replies": [ { "id": 293598, "author_name": "tarun", "author_url": "", "post_date": "2018-03-10T07:47:46.907000", "content": "Hey Zoubair, welcome to kaggle, There are kernel that kaggle offers which you could use, it burns cpus held by kaggle & wont put a load on your pc
", "votes": 1, "replies": [] }, { "id": 294221, "author_name": "zoubairOmar", "author_url": "", "post_date": "2018-03-11T15:36:14.563000", "content": "thank you.
", "votes": 0, "replies": [] } ] }, { "id": 293308, "author_name": "João Pedro Peinado", "author_url": "", "post_date": "2018-03-09T16:03:51.323000", "content": "Thanks for sharing all this information, this could be very helpful to everyone.
", "votes": 0, "replies": [] }, { "id": 300558, "author_name": "Gautham Prabhu", "author_url": "", "post_date": "2018-03-21T18:03:23.157000", "content": "Thanks for the insights
", "votes": 0, "replies": [] }, { "id": 296651, "author_name": "Wang Yongshuai", "author_url": "", "post_date": "2018-03-15T15:08:37.663000", "content": "wonderful idea, thanks!
", "votes": 0, "replies": [] }, { "id": 295595, "author_name": "", "author_url": "", "post_date": "2018-03-13T22:03:45.533000", "content": "Thanks! This is useful.
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "292758": "Hereby, I have gathered ideas on how to manage the huge amount of data without dropping (or not much) the predictive power. I will start with a very good observation from Konrad Banachewicz:\n\n 1. https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/51347\nNamely, get rid of the year and month as they are all the same\n 2. I will propose to get rid of the minutes and seconds as well. One can still detect clicks coming from the same IP within an \"unreasonable\" amount of time by just comparing the date + hour. Even in the beginning simply using only the hour will be more than good enough.\n 3. use dask instead of pandas (way faster!!!!). I have also just read about \"pandas on ray\", have never used it myself but looks extremely promising.\n 4. in my opinion the column \"attributed time\" is totally useless - don't even read it when loading the files (the train data). What does it matter when it was attributed if you have to predict whether it is attributed or not?\n 5. Count the clicks from each IP and store this information (as a person with experience in online advertisement I can guarantee you that this will be one of the most influential features). Afterwards delete IP.\n 6. Actually before deleting it you can also create a column indicating if the user made more than x (say, 5) clicks within an hour.\n 7. Use the correct datatypes when loading the data\n 8. Use gc (garbage collector). Extremely easy to use and very helpful. Constantly delete anything that is not necessary anymore.\n\nPlease, share more ideas/observations if you can think of smt.", "294940": "I created a tutorial kernel with some tips on how to manage data. It includes subsampling, loading in chunks, creative processing and basic dask tutorial. \nhttps://www.kaggle.com/yuliagm/how-to-work-with-big-datasets-on-16g-ram-dask. \nHope you guys find it useful!\n\n", "295308": "Step 2 i.e. removal of attribute_time column and keeping click_time hr and day. I have done this using python and sqlite3 (inbuilt package of python). Code of the same is here :- https://www.kaggle.com/c/talkingdata-adtracking-fraud-detection/discussion/51823\n\nFeel free to use this. I think after implementing the above stage, data size can be reduced to around 5 Gb which i think is quite manageable. ", "303115": "**'the column \"attributed time\" is totally useless'**\nI do not agree with this statement. How to know a priori whether the IPs in which that variable is a date one day after the last click are always reliable?\nThis is only a hypothetical example but I want to point out that it is algorithms and construction techniques that must decide whether a variable is important or not.\nObviously the intuition of the data scientist is important but I think it is good practice to verify the importance of having a variable within the model before eliminating it.", "296131": "I do not have enough computing resources to participate in this competition, 16 GB Ram, 16 GB Swap(SSD), even after using data size reduction techniques discussed in discussions I am not able to complete a full training pass on all data set.