{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"markdown","source":"# Backtick hands on kernel\nAfter some initial setup a few sectionsfollows, that upon completetion will allow you to implement a simple stock market price prediction pipeline.\n\nThe pandas API documenation is a useful source to solve some of the problems: https://pandas.pydata.org/pandas-docs/stable/reference/index.html\n\n## 1 Data investigation\n## 2 Preprocessing\n## 3 Feature Creation\n## 4 Binary Classification in LightGBM\n## 5 Confidence Interval & Scoring\n## 6 Model Evaluation\n## 7 Voting"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\nimport numpy as np\nimport pandas as pd\nimport matplotlib.pyplot as plt\n%matplotlib inline\nimport lightgbm as lgb\nfrom kaggle.competitions import twosigmanews\nfrom sklearn.metrics import confusion_matrix, accuracy_score","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"40f992c0e5b52a286a32712804b85adba5d01e8b"},"cell_type":"markdown","source":"# Load the data from the competition environment\nYou can only load the data from the env once - you'll need to completely restart your kernel if you lose it (second right button at the very bottom)"},{"metadata":{"trusted":true,"_uuid":"049d54a6bc75b2117e66c5f43eb7710bb5ab8c5b","scrolled":true},"cell_type":"code","source":"env = twosigmanews.make_env()\n\nmarket_orig, news_orig = env.get_training_data()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7faf77e7c9806b2a4e05d01b8524ee597268cae3"},"cell_type":"code","source":"# We'll use data from 2013 onwards to speed things up a bit\nmarket_orig = market_orig[market_orig['time'] >= \"2013-01-01\"]\nmarket_orig.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c0037c5ceafaf0d3cd9830cfc00806a1037cbaf7"},"cell_type":"markdown","source":"# Section 1, Data Investigation\n"},{"metadata":{"_uuid":"d4917988d26d1f3e0d6257498a660314a566fa7d"},"cell_type":"markdown","source":"##  1.1 How many unique assetNames are there?"},{"metadata":{"trusted":true,"_uuid":"efc6fbc7e2bdf30574e9d272c08c07b459d1d525"},"cell_type":"code","source":"def unique_asset_names(df):\n    return \"TODO\"\n\nunique_asset_names(market_orig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ad9029bdfd9279957ee10f299a2277aa2b86abab"},"cell_type":"markdown","source":"## 1.2 Find the most common assetName.\nHint:\n\n'noitagergga tnuoc htiw denibmoc ybpuorg noitcnuf sadnap eht esu nac uoy'[::-1]"},{"metadata":{"trusted":true,"_uuid":"66122a9b3f4bf5f88d3c7caf4d5363604a48d484"},"cell_type":"code","source":"def most_common_asset_name(df):\n    return \"TODO\"\n\nmost_common_asset_name(market_orig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"ee93f4ae90cc02904eca03ed353f0e8b7d829be2"},"cell_type":"markdown","source":"## 1.3  What is the maximum value of returnsOpenNextMktres10?"},{"metadata":{"trusted":true,"_uuid":"b1d835d6475822b8cf8e560351470dc43404b751"},"cell_type":"code","source":"def max_next10(df):\n    return \"TODO\"\n\nmax_next10(market_orig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"232606ff6ae3b72b543fd85b31d0addf1568c563"},"cell_type":"markdown","source":"## 1.4 Plot the close price of Facebook (assetCode=\"FB.O\") over time\n\nHint:\n\n'sixa x sa nmuloc emit eht ssap ,biltolptam fo noitcnuf tolp eht tuo kcehc'[::-1]"},{"metadata":{"trusted":true,"_uuid":"0b19f307892c299c10471343ab324ee8dae3a0ce"},"cell_type":"code","source":"# Matplotlib is available as \"plt\"\nplt.figure(figsize=(12,7))\n\ndef plot_fb(df):\n    return \"TODO\"\n\nplot_fb(market_orig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d1630c17b01df67a24fb39d90bd0c039476b6e4e"},"cell_type":"markdown","source":"## 1.5 What is the min, max and mean amount of days an asset is present in the dataset? (use \"assetCode\")"},{"metadata":{"trusted":true,"_uuid":"655eb82de96c410a50d53955ab147af96a536c40"},"cell_type":"code","source":"def asset_presence(df):\n    return \"TODO\"\n\nasset_presence(market_orig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"5728caf3c2eb23a18dd382fcee35615118acb733"},"cell_type":"markdown","source":"# Section 2. Preprocessing"},{"metadata":{"_uuid":"bc88fbda8f49110d4594480e5605c33687136a8d"},"cell_type":"markdown","source":"## 2.1 Remove all rows where assetName is \"Unknown\"\n*Hint:*\n\n'dnammoc nisi eht htiw emarfatad eht no gniksam naeloob esu ot si noitulos elbissop a'[::-1]"},{"metadata":{"trusted":true,"_uuid":"37255f01ce8b6f1b415adff24129d36072def277"},"cell_type":"code","source":"def remove_unknown(df):\n    return df\n\nmarket = remove_unknown(market_orig)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c8238451faa2332068a4a2b3f8ce63d5153e33de"},"cell_type":"markdown","source":"## 2.2 Limit the returnsOpenNextMktres10 column to values between -1 and 1\n\n*Hint:*\n\n'noitcnuf pilc sadnap eht yrt'[::-1]"},{"metadata":{"trusted":true,"_uuid":"1ccc69cbbb6559f385bcff8ef0b4148d71d87b10"},"cell_type":"code","source":"def clip_next10(df):\n    return df\n\nmarket = clip_next10(market)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"12b79fa4a421af16372b988dd1ceb8eff372cc62"},"cell_type":"markdown","source":"## 2.3 Remove instances where the asset is not present in the data for more than 30 days (use \"assetCode\")"},{"metadata":{"trusted":true,"_uuid":"428b28c0ec1d7d411343f9f99c100113b6641291"},"cell_type":"code","source":"def remove_short_lived(df):\n    return df\n\nmarket = remove_short_lived(market)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9d65d4114d171a7af586ec860b20e856df4fa351"},"cell_type":"markdown","source":"# Section 3. Feature Creation"},{"metadata":{"_uuid":"449624e3d3c01bb71604d7abf6fef7c19a059ca8"},"cell_type":"markdown","source":"## 3.1 Add the (RSI) indicator for a few different periods (default is 14)"},{"metadata":{"trusted":true,"scrolled":false,"_uuid":"9e47fe1ecb23f96bbc7da2337bd6a26ffd0d6ed2"},"cell_type":"code","source":"# https://github.com/bukosabino/ta/blob/master/ta/momentum.py\ndef ema(series, periods, fillna=False):\n    if fillna:\n        return series.ewm(span=periods, min_periods=0).mean()\n    return series.ewm(span=periods, min_periods=periods).mean()\n\ndef rsi(close, n=14, fillna=False):\n    \"\"\"Relative Strength Index (RSI)\n    Compares the magnitude of recent gains and losses over a specified time\n    period to measure speed and change of price movements of a security. It is\n    primarily used to attempt to identify overbought or oversold conditions in\n    the trading of an asset.\n    https://www.investopedia.com/terms/r/rsi.asp\n    Args:\n        close(pandas.Series): dataset 'Close' column.\n        n(int): n period.\n        fillna(bool): if True, fill nan values.\n    Returns:\n        pandas.Series: New feature generated.\n    \"\"\"\n    diff = close.diff()\n    which_dn = diff < 0\n\n    up, dn = diff, diff*0\n    up[which_dn], dn[which_dn] = 0, -up[which_dn]\n\n    emaup = ema(up, n, fillna)\n    emadn = ema(dn, n, fillna)\n\n    rsi = 100 * emaup / (emaup + emadn)\n    if fillna:\n        rsi = rsi.replace([np.inf, -np.inf], np.nan).fillna(50)\n    return pd.Series(rsi, name='rsi')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"0ec8dec90a7ffc7bc1ac3772212e1c8bdbe68f07"},"cell_type":"code","source":"# TODO: Add the RSI indicator to the market df","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"f8d879c17d0c60b54895a99b9de09d85a9400500"},"cell_type":"markdown","source":"## 3.2 Add features which measures the average volume of the last [5, 10, 20] days\n\n*Hint:*\n\n'emarfatad eht revo wodniw gnillor a etaerc nac uoy'[::-1]"},{"metadata":{"trusted":true,"_uuid":"f4879b8384fe2ef228b16fba1849b4675ee9bc4f"},"cell_type":"code","source":"def add_volume_avg(df):\n    return df\n\nadd_volume_avg(market)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"17838af95bf18dfa5cf4e0d913b622c19f1b1348"},"cell_type":"markdown","source":"## 3.3 (Optional) - Add additional lag features for some relevant columns. Compute min, max and mean for different window sizes\n\n*Hint:*\n\n'76-tpircs-ade/rogoegqq/moc.elggak.www//:sptth :lenrek siht ni noitaripsni dna spit ecnamrofrep doog emos ereht'[::-1]\n\n\n"},{"metadata":{"trusted":true,"_uuid":"9fdd406d12637e8544383d7690cadbd34b228335"},"cell_type":"code","source":"def add_lag_features(df):\n    return df\n    \nmarket = add_lag_features(market)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1136deeb3296cb8ab1ec459a14b12ef9f90d8602"},"cell_type":"markdown","source":"\n# Section 4 - Binary Classification in LightGBM "},{"metadata":{"_uuid":"826c2a24e34769ccf62264b4e0217c6398015829"},"cell_type":"markdown","source":"## 4.1 Separate the data into a train and test set based on reasonable dates\n\nIt is wise to have small gap between your train set and test set."},{"metadata":{"trusted":true,"_uuid":"1f965e4781b687552cac86009cda9ce85ccce6be"},"cell_type":"code","source":"# Returns:\n#    X: numpy matrix with relevant features only\n#    y: numpy array of class values, 0 if returnsOpenNextMktres10 is negative, else 1\n\nTRAIN_END_DATE  = \"\"\nTEST_START_DATE = \"\"\n\ndef get_data(df):\n    pass\n\nX_train, y_train = get_data()\nX_test, y_test   = get_data()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"f34d6e3f48f54069ae989ab21ffe859d4460c60a"},"cell_type":"markdown","source":"## 4.2 Familiarize yourself with the LightGBM API"},{"metadata":{"trusted":true,"_uuid":"0046767b15cb2f6a7a7d568f5cf7e393d48f334b"},"cell_type":"code","source":"def train_clf(X_train, y_train):\n    # https://lightgbm.readthedocs.io/en/latest/Parameters.html\n    params = {\n        'objective': 'binary',\n        'num_threads': 4\n    }\n\n    train_set = lgb.Dataset(X_train, y_train)\n\n    # https://lightgbm.readthedocs.io/en/latest/Python-API.html#training-api\n    lgb_clf = lgb.train(params, train_set)\n    \n    return lgb_clf\n\nclf = train_clf(X_train, y_train)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"03b7471f9094a6ed760b8909a6ab0f79d89154cc"},"cell_type":"markdown","source":"## 4.3 Report the test accuracy score of your classifier\n"},{"metadata":{"trusted":true,"_uuid":"8f128c35e63d23fc39579422fb15125448aa9448"},"cell_type":"code","source":"def accuracy(clf, X_test, y_test):\n    return 0\n\naccuracy(clf, X_test, y_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6697116b9fec088f71a0fe61102cfc38f1c3c6b2"},"cell_type":"markdown","source":"## 4.4 Plot the feature importances of your trained model"},{"metadata":{"trusted":true,"_uuid":"aebbb23eabef18d80abd696aac19acad8459e847"},"cell_type":"code","source":"# You can use this helper, if you want.\ndef plot_feature_importances(clf, feature_columns):\n    features_imp = pd.DataFrame()\n    features_imp['features'] = list(feature_columns)[:]\n    features_imp['importance'] = clf.feature_importance()\n    features_imp = features_imp.sort_values(by='importance', ascending=False).reset_index()\n    shape = features_imp.shape[0]\n    \n    y_plot = -np.arange(shape)\n    plt.figure(figsize=(10,7))\n    plt.barh(y_plot, features_imp.loc[:shape,'importance'].values)\n    plt.yticks(y_plot,(features_imp.loc[:shape,'features']))\n    plt.xlabel('Feature importance')\n    plt.title('Features importance')\n    plt.tight_layout()\n\n","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"74aa0296cfade8a51dcd34e0c889947fbce451cc"},"cell_type":"markdown","source":"# 5 Confidence Interval & Scoring"},{"metadata":{"trusted":true,"_uuid":"158c898451e21f1da9dc5f2abcb44bae447a4629"},"cell_type":"markdown","source":"## 5.1 Construct a confidence value for each sample in the test set"},{"metadata":{"trusted":true,"_uuid":"8cb097238c830c80326256f86ce1a6c83e2a5851"},"cell_type":"code","source":"# Returns a series of confidence values\ndef get_confidence(clf, X_test):\n    y_pred = clf.predict(X_test)\n    \n    return \"TODO\"\n\nconfidence = get_confidence(clf, X_test)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7017dd8e578332b0e455a7c3016adcd10dbde99d"},"cell_type":"markdown","source":"## 5.2 Compute the score of your strategy\n\nMake sure you group the predictions by day, and use the formula in the competition description."},{"metadata":{"trusted":true,"_uuid":"046d319c79129dfa0e8f66ddb585d5f628cd4863"},"cell_type":"code","source":"# You can use this helper:\ndef get_scoring_data(market):\n    test_df         = market[[market['time'] > TEST_START_DATE]\n    test_df['date'] = df['time'].dt.date\n                     \n    actual_returns  = test_df['returnsOpenNextMktres10'].values.clip(-1, 1)\n    universe        = test_df['universe']\n    dates           = test_df['date']\n\n    return actual_returns, universe, dates\n\nactual_returns, universe, dates = get_scoring_data(market)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a8e7360dc8f98435a6a3db08e40842b2e47524ec"},"cell_type":"code","source":"def score(confidence, actual_returns, universe, dates):\n    return 0\n\nscore(confidence, actual_returns, universe, dates)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"080939423eec7c0a988b530362fdc4fbaae06047"},"cell_type":"markdown","source":"## 5.3 Plot the daily returns of your strategy"},{"metadata":{"trusted":true,"_uuid":"1647bdf394f978d065da5fb8155eedbf23f84c26"},"cell_type":"code","source":"# Modify the \"score\" function above to plot your strategy's daily returns","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"1e7621d7c0b82819bea8bea06be81cf7d090cd5e"},"cell_type":"markdown","source":"# 6 Model Evaluation"},{"metadata":{"_uuid":"425cf17b0bb03fc035a28766b185c036b198a27d"},"cell_type":"markdown","source":"## 6.1 Cross-validate your model using an appropriate approach for time series data\nLets see if the model performs similarly on data from different time periods.\n\nHopyfully, you can reuse some of the functions you've previously created."},{"metadata":{"trusted":true,"_uuid":"20c2d31d0686e94340fc4501c3f5c0cfac36edfe"},"cell_type":"code","source":"def cross_validate():\n    # 1. Create folds based on date from the data\n    # 2. Train a classifier for each fold\n    # 3. Test against related test set\n    # 4. Evaluate the results\n    \n    pass\n\ncross_validate()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"73b8ada885dcd467535384ef6da8800a96357f88"},"cell_type":"markdown","source":"# 7. Voting\n"},{"metadata":{"_uuid":"a9fdb987bd69674422036994a56a0a982512a817"},"cell_type":"markdown","source":"## 7.1 Democratize your solution. \nImplement a way for multiple classifiers to have a say about the confidence interval"},{"metadata":{"trusted":true,"_uuid":"0e957f0498c359eea1d3f7984c997b12da6afc5c"},"cell_type":"code","source":"# Implement a voting strategy, you can reuse a lot of the code from the cross_validation step","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cd83ce79bf803ed5ca6329d18dc33444b45fa2d2"},"cell_type":"markdown","source":"# 8 Revise your pipeline\nNow you hopefully have a good sense of the problem, and you can back and try to improve each step a long the way. \n\nHere's a few things you can try for a better score:\n\n* Better preprocessing\n* Use the news data set\n* Multiclass classification\n* Probability Selection\n* Further post-preprocessing strategies\n* Any ideas that you might have"},{"metadata":{"trusted":true,"_uuid":"2a6bcc5097e05853cd532b8852524fd65c05429b"},"cell_type":"code","source":"\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"062364dc55f03ffce564eed21ba3069621620102"},"cell_type":"code","source":"","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}