{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **Crypto forecasting tutorial**","metadata":{"id":"8PNZS8gF3CVN"}},{"cell_type":"markdown","source":"# G-Research Crypto forecasting competition\n\nIn the [G-Research Crypto forecasting competition](link), participants have the challenge to predict  price returns across a bundle of major cryptocurrencies. To facilitate your participation, we have created this tutorial notebook covering some relevant concepts for the crypto forecasting challenge.\n\nThe notebook presents an introduction to crypto forecasting, describing the structure and elements of the dataset, some relevant statistical properties, as well as building a couple of ML baseline models and providing an example code submission.\n\n","metadata":{"id":"DO0FyA3q6rTl"}},{"cell_type":"markdown","source":"## The Cryptocurrency Market\n\nFirst, a quick introduction to the crypto world. Cryptocurrencies have become an extremely popular and volatile market, delivering massive returns (as well as losses) to investors. Thousands of cryptocurrencies have been created with a few major ones that many of you will have heard of including Bitcoin (BTC), Ether (ETH) or Dogecoin (DOGE).\n\nCryptocurrencies are traded extensively across crypto-exchanges, with an average volume of $41 billion traded daily over the last year, according to CryptoCompare (as of 25th July 2021). \n\nChanges in prices between different cryptocurrencies are highly interconnected. For example, Bitcoin has historically been a major driver of price changes across cryptocurrencies but other coins also impact the market.  \n","metadata":{"id":"x3HpXt5v-ko4"}},{"cell_type":"markdown","source":"\n## Forecasting returns\n\nA fundamental task in financial modeling is predicting how prices will behave in the near future. Using the time-series of historical prices as training data, we want to predict if prices will go up or down, and by how much, namely the asset *returns*.\n\nIn this competition, Kagglers are challenged to build machine learning models to predict the returns of 14 popular cryptocurrencies, in the time scale of minutes to hours. You will have access to millions of rows of minute-by-minute cryptocurrency trading data, with which you'll design your forecasting models for all 14 assets simultaneously. Your predictions will be evaluated by how much they correlate with real market data collected during the three-month evaluation period after the competition has closed. \n\nCryptocurrency returns prediction remains an open and extremely challenging forecasting task. This is a fascinating problem domain for the ML community given the extreme volatility of the assets, the non-stationary nature of the data, the market and meme manipulation, the correlation between assets and the very fast changing market conditions. We hope you find it as fascinating as we do! \n","metadata":{"id":"fDAWC9JNZITK"}},{"cell_type":"markdown","source":"# Dataset description\n\nNow, let's dive into the data! We start by loading the competition's dataset and inspecting its basic properties. ","metadata":{"id":"4e6OA0qc6x9h"}},{"cell_type":"markdown","source":"## Load the training set","metadata":{"id":"OevdFjmtar0j"}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nfrom datetime import datetime","metadata":{"id":"90fqnpNQ2wd8","execution":{"iopub.status.busy":"2022-07-16T18:26:42.414442Z","iopub.execute_input":"2022-07-16T18:26:42.414688Z","iopub.status.idle":"2022-07-16T18:26:42.419235Z","shell.execute_reply.started":"2022-07-16T18:26:42.414657Z","shell.execute_reply":"2022-07-16T18:26:42.418285Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_folder = \"../input/g-research-crypto-forecasting/\"\n!ls $data_folder","metadata":{"id":"Zw_oi4NETkKv","outputId":"e6fe292d-721e-4c6a-bc70-9026cfd5b915","execution":{"iopub.status.busy":"2022-07-16T18:26:42.420623Z","iopub.execute_input":"2022-07-16T18:26:42.420867Z","iopub.status.idle":"2022-07-16T18:26:43.221915Z","shell.execute_reply.started":"2022-07-16T18:26:42.420812Z","shell.execute_reply":"2022-07-16T18:26:43.220903Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"crypto_df = pd.read_csv(data_folder + 'train.csv')","metadata":{"id":"La0-ivRAUopA","scrolled":true,"execution":{"iopub.status.busy":"2022-07-16T18:26:44.625554Z","iopub.execute_input":"2022-07-16T18:26:44.625892Z","iopub.status.idle":"2022-07-16T18:27:37.231706Z","shell.execute_reply.started":"2022-07-16T18:26:44.625854Z","shell.execute_reply":"2022-07-16T18:27:37.230779Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"crypto_df.head(10)","metadata":{"id":"14ZSPE6daadW","outputId":"b6a4f262-4071-4c2f-dd42-12a253452297","execution":{"iopub.status.busy":"2022-07-16T18:27:37.233176Z","iopub.execute_input":"2022-07-16T18:27:37.233438Z","iopub.status.idle":"2022-07-16T18:27:37.260010Z","shell.execute_reply.started":"2022-07-16T18:27:37.233410Z","shell.execute_reply":"2022-07-16T18:27:37.259171Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that each row of the data set has the trading data for an asset, at a given minute timestamp, described in detail below. ","metadata":{"id":"ZUPxDsW8bCh1"}},{"cell_type":"markdown","source":"## Data features\nWe can see the different features included in the dataset. Specifically, the features included per asset are the following:\n*   **timestamp**: All timestamps are returned as second Unix timestamps (the number of seconds elapsed since 1970-01-01 00:00:00.000 UTC). Timestamps in this dataset are multiple of 60, indicating minute-by-minute data.\n*   **Asset_ID**: The asset ID corresponding to one of the crytocurrencies (e.g. `Asset_ID = 1` for Bitcoin). The mapping from `Asset_ID` to crypto asset is contained in `asset_details.csv`.\n*   **Count**: Total number of trades in the time interval (last minute).\n*   **Open**:\tOpening price of the time interval (in USD).\n*   **High**:\tHighest price reached during time interval (in USD).\n*   **Low**: Lowest price reached during time interval (in USD).\n*   **Close**:\tClosing price of the time interval (in USD).\n*   **Volume**:\tQuantity of asset bought or sold, displayed in base currency USD.\n*   **VWAP**: The average price of the asset over the time interval, weighted by volume. VWAP is an aggregated form of trade data.\n*   **Target**: Residual log-returns for the asset over a 15 minute horizon. \n\nThe first two columns define the time and asset indexes for this data row. The 6 middle columns are feature columns with the trading data for this asset and minute in time. The last column is the prediction target, which we will get to later in more detail.\n\nWe also view the asset information, including the list of all assets, the `Asset_ID` to asset mapping, and the weight of each asset used to weigh their relative importance in the evaluation metric.","metadata":{"id":"YjN-TfBUx2ml"}},{"cell_type":"code","source":"asset_details = pd.read_csv(data_folder + 'asset_details.csv')\nasset_details","metadata":{"execution":{"iopub.status.busy":"2022-07-16T18:27:37.261212Z","iopub.execute_input":"2022-07-16T18:27:37.261534Z","iopub.status.idle":"2022-07-16T18:27:37.278396Z","shell.execute_reply.started":"2022-07-16T18:27:37.261504Z","shell.execute_reply":"2022-07-16T18:27:37.277580Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Candlestick charts\n\nThe trading data format is an aggregated form of market data including for Open, High, Low and Close. We can visualize this data through the commonly used candlestick bar chart, which allows traders to perform technical analysis on intraday values. The bar's body length represents the price range between the open and close of that day's trading. When the bar is red, it means the close was lower than the open, and green otherwise. These are also referred to as bullish and bearish candlesticks. The wicks above and below the bars show the high and low prices of that interval's trading.","metadata":{"id":"QNWAfs_O0hyf"}},{"cell_type":"markdown","source":"We can visualize a slice of the Bitcoin prices using the `plotly` library. The bottom part of the plot shows a rangeslider, which you can use to zoom in the plot.","metadata":{"id":"570DeXQa0skY"}},{"cell_type":"code","source":"btc = crypto_df[crypto_df[\"Asset_ID\"]==1].set_index(\"timestamp\") # Asset_ID = 1 for Bitcoin\nbtc_mini = btc.iloc[-200:] # Select recent data rows","metadata":{"id":"tMxYx8ZO3fuy","execution":{"iopub.status.busy":"2022-07-16T18:27:37.280240Z","iopub.execute_input":"2022-07-16T18:27:37.280443Z","iopub.status.idle":"2022-07-16T18:27:37.668704Z","shell.execute_reply.started":"2022-07-16T18:27:37.280419Z","shell.execute_reply":"2022-07-16T18:27:37.667440Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import plotly.graph_objects as go\n\nfig = go.Figure(data=[go.Candlestick(x=btc_mini.index, open=btc_mini['Open'], high=btc_mini['High'], low=btc_mini['Low'], close=btc_mini['Close'])])\nfig.show()","metadata":{"id":"HCiq4HwD4ySs","outputId":"1d422d61-2382-41f0-ba2d-ca42946ae928","execution":{"iopub.status.busy":"2022-07-16T18:27:37.670177Z","iopub.execute_input":"2022-07-16T18:27:37.671188Z","iopub.status.idle":"2022-07-16T18:27:37.804239Z","shell.execute_reply.started":"2022-07-16T18:27:37.671137Z","shell.execute_reply":"2022-07-16T18:27:37.803438Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Preprocessing","metadata":{"id":"S-oceNYiivUi"}},{"cell_type":"markdown","source":"## Dealing with missing data\n","metadata":{"id":"K9ce7eWddHfM"}},{"cell_type":"markdown","source":"Let us inspect the data for another important asset, Ethereum.","metadata":{"id":"AOpyJzKgFuYb"}},{"cell_type":"code","source":"eth = crypto_df[crypto_df[\"Asset_ID\"]==6].set_index(\"timestamp\") # Asset_ID = 6 for Ethereum\neth.info(show_counts =True)","metadata":{"id":"cuWT1Suxzjvb","outputId":"04ff726c-8858-4d87-b592-ee1b036f0996","execution":{"iopub.status.busy":"2022-07-16T18:29:35.170718Z","iopub.execute_input":"2022-07-16T18:29:35.172837Z","iopub.status.idle":"2022-07-16T18:29:35.621187Z","shell.execute_reply.started":"2022-07-16T18:29:35.172693Z","shell.execute_reply":"2022-07-16T18:29:35.620197Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see the number of rows in the training set, and that there are missing values for the targets columns, which we will address later. Let's confirm that:\n","metadata":{"id":"3qryFWzMgXgo"}},{"cell_type":"code","source":"eth.isna().sum()","metadata":{"id":"0hVTUf1LflPM","outputId":"4c3c1993-1594-44c2-8457-52e94c199a5f","execution":{"iopub.status.busy":"2022-07-16T18:29:47.501538Z","iopub.execute_input":"2022-07-16T18:29:47.502032Z","iopub.status.idle":"2022-07-16T18:29:47.543231Z","shell.execute_reply.started":"2022-07-16T18:29:47.501981Z","shell.execute_reply":"2022-07-16T18:29:47.542436Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's check the time range for Bitcoin and Ethereum data, using the coversion from timestamp to `datetime`.","metadata":{"id":"sJ_CF60EfpQS"}},{"cell_type":"code","source":"btc.head()","metadata":{"execution":{"iopub.status.busy":"2022-07-16T18:30:01.950367Z","iopub.execute_input":"2022-07-16T18:30:01.950857Z","iopub.status.idle":"2022-07-16T18:30:01.967402Z","shell.execute_reply.started":"2022-07-16T18:30:01.950801Z","shell.execute_reply":"2022-07-16T18:30:01.966764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"beg_btc = btc.index[0].astype('datetime64[s]')\nend_btc = btc.index[-1].astype('datetime64[s]')\nbeg_eth = eth.index[0].astype('datetime64[s]')\nend_eth = eth.index[-1].astype('datetime64[s]')\n\nprint('BTC data goes from ', beg_btc, 'to ', end_btc)\nprint('Ethereum data goes from ', beg_eth, 'to ', end_eth)","metadata":{"id":"YrVaTbdtZ6Qv","outputId":"c640691c-075f-4573-e3a4-ad24ae468f72","execution":{"iopub.status.busy":"2022-07-16T18:30:12.837494Z","iopub.execute_input":"2022-07-16T18:30:12.837971Z","iopub.status.idle":"2022-07-16T18:30:12.845096Z","shell.execute_reply.started":"2022-07-16T18:30:12.837926Z","shell.execute_reply":"2022-07-16T18:30:12.844387Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Missing asset data, for a given minute, is not represented by NaN's, but instead by the absence of those rows. We can check the timestamp difference between consecutive rows to see if there is missing data.","metadata":{"id":"4PIgY9dtZ7BX"}},{"cell_type":"code","source":"(eth.index[1:]-eth.index[:-1]).value_counts().head()","metadata":{"id":"KOrD8uIked2c","outputId":"51d9c340-a88c-4476-fdca-cffa8edc5c39","execution":{"iopub.status.busy":"2022-07-16T18:34:31.804188Z","iopub.execute_input":"2022-07-16T18:34:31.805285Z","iopub.status.idle":"2022-07-16T18:34:31.836329Z","shell.execute_reply.started":"2022-07-16T18:34:31.805242Z","shell.execute_reply":"2022-07-16T18:34:31.835641Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Notice that there are many gaps in the data. To work with most time series models, we should preprocess our data into a format without time gaps. To fill the gaps, we can use the `.reindex()` method for forward filling, filling gaps with the previous valid value. ","metadata":{"id":"GFLjIGONhN9s"}},{"cell_type":"code","source":"eth = eth.reindex(range(eth.index[0],eth.index[-1]+60,60),method='pad')","metadata":{"id":"8h29S-1Hhp3Y","execution":{"iopub.status.busy":"2022-07-16T18:35:04.924765Z","iopub.execute_input":"2022-07-16T18:35:04.925231Z","iopub.status.idle":"2022-07-16T18:35:05.434518Z","shell.execute_reply.started":"2022-07-16T18:35:04.925197Z","shell.execute_reply":"2022-07-16T18:35:05.433543Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And check that are no time gaps now.","metadata":{"id":"4sX6uvcQiCC5"}},{"cell_type":"code","source":"(eth.index[1:]-eth.index[:-1]).value_counts().head()","metadata":{"id":"UDUygkngiBKT","outputId":"c9f12e5c-f07d-4063-c35e-2eec6d692438","execution":{"iopub.status.busy":"2022-07-16T18:35:07.162883Z","iopub.execute_input":"2022-07-16T18:35:07.163208Z","iopub.status.idle":"2022-07-16T18:35:07.213680Z","shell.execute_reply.started":"2022-07-16T18:35:07.163173Z","shell.execute_reply":"2022-07-16T18:35:07.212802Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data visualisation\n\nWe  will start by visualising the Close prices for the two assets we have selected.","metadata":{"id":"_TcwTqisePHB"}},{"cell_type":"code","source":"import matplotlib.pyplot as plt\n\n# plot vwap time series for both chosen assets\nf = plt.figure(figsize=(15,4))\n\n# fill missing values for BTC\nbtc = btc.reindex(range(btc.index[0],btc.index[-1]+60,60),method='pad')\n\nax = f.add_subplot(121)\nplt.plot(btc['Close'], label='BTC')\nplt.legend()\nplt.xlabel('Time')\nplt.ylabel('Bitcoin')\n\nax2 = f.add_subplot(122)\nax2.plot(eth['Close'], color='red', label='ETH')\nplt.legend()\nplt.xlabel('Time')\nplt.ylabel('Ethereum')\n\nplt.tight_layout()\nplt.show()","metadata":{"id":"PtVHC0YTxq-l","outputId":"525bc0d9-9673-42d0-ab73-aadc5e6afdad","execution":{"iopub.status.busy":"2022-07-16T18:40:55.540409Z","iopub.execute_input":"2022-07-16T18:40:55.541565Z","iopub.status.idle":"2022-07-16T18:40:59.112922Z","shell.execute_reply.started":"2022-07-16T18:40:55.541485Z","shell.execute_reply":"2022-07-16T18:40:59.111804Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The assets have quite different history, but we could check if they correlate in recent times.","metadata":{"id":"ECfuzVHyzEF2"}},{"cell_type":"code","source":"import time\n\n# auxiliary function, from datetime to timestamp\ntotimestamp = lambda s: np.int32(time.mktime(datetime.strptime(s, \"%d/%m/%Y\").timetuple()))\n\n# create intervals\nbtc_mini_2021 = btc.loc[totimestamp('01/06/2021'):totimestamp('01/07/2021')]\neth_mini_2021 = eth.loc[totimestamp('01/06/2021'):totimestamp('01/07/2021')]","metadata":{"id":"mcfpDT6vNzpT","execution":{"iopub.status.busy":"2022-07-16T18:41:27.707919Z","iopub.execute_input":"2022-07-16T18:41:27.708232Z","iopub.status.idle":"2022-07-16T18:41:27.718647Z","shell.execute_reply.started":"2022-07-16T18:41:27.708197Z","shell.execute_reply":"2022-07-16T18:41:27.717836Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# plot time series for both chosen assets\nf = plt.figure(figsize=(7,8))\n\nax = f.add_subplot(211)\nplt.plot(btc_mini_2021['Close'], label='btc')\nplt.legend()\nplt.xlabel('Time')\nplt.ylabel('Bitcoin Close')\n\nax2 = f.add_subplot(212)\nax2.plot(eth_mini_2021['Close'], color='red', label='eth')\nplt.legend()\nplt.xlabel('Time')\nplt.ylabel('Ethereum Close')\n\nplt.tight_layout()\nplt.show()","metadata":{"id":"EfZiXb74zQsK","outputId":"5039e885-cb14-487f-fe06-3a30e4cf7270","execution":{"iopub.status.busy":"2022-07-16T18:42:33.742963Z","iopub.execute_input":"2022-07-16T18:42:33.743255Z","iopub.status.idle":"2022-07-16T18:42:34.225587Z","shell.execute_reply.started":"2022-07-16T18:42:33.743227Z","shell.execute_reply":"2022-07-16T18:42:34.224751Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"On shorter intervals we can visually see some potential correlation between both assets, with some simultaneous ups and downs. A better format for analyzing such movements is by calculating asset returns. ","metadata":{"id":"O2-jEU3wPnNV"}},{"cell_type":"markdown","source":"## Log returns\n\nIn order to analyze price changes for an asset we can deal with the price difference. However, different assets exhibit different price scales, so that their returns are not readily comparable. We can solve this problem by computing the percentage change in price instead, also known as the return. This return coincides with the percentage change in our invested capital.\n\nReturns are widely used in finance, however log returns are preferred for mathematical modelling of time series, as they are additive across time. Also, while regular returns cannot go below -100%, log returns are not bounded.\n\nTo compute the log return, we can simply take the logarithm of the ratio between two consecutive prices. The first row will have an empty return as the previous value is unknown, therefore the empty return data point will be dropped.","metadata":{"id":"6WpWqQ-EReg2"}},{"cell_type":"code","source":"# define function to compute log returns\ndef log_return(series, periods=1):\n    return np.log(series).diff(periods=periods)","metadata":{"id":"DBg185h-SOwI","execution":{"iopub.status.busy":"2022-07-16T18:44:58.627893Z","iopub.execute_input":"2022-07-16T18:44:58.628416Z","iopub.status.idle":"2022-07-16T18:44:58.633720Z","shell.execute_reply.started":"2022-07-16T18:44:58.628357Z","shell.execute_reply":"2022-07-16T18:44:58.632742Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can visualize the log return for our two assets. See how the signal now looks more like white noise, with less drift than the time series for prices.","metadata":{"id":"pdAR2fjIrCge"}},{"cell_type":"code","source":"import scipy.stats as stats\n\nlret_btc = log_return(btc_mini_2021.Close)[1:]\nlret_eth = log_return(eth_mini_2021.Close)[1:]\nlret_btc.rename('lret_btc', inplace=True)\nlret_eth.rename('lret_eth', inplace=True)\n\nplt.figure(figsize=(8,4))\nplt.plot(lret_btc);\nplt.plot(lret_eth);\nplt.show()","metadata":{"id":"9WYTbi-LrDOM","outputId":"797d73c3-a320-4376-c1ff-4607a1adea29","execution":{"iopub.status.busy":"2022-07-16T18:45:01.323042Z","iopub.execute_input":"2022-07-16T18:45:01.323342Z","iopub.status.idle":"2022-07-16T18:45:02.543114Z","shell.execute_reply.started":"2022-07-16T18:45:01.323302Z","shell.execute_reply":"2022-07-16T18:45:02.542446Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Correlation between assets\n\nWe hypothesized before that crypto asset returns may exhibit some correlation. Let's check this in more detail now.\n\nWe can check how the correlation between Bitcoin and Ethereum change over time for the 2021 period we selected. ","metadata":{"id":"DnAM1L8dj5ir"}},{"cell_type":"code","source":"# join two asset in single DataFrame\n\nlret_btc_long = log_return(btc.Close)[1:]\nlret_eth_long = log_return(eth.Close)[1:]\nlret_btc_long.rename('lret_btc', inplace=True)\nlret_eth_long.rename('lret_eth', inplace=True)\ntwo_assets = pd.concat([lret_btc_long, lret_eth_long], axis=1)\n\n# group consecutive rows and use .corr() for correlation between columns\ncorr_time = two_assets.groupby(two_assets.index//(10000*60)).corr().loc[:,\"lret_btc\"].loc[:,\"lret_eth\"]\n\ncorr_time.plot();\nplt.xticks([])\nplt.ylabel(\"Correlation\")\nplt.title(\"Correlation between BTC and ETH over time\");","metadata":{"id":"pDB4cVz2u_kC","outputId":"7c483562-4cbd-46a1-d16d-94b3bc69d2c3","execution":{"iopub.status.busy":"2022-07-16T18:45:08.922152Z","iopub.execute_input":"2022-07-16T18:45:08.923005Z","iopub.status.idle":"2022-07-16T18:45:09.498568Z","shell.execute_reply.started":"2022-07-16T18:45:08.922961Z","shell.execute_reply":"2022-07-16T18:45:09.497793Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Note the high but variable correlation between the assets. Here we can see that there is some changing dynamics over time, and this would be critical for this time series challenge, that is, how to perform forecasts in a highly non-stationary environment.\n\nA stationary behaviour of a system or a process is characterized by non-changing statistical properties over time such as the mean, variance and autocorrelation. On the other hand, a non-stationary behaviour is characterized by a continuous change of statistical properties over time. Stationarity is important because many useful analytical tools and statistical tests and models rely on it.\n\nWe can also check the correlation between all assets visualizing the correlation matrix. Note how some assets have much higher pairwise correlation than others.","metadata":{"id":"oc7qBUdszXZf"}},{"cell_type":"code","source":"# create dataframe with returns for all assets\nall_assets_2021 = pd.DataFrame([])\nfor asset_id, asset_name in zip(asset_details.Asset_ID, asset_details.Asset_Name):\n  asset = crypto_df[crypto_df[\"Asset_ID\"]==asset_id].set_index(\"timestamp\")\n  asset = asset.loc[totimestamp('01/01/2021'):totimestamp('01/05/2021')]\n  asset = asset.reindex(range(asset.index[0],asset.index[-1]+60,60),method='pad')\n  lret = log_return(asset.Close.fillna(0))[1:]\n  all_assets_2021 = all_assets_2021.join(lret, rsuffix=asset_name, how=\"outer\")","metadata":{"id":"Dzv6CD3nz57z","execution":{"iopub.status.busy":"2022-07-16T18:45:17.832854Z","iopub.execute_input":"2022-07-16T18:45:17.833135Z","iopub.status.idle":"2022-07-16T18:45:22.326387Z","shell.execute_reply.started":"2022-07-16T18:45:17.833093Z","shell.execute_reply":"2022-07-16T18:45:22.325539Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"asset_details.Asset_ID, asset_details.Asset_Name","metadata":{"execution":{"iopub.status.busy":"2022-07-16T18:45:28.946658Z","iopub.execute_input":"2022-07-16T18:45:28.946970Z","iopub.status.idle":"2022-07-16T18:45:28.954322Z","shell.execute_reply.started":"2022-07-16T18:45:28.946939Z","shell.execute_reply":"2022-07-16T18:45:28.953414Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.imshow(all_assets_2021.corr());\nplt.yticks(range(14), asset_details.Asset_Name.values);\nplt.xticks(range(14), asset_details.Asset_Name.values, rotation='vertical');\nplt.colorbar();","metadata":{"execution":{"iopub.status.busy":"2022-07-16T18:45:40.724018Z","iopub.execute_input":"2022-07-16T18:45:40.724662Z","iopub.status.idle":"2022-07-16T18:45:41.222615Z","shell.execute_reply.started":"2022-07-16T18:45:40.724624Z","shell.execute_reply":"2022-07-16T18:45:41.220951Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We encourage participants to perform additional statistical analyses to have a stronger grasp on the dataset, including autocorrelation, time-series decomposition and stationarity tests.","metadata":{}},{"cell_type":"markdown","source":"# Building your prediction model","metadata":{"id":"EaIGlwUB9kPy"}},{"cell_type":"markdown","source":"## Prediction targets and evaluation","metadata":{"id":"9Akm2RTz-rOH"}},{"cell_type":"markdown","source":"This forecasting competition aims to predict returns in the near future for prices $P^a$, for each asset $a$. For each row in the dataset, we include the target for prediction, `Target`. `Target` is derived from log returns ($R^a$) over 15 minutes.\n\n$$R^a(t) = log (P^a(t+16)\\ /\\ P^a(t+1))$$\n\nCrypto asset returns are highly correlated, following to a large extend the overall crypto market. As we want to test your ability to predict returns for individual assets, we perform a linear residualization, removing the market signal from individual asset returns when creating the target. In more detail, if $M(t)$ is the weighted average market returns, the target is:\n\n$$M(t) = \\frac{\\sum_a w^a R^a(t)}{\\sum_a w^a}  \\\\\n\\beta^a = \\frac{\\langle M \\cdot R^a \\rangle}{\\langle M^2 \\rangle} \\\\\n\\text{Target}^a(t) = R^a(t) - \\beta^a M(t)$$\n\nwhere the bracket $\\langle .\\rangle$ represent the rolling average over time (3750 minute windows), and same asset weights $w^a$ used for the evaluation metric.\n\nSome rows have null values for targets due to missing values in future prices. Rows with nulls in the test set ground truth are ignored for scoring purposes.\n\nIn the competition, your predictions will be evaluated on a weighted version of the Pearson correlation coefficient, with weights given by the `Weight` column in the Asset Details file.\n\nIn this tutorial, we will simplify things and use correlation (without weights) for evaluation, and consider only two assets, BTC and ETH.","metadata":{"id":"hl29F0dC9mWc"}},{"cell_type":"markdown","source":"## Feature design\n\nWe first design a few relevant features to input to our model.","metadata":{"id":"Motsk80VF7kj"}},{"cell_type":"code","source":"# Select some input features from the trading data: \n# 5 min log return, abs(5 min log return), upper shadow, and lower shadow.\nupper_shadow = lambda asset: asset.High - np.maximum(asset.Close,asset.Open)\nlower_shadow = lambda asset: np.minimum(asset.Close,asset.Open)- asset.Low\n\nX_btc = pd.concat([log_return(btc.VWAP,periods=5), log_return(btc.VWAP,periods=1).abs(), \n               upper_shadow(btc), lower_shadow(btc)], axis=1)\ny_btc = btc.Target\n\nX_eth = pd.concat([log_return(eth.VWAP,periods=5), log_return(eth.VWAP,periods=1).abs(), \n               upper_shadow(eth), lower_shadow(eth)], axis=1)\ny_eth = eth.Target","metadata":{"id":"2kQz1YDeqsch","execution":{"iopub.status.busy":"2022-07-16T18:47:05.655601Z","iopub.execute_input":"2022-07-16T18:47:05.656587Z","iopub.status.idle":"2022-07-16T18:47:06.005642Z","shell.execute_reply.started":"2022-07-16T18:47:05.656537Z","shell.execute_reply":"2022-07-16T18:47:06.004459Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Preparing the data for building predictive models\n\nAs we will train linear regression parameters, we need to separate training and test sets. To do so, we will compute X and y and split this data into train and test splits. Note that the test split represents a later part of the data, as it is commonly done in time series. ","metadata":{"id":"26locv57nOiP"}},{"cell_type":"code","source":"# select training and test periods\ntrain_window = [totimestamp(\"01/05/2021\"), totimestamp(\"30/05/2021\")]\ntest_window = [totimestamp(\"01/06/2021\"), totimestamp(\"30/06/2021\")]\n\n# divide data into train and test, compute X and y\n# we aim to build simple regression models using a window_size of 1\nX_btc_train = X_btc.loc[train_window[0]:train_window[1]].fillna(0).to_numpy()  # filling NaN's with zeros\ny_btc_train = y_btc.loc[train_window[0]:train_window[1]].fillna(0).to_numpy()  \n\nX_btc_test = X_btc.loc[test_window[0]:test_window[1]].fillna(0).to_numpy() \ny_btc_test = y_btc.loc[test_window[0]:test_window[1]].fillna(0).to_numpy() \n\nX_eth_train = X_eth.loc[train_window[0]:train_window[1]].fillna(0).to_numpy()  \ny_eth_train = y_eth.loc[train_window[0]:train_window[1]].fillna(0).to_numpy()  \n\nX_eth_test = X_eth.loc[test_window[0]:test_window[1]].fillna(0).to_numpy() \ny_eth_test = y_eth.loc[test_window[0]:test_window[1]].fillna(0).to_numpy() ","metadata":{"id":"Ud_lz31L7S1x","execution":{"iopub.status.busy":"2022-07-16T18:47:42.951641Z","iopub.execute_input":"2022-07-16T18:47:42.952209Z","iopub.status.idle":"2022-07-16T18:47:42.966297Z","shell.execute_reply.started":"2022-07-16T18:47:42.952155Z","shell.execute_reply":"2022-07-16T18:47:42.965459Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We now standardize the input data. Standardization is the process of putting different variables on the same scale. In regression analysis, it is often crucial to standardize your independent variables or you may risk obtaining misleading results.","metadata":{"id":"iWGNGmX16l28"}},{"cell_type":"code","source":"from sklearn.preprocessing import StandardScaler\n# simple preprocessing of the data \nscaler = StandardScaler()\n\nX_btc_train_scaled = scaler.fit_transform(X_btc_train)\nX_btc_test_scaled = scaler.transform(X_btc_test)\n\nX_eth_train_scaled = scaler.fit_transform(X_eth_train)\nX_eth_test_scaled = scaler.transform(X_eth_test)","metadata":{"id":"90LRr1HC8vSH","execution":{"iopub.status.busy":"2022-07-16T18:47:52.994093Z","iopub.execute_input":"2022-07-16T18:47:52.994861Z","iopub.status.idle":"2022-07-16T18:47:53.143007Z","shell.execute_reply.started":"2022-07-16T18:47:52.994803Z","shell.execute_reply":"2022-07-16T18:47:53.141933Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Baseline model: Linear Regression\n\nWe will try a simple Linear Regression model on the features we designed. Note that Linear Regression is not commonly used in time series analysis, specially with only one time step! \n\nWe compare two Linear Regression baselines, one that considers each asset independently and one multiple inputs that models all assets together. \n","metadata":{"id":"lWJwz4t0SqiX"}},{"cell_type":"code","source":"from sklearn.linear_model import LinearRegression\n\n# implement basic ML baseline (one per asset)\nlr = LinearRegression()\nlr.fit(X_btc_train_scaled,y_btc_train)\ny_pred_lr_btc = lr.predict(X_btc_test_scaled)\n\nlr.fit(X_eth_train_scaled,y_eth_train)\ny_pred_lr_eth = lr.predict(X_eth_test_scaled)","metadata":{"id":"cc-Ntnit9TH5","execution":{"iopub.status.busy":"2022-07-16T18:47:59.084210Z","iopub.execute_input":"2022-07-16T18:47:59.085026Z","iopub.status.idle":"2022-07-16T18:47:59.249332Z","shell.execute_reply.started":"2022-07-16T18:47:59.084980Z","shell.execute_reply":"2022-07-16T18:47:59.248323Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# implement more complex baseline (multiple input/output regression model)\nfrom sklearn.multioutput import MultiOutputRegressor\n\n# we concatenate X and y for both assets\nX_both_train = np.concatenate((X_btc_train_scaled, X_eth_train_scaled), axis=1)\nX_both_test = np.concatenate((X_btc_test_scaled, X_eth_test_scaled), axis=1)\ny_both_train = np.column_stack((y_btc_train, y_eth_train))\ny_both_test = np.column_stack((y_btc_test, y_eth_test))\n\n# define the direct multioutput model and fit it\nmlr = MultiOutputRegressor(LinearRegression())\nmlr.fit(X_both_train,y_both_train)\ny_pred_lr_both = mlr.predict(X_both_test)","metadata":{"id":"3_RA5yy3-DOi","execution":{"iopub.status.busy":"2022-07-16T18:48:06.788630Z","iopub.execute_input":"2022-07-16T18:48:06.788947Z","iopub.status.idle":"2022-07-16T18:48:06.819862Z","shell.execute_reply.started":"2022-07-16T18:48:06.788911Z","shell.execute_reply":"2022-07-16T18:48:06.818623Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Evaluate baselines\n\nThe competition performance metric is weighted correlation. However, for now we will use simple correlation to evaluate the two baseline models built.","metadata":{"id":"4_LbXT3KEdv5"}},{"cell_type":"code","source":"print('Test score for LR baseline: BTC', f\"{np.corrcoef(y_pred_lr_btc, y_btc_test)[0,1]:.2f}\", \n                                ', ETH', f\"{np.corrcoef(y_pred_lr_eth, y_eth_test)[0,1]:.2f}\")\nprint('Test score for multiple output LR baseline: BTC', f\"{np.corrcoef(y_pred_lr_both[:,0], y_btc_test)[0,1]:.2f}\", \n                                                ', ETH', f\"{np.corrcoef(y_pred_lr_both[:,1], y_eth_test)[0,1]:.2f}\")","metadata":{"id":"UhOs6lITEuXC","outputId":"eead03ce-b36e-474a-a13a-42fffc67eac1","execution":{"iopub.status.busy":"2022-07-16T18:48:11.269596Z","iopub.execute_input":"2022-07-16T18:48:11.270617Z","iopub.status.idle":"2022-07-16T18:48:11.281065Z","shell.execute_reply.started":"2022-07-16T18:48:11.270570Z","shell.execute_reply":"2022-07-16T18:48:11.280398Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can see that, for the training and test periods selected, the multiple asset LR model performs better than simply modelling each asset separately. Note that because the data is highly non-stationary, these results might vary a lot for different periods.","metadata":{"id":"qzPBu6HUrr9-"}},{"cell_type":"markdown","source":"## Submission\n\nNote that this is a Code Competition, in which you must submit your notebook to be run against the hidden private data. Your notebook should use the provided python time-series API, which ensures that models do not peek forward in time. To use the API, follow the instructions and template in [Code Competition Detailed API instructions](https://www.kaggle.com/sohier/detailed-api-introduction) and [Basic Submission Template](https://www.kaggle.com/sohier/basic-submission-template).","metadata":{"id":"alEWivyFSY68"}}]}