{"cells":[{"metadata":{"_uuid":"905a7aaa9f0f5b18d5bf8367f291d86c01070587"},"cell_type":"markdown","source":"### Dask - Why every Data Scientist should use Dask?\n---\n![Dask](https://camo.githubusercontent.com/6bf060c071924b0d6cf6158048f12fde592f7939/68747470733a2f2f692e696d6775722e636f6d2f4d3342527268312e706e67)\n\n---\n#### [Why we talk about dask ?](#01)\n#### [Magic thing about dask ?](#02)\n#### [Dask Introduction](#03)\n\n### Index \n\n1. [**Dask Installation**](#1)\n1. [**Dask Data Science Interface**](#2)\n 1. [**Dataframe : mimics Pandas**](#21)\n 1. [**Talking Data Fraud Detection**](#12)\n 2. [**Microsoft Malware Prediction**](#13)\n 1. [**Array: mimics Numpy**](#22)\n 1. [**Dask ML**](#23)\n 1. [**Example of Dask XGBOOST**](#31)\n 1. [**Training on Large Dataset**](#32)\n 1. [**Voting Classifier**](#33)\n\n---\n\n#### Why we talk about dask ? \n* Dask is essentially the most progressive tool for data processing that I have experienced. On the off chance that you adore Pandas and Numpy however were at times battling with the data that would not fit into RAM then Dask is unquestionably what you require.\n* Dask underpins the Pandas dataframe and Numpy exhibit data structures and can either be kept running on your local computer or be scaled up to keep running on a cluster. Basically you compose code once and after that decide to either run it locally or convey to a multi-hub cluster utilizing a simply typical Pythonic syntax structure. This is an incredible element in itself, yet it isn't the reason I am composing this notebook and saying that each data Scientist (in any event the one's utilizing Python) should utilize Dask.\n\n---\n\n#### Magic thing about dask ? \n* Dask is popularly known as a ‘parallel computing’ python library that has been designed to run across multiple systems. Your next question would understandably be – what is parallel computing?\n* As in our example of separating the balls, 10 people doing the job simultaneously can be considered analogous to parallel computation. In technical terms, parallel computation is performing multiple tasks (or computations) simultaneously, using more than one resource.\n\n---\n\n### Dask Introduction \nReference : [**From official Dask site**](http://docs.dask.org/en)\n\n***Dask is composed of two parts:***\n\n1. **Dynamic task scheduling** optimized for computation. This is similar to Airflow, Luigi, Celery, or Make, but optimized for interactive computational workloads.\n2. **“Big Data” collections** like ***parallel arrays, dataframes, and lists*** that extend common interfaces like **NumPy, Pandas, or Python iterators to larger-than-memory** or distributed environments. These parallel collections run on top of dynamic task schedulers.\n\n\n### Dask emphasizes the following virtues:\n\n* **Familiar**: Provides parallelized NumPy array and Pandas DataFrame objects\n* **Flexible**: Provides a task scheduling interface for more custom workloads and integration with other projects.\n* **Native**: Enables distributed computing in pure Python with access to the PyData stack.\n* **Fast**: Operates with low overhead, low latency, and minimal serialization necessary for fast numerical algorithms\n* **Scales up**: Runs resiliently on clusters with 1000s of cores\n* **Scales down**: Trivial to set up and run on a laptop in a single process\n* **Responsive**: Designed with interactive computing in mind, it provides rapid feedback and diagnostics to aid humans\n\n![](http://docs.dask.org/en/latest/_images/collections-schedulers.png)\n\n---\n\n### Example to understand working of Dask \nReference : [Example](https://www.analyticsvidhya.com/blog/2018/08/dask-big-datasets-machine_learning-python/)\n* Suppose you have 4 balls (of different colors) and you are asked to separate them within an hour (based on the color) into different buckets. you can do it immidiately. but what if it's thousands of balls with different color and give you task to seperate it what would u do that?\n\n![](https://s3-ap-south-1.amazonaws.com/av-blog-media/wp-content/uploads/2018/07/image-21-300x169.jpg)\n\nQuestion : ***What if it's thousands of balls with different color and give you task to seperate it what would u do that?*** and ***What if you are given a hundred balls and you have to separate them in an hour’s time?*** \nAnswer : **The best bet would be to ask a few other people for help.** You can call 9 other friends, give each of them 100 balls and ask them to separate these based on the color. In this case, 10 people are simultaneously working on the assigned task and together would be able to complete it faster than a single person would have (here you had a huge amount of data which you distributed among a bunch of people).\n\n---\n\n### Famous Presentations talk On Dask\n\n* **SciPy 2018, July 2018**\n * [Scalable Machine Learning with Dask (30 minutes)](https://www.youtube.com/watch?v=ccfsbuqsjgI)\n* **PyCon 2018, May 2018**\n * [Democratizing Distributed Computing with Dask and JupyterHub (32 minutes)](https://www.youtube.com/watch?v=Iq72dt1gO9c)\n* **AMS & ESIP, January 2018**\n * [Pangeo quick demo: Dask, XArray, Zarr on the cloud with JupyterHub (3 minutes)](https://www.youtube.com/watch?v=rSOJKbfNBNk)\n * [Pangeo talk: An open-source big data science platform with Dask, XArray, Zarr on the cloud with JupyterHub (43 minutes)](https://www.youtube.com/watch?v=mDrjGxaXQT4)\n* **PYCON.DE 2017, November 2017**\n * [Dask: Parallelism in Python (1 hour, 2 minutes)](https://www.youtube.com/watch?v=rZlshXJydgQ)\n* **PYCON 2017, May 2017**\n * [Dask: A Pythonic Distributed Data Science Framework (46 minutes)](https://www.youtube.com/watch?v=RA_2qdipVng)\n* **PLOTCON 2016, December 2016**\n * [Visualizing Distributed Computations with Dask and Bokeh (33 minutes)](https://www.youtube.com/watch?v=FTJwDeXkggU)\n* **PyData DC, October 2016**\n * [Using Dask for Parallel Computing in Python (44 minutes)](https://www.youtube.com/watch?v=s4ChP7tc3tA)\n* **SciPy 2016, July 2016**\n * [Dask Parallel and Distributed Computing (28 minutes)](https://www.youtube.com/watch?v=PAGjm4BMKlk)\n* **PyData NYC, December 2015**\n * [Dask Parallelizing NumPy and Pandas through Task Scheduling (33 minutes)](https://www.youtube.com/watch?v=mHd8AI8GQhQ)\n* **PyData Seattle, August 2015**\n * [Dask: out of core arrays with task scheduling (1 hour, 50 minutes)](https://www.youtube.com/watch?v=ieW3G7ZzRZ0)\n* **SciPy 2015, July 2015**\n * [Dask Out of core NumPy:Pandas through Task Scheduling (16 minutes)](https://www.youtube.com/watch?v=1kkFZ4P-XHg)\n---\n### 1.Dask Installation \n[**Go to top**](#99)\n\n1. **Conda Installation** `conda install dask`\n2. **pip Installation** `pip install “dask[complete]”`\n3. **Source**\n` \n#clone git\ngit clone https://github.com/dask/dask.git\ncd dask\npython setup.py install\n`\n![](https://www.kaggle.com/ashishpatel26/dask-u/downloads/installation.JPG)\n\n---\n\n### 2.Dask Data Science Interface\n\n---\n[**Go to top**](#99)\n### 2.1.**Dataframe : mimics Pandas** \n* A DataFrame Dask is a large parallel DataFrame that consists of many smaller pandas DataFrames across the index. These pandas data frames can be stored on disk for a computation that is larger than the memory of a single machine or on many different machines in a cluster. A Dask DataFrame operation triggers many operations on the Pandas DataFrame components.\n\n![image.png](https://s3-ap-south-1.amazonaws.com/av-blog-media/wp-content/uploads/2018/08/dask-df-768x569.png)\n\n![](https://www.kaggle.com/ashishpatel26/dask-u/downloads/dataframe.JPG)\n\n### Useful Basic operations\n\n* **Trivially parallelizable operations (fast):**\n \n * Element-wise operations: **`df.x + df.y`, `df * df`**\n * Row-wise selections: **`df[df.x > 0]`**\n * Loc: **`df.loc[4.0:10.5]`**\n * Common aggregations: **`df.x.max()`, `df.max()`**\n * Is in: **`df[df.x.isin([1, 2, 3])]`**\n * Date time/string accessors: **`df.timestamp.month`**\n \n* **Cleverly parallelizable operations (fast):**\n \n * groupby-aggregate (with common aggregations): **`df.groupby(df.x).y.max()`, `df.groupby('x').max()`**\n * groupby-apply on index: **`df.groupby(['idx', 'x']).apply(myfunc)`, where `idx` is the index level name**\n * value\\_counts: **`df.x.value_counts()`**\n * Drop duplicates: **`df.x.drop_duplicates()`**\n * Join on index: **`dd.merge(df1, df2, left_index=True, right_index=True)`** or **`dd.merge(df1, df2, on=['idx', 'x'])`** where **`idx`** is the index name for both **`df1` and `df2`**\n * Join with Pandas DataFrames: **`dd.merge(df1, df2, on='id')`**\n * Element-wise operations with different partitions / divisions: **`df1.x + df2.y`**\n * Date time resampling: **`df.resample(...)`**\n * Rolling averages: **`df.rolling(...)`**\n * Pearson’s correlation: **`df[['col1', 'col2']].corr()`**\n \n* **Operations requiring a shuffle (slow-ish, unless on index)**\n \n * Set index: **`df.set_index(df.x)`**\n * groupby-apply not on index (with anything): **`df.groupby(df.x).apply(myfunc)`**\n * Join not on the index: **`dd.merge(df1, df2, on='name')`**\n\n## Two Example \n[**Go to top**](#99)\n\n1. [**Talking Data Fraud Detection**](#12)\n2. [**Microsoft Malware Prediction**](#13)"},{"metadata":{"_kg_hide-input":true,"trusted":true,"_uuid":"8e878d3faf47c346101a8b5d2018f3ebb475dc79"},"cell_type":"code","source":"from IPython.display import HTML\n#Define the javascript function and HTML to produce the show/hide code button\n# text = str('''\n# \n#
''')\n\n# HTML(text)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4dd9dd836b0ef56e68705baf1e2d466c5316ef1d","_kg_hide-input":true},"cell_type":"code","source":"HTML(\"\")","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true,"_kg_hide-output":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\nimport dask\nimport dask.dataframe as dd\nimport os\nprint(os.listdir(\"../input\"))\n\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n%matplotlib inline\nplt.style.use('fivethirtyeight')\n\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"01519e9577324a97ab173d0b95df56c2a97f8c5a"},"cell_type":"markdown","source":"## Talking Data Fraud Detection \n[**Go to top**](#99)\n\nTime to read dataframe: \n\n* **Dask - 179 ms**\n* **Pandas - my result kernel crash in kaggle**\n\n#### 1. Read data"},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"%%time\n# Loading in the train data\ndtypes = {'ip':'uint32',\n 'app': 'uint16',\n 'device': 'uint16',\n 'os': 'uint16',\n 'channel': 'uint16',\n 'is_attributed': 'uint8'}\n\ntrain = dd.read_csv(\"../input/talkingdata-adtracking-fraud-detection/train.csv\", dtype=dtypes, parse_dates=['click_time', 'attributed_time'])","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"9c584a543b1f09485c08a1c617c281569c7c9b02"},"cell_type":"markdown","source":"#### 2. Display data"},{"metadata":{"trusted":true,"_uuid":"3b4446ff0574924a1c48a83f39fa92d159a0652a"},"cell_type":"code","source":"train.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3fd456c0cf38e23824d13b21403b719ec21298cd"},"cell_type":"markdown","source":"#### 3.Information about dataframe"},{"metadata":{"trusted":true,"_uuid":"10447095082ca219ebe5aea59ff6f54b00ec8d52"},"cell_type":"code","source":"train.info()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7d9ccfe67492aef67468e0229be7a3818e0a58a7"},"cell_type":"markdown","source":"#### 4. Length of dataframe"},{"metadata":{"trusted":true,"_uuid":"1ec3f8643fa83e844079a33b951b5ac2badce5ac"},"cell_type":"code","source":"len(train)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"7f7471341b19905613edb44947a103c346e4e698"},"cell_type":"code","source":"train.describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d5bbfb9ab39496cb6c03a687f011fbe1d287f966"},"cell_type":"markdown","source":"### 5.Check datatype"},{"metadata":{"trusted":true,"_uuid":"93addde36dd3c106ffd43d64f6a125d41045d8aa"},"cell_type":"code","source":"train.dtypes","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"d4b81af90ab01d763be52ed1be7bc24e2a0212ab"},"cell_type":"code","source":"train.dtypes.value_counts().plot(kind=\"pie\", figsize = (10,10))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"603222f19ea28720c4942b4c03f5fe3567da13ac"},"cell_type":"markdown","source":"## Microsoft Malware Prediction \n[**Go to top**](#99)\n\nTime to read dataframe: \n\n* **Dask - 419 ms**\n* **Pandas - my result kernel crash in kaggle**"},{"metadata":{"_uuid":"a8ae3f80dfb475687e1fbfb05610009b17ac061c"},"cell_type":"markdown","source":"#### 1.Read data"},{"metadata":{"_kg_hide-input":true,"_kg_hide-output":true,"trusted":true,"_uuid":"00630017bf184c02ae29540435620525233e2cb2"},"cell_type":"code","source":"dtypes1 = {\n 'MachineIdentifier': 'category',\n 'ProductName': 'category',\n 'EngineVersion': 'category',\n 'AppVersion': 'category',\n 'AvSigVersion': 'category',\n 'IsBeta': 'int8',\n 'RtpStateBitfield': 'float16',\n 'IsSxsPassiveMode': 'int8',\n 'DefaultBrowsersIdentifier': 'float16',\n 'AVProductStatesIdentifier': 'float32',\n 'AVProductsInstalled': 'float16',\n 'AVProductsEnabled': 'float16',\n 'HasTpm': 'int8',\n 'CountryIdentifier': 'int16',\n 'CityIdentifier': 'float32',\n 'OrganizationIdentifier': 'float16',\n 'GeoNameIdentifier': 'float16',\n 'LocaleEnglishNameIdentifier': 'int8',\n 'Platform': 'category',\n 'Processor': 'category',\n 'OsVer': 'category',\n 'OsBuild': 'int16',\n 'OsSuite': 'int16',\n 'OsPlatformSubRelease': 'category',\n 'OsBuildLab': 'category',\n 'SkuEdition': 'category',\n 'IsProtected': 'float16',\n 'AutoSampleOptIn': 'int8',\n 'PuaMode': 'category',\n 'SMode': 'float16',\n 'IeVerIdentifier': 'float16',\n 'SmartScreen': 'category',\n 'Firewall': 'float16',\n 'UacLuaenable': 'float32',\n 'Census_MDC2FormFactor': 'category',\n 'Census_DeviceFamily': 'category',\n 'Census_OEMNameIdentifier': 'float16',\n 'Census_OEMModelIdentifier': 'float32',\n 'Census_ProcessorCoreCount': 'float16',\n 'Census_ProcessorManufacturerIdentifier': 'float16',\n 'Census_ProcessorModelIdentifier': 'float16',\n 'Census_ProcessorClass': 'category',\n 'Census_PrimaryDiskTotalCapacity': 'float32',\n 'Census_PrimaryDiskTypeName': 'category',\n 'Census_SystemVolumeTotalCapacity': 'float32',\n 'Census_HasOpticalDiskDrive': 'int8',\n 'Census_TotalPhysicalRAM': 'float32',\n 'Census_ChassisTypeName': 'category',\n 'Census_InternalPrimaryDiagonalDisplaySizeInInches': 'float16',\n 'Census_InternalPrimaryDisplayResolutionHorizontal': 'float16',\n 'Census_InternalPrimaryDisplayResolutionVertical': 'float16',\n 'Census_PowerPlatformRoleName': 'category',\n 'Census_InternalBatteryType': 'category',\n 'Census_InternalBatteryNumberOfCharges': 'float32',\n 'Census_OSVersion': 'category',\n 'Census_OSArchitecture': 'category',\n 'Census_OSBranch': 'category',\n 'Census_OSBuildNumber': 'int16',\n 'Census_OSBuildRevision': 'int32',\n 'Census_OSEdition': 'category',\n 'Census_OSSkuName': 'category',\n 'Census_OSInstallTypeName': 'category',\n 'Census_OSInstallLanguageIdentifier': 'float16',\n 'Census_OSUILocaleIdentifier': 'int16',\n 'Census_OSWUAutoUpdateOptionsName': 'category',\n 'Census_IsPortableOperatingSystem': 'int8',\n 'Census_GenuineStateName': 'category',\n 'Census_ActivationChannel': 'category',\n 'Census_IsFlightingInternal': 'float16',\n 'Census_IsFlightsDisabled': 'float16',\n 'Census_FlightRing': 'category',\n 'Census_ThresholdOptIn': 'float16',\n 'Census_FirmwareManufacturerIdentifier': 'float16',\n 'Census_FirmwareVersionIdentifier': 'float32',\n 'Census_IsSecureBootEnabled': 'int8',\n 'Census_IsWIMBootEnabled': 'float16',\n 'Census_IsVirtualDevice': 'float16',\n 'Census_IsTouchEnabled': 'int8',\n 'Census_IsPenCapable': 'int8',\n 'Census_IsAlwaysOnAlwaysConnectedCapable': 'float16',\n 'Wdft_IsGamer': 'float16',\n 'Wdft_RegionIdentifier': 'float16',\n 'HasDetections': 'int8'\n }","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1bbc118ed3960ac33382a63ff465dc8d6ab69942"},"cell_type":"code","source":"%%time\ntrain_df = dd.read_csv(\"../input/microsoft-malware-prediction/train.csv\", dtype = dtypes1)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"0ada180777b34b5cb2369475c2869381320155c0"},"cell_type":"markdown","source":"#### 2. Display header values"},{"metadata":{"trusted":true,"_uuid":"302bc1b6cb60e9efb6f9c4c921ffd6a6e4d5326c"},"cell_type":"code","source":"train_df.head()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"470bb35b7b3bf9004793c3391d534de04720b748"},"cell_type":"markdown","source":"#### 3.Information about dataframe"},{"metadata":{"trusted":true,"_uuid":"ed962fe311e3ea238473534c461eccb169ee76d1"},"cell_type":"code","source":"train_df.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"78a067bf13eb934eeafdfe7e6a40aa129dda10f5"},"cell_type":"code","source":"train_df.describe()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b049839ab50679f58717c2edf63b5db2b31f9ca8"},"cell_type":"markdown","source":"#### 4.Aggregation operation"},{"metadata":{"trusted":true,"_uuid":"550b00c313721e26f740b723d5d84e4b71dd6f6a"},"cell_type":"code","source":"train_df.groupby(train_df.Census_ChassisTypeName).Census_InternalPrimaryDiagonalDisplaySizeInInches.mean().compute()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"61a45866e256ca79bc7c9474ee0ca206fea3e429"},"cell_type":"code","source":"len(train_df)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"cc3603153bd9ad3f24eabdf994f1689937f8ae79"},"cell_type":"markdown","source":"#### 5. Data type Count plot"},{"metadata":{"trusted":true,"_uuid":"761c7a85b519be1bf37d11dc4a1aea566478b74c"},"cell_type":"code","source":"train_df.dtypes.value_counts().plot(kind=\"pie\", figsize = (10,10))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"22307fc1c3fcb96cab1a8ad122b0e8dc3a55b6a4"},"cell_type":"markdown","source":"#### 6.Reset Index"},{"metadata":{"trusted":true,"_uuid":"91be95d11e6c155f48334cf710fd011630d4fad0"},"cell_type":"code","source":"train_df=train_df.reset_index()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b53dbc1ca9dbdf2773f3b00e51d7f8ae1dad8bfe"},"cell_type":"markdown","source":"#### 7. Visualize Data\n\nNote : ***Double click on image to see the zoom version this image***"},{"metadata":{"trusted":true,"_uuid":"a23a1bc96eb9171a42dcc483d320840eb45afaf8"},"cell_type":"code","source":"train_df.Census_SystemVolumeTotalCapacity.max().visualize()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"50e8e3f1d4a5a0fa3deecd7fc57e1abcca936c5b"},"cell_type":"markdown","source":"### 2.2.**Array: mimics Numpy**\n---\n[**Go to top**](#99)\n* Dask Array implements a subset of the NumPy ndarray interface using blocked algorithms, cutting up the large array into many small arrays. This lets us compute on arrays larger than memory using all of our cores. We coordinate these blocked algorithms using Dask graphs.\n\n\n![](https://s3-ap-south-1.amazonaws.com/av-blog-media/wp-content/uploads/2018/07/image-4-e1532888794915.png)\n\n* Dask arrays coordinate many NumPy arrays arranged into a grid. These NumPy arrays may live on disk or on other machines.\n\n![](//github.com/dask/dask-tutorial/raw/33efceb9ba76b16ce3bf51d6210546e257f0874b/images/array.png) Dask array provides a parallel, larger-than-memory, n-dimensional array using blocked algorithms. Simply put: distributed Numpy.\n\n* **Parallel**: Uses all of the cores on your computer\n* **Larger-than-memory**: Lets you work on datasets that are larger than your available memory by breaking up your array into many small pieces, operating on those pieces in an order that minimizes the memory footprint of your computation, and effectively streaming data from disk.\n* **Blocked Algorithms**: Perform large computations by performing many smaller computations\n\n**Related Documentation**\n\n* [Documentation](http://dask.readthedocs.io/en/latest/array.html)\n* [API reference](http://dask.readthedocs.io/en/latest/array-api.html)\n\n![](https://www.kaggle.com/ashishpatel26/dask-u/downloads/array.JPG)\n\n* Dask Array is used in fields like atmospheric and oceanographic science, large scale imaging, genomics, numerical algorithms for optimization or statistics, and more.\n\n\nUse Basic operation of Dask Array\n---------------------------------------------\n\nDask arrays support most of the NumPy interface like the following:\n\n* Arithmetic and scalar mathematics: **`+, *, exp, log, ...`**\n* Reductions along axes: **`sum(), mean(), std(), sum(axis=0), ...`**\n* Tensor contractions / dot products / matrix multiply: **`tensordot`**\n* Axis reordering / transpose: **`transpose`**\n* Slicing: **`x[:100, 500:100:-2]`**\n* Fancy indexing along single axes with lists or NumPy arrays: **`x[:, [10, 1, 5]]`**\n* Array protocols like **`__array__` and `__array_ufunc__`**\n* Some linear algebra: **`svd, qr, solve, solve_triangular, lstsq`**\n* …\n\nHowever, Dask Array does not implement the entire NumPy interface. Users expecting this will be disappointed. Notably, Dask Array lacks the following features:\n\n* Much of **`np.linalg`** has not been implemented. This has been done by a number of excellent **BLAS/LAPACK** implementations, and is the focus of numerous ongoing academic research projects\n* Arrays with unknown shapes do not support all operations\n* Operations like **`sort`** which are notoriously difficult to do in parallel, and are of somewhat diminished value on very large data (you rarely actually need a full sort). Often we include parallel-friendly alternatives like **topk`**\n* Dask Array doesn’t implement operations like **`tolist`** that would be very inefficient for larger datasets. Likewise, it is very inefficient to iterate over a Dask array with for loops\n* Dask development is driven by immediate need, hence many lesser used functions have not been implemented. Community contributions are encouraged"},{"metadata":{"_uuid":"448552bf7d1f48ad6f63bad9e4e335b4cee6cae5"},"cell_type":"markdown","source":"### Create Random array using dask"},{"metadata":{"trusted":true,"_uuid":"87697fce250e59ac3dc5de5ac821c79455865b60"},"cell_type":"code","source":"import dask.array as da \n\n#using arange to create an array with values from 0 to 10\nX = da.arange(20, chunks=5)\nX.compute() ","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8477f1c9d6b4e81802d907bde5ef2afb6eff88fa"},"cell_type":"code","source":"#to see size of each chunk\nX.chunks","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"eaa84a5e1cd867b49db76ac3ba380d53f8247e8d"},"cell_type":"code","source":"result = X.sum()\nresult.compute()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7e421588ec917671dc4a0357be9f5733f6836526"},"cell_type":"markdown","source":"### Convert a numpy array to Dask array"},{"metadata":{"trusted":true,"_uuid":"e746731c55f2a745219c77826378f20a4a46aeea"},"cell_type":"code","source":"import numpy as np\nimport dask.array as da\nx = np.arange(100)\ny = da.from_array(x, chunks=5)\ny.compute() #results in a dask array","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7a1e88ff215a233deb22e86b6259fbfdf5a4fc1f"},"cell_type":"markdown","source":"### Calculating mean of the first 100 numbers"},{"metadata":{"trusted":true,"_uuid":"a3fc38fe256a659ecce7fe7a19c60b65c7b011c6"},"cell_type":"code","source":"x = np.arange(1000) #arange is used to create array on values from 0 to 1000\ny = da.from_array(x, chunks=(100)) #converting numpy array to dask array\n\ny.mean().compute() #computing mean of the array","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3dc8a7fe75e4add8a82d81b0593bdf106caf37d7"},"cell_type":"markdown","source":"### Slicing\n\n\n#### Dask.array supports most of the NumPy slicing syntax. In particular it supports the following:\n\n* Slicing by integers and slices **`x[0, :5]`**\n* Slicing by lists/arrays of integers **`x[[1, 2, 4]]`**\n* Slicing by lists/arrays of booleans **`x[[False, True, True, False, True]]`**\n\nIt does not currently support the following:\n\n* Slicing one `dask.array` with another **`x[x > 0]`**\n* Slicing with lists in multiple axes **`x[[1, 2, 3], [3, 2, 1]]`**\n\nBoth of these are straighforward to add though. If you have a use case then raise an issue."},{"metadata":{"trusted":true,"_uuid":"dfb323a26afd4137e21a4288bf03661825f5a66a"},"cell_type":"code","source":"x = da.ones((10000, 10000), chunks = 5000)\nda.exp(x)[:5000, :5000].compute()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"fdf2de18f741c035ca9675aa4aa77b781ff5df72"},"cell_type":"code","source":"y = x + x.T\nz = y[::2, 5000:].mean(axis=1)\nz.compute()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"2f7d0aa9b956270ed96fddd26b0f0a9301717a11"},"cell_type":"markdown","source":"### Persist data in memory\n* If you have the available RAM for your dataset then you can persist data in memory.\n* This allows future computations to be much faster."},{"metadata":{"trusted":true,"_uuid":"2a62b46f5710f2e15185e4775de948f52313f479"},"cell_type":"code","source":"y = y.persist()\n%time y[0, 0].compute()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"2561972939805fcf59e4b19cb8cf7b2f03516446"},"cell_type":"code","source":"%time z.sum().compute()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"594ca0cf1dcf15321894b51f6e5d547e6b1e0f6d"},"cell_type":"code","source":"x[[1,4,7]].compute()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"89ccc1af274c1afbffd812c2f44a0dabc720e246"},"cell_type":"markdown","source":"Stack and Concatenate\n======\n[**Go to top**](#99)\n* Often we have many arrays stored on disk that we want to stack together and think of as one large array. This is common with geospatial data in which we might have many HDF5/NetCDF files on disk, one for every day, but we want to do operations that span multiple days.\n\n\n### Stack\n\nWe stack many existing dask Arrays into a new array, creating a new dimension as we go."},{"metadata":{"trusted":true,"_uuid":"46bba8884f01767a742d4e05a8ac999b95beb50f"},"cell_type":"code","source":"import dask.array as da\ndata = [da.from_array(np.ones((4, 4)), chunks=(2, 2)) for i in range(3)] # A small stack of dask arrays","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"1df4fb6cf6d73ef64d9264e690a542e16e64af02"},"cell_type":"code","source":"x = da.stack(data, axis=0)\nx.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"85ade5806c3677aac75c90d7d62ce008e0692844"},"cell_type":"code","source":"da.stack(data, axis=1).shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"4b051e5e5b0da75ead86f5ee7cefb3f01333eb37"},"cell_type":"code","source":"da.stack(data, axis=-1).shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"6d5c83f8c7d7f8f4c3f9bfd7f6def3a9f15ea2f9"},"cell_type":"markdown","source":"This creates a new dimension with length equal to the number of slices\n\n### Concatenate\n\nWe concatenate existing arrays into a new array, extending them along an existing dimension"},{"metadata":{"trusted":true,"_uuid":"a000f70ec40d4a63849894afb1c586ea031f4c5e"},"cell_type":"code","source":"import dask.array as da\nimport numpy as np\n\ndata = [da.from_array(np.ones((4, 4)), chunks=(2, 2)) for i in range(3)] # small stack of dask arrays","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"8148d9d0906ae3aef089ddc03e1ef6127ee337ef"},"cell_type":"code","source":"x = da.concatenate(data, axis=0)\nx.shape","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"ba5036a1730eb7d0a278d99b4b003e01cde83e37"},"cell_type":"code","source":"da.concatenate(data, axis=1).shape","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"14c5fe20b81f629e8bf821f4e4135ac4541cea48"},"cell_type":"markdown","source":"### 2.3. Dask ML \n[**Go to top**](#99)\n\n* **Dask ML** provides scalable Python **machine learning algorithms** that are compatible with Scikit-Learn. First let's understand how Scikit-Learn deals with calculations, and then let's see how Dask performs these operations differently.\n\n![](https://s3-ap-south-1.amazonaws.com/av-blog-media/wp-content/uploads/2018/07/sklearn-11.png)\n\n* A user can do parallel computing with scikit-learn (on a single machine) by setting the parameter **`njobs = -1.`** Scikit-learn uses Joblib to perform these parallel calculations. **Joblib is a library in Python that provides support for parallelization.** When you call the **.fit () function** , which is based on the tasks to be performed (whether hyperparameters search or model matching), Joblib distributes the task to the available cores.\n\n* Although parallel calculations can be done with scikit-learn, they can not be resized on multiple machines. On the other hand, **Dask works well on a single machine and can even be resized to a group of machines.**\n\n![](https://s3-ap-south-1.amazonaws.com/av-blog-media/wp-content/uploads/2018/07/dask-1.png)\n\n* Dask has a central **task scheduler** and a **group of workers.** The scheduler assigns tasks to the workers. Each worker is assigned a number of cores in which he can perform calculations. \n**Workers provide two functions:**\n 1. The calculation tasks assigned by the programmer \n 2. deliver the results to other worker on request\n \n * Here's an example that explains what a conversation between a programmer and the workers looks like (by Matthew Dask, a developer of Dask):\n\n* The central task scheduler sends jobs (Python functions) to different work processes, either on the same machine or in a cluster:\n\n Worker A, compute x = f (1), worker B, y = g (2)\n Worker A, if g (2) is executed, get y from Worker B and compute z = h (x, y).\n This should give you a clear idea of how Dask works. Now the models of machine learning and the CV of the Dask search are discussed!\n \n\n### 3.1 Example of Dask XGBOOST \n[**Go to top**](#99)\n\n
\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
train(client, params, data, labels[, …])Train an XGBoost model on a Dask Cluster
predict(client, model, data)Distributed prediction with XGBoost
XGBClassifier([max_depth, learning_rate, …])\n\n\n\n\n\n\n
Attributes:
\n
XGBRegressor([max_depth, learning_rate, …])\n\n\n\n\n\n\n
Attributes:
\n
"},{"metadata":{"_uuid":"63f30ad7e54d7d65e6d92508f9a68bb3a2383d68","trusted":true},"cell_type":"code","source":"## dask_ml is different package from dask so you have to install this package\n## pip install dask_ml\nimport dask_ml","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4785ece249d6eec89bb17e219c84271cd9444b7a"},"cell_type":"markdown","source":"### XGB Setup\n* Dask and XGBoost can work together to train gradient boosted trees in parallel. This notebook shows how to use Dask and XGBoost together.\n\n* XGBoost provides a powerful prediction framework, and it works well in practice. It wins Kaggle contests and is popular in industry because it has good performance and can be easily interpreted (i.e., it’s easy to find the important features from a XGBoost model)."},{"metadata":{"trusted":true,"_uuid":"d324cbe8d5e9ffb3f265d77f75cac4c25080601b"},"cell_type":"code","source":"from dask.distributed import Client\n\nclient = Client(n_workers=4, threads_per_worker=1)\nclient","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"61be2bafd0faf35e786ddf8b49356da56d7d90f1"},"cell_type":"markdown","source":"### Create data\nFirst we create a bunch of synthetic data, with 100,000 examples and 20 features."},{"metadata":{"trusted":true,"_uuid":"3a2eef9eb865d3add5007aff5878ca70859ce9ac"},"cell_type":"code","source":"from dask_ml.datasets import make_classification\n\nX, y = make_classification(n_samples=100000, n_features=20,\n chunks=1000, n_informative=4,\n random_state=0)\nX","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"b48f26b2e2d2257b9469c8b671976fc9de28329b"},"cell_type":"markdown","source":"Dask-XGBoost works with both arrays and dataframes. For more information on creating dask arrays and dataframes from real data, see documentation on Dask arrays or Dask dataframes.\n\n### Split data for training and testing\nWe split our dataset into training and testing data to aid evaluation by making sure we have a fair test:"},{"metadata":{"trusted":true,"_uuid":"c5e36a4c507c43c38386a8462548d7cd8ae8f128"},"cell_type":"code","source":"from dask_ml.model_selection import train_test_split\n\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.15)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d22d32b129f5c743aed17b8180f0895b5e91c95b"},"cell_type":"markdown","source":"### Train Dask XGBOOST\n\ndask-xgboost is a small wrapper around xgboost. Dask sets XGBoost up, gives XGBoost data and lets XGBoost do it’s training in the background using all the workers Dask has available.\n\nLet’s do some training:"},{"metadata":{"trusted":true,"_uuid":"c68b849169b8e639aa6e04373613959da9d77fcd"},"cell_type":"code","source":"import dask_xgboost\nimport xgboost","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"a14eda76b0813babf2b12dd3afb6da37994444ea"},"cell_type":"code","source":"%%time\nparams = {'objective': 'binary:logistic',\n 'max_depth': 4, 'eta': 0.01, 'subsample': 0.5,\n 'min_child_weight': 0.5}\n\nbst = dask_xgboost.train(client, params, X_train, y_train, num_boost_round=10)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"3a0ab1a04b03d13bea21bd59940838486e81815e"},"cell_type":"markdown","source":"### Visualize results\nThe bst object is a regular xgboost.Booster object."},{"metadata":{"trusted":true,"_uuid":"f28220e899d052e3e02ca862ed975cbb40bd25b4"},"cell_type":"code","source":"bst","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"c549534c63628e9471f22a1e0c71407154d53a7d"},"cell_type":"markdown","source":"#### This means all the methods mentioned in the XGBoost documentation are available. We show two examples to expand on this, but these examples are of XGBoost instead of Dask."},{"metadata":{"trusted":true,"_uuid":"294247eb861c6865ff4b43f96a9327635a164458"},"cell_type":"code","source":"%matplotlib inline\nimport matplotlib.pyplot as plt\n\nax = xgboost.plot_importance(bst, height=0.8, max_num_features=9)\nax.grid(False, axis=\"y\")\nax.set_title('Estimated feature importance')\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7d96d045ffb43cbbe334ce83451a8301f03fe9d8"},"cell_type":"markdown","source":"We specified that only 4 features were informative while creating our data, and only 3 features show up as important."},{"metadata":{"_uuid":"b9e538cbbe12a1aef38b48ae5e374b28b0fd8663"},"cell_type":"markdown","source":"### Plot the Receiver Operating Characteristic curve"},{"metadata":{"trusted":true,"_uuid":"d7b1fb8343a3b1905a596b956f59bfac780de568"},"cell_type":"code","source":"y_hat = dask_xgboost.predict(client, bst, X_test).persist()\ny_hat","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"eb2e73794c5e6a35905c42df3d10290bd76c923f"},"cell_type":"code","source":"from sklearn.metrics import roc_curve\nfpr, tpr, _ = roc_curve(y_test, y_hat)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"99904bd904b625fe7dd1ca262263bbc4390c6c86"},"cell_type":"code","source":"from sklearn.metrics import auc\n\nfig, ax = plt.subplots(figsize=(12, 8))\nax.plot(fpr, tpr, lw=3,\n label='ROC Curve (area = {:.2f})'.format(auc(fpr, tpr)))\nax.plot([0, 1], [0, 1], 'k--', lw=2)\nax.set(\n xlim=(0, 1),\n ylim=(0, 1),\n title=\"ROC Curve\",\n xlabel=\"False Positive Rate\",\n ylabel=\"True Positive Rate\",\n)\nax.legend();\nplt.show()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7c6394ba6388ec8ac94977f4896cdcc3276be534"},"cell_type":"markdown","source":"### Reimplement Algorithms with Dask Array\n\n* For simple machine learning algorithms which use Numpy arrays, Dask ML re-implements these algorithms. Dask replaces numpy arrays with Dask arrays to achieve scalable algorithms. This has been implemented for:\n\n * Linear models (linear regression, logistic regression, poisson regression)\n * Pre-processing (scalers , transforms)\n * Clustering (k-means, spectral clustering)"},{"metadata":{"_uuid":"49aeb82e04949fee1a2f376721b54c6c33b73826"},"cell_type":"markdown","source":"### A. Linear model example\n\n from dask_ml.linear_model import LogisticRegression\n\n model = LogisticRegression()\n model.fit(data, labels)\n \n### B. Pre-processing example\n\n from dask_ml.preprocessing import OneHotEncoder\n\n encoder = OneHotEncoder(sparse=True)\n result = encoder.fit(data)\n\n### C. Clustering example\n\n from dask_ml.cluster import KMeans\n model = KMeans()\n model.fit(data)"},{"metadata":{"_uuid":"f71db988c8e10786889e8ec4a8673b97a32a8a67"},"cell_type":"markdown","source":"### 3.2. Train Models on Large Datasets \n[**Go to top**](#99)\n\nMost Scikit-Learn estimators are designed to work with NumPy arrays or sparse Scipy matrices. These data structures must fit into RAM on a single machine.\n\nEstimates implemented in Dask-ML work well with Dask arrays and DataFrames. This can be much larger than the memory of a single machine. They can be distributed on a cluster of machines in memory."},{"metadata":{"trusted":true,"_uuid":"d2018ae11c47081fa738ecb24c6ea3bfe5dd05b0"},"cell_type":"code","source":"%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"73fede4555e5f827795db06855f52119f9d0b338"},"cell_type":"code","source":"# from dask.distributed import Client\n\n# # Scale up: connect to your own cluster with more resources\n# # see http://dask.pydata.org/en/latest/setup.html\n# client = Client(processes=False, threads_per_worker=4,\n# n_workers=1, memory_limit='2GB')\n# client","execution_count":null,"outputs":[]},{"metadata":{"_kg_hide-output":true,"trusted":true,"_uuid":"e13a78d089bdf3583204a15fd0d961fa59a424ab"},"cell_type":"code","source":"import dask_ml.datasets\nimport dask_ml.cluster\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"7464d2a7fb812d717315efa524190fb8616f437c"},"cell_type":"markdown","source":"In this example, we’ll use `dask_ml.datasets.make_blobs` to generate some random dask arrays."},{"metadata":{"_kg_hide-output":false,"_kg_hide-input":false,"trusted":true,"_uuid":"87aeee36bc0bbe42cabb7742ce9ad47d50b26319"},"cell_type":"code","source":"# Scale up: increase n_samples or n_features\nX, y = dask_ml.datasets.make_blobs(n_samples=1000000,\n chunks=100000,\n random_state=0,\n centers=3)\nX = X.persist()\nX","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"a6aa73e7ce70803e111bf7f6cb980d874e5f0690"},"cell_type":"markdown","source":"We’ll use the k-means implemented in Dask-ML to cluster the points. It uses the k-means|| (read: **“k-means parallel”**) initialization algorithm, which scales better than **k-means++**. All of the computation, both during and after initialization, can be done in parallel."},{"metadata":{"trusted":true,"_uuid":"3afa218bace7c121949f9ab583f8d783c37151d2"},"cell_type":"code","source":"km = dask_ml.cluster.KMeans(n_clusters=3, init_max_iter=2, oversampling_factor=10)\nkm.fit(X)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5eb222492261ebc15b1b43620602af6ae3a4597f"},"cell_type":"code","source":"fig, ax = plt.subplots()\nax.scatter(X[::1000, 0], X[::1000, 1], marker='.', c=km.labels_[::1000],\n cmap='viridis', alpha=0.25);","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"aa65bd80eac0cb078fd4c8c1e5a2ce6105c2ceaa"},"cell_type":"markdown","source":"### For More update - [DASK ML API DOCUMENT](http://ml.dask.org/modules/api.html)"},{"metadata":{"_uuid":"e92da64d2cb34472141054ca3c622d15c96dada4"},"cell_type":"markdown","source":"### 3.3 Voting Classifier \n[**Go to Top**](#99)\n\nA [Voting classifier](http://scikit-learn.org/stable/modules/ensemble.html#voting-classifier) model combines multiple different models (i.e., sub-estimators) into a single model, which is (ideally) stronger than any of the individual models alone.\n\n[Dask](http://ml.dask.org/joblib.html) provides the software to train individual sub-estimators on different machines in a cluster. This enables users to train more models in parallel than would have been possible on a single machine. Note that users will only observe this benefit if they have a distributed cluster with more resources than their single machine (because sklearn already enables users to parallelize training across cores on a single machine)."},{"metadata":{"_kg_hide-output":true,"trusted":true,"_uuid":"f37340b6842256a69eaa0a0b122be08bc1c4db26"},"cell_type":"code","source":"from sklearn.ensemble import VotingClassifier\nfrom sklearn.linear_model import SGDClassifier\nfrom sklearn.linear_model import LogisticRegression\nfrom sklearn.svm import SVC\n\nimport sklearn.datasets","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"37c11bb2784783d3daa1f7f0dad872b2ad288edb"},"cell_type":"markdown","source":"We create a synthetic dataset (with 1000 rows and 20 columns) that we can give to the voting classifier model."},{"metadata":{"trusted":true,"_uuid":"785aa77b46d58f67649f4fbef099c6bf18e11002"},"cell_type":"code","source":"X, y = sklearn.datasets.make_classification(n_samples=1_000, n_features=20)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"5b15da66f19562e8380cfd37a9f69e7c10436f42"},"cell_type":"code","source":"classifiers = [\n ('sgd', SGDClassifier(max_iter=1000)),\n ('logisticregression', LogisticRegression()),\n ('svc', SVC(gamma='auto')),\n]\nclf = VotingClassifier(classifiers, n_jobs=-1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"6b4386da54915a6384e11886b9707c35bd761768"},"cell_type":"code","source":"%time clf.fit(X, y)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_uuid":"94d5bf1375d7693bc53f18580dd38b361d55a7e4"},"cell_type":"code","source":"from sklearn.externals import joblib\n# from distributed import Client\n\n# client = Client()\n# client","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"98bd210e97dea3d4974bfae393450e725f73a927"},"cell_type":"markdown","source":"To train the **`voting classifier`**, we call the classifier’s fit method, but enclosed in joblib’s parallel_backend context manager. This distributes training of sub-estimators acoss the cluster."},{"metadata":{"trusted":true,"_uuid":"b5b23093d5dacd097ba367678d51e9f6c73b99bc"},"cell_type":"code","source":"%%time\nwith joblib.parallel_backend(\"dask\"):\n clf.fit(X, y)\n\nprint(clf)","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"4cf02516b816b3e062f1bfc56ef5a87f2ecf318a"},"cell_type":"markdown","source":"### References : \n[**Go to top**](#99)\n1. **https://docs.dask.org/**\n2. **https://www.analyticsvidhya.com/blog/2018/08/dask-big-datasets-machine_learning-python/**\n3. **https://towardsdatascience.com/why-every-data-scientist-should-use-dask-81b2b850e15b**\n4. **https://towardsdatascience.com/trying-out-dask-dataframes-in-python-for-fast-data-analysis-in-parallel-aa960c18a915**\n5. **https://www.kdnuggets.com/2016/09/introducing-dask-parallel-programming.html**"},{"metadata":{"_uuid":"628ead0ff7219cce5a4dae2b6309a8c9f8d8c7dc"},"cell_type":"markdown","source":"## to be continue for further updates...!!!"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"name":"python","version":"3.6.6","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"}},"nbformat":4,"nbformat_minor":1}