{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# AMEX Export to Parquet with Apache Spark\n\n\"clipart\"\nClipart Designed by Freepik\n\n### Hi! 👋\n\nWe will be using [Apache Spark](https://spark.apache.org) to export CSV files to a convenient Parquet format.\n\n*The output of this code* will be a directory with Parquet files - they can be read by a) *pandas* and b) *Apache Spark*.\n\nThis notebook will show:\n\n- importing CSV, exporting Parquet\n- **an overview of column types** in AMEX dataset\n- description of some Apache Spark characteristics on a true dataset\n\nThis code will:\n\n1. Import raw CSV data\n2. Preprocess data\n3. Export to Parquet\n4. Perform some trivial EDA\n\nResulting Parquet dataset will be published as **a Kaggle dataset**, so you can use it in your notebooks. 👍","metadata":{}},{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-06-01T15:18:13.007809Z","iopub.execute_input":"2022-06-01T15:18:13.00896Z","iopub.status.idle":"2022-06-01T15:18:13.019241Z","shell.execute_reply.started":"2022-06-01T15:18:13.008904Z","shell.execute_reply":"2022-06-01T15:18:13.01809Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Installing Spark\n\nSpark can be pip installed.","metadata":{}},{"cell_type":"code","source":"!pip install pyspark==3.2.1","metadata":{"_kg_hide-output":true,"execution":{"iopub.status.busy":"2022-06-01T15:18:15.519014Z","iopub.execute_input":"2022-06-01T15:18:15.51981Z","iopub.status.idle":"2022-06-01T15:19:08.722988Z","shell.execute_reply.started":"2022-06-01T15:18:15.519759Z","shell.execute_reply":"2022-06-01T15:19:08.721321Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pyspark\nimport pyspark.sql.functions as F\n\npyspark.__version__","metadata":{"execution":{"iopub.status.busy":"2022-06-01T15:27:43.954646Z","iopub.execute_input":"2022-06-01T15:27:43.955665Z","iopub.status.idle":"2022-06-01T15:27:43.962237Z","shell.execute_reply.started":"2022-06-01T15:27:43.955623Z","shell.execute_reply":"2022-06-01T15:27:43.961308Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from pyspark.sql import SparkSession\n\nspark = SparkSession.builder\\\n .master(\"local[*]\")\\\n .config(\"spark.driver.memory\", \"14g\")\\\n .getOrCreate()\n\nspark.conf.set(\"spark.sql.shuffle.partitions\", 20) # it isn't a cluster, so 20 is enough\n\nspark","metadata":{"execution":{"iopub.status.busy":"2022-06-01T15:19:08.788295Z","iopub.execute_input":"2022-06-01T15:19:08.78862Z","iopub.status.idle":"2022-06-01T15:19:15.863089Z","shell.execute_reply.started":"2022-06-01T15:19:08.788592Z","shell.execute_reply":"2022-06-01T15:19:15.861908Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 0. Save train_labels to Parquet\n\nAs a quick warmup, let's export `train_labels.csv` to Parquet! 🎣\n\nI've choosen the smallest adequate column types - to save space.","metadata":{}},{"cell_type":"code","source":"import pyspark.sql.types\n\n# column types\nschema = pyspark.sql.types.StructType([\n pyspark.sql.types.StructField('customer_ID', pyspark.sql.types.StringType()),\n pyspark.sql.types.StructField('target', pyspark.sql.types.ByteType())\n])\n\ndf = (\n spark.read\n .schema(schema)\n .option(\"header\", True)\n .option(\"mode\", \"FAILFAST\")\n .csv(\"/kaggle/input/amex-default-prediction/train_labels.csv\")\n)\n\n# interpreting 0 and 1 as false and true\ndf = df.withColumn(\n 'target',\n F.col('target').cast(pyspark.sql.types.BooleanType())\n # F was earlier imported with \"import ... as F\"\n)\n\ndf.show()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T12:57:51.505429Z","iopub.execute_input":"2022-06-01T12:57:51.505904Z","iopub.status.idle":"2022-06-01T12:57:56.518381Z","shell.execute_reply.started":"2022-06-01T12:57:51.505856Z","shell.execute_reply":"2022-06-01T12:57:56.517398Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# coalesce makes sure, that output is in 1 file\ndf.coalesce(1).write.parquet('amexparq/train_labels')","metadata":{"execution":{"iopub.status.busy":"2022-06-01T12:57:56.519574Z","iopub.execute_input":"2022-06-01T12:57:56.520018Z","iopub.status.idle":"2022-06-01T12:57:59.96397Z","shell.execute_reply.started":"2022-06-01T12:57:56.519977Z","shell.execute_reply":"2022-06-01T12:57:59.961791Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ls -lh amexparq/train_labels","metadata":{"execution":{"iopub.status.busy":"2022-06-01T12:57:59.965255Z","iopub.execute_input":"2022-06-01T12:57:59.965695Z","iopub.status.idle":"2022-06-01T12:58:00.73288Z","shell.execute_reply.started":"2022-06-01T12:57:59.965661Z","shell.execute_reply":"2022-06-01T12:58:00.731806Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 1. Import raw CSV data\n\nNow we will get our hands dirty with training/test data!\n\nThe first step is to load raw CSV data.\n\n### Column types 🎈\n\nWe have 5 types of columns:\n\n||category|column names|description|example value|\n|-|-|-|-|-|\n|1.|Customer ID|`customer_ID`|string with hex digits|`'0000099d6bd597052cdcda90ffabf56573fe9d7c79be5fbac11a8ed792feb62a'`|\n|2.|Datetime|`S_2`|*YYYY-MM-DD* datetime|`2017-03-09`|\n|3.|Categorical columns, which are already numbers|`B_38`, `D_116`, ...|small integer represented as a float number|`2.0`|\n|4.|Categorical columns, which have string labels|`D_63`, `D_64`|string labels from a fixed set|`'CR'`, `'CO'`|\n|5.|Numerical columns|`R_1`, `D_58`, ...|float|`0.00922`|\n\n\nEach of them must be properly handled. These columns can have *nulls*.","metadata":{}},{"cell_type":"code","source":"# this cell is hidden\n# because this list is quite long... 😕\n\nall_columns = [\n 'customer_ID',\n 'S_2',\n 'P_2',\n 'D_39',\n 'B_1',\n 'B_2',\n 'R_1',\n 'S_3',\n 'D_41',\n 'B_3',\n 'D_42',\n 'D_43',\n 'D_44',\n 'B_4',\n 'D_45',\n 'B_5',\n 'R_2',\n 'D_46',\n 'D_47',\n 'D_48',\n 'D_49',\n 'B_6',\n 'B_7',\n 'B_8',\n 'D_50',\n 'D_51',\n 'B_9',\n 'R_3',\n 'D_52',\n 'P_3',\n 'B_10',\n 'D_53',\n 'S_5',\n 'B_11',\n 'S_6',\n 'D_54',\n 'R_4',\n 'S_7',\n 'B_12',\n 'S_8',\n 'D_55',\n 'D_56',\n 'B_13',\n 'R_5',\n 'D_58',\n 'S_9',\n 'B_14',\n 'D_59',\n 'D_60',\n 'D_61',\n 'B_15',\n 'S_11',\n 'D_62',\n 'D_63',\n 'D_64',\n 'D_65',\n 'B_16',\n 'B_17',\n 'B_18',\n 'B_19',\n 'D_66',\n 'B_20',\n 'D_68',\n 'S_12',\n 'R_6',\n 'S_13',\n 'B_21',\n 'D_69',\n 'B_22',\n 'D_70',\n 'D_71',\n 'D_72',\n 'S_15',\n 'B_23',\n 'D_73',\n 'P_4',\n 'D_74',\n 'D_75',\n 'D_76',\n 'B_24',\n 'R_7',\n 'D_77',\n 'B_25',\n 'B_26',\n 'D_78',\n 'D_79',\n 'R_8',\n 'R_9',\n 'S_16',\n 'D_80',\n 'R_10',\n 'R_11',\n 'B_27',\n 'D_81',\n 'D_82',\n 'S_17',\n 'R_12',\n 'B_28',\n 'R_13',\n 'D_83',\n 'R_14',\n 'R_15',\n 'D_84',\n 'R_16',\n 'B_29',\n 'B_30',\n 'S_18',\n 'D_86',\n 'D_87',\n 'R_17',\n 'R_18',\n 'D_88',\n 'B_31',\n 'S_19',\n 'R_19',\n 'B_32',\n 'S_20',\n 'R_20',\n 'R_21',\n 'B_33',\n 'D_89',\n 'R_22',\n 'R_23',\n 'D_91',\n 'D_92',\n 'D_93',\n 'D_94',\n 'R_24',\n 'R_25',\n 'D_96',\n 'S_22',\n 'S_23',\n 'S_24',\n 'S_25',\n 'S_26',\n 'D_102',\n 'D_103',\n 'D_104',\n 'D_105',\n 'D_106',\n 'D_107',\n 'B_36',\n 'B_37',\n 'R_26',\n 'R_27',\n 'B_38',\n 'D_108',\n 'D_109',\n 'D_110',\n 'D_111',\n 'B_39',\n 'D_112',\n 'B_40',\n 'S_27',\n 'D_113',\n 'D_114',\n 'D_115',\n 'D_116',\n 'D_117',\n 'D_118',\n 'D_119',\n 'D_120',\n 'D_121',\n 'D_122',\n 'D_123',\n 'D_124',\n 'D_125',\n 'D_126',\n 'D_127',\n 'D_128',\n 'D_129',\n 'B_41',\n 'B_42',\n 'D_130',\n 'D_131',\n 'D_132',\n 'D_133',\n 'R_28',\n 'D_134',\n 'D_135',\n 'D_136',\n 'D_137',\n 'D_138',\n 'D_139',\n 'D_140',\n 'D_141',\n 'D_142',\n 'D_143',\n 'D_144',\n 'D_145',\n]","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-06-01T15:19:15.867082Z","iopub.execute_input":"2022-06-01T15:19:15.867618Z","iopub.status.idle":"2022-06-01T15:19:15.890907Z","shell.execute_reply.started":"2022-06-01T15:19:15.867565Z","shell.execute_reply":"2022-06-01T15:19:15.890008Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# schema definition (column types)\n\nimport pyspark.sql.types\n\n# list \"all_columns\" is defined in a hidden cell above\nprint(f\"len(all_columns)={len(all_columns)}\")\n\ncategorical_columns = {\n 'B_30', 'B_38', 'D_114', 'D_116', 'D_117', 'D_120', 'D_126', 'D_63', 'D_64', 'D_66', 'D_68',\n}\ncustom_label_columns = {'D_64', 'D_63'} # these columns have string labels instead of integers\n\ndef get_type(col_name):\n \"\"\"Get appriopriate type for given column.\"\"\"\n \n # see https://spark.apache.org/docs/latest/sql-ref-datatypes.html\n\n if col_name == 'customer_ID':\n return pyspark.sql.types.StringType()\n elif col_name in categorical_columns:\n if col_name in custom_label_columns:\n # these columns have string labels instead of integers\n # we will later replace them with numbers\n return pyspark.sql.types.StringType()\n else:\n # these were written as floats, like \"1.0\"\n # we will later cast them to bytes\n return pyspark.sql.types.FloatType()\n elif col_name == 'S_2':\n # YYYY-MM-DD date type\n return pyspark.sql.types.DateType()\n else:\n return pyspark.sql.types.FloatType()\n\nschema = pyspark.sql.types.StructType(\n [pyspark.sql.types.StructField(c, get_type(c)) for c in all_columns]\n)\n","metadata":{"execution":{"iopub.status.busy":"2022-06-01T15:19:15.8931Z","iopub.execute_input":"2022-06-01T15:19:15.894191Z","iopub.status.idle":"2022-06-01T15:19:15.913198Z","shell.execute_reply.started":"2022-06-01T15:19:15.894141Z","shell.execute_reply":"2022-06-01T15:19:15.912101Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pyspark.sql.functions as F\n\n# limit lines for debugging purposes\nlimit = None\n\ndef load_raw_data(path):\n # see https://spark.apache.org/docs/3.2.0/sql-data-sources-csv.html\n df = (\n spark.read\n .schema(schema)\n .option(\"header\", True)\n .option(\"mode\", \"FAILFAST\")\n .csv(path)\n )\n\n if limit is not None:\n df = df.limit(limit)\n \n return df\n\n# notice, that in Spark dataframes are lazy-evaluated!\ntrain_df = load_raw_data(\"/kaggle/input/amex-default-prediction/train_data.csv\")\ntest_df = load_raw_data(\"/kaggle/input/amex-default-prediction/test_data.csv\")\n\ntrain_df","metadata":{"execution":{"iopub.status.busy":"2022-06-01T15:26:14.649652Z","iopub.execute_input":"2022-06-01T15:26:14.650107Z","iopub.status.idle":"2022-06-01T15:26:14.915766Z","shell.execute_reply.started":"2022-06-01T15:26:14.650073Z","shell.execute_reply":"2022-06-01T15:26:14.914859Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 2. Preprocess data\n\nThere are a few problems with the raw data.\n\n------\n\na) We had to parse some categorical columns as floats, because they were written with comma: `1.0`, `2.0`.\n\nByte would be better, because float has 4 bytes - 4x more! 😲\n\n------\n\nb) Columns `D_63` and `D_64` have strings instead of integers.\n\nTo speed up analytics and conserve memory, we will assign to each category a number.\n\nWe will use [StringIndexer](https://spark.apache.org/docs/latest/ml-features#stringindexer).\nIt's similar to [LabelEncoder from sklearn](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html).\n\n------\n\nNow we will solve these problems. 💫\n\nNotice, how schema changes after each step.","metadata":{}},{"cell_type":"code","source":"def cast_categoricals_to_byte(df): \n for c in categorical_columns - custom_label_columns:\n # these were written as floats, like \"1.0\"\n # however, they are in reality only small bytes\n df = df.withColumn(c, F.col(c).cast(pyspark.sql.types.ByteType()))\n \n return df\n\ntrain_df = cast_categoricals_to_byte(train_df)\ntest_df = cast_categoricals_to_byte(test_df)\n\ntrain_df","metadata":{"execution":{"iopub.status.busy":"2022-06-01T15:26:17.352084Z","iopub.execute_input":"2022-06-01T15:26:17.352618Z","iopub.status.idle":"2022-06-01T15:26:18.25672Z","shell.execute_reply.started":"2022-06-01T15:26:17.352569Z","shell.execute_reply":"2022-06-01T15:26:18.255787Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# columns 'D_64' and 'D_63' have string labels instead of integers\n# we replace them with integers\n\nfrom pyspark.ml.feature import StringIndexerModel\n\n# I've manually fitted StringIndexer.\n# This indexer is there recreated with `from_arrays_of_labels` function\nstring_indexer = StringIndexerModel.from_arrays_of_labels(\n [['CL', 'CO', 'CR', 'XL', 'XM', 'XZ'], ['-1', 'O', 'R', 'U']],\n inputCols=['D_63', 'D_64'],\n outputCols=['D_63_index', 'D_64_index'],\n handleInvalid=\"keep\", # nulls and labels not in the set\n)\n\ndef index_labels(df):\n df = string_indexer.transform(df)\n \n for c in custom_label_columns: \n index_col = F.col(c + '_index')\n \n # give null on the output, when original input was also null\n index_col = F.when(F.col(c).isNull(), None).otherwise(index_col)\n \n # indexer returns floats, we want bytes\n index_col = index_col.cast(pyspark.sql.types.ByteType())\n \n df = df.withColumn(c + '_index', index_col)\n df = df.drop(c)\n \n return df\n\ntrain_df = index_labels(train_df)\ntest_df = index_labels(test_df)\n\ntrain_df","metadata":{"execution":{"iopub.status.busy":"2022-06-01T15:26:19.240325Z","iopub.execute_input":"2022-06-01T15:26:19.24072Z","iopub.status.idle":"2022-06-01T15:26:20.134528Z","shell.execute_reply.started":"2022-06-01T15:26:19.240688Z","shell.execute_reply":"2022-06-01T15:26:20.133461Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 3. Export to Parquet\n\nNow we will save the data to Parquet.\n\n### Lazy evaluation\n\nUp to this point *no computation was done*.\nSpark was only building a query plan with *tranformations*, which will be run by an *action* - write to Parquet!\n\n### Partitioning\n\nWe are using option `repartition`, which will cause the result of the computation to be written to many files.\n\nSpark will make sure, that all records of a given customer will be in one partition.\n\nTherefore, you can load in Pandas only one Parquet file - and be sure, that you see all\ninteractions of a given customer.\n\nNotice, that each Parquet file will contain many customers.\n\nThere is [a blog post about repartitioning](https://medium.com/@vladimir.prus/spark-partitioning-the-fine-print-5ee02e7cb40b).","metadata":{}},{"cell_type":"code","source":"%%time\n\ndef write_parquet(df, name, partitions):\n (\n df\n .repartition(partitions, \"customer_ID\")\n .sortWithinPartitions(\"customer_ID\", \"S_2\")\n .write\n .parquet(\"amexparq/\" + name)\n )\n \nwrite_parquet(train_df, \"train\", 20)\nwrite_parquet(test_df, \"test\", 40)","metadata":{"execution":{"iopub.status.busy":"2022-06-01T15:26:22.446595Z","iopub.execute_input":"2022-06-01T15:26:22.447028Z","iopub.status.idle":"2022-06-01T15:26:54.298735Z","shell.execute_reply.started":"2022-06-01T15:26:22.446992Z","shell.execute_reply":"2022-06-01T15:26:54.297638Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!ls /kaggle/working/amexparq/train -lh","metadata":{"execution":{"iopub.status.busy":"2022-06-01T15:26:54.300527Z","iopub.execute_input":"2022-06-01T15:26:54.301056Z","iopub.status.idle":"2022-06-01T15:26:55.102988Z","shell.execute_reply.started":"2022-06-01T15:26:54.301007Z","shell.execute_reply":"2022-06-01T15:26:55.102025Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!ls /kaggle/working/amexparq/test -lh","metadata":{"execution":{"iopub.status.busy":"2022-06-01T12:59:00.868268Z","iopub.execute_input":"2022-06-01T12:59:00.869594Z","iopub.status.idle":"2022-06-01T12:59:01.647362Z","shell.execute_reply.started":"2022-06-01T12:59:00.869552Z","shell.execute_reply":"2022-06-01T12:59:01.646266Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"!du -sh /kaggle/working/amexparq/*","metadata":{"execution":{"iopub.status.busy":"2022-06-01T15:26:55.104893Z","iopub.execute_input":"2022-06-01T15:26:55.10529Z","iopub.status.idle":"2022-06-01T15:26:55.880406Z","shell.execute_reply.started":"2022-06-01T15:26:55.105241Z","shell.execute_reply":"2022-06-01T15:26:55.879224Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## 4. Perform some trivial EDA 🔎\n\nTo wheat your appetite, we will do some basic EDA. You can **use it/fork it** as a basis of *your EDA*! \n\nIt turns out, that Spark has an optional **pandas-like API**.\n\nFirst, we read our *fresh parquet files* for an analysis:","metadata":{}},{"cell_type":"code","source":"import pyspark.pandas as ps # enable pandas-like API in PySpark\n\ntrain_df = spark.read.parquet(\"/kaggle/working/amexparq/train\")\ntest_df = spark.read.parquet(\"/kaggle/working/amexparq/test\")\ntrain_labels_df = spark.read.parquet(\"/kaggle/working/amexparq/train_labels\").cache()\n\nps.set_option('compute.default_index_type', 'distributed')\n# magic to speedup pandas-API in Spark\n# see https://databricks.com/blog/2020/08/11/interoperability-between-koalas-and-apache-spark.html","metadata":{"execution":{"iopub.status.busy":"2022-06-01T15:26:55.883359Z","iopub.execute_input":"2022-06-01T15:26:55.884499Z","iopub.status.idle":"2022-06-01T15:26:56.620526Z","shell.execute_reply.started":"2022-06-01T15:26:55.884432Z","shell.execute_reply":"2022-06-01T15:26:56.618629Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### About Parquet/Spark performance\n\nSpark has a few optimizing techniques.\n\nIf your query reads only column `D_63_index`, then other columns will\nbe skipped during Parquet file reading - reducing I/O!\n\nIn the next query we calculate, how often given `D_63_index` was used.\n\nNotice, that in the physical plan we have a line:\n\n```\n +- FileScan parquet [D_63_index#7492] Batched: ...,\n```\n\nThe array of columns to read is `[D_63_index#7492]`.\n\nSpark will read only `D_63_index` column!","metadata":{}},{"cell_type":"code","source":"(\n train_df\n .select('D_63_index')\n .groupby('D_63_index')\n .count()\n).explain()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T13:07:57.287858Z","iopub.execute_input":"2022-06-01T13:07:57.289206Z","iopub.status.idle":"2022-06-01T13:07:57.420651Z","shell.execute_reply.started":"2022-06-01T13:07:57.289162Z","shell.execute_reply":"2022-06-01T13:07:57.41988Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"(\n train_df\n .select('D_63_index')\n .groupby('D_63_index')\n .count()\n).show()","metadata":{"execution":{"iopub.status.busy":"2022-06-01T15:41:07.514409Z","iopub.execute_input":"2022-06-01T15:41:07.515151Z","iopub.status.idle":"2022-06-01T15:41:07.723264Z","shell.execute_reply.started":"2022-06-01T15:41:07.515107Z","shell.execute_reply":"2022-06-01T15:41:07.721359Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Let's use pandas-API to do EDA!","metadata":{}},{"cell_type":"code","source":"%%time\n\n# train dataframe with pandas-API\ntrain_pf = train_df.to_pandas_on_spark()\n\ntrain_pf.agg(['min', 'max'])","metadata":{"execution":{"iopub.status.busy":"2022-06-01T15:40:17.813547Z","iopub.execute_input":"2022-06-01T15:40:17.814576Z","iopub.status.idle":"2022-06-01T15:40:22.858167Z","shell.execute_reply.started":"2022-06-01T15:40:17.814531Z","shell.execute_reply":"2022-06-01T15:40:22.857027Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Histogram of `D_63_index`**","metadata":{}},{"cell_type":"code","source":"%%time\ntrain_pf['D_63_index'].plot.hist(bins=np.arange(7) - 0.5)","metadata":{"execution":{"iopub.status.busy":"2022-06-01T15:32:44.375645Z","iopub.execute_input":"2022-06-01T15:32:44.377166Z","iopub.status.idle":"2022-06-01T15:32:44.928868Z","shell.execute_reply.started":"2022-06-01T15:32:44.377110Z","shell.execute_reply":"2022-06-01T15:32:44.927942Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Histogram of user interaction number**","metadata":{}},{"cell_type":"code","source":"%%time\ntrain_pf['customer_ID'].value_counts().plot.hist(bins=np.arange(100) - 0.5)","metadata":{"execution":{"iopub.status.busy":"2022-06-01T15:32:51.109908Z","iopub.execute_input":"2022-06-01T15:32:51.110773Z","iopub.status.idle":"2022-06-01T15:32:51.875322Z","shell.execute_reply.started":"2022-06-01T15:32:51.110713Z","shell.execute_reply":"2022-06-01T15:32:51.873932Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"These plots are **interactive** - use mouse to zoom in the previous one!","metadata":{}},{"cell_type":"markdown","source":"## Conclusion\n\nI hope you've enjoyed the notebook.\n\nResulting Parquet dataset will be published as **a Kaggle dataset**, so you can use it in your notebooks. 👍\n\n## Questions? Improvement ideas?\n\nPlease comment! I will try to answer. 😁\n","metadata":{}}]}