{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Agenda:\n1. [Introduction](#1)\n - 1.1 [Problem Statement](#2)\n - 1.2 [Columns Description](#3)\n - 1.3 [Challenges](#4)\n2. [Data Preparation & Cleaning](#5)\n - 2.1 [Packages & Helping Functions](#6)\n - 2.2 [Data Loading & Cleaning](#7) \n3. [General Exploration](#8)\n4. [Answering The Proposed Questions](#9)\n5. [Conclusion](#10)","metadata":{"papermill":{"duration":0.100538,"end_time":"2022-07-17T13:54:02.470231","exception":false,"start_time":"2022-07-17T13:54:02.369693","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"

1. Introduction

","metadata":{"papermill":{"duration":0.098543,"end_time":"2022-07-17T13:54:02.667989","exception":false,"start_time":"2022-07-17T13:54:02.569446","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"### 1.1 Problem Statement \nVisit problem statment subsection in the [discribtion](https://www.kaggle.com/competitions/learnplatform-covid19-impact-on-digital-learning/overview/description) of the dataset. ","metadata":{"papermill":{"duration":0.097672,"end_time":"2022-07-17T13:54:02.863305","exception":false,"start_time":"2022-07-17T13:54:02.765633","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"**Below are some examples of questions that relate to the problem statement:**\n\n - What is the picture of digital connectivity and engagement in 2020?\n - What is the effect of the COVID-19 pandemic on online and distance learning, and how might this also evolve in the future?\n - How does student engagement with different types of education technology change over the course of the pandemic?\n - How does student engagement with online learning platforms relate to different geography? Demographic context (e.g., race/ethnicity, ESL, learning disability)? Learning context? Socioeconomic status?\n - Do certain state interventions, practices or policies (e.g., stimulus, reopening, eviction moratorium) correlate with the increase or decrease online engagement?","metadata":{"papermill":{"duration":0.099619,"end_time":"2022-07-17T13:54:03.061422","exception":false,"start_time":"2022-07-17T13:54:02.961803","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"

2. Data Preparation & Cleaning

","metadata":{"papermill":{"duration":0.101963,"end_time":"2022-07-17T13:54:03.261510","exception":false,"start_time":"2022-07-17T13:54:03.159547","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"2.1 Important Packages & Some Helping Functions ","metadata":{"papermill":{"duration":0.096177,"end_time":"2022-07-17T13:54:03.455696","exception":false,"start_time":"2022-07-17T13:54:03.359519","status":"completed"},"tags":[]}},{"cell_type":"code","source":"import os \n\n# For Loading and Manipulating data\nimport pandas as pd\nimport numpy as np\n\n# For visualization purposes\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nimport missingno as msno # Visualizing Missingness.\nimport matplotlib.patches as mpatches\n\n%matplotlib inline\n\n# for coloring the printed output\nfrom termcolor import colored","metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","papermill":{"duration":1.209286,"end_time":"2022-07-17T13:54:04.786184","exception":false,"start_time":"2022-07-17T13:54:03.576898","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:34:31.095866Z","iopub.execute_input":"2022-08-14T16:34:31.096339Z","iopub.status.idle":"2022-08-14T16:34:31.721531Z","shell.execute_reply.started":"2022-08-14T16:34:31.096242Z","shell.execute_reply":"2022-08-14T16:34:31.720674Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def get_missingness(dataframe):\n total_missingness = dataframe.isna().sum().sort_values(ascending= False)\n #Exclude columns containing no missingness.\n #total_missingness = total_missingness[total_missingness.values !=0]\n\n #Get the percentage of missingess per column.\n percentage= np.round(total_missingness* 100/ len(dataframe), 2)\n per = []\n [per.append('{:.1f} %'.format(p)) for p in percentage]\n df = pd.DataFrame({\"Missing count\": total_missingness, \n \"Missing percentage\": per })\n #df.assign(Percentage = percentage)\n return df","metadata":{"papermill":{"duration":0.106579,"end_time":"2022-07-17T13:54:04.991514","exception":false,"start_time":"2022-07-17T13:54:04.884935","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:37:50.563470Z","iopub.execute_input":"2022-08-14T16:37:50.564365Z","iopub.status.idle":"2022-08-14T16:37:50.571154Z","shell.execute_reply.started":"2022-08-14T16:37:50.564313Z","shell.execute_reply":"2022-08-14T16:37:50.570070Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def show_countplot(dataframe, x, y, hue=None, order=None, title='Title', x_rotation=0):\n ax = sns.countplot(data=dataframe, x=x, y=y, order=order, hue=hue, color='#4d83de')\n \n plt.title(title, fontsize=20, color='brown')\n \n plt.xlabel(x.title() if x else 'Count', fontsize=15)\n plt.xticks(rotation=x_rotation, fontsize=12)\n plt.ylabel(y.title() if y else 'Count', fontsize=15)\n plt.yticks(fontsize=12)\n \n total = dataframe.shape[0]\n for patch in ax.patches:\n loc = patch.get_x() if x else patch.get_y()\n width = patch.get_width()\n height = patch.get_height()\n \n # text location\n loc_x = (loc+width/2) if x else (width*0.5)\n loc_y = (height*0.5) if x else (loc+height/2)\n \n # text\n percent = (height*100/total) if x else (width*100/total)\n \n ax.text(loc_x, loc_y, f'{percent:.2f}%', ha='center', weight='bold', fontsize=10, color='white')","metadata":{"papermill":{"duration":0.111796,"end_time":"2022-07-17T13:54:05.204183","exception":false,"start_time":"2022-07-17T13:54:05.092387","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:34:46.582811Z","iopub.execute_input":"2022-08-14T16:34:46.583246Z","iopub.status.idle":"2022-08-14T16:34:46.593790Z","shell.execute_reply.started":"2022-08-14T16:34:46.583208Z","shell.execute_reply":"2022-08-14T16:34:46.592442Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def fill_it(product_name, sector, PEF):\n global products_info\n products_info.loc[products_info['Product Name']==product_name, 'Sector(s)'] = sector\n products_info.loc[products_info['Product Name']==product_name, 'Primary Essential Function'] = PEF\n\ndef remove_it(product_name):\n global products_info\n products_info = products_info[products_info['Product Name']!=product_name].copy()\n \ndef fill_data(data):\n\n data.dropna(subset=['lp_id'], inplace=True)\n data.dropna(subset=['pct_access', 'engagement_index'], how='all', inplace=True)\n data.fillna(-1, inplace=True)\n \ndef check(product):\n missing_months = sorted(set(range(1, 13)) - set(product.index))\n product = product.values\n for mm in missing_months:\n product = np.insert(product, mm-1, 0)\n \n return pd.Series(product, index=range(1, 13))\n\n","metadata":{"papermill":{"duration":0.109242,"end_time":"2022-07-17T13:54:05.411749","exception":false,"start_time":"2022-07-17T13:54:05.302507","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:34:47.177600Z","iopub.execute_input":"2022-08-14T16:34:47.177992Z","iopub.status.idle":"2022-08-14T16:34:47.187752Z","shell.execute_reply.started":"2022-08-14T16:34:47.177959Z","shell.execute_reply":"2022-08-14T16:34:47.186605Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def eng_data():\n path = '../input/learnplatform-covid19-impact-on-digital-learning/engagement_data'\n files = os.listdir(path)\n\n for file in files:\n df = pd.read_csv(os.path.join(path,file))\n \n yield df","metadata":{"papermill":{"duration":0.106996,"end_time":"2022-07-17T13:54:05.615490","exception":false,"start_time":"2022-07-17T13:54:05.508494","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:34:48.873846Z","iopub.execute_input":"2022-08-14T16:34:48.874592Z","iopub.status.idle":"2022-08-14T16:34:48.880148Z","shell.execute_reply.started":"2022-08-14T16:34:48.874556Z","shell.execute_reply":"2022-08-14T16:34:48.879284Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def show_plots(dataframe, feature, title='Title'): \n fig = plt.figure(figsize=(25,35))\n fig.suptitle(title, fontsize=30, weight='bold', y=0.91)\n \n for i, product in enumerate(dataframe['Product Name'].unique()):\n data = dataframe[dataframe['Product Name']==product]\n data = data.groupby('month')[feature].mean()\n\n plt.subplot(5, 4, i+1)\n plt.plot(data.index, data.values)\n plt.title(product, color='brown')\n plt.xticks(ticks=range(1,13), labels=range(1, 13)); ","metadata":{"_kg_hide-input":true,"papermill":{"duration":0.107166,"end_time":"2022-07-17T13:54:05.820785","exception":false,"start_time":"2022-07-17T13:54:05.713619","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:34:49.529309Z","iopub.execute_input":"2022-08-14T16:34:49.530127Z","iopub.status.idle":"2022-08-14T16:34:49.538167Z","shell.execute_reply.started":"2022-08-14T16:34:49.530077Z","shell.execute_reply":"2022-08-14T16:34:49.537297Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def investigate(feature, size, title, order=None): \n fig, axes = plt.subplots(nrows=3, ncols=1, figsize=size)\n fig.suptitle(title, fontsize=30, weight='bold', y=0.93)\n\n for i, subset in enumerate(to_investigate): \n fig.sca(axes[i])\n sub_df = districts_info[districts_info['state'].isin(subset)].copy()\n\n show_countplot(sub_df, x=feature, y=None, order=order, title=titles[i])","metadata":{"papermill":{"duration":0.108848,"end_time":"2022-07-17T13:54:06.026484","exception":false,"start_time":"2022-07-17T13:54:05.917636","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:34:50.140229Z","iopub.execute_input":"2022-08-14T16:34:50.140625Z","iopub.status.idle":"2022-08-14T16:34:50.147248Z","shell.execute_reply.started":"2022-08-14T16:34:50.140591Z","shell.execute_reply":"2022-08-14T16:34:50.146409Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"2.2 Importing our Datasets & Cleaning them. \n

Products info dataset.

","metadata":{"papermill":{"duration":0.098931,"end_time":"2022-07-17T13:54:06.224670","exception":false,"start_time":"2022-07-17T13:54:06.125739","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# reading the data\nproduct_info_path= '../input/learnplatform-covid19-impact-on-digital-learning/products_info.csv'\nproducts_info = pd.read_csv(product_info_path)\nproducts_info.head()","metadata":{"_kg_hide-input":false,"papermill":{"duration":0.144838,"end_time":"2022-07-17T13:54:06.468473","exception":false,"start_time":"2022-07-17T13:54:06.323635","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:38:04.156772Z","iopub.execute_input":"2022-08-14T16:38:04.157389Z","iopub.status.idle":"2022-08-14T16:38:04.176517Z","shell.execute_reply.started":"2022-08-14T16:38:04.157351Z","shell.execute_reply":"2022-08-14T16:38:04.175433Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Abit more information.\nproducts_info.info()","metadata":{"_kg_hide-input":false,"papermill":{"duration":0.124607,"end_time":"2022-07-17T13:54:06.694334","exception":false,"start_time":"2022-07-17T13:54:06.569727","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:35:39.995711Z","iopub.execute_input":"2022-08-14T16:35:39.996196Z","iopub.status.idle":"2022-08-14T16:35:40.019630Z","shell.execute_reply.started":"2022-08-14T16:35:39.996157Z","shell.execute_reply":"2022-08-14T16:35:40.018745Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Well! It seems we have 372 entries, one row per each product, and we have a lot of missingness in ***Sector(s) and primary EEssential Function*** columns ","metadata":{"papermill":{"duration":0.102638,"end_time":"2022-07-17T13:54:06.897028","exception":false,"start_time":"2022-07-17T13:54:06.794390","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# investigating the missing values\nget_missingness(products_info)","metadata":{"_kg_hide-input":false,"papermill":{"duration":0.132719,"end_time":"2022-07-17T13:54:07.132155","exception":false,"start_time":"2022-07-17T13:54:06.999436","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:38:07.781332Z","iopub.execute_input":"2022-08-14T16:38:07.781958Z","iopub.status.idle":"2022-08-14T16:38:07.795829Z","shell.execute_reply.started":"2022-08-14T16:38:07.781911Z","shell.execute_reply":"2022-08-14T16:38:07.794142Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"For better intution, let's conveert this table into a nicer looking chart.","metadata":{"papermill":{"duration":0.09961,"end_time":"2022-07-17T13:54:07.332574","exception":false,"start_time":"2022-07-17T13:54:07.232964","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# Set a global style for our graphs.\nsns.set(font_scale= 1.5, rc={\"figure.figsize\":(8, 5)})\nsns.set_style({'axes.facecolor':'white'})","metadata":{"papermill":{"duration":0.167921,"end_time":"2022-07-17T13:54:07.599434","exception":false,"start_time":"2022-07-17T13:54:07.431513","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:36:08.757675Z","iopub.execute_input":"2022-08-14T16:36:08.759090Z","iopub.status.idle":"2022-08-14T16:36:08.766034Z","shell.execute_reply.started":"2022-08-14T16:36:08.759009Z","shell.execute_reply":"2022-08-14T16:36:08.764940Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"ax= msno.bar(products_info, figsize=(10,6), sort=\"ascending\")\nplt.title(\"The percentage and count of missingness\")\nfor p in ax.patches:\n percentage = '{:.1f}%'.format(100 - (100 * p.get_height()))\n x = p.get_x() \n y = p.get_y() + p.get_height()/2\n ax.annotate(percentage, (x, y), size = 15, weight=\"bold\", color=\"red\")","metadata":{"papermill":{"duration":1.02882,"end_time":"2022-07-17T13:54:08.730623","exception":false,"start_time":"2022-07-17T13:54:07.701803","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:36:19.432483Z","iopub.execute_input":"2022-08-14T16:36:19.433260Z","iopub.status.idle":"2022-08-14T16:36:20.052323Z","shell.execute_reply.started":"2022-08-14T16:36:19.433219Z","shell.execute_reply":"2022-08-14T16:36:20.051118Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"missingness = products_info[products_info.isna().any(axis= 1)].copy(deep= True) # Extract records containing NANs.\n\n# Exclude corresponding columns ['LP ID', 'URL', 'Product Name'], as containing no missingness.\n#missingness.drop(['LP ID', 'URL', 'Product Name'], axis= 1, inplace= True)\n# sort values by gunlaw column to get a better intution.\nmissingness= missingness.sort_values(by= 'Provider/Company Name')","metadata":{"papermill":{"duration":0.114062,"end_time":"2022-07-17T13:54:08.947990","exception":false,"start_time":"2022-07-17T13:54:08.833928","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:36:35.940039Z","iopub.execute_input":"2022-08-14T16:36:35.940451Z","iopub.status.idle":"2022-08-14T16:36:35.948841Z","shell.execute_reply.started":"2022-08-14T16:36:35.940416Z","shell.execute_reply":"2022-08-14T16:36:35.947811Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Now visualize missingness.\nmsno.matrix(missingness, figsize=(10,10), sparkline= False)\n\n# Identify patches for better eye_view.\ngray_patch, white_patch = mpatches.Patch(color='gray', label='remain values'), mpatches.Patch(color='white', label='missingness')\n\n# Set a legend\nplt.legend(handles=[gray_patch, white_patch], loc='center left', bbox_to_anchor=(1, 0.5))\n\nplt.title(\"The distribution of missingness\");","metadata":{"papermill":{"duration":0.498456,"end_time":"2022-07-17T13:54:09.552502","exception":false,"start_time":"2022-07-17T13:54:09.054046","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:36:37.542409Z","iopub.execute_input":"2022-08-14T16:36:37.543178Z","iopub.status.idle":"2022-08-14T16:36:37.859783Z","shell.execute_reply.started":"2022-08-14T16:36:37.543138Z","shell.execute_reply":"2022-08-14T16:36:37.858780Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"msno.dendrogram(missingness, figsize=(8, 6))\nplt.title(\"Fully correlate missingness completion\", color=\"red\");","metadata":{"papermill":{"duration":0.399942,"end_time":"2022-07-17T13:54:10.057363","exception":false,"start_time":"2022-07-17T13:54:09.657421","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:36:45.682342Z","iopub.execute_input":"2022-08-14T16:36:45.683048Z","iopub.status.idle":"2022-08-14T16:36:45.844467Z","shell.execute_reply.started":"2022-08-14T16:36:45.683009Z","shell.execute_reply":"2022-08-14T16:36:45.843238Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From this dendogram and matrix above, we can observe that misisngness in Sector(s) and primar Essential Function are heighly correlated.","metadata":{"papermill":{"duration":0.103405,"end_time":"2022-07-17T13:54:10.281351","exception":false,"start_time":"2022-07-17T13:54:10.177946","status":"completed"},"tags":[]}},{"cell_type":"markdown","source":"##### Let's fill the NaNs.\nWe will try to fill the NaNs Manually by going to each URL and trying to estimate `Sector` and `Primary Essential Function`","metadata":{"papermill":{"duration":0.10383,"end_time":"2022-07-17T13:54:10.489108","exception":false,"start_time":"2022-07-17T13:54:10.385278","status":"completed"},"tags":[]}},{"cell_type":"code","source":"# http://www.ixl.com/\nfill_it('IXL Language', 'PreK-12', 'LC - Digital Learning Platforms')\n\n# https://www.yelp.com/\n# It's not something for learning\nremove_it('Yelp')\n\n# http://www.learnplatform.com/\nfill_it('LearnPlatform', 'PreK-12; Corporate', 'SDO - Data, Analytics & Reporting - Student Information Systems (SIS)')\n\n# http://genius.com/static/education\n# It's not something for learning\nremove_it('Education Genius')\n\n# http://www.microsoft.com/en-us/education/products/office/default.aspx\nfill_it('Microsoft Office 365', 'PreK-12; Higher Ed; Corporate', 'LC - Study Tools')\n\n# http://www.classzone.com/cz/index.htm\n# It has been retired and is no longer accessible\nremove_it('ClassZone')\n\n# http://student.classdojo.com/#/login\nfill_it('ClassDojo for Students', 'PreK-12; Higher Ed', 'CM - Classroom Engagement & Instruction - Assessment & Classroom Response')\n\n# https://play.google.com/music/listen?u=0#/sulp\n# Google Play Music is no longer available\nremove_it('Google Play Music')\n\n# https://sciencejournal.withgoogle.com/\nfill_it('Google Science Journal', 'PreK-12; Higher Ed', 'LC - Study Tools - Tutoring')\n\n# https://edutrainingcenter.withgoogle.com/\n# It is no longer available\nremove_it('Google Training Center')\n\n# https://info.flipgrid.com/\nfill_it('Flipgrid One', 'PreK-12; Higher Ed; Corporate', 'CM - Virtual Classroom - Video Conferencing & Screen Sharing')\n\n# https://spark.adobe.com/about/page\n# It's not something for learning\nremove_it('Adobe Spark Page')\n\n# https://www.usnews.com/best-colleges/myfit\nfill_it('College Compass', 'Higher Ed; Corporate', 'SDO - Data, Analytics & Reporting - Site Hosting & Data Warehousing')\n\n# https://chrome.google.com/webstore/detail/grammarly-for-chrome/kbfnbcaeplbcioakkpcpgfkobkghlhen?hl=en\nfill_it('Grammarly for Chrome', 'PreK-12; Higher Ed; Corporate', 'LC - Content Creation & Curation')\n\n# https://www.maxpreps.com/state/connecticut.htm\n# It is no longer available\nremove_it('MaxPreps: Connecticut')\n\n# https://www.ducksters.com/history/\nfill_it('History for Kids', 'PreK-12', 'LC - Digital Learning Platforms')\n\n# https://safeyoutube.net/\n# It is no longer accessible\nremove_it('SafeYouTube')\n\n# https://studio.code.org\nfill_it('Studio Code', 'PreK-12; Higher Ed', 'LC - Sites, Resources & Reference - Games & Simulations')\n\n# http://edpuzzle.com\nfill_it('Edpuzzle - Free (Basic Plan)', 'PreK-12; Higher Ed; Corporate', 'LC - Sites, Resources & Reference - Digital Collection & Repository')\n\n# http://www.truenorthlogic.com/\n# It is no longer accessible\nremove_it('True North Logic')","metadata":{"_kg_hide-input":false,"papermill":{"duration":0.141523,"end_time":"2022-07-17T13:54:10.736149","exception":false,"start_time":"2022-07-17T13:54:10.594626","status":"completed"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:38:16.647493Z","iopub.execute_input":"2022-08-14T16:38:16.647911Z","iopub.status.idle":"2022-08-14T16:38:16.677380Z","shell.execute_reply.started":"2022-08-14T16:38:16.647874Z","shell.execute_reply":"2022-08-14T16:38:16.676381Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"- **Check our modifications**","metadata":{"papermill":{"duration":0.108302,"end_time":"2022-07-17T13:54:10.950388","exception":false,"start_time":"2022-07-17T13:54:10.842086","status":"completed"},"tags":[]}},{"cell_type":"code","source":"get_missingness(products_info)","metadata":{"_kg_hide-input":false,"papermill":{"duration":0.213791,"end_time":"2022-07-17T13:54:11.271267","exception":true,"start_time":"2022-07-17T13:54:11.057476","status":"failed"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:38:33.337080Z","iopub.execute_input":"2022-08-14T16:38:33.337509Z","iopub.status.idle":"2022-08-14T16:38:33.350040Z","shell.execute_reply.started":"2022-08-14T16:38:33.337470Z","shell.execute_reply":"2022-08-14T16:38:33.349119Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Last but not least, Is there any complete duplicates?\nproducts_info.duplicated().sum()","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:38:44.346976Z","iopub.execute_input":"2022-08-14T16:38:44.347376Z","iopub.status.idle":"2022-08-14T16:38:44.356436Z","shell.execute_reply.started":"2022-08-14T16:38:44.347343Z","shell.execute_reply":"2022-08-14T16:38:44.355601Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Remove URL column, as it contains no useful information.\nproducts_info.drop('URL', axis=1, inplace=True)","metadata":{"_kg_hide-input":false,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:38:46.946976Z","iopub.execute_input":"2022-08-14T16:38:46.947376Z","iopub.status.idle":"2022-08-14T16:38:46.953924Z","shell.execute_reply.started":"2022-08-14T16:38:46.947342Z","shell.execute_reply":"2022-08-14T16:38:46.952627Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now product_info dataset is ready.","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"

Districts Info dataset

","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"# reading the data\ndistricts_inf_path= '../input/learnplatform-covid19-impact-on-digital-learning/districts_info.csv'\ndistricts_info = pd.read_csv(districts_inf_path)\ndistricts_info.head()","metadata":{"_kg_hide-input":false,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:38:52.111963Z","iopub.execute_input":"2022-08-14T16:38:52.112407Z","iopub.status.idle":"2022-08-14T16:38:52.131757Z","shell.execute_reply.started":"2022-08-14T16:38:52.112366Z","shell.execute_reply":"2022-08-14T16:38:52.130711Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# get more info\ndistricts_info.info()","metadata":{"_kg_hide-input":false,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:38:52.781492Z","iopub.execute_input":"2022-08-14T16:38:52.782281Z","iopub.status.idle":"2022-08-14T16:38:52.795970Z","shell.execute_reply.started":"2022-08-14T16:38:52.782240Z","shell.execute_reply":"2022-08-14T16:38:52.794790Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"get_missingness(districts_info)","metadata":{"_kg_hide-input":false,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:38:57.401231Z","iopub.execute_input":"2022-08-14T16:38:57.401606Z","iopub.status.idle":"2022-08-14T16:38:57.414945Z","shell.execute_reply.started":"2022-08-14T16:38:57.401575Z","shell.execute_reply":"2022-08-14T16:38:57.413595Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Again! We've alot of missingness. So, Let's fill those NaNs.","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"msno.dendrogram(districts_info, figsize=(8, 6))\nplt.title(\"Fully correlate missingness completion\", color=\"red\");","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:39:45.736199Z","iopub.execute_input":"2022-08-14T16:39:45.737212Z","iopub.status.idle":"2022-08-14T16:39:45.966416Z","shell.execute_reply.started":"2022-08-14T16:39:45.737167Z","shell.execute_reply":"2022-08-14T16:39:45.965038Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Missingness are heightly correlated. So, we're going to drop records containing NaNs in all features except for the district id columns.","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"districts_info = districts_info.dropna(how='all', subset=['state', 'locale', 'pct_black/hispanic', 'pct_free/reduced',\n 'county_connections_ratio', 'pp_total_raw']) ","metadata":{"_kg_hide-input":false,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:39:57.183564Z","iopub.execute_input":"2022-08-14T16:39:57.184363Z","iopub.status.idle":"2022-08-14T16:39:57.193532Z","shell.execute_reply.started":"2022-08-14T16:39:57.184314Z","shell.execute_reply":"2022-08-14T16:39:57.192486Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we discussed above ... let's fill the NaNs with \"Missied\" Value","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"districts_info.fillna(\"Missied\", inplace=True)","metadata":{"_kg_hide-input":false,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:40:01.839603Z","iopub.execute_input":"2022-08-14T16:40:01.840007Z","iopub.status.idle":"2022-08-14T16:40:01.846487Z","shell.execute_reply.started":"2022-08-14T16:40:01.839973Z","shell.execute_reply":"2022-08-14T16:40:01.845145Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Check about missingness existence.","metadata":{}},{"cell_type":"code","source":"get_missingness(districts_info)","metadata":{"_kg_hide-input":false,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:40:21.644429Z","iopub.execute_input":"2022-08-14T16:40:21.644826Z","iopub.status.idle":"2022-08-14T16:40:21.658691Z","shell.execute_reply.started":"2022-08-14T16:40:21.644796Z","shell.execute_reply":"2022-08-14T16:40:21.657422Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Engagement Data

","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"to_merge = [] # putting the engagement data in one place for concatenation\neng_df = eng_data() # defining the generator","metadata":{"_kg_hide-input":false,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:40:50.693006Z","iopub.execute_input":"2022-08-14T16:40:50.693895Z","iopub.status.idle":"2022-08-14T16:40:50.699318Z","shell.execute_reply.started":"2022-08-14T16:40:50.693852Z","shell.execute_reply":"2022-08-14T16:40:50.697929Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for data in eng_df:\n fill_data(data) # filling NaNs\n to_merge.append(data)\n \nall_eng = pd.concat(to_merge)","metadata":{"_kg_hide-input":false,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:40:52.191569Z","iopub.execute_input":"2022-08-14T16:40:52.192007Z","iopub.status.idle":"2022-08-14T16:41:10.943109Z","shell.execute_reply.started":"2022-08-14T16:40:52.191969Z","shell.execute_reply":"2022-08-14T16:41:10.941756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_eng.head()","metadata":{"_kg_hide-input":false,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:41:10.944895Z","iopub.execute_input":"2022-08-14T16:41:10.945275Z","iopub.status.idle":"2022-08-14T16:41:10.959255Z","shell.execute_reply.started":"2022-08-14T16:41:10.945235Z","shell.execute_reply":"2022-08-14T16:41:10.958299Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_eng.info()","metadata":{"_kg_hide-input":false,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:41:14.973818Z","iopub.execute_input":"2022-08-14T16:41:14.974644Z","iopub.status.idle":"2022-08-14T16:41:14.989397Z","shell.execute_reply.started":"2022-08-14T16:41:14.974593Z","shell.execute_reply":"2022-08-14T16:41:14.988090Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Check for missingness.\nget_missingness(all_eng)","metadata":{"_kg_hide-input":false,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:41:26.483901Z","iopub.execute_input":"2022-08-14T16:41:26.484307Z","iopub.status.idle":"2022-08-14T16:41:27.578982Z","shell.execute_reply.started":"2022-08-14T16:41:26.484275Z","shell.execute_reply":"2022-08-14T16:41:27.577863Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's convert the type of \"time\" column from `object` to `datetime`","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"all_eng['time'] = pd.to_datetime(all_eng['time'])","metadata":{"_kg_hide-input":false,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:41:43.186125Z","iopub.execute_input":"2022-08-14T16:41:43.186519Z","iopub.status.idle":"2022-08-14T16:41:46.787470Z","shell.execute_reply.started":"2022-08-14T16:41:43.186486Z","shell.execute_reply":"2022-08-14T16:41:46.786439Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Chech for complete duplicates.\nall_eng.duplicated().sum()","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:41:46.789450Z","iopub.execute_input":"2022-08-14T16:41:46.789885Z","iopub.status.idle":"2022-08-14T16:41:54.868187Z","shell.execute_reply.started":"2022-08-14T16:41:46.789844Z","shell.execute_reply":"2022-08-14T16:41:54.856673Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### As we can see there are: \n - _Columns has wrong datatype:_\n - time\n \n - _Duplicates:_\n - 5056710 duplicated rows","metadata":{}},{"cell_type":"code","source":"# Drop duplicates\nall_eng.drop_duplicates(inplace=True)","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:42:25.516550Z","iopub.execute_input":"2022-08-14T16:42:25.516966Z","iopub.status.idle":"2022-08-14T16:42:33.833365Z","shell.execute_reply.started":"2022-08-14T16:42:25.516932Z","shell.execute_reply":"2022-08-14T16:42:33.832120Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> Great, Now we're **ready** for some exploration :).","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"Before doing anything, Let's first combine \"all_eng\" and \"products_info\" datasets to have more info about each product. ","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"eng_product_merge = pd.merge(all_eng, products_info, left_on='lp_id',right_on='LP ID',how='inner').drop(columns='LP ID')\neng_product_merge.head()","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:42:33.835194Z","iopub.execute_input":"2022-08-14T16:42:33.835574Z","iopub.status.idle":"2022-08-14T16:42:39.348008Z","shell.execute_reply.started":"2022-08-14T16:42:33.835540Z","shell.execute_reply":"2022-08-14T16:42:39.346902Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"eng_product_merge.info()","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:42:39.350261Z","iopub.execute_input":"2022-08-14T16:42:39.350593Z","iopub.status.idle":"2022-08-14T16:42:39.366416Z","shell.execute_reply.started":"2022-08-14T16:42:39.350562Z","shell.execute_reply":"2022-08-14T16:42:39.365033Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"extracting the `month` from \"time\" feature","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"eng_product_merge['month'] = eng_product_merge['time'].dt.month\neng_product_merge.head()","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:42:39.368346Z","iopub.execute_input":"2022-08-14T16:42:39.370504Z","iopub.status.idle":"2022-08-14T16:42:40.254232Z","shell.execute_reply.started":"2022-08-14T16:42:39.370468Z","shell.execute_reply":"2022-08-14T16:42:40.253162Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Before anwering any questions, we need to explore the data a little bit to know what are we dealing with.","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"

Products Info

","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"_Provider/Company Name_","metadata":{"execution":{"iopub.execute_input":"2021-09-24T19:12:09.577834Z","iopub.status.busy":"2021-09-24T19:12:09.577466Z","iopub.status.idle":"2021-09-24T19:12:09.585955Z","shell.execute_reply":"2021-09-24T19:12:09.583948Z","shell.execute_reply.started":"2021-09-24T19:12:09.577802Z"},"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"products_info['Provider/Company Name'].nunique()","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:42:46.700403Z","iopub.execute_input":"2022-08-14T16:42:46.700837Z","iopub.status.idle":"2022-08-14T16:42:46.708328Z","shell.execute_reply.started":"2022-08-14T16:42:46.700800Z","shell.execute_reply":"2022-08-14T16:42:46.707337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> As we know, this number is too large for visualization. so let's plot the most frequent 20 companies.","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"# Flip it\norder = products_info['Provider/Company Name'].value_counts().index[:20]\n\nplt.figure(figsize=(15, 12))\nshow_countplot(products_info, x=None, y='Provider/Company Name', \n order=order, title='Provide/Company Name')","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:42:57.011001Z","iopub.execute_input":"2022-08-14T16:42:57.011454Z","iopub.status.idle":"2022-08-14T16:42:57.476635Z","shell.execute_reply.started":"2022-08-14T16:42:57.011415Z","shell.execute_reply":"2022-08-14T16:42:57.475469Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"_Sector(s)_ ","metadata":{"execution":{"iopub.execute_input":"2021-09-25T16:31:53.995599Z","iopub.status.busy":"2021-09-25T16:31:53.994865Z","iopub.status.idle":"2021-09-25T16:31:54.005225Z","shell.execute_reply":"2021-09-25T16:31:54.003486Z","shell.execute_reply.started":"2021-09-25T16:31:53.995527Z"},"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"sectors = eng_product_merge['Sector(s)'].str.split(';',expand=True)\n\n# removing unnecessary spaces\nfor col in sectors.columns:\n sectors[col] = sectors[col].str.strip()\n\n# Counting the occurences of each sector in each column \ndic={}\nfor col in sectors.columns:\n dic[col]=(dict(sectors[col].value_counts()))\n\n# Summing all together\nsectors = pd.DataFrame(dic)\nsectors_count = pd.DataFrame(sectors.fillna(0).values.sum(axis=1), index=sectors.index, columns=['Count'])\nsectors_count","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:43:04.276814Z","iopub.execute_input":"2022-08-14T16:43:04.277249Z","iopub.status.idle":"2022-08-14T16:43:37.720236Z","shell.execute_reply.started":"2022-08-14T16:43:04.277209Z","shell.execute_reply":"2022-08-14T16:43:37.718970Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.pie(sectors_count.values.ravel(), labels=sectors_count.index, startangle = 90, autopct='%1.2f%%', \n counterclock = False, radius = 1.2, textprops={'fontsize': 14})\n\nplt.title('Sector(s)', fontsize=15, color='brown');","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:43:37.722494Z","iopub.execute_input":"2022-08-14T16:43:37.723220Z","iopub.status.idle":"2022-08-14T16:43:38.377173Z","shell.execute_reply.started":"2022-08-14T16:43:37.723174Z","shell.execute_reply":"2022-08-14T16:43:38.375978Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# For memory size reasons\ndel sectors, sectors_count","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:44:22.770122Z","iopub.execute_input":"2022-08-14T16:44:22.770549Z","iopub.status.idle":"2022-08-14T16:44:22.775567Z","shell.execute_reply.started":"2022-08-14T16:44:22.770516Z","shell.execute_reply":"2022-08-14T16:44:22.774456Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"_Primary Essential Function_ ","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"Main Categories ['LC', 'SDO', 'CM']","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"# Let's get the primary eseential function main categories ['LC', 'SDO', 'CM'] alone \neng_product_merge['PEF'] = eng_product_merge['Primary Essential Function'].str.split('-')\neng_product_merge['PEF'] = eng_product_merge['PEF'].apply(lambda x: x[0])\neng_product_merge['PEF'] = eng_product_merge['PEF'].str.strip()","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:44:28.328492Z","iopub.execute_input":"2022-08-14T16:44:28.329341Z","iopub.status.idle":"2022-08-14T16:44:48.501622Z","shell.execute_reply.started":"2022-08-14T16:44:28.329296Z","shell.execute_reply":"2022-08-14T16:44:48.500581Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"pef_count = eng_product_merge['PEF'].value_counts()\n\nplt.figure(figsize=(8, 8))\nplt.pie(pef_count.values, labels=pef_count.index, startangle = 90, autopct='%1.2f%%', \n counterclock = False, radius = 1.2, textprops={'fontsize': 14});\n\nplt.title('Primary Essential Function', fontsize=15, color='brown', y=1.1);","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:44:48.503611Z","iopub.execute_input":"2022-08-14T16:44:48.504135Z","iopub.status.idle":"2022-08-14T16:44:49.457564Z","shell.execute_reply.started":"2022-08-14T16:44:48.504090Z","shell.execute_reply":"2022-08-14T16:44:49.456299Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's explore the subcategories of each Primary Essential Function main categories.","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"eng_product_merge['sub_PEF'] = eng_product_merge['Primary Essential Function'].str.split('-')\neng_product_merge['sub_PEF'] = eng_product_merge['sub_PEF'].apply(lambda y: ' '.join(map(lambda x: x.strip(), y[1:])))","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:44:49.459505Z","iopub.execute_input":"2022-08-14T16:44:49.460286Z","iopub.status.idle":"2022-08-14T16:45:15.840142Z","shell.execute_reply.started":"2022-08-14T16:44:49.460237Z","shell.execute_reply":"2022-08-14T16:45:15.839052Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, axes = plt.subplots(nrows=3, ncols=1, figsize=(15, 25))\n\nfig.suptitle('Subcategories of each Primary Essential Function main categories', fontsize=25, y=0.92, x=0.3)\n\nfor i, pef in enumerate(['LC', 'CM', 'SDO']):\n fig.sca(axes[i])\n \n temp = eng_product_merge[eng_product_merge['PEF']==pef].copy()\n order = temp['sub_PEF'].value_counts().index\n \n show_countplot(temp, x=None, y='sub_PEF', order=order, title=pef)","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:45:15.842429Z","iopub.execute_input":"2022-08-14T16:45:15.842767Z","iopub.status.idle":"2022-08-14T16:45:29.312202Z","shell.execute_reply.started":"2022-08-14T16:45:15.842737Z","shell.execute_reply":"2022-08-14T16:45:29.310830Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Districts Info

","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"fig, axes = plt.subplots(6, 1, figsize=(25,65))\nfig.suptitle('Districts Info', fontsize=25, y=0.9)\naxes = axes.ravel()\n\n# which axes to plot on\nto_plot=[(None, 'state'),\n ('locale', None),\n ('pct_black/hispanic', None),\n ('pct_free/reduced', None),\n ('county_connections_ratio', None),\n ('pp_total_raw', None)]\n\nfor i,col in enumerate(districts_info.columns[1:]):\n fig.sca(axes[i])\n \n order=districts_info[col].value_counts().index\n show_countplot(districts_info, x=to_plot[i][0], y=to_plot[i][1], order=order, title=f'{col.title()} Percentage')","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:45:29.313911Z","iopub.execute_input":"2022-08-14T16:45:29.314353Z","iopub.status.idle":"2022-08-14T16:45:31.173124Z","shell.execute_reply.started":"2022-08-14T16:45:29.314311Z","shell.execute_reply":"2022-08-14T16:45:31.171890Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> From the previous plot we can see that \"Country connections ratio\" column is not important at all.That's why we will drop it","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"districts_info.drop('county_connections_ratio', axis=1, inplace=True)","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:45:31.174632Z","iopub.execute_input":"2022-08-14T16:45:31.175034Z","iopub.status.idle":"2022-08-14T16:45:31.182255Z","shell.execute_reply.started":"2022-08-14T16:45:31.174995Z","shell.execute_reply":"2022-08-14T16:45:31.181380Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Engagement Data

","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"Let's explore \"engagement_index\" and \"perecentage access\" over the whole year","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"all_eng_copy = all_eng.set_index('time')\n\nfig, axes = plt.subplots(nrows=2, ncols=1, figsize=(15, 12))\n\nfor i, col in enumerate(['engagement_index', 'pct_access']):\n fig.sca(axes[i])\n all_eng_copy[col].plot()\n plt.title(col, fontsize=15, color='brown')\n\nplt.tight_layout()\ndel all_eng_copy","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:45:31.183679Z","iopub.execute_input":"2022-08-14T16:45:31.184052Z","iopub.status.idle":"2022-08-14T16:48:15.996029Z","shell.execute_reply.started":"2022-08-14T16:45:31.184007Z","shell.execute_reply":"2022-08-14T16:48:15.994839Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> We can see that their behaviour approximately the same over the year.Before finishing the exploration let's see the correlation between them.","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"corr = all_eng['engagement_index'].corr(all_eng['pct_access'])\n\nprint(f\"Pearson's correlation between engagement index and perecentage access is: {colored(corr, 'green')}\")","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:48:15.997965Z","iopub.execute_input":"2022-08-14T16:48:15.998699Z","iopub.status.idle":"2022-08-14T16:48:16.329735Z","shell.execute_reply.started":"2022-08-14T16:48:15.998664Z","shell.execute_reply":"2022-08-14T16:48:16.328594Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> As expected, Quite high correlation. Which makes us use only one of them in our next analysis (By choosing the most suitable one for answering our questions)","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"

Merged Data

","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"Let's see, what are the most used products in 2020 ?","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"top_products = eng_product_merge.groupby('Product Name')['engagement_index'].mean().sort_values(ascending=False)\ntop_10=pd.DataFrame(top_products[:10])\n\nplt.figure(figsize=(15,9))\n\nsns.barplot(x=top_10['engagement_index'],y=top_10.index, color='#4d83de')\nplt.title('Top 10 Products', color='brown', fontsize=15)\nplt.xlabel('Engagment Index', fontsize=12)\nplt.ylabel('Product Name', fontsize=12)\n\ndel top_products","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:48:16.332881Z","iopub.execute_input":"2022-08-14T16:48:16.333245Z","iopub.status.idle":"2022-08-14T16:48:17.582720Z","shell.execute_reply.started":"2022-08-14T16:48:16.333213Z","shell.execute_reply":"2022-08-14T16:48:17.581584Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"

Let's Answer the proposed Questions first then we will dive deeper in our analysis

","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"Actually we can answer the first proposed question **\"What is the picture of digital connectivity and engagement in 2020?\"** by the [previous exploration](#pe).from the engagement index plot we can see that the engagement with e-learning platform began low then was increasing with time then there was a drop (between July and September) then the engagement began to increase again.Two questions should be answered:
\n    (1) what caused that drop?
\n    (2) why the engagement began to rise again?
\n\n**For the first question, May be due to:**\n- Most of the american universities take the holiday between July and September.\n- According to [WHO](https://www.who.int/emergencies/diseases/novel-coronavirus-2019/interactive-timeline?gclid=CjwKCAjwyvaJBhBpEiwA8d38vF9MS1vRVTg1UPNPzmP8DNAgsBywY6M_Q3Q9giUSd5xtato7Y79zNBoC2mIQAvD_BwE#!) the number of corona cases passed 200000 during that period which ,in turn, attracted the attention to \"what we should do and how can we protect ourselves\". May all that distracted the students from learning. \n\n**For the second one:**\n- If the cause was the holiday. So obviously we can say that the rise was due to the begging of a new semester.\n- But if the cause was due to the distraction caused by the number of deaths So may people began used to it.\n\nLet's dive deeper to **see the details of this picture :)**","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"# number of products we have\neng_product_merge['Product Name'].nunique()","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:48:17.584339Z","iopub.execute_input":"2022-08-14T16:48:17.585351Z","iopub.status.idle":"2022-08-14T16:48:18.295028Z","shell.execute_reply.started":"2022-08-14T16:48:17.585305Z","shell.execute_reply":"2022-08-14T16:48:18.293851Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> 361 is a very large number for visualization. so we will take 20 random chosen products to tell us the overall picture.","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"np.random.seed(42) # to have a consistent output every time we run the code\n\nsampled_products = np.random.choice(eng_product_merge['Product Name'].unique(), size=20, replace=False)\nsampled_products = eng_product_merge[eng_product_merge['Product Name'].isin(sampled_products)].copy()","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:48:18.296347Z","iopub.execute_input":"2022-08-14T16:48:18.296684Z","iopub.status.idle":"2022-08-14T16:48:19.519124Z","shell.execute_reply.started":"2022-08-14T16:48:18.296655Z","shell.execute_reply":"2022-08-14T16:48:19.517765Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"show_plots(sampled_products, 'engagement_index', title='Engagement Index for 20 random sampled products')","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:48:19.520827Z","iopub.execute_input":"2022-08-14T16:48:19.521573Z","iopub.status.idle":"2022-08-14T16:48:23.696016Z","shell.execute_reply.started":"2022-08-14T16:48:19.521525Z","shell.execute_reply":"2022-08-14T16:48:23.695065Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"_From the previous exploration we knew that there is no need for investigating the percentage access but let's see if it will add something_","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"show_plots(sampled_products, 'pct_access',\n title='Percentage Access for 20 random sampled products')","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:48:23.697183Z","iopub.execute_input":"2022-08-14T16:48:23.697500Z","iopub.status.idle":"2022-08-14T16:48:28.089136Z","shell.execute_reply.started":"2022-08-14T16:48:23.697471Z","shell.execute_reply":"2022-08-14T16:48:28.088192Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> We can observe that **the two plots are very similar**, so in the next plots, We will use `engagement index` because the percentage access is related to a certain district but we want now to make a general analysis for product usage not for specific districts.\n\n> **We can observe that the engagement index behaviour changes from product to product the:
**\n     ○ Some products, their usage after the mentioned drop couldn't reach its original state before that drop like:
\n         • CoolMath Games
\n         • CK-12
\n         • Adobe Character Animator
\n         • Microsoft Outlook

\n     ○ Some products, their usage after the mentioned drop approximately reached its original state before that drop like:
\n         • Diax

\n     ○ Other products, their usage after the mentioned drop surpassed its original state before that drop like:
\n         • Remind
\n         • Ellevation
\n         • ZOOM Cloud Meetings
\n\nSo now let's dig deeper and see what we will get.","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"# For memory usage\ndel sampled_products","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:48:28.090353Z","iopub.execute_input":"2022-08-14T16:48:28.091213Z","iopub.status.idle":"2022-08-14T16:48:28.111463Z","shell.execute_reply.started":"2022-08-14T16:48:28.091174Z","shell.execute_reply":"2022-08-14T16:48:28.110086Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**_Let's see the impact of COVID-19 on \"LC\" Websites_**","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"trunc_df = eng_product_merge[eng_product_merge['PEF']=='LC'].copy()\n\n#Let's take the 20 most used products for \"LC\" category and analyse their usage along the year\nsampled_products = trunc_df['Product Name'].value_counts().index[:20]\nsampled_products = trunc_df[trunc_df['Product Name'].isin(sampled_products)].copy()\n \nshow_plots(sampled_products, 'engagement_index', title='LC: Engagement Index for 20 random sampled products')\n\n# for memory usage\ndel sampled_products, trunc_df","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:48:28.113215Z","iopub.execute_input":"2022-08-14T16:48:28.113715Z","iopub.status.idle":"2022-08-14T16:48:37.542207Z","shell.execute_reply.started":"2022-08-14T16:48:28.113668Z","shell.execute_reply":"2022-08-14T16:48:37.541103Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**_Let's see the impact of COVID-19 on \"CM\" Websites_**","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"trunc_df = eng_product_merge[eng_product_merge['PEF']=='CM'].copy()\n\n#Let's take the 20 most used products for \"LC\" category and analyse their usage along the year\nsampled_products = trunc_df['Product Name'].value_counts().index[:20]\nsampled_products = trunc_df[trunc_df['Product Name'].isin(sampled_products)].copy()\n \nshow_plots(sampled_products, 'engagement_index', title='CM: Engagement Index for 20 random sampled products')\n\n# for memory usage\ndel sampled_products, trunc_df","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:48:37.544179Z","iopub.execute_input":"2022-08-14T16:48:37.544566Z","iopub.status.idle":"2022-08-14T16:48:42.796521Z","shell.execute_reply.started":"2022-08-14T16:48:37.544529Z","shell.execute_reply":"2022-08-14T16:48:42.795149Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**_Let's see the impact of COVID-19 on \"SDO\" Websites_**","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"trunc_df = eng_product_merge[eng_product_merge['PEF']=='SDO'].copy()\n\n#Let's take the 20 most used products for \"LC\" category and analyse their usage along the year\nsampled_products = trunc_df['Product Name'].value_counts().index[:20]\nsampled_products = trunc_df[trunc_df['Product Name'].isin(sampled_products)].copy()\n \nshow_plots(sampled_products, 'engagement_index', title='SDO: Engagement Index for 20 random sampled products')\n\n# for memory usage\ndel sampled_products","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:48:42.802189Z","iopub.execute_input":"2022-08-14T16:48:42.802703Z","iopub.status.idle":"2022-08-14T16:48:48.171759Z","shell.execute_reply.started":"2022-08-14T16:48:42.802656Z","shell.execute_reply":"2022-08-14T16:48:48.170005Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**_Let's see if the impact of COVID-19 differs between \"LC\", \"CM\",and \"SDO\" Websites_**","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"colors = ['red', 'blue', 'green']\n\nfig = plt.figure(figsize=(15,20))\nfig.suptitle('Impact of COVID-19 on (\"LC\", \"CM\", \"SDO\") websites', fontsize=30, weight='bold', y=0.93)\n\nfor i, pef in enumerate(['LC', 'CM', 'SDO']):\n trunc_df = eng_product_merge[eng_product_merge['PEF']==pef].copy()\n \n most_used = trunc_df['Product Name'].value_counts().index[:20]\n \n plt.subplot(3, 1, i+1)\n for product in most_used:\n product_list = trunc_df[trunc_df['Product Name']== product].groupby('month')['pct_access'].mean()\n plt.plot(product_list.index, product_list.values, color=colors[i]);\n \n plt.title(pef, fontsize=20, color='brown')\n plt.xticks(ticks=range(1,13), labels=range(1, 13));","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:48:48.173490Z","iopub.execute_input":"2022-08-14T16:48:48.174189Z","iopub.status.idle":"2022-08-14T16:49:04.880363Z","shell.execute_reply.started":"2022-08-14T16:48:48.174151Z","shell.execute_reply":"2022-08-14T16:49:04.879204Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> As we can see, it's hard to plot all these lines in one plot because of the wide range between the values in `engagement_index` and of course `pct_access` so we needed to take another measure of the usage of e-learning platforms and online learning.That's why we've used **the number of e-learning products used during each month** as a measure of how active the e-learning was during that month.","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"colors = ['red', 'blue', 'green']\n\nfig = plt.figure(figsize=(15,20))\nfig.suptitle('Impact of COVID-19 on (\"LC\", \"CM\", \"SDO\") websites', fontsize=30, weight='bold', y=0.93)\n\nfor i, pef in enumerate(['LC', 'CM', 'SDO']):\n trunc_df = eng_product_merge[eng_product_merge['PEF']==pef].copy()\n \n most_used = trunc_df['Product Name'].value_counts().index[:20]\n \n plt.subplot(3, 1, i+1)\n for product in most_used:\n product_list = trunc_df[trunc_df['Product Name']== product].groupby('month')['lp_id'].count()\n plt.plot(product_list.index, product_list.values, color=colors[i]);\n \n plt.title(pef, fontsize=20, color='brown')\n plt.xticks(ticks=range(1,13), labels=range(1, 13));","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:49:04.882006Z","iopub.execute_input":"2022-08-14T16:49:04.882399Z","iopub.status.idle":"2022-08-14T16:49:23.502129Z","shell.execute_reply.started":"2022-08-14T16:49:04.882364Z","shell.execute_reply":"2022-08-14T16:49:23.500893Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's plot the average line for each category","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"colors = ['red', 'blue', 'green']\n\nfig = plt.figure(figsize=(12,7))\n\nfor i, pef in enumerate(['LC', 'CM', 'SDO']):\n trunc_df = eng_product_merge[eng_product_merge['PEF']==pef].copy()\n\n most_used = trunc_df['Product Name'].value_counts().index[:20]\n \n product_month = {}\n for product in most_used:\n product_month[product] = trunc_df[trunc_df['Product Name']== product].groupby('month')['lp_id'].count()\n product_month[product] = check(product_month[product])\n \n product_month = pd.DataFrame(product_month)\n \n average_plot = product_month.mean(axis=1)\n \n plt.plot(average_plot.index, average_plot.values, color=colors[i], label=pef);\n \nplt.title('Impact of COVID-19 on (\"LC\", \"CM\", \"SDO\") websites (Averages)', fontsize=20, color='brown')\nplt.legend();\n\n# for memory usage\ndel product_month","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:49:23.503510Z","iopub.execute_input":"2022-08-14T16:49:23.504127Z","iopub.status.idle":"2022-08-14T16:49:40.027208Z","shell.execute_reply.started":"2022-08-14T16:49:23.504094Z","shell.execute_reply":"2022-08-14T16:49:40.026092Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Regardless of this drop we've talked about before. It seems that approximately:\n- The usage of `LC` webistes **returns** to its original state before that drop\n- The usage of `CM` websites **Increased**\n- The usage of `SDO` webistes **decreased**","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"### This analysis has been done on the whole analysis Let's investigate each state sperately and see if the that will lead us to something.","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"Combining districts for each state together","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"# so that our results always be the same\nnp.random.seed(42)\n\nstates = {} \nstates_to_take = np.random.choice(districts_info['state'].unique(), size=10, replace=False)\n\nfor state in states_to_take:\n districts = districts_info[districts_info['state']==state].district_id.values\n \n to_merge = []\n for district in districts:\n to_merge.append(pd.read_csv(f'../input/learnplatform-covid19-impact-on-digital-learning/engagement_data/{district}.csv'))\n \n states[state] = pd.concat(to_merge)","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:49:40.028681Z","iopub.execute_input":"2022-08-14T16:49:40.029282Z","iopub.status.idle":"2022-08-14T16:49:44.599199Z","shell.execute_reply.started":"2022-08-14T16:49:40.029249Z","shell.execute_reply":"2022-08-14T16:49:44.598005Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for state in states:\n states[state] = pd.merge(states[state], products_info, left_on='lp_id', right_on='LP ID', how='inner').drop('LP ID', axis=1)\n \n # doing the same processes\n states[state]['time'] = pd.to_datetime(states[state]['time'])\n states[state]['month'] = states[state]['time'].dt.month\n \n states[state]['PEF'] = states[state]['Primary Essential Function'].str.split('-')\n states[state]['PEF'] = states[state]['PEF'].apply(lambda x: x[0])\n states[state]['PEF'] = states[state]['PEF'].str.strip()","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:49:44.602109Z","iopub.execute_input":"2022-08-14T16:49:44.602762Z","iopub.status.idle":"2022-08-14T16:49:55.975048Z","shell.execute_reply.started":"2022-08-14T16:49:44.602722Z","shell.execute_reply":"2022-08-14T16:49:55.973886Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig, axes = plt.subplots(nrows=10, ncols=3, figsize=(25,65))\nfig.suptitle('Impact of COVID-19 on (\"LC\", \"CM\", \"SDO\") websites (from each state perspective)', fontsize=30, weight='bold', y=0.91)\n\ncolors = ['red', 'blue', 'green']\nfor i, state in enumerate(states):\n for j, pef in enumerate(['LC', 'CM', 'SDO']):\n data = states[state]\n trunc_df = data[data['PEF']==pef].copy()\n\n most_used = trunc_df['Product Name'].value_counts().index[:20]\n\n for product in most_used:\n product_list = trunc_df[trunc_df['Product Name']== product].groupby('month')['lp_id'].count()\n axes[i, j].plot(product_list.index, product_list.values, color=colors[j], alpha=0.5);\n\n axes[i, j].set_title(f'{state}\\n({pef})', fontsize=20, color=colors[j])\n axes[i, j].set_xticks(range(1, 13));","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:49:55.976739Z","iopub.execute_input":"2022-08-14T16:49:55.977757Z","iopub.status.idle":"2022-08-14T16:50:10.993277Z","shell.execute_reply.started":"2022-08-14T16:49:55.977708Z","shell.execute_reply":"2022-08-14T16:50:10.992349Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's plot the average lines instead of all that mess.","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"fig, axes = plt.subplots(nrows=10, ncols=1, figsize=(25,65))\nfig.suptitle('Impact of COVID-19 on (\"LC\", \"CM\", \"SDO\") websites (Averages-from each state perspective)', fontsize=30, weight='bold', y=0.91)\n\ncolors = ['red', 'blue', 'green']\nfor i, state in enumerate(states):\n for j, pef in enumerate(['LC', 'CM', 'SDO']):\n data = states[state]\n trunc_df = data[data['PEF']==pef].copy()\n\n most_used = trunc_df['Product Name'].value_counts().index[:20]\n \n product_month = {}\n for product in most_used:\n product_month[product] = trunc_df[trunc_df['Product Name']== product].groupby('month')['lp_id'].count()\n product_month[product] = check(product_month[product])\n\n product_month = pd.DataFrame(product_month)\n\n average_plot = product_month.mean(axis=1)\n \n axes[i].plot(average_plot.index, average_plot.values, color=colors[j], label=pef);\n\n axes[i].set_title(state, fontsize=20, color=colors[j])\n axes[i].legend()","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:50:10.994576Z","iopub.execute_input":"2022-08-14T16:50:10.995525Z","iopub.status.idle":"2022-08-14T16:50:21.381344Z","shell.execute_reply.started":"2022-08-14T16:50:10.995488Z","shell.execute_reply":"2022-08-14T16:50:21.380262Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"From the plots above, we can observe that the usage of learning platforms has **three** common behaviour ,_after the drop we've taked before about_,:\n* The usage has decreased from its original value ( Before the drop happened ):\n - New York\n - WisConsin\n - Minnesota\n - Washington\n* The usage returns to its original value ( Before the drop happened ):\n - Utah\n - New Jersey\n - California\n* The usage has increased from its original value ( Before the drop happened ):\n - Illinois\n - Texas\n \n> The Last state `Indiana` was a little strange because the CM websites curve went up after the \"drop\" period but the other two curves went down.","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"# for memory usage\ndel states, product_month","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:50:21.383027Z","iopub.execute_input":"2022-08-14T16:50:21.383730Z","iopub.status.idle":"2022-08-14T16:50:21.536279Z","shell.execute_reply.started":"2022-08-14T16:50:21.383682Z","shell.execute_reply":"2022-08-14T16:50:21.535115Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's dig deeper and see if we can know the reason for the previous phenomena","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"to_investigate = [['New York', 'Wisconsin', 'Minnesota', 'Washington'],\n ['Utah', 'New Jersey', 'California'], \n ['Illinois', 'Texas']] \n\ntitles = ['Decreasing the usage from its original value (before the drop)',\n 'Returning the usage to its original value (before the drop)',\n 'Increasing the usage from its original value (before the drop)']","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:50:21.537796Z","iopub.execute_input":"2022-08-14T16:50:21.538194Z","iopub.status.idle":"2022-08-14T16:50:21.546485Z","shell.execute_reply.started":"2022-08-14T16:50:21.538158Z","shell.execute_reply":"2022-08-14T16:50:21.545358Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**From \"Locale\" Perspective**","metadata":{"execution":{"iopub.execute_input":"2021-09-15T14:49:37.760559Z","iopub.status.busy":"2021-09-15T14:49:37.760164Z","iopub.status.idle":"2021-09-15T14:49:37.76627Z","shell.execute_reply":"2021-09-15T14:49:37.76561Z","shell.execute_reply.started":"2021-09-15T14:49:37.760527Z"},"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"order = ['City', 'Town', 'Suburb', 'Rural']\n\ninvestigate('locale', (15,20), 'From \"Locale\" Perspective', order=order)","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:50:21.551310Z","iopub.execute_input":"2022-08-14T16:50:21.551882Z","iopub.status.idle":"2022-08-14T16:50:22.030977Z","shell.execute_reply.started":"2022-08-14T16:50:21.551847Z","shell.execute_reply":"2022-08-14T16:50:22.029819Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**We can see that:**
\n* **The states where the learning platforms usage has decreased from its original value ( Before the drop ) tends to have**:**
\n - Less towns (0% compared to 14% and 5%)
\n - More Rural areas (28% compared to 7% and 15%)
\n
\n* **The states where the learning platforms usage has returned to its original value ( Before the drop ) tends to have: **
\n - More towns (14% compared to 0% and 5%).
\n - Less Rural areas (7% compared to 28% and 15%)
\n
\n* **The states where the learning platforms usage has increased from its original value ( Before the drop ) tends to have**:**
\n - More Suburb (75% compared to 53% and 50%)
\n - Less Cities (5% compared to 26% and 22%)
","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"**From \"pct_black/hispanic\" Perspective**","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"order = ['[0, 0.2[', '[0.2, 0.4[', '[0.4, 0.6[', '[0.6, 0.8[', '[0.8, 1[']\n \ninvestigate('pct_black/hispanic', (15,20), 'From \"pct_black/hispanic\" Perspective', order=order)","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:50:22.032631Z","iopub.execute_input":"2022-08-14T16:50:22.033208Z","iopub.status.idle":"2022-08-14T16:50:22.575515Z","shell.execute_reply.started":"2022-08-14T16:50:22.033160Z","shell.execute_reply":"2022-08-14T16:50:22.574317Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**We can see that:**\n* **The states where the learning platforms usage has decreased from its original value ( Before the drop ) tends to have:**\n - A relatively low percentage of black/Hispanic (About 5.56% of their districts have these students with a percentage of 60% or higher).\n
\n* **The states where the learning platforms usage has returned to its original value ( Before the drop ) tends to have:**\n - A a bit higher percentage of black/Hispanic (About 9.31% of their districts have these students with a percentage of 60% or higher).\n
\n* **The states where the learning platforms usage has increased from its original value ( Before the drop ) tends to have:**\n - A relatively high percentage of black/Hispanic (About 25% of their districts have these students with a percentage of 60% or higher)\n \n_Note:_\n> The states where the learning platforms usage has decreased from its original value ( Before the drop ) have the least chance of having a district with 60% or higher of black/hispanic. which can indicate that may be the policies in that states (how they are treated by the law) are not the best or even some bad deeds from the white people there like: bullying and racism. All that we not give these students the right environment for learning.let us explain a bit more, If they have patients among their families or even themselves, these polices or bad deeds wouldn't let them be treated equally with white patients.
","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"**From \"pct_free/reduced\" Perspective**","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"order = ['[0, 0.2[', '[0.2, 0.4[', '[0.4, 0.6[', '[0.6, 0.8[', '[0.8, 1[']\n\ninvestigate('pct_free/reduced', (15,20), 'From \"pct_free/reduced\" Perspective', order=order)","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:50:22.576860Z","iopub.execute_input":"2022-08-14T16:50:22.577227Z","iopub.status.idle":"2022-08-14T16:50:23.167621Z","shell.execute_reply.started":"2022-08-14T16:50:22.577195Z","shell.execute_reply":"2022-08-14T16:50:23.166478Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**We can see that:**\n* **The states where the learning platforms usage has decreased from its original value ( Before the drop ) tends to have:**\n - A relatively high percentage of students eligible for free or reduced-price lunch (About 83.33% of their districts have these students with a percentage of 20% or higher).\n
\n* **The states where the learning platforms usage has returned to its original value ( Before the drop ) tends to have:**\n - A relatively moderate percentage of students eligible for free or reduced-price lunch (About 67.44% of their districts have these students with a percentage of 20% or higher).\n
\n* **The states where the learning platforms usage has increased from its original value ( Before the drop ) tends to have:**\n - A bit lower percentage of students eligible for free or reduced-price lunch (About 60% of their districts have these students with a percentage of 20% or higher).\n \n_Note:_\n> Having a large percentage of the students eligible for free or reduced-price lunches leads to lower usage of learning platforms. Maybe the reason is the cost of having good internet access (As during COVID-19 the load on the companies that provide internet has increased. Which made the service that has a humble cost isn't good anymore)","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"**From \"pp_total_raw\" Perspective**","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"order = ['[4000, 6000[', '[6000, 8000[', '[8000, 10000[', '[10000, 12000[', '[12000, 14000[', '[14000, 16000[', \n '[16000, 18000[', '[18000, 20000[']\n\ninvestigate('pp_total_raw', (15,20), 'From \"pp_total_raw\" Perspective', order=order)","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:50:23.169330Z","iopub.execute_input":"2022-08-14T16:50:23.169696Z","iopub.status.idle":"2022-08-14T16:50:23.944850Z","shell.execute_reply.started":"2022-08-14T16:50:23.169663Z","shell.execute_reply":"2022-08-14T16:50:23.943565Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**We can see that:**\n* **The states where the learning platforms usage has decreased from its original value ( Before the drop ) tends to have:**\n - Most of the expenditure lies between 14000-16000\\$\n
\n* **The states where the learning platforms usage has returned to its original value ( Before the drop ) tends to have:**\n - Most of the expenditure lies between 6000-10000\\$\n
\n* **The states where the learning platforms usage has increased from its original value ( Before the drop ) tends to have:**\n - Most of the expenditure lies between 12000-14000\\$\n \n_Note:_\n> It's not a must to increase the fees of a school to have a good student (that follows his lessons despite the circumstances) and of course we shouldn't lower it very much to provide a good education services.","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"### We've seen the impact of COVID-19 on different \"PEF\" and different states. What about Educational Sectors.","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"_Investigating the Impact of COVID-19 on each PEF for each sector separately_","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"sec_df = eng_product_merge.copy()\nsec_df['Sec'] = sec_df['Sector(s)'].str.split(';')\nsec_df = sec_df.explode('Sec')\nsec_df['Sec'] = sec_df['Sec'].str.strip()","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:50:23.946762Z","iopub.execute_input":"2022-08-14T16:50:23.947571Z","iopub.status.idle":"2022-08-14T16:51:09.449458Z","shell.execute_reply.started":"2022-08-14T16:50:23.947521Z","shell.execute_reply":"2022-08-14T16:51:09.448230Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig = plt.figure(figsize=(17, 40))\nfig.suptitle('Impact of COVID-19 on (\"LC\", \"CM\", \"SDO\") websites (Averages-from each sector perspective)', fontsize=20, weight='bold', y=0.91)\nfor i, sector in enumerate(sec_df['Sec'].unique()):\n plt.subplot(6, 1, i+1)\n data = sec_df[sec_df['Sec']==sector]\n \n for pef in ['CM','SDO','LC']:\n result = data[data['PEF']==pef].groupby('month')['lp_id'].count()\n plt.plot(result.index, result.values, label=pef)\n plt.title(sector, fontsize=12, color='brown')\n plt.legend()","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:51:09.450845Z","iopub.execute_input":"2022-08-14T16:51:09.451238Z","iopub.status.idle":"2022-08-14T16:51:26.018991Z","shell.execute_reply.started":"2022-08-14T16:51:09.451204Z","shell.execute_reply":"2022-08-14T16:51:26.017768Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"> As we can see the Effect of COVID-19 is almost the same for each Educational Sector.","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"# for memory usage\ndel sec_df","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:51:26.020523Z","iopub.execute_input":"2022-08-14T16:51:26.021580Z","iopub.status.idle":"2022-08-14T16:51:26.846541Z","shell.execute_reply.started":"2022-08-14T16:51:26.021544Z","shell.execute_reply":"2022-08-14T16:51:26.845427Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Let's investigate the usage of 6 random chosen sub categories of the three main categories. ","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"code","source":"for pef in ['LC', 'CM', 'SDO']:\n temp = eng_product_merge[eng_product_merge['PEF']==pef].copy()\n\n np.random.seed(42)\n random_chosen = np.random.choice(temp['sub_PEF'].unique(), size=6, replace=False)\n\n fig, axes = plt.subplots(nrows=2, ncols=3, figsize=(20, 8))\n\n fig.suptitle(pef, fontsize=15, y=0.96)\n\n axes = axes.ravel()\n for i, spef in enumerate(random_chosen):\n fig.sca(axes[i])\n\n sub_pef = temp[temp['sub_PEF']== spef].groupby('month')['lp_id'].count()\n sub_pef = check(sub_pef)\n\n plt.plot(sub_pef.index, sub_pef.values)\n plt.title(spef, color='brown')\n\n del temp\n del sub_pef\n","metadata":{"_kg_hide-input":true,"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[],"execution":{"iopub.status.busy":"2022-08-14T16:51:26.847802Z","iopub.execute_input":"2022-08-14T16:51:26.848650Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## LC \n>**Most of the subcategories increased from their original values after the drop or at least returns to their original state like:**
\n - Digital Learning Platform. \n - Sites, Resources & Reference Encyclopedia. \n - Study Tools.\n \n
\n\n>**So from the above plot we can conclude that:**
\n
  • Most of the subcategories,if not all, returned to their original values or even surpassed it (the peak is most likely to be in October) \n
  • Before the drop`The career planning and job search` subcategory was decreasing (may be that is because of the spread of COVID-19 and the Emergency Declaration at that time) But after that drop it went up rapidly and surpassed its previous state (And of course that is because people became open to the idea of working from home and how they can keep working despite the circumstances).\n","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"## CM \n>**As expected most of the subcategories increased from their original values after the drop like:**
    \n - Classroom Engagement & Instruction Assessment & Classroom Response. \n - Teacher Resources Professional Learning. \n - Virtual Classroom Video Conferencing & Screen Sharing.\n \n
    \n\n>**So from the above plot we can conclude that:**
    \n
  • Also here most of the subcategories returned to their original values or even surpassed it (the peak is most likely to be in October) except `Classroom Engagement & Instruction Communicaton & Messaging` which is kind of weird actually as `Classroom Engagement & Instruction Assessment & Classroom Response` or `Classroom engagement & Instruction Classroom Management` surpassed their orignal state before the drop which make us wonder why such a thing happend.If we think about it alittle deeper we will find that this is reasonable as `Classroom Engagement & Instruction Communicaton & Messaging` only providing one additional feature instead of Classroom engagment which is Communication and messaging but as we know it's not an important feature anymore we can do that by Facebook or Whatsapp.On the other hand, the `Classroom Engagement & Instruction Assessment & Classroom Response` or `Classroom engagement & Instruction Classroom Management` provides new features like Instruction assessment ( in the first one ) or Instruction Classroom management ( in the second one ).

    \n
  • The spread of virtual classroom video conferencing & screen sharing start increasing in March and it's still increasing and that's because of online learning from home. This is the way to communicate between the lecturer and their students","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"## SDO \n>**Also here most of the subcategories increased from their original values after the drop like:**
    \n - Data Analytics & Reporting Student Information Systems (SIS). \n - Environmental, Health & Safety (EHS) Compilance. \n - School Management Software SSO.\n \n
    \n\n>**So from the above plot we can conclude that:**
    \n
  • Also here most of the subcategories returned to their original values or even surpassed it (the peak is most likely to be in October) except `Learning Management Systems (LMS)`.By diving deeper into that subcategory we've found that its products provide features that are used in many other websites not only that but the other products provide more than that.
    \n
  • Products like \"SafeSchool\", \"Clever\",and \"Infinite Campus\" have got more attention because of COVID-19.
    ","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"

    5. Conclusion

    ","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"○ Products that are most used in 2020 are for the PreK-12 sector, and that’s good because if students in this sector adapted themselves to use online tools and new technologies, that will help them in the future. >> [Reference](#sec)
    \n\n○ Most of the products that are used fall under the service of LC (Learning & Curriculum). >> [Reference](#pef)
    \n\n○ For LC category: >> [Reference](#sub)\n>
  • Most of the subcategories,if not all, returned to their original values or even surpassed it (the peak is most likely to be in October) \n
  • Before the dropThe career planning and job search subcategory was decreasing (may be that is because of the spread of COVID-19 and the Emergency Declaration at that time) But after that drop it went up rapidly and surpassed its previous state (And of course that is because people became open to the idea of working from home and how they can keep working despite the circumstances).\n\n○ For CM category: >> [Reference](#sub)\n>
  • Also here most of the subcategories returned to their original values or even surpassed it (the peak is most likely to be in October) except Classroom Engagement & Instruction Communicaton & Messaging which is kind of weird actually as Classroom Engagement & Instruction Assessment & Classroom Response or Classroom engagement & Instruction Classroom Management surpassed their orignal state before the drop which make us wonder why such a thing happend.If we think about it alittle deeper we will find that this is reasonable as Classroom Engagement & Instruction Communicaton & Messaging only providing one additional feature instead of Classroom engagment which is Communication and messaging but as we know it's not an important feature anymore we can do that by Facebook or Whatsapp.On the other hand, the Classroom Engagement & Instruction Assessment & Classroom Response or Classroom engagement & Instruction Classroom Management provides new features like Instruction assessment ( in the first one ) or Instruction Classroom management ( in the second one ).

    \n
  • The spread of virtual classroom video conferencing & screen sharing start increasing in March and it's still increasing and that's because of online learning from home. This is the way to communicate between the lecturer and their students.\n \n○ For SDO category: >> [Reference](#sub)\n>
  • Also here most of the subcategories returned to their original values or even surpassed it (the peak is most likely to be in October) except Learning Management Systems (LMS).By diving deeper into that subcategory we've found that its products provide features that are used in many other websites not only that but the other products provide more than that.
    \n
  • Products like \"SafeSchool\", \"Clever\",and \"Infinite Campus\" have got more attention because of COVID-19.
    \n \n \n○ **The states where the learning platforms usage has decreased from its original value ( Before the drop ) tends to have the following features:**
    \n    • A relatively low percentage of black/Hispanic (About 5.56% of their districts have these students with a percentage of 60% or higher). >> [Reference](#bh)
    \n    • A relatively high percentage of students eligible for free or reduced-price lunch (About 83.33% of their districts have these students with a percentage of 20% or higher). >> [Reference](#fr) \n    • Most of the expenditure lies between 14000-16000\\$. >>[Reference](#tr)
    \n\n○ **The states where the learning platforms usage has returned to its original value ( Before the drop ) tends to have the following features:**
    \n    • A a bit higher percentage of black/Hispanic (About 9.31% of their districts have these students with a percentage of 60% or higher). >> [Reference](#bh)
    \n    • A relatively moderate percentage of students eligible for free or reduced-price lunch (About 67.44% of their districts have these students with a percentage of 20% or higher). >>[Reference](#fr)
    \n    • Most of the expenditure lies between 6000-10000\\$. >>[Reference](#tr)
    \n\n○ **The states where the learning platforms usage has increased from its original value ( Before the drop ) tends to have the following features:**
    \n    • A relatively high percentage of black/Hispanic (About 25% of their districts have these students with a percentage of 60% or higher). >> [Reference](#bh)
    \n    • A bit lower percentage of students eligible for free or reduced-price lunch (About 60% of their districts have these students with a percentage of 20% or higher). >>[Reference](#fr)
    \n    • Most of the expenditure lies between 12000-14000\\$ >>[Reference](#tr)
    ","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"**From that we can conclude the following:**\n\n> The states where the learning platforms usage has decreased from its original value ( Before the drop ) have the least chance of having a district with 60% or higher of black/hispanic >>[Reference](#bh)<< .which can indicate that may be the policies in that states (how they are treated by the law) are not the best or even some bad deeds from the white people there like: bullying and racism. All that we not give these students the right environment for learning.let us explain a bit more, If they have patients among their families or even themselves, these polices or bad deeds wouldn't let them be treated equally with white patients.
    \n\n> Having a large percentage of the students eligible for free or reduced-price lunches leads to lower usage of learning platforms. Maybe the reason is the cost of having good internet access (As during COVID-19 the load on the companies that provide internet has increased. Which made the service that has a humble cost isn't good anymore). >>[Reference](#fr)
    \n\n\n> It's not a must to increase the fees of a school to have a good student (that follows his lessons despite the circumstances) and of course we shouldn't lower it very much to provide a good education services. >>[Reference](#tr)
    ","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"**Suggested Solutions:**\n\n Human rights should be granted for every human,not just a word everybody say to seek a position.We don't know exactly how, we don't have such an experience in these kinda things but more efforts should be exerted to guarantee that every human had his own rights.
    \n\n> Poverity should not be an obstacle in the road of learning. This issue may be solved by:
    \n    • The material of each week (Lectures and assignment) can be provide in DVDs in the begging of each week so that it the student can't afford a good internet connect he can buy this DVD.
    \n    • If these people have something to identify them ( like a card or something ) we can provide them with a place with a good internet connection ( in the school or even in every district )
    \n\n> A limit should be put to school expenditure so that the fees doesn't go up. May be high fees makes it hard to a family to afford a good internet connection. Or may be high fees makes the school for rich people only and that will upset poor people which, in turn, makes them wants to work instead of leaning to make more money.
    ","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}},{"cell_type":"markdown","source":"Thank you for keeping reading the notebook to the very end, I hope you enjoyed, and I would be grateful to receive your feedback.","metadata":{"papermill":{"duration":null,"end_time":null,"exception":null,"start_time":null,"status":"pending"},"tags":[]}}]}