{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python Docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load\n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the read-only \"../input/\" directory\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n        \ndata_path = '/kaggle/input/competitive-data-science-predict-future-sales/'\n\nsales_train = pd.read_csv(data_path + 'sales_train.csv')\nshops = pd.read_csv(data_path + 'shops.csv')\nitems = pd.read_csv(data_path + 'items.csv')\nitem_categories = pd.read_csv(data_path + 'item_categories.csv')\ntest = pd.read_csv(data_path + 'test.csv')\nsubmission = pd.read_csv(data_path + 'sample_submission.csv')\n\n# You can write up to 20GB to the current directory (/kaggle/working/) that gets preserved as output when you create a version using \"Save & Run All\" \n# You can also write temporary files to /kaggle/temp/, but they won't be saved outside of the current session","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-08-07T07:55:52.563541Z","iopub.execute_input":"2022-08-07T07:55:52.563935Z","iopub.status.idle":"2022-08-07T07:55:53.711617Z","shell.execute_reply.started":"2022-08-07T07:55:52.563881Z","shell.execute_reply":"2022-08-07T07:55:53.710646Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"1. 기본 데이터","metadata":{}},{"cell_type":"code","source":"sales_train.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T07:55:53.713509Z","iopub.execute_input":"2022-08-07T07:55:53.713883Z","iopub.status.idle":"2022-08-07T07:55:53.727305Z","shell.execute_reply.started":"2022-08-07T07:55:53.713848Z","shell.execute_reply":"2022-08-07T07:55:53.725536Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"date_block_num: 월까지를 나타낸 순서표. 2013년 1월은 0번, 2013년 2월은 2번 이런 식....\n\n-> 월별 판매량 예측이 문제에서 구하는 것이므로 date는 필요 없어서 제거할 피처\n\nshop_id는 상점 id, item_id는 상품 id, item_price는 가격\n\nitem_cnt_day: 당일 판매량 -> 월별 판매량이 아님 -> date_block_num을 기준으로 item_cnt_day 모두 합치기\n","metadata":{}},{"cell_type":"markdown","source":"info를 불러서 비결측값 개수를 알아볼 것인데 행이나 열이 너무 많으면 출력되지 않음 -> show_counts = True를 전달","metadata":{}},{"cell_type":"code","source":"sales_train.info(show_counts = True)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T07:55:53.728993Z","iopub.execute_input":"2022-08-07T07:55:53.730166Z","iopub.status.idle":"2022-08-07T07:55:53.866632Z","shell.execute_reply.started":"2022-08-07T07:55:53.730084Z","shell.execute_reply":"2022-08-07T07:55:53.865470Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"모든 피처의 non_null 개수가 전체 데이터 수와 같으므로 모든 피처에 결측값이 없다는 것을 알 수 있음\n\n다만 데이터 수가 300만개정도나 되어서 메모리를 관리할 필요성이 있어 보임","metadata":{}},{"cell_type":"markdown","source":"본 데이터는 시계열 데이터(시간 순으로 기록)이므로 시간의 흐름이 중요함! 따라서 oof같이 여러 폴드로 나누어서 검증하면 과거와 미래가 뒤섞이기 때문에 가장 마지막인 15년 10월 데이터를 검증 데이터로 지정","metadata":{}},{"cell_type":"markdown","source":"2. 상점 데이터","metadata":{}},{"cell_type":"code","source":"shops.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T07:55:53.869573Z","iopub.execute_input":"2022-08-07T07:55:53.870625Z","iopub.status.idle":"2022-08-07T07:55:53.880253Z","shell.execute_reply.started":"2022-08-07T07:55:53.870589Z","shell.execute_reply":"2022-08-07T07:55:53.879032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"상점 명에서 새로운 피처를 구하 수 있지만 러시아어이므로 다른 캐글러의 아이딩를 서치해서 활용 (상점명의 첫 단어는 도시 이름을 뜻함) -> 첫 단어 추출해서 도시 피처 만들기\n\nshop_id는 train 데이터에도 있으므로 shop_id를 기준으로 train과 shops 병합 가능","metadata":{}},{"cell_type":"code","source":"shops.info()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T07:55:53.881816Z","iopub.execute_input":"2022-08-07T07:55:53.882379Z","iopub.status.idle":"2022-08-07T07:55:53.894346Z","shell.execute_reply.started":"2022-08-07T07:55:53.882346Z","shell.execute_reply":"2022-08-07T07:55:53.893359Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"3. 상품 데이터","metadata":{}},{"cell_type":"code","source":"items.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T07:55:53.895751Z","iopub.execute_input":"2022-08-07T07:55:53.896500Z","iopub.status.idle":"2022-08-07T07:55:53.908851Z","shell.execute_reply.started":"2022-08-07T07:55:53.896467Z","shell.execute_reply":"2022-08-07T07:55:53.907860Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"item_id를 기준으로 sales_train과 items를 합칠 수 있음","metadata":{}},{"cell_type":"code","source":"items.info()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T07:55:53.910161Z","iopub.execute_input":"2022-08-07T07:55:53.911083Z","iopub.status.idle":"2022-08-07T07:55:53.926298Z","shell.execute_reply.started":"2022-08-07T07:55:53.911051Z","shell.execute_reply":"2022-08-07T07:55:53.925109Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"4. 상품 분류 데이터","metadata":{}},{"cell_type":"code","source":"item_categories.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T07:55:53.927611Z","iopub.execute_input":"2022-08-07T07:55:53.928050Z","iopub.status.idle":"2022-08-07T07:55:53.937698Z","shell.execute_reply.started":"2022-08-07T07:55:53.928014Z","shell.execute_reply":"2022-08-07T07:55:53.936617Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"item_category_id를 기준으로 sales_train과 item_categories 병합\n\n상품분류명의 첫 단어는 대분류 의미함 -> 대분류 피처로 만들 수 있음","metadata":{}},{"cell_type":"code","source":"item_categories.info()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T07:55:53.939374Z","iopub.execute_input":"2022-08-07T07:55:53.940562Z","iopub.status.idle":"2022-08-07T07:55:53.953490Z","shell.execute_reply.started":"2022-08-07T07:55:53.940529Z","shell.execute_reply":"2022-08-07T07:55:53.951963Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"6. 테스트 데이터","metadata":{}},{"cell_type":"code","source":"test.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T07:55:53.957783Z","iopub.execute_input":"2022-08-07T07:55:53.958086Z","iopub.status.idle":"2022-08-07T07:55:53.969139Z","shell.execute_reply.started":"2022-08-07T07:55:53.958051Z","shell.execute_reply":"2022-08-07T07:55:53.968133Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 데이터 병합\n\nmerge를 히용하여 '하나 이상의 열을 기준으로 dataframe 행을 합치기'\n\n기준이 되는 dataframe에서 merge 호출하고 병합할 dataframe을 인수로 넣기, on 파라미터에는 벙합 기준이 되는 피처 넣기, how 파라미터에 left를 전달하면 왼 쪽 dataframe의 모든 행을 포함하는 결과 반환함","metadata":{}},{"cell_type":"code","source":"train = sales_train.merge(shops, on = 'shop_id', how = 'left')\ntrain = train.merge(items, on = 'item_id', how = 'left')\ntrain = train.merge(item_categories, on = 'item_category_id', how = 'left')\n\ntrain.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T07:55:53.970634Z","iopub.execute_input":"2022-08-07T07:55:53.971582Z","iopub.status.idle":"2022-08-07T07:55:55.419870Z","shell.execute_reply.started":"2022-08-07T07:55:53.971548Z","shell.execute_reply":"2022-08-07T07:55:55.418844Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# 피처 요약표 만들기","metadata":{}},{"cell_type":"code","source":"def resumtable(df): \n    print(f'데이터셋 형상: {df.shape}')\n    summary = pd.DataFrame(df.dtypes, columns = ['데이터 타입'])\n    summary = summary.reset_index()\n    summary = summary.rename(columns = {'index': '피처'})\n    summary['결측값 개수'] = df.isnull().sum().values\n    summary['고윳값 개수'] = df.nunique().values\n    summary['첫 번째 값'] = df.loc[0].values\n    summary['두 번째 값'] = df.loc[1].values\n    \n    return summary\n\nresumtable(train)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T08:00:28.774901Z","iopub.execute_input":"2022-08-07T08:00:28.775466Z","iopub.status.idle":"2022-08-07T08:00:30.657970Z","shell.execute_reply.started":"2022-08-07T08:00:28.775430Z","shell.execute_reply":"2022-08-07T08:00:30.656881Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**_id와 **_name의 데이터 수가 같다: 일대일 매칭이다 \n\n-> 하나만 남겨두어도 되지만, name을 이용하여 모델링에 도움 되는 파생 피처를 만들 수 있는 경우도 있음 -> improved에서 다루기","metadata":{}},{"cell_type":"markdown","source":"# 데이터 시각화","metadata":{}},{"cell_type":"markdown","source":"일별 판매량 시각화","metadata":{}},{"cell_type":"code","source":"import seaborn as sns\nimport matplotlib as mpl\nimport matplotlib.pyplot as plt\n%matplotlib inline\n\nsns.boxplot(y= 'item_cnt_day', data = train)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T08:05:18.123356Z","iopub.execute_input":"2022-08-07T08:05:18.123719Z","iopub.status.idle":"2022-08-07T08:05:19.280467Z","shell.execute_reply.started":"2022-08-07T08:05:18.123690Z","shell.execute_reply":"2022-08-07T08:05:19.279555Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"이상치가 많아서 모양이 이상함: 나머지 부분이 납작해짐 -> 1000이상 정도는 이상치로 보고 제거하기","metadata":{}},{"cell_type":"markdown","source":"상품 가격 시각화","metadata":{}},{"cell_type":"code","source":"sns.boxplot(y = 'item_price', data = train)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T08:08:50.979950Z","iopub.execute_input":"2022-08-07T08:08:50.980308Z","iopub.status.idle":"2022-08-07T08:08:51.641835Z","shell.execute_reply.started":"2022-08-07T08:08:50.980281Z","shell.execute_reply":"2022-08-07T08:08:51.640946Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"이상치 제거: 판매가 50000 이상","metadata":{}},{"cell_type":"markdown","source":"그룹화: 특정 피처를 기준으로 데이터 그룹화하여 그리기\n\ntrain의 data_block_num 피처를 기준으로 그룹화해서 item_cnt_day 피처 값의 합을 구하기","metadata":{}},{"cell_type":"code","source":"group = train.groupby('date_block_num').agg({'item_cnt_day':'sum'}) \n#groupby: 기준 피처, agg()메서드 이용하여 item_cnt_day 피처에 sum 함수 적용\n\ngroup.reset_index() # 인덱스 재설정\n\n# 왜 월별 판매량을 새 피처로 만들지는 않는 것이지???","metadata":{"execution":{"iopub.status.busy":"2022-08-07T08:23:36.811758Z","iopub.execute_input":"2022-08-07T08:23:36.812228Z","iopub.status.idle":"2022-08-07T08:23:36.883306Z","shell.execute_reply.started":"2022-08-07T08:23:36.812170Z","shell.execute_reply":"2022-08-07T08:23:36.882162Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"groupby에 적용 가능한 집계 함수 예시\n\nsum, mean, median(중간값), min, max, std(표준편차), var(분산), count(개수)","metadata":{}},{"cell_type":"code","source":"mpl.rc('font', size = 13)\nfigure, ax = plt.subplots()\nfigure.set_size_inches(11, 5)\n\ngroup_month_sum = train.groupby('date_block_num').agg({'item_cnt_day': sum})\ngroup_month_sum = group_month_sum.reset_index()\n\nsns.barplot(x = 'date_block_num', y = 'item_cnt_day', data = group_month_sum)\n\nax.set(title = 'Distribution of monthly item counts by date block number', \n      xlabel = 'Data block number', \n      ylabel = 'Monthly item counts');","metadata":{"execution":{"iopub.status.busy":"2022-08-07T08:32:35.159578Z","iopub.execute_input":"2022-08-07T08:32:35.159972Z","iopub.status.idle":"2022-08-07T08:32:35.588198Z","shell.execute_reply.started":"2022-08-07T08:32:35.159936Z","shell.execute_reply":"2022-08-07T08:32:35.587276Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"11번: 12월, 23번: 12월 -> 연말이라 판매량 급증","metadata":{}},{"cell_type":"markdown","source":"# 상품 분류별 판매량","metadata":{}},{"cell_type":"code","source":"# 상품 분류 : 피처 고윳값 개수 구하여 알아내기\n\ntrain['item_category_id'].nunique()","metadata":{"execution":{"iopub.status.busy":"2022-08-07T08:36:50.581189Z","iopub.execute_input":"2022-08-07T08:36:50.581554Z","iopub.status.idle":"2022-08-07T08:36:50.603802Z","shell.execute_reply.started":"2022-08-07T08:36:50.581525Z","shell.execute_reply":"2022-08-07T08:36:50.602820Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"84개 상품에 대햇서 다 그리기에는 너무 많으니 판매량이 10000개를 초과하는 상품에 대해서만 추출하여 그리기","metadata":{}},{"cell_type":"code","source":"figure, ax = plt.subplots()\nfigure.set_size_inches(11, 5)\n\ngroup_cat_sum = train.groupby('item_category_id').agg({'item_cnt_day': 'sum'})\ngroup_cat_sum = group_cat_sum.reset_index()\n\ngroup_cat_sum = group_cat_sum[group_cat_sum['item_cnt_day']>10000]\n\nsns.barplot(x = 'item_category_id', y= 'item_cnt_day', data = group_cat_sum)\nax.set(title = 'Distribution of total item counts by item category id', \n      xlabel = 'item category id', \n      ylabel = 'Total item counts')\nax.tick_params(axis = 'x', labelrotation = 90)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T08:49:56.933816Z","iopub.execute_input":"2022-08-07T08:49:56.934414Z","iopub.status.idle":"2022-08-07T08:49:57.441768Z","shell.execute_reply.started":"2022-08-07T08:49:56.934379Z","shell.execute_reply":"2022-08-07T08:49:57.440842Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"다그리기","metadata":{}},{"cell_type":"code","source":"figure, ax = plt.subplots()\nfigure.set_size_inches(25, 5)\n\ngroup_cat_sum = train.groupby('item_category_id').agg({'item_cnt_day': 'sum'})\ngroup_cat_sum = group_cat_sum.reset_index()\n\nsns.barplot(x = 'item_category_id', y= 'item_cnt_day', data = group_cat_sum)\nax.set(title = 'Distribution of total item counts by item category id', \n      xlabel = 'item category id', \n      ylabel = 'Total item counts')\nax.tick_params(axis = 'x', labelrotation = 90)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T08:50:01.015824Z","iopub.execute_input":"2022-08-07T08:50:01.016407Z","iopub.status.idle":"2022-08-07T08:50:02.548859Z","shell.execute_reply.started":"2022-08-07T08:50:01.016371Z","shell.execute_reply":"2022-08-07T08:50:02.547398Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"특별히 잘 팔리는 몇몇 아이템이 있는 모양임","metadata":{}},{"cell_type":"markdown","source":"# 상점별 판매량","metadata":{}},{"cell_type":"code","source":"figure, ax = plt.subplots()\nfigure.set_size_inches(11, 5)\n\ngroup_shop_sum = train.groupby('shop_id').agg({'item_cnt_day': 'sum'})\ngroup_shop_sum = group_shop_sum.reset_index()\n\ngroup_shop_sum = group_shop_sum[group_shop_sum['item_cnt_day']>10000]\n\nsns.barplot(x = 'shop_id', y= 'item_cnt_day', data = group_shop_sum)\nax.set(title = 'Distribution of total item counts by shop id', \n      xlabel = 'shop id', \n      ylabel = 'Total item counts')\nax.tick_params(axis = 'x', labelrotation = 90)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T08:50:21.270178Z","iopub.execute_input":"2022-08-07T08:50:21.270536Z","iopub.status.idle":"2022-08-07T08:50:22.111776Z","shell.execute_reply.started":"2022-08-07T08:50:21.270506Z","shell.execute_reply":"2022-08-07T08:50:22.110801Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"figure, ax = plt.subplots()\nfigure.set_size_inches(20, 5)\n\ngroup_shop_sum = train.groupby('shop_id').agg({'item_cnt_day': 'sum'})\ngroup_shop_sum = group_shop_sum.reset_index()\n\nsns.barplot(x = 'shop_id', y= 'item_cnt_day', data = group_shop_sum)\nax.set(title = 'Distribution of total item counts by shop id', \n      xlabel = 'shop id', \n      ylabel = 'Total item counts')\nax.tick_params(axis = 'x', labelrotation = 90)","metadata":{"execution":{"iopub.status.busy":"2022-08-07T08:51:09.171453Z","iopub.execute_input":"2022-08-07T08:51:09.171814Z","iopub.status.idle":"2022-08-07T08:51:09.916300Z","shell.execute_reply.started":"2022-08-07T08:51:09.171785Z","shell.execute_reply":"2022-08-07T08:51:09.914138Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"10000개 이하로 판 상점들도 있음, 31번, 25번 등 몇몇 상점이 특출나게 잘 팜. ","metadata":{}}]}