{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.13","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"none","dataSources":[{"sourceId":35332,"databundleVersionId":3723648,"sourceType":"competition"}],"dockerImageVersionId":30732,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":false}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Yarışma Hakkında Bilgi ##","metadata":{}},{"cell_type":"markdown","source":"- **Yarışma Başlangıcı**: 25 Mayıs 2022'de başlayan **American Express - Default Prediction** yarışması, katılımcıların gelecekteki müşteri temerrütlerini tahmin etmelerini amaçlamaktadır.\n- **Değerlendirme Metriği**: Yarışmanın değerlendirme metriği, Normalleştirilmiş Gini Katsayısı ve %4'lük varsayılan oranı içeren bir formüldür: $$ M = 0.5 \\cdot (G + D) $$.\n- **Ödüller ve İşe Alım**: Kazananlara toplamda **100,000 dolar** ödül dağıtılacak ve yüksek sıralamada olan yarışmacılar American Express tarafından iş görüşmesi için değerlendirilecek.\n- **Katılım ve İstatistikler**: Yarışmaya **29,924 giriş**, **6,003 katılımcı** ve **4,874 takım** katılmıştır.","metadata":{}},{"cell_type":"markdown","source":"Addison Howard, AritraAmex, Di Xu, Hossein Vashani, inversion, Negin, Sohier Dane. (2022). American Express - Default Prediction. Kaggle. https://kaggle.com/competitions/amex-default-prediction","metadata":{}},{"cell_type":"markdown","source":"## Kütüphanleri Kur & Yükle ##","metadata":{}},{"cell_type":"markdown","source":"### Veri Seti Çok Büyük\n\nSpark üzerinden ML yapılacaktır.","metadata":{"execution":{"iopub.status.busy":"2024-06-21T09:58:47.388007Z","iopub.execute_input":"2024-06-21T09:58:47.389549Z","iopub.status.idle":"2024-06-21T09:58:47.39492Z","shell.execute_reply.started":"2024-06-21T09:58:47.389503Z","shell.execute_reply":"2024-06-21T09:58:47.393528Z"}}},{"cell_type":"code","source":"import re\nfrom pyspark import SparkContext\nfrom pyspark.sql import SparkSession\nfrom pyspark.sql.types import *\nfrom pyspark.ml.feature import VectorAssembler\nfrom pyspark.ml.classification import LogisticRegression\nfrom pyspark.ml.evaluation import BinaryClassificationEvaluator","metadata":{"execution":{"iopub.status.busy":"2024-06-21T13:36:12.155637Z","iopub.execute_input":"2024-06-21T13:36:12.156231Z","iopub.status.idle":"2024-06-21T13:36:12.840216Z","shell.execute_reply.started":"2024-06-21T13:36:12.156183Z","shell.execute_reply":"2024-06-21T13:36:12.838688Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Spark'ı Başlat","metadata":{}},{"cell_type":"code","source":"!pip install spark","metadata":{"execution":{"iopub.status.busy":"2024-06-21T13:36:14.771733Z","iopub.execute_input":"2024-06-21T13:36:14.772378Z","iopub.status.idle":"2024-06-21T13:36:33.315251Z","shell.execute_reply.started":"2024-06-21T13:36:14.772341Z","shell.execute_reply":"2024-06-21T13:36:33.313435Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sc = SparkContext.getOrCreate()\nspark = SparkSession.builder.getOrCreate()","metadata":{"execution":{"iopub.status.busy":"2024-06-21T13:36:33.317692Z","iopub.execute_input":"2024-06-21T13:36:33.31817Z","iopub.status.idle":"2024-06-21T13:36:41.58281Z","shell.execute_reply.started":"2024-06-21T13:36:33.318126Z","shell.execute_reply":"2024-06-21T13:36:41.581142Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Veri Setini Yükle ##","metadata":{"execution":{"iopub.status.busy":"2024-06-21T10:02:13.802168Z","iopub.execute_input":"2024-06-21T10:02:13.802673Z","iopub.status.idle":"2024-06-21T10:02:13.810461Z","shell.execute_reply.started":"2024-06-21T10:02:13.802637Z","shell.execute_reply":"2024-06-21T10:02:13.809062Z"}}},{"cell_type":"markdown","source":"## Eğitim Veri Setini Yükle ##","metadata":{}},{"cell_type":"code","source":"%time\ntrain=spark.read.csv(\"/kaggle/input/amex-default-prediction/train_data.csv\",header=True,inferSchema=True)\ntrain.show()","metadata":{"execution":{"iopub.status.busy":"2024-06-21T13:38:49.19164Z","iopub.execute_input":"2024-06-21T13:38:49.192232Z","iopub.status.idle":"2024-06-21T13:43:12.864773Z","shell.execute_reply.started":"2024-06-21T13:38:49.192186Z","shell.execute_reply":"2024-06-21T13:43:12.863337Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Eğitm Seti Etiketlerini Yükle ###","metadata":{}},{"cell_type":"code","source":"train_labels=spark.read.csv(\"/kaggle/input/amex-default-prediction/train_labels.csv\",header=True, inferSchema=True)\ntrain_labels.show()","metadata":{"execution":{"iopub.status.busy":"2024-06-21T13:43:12.867089Z","iopub.execute_input":"2024-06-21T13:43:12.867584Z","iopub.status.idle":"2024-06-21T13:43:14.183567Z","shell.execute_reply.started":"2024-06-21T13:43:12.867536Z","shell.execute_reply":"2024-06-21T13:43:14.182295Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Eğitim Seti ile Etiketleri Birleştir ##","metadata":{}},{"cell_type":"code","source":"train=train.join(train_labels, on=\"customer_ID\")\ntrain.show()","metadata":{"execution":{"iopub.status.busy":"2024-06-21T13:43:14.184952Z","iopub.execute_input":"2024-06-21T13:43:14.185427Z","iopub.status.idle":"2024-06-21T13:46:28.668001Z","shell.execute_reply.started":"2024-06-21T13:43:14.185388Z","shell.execute_reply":"2024-06-21T13:46:28.662125Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Test Versini Yükle ##","metadata":{}},{"cell_type":"code","source":"test=spark.read.csv(\"/kaggle/input/amex-default-prediction/test_data.csv\",header=True, inferSchema=True)\ntest.show()","metadata":{"execution":{"iopub.status.busy":"2024-06-21T13:46:28.672602Z","iopub.status.idle":"2024-06-21T13:46:28.673744Z","shell.execute_reply.started":"2024-06-21T13:46:28.67337Z","shell.execute_reply":"2024-06-21T13:46:28.673403Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## SparK EDA #","metadata":{}},{"cell_type":"code","source":"## Spark EDA ##\ntrain.describe().toPandas().transpose()","metadata":{"execution":{"iopub.status.busy":"2024-06-21T13:46:28.675539Z","iopub.status.idle":"2024-06-21T13:46:28.67619Z","shell.execute_reply.started":"2024-06-21T13:46:28.675843Z","shell.execute_reply":"2024-06-21T13:46:28.675867Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Özellikleri ve Hedef Sütunu Seçme ##","metadata":{}},{"cell_type":"code","source":"#özellikleri seçme\npredictors=train.columns\npredictors.remove('customer_ID')\npredictors.remove('S_2')\npredictors.remove('target')","metadata":{"execution":{"iopub.status.busy":"2024-06-21T13:46:28.677515Z","iopub.status.idle":"2024-06-21T13:46:28.678126Z","shell.execute_reply.started":"2024-06-21T13:46:28.677795Z","shell.execute_reply":"2024-06-21T13:46:28.677818Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predictors.remove('D_63')\npredictors.remove('D_64')","metadata":{"execution":{"iopub.status.busy":"2024-06-21T13:46:28.679419Z","iopub.status.idle":"2024-06-21T13:46:28.680101Z","shell.execute_reply.started":"2024-06-21T13:46:28.679785Z","shell.execute_reply":"2024-06-21T13:46:28.679809Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Özellikleri Vektöre Çevir ##","metadata":{}},{"cell_type":"code","source":"#Kategorik sütunlar\n#from pyspark.sql.types import StringType\n\n#cat_cols = [col for col, dtype in train.dtypes if dtype == 'string']\n#cat_cols","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Kategorik Sütunlar Label Encoding ile sayısal verilere çevir\n#cat_features = ['D_63', 'D_64']\n\n#from pyspark.ml.feature import StringIndexer\n#indexer = StringIndexer(inputCols=cat_features, outputCols=[f\"{c}_indexed\" for c in cat_features])\n#train_leo = indexer.fit(train).transform(train)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#train_leo.show(5)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Özellikleri vektöre çevir \nassembler=VectorAssembler(inputCols=predictors,outputCol='features')\ntrain=assembler.transform(train).select('features','target')","metadata":{"execution":{"iopub.status.busy":"2024-06-21T13:46:28.683606Z","iopub.status.idle":"2024-06-21T13:46:28.69371Z","shell.execute_reply.started":"2024-06-21T13:46:28.693239Z","shell.execute_reply":"2024-06-21T13:46:28.693282Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Veri Setini Böl ##","metadata":{}},{"cell_type":"code","source":"train_data,test_data=train.randomSplit([0.8,0.2], seed=42)","metadata":{"execution":{"iopub.status.busy":"2024-06-21T13:46:28.695729Z","iopub.status.idle":"2024-06-21T13:46:28.696604Z","shell.execute_reply.started":"2024-06-21T13:46:28.696255Z","shell.execute_reply":"2024-06-21T13:46:28.696285Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Modeli OLuştur ve Eğit ##","metadata":{}},{"cell_type":"code","source":"lr=LogisticRegression(featuresCol='features',labelCol='target',maxIter=10)\nmodel=lr.fit(train_data)","metadata":{"execution":{"iopub.status.busy":"2024-06-21T13:46:28.698812Z","iopub.status.idle":"2024-06-21T13:46:28.699785Z","shell.execute_reply.started":"2024-06-21T13:46:28.699431Z","shell.execute_reply":"2024-06-21T13:46:28.699458Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Modeli Test Et & Değerlendir ##","metadata":{}},{"cell_type":"code","source":"# Veri seti üzerinde tahminleri yap \npredictions=model.transform(test_data)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"evaluator=BinaryClassificationEvaluator(labelCol='target')\nevaluator.evaluate(predictions)\nprint('Accuracy:',evaluator.evaluate(predictions))","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Test setinde tahminler yap ve gönder##","metadata":{}},{"cell_type":"code","source":"# Değerlendir\nsub=pd.read_csv('/kaggle/input/amex-default-prediction/sample_submission.csv')\nsub.head()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"predictions=model.transform(test)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Tahminleri ekle","metadata":{}},{"cell_type":"code","source":"sub['prediction'] = predictions.select('prediction').collect()\nsub.head()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Submission Dosyasını Oluştur ve Gönder","metadata":{}},{"cell_type":"code","source":"sub.to_csv('submission.csv',index=False)","metadata":{},"execution_count":null,"outputs":[]}]}