{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"### **BayesianGMMClassifier**","metadata":{}},{"cell_type":"markdown","source":"Public LB Score : 0.77025","metadata":{}},{"cell_type":"markdown","source":"#### References:\n\n* https://scikit-lego.netlify.app/api/mixture.html\n* https://scikit-lego.netlify.app/_modules/sklego/mixture/bayesian_gmm_classifier.html#BayesianGMMClassifier\n\n#### Credits:\n\n* https://www.kaggle.com/code/pourchot/simple-soft-voting\n* https://www.kaggle.com/code/ricopue/tps-jul22-clusters-and-lgb\n* https://www.kaggle.com/code/hiro5299834/tps-jul-2022-unsupervised-and-supervised-learning\n","metadata":{}},{"cell_type":"markdown","source":"The BayesianGMMClassifier trains a Gaussian Mixture Model for each class in y on a dataset X. Once a density is trained for each class we can evaluate the likelihood scores to see which class is more likely. All parameters of the model are an exact copy of the parameters in scikit-learn.\n\nFollowing all the productive discussions and reviewing each notebook that was shared within this competition a decent model was produced with a decent public LB score. This output was used to feed a Classifier developed by scikit-lego.  There is a big probability that this method is overfitting the LB sample but it is a good start. \n\nAlso note that to utilise k-means++ as init_params in BGMM you need to have sci-kit learn v 1.1.","metadata":{}},{"cell_type":"markdown","source":"#### *Libraries*","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\n\n\n# For Visualization\nimport matplotlib.pyplot as plt\nimport seaborn as sns\n\nfrom sklearn.mixture import BayesianGaussianMixture\nfrom sklego.mixture import BayesianGMMClassifier\nfrom sklearn.preprocessing import PowerTransformer","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Reading the dataset\n\ndata = pd.read_csv(\"C:/Users/karlc/.jupyter/lab/Notebooks/Tabular_Jul22/data.csv\", index_col='id')\nsubmission = pd.read_csv(\"C:/Users/karlc/.jupyter/lab/Notebooks/Tabular_Jul22/sample_submission.csv\")","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"best_data =['f_07','f_08', 'f_09', 'f_10','f_11', 'f_12', 'f_13', 'f_22','f_23', 'f_24', 'f_25','f_26','f_27', 'f_28']","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"data_scaled = pd.DataFrame(PowerTransformer().fit_transform(data), columns=data.columns)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Loading decent score submission\npred_test = pd.read_csv(\"C:/Users/karlc/.jupyter/lab/Notebooks/Tabular_Jul22/pred_test_bestdata_kmeasnplus200.csv\", index_col=[0])\npredict_soft = np.argmax(np.array(pred_test), axis=1)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"","metadata":{}},{"cell_type":"code","source":"X = np.array(data_scaled[best_data])\ny = np.array(predict_soft)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"for seed in tqdm(range(0,1)):\n    \n       \n    bgm = BayesianGMMClassifier(\n            n_components=7,\n            random_state = seed,\n            tol =1e-3,\n            covariance_type = 'full',\n            max_iter = 200,\n            n_init=3,\n            init_params='k-means++'\n                     )\n               \n\n    \n    # fitting and probability prediction\n    bgm.fit(X,y)\n    predict = bgm.predict(X)\n    pred_seed = bgm.predict_proba(X) ","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.set(rc={'figure.figsize':(15,10)})\npl = sns.countplot(x=predict)\npl.set_title(\"Distribution of clusters - Best features\")\nplt.show()","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"submission['Predicted'] = predict\nsubmission.to_csv('TabJul22_sub3_bgmc_220722.csv', index=False)\nsubmission.head(20)","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Other improvements:**\n\n* Possible integration and/or ensembling with other classifiers\n* Tuning BGMM hyperparameters\n\nHappy Kaggling !\n","metadata":{}}]}