{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "churn-00",
   "metadata": {},
   "source": "# Predict which customers may leave with Python\n\n**Your first AutoML classification model: customer churn**\n\nImagine you work for a phone company. You have a spreadsheet of how customers\nused the service, and you want to identify customers who might leave. Could\nprevious usage help you decide which customers deserve a closer look?\n\nWe will train a model to estimate **churn**, meaning a customer leaving the\nservice. Predicting a category is called *classification*. Our two categories\nare stayed and churned. We will also predict a probability, then check how often\nthe model misses customers who leave or flags customers who stay.\n\nYou will save a model, read its report, and reuse it for another CSV. The example\nteaches prediction; it does not show that contacting someone will prevent churn.\n\n## First, look at the customers\n\nWe use [UCI's Iranian Churn dataset](https://doi.org/10.24432/C5JW3Z). It contains\n3,150 customer records. These are five real rows, showing a few of the columns:\n\n| Subscription (months) | Call seconds | Text messages | Tariff | Churn |\n| ---: | ---: | ---: | --- | ---: |\n| 38 | 4370 | 5 | Pay as you go | 0 |\n| 39 | 318 | 7 | Pay as you go | 0 |\n| 37 | 2453 | 359 | Pay as you go | 0 |\n| 38 | 4198 | 1 | Pay as you go | 0 |\n| 38 | 2393 | 2 | Pay as you go | 0 |\n\nThe first customer had subscribed for 38 months and did not churn. `Churn` is\nour **target**: 1 means churned and 0 means stayed. The other selected columns\nare **inputs**, also called features. Never include the target among the inputs\nwhen asking the model for a new prediction.\n\nOnly **495 of 3,150 rows (15.7%)** have churn equal to 1. Predicting “stayed” for\neveryone would therefore look accurate while missing every customer who leaves.\nThat is why we will look beyond accuracy.\n\nThe publisher describes inputs from the first nine months and outcomes at month\ntwelve. There are no dates or customer IDs in the downloaded file. Our split\nchecks held-out profiles from this historical sample, not performance on a later\nperiod. See the [dataset review](https://supervised.mljar.com/assets/customer-churn/dataset-review/).\n"
  },
  {
   "cell_type": "markdown",
   "id": "churn-01",
   "metadata": {},
   "source": "## Set up your notebook\n\nUse Python 3.11 or 3.12. Install into the notebook's environment before launching it:\n\n```bash\npython -m pip install 'mljar-supervised==1.4.0b1' jupyterlab ipywidgets\npython -m jupyterlab\n```\n\nRun cells from top to bottom. This notebook includes all code; it does not need a\nhelper script. Standard CPU algorithms use a 120-second training budget and two\nthreads; total time also includes imports, reports, and reload. There is no GPU\nrequirement. Package installation is larger than the 132 kB dataset download.\nUse a new output directory each time you train. If a cell fails, fix the cause,\nrestart the kernel, choose a new directory, and run from the top.\n"
  },
  {
   "cell_type": "markdown",
   "id": "churn-02",
   "metadata": {},
   "source": "## 1. Inspect customers and choose inputs\n\n## Choose information available before churn\n\nThe ten inputs describe call failures, complaints, subscription length, charge\nband, call seconds and counts, text messages, distinct called numbers, tariff,\nand age. Tariff is converted from numeric codes to meaningful category names.\nThe charge band is an ordinal code, not a money amount.\n\nWe omit account `Status`, which may be too close to the outcome for the intended\nprediction point; `Customer Value`, whose formula is not documented; and the\nredundant `Age Group`. These are deliberate teaching choices. They do not prove\nthat the original columns leak future information. Review the actual availability\nof every input when adapting the example to your data.\n"
  },
  {
   "cell_type": "code",
   "id": "churn-03",
   "metadata": {},
   "source": "import argparse\nimport hashlib\nimport importlib.metadata\nimport json\nimport platform\nimport subprocess\nimport sys\nimport textwrap\nimport time\nfrom pathlib import Path\nfrom urllib.request import urlopen\n\nimport matplotlib\nmatplotlib.use(\"Agg\")\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nfrom sklearn.dummy import DummyClassifier\nfrom sklearn.metrics import (\n    ConfusionMatrixDisplay, accuracy_score, average_precision_score,\n    brier_score_loss, confusion_matrix, log_loss, precision_score,\n    recall_score, roc_auc_score,\n)\nfrom sklearn.model_selection import StratifiedGroupKFold\nfrom threadpoolctl import threadpool_limits\nfrom supervised import AutoML\n\nDATA_URL = \"https://archive.ics.uci.edu/static/public/563/data.csv\"\nDATA_SHA256 = \"90d5fb6bd1630cd4de4b4d28fcf8b4cb92a8f6ab7484605b0799d47386f7dbe1\"\nFEATURES = [\n    \"Call Failure\", \"Complains\", \"Subscription Length\", \"Charge Amount\",\n    \"Seconds of Use\", \"Frequency of use\", \"Frequency of SMS\",\n    \"Distinct Called Numbers\", \"Tariff Plan\", \"Age\",\n]\nTARGET = \"Churn\"\nSEED = 42\nTHRESHOLD = 0.5  # Chosen before evaluation; not tuned on the test set.\n\n\ndef load_data(path):\n    \"\"\"Download the reviewed snapshot once; retain the original CSV unchanged.\"\"\"\n    if not path.exists():\n        content = urlopen(DATA_URL, timeout=60).read()\n        if hashlib.sha256(content).hexdigest() != DATA_SHA256:\n            raise ValueError(\"UCI snapshot changed; review the dataset before continuing.\")\n        path.parent.mkdir(parents=True, exist_ok=True)\n        path.write_bytes(content)\n    if hashlib.sha256(path.read_bytes()).hexdigest() != DATA_SHA256:\n        raise ValueError(\"Dataset checksum differs from the reviewed snapshot.\")\n    data = pd.read_csv(path)\n    data.columns = [\" \".join(name.split()) for name in data.columns]\n    expected = set(FEATURES + [TARGET, \"Status\", \"Customer Value\", \"Age Group\"])\n    if data.shape != (3150, 14) or set(data.columns) != expected:\n        raise ValueError(\"Unexpected churn dataset schema.\")\n    if not np.isfinite(data.to_numpy(dtype=float)).all():\n        raise ValueError(\"The reviewed snapshot should contain only finite numbers.\")\n    if set(data[TARGET]) != {0, 1}:\n        raise ValueError(\"Expected 0 (no churn) and 1 (churn).\")\n    return data\n\n\ndef evaluate(y, probabilities):\n    \"\"\"Use the same metrics and fixed threshold for baseline and AutoML.\"\"\"\n    labels = (probabilities >= THRESHOLD).astype(int)\n    return {\n        \"roc_auc\": roc_auc_score(y, probabilities),\n        \"average_precision\": average_precision_score(y, probabilities),\n        \"log_loss\": log_loss(y, probabilities, labels=[0, 1]),\n        \"brier_score\": brier_score_loss(y, probabilities),\n        \"accuracy\": accuracy_score(y, labels),\n        \"precision\": precision_score(y, labels, zero_division=0),\n        \"recall\": recall_score(y, labels, zero_division=0),\n        \"confusion_matrix\": confusion_matrix(y, labels, labels=[0, 1]).tolist(),\n    }\n\n\n\nfrom IPython.display import Image, display\n",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "id": "churn-04",
   "metadata": {},
   "source": "output = Path(\"AutoML_CustomerChurn_Notebook\").resolve()\ndata_path = Path(\"AutoML_ChurnData/customer-churn.csv\").resolve()\nbudget = 120\nthread_limits = threadpool_limits(limits=2)\n# Step 1. Inspect the customers and choose the target.\nstarted = time.perf_counter()\noutput.mkdir(parents=True, exist_ok=False)\ndata = load_data(data_path)\nprint(f\"Customers: {len(data)}; churned: {data[TARGET].sum()} ({data[TARGET].mean():.1%}).\", flush=True)\nprint(data.head().to_string(index=False), flush=True)\n# Omit Status (proximity to churn), undocumented Customer Value, and redundant Age Group.\nX, y = data[FEATURES].copy(), data[TARGET]\n# Tariff codes name plans, not a numeric scale. AutoML handles their encoding.\nX[\"Tariff Plan\"] = X[\"Tariff Plan\"].map({1: \"Pay as you go\", 2: \"Contractual\"})\ndisplay(X.head())\n",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "churn-05",
   "metadata": {},
   "source": "## 2. Reserve test profiles\n\n## Keep repeated profiles together\n\nThis file contains repeated records. We group identical combinations of the ten\ninputs so the same profile cannot occur in both training and testing. We keep\nall rows, including profiles with differing outcomes; grouping never uses the\ntarget. Without IDs, we cannot establish whether matching rows are the same person.\n\nThe fixed split uses 2,511 rows for training and 639 for testing. Within training,\nthree grouped folds supply the validation data used to choose models. The script\nchecks group separation and saves `splits.json`. Grouping and stratification\napproximately preserve the churn proportion; they do not create a time split.\n"
  },
  {
   "cell_type": "code",
   "id": "churn-06",
   "metadata": {},
   "source": "# Step 2. Reserve test customers and keep identical input profiles together.\n# There are no customer IDs or dates. These groups are only duplicate-profile proxies.\ngroups = X.groupby(FEATURES, sort=True, dropna=False).ngroup()\nouter = StratifiedGroupKFold(n_splits=5, shuffle=True, random_state=SEED)\ntrain, test = next(outer.split(X, y, groups))\nX_train, y_train = X.iloc[train].reset_index(drop=True), y.iloc[train].reset_index(drop=True)\nX_test, y_test = X.iloc[test], y.iloc[test]\ntrain_groups = groups.iloc[train].reset_index(drop=True)\ncv = list(StratifiedGroupKFold(n_splits=3, shuffle=True, random_state=SEED).split(\n    X_train, y_train, train_groups))\nassert set(groups.iloc[train]).isdisjoint(groups.iloc[test])\nfor fit_rows, valid_rows in cv:\n    assert set(train_groups.iloc[fit_rows]).isdisjoint(train_groups.iloc[valid_rows])\nsplit_record = {\n    \"seed\": SEED, \"train_rows\": train.tolist(), \"test_rows\": test.tolist(),\n    \"profile_groups\": groups.tolist(),\n    \"cv_indices_relative_to_train\": [\n        {\"train\": a.tolist(), \"validation\": b.tolist()} for a, b in cv\n    ],\n}\n(output / \"splits.json\").write_text(json.dumps(split_record, indent=2) + \"\\n\")\nprint(f\"Train: {len(train)} rows; test: {len(test)} rows ({y_test.mean():.1%} churn).\", flush=True)\n",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "churn-07",
   "metadata": {},
   "source": "## 3. Train and compare\n\n## Train and compare\n\nOur baseline gives every row the churn proportion observed in training. It uses\nno test labels. AutoML compares Baseline, Linear, Decision Tree, and LightGBM,\nand tries an ensemble. It selects by **validation log loss**, which penalizes\nconfident wrong probabilities. Lower is better.\n\nFeature discovery is included in MLJAR's base installation, but is disabled here\nbecause this example uses custom grouped validation. Other feature engineering\nand stacking are disabled to keep this first example focused.\n"
  },
  {
   "cell_type": "code",
   "id": "churn-08",
   "metadata": {},
   "source": "# Step 3. Train a constant-probability baseline and compare AutoML models.\nbaseline = DummyClassifier(strategy=\"prior\").fit(X_train, y_train)\nautoml = AutoML(\n    results_path=str(output / \"automl\"), mode=\"Explain\",\n    algorithms=[\"Baseline\", \"Linear\", \"Decision Tree\", \"LightGBM\"],\n    ml_task=\"binary_classification\", eval_metric=\"logloss\", total_time_limit=budget,\n    validation_strategy={\"validation_type\": \"custom\"},\n    feature_discovery=False, golden_features=False, kmeans_features=False,\n    features_selection=False, stack_models=False, train_ensemble=True,\n    explain_level=1, n_jobs=2, random_state=SEED,\n)\nfit_started = time.perf_counter()\nautoml.fit(X_train, y_train, cv=cv)\nfit_seconds = time.perf_counter() - fit_started\n",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "churn-09",
   "metadata": {},
   "source": "## 4. Evaluate probabilities and errors\n\n## Understand the predictions\n\n`predict_proba()` returns probabilities for both classes. For this numeric 0/1\ntarget, column 1 represents churn:\n\n```python\nprobability_of_churn = automl.predict_proba(X_test)[:, 1]\n```\n\nA value of 0.7 is the model's estimated churn probability. It is not a guarantee\nthat exactly 70% of such customers will leave: calibration on representative\nfuture data still needs checking. We turn probabilities into labels using a\n**0.5 threshold fixed before evaluation**, rather than tuning against the test set.\n\n- A **false positive** flags a customer who stayed, potentially wasting outreach.\n- A **false negative** misses a customer who churned.\n- **Precision** asks how many flagged customers actually churned.\n- **Recall** asks how many customers who churned were flagged.\n\nA real threshold depends on outreach capacity and error costs. Choose it on\nseparate validation data, then assess it once on untouched test data. This tutorial\nuses its explicit threshold rather than `predict()`, which can use MLJAR's own\nselected classification threshold.\n"
  },
  {
   "cell_type": "code",
   "id": "churn-10",
   "metadata": {},
   "source": "# Step 4. Evaluate probabilities and classification errors on untouched rows.\n# For our numeric target 0/1, the second probability column is class 1 (churn).\nprobabilities = automl.predict_proba(X_test)[:, 1]\nbaseline_probabilities = baseline.predict_proba(X_test)[:, 1]\nscores = {\"baseline\": evaluate(y_test, baseline_probabilities),\n          \"automl\": evaluate(y_test, probabilities)}\nprint(pd.DataFrame(scores).drop(index=\"confusion_matrix\").to_string(), flush=True)\nX_test.assign(source_row=test, actual_churn=y_test,\n              churn_probability=probabilities,\n              predicted_churn=(probabilities >= THRESHOLD).astype(int),\n              baseline_probability=baseline_probabilities).to_csv(output / \"test_predictions.csv\", index=False)\nfig, axes = plt.subplots(1, 2, figsize=(9, 4), layout=\"constrained\")\nfor ax, (name, result) in zip(axes, scores.items()):\n    ConfusionMatrixDisplay(np.array(result[\"confusion_matrix\"]),\n                           display_labels=[\"Stayed\", \"Churned\"]).plot(ax=ax, colorbar=False, cmap=\"Blues\")\n    ax.set_title(f\"{name.title()} · threshold {THRESHOLD}\")\nfig.savefig(output / \"confusion-matrix.png\", dpi=160)\nplt.close(fig)\ndisplay(Image(filename=str(output / \"confusion-matrix.png\")))\n",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "churn-11",
   "metadata": {},
   "source": "The matrices compare the baseline and model on the same untouched rows.\nRows are actual outcomes; columns are predictions. Baseline precision is reported\nas zero by convention because it flags no customers at 0.5. It is not an estimate\nof the reliability of any flagged group. All scores are row-weighted.\n\nThe printed metrics are from **your run**. Compare them with the recorded tutorial\nwithout treating that single split as a promise of future performance. Do not tune\nmodels or thresholds against these test results.\n"
  },
  {
   "cell_type": "markdown",
   "id": "churn-12",
   "metadata": {},
   "source": "## 5. Predict and reload\n\nKeep the full `automl` directory. These example inputs are held-out rows without their labels. The next cell verifies all test probabilities after loading in a separate interpreter.\n"
  },
  {
   "cell_type": "code",
   "id": "churn-13",
   "metadata": {},
   "source": "# Step 5. Predict example inputs and verify reload in a fresh interpreter.\n# These are held-out examples with the label removed, not newly collected customers.\nnew_rows = X_test.head(3)\nnew_rows.to_csv(output / \"new_customers.csv\", index=False)\nprint(\"Example churn probabilities:\", automl.predict_proba(new_rows)[:, 1], flush=True)\nX_test.to_csv(output / \"reload_inputs.csv\", index=False)\nsubprocess.run([\n    sys.executable, \"-c\", textwrap.dedent(\"\"\"\n    import sys\n    import numpy as np\n    import pandas as pd\n    from supervised import AutoML\n    model = AutoML(results_path=sys.argv[1])\n    inputs = pd.read_csv(sys.argv[2])\n    np.save(sys.argv[3], model.predict_proba(inputs)[:, 1])\n    \"\"\"),\n    str(output / \"automl\"), str(output / \"reload_inputs.csv\"),\n    str(output / \"reloaded_probabilities.npy\"),\n], check=True)\nreloaded = np.load(output / \"reloaded_probabilities.npy\")\nnp.testing.assert_allclose(probabilities, reloaded, rtol=1e-7, atol=1e-7)\nprint(\"Fresh-process reload agrees with the original probabilities.\", flush=True)\n",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "churn-14",
   "metadata": {},
   "source": "## 6. Read the report\n\nThe next cell saves versions and evaluation metrics and displays the report. Click a model name to inspect its validation metrics and settings. Validation scores use different rows from the test scores above. Importance describes model reliance, not causation.\n"
  },
  {
   "cell_type": "code",
   "id": "churn-15",
   "metadata": {},
   "source": "# Step 6. Create the HTML report and record this run.\nreport = automl.report()\nmetrics = {\n    \"dataset_url\": DATA_URL, \"dataset_sha256\": DATA_SHA256,\n    \"features\": FEATURES, \"seed\": SEED, \"threshold\": THRESHOLD,\n    \"train_rows\": len(train), \"test_rows\": len(test),\n    \"train_churned\": int(y_train.sum()), \"test_churned\": int(y_test.sum()),\n    \"profile_groups\": int(groups.nunique()), \"scores\": scores,\n    \"training_budget_seconds\": budget, \"fit_wall_seconds\": fit_seconds,\n    \"workflow_seconds_excluding_imports\": time.perf_counter() - started,\n    \"reload_max_absolute_difference\": float(np.max(np.abs(probabilities - reloaded))),\n    \"python\": platform.python_version(), \"platform\": platform.platform(),\n    \"package_versions\": {name: importlib.metadata.version(name) for name in (\n        \"mljar-supervised\", \"numpy\", \"pandas\", \"scikit-learn\", \"lightgbm\", \"features-goldmine\", \"joblib\")},\n}\n(output / \"metrics.json\").write_text(json.dumps(metrics, indent=2) + \"\\n\")\nprint(f\"Open the report: {(output / 'automl' / 'README.html').as_uri()}\", flush=True)\nprint(\"Keep the complete automl directory for future predictions.\", flush=True)\nthread_limits.restore_original_limits()\ndisplay(report)\n",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "churn-16",
   "metadata": {},
   "source": "## Predict another CSV and reload\n\nThe script saves `new_customers.csv`: three held-out example rows without labels,\nnot independently collected customers. In a new Python process:\n\n```python\nimport pandas as pd\nfrom supervised import AutoML\n\nmodel = AutoML(results_path=\"AutoML_CustomerChurn_Notebook/automl\")\ncustomers = pd.read_csv(\"AutoML_CustomerChurn_Notebook/new_customers.csv\")\nprint(model.predict_proba(customers)[:, 1])\n```\n\nProvide the same ten column names and units. Use tariff labels `Pay as you go`\nand `Contractual`, as in the saved CSV. The complete script verifies all held-out\nprobabilities against a fresh-process reload with a small numerical tolerance.\n"
  },
  {
   "cell_type": "markdown",
   "id": "churn-17",
   "metadata": {},
   "source": "## Adapt it to your own customers\n\nReplace the download helper with your CSV reader, choose a binary target, and\nkeep only inputs available at the prediction point. Replace profile grouping\nwith real customer/account groups or a time split when those are available.\nSet aside test data before training, compare against a baseline, and review\nerrors and probability quality. Use a new output path for every training run.\n\nIf a checksum check fails, inspect the source rather than deleting the check.\nIf the output directory exists, choose a new `--output` name or reload the saved\nmodel. The `--data` option accepts a previously downloaded, verified source CSV.\n\nFor Studio, an intended starting prompt is: “Help me run this churn notebook,\nexplain the probabilities and errors, and show the report.” The Studio route and\nan unassisted beginner walkthrough remain unvalidated. The Python and notebook\nruns have their own recorded evidence below.\n"
  },
  {
   "cell_type": "markdown",
   "id": "churn-18",
   "metadata": {},
   "source": "## Attribution\n\nIranian Churn [Dataset] (2020), UCI Machine Learning Repository. [DOI: 10.24432/C5JW3Z](https://doi.org/10.24432/C5JW3Z), [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). Snapshot retrieved 27 October 2026. The source download is unchanged; column spacing and tariff labels are normalized in memory. Full [tutorial and evidence](https://supervised.mljar.com/tutorials/customer-churn/).\n"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}