{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "concrete-00",
   "metadata": {},
   "source": "# Predict a number from a CSV with Python\n\n**Your first AutoML regression model: concrete strength**\n\nImagine you have a spreadsheet of concrete test results. Each row describes\nthe ingredients, the specimen's age, and its measured strength. For another\nmixture, you know the ingredients and age but want an estimate of strength.\nCan a model learn from those past measurements to make that prediction?\n\nPredicting a number is called **regression**. In this notebook, you will look\nat the data, train several models with AutoML, and compare their predictions\nwith measurements that were kept out of training. Then you will read the\nreports and reload the saved model. Basic familiarity with running Python\nis enough to begin; we explain the machine-learning concepts as we use them.\n\n## A first look at the data\n\nThese are the first five rows of the real CSV, showing four of its nine columns:\n\n| Cement (kg/m³) | Water (kg/m³) | Age (days) | Measured strength (MPa) |\n| ---: | ---: | ---: | ---: |\n| 540.0 | 162.0 | 28 | 79.99 |\n| 540.0 | 162.0 | 28 | 61.89 |\n| 332.5 | 228.0 | 270 | 40.27 |\n| 332.5 | 228.0 | 365 | 41.05 |\n| 198.6 | 192.0 | 360 | 44.30 |\n\nThe first row describes a mixture with 540 kg of cement and 162 kg of water\nper cubic metre, tested at 28 days, with measured strength 79.99 MPa.\nThe other five ingredient columns are omitted from this short preview;\nfor example, the first two rows have different coarse aggregate amounts.\nThe full table appears after loading below.\n\nWe give the model eight **inputs**, also called *features*: seven ingredient\nquantities and age. We want it to predict one **target**: strength in MPa\n(megapascals). Do not include that measured target among the prediction inputs.\n\n## Set up your notebook\n\nUse Python 3.11 or 3.12. Before starting this notebook, install into its Python\nenvironment: `python -m pip install 'mljar-supervised==1.4.0b1' jupyterlab ipywidgets`.\nStart Jupyter with `python -m jupyterlab`. No GPU or optional Torch packages\nare needed. The UCI CSV download is 41.5 kB; package installation is larger.\n\nThis notebook uses a 120-second training budget and two CPU threads. Training\ncan finish early; total runtime also includes imports, reports, and reload.\nScript runs on the recorded Linux server took 16–21 seconds. Laptop and 8 GB\nmeasurements are still pending. Your environment can produce different timings.\n\nRun cells in order. Use a new output directory for each run. The code below\nis generated from `examples/scripts/concrete_strength.py`; it is fully\nincluded here, so you do not need that file to execute this notebook.\n"
  },
  {
   "cell_type": "markdown",
   "id": "concrete-01",
   "metadata": {},
   "source": "## 1. Load and inspect the measurements\n\nOne row contains seven ingredient amounts (kg/m³), specimen age (days), and\nmeasured compressive strength (MPa). The **target** is the quantity to predict:\n`Concrete compressive strength`. All other columns are numeric inputs.\n\nSource: Yeh, I. (1998), *Concrete Compressive Strength*,\n[UCI Machine Learning Repository](https://doi.org/10.24432/C5PK67),\n[CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).\nWe download UCI's CSV unchanged and check the reviewed file's checksum.\nThe 1,030 rows contain no missing values. Group labels and predictions are\nadditions made by this example. A changed checksum stops the example so a\nchanged source cannot silently alter the results.\n"
  },
  {
   "cell_type": "code",
   "id": "concrete-02",
   "metadata": {},
   "source": "import argparse\nimport hashlib\nimport importlib.metadata\nimport json\nimport platform\nimport subprocess\nimport sys\nimport textwrap\nimport time\nfrom pathlib import Path\nfrom urllib.request import urlopen\n\nimport matplotlib\n\nmatplotlib.use(\"Agg\")\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport pandas as pd\nfrom sklearn.dummy import DummyRegressor\nfrom sklearn.metrics import mean_absolute_error\nfrom sklearn.model_selection import GroupKFold, GroupShuffleSplit\nfrom threadpoolctl import threadpool_limits\n\nfrom supervised import AutoML\n\n\nDATA_URL = \"https://archive.ics.uci.edu/static/public/165/data.csv\"\nDATA_SHA256 = \"8d4b15b6fc68cd932d745cbd663d5ceae66dd54422e99c1e4865f2936ab7e2af\"\nINGREDIENTS = [\n    \"Cement\", \"Blast Furnace Slag\", \"Fly Ash\", \"Water\", \"Superplasticizer\",\n    \"Coarse Aggregate\", \"Fine Aggregate\",\n]\nFEATURES = INGREDIENTS + [\"Age\"]\nTARGET = \"Concrete compressive strength\"\nSEED = 42\n\n\ndef load_data(path):\n    \"\"\"Download once, then check the exact UCI snapshot on every run.\"\"\"\n    if not path.exists():\n        with urlopen(DATA_URL, timeout=60) as response:\n            content = response.read()\n        if hashlib.sha256(content).hexdigest() != DATA_SHA256:\n            raise ValueError(\"UCI data changed; review the new snapshot before using it.\")\n        path.parent.mkdir(parents=True, exist_ok=True)\n        path.write_bytes(content)\n    if hashlib.sha256(path.read_bytes()).hexdigest() != DATA_SHA256:\n        raise ValueError(\"Dataset checksum differs from the reviewed UCI snapshot.\")\n    data = pd.read_csv(path)\n    if list(data.columns) != FEATURES + [TARGET] or data.shape != (1030, 9):\n        raise ValueError(\"Unexpected concrete dataset schema.\")\n    if not np.isfinite(data.to_numpy(dtype=float)).all():\n        raise ValueError(\"The reviewed dataset should contain only finite numbers.\")\n    return data\n\n\n\nfrom IPython.display import display\n",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "id": "concrete-03",
   "metadata": {},
   "source": "output = Path(\"AutoML_ConcreteStrength_Notebook\").resolve()\ndata_path = Path(\"AutoML_ConcreteData/concrete.csv\").resolve()\nbudget = 120\nthread_limits = threadpool_limits(limits=2)\n# Step 1. Load the measurements and choose inputs and target.\n# Seven ingredient quantities (kg/m³) and age (days) predict strength (MPa).\nstarted = time.perf_counter()\noutput.mkdir(parents=True, exist_ok=False)\ndata = load_data(data_path)\nX, y = data[FEATURES], data[TARGET]\ndisplay(data.head())\n",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "concrete-04",
   "metadata": {},
   "source": "## 2. Reserve mixtures for testing\n\nSome mixtures were measured at multiple ages. A random row split could put\nthe same recipe in training and testing. Here we ask about **unseen mixtures**,\nso we group rows by the seven ingredient amounts and keep each group together.\n\nWe reserve 86 groups (192 rows) for final testing. The remaining 341 groups\n(838 rows) train the models. Within training, three grouped folds rotate which\nrows train and validate each candidate. AutoML uses those validation results\nto select a model. The held-out test target is used only for final evaluation.\n\nThese exact ingredient matches are proxy groups, not verified experiment IDs.\nNear-identical recipes or shared laboratories may still cross the split.\nNo batch/site identifiers are available to check those relationships.\n"
  },
  {
   "cell_type": "code",
   "id": "concrete-05",
   "metadata": {},
   "source": "# Keep the same mixture at different ages on one side of every split (step 2).\n# These are ingredient-based proxy groups, not verified experiment IDs.\ngroups = data.groupby(INGREDIENTS, sort=True).ngroup()\ntrain, test = next(GroupShuffleSplit(\n    n_splits=1, test_size=0.2, random_state=SEED\n).split(X, y, groups))\nX_train, y_train = X.iloc[train].reset_index(drop=True), y.iloc[train].reset_index(drop=True)\nX_test, y_test = X.iloc[test], y.iloc[test]\ntrain_groups = groups.iloc[train].reset_index(drop=True)\ncv = list(GroupKFold(n_splits=3).split(X_train, y_train, train_groups))\nassert set(groups.iloc[train]).isdisjoint(groups.iloc[test])\nfor fit_rows, valid_rows in cv:\n    assert set(train_groups.iloc[fit_rows]).isdisjoint(train_groups.iloc[valid_rows])\n\nsplit_record = {\n    \"seed\": SEED, \"train_rows\": train.tolist(), \"test_rows\": test.tolist(),\n    \"mixture_groups\": groups.tolist(),\n    \"cv_indices_relative_to_train\": [\n        {\"train\": a.tolist(), \"validation\": b.tolist()} for a, b in cv\n    ],\n}\n(output / \"splits.json\").write_text(json.dumps(split_record, indent=2) + \"\\n\")\nprint(f\"Concrete: {len(data)} measurements, {groups.nunique()} mixture groups.\", flush=True)\nprint(f\"Train: {len(train)} rows; held-out test: {len(test)} rows. Target: MPa.\", flush=True)\nprint(data.head().to_string(index=False), flush=True)\n",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "concrete-06",
   "metadata": {},
   "source": "## 3. Train a baseline and compare models\n\nA baseline always predicts the median training strength. The median minimizes\nconstant absolute error, giving us a useful reference for **MAE**, the mean\nabsolute difference between a prediction and its measured value.\n\nAutoML compares Baseline, Linear, Decision Tree, and LightGBM using the three\ntraining folds. It also tries an ensemble that combines candidate predictions.\n`n_jobs=2` limits model parallelism, and `total_time_limit=120` assigns a\ntraining budget. Keep the seed and split fixed when comparing configurations.\n\nFeature discovery is included in the base package, but currently does not\nsupport custom grouped validation, so it is disabled here. Other feature\nengineering and stacking are also disabled to keep this first run focused.\nThe `Baseline` in AutoML's leaderboard uses its own implementation; the\nseparately fitted median is our held-out evaluation reference.\n"
  },
  {
   "cell_type": "code",
   "id": "concrete-07",
   "metadata": {},
   "source": "# A training median is our baseline (step 3): the test target is unused.\n# AutoML compares models using only the three training-validation folds.\nbaseline = DummyRegressor(strategy=\"median\").fit(X_train, y_train)\nautoml = AutoML(\n    results_path=str(output / \"automl\"), mode=\"Explain\",\n    algorithms=[\"Baseline\", \"Linear\", \"Decision Tree\", \"LightGBM\"],\n    ml_task=\"regression\", eval_metric=\"mae\", total_time_limit=budget,\n    validation_strategy={\"validation_type\": \"custom\"},\n    feature_discovery=False, golden_features=False, kmeans_features=False,\n    features_selection=False, stack_models=False, train_ensemble=True,\n    explain_level=1, n_jobs=2, random_state=SEED,\n)\nfit_started = time.perf_counter()\nautoml.fit(X_train, y_train, cv=cv)\nfit_seconds = time.perf_counter() - fit_started\n",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "concrete-08",
   "metadata": {},
   "source": "## 4. Evaluate on the held-out rows and predict example inputs\n\nThese rows were not used for training or model selection. We save predictions\nwith their measurements for inspection. The three example inputs are held-out\nrows with the target removed, not proposed concrete designs or new experiments.\n"
  },
  {
   "cell_type": "code",
   "id": "concrete-09",
   "metadata": {},
   "source": "# Step 4. Evaluate on rows excluded from training and model selection.\npredictions = automl.predict(X_test)\nbaseline_predictions = baseline.predict(X_test)\nprint(f\"Baseline test MAE: {mean_absolute_error(y_test, baseline_predictions):.2f} MPa\", flush=True)\nprint(f\"AutoML test MAE: {mean_absolute_error(y_test, predictions):.2f} MPa\", flush=True)\nX_test.assign(\n    source_row=test, measured_strength_mpa=y_test,\n    predicted_strength_mpa=predictions, baseline_strength_mpa=baseline_predictions,\n).to_csv(output / \"test_predictions.csv\", index=False)\n\n# Demonstrate new, unlabeled inputs using held-out examples (not invented designs).\nnew_rows = X_test.head(3)\nnew_rows.to_csv(output / \"new_rows.csv\", index=False)\nnew_predictions = automl.predict(new_rows)\nprint(\"Example input predictions (MPa):\", new_predictions, flush=True)\n",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "concrete-10",
   "metadata": {},
   "source": "The recorded beta script runs produced **4.30 MPa MAE**, compared with\n**12.59 MPa** for the training-median baseline. A 4.30 MPa MAE means an average\nabsolute error of 4.30 MPa across these test rows; it does not bound every error.\nWhether this is useful depends on the intended use and acceptable errors.\nMixtures with more measured rows contribute more to this row-weighted metric.\n\nDo not repeatedly tune against this test result. Use training validation for\nchanges and reserve fresh evaluation data for a new final assessment.\n"
  },
  {
   "cell_type": "markdown",
   "id": "concrete-11",
   "metadata": {},
   "source": "## 5. Reload in a fresh Python process\n\nKeep the entire `automl/` directory, including preprocessing and all ensemble\ndependencies. This check starts a separate interpreter, reads the saved model\nand input CSV, then verifies that predictions agree with the in-memory model.\nIt raises an error if they differ beyond a small numerical tolerance.\n"
  },
  {
   "cell_type": "code",
   "id": "concrete-12",
   "metadata": {},
   "source": "# A new interpreter loads only the saved model directory and input CSV (step 5).\n# This checks that future predictions work without our training-session state.\nX_test.to_csv(output / \"reload_inputs.csv\", index=False)\nsubprocess.run([\n    sys.executable, \"-c\",\n    textwrap.dedent(\"\"\"\n    import sys\n    import numpy as np\n    import pandas as pd\n    from supervised import AutoML\n\n    model = AutoML(results_path=sys.argv[1])\n    inputs = pd.read_csv(sys.argv[2])\n    np.save(sys.argv[3], model.predict(inputs))\n    \"\"\"),\n    str(output / \"automl\"), str(output / \"reload_inputs.csv\"),\n    str(output / \"reloaded_predictions.npy\"),\n], check=True)\nreloaded = np.load(output / \"reloaded_predictions.npy\")\nnp.testing.assert_allclose(predictions, reloaded, rtol=1e-7, atol=1e-7)\nprint(\"Fresh-process reload agrees with the original predictions.\", flush=True)\n",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "concrete-13",
   "metadata": {},
   "source": "## 6. Inspect errors and save the run record\n\nPoints near the diagonal have predictions close to measured strength. Look\nfor large errors and ranges where predictions tend to be too high or low.\nThe example saves the figure, metrics, split indices, and package versions.\nThese records identify the run behind any reported number or chart.\n"
  },
  {
   "cell_type": "code",
   "id": "concrete-14",
   "metadata": {},
   "source": "# Step 6. Plot errors and save the evaluation and environment record.\nfig, ax = plt.subplots(figsize=(6, 5), layout=\"constrained\")\nax.scatter(y_test, predictions, s=20, alpha=0.65)\nlow, high = min(y_test.min(), predictions.min()), max(y_test.max(), predictions.max())\nax.plot([low, high], [low, high], color=\"black\", linestyle=\"--\", label=\"Perfect prediction\")\nax.set(xlabel=\"Measured strength (MPa)\", ylabel=\"Predicted strength (MPa)\",\n       title=\"Concrete strength: held-out mixture groups\")\nax.legend()\nfig.savefig(output / \"predicted-vs-measured.png\", dpi=160)\nplt.close(fig)\nfrom IPython.display import Image, display\ndisplay(Image(filename=str(output / \"predicted-vs-measured.png\")))\n",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "code",
   "id": "concrete-15",
   "metadata": {},
   "source": "metrics = {\n    \"dataset_url\": DATA_URL, \"dataset_sha256\": DATA_SHA256,\n    \"train_rows\": len(train), \"test_rows\": len(test),\n    \"train_groups\": int(groups.iloc[train].nunique()),\n    \"test_groups\": int(groups.iloc[test].nunique()),\n    \"baseline_strategy\": \"training median\",\n    \"baseline_mae_mpa\": mean_absolute_error(y_test, baseline_predictions),\n    \"automl_mae_mpa\": mean_absolute_error(y_test, predictions),\n    \"training_budget_seconds\": budget, \"fit_wall_seconds\": fit_seconds,\n    \"workflow_seconds_excluding_imports\": time.perf_counter() - started,\n    \"reload_max_absolute_difference\": float(np.max(np.abs(predictions - reloaded))),\n    \"python\": platform.python_version(), \"platform\": platform.platform(),\n    \"package_versions\": {name: importlib.metadata.version(name) for name in (\n        \"mljar-supervised\", \"numpy\", \"pandas\", \"scikit-learn\", \"lightgbm\",\n        \"features-goldmine\", \"joblib\",\n    )},\n}\n(output / \"metrics.json\").write_text(json.dumps(metrics, indent=2) + \"\\n\")\nprint(json.dumps(metrics, indent=2), flush=True)\nthread_limits.restore_original_limits()\n",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "concrete-16",
   "metadata": {},
   "source": "## 7. Read the reports\n\nRun the next cell to display the interactive HTML report here. Click a model\nname to open its individual report and **Go back** to return to the overview.\nThis also writes `AutoML_ConcreteStrength_Notebook/automl/README.html`, which\nyou can open in a browser. The Markdown version is available as `README.md`.\nThe leaderboard compares **training validation** MAE, not the separate test\nscore above. Lower MAE is better. In the recorded run the selected ensemble\nhad validation MAE 4.66 MPa and test MAE 4.30 MPa, measured on different rows.\n\nFollow the `4_Default_LightGBM` link to see its settings, validation metrics,\nlearning curves, permutation importance, and residual plots. Importance\ndescribes model dependence on inputs; it does not establish causal effects.\nThe exact artifacts available depend on the algorithm and explanation level.\n"
  },
  {
   "cell_type": "code",
   "id": "concrete-17",
   "metadata": {},
   "source": "# Step 7. Create the HTML report and print its browser location.\nreport = automl.report()\nprint(f\"Open the run report: {(output / 'automl' / 'README.html').as_uri()}\", flush=True)\nprint(\"Keep the complete automl directory for future predictions.\", flush=True)\ndisplay(report)\n",
   "execution_count": null,
   "outputs": []
  },
  {
   "cell_type": "markdown",
   "id": "concrete-18",
   "metadata": {},
   "source": "## 8. Adapt this workflow to your CSV\n\n1. Replace `load_data(data_path)` with `pd.read_csv(\"your_data.csv\")`; the UCI\n   checksum and schema checks apply only to the concrete dataset.\n2. Set `TARGET` to your measured outcome and `FEATURES` to inputs available\n   when predicting. For this regression example, the outcome must be numeric.\n3. Replace `INGREDIENTS` and the grouping rule with your real experiment,\n   customer, batch, or subject identifier. Keep identifiers out of model\n   inputs unless their use is justified. Time-dependent data needs a time\n   split; do not blindly reuse this mixture split.\n4. Choose an untouched test set, a baseline, a budget, and an error measure\n   meaningful in your units. Set a new output path before running.\n5. Inspect errors, read reports, and verify prediction/reload with the same\n   input columns and units. Domain knowledge determines acceptable errors.\n\nTo predict a later CSV in a new session:\n\n```python\nimport pandas as pd\nfrom supervised import AutoML\nmodel = AutoML(results_path=\"AutoML_ConcreteStrength_Notebook/automl\")\nnew_rows = pd.read_csv(\"AutoML_ConcreteStrength_Notebook/new_rows.csv\")\npredictions_mpa = model.predict(new_rows)\n```\n\nIf the output directory already exists, choose a new name to retrain or load\nthe existing model as above. If a notebook cell fails, fix the cause, restart\nthe kernel, choose a new output directory, and run from the top.\n\n**Limits:** exact recipe grouping cannot establish independence by experiment\nor laboratory. A single split does not establish general performance.\nObservational predictions do not prove that changing an ingredient causes\na strength change or replace physical testing and engineering assessment.\n\n**Studio starting prompt (not yet validated in Studio):** “Help me run this\nconcrete-strength notebook. Explain the target and mixture groups, compare\nthe held-out error with the baseline, and show the saved model report.”\nThe Python/Jupyter workflow is tested separately; Studio execution remains\npart of the release checklist.\n"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}