{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0280db8a",
   "metadata": {},
   "source": [
    "# Adaptive Gradient Boosting (AGB) Explained\n",
    "\n",
    "Pega's **Adaptive Decision Manager (ADM)** uses **Adaptive Gradient Boosting\n",
    "(AGB)** as its default predictive algorithm for Next-Best-Action recommendations.\n",
    "AGB is an online-learning variant of gradient boosted trees that continuously\n",
    "updates as new customer interactions arrive — no batch retraining required.\n",
    "\n",
    "**Pega documentation:**\n",
    "- [Adaptive Gradient boosting overview](https://docs.pega.com/bundle/alerts/page/platform/decision-management/adaptive-boosting-algorithm.html) — what AGB is and why it replaces Naïve Bayes\n",
    "- [The Gradient boosting technique](https://docs.pega.com/bundle/alerts/page/platform/decision-management/adaptive-gradient-boosting.html) — deep dive: ensemble construction, gain formula, ADWIN pruning\n",
    "- [Downloading a Gradient boosting Adaptive Model](https://docs.pega.com/bundle/alerts/page/platform/decision-management/gradient-boosting-model.html) — how to export the JSON from Prediction Studio\n",
    "- [Interpreting Gradient boosting predictor importance](https://docs.pega.com/bundle/alerts/page/platform/decision-management/gradient-boosting-explanations.html) — reading the feature importance analysis in Prediction Studio\n",
    "\n",
    "This notebook explains:\n",
    "\n",
    "1. [**How a single decision tree works**](#1.-The-Building-Block:-A-Single-Decision-Tree) — nodes, splits, and the gain formula.\n",
    "2. [**How trees combine**](#2.-The-Ensemble:-How-Trees-Build-on-Each-Other) into an ensemble, how the model grows over time, and\n",
    "   the cold-start / warm-start behaviour.\n",
    "3. [**How a customer is scored**](#3.-Scoring-a-Customer) — tracing the path through every tree.\n",
    "4. [**Which predictors matter**](#4.-Feature-Importance:-Which-Predictors-Drive-the-Model?) — feature importance from split gains.\n",
    "5. [**How to read model health**](#5.-Model-Health-at-a-Glance) — key metrics and what they signal.\n",
    "6. [**Model AUC and calibration**](#6.-Model-AUC-and-Calibration) — how the model is evaluated and what the AUC score means.\n",
    "\n",
    "All examples use the `pdstools` sample AGB model.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "548976e2",
   "metadata": {},
   "source": [
    "> **📦 Optional dependencies**\n",
    ">\n",
    "> This article uses features from the pdstools `adm` extra, and requires `pydot` and\n",
    "> `graphviz` for tree visualizations. Install with your favorite package\n",
    "> manager, e.g. `uv pip install \"pdstools[adm]\" pydot graphviz`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fba2fb28",
   "metadata": {
    "tags": [
     "remove_input"
    ]
   },
   "outputs": [],
   "source": [
    "# These lines are only for rendering in the docs, and are hidden through Jupyter tags\n",
    "# Do not run if you're running the notebook separately\n",
    "\n",
    "import plotly.io as pio\n",
    "\n",
    "pio.renderers.default = \"notebook_connected\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32174ba2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import polars as pl\n",
    "from math import exp\n",
    "from great_tables import GT\n",
    "from pdstools import datasets\n",
    "\n",
    "AGBModel = datasets.sample_trees()\n",
    "# To use your own model, export the JSON from Prediction Studio and load it with:\n",
    "# from pdstools.adm.trees import ADMTreesModel\n",
    "# AGBModel = ADMTreesModel.from_file(\"path/to/model_export.json\")\n",
    "# See: https://docs.pega.com/bundle/platform/page/platform/pega-ai-tools/export-model-data-prediction-studio.html\n",
    "print(f\"Loaded model: {len(AGBModel.model)} trees, Pooled AUC={AGBModel.metrics['auc']:.4f}\")\n",
    "print(f\"Training data: {AGBModel.metrics['response_positive_count']:,} positives, \"\n",
    "      f\"{AGBModel.metrics['response_negative_count']:,} negatives\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "quick-sanity-caveat",
   "metadata": {},
   "source": [
    "**Caveat:** Pooled AUC—computed by mixing prediction scores and outcomes across all actions before drawing a single ROC curve—is systematically inflated by action base-rate differences and action-mix. It is therefore not a reliable measure of the discriminative power of any individual action model. For model-performance assessment, prefer the response-count weighted average of per-action AUC values: this is the AUC shown in Prediction Studio's Adaptive Model Performance view. The standalone AGB JSON export used in this notebook does not contain the per-action AUCs needed to calculate that weighted value; it only exposes the exported pooled AUC scalar. See **Pooled vs weighted-average AUC** below."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "quick-sanity-intro",
   "metadata": {},
   "source": [
    "## 0. Quick Sanity Check\n",
    "\n",
    "Before diving into the detailed walkthrough below, it's worth checking whether\n",
    "a model has actually learned anything meaningful. A handful of `metrics`\n",
    "values, taken together, tell you that at a glance:\n",
    "\n",
    "- **Too few trees / mostly stumps** — the model hasn't had enough responses\n",
    "  (or enough signal) to build real structure yet.\n",
    "- **Pooled AUC ≈ 0.5** — no discriminative power; predictions are\n",
    "  indistinguishable from random. (As elsewhere in this notebook, this is the\n",
    "  *pooled* AUC exported with the model — not the weighted-average AUC across\n",
    "  actions.)\n",
    "- **Negatives ≤ positives** — for a typical NBA response model this is\n",
    "  inverted (responses are usually rare), and can indicate a sampling issue,\n",
    "  a very high base rate, or a still-forming candidate model.\n",
    "- **Very few active predictors** — the model is only using a sliver of the\n",
    "  available data to make decisions.\n",
    "\n",
    "None of these are fatal on their own (a brand-new candidate model is\n",
    "*supposed* to look like this for a while), but several at once mean a\n",
    "model's plots and metrics are not yet a reliable read on its real-world\n",
    "performance. `ADMTreesModel.sanity_check()` runs these checks for you:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "quick-sanity-code",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Quick sanity check — flags a handful of \"this model may not have learned\n",
    "# anything useful yet\" signals.\n",
    "result = AGBModel.sanity_check()\n",
    "\n",
    "print(f\"Sanity check ({result['n_trees']} trees, Pooled AUC={result['pooled_auc']:.4f}, \"\n",
    "      f\"{result['positive_count']:,} pos / {result['negative_count']:,} neg):\\n\")\n",
    "if result[\"flags\"]:\n",
    "    print(f\"⚠ {len(result['flags'])} red flag(s):\")\n",
    "    for msg in result[\"flags\"]:\n",
    "        print(f\"  ⚠ {msg}\")\n",
    "    print(\"\\nTreat the detailed sections below as illustrative of the mechanics,\")\n",
    "    print(\"not as a reliable read on this model's real-world performance yet.\")\n",
    "else:\n",
    "    print(\"✓ No red flags from this quick check — proceed to the detailed walkthrough below.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4f9e376",
   "metadata": {},
   "source": [
    "## 1. The Building Block: A Single Decision Tree\n",
    "\n",
    "AGB is an **ensemble** of binary decision trees. Each tree is a directed graph\n",
    "where internal nodes split the population by a predictor threshold and leaf\n",
    "nodes hold a **score** (a real number in log-odds space).\n",
    "\n",
    "The model JSON stores each tree as a nested dict. Here is the root node of\n",
    "the first tree:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5fc80941",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show the root node of tree 0 — the first split the model makes\n",
    "root = AGBModel.model[0]\n",
    "print(json.dumps(\n",
    "    {k: v for k, v in root.items() if k not in (\"left\", \"right\")},\n",
    "    indent=2,\n",
    "))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdfa8374",
   "metadata": {},
   "source": [
    "Each node contains:\n",
    "\n",
    "| Field | Meaning |\n",
    "|---|---|\n",
    "| `split` | The branching condition (`predictor op threshold`). |\n",
    "| `gain` | How much this split reduces prediction error (see formula below). |\n",
    "| `score` | The log-odds leaf score assigned to customers reaching this node. |\n",
    "| `sampleCount` | Number of training responses routed through this node. |\n",
    "| `left` / `right` | Child sub-trees (condition true → left, false → right). |\n",
    "\n",
    "### The Split Gain Formula\n",
    "\n",
    "AGB uses the XGBoost-style **gradient-based split gain**. For a candidate\n",
    "split that partitions the training responses into left ($L$) and right ($R$)\n",
    "groups:\n",
    "\n",
    "$$\n",
    "\\text{Gain} = \\frac{1}{2}\\left[\n",
    "  \\frac{G_L^2}{H_L + \\lambda} +\n",
    "  \\frac{G_R^2}{H_R + \\lambda} -\n",
    "  \\frac{(G_L + G_R)^2}{H_L + H_R + \\lambda}\n",
    "\\right] - \\gamma\n",
    "$$\n",
    "\n",
    "Where:\n",
    "- $G = \\sum g_i$ with $g_i = p_i - y_i$ — the **prediction error**\n",
    "  (residual) of the current model for response $i$.\n",
    "- $H = \\sum h_i$ with $h_i = p_i(1 - p_i)$ — the **Hessian** (curvature).\n",
    "- $\\lambda$ — L2 regularisation on leaf scores.\n",
    "- $\\gamma$ — the **complexity threshold**: a split is only created when\n",
    "  Gain $> 0$ after subtracting $\\gamma$. This prevents over-splitting.\n",
    "\n",
    "A split is only added to the tree if this gain is positive.  High gain at\n",
    "the root means the split strongly separates positives from negatives.\n",
    "\n",
    "**ADWIN — the adaptation mechanism:** Each node continuously monitors its\n",
    "prediction error using an Adaptive Sliding Window (ADWIN).  When the error\n",
    "rises — signalling concept drift — the window shrinks and the effective gain\n",
    "falls. Branches whose gain drops below $\\gamma$ are **pruned**, allowing the\n",
    "model to forget stale patterns and grow fresh splits.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a01aaaf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predictors used by this model, by type\n",
    "print(f\"Total predictors: {len(AGBModel.predictors)}\")\n",
    "for ptype in [\"symbolic\", \"numeric\"]:\n",
    "    names = [k for k, v in AGBModel.predictors.items() if v == ptype]\n",
    "    print(f\"  {ptype}: {len(names)} — e.g. {names[:3]}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "564a3b5d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualise tree 0 (requires pydot + graphviz)\n",
    "AGBModel.plot.tree(0)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54b8b8fe",
   "metadata": {},
   "source": [
    "## 2. The Ensemble: How Trees Build on Each Other\n",
    "\n",
    "AGB is an **additive model**.  Each new tree is trained on the *residuals* of\n",
    "all previous trees — it corrects whatever the ensemble got wrong so far.\n",
    "\n",
    "### The Scoring Formula\n",
    "\n",
    "For a customer $x$, the raw log-odds score is the sum of all leaf scores:\n",
    "\n",
    "$$\n",
    "s = \\sum_{k=1}^{K} \\text{score}_k(x)\n",
    "$$\n",
    "\n",
    "where $\\text{score}_k(x)$ is the leaf score of tree $k$ for input $x$.\n",
    "\n",
    "The **propensity** (predicted probability of a positive outcome) is:\n",
    "\n",
    "$$\n",
    "p = \\sigma(s) = \\frac{1}{1 + e^{-s}}\n",
    "$$\n",
    "\n",
    "The leaf scores stored in the model JSON already incorporate the learning\n",
    "rate ($\\eta$) applied during training — there is no separate $\\eta$ factor\n",
    "at scoring time.\n",
    "\n",
    "### Cold Start and Warm Start\n",
    "\n",
    "**Cold start:** A new model starts with zero trees, so $s = 0$ and the\n",
    "initial propensity is exactly $\\sigma(0) = 0.5$:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3d460f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from math import exp\n",
    "\n",
    "# Cold-start propensity: no trees → score sum is 0\n",
    "cold_start_propensity = 1 / (1 + exp(0))\n",
    "print(f\"Cold-start propensity: {cold_start_propensity}\")  # exactly 0.5\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22ca227b",
   "metadata": {},
   "source": [
    "**Warm start:** When a new treatment or action is introduced with similar\n",
    "attributes to an existing one, the model immediately benefits from predictors\n",
    "it has already learned.  The new treatment reuses splits built for similar\n",
    "treatments, giving it an advantage over a blank-slate model.\n",
    "\n",
    "### Per-Tree Statistics\n",
    "\n",
    "The table below summarises each tree.  Early trees capture the strongest\n",
    "signals (high gain, large root score magnitude).  Later trees make finer\n",
    "corrections.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9eed0404",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Per-tree summary — drop the raw gains list for display\n",
    "(\n",
    "    GT(\n",
    "        AGBModel.tree_stats\n",
    "        .select(\"treeID\", \"score\", \"depth\", \"nsplits\", \"meangains\")\n",
    "        .rename({\"treeID\": \"Tree\", \"score\": \"Root score\", \"depth\": \"Depth\",\n",
    "                 \"nsplits\": \"Splits\", \"meangains\": \"Mean gain\"})\n",
    "        .head(10)\n",
    "    )\n",
    "    .tab_header(title=\"First 10 AGBModel — Summary\")\n",
    "    .fmt_number(columns=[\"Root score\", \"Mean gain\"], decimals=4)\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3708a00",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Gain contribution per tree — early trees dominate\n",
    "AGBModel.plot.gain_per_tree()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0747ccd4",
   "metadata": {},
   "source": [
    "**Interpreting this chart:** A healthy ensemble shows a steep decline from left to right — the first 10–20 trees capture the bulk of the gain and later trees make progressively smaller corrections. A flat or rising tail suggests the model is still actively learning or that concept drift is continuously introducing new signal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8e1f89f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cumulative gain share — S-curve showing how quickly the ensemble saturates\n",
    "AGBModel.plot.cumulative_gain_share()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b50374b0",
   "metadata": {},
   "source": [
    "**Interpreting this chart:** The dashed line marks where 50% of total gain has accumulated. **Good:** this crossover should fall well before the midpoint of the tree count — in this model around tree 38 out of 83 — confirming that a small core of early trees does most of the work and later trees add diminishing corrections. A late crossover (past the halfway point) indicates an unusually flat gain distribution."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb413676",
   "metadata": {},
   "source": [
    "### Training Stream\n",
    "\n",
    "Because AGB trains online, we can reconstruct the timeline of tree additions\n",
    "from the `sampleCount` stored in each tree's root node.  A new tree is added\n",
    "after the ensemble has processed enough new responses to detect an improvement\n",
    "opportunity.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "645995ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Total responses seen when each tree was added\n",
    "AGBModel.plot.training_stream_timeline()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "325340cd",
   "metadata": {},
   "source": [
    "**Interpreting this chart:** Each point is a tree; the y-axis shows the total responses seen at the moment that tree was added. **Good:** a gently declining curve — the first tree was trained on the most data, and each subsequent tree refines a gradually changing stream. Steep steps indicate a burst of new data that triggered a new tree; a very flat curve suggests data was arriving slowly during that period."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b2b2a38",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Response volume between consecutive tree additions\n",
    "# Spikes indicate bursts of activity; long gaps indicate slow learning periods\n",
    "AGBModel.plot.inter_tree_gaps()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "732599f0",
   "metadata": {},
   "source": [
    "**Interpreting this chart:** Each bar is the change in root `sampleCount` from the previous tree. Most values are negative (the ADWIN window shrinks between tree additions as old responses age out). **Good:** gaps cluster near zero with occasional small negative or positive spikes. Large positive spikes mark a flood of new training data that triggered a new tree; large negative spikes indicate significant pruning of stale observations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71be49a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Gain decay — how much each tree contributes relative to its age\n",
    "# x-axis: responses seen since that tree was added\n",
    "AGBModel.plot.gain_decay_dual_lens()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d652572",
   "metadata": {},
   "source": [
    "**Interpreting this chart:** The steelblue trace shows gain by tree-index order; the orange dotted trace replots the same gain against *training age* (responses seen since that tree was added). **Good:** gain should be highest for the youngest trees (low training age, right side of the orange trace) and decay as trees age, confirming the model continuously renews itself. If the two traces look nearly identical in shape, the training rate has been roughly constant."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "853a39ff",
   "metadata": {},
   "source": [
    "## 3. Scoring a Customer\n",
    "\n",
    "To score a customer, the model traverses every tree from root to leaf,\n",
    "collecting the leaf score for each tree, then applies the sigmoid.\n",
    "\n",
    "Let's trace a concrete customer through the model.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23d8b25a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build a concrete customer profile.\n",
    "# For each predictor we pick the first threshold value seen in the model's splits\n",
    "# (giving a real, self-consistent set of inputs), then override a few key fields.\n",
    "x = {\n",
    "    pred: (\n",
    "        float(next(iter(AGBModel.all_values_per_split[pred])))\n",
    "        if ptype == \"numeric\" and pred in AGBModel.all_values_per_split\n",
    "        else next(iter(AGBModel.all_values_per_split.get(pred, {\"Unknown\"})))\n",
    "    )\n",
    "    for pred, ptype in AGBModel.predictors.items()\n",
    "}\n",
    "# Fix the treatment so tree 0 takes its left (positive) branch\n",
    "x[\"pyTreatment\"] = \"Action_01\"\n",
    "\n",
    "# Show just the key context-key predictors\n",
    "print(\"Key predictor values for this customer:\")\n",
    "for pred in [\"pyTreatment\", \"pyName\", \"pyGroup\"]:\n",
    "    print(f\"  {pred} = {x[pred]!r}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61ee63b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Highlight the path taken through tree 0 (green nodes = visited)\n",
    "AGBModel.plot.tree(0, highlighted=x)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4488c919",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Per-tree leaf scores for this customer\n",
    "visited = AGBModel.get_all_visited_nodes(x)\n",
    "(\n",
    "    GT(\n",
    "        visited.select(\"treeID\", \"score\")\n",
    "        .rename({\"treeID\": \"Tree\", \"score\": \"Leaf score\"})\n",
    "        .head(10)\n",
    "    )\n",
    "    .tab_header(title=\"First 10 Leaf Scores for Customer x\")\n",
    "    .fmt_number(columns=[\"Leaf score\"], decimals=6)\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d6cb0b8b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Manual verification: sum all leaf scores, apply sigmoid\n",
    "raw_score = visited.get_column(\"score\").sum()\n",
    "propensity_manual = 1 / (1 + exp(-raw_score))\n",
    "propensity_library = AGBModel.score(x)\n",
    "\n",
    "print(f\"Sum of leaf scores: {raw_score:.6f}\")\n",
    "print(f\"Propensity (manual σ(s)): {propensity_manual:.6f}\")\n",
    "print(f\"Propensity (AGBModel.score): {propensity_library:.6f}\")\n",
    "print(f\"Match: {abs(propensity_manual - propensity_library) < 1e-10}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd96ec89",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Running propensity as each tree is added — shows convergence\n",
    "AGBModel.plot.contribution_per_tree(x)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48bb7442",
   "metadata": {},
   "source": [
    "**Interpreting this chart:** Each bar is the incremental propensity change added by one tree for this specific customer. **Good:** the running total (visible in the cumulative trace) converges quickly — typically after 20–30 trees — and subsequent trees make only small adjustments. A propensity that is still oscillating late in the ensemble indicates a borderline customer where the model is genuinely uncertain."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09b3e69a",
   "metadata": {},
   "source": [
    "## 4. Feature Importance: Which Predictors Drive the Model?\n",
    "\n",
    "A predictor's importance is measured by its **total split gain** — the sum of gain values at every split node that uses that predictor, across all trees.\n",
    "\n",
    "> **Note on Prediction Studio:** In the Prediction Studio UI, feature importance is shown as a **0–100 score that sums to 100** across all active predictors. The plots below use raw total gain (absolute scale), which is more useful for analysis but differs from the normalized UI value. Prediction Studio also offers **treatment-level feature importance**, showing which predictors matter most for each individual treatment.\n",
    "\n",
    "Predictors are grouped into ADM-style **PredictorCategory** values that indicate the data source. The base categories and colors are the same defaults used by ADM plots:\n",
    "\n",
    "| Prefix or match | Predictor category | Source |\n",
    "|---|---|---|\n",
    "| *(no dot)* | Primary | Primary/context predictors such as `pyTreatment`, `pyGroup`, and `pyName` |\n",
    "| `IH.*` | IH | Interaction History predictors |\n",
    "| `Customer.*` | Customer | Customer attribute predictors |\n",
    "| `Param.*` | Param | Parameter predictors |\n",
    "| `<other>.*` | `<other>` | Any other dot-prefixed namespace |\n",
    "| HC external-score keywords | External Model | Predictors whose names contain terms such as `Propensity`, `Score`, `Prediction`, or `ModelScore` |\n",
    "\n",
    "The next cell applies the same external-model keyword defaults used by the Health Check import screen, so score-like predictors are colored and labelled consistently with ADM Health Check outputs.\n",
    "\n",
    "### Predictors pre-processing\n",
    "\n",
    "- **Symbolic predictors** use a custom encoding that allows for more granular bins than the classic Naïve Bayes algorithm.\n",
    "- **Numeric predictors** are pre-processed using **percentile streaming**, which is robust against extreme outliers.\n",
    "\n",
    "Both pre-processing steps contribute to AGB's higher predictive power compared to the classic algorithm."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "177a2679",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Top split patterns by total gain\n",
    "top_splits = (\n",
    "    AGBModel.grouped_gains_per_split\n",
    "    .with_columns(pl.col(\"gains\").list.sum().alias(\"total_gain\"))\n",
    "    .sort(\"total_gain\", descending=True)\n",
    "    .head(10)\n",
    ")\n",
    "(\n",
    "    GT(top_splits.select(\"split\", \"predictor\", \"n\", \"mean\", \"total_gain\"))\n",
    "    .tab_header(title=\"Top 10 Split Patterns by Total Gain\")\n",
    "    .cols_label(split=\"Split condition\", predictor=\"Predictor\", n=\"# occurrences\",\n",
    "                mean=\"Mean gain\", total_gain=\"Total gain\")\n",
    "    .fmt_number(columns=[\"mean\", \"total_gain\"], decimals=1)\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0e2a64d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Gain distribution per split condition for the top-5 predictors by total gain\n",
    "top_preds = (\n",
    "    AGBModel.grouped_gains_per_split\n",
    "    .with_columns(pl.col(\"gains\").list.sum().alias(\"total_gain\"))\n",
    "    .group_by(\"predictor\")\n",
    "    .agg(pl.col(\"total_gain\").sum())\n",
    "    .sort(\"total_gain\", descending=True)\n",
    "    .head(5)\n",
    "    .get_column(\"predictor\")\n",
    "    .to_list()\n",
    ")\n",
    "AGBModel.plot.splits_per_variable(subset=set(top_preds));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6261eff",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Health Check uses these default keyword matches for external model scores.\n",
    "# Keep this notebook aligned so AGB and ADM Health Check plots label score-like\n",
    "# predictors consistently.\n",
    "from pdstools.adm.trees._plots import _PREDICTOR_CATEGORY_VALUE\n",
    "\n",
    "HEALTH_CHECK_EXTERNAL_MODEL_KEYWORDS = (\n",
    "    \"Propensity\",\n",
    "    \"Score\",\n",
    "    \"Class\",\n",
    "    \"Classifier\",\n",
    "    \"Classification\",\n",
    "    \"Probability\",\n",
    "    \"Prediction\",\n",
    "    \"Predicted\",\n",
    "    \"ModelScore\",\n",
    ")\n",
    "\n",
    "external_model_score_expr = pl.any_horizontal(\n",
    "    *[\n",
    "        pl.col(\"predictor\").cast(pl.Utf8).str.contains(keyword, literal=True)\n",
    "        for keyword in HEALTH_CHECK_EXTERNAL_MODEL_KEYWORDS\n",
    "    ]\n",
    ")\n",
    "\n",
    "AGBModel.plot.predictor_category_expr = (\n",
    "    pl.when(external_model_score_expr)\n",
    "    .then(pl.lit(\"External Model\"))\n",
    "    .otherwise(_PREDICTOR_CATEGORY_VALUE)\n",
    "    .alias(\"PredictorCategory\")\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "064a95f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Overall feature importance ranked by total gain\n",
    "AGBModel.plot.feature_importance_by_gain()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85e2dc26",
   "metadata": {},
   "source": [
    "**Interpreting this chart:** Predictors are sorted by their total split gain across all trees. **Good:** model-context fields (`pyGroup`, `pyIssue`, `pyTreatment`) dominating the top is expected and healthy — AGB uses a single model for all actions, so action identity is a powerful discriminator. Below those, interaction-history (`IH.*`) and customer-attribute predictors should contribute meaningfully. A single non-context predictor taking >50% of the gain warrants investigation for potential data leakage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "87f9adc0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Gain share by predictor category (IH.* vs Customer.* vs Primary / Param)\n",
    "AGBModel.plot.gain_by_namespace()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64538517",
   "metadata": {},
   "source": [
    "**Interpreting this chart:** Each bar shows one predictor category's share of total split gain. **Good:** several categories each claiming 15–40% of the gain, indicating the model draws on diverse signals. A single category above 70–80% may indicate that other data sources are unavailable or that those predictors are inactive.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "645ac1f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Early learner vs late refiner — which predictors drive the first quarter of trees\n",
    "# vs the last quarter?  Persistent predictors appear in both halves.\n",
    "AGBModel.plot.early_vs_late_gain()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d474aec",
   "metadata": {},
   "source": [
    "**Interpreting this chart:** Points above the diagonal are *late refiners* — their gain increases after many responses have arrived. Points below are *early specialists* — they drive the initial rapid learning. **Good:** model-context fields (`pyGroup`, `pyIssue`) typically land near the upper-left (early, dominant); behavioural predictors (`IH.*`) gradually strengthen over time and appear near or above the diagonal. Predictors clustered near the bottom-left contributed little in either phase and may be inactive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3192f285",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Feature role map:\n",
    "#   x-axis = mean split depth (shallow = high-level router, deep = specialist refiner)\n",
    "#   y-axis = fraction of trees where the predictor appears (coverage)\n",
    "#   size   = total gain\n",
    "AGBModel.plot.feature_role_map()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "487e2eaf",
   "metadata": {},
   "source": [
    "**Interpreting this chart:** The x-axis is mean split depth — shallow predictors (near 0) act as high-level routers at the top of every tree; deeper predictors fire only for specific sub-populations. The y-axis shows the fraction of trees the predictor appears in (coverage); bubble size is total gain. **Good:** model-context fields such as `pyGroup` should be shallow and widely covered; individual IH or customer predictors should be deeper and more selective. A large bubble with shallow depth and broad coverage is a powerful universal signal. A predictor with high coverage but a tiny bubble is splitting often without much gain — a candidate for review."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1d5c39d",
   "metadata": {},
   "source": [
    "> **Single-case attribution (SHAP):** For individual propensity attribution —\n",
    "> \"why did this customer receive this propensity?\" — Prediction Studio uses\n",
    "> **Shapley values (SHAP)**.  The split-gain importance shown above is a\n",
    "> model-level signal; SHAP provides the complementary per-customer view.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32afc65e",
   "metadata": {},
   "source": [
    "## 5. Model Health at a Glance\n",
    "\n",
    "The `metrics` dictionary captures a comprehensive set of health indicators\n",
    "computed from the model structure.  In production, many of these metrics are\n",
    "exposed through PEGA_ADM05 telemetry and visible in Prediction Studio model\n",
    "reports.\n",
    "\n",
    "Key signals to watch:\n",
    "\n",
    "| Metric | Healthy range | Signal |\n",
    "|---|---|---|\n",
    "| `score_decay_ratio` | < 1 | Converging model; > 1 may indicate instability |\n",
    "| `top_predictor_gain_share` | < 0.5 | High value = over-reliance on one predictor |\n",
    "| `predictor_gain_entropy` | High | Low entropy = gain concentrated in few predictors |\n",
    "| `number_of_stump_trees` | Low | High = heavy pruning, possible concept drift |\n",
    "| `mean_gain_last_half` < `mean_gain_first_half` | — | Expected; model converging |\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c23a63c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build a readable metrics table\n",
    "from IPython.display import HTML\n",
    "\n",
    "descriptions = AGBModel.metric_descriptions()\n",
    "metrics_df = pl.DataFrame({\n",
    "    \"Metric\": list(AGBModel.metrics.keys()),\n",
    "    \"Value\": [\n",
    "        f\"{v:.4f}\" if isinstance(v, float) else str(v)\n",
    "        for v in AGBModel.metrics.values()\n",
    "    ],\n",
    "    \"Description\": [descriptions.get(k, \"\") for k in AGBModel.metrics.keys()],\n",
    "})\n",
    "table_html = (\n",
    "    GT(metrics_df)\n",
    "    .tab_header(title=\"Model Health Metrics\")\n",
    "    .cols_width({\"Metric\": \"25%\", \"Value\": \"15%\", \"Description\": \"60%\"})\n",
    "    .as_raw_html()\n",
    ")\n",
    "HTML(f'<div style=\"max-height:420px;overflow-y:auto\">{table_html}</div>')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c4cb964",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Key \"model not developing\" diagnostics (SOP-ADM009)\n",
    "# The split/tree ratio is computable from any model export.\n",
    "# Saturation metrics require a datamart Modeldata blob — see note below.\n",
    "m = AGBModel.metrics\n",
    "total_splits = m[\"number_of_numeric_splits\"] + m[\"number_of_symbolic_splits\"]\n",
    "ratio = total_splits / m[\"number_of_trees\"]\n",
    "\n",
    "print(f\"Avg splits / tree:   {ratio:.1f}  ({'✓ OK' if ratio > 1 else '⚠ ALERT — model not developing'})\")\n",
    "print(f\"Active predictors:   {m['total_number_of_active_predictors']}  \"\n",
    "      f\"({'✓' if m['total_number_of_active_predictors'] >= 10 else '⚠ < 10 — may be too few to develop'})\")\n",
    "if \"number_of_saturated_context_key_predictors\" in m:\n",
    "    print(f\"Saturated context-key:     {m['number_of_saturated_context_key_predictors']}\")\n",
    "    print(f\"Saturated symbolic:        {m['number_of_saturated_symbolic_predictors']}\")\n",
    "    print(f\"Max context-key fill rate: {m['max_saturation_rate_on_context_key_predictors']:.0f} %\")\n",
    "else:\n",
    "    print(\"Saturation metrics: not available for Prediction Studio exports.\")\n",
    "    print(\"Load via ADMDatamart.agb to see encoder fill rates.\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ee9b83a",
   "metadata": {},
   "source": [
    "**Interpreting this output:** Avg splits/tree is the primary \"not developing\" signal — at or below 1 means most trees are stumps and the model is not learning structure from the data despite receiving responses.  If saturation metrics are available, check them first: a predictor whose encoder table is full cannot contribute new splits regardless of data volume, which explains why gain flatlines even as response counts climb."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f0092835",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Splits by predictor type over the ensemble — symbolic vs numeric usage\n",
    "AGBModel.plot.splits_per_variable_type()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32a3ed24",
   "metadata": {},
   "source": [
    "**Interpreting this chart:** The Absolute view shows raw split counts per predictor type across trees; click **Relative** to compare proportional mix. A **healthy** model shows a stable percentage mix throughout the ensemble — a sudden shift late in training (e.g. symbolic splits spiking) can indicate a new data source becoming active or an imbalance in predictor coverage."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8f17543",
   "metadata": {},
   "source": [
    "## 6. Model AUC and Calibration\n",
    "\n",
    "After each new response, AGB applies a final calibration layer: the ensemble's raw log-odds score $s$ is passed through a **PAVA-calibrated propensity mapping** — the same Isotonic Regression (Pool Adjacent Violators) step used by the classic Naïve Bayes algorithm. The resulting propensity is the predicted probability exposed to the Next-Best-Action engine.\n",
    "\n",
    "### Validated AUC: test-then-train\n",
    "\n",
    "The model's AUC is a **validated** metric, not a training-set score. AGB uses **test-then-train**:\n",
    "\n",
    "1. The incoming response is **first scored** by the current ensemble (producing a propensity via PAVA).\n",
    "2. The predicted outcome is compared to the actual outcome, and the validated result contributes to the running AUC.\n",
    "3. **Then** the model weights are updated.\n",
    "\n",
    "This order ensures the AUC always reflects predictions made on data the model had not yet seen — it is a live, unbiased performance estimate.\n",
    "\n",
    "### Pooled vs weighted-average AUC\n",
    "\n",
    "**Pooled AUC** mixes prediction scores and outcomes from multiple actions into one combined ROC calculation. This number is structurally inflated by cross-action base-rate separation, so a high pooled AUC can coexist with weak per-action models.\n",
    "\n",
    "**Weighted-average AUC** is the response-count weighted average of per-action AUC values. This is the AUC displayed in Prediction Studio's **Adaptive Model Performance** view, and it is the honest, actionable portfolio metric because it answers the operational question that matters in NBA: does each action model rank likely responders above likely non-responders within its own action?\n",
    "\n",
    "The standalone AGB JSON export used in this notebook does **not** include the per-action AUC values and response counts needed to calculate weighted-average AUC. It only exposes the exported pooled AUC scalar. To inspect weighted-average AUC, use Prediction Studio's Adaptive Model Performance view or an ADM/datamart source that contains per-action model performance rows.\n",
    "\n",
    "In practice, report weighted-average AUC as the primary portfolio measure and treat pooled AUC only as a descriptive aggregate, not as evidence of model quality.\n",
    "\n",
    "### AGB vs Naïve Bayes: calibration observability\n",
    "\n",
    "| Aspect | Naïve Bayes | AGB |\n",
    "|---|---|---|\n",
    "| PAVA bins exported? | ✅ Yes — visible in ADM datamart | ❌ No — internal to engine |\n",
    "| AUC derivable from bins? | ✅ `auc_from_bincounts()` | ❌ Only the pre-computed scalar |\n",
    "| AUC is validated? | ✅ test-then-train | ✅ test-then-train |\n",
    "\n",
    "> **Note:** PAVA bins are not exported for AGB models — only the pre-computed AUC scalar. For Naïve Bayes models, `auc_from_bincounts()` can re-derive AUC from the exported bins."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pdstools (3.12.6.final.0)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}