{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "0280db8a",
   "metadata": {},
   "source": [
    "# Adaptive Gradient Boosting (AGB) Explained\n",
    "\n",
    "Pega's **Adaptive Decision Manager (ADM)** uses **Adaptive Gradient Boosting\n",
    "(AGB)** as its default predictive algorithm for Next-Best-Action recommendations.\n",
    "AGB is an online-learning variant of gradient boosted trees that continuously\n",
    "updates as new customer interactions arrive — no batch retraining required.\n",
    "Just as important, AGB changes the **learning unit**: instead of treating each\n",
    "action/channel slice as an isolated model, it can use action and context fields\n",
    "as predictors in one ensemble. Sparse or newly introduced actions can therefore\n",
    "benefit from patterns learned elsewhere, rather than starting from a completely\n",
    "separate blank slate.\n",
    "\n",
    "**Pega documentation:**\n",
    "- [Adaptive Gradient boosting overview](https://docs.pega.com/bundle/platform/page/platform/decision-management/adaptive-boosting-algorithm.html) — what AGB is and why it replaces Naïve Bayes\n",
    "- [The Gradient boosting technique](https://docs.pega.com/bundle/platform/page/platform/decision-management/adaptive-gradient-boosting.html) — deep dive: ensemble construction, gain formula, ADWIN pruning\n",
    "- [Downloading a Gradient boosting Adaptive Model](https://docs.pega.com/bundle/platform/page/platform/decision-management/view-model-report.html#ariaid-title5) — how to export the JSON from Prediction Studio\n",
    "- [Interpreting Gradient boosting predictor importance](https://docs.pega.com/bundle/platform/page/platform/decision-management/gradient-boosting-explanations.html) — reading the feature importance analysis in Prediction Studio\n",
    "\n",
    "This notebook explains:\n",
    "\n",
    "- [Quick Sanity Check](#Quick-Sanity-Check) — early red flags before reading detailed plots.\n",
    "- [How a Single Decision Tree Works](#How-a-Single-Decision-Tree-Works) — nodes, splits, and the gain formula.\n",
    "- [How Trees Build on Each Other](#How-Trees-Build-on-Each-Other) — ensemble growth, training stream, and cold-start / warm-start behaviour.\n",
    "- [Scoring a Customer](#Scoring-a-Customer) — tracing the path through every tree.\n",
    "- [Feature Importance](#Feature-Importance) — feature importance from split gains and predictor-encoding diagnostics.\n",
    "- [Model Health at a Glance](#Model-Health-at-a-Glance) — key metrics and what they signal.\n",
    "- [Model AUC and Calibration](#Model-AUC-and-Calibration) — how the model is evaluated and what the AUC score means.\n",
    "\n",
    "All examples use the `pdstools` sample AGB model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "548976e2",
   "metadata": {},
   "source": [
    "> **📦 Optional dependencies**\n",
    ">\n",
    "> This article uses features from the pdstools `adm` extra, and requires `pydot` and\n",
    "> `graphviz` for tree visualizations. Install with your favorite package\n",
    "> manager, e.g. `uv pip install \"pdstools[adm]\" pydot graphviz`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fba2fb28",
   "metadata": {
    "tags": [
     "remove_input"
    ]
   },
   "outputs": [],
   "source": [
    "# These lines are only for rendering in the docs, and are hidden through Jupyter tags\n",
    "# Do not run if you're running the notebook separately\n",
    "\n",
    "import plotly.io as pio\n",
    "\n",
    "pio.renderers.default = \"notebook_connected\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "32174ba2",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import polars as pl\n",
    "from math import exp\n",
    "from great_tables import GT\n",
    "from pdstools import datasets\n",
    "\n",
    "AGBModel = datasets.sample_trees()\n",
    "# To use your own model, export the JSON from Prediction Studio and load it with:\n",
    "# from pdstools.adm.trees import ADMTreesModel\n",
    "# AGBModel = ADMTreesModel.from_file(\"path-to-your-model.json\")  # <-- Replace with your model file path\n",
    "# See: https://docs.pega.com/bundle/platform/page/platform/decision-management/view-model-report.html#ariaid-title5\n",
    "print(f\"Loaded model: {len(AGBModel.model)} trees, Pooled AUC={AGBModel.metrics['auc']:.4f}\")\n",
    "print(f\"Training data: {AGBModel.metrics['response_positive_count']:,} positives, \"\n",
    "      f\"{AGBModel.metrics['response_negative_count']:,} negatives\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "quick-sanity-caveat",
   "metadata": {},
   "source": [
    "**Caveat:** Pooled AUC—computed by mixing prediction scores and outcomes across all actions before drawing a single ROC curve—is systematically inflated by action base-rate differences and action-mix. It is therefore not a reliable measure of the discriminative power of any individual action model. For model-performance assessment, prefer the response-count weighted average of per-action AUC values: this is the AUC shown in Prediction Studio's Adaptive Model Performance view. The standalone AGB JSON export used in this notebook does not contain the per-action AUCs needed to calculate that weighted value; it only exposes the exported pooled AUC scalar. See **Pooled vs weighted-average AUC** below."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "quick-sanity-intro",
   "metadata": {},
   "source": [
    "## Quick Sanity Check\n",
    "\n",
    "Before diving into the detailed walkthrough below, it's worth checking whether\n",
    "a model has actually learned anything meaningful. A handful of `metrics`\n",
    "values, taken together, tell you that at a glance:\n",
    "\n",
    "- **Too few trees / mostly stumps** — the model hasn't had enough responses\n",
    "  (or enough signal) to build real structure yet.\n",
    "- **Pooled AUC ≈ 0.5** — no discriminative power; predictions are\n",
    "  indistinguishable from random. (As elsewhere in this notebook, this is the\n",
    "  *pooled* AUC exported with the model — not the weighted-average AUC across\n",
    "  actions.)\n",
    "- **Negatives ≤ positives** — for a typical NBA response model this is\n",
    "  inverted (responses are usually rare), and can indicate a sampling issue,\n",
    "  a very high base rate, or a still-forming candidate model.\n",
    "- **Very few active predictors** — the model is only using a sliver of the\n",
    "  available data to make decisions.\n",
    "\n",
    "None of these are fatal on their own (a brand-new candidate model is\n",
    "*supposed* to look like this for a while), but several at once mean a\n",
    "model's plots and metrics are not yet a reliable read on its real-world\n",
    "performance. The cell also gives an early count of predictor-encoding flags;\n",
    "the detailed predictor table appears in the feature-importance section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "quick-sanity-code",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Quick sanity check — flags a handful of \"this model may not have learned\n",
    "# anything useful yet\" signals, plus an early count of predictor-design flags.\n",
    "result = AGBModel.sanity_check()\n",
    "predictor_diagnostics = AGBModel.predictor_diagnostics()\n",
    "predictor_flags = predictor_diagnostics.filter(pl.col(\"flags\") != \"\")\n",
    "\n",
    "print(f\"Sanity check ({result['n_trees']} trees, Pooled AUC={result['pooled_auc']:.4f}, \"\n",
    "      f\"{result['positive_count']:,} pos / {result['negative_count']:,} neg):\\n\")\n",
    "if result[\"flags\"]:\n",
    "    print(f\"⚠ {len(result['flags'])} red flag(s):\")\n",
    "    for msg in result[\"flags\"]:\n",
    "        print(f\"  ⚠ {msg}\")\n",
    "    print(\"\\nTreat the detailed sections below as illustrative of the mechanics,\")\n",
    "    print(\"not as a reliable read on this model's real-world performance yet.\")\n",
    "else:\n",
    "    print(\"✓ No red flags from this quick check — proceed to the detailed walkthrough below.\")\n",
    "\n",
    "if predictor_flags.is_empty():\n",
    "    print(\"\\nPredictor diagnostics: no high-cardinality or numeric-looking symbolic predictor flags detected.\")\n",
    "else:\n",
    "    print(f\"\\nPredictor diagnostics: {predictor_flags.height} predictor flag(s) found — see Feature Importance for details.\")\n",
    "    for predictor in predictor_flags.get_column(\"predictor\").head(5).to_list():\n",
    "        print(f\"  - {predictor}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4f9e376",
   "metadata": {},
   "source": [
    "## How a Single Decision Tree Works\n",
    "\n",
    "AGB is an **ensemble** of binary decision trees. Each tree is a directed graph\n",
    "where internal nodes split the population by a predictor threshold and leaf\n",
    "nodes hold a **score** (a real number in log-odds space).\n",
    "\n",
    "The model JSON stores each tree as a nested dict. Here is the root node of\n",
    "the first tree:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5fc80941",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Show the root node of tree 0 — the first split the model makes\n",
    "root = AGBModel.model[0]\n",
    "print(json.dumps(\n",
    "    {k: v for k, v in root.items() if k not in (\"left\", \"right\")},\n",
    "    indent=2,\n",
    "))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fdfa8374",
   "metadata": {},
   "source": [
    "Each node contains:\n",
    "\n",
    "| Field | Meaning |\n",
    "|---|---|\n",
    "| `split` | The branching condition (`predictor op threshold`). |\n",
    "| `gain` | How much this split reduces prediction error (see formula below). |\n",
    "| `score` | The log-odds leaf score assigned to customers reaching this node. |\n",
    "| `sampleCount` | Number of training responses routed through this node. |\n",
    "| `left` / `right` | Child sub-trees (condition true → left, false → right). |\n",
    "\n",
    "### The Split Gain Formula\n",
    "\n",
    "AGB uses the XGBoost-style **gradient-based split gain**. For a candidate\n",
    "split that partitions the training responses into left ($L$) and right ($R$)\n",
    "groups:\n",
    "\n",
    "$$\n",
    "\\text{Gain} = \\frac{1}{2}\\left[\n",
    "  \\frac{G_L^2}{H_L + \\lambda} +\n",
    "  \\frac{G_R^2}{H_R + \\lambda} -\n",
    "  \\frac{(G_L + G_R)^2}{H_L + H_R + \\lambda}\n",
    "\\right] - \\gamma\n",
    "$$\n",
    "\n",
    "Where:\n",
    "- $G = \\sum g_i$ with $g_i = p_i - y_i$ — the **prediction error**\n",
    "  (residual) of the current model for response $i$.\n",
    "- $H = \\sum h_i$ with $h_i = p_i(1 - p_i)$ — the **Hessian** (curvature).\n",
    "- $\\lambda$ — L2 regularisation on leaf scores.\n",
    "- $\\gamma$ — the **complexity threshold**: a split is only created when\n",
    "  Gain $> 0$ after subtracting $\\gamma$. This prevents over-splitting.\n",
    "\n",
    "Advanced settings such as learning rate, regularisation, minimum split gain,\n",
    "maximum depth, and maximum tree count constrain how trees grow and how much\n",
    "each tree can contribute. They do not turn AGB into a scheduled batch-training\n",
    "process; the model still learns from the incoming response stream.\n",
    "\n",
    "A split is only added to the tree if this gain is positive.  High gain at\n",
    "the root means the split strongly separates positives from negatives.\n",
    "\n",
    "**ADWIN — the adaptation mechanism:** Each node continuously monitors its\n",
    "prediction error using an Adaptive Sliding Window (ADWIN).  When the error\n",
    "rises — signalling concept drift — the window shrinks and the effective gain\n",
    "falls. Branches whose gain drops below $\\gamma$ are **pruned**, allowing the\n",
    "model to forget stale patterns and grow fresh splits.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7a01aaaf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predictors used by this model, by type\n",
    "print(f\"Total predictors: {len(AGBModel.predictors)}\")\n",
    "for ptype in [\"symbolic\", \"numeric\"]:\n",
    "    names = [k for k, v in AGBModel.predictors.items() if v == ptype]\n",
    "    print(f\"  {ptype}: {len(names)} — e.g. {names[:3]}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "564a3b5d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Visualise tree 0 (requires pydot + graphviz)\n",
    "AGBModel.plot.tree(0)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54b8b8fe",
   "metadata": {},
   "source": [
    "## How Trees Build on Each Other\n",
    "\n",
    "AGB is an **additive model**. Each new tree is trained on the *residuals* of\n",
    "all previous trees — it corrects whatever the ensemble got wrong so far.\n",
    "\n",
    "### The Scoring Formula\n",
    "\n",
    "For a customer $x$, the raw log-odds score is the sum of all leaf scores:\n",
    "\n",
    "$$\n",
    "s = \\sum_{k=1}^{K} \\text{score}_k(x)\n",
    "$$\n",
    "\n",
    "where $\\text{score}_k(x)$ is the leaf score of tree $k$ for input $x$.\n",
    "\n",
    "The **propensity** (predicted probability of a positive outcome) is:\n",
    "\n",
    "$$\n",
    "p = \\sigma(s) = \\frac{1}{1 + e^{-s}}\n",
    "$$\n",
    "\n",
    "The leaf scores stored in the model JSON already incorporate the learning\n",
    "rate ($\\eta$) applied during training — there is no separate $\\eta$ factor\n",
    "at scoring time.\n",
    "\n",
    "### Cold Start and Warm Start\n",
    "\n",
    "**Cold start:** A new model starts with zero trees, so $s = 0$ and the\n",
    "initial propensity is exactly $\\sigma(0) = 0.5$:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b3d460f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "from math import exp\n",
    "\n",
    "# Cold-start propensity: no trees → score sum is 0\n",
    "cold_start_propensity = 1 / (1 + exp(0))\n",
    "print(f\"Cold-start propensity: {cold_start_propensity}\")  # exactly 0.5\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22ca227b",
   "metadata": {},
   "source": [
    "**Warm start:** When a new treatment or action is introduced with similar\n",
    "attributes to an existing one, the model immediately benefits from predictors\n",
    "it has already learned.  The new treatment reuses splits built for similar\n",
    "treatments, giving it an advantage over a blank-slate model. This is the\n",
    "practical payoff of learning across action and context combinations in one\n",
    "ensemble rather than maintaining a fully isolated model per slice.\n",
    "\n",
    "### Per-Tree Statistics\n",
    "\n",
    "The table below summarises each tree.  Early trees capture the strongest\n",
    "signals (high gain, large root score magnitude).  Later trees make finer\n",
    "corrections.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9eed0404",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Per-tree summary — drop the raw gains list for display\n",
    "(\n",
    "    GT(\n",
    "        AGBModel.tree_stats\n",
    "        .select(\"treeID\", \"score\", \"depth\", \"nsplits\", \"meangains\")\n",
    "        .rename({\"treeID\": \"Tree\", \"score\": \"Root score\", \"depth\": \"Depth\",\n",
    "                 \"nsplits\": \"Splits\", \"meangains\": \"Mean gain\"})\n",
    "        .head(10)\n",
    "    )\n",
    "    .tab_header(title=\"First 10 AGBModel — Summary\")\n",
    "    .fmt_number(columns=[\"Root score\", \"Mean gain\"], decimals=4)\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a3708a00",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Gain contribution per tree — early trees dominate\n",
    "AGBModel.plot.gain_per_tree()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0747ccd4",
   "metadata": {},
   "source": [
    "**Interpreting this chart:** A healthy ensemble shows a steep decline from left to right — the first 10–20 trees capture the bulk of the gain and later trees make progressively smaller corrections. A flat or rising tail suggests the model is still actively learning or that concept drift is continuously introducing new signal."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a8e1f89f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Cumulative gain share — S-curve showing how quickly the ensemble saturates\n",
    "AGBModel.plot.cumulative_gain_share()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b50374b0",
   "metadata": {},
   "source": [
    "**Interpreting this chart:** The dashed line marks where 50% of total gain has accumulated. **Good:** this crossover should fall well before the midpoint of the tree count — in this model around tree 38 out of 83 — confirming that a small core of early trees does most of the work and later trees add diminishing corrections. A late crossover (past the halfway point) indicates an unusually flat gain distribution."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb413676",
   "metadata": {},
   "source": [
    "### Training Stream\n",
    "\n",
    "Because AGB trains online, we can reconstruct the timeline of tree additions\n",
    "from the `sampleCount` stored in each tree's root node.  A new tree is added\n",
    "after the ensemble has processed enough new responses to detect an improvement\n",
    "opportunity.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "645995ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Total responses seen when each tree was added\n",
    "AGBModel.plot.training_stream_timeline()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "325340cd",
   "metadata": {},
   "source": [
    "**Interpreting this chart:** Each point is a tree; the y-axis shows the total responses seen at the moment that tree was added. **Good:** a gently declining curve — the first tree was trained on the most data, and each subsequent tree refines a gradually changing stream. Steep steps indicate a burst of new data that triggered a new tree; a very flat curve suggests data was arriving slowly during that period."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b2b2a38",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Response volume between consecutive tree additions\n",
    "# Spikes indicate bursts of activity; long gaps indicate slow learning periods\n",
    "AGBModel.plot.inter_tree_gaps()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "732599f0",
   "metadata": {},
   "source": [
    "**Interpreting this chart:** Each bar is the change in root `sampleCount` from the previous tree. Most values are negative (the ADWIN window shrinks between tree additions as old responses age out). **Good:** gaps cluster near zero with occasional small negative or positive spikes. Large positive spikes mark a flood of new training data that triggered a new tree; large negative spikes indicate significant pruning of stale observations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71be49a2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Gain decay — how much each tree contributes relative to its age\n",
    "# x-axis: responses seen since that tree was added\n",
    "AGBModel.plot.gain_decay_dual_lens()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d652572",
   "metadata": {},
   "source": [
    "**Interpreting this chart:** The steelblue trace shows gain by tree-index order; the orange dotted trace replots the same gain against *training age* (responses seen since that tree was added). **Good:** gain should be highest for the youngest trees (low training age, right side of the orange trace) and decay as trees age, confirming the model continuously renews itself. If the two traces look nearly identical in shape, the training rate has been roughly constant."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "853a39ff",
   "metadata": {},
   "source": [
    "## Scoring a Customer\n",
    "\n",
    "To score a customer, the model traverses every tree from root to leaf,\n",
    "collecting the leaf score for each tree, then applies the sigmoid.\n",
    "\n",
    "Let's trace a concrete customer through the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "23d8b25a",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build a concrete customer profile.\n",
    "# For each predictor we pick the first threshold value seen in the model's splits\n",
    "# (giving a real, self-consistent set of inputs), then override a few key fields.\n",
    "x = {\n",
    "    pred: (\n",
    "        float(next(iter(AGBModel.all_values_per_split[pred])))\n",
    "        if ptype == \"numeric\" and pred in AGBModel.all_values_per_split\n",
    "        else next(iter(AGBModel.all_values_per_split.get(pred, {\"Unknown\"})))\n",
    "    )\n",
    "    for pred, ptype in AGBModel.predictors.items()\n",
    "}\n",
    "# Fix the treatment so tree 0 takes its left (positive) branch\n",
    "x[\"pyTreatment\"] = \"Action_01\"\n",
    "\n",
    "# Show just the key context-key predictors\n",
    "print(\"Key predictor values for this customer:\")\n",
    "for pred in [\"pyTreatment\", \"pyName\", \"pyGroup\"]:\n",
    "    print(f\"  {pred} = {x[pred]!r}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "61ee63b1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Highlight the path taken through tree 0 (green nodes = visited)\n",
    "AGBModel.plot.tree(0, highlighted=x)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4488c919",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Per-tree leaf scores for this customer\n",
    "visited = AGBModel.get_all_visited_nodes(x)\n",
    "(\n",
    "    GT(\n",
    "        visited.select(\"treeID\", \"score\")\n",
    "        .rename({\"treeID\": \"Tree\", \"score\": \"Leaf score\"})\n",
    "        .head(10)\n",
    "    )\n",
    "    .tab_header(title=\"First 10 Leaf Scores for Customer x\")\n",
    "    .fmt_number(columns=[\"Leaf score\"], decimals=6)\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d6cb0b8b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Manual verification: sum all leaf scores, apply sigmoid\n",
    "raw_score = visited.get_column(\"score\").sum()\n",
    "propensity_manual = 1 / (1 + exp(-raw_score))\n",
    "propensity_library = AGBModel.score(x)\n",
    "\n",
    "print(f\"Sum of leaf scores: {raw_score:.6f}\")\n",
    "print(f\"Propensity (manual σ(s)): {propensity_manual:.6f}\")\n",
    "print(f\"Propensity (AGBModel.score): {propensity_library:.6f}\")\n",
    "print(f\"Match: {abs(propensity_manual - propensity_library) < 1e-10}\")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd96ec89",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Running propensity as each tree is added — shows convergence\n",
    "AGBModel.plot.contribution_per_tree(x)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48bb7442",
   "metadata": {},
   "source": [
    "**Interpreting this chart:** Each bar is the incremental propensity change added by one tree for this specific customer. **Good:** the running total (visible in the cumulative trace) converges quickly — typically after 20–30 trees — and subsequent trees make only small adjustments. A propensity that is still oscillating late in the ensemble indicates a borderline customer where the model is genuinely uncertain."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "09b3e69a",
   "metadata": {},
   "source": [
    "## Feature Importance\n",
    "\n",
    "A predictor's importance is measured by its **total split gain** — the sum of gain values at every split node that uses that predictor, across all trees. This is an operational monitoring signal, not a causal explanation: it shows which predictors the ensemble used to reduce error, not whether those predictors caused the outcome or why one individual customer received a specific propensity.\n",
    "\n",
    "> **Note on Prediction Studio:** In the Prediction Studio UI, feature importance is shown as a **0–100 score that sums to 100** across all active predictors. The plots below use raw total gain (absolute scale), which is more useful for analysis but differs from the normalized UI value. Prediction Studio also offers **treatment-level feature importance**, showing which predictors matter most for each individual treatment.\n",
    "\n",
    "Predictors are grouped into ADM-style **PredictorCategory** values that indicate the data source. AGB model-context fields get their own category so action/context identifiers stand out from other undotted predictors. The remaining base categories and colors are the same defaults used by ADM plots:\n",
    "\n",
    "| Prefix or match | Predictor category | Source |\n",
    "|---|---|---|\n",
    "| AGB model-context fields | Model context | `pyIssue`, `pyGroup`, `pyName`, `pyTreatment`, `pyDirection`, and `pyChannel` |\n",
    "| *(no dot)* | Primary | Other undotted primary predictors |\n",
    "| `IH.*` | IH | Interaction History predictors |\n",
    "| `Customer.*` | Customer | Customer attribute predictors |\n",
    "| `Param.*` | Param | Parameter predictors |\n",
    "| `<other>.*` | `<other>` | Any other dot-prefixed namespace |\n",
    "| HC external-score keywords | External Model | Predictors whose names contain terms such as `Propensity`, `Score`, `Prediction`, or `ModelScore` |\n",
    "\n",
    "The next cell applies the same external-model keyword defaults used by the Health Check import screen, so score-like predictors are colored and labelled consistently with ADM Health Check outputs.\n",
    "\n",
    "### Predictors pre-processing\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e864b81e",
   "metadata": {},
   "source": [
    "### Predictor Diagnostics\n",
    "\n",
    "A few predictor-design rules of thumb are worth checking before ranking predictors by gain:\n",
    "\n",
    "- **Use enough predictors** from different perspectives of the customer. ADM/AGB performs predictor selection, so a broad candidate set is better than an overly sparse one.\n",
    "- **Keep ordinary numeric fields numeric.** Values such as age, income, price, counts, balances, or durations can be passed as numeric predictors; AGB bins them into useful value ranges.\n",
    "- **Keep modest-cardinality symbolic fields as-is.** String fields with a manageable number of categories do not need one-hot encoding.\n",
    "- **Review high-cardinality identifiers and codes.** Full postal codes, product SKUs, or other code-like fields may fill the symbolic bin space. AGB can still hash/reuse bins, but a coarser derived predictor, such as postal-code prefix or product family, is often easier to monitor and generalise.\n",
    "- **Verify numeric-looking symbolic values.** If a symbolic predictor contains values such as `44.95`, `85.05`, and `94.95`, confirm whether it is really a category/code or whether it should be configured as numeric."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6757319b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Predictor diagnostics — high-cardinality symbolic and numeric-looking categorical checks\n",
    "from IPython.display import display\n",
    "\n",
    "predictor_diagnostics = AGBModel.predictor_diagnostics()\n",
    "predictor_flags = predictor_diagnostics.filter(pl.col(\"flags\") != \"\")\n",
    "\n",
    "if predictor_flags.is_empty():\n",
    "    print(\"No high-cardinality or numeric-looking symbolic predictor flags detected.\")\n",
    "else:\n",
    "    display(\n",
    "        GT(\n",
    "            predictor_flags.select(\n",
    "                \"predictor\",\n",
    "                \"predictor_type\",\n",
    "                \"split_count\",\n",
    "                \"total_gain\",\n",
    "                \"observed_values\",\n",
    "                \"used_bins\",\n",
    "                \"max_bins\",\n",
    "                \"flags\",\n",
    "            ).rename(\n",
    "                {\n",
    "                    \"predictor\": \"Predictor\",\n",
    "                    \"predictor_type\": \"Type\",\n",
    "                    \"split_count\": \"Splits\",\n",
    "                    \"total_gain\": \"Total gain\",\n",
    "                    \"observed_values\": \"Observed values\",\n",
    "                    \"used_bins\": \"Used bins\",\n",
    "                    \"max_bins\": \"Max bins\",\n",
    "                    \"flags\": \"Flag\",\n",
    "                }\n",
    "            )\n",
    "        )\n",
    "        .tab_header(title=\"Predictor Diagnostics — Flags\")\n",
    "        .fmt_number(columns=[\"Total gain\"], decimals=1)\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "177a2679",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Top split patterns by total gain\n",
    "top_splits = (\n",
    "    AGBModel.grouped_gains_per_split\n",
    "    .with_columns(pl.col(\"gains\").list.sum().alias(\"total_gain\"))\n",
    "    .sort(\"total_gain\", descending=True)\n",
    "    .head(10)\n",
    ")\n",
    "(\n",
    "    GT(top_splits.select(\"split\", \"predictor\", \"n\", \"mean\", \"total_gain\"))\n",
    "    .tab_header(title=\"Top 10 Split Patterns by Total Gain\")\n",
    "    .cols_label(split=\"Split condition\", predictor=\"Predictor\", n=\"# occurrences\",\n",
    "                mean=\"Mean gain\", total_gain=\"Total gain\")\n",
    "    .fmt_number(columns=[\"mean\", \"total_gain\"], decimals=1)\n",
    ")\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0e2a64d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Gain distribution per split condition for the top-5 predictors by total gain\n",
    "top_preds = (\n",
    "    AGBModel.grouped_gains_per_split\n",
    "    .with_columns(pl.col(\"gains\").list.sum().alias(\"total_gain\"))\n",
    "    .group_by(\"predictor\")\n",
    "    .agg(pl.col(\"total_gain\").sum())\n",
    "    .sort(\"total_gain\", descending=True)\n",
    "    .head(5)\n",
    "    .get_column(\"predictor\")\n",
    "    .to_list()\n",
    ")\n",
    "AGBModel.plot.splits_per_variable(subset=set(top_preds));"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6261eff",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Health Check uses these default keyword matches for external model scores.\n",
    "# Keep this notebook aligned so AGB and ADM Health Check plots label score-like\n",
    "# predictors consistently.\n",
    "from pdstools.adm.trees._plots import (\n",
    "    _PREDICTOR_CATEGORY_COLORMAP,\n",
    "    _PREDICTOR_CATEGORY_VALUE,\n",
    ")\n",
    "\n",
    "HEALTH_CHECK_EXTERNAL_MODEL_KEYWORDS = (\n",
    "    \"Propensity\",\n",
    "    \"Score\",\n",
    "    \"Class\",\n",
    "    \"Classifier\",\n",
    "    \"Classification\",\n",
    "    \"Probability\",\n",
    "    \"Prediction\",\n",
    "    \"Predicted\",\n",
    "    \"ModelScore\",\n",
    ")\n",
    "\n",
    "external_model_score_expr = pl.any_horizontal(\n",
    "    *[\n",
    "        pl.col(\"predictor\").cast(pl.Utf8).str.contains(keyword, literal=True)\n",
    "        for keyword in HEALTH_CHECK_EXTERNAL_MODEL_KEYWORDS\n",
    "    ]\n",
    ")\n",
    "\n",
    "AGBModel.plot.predictor_category_expr = (\n",
    "    pl.when(external_model_score_expr)\n",
    "    .then(pl.lit(\"External Model\"))\n",
    "    .otherwise(_PREDICTOR_CATEGORY_VALUE)\n",
    "    .alias(\"PredictorCategory\")\n",
    ")\n",
    "AGBModel.plot.predictor_category_colormap = dict(_PREDICTOR_CATEGORY_COLORMAP)\n",
    "AGBModel.plot.predictor_category_order = list(_PREDICTOR_CATEGORY_COLORMAP)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "064a95f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Overall feature importance ranked by total gain\n",
    "AGBModel.plot.feature_importance_by_gain()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85e2dc26",
   "metadata": {},
   "source": [
    "**Interpreting this chart:** Predictors are sorted by their total split gain across all trees. **Good:** model-context fields (`pyGroup`, `pyIssue`, `pyTreatment`) dominating the top is expected and healthy — AGB uses a single model for all actions, so action identity is a powerful discriminator. Below those, interaction-history (`IH.*`) and customer-attribute predictors should contribute meaningfully. A single non-context predictor taking >50% of the gain warrants investigation for potential data leakage."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "87f9adc0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Gain share by predictor category (IH.* vs Customer.* vs Primary / Param)\n",
    "AGBModel.plot.gain_by_namespace()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64538517",
   "metadata": {},
   "source": [
    "**Interpreting this chart:** Each bar shows one predictor category's share of total split gain. **Good:** several categories each claiming 15–40% of the gain, indicating the model draws on diverse signals. A single category above 70–80% may indicate that other data sources are unavailable or that those predictors are inactive.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "645ac1f8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Early learner vs late refiner — which predictors drive the first quarter of trees\n",
    "# vs the last quarter?  Persistent predictors appear in both halves.\n",
    "AGBModel.plot.early_vs_late_gain()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d474aec",
   "metadata": {},
   "source": [
    "**Interpreting this chart:** Points above the diagonal are *late refiners* — their gain increases after many responses have arrived. Points below are *early specialists* — they drive the initial rapid learning. **Good:** model-context fields (`pyGroup`, `pyIssue`) typically land near the upper-left (early, dominant); behavioural predictors (`IH.*`) gradually strengthen over time and appear near or above the diagonal. Predictors clustered near the bottom-left contributed little in either phase and may be inactive."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3192f285",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Feature role map:\n",
    "#   x-axis = mean split depth (shallow = high-level router, deep = specialist refiner)\n",
    "#   y-axis = fraction of trees where the predictor appears (coverage)\n",
    "#   size   = total gain\n",
    "AGBModel.plot.feature_role_map()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "487e2eaf",
   "metadata": {},
   "source": [
    "**Interpreting this chart:** The x-axis is mean split depth — shallow predictors (near 0) act as high-level routers at the top of every tree; deeper predictors fire only for specific sub-populations. The y-axis shows the fraction of trees the predictor appears in (coverage); bubble size is total gain. **Good:** model-context fields such as `pyGroup` should be shallow and widely covered; individual IH or customer predictors should be deeper and more selective. A large bubble with shallow depth and broad coverage is a powerful universal signal. A predictor with high coverage but a tiny bubble is splitting often without much gain — a candidate for review."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a1d5c39d",
   "metadata": {},
   "source": [
    "> **Single-case attribution (SHAP):** For individual propensity attribution —\n",
    "> \"why did this customer receive this propensity?\" — Prediction Studio uses\n",
    "> **Shapley values (SHAP)**.  The split-gain importance shown above is a\n",
    "> model-level signal; SHAP provides the complementary per-customer view.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32afc65e",
   "metadata": {},
   "source": [
    "## Model Health at a Glance\n",
    "\n",
    "The `metrics` dictionary captures a comprehensive set of health indicators\n",
    "computed from the model structure. In production, many of these metrics are\n",
    "exposed through Prediction Studio telemetry and model reports.\n",
    "\n",
    "Key signals to watch:\n",
    "\n",
    "| Metric | Healthy range | Signal |\n",
    "|---|---|---|\n",
    "| `score_decay_ratio` | < 1 | Converging model; > 1 may indicate instability |\n",
    "| `top_predictor_gain_share` | < 0.5 | High value = over-reliance on one predictor |\n",
    "| `predictor_gain_entropy` | High | Low entropy = gain concentrated in few predictors |\n",
    "| `number_of_stump_trees` | Low | High = heavy pruning, possible concept drift |\n",
    "| `mean_gain_last_half` < `mean_gain_first_half` | — | Expected; model converging |"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c23a63c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Build a readable metrics table\n",
    "from IPython.display import HTML\n",
    "\n",
    "descriptions = AGBModel.metric_descriptions()\n",
    "metrics_df = pl.DataFrame({\n",
    "    \"Metric\": list(AGBModel.metrics.keys()),\n",
    "    \"Value\": [\n",
    "        f\"{v:.4f}\" if isinstance(v, float) else str(v)\n",
    "        for v in AGBModel.metrics.values()\n",
    "    ],\n",
    "    \"Description\": [descriptions.get(k, \"\") for k in AGBModel.metrics.keys()],\n",
    "})\n",
    "table_html = (\n",
    "    GT(metrics_df)\n",
    "    .tab_header(title=\"Model Health Metrics\")\n",
    "    .cols_width({\"Metric\": \"25%\", \"Value\": \"15%\", \"Description\": \"60%\"})\n",
    "    .as_raw_html()\n",
    ")\n",
    "HTML(f'<div style=\"max-height:420px;overflow-y:auto\">{table_html}</div>')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6c4cb964",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Key \"model not developing\" diagnostics (SOP-ADM009)\n",
    "# The split/tree ratio is computable from any model export.\n",
    "# Saturation metrics require a datamart Modeldata blob — see note below.\n",
    "m = AGBModel.metrics\n",
    "total_splits = m[\"number_of_numeric_splits\"] + m[\"number_of_symbolic_splits\"]\n",
    "ratio = total_splits / m[\"number_of_trees\"]\n",
    "\n",
    "print(f\"Avg splits / tree:   {ratio:.1f}  ({'✓ OK' if ratio > 1 else '⚠ ALERT — model not developing'})\")\n",
    "print(f\"Active predictors:   {m['total_number_of_active_predictors']}  \"\n",
    "      f\"({'✓' if m['total_number_of_active_predictors'] >= 10 else '⚠ < 10 — may be too few to develop'})\")\n",
    "if \"number_of_saturated_context_key_predictors\" in m:\n",
    "    print(f\"Saturated context-key:     {m['number_of_saturated_context_key_predictors']}\")\n",
    "    print(f\"Saturated symbolic:        {m['number_of_saturated_symbolic_predictors']}\")\n",
    "    print(f\"Max context-key fill rate: {m['max_saturation_rate_on_context_key_predictors']:.0f} %\")\n",
    "else:\n",
    "    print(\"Saturation metrics: not available for Prediction Studio exports.\")\n",
    "    print(\"Load via ADMDatamart.agb to see encoder fill rates.\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ee9b83a",
   "metadata": {},
   "source": [
    "**Interpreting this output:** Avg splits/tree is the primary \"not developing\" signal — at or below 1 means most trees are stumps and the model is not learning structure from the data despite receiving responses.  If saturation metrics are available, check them first: a predictor whose encoder table is full cannot contribute new splits regardless of data volume, which explains why gain flatlines even as response counts climb."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f0092835",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Splits by predictor type over the ensemble — symbolic vs numeric usage\n",
    "AGBModel.plot.splits_per_variable_type()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32a3ed24",
   "metadata": {},
   "source": [
    "**Interpreting this chart:** The Absolute view shows raw split counts per predictor type across trees; click **Relative** to compare proportional mix. A **healthy** model shows a stable percentage mix throughout the ensemble — a sudden shift late in training (e.g. symbolic splits spiking) can indicate a new data source becoming active or an imbalance in predictor coverage."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8f17543",
   "metadata": {},
   "source": [
    "## Model AUC and Calibration\n",
    "\n",
    "After each new response, AGB applies a final calibration layer: the ensemble's raw log-odds score $s$ is passed through a **PAVA-calibrated propensity mapping** — the same Isotonic Regression (Pool Adjacent Violators) step used by the classic Naïve Bayes algorithm. The resulting propensity is the predicted probability exposed to the Next-Best-Action engine.\n",
    "\n",
    "### Validated AUC: test-then-train\n",
    "\n",
    "The model's AUC is a **validated** metric, not a training-set score. AGB uses **test-then-train**:\n",
    "\n",
    "1. The incoming response is **first scored** by the current ensemble (producing a propensity via PAVA).\n",
    "2. The predicted outcome is compared to the actual outcome, and the validated result contributes to the running AUC.\n",
    "3. **Then** the model weights are updated.\n",
    "\n",
    "This order ensures the AUC always reflects predictions made on data the model had not yet seen — it is a live, unbiased performance estimate.\n",
    "\n",
    "### Pooled vs weighted-average AUC\n",
    "\n",
    "**Pooled AUC** mixes prediction scores and outcomes from multiple actions into one combined ROC calculation. This number is structurally inflated by cross-action base-rate separation, so a high pooled AUC can coexist with weak per-action models.\n",
    "\n",
    "**Weighted-average AUC** is the response-count weighted average of per-action AUC values. This is the AUC displayed in Prediction Studio's **Adaptive Model Performance** view, and it is the honest, actionable portfolio metric because it answers the operational question that matters in NBA: does each action model rank likely responders above likely non-responders within its own action?\n",
    "\n",
    "The standalone AGB JSON export used in this notebook does **not** include the per-action AUC values and response counts needed to calculate weighted-average AUC. It only exposes the exported pooled AUC scalar. To inspect weighted-average AUC, use Prediction Studio's Adaptive Model Performance view or an ADM/datamart source that contains per-action model performance rows.\n",
    "\n",
    "In practice, report weighted-average AUC as the primary portfolio measure and treat pooled AUC only as a descriptive aggregate, not as evidence of model quality.\n",
    "\n",
    "### AGB vs Naïve Bayes: calibration observability\n",
    "\n",
    "| Aspect | Naïve Bayes | AGB |\n",
    "|---|---|---|\n",
    "| PAVA bins exported? | ✅ Yes — visible in ADM datamart | ❌ No — not included in the standalone AGB export |\n",
    "| AUC derivable from bins? | ✅ `auc_from_bincounts()` | ❌ Only the pre-computed scalar |\n",
    "| AUC is validated? | ✅ test-then-train | ✅ test-then-train |\n",
    "\n",
    "> **Note:** PAVA bins are not exported for AGB models — only the pre-computed AUC scalar. For Naïve Bayes models, `auc_from_bincounts()` can re-derive AUC from the exported bins."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pdstools (3.13.8)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}