{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "1c1d8da6a687",
   "metadata": {},
   "source": [
    "> **📦 Optional dependencies**\n",
    ">\n",
    "> This article uses features from the pdstools `explanations` extra. Install with your favorite package manager, e.g. `uv pip install \"pdstools[explanations]\"`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1a99ea0b",
   "metadata": {
    "nbsphinx": "hidden",
    "tags": [
     "remove_input"
    ]
   },
   "outputs": [],
   "source": [
    "# These lines are only for rendering in the docs, and are hidden through Jupyter tags\n",
    "# Do not run if you're running the notebook seperately\n",
    "\n",
    "import plotly.io as pio\n",
    "\n",
    "pio.renderers.default = \"notebook_connected\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "y5ffwchvqi",
   "metadata": {},
   "source": [
    "# Global Explanations for Adaptive Gradient Boosting Models\n",
    "\n",
    "This notebook demonstrates how to analyze and visualize global explanations for Adaptive Gradient Boosting models in Pega Adaptive Decision Manager (ADM). Global explanations provide insights into which predictors have the most influence on model predictions and how different predictor values correlate with the model's scores.\n",
    "\n",
    "**Important:** The SHAP explanation datasets used in this notebook are available starting with **Pega Infinity '25**. In earlier versions, the explanation files are not written to the repository.\n",
    "\n",
    "The explanation files contain SHAP (SHapley Additive exPlanations) contributions for a sample of model executions. This notebook shows how these contributions are aggregated to provide global explanations for Gradient Boosting models. In Pega Infinity '25, a Global Explanations report can also be generated directly from [Prediction Studio](https://docs.pega.com/bundle/platform/page/platform/decision-management/gradient-boosting-explanations.html).\n",
    "\n",
    "**Note on earlier versions:** In versions prior to Infinity '25, Feature Importance (also known as Predictor Importance) is already available for Gradient Boosting models and uses SHAP values. In Infinity '25, we enhance this by providing detailed insights into the correlation between predictor _values_ and the model's score (for example, high income may correlate with a high contribution to the propensity score for a platinum credit card offer) rather than just the importances of predictors. See: https://docs.pega.com/bundle/platform/page/platform/decision-management/view-summarized-reports-adm.html\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "730bffd4",
   "metadata": {},
   "source": [
    "## Pre-Aggregated data exported from Infinity\n",
    "\n",
    "**Prerequisite**: You have already exported pre-aggregated explanation files from Infinity.\n",
    "\n",
    "Parameters:\n",
    "- `base_path`: the folder that contains the pre-aggregated explanation files\n",
    "- `model_name`: `optional` - the model rule to check for explanations; if not passed, will pick up any file in the folder\n",
    "- `from_date`: `optional` - if not passed, will be today - 7 days\n",
    "- `to_date`: `optional` - if not passed, will be today"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "45f9f307",
   "metadata": {},
   "outputs": [],
   "source": [
    "from pdstools import datasets\n",
    "\n",
    "import datetime\n",
    "import polars as pl\n",
    "import logging\n",
    "# logging.basicConfig(level=logging.INFO) # Uncomment to see progress for large files\n",
    "\n",
    "# Fetched over HTTPS from the pdstools repository, so this works from an\n",
    "# installed package as well as from a repository checkout.\n",
    "explanations = datasets.sample_explanations(\n",
    "    model_name='AdaptiveBoostCT',\n",
    "    from_date=datetime.datetime(2024, 3, 28),\n",
    "    to_date=datetime.datetime(2025, 3, 28),\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "954a4362",
   "metadata": {},
   "source": [
    "## Plotting contributions\n",
    "\n",
    "The [contributions_overall()](https://pegasystems.github.io/pega-datascientist-tools/autoapi/pdstools/explanations/Plots/index.html#pdstools.explanations.Plots.Plots.contributions_overall) method plots the top-n most influential predictors and, for each of them, shows how individual predictor values contribute to the model score. Numeric values are binned (max 10 bins), symbolic predictors show the top-k categories.\n",
    "\n",
    "Results can be shown for the model overall or drilled down into the action hierarchy (direction, channel, issue, group, action name), referred to as \"context\" in the API."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ef8a504",
   "metadata": {},
   "source": [
    "### Overall model\n",
    "\n",
    "Calling [contributions_overall()](https://pegasystems.github.io/pega-datascientist-tools/autoapi/pdstools/explanations/Plots/index.html#pdstools.explanations.Plots.Plots.contributions_overall) without selecting a level in the action hierarchy aggregates over all actions.\n",
    "\n",
    "Parameters:\n",
    "- `top_n`: Number of top predictors to plot.\n",
    "- `top_k`: Number of top predictor values for symbolic predictors to plot.\n",
    "- `remaining`: If `True`, the remaining predictors will be plotted as a single bar.\n",
    "- `missing`: If `True`, the missing values will be plotted as a separate bar.\n",
    "- `descending`: If `True`, the predictors will be sorted in descending order of their contributions, i.e. least contributing predictors will be plotted first.\n",
    "- `sort_by`: Method to calculate contributions. Options include `contribution`, `contribution_abs`, `contribution_weighted`. Default is `contribution_abs` (average absolute contributions to predictions).\n",
    "- `display_by`: Contribution type to display when plotting. Options include `contribution`, `contribution_abs`, `contribution_weighted`. Default is `contribution` (average contributions to predictions)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "01768e07",
   "metadata": {},
   "outputs": [],
   "source": [
    "overall_model, model_top_predictors = explanations.plot.contributions_overall(top_n=3, top_k=5, remaining=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5bd1fd03",
   "metadata": {},
   "outputs": [],
   "source": [
    "overall_model.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "672f3e49",
   "metadata": {},
   "outputs": [],
   "source": [
    "for top_predictor in model_top_predictors:\n",
    "    top_predictor.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78dfa154",
   "metadata": {},
   "source": [
    "### By level in the action hierarchy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ed529772",
   "metadata": {},
   "outputs": [],
   "source": [
    "selected_context = explanations.aggregates.unique_contexts()[0]\n",
    "selected_context"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b6e45ff1",
   "metadata": {},
   "outputs": [],
   "source": [
    "context_header, overall_by_context, context_top_predictors = explanations.plot.contributions_by_context(\n",
    "    context=selected_context,\n",
    "    top_n=3,\n",
    "    top_k=5,\n",
    ")\n",
    "display(context_header)\n",
    "display(overall_by_context)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d44be30",
   "metadata": {},
   "outputs": [],
   "source": [
    "for top_predictor in context_top_predictors:\n",
    "    top_predictor.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9d1ccec",
   "metadata": {},
   "source": [
    "## Advanced data exploration\n",
    "\n",
    "For more control, you can work with the [aggregates](https://pegasystems.github.io/pega-datascientist-tools/autoapi/pdstools/explanations/Aggregates/index.html#pdstools.explanations.Aggregates.Aggregates) object directly, inspect the underlying data, and build custom analyses."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a106a296",
   "metadata": {},
   "outputs": [],
   "source": [
    "aggregates = explanations.aggregates # load the aggregated data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b9c5bfa",
   "metadata": {},
   "source": [
    "### Inspect data for the overall model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39907165",
   "metadata": {},
   "source": [
    "Get the top-n predictors and their contributions using [predictor_contributions()](https://pegasystems.github.io/pega-datascientist-tools/autoapi/pdstools/explanations/Aggregates/index.html#pdstools.explanations.Aggregates.Aggregates.predictor_contributions)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "34aec57c",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_overall = aggregates.predictor_contributions(top_n = 3, remaining=False)\n",
    "df_overall\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85d3f460",
   "metadata": {},
   "source": [
    "Inspect the most influential values of those predictors using [predictor_value_contributions()](https://pegasystems.github.io/pega-datascientist-tools/autoapi/pdstools/explanations/Aggregates/index.html#pdstools.explanations.Aggregates.Aggregates.predictor_value_contributions)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6d6fd0d",
   "metadata": {},
   "outputs": [],
   "source": [
    "top_n_predictors = df_overall.select(pl.col('predictor_name')).unique().to_series().to_list()\n",
    "aggregates.predictor_value_contributions(\n",
    "    predictors=top_n_predictors, \n",
    "    top_k = 2, \n",
    "    remaining=False\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0919edd5",
   "metadata": {},
   "source": [
    "### Inspect data by level in the action hierarchy"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "abe47103",
   "metadata": {},
   "source": [
    "The same methods can drill down into the action hierarchy instead of looking at the overall model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "40217fb3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "context_info = random.choice(aggregates.unique_contexts())\n",
    "print('Selected random context: \\n')\n",
    "for key, value in context_info.items():\n",
    "    print(f'{key}: {value}')\n",
    "df_by_context = aggregates.predictor_contributions(\n",
    "    context=context_info, \n",
    "    top_n=3, \n",
    "    remaining=False)\n",
    "df_by_context\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5e15553f",
   "metadata": {},
   "outputs": [],
   "source": [
    "top_n_predictors = df_by_context.select(pl.col('predictor_name')).unique().to_series().to_list()\n",
    "aggregates.predictor_value_contributions(\n",
    "    predictors=top_n_predictors, \n",
    "    top_k=2, \n",
    "    context=context_info, \n",
    "    remaining=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2acd3479",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "pdstools (3.12.7)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}