📦 Optional dependencies

This article uses features from the pdstools explanations extra. Install with your favorite package manager, e.g. uv pip install "pdstools[explanations]".

Global Explanations for Adaptive Gradient Boosting Models

This notebook demonstrates how to analyze and visualize global explanations for Adaptive Gradient Boosting models in Pega Adaptive Decision Manager (ADM). Global explanations provide insights into which predictors have the most influence on model predictions and how different predictor values correlate with the model’s scores.

Important: The SHAP explanation datasets used in this notebook are available starting with Pega Infinity ‘25. In earlier versions, the explanation files are not written to the repository.

The explanation files contain SHAP (SHapley Additive exPlanations) contributions for a sample of model executions. This notebook shows how these contributions are aggregated to provide global explanations for Gradient Boosting models. In Pega Infinity ‘25, a Global Explanations report can also be generated directly from Prediction Studio.

Note on earlier versions: In versions prior to Infinity ‘25, Feature Importance (also known as Predictor Importance) is already available for Gradient Boosting models and uses SHAP values. In Infinity ‘25, we enhance this by providing detailed insights into the correlation between predictor values and the model’s score (for example, high income may correlate with a high contribution to the propensity score for a platinum credit card offer) rather than just the importances of predictors. See: https://docs.pega.com/bundle/platform/page/platform/decision-management/view-summarized-reports-adm.html

Pre-Aggregated data exported from Infinity

Prerequisite: You have already exported pre-aggregated explanation files from Infinity.

Parameters:

  • base_path: the folder that contains the pre-aggregated explanation files

  • model_name: optional - the model rule to check for explanations; if not passed, will pick up any file in the folder

  • from_date: optional - if not passed, will be today - 7 days

  • to_date: optional - if not passed, will be today

[2]:

from pdstools import datasets

import datetime
import polars as pl
import logging
# logging.basicConfig(level=logging.INFO) # Uncomment to see progress for large files

# Fetched over HTTPS from the pdstools repository, so this works from an
# installed package as well as from a repository checkout.
explanations = datasets.sample_explanations(
    model_name='AdaptiveBoostCT',
    from_date=datetime.datetime(2024, 3, 28),
    to_date=datetime.datetime(2025, 3, 28),
)

Plotting contributions

The contributions_overall() method plots the top-n most influential predictors and, for each of them, shows how individual predictor values contribute to the model score. Numeric values are binned (max 10 bins), symbolic predictors show the top-k categories.

Results can be shown for the model overall or drilled down into the action hierarchy (direction, channel, issue, group, action name), referred to as “context” in the API.

Overall model

Calling contributions_overall() without selecting a level in the action hierarchy aggregates over all actions.

Parameters:

  • top_n: Number of top predictors to plot.

  • top_k: Number of top predictor values for symbolic predictors to plot.

  • remaining: If True, the remaining predictors will be plotted as a single bar.

  • missing: If True, the missing values will be plotted as a separate bar.

  • descending: If True, the predictors will be sorted in descending order of their contributions, i.e. least contributing predictors will be plotted first.

  • sort_by: Method to calculate contributions. Options include contribution, contribution_abs, contribution_weighted. Default is contribution_abs (average absolute contributions to predictions).

  • display_by: Contribution type to display when plotting. Options include contribution, contribution_abs, contribution_weighted. Default is contribution (average contributions to predictions).

[3]:

overall_model, model_top_predictors = explanations.plot.contributions_overall(top_n=3, top_k=5, remaining=True)

[4]:

overall_model.show()

[5]:

for top_predictor in model_top_predictors:
    top_predictor.show()

By level in the action hierarchy

[6]:

selected_context = explanations.aggregates.unique_contexts()[0]
selected_context

[6]:

{'pyChannel': 'PegaBatch',
 'pyDirection': 'E2E Test',
 'pyGroup': 'E2E Test',
 'pyIssue': 'Batch',
 'pyName': 'P13'}

[7]:

context_header, overall_by_context, context_top_predictors = explanations.plot.contributions_by_context(
    context=selected_context,
    top_n=3,
    top_k=5,
)
display(context_header)
display(overall_by_context)

[8]:

for top_predictor in context_top_predictors:
    top_predictor.show()

Advanced data exploration

For more control, you can work with the aggregates object directly, inspect the underlying data, and build custom analyses.

[9]:

aggregates = explanations.aggregates # load the aggregated data

Inspect data for the overall model

Get the top-n predictors and their contributions using predictor_contributions().

[10]:

df_overall = aggregates.predictor_contributions(top_n = 3, remaining=False)
df_overall

[10]:
shape: (3, 10)
context_partitionpredictor_namepredictor_typecontributioncontribution_abscontribution_weightedcontribution_weighted_absfrequencycontribution_mincontribution_max
strstrstrf64f64f64f64i64f64f64
"whole_model""pyName""SYMBOLIC"-0.0212420.021307-0.0212450.021319214-0.0425130.02603
"whole_model""Age""NUMERIC"-0.0111850.011656-0.0111080.0114889136-0.0320990.023485
"whole_model""Occupation""SYMBOLIC"-0.0088610.010097-0.0089210.0100729152-0.0297880.06411

Inspect the most influential values of those predictors using predictor_value_contributions().

[11]:

top_n_predictors = df_overall.select(pl.col('predictor_name')).unique().to_series().to_list()
aggregates.predictor_value_contributions(
    predictors=top_n_predictors,
    top_k = 2,
    remaining=False
)

[11]:
shape: (7, 14)
context_partitionpredictor_namepredictor_typebin_orderbin_contentscontributioncontribution_abscontribution_weightedcontribution_weighted_absfrequencycontribution_mincontribution_maxsort_columnsort_value
strstrstri64strf64f64f64f64i64f64f64strf64
"whole_model""Age""NUMERIC"0"MISSING"-0.0124880.01451-0.0004660.000542341-0.0320990.023485"bin_order"0.0
"whole_model""Age""NUMERIC"4"[38.000:43.000]"-0.0128950.013182-0.0012420.00127880-0.0267610.014212"bin_order"4.0
"whole_model""Occupation""SYMBOLIC"5"MISSING"-0.0119890.012913-0.0003140.000339240-0.02680.029334"contribution_abs"0.012913
"whole_model""Occupation""SYMBOLIC"45"Psychotherapist, child"-0.0163640.016364-0.0002450.000245137-0.029788-0.007757"contribution_abs"0.016364
"whole_model""Occupation""SYMBOLIC"31"TEFL teacher"0.0189420.0191320.0003390.000343164-0.0118960.06411"contribution_abs"0.019132
"whole_model""pyName""SYMBOLIC"4"P18"-0.0245490.024549-0.0012920.001292485-0.037306-0.009315"contribution_abs"0.024549
"whole_model""pyName""SYMBOLIC"2"P1"-0.025620.02562-0.0013960.001396502-0.042513-0.010836"contribution_abs"0.02562

Inspect data by level in the action hierarchy

The same methods can drill down into the action hierarchy instead of looking at the overall model.

[12]:

import random
context_info = random.choice(aggregates.unique_contexts())
print('Selected random context: \n')
for key, value in context_info.items():
    print(f'{key}: {value}')
df_by_context = aggregates.predictor_contributions(
    context=context_info,
    top_n=3,
    remaining=False)
df_by_context

Selected random context:

pyChannel: PegaBatch
pyDirection: E2E Test
pyGroup: E2E Test
pyIssue: Batch
pyName: P5

[12]:
shape: (3, 10)
context_partitionpredictor_namepredictor_typecontributioncontribution_abscontribution_weightedcontribution_weighted_absfrequencycontribution_mincontribution_max
strstrstrf64f64f64f64i64f64f64
"{"partition":{"pyChannel":"Peg…"pyName""SYMBOLIC"-0.0225710.022571-0.0225710.022571414-0.033339-0.004782
"{"partition":{"pyChannel":"Peg…"Occupation""SYMBOLIC"-0.00940.010793-0.0091720.010597464-0.0229030.046346
"{"partition":{"pyChannel":"Peg…"Age""NUMERIC"-0.0101680.010452-0.0102910.010496439-0.0210890.015328
[13]:

top_n_predictors = df_by_context.select(pl.col('predictor_name')).unique().to_series().to_list()
aggregates.predictor_value_contributions(
    predictors=top_n_predictors,
    top_k=2,
    context=context_info,
    remaining=False)

[13]:
shape: (7, 14)
context_partitionpredictor_namepredictor_typebin_orderbin_contentscontributioncontribution_abscontribution_weightedcontribution_weighted_absfrequencycontribution_mincontribution_maxsort_columnsort_value
strstrstri64strf64f64f64f64i64f64f64strf64
"{"partition":{"pyChannel":"Peg…"Age""NUMERIC"0"MISSING"-0.0083470.009816-0.0002470.00029113-0.018750.004834"bin_order"0.0
"{"partition":{"pyChannel":"Peg…"Age""NUMERIC"5"[42.500:48.000]"-0.0132010.013201-0.0012930.00129343-0.021089-0.006434"bin_order"5.0
"{"partition":{"pyChannel":"Peg…"Age""NUMERIC"6"[48.000:53.000]"-0.0134030.013403-0.0013130.00131343-0.019928-0.00803"bin_order"6.0
"{"partition":{"pyChannel":"Peg…"Occupation""SYMBOLIC"48"MISSING"-0.0064840.009999-0.0000420.0000653-0.0156160.005273"contribution_abs"0.009999
"{"partition":{"pyChannel":"Peg…"Occupation""SYMBOLIC"12"Psychotherapist, child"-0.0180790.018079-0.0004680.00046812-0.022903-0.014263"contribution_abs"0.018079
"{"partition":{"pyChannel":"Peg…"Occupation""SYMBOLIC"22"TEFL teacher"0.02460.02460.000530.00053100.0161590.046346"contribution_abs"0.0246
"{"partition":{"pyChannel":"Peg…"pyName""SYMBOLIC"1"P5"-0.0225710.022571-0.0225710.022571414-0.033339-0.004782"contribution_abs"0.022571
[ ]: