Value Finder Analysis¶
Every Value Finder simulation populates a dataset, the pyValueFinder dataset. This dataset contains a lot more information than is what is currently presented on screen.
The data held in this dataset can be analysed to uncover insights into your decision framework. This notebook provides a sample analysis of the Value Finder simulation results.
In the data folder we’ve stored a copy of such a dataset, generated from an (internal) demo application (CDHSample). To run this notebook on your own data, you should export the pyValueFinder dataset from Dev Studio then follow the instructions below.
This is how Value Finder results of the sample data are presented in Pega (8.6, it may look different in other versions):
For the sample provided, the relevant action setting is 1.2%. There are 10.000 customers, 3491 without actions, 555 with only irrelevant actions and 5954 with at least one relevant action.
PDSTools defines a class ValueFinder that wraps the operations on this dataset. The “datasets” import is used for the example but you won’t need this if you load your own Value Finder dataset.
Just like with the ADMDatamart class, you can supply your own path and filename as such:
vf = ValueFinder(path = '[PATH TO DATA]', filename="[NAME OF DATASET EXPORT]")
If only a path is supplied, it will automatically look for the latest file.
It is also possible to supply a dataframe as the ‘df’ argument directly, in which case it will use that instead.
[2]:
from pdstools import ValueFinder, datasets
import polars as pl
import numpy as np
# Load sample data - replace with your own data:
# vf = ValueFinder(path='...', filename='...')
vf = datasets.sample_value_finder()
When reading the data, we filter out unnecessary data, and the result is kept in the df property:
[3]:
vf.df.head(5).collect()
[3]:
| Adminputsource | TreatmentKeyCode | DynamicTemplateName | IsEmailEnabled | Value | Issue | Quantity | Bundlehead | CustomerHash | Component | Istransactional | Direction | OptimizePredictionOutcome | Propensity | Weight | Channel | Operator | Group | LastStage | ModelControlGroup | Division | Updateoperator | SpecifySMSMessage | ModelEvidence | StartingEvidence | TemplateName | ImageURL | ModelExecutionError | Label | SubjectType | AdjustedPropensityOmni | Applyanalytics | Pricing | EligibilityDescription | ActionLimit | ModelPropensity | Prediction | … | DecisionReference | MaturityCap | ModelPerformance | Unit | WorkID | InteractionID | StartingPropensity | Benefits | Treatment | PropensityThreshold | SMSMessage | ShortDescription | Agentcompensation | Lastchannel | Application | Segment | Strategy | IsSMSEnabled | Name | ActionContext | ContextWeight | Journey | Priority_2 | Interaction | TemplateType | ClickthroughURL | DeliverOffline | SubjectHash | UpdateDateTime | KeyCode | IsWebEnabled | BundleName | EmailSubject | SpecifyEmailSubject | Communicateto | IsPaidEnabled | Cost |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| str | str | str | bool | f64 | cat | i64 | bool | i64 | str | bool | cat | bool | f64 | f64 | cat | str | cat | str | str | str | str | bool | i64 | u32 | str | str | bool | str | cat | f64 | bool | str | str | i64 | f64 | str | … | str | f64 | str | str | str | str | str | str | str | f64 | str | str | str | str | str | str | str | bool | str | str | str | str | f64 | str | str | str | bool | str | i64 | str | bool | str | str | bool | str | bool | f64 |
| "modelReferences" | "" | "ChannelAction_Template" | true | null | "Sales" | null | false | 10 | "Customers" | false | "Outbound" | true | 0.269231 | null | "SMS" | "Otto Perdeck" | "DepositAccounts" | null | "Test" | "Div" | "vashp" | false | 96 | 100 | "" | "web/DepositAccountOffer.png" | null | "Student Checking" | "CDHSample-Data-Customer" | null | true | "No monthly maintenance fee whe… | "For students ages 17-24, with … | null | 0.269231 | null | … | " 6320879002230024255" | 0.1768 | "0.5" | "Unit" | "Opp_NBA_AlDF_Simulat-1" | "6320879002230024255" | "0.5" | "Students under age 23 are elig… | "Deposit_SMSTreatment" | 0.0 | "" | "Best of all, with Student Chec… | "2" | null | "CDHSample" | null | "Opp_NBA_All" | true | "StudentChecking" | "Customer" | null | null | null | "——" | "" | "/MS/index.html?Site=NonBundled… | false | "0.4388105018174723" | 1629803581090 | "SDSC2" | null | null | null | null | null | null | null |
| "modelReferences" | "MOBILE" | "Usage_ChannelAction_Template" | true | null | "Usage" | null | null | 1 | "Customers" | false | "Outbound" | true | 0.5 | null | "SMS" | "Otto Perdeck" | "Mobilebanking" | null | "Test" | "Div" | "vashp" | true | 0 | 25 | "Mobile_SMSDBTemplate" | null | null | "Get the U+ Mobile App" | "CDHSample-Data-Customer" | null | true | null | null | null | 0.5 | null | … | " 6320879002230024184" | 0.02 | "0.5" | "Unit" | "Opp_NBA_AlDF_Simulat-1" | "6320879002230024184" | "1" | null | "Mobile_ExternalSMSTreatment" | 0.0 | "You are entitled for an offer" | "Manage accounts from your mobi… | null | null | "CDHSample" | null | "Opp_NBA_All" | true | "GetTheUMobileApp" | "Customer" | null | null | null | "——" | "DB" | null | true | "0.940830237245977" | 1629803581074 | "UMGUMA1" | null | null | null | null | null | null | null |
| "modelReferences" | "" | "Collections_ChannelAction_Temp… | null | null | "Collections" | null | null | 0 | "Customers" | false | "Outbound" | true | 0.5 | null | "SMS" | "Otto Perdeck" | "Recommendations" | null | "Test" | "Div" | "vashp" | false | 0 | 25 | "" | null | null | "Setup Autopay today" | "CDHSample-Data-Customer" | null | true | null | null | null | 0.5 | null | … | " 6320879002230023934" | 0.02 | "0.5" | "Unit" | "Opp_NBA_AlDF_Simulat-1" | "6320879002230023934" | "1" | null | "Recommendations_SMSTreatment" | 0.0 | "" | "Setup autopay to avoid missing… | null | null | "CDHSample" | null | "Opp_NBA_All" | true | "SetupAutopayToday" | "Customer" | null | null | null | "——" | "" | null | false | "0.8026197582592971" | 1629803581035 | "CRSAT1" | true | null | null | null | null | null | null |
| "modelReferences" | "" | "ChannelAction_Template" | true | null | "Sales" | null | false | 10 | "Customers" | false | "Outbound" | true | 0.269231 | null | "SMS" | "Otto Perdeck" | "DepositAccounts" | null | "Test" | "Div" | "vashp" | false | 96 | 100 | "" | "web/DepositAccountOffer.png" | null | "Student Checking" | "CDHSample-Data-Customer" | null | true | "No monthly maintenance fee whe… | "For students ages 17-24, with … | null | 0.269231 | null | … | " 6320879002230023873" | 0.1768 | "0.5" | "Unit" | "Opp_NBA_AlDF_Simulat-1" | "6320879002230023873" | "0.5" | "Students under age 23 are elig… | "Deposit_SMSTreatment" | 0.0 | "" | "Best of all, with Student Chec… | "2" | null | "CDHSample" | null | "Opp_NBA_All" | true | "StudentChecking" | "Customer" | null | null | null | "——" | "" | "/MS/index.html?Site=NonBundled… | false | "0.2919499647730856" | 1629803581030 | "SDSC2" | null | null | null | null | null | null | null |
| "modelReferences" | "" | "ChannelAction_Template" | null | null | "Sales" | null | null | 5 | "Customers" | false | "Outbound" | true | 0.15 | null | "SMS" | "Otto Perdeck" | "Bundles" | null | "Test" | "Div" | "vashp" | false | 91 | 100 | "" | null | null | "Student Choice" | "CDHSample-Data-Customer" | null | true | "No fee for all your basic need… | null | null | 0.15 | null | … | " 6320879002230023923" | 0.1082 | "0.6029929577464789" | "Unit" | "Opp_NBA_AlDF_Simulat-1" | "6320879002230023923" | "0.5" | "Free Online Banking with e-Sta… | "Bundles_SMSTreatment" | 0.0 | "" | "All the essentials are include… | null | null | "CDHSample" | null | "Opp_NBA_All" | true | "StudentChoice" | "Customer" | null | null | null | "——" | "" | null | false | "0.45582757506999766" | 1629803581034 | "SBSC1" | true | "Student Choice" | null | null | null | null | null |
Basic Analysis: Pie Charts and Customer Segmentation¶
The piechart shown in platform is based on a propensity threshold. For the sample data, this threshold follows from a propensity quantile of 5.2%.
The plot.pie_charts() function shows piecharts for all stages in the engagement policies (in platform you only see the last one). The threshold is calculated automatically, but you can specify it explicitly.
[4]:
# Automatic threshold calculation
vf.plot.pie_charts()
# Or specify threshold explicitly (e.g., 5.2% quantile)
vf.plot.pie_charts(quantiles=[0.052])
Hover over the charts to see the details. For the sample data, the rightmost pie chart corresponds to the numbers in Pega as shown in the screenshot above.
Red = customers not receiving any action
Yellow = customers not receiving any “relevant” actions, sometimes also called “under served”
Green = customers that receive at least one “relevant” action, sometimes also called “well served”
With “relevant” defined as having a propensity above the threshold. This defaults to the 5th percentile.
Propensity Analysis¶
Insights into the propensity distribution per stage are crucial. The plot.propensity_threshold() function visualizes this distribution. You often see a spike at 0.5, which corresponds to models without responses (their propensity defaults to 0.5/1 = 0.5).
The dotted vertical line represents the computed threshold.
[5]:
vf.plot.propensity_threshold()
These different propensities represent
pyModelPropensity = the actual propensities from the models
pyPropensity = model or random propensity, depending on the ModelControl (or, when models are executed from an extension point after the standard Predictions, their propensity, but such a configuration is not supported by Value Finder)
FinalPropensity = the propensity after possible adjustments by Thompson Sampling; Thompson Sampling basically smoothes the propensities, you would expect any peak at 0.5 caused by empty models to be smoothed out
We can also look at the propensity distributions across the different stages. This is based on the model propensities, not any of the subsequent overrides:
[6]:
vf.plot.propensity_distribution()
Threshold Simulation¶
The effect of the propensity threshold selection on the number of actions for a customer can be simulated by supplying a list of quantiles or propensities. This generates aggregated counts per stage for different threshold values.
[7]:
# Simulate thresholds from 1% to 99% in 1% increments
vf.plot.pie_charts(quantiles=np.arange(0.01, 1, 0.01))
The further to the left you put the slider threshold, the more “green” you will see. As you raise the threshold, more customers will be reported as getting “not relevant” actions.
The same effect can be visualized as a funnel. Use plot.distribution_per_threshold() to show the threshold on the x-axis.
[8]:
# Default threshold range
vf.plot.distribution_per_threshold()
# Custom threshold range (1% to 99% in 1% increments)
vf.plot.distribution_per_threshold(quantiles=np.arange(0.01, 1, 0.01))
Funnel Analysis¶
You can analyze how individual actions are distributed across stages. Since there are typically many actions, it’s useful to filter to a specific group or issue.
The plot.funnel_chart() function shows how actions flow through the stages. Rule of thumb: If there are only a few actions in each stage, or if certain actions are completely filtered out between stages, this may indicate strong filtering issues.
[9]:
# Filter to Sales issue actions (adjust query for your data)
vf.plot.funnel_chart("Name", query=pl.col("Issue") == "Sales")
Hierarchical Funnel Views¶
Start with a coarse-grained view at the Issue level, then drill down to Group or individual Action level as needed.
[10]:
# High-level view: Issue level
vf.plot.funnel_chart("Issue")
# Mid-level view: Groups within Sales issue
vf.plot.funnel_chart("Group", query=pl.col("Issue") == "Sales")