Creating Metrics & Storyboards for High Performer Likelihood Models

Description: This guide is intended to provide users with instructions on creating metrics and storyboards when deploying your first machine learning model of a new type, such as classification or regression. This process requires coordination with your CS lead and data engineer to add the necessary code to your processing script to build the tables and dimensions that are used to create the metrics and storyboards and receive access to our Demo One Model site to view the template storyboards. In many cases, if you are working with a different recipe within the same type of model, you will use the same tables and dimensions, so data engineering work is not necessary. In some cases, you may need something custom and new, in which you would need to work with your CS team.

Module Type: Functional

Level: Intermediate-Advanced

Audience: Model & storyboard creators

Prerequisites: Access to and experience creating metrics & storyboards in One Model. "One AI Recipes", "High Performer Likelihood Model" & "Model Deployment" modules 

Before You Get Started

1. Before your data engineer can add the necessary tables and dimensions to your processing script, you must deploy a high performer model with SHAP enabled in the global settings. 
2. Submit a ticket to have your data engineer add the required code to your processing script.
3. Allow your One Model site to reprocess. 
4. Submit a ticket to request One AI access to the Demo One Model site. 
5. Review the templated storyboards and metrics to determine which ones you are interested in building. You do not need to build every storyboard page from the template—only create what you need to share your model's insights. Note: Only create the metrics and storyboard for sharing model results at the individual-level if you have internal approval to do so. 
6. Using the metric guide, create the necessary metrics.
7. Using the storyboard guide, create the color palettes (if desired) and storyboard pages of interest. Note: Pay close attention to the notes and guidance for each storyboard to ensure you build each tile correctly. 
8. Ensure storyboard viewers have the appropriate data access to review. 

The One Model team is here if you get stuck!

High Performer Likelihood Metric Guide

Follow the Metrics for High Performer Likelihood Models Google Sheet for guidance on creating each metric. You are welcome to modify metrics for your organizational needs. For example, the low/medium/high likelihood buckets for the likelihood level metrics can be adjusted by editing the metrics based on the your needs and risk tolerance, but the parameters described below provide a good starting point. 

Ensure you are only providing metric and storyboard permission to appropriate data access roles. 

Note 1:Many metrics are filtered by label values. Model creators set these labels in the One AI Recipe Screen during the "Would you like to add more meaningful labels to the values you’re predicting?" step. In this guide, the label for employees predicted to be high performers is "High Performer," while the label for those not predicted to be high performers is "Not High Performer." You can customize these names to fit your organization's needs; however, we recommend keeping label names consistent across models of the same recipe type to avoid needing new metrics for each model.

Note 2: Time Functions Type for all metrics in this guide are ‘None’ and Restrictions should not be applied.  

Storyboard & Color Palette Guide

Custom color palettes

#3680D5

#E15759

#E15759

#F28E2B

#59A14F

#E15759

#59A14F

Storyboard Page 1: Performance Management

https://demo-1.onemodel.us/app/storyboard/38214/28018 

Notes:

• This storyboard page does not contain predictive analytics. It is a summary of relevant performance management to provide context to the high performer model. You can use something you already have on your site or create one using ideas from this storyboard page. 

Storyboard Page 2: Model Info + Insights (Summarized)

https://demo-1.onemodel.us/app/storyboard/38214/28018 

Notes:

• Go into edit mode, add an 'Augmentation' storyboard filter, and filter to a model that you intend to use. Then save (this will make reading the charts easier)
• Update the Recipe Summary Statement or remove (it’s not dynamic, so will need to be updated if the model was edited). 
• Disable Pagination on all tables
• Ensure hyperlinks are pasted linked and make them bold and blue so they stand out
• Optional conditional formatting in the performance table
• Ensure hyperlinks are pasted linked and make them bold and blue so they stand out
• Feature Importance (sorted from highest impact to lowest) and directionality tile
  • Several of the columns should be renamed, either in the Table & Column Label Editor for the site or directly in the tile
• Chart tiles
  • Move the legends to the top and reverse order in some cases
• Specifically for the Where Does High Performer Potential Sit? section
  • Dimensions can be changed to whatever you interested in analyzing, as long as it's joined properly
    • Risk of Terminating Red/Orange/Green tiles
      • Percent Axis - set Max Value to 100
      • Move legend to top and reverse legend
    • Predictions horizontal bar charts
      • Move legend to top
      • Remove axis labels
• Specifically for the Forecasts section
  • Trend tile(s), turn on forecast
  • Update text/insights to be more relevant to your customer. These are just general ideas to drive home why this model matters

Storyboard Page 3: Model Info + Insights (Detailed)

https://demo-1.onemodel.us/app/storyboard/38214/28084 

Notes:

• This is simply a more detailed version of Storyboard Page 2. You should create it in the same way, incorporating all the notes mentioned above.

Storyboard Page 4: Individual Predictions

https://demo-1.onemodel.us/app/storyboard/38214/28088 

Notes:

• Review the demo version internally with your legal team or other relevant stakeholders to ensure compliance
• Disable Pagination on all tables
• Add 'Person' dimension to the filter bar in design mode and save - Storyboard viewers must select one and only one person from the model population with this filter. Nothing will be displayed on this storyboard until the filter is applied.
• Feature Impacts on This Person's Prediction (sorted from most impact to least) tile
  • Several of the columns should be renamed, either in the Table & Column Label Editor for the site or directly in the tile
  • Average Absolute SHAP Value tile is hidden
• Feature Impacts on Prediction and Feature Value Difference from Mean tiles
  • Move legend to top