Creating Metrics & Storyboards for New Hire Failure Risk Models

Description: This guide is intended to provide users with instructions on on creating metrics and storyboards when deploying your first machine learning model of a new type, such as classification or regression. This process requires coordination with your CS lead and data engineer to add the necessary code to your processing script to build the tables and dimensions that are used to create the metrics and storyboards and receive access to our Demo One Model site to view the template storyboards. In many cases, if you are working with a different recipe within the same type of model, you will use the same tables and dimensions, so data engineering work is not necessary. In some cases, you may need something custom and new, in which you would need to work with your CS team.

Module Type: Functional function sym.png

Level: Intermediate-Advanced I-Spaceship.svg

Audience: Model & storyboard creators

Prerequisites: Access to and experience creating metrics & storyboards in One Model. "One AI Recipes", "Model Deployment", & "New Hire Failure Risk Model" modules 

New Hire Success / Failure Risk Model Metric & Storyboard Installation Guide

  1. Before your data engineer can add the necessary tables and dimensions to your processing script, you must deploy a new hire success / failure model with SHAP enabled in the global settings. 
  2. Submit a ticket to have your data engineer add the required code to your processing script.
  3. Allow your One Model site to reprocess. 
  4. Submit a ticket to request One AI access to the Demo One Model site. 
  5. Review the templated storyboards and metrics to determine which ones you are interested in building. You do not need to build every storyboard page from the template—only create what you need to share your model's insights. Note: Only create the metrics and storyboard for sharing model results at the individual-level if you have internal approval to do so. 
  6. Using the metric guide, create the necessary metrics.
  7. Using the storyboard guide, create pages of interest. Note: Pay close attention to the notes and guidance for each storyboard to ensure you build each tile correctly. 
  8. Ensure storyboard viewers have the appropriate data access to review. 

The One Model team is here if you get stuck!

New Hire Success / Failure Metric Guide

Follow the Metrics for New Hire Success / Failure Risk Models Google Sheet for guidance on creating each metric. You are welcome to modify metrics for your organizational needs. For example, the low/medium/high risk buckets for the risk level metrics can be adjusted by editing the metrics based on the your needs and risk tolerance, but the parameters described below provide a good starting point. 

Ensure you are only providing metric and storyboard permission to appropriate data access roles. 

Note: Many metrics are filtered by the label value. Model creators set these labels in the One AI Recipe Screen during the "Give your prediction target meaningful labels" step. In this guide, the label for new hire failures is referred to as "Failure", and the label for new hire successes is called "Success". You can customize these names to fit your organization's needs, but we recommend keeping label names consistent across models of the same recipe type to avoid the need to create new metrics for each model.

Storyboard Guide

Storyboard Page 1: Model Information & Performance

https://demo-1.onemodel.us/app/storyboard/35263/21212 

Notes:

  • Go into edit mode, add an 'Augmentation' storyboard filter, and filter to a model that you intend to use. Then save (this will make reading the charts easier)
  • Update the Recipe Summary Statement or remove (it’s not dynamic, so will need to be updated if the model was edited). 
  • Disable Pagination on all tables
  • Ensure hyperlinks are pasted linked and make them bold and blue so they stand out
  • Optional conditional formatting in the performance table

Storyboard Page 2: Model Details & Drivers

https://demo-1.onemodel.us/app/storyboard/35263/21214

Notes:

  • Disable Pagination on all tables
  • Ensure hyperlinks are pasted linked and make them bold and blue so they stand out
  • Feature Importance (sorted from highest impact to lowest) and directionality tile
    • Several of the columns should be renamed, either in the Table & Column Label Editor for the site or directly in the tile
  • Chart tiles
    • Move the legends to the top and reverse order in some cases

Storyboard Page 3: Where Does Risk Sit?

https://demo-1.onemodel.us/app/storyboard/35263/21215

Notes:

  • Dimensions can be changed to whatever you interested in analyzing, as long as it's joined properly
  • Risk of Terminating Red/Orange/Green tiles
    • Percent Axis - set Max Value to 100
    • Move legend to top and reverse legend
  • Predictions horizontal bar charts
    • Move legend to top
    • Remove axis labels

Storyboard Page 4: Filter View

https://demo-1.onemodel.us/app/storyboard/35263/21266

Notes:

  • Disable Pagination on all tables
  • Add key dimensions to filter bar
  • Risk Level Breakout Red/Orange/Green tiles
    • Percent Axis - set Max Value to 100
    • Move legend to top and reverse legend
  • Directionality and Feature Importance for Predicting Risk tiles
    • Move legend to top
  • Trend tiles
    • Turn on forecast

Storyboard Page 5: Forecasts

https://demo-1.onemodel.us/app/storyboard/35263/21265

Notes:

  • Ensure hyperlinks are pasted linked and make them bold and blue so they stand out
  • Disable Pagination on all tables
  • Trend tiles
    • Turn on forecast
  • Update text/insights to be more relevant to your customer. These are just general ideas to drive home why this model matters

Storyboard Page 6: Individual Predictions

https://demo-1.onemodel.us/app/storyboard/35263/21321

Notes:

  • Review the demo version internally with your legal team or other relevant stakeholders to ensure compliance
  • Disable Pagination on all tables
  • Add 'Person' dimension to the filter bar in design mode and save - Storyboard viewers must select one and only one person from the model population with this filter. Nothing will be displayed on this storyboard until the filter is applied.
  • Feature Impacts on This Person's Prediction (sorted from most impact to least) tile
    • Several of the columns should be renamed, either in the Table & Column Label Editor for the site or directly in the tile
    • Average Absolute SHAP Value tile is hidden
  • Feature Impacts on Prediction and Feature Value Difference from Mean tiles
    • Move legend to top

Was this article helpful?

0 out of 0 found this helpful

Comments

0 comments

Please sign in to leave a comment.