Predictive Model Evaluation Using Receiver Operating Characteristic Curves

Track: Data Analytics Symposium

Session Number: 2009
Date: Thu, Aug 1st, 2019
Time: 10:30 AM - 12:00 PM
Room: Grand Saguaro West

Description:

More important than simply building a model is the ability to determine its accuracy in predicting outcomes. Before releasing scores to your stakeholders, you want to be sure that you have selected the optimal modeling approach for your purposes.

Receiver Operating Characteristic (ROC) Curves and their associated Area Under the ROC Curve (AUC) metric are useful tools for both graphically identifying the optimal classification threshold, and quantitatively determining the effectiveness of a given predictive model in correctly classifying individual cases.

This sixty-minute presentation will demonstrate the effective use of ROC/AUC in assessing model performance by showcasing several packages in R, as well as by demonstrating a method by which both an ROC/AUC can be generated using SQL to populate a Tableau report. Participants will come away with a better understanding of this simple and useful tool, with snippets of code that they can immediately implement in their own shops using either R or SQL.
Session Type: Breakout Session (90min)

Primary Competency: DA:Competency 4: Statistical Techniques and Competencies
Secondary Competency: DA:Competency 3: Data Manipulation Skills
Tertiary Competency: DA:Competency 5: Visualization/Reporting Techniques and Competencies
Intended Audience Level: Level I
Recommended Prerequisites: Cursory Statistical Background
Knowledge of SQL
Knowledge of R
Learning Objective #1: Attendees will come away with an understanding of predictive model evaluation techniques, and a how-to guide for implementing ROC/AUC evaluations in their own shops.
Learning Objective #2: Attendees will be able to generate ROC/AUC visualizations for both internal and external consumption using either R or Tableau.
Shop Size: Small, Mid-Size/Large, All Shop Sizes
Session Type: Breakout Session (90min)

Primary Competency: DA:Competency 4: Statistical Techniques and Competencies
Secondary Competency: DA:Competency 3: Data Manipulation Skills
Tertiary Competency: DA:Competency 5: Visualization/Reporting Techniques and Competencies
Intended Audience Level: Level I
Recommended Prerequisites: Cursory Statistical Background
Knowledge of SQL
Knowledge of R
Learning Objective #1: Attendees will come away with an understanding of predictive model evaluation techniques, and a how-to guide for implementing ROC/AUC evaluations in their own shops.
Learning Objective #2: Attendees will be able to generate ROC/AUC visualizations for both internal and external consumption using either R or Tableau.
Shop Size: Small, Mid-Size/Large, All Shop Sizes