Neural Network Lexicon

Utility Curves

Short Definition

Utility curves visualize how the usefulness of a model changes across decision thresholds or operating conditions.

Definition

Utility curves plot the expected utility (benefit minus cost) of a model as a function of decision thresholds, class prevalence, or other operating parameters. Unlike error-based metrics, utility curves explicitly encode the value of correct decisions and the cost of incorrect ones.

Utility is defined by the application and reflects real-world outcomes rather than abstract performance.

Why It Matters

In practice, different decisions have different consequences. Utility curves help practitioners:

  • align model behavior with business or safety goals
  • compare models using outcome-based criteria
  • select thresholds that maximize real-world value

They provide a decision-centric alternative to accuracy or AUC.

How It Works (Conceptually)

  • Assign utilities (or costs) to outcomes (TP, FP, TN, FN)
  • Compute expected utility for each threshold or condition
  • Plot utility against the operating parameter
  • Identify regions of maximum utility

The optimal model behavior maximizes expected utility, not accuracy.

Conceptual Formula

Expected Utility =
TP × Utility_TP +
TN × Utility_TN +
FP × Utility_FP +
FN × Utility_FN

Minimal Python Example


Python
expected_utility = (
tp * u_tp +
tn * u_tn +
fp * u_fp +
fn * u_fn
)


Common Pitfalls

  • Defining arbitrary or poorly justified utilities
  • Ignoring uncertainty in utility estimates
  • Assuming utility is static across contexts
  • Comparing utilities without consistent definitions

Related Concepts

  • Expected Cost Curves
  • Cost-Sensitive Learning
  • Decision Thresholding
  • Operating Point Selection
  • Evaluation Metrics