Calibrating your decision-making: A guide for risk professionals RISK-ACADEMY Blog

One of the fundamental problems with heatmaps and other qualitative risk assessment methodologies is the inconsistency and subjectivity in human perceptions of risk and uncertainty. This complexity is highlighted in the study by Shou and Olney (2020), which found that self-reported risk attitudes often intertwine perceived uncertainty and risk, with variations influenced by prior knowledge and emotional reactions. Similarly, research on illness risk perceptions (NCBI) emphasizes the psychological factors that shape perceptions, while the Frontiers study explores the emotional connection between uncertainty and decision-making. The Slovic & Weber study further underscores the multifaceted nature of risk perception, especially in the context of extreme events. These studies collectively reveal that human perceptions of risk are far from uniform and are shaped by a blend of cognitive and emotional factors.

This inherent subjectivity and inconsistency pose challenges not only for qualitative but for quantitative risk assessments, since there are plenty of expert judgements in monte-carlo models, starting from methodological choices to input assumptions.

Luckily, calibration, the practice of aligning our judgments with reality, can help risk professional and decision makers to improve their estimates.

What is calibration?

Calibration is the process of aligning one’s judgments and predictions with actual outcomes. It’s about understanding how confident we should be in our predictions and adjusting our confidence levels to match reality. In other words, if you’re 90% confident in a prediction, you should be right about 90% of the time.

Why is calibration important?

Calibration is not just a theoretical concept; it has practical applications in risk management, decision-making, planning, forecasting, and more. A well-calibrated professional can make more accurate predictions, better appreciate uncertainty and the limits of knowledge, leading to better decisions and outcomes.

How can you calibrate yourself?

The Open Philanthropy Project offers a free and user-friendly tool for calibration training, available at Open Philanthropy Calibration. This tool provides a series of questions and feedback to help you understand how well your confidence levels align with reality.

Here’s how you can use it:

Take the test: Answer a series of questions with varying levels of difficulty.
Assess your confidence: For each question, you’ll provide a confidence level for your answer.
Receive feedback: After completing the test, you’ll receive feedback on how well your confidence levels matched the correct answers.
Repeat and improve: Regular practice helps in honing your calibration skills.

Applying calibration in your company

Calibration is not just for individual growth; it can be a powerful tool within your organization. Here’s how you can apply it:

Train decision makers: Encourage decision-makers within your company to take the calibration test.
Integrate into processes: Incorporate calibration into decision-making, planning, and forecasting processes before any important assumptions need to be made.
Monitor and adjust: Regularly assess and adjust calibration levels to ensure ongoing accuracy.

Calibration is a practical and valuable skill for risk professionals. By aligning our judgments with reality and by calibrating decision makers, we can make more accurate and effective risk-based decisions. The Open Philanthropy calibration tool provides an accessible way to develop this skill. Enjoy!