Whenever subject matter experts make quantitative assessments in a business process, we check those assessments after the fact and report back on precision and bias so that our SMEs can improve their future assessments. This is rather more difficult where the assessments are probabilities because the verification is necessarily statistical. Traditional methods can reveal optimism and pessimism in predictions, but they are not so effective at revealing cases where the impact of positive or negative evidence is over- or under-incorporated into an assessment and probabilities are either polarized – i.e. too extreme – or sitting on the fence – i.e .not extreme enough. They also demand substantial numbers of assessments.
I present an alternative approach to exposing systematic bias, which is based on a deeper look at how systematic bias effects probabilities. The tool produces a “bias footprint” which in a single figure shows not only the optimism or pessimism in a sequence of assessments, but also the tendency to polarize or fence-sit, together with a clear sense of the extent to which the data support those conclusions, which is dependent on both the number of assessments and the consistency of the bias.
I will demonstrate the method on Hubbard Decision Research’s FrankenSME aggregation methodology, showing how the aggregation method substantially improves predictions by removing systematic biases.
https://2021.riskawarenessweek.com/talks/how-expert-are-your-experts/
Check out other risk management books
RISK-ACADEMY offers online courses
Informed Risk Taking
Learn 15 practical steps on integrating risk management into decision making, business processes, organizational culture and other activities!
ISO31000 Integrating Risk Management
Alex Sidorenko, known for his risk management blog http://www.riskacademy.blog, has created a 25-step program to integrate risk management into decision making, core business processes and the overall culture of the organization.
Advanced Risk Governance
This course gives guidance, motivation, critical information, and practical case studies to move beyond traditional risk governance, helping ensure risk management is not a stand-alone process but a change driver for business.
