How to measure anything in Global Health: 5. Calibration techniques
Calibrate because you care!
In my previous essay, I discussed how calibrating your judgments is a crucial step when measuring anything in global health.
For instance, if you want to estimate the likelihood a new cancer treatment causes severe side effects, then a calibrated estimator can offer a 90% uncertainty interval about a plausible range of side effect rates as a starting place for measurement.
Although it is essential in the measurement process, calibration can be a tricky state to achieve.
Here are 5 recommendations for how to achieve calibration with your judgment.
This material comes from Douglas Hubbards marvelous book, How to Measure Anything. I recommend you read the book if these posts interest you. You will find much more thorough explanations of these concepts there.
1. The equivalent bet test
Ask yourself, would you rather accept a bet to win $1000 if the true value falls in your uncertainty interval or accept a bet to spin a roulette wheel with a wedge of 10% of its area where you win nothing and 90% of its area where you win $1000. If you are calibrated, you should be indifferent between these bets. If you prefer your uncertainty interval, you’re overconfident. If you prefer the wheel, you’re underconfident.
2. The premortem test
Explain what would be wrong with your interval if the true value turns out to be outside your interval. If you can come up with a plausible failure mode, then perhaps your interval is too narrow.
3. Test the limits
Exam the lower limit, then the upper limit of the interval. For a 90% interval, 95% of the plausible values should be above the lower limit. Does your limit reflect that estimate? Then move to the upper limit and assess it its plausible that 95% of values are below that limit. This helps to avoid anchoring bias, something I found myself susceptible to as I started exploring calibration.
4. The absurdity test
Start with an absurdly wide range and progressively narrow it, explaining why it can’t be that wide along the way. This test helps to locate the edge of our knowledge about the estimate by eliminating things that are obviously not correct.
5. Repetition and feedback
Go to the HTMA website and download the calibration tests to take, or make up your own questions, estimate the answers, and grade them. Calibration is not a skill that comes naturally to most people, but is a skill that must be practiced. There’s a large body of literature on the topic you can also reference, something I was unfamiliar with until recently.
It takes a little time and effort, but soon you can calibrate your judgment to measure anything in global health that will improve the care you provide to your patients!