Set and defend the confidence column. Use when choosing confidence values, and before any recalibration. The counter-intuitive rule: calibrate to the labeling policy, not to correctness.
