New York Site views —Page views —
← All series

The confidence lab

Measure calibration, rank errors and compare prediction sets on sentiment data.

Reading order

  1. Does 90% confidence mean 90% accuracy? A baseline with 256 sentiment predictions
  2. Why can the same predictions produce different calibration errors?
  3. When accuracy stays the same, does probability quality stay the same?
  4. Which examples attract the model’s most confident errors?
  5. Can we compare confidence from a classification head and a generative model?
  6. How much overconfidence can one temperature parameter correct?
  7. How stable is a temperature estimate from a few dozen examples?
  8. Temperature scaling or isotonic regression: does more flexibility help?
  9. Is one global temperature enough under class imbalance?
  10. Can calibration parameters transfer across models and templates?
  11. Can a model make fewer mistakes by answering fewer questions?
  12. Maximum probability, entropy and score margin: which ranks errors best?
  13. How reliable is an abstention gain when there are few errors?
  14. Choosing an abstention threshold without leaking test data
  15. Does better calibration improve the ranking used for abstention?
  16. Beyond one answer: building a minimal conformal prediction set
  17. Who is covered by a 90% coverage guarantee?
  18. Why small calibration sets can produce much larger prediction sets
  19. When overall coverage meets its target, what happens to subgroups?
  20. LAC and APS: are smaller sets better at the same coverage?
  21. Does calibration transfer from short sentences to long film reviews?
  22. What changes when a classifier reads less?