The confidence lab
Measure calibration, rank errors and compare prediction sets on sentiment data.
Reading order
- Does 90% confidence mean 90% accuracy? A baseline with 256 sentiment predictions
- Why can the same predictions produce different calibration errors?
- When accuracy stays the same, does probability quality stay the same?
- Which examples attract the model’s most confident errors?
- Can we compare confidence from a classification head and a generative model?
- How much overconfidence can one temperature parameter correct?
- How stable is a temperature estimate from a few dozen examples?
- Temperature scaling or isotonic regression: does more flexibility help?
- Is one global temperature enough under class imbalance?
- Can calibration parameters transfer across models and templates?
- Can a model make fewer mistakes by answering fewer questions?
- Maximum probability, entropy and score margin: which ranks errors best?
- How reliable is an abstention gain when there are few errors?
- Choosing an abstention threshold without leaking test data
- Does better calibration improve the ranking used for abstention?
- Beyond one answer: building a minimal conformal prediction set
- Who is covered by a 90% coverage guarantee?
- Why small calibration sets can produce much larger prediction sets
- When overall coverage meets its target, what happens to subgroups?
- LAC and APS: are smaller sets better at the same coverage?
- Does calibration transfer from short sentences to long film reviews?
- What changes when a classifier reads less?