Question 1

What is self-consistency and why does it cost more?

Accepted Answer

Self-consistency is a prompting technique where, instead of trusting a single answer, you sample the model several times at a non-zero temperature and take the answer the samples agree on most — the majority vote. It works because a good model is right more often than any particular way of being wrong, so the correct answer tends to be the one that recurs. The cost is blunt: running N samples costs N times as much as one call, because you pay for every token of every sample. The accuracy, meanwhile, does not rise linearly — it climbs fast from one to three samples, then flattens, so the fifth and seventh samples buy far less accuracy than the first extra one did. That mismatch — linear cost, flattening benefit — is the whole economics of the technique, and it is what this calculator prices: the cost per correct answer at each N, so you can see where the extra samples stop being worth it.

Question 2

How much does majority voting actually improve accuracy?

Accepted Answer

It depends entirely on the single-sample accuracy, and the direction flips at 50%. If one sample is right more than half the time, majority voting pushes accuracy upward toward 100% as you add samples — this is the same law of large numbers that makes a slightly biased coin land its favoured side in a long run of flips. A model at 70% single-sample accuracy might reach roughly 78% with three votes and around 84% with five. But if one sample is right less than half the time, voting makes it worse, not better: the majority now reinforces the wrong answer, and more samples drive accuracy toward zero. This calculator models the voted accuracy with a binomial majority — the probability that more than half of N independent samples are correct — and flags the case where your single-sample accuracy is below 50%, because there voting is actively harmful and you should fix the prompt or the model instead of spending on samples.

Question 3

Is self-consistency cheaper than just using a better model?

Accepted Answer

Often not, and that is the comparison that matters. Voting buys accuracy, but it buys it at N times the cost, so its cost per correct answer usually rises even as its raw accuracy improves — five samples of a cheap model can cost more per usable result than one call to a pricier, more accurate model. Voting tends to win only when a stronger single model is either much more expensive or simply unavailable at the accuracy you need, or when the task is one where extra samples cheaply resolve a genuine ambiguity. This calculator puts the numbers side by side: the cost per correct answer for N samples of your model against a single call to an alternative you specify, so you can see whether you are better off paying for votes or paying for a better model. The rule of thumb it makes concrete is that self-consistency is an accuracy tool first and a cost tool almost never.

Question 4

When is paying for self-consistency worth it?

Accepted Answer

When a wrong answer costs you far more than a few extra samples do. If you are extracting a value that feeds a financial decision, gating an automated action, or grading at scale where one bad output has a real downstream cost, spending three or five times on samples to move accuracy from, say, 85% to 95% can be cheap insurance — the expected cost of the mistakes you avoid exceeds the cost of the votes. Where it rarely pays is on high-volume, low-stakes work: a wrong answer that a human quickly corrects, or output that gets reviewed anyway, does not justify multiplying every call. This calculator helps you draw that line by showing both the cost per correct answer and the accuracy each N buys, so you can weigh the extra spend against what a mistake actually costs you — the number that is not on the calculator but should be in your head.

Calculadora de custos de autoconsistência

Precisão e custo em cada N

Votar compra precisão, quase nunca economia

Como funciona esta calculadora

Perguntas frequentes

O que é autoconsistência e por que custa mais?

Quanto a votação por maioria realmente melhora a precisão?

A autoconsistência é mais barata do que apenas usar um modelo melhor?

Quando vale a pena pagar pela autoconsistência?