Rerank the shortlist, not the haystack
A reranker improves RAG quality cheaply if you feed it a small candidate set. Reranking hundreds per query is the cost trap. Size the retrieval side on the RAG chatbot cost calculator.
Reranking: small model, real bill
Rerankers price per 1,000 searches ($1–$2) or per document scored. The multiplication that surprises: reranking 100 candidates per query at $1/1k searches with 50k monthly queries is $50 — fine. But naive pipelines rerank on every keystroke of search-as-you-type, multiplying query volume 5–10×. Debouncing and reranking only the final query restores sanity. The cost-effectiveness case is strong though: a reranker in front of a RAG pipeline typically lets you retrieve fewer, better chunks — cutting LLM input tokens enough that the reranker pays for itself. Test top-20 rerank versus top-5 raw retrieval; the token savings usually win.