Most guides take you from “here is a PDF” to “here is a chatbot” in fifteen minutes, then stop. You get something that works on the demo questions and fails on Tuesday. This one goes past the demo: every component, published prices, a monthly cost model with the arithmetic shown, and an eval plan you can run on Monday.
Short answer: to build an AI knowledge assistant using RAG you need nine components working in sequence: ingest, chunk on structure, embed, retrieve with BM25 and dense vectors together, filter by metadata, rerank, assemble context, generate, cite. Hybrid retrieval plus reranking is the configuration that survives contact with real users. And the bill is dominated by generation tokens, not storage.
I costed one scenario end to end: 12,000 documents, roughly 14.4 million tokens, about 32,000 chunks, 20,000 questions a month. Managed infrastructure lands between $62 and $319 a month depending on stack. Vector storage for the whole corpus is about six cents a month. Nobody budgets the retrieval half properly, so they get surprised by generation.
How to build an AI knowledge assistant using RAG in 2026
RAG (short for retrieval-augmented generation) is the pattern of fetching relevant passages from your own data at query time and putting them in the model’s context window, so answers are grounded in sources you control rather than frozen in model weights. The term comes from Lewis et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (arXiv:2005.11401), first submitted 22 May 2020 and accepted at NeurIPS 2020. Over six years old, still the canonical citation.
What changed since 2020 is the plumbing. Current LangChain retrieval documentation describes three architectures worth choosing between deliberately:
- Two-step RAG: retrieval always runs before generation. Predictable, low latency, right for FAQs and documentation bots.
- Agentic RAG: an LLM decides when and how to retrieve during reasoning. Flexible, variable latency, right for research assistants with multiple tools.
- Hybrid RAG: query preprocessing, retrieval validation, post-generation checks. Medium on both counts.
Start with two-step. Move to agentic only when an eval set shows failures that fixed retrieval causes, because agentic RAG multiplies latency and eval cost.
Sources for the architecture and cost model
Pricing reflects vendor pages and Microsoft’s Retail Prices API checked on 16 October 2026. Package versions come from PyPI, npm and GitHub releases. Research results are attributed to the cited papers.
The cost model calculates from published unit prices. It does not represent a deployed system’s invoice, and no retrieval or latency benchmarks were run for this guide. Cohere’s pay-as-you-go Embed and Rerank rates remain unconfirmed here.
The pipeline, honestly: nine stages
- Ingestion: pull documents from their sources, parse them, and keep a stable document ID and source URL per chunk. Without that you cannot produce a working citation later.
- Chunking: split documents into retrievable units. See the next section; this is where most tutorials mislead you.
- Embedding: turn each chunk into a dense vector.
text-embedding-3-smallreturns 1,536 dimensions andtext-embedding-3-largereturns 3,072 by default, both capped at 8,192 tokens of input, per OpenAI’s embeddings documentation. - Hybrid retrieval: run BM25 keyword search and dense vector search, then fuse the two ranked lists.
- Metadata filtering: restrict the candidate pool by tenant, permission, date or document type before ranking.
- Reranking: pass the top 50 candidates through a cross-encoder and keep the best 5 to 10.
- Context assembly: pack the surviving chunks, ordered so the strongest evidence is not buried.
- Generation: send the assembled context with a grounded system prompt.
- Citation: return source links and quote spans alongside the answer, not as an afterthought.
Why fixed-size chunking is the wrong default
A chunk is the unit of retrieval, so anything that makes a chunk unanswerable on its own is a bug, not a trade-off. Most tutorials open with a fixed splitter (512 tokens, 64 overlap) and never revisit it.
The failure is concrete. Chunk a quarterly report and you get “Revenue grew by 3% over the previous quarter.” Which company? Which quarter? That chunk is unretrievable: a dense embedding of it is nearly meaningless, and BM25 cannot match words that are not in the chunk.
Four fixes, in rough order of effort-to-payoff:
- Chunk on document structure. Headings, table rows, FAQ pairs, numbered clauses. Free, and usually the biggest single win.
- Prepend context per chunk. Anthropic’s Contextual Retrieval prepends a short generated description of where the chunk sits before embedding and indexing. Vendor-reported, from August 2024: using
1 − recall@20as the metric, with Gemini Text 004 embeddings and Cohere reranking: contextualised embeddings alone cut top-20 retrieval failure from 5.7% to 3.7%, and adding contextual BM25 cut it to 2.9%. Anthropic reports a one-time cost of $1.02 per million document tokens to generate that context, using prompt caching. - Late chunking. Günther et al., “Late Chunking” (arXiv:2409.04701, submitted 7 September 2024) embeds the whole document first, then pools token vectors into chunk vectors, so each chunk embedding carries full-document context.
- Summarised parents. Retrieve small leaf chunks, hand the model the parent section. RAPTOR (arXiv:2401.18059, 31 January 2024) builds a recursive tree of summaries for this.
Use them together if your evaluation says so. Not because a blog said so.
Why hybrid retrieval usually beats dense-only
BM25 is a lexical ranking function scoring documents by term frequency, inverse document frequency and a length-normalising saturation; it matches exact strings that dense vectors blur. Ask it for “error code TS-999” and it finds TS-999. Ask an embedding model and you get documents about error codes.
The gap is structural, not incidental. Dense retrieval compresses meaning into a fixed-length vector; lexical retrieval preserves surface form. Neither dominates:
- BM25 catches: product codes, SKUs, error strings, statute numbers, people’s names, exact phrases, rare proper nouns.
- Dense catches: paraphrases, cross-language queries, conceptual matches, questions phrased nothing like the document.
Fuse them with reciprocal Rank Fusion, which merges ranked lists by summing 1/(k + rank). No score normalisation required: the practical reason it dominates weighted score blending. Recommended by Elastic, implemented in Qdrant’s Query API as rrf and dbsf, and described in Pinecone’s hybrid search guide as one of three ways to combine a keyword signal with a dense signal.
The broader evidence is consistent but mostly vendor-run. BEIR (arXiv:2104.08663, 17 April 2021) showed zero-shot retrieval is far harder than benchmarks suggested and that BM25 is a genuinely strong baseline. That paper is over five years old. Treat “hybrid beats dense” as a well-supported default you should still verify, not a law.
What is established versus what is hype
Four claims circulate constantly. Here is what the primary literature actually supports.
Agentic RAG: established, and harder than it looks. Letting a model decide when to retrieve is a documented architecture with a documented trade-off, not a buzzword. LangChain’s current docs price it explicitly: low control, high flexibility, variable latency. Self-RAG (arXiv:2310.11511, 17 October 2023) and CRAG (arXiv:2401.15884, 29 January 2024) both showed real gains, and both predate current models. Ship two-step first.
Late-interaction retrieval (ColBERT-style): effective, usually not worth it here. ColBERT (arXiv:2004.12832, SIGIR 2020) encodes query and document separately into many token-level vectors and matches them late, keeping BERT-level effectiveness at two orders of magnitude less latency per query. It needs an index sized for multi-vector storage though, and for a 32,000-chunk corpus a cross-encoder gets most of the benefit for a fraction of the complexity.
GraphRAG: real, and scoped to one question type. Microsoft’s GraphRAG (arXiv:2404.16130, submitted 24 April 2024) builds an entity graph plus community summaries to answer global questions (“what are the main themes across these 10,000 documents”) where conventional RAG fails because the task is query-focused summarisation, not lookup. The paper’s own claim is narrow: gains over a conventional RAG baseline on comprehensiveness and diversity for global sensemaking questions over roughly one-million-token datasets. If your user asks “what does our refund policy say”, GraphRAG is the wrong tool.
Long context versus RAG: settled enough to act on. Longer context windows do not remove the need for retrieval above a few hundred thousand tokens; they shift the break-even point. Retrieval meets Long Context LLMs (arXiv:2310.03025, 4 October 2023) framed it as a trade-off and Lost in the Middle (arXiv:2307.03172, 6 July 2023) showed position-dependent degradation. Both predate current architectures: re-test the threshold with your own tokens.
Choosing a vector database: the comparison
Every option below can answer a knowledge-base query. They differ on who runs it, what it costs at rest, and whether hybrid search and reranking are built in or something you assemble.
| Option | Free tier / floor | Storage and query price (16 Oct 2026) | Hybrid search | Reranking included | Licence / hosting |
|---|---|---|---|---|---|
| Pinecone | Starter $0 (2 GB, 1M read + 1M write units/mo); Builder $20/mo; Standard $50/mo minimum | Storage $0.33/GB/mo; read units $16–$18 per million; write units $4–$4.50 per million | Dense, sparse and full-text in one index; RRF or filter-then-rank | Yes: bge-reranker-v2-m3, pinecone-rerank-v0, cohere-rerank-4-fast at $2 per 1,000 requests | Managed |
| Qdrant | Free forever: single node, 0.5 vCPU / 1 GB RAM / 4 GB disk | Usage-based hourly on vCPU, RAM, storage and backups; Qdrant publishes no flat per-GB rate | Sparse + dense, fused with rrf or dbsf | Cloud Inference free tier; paid models carry free tokens | Apache-2.0 OSS or managed |
| Weaviate | Free forever: 100,000 objects, 1 GB RAM, 10 GB disk, 1 collection | Flex from $45/mo; vector dimensions from $0.00465 per 1M; storage from $0.12/GiB; backups from $0.03036/GiB | Hybrid on all paid plans | No: use its Query Agent or an external reranker | Managed (open-source core) |
| Zilliz Cloud / Milvus | Free: 5 GB storage, 2.5M vCU/mo, up to 5 collections | Dedicated clusters from $63 / $16 / $5 per million vectors/mo by tier; rates assume 768-dim vectors | Via sparse + dense indexes | No | Milvus is Apache-2.0 OSS; Zilliz is managed |
| Chroma | Starter $0/mo + $5 credits; 10 databases | $0.33/GiB/mo storage; $2.50/GiB written; $0.0075 per TiB queried | Vector, full-text and metadata search; scale plan $250/mo | No | Apache-2.0 OSS or cloud |
| pgvector | Free (it is a Postgres extension | Postgres instance cost only | Yes) combine with Postgres full-text search, RRF or a cross-encoder | No | PostgreSQL licence |
| LanceDB | OSS is free | OSS free; managed pricing needs a live calculator | Unified vector, full-text and hybrid with SQL filters | No | Apache-2.0 OSS |
| FAISS | Free | Library only: you own the serving layer | No | No | C++/Python library from Meta FAIR |
| Elasticsearch / OpenSearch | Free self-hosted; hosted tiers vary | Elastic Cloud is feature-gated by tier; OpenSearch bills per instance-hour (r6g.xlarge.search at $0.418/hr in AWS’s own worked example) | First-class hybrid with RRF recommended by Elastic | Elastic has a semantic ranker tier; OpenSearch has neural plugins | Commercial / Apache-2.0 |
| Vespa | Free to self-host | Self-host free; Vespa Cloud commercial, on request | nearestNeighbor composes with BM25 in one YQL query | Custom ranking profiles | Apache-2.0 |
| Vectara | 30-day trial only | SaaS from $100,000/year; VPC from $250,000/year; on-prem from $500,000/year | Built in | Built in | Managed |
Three things this table tells you that marketing pages do not. The free tiers are genuinely usable: a 32,000-chunk corpus is a rounding error against Weaviate’s 100,000-object free tier, Zilliz’s 5 GB free tier or Pinecone’s 2 GB Starter allowance. The managed floors dominate your bill early: Pinecone’s $50 minimum and Weaviate’s $45 exceed everything else combined at small scale. And Vectara has left the self-serve market, with its cheapest SaaS tier starting at six figures a year.
One caveat on Pinecone: its pricing page illustrates throughput with worked examples (roughly 15,000 searches a day for a 30,000-document knowledge base with 1024-dimension embeddings on Starter) with an explicit disclaimer that these are illustrative only and subject to change. Sanity check, not quote.
Frameworks and managed platforms: own one layer, not five
The frameworks are free and mostly interchangeable. What you are really choosing is who owns the ingestion logic and the citation contract.
| Tool | Current version (16 Oct 2026) | Licence | What it is actually good for |
|---|---|---|---|
| LlamaIndex | 0.14.25 (PyPI, 21 Sep 2026) | MIT | Data connectors and ingestion; IngestionPipeline caches per-node transformations |
| LangChain | 1.4.6 (PyPI, 14 Oct 2026) | MIT | Retrieval abstractions and broad provider coverage |
| LangGraph | 1.2.15 (PyPI, 12 Oct 2026) | MIT | Stateful multi-step agentic RAG with cycles and persistence |
| Haystack | 3.4.0 (PyPI, 15 Oct 2026) | Apache-2.0 | Pipelines you can inspect and unit-test |
| DSPy | 3.4.1 (PyPI, 12 Oct 2026) | MIT | Optimising prompts and retrieval programs from a labelled set |
| RAGFlow | v1.0.0-rc2 (GitHub, 16 Oct 2026) | Apache-2.0 | Document-heavy ingestion with layout-aware parsing |
| Dify | Active, 158,658 GitHub stars | Mixed | Low-code app builder; self-hosting needs 2 CPU cores and 4 GiB RAM |
| Flowise | 3.1.4 (GitHub, 29 Jul 2026) | Mixed | Visual drag-and-drop RAG builder |
| Langflow | v1.12.6 (GitHub, 13 Oct 2026) | MIT | Visual builder with a large component ecosystem |
| Open WebUI | 0.12.3 (PyPI and GitHub, 16 Oct 2026) | Mixed | Self-hosted chat UI with built-in document RAG |
If a managed RAG service already covers your compliance boundary, use it. The economics:
| Managed RAG service | Price (16 Oct 2026) | What it includes |
|---|---|---|
| AWS Bedrock Managed Knowledge Base | Index storage $5.00 per GB of raw data/month; standard Retrieve API $1.00 per 1,000 calls; managed parsing, embeddings and reranking $0; agentic retrieval $4.00 per 1,000 agentic calls plus $1.00 per 1,000 underlying Retrieve calls | Ingestion, multimodal parsing, vector storage, hybrid retrieval, managed reranker |
| Google Vertex AI: grounding on your data | $2.50 per 1,000 prompts | Grounded generation over your own corpus; you supply and pay for the model |
| Azure AI Search | Free tier with 1,000 queries a month; Basic S1 $0.336/hour; Standard S2 $1.344/hour; Standard S3 $2.688/hour; semantic ranker $1.00 per 1,000 queries; serverless tier at $0.24/hour compute plus $0.20/GB/month storage, effective 1 September 2026 | Search engine with hybrid retrieval and semantic rank |
| Snowflake Cortex Search | Serving billed per GB/month of indexed data; embedding via EMBED_TEXT_768 or EMBED_TEXT_1024 at a per-token credit cost; virtual warehouse refresh credits on top | Search service over Snowflake tables |
Azure figures come from Microsoft’s official Retail Prices API, filtered to productName eq 'Azure AI Search' and armRegionName eq 'eastus'. Many meters carry 2025 effective dates: that is the API reporting last-change dates, not new pricing.
Embeddings and rerankers: current options and unit prices
An embedding model converts text into a fixed-length vector so you can measure semantic similarity with cosine distance; a reranker is a slower, more accurate model that scores query-document pairs directly. Both now have serious open-weight options.
| Provider / model | Price (16 Oct 2026) | Notes |
|---|---|---|
OpenAI text-embedding-3-small | $0.02 per 1M tokens | 1,536 dims, 8,192 token max input, 62.3% on OpenAI’s cited MTEB figure |
OpenAI text-embedding-3-large | $0.13 per 1M tokens | 3,072 dims, 64.6% MTEB; dimensions parameter for Matryoshka truncation |
Voyage voyage-4-lite | $0.02 per 1M tokens | First 200 million tokens free per account |
Voyage voyage-4 / -4-large / -context-4 | $0.06 / $0.12 / $0.12 per 1M | Same 200M free-token allowance |
Voyage rerank-3 | $0.05 per 1M processed tokens | First 200M tokens free; Voyage estimates $0.0025 per request at 100 documents of ~500 tokens |
Voyage rerank-3-lite | $0.02 per 1M | Same free allowance |
Cohere Embed 5 (embed-v5.0-pro / -fast) | Per-token billing; rate not readable from the public page | 128k context; output dimensions 256–2048 |
Cohere Rerank 4 (rerank-v4.0-pro / -fast) | Priced per search, not per token; rate not readable from the public page | Model Vault dedicated: Rerank 4 Pro Medium $5.00/hour or $3,250/month |
| Pinecone Inference | llama-text-embed-v2 $0.16/M tokens; multilingual-e5-large $0.08/M | 5M tokens/mo included on Starter, 10M on Builder |
Jina AI jina-embeddings-v5-text-small / -nano | Pricing page renders client-side; free tier available | 1024-dim and 768-dim, 32K and 8K context |
BAAI/bge-m3 | Free, open weights | Dense + multi-vector + sparse in one model, 100+ languages |
Qwen/Qwen3-Embedding-8B | Free, open weights | Per arXiv:2506.05176 (5 June 2025) |
Two notes. OpenAI’s embeddings page states both v3 models have no knowledge of events after September 2021: fine for retrieval, but do not expect them to know anything current. And Hugging Face’s Inference Providers route through many vendors with no markup, showing $2.00 monthly credits for PRO users and $2.00 per seat for Team and Enterprise.
Evaluation: the section most guides skip
You cannot improve a RAG system you cannot measure, and you cannot debug one whose retrieval and generation scores are fused into a single number.
Split the metrics in two
Retrieval metrics need labelled relevant passages per query. Three matter:
- Recall@k: of the passages that genuinely answer the question, what fraction appear in the top k returned? This tells you whether the right chunk is even reachable. If recall@20 is 0.6, reranking cannot save you; retrieval is the problem.
- MRR (mean reciprocal rank): average of
1/rankof the first correct hit. Sensitive to ordering, which recall@k is not. Reranking moves MRR a lot and recall@k barely, which is how you prove the reranker worked. - nDCG@k: normalised discounted cumulative gain. Rewards putting the best passage at rank one rather than rank four. Use it when queries have several valid answers of differing quality.
Generation metrics need an LLM judge and can be reference-free:
- Faithfulness / groundedness: does every claim trace back to a retrieved passage? The hallucination detector.
- Answer relevancy: does the answer address the question asked?
RAGAS’s metrics catalogue lists these under its RAG section (Context Precision, Context Recall, Context Entities Recall, Noise Sensitivity, Response Relevancy, Faithfulness, plus multimodal variants) all reference-free, per the RAGAS paper (arXiv:2309.15217, submitted 26 September 2023).
The frameworks, as installed on 16 October 2026
| Framework | Version | Licence | Best for |
|---|---|---|---|
| RAGAS | 0.4.3 (13 Jan 2026) | Apache-2.0 | Reference-free RAG metrics in Python; broadest metric catalogue |
| TruLens | 2.15.0 (5 Oct 2026) | See PyPI | App tracing plus RAG triad scoring (context relevance, groundedness, answer relevance) |
| DeepEval | 4.3.0 (15 Oct 2026) | Apache-2.0 | Pytest-style assertions; works like a test suite for prompts |
| promptfoo | 0.124.1 (npm, 8 Oct 2026) | MIT | Declarative YAML configs, red-teaming, CI diffing across prompt versions |
| ARES | Paper, NAACL 2024 (submitted 16 Nov 2023) | Research code | The academic option: trains lightweight judges on synthetic data and needs only a few hundred human labels, using prediction-powered inference for calibration |
Promptfoo is npm while the others are Python: that is a real operational difference. If your team is Node-first, promptfoo is the path of least resistance.
A worked evaluation set for a business
Build this before you build the chatbot.
- Sample 200 real questions. Pull from support tickets, sales call notes or search logs. Include 20% you know the current system cannot answer: those are where faithfulness fails.
- Label the gold passages. Mark which chunks answer each question. Two analysts on the first 50, reconcile disagreements, then use that as a quality bar for the rest. Budget roughly a day per 100 questions.
- Measure retrieval cold. Record recall@5, recall@20 and MRR@10 for the current configuration before changing anything. This baseline beats any vendor benchmark.
- Measure generation. Run faithfulness and answer relevancy with RAGAS or DeepEval. Cheap judge model routinely, expensive one only when validating a release.
- Change one thing. Chunk size, embedding model, top-k, reranker: one at a time. Two changes and you have learned nothing.
- Re-run and diff. Track metrics against the config hash. promptfoo does this natively.
- Gate CI. Fail the build when faithfulness drops past a set threshold. The step teams skip and regret.
Keep this set forever. It is the highest-leverage asset in a RAG project: every future model or configuration decision becomes an A/B test rather than an argument.
What a RAG assistant costs per month: the worked model
Assume:
- Corpus: 12,000 documents at 1,200 tokens each = 14,400,000 tokens
- Chunking: 512-token windows, 64-token overlap → 448-token stride → 14,400,000 ÷ 448 = 32,142 chunks
- Embedding: each chunk embedded once = 32,142 × 512 = 16,456,704 tokens
- Vectors:
text-embedding-3-small, 1,536 dims, float32 = 6,144 bytes each → 197 MB ≈ 0.20 GB - Volume: 20,000 questions a month; top-50 candidates → hybrid fusion → rerank → top-8 to the model
- Per query: 4,100 uncached input tokens + 600 cacheable tokens, 350 output tokens
- Rerank per query: 50 documents × 512 tokens, plus query tokens counted per document (50 × 20) = 26,600 processed tokens
One-time ingestion cost
| Model | Rate | 16,456,704 tokens | Cost |
|---|---|---|---|
text-embedding-3-small | $0.02 / 1M | 16.46M | $0.33 |
text-embedding-3-large | $0.13 / 1M | 16.46M | $2.14 |
Vector storage for 32,142 chunks of a 12,000-document corpus is about $0.06 a month on Pinecone Standard ($0.33/GB) and $0.23 a month on Weaviate Flex ($0.00465 per million vector dimensions, 49.4M dimensions). Generation tokens are where the money actually is: plan your budget around the model, not the database.
Recurring monthly cost, per line item (20,000 questions)
| Line item | Calculation | Cost |
|---|---|---|
| Query embeddings | 20,000 × 20 tokens = 0.4M × $0.02/M | $0.01 |
| Retrieval: Pinecone Standard | Within the $50/mo minimum | $50.00 |
| Retrieval: Bedrock Managed KB | 20 × $1.00 per 1,000 calls | $20.00 |
| Retrieval: Vertex grounding | 20 × $2.50 per 1,000 prompts | $50.00 |
Rerank: Voyage rerank-3 | 532.0M tokens × $0.05/M | $26.60 |
| Rerank: Pinecone-hosted | 20 × $2.00 per 1,000 requests | $40.00 |
| Rerank: Bedrock managed reranker | $0 | $0.00 |
Generation: gpt-5-mini | 82M × $0.25 + 12M × $0.025 + 7M × $2.00 | $34.80 |
Generation: gpt-6-luna | 82M × $0.10 + 12M × $0.01 + 7M × $0.20 | $9.72 |
Generation: gpt-6.1-sol | 82M × $2.00 + 12M × $0.10 + 7M × $10.00 | $235.20 |
| Evaluation: 200 Q × 4 configs × 4 runs | answers $6.05 + judges $1.15 | $7.20 |
The five stacks, totalled
| Stack | Components | Monthly total |
|---|---|---|
| 1. Cheapest managed | Pinecone Standard + Voyage rerank-3 + gpt-5-mini + eval | $118.60 |
| 2. Frontier generation | Pinecone Standard + Voyage rerank-3 + gpt-6.1-sol + eval | $319.00 |
| 3. Managed RAG service | Bedrock Managed Knowledge Base + gpt-5-mini + eval | $62.29 |
| 4. Self-hosted | pgvector on a $24/mo DigitalOcean Basic Droplet (4 GiB, 2 vCPU) + local bge-m3 embeddings + local bge-reranker-v2-m3 + gpt-5-mini + eval | $66.00 |
| 5. Batch-optimised evals | Stack 1, eval on the OpenAI Batch API (50% of standard pricing) | $114.76 |
Read those numbers again. Switching from gpt-5-mini to gpt-6.1-sol costs $200 a month more than your entire vector database. Conversely, swapping Pinecone for Bedrock Knowledge Bases saves $30: real, but an order of magnitude less than the generation decision.
One correction to a common intuition: going cheaper on generation is not automatically the answer. gpt-6-luna at $9.72 a month is a third the cost of gpt-5-mini, and if it also answers more questions correctly you save on support deflection. Cost per correct answer is the metric, which is why the eval set comes before the cost model.
The free tier path, stated exactly
You can build and demo this for $0 to $20 a month:
- Vector store: Chroma Cloud Starter ($0 + $5 credits), Zilliz Cloud Free (5 GB, 2.5M vCU/mo), Pinecone Starter ($0, 2 GB), Weaviate Free (100,000 objects, 10 GB disk) or Qdrant Cloud Free (1 GB RAM, 4 GB disk)
- Embeddings: Voyage’s first 200M tokens free per account, or open weights like
bge-m3 - Orchestration: Dify, Flowise or Langflow self-hosted; Dify’s documented floor is 2 CPU cores and 4 GiB RAM
- Generation: whatever you already pay OpenAI for, or a local model
The part that isn’t a dollar figure: engineering time
Estimate the schedule from the work your data and integrations need. Budget for these workstreams:
- Source connectors and parser hardening (the long pole: PDFs are never clean)
- Chunking strategy per document type
- Hybrid retrieval plus fusion implementation
- Reranker integration and latency budget
- Permission-aware metadata filtering
- The 200-question labelled eval set
- Citation rendering in your UI
- CI gates, monitoring and a re-indexing runbook
- Rate limiting, caching and cost alerting
Multiply that by your blended engineering day rate. That is your real number, and it will exceed the infrastructure bill in year one.
Build it: the nine-step implementation plan
- Stand up the eval set first. 200 labelled questions. Nothing else matters until this exists.
- Ingest with provenance. Every chunk gets a document ID, source URL, title and section path. You cannot cite what you cannot locate.
- Chunk on structure, with 512/64 as the fallback for unstructured prose. Record chunk size in your config hash.
- Embed once with a cheap model.
text-embedding-3-smallat $0.33 for this corpus is a rounding error. - Implement hybrid retrieval. BM25 plus dense, fused with RRF. Return 50 candidates.
- Add metadata filters (tenant, permission, date) applied before ranking.
- Rerank to 8. Voyage
rerank-3-liteor a localbge-reranker-v2-m3. Re-measure MRR to prove it worked. - Generate with citations enforced in the prompt, then verify every returned citation resolves.
- Wire the CI gate and re-run the eval set on every prompt, model or chunk-size change.
What breaks: failure modes and gotchas
- Approximate nearest-neighbour recall silently degrades. With pgvector,
hnsw.ef_searchdefaults to 40 and results are limited by it;ivfflat.probesdefaults to 1. Filtering after an approximate index scan is worse: if a filter matches 10% of rows you get roughly 4 matches out of 40 candidates. pgvector 0.8.0 added iterative index scans for this; turn them on. - Changing embedding model requires a full re-index. Vectors from one model are meaningless to an index built with another. Cheap here ($0.33) but a migration, not a config flip.
- Fixed chunks break tables and code. A chunked table row loses its header; a chunked function loses its docstring.
- Context stuffing hurts. More chunks raise the odds of including the answer and of distracting the model. Anthropic’s guidance, vendor-reported from August 2024, was that top-20 beat top-10 and top-5: one team’s result on their data.
- Reranking is a latency and cost tax. 150 chunks per query costs far more than 20. Measure the curve on your own questions.
- “Lost in the Middle” still bites. Long-context models degrade when relevant information sits mid-context. That evidence is over three years old, but put your strongest evidence at the top and the end rather than trusting uniform ordering.
- Permission filtering is the security bug. Filtering after retrieval, or on a field that is not indexed, leaks documents across tenants. Filter at the query.
- Eval drift. A judge model upgrade can move your faithfulness scores with no change to your system. Pin judge snapshots: OpenAI recommends pinning production snapshots such as
gpt-5.5-2026-04-23and building eval suites alongside.
Legal, policy and compliance
Not legal advice (get counsel for your situation) but three things are load-bearing.
EU AI Act transparency. Article 50 came into force on 2 August 2026 under Article 113. It requires providers to ensure AI systems intended to interact directly with natural persons are designed so those people are informed they are interacting with an AI system, unless it is obvious to a reasonably well-informed person. A customer-facing knowledge assistant almost certainly triggers this. A one-line disclosure in the chat header satisfies the letter of it.
Data protection. Everything you index is data you process. Under UK and EU GDPR that needs a lawful basis, and personal data in the corpus needs one too.
Provider terms. OpenAI’s embeddings documentation states that customers own their input and output including for embeddings, and are responsible for ensuring content complies with applicable law. Read the same clause at your provider, then check whether regulated data touches the pipeline: most knowledge-base corpora contain more than people expect.
The bottom line
Build an AI knowledge assistant using RAG with hybrid retrieval, structure-aware chunking, and a reranker between retrieval and generation. Those three choices account for most of the difference between a demo and something people rely on.
Budget $62 to $319 a month for 20,000 questions a month on managed infrastructure, and expect engineering time to dwarf that in year one. Vector storage is pennies. Generation tokens are the line item. If your bill feels wrong, the fix is almost never a different database.
Build the 200-question eval set before you write the retriever. Everything else here is reversible; that set is the asset that compounds.
FAQ
What is the cheapest way to build a RAG knowledge assistant?
The cheapest real path is a free vector tier plus a database you already pay for. Chroma Cloud Starter is $0 a month with $5 of credits, Zilliz Cloud Free includes 5 GB of storage and 2.5 million vCU a month, Pinecone Starter is free with 2 GB of storage and 1 million read units a month, and Weaviate’s free tier holds 100,000 objects. Running pgvector inside existing Postgres drops the vector line to zero. You still pay for generation tokens and, if you rerank, rerank calls. Budget $0 to $20 a month for a working prototype.
How much does it cost to run a RAG assistant per month?
For the scenario modelled here (12,000 documents, roughly 32,000 chunks, 20,000 questions a month) managed options run from about $62 a month on AWS Bedrock Managed Knowledge Base with a small OpenAI model, to about $119 a month on Pinecone Standard, to about $319 a month on Pinecone Standard with a frontier-class generation model. A self-hosted pgvector setup on a $24 a month DigitalOcean droplet plus a small hosted model comes to roughly $66 a month. All figures exclude engineering time.
Is hybrid search actually better than pure vector search?
For knowledge bases, usually yes, and the mechanism is simple. BM25 matches exact strings (error codes, SKU numbers, statute numbers, product names) that dense embeddings blur. Vendor benchmarking supports the direction: Anthropic reported in August 2024 that adding BM25 to dense retrieval cut its top-20 retrieval failure rate from 5.7% to 4.2%, with larger gains on exact-match-heavy corpora. Treat that as a vendor benchmark from about 24 months ago, not settled science, and measure it on your own data.
What chunk size should I use for RAG?
Do not start from a number. Chunk along the structure your documents already have (headings, table rows, question-and-answer pairs) and then measure whether a retrieved chunk can answer a question on its own. A defensible starting configuration is 512 tokens with 64 tokens of overlap, which this article uses throughout the cost model. If you need the parent document’s heading to make a chunk make sense, the chunk is too decontextualised. Late chunking and contextual retrieval are the two well-documented fixes when structure alone is not enough.
Do I need a reranker if I already use hybrid search?
For anything user-facing, yes. Retrieval must cast a wide net to avoid missing the right passage, but you only want a narrow set in the prompt. A cross-encoder reranker sits between the two and is the cheapest quality win available. Anthropic reported that adding reranking on top of contextualised embeddings plus BM25 took top-20 retrieval failure from 5.7% to 1.9%: again a vendor benchmark from August 2024. Confirm the latency and cost trade-off on your own candidate-set sizes.
How do I evaluate whether my RAG system is good?
Measure retrieval and generation separately. Retrieval metrics (recall@k, mean reciprocal rank, nDCG@k) require labelled relevant passages per query. Generation metrics (faithfulness, answer relevancy) need an LLM judge and can run reference-free. RAGAS 0.4.3, TruLens 2.15.0, DeepEval 4.3.0 and promptfoo 0.124.1 implement these. ARES (arXiv:2311.09476, NAACL 2024) is the research-backed option that needs only a few hundred human labels via prediction-powered inference.
Is RAG still worth it with long-context models?
Usually yes for corpora above a few hundred thousand tokens, because long context does not fix retrieval precision and scales poorly on cost. But it is not automatic. Anthropic’s own guidance, published in August 2024, was that under roughly 200,000 tokens (about 500 pages) you can skip retrieval entirely and place the whole corpus in the prompt with prompt caching. Above that, retrieval wins on cost and latency. Verify against your own token bill rather than assuming.
Which vector database should I pick?
Pick by constraint, not by benchmark. Already on Postgres means pgvector at v0.8.7 is usually right and costs nothing incremental. Pure managed with no ops burden points to Pinecone Standard at a $50 monthly minimum or Weaviate Flex at $45. Self-hosted control with good hybrid-query ergonomics points to Qdrant under Apache-2.0. Enterprise search you already own points to Elasticsearch or OpenSearch hybrid search with reciprocal rank fusion. Vectara’s SaaS now starts at $100,000 a year, which excludes it for most teams.
Can I switch embedding models later?
Yes, but treat it as a migration rather than a config change. Changing the embedding model changes the vector space, so the entire corpus must be re-embedded and the index rebuilt. You cannot query an index built with one model using vectors from another. At this scale the embedding cost is trivial (32,142 chunks with text-embedding-3-small is about 33 cents) so schedule the downtime, re-run the eval set, and compare before and after recall@k.
What legal or compliance issues should I check?
Three at minimum. EU AI Act Article 50 has been in force since 2 August 2026 and requires providers of AI systems intended to interact directly with natural persons to tell those people they are interacting with an AI system unless it is obvious. You also need a lawful basis for any personal data you index under UK and EU GDPR. And your model provider’s terms govern what you may send: OpenAI states customers own input and output but are responsible for complying with applicable law. Get qualified counsel for regulated use cases.
Sources
- Pinecone Pricing: plan floors, storage/read/write unit rates, integrated embedding and rerank prices, throughput illustrations
- Qdrant Pricing: free-tier specs, usage-based billing model, uptime SLAs
- Weaviate Cloud Pricing: Free/Flex/Premium tiers, vector-dimension and storage rates, 1 September 2026 price-book change
- Zilliz Cloud Pricing: free tier, dedicated cluster tiers, per-million-vector rates at 768 dimensions
- Chroma Pricing: Starter and Scale plans, per-GiB storage and write charges
- pgvector on GitHub: v0.8.7, HNSW and IVFFlat tuning, iterative index scans, hybrid search recipe
- OpenAI API Pricing: text-embedding-3 prices, generation model rates, cached-input rates, Batch API rates
- OpenAI Vector Embeddings: 1,536 and 3,072 default dimensions, 8,192 token input cap, MTEB figures, pre-September-2021 knowledge caveat
- Voyage AI Pricing: voyage-4 family and rerank-3 rates, 200M free-token allowances, per-request estimate
- Cohere: How Does Pricing Work: Rerank billed per search, Embed billed per token embedded
- Cohere Pricing and Model Vault: dedicated instance hourly and monthly rates for Embed 5 and Rerank 4
- Cohere Embed docs: embed-v5.0-pro and -fast specifications, 128k context, Matryoshka dimensions
- AWS Bedrock Pricing: Managed Knowledge Base index storage, Retrieve API, managed parsing/embedding/reranking, agentic retrieval
- Google Cloud Vertex AI Pricing: grounding on your data, Grounding with Google Search, Gemini 4 Argon token rates
- Microsoft Azure Retail Prices API: Azure AI Search meter rates and effective dates for East US
- Snowflake Cortex Search overview, which components consume credits and how
- DigitalOcean Droplet Pricing: Basic Droplet monthly and hourly rates, per-second billing change from 1 January 2026
- Dify self-hosting docs: minimum 2 CPU cores and 4 GiB RAM
- Vectara Pricing: SaaS from $100,000/year, VPC and on-premise tiers
- LangChain retrieval docs: 2-step, agentic and hybrid RAG architectures
- LlamaIndex Ingestion Pipeline docs: transformation caching, document management, parallel processing
- Pinecone hybrid search guide: filter-then-rank, client-side RRF, server-side dense/sparse combination
- Elastic hybrid search docs: recommendation to use reciprocal rank fusion
- Qdrant hybrid queries docs: prefetch API and rrf/dbsf fusion
- RAGAS metrics catalogue: full list of reference-free RAG metrics
- Hugging Face Inference Providers pricing: monthly credits, routed versus custom provider keys
- Anthropic, Introducing Contextual Retrieval: vendor-reported 49% and 67% retrieval failure reductions, August 2024
- EU AI Act Article 50: transparency obligations, in force 2 August 2026
- arXiv:2005.11401: Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: Lewis et al., submitted 22 May 2020, NeurIPS 2020
- arXiv:2004.12832: ColBERT: late interaction retrieval, SIGIR 2020
- arXiv:2402.03216: M3-Embedding (bge-m3): dense, multi-vector and sparse in one model
- arXiv:2409.04701: Late Chunking: contextual chunk embeddings, submitted 7 September 2024
- arXiv:2506.05176: Qwen3 Embedding: embedding and reranking foundation models, 5 June 2025
- arXiv:2401.18059: RAPTOR: recursive summarisation trees, 31 January 2024
- arXiv:2104.08663: BEIR: zero-shot retrieval benchmark and BM25 baseline, 17 April 2021
- arXiv:2311.09476: ARES: automated RAG evaluation with few hundred human labels, NAACL 2024
- arXiv:2309.15217: RAGAS: reference-free RAG evaluation metrics, 26 September 2023
- arXiv:2307.03172: Lost in the Middle: long-context position bias, 6 July 2023
- arXiv:2310.03025: Retrieval meets Long Context LLMs: retrieval and long-context trade-off, 4 October 2023
- PyPI JSON API: release versions and upload dates for RAGAS, TruLens, DeepEval, LlamaIndex, Haystack, DSPy and others
