Build a Private AI Chatbot That Answers Questions From Your Documents

You can build a private AI chatbot that answers questions from your own documents this weekend, with three privacy tiers to choose from. Cloud tools (Gemini Notebook (formerly NotebookLM) and ChatGPT Projects) take about five minutes and promise not to train on your files, but the files still sit on Google’s or OpenAI’s servers. AnythingLLM Desktop keeps everything on your machine in roughly 15 minutes. A full local stack of Ollama plus Open WebUI takes under an hour and gives you the most control. All cost nothing to start.

What is RAG, in plain language?

RAG is the trick that lets a chatbot answer from your files instead of guessing from memory. RAG (retrieval-augmented generation) is a technique where the model fetches relevant facts from an external source before writing its answer, rather than relying only on its training. Explainers from NVIDIA and IBM describe the same four steps every one of these tools runs:

1. Chunking. Chunking is cutting a document into segments (often 500 to 1,500 tokens) small enough to search and to fit in the model’s context window.
2. Embedding. An embedding is a list of numbers capturing what a piece of text means; similar text produces similar numbers, so “refund window” lands near “you may return items within 30 days.”
3. Retrieval. Your question gets embedded the same way, and the system pulls back the chunks closest to it.
4. Generation. The retrieved chunks go into the model’s prompt, and it answers using your actual text.

Chunking exists because a model’s context window is finite: you can’t dump a 400-page contract in whole, so the system feeds it the most relevant paragraphs instead. Cloud tools run all four steps invisibly. Local tools expose them as settings: a blessing for control, a trap when a default silently sabotages you (more below).

Which path should you pick?

Pick by one question first: can your documents touch someone else’s server? Everything else (cost, quality, effort) is secondary. Here’s the comparison, with limits from official pages in November 2026:

Path	Setup effort	Does document content leave your device?	Answer quality	Cost
Gemini Notebook (cloud, zero-code)	~5 minutes	Yes. Google’s servers; not used for training unless you submit feedback	Excellent, with citations to exact passages	Free tier; paid Google AI tiers raise limits
ChatGPT Projects (cloud, zero-code)	~5 minutes	Yes. OpenAI’s servers; no training on Business/Edu/Enterprise by default; personal accounts must opt out	Excellent	Free (5 files/project); Plus $20/mo; Business from $25/user/mo
AnythingLLM Desktop (local, low-code)	~15 minutes	No, on-device; anonymous telemetry on by default, can be disabled	Good, below frontier cloud on messy documents	Free, open source (MIT)
Ollama + Open WebUI (local, full stack)	~45-60 minutes	No. Ollama collects nothing by default; Open WebUI is self-hosted	Good with tuning	Free, open source

Quick fit: Gemini Notebook for non-sensitive research; ChatGPT Projects for Business teams; AnythingLLM Desktop for confidential solo use; Ollama + Open WebUI for small teams who want accounts and roles.

The zero-code paths: private-ish, not private

Zero-code tools are the fastest route and their privacy terms have genuinely improved, but “not used for training” is not “stays on your machine.” Documents are uploaded, processed, and stored on the vendor’s infrastructure either way. For client contracts, patient records, or anything under an NDA, that usually isn’t a fine trade.

Gemini Notebook: the strongest zero-code option

Google renamed NotebookLM to Gemini Notebook on July 16, 2026: same product, existing notebooks carried over. Per Google’s plans and limits page (checked November 2026): free accounts get 100 notebooks with 50 sources each; paid tiers scale that to 300 sources on Pro. The official FAQ caps each source at 500,000 words or 200MB.

A single Gemini Notebook source can hold up to 500,000 words or 200MB: roughly ten full-length books in one notebook, per Google’s own FAQ.

The privacy wording is unusually specific: “Your data is protected, and is not used to train Gemini Notebook unless you provide feedback,” per the same help center. Work and school accounts get stronger terms: no human review of uploads, chats, or outputs, and no training use. Answers carry inline citations to the exact passage, and a May 2026 update added automatic Drive syncing for Docs, Sheets, and Slides.

ChatGPT Projects: good for teams, one personal-account trap

Per OpenAI’s Projects help page, file caps per project are 5 on Free, 25 on Go and Plus, and 40 on Edu, Pro, Business, and Enterprise, with 10 files per upload. Deleted projects are purged within 30 days.

The privacy split matters. On personal accounts (Free, Plus, Pro), project content may be used for training if “Improve the model for everyone” is on, and per OpenAI’s data controls page, that setting is on by default. Turn it off under Settings → Data controls first. Business, Enterprise, and Edu workspaces don’t train on workspace content by default, which is why Business (from about $25 per user monthly per OpenAI’s pricing) makes sense for teams. One quirk: Drive files added to a project don’t auto-sync. ChatGPT accesses them on demand.

The easiest private build: AnythingLLM Desktop

For one person who wants documents kept on the machine, AnythingLLM Desktop is the shortest verified path. AnythingLLM is an open-source (MIT-licensed) desktop app from Mintplex Labs for Mac, Windows, and Linux that bundles a local LLM provider, a local embedding engine, and a local vector database: no accounts, no API keys. Per the official docs, the current release is v1.19.0 and the backend listens only on 127.0.0.1 by default.

The build sequence, per the docs:

1. Download the installer from anythingllm.com and install it.
2. On first launch, pick the built-in “AnythingLLM Default” LLM provider, or connect Ollama, LM Studio, or a cloud API.
3. Download a model sized for your hardware. An 8B-class model is comfortable on a 16GB machine; Google’s Gemma 4 E4B (4.5B effective parameters, 128K context) is in Ollama’s library and widely recommended there.
4. Create a workspace: workspaces group documents and chats.
5. Open the document picker and drop in PDFs, DOCX, or text files. Choose “embed” for large sets (that’s RAG mode) or “attach” to place one or two small files fully in context.
6. Wait for embedding, then chat inside the workspace.
7. If answers feel thin, raise the retrieved-chunk count in the workspace’s chat settings: community testing found only about 4-6 chunks retrieved per query by default (GitHub issue #2007).
8. For zero outbound calls, switch off anonymous telemetry in settings: the privacy page says it collects anonymous usage events via PostHog and none of your chats or documents.

Two honest caveats: the built-in embedder can choke on very large PDFs (reports mention 5,000-plus-page files failing), so split monsters before importing, and split loosely related files across workspaces by topic or retrieval degrades.

The full local stack: Ollama + Open WebUI

If you want multiple user accounts, deeper retrieval settings, or a Docker deployment on an always-on machine, pair Ollama with Open WebUI. Ollama is a free, open-source tool that downloads and serves local models with one command; Open WebUI is a self-hosted web interface that connects to it. Ollama shipped 34 stable releases between August 10 and November 3, 2026, reaching v0.40.14, and Open WebUI hit v0.13.0 on November 4, 2026. Ollama’s privacy page says it does not collect or transmit your prompts or responses, and ships no telemetry.

The build, per Open WebUI’s quick-start docs:

1. Install Ollama, then pull a chat model: ollama pull gemma4:e4b is a solid default for 16GB machines; step up to a 12B or 26B variant if you have the memory.
2. Pull an embedding model too: ollama pull nomic-embed-text (small, fast, 768-dimension vectors) or ollama pull mxbai-embed-large (~670MB, better retrieval quality). Ollama’s library lists nomic-embed-text as its most-pulled embedding model.
3. Start Open WebUI with Docker: docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data -e WEBUI_SECRET_KEY=<openssl rand -hex 32> --name open-webui --restart always ghcr.io/open-webui/open-webui:main. Never skip the data volume, or you lose chats on every container recreation.
4. Open http://localhost:3000 and create your account: the first account becomes admin, and sign-ups then close automatically.
5. Go to Workspace → Knowledge, create a collection, and upload documents; files are auto-indexed for RAG on arrival.
6. Tune chunking under Settings → Admin → Documents. For local models with 8K or smaller context, Open WebUI’s RAG troubleshooting docs recommend token splitting, chunk size 1,000, overlap 100, and Top K of 3-5.
7. Fix the context trap: Ollama defaults a model’s context to 4,096 tokens on GPUs under 24GB, which trims the very chunks RAG just retrieved. Raise it to 16,384 or more in the model’s Advanced Params, per the same docs.
8. Attach the knowledge collection to a model, or reference it in chat with # followed by the collection name.

One licensing note: Open WebUI uses a custom license: free to self-host below 50 users with its branding intact; whitelabeling or large commercial deployments need an enterprise license.

What actually leaves your machine in each path

“Private” is a spectrum, so here’s the data flow per path, stated plainly:

• Gemini Notebook: the full text of your uploads goes to Google’s cloud. It isn’t used for training unless you submit feedback, and feedback may be reviewed with your queries and uploads included, per Google’s help center.
• ChatGPT Projects: uploads go to OpenAI’s cloud. Personal accounts may be used for training unless you opt out; Business/Edu/Enterprise defaults to off (Projects docs).
• AnythingLLM Desktop: documents, chats, embeddings, and the vector store stay on disk. The exception is anonymous telemetry (feature-usage counts, app version, OS) on by default with an in-app off switch. Optional “network discovery” exposes an unauthenticated API to your LAN; leave it off.
• Ollama + Open WebUI: nothing leaves the machine during inference, and Ollama ships no telemetry. Model weights download once from a registry; after that, everything runs offline.

One security flag: the classic local-AI mistake is binding your model server to 0.0.0.0, which puts it on your network: keep everything on 127.0.0.1. And if you handle regulated client data, this article is general guidance, not compliance advice.

Honest quality tradeoffs versus the cloud

A local RAG bot is private, not magic. For clean documents and factual questions, the good 8B-class models answer convincingly. But hallucination remains an open research problem even in 2026 (a comprehensive survey published October 2025 catalogs how often models invent details despite grounding) and industry benchmark coverage this year has put ungrounded hallucination rates at 15-25% on factual queries. RAG cuts that down a lot, but not to zero, and smaller models inherit more of the risk.

Where local setups lose to cloud, concretely:

• Retrieval depth. AnythingLLM pulls roughly 4-6 chunks per question by default; cloud tools retrieve and re-rank across your whole library. Multi-document synthesis is where local setups stumble most.
• Silent truncation. Ollama’s 4,096-token default context on GPUs under 24GB trims retrieved text before the model sees it: the failure looks like a confident partial answer, not an error. The Open WebUI docs flag it as the most common RAG failure they see.
• Messy inputs. Scanned PDFs need OCR; Open WebUI’s docs recommend Apache Tika or Docling for poor extraction. Cloud services handle this for you.
• Tables and numbers. Small models misread tabular data and mix up figures across chunks more often than Gemini-class cloud models. That follows from the documented retrieval limits, not from a published benchmark.

Against all that: Gemini Notebook’s citation-to-passage links make cloud answers unusually auditable, and AnythingLLM’s workspaces keep local answers traceable to their source chunks. Treat every answer as a draft that cites its evidence.

What to do next

Non-sensitive documents: open Gemini Notebook, add sources, judge the quality for your material: five minutes, zero cost. Confidential documents: install AnythingLLM Desktop, embed one real document set, and ask it five questions you already know the answers to. That test tells you more than any benchmark. Move to Ollama + Open WebUI only when you outgrow single-player: multiple users, custom chunking, or a shared server. And flip off training settings on any personal cloud account before it sees a single file.

FAQ

What is a private AI chatbot for documents?

It’s a chatbot that answers questions using only documents you supply (PDFs, DOCX files, notes) instead of its general training knowledge. Most use RAG: your files are split into chunks, converted to embeddings, and the most relevant chunks are fed to the model with each question. “Private” spans two levels: cloud tools with no-training policies, and fully local setups where files never leave your machine.

What is the best way to build one?

For one person who wants files kept on-device with minimal setup, AnythingLLM Desktop is the shortest verified path: one installer with a built-in LLM provider, embedder, and local vector database. If you need multi-user access or deeper retrieval controls, run Ollama with Open WebUI. If your documents are not sensitive, Gemini Notebook gives the best answer quality with passage-level citations in about five minutes.

How does RAG work?

Retrieval-augmented generation has four steps. Your documents are split into chunks (often 500-1,500 tokens). An embedding model converts each chunk into a numeric vector capturing its meaning. When you ask a question, the question is embedded too, and the system retrieves the most similar chunks. Those chunks are inserted into the model’s prompt, so the answer is grounded in your actual text rather than the model’s memory.

How much does it cost?

All the core tools are free. Gemini Notebook’s free tier gives 100 notebooks with 50 sources each; paid Google AI tiers raise limits. ChatGPT Projects work on free accounts with 5 files per project, 25 on Plus ($20/month), and 40 on Business. AnythingLLM, Ollama, and Open WebUI are free and open source; your only cost is electricity and hardware you likely own.

What are the main alternatives?

Beyond the paths covered here: LM Studio is a free-for-work desktop app with an OpenAI-compatible local server, Jan is an active offline-first open-source alternative, and GPT4All (once the default recommendation) has been dormant since February 2025. Cloud-side, Claude Projects and Gemini’s main app also accept files, but they lack Notebook-style source-grounding controls.

Which option is best for beginners?

If your documents are not confidential, start with Gemini Notebook: it is point-and-click and answers come with citations to the exact source passage. If confidentiality matters, AnythingLLM Desktop is the beginner pick: the installer handles the model download, and you drag documents into a workspace. Expect about 15 minutes from download to first answer, no command line required.

What are the main limitations?

Local setups retrieve only a handful of chunks per question (roughly 4-6 in AnythingLLM by default), struggle with scanned PDFs unless OCR runs first, and small models misread tables and cross-references more often than frontier cloud models. Cloud paths cap files per project (5 on free ChatGPT, 40 on Business) and process your documents on external servers. Ollama’s default context length can silently truncate retrieved chunks.

Can I use it for commercial or confidential work?

The local tools are cleared for it: AnythingLLM is MIT-licensed, Open WebUI is free below 50 users with its branding intact, and LM Studio has been free for work use since July 2025. Check each downloaded model’s own license (Gemma and Llama models carry community terms). For regulated data such as client health or financial records, fully local is the only path that keeps files off third-party servers; consult your compliance officer for formal obligations.

How can users get better results?

Keep source PDFs text-based rather than scanned. For local models with 8K context, Open WebUI’s docs recommend chunk sizes near 1,000 tokens with 100 overlap and retrieving 3-5 chunks. Raise the model’s context length to 16,384+ so retrieved text is not trimmed. Prefer a purpose-built embedding model such as nomic-embed-text or mxbai-embed-large. Ask specific questions, and audit every answer against the cited passage.

What should users choose or do next?

This weekend: if files are non-sensitive, spend five minutes in Gemini Notebook and judge quality first. If they are sensitive, install AnythingLLM Desktop, pick a small model, and embed one real document set. Only move to the Ollama + Open WebUI stack if you outgrow it: multiple users, custom retrieval settings, or a Docker server. Budget a 16GB-RAM machine for comfortable 8B-class models.

Sources

• Google blog: NotebookLM is now Gemini Notebook
• Google Workspace Updates: NotebookLM now Gemini Notebook
• Google Help: Gemini Notebook plans and limits
• Google Help: Gemini Notebook source limits FAQ
• Google Workspace Updates: automatic Drive syncing in NotebookLM
• OpenAI Help: Projects in ChatGPT
• OpenAI Help: Data controls in ChatGPT
• OpenAI pricing
• AnythingLLM docs: Desktop installation overview
• AnythingLLM docs: Privacy and data handling
• AnythingLLM site
• AnythingLLM GitHub issue #2007: document retrieval limits per query
• Open WebUI docs: Quick start
• Open WebUI docs: Troubleshooting document retrieval (RAG)
• Open WebUI docs: License
• Open WebUI blog: v0.13.0 release
• OllamaHerd: Ollama release notes tracker
• Ollama privacy page
• Ollama library: gemma4
• Ollama library: nomic-embed-text
• LM Studio blog: free for work
• GPT4All GitHub issue #3558: development stopped
• Jan blog: how to run AI models locally
• arXiv: Comprehensive LLM hallucination survey (October 2025)
• NVIDIA: What is retrieval-augmented generation?
• IBM: What is RAG?
• Micro Center: best local LLMs by memory size