How to Build a Custom LLM Chatbot for Your Business (No-Hallucination Playbook)

A retrieval-augmented chatbot that answers correctly 87% of the time is not an 87% solution — it is a liability generator. The failure mode is asymmetric: a single confident, fabricated refund policy or SLA commitment surfaced to a customer or an auditor costs more than every correct answer saved. The engineering question is therefore not "how do we raise accuracy?" but "how do we make wrong answers structurally impossible to emit?"

That constraint changes the architecture. Prompt instructions like "only use the provided context" are advisory, not enforcing. To build a custom LLM chatbot for business use where correctness is contractual, you need an abstention-first pipeline: retrieval that knows when it has failed, generation that is forced into a typed output contract, and a deterministic verification pass that runs after the model and has veto power over the response.

The Hallucination Is Not a Prompt Problem — It Is an Architecture Problem

Three failure classes dominate production incidents, and each requires a different layer of defense:

• Retrieval miss. The correct document exists but ranks below top-k because the user's vocabulary diverges from the corpus. The model then answers from parametric memory. No prompt can fix this.
• Context contamination. Retrieved chunks contain near-miss information (an older policy version, a different region's pricing). The model blends them into a plausible hybrid that matches no source.
• Over-extrapolation. The context answers 80% of the question; the model silently fills the remaining 20% to produce a complete-sounding reply.

Only the third is partially addressable at the prompt layer. The first two are retrieval and verification problems. Senior teams treat the LLM as an untrusted renderer — a component that formats verified facts into language, never a component that decides what is true.

Architectural Note: Design for abstention as a first-class success path, not an error. A chatbot that says "I don't have a verified answer — here is the source document" and routes to a human preserves trust. A chatbot that guesses destroys it permanently.

Reference Architecture for a Grounded Custom LLM Chatbot

The pipeline has seven stages, each with a hard latency budget and a defined failure contract.

Production Caveat: Do not collapse the reranker to save 150 ms. In our benchmarks, removing cross-encoder reranking increases top-1 retrieval misses by roughly 30–40% on domain corpora with high lexical overlap (legal, medical, financial policy). That latency is the cheapest hallucination insurance you will ever buy.

Ingestion and Chunking Strategy

Chunking is where most RAG systems silently lose accuracy. Fixed 512-token windows split tables from headers and orphan numbered clauses from their parent section. Use structure-aware chunking: parse the source into a document tree (H1 → H2 → clause), emit leaf chunks of 200–400 tokens, and prepend a breadcrumb header to every chunk's embedding text.

Always scope retrieval by tenant_id and an ACL predicate. A cross-tenant leak in a multi-tenant deployment is a data breach, not a bug.

Implementing the Grounded Answer Contract in TypeScript

The generator must return a typed, schema-validated object — never free text. Every factual claim carries a chunk_id and a verbatim quote. The verifier then proves the quote exists in the cited chunk. If proof fails, the response is discarded.

// src/answer/grounded-answer.ts
import { z } from "zod";
import OpenAI from "openai";

const GroundedAnswer = z.object({
  status: z.enum(["answered", "insufficient_evidence"]),
  answer_markdown: z.string().min(1).max(4000),
  claims: z.array(
    z.object({
      chunk_id: z.string(),
      quote: z.string().min(20), // verbatim, must appear in the chunk
      text: z.string().min(1),
    })
  ).max(12),
});
export type GroundedAnswer = z.infer<typeof GroundedAnswer>;

export interface RetrievedChunk