A dispatch assistant that cites the company's own documents

Challenge

Dispatchers handled exceptions all day: a damaged load, a missed appointment window, a carrier disputing a charge. The answer to each one lived in 1,400 pages of carrier contracts, rate sheets and standard operating procedures. Finding it took 40 minutes a shift on average, and new dispatchers took four months to become fast. A previous vendor had built a chatbot demo that impressed the board and was never deployed because legal could not tell what it would say or where the data went.

Approach

The two-week review scored four use cases and picked one: exceptions guidance for dispatchers. We collected 340 real dispatcher questions with the answers a senior dispatcher gave, which became the evaluation suite before any code was written.

The system runs inside the carrier's AWS account. Documents are chunked by section, embedded and stored in Postgres with pgvector, with metadata for carrier, effective date and document type so retrieval can filter before it searches. Claude on Bedrock generates answers through a private endpoint. Presidio redacts driver and customer names before any model call. Every answer must cite the chunk it came from, and the interface shows the clause beside the answer.

Evaluation runs on every change: retrieval recall, answer faithfulness, relevance and correct refusal on the 40 questions that should not be answered. Langfuse tracks cost and latency per query with a hard ceiling. A red-team pass in week seven produced eleven prompt-injection findings, all closed before launch.

Result

Faithfulness reached 92 percent on the evaluation suite, against 71 percent for the vendor demo when we ran it through the same tests. Queries cost $0.004 each at 1.8 seconds p95. Dispatchers spend under five minutes a shift searching. The eval suite and cost dashboard are now the acceptance criteria for every AI change the company makes, and the second use case, carrier invoice audit, shipped eight weeks later on the same platform.