An agency accumulates institutional memory fast: call transcripts, contracts, client email, scripts, SOPs. The question people actually ask is small and specific: what did we agree on turnaround, what does this contract say about revisions. The risk is equally specific: the answer engine must never surface content the asker is not permissioned to see.
Off-the-shelf answer bots do not clear that bar, so this one was built to run inside our own walls: a permission wall enforced against our own role map, retrieval and inference on our own hardware, and a marginal cost per answer of zero, so the team never has to meter its questions.
It is retrieval-augmented generation over 233 company documents: it takes a question, finds the exact source, and quotes it with a link. Retrieval is hybrid, pairing local dense embeddings with BM25 lexical search merged through reciprocal rank fusion, over type-aware chunks: transcripts split by speaker turn, contracts by clause. Every chunk carries a sensitivity tier, and role-based access control is enforced at retrieval time against a per-person clearance map, so content someone is not cleared to see never reaches the model. The corpus is fed from the company's canonical documents, keeping every answer on the same source of truth the rest of the operating system runs on, and when that corpus lacks grounded support for a question, it abstains rather than guesses.
Inference runs on-premise, on local models on the office Mac Studio, which puts the marginal cost of an answer at zero. The team reaches it through Slack, an MCP server, or a CLI, and every answer arrives quoting its source. Development was gated by a dated evaluation changelog and a hand-written suite of 43 cases spanning lookups, multi-client disambiguation, time-scoped questions, unanswerables, and forbidden-extraction attacks, with a hard regression gate: no change ships if false answers rise or a single leak attempt succeeds.
Eval discipline / dated changelog, hard regression gateEVAL SUITE ........... 43 HAND-WRITTEN CASES, 5 CATEGORIES LEAK ATTEMPTS ........ ALL BLOCKED, 0 GATED CHUNKS REACHED THE MODEL FALSE ANSWERS ........ 0, ABSTAINS WHEN THE CORPUS LACKS THE ANSWER PERMISSIONS .......... ROLE-BASED, ENFORCED AT RETRIEVAL TIME REGRESSION GATE ...... NO SHIP IF A SINGLE LEAK ATTEMPT SUCCEEDS MODELS ............... LOCAL, ON-PREMISE, $0 PER ANSWER STATUS ............... LIVE IN THE TEAM SLACK/ Shipped as a handoff
The launch itself was a collaboration. The code traveled through a private GitHub repo with the corpus and secrets moved out of band, because real client data never belongs in version control, alongside a turn-key deployment runbook covering environment, secrets, smoke tests, and the always-on service. Our COO picked up that handoff and, following the runbook we provided, had it running on local models on the office Mac Studio.