Company knowledge that cites its sources, and knows its limits.

Who it’s for

This engagement is for organisations with a document estate, source owners and access rules that matter. The job is to make policies, procedures, contracts or internal notes answerable without treating every document as available to every user.

You might be replacing a pilot that answered without evidence, extending search across disconnected sources or investigating whether an assistant respects document permissions. We scope the questions, the permitted sources and the cost of a wrong answer before choosing the retrieval design.

A public FAQ over a small, stable set of pages may fit a hosted product. Custom retrieval needs a reason: source coverage, permission behaviour, an evaluation requirement or an integration that the simpler option cannot meet.

A knowledge system.
Beyond a chat box.

• Source connectors and ingestion. Select an API or export path per source. Preserve document identity, owner, effective date and access metadata. Split content with its structure in mind so headings, tables and their context stay connected.
• Permission-aware retrieval. Map source access rules to the querying identity and apply an access filter before candidate retrieval. Keep that mapping outside the language model.
• Retrieval and answers. Combine keyword and semantic retrieval where the case set supports it, rerank candidates and generate answers with passage citations or an explicit refusal.
• An operating surface. Include source inventory, sync status and failure visibility, a feedback queue and an evaluation set. A source that fails to sync needs an owner and a visible state.

The engagement does not include rewriting the document estate, producing legal or medical advice products, or exposing private material through a public bot without a separate review. Content owners resolve outdated or contradictory source material. Training on documents is not a substitute for defining retrieval and access behaviour.

Assistant memory needs its own scope: what is stored, whose context it belongs to, how it expires and how a user can remove it. Permission to retrieve a document does not by itself grant permission to retain its contents in another user’s conversation.

When a simpler
alternative is better.

If most knowledge already lives in an office suite with well-maintained permissions, evaluate its existing assistant against your questions first. Check source coverage, citations, deletion handling and restricted-user cases. A custom system earns its place by addressing a specific unmet requirement.

For a small, stable corpus, test whether supplying the permitted documents directly in model context is sufficient. An index adds ingestion, sync and access-mapping work; do not add it without testing the need.

If the problem is tone, format or a repeated output pattern, begin with prompting and validation. Consider fine-tuning only against that defined behaviour. Retrieval is the part of the design that supplies permitted source material at answer time.

Prerequisites

• A source inventory with content owners, approximate sizes, formats and an API or export path to investigate. Named connectors are selected during scoping; a source name alone does not establish integration support.
• A documented identity and group model. Define whose access applies, how source rules map to that identity and what happens when membership changes.
• Read access through scoped accounts, with permission to inspect the access metadata required for the design.
• Real questions and known good answers from the people who ask them. Include questions that should be refused, ambiguous wording and answers that depend on an effective date.

Use the RAG readiness checklist to record evidence for sources, access, conflicting documents and operating ownership.

Permission first.
Evidence before prose.

Resolve identity, determine permitted content, retrieve candidates and rerank them. Assemble an answer from the supporting passages and check the citations. The refusal path is part of the answer contract, including when evidence is missing or too weak to support a useful answer.

Ingestion records source versions and deletion markers. Sync updates content and access metadata, and invalidates derived chunks and answer caches affected by a change. The design sets a freshness boundary and blocks affected retrieval when access metadata is too stale to trust.

The environment and any model-provider data flow are agreed during scoping. Private AI infrastructure covers a stack your team owns and operates. If an answer should become an action, AI agent development adds a separate authority and execution boundary.

Access can change.
The index must follow.

Revocations and permission drift

Define how each connector detects a revoked share, a group change or a deleted document. Use available source change events or polling, reconcile periodically and record the last successful permission sync. When the source allows a live access check, recheck selected passages before sending them to generation.

A copied permission model can become stale between updates. The acceptable freshness interval, unavailable-source behaviour and affected-cache invalidation are agreed per engagement and tested. If the required revocation behaviour cannot be established for a source, exclude that source or restrict the deployment until it can.

Conflicts, citations and refusal

Source owners identify which records are authoritative and how effective dates should be interpreted. A later upload date is not sufficient to establish a newer policy. When permitted records disagree and precedence is unclear, show the conflict with dates and owner references rather than silently choosing a position.

Evaluate whether each cited passage supports the associated claim, whether qualifiers survived summarisation and whether the source was accessible to that user. A plausible answer with a decorative citation is a failed case.

“Not found in sources you can access” is a valid result. Refusal wording should avoid revealing whether restricted content exists. Test titles, snippets, citations and cached answers as possible disclosure paths, as well as the body of the answer.

Illustrative example: two permitted procedures give different approval rules. One has an effective date; the other is an undated working copy. The answer identifies the disagreement and asks the owner to resolve precedence. It does not invent a combined rule.

Logs and document instructions

Agree whether logs retain full queries, redacted queries or references only. Record retrieved source versions and access decisions under an explicit retention and access policy. Treat logs, feedback and stored conversation history as sensitive stores with their own deletion requirements.

Retrieved text is evidence to inspect, not authority to change system instructions. Separate it from control instructions and enforce identity, retrieval scope and tool permissions outside the model. Include adversarial document text in evaluation; a prompt alone is not an access control.

Deliverables
and handover.

The handover includes pipeline source and infrastructure definitions in your repository, connector configuration, the permission-mapping document and the evaluation set with its harness. The source inventory records what was connected, what was excluded and the agreed freshness behaviour.

Runbooks cover adding a source, re-indexing, rotating credentials, investigating a wrong answer and processing a deletion request across derived stores. An admin walkthrough covers sync failures, access checks and the feedback queue.

What your team
operates.

Your content owners keep source material current and resolve flagged contradictions. The system operator monitors sync failures and freshness, reviews access mappings and runs the evaluation before a model, prompt, chunking or retrieval change. Feedback needs a named reviewer and a route back to the source owner.

Your team also owns access reviews and retention decisions for indexes, answer caches, conversations and logs. Changing the source estate or identity model requires another review of the permission boundary. Ongoing support is agreed per engagement.

Test the answer.
Test the refusal.

Before building, agree the question set, its authors, the permitted sources and the acceptance thresholds. Separate tuning examples from held-out cases. Review retrieval and answer quality separately so a good-looking answer does not hide a failed search.

• Retrieval: check whether the relevant permitted passages appear in the candidates for answerable questions.
• Citations: inspect whether passages support the claims, including dates, conditions and exceptions.
• Abstention: ask questions absent from the permitted corpus. Check that the system refuses rather than filling gaps with unsupported facts.
• Permissions: ask a restricted-document question as a user without access. The acceptance condition is no answer from that document and no disclosure of its existence through text, title, citation or cache.
• Freshness: edit, revoke and delete source content. Check the index, caches and displayed sources after the agreed sync boundary, including a failed sync.
• Conflicts and hostile content: test contradictory records and instructions embedded in a document. Inspect the answer and the access decisions.

The evaluation records results on the agreed sources and cases. It does not establish universal accuracy. Changes to the corpus, user groups or retrieval design need regression evaluation.

Use the knowledge-system section of the business AI acceptance scorecard to define retrieval, citation, refusal, access and freshness tests.

What to prepare

Bring the source list with owners, formats and approximate sizes; the identity provider and group model; representative questions with known answers; and questions that should be refused. If a pilot exists, include examples of wrong answers and permission concerns. Redact sensitive content from the first email.

Bring your scored RAG readiness checklist, including the evidence gaps, to the scoping discussion.

Start with
the sources.