Insight · Knowledge readiness
RAG Readiness: A Scored Checklist for Business Knowledge
Score your sources, permissions, content quality, questions and operations, with evidence examples and a checklist that also works without JavaScript.
Before connecting documents to a model, ask three questions: can we find the right passage, may this person see it, and will it still be right next month? This checklist assumes an internal question-answering system over a bounded set of sources. Its scores are planning prompts, not a validated readiness test.
Ready means evidence for a bounded use.
RAG retrieves source passages to help a model answer. Useful retrieval needs more than readable files: the right passage must be available under the requester’s permissions, and the system needs a rule for outdated or conflicting guidance. A good answer to one question does not establish that these controls work for another user or after a source change.
Choose one corpus, user group and decision to assess. Identify the source owner and the person who can accept the resulting answers. Score what is evidenced today, then repeat for another scope if its sources or access rules differ. This avoids giving a large repository one average score that hides a restricted folder or an unowned policy.
Score the evidence, not the intention.
For each item, choose 0 for absent or unknown, 1 for partial evidence, or 2 for the evidence described. An unscored item is not evidence. Record the actual evidence location in your own worksheet; this page does not collect it.
Checklist scores stay in this page’s memory while it is open. This checklist does not store or send your selections. With JavaScript disabled, use the same tables and add your scores manually; each section is out of 8 and the whole checklist is out of 40.
Sources
| Item | 2 looks like… | Your score (0–2) |
|---|---|---|
| A register lists repositories, content types, scope and a stable reference for each source. | ___ / 2 | |
| Each included source has an owner who can authorise use and answer questions about it. | ___ / 2 | |
| Representative files, including scans and tables, yield readable passages with usable source references. | ___ / 2 | |
| A repeated document maps to a defined canonical version rather than competing copies. | ___ / 2 |
Section total:
First repair: Start with a source register for one bounded corpus.
Permissions
| Item | 2 looks like… | Your score (0–2) |
|---|---|---|
| The owner identifies where access rules live and supplies allowed/denied examples for the retrieval layer. | ___ / 2 | |
| Test users map to current groups, including nested groups and explicit exceptions where used. | ___ / 2 | |
| A removed user loses access to answers, snippets and cited content within an agreed, tested boundary. | ___ / 2 | |
| When identity or permissions cannot be checked, restricted retrieval stops and an owner receives the case. | ___ / 2 |
Section total:
First repair: Map access for one source and test a removed user before adding more.
Content quality
| Item | 2 looks like… | Your score (0–2) |
|---|---|---|
| Active and superseded documents are distinguishable, and tests exclude obsolete guidance from current answers. | ___ / 2 | |
| Time-sensitive policies carry effective dates and an applicable team, product or jurisdiction where relevant. | ___ / 2 | |
| Priority · conflicting guidance | A conflicting pair has an owner-approved precedence rule or a tested path that exposes the conflict and declines to choose. | ___ / 2 |
| A retrieved passage opens at a stable, authorised source location with its version or update reference. | ___ / 2 |
Section total:
First repair: Resolve one conflicting policy pair and define the active-version rule.
Questions
| Item | 2 looks like… | Your score (0–2) |
|---|---|---|
| A permitted sample represents the actual roles, wording and tasks the system is meant to support. | ___ / 2 | |
| Source owners identify supporting passages and an acceptable answer or escalation for each test case. | ___ / 2 | |
| The set includes absent answers, out-of-scope requests and questions that should be refused for a given user. | ___ / 2 | |
| An owner keeps a separate acceptance set and records the rubric, critical failures and who will sign off. | ___ / 2 |
Section total:
First repair: Collect real questions and expected evidence from the people doing the work.
Operations
| Item | 2 looks like… | Your score (0–2) |
|---|---|---|
| Named owners can update sources and a tested refresh process handles changes and failed imports. | ___ / 2 | |
| Users can flag an answer with a source reference and an owner can investigate, correct and rerun the case. | ___ / 2 | |
| A drill removes a document from retrieval, derived indexes and relevant caches, with retained backups governed explicitly. | ___ / 2 | |
| An operator can pause retrieval, inspect failures and restore the accepted configuration using an accessible runbook. | ___ / 2 |
Section total:
First repair: Name an operator and rehearse a source change from ingestion through answer.
Read the weak sections before the total.
Each section has four items and a maximum of 8. As an illustrative planning rule, 0–3 means fix foundations before a pilot; 4–6 means close the evidence gaps; 7–8 means evidence is recorded and should be validated in a scoped test. Incomplete sections remain incomplete. These bands are editorial prompts, not a measured prediction of project success.
The total is a way to track repair work, not a release threshold. Strong source inventory does not compensate for weak permissions. Before a restricted-data pilot, establish revocation and unavailable-permission behaviour. Before answering disputed policy questions, establish precedence or an explicit conflict/refusal path. A score of 2 means the stated evidence exists; acceptance tests still need to exercise it.
Give contradictory documents their own test. Choose two plausible sources that disagree about the same case, identify their effective dates and owners, and ask the question as an affected user. If the owner cannot establish which applies, require the system to expose the disagreement or decline to choose. Silently blending both answers is not an acceptable conflict policy.
Repair one boundary at a time.
Start with the blocker, then repeat the affected tests. For sources, make a small register before writing another connector. For permissions, prove access and removal on one source. For content quality, resolve one conflicting pair and mark the active version. For questions, collect actual requests with expected supporting passages. For operations, rehearse an update and deletion from source through index and cache.
Treat deletion as a data-flow exercise. Removing a source link may leave extracted text, embeddings, cached answers or retained logs. List those copies, their owners and their deletion or retention rules, including backups. Test the live retrieval path after deletion and document copies governed by a separate retention process.
Use the knowledge-system acceptance scorecard after the repairs. Keep the readiness evidence and the evaluation cases together so source changes can trigger the right regression checks.
Sometimes the next step is not RAG.
If a small set of policies contradicts itself and no owner can resolve it, fix the policies first. If the real task is calculating from structured records, assess a permission-aware query or ordinary report. If staff only need a clear directory of a few documents, improved navigation and search may address the problem with less operating work.
An off-the-shelf tool may be worth testing when it can reach the required sources, preserve their permissions and meet your citation, refusal and deployment requirements. Verify those behaviours with your own cases and contract review; no product’s capabilities are assumed here. Custom work needs a reason grounded in a gap the test exposes.
For a bounded knowledge system, bring the weakest section, source register and real question set to a RAG and knowledge systems discussion. Include the known conflicts rather than cleaning them out of the demonstration set.
What to include
Include the decision you are working through, the systems involved and the constraint you need to resolve.