---
title: "Docs-as-code Knowledge Base and Context Graph: a case study across a 15-project portfolio"
description: "Docs for 15 projects had drifted from the code. A docs-as-code KB, a context graph and an MCP server for AI agents, measured in 126 blind-graded sessions."
canonical: "https://tmtuan.dev/en/articles/knowledge-base-docs-as-code-context-graph"
language: "en"
image: "https://tmtuan.dev/_img/articles/knowledge-base-docs-as-code-context-graph/knowledge-base-docs-as-code-context-graph-cover.5648dd27-og.jpg"
updated: "2026-09-25"
tokens: 5364
---

# Case study — Docs-as-code Knowledge Base + Context Graph for a 15-project portfolio

> This case study intentionally omits the client's name and every internal identifier. What remains is the system design, the measurement method and the results — the part that is safe to share publicly.

> Role: Technical Lead, designed and built the whole system myself (2026-08 → 2026-09, roughly 7 weeks alongside project management work). Personal repo: `tmt-projects-kb`. Stack: Markdown + YAML frontmatter, Python (tree-sitter, networkx, SQLite), MCP server, d3-force viewer.

---

## 1\. Context

I am the Technical Lead on several web projects for a Japanese enterprise client. Each project has 3–4 microservice-style repos: web frontend, contact API, Terraform infra, CMS deployment. The tech stack and deployment environment differ per project **and per env**: Cloud Run, Google App Engine, Firebase Hosting/Functions, Cloud Functions gen1/gen2, with some still on CircleCI or AWS S3 + CloudFront.

Management documentation was scattered across Confluence, Google Sheets/Slides/Docs. After years of hand-offs through several generations of developers, most of it **had drifted from the running code**.

## 2\. Measured pain points (before the system)

| Pain | What it looked like in practice |
| --- | --- |
| Updating a feature | Every time, hunting and reassembling information from Confluence, Sheets and 3–4 repos, with no way to tell which source was still correct. |
| Simple client questions | "Which projects run on Cloud Run in prod?", "How many projects still use Nuxt?" meant opening each repo to check — tens of minutes each time. |
| AI coding agents | Every session needed context fed in by hand. The agent read dozens of files to reconstruct relationships between repos, burned tokens, and still missed cross-repo hops or confused components with the same name in different projects. |
| Technical decisions | Root causes and trade-offs behind past fixes were never recorded; months later the same question came back — "why did we do it this way?". |

## 3\. Goals and self-imposed constraints

**Goal**: every project-knowledge question (infrastructure, architecture, tech stack, deployment, env, convention, decisions) answerable **straight from the KB**, without spending time or tokens scanning source repos. People and agents share one source. **Each class of question has one cheapest layer that answers it**, and the agent has to pick the right layer.

**Self-imposed constraints**:

- **Local-first, docs-as-code**: Markdown + YAML frontmatter in git, no SaaS dependency. Reviewable as a diff, committed with the `kb:` prefix.
- **The running code is the referee**: imported documentation is only a hint until it has been checked against code and pinned to a commit hash.
- **No vector DB or embeddings** at a scale of tens of projects × tens of files; structured progressive disclosure instead of fuzzy search.
- **Never store a secret**: only the name, the location (Secret Manager @ project / name), a hint and a SHA-256 prefix for comparison.
- **Build cost = 0 LLM tokens**: every derived artifact (index, graph) is generated deterministically from existing sources, and never edited by hand.

## 4\. Solution — four layers, each for a class of question

### 4.1 Knowledge layer: docs-as-code KB by project × component × env

**Answers**: single facts about one project (GCP project, URL, runtime, deploy flow, secret location, convention) and the internal architecture of one component.

- One folder per project, each component mapping 1:1 to a code repo. Every doc carries controlled frontmatter: runtime, GCP project, access control, deploy flow, integrations, secrets (pointers only), tags, `source` and `verification`.
- **Auto-generated index** `index/projects.yaml` (project × component × env × attribute): a single-fact question is answered by one file read, with no MCP involved.
- **Controlled vocabulary** (`shared/vocabulary.yaml`): the validator fails the build if frontmatter uses an unknown value (for example `cloudrun` instead of `cloud-run`). This is the precondition for the index and the graph to be generated deterministically.
- **ADRs in MADR 4.0.0** for every change with a decision behind it: `status` (proposed / accepted / superseded…), and a 1–2 sentence `summary` indexed verbatim into `index/decisions.md` so an agent can answer without opening the body. The validator warns on ADRs left `proposed` for over 30 days.
- Clear scope rules: the KB holds architecture, tech stack, feature specs, env/infra, conventions and decisions. Effort, schedule, assignments and meeting notes **never** go in.

### 4.2 Import and verification layer

**Answers**: "is this fact still true, and true as of which commit?"

- A pipeline exports Confluence (by hub page, following links) and Google Drive into `_imports/raw/<slug>/`, keeping originals intact and recording the source of each file.
- A four-step process per new project: **excavate the code repo** (produce `verified` docs from code) → **import existing documentation** → **distil** into the right component (all `unverified`) → **a verification session** against the code, recording every discrepancy.
- Each doc carries `verification.status` and `verified_against: <commit hash>`. The `verification-report.md` dashboard is generated automatically: verified coverage per project, files stale by source date, docs verified more than 90 days ago.
- Verification surfaced real doc-vs-code drift: Confluence said "purge by URL" while the code purged everything by CP code (Akamai CCU v3); Confluence held a plaintext credential, which the KB refused to copy.

### 4.3 Graph layer: a two-layer context graph, partitioned per project

**Answers**: **relationship** questions — where does form X call into across repos, which files does rotating secret Y affect, which projects use package / middleware / host Z, "does X call Y".

- The **knowledge layer** is generated from validated frontmatter: Project, Component, EnvInstance, GcpProject, Runtime, Integration, Secret, ADR, SourceDoc.
- The **code layer** is generated from local repos with **tree-sitter** (10 extractors: TypeScript/JS, Next.js, Vue SFC, Express, Firebase Functions, Terraform HCL, Python, Contentful, env scan, HTTP calls): CodeFile, Symbol, Route, Middleware, EnvVar, HttpCall, TfResource.
- **Bridge edges** join the two layers: HAS\_CODE, RESOLVES\_TO (env var ↔ secret), CALLS (a frontend HTTP call resolved through an env var URL to another API's route), PROVISIONS (Terraform ↔ GCP project).
- **Partitioned per project from the start**: `_global.json` holds only the knowledge layer plus a **catalog layer** (package + version, middleware, outbound hosts, Terraform resource types, env var names per component) so cross-project questions can be answered **without ever exposing another project's code internals**. An edge joining two different projects is a build error.
- **Token budget is first-class**: each tool response stays under ~2,000 tokens; over budget, edges are grouped by type, files collapsed by directory, and Route / Symbol / EnvVar in the trace chain are kept before distant nodes are cut.
- A single `kb_refresh.py` command rebuilds exactly the stale part (components with new commits, the index when docs change): 1–2 seconds when nothing is stale, tens of seconds when forcing a full rebuild of all 46 components. A git hook runs it after commit.

### 4.4 Access layer: a multi-client MCP server and a visual viewer

- **MCP server `kb-graph`**, 8 tools, one per shape of question: `kb_trace` (form → route → controller → service across repos), `kb_impact` (what does changing or rotating X affect), `kb_neighbors` (one node's neighbours, for checking negatives), `kb_find` / `kb_compare` (cross-project over the catalog: "how many projects use Nuxt", "compare Express versions"), `kb_sql` (read-only SQL over the partition's SQLite for questions no tool covers), `kb_entity`, `kb_status`. The server reloads on mtime and reports `code_graph_stale` when the local repo HEAD differs from the commit the graph was built from.
- Registered for **Claude Code, Gemini CLI, GitHub Copilot CLI and VS Code Copilot Chat** from the same binary. The agent's layer-selection rules (single fact → index; relationship → graph; never conclude "no edge means no relation") are written once in `AGENTS.md`; `CLAUDE.md`, `GEMINI.md` and `copilot-instructions.md` are just pointers.
- A **force-directed viewer** (canvas + d3-force, reading partition JSON directly): pick a partition, find a node, highlight 1- and 2-hop neighbours, an evidence panel with edges grouped by type, a legend to show and hide types, and state kept in the URL so a view can be shared. Used to present architecture to clients and to debug extractors.

### 4.5 Layer selection: which question goes to which layer

| Class of question | Answering layer | Why (per the benchmark in §5) |
| --- | --- | --- |
| A single fact about one project: GCP project, URL, runtime, deploy, secret location | `index/projects.yaml`, no MCP | One file read, cheapest, verified against a commit |
| Internal architecture of one component: route → handler → middleware | KB docs (`architecture.md`), opening code when a specific line is needed | KB docs suffice (completeness 0.72); the graph currently loses here (0.64) because it has no route → handler function name edge and no per-route middleware |
| Cross-repo trace, impact of rotating a secret, isolation between projects | `kb_trace`, `kb_impact` | The graph recovers the completeness KB docs lose (0.44 → 0.72), gives `file:line`, and is cheaper than reading source |
| Cross-project: "how many projects use Nuxt / package X / middleware Y", version comparison | `kb_find`, `kb_compare` with `project="*"` | Cheapest and fastest of all three modes ($0.05 / 11 s), no grepping 35 repos, no exposure of other projects' code |
| Negatives: "does component X call Y" | `kb_neighbors` plus the phrase "per the static graph" | Without the graph, the agent confused same-named components across projects (2/3 and 1/3 runs); with it, 3/3 picked the right component |
| Decisions / ADRs | `index/decisions.md` | Summaries are indexed verbatim, so the body never has to be opened |
| Anything no tool covers yet | `kb_sql` | Read-only SQL over the partition's SQLite copy |

## 5\. Results

### Current scale (2026-09-25)

| Metric | Value |
| --- | --- |
| Projects / component repos | 15 / 51 (50 components have docs) |
| Distilled knowledge docs | 153 files, 135 verified against a commit hash (75% of the 180 files with a status) |
| Indexed ADRs | 27 project ADRs + 4 about the KB itself |
| Source documents imported | 184 files from Confluence / Drive |
| Code graph | 46 components, 10 tree-sitter extractors, 0 LLM tokens |
| Graph | ~25,000 nodes / ~45,500 edges across 15 partitions; `_global` 1,026 nodes / 2,679 edges |
| MCP | 8 tools, 4 clients, budget ≤ 2,000 tokens per response |
| Tests | 33 pytest (mini KB fixture + mini repo) |

### Benchmark v2 — measured with real agent sessions, blind-graded

**Method** (2026-09-24): 9 questions representing 5 use-case groups, each run in 3 modes × 3 cold runs (81 sessions), plus 3 long sessions of 5 consecutive questions × 3 modes (45 sessions). Model `claude-opus-5-5`, Claude Code 2.1.281, 35 repos pinned to a HEAD. Ground truth was frozen before the first run; completeness (0–1) was graded blind by a separate agent session that saw only the truth and the anonymised answers; hallucinations were counted. Total cost of measuring and grading: $40.74.

Three modes: **A Bare** (reading source repos only), **B KB docs** (KB, no MCP), **C KB + Graph** (MCP). B − A is what the KB docs are worth on their own; C − B is what the graph adds.

| Question group | Metric | A Bare | B KB docs | C KB + Graph | How to read it |
| --- | --- | --- | --- | --- | --- |
| Inside one component (3 questions) | cost / time / compl | $0.19 / 48 s / 0.93 | $0.14 / 25 s / 0.72 | $0.13 / 25 s / 0.64 | KB docs suffice; the graph does not help and in fact loses |
| Trace / impact / isolation (3 questions) | cost / time / compl | $0.19 / 51 s / 0.72 | $0.12 / 23 s / 0.44 | $0.15 / 28 s / 0.72 | The graph recovers the completeness of reading source, cheaper and faster |
| Honest negatives (1 question) | compl | 0.11 | 0.11 | 0.33 | Without the graph: same-named components confused in 2/3 and 1/3 runs |
| Cross-project (2 questions) | cost / time / payload | $0.20 / 61 s / 5,604 tok | $0.14 / 23 s / 5,698 tok | $0.11 / 19 s / 3,372 tok | The catalog replaces grepping 35 repos, payload −2,327 tok |
| **All 9 questions (median)** | cost / time / turns / compl / hallu | $0.17 / 49 s / 10 / 0.70 / 2/27 | $0.13 / 24 s / 4 / 0.55 / 3/27 | $0.14 / 25 s / 5 / 0.65 / 2/27 |  |
| **Long 5-question session** | session cost / per question from Q2 / compl / hallu | $3.65 / $0.84 / 0.92 / 3 | $2.73 / $0.64 / 0.72 / 2 | $2.06 / $0.48 / 0.79 / 1 | The graph is 44% cheaper than reading source, and it compounds with question count |

One concrete cross-repo trace question (contact form → route → controller → service across 2 repos): KB docs scored 0.25 in completeness because they had the architecture but no `file:line`; the graph scored 1.00 thanks to the `CALLS` edge from the frontend call site to the API's route.

```
ContactForm (contact-form.tsx:105)
  --CALLS_HTTP--> POST {NEXT_PUBLIC_CONTACT_API_URL}jp/contact
  --CALLS (envvar-resolved, dev+prod)--> route POST /api/jp/contact
  --ROUTES_TO--> contactController.sendContactEmailJp
  --IMPORTS--> zendeskService.ts
caveats: static trace — runtime middleware/DI and dynamic dispatch do not appear here
```

**Honest conclusions from the numbers**:

- **The KB docs buy all of the savings**: time −50%, turns 10 → 4, cost −25% compared with reading source — paid for with −0.15 completeness, because the agent stops at the documentation and never opens the code.
- **The graph does not change total cost or time** but recovers +0.10 completeness, concentrated in exactly the group it was designed for: trace / impact / isolation (+0.28, matching source reading) and honest negatives (+0.22). On cross-project questions the graph is both cheapest and fastest without losing quality.
- **The graph genuinely loses on internal component architecture**: route → handler edges stop at the file or module, with no function name and no per-route validator middleware; the agent trusts the graph and therefore does not open the code. This is recorded backlog, not a selling point.
- **Long sessions are clearly cheaper with the graph**: 5 consecutive questions at $2.06 versus $3.65 reading source, 1 hallucination versus 3 — and asking about one project right after another did not bleed facts from one into the other.
- Sources of error worth stating: n = 3 runs per cell, with a wide min–max; one entry in the ground truth for the summary question disagreed with the verified KB, so most of that question's hallucination flags were truth errors rather than the model inventing things; and the results hold only for the pairing of Claude Code 2.1.281 with `claude-opus-5-5`.

### Operational impact

- Every agent session, opened in any repo, gets the same context through a `CLAUDE.local.md` pointer and MCP; nothing is fed in by hand any more.
- Client questions like "which projects run Cloud Run in prod" are answered from the index in one read; "how many projects use Nuxt / still use Akamai" is answered by one tool call over the catalog, without opening a repo.
- 27 ADRs record the root causes and trade-offs of security audits, CVE patches and CI migrations — including findings that reversed the original design (for example, `npm audit = 0` does not mean clean because it misses advisories that exist only on GitHub; `min-release-age` does not apply to `npm ci`).
- The verification report names exactly which project's documentation is still stale, so verification can be prioritised instead of suspecting everything.

## 6\. Design decisions worth stating, and why

| Decision | Rejected alternative | Reason |
| --- | --- | --- |
| Frontmatter with a controlled vocabulary + auto-generated index | Full-text search / vector DB | Small scale, and most questions are structured facts; deterministic, no embeddings needed, reviewable as a diff. |
| Split layers by class of question, agent must choose the layer | A single access point (MCP only, or docs only) | The benchmark shows each layer wins on a different group; collapsing to one either loses completeness or costs turns. |
| Derive-only graph, 0 LLM tokens | Using an LLM to extract relations from code | Build cost is zero and reproducible; when it is wrong you fix the extractor, not the graph by hand. |
| Per-project partitions + a separate catalog layer | One global graph | Data isolation between clients, while cross-project questions are still answerable through the catalog without exposing code. |
| A hard token budget on every tool response | Returning the full subgraph | The agent gets just enough, without blowing up its context; collapsing happens in priority order rather than by arbitrary truncation. |
| Negative conclusions must say "per the static graph" | Treating a missing edge as a missing relation | Static analysis does not see dynamic dispatch or concatenated URLs; the build report lists unmatched and dynamic calls to check. |
| Agent rules written once in `AGENTS.md` | One rulebook per agent | Multiple clients (Claude / Gemini / Copilot) share one set of rules, so they cannot drift apart. |
| Dropped the `commits` field in ADRs, kept `prs` | Recording commit hashes | Hashes on a feature branch change after squash or rebase, producing "fill in after" notes that hang around for months. |

## 7\. Lessons

- **Distilling and verifying buys the savings; the graph buys the correctness.** KB docs halve time and turns for every question; the graph is only worth it for relationship and cross-project questions. Do not build a graph before the docs are clean, and do not force the graph to answer what the docs already cover.
- **Measure with ablation and blind grading before claiming anything.** The first measurement (4 questions, not blind-graded) gave a rosier conclusion than reality; v2 with frozen ground truth exposed where the graph genuinely loses, on internal component architecture. Unfavourable results went into the benchmark verbatim and became backlog.
- **Total session usage reflects the harness's turn count**, not how much information the tools returned. The right quantities to compare are cost, turns, retrieved payload and blind-graded completeness.
- **The source of hallucination was not "the model made it up" but same names across projects and a wrong truth**: without the graph, the agent answered about a same-named component in a different project; and one batch of hallucination flags turned out to be ground truth pinned to the code while the rate limit actually lived in Cloud Armor.
- **A controlled vocabulary** was the cheapest investment with the largest return: it is what makes the index, the graph, the validator and the dashboard all generate deterministically.
- **Staleness has to be visible**: `code_graph_stale`, `kb_index_stale`, ADRs `proposed` for over 30 days, docs verified more than 90 days ago. A knowledge system that cannot report its own age will drift exactly like Confluence did.

## 8\. Tech stack

Markdown + YAML frontmatter · Python 3.13 · tree-sitter (language pack: TypeScript/TSX/JS, Python, HCL, Vue, YAML, Dockerfile) · networkx · SQLite · MCP Python SDK (stdio) · pytest · HTML canvas + d3-force · GitHub / GHES · GCP (Cloud Run, GAE, Firebase, Cloud Functions, Secret Manager) · Terraform · Contentful · Akamai.

## Related projects

- [Dr.Well-B](https://tmtuan.dev/en/projects/dr-well-b)
- [TaxPal](https://tmtuan.dev/en/projects/taxpal)
- [Waroku Hospital Karte](https://tmtuan.dev/en/projects/waroku-hospital-karte)
- [Waroku Home-Visit Nursing](https://tmtuan.dev/en/projects/waroku-home-visit-nursing)

## Structured data (JSON-LD)

```json
{"@context":"https://schema.org","@graph":[{"@type":"Article","@id":"https://tmtuan.dev/en/articles/knowledge-base-docs-as-code-context-graph#article","headline":"Docs-as-code Knowledge Base + Context Graph for a 15-project portfolio","description":"Docs for 15 projects had drifted from the code. A docs-as-code KB, a context graph and an MCP server for AI agents, measured in 126 blind-graded sessions.","datePublished":"2026-09-25","dateModified":"2026-09-25","inLanguage":"en","url":"https://tmtuan.dev/en/articles/knowledge-base-docs-as-code-context-graph","mainEntityOfPage":"https://tmtuan.dev/en/articles/knowledge-base-docs-as-code-context-graph","image":"https://tmtuan.dev/_img/articles/knowledge-base-docs-as-code-context-graph/knowledge-base-docs-as-code-context-graph-cover.5648dd27-og.jpg","author":{"@type":"Person","@id":"https://tmtuan.dev/#person","name":"Tran Manh Tuan","url":"https://tmtuan.dev/en"}},{"@type":"BreadcrumbList","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https://tmtuan.dev/en"},{"@type":"ListItem","position":2,"name":"Articles","item":"https://tmtuan.dev/en/articles"},{"@type":"ListItem","position":3,"name":"Docs-as-code Knowledge Base + Context Graph for a 15-project portfolio","item":"https://tmtuan.dev/en/articles/knowledge-base-docs-as-code-context-graph"}]}]}
```