Case study — Docs-as-code Knowledge Base + Context Graph for a 15-project portfolio

This case study intentionally omits the client's name and every internal identifier. What remains is the system design, the measurement method and the results — the part that is safe to share publicly.

Role: Technical Lead, designed and built the whole system myself (2026-08 → 2026-09, roughly 7 weeks alongside project management work). Personal repo: tmt-projects-kb. Stack: Markdown + YAML frontmatter, Python (tree-sitter, networkx, SQLite), MCP server, d3-force viewer.

1. Context

I am the Technical Lead on several web projects for a Japanese enterprise client. Each project has 3–4 microservice-style repos: web frontend, contact API, Terraform infra, CMS deployment. The tech stack and deployment environment differ per project and per env: Cloud Run, Google App Engine, Firebase Hosting/Functions, Cloud Functions gen1/gen2, with some still on CircleCI or AWS S3 + CloudFront.

Management documentation was scattered across Confluence, Google Sheets/Slides/Docs. After years of hand-offs through several generations of developers, most of it had drifted from the running code.

2. Measured pain points (before the system)

3. Goals and self-imposed constraints

Goal: every project-knowledge question (infrastructure, architecture, tech stack, deployment, env, convention, decisions) answerable straight from the KB, without spending time or tokens scanning source repos. People and agents share one source. Each class of question has one cheapest layer that answers it, and the agent has to pick the right layer.

Self-imposed constraints:

• Local-first, docs-as-code: Markdown + YAML frontmatter in git, no SaaS dependency. Reviewable as a diff, committed with the kb: prefix.
• The running code is the referee: imported documentation is only a hint until it has been checked against code and pinned to a commit hash.
• No vector DB or embeddings at a scale of tens of projects × tens of files; structured progressive disclosure instead of fuzzy search.
• Never store a secret: only the name, the location (Secret Manager @ project / name), a hint and a SHA-256 prefix for comparison.
• Build cost = 0 LLM tokens: every derived artifact (index, graph) is generated deterministically from existing sources, and never edited by hand.

4. Solution — four layers, each for a class of question

4.1 Knowledge layer: docs-as-code KB by project × component × env

Answers: single facts about one project (GCP project, URL, runtime, deploy flow, secret location, convention) and the internal architecture of one component.

• One folder per project, each component mapping 1:1 to a code repo. Every doc carries controlled frontmatter: runtime, GCP project, access control, deploy flow, integrations, secrets (pointers only), tags, source and verification.
• Auto-generated index index/projects.yaml (project × component × env × attribute): a single-fact question is answered by one file read, with no MCP involved.
• Controlled vocabulary (shared/vocabulary.yaml): the validator fails the build if frontmatter uses an unknown value (for example cloudrun instead of cloud-run). This is the precondition for the index and the graph to be generated deterministically.
• ADRs in MADR 4.0.0 for every change with a decision behind it: status (proposed / accepted / superseded…), and a 1–2 sentence summary indexed verbatim into index/decisions.md so an agent can answer without opening the body. The validator warns on ADRs left proposed for over 30 days.
• Clear scope rules: the KB holds architecture, tech stack, feature specs, env/infra, conventions and decisions. Effort, schedule, assignments and meeting notes never go in.

4.2 Import and verification layer

Answers: "is this fact still true, and true as of which commit?"

• A pipeline exports Confluence (by hub page, following links) and Google Drive into _imports/raw/<slug>/, keeping originals intact and recording the source of each file.
• A four-step process per new project: excavate the code repo (produce verified docs from code) → import existing documentation → distil into the right component (all unverified) → a verification session against the code, recording every discrepancy.
• Each doc carries verification.status and verified_against: <commit hash>. The verification-report.md dashboard is generated automatically: verified coverage per project, files stale by source date, docs verified more than 90 days ago.
• Verification surfaced real doc-vs-code drift: Confluence said "purge by URL" while the code purged everything by CP code (Akamai CCU v3); Confluence held a plaintext credential, which the KB refused to copy.

4.3 Graph layer: a two-layer context graph, partitioned per project

Answers: relationship questions — where does form X call into across repos, which files does rotating secret Y affect, which projects use package / middleware / host Z, "does X call Y".

• The knowledge layer is generated from validated frontmatter: Project, Component, EnvInstance, GcpProject, Runtime, Integration, Secret, ADR, SourceDoc.
• The code layer is generated from local repos with tree-sitter (10 extractors: TypeScript/JS, Next.js, Vue SFC, Express, Firebase Functions, Terraform HCL, Python, Contentful, env scan, HTTP calls): CodeFile, Symbol, Route, Middleware, EnvVar, HttpCall, TfResource.
• Bridge edges join the two layers: HAS_CODE, RESOLVES_TO (env var ↔ secret), CALLS (a frontend HTTP call resolved through an env var URL to another API's route), PROVISIONS (Terraform ↔ GCP project).
• Partitioned per project from the start: _global.json holds only the knowledge layer plus a catalog layer (package + version, middleware, outbound hosts, Terraform resource types, env var names per component) so cross-project questions can be answered without ever exposing another project's code internals. An edge joining two different projects is a build error.
• Token budget is first-class: each tool response stays under ~2,000 tokens; over budget, edges are grouped by type, files collapsed by directory, and Route / Symbol / EnvVar in the trace chain are kept before distant nodes are cut.
• A single kb_refresh.py command rebuilds exactly the stale part (components with new commits, the index when docs change): 1–2 seconds when nothing is stale, tens of seconds when forcing a full rebuild of all 46 components. A git hook runs it after commit.

4.4 Access layer: a multi-client MCP server and a visual viewer

• MCP server kb-graph, 8 tools, one per shape of question: kb_trace (form → route → controller → service across repos), kb_impact (what does changing or rotating X affect), kb_neighbors (one node's neighbours, for checking negatives), kb_find / kb_compare (cross-project over the catalog: "how many projects use Nuxt", "compare Express versions"), kb_sql (read-only SQL over the partition's SQLite for questions no tool covers), kb_entity, kb_status. The server reloads on mtime and reports code_graph_stale when the local repo HEAD differs from the commit the graph was built from.
• Registered for Claude Code, Gemini CLI, GitHub Copilot CLI and VS Code Copilot Chat from the same binary. The agent's layer-selection rules (single fact → index; relationship → graph; never conclude "no edge means no relation") are written once in AGENTS.md; CLAUDE.md, GEMINI.md and copilot-instructions.md are just pointers.
• A force-directed viewer (canvas + d3-force, reading partition JSON directly): pick a partition, find a node, highlight 1- and 2-hop neighbours, an evidence panel with edges grouped by type, a legend to show and hide types, and state kept in the URL so a view can be shared. Used to present architecture to clients and to debug extractors.

4.5 Layer selection: which question goes to which layer

5. Results

Current scale (2026-09-25)

Benchmark v2 — measured with real agent sessions, blind-graded

Method (2026-09-24): 9 questions representing 5 use-case groups, each run in 3 modes × 3 cold runs (81 sessions), plus 3 long sessions of 5 consecutive questions × 3 modes (45 sessions). Model claude-opus-5-5, Claude Code 2.1.281, 35 repos pinned to a HEAD. Ground truth was frozen before the first run; completeness (0–1) was graded blind by a separate agent session that saw only the truth and the anonymised answers; hallucinations were counted. Total cost of measuring and grading: $40.74.

Three modes: A Bare (reading source repos only), B KB docs (KB, no MCP), C KB + Graph (MCP). B − A is what the KB docs are worth on their own; C − B is what the graph adds.

One concrete cross-repo trace question (contact form → route → controller → service across 2 repos): KB docs scored 0.25 in completeness because they had the architecture but no file:line; the graph scored 1.00 thanks to the CALLS edge from the frontend call site to the API's route.

ContactForm (contact-form.tsx:105)
  --CALLS_HTTP--> POST {NEXT_PUBLIC_CONTACT_API_URL}jp/contact
  --CALLS (envvar-resolved, dev+prod)--> route POST /api/jp/contact
  --ROUTES_TO--> contactController.sendContactEmailJp
  --IMPORTS--> zendeskService.ts
caveats: static trace — runtime middleware/DI and dynamic dispatch do not appear here

Honest conclusions from the numbers:

• The KB docs buy all of the savings: time −50%, turns 10 → 4, cost −25% compared with reading source — paid for with −0.15 completeness, because the agent stops at the documentation and never opens the code.
• The graph does not change total cost or time but recovers +0.10 completeness, concentrated in exactly the group it was designed for: trace / impact / isolation (+0.28, matching source reading) and honest negatives (+0.22). On cross-project questions the graph is both cheapest and fastest without losing quality.
• The graph genuinely loses on internal component architecture: route → handler edges stop at the file or module, with no function name and no per-route validator middleware; the agent trusts the graph and therefore does not open the code. This is recorded backlog, not a selling point.
• Long sessions are clearly cheaper with the graph: 5 consecutive questions at $2.06 versus $3.65 reading source, 1 hallucination versus 3 — and asking about one project right after another did not bleed facts from one into the other.
• Sources of error worth stating: n = 3 runs per cell, with a wide min–max; one entry in the ground truth for the summary question disagreed with the verified KB, so most of that question's hallucination flags were truth errors rather than the model inventing things; and the results hold only for the pairing of Claude Code 2.1.281 with claude-opus-5-5.

Operational impact

• Every agent session, opened in any repo, gets the same context through a CLAUDE.local.md pointer and MCP; nothing is fed in by hand any more.
• Client questions like "which projects run Cloud Run in prod" are answered from the index in one read; "how many projects use Nuxt / still use Akamai" is answered by one tool call over the catalog, without opening a repo.
• 27 ADRs record the root causes and trade-offs of security audits, CVE patches and CI migrations — including findings that reversed the original design (for example, npm audit = 0 does not mean clean because it misses advisories that exist only on GitHub; min-release-age does not apply to npm ci).
• The verification report names exactly which project's documentation is still stale, so verification can be prioritised instead of suspecting everything.

6. Design decisions worth stating, and why

7. Lessons

• Distilling and verifying buys the savings; the graph buys the correctness. KB docs halve time and turns for every question; the graph is only worth it for relationship and cross-project questions. Do not build a graph before the docs are clean, and do not force the graph to answer what the docs already cover.
• Measure with ablation and blind grading before claiming anything. The first measurement (4 questions, not blind-graded) gave a rosier conclusion than reality; v2 with frozen ground truth exposed where the graph genuinely loses, on internal component architecture. Unfavourable results went into the benchmark verbatim and became backlog.
• Total session usage reflects the harness's turn count, not how much information the tools returned. The right quantities to compare are cost, turns, retrieved payload and blind-graded completeness.
• The source of hallucination was not "the model made it up" but same names across projects and a wrong truth: without the graph, the agent answered about a same-named component in a different project; and one batch of hallucination flags turned out to be ground truth pinned to the code while the rate limit actually lived in Cloud Armor.
• A controlled vocabulary was the cheapest investment with the largest return: it is what makes the index, the graph, the validator and the dashboard all generate deterministically.
• Staleness has to be visible: code_graph_stale, kb_index_stale, ADRs proposed for over 30 days, docs verified more than 90 days ago. A knowledge system that cannot report its own age will drift exactly like Confluence did.

8. Tech stack

Markdown + YAML frontmatter · Python 3.13 · tree-sitter (language pack: TypeScript/TSX/JS, Python, HCL, Vue, YAML, Dockerfile) · networkx · SQLite · MCP Python SDK (stdio) · pytest · HTML canvas + d3-force · GitHub / GHES · GCP (Cloud Run, GAE, Firebase, Cloud Functions, Secret Manager) · Terraform · Contentful · Akamai.