Announcing Open Knowledge Compiler: a compiler for the Open Knowledge Format

Raw engineering artifacts go in. A living, query-able, agent-readable knowledge base comes out.

TL;DR — Open Knowledge Compiler (open-knowledge-compiler, Apache-2.0) is a new open-source tool that continuously compiles a Git repository — its code, pull requests, and optionally Jira — into a knowledge base that is simultaneously a browsable wiki for humans and a structured, query-able graph for AI agents. The output is a conformant OKF (Open Knowledge Format) bundle, checked by the tool's own conformance validator, not just claimed. This post explains what OKF is, what Knowledge Compiler actually does today, and where the two genuinely different roles — producer and consumer — split apart.

The problem this is solving

If you've tried to wire an AI coding agent into a real codebase, you've hit the same wall from three different directions:

• The agent can read code, but it can't tell you why a piece of code exists — which business rule it enforces, which Jira ticket motivated the change, whether it's actually tested.
• Documentation about "what this system does" lives in five places — a wiki nobody updates, a Confluence page from eighteen months ago, tribal knowledge in someone's head, and scattered code comments — and none of them are current.
• Every new agent, every new tool, every new integration re-solves the same problem: how do I assemble enough context about this codebase to be useful?

This is exactly the gap the Open Knowledge Format (OKF) was designed to close, and exactly the gap Open Knowledge Compiler exists to fill automatically, rather than asking a human to maintain it by hand.

What OKF actually is

Open Knowledge Format is an open specification — not a product, not a vendor SDK — published by Google Cloud's Data Analytics and Data Cloud teams (Sam McVeety and Amir Hormati), building on an idea Andrej Karpathy sketched in an LLM-wiki gist. It formalizes a pattern a lot of people were already converging on independently — Obsidian vaults, AGENTS.md/CLAUDE.md files, "metadata as code" repos — into something interoperable.

The spec (currently v0.2) is deliberately small. An OKF bundle is a directory tree of Markdown files. Each file is a concept — a table, a metric, a component, a business rule, whatever your domain needs — with YAML frontmatter on top:

---
type: Component
title: Billing rules
description: Discount calculation and eligibility checks.
tags: [billing, revenue]
generated:
  by: process:some-producer/1.0
  at: 2026-08-05T14:31:43+00:00
---
# Billing rules

Applies the discount cap and eligibility checks before a claim is saved.

Only one field is universally required: type. Everything else — title, description, resource, tags, the provenance-oriented sources/generated/verified fields, the lifecycle status/stale_after fields — is optional or recommended, not mandated. Two filenames are reserved with exact structure: index.md (a directory listing, no general frontmatter) and log.md (a date-grouped, prose changelog, also no frontmatter). Everything else about the file's shape is designed to be tolerant: unknown types, broken links, missing optional fields — none of it is grounds for a consumer to reject a bundle.

That minimalism is the point. OKF's own design principles state it plainly: producers and consumers are independent. Anyone can write a producer (something that emits OKF bundles) or a consumer (something that reads them — a viewer, a search index, an agent) without coordinating with anyone else, as long as both sides honor the spec.

Google shipped a reference producer alongside the spec: an agent that walks Big Query datasets and drafts OKF concept documents for each table, plus a static HTML bundle visualizer. Both are useful demonstrations. Neither is aimed at software engineering repositories, and neither continuously re-syncs — they're one-shot generation tools.

That's the gap Open Knowledge Compiler fills.

What Open Knowledge Compiler is

Open Knowledge Compiler (package name open-knowledge-compiler, CLI command kc) is an independent, third-party producer for the OKF ecosystem — not affiliated with Google, not part of the OKF spec repository. It has one job: take a software repository's raw artifacts and continuously compile them into an OKF-conformant knowledge base that stays in sync as the repository changes.

The "compiler" framing is deliberate and precise, not marketing. A compiler is a piece of software with a defined input grammar, a deterministic pass over that input, and a reproducible output — not a one-shot LLM summarization job you re-run and hope produces something similar to last time. Open Knowledge Compiler treats knowledge extraction the same way a real compiler treats source code: parse what you can deterministically, treat inference (the LLM layer) as an optional enrichment pass on top, and never let inference decide identity.

Six stages, always in this order: 

Collect (Git, PRs, optionally Jira) → Extract (deterministic parsing, then an optional LLM pass) → Normalize (turn raw facts into stable, identified entities) → Diff (compute what changed) → Persist (one atomic transaction into Postgres) → Emit (render the OKF wiki bundle from whatever's now in the database).

The database is the durable source of truth. The OKF bundle is a render of it — disposable, regenerated wholesale on every compile, never hand-edited. If a page in the wiki is wrong, the fix is in the compiler or the source data, not the Markdown file.

What's actually implemented today

This is the section where I'd rather under-claim than over-claim. Here's a plain accounting of what exists and runs, versus what's designed-for but not yet built.

The deterministic core (works with zero LLM, zero API keys)

• Language analysis: Python, TypeScript and JavaScript, parsed via tree-sitter running in-process — no Node.js runtime required. This produces component, api, and test_coverage entities directly from AST structure: modules, symbols, HTTP routes and their handlers, which tests cover which code.
• Git and PR history: a Git collector plus a forge (GitHub) collector that associates pull requests with the entities their diffs touch — via the forge API, not commit parentage, because squash/rebase merges rewrite history and would otherwise silently break this.
• Stable identity across recompiles: this is the unglamorous, hardest-to-get-right piece. Every entity gets a slug — component/billing-rules, api/get-discount — and that slug has to survive a second, third, hundredth recompile without churning, or every downstream link, every agent query, every wiki cross-reference breaks on every run. Deterministic entities get natural keys (a component's slug comes from its file path and symbol structure). LLM-derived entities go through a match-then-mint cascade — try to match an existing entity by external key, then anchor overlap, then name similarity, and only mint a new slug if nothing matches. The LLM never assigns identity; identity is decided by deterministic rules that happen to consult LLM output.
• An append-only delta log: every compile records exactly what was added, changed, removed, or moved, in one atomic transaction. "What changed in this repo in the last week" is a database query, not a diff of Markdown files.

The optional semantic layer (opt-in, requires an LLM provider)

• LLM-derived entities: feature, business_rule, and risk — things no AST can see, because they're about intent, not structure. "This function exists because refunds must be capped at 20% of order value" is a business rule; tree-sitter cannot extract that, an LLM reading the code and its context can.
• Four provider integrations: Anthropic, OpenAI, Azure OpenAI, and Cloudflare Workers AI, behind one thin interface — so swapping providers is a config change, not a rewrite.
• A content-addressed cache: every LLM call is cached by a hash of (prompt template version, model, input content). Re-running a compile on unchanged files costs nothing — the cache serves the prior answer, byte-identical, which is also what keeps entity identity stable across runs. This is the mechanism that makes "recompile the whole repo cheaply" true rather than aspirational.
• A Jira collector: opt-in, fetches issues linked from a merged PR's title or body, mints jira_story entities and links them to the PRs that closed them.

Retrieval and the MCP server

• Hybrid search: PostgreSQL full-text search always available; optional embeddings (pgvector) fused with keyword results via reciprocal rank fusion when enabled. No embeddings configured? Search still works, keyword-only — retrieval degrades gracefully, it doesn't fail.
• A read-only MCP server (kc serve) — this is the part that turns the OKF bundle from something an agent reads into something an agent queries. It exposes:

The server never compiles anything — it's read-only, always. Compilation is a separate, CI-triggered step. This matters for the same reason a database's read replica and its write path are usually different concerns: you don't want an agent's queries able to mutate the knowledge base it's querying.

OKF conformance — checked, not assumed

This is the piece I want to be most transparent about, because it's also the most interesting story from actually building this.

The wiki emitter was originally built against an early draft of the OKF spec. While writing this announcement and researching the spec's actual GitHub repository directly, it became clear the authoritative spec had already moved to v0.2 — with real, breaking changes from that earlier draft (a timestamp field renamed to a generated: {by, at} object; provenance moved from a body-text # Citations section into a sources frontmatter array; stricter rules for the two reserved filenames).

That discovery surfaced actual bugs — not stylistic drift, real conformance failures: the wiki's index.md was emitting frontmatter fields the spec explicitly forbids there; a sources key was colliding in name (but not shape) with the spec's own provenance vocabulary. Those got fixed. Two things came out of fixing them:

1. kc validate-okf — a command that checks an emitted bundle against the actual conformance rules (parseable frontmatter, a non-empty type field, reserved-filename structure) rather than assuming the emitter got it right.
2. kc compile --emit-only — because a spec-version bump only ever needs to change how already-compiled knowledge is rendered, this re-runs just the Emit stage against the database, with no new compile, no re-parsing, no LLM calls. Rolling out a spec update across every repo you've compiled is cheap by construction, not something you re-pay full compile cost for.

Every compile run now records which OKF version its wiki targeted (okf_spec_version), the same way it already tracked its own fact and entity schema versions. This is recorded in ADR-013, one of thirteen architectural decision records that document why the system is built the way it is, not just what it does.

Honest nuance: conformance here means the required and recommended fields (type, title, description, resource, tags) plus the correctly-shaped generated provenance block. The newer optional OKF v0.2 families — verified trust tiers, status/stale_after lifecycle fields, the Attested Computation concept type for verifiable, executable knowledge — are not populated by Knowledge Compiler today. They're real, interesting extension points; they're future work, not shipped features.

Where the producer/consumer split actually matters

OKF's own design insists producers and consumers stay independent, and this is worth taking seriously rather than treating as a spec footnote — it's the reason this tool is useful to people who will never install it.

Anyone can point a generic OKF viewer, a search indexer, or a completely different agent framework at a bundle Open Knowledge Compiler produced, without installing Knowledge Compiler or touching its database. Conversely, Knowledge Compiler's compiled Postgres database is genuinely useful (via kc serve's MCP tools) even to someone who's never heard of OKF — the OKF bundle is one output of the system, not the whole point of it.

For humans, and for agents — genuinely different experiences of the same knowledge

For a human, the output is a wiki. Push the knowledge/wiki branch (the reference publisher target — the destination is pluggable, GitHub Pages and Confluence publishers are additive, not built yet) and GitHub renders it as ordinary Markdown pages, cross-linked, with an index.md you can click through. No special tooling to read it. That's the whole point of choosing "just Markdown, just files" as the substrate.

For an agent, there are two genuinely different ways in, and it's worth being precise about which one you're using:

1. Read the OKF bundle directly — any agent, any tool, zero KC-specific integration. It's Markdown with YAML frontmatter; every modern coding agent already knows how to read that.
2. Query the MCP server — this is richer than reading files, because it's not just retrieval, it's structured traversal of a graph: "what does changing this component affect," "resolve this import to the repo it actually lives in," "what covers this with tests." That's a live query over the compiled database, not a grep over rendered Markdown.

Neither path requires the other. An agent that only speaks "read files" still gets a synchronized, provenance-carrying knowledge base. An agent that speaks MCP gets the graph underneath the Markdown.

Real numbers, not hypotheticals

Knowledge Compiler has been dogfooded on real, private codebases — one primary service repository, one library it depends on, and on its own source tree. These are actual compiled states as of this writing, not illustrative examples:

That third row is worth pausing on: the tool compiles itself. kc verify — a zero-write shadow compile that checks the incrementally-maintained state against a full recompile — reports the compiled state of Open Knowledge Compiler's own repository as equivalent to a fresh full compile. kc validate-okf reports the resulting bundle as fully conformant against OKF v0.2. Self-hosting a compiler on its own source is a real, verifiable claim, not a metaphor.

The cross-repo case is a real scenario worth spelling out, because it's the one that comes up constantly in any codebase with more than one repository: Repo A depends on a library, Repo B. Compile both into the same database, add one line to Repo A's config mapping the import prefix to Repo B's registered name, and resolve_dependency will resolve some_lib.SomeClass to the actual compiled entity in Repo B — its payload, its relationships, its provenance — without Repo A's compile ever reading across the repository boundary at compile time. This is deliberately a query-time-only mechanism (ADR-011) — no shared schema changes, no cross-repository writes, nothing that risks one repo's compile depending on another's compile order.

What this is not (yet)

Being specific about the boundary matters more than the feature list, so here it is plainly:

• Not a plugin ecosystem — yet. The architecture is designed around pluggable stages (collectors, extractors, LLM providers, retrieval strategies, publishers), discovered via standard Python packaging entry points and activated only by explicit configuration — never by merely being installed, which is the property that keeps compilation reproducible. Today, the built-in plugins (the Python/TypeScript/JavaScript analyzers, the Git and Jira collectors, the four LLM providers) are wired directly rather than through that discovery mechanism; the entry-point activation path is designed but not yet the live path. Third-party plugins aren't a real, working story yet.
• Not proven at large scale. The architecture targets repositories up to roughly 5 million lines of code. The actual dogfood evidence above is on repositories in the hundreds to low thousands of compiled entities. That's real, verifiable evidence at the scale it exists at — it is not evidence of behavior at 5M LOC.
• No version-pinned or release-aware queries. "What changed in this dependency between the version I'm pinned to and the latest" is not answerable today. It's an explicitly deferred design item, not a bug.
• No database-less mode. Every compile requires PostgreSQL. There's an explicit, written-down design exploration for a lighter-weight mode that would trade away cross-run identity stability and the delta log for zero infrastructure — but it's a parked brainstorm document, not a roadmap commitment.
• Not affiliated with Google or the OKF spec authors. This is an independent, third-party implementation targeting a spec Google published and continues to evolve. If OKF changes again — and its own versioning policy says it will — Knowledge Compiler has to track that, the same ongoing-maintenance category of cost it already accepts for tree-sitter grammars and LLM provider APIs changing underneath it.

Trying it

git clone https://github.com/kushal-omnius/open-knowledge-compiler.git
cd open-knowledge-compiler
python -m venv venv && venv\Scripts\activate   # or source venv/bin/activate
pip install -e .[all]
docker compose up -d                            # Postgres + pgvector

# Point it at any local git repo
kc init --slug my-service --forge-ref github.com/you/my-service --dir /path/to/repo
kc compile --full --no-llm --dir /path/to/repo  # deterministic pass, no API keys needed
kc inspect --dir /path/to/repo                  # see what got compiled
kc validate-okf --dir /path/to/repo             # confirm the emitted bundle is OKF-conformant

No API keys, no LLM provider, no embeddings configuration required to see real output. Enabling the semantic layer ([llm] enabled = true in the generated kc.toml) and embeddings ([embeddings] enabled = true) are both later, opt-in steps, each independently toggleable.

The direction this is heading — clearly marked as direction, not done

Everything above this line is real and running. Everything in this section is where the project is aimed, and none of it should be read as already built:

• A real plugin ecosystem, once the built-in plugins have proven the interfaces stable enough to hand to third parties — the current wiring is deliberately the simple path until that trigger fires, not a permanent decision.
• Version-aware queries across compiled repositories — "what changed in this dependency between the two versions I care about" — as a thin layer over the existing delta log rather than a new storage mechanism.
• Additional OKF v0.2 field families — trust tiers, staleness lifecycle, and eventually mapping something like mutation-kill-rate data onto the Attested Computation concept type, so a business rule's compiled entity could carry a verifiable, re-runnable check rather than just a description.
• The long-term evaluation question: whether knowledge compiled this way measurably improves what AI agents produce when working in a codebase — code changes, generated tests, answers to engineering questions — compared to an agent with no compiled context at all. That's the actual north star. Everything described in this post is the infrastructure a fair test of that question requires; it is not, itself, evidence that the test has been run or won.

Closing

Open Knowledge Format gives the ecosystem a shared, minimal answer to "what should a knowledge file look like." Open Knowledge Compiler answers a narrower, harder question underneath that: for a living software repository, where does that file's content come from, and how does it stay true as the code keeps changing?

The answer here is: deterministically wherever possible, an LLM only where structure genuinely can't reach, identity that survives recompilation, provenance on every fact, and a conformance checker that verifies the output rather than trusting the emitter. That's a compiler, in the actual sense of the word — and it's now open source, Apache-2.0, and running on real code today.

Repository: github.com/kushal-omnius/open-knowledge-compiler License: Apache-2.0 Spec it targets: OKF v0.2

Contributions, issues, and honest skepticism are all welcome — see CONTRIBUTING.md in the repo.