Most enterprise AI pilots follow the same arc. The demo is extraordinary. The pilot is good. Then someone from legal asks a question that spans four documents and a policy exception, the system answers confidently and wrongly, and the project quietly loses its sponsor. The failure is almost never the model. It is the retrieval.
Standard retrieval augmented generation embeds your documents into vectors, embeds the question the same way, and returns the passages that sit closest in that space. The approach was formalised in the original retrieval augmented generation paper and it works remarkably well for a certain shape of question: ones whose answer lives in a single passage that talks about the same things the question talks about.
Enterprise questions are usually not that shape.
Where vector-only retrieval runs out
Three failure modes show up again and again, and they are all the same failure underneath.
Multi-hop questions. "Which of our contracts inherit the 2024 liability cap?" The cap is defined once, in an amendment. Four contracts inherit it through references that never restate the number. No passage contains both the question's terms and the answer, so no amount of semantic similarity will find it.
Aggregation. "How many open incidents touch the payments service?" This is a counting question over a set. Retrieval returns twenty passages that each mention one incident, the model counts the ones it can see, and reports a number that is confidently too small.
Disambiguation. Three entities in your corpus are called Atlas. Vector search has no concept of identity, so it returns passages about all three and the model blends them into a single plausible paragraph about a thing that does not exist.
What connects these is that the answer depends on relationships between pieces of content, and an embedding is a description of one piece of content in isolation. Chunking strategies, larger context windows and contextual retrieval all help at the margin, and none of them change the underlying fact that similarity is not connection.
What the graph adds
GraphRAG keeps the vector index and adds a second structure: a knowledge graph extracted from the same documents, where entities are nodes and the relationships between them are edges. The approach was popularised by Microsoft Research and set out in detail in their 2024 paper, and the core insight is simple. Retrieve candidates by similarity, then expand along relationships to gather the context those candidates depend on.
In the liability cap example, the vector step finds the amendment. The graph step follows the AMENDS and REFERENCES edges outward and returns the four contracts, the clause that creates the inheritance, and the two statements of work underneath. The model now has the whole subgraph rather than three passages that happened to use similar words.
Building the graph is the part teams underestimate. Entity extraction from documents is imperfect, and the failure is not random: it systematically merges entities with similar names and splits entities referred to inconsistently. Which means the quality ceiling of your GraphRAG system is set by your entity resolution, not your model. If that phrase is unfamiliar, our guide to entity resolution techniques covers it directly, because it is the same problem wearing a different hat.
A retrieval path that is actually buildable
The pipeline that works in production has five steps, and the discipline is in keeping each step narrow.
- Interpret the question. Extract the entities it mentions and the relationship it is asking about. This can be a small model call; it does not need the frontier one.
- Vector search for candidates. Top twenty, tuned for recall rather than precision. You are casting a wide net on purpose.
- Expand along the graph. One or two hops from the candidate nodes. Two is usually the limit before the subgraph stops being an answer and starts being the whole database.
- Re-rank by path relevance. A passage three hops away through a weak relationship is less useful than one a single strong edge away. Score the path, not just the node.
- Generate with citations attached. Every claim in the answer carries the node and edge that supports it.
Both the Neo4j GraphRAG package and the property graph tooling in LlamaIndex implement most of this, and LangChain covers the orchestration. You will still write the extraction and the re-ranking yourself, because both are domain specific and neither generalises the way the demos suggest.
Citations are the whole point
The reason to do any of this is not answer quality in the abstract. It is that a graph-grounded answer can be checked.
When retrieval is a similarity score, the only evidence for an answer is that a passage scored 0.87. Nobody can act on that. When retrieval is a traversal, the answer comes with a path: this clause, in this document, amended by this one, which these four contracts reference. A lawyer can follow it in ninety seconds and tell you whether it is right.
That property is what turns an interesting internal tool into something a regulated business will actually deploy. The NIST AI Risk Management Framework is explicit about traceability and documentation as risk controls, and auditors increasingly ask the same questions. A system that cannot show its reasoning path will not pass that review, however good its answers are.
It also changes how you debug. A wrong vector-only answer is close to un-diagnosable; you can look at the retrieved chunks and shrug. A wrong graph answer tells you which hop went wide, and you fix that edge type rather than re-embedding your entire corpus and hoping.
Extraction quality is the ceiling
The graph you retrieve over is only as good as the extraction that built it, and extraction is where the unglamorous engineering lives. Three decisions determine most of the outcome.
What counts as an entity. Resist the temptation to extract everything. A schema with sixty node types produces a graph nobody can query and an extractor that is wrong in sixty ways. Most useful enterprise graphs run on somewhere between six and fifteen types, chosen because a real question needs them.
How relationships get named. Free-form relationship extraction produces REFERENCES, REFERS_TO, MENTIONS and CITES as four separate edge types meaning the same thing, and every query then has to know all four. Constrain the extractor to a fixed vocabulary and normalise aggressively.
What happens to the text. Keep the source passage attached to the nodes it produced. You need it for citation, and you need it when the extraction turns out to be wrong and somebody has to see what the document actually said.
Budget real time for evaluating the extraction on its own, before any retrieval is involved. Sample fifty documents, have a person check the entities and edges the pipeline produced, and fix the systematic errors. Every hour spent here saves several at the retrieval end, where the same errors are much harder to see.
When not to use it
GraphRAG is the wrong answer for a good number of problems, and saying so early saves everyone money.
If your questions are answered by a single passage, and they mostly are for policy lookup, FAQ deflection and simple document search, plain vector retrieval is faster to build, cheaper to run and easier to maintain. Adding a graph to that is engineering for its own sake.
If your corpus has no meaningful relationships between documents, there is no graph to extract. A pile of independent product reviews is a pile of independent product reviews. The technique needs structure that genuinely exists in the domain.
And if your documents change hourly, weigh the re-extraction cost honestly. Incremental extraction is possible but it is real work, and a nightly rebuild is not always acceptable. The general survey in Gao et al. is a good map of the alternatives if this describes your situation.
Cost, latency, and what to do first
GraphRAG is more expensive than vector-only retrieval on ingestion and cheaper on inference, which is usually the right trade. Extraction is a one-off cost per document. At query time you are retrieving a precise subgraph instead of stuffing twenty long passages into a prompt, and smaller prompts are faster and cheaper.
The pragmatic sequencing matters. Do not build the whole ontology first. Pick one question your business actually asks, that currently takes a person a day to answer, and build only the entity types and relationships that question requires. Ship that, measure it against the human answer, then widen. Teams that model the domain exhaustively before retrieving anything spend six months and produce a schema rather than a system.
Evaluate with a question set, not with vibes. Twenty questions with known correct answers, half of them deliberately multi-hop, run on every change. It is unglamorous and it is the only thing that tells you whether last week's tuning helped.
The same architecture sits underneath Bridgr, which is where we ended up after building it repeatedly for clients. If you want to talk through whether your corpus is graph shaped, Our Neo4j consulting practice is the short version and a call is the faster one.