Series: RAG 系統架構 (3/5)

← RAG's Five Stages: From Pipeline to Reasoning Retrieval, and the Naive RAG on My Own Site PageIndex Deep Dive: A RAG Architecture That Replaces Vector Search with LLM Reasoning →
Key Points 11 min read
  • InfiniFlow's 2024 year-in-review distills into 5 infra lessons: document parsing, chunk contextualization, three-lane hybrid search, tensor reranker, and GraphRAG.
  • Pure length-based chunking has flat-lined — real gains come from letting an LLM tag each chunk with global context (Contextual Retrieval / dsRAG).
  • 'Vector + sparse vector + BM25' three-lane hybrid is the recall ceiling; missing any lane leaves a clear gap.
  • Tensor Rerankers (the ColBERT family) push reranking into the database layer and can rerank a much larger candidate set — but Cloudflare's stack doesn't natively support it yet.
  • For engineer-news, the priority is Cross-Encoder Reranker + Contextual Retrieval first, then SQLite FTS5 for BM25; GraphRAG stays off the roadmap for now.
Table of Contents

The previous post zoomed out for a five-stage panorama of RAG’s two-year evolution. This one zooms in on the actual engineering decisions of building a working RAG. The material comes mostly from InfiniFlow (the RAGFlow team)‘s late-2024 year-in-review, which I distill into “five infra lessons” — each cross-checked against engineer-news’s current stack.

The previous post concluded the site sits at the Naive RAG edge. This post unpacks that sentence into a concrete checklist.

Lesson 1: Document Ingestion Quality — Document Intelligence

Whether you’re feeding pure text or a PDF full of tables, formulas and flowcharts to the LLM, there’s a common prerequisite — input quality caps output quality. Garbage In, Garbage Out becomes Quality In, Quality Out in RAG.

Most enterprise data is PDFs, PPTs, Word docs, and magazines with mixed text and images — not plain text. Early LLMOps stacks (LangChain + vector DB + plain-text chunker) only handle plain text, which crushes RAG’s ceiling.

This class of problems is historically called Document Intelligence, covering several subtasks:

  • Layout Analysis
  • Table Structure Recognition (TSR)
  • Formula recognition
  • Flowchart / pie chart recognition

Each used to have specialized models. RAG bundles them into a broad-sense OCR as the input layer. Two generations of approach:

Gen 1 (CNN + traditional vision): PaddleOCR, RAGFlow DeepDoc, MinerU, Docling. CPU-friendly, cheap, but poor generalization — each scenario needs its own model, sometimes called “carving flowers.”

Gen 2 (Encoder-Decoder Transformer): Meta’s Nougat, GOT-OCR 2.0, StructEqTable, M2Doc. A single generative model handles diverse document types with stronger generalization; needs GPU. Its architecture closely mirrors VLMs, and 2025 will likely see convergence into unified multimodal document parsing models.

Against my site: engineer-news only consumes Markdown (hand-written or AI-normalized), bypassing OCR entirely. This lesson is N/A — no gap. On the flip side, that limits what RAG can consume — if I ever want to ingest PDF papers, slides, or whiteboard photos, this layer needs to come back. Priority: very low.

Lesson 2: Chunking Moves from Length-Split to Context-Augmented

Naive Text Chunking means “split by character count” — exactly what this site does today. scripts/sync-to-d1.ts’s chunkText():

function chunkText(text: string, maxLength = 1000): string[] {
  const paragraphs = text.split(/\n\n+/);
  const chunks: string[] = [];
  let current = '';
  for (const p of paragraphs) {
    if ((current + p).length > maxLength) {
      if (current) chunks.push(current.trim());
      current = p;
    } else {
      current += (current ? '\n\n' : '') + p;
    }
  }
  if (current) chunks.push(current.trim());
  return chunks;
}

Merge paragraphs up to 1000 chars, done. This has a structural pain point — chunks lack the article’s full context.

Example: an article about “solving D1 batch timeout” has a paragraph “just add this setting in wrangler.jsonc”. Pulled out alone, you have no idea which D1, which timeout, or which setting is being discussed. A query “D1 batch timeout” struggles to hit that chunk.

2024 saw a wave of chunking improvements:

  • Late Chunking (Jina): encode the whole document with the embedding model first, then split boundaries before the final mean pooling. Every token before the boundary can “see” context, preserving semantics better. Requires the embedding model to use mean pooling (bge-m3 is CLS pooling), so it doesn’t just plug in.
  • dsRAG: an LLM writes an auto-context for each chunk, fixing the “no clues in the chunk itself” problem.
  • Contextual Retrieval (Anthropic): similar idea to dsRAG — an LLM writes a short chunk-specific context, concatenated with the original text before embedding. Effective, direct to implement, and has become a de-facto standard for context-augmented chunking since late 2024.
  • Meta-Chunking (RUC & Shanghai AILab): an LLM decides sentence boundaries by logical coherence.
  • Mix-of-Granularity (Shanghai AI + BUAA): multi-granularity chunking + dynamic traversal depth, minimizing context redundancy.

Bottom line: tweaking chunk size has flat-lined. The real value is tagging chunks with context — only LLMs can do this, and the cost is acceptable (one-time investment at sync).

Against my site: pure length-split is a clear gap. The most direct fix is Contextual Retrieval: at sync time, call llama-3.1-8b (or something smaller) once per chunk to generate a 50–100 char context blurb, concatenate with the chunk, then embed. Sync cost doubles, but it’s offline and one-shot.

Lesson 3: Hybrid Search — Vector + Sparse Vector + BM25

IBM Research’s 2024 BlendedRAG proved one thing: vector + sparse vector + BM25 three-lane hybrid is the recall ceiling, beating any single lane or two-lane setup.

Why do we need all three?

  • Vector: semantic recall. Great at “similar meaning, different wording.” Inherently bad at precise matches — “our company’s March 2024 financial plan” will happily recall content from other months.
  • Sparse vector (like SPLADE): a pretrained model outputs a fixed-dim sparse vector — treat it as standardized keyword expansion. Great on general queries, but misses domain-specific terms (model numbers, internal codes) that weren’t in pretraining vocabulary.
  • BM25: a 30-year-old algorithm, the most direct answer for exact keyword matching. Sparse vectors can’t replace it.

Three lanes, each specialized. Not one replacing another.

The hard engineering question isn’t “does it support BM25” but does it support proper BM25:

  • Phrase query — inverted indexes must store positions
  • Dynamic pruning — prevent OR queries from exploding
  • Chinese segmentation — including bigram tokenization, term weights, stop-word filtering

Elasticsearch is the gold standard here. RAGFlow picked ES as its sole backend from day one for these reasons. OpenAI’s June 2024 Rockset acquisition was in large part because Rockset is cloud-native and offers close-to-ES full-text retrieval. Many pure vector DBs (Milvus, Qdrant) now claim BM25 support, but few actually deliver “phrase query + dynamic pruning + CJK tokenization.”

Against my site: D1 is SQLite, and SQLite ships a very useful built-in full-text index — FTS5, supporting tokenizers, phrase queries, and rank functions. To add BM25:

  1. At sync, create a posts_fts virtual table indexing title + tldr + content
  2. In /api/search, add a lane: SELECT ... FROM posts_fts WHERE posts_fts MATCH ? ORDER BY rank
  3. Merge vector-lane and BM25-lane results with RRF (Reciprocal Rank Fusion)

More work than a Reranker (build FTS5, handle mixed Chinese-English tokenization), but this is the watershed between “has hybrid search” and “doesn’t.” Sparse vectors can wait — Cloudflare’s stack doesn’t ship a SPLADE service.

Lesson 4: Tensor Reranker — Late Interaction

Reranker evolved rapidly this year, in three generations:

Cross-Encoder (BGE-Reranker): concatenate query and doc into BERT, capture token interactions. High quality, moderate cost. The current mainstream.

LLM-based Reranker (gte-Qwen2-7B): a 7B LLM scores directly. Better quality but doubled inference cost.

Late Interaction / Tensor Reranker (the ColBERT family): store per-token embeddings at index time (one tensor per doc); at query time, sum pairwise similarities between query and doc tokens.

Tensor Reranker has an engineering advantage: it can live in the database layer. Query-time inference is just dot products, so it’s fast. That means the coarse-ranking result set doesn’t have to be tightly capped at 5–10 — it can expand to hundreds or thousands for reranking, salvaging poor coarse-ranking.

Vespa was the earliest DB to engineer tensor support; Infinity (RAGFlow’s own) added it mid-2024. Model side: ColBERT / ColBERT v2 / JaColBERT (Japanese) / jina-colbert-v2 (multilingual) are all shipping fast.

Against my site: Cloudflare Workers AI currently offers @cf/baai/bge-reranker-base (Cross-Encoder); Tensor Reranker has no native support. So the pragmatic path is:

  1. Short-term: add Cross-Encoder (BGE-Reranker) — already a huge upgrade
  2. Long-term: wait for Cloudflare to ship the ColBERT family, or self-host an embedding service

Tensor Reranker’s ROI at this site’s stage (150 articles) is low; priority: low.

Lesson 5: GraphRAG and the Semantic Gap

The previous post walked through GraphRAG’s spectrum in detail (Microsoft GraphRAG → LightRAG → LazyGraphRAG → HippoRAG → KAG); here I add three engineering observations.

RAPTOR is a pre-GraphRAG transition: cluster the text, have an LLM summarize each cluster, then feed both summaries and originals into the search index. It’s already tackling “cross-chunk macroscopic questions”, just without an explicit graph. RAGFlow mid-year adopted RAPTOR as a GraphRAG stand-in.

SiReRAG proposes a two-axis recall: text has two dimensions — similarity (semantic distance) and relatedness (entity/relation association). RAPTOR sits on the similarity axis, GraphRAG on the relatedness axis; SiReRAG merges them. The framing is clean, and most Graph-RAG variants can be placed on this coordinate system.

HybridRAG’s schema insight: a fully featured database doesn’t actually need a graph DB to implement GraphRAG. Edges, entities, community summaries — they’re all text, and one table with full-text + vector indexes can carry them all with a type column distinguishing kind. This is exactly why RAGFlow stuck with Elasticsearch / Infinity instead of adding Neo4j. Useful reminder for this site: if I ever want GraphRAG on D1, one table is enough.

Against my site: not yet. GraphRAG’s fit is “strong cross-document association + global understanding” — 150 personal tech articles don’t satisfy that. Once the corpus reaches thousands, or readers start asking “what does this site think about topic X overall”, the investment starts making sense.

Aside: Agentic + Memory in one line

Mem0 got a huge star bump just by defining a Memory API — evidence that Memory as a primitive is genuinely in demand. But the Memory infra itself is mature (real-time filter + search) — the scarce piece is “combining Memory with Reasoning”. That’s the hot space for 2025 but three to five years out for a personal site.

Aside: Multimodal RAG in one line

VLMs went in two years from “recognize everyday objects” to “understand enterprise-grade multimodal documents”. ColPali pioneered the “skip OCR, generate tensor embeddings directly from images” route, and paired with Tensor Reranker delivers end-to-end multimodal RAG. The ColPali paper recommends dropping OCR, but that’s compared to Gen-1 CNN OCR — versus Gen-2 Encoder-Decoder OCR, both routes fit different scenarios and will run in parallel for a while.

The five lessons, ranked for a personal site

Reranked from engineer-news’s perspective:

PriorityLessonWhy
★★★Reranker (start with Cross-Encoder, skip Tensor)Workers AI already ships bge-reranker-base; an evening’s work, immediate impact
★★★Contextual Retrieval (chunk context-augmentation)Offline at sync time, one-shot investment, directly closes the most basic semantic gap
★★BM25 hybrid (via SQLite FTS5)Essential rescue for proper nouns, moderate effort
★Document IntelligenceOnly ingest Markdown right now; N/A
✕GraphRAG / Agentic RAG / Tensor RerankerCorpus size and query complexity haven’t crossed the threshold

Principle: do the “one-time change with permanent gain” work first (Chunking, Reranker); then the “keeps costing to maintain” work (Hybrid Search needs FTS5 index upkeep); finally the “architectural change” work (GraphRAG).

Closing: RAG is a whole-stack collaboration

The most valuable judgment from InfiniFlow’s year-in-review:

RAG is not a simple application. It’s a complex system centered on search, orchestrating diverse data, foundational components, and models large and small to work together.

The previous post’s System 1 → System 2 thesis is about the paradigm shift — where imagination’s boundary lies. This post’s five lessons are the foundation — before you can walk toward that boundary, every brick under your feet has to be laid straight.

RAG is like the database of old — the external interface is trivial, the internals are absurd. Real RAG quality isn’t decided by which embedding model you picked, but by whether these five lessons have each been done well. engineer-news’s next step is filling in this five-lesson checklist, one box at a time.

References

Ask this article

Answers come from this article only. Click any prompt below or open the chat at the bottom right.

🇺🇸 English

The last piece we did was a big zoom-out — the five-stage panorama of how RAG has evolved over two years. This one goes the other direction. We're zooming all the way in, right down to the engineering decisions you actually have to make if you want to build a working RAG system. Most of the material comes from InfiniFlow — the team behind RAGFlow — and their late-2024 year-in-review. I've boiled it down into five infrastructure lessons, and I'm going to check each one against my own site, engineer-news, as we go.

Quick recap: last time I concluded that this site sits right at the edge of what's called Naive RAG. Today we're going to unpack that sentence into a very concrete checklist.

Lesson one — document ingestion quality. Or as the field calls it, Document Intelligence.

Here's the golden rule. Whether you're feeding pure text or a PDF stuffed with tables, formulas and flowcharts to a language model, input quality caps output quality. Garbage in, garbage out — in RAG, that becomes quality in, quality out.

And the awkward truth is, most enterprise data is not plain text. It's PDFs, PowerPoints, Word docs, magazines with mixed text and images. Early LLMOps stacks — LangChain plus a vector database plus a plain-text chunker — only handled clean text, which basically caps how good your RAG can get.

There are two generations of solution here. First generation is the traditional computer-vision approach — things like PaddleOCR, RAGFlow's DeepDoc, MinerU, Docling. CPU-friendly, cheap, but they don't generalize well. Every new document type needs its own model. Second generation is Encoder-Decoder Transformers — Meta's Nougat, GOT-OCR 2.0, and friends. One generative model handles many document types, generalizes much better, but you need a GPU. And the architecture looks a lot like a vision-language model, so in 2025 I expect these to converge into unified multimodal document parsers.

For my site? Engineer-news only takes Markdown — either hand-written or AI-normalized — so I bypass OCR entirely. This lesson is basically not applicable. The flip side, though, is that I can't ingest PDF papers, slides, or whiteboard photos. If I ever want to, this whole layer comes back into play. For now, priority: very low.

Lesson two — chunking. Moving from length-splitting to context-augmented chunking.

Naive chunking is just "split by character count." That's exactly what my site does today. My sync script merges paragraphs together until they hit about a thousand characters, then cuts. That's it. Simple, but it has a structural pain point — the chunks lack the article's overall context.

Here's what I mean. Imagine an article about "solving a Cloudflare D1 batch timeout," and one paragraph says "just add this setting in wrangler.jsonc." Pull that paragraph out on its own and you have no idea which D1, which timeout, or which setting the author is talking about. A query for "D1 batch timeout" will struggle to hit that chunk, because the chunk itself doesn't contain those words.

2024 brought a wave of fixes. Jina's Late Chunking encodes the whole document with the embedding model before deciding boundaries, so every token gets to "see" context. There's dsRAG, which uses an LLM to write an auto-context for every chunk. Anthropic's Contextual Retrieval is the same idea — an LLM writes a short context blurb, you glue it onto the original text, then embed the combined thing. That last one is direct, effective, and has basically become the de-facto standard for context-augmented chunking since late 2024. There are also approaches like Meta-Chunking and Mix-of-Granularity that use LLMs to pick chunk boundaries by logical coherence.

The bottom line: fiddling with chunk size has plateaued. The real gain is tagging chunks with context. Only LLMs can do that well, and the cost is fine, because it's a one-time offline job at sync time.

For my site, this is a clear gap. The most direct fix is Contextual Retrieval — at sync time, call a small model like llama-3.1-8b once per chunk, generate a fifty-to-a-hundred-character context blurb, glue it onto the chunk, then embed. My sync cost roughly doubles, but it's offline, and I only pay it once.

Lesson three — hybrid search. And by hybrid, I don't mean two lanes. I mean three.

IBM Research put out a paper called BlendedRAG in 2024 that proved something important. Vector plus sparse vector plus BM25 — three lanes together — is the recall ceiling. It beats any single lane. It beats any two-lane combo.

Why do you need all three? Vector search handles semantic recall — great when the wording is different but the meaning is the same. But it's bad at precise matches. Search for "our company's March 2024 financial plan" and it'll happily return the February one too. Sparse vectors like SPLADE are basically standardized keyword expansion — a pretrained model outputs a fixed sparse representation. Good on general queries, weak on domain-specific terms the pretraining never saw, like model numbers or internal codes. And BM25 — a thirty-year-old algorithm — is still the most direct answer for exact keyword matching. Sparse vectors can't replace it. Three lanes, each specialized. Nothing is redundant.

Now the hard engineering question isn't "do you support BM25," it's "do you support proper BM25." Proper means phrase queries with position information in the inverted index. Dynamic pruning so OR queries don't explode. And for Chinese, real segmentation — bigram tokenization, term weights, stop-word filtering. Elasticsearch is the gold standard here, which is why RAGFlow picked ES as its only backend from day one. It's also a big part of why OpenAI acquired Rockset in June 2024 — Rockset is cloud-native and offers close-to-ES full-text search. Plenty of pure vector databases claim BM25 support now, but very few actually deliver phrase queries plus dynamic pruning plus CJK tokenization.

For my site — I'm on D1, which is SQLite under the hood. And SQLite ships a really useful built-in full-text index called FTS5 that supports tokenizers, phrase queries, and rank functions. So to add BM25, I create a virtual table at sync time indexing title, TL;DR, and content. Add a BM25 lane to my search endpoint. Then merge the vector results and the BM25 results using Reciprocal Rank Fusion. It's more work than adding a reranker, but this is the watershed between "has hybrid search" and "doesn't." Sparse vectors can wait, because Cloudflare's stack doesn't ship a SPLADE service anyway.

Lesson four — reranking. And specifically, the rise of the tensor reranker.

Rerankers evolved fast this year, in three generations. First, Cross-Encoders like BGE-Reranker — you concatenate the query and the document into BERT and let the tokens interact. High quality, moderate cost, this is today's mainstream. Second, LLM-based rerankers, like gte-Qwen2-7B — a seven-billion-parameter LLM scores directly. Better quality but roughly doubled inference cost. Third, late interaction, or tensor rerankers — the ColBERT family. You store per-token embeddings at index time — one tensor per document — and at query time, you just sum up pairwise similarities between query tokens and document tokens.

Tensor rerankers have a huge engineering advantage — they can live inside the database. Query-time inference is just dot products. So your coarse-ranking result set doesn't have to be capped tight at five or ten items — it can expand to hundreds, even thousands. Which means even if the coarse ranking was sloppy, the reranker can rescue it. Vespa was the first database to engineer real tensor support; RAGFlow's own Infinity added it mid-2024. And on the model side, ColBERT, ColBERT v2, JaColBERT for Japanese, jina-colbert-v2 for multilingual — all shipping fast.

For my site — Cloudflare Workers AI currently ships bge-reranker-base, which is a Cross-Encoder. Tensor reranker, no native support. So short-term, I add the Cross-Encoder — that alone is a huge upgrade. Long-term, either Cloudflare ships the ColBERT family, or I self-host. Given I only have 150 articles, tensor reranker ROI is low, so priority: low.

Lesson five — GraphRAG, and the semantic gap it's trying to close.

The previous post already walked through the whole spectrum — Microsoft GraphRAG, LightRAG, LazyGraphRAG, HippoRAG, KAG. Here I want to add three engineering observations.

First, RAPTOR is really a pre-GraphRAG transition. You cluster the text, have an LLM summarize each cluster, then feed both the summaries and the originals into the index. It's already tackling cross-chunk macroscopic questions, just without an explicit graph. RAGFlow adopted RAPTOR mid-year as a GraphRAG stand-in.

Second, SiReRAG proposes a nice two-axis framing. Text has two dimensions — similarity, meaning semantic distance, and relatedness, meaning entity or relation association. RAPTOR sits on the similarity axis, GraphRAG sits on the relatedness axis, SiReRAG merges them. It's a clean way to think about it, and most Graph-RAG variants fit somewhere on that coordinate system.

Third, and this is the one I love — HybridRAG's schema insight. A fully featured database doesn't actually need a graph database to do GraphRAG. Edges, entities, community summaries — they're all text. One table, full-text index plus vector index, and a column called "type" to distinguish what kind of row it is. Done. This is exactly why RAGFlow stuck with Elasticsearch and Infinity instead of bolting on Neo4j. Which is a useful reminder for my site — if I ever want GraphRAG on D1, one table is enough.

For my site — not yet. GraphRAG shines when you have strong cross-document association and need global understanding. A hundred and fifty personal tech articles don't clear that bar. Once the corpus hits thousands, or readers start asking "what does this site think about topic X overall," then the investment starts to make sense.

Two quick asides before I wrap up. On Agentic and Memory — Mem0 got a big star bump just by defining a Memory API. That's evidence memory as a primitive is genuinely in demand. But the memory infrastructure itself is already mature — real-time filter plus search. The scarce piece is combining memory with reasoning. That's the hot space for 2025, but it's three to five years away for a personal site. On multimodal RAG — vision-language models went from "recognize a cat" to "understand enterprise multimodal documents" in two years. ColPali pioneered skipping OCR entirely and generating tensor embeddings directly from images. Paired with a tensor reranker, you get end-to-end multimodal RAG. The paper recommends dropping OCR, but that's compared to first-generation OCR — versus second-generation Encoder-Decoder OCR, both routes fit different scenarios and will coexist for a while.

Now — my ranked priority list for the site. Top of the list, three stars each — Reranker, starting with the Cross-Encoder, skipping tensor for now. Workers AI already ships bge-reranker-base; it's an evening's work for immediate impact. Also three stars — Contextual Retrieval for chunking, because it's offline at sync time, one-shot investment, and it directly closes the most basic semantic gap. Two stars — BM25 hybrid search via SQLite's FTS5. Essential for proper nouns, moderate effort. One star — Document Intelligence, since I only ingest Markdown right now. Crossed out — GraphRAG, Agentic RAG, Tensor Reranker. The corpus and query complexity just haven't crossed the threshold yet.

The principle underneath that ordering — do the "one-time change, permanent gain" work first, like chunking and reranker. Then the "keeps costing to maintain" work, like hybrid search, which needs FTS5 index upkeep. And finally the "architectural change" work like GraphRAG.

So, three takeaways to walk away with.

One — RAG quality isn't decided by which embedding model you picked. It's decided by whether these five lessons — ingestion, chunking, hybrid search, reranking, and knowledge graphs — have each been done well. It's a whole-stack collaboration.

Two — order matters. Do the offline, one-shot changes first. Contextual Retrieval at sync time, plus a Cross-Encoder reranker. Those two together are probably the single biggest upgrade any Naive RAG can make in a weekend.

And three — architecture doesn't have to grow to match ambition. HybridRAG showed that GraphRAG doesn't need a graph database. FTS5 shows BM25 doesn't need Elasticsearch. Before you add a new system, check if the one you have can carry the extra weight. Very often, it can. RAG is like the databases of old — the interface is trivial, the internals are absurd. The next step for engineer-news is filling in this checklist, one box at a time.

🇹🇼 中文

上一集我們用五階段全景，把 RAG 這兩年的演進看了一輪，那是拉高鏡頭看趨勢。這一集要拉近，看一件更具體的事——如果你今天真的要蓋一個 RAG 系統，你會面對哪些工程決策。

素材主要來自 InfiniFlow，也就是 RAGFlow 團隊 2024 年底那篇年度總結。我把它整理成五個 infra 功課，每一個都對照 engineer-news 這個站的現況，看哪些欠、哪些不欠、該先補哪個。

上一集我說本站落在 Naive RAG 的邊界，這一集就是那句話的展開清單。

先講功課一，文件入口的品質，也就是 Document Intelligence。

RAG 有一個很殘酷的前提——資料入口的品質，決定最終品質。Garbage In, Garbage Out 在 RAG 場景就變成 Quality In, Quality Out。

企業裡大部分的資料，不是純文字，是 PDF、PPT、Word、圖文混排的雜誌。早期 LLMOps 那套 LangChain 加向量庫加純文字 chunker，只能處理純文字，這就把 RAG 的商業價值天花板壓得很低。

過去這類問題叫 Document Intelligence，底下有一堆子任務——布局辨識、表格結構、公式辨識、流程圖辨識，每個都有專用模型。RAG 把它們整合起來，形成廣義 OCR，當作 RAG 的入口。方法上分兩代。第一代是 CNN 加傳統視覺，像 PaddleOCR、DeepDoc、MinerU、Docling，可以跑 CPU、成本低，但泛化差，每個場景都要單獨訓，被戲稱為雕花。第二代是 Encoder-Decoder Transformer，像 Meta 的 Nougat、GOT-OCR 2.0，用生成式模型統一處理，泛化強，但需要 GPU。這條路跟 VLM 架構高度相似，2025 年很可能就收斂成統一的多模態文件解析模型。

對照 engineer-news 呢？這個站只吃 Markdown，作者手寫或 AI 整理過的 Markdown，完全繞過 OCR。所以這個功課對我來說是 N/A，不欠。但反過來說，這也限制了它能吃的資料類型——如果之後想擴展到 PDF 論文、投影片、白板照片，就得補這一層。現階段優先度極低。

功課二，Chunking 從長度切分走向上下文化。

Naive Text Chunking 就是按字數硬切，這是我現在的做法。sync-to-d1 裡的 chunkText 函式，邏輯就是把段落合併到大約一千字上限就切，就這樣。

這種切法有一個結構性痛點——chunk 內部沒有全文語境。舉個例子，一篇文章講 D1 batch timeout 的解法，其中一段寫「在 wrangler.jsonc 加上這個設定即可」。這段 chunk 拿出來單獨看，完全不知道在講什麼 D1、什麼 timeout、什麼設定。查詢的時候要命中這段，非常難。

2024 年 Chunking 這一層有一連串進化。Jina 的 Late Chunking，先對整份文檔編碼，在最後 mean pooling 之前才切邊界，讓 chunk 邊界之前所有 token 都能看到上下文。但它要求 embedding 模型走 mean pooling，我用的 bge-m3 是 CLS pooling，直接搭不上。dsRAG 是 LLM 為每個 chunk 補一段 auto-context。Anthropic 的 Contextual Retrieval，概念類似 dsRAG——讓 LLM 幫每個 chunk 生成一小段專屬的上下文說明，跟原文一起 embed。這個做法效果好、實作直觀，已經是 2024 下半年的事實標準之一。還有 Meta-Chunking、Mix-of-Granularity 這些學術路線。

整體結論很清楚——單純調 chunk 大小的收益已經到頂了。真正有價值的是給 chunk 補上下文標籤，這件事只有 LLM 能做，成本可以接受，因為是 sync 階段一次性投入。

對照本站，這是明確缺口。最直接的補法就是走 Contextual Retrieval——sync 的時候，每個 chunk 呼叫一次 llama-3.1-8b 或更小的模型，生成大約五十到一百字的上下文摘要，跟原文串接後再 embed。sync 成本翻倍，但因為離線只做一次，划算。

功課三，混合搜尋——向量、稀疏向量、BM25 三路。

2024 年 IBM Research 的 BlendedRAG 論證了一件事——向量加稀疏向量加 BM25，三路混合就是召回品質的上限，比任何單路或兩路都好。

為什麼一定要三路？向量負責語意召回，擅長「意思相近但用詞不同」，但天生沒辦法處理精確查詢，比如「2024 年 3 月我們公司財務計畫」，很可能召回其他時間段的內容。稀疏向量像 SPLADE，是預訓練模型輸出的固定維度稀疏向量，可以看成標準化的關鍵字擴展，通用查詢不錯，但遇到 domain-specific 詞彙、型號、代號、內部術語就會漏，因為那些詞不在預訓練詞彙裡。BM25 是三十年前的老演算法，對精確關鍵字匹配最直接，稀疏向量取代不了它。三路各有所長，不是誰替代誰。

工程上真正難的不是「支援 BM25」，而是「支援合格的 BM25」——短語查詢需要倒排索引存位置、動態剪枝避免 OR 查詢爆炸、中文分詞、詞權重、停用詞。Elasticsearch 是黃金標準，RAGFlow 一開始就選 ES 當唯一後端就是這個理由。純向量資料庫像 Milvus、Qdrant 宣稱支援 BM25 的越來越多，但真正做到位的很少。

對照本站，我用的 D1 是 SQLite，SQLite 有一個非常好用的內建全文索引——FTS5，支援 tokenizer、短語查詢、rank function。要補 BM25 這一路，步驟是——sync 時多建一張 posts_fts 虛擬表，索引 title、tldr、content；搜尋 API 再加一路 FTS5 查詢；最後用 Reciprocal Rank Fusion 把向量路和 BM25 路合起來。工作量比 Reranker 大，但這是「有沒有 hybrid search」的分水嶺。稀疏向量那一路暫時可以先不做，Cloudflare 生態沒有現成的 SPLADE 服務。

功課四，Tensor Reranker，也就是延遲交互模型。

Reranker 這一年也在快速演化，分成三代。Cross-Encoder，像 BGE-Reranker，把 query 和 doc 拼起來丟進 BERT，捕捉 token 交互，品質高、成本適中，目前主流。LLM-based Reranker，直接用 7B 級 LLM 打分，品質更好，成本翻倍。第三代是 Late Interaction，也叫 Tensor Reranker，ColBERT 系列——索引階段就把每個 token 的 embedding 存下來，一份文件用一個 tensor 表示，查詢時只算 query token 和 doc token 兩兩相似度再累加。

Tensor Reranker 有一個工程優勢——它可以放進資料庫層做。查詢階段沒有 LLM 推理，只有 tensor 內積，很快。這意味著粗排結果不用嚴格控制到五到十個，可以擴大到幾百上千個做 rerank，補救粗排品質不佳。Vespa 是最早工程化 tensor 的資料庫，Infinity 也在 2024 年中補上。

對照本站，Cloudflare Workers AI 目前有 bge-reranker-base，是 Cross-Encoder，Tensor Reranker 這一層沒有原生支援。所以我的實務路徑是——短期先加 Cross-Encoder，這已經是巨大進步；長期等 Cloudflare 推出 ColBERT 系列，或自架 embedding service。以現在 150 篇文章的規模，Tensor Reranker 投入產出比不高，優先度低。

功課五，GraphRAG 與語意鴻溝。

上一集已經把 GraphRAG 的光譜講過，這裡只補三個工程觀察。第一，RAPTOR 是 GraphRAG 之前的過渡——先做文本聚類、LLM 為每個聚類生成摘要，摘要跟原文一起餵給搜尋系統，它已經在解決「跨 chunk 的宏觀提問」問題，只是沒有明確圖結構。第二，SiReRAG 提出雙軸召回，文本之間有兩個維度——相似度和相關性；RAPTOR 走相似度側，GraphRAG 走相關性側，SiReRAG 合起來。這個切分很清晰，之後的 Graph-RAG 變種都可以放到這個座標系裡看。第三，HybridRAG 有個 schema 洞見——一個功能完備的資料庫不需要圖資料庫來實作 GraphRAG，邊、實體、社群摘要，這些都是文字，只要一張具備全文索引加向量索引的表就能承載，多加一個 type 欄位區分類型即可。這也是為什麼 RAGFlow 繼續用 Elasticsearch，而不加 Neo4j。

對本站來說這是重要提示——如果哪天真的要做 GraphRAG，D1 一張表就夠了。但這階段暫時不做。GraphRAG 適用的是「跨文件關聯強、需要全域理解」，對 150 篇個人技術文不成立。要等資料量上千、或者讀者開始問「這個站對某某主題整體是什麼觀點」，才值得投入。

兩個番外快速講一下。Mem0 只定義了一組 memory 管理 API 就爆紅，說明 memory 這個 primitive 需求很強，但 Memory infra 本身已經很成熟——真正稀缺的是怎麼把 Memory 跟 Reasoning 結合，這是 2025 年的熱區，但對個人站來說是三五年後的問題。多模態這邊，VLM 兩年內從識別日常用品進化到理解企業級多模態文檔，ColPali 開創了跳過 OCR、直接對圖片生成 tensor embedding 的路線，但那是跟第一代 CNN OCR 比的，跟第二代 Encoder-Decoder OCR 比，兩條路會並行很久。

最後把五個功課排一下優先順序。

最高優先，Reranker——先上 Cross-Encoder，不用等 Tensor。Workers AI 已經有 bge-reranker-base，一個晚上能加，效果立即可見。並列最高，Contextual Retrieval——sync 階段離線做，一次性投入，直接補「語意鴻溝」最基礎的一層。第二層，BM25 三路混合，走 SQLite FTS5——專有名詞救援必備，工作量中等。第三層，Document Intelligence——目前只吃 Markdown，暫時 N/A。最後，GraphRAG、Agentic RAG、Tensor Reranker，資料量和需求都沒到。

排序背後的原則是——先做「單次修改能永久改善」的功課，比如 Chunking 和 Reranker；再做「需要持續投入才有回報」的功課，比如 Hybrid Search 要維護 FTS5 索引；最後才是「需要架構級改動」的功課，比如 GraphRAG。

收個尾。給你帶走三件事。

第一，RAG 的品質天花板不在你選了哪個 embedding 模型，而在文件入口、Chunking、混合搜尋、Reranker、GraphRAG 這五塊地基有沒有分別做好。第二，對一個 150 篇規模的個人站，最划算的兩件事是——上 Cross-Encoder Reranker，和幫每個 chunk 補上下文摘要。這兩件都能離線做、一次投入。第三，InfiniFlow 那句話值得反覆咀嚼——RAG 不是一個簡單的應用，它是以搜尋為中心，把各類資料、基礎組件、大小模型協同起來的複雜系統。想像力再遠，都要從腳下這幾塊磚開始鋪。

Tags

#ai#llm#rag#chunking#reranker#hybrid-search#cloudflare#system-design

Related Articles

tech

RAG's Five Stages: From Pipeline to Reasoning Retrieval, and the Naive RAG on My Own Site

Over the past two years RAG evolved from a 'linear pipeline' to 'loop-based reasoning'. It maps cleanly to five stages: Naive, Advanced, Modular, Graph, Agentic. The real inflection point is control moving from pipeline to agent — a System 1 → System 2 shift. Looking back at engineer-news's own RAG stack, it's stuck at the Naive edge — so this post also lays out what to fix next.

#ai#llm#rag#graph-rag#agentic-rag#system-design#cloudflare